Minio 高可用部署
· 阅读需 12 分钟
Keepalived+Nginx 高可用搭建
安装主备 nginx
基础配置
# 1、设置时区
timedatectl set-timezone Asia/Shanghai
yum -y install ntpdate ntp
ntpdate cn.ntp.org.cn
systemctl restart ntpd && systemctl enable ntpd
# 2、关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
# 3、关闭selinux
sed -i "/^SELINUX/s/enforcing/disabled/" /etc/selinux/config
setenforce 0
安装 pcre
# 1、安装依赖
yum -y install make zlib zlib-devel gcc-c++ libtool openssl openssl-devel
# 2、安装pcre
wget http://downloads.sourceforge.net/project/pcre/pcre/8.35/pcre-8.35.tar.gz
tar zxvf pcre-8.35.tar.gz
mv pcre-8.35 /usr/local/
cd /usr/local/pcre-8.35
./configure
make && make install
pcre-config --version
安装 nginx
# 1、安装之前，最好检查一下是否已经安装有nginx
find -name nginx
# 2、如果系统已经安装了nginx，那么就先卸载
yum remove nginx
# 3、从官网下载最新版的nginx
wget http://nginx.org/download/nginx-1.9.5.tar.gz
tar -zxvf nginx-1.9.5.tar.gz
mv nginx-1.9.5 /usr/local/
cd /usr/local/nginx-1.9.5
# 4、使用--prefix参数指定nginx安装的目录,make、make install安装
./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-pcre=/usr/local/pcre-8.35
make && make install
# 5、启动
cd /usr/local/nginx/sbin
./nginx // 启动
加入开机自启动
cd /lib/systemd/system/
vi nginx.service
[Unit]
Description=nginx service
After=network.target
[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s quit
PrivateTmp=true
[Install]
WantedBy=multi-user.target
主备分别安装 Keepalived
yum 安装
yum -y install keepalived
systemctl enable keepalived # 加入开机启动
# 安装成功后keepalived配置文件地址在：/etc/keepalived/keepalived.conf
配置Nginx检查脚本
vim /usr/local/keeplived/check_list
#!/bin/sh
nginxpid=$(ps -C nginx --no-header|wc -l)
#1.判断Nginx是否 存活,如果不存活则尝试启动Nginx
if [ $nginxpid -eq 0 ];then
systemctl start nginx
sleep 3
#2.等待3秒后再次获取一次Nginx状态
nginxpid=$(ps -C nginx --no-header|wc -l)
#3.再次进行判断, 如Nginx还不存活则停止Keepalived,让地址进行漂移,并退出脚本
if [ $nginxpid -eq 0 ];then
systemctl stop keepalived
fi
fi
keepalived 配置
# 在两台机器上安装keeplived并分别配置。master配置(/etc/keepalived/keepalived.conf)：
global_defs {
router_id minio1 #hostname
}
vrrp_script chk_nginx {
script "/usr/local/keeplived/check_list" #检测nginx的脚本
interval 2 #每2秒检测一次
weight -20 #如果某一个nginx宕机 则权重减20
}
vrrp_instance VI_1 {
state MASTER #状态 MASTER BACKUP
interface ens33 #绑定的网卡
virtual_router_id 51 #虚拟路由的ID号,两个节点设置必须一样
mcast_src_ip 192.168.2.245 #本机的IP
priority 100 # 优先级，谁大谁是Master
advert_int 1
# 设置验证信息，两个节点必须一致
authentication {
auth_type PASS
auth_pass 10086
}
# 虚拟IP，两个节点设置必须一样。
virtual_ipaddress {
192.168.2.250 # VIP
}
# nginx存活状态检测脚本
track_script {
chk_nginx
}
}
backup配置(/etc/keepalived/keepalived.conf)：
global_defs {
router_id minio2 #hostname
}
vrrp_script chk_nginx {
script "/usr/local/keeplived/check_list" #检测nginx的脚本
interval 2 #每2秒检测一次
weight -20 #如果某一个nginx宕机 则权重减20
}
vrrp_instance VI_1 {
state BACKUP #状态 MASTER BACKUP
interface ens33 #绑定的网卡
virtual_router_id 51
mcast_src_ip 192.168.2.246
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 10086
}
virtual_ipaddress {
192.168.2.250 # VIP
}
track_script {
chk_nginx
}
}
测试 Keepalived
# 两台服务器启动 keepalived
systemctl restart keepalived
ip a
# 关掉 mino1 查看 minio2 的 vip 是否转移到此机器上
搭建分布式 Minio
格式化文件系统
分布式Minio可以让你将多块硬盘（甚至在不同的机器上）组成一个对象存储服务。由于硬盘分布在不同的节点上，分布式Minio避免了单点故障。分布式Minio的好处如下： 数据保护
分布式Minio采用 纠删码来防范多个节点宕机和位衰减bit rot。分布式Minio至少需要4个硬盘，使用分布式Minio自动引入了纠删码功能。 高可用
单机Minio服务存在单点故障，相反，如果是一个有N块硬盘的分布式Minio,只要有N/2硬盘在线，你的数据就是安全的。不过你需要至少有N/2+1个硬盘来创建新的对象。 例如，一个16节点的Minio集群，每个节点16块硬盘，就算8台服務器宕机，这个集群仍然是可读的，不过你需要9台服务器才能写数据。
注意，只要遵守分布式Minio的限制，你可以组合不同的节点和每个节点几块硬盘。比如，你可以使用2个节点，每个节点4块硬盘，也可以使用4个节点，每个节点两块硬盘，诸如此类。 一致性
Minio在分布式和单机模式下，所有读写操作都严格遵守read-after-write一致性模型
查看磁盘情况
将磁盘挂入虚拟机后需要先将磁盘格式化，并建立文件系统（本文将一块磁盘建立一个分区）
fdisk -l
建立磁盘分区
1、步骤一 对磁盘进行分区
# fdisk /dev/sdb （回车）
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x48d97cfd.
Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
g create a new empty GPT partition table
G create an IRIX (SGI) partition table
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)
2、步骤二 列出当前分区使用情况
Command (m for help): p
Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x48d97cfd
Device Boot Start End Blocks Id System
3、步骤三 添加磁盘分区
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
// 起始扇区和Last扇区为区分磁盘分区大小的重要参数，如果需要将整个块作为一个分区使用默认值即可。如果需要精细化定制则计算分区大小。
First sector (2048-41943039, default 2048):
Using default value 2048
// last扇区值可以使用：+扇区 的格式，例如：+2G 表示分配2G的分区
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039):
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set
4、步骤四 保存/退出分区
// 参数w为保持当前分区，q为取消退出分区
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
建立文件系统
1、步骤一 格式化（注意，分区格式化会造成数据丢失，一定要保障分区名称的正确性。）
mkfs.ext4 /dev/sdb1
2、步骤二 mount进行目录挂载
// 将创建的分区挂载到一个新的目录
mkdir /mnt/sdb1
// 将/dev/sdb1分区挂载到目录：/mnt/sdb1
mount /dev/sdb1 /mnt/sdb1
// 使用mount命令查看挂载情况
mount
/dev/sdb1 on /mnt/sdb1 type ext4 (rw,relatime,seclabel,data=ordered)
// 使用df命令查看磁盘使用情况
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 485M 143M 343M 30% /boot
/dev/sdb1 20G 45M 19G 1% /mnt/sdb1
3、步骤三 fstab进行永久挂载
// 通过mount挂载的目录是保存在内存中的，重启之后就不会生效了 ，所以要通过修改：/etc/fstab 配置文件进行永久挂载。
# vi /etc/fstab
/dev/sdb1 /mnt/sdb1 ext4 defaults 0 0
// 说明：配置文件参数格式：[分区名称] [挂载目录名] [文件系统类型] [权限] [磁盘配额] [磁盘配额]
完成对sdb文件块格式化分区后，按照上述流程完成对sbc和另外一台服务器的分区格式化，然后两台服务器分区如下：
ls -l /dev/sd?
ls -l /dev/sd\??
创建Minio实现分布式
完成磁盘格式化分区后就可以创建分布式Minio了，创建之前需要注意以下几点：
- 分布式Minio里所有的节点需要有同样的access秘钥和secret秘钥，这样这些节点才能建立联接。为了实现这个，你需要在执行minio server命令之前，先将access秘钥和secret秘钥export成环境变量。
- 分布式Minio使用的磁盘里必须是干净的，里面没有数据。
- 分布式Minio里的节点时间差不能超过3秒，你可以使用NTP 来保证时间一致。
# 1、分别在两台服务器下载Minio
wget https://dl.min.io/server/minio/release/linux-amd64/minio.RELEASE.2022-10-05T14-58-27Z
# 2、设置权限
chmod 755 minio
mv minio /usr/bin/
# 3、创建目录,编写启动脚本
mkdir /usr/local/minio -p
vim /usr/local/minio/run.sh
#!/bin/bash
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=Test123!
minio server --config-dir /usr/local/minio/conf \
http://192.168.2.245/mnt/sdb1 \
http://192.168.2.246/mnt/sdb1
# 不加端口号默认 9000
# 4、编写启动服务
vi /usr/lib/systemd/system/minio.service
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/
[Service]
WorkingDirectory=/usr/local/minio
ExecStart=/usr/local/minio/run.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
# 5、启动Minio
chmod +x /usr/local/minio/run.sh
systemctl start minio
systemctl status minio
6、加入开启启动
systemctl enable minio
修改Nginx配置文件实现负载均衡
# 修改nginx配置文件
vi /usr/local/nginx/conf/nginx.conf
#user nobody;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
access_log logs/access.log;
sendfile on;
keepalive_timeout 65;
#gzip on;
upstream minioserver {
server 192.168.2.245:9000;
server 192.168.2.246:9000;
}
server {
listen 80;
server_name localhost;
#access_log logs/host.access.log main;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minioserver;
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
# 2、重启nginx
systemctl restart nginx
在两台 Minio 服务器修改配置并重启之后，就可以使用 VIP 进行访问了：
http://192.168.2.250/minio/login
账号为配置的：admin 密码：Test123!
容器快速部署
version: "3.7"
services:
minio:
image: "quay.io/minio/minio:RELEASE.2022-08-02T23-59-16Z"
ports:
- "9000:9000"
- "9001:9001"
volumes:
- "./minio/data1:/data1"
- "./minio/data2:/data2"
- "./minio/data3:/data3"
- "./minio/data4:/data4"
command: server --console-address ":9001" http://192.168.2.231:9000/data{1...4} http://192.168.2.232:9000/data{1...4} http://192.168.2.233:9000/data{1...4} http://192.168.2.234:9000/data{1...4}
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=Test123!
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
部署负载均衡服务nginx
version: '3.7'
services:
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9001:9001"
nginx配置文件
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server 192.168.2.231:9000;
server 192.168.2.232:9000;
server 192.168.2.233:9000;
server 192.168.2.234:9000;
}
upstream console {
ip_hash;
server 192.168.2.231:9001;
server 192.168.2.232:9001;
server 192.168.2.233:9001;
server 192.168.2.234:9001;
}
server {
listen 9000;
listen [::]:9000;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9001;
listen [::]:9001;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
}