Keepalived+Nginx 高可用搭建

安装主备 nginx

基础配置

1
2
3
4
5
6
7
8
9
10
11
12
13
# 1、设置时区
timedatectl set-timezone Asia/Shanghai
yum -y install ntpdate ntp
ntpdate cn.ntp.org.cn
systemctl restart ntpd && systemctl enable ntpd

# 2、关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 3、关闭selinux
sed -i "/^SELINUX/s/enforcing/disabled/" /etc/selinux/config
setenforce 0

安装 pcre

1
2
3
4
5
6
7
8
9
10
11
# 1、安装依赖
yum -y install make zlib zlib-devel gcc-c++ libtool  openssl openssl-devel

# 2、安装pcre
wget http://downloads.sourceforge.net/project/pcre/pcre/8.35/pcre-8.35.tar.gz
tar zxvf pcre-8.35.tar.gz 
mv pcre-8.35 /usr/local/ 
cd /usr/local/pcre-8.35
./configure
make && make install
pcre-config --version

安装 nginx

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 1、安装之前，最好检查一下是否已经安装有nginx
find -name nginx  

# 2、如果系统已经安装了nginx，那么就先卸载
yum remove nginx  

# 3、从官网下载最新版的nginx
wget http://nginx.org/download/nginx-1.9.5.tar.gz  
tar -zxvf nginx-1.9.5.tar.gz  
mv nginx-1.9.5 /usr/local/
cd  /usr/local/nginx-1.9.5 

# 4、使用--prefix参数指定nginx安装的目录,make、make install安装
./configure --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-pcre=/usr/local/pcre-8.35
make && make install    

# 5、启动
cd /usr/local/nginx/sbin
./nginx    // 启动

加入开机自启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
cd /lib/systemd/system/

vi nginx.service
[Unit]
Description=nginx service
After=network.target 

[Service] 
Type=forking 
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s quit
PrivateTmp=true 

[Install] 
WantedBy=multi-user.target

主备分别安装 Keepalived

yum 安装

1
2
3
4
yum -y install keepalived
systemctl enable keepalived   # 加入开机启动

# 安装成功后keepalived配置文件地址在：/etc/keepalived/keepalived.conf

配置Nginx检查脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
vim /usr/local/keeplived/check_list

#!/bin/sh

nginxpid=$(ps -C nginx --no-header|wc -l)
#1.判断Nginx是否存活,如果不存活则尝试启动Nginx
if [ $nginxpid -eq 0 ];then
    systemctl start nginx
    sleep 3
    #2.等待3秒后再次获取一次Nginx状态
    nginxpid=$(ps -C nginx --no-header|wc -l) 
    #3.再次进行判断, 如Nginx还不存活则停止Keepalived,让地址进行漂移,并退出脚本  
    if [ $nginxpid -eq 0 ];then
        systemctl stop keepalived
   fi
fi

keepalived 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 在两台机器上安装keeplived并分别配置。master配置(/etc/keepalived/keepalived.conf)：

global_defs {
   router_id minio1 #hostname 
}
vrrp_script chk_nginx {
    script "/usr/local/keeplived/check_list" #检测nginx的脚本
    interval 2 #每2秒检测一次
    weight -20 #如果某一个nginx宕机 则权重减20
}
vrrp_instance VI_1 {
    state MASTER #状态 MASTER BACKUP
    interface ens33 #绑定的网卡
    virtual_router_id 51 #虚拟路由的ID号,两个节点设置必须一样
    mcast_src_ip 192.168.2.245 #本机的IP
    priority 100  # 优先级，谁大谁是Master
    advert_int 1
# 设置验证信息，两个节点必须一致
    authentication {
        auth_type PASS
        auth_pass 10086
    }
 # 虚拟IP，两个节点设置必须一样。
    virtual_ipaddress {
        192.168.2.250  # VIP
    }
# nginx存活状态检测脚本
    track_script {
       chk_nginx
    }
}

backup配置(/etc/keepalived/keepalived.conf)：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
global_defs {
   router_id minio2  #hostname 
}
vrrp_script chk_nginx {
    script "/usr/local/keeplived/check_list" #检测nginx的脚本
    interval 2 #每2秒检测一次
    weight -20 #如果某一个nginx宕机 则权重减20
}
vrrp_instance VI_1 {
    state BACKUP #状态 MASTER BACKUP
    interface ens33 #绑定的网卡
    virtual_router_id 51
    mcast_src_ip 192.168.2.246
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 10086
    }
    virtual_ipaddress {
        192.168.2.250  # VIP
    }

    track_script {
       chk_nginx
    }
}

测试 Keepalived

1
2
3
4
5
6
# 两台服务器启动 keepalived
systemctl restart keepalived

ip a 

# 关掉 mino1 查看 minio2 的 vip 是否转移到此机器上

搭建分布式 Minio

格式化文件系统

　　分布式Minio可以让你将多块硬盘（甚至在不同的机器上）组成一个对象存储服务。由于硬盘分布在不同的节点上，分布式Minio避免了单点故障。分布式Minio的好处如下：
数据保护

　　分布式Minio采用 纠删码来防范多个节点宕机和位衰减bit rot。分布式Minio至少需要4个硬盘，使用分布式Minio自动引入了纠删码功能。
高可用

　　单机Minio服务存在单点故障，相反，如果是一个有N块硬盘的分布式Minio,只要有N/2硬盘在线，你的数据就是安全的。不过你需要至少有N/2+1个硬盘来创建新的对象。
例如，一个16节点的Minio集群，每个节点16块硬盘，就算8台服務器宕机，这个集群仍然是可读的，不过你需要9台服务器才能写数据。

　　注意，只要遵守分布式Minio的限制，你可以组合不同的节点和每个节点几块硬盘。比如，你可以使用2个节点，每个节点4块硬盘，也可以使用4个节点，每个节点两块硬盘，诸如此类。
一致性

　　Minio在分布式和单机模式下，所有读写操作都严格遵守read-after-write一致性模型

查看磁盘情况

将磁盘挂入虚拟机后需要先将磁盘格式化，并建立文件系统（本文将一块磁盘建立一个分区）

1
fdisk -l

建立磁盘分区

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
1、步骤一  对磁盘进行分区
# fdisk /dev/sdb  （回车）
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x48d97cfd.

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   g   create a new empty GPT partition table
   G   create an IRIX (SGI) partition table
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

  2、步骤二  列出当前分区使用情况
Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x48d97cfd

   Device Boot      Start         End      Blocks   Id  System

3、步骤三  添加磁盘分区
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1 
// 起始扇区和Last扇区为区分磁盘分区大小的重要参数，如果需要将整个块作为一个分区使用默认值即可。如果需要精细化定制则计算分区大小。
First sector (2048-41943039, default 2048): 
Using default value 2048
// last扇区值可以使用：+扇区 的格式，例如：+2G 表示分配2G的分区
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): 
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set

4、步骤四   保存/退出分区
// 参数w为保持当前分区，q为取消退出分区
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

建立文件系统

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
1、步骤一  格式化（注意，分区格式化会造成数据丢失，一定要保障分区名称的正确性。）
mkfs.ext4 /dev/sdb1

2、步骤二  mount进行目录挂载
// 将创建的分区挂载到一个新的目录
mkdir /mnt/sdb1
// 将/dev/sdb1分区挂载到目录：/mnt/sdb1
mount /dev/sdb1 /mnt/sdb1
// 使用mount命令查看挂载情况
mount
/dev/sdb1 on /mnt/sdb1 type ext4 (rw,relatime,seclabel,data=ordered)
// 使用df命令查看磁盘使用情况
df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda1                485M  143M  343M  30% /boot
/dev/sdb1                 20G   45M   19G   1% /mnt/sdb1

3、步骤三  fstab进行永久挂载
// 通过mount挂载的目录是保存在内存中的，重启之后就不会生效了，所以要通过修改：/etc/fstab 配置文件进行永久挂载。
# vi  /etc/fstab
/dev/sdb1  /mnt/sdb1  ext4  defaults  0  0
// 说明：配置文件参数格式：[分区名称] [挂载目录名] [文件系统类型] [权限] [磁盘配额] [磁盘配额]

完成对sdb文件块格式化分区后，按照上述流程完成对sbc和另外一台服务器的分区格式化，然后两台服务器分区如下：

1
2
ls -l /dev/sd?
ls -l /dev/sd??

创建Minio实现分布式

　完成磁盘格式化分区后就可以创建分布式Minio了，创建之前需要注意以下几点：

  1. 分布式Minio里所有的节点需要有同样的access秘钥和secret秘钥，这样这些节点才能建立联接。为了实现这个，你需要在执行minio server命令之前，先将access秘钥和secret秘钥export成环境变量。
  2. 分布式Minio使用的磁盘里必须是干净的，里面没有数据。
  3. 分布式Minio里的节点时间差不能超过3秒，你可以使用NTP 来保证时间一致。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 1、分别在两台服务器下载Minio
wget https://dl.min.io/server/minio/release/linux-amd64/minio.RELEASE.2022-10-05T14-58-27Z

# 2、设置权限
chmod 755 minio
mv minio /usr/bin/

# 3、创建目录,编写启动脚本
mkdir /usr/local/minio -p
vim /usr/local/minio/run.sh
#!/bin/bash
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=Test123!

minio server --config-dir /usr/local/minio/conf \
http://192.168.2.245/mnt/sdb1 \
http://192.168.2.246/mnt/sdb1 
# 不加端口号默认 9000

# 4、编写启动服务
vi /usr/lib/systemd/system/minio.service
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/

[Service]
WorkingDirectory=/usr/local/minio
ExecStart=/usr/local/minio/run.sh

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

# 5、启动Minio
chmod +x /usr/local/minio/run.sh
systemctl start minio
systemctl status minio

6、加入开启启动
systemctl enable minio

修改Nginx配置文件实现负载均衡

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# 修改nginx配置文件
vi /usr/local/nginx/conf/nginx.conf

#user  nobody;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    access_log  logs/access.log;
    sendfile        on;
    keepalive_timeout  65;
    #gzip  on;

    upstream minioserver {
        server 192.168.2.245:9000;
        server 192.168.2.246:9000;
    }

    server {
        listen       80;
        server_name  localhost;
        #access_log  logs/host.access.log  main;
        location / {
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Host $http_host;
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
            proxy_pass http://minioserver;
            root   html;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

# 2、重启nginx
systemctl restart nginx

在两台 Minio 服务器修改配置并重启之后，就可以使用 VIP 进行访问了：

http://192.168.2.250/minio/login

账号为配置的：admin 密码：Test123!

容器快速部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: "3.7"
services:
  minio:
    image: "quay.io/minio/minio:RELEASE.2022-08-02T23-59-16Z"
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - "./minio/data1:/data1"
      - "./minio/data2:/data2"
      - "./minio/data3:/data3"
      - "./minio/data4:/data4"
    command: server --console-address ":9001" http://192.168.2.231:9000/data{1...4} http://192.168.2.232:9000/data{1...4} http://192.168.2.233:9000/data{1...4} http://192.168.2.234:9000/data{1...4}
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=Test123!
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

部署负载均衡服务nginx

1
2
3
4
5
6
7
8
9
10
11
version: '3.7'
services:
  nginx:
    image: nginx:1.19.2-alpine
    hostname: nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
      - "9001:9001"

nginx配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  65;

    # include /etc/nginx/conf.d/*.conf;

    upstream minio {
        server 192.168.2.231:9000;
        server 192.168.2.232:9000;
        server 192.168.2.233:9000;
        server 192.168.2.234:9000;
    }
 
   upstream console {
        ip_hash;
        server 192.168.2.231:9001;
        server 192.168.2.232:9001;
        server 192.168.2.233:9001;
        server 192.168.2.234:9001;
    }

    server {
        listen       9000;
        listen  [::]:9000;
        server_name  localhost;

        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
            proxy_pass http://minio;
        }
    }

    server {
        listen       9001;
        listen  [::]:9001;
        server_name  localhost;
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
        proxy_request_buffering off;
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;
            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;
            proxy_connect_timeout 300;
            
            # To support websocket
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            
            chunked_transfer_encoding off;
            proxy_pass http://console;
        }
    }
}