概述

可以通过 rest api 添加或删除后端服务器，实现 nginx 后端服务器的动态更新

背景:

服务实例的 ip 会频繁更换, 需要想办法让发布后的实例 ip 自动更新到 nginx 的配置中去,并且能够让其自动生效nginx reload 会重新开一组进程来接收请求，让老进程完事后再退出. 切记如果你线上流量较大，不要频繁地 reload,因为 nginx 新老配置交替的时候会掉 15%左右的 qps; nginx reload 也会造成 nginx 与 client 和 backend 的长链接关闭组件版本nginx-1.20.1etcd-3.4.16nginx-upsync-module_v2.1.3 当前仅支持 etcd V2 API, 从 consul 或 etcd 同步后端服务器配置，动态修改后端服务器的属性, 不需要重新加载 nginxnginx_upstream_check_module: 主动检查后端服务器是否健康Consul

1、二进制方式部署

# 下载预编译二进制安装wget  consul_1.8.15_linux_amd64.zipmv consul /usr/local/bin/

2、创建运行环境及启动脚本

# 创建运行用户及相关目录权限useradd -M -s /sbin/nologin consul && mkdir /var/lib/consul/ && chown -R consul.consul /var/lib/consul/mkdir /var/run/consul && chown -R consul.consul /var/run/consul/mkdir /etc/consul# 启动脚本cat /etc/systemd/system/consul.service[Unit]Description=Consul service discovery agentRequires=network-online.targetAfter=network-online.target[Service]User=consulGroup=consulPIDFile=/var/run/consul/consul.pidEnvironmentFile=-/etc/default/consulEnvironment=GOMAXPROCS=2Restart=on-failureExecStart=/usr/local/bin/consul agent -dev -bind=192.168.31.69 -client 0.0.0.0 -config-dir /etc/consul/ -data-dir=/var/lib/consul -pid-file=/var/run/consul/consul.pidExecReload=/usr/local/bin/consul reloadKillSignal=SIGTERMTimeoutStopSec=5[Install]WantedBy=multi-user.target

3、启动 consul

systemctl start consul# consul web 界面:

1、下载解压安装

wget  xzf etcd-v3.4.16-linux-amd64.tar.gzmv etcd-v3.4.16-linux-amd64 /srv/etcd && cd /srv/etcd

2、设置工作及配置目录

mkdir -p /var/lib/etcd/ && mkdir -p /srv/etcd/config/cat <<EOF | sudo tee /srv/etcd/config/etcd.conf#节点名称ETCD_NAME=$(hostname -s)# 数据存放位置ETCD_DATA_DIR=/var/lib/etcdETCD_LISTEN_PEER_URLS=

3、设置 systemd 启动脚本

; 添加运行用户、文件目录useradd -M -r -s /sbin/nologin etcdmkdir /var/lib/etcd/ && chown etcd.etcd /var/lib/etcd/ -R; 启动脚本cat <<EOF | sudo tee /usr/lib/systemd/system/etcd.service[Unit]Description=Etcd ServerDocumentation=[Service]Type=notifyUser=etcdGroup=etcdEnvironmentFile=-/srv/etcd/config/etcd.confExecStart=/srv/etcd/etcd --enable-v2=trueExecReload=/bin/kill -HUP \$MAINPIDRestart=on-failureRestartSec=10sKillMode=processLimitNOFILE=40000[Install]WantedBy=multi-user.targetEOF

4、启动 systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

Nginx

1、下载扩展模块 nginx-upsync-module

yum -y install gcc-c++ zlib-devel openssl-devel pcre-devel gd-devel patchwget  -O nginx-upsync-module_v2.1.3.tar.gzwget   xf nginx-1.20.1.tar.gz && cd nginx-1.20.1 && unzip master.zip# 编译之前部分模块需要先给 nginx 打补丁patch -p1 < patch -p1 < /root/src/nginx_upstream_check_module-master/check_1.12.1+.patch./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' \--add-dynamic-module=/root/src/nginx-upsync-module-2.1.3 \--add-module=/root/src/nginx_upstream_check_module-master# nginx 编译选项说明:# --add-module    静态加载, 编译后需要替换正式环境中的二进制文件 nginx# --add-dynamic-module 动态加载, 将模块编译为 objs/*.so(make 编译后即可生成) 然后在 nginx 的配置文件中加载就行, 不需要替换 nginx# 编译安装 nginx_upstream_check_modulemake && make install# 添加运行用户及必要的文件目录信息useradd -M -r -s /sbin/nologin nginx && mkdir /etc/nginx/conf.d/ && touch /etc/nginx/conf.d/serversmkdir /var/cache/nginx/client_temp -p

2、nginx 配置文件主要部分

load_module "/usr/lib64/nginx/modules/ngx_http_upsync_module.so";# http 模块下upstream test {    # 后端使用 consul 存储    upsync 192.168.31.69:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;    # 后端使用 etcd 存储    #upsync 192.168.31.69:2379/v2/keys/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=etcd strong_dependency=off;    upsync_dump_path /etc/nginx/conf.d/servers;    include /etc/nginx/conf.d/servers;    check interval=5000 rise=2 fall=3 timeout=3000 type=http;    check_http_send "HEAD /test HTTP/1.0\r\n\r\n";    check_http_expect_alive http_2xx http_3xx;}server {    listen 80;    server_name c22.it123.com;    location /test {        proxy_pass ;    }    location /upstream_show {        upstream_show;    }    # 查看后端服务器状态    location /backend_status {        check_status;        # check_status json;        access_log off;        #allow 172.1.1.1;        #deny all;    }}

3、初始后端服务器列表文件

cat /etc/nginx/conf.d/servers> server 192.168.31.17:8081 weight=1 fail_timeout=10 max_fails=3;

4、启动 nginx systemctl start nginx

验证

1、etcd 方式

# 添加后端上游服务器, 对应配置项会立即写入文件 /etc/nginx/conf.d/serverscurl -X PUT -d value="{\"weight\":3, \"max_fails\":3, \"fail_timeout\":15}"  -X PUT -d value="{\"weight\":3, \"max_fails\":3, \"fail_timeout\":15}"  同步上游服务器配置到文件cat /etc/nginx/conf.d/serversserver 192.168.31.17:8081 weight=1 fail_timeout=10 max_fails=3;server 192.168.31.17:8082 weight=1 fail_timeout=10 max_fails=3;server 192.168.31.17:8083 weight=1 fail_timeout=10 max_fails=3;# 查看已添加的配置export ETCDCTL_API=2etcdctl --endpoints=//192.168.31.69:2379 -r ls /

2、Consul 方式

# 增加一个后端服务器curl -X PUT  -X PUT  -X PUT  同步上游服务器配置到文件cat /etc/nginx/conf.d/serversserver 192.168.31.17:8081 weight=1 fail_timeout=10 max_fails=3;server 192.168.31.17:8082 weight=1 fail_timeout=10 max_fails=3;server 192.168.31.17:8083 weight=1 fail_timeout=10 max_fails=3;# 删除一个服务器curl -X DELETE 

3、业务访问测试