基于Nginx负载均衡方案

基于Nginx负载均衡方案项目背景公司一直使用商用负载均衡（LB），基于以下几点原因考虑用开源产品来替代：价格昂贵，HTTPS支持并发数太低技术门槛比较高，学习成本大技术Bug修复方面都太慢商用产品在新功能技术支持方面（如H2,protocol_proxy支持）滞后技术选型主要调研了lvs/haproxy/nginx这三种开源产品在四层负载方面功能特性，新LB方案水平扩展相对容易，选型阶段主要考察功能支持情况，情况如下：综上，同时负载均衡主要业务对象是web类型，现有运维人员对nginx比较熟悉，最终先把Nginx做四层负载。方案设计四层负载在最前端，后端七层负载。四层负载主要负责对外暴露公网IP，七层负载主要业务规则重写。同时考虑多机房的容灾，架构设计如下：上图主要做到以下组件冗余：机房A和机房B即可主备也可双主，避免单点四层接入方面，当OSPF发生故障，可以用Nginx做备用，直接指向L7服务器组双机房L7服务器组也可以同时提供服务，避免单点双机房L7服务器组配置保持同步，安装方式采用静态编译安装，复制考贝启动即可相关配置网络相关配置服务器部分配置如下：#OS基于Centos7，测试环境，生产环境根据实际情况修改#安装路由软件yuminstallquagga#配置zebra#cat/etc/quagga/zebra.conf!!Zebraconfigurationsavedfromvty!2017/09/2815:57:12!hostnametest-ssl-10-231.test.org#这个每台名字要不同password8WuN0UOEsh./0Uenablepassword8g9UPXyneQv2n.logfile/var/log/quagga/zebra.logservicepassword-encryption#配置ospfd#cat/etc/quagga/ospfd.confhostnametest-ssl-10-231.test.org#每台要不同password8cQGHF4e9QbcAenablepassword8RBUKMtvgMhU3Mlogfile/var/log/quagga/ospfd.logservicepassword-encryption!!!interfaceeth2ipospfauthenticationmessage-digestipospfmessage-digest-key1md5pIW87ypU3d4v3pG7#此处密码告知网络工程师ipospfhello-interval1ipospfdead-interval4ipospfpriority0routerospfospfrouter-id10.10.41.130#每台router-id要不一样log-adjacency-changesnetwork10.10.41.0/24area0.0.0.0network10.10.100.100/32area0.0.0.0#宣告自己的ospf互边地址和VIP地址，新增地址都在此处添加area0.0.0.0authenticationmessage-digest!linevty!#启动服务systemctlenablezebra.servicesystemctlenableospfd.servicesystemctlstartzebra.servicesystemctlstartospfd.service#添加ospf和zebra保活,打开配置文件打开如下行行vim/etc/sysconfig/quaggaWATCH_DAEMONS='zebraospfd'######策略路由配置，eth0指向默认路由，在eth1模拟公网进行配置#######cat/etc/iproute2/rt_tables增加100wan41#增加路由表相关配置iprouteadd10.10.41.0/24deveth1src10.10.41.130tablewan41iprouteadddefaultvia10.10.41.250tablewan41ipruleaddfrom10.10.41.130tablewan41持久化到配置文件catroute-eth110.10.41.0/24deveth2src10.10.41.130tablewan41defaultvia10.10.41.250table100catrule-eth1from10.10.41.130tablewan41######策略路由配置结束######交换机配置部分（略）增加zebraospfd保活打开/etc/sysconfig/quagga注释以下行：WATCH_DAEMONS='zebraospfd'nginx七层配置，关键是日志配置获取ClientIP如下：servercontextlisten增加如下：listen80proxy_protocol;listen443http2proxy_protocol;#log_format,要配置$proxy_protocol_addr$proxy_protocol_port,log_formatxff'$proxy_protocol_addr:$proxy_protocol_port$http_x_forwarded_for-$remote_user[$time_local]'$request'''$status$body_bytes_sent'$http_referer''$http_user_agent''$host'''$request_time'$upstream_addr''$upstream_response_time''$server_protocol'';nginxtcp四层代理配置stream{log_formatproxy'$remote_addr:$remote_port[$time_local]''$protocol$status$bytes_sent$bytes_received''$session_time'$upstream_addr''''$upstream_bytes_sent''$upstream_bytes_received''$upstream_connect_time'';upstreambackend-test{server10.x.x.233:80;}upstreambackend-test_ssl{server10.x.x.233:443;}server{listen80;proxy_protocolon;proxy_passbackend-test;access_log/opt/test/logs/nginx/m.test.com.logproxy;}server{listen443;proxy_protocolon;proxy_passbackend-test_ssl;access_log/opt/test/logs/nginx/m.test.com.logproxybuffer=1kflush=1s;}}nginx加入sysctemctl管理，并加入开机启动[Unit]Description=nginxAfter=network.target[Service]Type=forkingExecStart=/opt/test/nginx/sbin/nginxExecReload=/opt/test/nginx/sbin/nginx-sreloadExecStop=/opt/test/nginx/sbin/nginx-sstopPrivateTmp=true[Install]WantedBy=multi-user.target#开机启动systemctlenablenginx.service运维管理新增IPcataddip.sh#!/bin/baship=$1pswd='test123'expect-c'settimeout30evalspawn-noechotelnet127.0.0.12604expect\'Password:\'send\'$pswd\r\'expect\'*\'send\'enable\r\'expect\'Password:\'send\'$pswd\r\'expect\'*#\'send\'configuret\r\'expect\'*(config)#\'send\'routerospf\r\'expect\'*(config-router)#\'send\'network$ip/32area0.0.0.0\r\'expect\'*(config-router)#\'send\'w\r\'send\'exit\r\'send\'exit\r\'send\'exit\r\'interact'/dev/null###增加策略路由ipaddradd10.10.100.103/32devlo:1ipruleaddfrom10.10.100.103tablewan41###持久化到配置文件#rule-lo:1from10.10.100.103tablewan41保活#故障，OSPF下线，恢复自动上线，监控setmailservermail.test.comport25setmail-format{from:[e-mail]:Nginx-L4$SERVICE$EVENTat$DATEmessage:Monit$ACTION$SERVICEat$DATEon$HOST:$DESCRIPTION.}[e-mail]pidfile/opt/test/nginx/logs/nginx.pidifdoesnotexistfor3cyclesthenexec'/bin/systemctlstopzebra'elseifsucceededfor3cyclesthenexec'/bin/sh/opt/test/sysadmin/ospf_start.sh'checkhostNginx-L4withaddress10.x.x.250iffailedpingcount5withtimeout1secondsthenexec'/bin/systemctlstopzebra'elseifsucceededthenexec'/bin/sh/opt/test/sysadmin/ospf_start.sh'性能测试数据主要测试七层SSLRSA2048位加解密能力，2620CPU加装了加速卡之后，并发TPS能达到26000。数据分析基于ESAPI获取带宽，流量，PV数据汇总后再次存入ES，最终使用grafana进行展示。出处：