Linux服务器性能调优

热身讨论服务器由什么组成？如何最大化服务器性能？服务器：核心三大核心CPUIO存储服务器：配合性能最大化CPUIO存储服务器：架构架构服务器：架构接入逻辑代理数据控制数据服务器：架构常用架构讲解服务器：架构任务/操作序列步骤0netIO(epoll)步骤1diskIO步骤2CPU步骤3DB(mysql)步骤4CPU步骤5RPC服务器：架构传统架构线程0线程1线程2步骤0netIO(epoll)步骤1diskIO步骤5RPC…步骤0netIO(epoll)步骤1diskIO步骤5RPC…步骤0netIO(epoll)步骤1diskIO步骤5RPC…开发简单，逻辑清楚，不容易出错并发量小，容错性差，性能差，可扩展性差优点：缺点：服务器：架构功能组件架构（流水线架构）线程0网络线程1…disk线程2CPU线程3…DB步骤0netIO(epoll)步骤1diskIO步骤2CPU步骤3DB(mysql)步骤4CPU步骤5RPC开发量大，容错性强，高性能，可扩展性好；开发简单，不容易出错逻辑不直观，优点：缺点：性能：指标吞吐量Throughput延迟Latency性能：指标吞吐量Throughput延迟Latency并发量时间性能：指标吞吐量Throughput延迟Latency个/秒秒/个性能：指标吞吐量Throughput延迟Latency性能：流程满足性能目标？定位瓶颈调优否退出开始是迭代性能：分析三大核心CPUIO存储性能：分析三大核心CPUIO存储freedfnetstatiostatvnstatlsoftcpdumpifconfigethtooltoppsvmstatstraceltracegdb……性能：瓶颈定位CPU/内存：free,vmstat,topIO：iostat,netstat(EST2w,SYN1k,TIME_WAIT10),tcpdump,lsof,vnstat,ifconfig,ethtool进程：strace,ltrace,ps性能：瓶颈定位问题举例及实例讲解：内存不足、CPU不够、连接数太多、TIME_WAIT太多、句柄数太多、IO太多上面每种情况，一个截图来说明性能：瓶颈定位问题举例及实例讲解：内存不足、CPU不够、连接数太多、半开连接数太多、句柄数太多、IO太多上面每种情况，一个截图来说明[hepengfei@test12tmp]#toptop-17:43:27up115days,19:38,1user,loadaverage:12.68,12.19,12.58Tasks:136total,6running,129sleeping,0stopped,1zombieCpu(s):47.9%us,7.0%sy,0.0%ni,23.1%id,13.5%wa,0.2%hi,8.2%si,0.0%stMem:16426616ktotal,16373692kused,52924kfree,68104kbuffersSwap:20972848ktotal,5401524kused,15571324kfree,7799216kcachedPIDUSERPRNIVIRTRESSHRS%CPU%MEMTIME+COMMAND23462root250488m180m4160R92.71.1290:41.36ffmpeg_new16782root180294m76m3056R88.40.51:50.86videotext_detec16996root190421m115m4168R86.40.71:06.23ffmpeg_new15261root1502204m859m63mR55.55.425724:51chunk_server4082root1506480m2.2g2576S54.814.269143:39http_down_44331201root1503738m1.9g3240R52.811.84282:14vip_cdn4075root1505771m1.9g2476S47.511.970235:04http_down444root10-5000S1.70.01805:47kswapd1443root10-5000S1.30.01473:22kswapd08266root1501857m103m2944S1.30.63327:59gdrive.store.no3610root10-5000S0.30.028:34.76kjournald1root15010348588556S0.00.04:25.55init2rootRT-5000S0.00.00:15.18migration/03root3419000S0.00.00:15.92ksoftirqd/04rootRT-5000S0.00.00:00.00watchdog/05rootRT-5000S0.00.00:06.24migration/16root3419000S0.00.00:00.95ksoftirqd/17rootRT-5000S0.00.00:00.00watchdog/18rootRT-5000S0.00.00:13.28migration/2性能：瓶颈定位问题举例及实例讲解：内存不足、CPU不够、连接数太多、半开连接数太多、句柄数太多、IO太多上面每种情况，一个截图来说明[hepengfei@test12tmp]#toptop-15:36:25up31days,8:27,5users,loadaverage:9.10,10.58,10.51Tasks:213total,2running,209sleeping,0stopped,2zombieCpu0:13.7%us,15.7%sy,0.1%ni,68.3%id,2.1%wa,0.0%hi,0.1%si,0.0%stCpu1:18.1%us,16.9%sy,0.0%ni,60.0%id,3.0%wa,0.1%hi,1.8%si,0.0%stCpu2:17.3%us,15.6%sy,0.0%ni,62.1%id,3.0%wa,0.1%hi,1.9%si,0.0%stCpu3:16.9%us,15.1%sy,0.0%ni,62.7%id,2.3%wa,0.1%hi,2.9%si,0.0%stCpu4:18.1%us,14.7%sy,0.0%ni,47.4%id,15.5%wa,0.2%hi,4.0%si,0.0%stCpu5:17.7%us,16.4%sy,0.0%ni,50.3%id,13.3%wa,0.3%hi,2.1%si,0.0%stCpu6:19.2%us,14.5%sy,0.0%ni,47.5%id,14.3%wa,0.2%hi,4.2%si,0.0%stCpu7:18.8%us,13.7%sy,0.0%ni,48.2%id,14.6%wa,0.2%hi,4.4%si,0.0%stMem:16411876ktotal,11161960kused,5249916kfree,6280kbuffersSwap:10482404ktotal,5117080kused,5365324kfree,780392kcachedPIDUSERPRNIVIRTRESSHRS%CPU%MEMTIME+COMMAND10686root1508624m8.1g1764S2.052.033:59.84res_del538root150927m342m3052S0.02.10:01.37scene8724root150705m305m2788S0.01.90:11.09btbox3472root150831m228m1876S0.01.4209:36.40gdrive.gim23927root150672m103m3532S2.00.6218:55.67gdrive.gdm_pt5060root150463m51m1932S0.00.3349:51.92gsmReport9243root1502766m42m2812S0.00.3110:13.83client_interfac31843root1502769m39m2868S0.00.2105:05.03client_interfac6303root150519m27m744S0.00.2237:24.72httpsqs4318root180298m21m1532S4.00.12057:35named23731root150149m18m2340S0.00.129:17.59gcid_checker.rb23604root150149m18m2340S0.00.131:11.30gcid_checker.rb性能：瓶颈定位问题举例及实例讲解：内存不足、CPU不够、连接数太多、半开连接数太多、句柄数太多、IO太多上面每种情况，一个截图来说明[root@test12tmp]#netstat-anp|gawk'{print$6}'|sort|uniq-c|sort-nr|head8733TIME_WAIT2433ESTABLISHED157LAST_ACK120FIN_WAIT129CLOSE_WAIT21FIN_WAIT219CLOSING10SYN_RECV10LISTEN2SYN_SENT性能：瓶颈定位问题举例及实例讲解：内存不足、CPU不够、连接数太多、半开连接数太多、句柄数太多、IO太多上面每种情况，一个截图来说明[root@test12tmp]#strace-p312012&1|grep-E'^(read|write|epoll|futex)'epoll_wait(12,{{EPOLLIN,{u32=2,u64=2}}},51200,0)=1read(7,\1\0\0\0,4)=4read(7,!\1\0\0pq\4\231\1\0\0\0\0\0\0\0D\0\0\0D@\0\0\0\0\0\0t\224\324\31...,40)=40epoll_ctl(12,EPOLL_CTL_MOD,289,{EPOLLIN,{u32=520,u64=520}})=0read(7,0x7fff8d99db5c,4)=-1EAGAIN(Resourcetemporarilyunavailable)epoll_wait(12,{},51200,0)=0epoll_wait(12,{},51200,0)=0epoll_wait(12,{},51200,0)=0epoll_wait(12,{},51200,0)=0epoll_wait(12,{{EPOLLOUT,{u32=466,u64=466}},{EPOLLIN,{u32=261,u64=261}}},51200,0)=2epoll_ctl(12,EPOLL_CTL_DEL,315,NULL)=0epoll_ctl(12,EPOLL_CTL_DEL,695,NULL)=0futex(0x2aaab40109fc,FUTEX_WAKE_OP_PRIVATE,1,1,0x2aaab40109f8,{FUTEX_OP_SET,0,FUTEX_OP_CMP_GT,1})=1futex(0x2aaab40109a0,FUTEX_WAKE_PRIVATE,1)=1epoll_wait(12,{{EPOLLIN,{u32=2,u64=2}}},51200,0)=1read(7,\1\0\0\0,4)=4epoll_ctl(12,EPOLL_CTL_ADD,315,{EPOLLIN,{u32=261,u64=261}})=0read(7,0x7fff8d99db5c,4)=-1EAGAIN(Resourcetemporarilyunavailable)性能：瓶颈定位问题举例及实例讲解：内存不足、CPU不够、连接数太多、半开连接数太多、句柄数太多、IO太多上面每种情况，一个截图来说明[hepengfei@test12workspace]$ltrace-p11261[pi