搜索
互联⺴⽹网架构优化@TimYang1PartI背景知识2Softwarestack(Preferred)•LAMPLinuxApache(Tomcat)MySQLPHP(JavaC…)•Memcache/Redis•SOA/OpenAPI•RPC:ThriftandProtocolBuffers•MySQL,HBase分布式⼤大规模数据存储3PresentationTierArch(Preferred)Apache/Nginx/PHPVarnish/HAProxy浏览器LVS4层7层应⽤用层Service/Database服务/数据层4•L4•LVS,HAProxy•Loadbalance•Highavailability•L7•Squid,Varnish•Cache•SlowconnectionmanageLoadBalancer5L7LevelCaching•Varnish:AHTTPaccelerator(reverseproxy)thatcachesdatainvirtualmemory.•Squid:AcachingserverthatcancacheDNS,FTP,HTTPS,HTTP,TSL,andSSLprotocols.•ApacheTrafficServer:Areverseproxyandforwardproxycachingserverthatdespiteitsname,workswithNginx.6Language•PHP⾯面向⽤用户端产品，开发快速，PHPextension•Java后端服务，适合业务逻辑实现，只使⽤用最基本语⾔言特性•C底层服务，存储、计算，关注⾼高性能•其他选择？Erlang,Scala,Go……7MySQL•久经考验•性能与访问量保持线性•通过sharding存储海量数据，可扩展•活跃的社区免费•但扩展需要投⼊入较多⼈人⼒力Reference:•基于MySQL的分布式数据库实践•成熟稳定•⾼高性能•⼲⼴广泛使⽤用•免费9Redis•丰富的数据结构，List,Set……•可持久保存•性能好•稳定•免费Reference:•新浪微博开放平台中的Redis实践fl-sina-weibo-platform-redis-practice•Redis运维之道案例：博客功能1.博⽂文2.评论3.⽤用户需求•100万⽤用户•500QPSThefirstblogpostSep2,2013Comments(123)Page1.2.3.4.5123Tim'sBlog311DatabaseschemaUserid:uint64nick:stringavatar:stringPostid:uint64subject:stringcontent:stringuid:uint64ctime:uint32ctime:uint32Commentsid:uint64pid:uint64content:stringuid:uint64ctime:uint3212CreateInsertintoPosts(uid,subject,content)values(…)Id⾃自增主键ReadSelect*frompostswhereuid=xxxorderbyctimedescSelect*frompostswhereid=xxx;UpdateUpdatepostssetsubject=xxxandcontent=xxxwhereid=xxxDeleteDeletefrompostswhereid=xxx13MySQL(TableUser/Post/Cmt)LoadbalancerSystemDesignApache/PHP(Tomcat/Java)MemcachedMS14Cache及容量设计TypeKeyValueUseruiduserobjectPostrecentlistuidvector(n)Commentrecentlistpostidvector(n)Postcontentpostidpostobject15容量评估表数据类型单位（如每⽤用户）单位容量（当前/最⼤大）数量（当前/最⼤大）Write/s(max)Read/s(max)NodesMemory(⽅方案结论)(⽅方案结论)16CacheDeployment•⽅方案关键点•服务器数LoadBalance&HighAvailbility•端⼝口数Throughput•HashbyKey•Key%ports•CRC32•ConsistentHash17•为什么新发表的博客刷新后不出来？•为什么评论跟计数不同？Reference:•CAPtheorem~brewer/cs262b-2004/PODC-keynote.pdf•BASE(BasicallyAvailable,Softstate,Eventualconsistency)vs.ACID(Atomicity,Consistency,Isolation,Durability)•EventuallyConsistent,WernerVogels,CTOAmazon,://forge.mysql.com/wiki/Top10SQLPerformanceTipsinnodb_buffer_pool_sizequery_cache_sizeEXPLAINIndexesSELECT*DISTINCTvs.GROUPBY20Reference:•Scale-outvsScale-up•ScalingMySQL:ScaleUpversusScaleOut•ContinuousAvailability/Redundancy•Cost/PerformanceFlexibility•ContinuousUpgrades•GeographicalDistribution21ScaleOutFeaturesandBenefits22案例-CascadedReplication如Yahoo!新浪微博……23MySQL(TableUser/Post/Cmt)LoadbalancerSystemDesignApache/PHPMemcachedMMSSMSSMSSMSS24SOA•功能耦合⼤大包上线互相依赖•连接数限制•并⾏行开发25ServiceTierArch(Preferred)ServiceApacheTomcat展⽰示层LVS/Varnish/HAProxy4/7层应⽤用层服务层Master数据层SlaveRedisHBaseQueueMemcachedRESTfulAPIRPC26“⽆无极⽣生太极太极⽣生两仪两仪⽣生四象四象⽣生⼋八卦⼋八卦⽣生六⼗十四卦六⼗十四卦⽣生宇宙万象”⽆无限扩展为永恒之道郭欣，构建⾼高性能Web站点27PartII性能优化28Whatdowethinkaboutarchitecture?1.Performance29热点分析ThefirstblogpostSep2,2013Comments(123)Page1.2.3.4.5123Tim'sBlog341.Listofposts2.Listofcomments3.Userdata4.Counter30Reference:•JeffDean：NumbersEveryoneShouldKnow“EverythingrunsfrommemoryinWeb2.0”EvanWeaver,Twitter,March200933Cache演进⽆无cache进程内cache分布式cachecachewithSSDCacheaaSReference:•Twemproxy:Afast,light-weightproxyformemcached•腾讯⺴⽹网相关的云平台之⾼高性能⾼高可⽤用cache平台复制Update/PurgeproblemforLRUhotdata34cache的性能与⾼高可⽤用•没有好的更新⼿手段不能⾃自我修复、不能全部清除•缺少好的监控连接数、访问量•CacheHA策略replication,中间层•⼀一致性保证校验及修复机制•LRUfailed,MemorystoreeraReference:•Missingcachestorm•cachemiss⻛风暴上千个并发线程同时穿透到db•key_mutex先add成功，再loaddb•mutex失败则等待，重试•添加结束，删除mutex36存储的扩展性与优化37Database•如何存储每天上亿数据？•MySQL久经考验HandlerSocket,SSD,bin-logtools•适合场景读密集？写密集？Likebutton,batchinsert(ugly)•按峰值访问部署设备利⽤用率的问题•Schema问题修改沉重→改变含义→keyvalue•主从延迟问题⼤大部分架构不能承受⼤大规模延迟开发与运维的噩梦•性能问题QPS较低38Sharding•Shardingbyhash/key•双份冗余的困惑socialgraph•多级索引按热度设计userprofileindex•持续拆分的成本⼈人员维护成本与稳定性的⻛风险39⼀一致性的烦恼•Master/slave多机房,trigger,预设缓存•多份存储评论列表与计数，最终不⼀一致•队列复制异地，对账•Notify与内容⼀一致性难题,since_id40MySQL+Memcached问题•⼀一致性脏数据……⼿手⼯工修改……防雪崩……如何修改？•代码优雅性调⽤用程序维护⼆二者⼀一致性ugly,writethrough,proxy•雪崩的⻛风险重启cache与批量清除的危险,consistenthash?•⼤大多序列化list,hash……增量……修改41NoSQL•one-sizefitsallnolongerapplies•graph,100kQPS•list,set...•Redis⾼高性能(600kQPS)数据结构丰富failover突然变慢适合场景(中转箱？)•NoS