豆瓣网技术架构

[e-mail]•20053••••2.8M1/4••20M/500~600/sec•23PC(1U*15/2U*8)•12•38Gmemcached•1U(frodo)•AMDAthlon641.8GHz•1G160GSATA*2•GentooLinux•MySQL5•Quixote(aPythonwebframework)•Lighttpd+SCGI(shire)•Memcached(!)InternetMySQLLighttpdAppSCGIMemcacheStaticFilesFSGentooLinux••emergemysql•ebuildpatch•••GLSA(GentooLinuxSecurityAdvisories)MySQL•Theworld’smostpopularopensourcedatabase•/==MyISAM•==InnoDB•ReplicateforbackupPython••BatteryIncluded•••CPUG:•RESTURL•Django,TurboGears,PylonsZOPE•(request,name):subject=get_subject(name)returnlambdareq:subject_ui(req,subject)#luz/subject/subject_ui.ptldefsubject_ui[html](request,subject):site_header(request)“h1%s/h1”%subject.titlesite_footer(request)Lighttpd••SCGI•SCGI:FastCGIQuixote•80lighttpdSCGIlocalhostQuixoteMemcache•MySQL•libmemcachepythonPyrexpython3x+defget_subject(subject_id):subject=mc.get(‘s:’+subject_id)ifsubjectisNone:store.farm.execute(“selectxxx,xxxfromsubjectwhereid=%s”,subject_id)subject=Subject(*store.farm.fetchone())mc.set(‘s:’+subject_id,subject)returnsubject•1.2M/•IO••1U•pippinmeriadoc(merry)•,4G250GSATA*3••IPDNSIP-_-b•frodo(subversion,trac,etc...)InternetDNSLighttpd(!)Lighttpd(#$)AppMySQLHTTPProxyMemcacheStaticFilesFSSCGI••innodb_buffer_pool_size••IP•1.5M/••IP•IP(BGP)•(arwen)•74G1wSATA*3••InternetLighttpdAppMemcacheDataMiningMySQLMasterMySQLSlaveReplicateStaticFilesSCGIreadwrite•2M/•IO•,etc...••4G250GSATA*3•10000•mod_rewriteURL•lighttpdmod_memcache•IO•web••WebServiceInternetLighttpdAppMemcacheMySQLMasterMySQLSlaveReplicateStaticFilesLighttpdWebDAVWebDAVSCGI!#$%MemcacheSpidersDataMiningMySQLSlaveReplicateLighttpd(w/mod_memcache)HTTPProxystore.farmstore.farmrwriteread•storefarmr•replicatedelay•••cache•••......butitworksreplicatedelaydefget_subject(sid):sbj=mc.get(‘s:’+sid)ifsbjisNone:sbj=flush_subject(sid,store.farmr)returnsbjdefflush_subject(sid,cursor=None):cursor=cursororstore.farmcursor.execute(“select...fromsubject”)subject=Subject(*cursor.fetchone())mc.set(‘s:’+sid,subject)returnsubjectdefupdate_subject(subject,props):store.farm.execute(“updatesubject...”)store.farm.connection.commit()flush_subject(subject.id,store.farm)•2.5M/••/•SATA••ScaleUp1U•16G147GSCSI*2+500GSATA•SCSIRAID-0•MySQLSlave•memcached•MyISAMInnoDB••SphinxInternetLighttpdAppMemcacheMySQLMasterMySQLSlaveReplicateStaticFilesLighttpdWebDAVWebDAVSCGIMemcacheSpidersLighttpd(w/mod_memcache)HTTPProxystore.farmstore.farmrSphinxWebServiceMemcacheMemcacheWebService•5.2M/••WebIO•••:)••••31U432G1TSATA*3•otho.douban.comlotho.douban.com•lighttpd1.5withaiosupport•LVS•ScaleUp:4G-8GInternetLighttpdStaticFilesLighttpdWebDAVLighttpdHTTPProxyLighttpd1.5(w/mod_cache)Lighttpd1.5(w/mod_cache)LVSLB(Master)LVSLB(backup)Keepalived!#$%DataMining!#$%DataMiningMySQLSlavereplicatereadreadwritewrite•6.4M/(5MPV)•••CPUmemcache/••lighttpdmod_scgiround-robin•lighttpd1.5•mod_proxy•proxy.balance=fair(loadbased,passivebalancing)••spreadLighttpdStaticFilesLighttpdWebDAVLighttpdHTTPProxyAppMemcacheLighttpdAppMemcacheLighttpdInternetSCGISCGIHTTPProxyHTTPProxyLogAggregatorspreadspread•11M/3••Sphinx••load••••Sphinx-Xapian•MogileFS•libmemcache-libmemcachedconsistenthashmemcache•libmemcachedconsistenthashbug•CPU•libmemcachedfailoverbug•nginxlighttpdloadbalance•spread•nginx!Master#$Master%&'()DataMining%&'()DataMining#$Slave!Slave*+,-Slave!Slave#$Slave*+,-Masterreplicatewritewritereplicatereplicatereadreadreplicatereplicate••store.farm[r]-store.get_cursor(table=‘xxx’,ro=True/False)defflush_subject(sid,ro=False):cursor=store.get_cursor(table=‘subject’,ro=ro)cursor.execute(“select...fromsubject”)subject=Subject(*cursor.fetchone())mc.set(‘s:’+sid,subject)returnsubject•MogileFSMasterUploaderMogileFSNodeMogileFSNodeLighttpd1.5(w/mod_cache)MogileFSTrackerFileStorageGatewayInternetupload.douban.comotho.douban.comAppsignedPOSTformHTTPredirectLighttpdStaticFilesLighttpdWebDAVLighttpdHTTPProxyAppLighttpdInternetSCGIHTTPProxyNginxHTTPProxyAppLighttpdSCGIHTTPProxyAppLighttpdSCGIHTTPProxy•13M/•MogileFS•TrackerDB••8•32GCPU•(300GSCSI×2+1TSATA)×3•(1TSATA×3)×5•62•DoubanFSDoubanFS•hash•hashhash•MerkleTree•consistenthash•WebDAV•MogileFS350ConsistentHashMerkleTreeUploaderDoubanFSNodeDoubanFSNodeLighttpd1.5(w/mod_cache)FileStorageGatewayInternetupload.douban.comotho.douban.comAppsignedPOSTformHTTPredirect•16M/••DoubanFSIO••DoubanDB••MySQL•MySQLMaster•failover•replicatedelayDoubanDB•Key-Value•AmazonDynamo•set(key,value),get(key),delete(key)•memcache•MerkleTree•ConsistentHash•TokyoCabinet•DoubanDB•DoubanFS2.0DoubanDB!Master1#$Master1%&'()DataMining%&'()DataMining#$Master2!Master2*+,-Slave!Slave#$Slave*+,-Masterreplicatewritewritereplicatereplicatereadreadreplicatereplicate•DoubanFS•••ngnix••RabbitMQspread••profile•memcachecache•join••