大众点评网高可用数据架构

HAArchitectureinDPMMM&Memcached卢钧轶@[e-mail]:米雪儿侬好的cenaluluBlog::●Memcachedfailed(physical)●Keyexpired(logical)CacheMissStormIdealCacheMissProcedure1.getmemcachedmiss2.queryMySQL3.setvalueintomemcachedCacheMissStormInFact!1.getmemcachedmiss2.massiveconcurrentqueryonMySQL(timeout)3.nothingcanbesetintomemcached4.cachemissforever....CacheMissStorm--Fix1Node1Node2Node3Node4MainRingBackupRingFixforphysicalcachemiss--DoubleWriteCacheMissStorm--Fix2Fixforlogicalcachemiss--HotKey0.get&&setvalueintoweblocalcache1.getmemcachedmiss2.trytoaddalockkeyintomemcachea.if(success)queryMySQL&setmemcacheb.if(failed)returnlocalcache*OnlyonewebcanqueryMySQLformissedkeyatthesametime.Problem2MultiGetHoleMultiGetHoleMultiGet/Gets:getcommandwithmultiplekeysPurpose:Omitthemultiplenetworkround-trips,Problem:Thegetscommandwillbeslowerwhenweaddmorenodesintothecluster.!!!!Node4Resultv1,v5,v9v2,v6,v10v3,v7,v11v4,v8,v12MultiGetHole--FixBefore:clientusekeynameto●decidewhichhosttostore●storevalueNow:clientusemasterkeynameto●decidewhichhosttostoreusekeynameto●storevalue*PutthekeysofthesamegroupintoonlyonenodeMultiGetHole--FixNode1Node2Node3Clientcmd:setkey:key1master_key:group1value:v1Hashusingonlymaster_keyMultiGetHole--FixNode1Node2Node3Clientcmd:getkey:key1...key12master_key:group1MultiGetHole--FixNode1Node2Node3Clientcmd:getkey:key1...key12master_key:group1node3getkey1...key12Hashusingonlymaster_keyMultiGetHole--FixNode1Node2Node3ClientResultv1,v4,v7,v10v2,v5,v8,v11v3,v6,v9,v12node3getkey1...key12Problem3VirtualPacketLossVPLVPL:virtualpacketlossNoactualpacketloss,butvmresponsetimeexceedstheretransmissiontimeoutTwonetwork-boundedvirtualmachineputtogetherresultinhugegettimeout.VPLAnormalretransmissionconsume50ms,whichexceedsourMemcachedtimeout.timeout==noresult==cachemissResult:anotherkindofcachemissstormVPL--Fix●SplitNetwork-Boundbizondifferentrealmachine.●MaybeUDP?●Maybefastretransmission?ShortQuestions?MMMWhatisMMM●Perl●MessagebetweenMonitor&Agent●AutoFailoverforM/SbutMMMisnot:●SQLrouter●LoadBalancerWhat'swrongwithMMMMMMis1)fundamentallybrokenandunsuitableforuseasaHAtool2)absolutelycannotbefixed.BaronSchwartz:(){process_check_resultscheck_host_statesprocess_commandsdistribute_rolesend_status_to_agents}Agentwhile(readsocket){handle_command}MMMarchitectureMonitorSlavevip3Mastervip1Slavevip4Mastervip2HowMMMDoFailoverMonitorSlavevip3Mastervip1Slavevip4Mastervip2HowMMMDoFailoverMonitorSlavevip3Mastervip1Slavevip4Mastervip2setglobalread_only=1HowMMMDoFailoverMonitorSlavevip3MasterSlavevip4Mastervip2removeVIPHowMMMDoFailoverMonitorSlavevip3MasterSlavevip4Mastervip2selectMASTER_POS_WAIT()HowMMMDoFailoverMonitorSlavevip3MasterSlavevip4Mastervip2showmasterstatusHowMMMDoFailoverMonitorSlavevip3MasterSlavevip4Mastervip2changemastertoHowMMMDoFailoverMonitorSlavevip3MasterSlavevip4Mastervip1&vip2vip1onlineMMMMMMinDPMMMinDPFrontendGroupvip1&vip2BackendGroupvip3&vip4JobGroupvip5Slavevip3/vip5Mastervip1Slavevip4Mastervip2MMMProblemsinMMMMMMProblem1setread_onlyisdifficultonbusyserversetread_onlywillbeblockedbylongrunningSQLMonitorSlavevip3Mastervip1Slavevip4Mastervip2setglobalread_only=1MMMProblem1MonitorSlavevip3Mastervip1Slavevip4Mastervip2setglobalread_only=1MMMProblem1--FixMonitorSlavevip3Mastervip1Slavevip4Mastervip2removevipMMMProblem1--FixMonitorSlavevip3MasterSlavevip4Mastervip2killuncommitedprocessMMMProblem1--FixMonitorSlavevip3MasterSlavevip4Mastervip2killuncommitedprocessMMMProblem1--FixMonitorSlavevip3MasterSlavevip4Mastervip1&vip2showmasterstatschangemastertoMMMProblem2MonitorSlave30mBehindMasterSlavevip4Mastervip2selectMASTER_POS_WAIT()WriterVIPcannotbeaccessedwhenslaveisfarbehindmasterMMMProblem2MonitorSlavevip3MasterSlavevip4Mastervip1&2WriterVIPcannotbeaccessedwhenslaveisfa