搜索
TheeBayArchitectureStrikingabalancebetweensitestability,featurevelocity,performance,andcostPresentedBy:RandyShoupandDanPritchettDate:November29,2006SDForum2006eBay,Inc.2©2006eBayInc.Whatwe’reupagainst•eBaymanages…–Over212,000,000registeredusers–Over1Billionphotos–eBayusersworldwidetrademorethan$1590worthofgoodseverysecond–eBayaveragesover1billionpageviewsperday–Atanygiventime,thereareapproximately105millionlistingsonthesite–eBaystoresover2Petabytesofdata–over200timesthesizeoftheLibraryofCongress!–TheeBayplatformhandles3billionAPIcallspermonth•Inadynamicenvironment–300+featuresperquarter–Weroll100,000+linesofcodeeverytwoweeks•In33countries,insevenlanguages,24x726BillionSQLexecutions/day!AnSUVissoldevery5minutesAsportinggoodsellsevery2secondsOver½MillionpoundsofKimchiaresoldeveryyear!eBay,Inc.4©2006eBayInc.Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3Q4Q1Q2Q3eBay’sExponentialGrowth212MillionUsers19992000200120022003199820042005105MillionListings2006eBay,Inc.5©2006eBayInc.VelocityofeBay--SoftwareDevelopmentProcess•Oursiteisourproduct.Wechangeitincrementallythroughimplementingnewfeatures.•Verypredictabledevelopmentprocess–trainsleaveon-timeatregularintervals(weekly).•Paralleldevelopmentprocesswithsignificantoutput--100,000LOCperrelease.•Alwayson–over99.94%available.FeatureFeatureFeatureTrain6MLOC100KLOC/Wk99.94%212MUsers300+FeaturesPerQuarterAllwhilesupportinga24x7environmenteBay,Inc.6©2006eBayInc.SystemicRequirementsMaintainabilityFasterProductDeliveryEnablerapidbusinessinnovationEnableseamlessgrowthDeliverqualityfunctionalityatacceleratingratesArchitectforthefuture10XGrowthAvailabilityReliabilityMassiveScalabilitySecurityeBay,Inc.7©2006eBayInc.ArchitecturalLessons•ScaleOut,NotUp–Horizontalscalingateverytier.–Functionaldecomposition.•PreferAsynchronousIntegration–Minimizeavailabilitycoupling.–Improvescalingoptions.•VirtualizeComponents–Reducephysicaldependencies.–Improvedeploymentflexibility.•DesignforFailure–Automatedfailuredetectionandnotification.–“Limpmode”operationofbusinessfeatures.eBay,Inc.8©2006eBayInc.19981999200020012002200320042005Q32006OngoingPlatformEvolution…V1V2.0V2.4V3V2.3eBayarchitectureversionsRegisteredUsers212MV4V2.5eBay,Inc.9©2006eBayInc.V1.01995-September1997•BuiltoveraweekendinPierreOmidyar’slivingroomin1995•SystemhardwarewasmadeupofpartsthatcouldbeboughtatFry's•Everyitemwasaseparatefile,generatedbyaPerlscript•Nosearchfunctionality,onlycategorybrowsing19951996199719981999200020012002200320042005Thissystemmaxedoutat50,000activeitemseBay,Inc.10©2006eBayInc.V2.0September1997-February1999•3-tieredconceptualarchitecture(separationofbus/presanddbaccesstiers)•2-tieredphysicalimplementation(noapplicationserver)•C++Library(eBayISAPI.dll)runningonIISonWindows•Microsoftindexserverusedforsearch•ItemsmigratedfromGDBMtoanOracledatabaseonSolaris19951996199719981999200020012002200320042005eBay,Inc.11©2006eBayInc.V2.1February1999-November1999•Serversgroupedintopools(smallsoldiers)•Resonateusedforfrontendloadbalancingandfailover•SearchfunctionalitymovedtotheThunderstoneindexingsystem•Back-endOracledatabaseserverscaledverticallytoalargermachine(SunE10000)19951996199719981999200020012002200320042005eBay,Inc.12©2006eBayInc.V2.3June1999-November1999•SecondDatabaseaddedforfailover•CGIpools,Listings,Pages,andSearchcontinuedtoscalehorizontallyHowever…ByNovember1999,thedatabaseserversapproachedtheirlimitsofphysicalgrowth.eBay,Inc.13©2006eBayInc.V2.4November1999-April2001•Databasesplittechnology.•Logicallypartitiondatabaseintoseparateinstances.•Horizontalscalabilitythrough2000,butnotbeyond.eBay,Inc.14©2006eBayInc.V2.5April2001–December2002BearSUNA3500BullSUNA3500SUNA3500November,1999December,2002ACCOUNTSFEEDBACKARCHIVECATY1CATY2CATY3CATY4BATCHJOBSCATY5CATY6CATY7CATY8CATY12CATY11CATY10CATY9UserWriteUserReadTranScratchBATCHJOBS•Horizontalscalabilitythroughdatabasesplits•Itemssplitbycategory•SPOFeliminationeBay,Inc.15©2006eBayInc.NowthatwehavetheDatabasetakencareof….•ApplicationServer–Monolithic2-tierArchitecture–3.3MillionLineC++ISAPIDLL(150MBbinary)–Hundredsofdevelopers,allworkingonthesamecode–Hittingcompilerlimitsonnumberofmethodsperclass(!!)eBay,Inc.16©2006eBayInc.V3–ReplaceC++/ISAPIwithJava2002-present•Re-wrotetheentireapplicationinJ2EEapplicationserverframework–Gaveusachancetoarchitectthecodeforreuseandseparationofduties•LeveragedtheMSXMLframeworkforthepresentationlayer–Minimizingthedevelopmentcostformigration•Implementedadevelopmentkernelasafoundationforprogrammers–AllowedforrapidtraininganddeploymentofnewengineerseBay,Inc.ScalingtheDataTiereBay,Inc.18©2006eBayInc.ScalingtheDataTier:Overview•SpreadtheLoad–Segmentationbyfunction.–Horizontalsplitswithinfunctions.•MinimizetheWork–Limitindatabasework•TheTrickstoScaling–Howtosurvivewithouttransactions.–Creatingalternatedatabasestructures.eBay,Inc.19©2006eBayInc.ScalingtheDataTier:FunctionalSegmentation•Segmentdatabasesintofunctionalareas–Userhosts–Itemhosts–Accounthosts–Feedbackhosts–Transactionhosts–Andabout70morefunctionalcategories•Rationale–Pa