基于Hadoop平台的亿贝用户邮件数据分析(苏立)

eBay  M2M  Email  Analysis  on  Hadoop  Pla4orm  ForestSu,DanielZhangOctober  2011  eBay  Inc.  Confiden.al  TheBirthofeBay...  eBay  Inc.  Confiden.al  2  ...soldfor$14.83USDStartedwithaBrokenLaserPointer...AuctionWebwasbornontheLaborDayweekendinSeptember1995PierreOmidyar$30eBayFoundereBay  Fact  –  15  Years  A=er  …  eBay  Inc.  Confiden.al  3  450+MillionRegisteredUsersOver2BillionPhotos220+MillionActiveItemListingforsale50,000Categories2PetabytesStored25PetabytesProcesseddaily300+Featuresperquarter100,000linesofcoderolledoutevery2weeks48BillionSQLCallsPerday5.5BillionAPICallsPermonth4.4GBSourceCodeGlobalPresentsIn33InternationalMarkets10+MillionNewItemsAddedPerDay$2,000+USDTradingValuePerSecond4VelocityoftradingOnanaveragedayoneBay…ADiamondRingissoldeverytwominutes5OnanaveragedayoneBay…Over3600MP3playersaresoldVelocityoftrading6VelocityoftradingOnanaveragedayoneBay…Over300stampsaresoldeveryhour7VelocityoftradingOnanaveragedayoneBay…AnautomobileissoldeveryminuteHadoopontheLandscape  eBay  Inc.  Confiden.al  8  Dual  LoadDual  LoadDual  LoadPrimary/SecondaryQuery  DirectorDual  LoadTransactional  DatabasesDeep  SystemResearch  PlatformUnstructured  DataStructured  DataStructured  DataDual  LoadSocPArcETL  LoadBusiness  UsersDual  LoadStructured/Unstructured  DataDataFeedDual  LoadUnstructured  DataDataFeedDual  LoadSearch  AnalystsHadoop  Data  Intensive  PlatformNode:  500+Disk  Space:  Approx.  4.25PBComponent:  HDFS/HiveHardware:  Commodity  HardwareKnowledgeFeedbackKnowledgeFeedbackeBay  Email  Universe  • Members  write  to  each  other  (“M2M”)  • Members  write  to  eBay  • eBay  writes  to  Members  -­‐ Millions  of  times  per  day!  • Where  are  these  messages?  o In  My  Messages:  Messages  through  eBay  o In  a  user’s  private  system:  Messages  outside  of  eBay  eBay  Inc.  Confiden.al  9  Single  User  Study  • What  does  a  typical  user  send  /  receive?  • Follow  a  single  user  for  50  days  during  Oct/Nov  2010  • User  ProNile:  ― User  subscribed  to  daily  deal  ― User  was  member  of:  § eBay  Bucks  § eBay  VIP  § PayPal  Advantage  ― User  is  mainly  a  buyer  but  also  a  casual  seller  eBay  Inc.  Confiden.al  10  Single  User  Study:  Volume  eBay  Inc.  Confiden.al  11  Count  Grouping  Percentage  Percentage  excl  Daily  Deal  Daily  Average  53  eBay  Daily  Deal  25.1%      1.06  35  eBay  Marke.ng  16.6%  22.2%  0.70  16  PayPal  Marke.ng  7.6%  10.1%  0.32  87  Transac.on  Related  41.2%  55.1%  1.74  8  Watched  item  ending  3.8%  5.1%  0.16  9  Selling  ac.vity  4.3%  5.7%  0.18  3  Other  1.4%  1.9%  0.06  211  Total  4.22  Single  User  Study:  Visibility  eBay  Inc.  Confiden.al  12  Count  In  MyMessages  Percentage  Percentage  excl.  Daily  Deal  53  No  (Daily  Deal)  25.1%      124  No  58.8%  78.5%  26  Inbox  12.3%  16.5%  8  Sent  3.8%  5.1%  211  Total  Only21.5%arevisiblefromwithineBayEmail  Data:  Where  is  it?  eBay  Inc.  Confiden.al  13  Emailsarestoredinamixedsystem:-MetadatainDW-MessagebodyonNASbefore10/2010,onGrid(LowCostStorage)since10/2010Emaildatanotminableincurrentstorage-QueriesonEmailSubjectsinTeradatasufferCPUskew-Emailbodyinindividualfiles,doesnotscaleatalleBay  Sample  Email  eBay  Inc.  Confiden.al  14  Major  Challenges  • Data  is  large  ― Data  is  in  individual  message  Niles:  § millions  of  Niles  per  day,    1  billion  Niles  per  year  • Data  contains  Personally  IdentiNiable  Information  (PII)  ― Hadoop  current  release  does  not  support  effective  § User  Access  Permissions  -­‐  Need  Encryption    eBay  Inc.  Confiden.al  15  Challenges  Answered  eBay  Inc.  Confiden.al  16  Thiscopiedover1billionfiles,about15TBcompressedHadoop  Data  Storage  Use  eBay  Inc.  Confiden.al  17  Year  Month  Raw  M2M  GigaBytes  Clean  M2M  GigaBytes  2009  9-­‐12  4,060  242  2010  1-­‐12  14,959  944  2011  1-­‐7  10,249  1,282  Total      29,268  2,467  2011  1  1,274  85  2011  2  1,271  81  2011  3  1,650  233  2011  4  1,484  219  2011  5  1,560  230  2011  6  1,506  217  2011  7  1,505  218  eBay  M2M  Emails  /  Year  eBay  Inc.  Confiden.al  18  01,000,0002,000,0003,000,0004,000,0005,000,0006,000,0007,000,0008,000,0009,000,0007-1-20108-1-20109-1-201010-1-201011-1-201012-1-20101-1-20112-1-20113-1-20114-1-20115-1-20116-1-2011M2M1yeareBay  System  Emails  eBay  Inc.  Confiden.al  19  05,000,00010,000,00015,000,00020,000,00025,000,00030,000,00035,000,0003-1-20114-1-20115-1-20116-1-2011eBaySystemeBay  System  Emails  eBay  Inc.  Confiden.al  20  20%15%14%8%8%5%4%4%4%3%3%12%AgreementUpdateMarketingOffersPaymentsSeller'sReturnPolicyeBayBucksSurveysUPIPassword/Email/UseridCancellingPurchaseWelcomeeBay/MyMessagesMiscellaneouseBay  System  Emails  eBay  Inc.  Confiden.al  21  0.0%5.0%10.0%15.0%20.0%25.0%30.0%35.0%40.0%02468101214161820MessagesperconversationHowmanyemailsperconversation?eBay  Emails  eBay  Inc.  Confiden.al  22  InanM2Mconversation,howlongforthefirstreply?0.0%5.0%10.0%15.0%20.0%25.0%30.0%05101520253035Pre-