20th.陈晓明-百度海量日志分析架构及处理经验分享

Òu ~4²«4Ģ¤īăÈlÜoÒuVØ´±ė@Ģ¤īSÒ[e-mail] ò¨Ĭěk ~4²VªČÜÒu ~4²†ĝBÜÈ&LSPqIÈ&DISQLþø‚ëĞàLOG ò¨Ĭěk
¯ ~ 46.70.93.94--[11/Nov/2011:11:11:11-1100]GET/book/1984.htmlHTTP/1.1“4042326=1984&rsv_bp=0&rsv_spt=3&inputT=947Mozilla/5.0(iPad;U;CPUiPhoneOS3_2likeMacOSX;en-us)AppleWebKit/531.21.10(KHTML,likeGecko)Version/4.0.4Mobile/7B314Safari/531.21.10“÷щ——TU°É 46.70.93.94-- [11/Nov/2011:11:11:11-1100] GET/book/1984.htmlHTTP/1.1 404 2326 =1984&rsv_bp=0&rsv_spt=3&inputT=947 Mozilla/5.0(iPad;U;CPUiPhoneOS3_2likeMacOSX;en-us)AppleWebKit/531.21.10(KHTML,likeGecko)Version/4.0.4Mobile/7B314Safari/531.21.10“÷щ——Ćć°É 46.70.93.94-- [11/Nov/2011:11:11:11-1100] GET/book/1984.htmlHTTP/1.1 404 2326 =1984&rsv_bp=0&rsv_spt=3&inputT=947 Mozilla/5.0(iPad;U;CPUiPhoneOS3_2likeMacOSX;en-us)AppleWebKit/531.21.10(KHTML,likeGecko)Version/4.0.4Mobile/7B314Safari/531.21.10“÷щ——ÑêÞ 46.70.93.94-- [11/Nov/2011:11:11:11-1100] GET/book/1984.htmlHTTP/1.1 404 2326 =1984&rsv_bp=0&rsv_spt=3&inputT=947 Mozilla/5.0(iPad;U;CPUiPhoneOS3_2likeMacOSX;en-us)AppleWebKit/531.21.10(KHTML,likeGecko)Version/4.0.4Mobile/7B314Safari/531.21.10÷òm——üĞĚ 46.70.93.94-- [11/Nov/2011:11:11:11-1100] GET/book/1984.htmlHTTP/1.1 404 2326 =1984&rsv_bp=0&rsv_spt=3&inputT=947 Mozilla/5.0(iPad;U;CPUiPhoneOS3_2likeMacOSX;en-us)AppleWebKit/531.21.10(KHTML,likeGecko)Version/4.0.4Mobile/7B314Safari/531.21.10÷òm——¢Ñ 46.70.93.94-- [11/Nov/2011:11:11:11-1100] GET/book/1984.htmlHTTP/1.1 404 2326 =1984&rsv_bp=0&rsv_spt=3&inputT=947 Mozilla/5.0(iPad;U;CPUiPhoneOS3_2likeMacOSX;en-us)AppleWebKit/531.21.10(KHTML,likeGecko)Version/4.0.4Mobile/7B314Safari/531.21.10÷òm——Hс 46.70.93.94-- [11/Nov/2011:11:11:11-1100] GET/book/1984.htmlHTTP/1.1 404 2326 =1984&rsv_bp=0&rsv_spt=3&inputT=947 Mozilla/5.0(iPad;U;CPUiPhoneOS3_2likeMacOSX;en-us)AppleWebKit/531.21.10(KHTML,likeGecko)Version/4.0.4Mobile/7B314Safari/531.21.10Ô¶——¥\T©=Ô¶——¥\TĄĜLOG ò¨Ĭěk ~4²VªČÜÒu ~4²†ĝBÜÈ&LSPqIÈ&DISQLþø‚ëĞà ~4²VªČÜ ~4²VªČÜV
 7
 h
q%J
 `B
 iH
bS
 q*bS
 Ce
K
 3^
 !^
 rV
 
LOG ò¨Ĭěk ~4²VªČÜÒu ~4²†ĝBÜÈ&LSPqIÈ&DISQLþø‚ëĞàÒu ~4²†ĝBÜÑ› P/M;0B@ D/co N? WTJ WjX Ek]
›”ĚÓĞħ UM6l/fa Gj] S)M-/+I K2[*984  :Mj] [*984 ='S)84 5,84 (mMLOG ò¨Ĭěk ~4²VªČÜÒu ~4²†ĝBÜÈ&LSPqIÈ&DISQLþø‚ëĞàÈ&LSPqIæì´±qIUIîĉ»xá@»x
DQuery»x
Y­»x
LOG ò¨Ĭěk ~4²VªČÜÒu ~4²†ĝBÜÈ&LSPqIÈ&DISQLþø‚ëĞàÈ&DISQLþø!ĴĠÝËüĞĚOsNĚìù
 ‹ó¿ĩ ĀE ~›” ĒE2_Urlĭ_Res(sN›)6 î-
3›ij_Url E2_Site Ѿ7ôċxČÊ2ĠÝËӛ” ’ÝË4éijÁé%Ĵ ùâüĞĚ isN›ÂO Ċ2›”ijÁó£
JSON›”!ĴDQuery×
 ĀE ~›” ĒE2_Urlĭ_Res(sN›)6 î-
3›ij_Url E2_S Ѿ7ôċxČÊ2ĠÝËӛ ’ÝË4éijÁé%Ĵ ùâüĞĚ isN›ÂO Ċ2›”ijÁó£
JSON›”PHP-CallbackC-callback1ÌË ÑPHPôċÓåSQLĔĉ(ĥpáç) hõŠ¨SQLâ_ÓM/R4nxeÎĴ 4éĭôđ•ĭó6ČÊĭģK˜ĭĊ&Ċ2ºxć“ ēČđí3›āÑôċDAG›”Ç òïý
ĈˆZĈMapReduce Hïý@¬ù∛”ÇR ÑĔĉĦtðĥSQLĦtôċĔĉ ™‘PHPòd3›ıáÆIJ ™‘C++òd3›ıL¸áÆ+ĪšIJOC-RuntimeNEW! 'òĪš,aãÏıRAII+,aÃIJ vfĂY:ıCopyOnWriteIJ `À˜ïý›é˜ijŸ`+µŒıschema–gIJ C++ӁñijPHPÓwDİ
XÏġÀ
Qfa%J
pfaZd

j]
;ÞXÏ ŽÑ‰î-ÓùâĔĉïýîýÜtÏ÷Ó ğ×ıþÄ·ĭ›”ÇRIJ ;Þ×čó
ĕijÐë³£ ğ× Õyîý« Óparser  ğ×ÑJSONôÙ[{cmd:load“,path:nullusing:SchemaReaderfrom:17options:{max_item_in_mem“:100000}include:[25]},{cmd:filter……},{cmd:join……},…………]
SQLþø•Gĵ[{cmd:load“,path:nullusing:SchemaReaderfrom:17options:{max_item_in_mem“:100000}include:[25]},{cmd:filter……},{cmd:join……},…………] 

 ğþøXÏ f›”ÇRZ¼ßF“ ¾ö? i›”ÇRF†cœÓžMíXÏӛ”ÇR â_¦“ ieÎY­Óâ_ߦ“†Zá@â_ ? f›”ÇRĐóJÚ?ij‹óšÍ—Ī ġÀ54(HĒ) 54ZMapReduce‹óġÀ Schema–gĭ`À$Û̖g –gÁ
â_2ÓôschemaijC`À$ÛĚ ×ІıC++ĭPHPĭDOTIJ І־H‹óÓ×
 
 
â_¦“














Schema–gĭ¶–g




































? Z=Kr ßâ_Kr Combiner? CachedCombiner? LkeyJoinKr? ()_ôċx—E ¹}… 0jĈ›ĭ0jI/Oĭ0jęYù⛔ÇRІ×І PHP C++ DOT @¬/MapReducečó¡8^?
雔XÏXÏ
é Ó
¯›”ûzıZ¼āÑIJë®
雔XÏ 















 



Processor——Pipes&FilterĘф/ 4²Üt›ĚWĝ 4²ÜtĊ&ĚOO..UU##

;;

;;

$$nn

$$nnLLFF

RYg




D/




&AYg




OO..UU##

FF

RYg

D/

&AYg

 LSPqIщ›Ě__\\

99mm

FF













19



LOG ò¨Ĭěk ~4²VªČÜÒu ~4²†ĝBÜÈ&LSPqIÈ&DISQLþø‚ëĞà‚ëĞà‚ëĞà  ~Ó# ÷щ●÷òm  ~4²VªČÜ —EĊ●÷²ČÊ●ùâ●ÑıôĭRôĭQĨè…IJ Òu ~4²†ĝBÜ È&LSPqI qI´±●qIUI●Úîĉ»x È&DISQLþø 
!_●1Ì