3-5-美团大数据平台架构实践-谢语宸

[e-mail]•11,121314••••BI…-BI••Binlog••850+•+•••••1100+••,•••42P+•150K/•2500+,3•16KETL•••SLA•BIBI-BI-201120122013201420152016“”PHPSQLETLSQLHadoopETLBinlogHadoop2.0YARNSparkHadoop&SLAOLAP201120122013201420152016“”PHPSQLETLSQLHadoopETLBinlogHadoop2.0YARNSparkHadoop&SLAOLAP201120122013201420152016“”PHPSQLETLSQLHadoopETLBinlogHadoop2.0YARNSparkHadoop&SLAOLAP201120122013201420152016“”PHPSQLETLSQLHadoopETLBinlogHadoop2.0YARNSparkHadoop&SLAOLAP201120122013201420152016“”PHPSQLETLSQLHadoopETLBinlogHadoop2.0YARNSparkHadoop&SLAOLAP201120122013201420152016“”PHPSQLETLSQLHadoopETLBinlogHadoop2.0YARNSparkHadoop&SLAOLAP•HadoopNameSpace••OLAPHadoop••15,500•9•1561000,15121500•,Hadoop•Hadoop•,•Hadoop,•Hadoop•Appcontainer•DataNode•HDFSpipelineHadoop••,NameSpace••YARN,•HDFSaddBlock,clientDataNodepipeline.client.•HDFSBalancer•Balancer,Block/Hadoop-24642061111101894198524600500100015002000250030002015.032015.052015.072015.092015.112016.012016.03cqdxrzallHadoop••,•,•••Hadoop/Spark•,,•,•/,•Spark,/•••,,••ipythonnotebook+sparkzeppelin•,bug/&•,usercode…gitpush…托管平台注册…Hadoop作业管理Hadoop作业编译、执⾏…OLAP••,50•,20•TP993S••OLAP••Presto/Hive/SparkonORCFile•HivegroupingsetHBase+•Druid•ElasticSearch•KylinOLAP••,,,,Kylin,•StarSchemaBenchmark,OLAP,•,Kylin-OLAPStarSchemaBenchmarkOLAP••PrestoKylin1.3Kylin1.5Druid•BI7•7-1~2•3,TP95%1s,2••,•,•,•••,•,••&•patch••&