strata_spark_streaming

SparkStreamingLarge-scalenear-real-timestreamprocessingTathagataDas(TD)UCBerkeleyUCBERKELEYWhatisSparkStreaming?Frameworkforlargescalestreamprocessing-Scalesto100sofnodes-Canachievesecondscalelatencies-IntegrateswithSpark’sbatchandinteractiveprocessing-Providesasimplebatch-likeAPIforimplementingcomplexalgorithm-CanabsorblivedatastreamsfromKafka,Flume,ZeroMQ,etc.MotivationManyimportantapplicationsmustprocesslargestreamsoflivedataandprovideresultsinnear-real-time-Socialnetworktrends-Websitestatistics-Intrustiondetectionsystems-etc.RequirelargeclusterstohandleworkloadsRequirelatenciesoffewsecondsNeedforaframework……forbuildingsuchcomplexstreamprocessingapplicationsButwhataretherequirementsfromsuchaframework?RequirementsScalabletolargeclustersSecond-scalelatenciesSimpleprogrammingmodelCasestudy:Conviva,Inc.Real-timemonitoringofonlinevideometadata-HBO,ESPN,ABC,SyFy,…TwoprocessingstacksCustom-builtdistributedstreamprocessingsystem•1000scomplexmetricsonmillionsofvideosessions•RequiresmanydozensofnodesforprocessingHadoopbackendforofflineanalysis•Generatingdailyandmonthlyreports•SimilarcomputationasthestreamingsystemCustom-builtdistributedstreamprocessingsystem•1000scomplexmetricsonmillionsofvideossessions•RequiresmanydozensofnodesforprocessingHadoopbackendforofflineanalysis•Generatingdailyandmonthlyreports•SimilarcomputationasthestreamingsystemCasestudy:XYZ,Inc.AnycompanywhowantstoprocesslivestreamingdatahasthisproblemTwicetheefforttoimplementanynewfunctionTwicethenumberofbugstosolveTwicetheheadacheTwoprocessingstacksRequirementsScalabletolargeclustersSecond-scalelatenciesSimpleprogrammingmodelIntegratedwithbatch&interactiveprocessingStatefulStreamProcessingTraditionalstreamingsystemshaveaevent-drivenrecord-at-a-timeprocessingmodel-Eachnodehasmutablestate-Foreachrecord,updatestate&sendnewrecordsStateislostifnodedies!Makingstatefulstreamprocessingbefault-tolerantischallengingmutablestatenode1node3inputrecordsnode2inputrecords9ExistingStreamingSystemsStorm-Replaysrecordifnotprocessedbyanode-Processeseachrecordatleastonce-Mayupdatemutablestatetwice!-Mutablestatecanbelostduetofailure!Trident–Usetransactionstoupdatestate-Processeseachrecordexactlyonce-Perstatetransactionupdatesslow10RequirementsScalabletolargeclustersSecond-scalelatenciesSimpleprogrammingmodelIntegratedwithbatch&interactiveprocessingEfficientfault-toleranceinstatefulcomputationsSparkStreaming12DiscretizedStreamProcessingRunastreamingcomputationasaseriesofverysmall,deterministicbatchjobs13SparkSparkStreamingbatchesofXsecondslivedatastreamprocessedresultsChopupthelivestreamintobatchesofXsecondsSparktreatseachbatchofdataasRDDsandprocessesthemusingRDDoperationsFinally,theprocessedresultsoftheRDDoperationsarereturnedinbatchesDiscretizedStreamProcessingRunastreamingcomputationasaseriesofverysmall,deterministicbatchjobs14SparkSparkStreamingbatchesofXsecondslivedatastreamprocessedresultsBatchsizesaslowas½second,latency~1secondPotentialforcombiningbatchprocessingandstreamingprocessinginthesamesystemExample1–GethashtagsfromTwittervaltweets=ssc.twitterStream(Twitterusername,Twitterpassword)DStream:asequenceofRDDrepresentingastreamofdatabatch@t+1batch@tbatch@t+2tweetsDStreamstoredinmemoryasanRDD(immutable,distributed)TwitterStreamingAPIExample1–GethashtagsfromTwittervaltweets=ssc.twitterStream(Twitterusername,Twitterpassword)valhashTags=tweets.flatMap(status=getTags(status))flatMapflatMapflatMap…transformation:modifydatainoneDstreamtocreateanotherDStreamnewDStreamnewRDDscreatedforeverybatchbatch@t+1batch@tbatch@t+2tweetsDStreamhashTagsDstream[#cat,#dog,…]Example1–GethashtagsfromTwittervaltweets=ssc.twitterStream(Twitterusername,Twitterpassword)valhashTags=tweets.flatMap(status=getTags(status))hashTags.saveAsHadoopFiles(hdfs://...)outputoperation:topushdatatoexternalstorageflatMapflatMapflatMapsavesavesavebatch@t+1batch@tbatch@t+2tweetsDStreamhashTagsDStreameverybatchsavedtoHDFSJavaExampleScalavaltweets=ssc.twitterStream(Twitterusername,Twitterpassword)valhashTags=tweets.flatMap(status=getTags(status))hashTags.saveAsHadoopFiles(hdfs://...)JavaJavaDStreamStatustweets=ssc.twitterStream(Twitterusername,Twitterpassword)JavaDstreamStringhashTags=tweets.flatMap(newFunction...{})hashTags.saveAsHadoopFiles(hdfs://...)FunctionobjecttodefinethetransformationFault-toleranceRDDsarerememberthesequenceofoperationsthatcreateditfromtheoriginalfault-tolerantinputdataBatchesofinputdataarereplicatedinmemoryofmultipleworkernodes,thereforefault-tolerantDatalostduetoworkerfailure,canberecomputedfrominputdatainputdatareplicatedinmemoryflatMaplostpartitionsrecomputedonotherworkerstweetsRDDhashTagsRDDKeyconceptsDStream–sequenceofRDDsrepresentingastreamofdata-Twitter,HDFS,Kafka,Flume,ZeroMQ,AkkaActor,TCPsocketsTransformations–modifydatafromonDStreamtoanother-StandardRDDoperations–map,countByValue,reduce,join,…-Statefuloperations–window,countByValueAndWindow,…OutputOperations–senddatatoexternalentity-saveAsHadoopFiles–savestoH