Dzone2018年大数据技术指南英文版201838页

THE2018DZONEGUIDETOBigDataSTREAMPROCESSING,STATISTICS,&SCALABILITYVOLUMEVBROUGHTTOYOUINPARTNERSHIPWITHTHEDZONEGUIDETOBIGDATA:STREAMPROCESSING,STATISTICS,ANDSCALABILITYDZONE.COM/GUIDESDZONE.COM/GUIDESPAGE2OF35THEDZONEGUIDETOBIGDATA:STREAMPROCESSING,STATISTICS,ANDSCALABILITYDZONE.COM/GUIDESDearReader,Ifirstheardtheterm“BigData”almostadecadeago.Atthattime,itlookedlikeitwasnothingnew,andourdatabaseswouldjustbeup-gradedtohandlesomemoredata.Nobigdeal.Butsoon,itbecameclearthattraditionaldatabaseswerenotdesignedtohandleBigData.Theterm“BigData”hasmoredimensionsthanjust“somemoredata.”Itencompassesbothstructuredandunstructureddata,fastmovingandhistoricaldata.Now,withtheseelementsaddedtothedata,someoftheotherproblemssuchasdatacontextualization,datavalidity,noise,andabnormalityinthedatabecamemoreprominent.Sincethen,BigDatatechnologieshasgonethroughseveralphasesofdevel-opmentandtransformation,andtheyaregraduallymaturing.Atermthatwasconsideredasafadandatechnologyecosystemthatwasconsideredaluxuryareslowlyestablishingthemselvesasnecessaryneedsfortoday’sbusinessactivities.BigDataisthenewcompetitiveadvantageanditmattersforourbusinesses.Themoreweprogressandthemoreautomationweimplement,dataisalwaysgoingtotransformandgrow.Blockchaintechnologies,Cloud,andIoTareaddingnewdimensionstotheBigDatatrend.HatsoffthedeveloperswhoarecontinuallyinnovatingandcreatingnewBigDataStorageandAnalyticsapplicationstoderivevalueoutofthisdata.Thefast-paceddevelopmenthasmadeiteasierforustotamefast-grow-ingmassivedataandintegrateourexistingEnterpriseITinfrastructurewiththesenewdatasources.ThesesuccessesaredrivenbybothEn-terprisesandOpenSourcecommunities.Withoutopensourceproj-ectslikeApacheHadoop,ApacheSpark,andKafka,tonameafew,thelandscapewouldhavebeentirelydifferent.TheuseofMachineLearn-ingandDatavisualizationmethodspackagedforanalyzingBigDataisalsomakinglifeeasierforanalystsandmanagement.However,westillhearthefailureofanalyticsprojectsmoreoftenthanthesuccess-es.Thereareseveralreasonswhy.So,webringyouthisguide,wherethesearticleswrittenbyDZonecontributorsaregoingtoprovideyouwithmoresignificantinsightsintothesetopics.TheBigDataguideisanattempttohelpreadersdiscoverandhelpun-derstandthecurrentlandscapeoftheBigDataecosystem,wherewestand,andwhatamazinginsightsandapplicationspeoplearediscov-eringinthisspace.Wewishthateveryonewhoreadsthisguidefindsitworthyandinformative.Happyreading!BySibanjanDasBUSINESSANALYTICS&DATASCIENCECONSULTANT&DZONEZONELEADERExecutiveSummaryBYMATTWERNER_______________________________3KeyResearchFindingsBYG.RYANSPAIN_______________________________4TakeBigDatatotheNextLevelwithBlockchainNetworksBYARJUNACHALA______________________________6SolvingDataIntegrationatStitchFixBYLIZBENNETT_______________________________10Checklist:TenTipsforEnsuringYourNextDataAnalyticsProjectisaSuccessBYWOLFRUZICKA,______________________________13Infographic:BigDataRealizationwithSanitation______14WhyDevelopersShouldBetBigonStreamingBYJONASBONÉR_______________________________16IntroductiontoBasicStatisticsMeasurementsBYSUNILKAPPAL_______________________________20DivingDeeperintoBigData_____________________23ExecutiveInsightsontheStateofBigDataBYTOMSMITH_________________________________24BigDataSolutionsDirectory____________________26Glossary__________________________________36DZONEIS...PRODUCTIONCHRISSMITH,DIR.OFPRODUCTIONANDREPOWELL,SR.PRODUCTIONCOORDINATORG.RYANSPAIN,PRODUCTIONCOORDINATORASHLEYSLATE,DESIGNDIR.BILLYDAVIS,PRODUCTIONASSISSTANTMARKETINGKELLETATKINSON,DIR.OFMARKETINGLAURENCURATOLA,MARKETINGSPECIALISTKRISTENPAGÀN,MARKETINGSPECIALISTNATALIEIANNELLO,MARKETINGSPECIALISTJULIANMORRIS,MARKETINGSPECIALISTBUSINESSRICKROSS,CEOMATTSCHMIDT,PRESIDENTJESSEDAVIS,EVPSALESMATTO’BRIAN,DIR.OFBUSINESSDEV.ALEXCRAFTS,DIR.OFMAJORACCOUNTSJIMHOWARD,SRACCOUNTEXECUTIVEJIMDYER,ACCOUNTEXECUTIVEANDREWBARKER,ACCOUNTEXECUTIVEBRIANANDERSON,ACCOUNTEXECUTIVERYANMcCOOK,ACCOUNTEXECUTIVECHRISBRUMFIELD,SALESMANAGERTOMMARTIN,ACCOUNTMANAGERJASONBUDDAY,ACCOUNTMANAGEREDITORIALCAITLINCANDELMO,DIR.OFCONTENT&COMMUNITYMATTWERNER,PUBLICATIONSCOORD.MICHAELTHARRINGTON,CONTENT&COMMUNITYMANAGERKARAPHELPS,CONTENT&COMMUNITYMANAGERMIKEGATES,SR.CONTENTCOORD.SARAHDAVIS,CONTENTCOORD.TOMSMITH,RESEARCHANALYSTJORDANBAKER,CONTENTCOORD.ANNEMARIEGLEN,CONTENTCOORD.ANDRELEE-MOYE,CONTENTCOORD.TableofContentsTHEDZONEGUIDETOBIGDATA:STREAMPROCESSING,STATISTICS,ANDSCALABILITYDZONE.COM/GUIDESPAGE3OF35THEDZONEGUIDETOBIGDATA:STREAMPROCESSING,STATISTICS,ANDSCALABILITYClassically,BigDatahasbeendefinedbythreeV’s:Volume,orhowmuchdatayouhave;Velocity,orhowfastdataiscollected;andVariety,orhowheterogeneousthedatasetis.AsmovementsliketheInternetofThingsprovideconstantstreamsofdatafromhardware,andAIinitiativesrequiremassivedatasetstoteachmachinestothink,thewayinwhichBigDataisstoredandutilizedcontinuestochange.Tofindouthowdevelopersareapproachingthesechallenges,weasked540DZonememberstotellusaboutwhattoolsthey’reusingtoovercomethem,andhowtheirorganizationsmeasuresuccessfulimplementations.THEPAINSOFTHETHREEV’SDATAOfseveraldatasourc