数据仓库与数据挖掘

August1,2019DataMining:ConceptsandTechniques1数据仓库与数据挖掘August1,2019DataMining:ConceptsandTechniques2DataMining:ConceptsandTechniques—Chapter1——Introduction—JiaweiHanandMichelineKamberDepartmentofComputerScienceUniversityofIllinoisatUrbana-Champaignwww.cs.uiuc.edu/~hanj©2006JiaweiHanandMichelineKamber.Allrightsreserved.August1,2019DataMining:ConceptsandTechniques3August1,2019DataMining:ConceptsandTechniques4August1,2019DataMining:ConceptsandTechniques5Chapter1.IntroductionMotivation:Whydatamining?Whatisdatamining?DataMining:Onwhatkindofdata?DataminingfunctionalityAreAllthe“Discovered”PatternsInteresting?ClassificationofdataminingsystemsMajorissuesindataminingOverviewofthecourseSupplementaryLectureSlidesAugust1,2019DataMining:ConceptsandTechniques6WhyDataMining?TheExplosiveGrowthofData:fromterabytestopetabytesDatacollectionanddataavailabilityAutomateddatacollectiontools,databasesystems,Web,computerizedsocietyMajorsourcesofabundantdataBusiness:Web,e-commerce,transactions,stocks,…Science:Remotesensing,bioinformatics,scientificsimulation,…Societyandeveryone:news,digitalcameras,YouTubeWearedrowningindata,butstarvingforknowledge!“Necessityisthemotherofinvention”—Datamining—AutomatedanalysisofmassivedatasetsAugust1,2019DataMining:ConceptsandTechniques7EvolutionofSciencesBefore1600,empiricalscience1600-1950s,theoreticalscienceEachdisciplinehasgrownatheoreticalcomponent.Theoreticalmodelsoftenmotivateexperimentsandgeneralizeourunderstanding.1950s-1990s,computationalscienceOverthelast50years,mostdisciplineshavegrownathird,computationalbranch(e.g.empirical,theoretical,andcomputationalecology,orphysics,orlinguistics.)ComputationalSciencetraditionallymeantsimulation.Itgrewoutofourinabilitytofindclosed-formsolutionsforcomplexmathematicalmodels.1990-now,datascienceThefloodofdatafromnewscientificinstrumentsandsimulationsTheabilitytoeconomicallystoreandmanagepetabytesofdataonlineTheInternetandcomputingGridthatmakesallthesearchivesuniversallyaccessibleScientificinfo.management,acquisition,organization,query,andvisualizationtasksscalealmostlinearlywithdatavolumes.Dataminingisamajornewchallenge!JimGrayandAlexSzalay,TheWorldWideTelescope:AnArchetypeforOnlineScience,Comm.ACM,45(11):50-54,Nov.2002August1,2019DataMining:ConceptsandTechniques8EvolutionofDatabaseTechnology1960s:Datacollection,databasecreation,IMSandnetworkDBMS1970s:Relationaldatamodel,relationalDBMSimplementation1980s:RDBMS,advanceddatamodels(extended-relational,OO,deductive,etc.)Application-orientedDBMS(spatial,scientific,engineering,etc.)1990s:Datamining,datawarehousing,multimediadatabases,andWebdatabases2000sStreamdatamanagementandminingDatamininganditsapplicationsWebtechnology(XML,dataintegration)andglobalinformationsystemsAugust1,2019DataMining:ConceptsandTechniques9WhyNotTraditionalDataAnalysis?TremendousamountofdataAlgorithmsmustbehighlyscalabletohandlesuchastera-bytesofdataHigh-dimensionalityofdataMicro-arraymayhavetensofthousandsofdimensionsHighcomplexityofdataDatastreamsandsensordataTime-seriesdata,temporaldata,sequencedataStructuredata,graphs,socialnetworksandmulti-linkeddataHeterogeneousdatabasesandlegacydatabasesSpatial,spatiotemporal,multimedia,textandWebdataSoftwareprograms,scientificsimulationsAugust1,2019DataMining:ConceptsandTechniques10WhyDataMining?—PotentialApplicationsDataanalysisanddecisionsupportMarketanalysisandmanagementTargetmarketing,customerrelationshipmanagement(CRM),marketbasketanalysis,crossselling,marketsegmentationRiskanalysisandmanagementForecasting,customerretention,improvedunderwriting,qualitycontrol,competitiveanalysisFrauddetectionanddetectionofunusualpatterns(outliers)OtherApplicationsTextmining(newsgroup,email,documents)andWebminingStreamdataminingBioinformaticsandbio-dataanalysis2019年8月1日星期四DataMining:ConceptsandTechniques11WhatIsDataMining?Datamining(knowledgediscoveryfromdata)Extractionofinteresting(non-trivial,implicit,previouslyunknownandpotentiallyuseful)patternsorknowledgefromhugeamountofdatathenon-trivialprocessofidentifyingvalid,novel,potentiallyusefulandultimatelyunderstandablepatternsindata(从数据集中识别出有效的、新颖的、潜在有用的，以及最终可理解的模式的非平凡过程)AlternativenamesKnowledgediscovery(mining)indatabases(KDD),knowledgeextraction,data/patternanalysis,dataarcheology,datadredging,etc.August1,2019DataMining:ConceptsandTechniques12KnowledgeDiscovery(KDD)ProcessDatamining—coreofknowledgediscoveryprocessDataCleaningDataIntegrationDatabasesDataWarehouseTask-relevantDataSelectionDataMiningPatternEvaluationAugust1,2019DataMining:ConceptsandTechniques13KDDProcess:SeveralKeyStepsLearningtheapplicationdomainrelevantpriorknowledgeandgoalsofapplicationCreatingatargetdataset:dataselectionDatacleaningandpreprocessing:(maytake60%ofeffort!)DatareductionandtransformationFindusefulfeatures,dimensionality/variablereduction,invariantrepresentationChoosingfunctionsofdataminingsummarization,classification,regression,association,clusteringChoosingtheminingalgorithm(s)Datamining:searchfor