文本挖掘-话题跟踪

1文本挖掘技术(2010春)杨建武Email:[e-mail]第六章：北京大学计算机科学技术研究所话题检测跟踪技术2话题检测跟踪技术TopicDetectionandTracking(TDT)话题检测跟踪技术主题检测与追踪技术TDTtechniqueexplorefordetectingtheappearanceofnewtopicsandfortrackingthereappearanceandevolutionofthem.3相关概念4术语Event(事件)Aspecificthingthathappensataspecifictimeandplacealongwithallnecessarypreconditionsandunavoidableconsequences.Specificelections,accidents,crimesandnaturaldisastersareexamplesofevents.Eg.911事件Activity(活动)Anactivityisaconnectedsetofactionsthathaveacommonfocusorpurpose.Specificcampaigns,investigations,anddisasterreliefeffortsareexamplesofactivities.Eg.党的十七大、北京奥运会5术语Topic（话题、主题）aneventoractivity,alongwithalldirectlyrelatedeventsandactivitiesTopicEvent,1n,n≥1Sometimes,Topic≈Event(99年前)TopicExampleWHAT:35or40youngMountainHikerswerelostinanavalancheinFrancearoundthe20thofJanuary.WHERE:Orres,FranceWHEN:January4,19986术语Story(报道)Astoryisanewswirearticleorasegmentofanewsbroadcastwithacoherent(连贯的)newsfocus.Theymustcontainatleasttwoindependent,declarativeclauses(子句).7术语StorySegmentationdividingthetranscriptofanewsshowintoindividualstoriesFirstStoryDetectionrecognizingtheonset(开始)ofanewtopicinthestreamofnewsstoriesStoryLinkDetectiondecidingwhethertworandomlyselectedstoriesdiscussthesamenewstopic8术语TopicDetectiongroupingallstoriesastheyarrive,basedonthetopicstheydiscussTopicTrackingmonitoringthestreamofnewsstoriestofindadditionalstoriesonatopicthatwasidentifiedusingseveralsamplestories91ststory2ndstory3rdstory……StorySegmentationgoaltakeashowofnewsandtodetecttheboundariesbetweenstoriesautomaticallytypesdoneontheaudiosourcedirectlyusingatexttranscriptoftheshow—eitherclosedcaptionsorspeechrecognizeroutput10StorySegmentationapproacheslookforchangesinthevocabularythatisusedlookforwords,phrases,pauses,orotherfeaturesthatoccurnearstoryboundaries,toseeiftheycanfindsetsoffeaturesthatreliablydistinguishthemiddleofastoryfromitsbeginningorend,andclusteringthosesegmentstofindlargerstory-likeunits11TextTilingmethodThesimilaritycalculationcanbeillustratedasthefollowingfigurewherethex-axisisthegapnumber.12FirstStoryDetectiongoalrecognizewhenanewstopicappearsthathadnotbeendiscussedearlierDetectthatfirstnewsstorythatreportsabomb’sexplosion,avolcano’seruption,orabrewingpoliticalscandalapplicationsinteresttoinformation,security,orstockanalystswhosejobislookforneweventsthatareofsignificanceintheirarea13FirstStoryDetectionapproach(1)Reducestoriestoasetoffeatures,eitherasavectororaprobabilitydistribution.(2)Whenanewstoryarrives,itsfeaturesetiscomparedtothoseofallpaststories.(3)Ifthereissufficientdifferencethestoryismarkedasafirststory;otherwise,not.14StoryLinkDetectiongoalhandedtwonewsstories,determinewhetherornottheydiscussthesametopic今天上午近百位達悟人又回到蘭嶼核廢料貯存場，聚集在核廢桶儲存溝的草原上，等待高金素梅等..今天上午近百位達悟人又回到蘭嶼核廢料貯存場，聚集在核廢桶儲存溝的草原上，等待高金素梅等..Officialssaythatthenation'swaterresourcesshouldkeepuntiltheendofJune;andifitrainsalittlethismonth,theOfficialssaythatthenation'swaterresourcesshouldkeepuntiltheendofJune;andifitrainsalittlethismonth,theTaiwanshouldnottopushtoohardincapitalizingonthecurrentgoodrelationswiththeUSgovernment,aUSscholar.TaiwanshouldnottopushtoohardincapitalizingonthecurrentgoodrelationswiththeUSgovernment,aUSscholar.世界盃足球賽即將在五月底在日本和韓國揭幕，Ｃ組的中國大陸代表隊將在六月四日迎戰哥斯大黎加，哥世界盃足球賽即將在五月底在日本和韓國揭幕，Ｃ組的中國大陸代表隊將在六月四日迎戰哥斯大黎加，哥Therulingparty'scommitteeonreformofthelegislaturesupportsareductioninthenumberTherulingparty'scommitteeonreformofthelegislaturesupportsareductioninthenumber林義夫今天上午前往蘭嶼與當地居民面對面溝通，並達成核廢場址遷移的共識，傍晚返回台北，在行政..林義夫今天上午前往蘭嶼與當地居民面對面溝通，並達成核廢場址遷移的共識，傍晚返回台北，在行政..呂秀蓮針對台灣核廢料送往大陸處理的問題指出，，如果大陸當局確實願意伸出援手，協助台灣解決..呂秀蓮針對台灣核廢料送往大陸處理的問題指出，，如果大陸當局確實願意伸出援手，協助台灣解決..Thedam'sdeadstorageamountstoabout15,000tonnesofwater,whichisenoughtosupportusuntiltheendofJune,saidKuoYao-chi(郭瑤琪),executive-generalofthecenter.Thedam'sdeadstorageamountstoabout15,000tonnesofwater,whichisenoughtosupportusuntiltheendofJune,saidKuoYao-chi(郭瑤琪),executive-generalofthecenter.Yes/NoYes/NoYes/NoYes/No15TopicDetectiongoaltoclusterstoriesonthesametopicintobinsthecreationofbinsisanunsupervisedtask……經濟部長林義夫今天傍晚表示，政府將在一個月內成立蘭嶼核廢料遷址委員會，同時成立蘭嶼社區...經濟部長林義夫今天傍晚表示，政府將在一個月內成立蘭嶼核廢料遷址委員會，同時成立蘭嶼社區...今天上午近百位達悟人又回到蘭嶼核廢料貯存場，聚集在核廢桶儲存溝的草原上，等待高金素梅等..今天上午近百位達悟人又回到蘭嶼核廢料貯存場，聚集在核廢桶儲存溝的草原上，等待高金素梅等..陳水扁總統今天上午在總統府接見第九屆十大傑出愛心媽媽，對她們無私的奉獻，表達感佩之意…陳水扁總統今天上午在總統府接見第九屆十大傑出愛心媽媽，對她們無私的奉獻，表達感佩之意…Officialssaythatthenation'swaterresourcesshouldkeepuntiltheendofJune;andifitrainsalittlethismonth,theOfficialssaythatthenation'swaterresourcesshouldkeepuntiltheendofJune;andifitrainsalittlethismonth,theTaiwanshouldnottopushtoohardincapitalizingonthecurrentgoodrelationswiththeUSgovernment,aUSscholar.TaiwanshouldnottopushtoohardincapitalizingonthecurrentgoodrelationswiththeUSgovernment,aUSscholar.世界盃足球賽即將在五月底在日本和韓國揭幕，Ｃ組的中國大陸代表隊將在六月四日迎戰哥斯大黎加，哥世界盃足球賽即將在五月底在日本和韓國揭幕，Ｃ組的中國大陸代表隊將在六月四日迎戰哥斯大黎加，哥Therulingparty'scommitteeonreformofthelegislaturesupportsareductioninthenumberTherulingparty'scommitteeonreformofthelegislaturesupportsareductioninthenumber16T