Statistical models of language learning and use

InCognitiveSiene26:3,2002StatistialModelsofLanguageLearningandUseMarkJohnsonStefanRiezlerBrownUniversityXeroxPARCMarkJohnsonBrown.eduriezlerpar.xerox.om1IntrodutionThispapersummarizesourreentworkindevelopingstatistialmodelsoflan-guagewhihareompatiblewiththekindsoflinguististruturespositedbyurrentlinguistitheories.Inaseriesofpaperswehavedevelopedtoolsforesti-matingor\learningsuhmodelsfromdata(Johnsonetal.,1999;JohnsonandRiezler,2000;Riezleretal.,2000)andthispaperprovidesahigh-leveloverviewofboththegeneralapproahandthemethodswedeveloped.Turningtotheoretialresultsonlearning,itseemsthatstatistiallearn-ersmaybemorepowerfulthannon-statistiallearners.Forexample,whileGold’sfamousresultsshowedthatneithernitestatenorontextfreelan-guagesanbelearntfrompositiveexamplesalone(Gold,1967),itturnsoutthatprobabilistiontextfreelanguagesanbelearntfrompositiveexamplesalone(Horning,1969).Informally,alassoflanguagesmaybestatistiallylearnableeventhoughitsategorialounterpartisnotbeausethestatistiallearningframeworkmakesstrongerassumptionsaboutthetrainingdata(i.e.,itisdistributedaordingtosomeprobabilistigrammarfromthelass)andaeptsaweakerriterionforsuessfullearning(onvergeneinprobability).Statistisprovidesthetheoryofoptimallearnersandoptimalomprehenders(optimalinaninformation-theoretisense)whihserveasidealizationsof,andupperboundsto,humanperformane.Ifanoptimalstatistiallearnerfailstolearnalanguagegivenertainkindsofinputs(say,phonologialformsalone)underertainassumptionsaboutuniversalgrammar,thenweanbefairlyer-tainthathumanbeingseitherhaveaesstoriherdataorhavestrongerbiasesthatrestritthelassofpossiblegrammars.Animmediategoalofthisresearhistondawayofdeningprobabilitydistributionsoverlinguistiallyrealististruturesinawaythatpermitsustodenelanguagelearningandlanguageomprehensionasstatistialproblems,andtherestofthispaperonentratesonthesequestions.Thenextsetiondesribesthelinguistitheory,Lexial-FuntionalGrammar,whihdenesthelinguististruturesusedinthisresearh,andthefollowingsetionexplainshowwedeneaprobabilitydistributionoverthesestrutures.Setion4desribes1howoneanlearntheparametersthatdeneprobabilitydistributionsoverthesestruturesinpriniple,andpointsoutsomeofthepratialproblemsthatmakestraight-forwardwaysofestimatingthesedistributionsinfeasible.Thisleadsustothe\pseudo-likelihoodestimationmethodsdesribedinsetion5,whihalsoraiseinterestingquestionsonerningthenatureofthedataavailabletothehildandmodularityoflanguagelearningandproessing.2Lexial-funtionalgrammarThisresearhdiersfrommostworkinstatistialomputationallinguistisinthatitisompatiblewithandbuildsontheresultsofmodernlinguistitheory.Whileourapproahisompatiblewithvirtuallyallexistingtheoriesofgrammar(inludingtransformationalgrammarandminimalistgrammars),wehaveadoptedtheframeworkandstruturesofLexial-FuntionalGrammar(LFG)inourresearh.LFGhasseveralpropertiesthatmakeitespeiallywell-suitedforresearhinvolvinglinguistially-orientedprobabilistigrammars.TheformaldenitionofLFGsandthestruturestheygenerateislearandpreise(Kaplan,1995),andLFGprovidessimple,leandesriptionsofawiderangeoftypologiallydiverselinguistiphenomena(Bresnan,1982).ThereisalsoasubstantialamountofexistingomputationalresearhonLFG,inludingoneÆientparsingwithlargegrammars(MaxwellIIIandKaplan,1993),whihweexploitinourresearh.AnLFGstrutureofasenteneonsistsofasmallnumberofdistintompo-nents,suhasthephonologialstruture,thesyntatistruture,thesemantiinterpretation,et.Tokeepthingssimpleinthispaper,however,wewillonlyuseasubsetoftheseomponentsandsimplifythemwhereappropriate.Forexample,wetakethephonologialomponentofasentenetobejustastringofwords,andignoreprosodyandotherphonologialdetails.Similarly,wetakethesemantiinterpretationofasentenetobeitsprediate-argumentstruture(roughly,\whodidwhattowhom),andignoremood,tense,et.Wemakeextensiveuseoftwoomponentsinthispaper.Theonstituentor-strutureofasenteneshowsthetemporalarrangementofwords,phrasesandlausesorganizedasatreestruture.Thefuntionalorf-strutureofasenteneisanattribute-valuestruturethatshowsthegrammatialfuntionrelationshipsbe-tweenthephrasesandlausesofasentene,abstratingawayfromdetailsoflinearorder.Thepartiulargrammatialfuntionrelationshipinvolved(e.g.,subjet,objet,et.)isrepresentedbytheattributename,andf-struturesalsoenodetheargument-adjuntdistintion.Althoughitprobablydeservestobeaomponentinitsownright,forsimpliitywefollowearlyworkinLFGthatenodestheprediate-argumentstrutureofaphraseorsenteneasthevalueoftheprediateattributeinanf-struture.Figure1depitsthe-strutureandf-strutureoftheEnglishsenteneSandywantstodrinkwine.Oneofthereasonsforadoptinganattribute-valuerepresentationoff-strutureinLFGisthatsuhstruturesandesribethemultiplefuntionalrolesthatasingleonstituentanplayinasinglesentene.Forexample,inSandywants2SNPSandyVPVwantsVPVtoVPVdrinkNPwine26666666666666664prediatewant(Sandy;drink(Sandy;wine))subjet24prediateSandyperson3rd1numbersingular35omplement2666664prediatedrink(Sandy;wine)subjet1objet24prediatewineperson3rdnumbersingular35377777537777777777777775Figure1:The-strutureandthef-struturefortheEngli