搜索
BootstrapEstimateofKullback-LeiblerInformationforModelSelectionByRiteiShibataTechnicalReportNo.424January1995DepartmentofStatisticsUniversityofCaliforniaBerkeley,CaliforniaBootstrapEstimateofKullback-LeiblerInformationforModelSelectionRiteiSHIBATA DepartmentofMathematics,KeioUniversity3-14-1Hiyoshi,Kohoku,Yokohama,223,JapanAbstractEstimationofKullback-Leibleramountofinformationisacru-cialpartofderivingastatisticalmodelselectionprocedurewhichisbasedonlikelihoodprinciplelikeAIC.Todiscriminatenestedmod-els,wehavetoestimateituptotheorderofconstantwhiletheKullback-Leiblerinformationitselfisoftheorderofthenumberofobservations.AcorrectiontermemployedinAICisanexampletoful llthisrequirementbutitisasimplemindedbiascorrectiontothelogmaximumlikelihood.ThereforethereisnoassurancethatsuchabiascorrectionyieldsagoodestimateofKullback-Leiblerinformation.Inthispaperasanalternative,bootstraptypeestimationisconsid-ered.Wewill rstshowthatbothbootstrapestimatesproposedbyEfron(1983,1986,1993)andCavanaughandShumway(1994)areatleastasymptoticallyequivalentandthereexistmanyotherequiva-lentbootstrapestimates.Wealsoshowthatallsuchmethodsareasymptoticallyequivalenttoanon-bootstrapmethod,knownasTIC(Takeuchi’sInformationCriterion)whichisageneralizationofAIC.KeyWordsandPhrases:Kullback-LeiblerInformation,Informa-tionCriterion,Bootstrap,BiasEstimation NowstayingatDepartmentofStatistics,U.C.Berkeley.11IntroductionEstimationofKullback-Leiblerinformationisakeytoderivingsocalledin-formationcriterionwhichisnowwidelyusedforselectingastatisticalmodel.Inparticular,Kullback-Leiblerinformationde nedasinthefollowing(1.1)isconsideredameasureofgoodnessof tofastatisticalmodel.Therefore,oneofstrategiesistoselectamodelsoastominimize(1.1).Throughoutthispaper,wemeanbyastatisticalmodelaparametricfamilyofdensitieswithrespecttoa - nitemeasure onndimensionalEuclideanspace,M=ff(x; )=Yifi(xi; ); 2 g;wherex=(x1;x2;:::;xn)Tand =( 1; 2;:::; p)T.Weassume,ontheotherhand,thatthejointdistributionofindependentobservationsy=(y1;y2;:::;yn)TisGwhichhasadensityg(y)=Qigi(yi)withrespectto .Denoting^ =^ (y)themaximumlikelihoodestimateof underamodelM,wede neKullback-LeiblerinformationformodelMasIn(g( );f( ;^ (y)))=Zg(x)logg(x)f(x;^ (y))d (x)(1.1)=Zg(x)logg(x)d (x) Zg(x)logf(x;^ (y))d (x):Sincethe rsttermontherighthandsideofthelastequationin(1.1)isindependentofanyparticularmodel,minimizingtheKullback-Leiblerinfor-2mation(1.1)isequivalenttomaximizingatargetvariable,T=T(y)=Zg(x)logf(x;^ (y))d (x):(1.2)ByasimpleTaylorexpansion,wehaveanapproximationofT,T=Zg(x)logf(x; )d (x) 12Q+op(1);(1.3)where isapseudotrueparameter,thatis,the whichminimizesI(g( );f( ; ))ormaximizesZg(x)logf(x; )d (x):Herewehaveusedthenotations,Q=(^ (y) )T^J(y; )(^ (y) )and^J(y; )= @2@ @ Tlogf(y; ):HoweverinpracticewehavetoestimateT,becausetheTdependsonanunknowng( ).ThelogmaximumlikelihoodisanaiveestimateofTanditcanbeagoodplatform.Itisapproximatedaslogf(y;^ (y))=logf(y; )+12Q+op(1)(1.4)=Zg(x)logf(x; )d (x)3+flogf(y; ) Zg(x)logf(x; )d (x)g+12Q+op(1):Theorderofmagnitudeofthe rstthreetermsontherighthandsideofthelastequationin(1.4)areO(n),Op(pn)andOp(1),respectively.Therefore,onlythe rsttermissigni cantasfarascompetitivemodelsarenotnestedeachother.However,ifmodelsM1 M2arenestedandg( )isamemberofM1,thenthepseudotrueparameter becomesthesameforbothmodels,sothatonlythelastterm12Qin(1.4)remainssigni cant.Infact,denotingthemaximumlikelihoodestimateof undereachmodelby^ 1and^ 2wecanwritethedi erenceofthecorrespondinglogmaximumlikelihoodsaslogf(y;^ 1) logf(y;^ 2)=12(Q1 Q2)+op(1):(1.5)Ontheotherhand,thedi erenceofvaluesofthetargetvariableTiswrittenasT1 T2= 12(Q1 Q2)+op(1):(1.6)Therefore,asimplemindedcorrectiontothelogmaximumlikelihoodiscor-rectingonlyasigni cantpartofthebiasof(1.5)to(1.6), E(Q1 Q2);whichisasymptoticallyequalto (p1 p2),wherep1andp2arethenumberofparametersofmodelsM1andM2respectively.Thisyieldsabiascorrec-tion ptothemaximumloglikelihoodlogf(y;^ (y)).Ifthecorrectedlog4maximumlikelihoodismultipliedby-2forconvenience,Akaike’sinformationcriterionAIC= 2logf(y;^ (y))+2pfollows.Ofcourse,suchasimplemindedcorrectiondoesnotnecessarilyyieldagoodestimate.Alotofworkshavebeendoneto ndabettercorrection.Oneofsuchapproachesistoevaluatethebiasaspreciselyaspossible.InspiredbythepioneeringworkbySugiura(1978),HurvichandTsai(1989,1991,1993)derivedamoreprecisebiascorrection,p+(p+1)(p+2)n p 2;thanthepinAICfornormallinearmodels.Inpractice,suchacorrectionisquitee ective,particularlywhenthepiscloseton.Alsonon-asymptoticbiascorrectionisimportantinselectingadiscretemodellikebinomialormultinomialmodels,wherethedistributionisoftenskewedandnormalap-proximationworkswellonlyforquitelargenumberofobservations.Butinthispaperwedon’tgofurtherintothisproblem.TheauthorshowedanoptimalityoftheselectionsoastominimizeAICundertheassumptionthatthenumberofparametersof increasesasthenumberofobservationsnincreases(Shibata(1980,1981)).Thisisforex-amplethecasewheng( )isoutsideofanymodel.Thenmoreandmoreparametersareneededtogetcloserapproximationtog( ).Unde