Long-Run Performance of Bayesian Model Averaging

Long-RunPerformaneofBayesianModelAveraging1AdrianE.RafteryUniversityofWashington,SeattleYingyeZhengFredHuthinsonCanerResearhCenter,SeattleWorkingPaperno.34CenterforStatistisandtheSoialSienesUniversityofWashingtonJuly17,20031AdrianE.RafteryisProfessorofStatistisandSoiology,DepartmentofStatistis,Univer-sityofWashington,Box354322,SeattleWA98195-4322;email:rafterystat.washington.edu;Web::yzhengfhr.org.ThisartileisaninviteddisussionoftheJournaloftheAmerianStatistialAssoiation|TheoryandMethodsInvitedPapersfor2003,\FrequentistModelAverageEstima-tors,byNilsLidHjortandGerdaClaeskens,and\TheFoussedInformationCriterion,byGerdaClaeskensandNilsLidHjort.WearegratefultoJASA|T&MEditorFrankSamaniegoforinvitingustoprepareit.ThisresearhwassupportedbyNIHGrant1R01CA094212-01andONRGrantN00014-01-10745.WearegratefultoMerliseClyde,EdGeorge,JenniferHoeting,DavidMadiganandChrisVolinskyforhelpfulomments.AbstratHjortandClaeskens(HC)arguethatstatistialinfereneonditionalonasingleseletedmodelunderestimatesunertainty,andthatmodelaveragingisthewaytoremedythis;westronglyagree.TheypointoutthatBayesianmodelaveraging(BMA)hasbeenthedominantapproahtothis,butarguethatitsperformanehasbeeninadequatelystudied,andproposeanalternative,FrequentistModelAveraging(FMA).Wepointout,however,thatthereisasubstantialliteratureontheperformaneofBMA,onsistingofthreemainthreads:generaltheoretialresults,simulationstudies,andevaluationofout-of-sampleperformane.Thetheoretialresultsaresattered,andwesummarizethem.Theresultshavebeenquiteonsistent:BMAhastendedtooutperformompetingmethodsformodelseletionandtakingaountofmodelunertainty.Thetheoretialresultsdependontheassumptionthatthe\pratialdistributionoverwhihtheperformaneofmethodsisassessedisthesameasthepriordistributionused,andweinvestigatesensitivityofresultstothisassumptioninasimplenormalexample;theyturnoutnottobeundulysensitive.WepointoutthatHC’sriskresults,thatAIC-modelaveragingandsimilarmethodssuhasFIC-basedmodelaveragingperformwell,dependruiallyontheirloalmisspeiationassumption(2.2),namelythatallnuisaneparametersaresmallanddelinewithsamplesize,atrateO(1pn).Thekeyquestionisthustherealismofthisassumption.Wequestionthisassumptiononthegroundsofitslakoffaevalidityinsomesituations,thegrowingseparationbetweendataolletionandresearh,theinreasingtendenyforresearhondierentquestionstobebasedonafewlargehigh-qualitypublidatasets,andthestatistialliterature,wheresamplesizeandparametervaluesrarelyovaryinthedesignofsimulationstudies.Finally,wereanalyzeHC’sdataexample,onriskfatorsforlowbirthweight.Contents1Introdution12PerformaneofBayesianModelSeletionandBayesianModelAveraging:TheoretialResults33NormalExample54TheLoalMisspeiationAssumption,AICandFMA85ModelAveragingforLogistiRegression145.1BayesianModelAveragingforCase-ControlStudies..............145.2BayesianModelAveragingfortheLowBirthweightExample.........155.3AnalysisofCompleteLowBirthweightData..................16ListofFigures1TotalErrorRateintheSimpleNormalExampleforn=100.ModelhoieisbasedonaBayesFator(solidline),a5%signianetest(dashes),BIC(dots),andAIC(dotsanddashes).Thex-axisshowsthepriorvariane2..72TotalErrorRateintheSimpleNormalExampleforn=100;000.......83BMAEstimationofintheSimpleNormalExample:MeanSquaredErrors.ThesolidlineshowstheMSEforthestandardestimator^=y,whihis1=n=:01......................................94Coverageof95%CondeneIntervalsforintheSimpleNormalExample:(a)BMAinterval,and(b)standardnormalondeneinterval........105AverageLengthsofCondeneIntervalsforintheSimpleNormalExample11ListofTables1StandardGLIMAnalysisandPosteriorModelProbabilitiesforHC’sSubsetoftheLowBirthweightData...........................162BMAEstimatesandPosteriorStandardDeviationsforHC’sFousParametersforHC’sSubsetoftheLowBirthweightData.................173PosteriorEetProbabilities,BMAPosteriorMeans,andBMAPosteriorStandardDeviationsfortheFullLowBirthweightDataset..........18i1IntrodutionIntheirartile,\FrequentistModelAverageEstimators,HjortandClaeskens|hereafterHC|makethepointthatstatistialinfereneonditionalonamodelseletedamongseveralonthebasisofdatawilltendtounderestimatevariability.Westronglyagree.Theyarguethatthewaytooveromethisisbymodelaveraging,andagainweagree.Thereismuhsupportforthesearguments:thesepointshavebeenmadebymanyauthorsinalonglineofliteraturegoingbakatleasttoLeamer(1977).HCpointoutthatBayesianmodelaveraging(BMA)dominatestheliteratureonaountingformodelunertaintyinstatistialinferene.TheirsearhforafrequentistalternativeislargelymotivatedbythefeelingthattheperformaneofBMAinrepeateddatasetsorexperimentshasbeeninadequatelystudied.Or,astheyputit,\eventhoughBMA‘works’,...,ratherlittleappearstobeknownabouttheatualperformaneorbehavioroftheonsequentinferenes,likeestimatorpreision.Thisisasomewhatsurprisingstatement,astheperformaneofBayesianmodelseletionandBMAhas,infat,beenextensivelystudied.Therearethreemainstrandsofresults:generalthe