A Principal Components Approach to Combining Regre

MachineLearning,0,1{25(1997)c1997KluwerAcademicPublishers,Boston.ManufacturedinTheNetherlands.APrincipalComponentsApproachtoCombiningRegressionEstimatesCHRISTOPHERJ.MERZANDMICHAELJ.PAZZANIfcmerz,[e-mail]onandComputerScience,UniversityofCalifornia,Irvine,CA92697-3425ReceivedOctober1,1997;RevisedJuly17,1998Editor:SalStolfoAbstract.Thegoalofcombiningthepredictionsofmultiplelearnedmodelsistoformanim-provedestimator.Acombiningstrategymustbeabletorobustlyhandletheinherentcorrelation,ormulticollinearity,ofthelearnedmodelswhileidentifyingtheuniquecontributionsofeach.Aprogressionofexistingapproachesandtheirlimitationswithrespecttothesetwoissuesaredis-cussed.Anewapproach,PCR*,basedonprincipalcomponentsregressionisproposedtoaddresstheselimitations.Anevaluationofthenewapproachonacollectionofdomainsrevealsthat1)PCR*wasthemostrobustcombiningmethod,2)correlationcouldbehandledwithouteliminat-inganyofthelearnedmodels,and3)theprincipalcomponentsofthelearnedmodelsprovidedacontinuumof\regularizedweightsfromwhichPCR*couldchoose.Keywords:Regression,principalcomponents,multiplemodels,combiningestimates.1.IntroductionCombiningasetoflearnedmodelstoimproveclassicationandregressionestimateshasbeenanareaofmuchresearchinmachinelearningandneuralnetworksAlearnedmodelmaybeanythingfromadecision/regressiontreetoaneuralnetwork.Thechallengeofthisproblemistodecidewhichmodelstorelyonforpredictionandhowmuchweighttogiveeach.Supposeaphysicianwishestopredictaperson’spercentageofbodyfat,PBF.S/hehasacollectionofpatientrecordswithsimplemeasurements/attributessuchasheight,weight,chestcircumference,legcircumference,etc.,alongwithamea-surementofPBFderivedfromawaterdisplacementtest.ThetaskistopredictPBFforfuturepatientsusingonlythesimplemeasurementswithoutperformingtheexpensivewaterdisplacementtest.ThephysicianhasderivedseveralmodelsforpredictingPBFusingvariouslinearregressionmethods,severalneuralnetworkcongurations,andsomeexistingheuristicfunctions.Thegoalistocombinethelearnedmodelstoobtainamoreaccuratepredictionthancanbeobtainedfromanysinglemodel.Thegeneralproblemofcombiningestimatesrobustlyisthefocusofthispaper.Onemajorissueincombiningasetoflearnedmodelsistheamountofcorrelationinthesetofpredictors.Ahighdegreeofcorrelationisexpectedbecausethelearnedmodelsareattemptingtoperformthesamepredictiontask.Correlationreectstheamountofagreementorlineardependencebetweenmodelswhenmakingaset2CHRISTOPHERMERZANDMICHAELPAZZANIofpredictions.Themorethemodelsagree,themorecorrelation,orredundancy,ispresent.Insomecases,one(ormore)modelsmaybeexpressedasalinearcombination(withvariousnumericalcoecients)oftheothermodels.Suchahighdegreeofcorrelationinthemodelsetcancausesomecombiningschemestoproduceunreliableestimates.Instatisticalterms,thisisreferredtoasthemulticollinearityproblem.Anotherissueincombiningthepredictionsoflearnedmodelsisdetectingeachmodel’suniquecontributiontopredictingthetargetoutcome.Modelsgeneratedusingdierentlearningalgorithmsaremorelikelytomakesuchcontributions.Forexample,aneuralnetworkmaydiscoverusefulnon-linearinteractionsamongsttheinitialattributes,whereasastandardlinearregressionmethodmayemployanattributedeletionstrategywhichsimpliesthepredictiontask.Agoodcombiningstrategymustbeabletoweigheachmodelaccordingtoitsuniquecontribution.Atradeoexistsinsolvingtheproblemsmentionedabove.Solutionstothemul-ticollinearityproblemarelikelytoignoretheuniquecontributionsofeachmodel.Ontheotherhand,methodsthataregoodatndingtheuniquecontributionsofeachmodelaremoresusceptibletothemulticollinearityproblem.Apointbetweenthesetwoextremeswherepredictionerrorisminimizedissought.Thefocusofthispaperistopresentandstudyanalgorithmforsolvingtheprob-lemsofmulticollinearityanddiscoveringuniquecontributions.Thepaperbeginsbydeningthetaskofcombiningregressionestimates(Section2)anddiscussingthelimitationsofexistingapproacheswithrespecttotheproblemsdiscussedabove.MoreadvancedapproachestosolvingthemulticollinearityproblemaredescribedinSection3.Anovelapproach,calledPCR*,basedonprincipalcomponentsre-gressionisoutlinedinSection4.AnalyticalandempiricalanalysesaregiveninSections5and6,respectively.RelatedworkisdiscussedinSection7.DirectionsforfutureworkaregiveninSection8,andconcludingremarksaregiveninSection9.2.MotivationTheproblemofcombiningasetoflearnedmodelsisdenedusingtheterminologyof[25].Supposetwosetsofdataaregiven:atrainingsetDTrain=(xm;ym)andatestsetDTest=(xl;yl).NowsupposeDTrainisusedtobuildasetoffunctions,F=^fi(x),eachelementofwhichapproximatesf(x).Thegoalistondthebestapproximationoff(x)usingF.Mostapproachestothisproblemlimitthespaceofapproximationsoff(x)tolinearcombinationsoftheelementsofF,i.e.,^f(x)=NXi=1i^fi(x)whereiisthecoecientorweightof^fi(x).Thefocusofthispaperistodevelopamethodforsettingthesecoecientsthatovercomesthelimitationsofearlierapproaches.Todoso,abriefsummaryoftheseCOMBININGREGRESSIONESTIMATES3approachesisnowprovidedprogressingfromsimplertomorecomplexmethodspointingouttheirlimitationsalongtheway.Thesimplestmethodforcombiningthememb