Weighted least squares fitting using ordinary leas

PSYCHOMETRIKA--VOL.62,NO.2,251-266JUNE1997WEIGHTEDLEASTSQUARESFITTINGUSINGORDINARYLEASTSQUARESALGORITHMSHENKA.L.KIERSUNIVERSITYOFGRONINGENAgeneralapproachforfittingamodeltoadatamatrixbyweightedleastsquares(WLS)isstudied.Thisapproachconsistsofiterativelyperforming(stepsof)existingalgorithmsforordinaryleastsquares(OLS)fittingofthesamemodel.TheapproachisbasedonminimizingafunctionthatmajorizestheWLSlossfunction.Thegeneralityoftheapproachimpliesthat,foreverymodelforwhichanOLSfittingalgorithmisavailable,thepresentapproachyieldsaWLSfittingalgorithm.InthespecialcasewheretheWLSweightmatrixisbinary,theapproachreducestomissingdataimputation.Keywords:weightedleastsquares,alternatingleastsquares,missingdata,algorithms,majorization,matrixapproximation,maximumlikelihoodestimation.Manymethodsformultivariateanalysisinvolvefittingamodeltoadatamatrix.Often,themodelisfitintheordinaryleastsquares(OLS)sense.Thatis,letXdenoteagivenn×mdatamatrixandletMdenoteannxmmodeldescriptionforthesedata.Thenleastsquaresfittingofthismodeltothedataconsistsofminimizingf(MIX)=llx-MH2---Z2(xii-mij)2(1)i=lj=toverM,subjecttocertainconstraintsonM.Forinstance,incaseofprincipalcomponentsanalysis(PCA),Misconstrainedtohaverankr,whererisafixedvalue,usuallysmallerthanmin(m,n).OnewayofdealingwiththisconstraintisbywritingM=AB'forann×rmatrixAandanmxrmatrixB,andreformulatingtheproblemasthatofminimizingg(A,BIX)=IIx-AB'll2.(2)Invariousapplications,itisdesiredtofitamodelbyminimizingaweightedleastsquares(WLS)lossfunction.Forinstance,BaileyandGower(1990,alsoseetenBerge&Kiers,1993)proposetoapproximateasymmetricmatrixbyamatrixoflowrankwhileusingdifferentialweightingforthediagonalelements.GabrielandZamir(1979)approx-imateanasymmetricmatrixbyamatrixoflowrank(asin(2)),whileusingdifferentialweightingforallelements.Similarly,differentialweightsfortheelementsareusedbyCarroll,DeSoeteandPruzanksy's(1989)procedureforWLSfittingofanN-wayarraybyamultilinearmodel,andbyR.A.Harshman's(PersonalCommunication,October14,1994)procedureforfittingthetrilinearmodeltoathree-wayarray(which,infact,gen-eralizesoneofGabrielandZamir'sprocedures).AdifferenttypeofapplicationofWLSisinVerboon's(1994;seealsoVerboon&Heiser,1992,1994)seriesofmethodsforrobustmultivariateanalysis:HeproposestominimizecertainrobustlossfunctionsbyiterativelyminimizingaWLSlossfunction.BesidestheaboveexplicitapplicationsofWLS,somemethodsexistthatareimplicitlyThisresearchhasbeenmadepossiblebyafellowshipfromtheRoyalNetherlandsAcademyofArtsandSciencestotheauthor.RequestsforreprintsshouldbesenttoHenkA.L.Kiers,DepartmentofPsychology(SPA),GroteKruisstraat2/1,9712TSGroningen,THENETHERLANDS.0033-3123/97/0600-95264500.75/0©1997ThePsychometricSociety251252PSYCHOMETRIKAbasedonWLS.Forinstance,acommonapproachforfittingdatawhichcontainmissingvalues(e.g.,Commandeur,1991;Girl,1990;tenBerge,Kiers&Commandeur,1993)consistsofsettingthemissingdataelementsinitiallyatarbitraryvalues,and,inthelossfunction,assigningzeroweightstotheresidualsthatbelongtotheseelements.Suchlossfunctionscanbeminimizedbyamissingdataimputationapproach,whichisaspecialinstanceoftheEMalgorithm(Dempster,Laird,&Rubin,1977):Byalternatelyfittingthemodeltothefulldataset(includingestimatesforthemissingvalues),andreplacingthemissingelementsbythecurrentmodelestimatesfortheseelements,theweightedlossfunctionisdecreasedmonotonically(andassumedtobeminimizedatleastlocally).An-otherimplicitapplicationofWLSfittingismaximumlikelihoodestimationincaseswheretheresidualsareassumedtobeindependentlynormallydistributedwithzeromeanandprespecifiedvariance(seeVerboon,1994,pp.38-40).InsuchcasesmaximumlikelihoodestimationisequivalenttoWLSfittingwiththeweightstakenequaltotheinverseofthestandarddeviations.Also,WLSissometimesusedformaximumlikelihoodfittingincaseswherethevarianceisnotprespecified.Insuchcasestheweightsareadjustedineachiterativecycle.Thisisdone,forinstance,intheCarrolletal.(1989)procedure.Inallthesemethods,aWLSlossfunctionisminimized.Lettheweights(whichareconsideredfixedandnonnegativehere)becollectedinann×rnmatrixW.ThentheWLSlossfunctioncanbedescribedgenerallyash(MlX,W)=II(x-M)*wll2=w2(xi-mii)2,i=1j=l(3)where*denotestheHadamard(orelementwise)product.Here,asin(1),MrepresentsalargevarietyofmodelsthatareobtainedbyvariouschoicesofconstraintsonM.TheproblemofminimizingtheWLSfunction(3)isoftenmuchmorecomplicatedthanthatofminimizingtheOLSfunction(1).AlthoughspecialalgorithmsareavailableforsomeoftheabovementionedWLSproblems,thereisamultitudeofmodelsforwhichWLSalgorithmshavenot(yet)beenproposed.Ratherthanresortingtogeneralgradientbasedoptimizationtechniques,whichusuallydependheavilyontheavailabilityofagoodstartingconfiguration,inthepresentpaperanapproachisusedthatisbasedonOLS.Specifically,ageneralprocedureisofferedthatcanbeusedtoobtainalgorithmsforWLSfittingofeverymodelforwhichanOLSfittingalgorithmisavailable.Thisproceduredecreasesh(MIX,W)monotonically,untila(possiblylocal)minimumisobtained.Thisisachievedbyiterativelydecreasingf(MtX),whereXisamatrixthatdependsonX,WandthevaluesofMatthecurrentiteration.Itw