Eigentracking Robust matching and tracking of arti

P1:SGRP1:SGRInternationalJournalofComputerVisionKL528-04-blackDecember18,199716:49InternationalJournalofComputerVision26(1),63–84(1998)c°1998KluwerAcademicPublishers.ManufacturedinTheNetherlands.EigenTracking:RobustMatchingandTrackingofArticulatedObjectsUsingaView-BasedRepresentationMICHAELJ.BLACKXeroxPaloAltoResearchCenter,3333CoyoteHillRoad,PaloAlto,[e-mail]¤DepartmentofComputerScience,UniversityofToronto,Toronto,[e-mail]15,1996;RevisedOctober17,1996;AcceptedNovember18,1996Abstract.Thispaperdescribesanapproachfortrackingrigidandarticulatedobjectsusingaview-basedrepre-sentation.Theapproachbuildsonandextendsworkoneigenspacerepresentations,robustestimationtechniques,andparameterizedopticalflowestimation.First,wenotethattheleast-squaresimagereconstructionofstandardeigenspacetechniqueshasanumberofproblemsandwereformulatethereconstructionproblemasoneofrobustestimation.Secondwedefinea“subspaceconstancyassumption”thatallowsustoexploittechniquesforpara-meterizedopticalflowestimationtosolveforboththeviewofanobjectandtheaffinetransformationbetweentheeigenspaceandtheimage.Toaccountforlargeaffinetransformationsbetweentheeigenspaceandtheim-agewedefineamulti-scaleeigenspacerepresentationandacoarse-to-finematchingstrategy.Finally,weusethesetechniquestotrackobjectsoverlongimagesequencesinwhichtheobjectssimultaneouslyundergobothaffineimagemotionsandchangesofview.Inparticularweusethis“EigenTracking”techniquetotrackandrecognizethegesturesofamovinghand.Keywords:eigenspacemethods,robustestimation,view-basedrepresentations,gesturerecognition,parametricmodelsofopticalflow,tracking,objectrecognition,motionanalysis1.IntroductionThispaperaddressestheproblemoftrackingaprevi-ouslyviewedobjectinanimagesequenceastheviewoftheobjectchangesduetoitsmotionorthemotionofthecamera.Traditionalopticalflowtechniquestreatanimageregionsimplyasmoving“stuff”(AdelsonandBergen,1991)andhencecannotdistinguishbetweenchangesinviewpointorconfigurationoftheobjectandchangesinpositionrelativetothecamera.Trackersthatusetemplatesofoneformoranotherhaveanotionof¤Also:CanadianInstituteforAdvancedResearch.the“thing”beingtracked,butiftheviewchangessig-nificantlythenthe“thing”isnolongerthesameandtrackingcanfail.Recovering3Dmotionortrackinga3Dmodelofanobjectarepossiblealternativesfortrackingrigidobjects,butfortrackingandrecognizingmovingarticulatedobjectssuchashumanhandsthesesolutionsarecomputationallyexpensive.Wewouldpreferthecomputationalsimplicityofworkingwith2Dimage-basedmodelsbutweneedtoextendthemtoaccountforchangingviewsorchangingstructure.Wewouldlikeaview-basedrepresentation(ormodel)ofobjectswithasmallsetofviewsandamethodthatwilltakeanimageandfindboththeviewoftheobjectP1:SGRP1:SGRInternationalJournalofComputerVisionKL528-04-blackDecember18,199716:4964BlackandJepsonandthetransformationthatmapstheimageontothemodel.Toachievethis,wecombinelinesofresearchfromobjectrecognitionusingeigenspaces,parameter-izedopticalflowmodels,androbustestimationtech-niquesintoanovelmethodfortrackingobjectsusingaview-basedrepresentation.View-based,orappearance-based,objectrepresen-tationshavefoundanumberofexpressionsinthecom-putervisionliterature,inparticularintheworkoneigenspacerepresentations(MuraseandNayar,1995;TurkandPentland,1991).Eigenspacerepresentationscanprovideacompactapproximateencodingofalargesetofimagesintermsofasmallnumberoforthogonalbasisimages.Thesebasisimagesspanasubspaceofthetrainingsetcalledtheeigenspaceandalinearcom-binationoftheseimagescanbeusedtoapproximatelyreconstructanyofthetrainingimages.Previousworkoneigenspacerepresentationshasfocusedontheprob-lemofobjectrecognitionandhasonlyperipherallyaddressedtheproblemoftrackingobjectsovertime.Additionally,theseeigenspacereconstructionmethodsarenotinvarianttoimagetransformationssuchastrans-lation,scaling,androtation.Previousapproacheshavetypicallyassumedthattheobjectofinterestcanbelo-catedinthescene,segmented,andtransformedintoacanonicalformformatchingwiththeeigenspace.Inthispaperwegeneralizeandextendthepreviousworkintheareatoamelioratesomeoftheseproblems.Therearethreeprimaryobservationsunderlyingthiswork.First,standardeigenspacetechniquesrelyonaleast-squaresfitbetweenanimageandtheeigenspace(MuraseandNayar,1995),andthiscanleadtopoorresultswhenthereisstructurednoiseintheinputimage.Wereformulatetheeigenspacematchingproblemasoneofrobustestimationandshowhowitovercomessomeoftheproblemsoftheleast-squaresapproach.Thismakeseigenspacemethodsmorepracticalinthattheycancopewithproblemsinwhichthestandardleast-squaresformulationgiveserroneousresults.Second,weobservethatratherthantrytorepresentallpossibleviewsofanobjectfromallpossibleview-ingpositions,itismorepracticaltorepresentasmallersetofcanonicalviewsandallowaparameterizedtrans-formation(e.g.,affine)betweenaninputimageandtheeigenspace.Whatthisimpliesisthatmatchingusinganeigenspacerepresentationinvolvesbothestimatingtheviewofobjectaswellasthetransformationthattakesthisviewintotheimage.Thisallowsamultiple-viewsplustransform