A Survey of Computer Vision-Based Human Motion Cap

ComputerVisionandImageUnderstanding81,231–268(2001)doi:10.1006/cviu.2000.0897,availableonlineat:[e-mail],[e-mail]2,1999;acceptedSeptember27,2000Acomprehensivesurveyofcomputervision-basedhumanmotioncapturelitera-turefromthepasttwodecadesispresented.Thefocusisonageneraloverviewbasedonataxonomyofsystemfunctionalities,brokendownintofourprocesses:initial-ization,tracking,poseestimation,andrecognition.Eachprocessisdiscussedanddividedintosubprocessesand/orcategoriesofmethodstoprovideareferencetode-scribeandcomparethemorethan130publicationscoveredbythesurvey.Referencesareincludedthroughoutthepapertoexemplifyimportantissuesandtheirrelationstothevariousmethods.Anumberofgeneralassumptionsusedinthisresearchfieldareidentifiedandthecharacteroftheseassumptionsindicatesthattheresearchfieldisstillinanearlystageofdevelopment.Toevaluatethestateoftheart,themajorapplicationareasareidentifiedandperformancesareanalyzedinlightofthemeth-odspresentedinthesurvey.Finally,suggestionsforfutureresearchdirectionsareoffered.c°2001AcademicPressCONTENTS1.Introduction.2.SurveysandTaxonomies.3.Initialization.4.Tracking.5.PoseEstimation.6.Recognition.7.Discussion.8.Conclusion.1.INTRODUCTIONTheanalysisofhumanactionsbyacomputerisgainingmoreandmoreinterest.Asignificantpartofthistaskistoregisterthemotion,aprocessknownashumanmotion2311077-3142/01$35.00Copyrightc°2001byAcademicPressAllrightsofreproductioninanyformreserved.232MOESLUNDANDGRANUMcapture.Eventhoughthistermcoversmanyaspects,itismainlyusedinconnectionwithcapturinglargescalebodymovements,whicharethemovementsofthehead,arms,torso,andlegs.Formallyweheredefinehumanmotioncaptureastheprocessofcapturingthelargescalebodymovementsofasubjectatsomeresolution.Weincludedatsomeresolutiontoemphasizethattrackingofasubject’slimbs,aswellasoveralltrackingofasubject,areconsideredtofallwithintheabovedefinition.Hence,humanmotioncaptureisusedbothwhenthesubjectisviewedasasingleobjectandwhenviewedasarticulatedmotionofahighdegreeoffreedomskeletonstructurewithanumberofjoints.Whatisnotcoveredbytheabovedefinitionissmallscalebodymovementssuchasfacialexpressionsandhandgestures.AthoroughreviewofhandgesturescanbefoundinthesurveybyPavlovicetal.[116].1.1.ApplicationAreasThepotentialapplicationsofhumanmotioncapturearethedrivingforceofsystemdevelopment,andweconsiderthefollowingthreemajorapplicationareas:surveillance,control,andanalysis.Thesurveillanceareacoversapplicationswhereoneormoresubjectsarebeingtrackedovertimeandpossiblymonitoredforspecialactions.Aclassicexampleisthesurveillanceofaparkinglot,whereasystemtrackssubjectstoevaluatewhethertheymaybeabouttocommitacrime,e.g.,stealacar.Thecontrolarearelatestoapplicationswherethecapturedmotionisusedtoprovidecontrollingfunctionalities.Itcouldbeusedasaninterfacetogames,virtualenvironments,oranimationortocontrolremotelylocatedimplements.Foracomprehensivediscussionofmotioncaptureinthecontrolapplicationarea,see[99].Thethirdapplicationareaisconcernedwiththedetailedanalysisofthecapturedmotiondata.Thismaybeusedinclinicalstudiesof,e.g.,diagnosticsoforthopedicpatientsortohelpathletesunderstandandimprovetheirperformance.1.2.AlternativeTechnologiesforMotionCaptureThesystemsusedtocapturehumanmotionconsistofsubsystemsforsensingandpro-cessing,respectively.Theoperationalcomplexityofthesesubsystemsistypicallyrelated,sothathighcomplexityofoneofthemallowsforacorrespondingsimplicityoftheother.Thistrade-offbetweenthecomplexitiesalsorelatestotheuseofactiveversuspassivesensing.Activesensingoperatesbyplacingdevicesonthesubjectandinthesurroundingswhichtransmitorreceivegeneratedsignals,respectively[99].Activesensingallowsforsimplerprocessingandiswidelyusedwhentheapplicationsaresituatedinwell-controlledenvironments.Thatisinparticularthecaseforthethirdapplicationarea,analysis,andinsomeofthecontrolapplications.Passivesensingisbasedon“natural”signalsources,e.g.,visuallightorotherelectro-magneticwavelengths,andrequiresnowearabledevices.Anexceptioniswhenmarkersareattachedtothesubjecttoeasethemotioncaptureprocess.Markersarenotasintrusiveasthedevicesusedinactivesensing.Passivesensingismainlyusedinsurveillanceandsomecontrolapplicationswheremountingdevicesonthesubjectisnotanoption.Computervisionwiththepassivesensingapproachhaschallengedactivesensingwithinallthreeapplicationareas.EventhoughtheuseofmarkersmayseemagoodcompromiseCOMPUTERVISION-BASEDHUMANMOTIONCAPTURE233betweenpassiveandactivesensing,itisstillinconvenientforthesubject(sometimesim-possible)andcomputervisionallowsinprinciplefortouchfreeandmorediscrete“pure”motioncapturesystems.1.3.ContentofThisPaperThispaperisonlyconcernedwithcomputervision-basedapproaches,i.e.,passivesens-ing.Itprovidesacompressivesurveyofpublicationsincomputervision-basedhumanmotioncapturefrom1980intothefirsthalfof2000.Thefocusisonageneraloverviewinrelationtoafun