Stereo person tracking with adaptive plan-view tem

StereoPersonTrakingwithAdaptivePlan-ViewStatistialTemplatesMihaelHarvilleHewlett-PakardLaboratories1501PageMillRd.,ms1181PaloAlto,CA94304UnitedStatesAbstratAstheostofomputingper-pixeldepthimageryfromstereoamerasinrealtimehasfallenrapidlyinreentyears,interestinusingstereovisionforpersontrakinghasgreatlyinreased.Methodsthatattempttotrakpeoplediretlyinthese\amera-viewdepthimagesareonfrontedbytheirsubstantialamountsofnoiseandunreliabledata.Somereentmethodshavethereforefounditusefultorstomputeoverhead,\plan-viewstatistisofthedepthdata,andthentrakpeopleinimagesofthesestatistis.Wedesribeanewombinationofplan-viewstatististhatbetterrepresentstheshapeoftrakedobjetsandprovidesamorerobustsubstrateforpersondetetionandtrakingthanpriorplan-viewalgorithms.Wealsointrodueanewmethodofplan-viewpersontraking,usingadaptivestatis-tialtemplatesandKalmanpredition.AdaptivetemplatesprovidemoredetailedmodelsoftrakedobjetsthanpriorhoiessuhasGaussians,andweillustratethatthetypialproblemswithtemplate-basedtrakinginamera-viewimagesareeasilyavoidedinaplan-viewframework.Weompareresultsofourmethodwiththosefortehniquesusingdierentplan-viewstatistisorpersonmodels,andndourmethodtoexhibitsuperiortrakingthroughhallengingphenomenasuhasomplexinter-personolusionsandloseinterations.Reasonablevaluesformostsystemparametersmaybederivedfromphysiallymeasurablequantitiessuhasaveragepersondimensions.Keywords:persontraking,plan-viewstatistis,stereodepthimages,adaptivetemplate,KalmanlterEmailaddress:harvillehpl.hp.om(MihaelHarville).URL:(MihaelHarville).PreprintsubmittedtoElsevierSiene20August20031IntrodutionManymethodsforreal-timemulti-persondetetionandtrakingwithvideoamerashavebeendesribedintheliterature.Unfortunately,fewofthese,ifany,produereliableresultsforlongperiodsoftimeinunonstrainedenviron-ments.ThispoorperformanestemsfromthemanydiÆulthallengesthatommonlybesettheproblem,amongthemostsigniantofwhihare:Segmentingthenovelordynamiobjets(\foreground)inthevideofromtherestofthesene(\bakground)Distinguishingpeoplefromotherforegroundobjetssuhasars,shoppingarts,orurtainsblowinginthewindAvoidingdistrationandonfusionduetolighting-relatedseneappearanehangessuhasshadows,inter-reetions,andglobalilluminationvariationTrakingpeoplethroughtemporaryolusions,eitherinpartorinfull,byotherpeopleorbystatiobjetsintheseneMaintainingtrakintegritywhenpeopleengageinloseinterations,ael-eraterapidly,orquiklyhangetheirbodyposeorappearanePer-pixeldepthordisparityimageryfromstereoamerasoersmuhpromisefordealingwiththeseissues.Forexample,thedistaneinformationinherentintheseimagesallowsforstraightforwardassessment,inomparisonwithtehniquesbasedonmonoularvideo,ofthe3Dloationsoftrakedobjets.Inaddition,depthdataIsapowerfulueforforegroundsegmentationIsrelativelyinsensitivetolightingeetssuhasshadowsandglobalillumi-nationhangesProvidesshapeandmetrisizeinformationthatanbeusedtodistinguishpeoplefromotherforegroundobjetsAllowsolusionsofpeoplebyeahotherorbybakgroundobjetstobedetetedandhandledmoreexpliitlyPermitsthequikomputationofnewtypesoffeaturesformathingpersondesriptionsarosstimeProvidesathird,disambiguatingdimensionofpreditionintrakingInreentyears,ashardwareandsoftwareforomputingdepthimageryfromstereoamerashasbeomeinreasinglyfastandheap[2{4,1,5℄,severalper-sondetetionandtrakingmethodsthatmakeuseofreal-timedepthdatahavebeenpresented.Mostoftheseanalyzeandtrakfeatures,gradients,andsmoothlyonnetedregionsdiretlyinthedepthimagesthemselves[6{9℄.Whenthedepthimagesareaompaniedbyaspatially-andtemporally-registeredolororgraysalevideostream,theresultsofthedepth-basedanaly-sisareeasilyintegratedwiththoseextratedfromtheolororluminanedata.2Fig.1.Exampleofolor-with-depthvideoinput,obtainedusingthePointGreyTrilopsamera[1℄.Inthedepthimage,brighterpixelsindiategreaterdistanefromtheamera,andinvalid(unreliable)depthdataisshowninblak.Manyofthetraditionalframeworksfortrakinginmonoularviewsmaythenbeapplied,buttothemuhriherper-pixelfeaturespaeofappearane(olororluminane)plusshape(depth).Thismethodologyisnotasfruitfulasonemighthope,however,beauseto-day’sstereoamerasproduedepthimageswhosestatistisarefarlessleanthanthoseofstandardolorormonohromevideo.Formulti-amerastereoimplementations,whihomputedepthbyndingsmallareaorrespondenesbetweenimagepairs,unreliablemeasurementsoftenourinimageregionsoflittlevisualtexture,asisoftentheaseforwalls,oors,orpeoplewearinguniformly-oloredlothing.Thisusuallyausesmuhofadepthimagetobeunusable.Also,itisnotpossibletondtheorretorrespondenesinregions,usuallyneardepthdisontinuitiesinthesene,thatarevisibleinonestereoinputimagebutnottheother.Thisresultsinadditionalregionsofunreliabledata,andausestheedgesofanobjetinadepthimagetobenoisyandpoorlyalignedwiththeobjet’solorimageedges.AlloftheseproblemsareevidentinthetypialoloranddepthimagepairofFigure1.Evenatseneloationswheredepthmeasurementsareinformative,thesen-sitivityofthestereoorrespondeneomputationtoverylowlevelsofimagernoise,lightingutuation,andsenemotionleadstosubstantialdepthnois