Robust real-time 3D head pose estimation from rang

RobustReal-time3DHeadPoseEstimationfromRangeDataSotirisMalassiotisandMichaelG.StrintzisInformatics&TelematicsInstitute,1stKmThermi-PanoramaRd.,57001Thessaloniki,Greece,Email:[e-mail]al-time3Dposeestimationalgorithmusingrangedataisdescribed.Thesystemreliesonanovel3Dsensorthatgeneratesadenserangeimageofthescene.Bynotrelyingonbrightnessinformation,theproposedsystemguaranteesrobustnessunderavari-etyofilluminationconditions,andscenecontents.Efficientfacedetectionusingglobalfea-turesandexploitationofpriorknowledgealongwithnovelfeaturelocalizationandtrackingtechniquesaredescribed.Experimentalresultsdemonstrateaccurateestimationofthe6de-greesoffreedomoftheheadandrobustnessunderocclusions,facialexpressions,andheadshapevariability.Keywords:rangedata,head,face,pose,tracking1IntroductionCapturingandunderstandinghumanmotionhasbecomeoneofthemostactiveresearchareasincomputervision,duetothelargenumberofpotentialapplications.Inparticular,trackingthe3Dlocationandorientationofthehumanface,whichisPreprintsubmittedtoElsevierScience22November2004thesubjectofthispaper,isveryimportantforapplicationssuchasmulti-modalhuman-computerinteraction,facerecognition,analysisoffacialexpressionsandvideo-conferencing.Thereareseveralcommercialproductscapableofaccurateandreliable3Dheadpositionandorientationestimation.Theseareeitherbasedonmagneticsensorsoronspecialmarkersplacedontheface;bothpracticesareencumbering,causingdiscomfortandlimitingnaturalmotion.Also,commercialsystemsbasedongazetrackingemployinginfraredillumination,doguaranteereliabledetectionofeyelocation,atthecost,howeverofrestrictionsplacedonheadpositionandorientation.Vision-based3Dheadtrackingprovidesanattractivealternative,buttherearestillseveralchallengestobeaddressedsuchasrobustnessunderarbitraryilluminationofthescene,copingwithclutteredbackgroundsanddealingwithocclusions.RelatedWorkFacedetectionandlocalization,examiningthepresenceandestimatingtheposi-tionofoneormorefacesinanimage,maybeconsideredasafirststepforfacetrackingandisusuallyexaminedseparatelyintheliterature.Severalfacedetectiontechniqueshavebeenproposedforgrey-scaleimages[1].Thesemayberoughlycategorizedtothosebasedonthedetectionoffacialfeatures,possiblyexploitingtheirrelativegeometricarrangement,andthosebasedontheclassificationofthebrightnesspatterninsideanimagewindow(obtainedbyexhaustivelysweepingthewholeimage)asfaceornon-face.Techniquesbelongingtothesecondcategorywererecentlyshowntobemoresuccessfulindetectingfacesinclutteredback-grounds[2];howeverthecorrectdetectionratesreportedwerebelow90%.Furthershortcomingsofexistingfacedetectionalgorithmsincludetheirsensitivitytopar-tialocclusionoftheface(e.g.glasses,hair)andtohardilluminationandheadpose2andtheirtendencytobecomputationallydemandinganddifficulttotrain.Colorin-formation,whenavailable,isapowerfulcueforlocatingtheface[3].Whentrans-formedtotheappropriatecolorspace(e.g.HSV),pixelvaluesformtightclustersandthusefficientprobabilisticmodellingtechniquesmaybeapplied[4].However,theparametersofthecolordistributionwereshowntorelyontheenvironmentalil-luminationandtheresponsecharacteristicsoftheacquisitiondevice.Furthermore,skin-coloredimageregionsonthebackgroundwillproduceerroneousfacecandi-dates.Inthispaper,ahighlyrobustfacedetection/localisationprocedureisfirstproposedbasedondepthinformation.Byexploitingdepthinformationthehumanbodymaybeeasilyseparatedfromthebackground,whilebyusinga-prioriknowledgeofitsgeometricstructureefficientsegmentationoftheheadfromthebody(neckandshoulders)isachieved.Amethodfor3Dfacetracking,i.e.thedynamicestimationofthe6degreesoffreedomofrigidheadmotionissubsequentlyexamined.Recovering3Dfaceposefromasinglevideocamera(uptoascalingfactor)isadifficultproblemthatisusuallyaddressedbyexploitinga-priorifacegeometrymodels.Proposedtrack-ingtechniquesmayberoughlyclassifiedtothosebasedonopticalflowandthosebasedontrackingofsalientimagefeaturessuchastheeyesandmouth.Inthefirstapproachconstraintsareposedtotheopticalflowfieldbyexplicitly[5]orimplic-itly[6]incorporatingheadgeometrymodels.Thisapproach,reliesontheassump-tionofconstantpixelbrightnessacrossframes,andthereforesufferswhenillumina-tionvariations,shadows,andocclusionsarepresent.Moreoversuchtechniquesarecomputationallydemanding.Withthesecondapproachtheeffectofilluminationconditionsisrelaxedbyexploitingfacialfeaturesandaparametric3Dfacemodelpossiblyreconstructeddirectlyfromthem[7,8].Thisapproachhaslimitationses-3peciallyforlargerotationsoftheheadsincesomeofthefeaturesmaybeoccludedorseriouslydistorted.Stereosystemsfor3Dfaceposeestimationhavealsobeenproposed(e.g.[9])relyingonfacialfeaturetracking.Byestablishingcorrespon-denceofthesefeaturesinthestereoframes,3Dcoordinatesofthesefeaturescanbeestimated.The3Dcoordinatesofthreeormoresuitablychosenfacialfeaturessufficetoestimatethe3Dposeoftheface.Although,atwocameraapproachlimitstheambiguityin3Dfaceposerecovery,trackingisstillbasedonthebrightnessfunctionandisthereforesensitivetoilluminationconditions,backgroundclutterandocclusions.Inthispaperanovel3Dsensorca