1 Introduction Computers Seeing People

ComputersSeeingPeopleIrfanA.EssaCollegeofComputing,GVUCenterGeorgiaInstituteofTechnologyAtlanta,[e-mail]1IntroductionOneofthemostexcitingandchallengingresearchquestsofthelastthirtyyearshasbeenthebuildingofmachinesthatcansee.Muchefforthasbeenexpendedon“automaticdeductionofstructureofapossi-blydynamicthree-dimensionalworldfromtwo-dimensionalimages.”[65].Therehasbeenconsiderableprogressintheareasofobjectrecognition,imageunderstandingandscenereconstructionfromsingleandmultipleimages.Thisprogresscoupledwiththeimprovementsincomputationalpowerhaspromptedanewresearchfocusofmakingmachinesthatcanseepeople,recognizethem,andinterprettheirgestures,expressions,andactions.Inthispaper,wepresentmethodsthatgivemachinestheabilitytoseepeople,in-terprettheiractionsandinteractwiththem.Wepresentthemotivatingfactorsbehindthiswork,examplesofhowsuchcomputationalmethodsaredevelopedandtheirapplications.Thebasicreasonforprovidingmachinestheabilitytoseepeoplereallydependsonthetaskweareassociatingwithamachine.Anindustrialvisionsystemaimedatextractingdefectsonanassemblylineneednotknowanythingaboutpeople.Similarly,acomputerusedforemailandtextwritingneednotseeandperceivetheusersgesturesandexpressions.However,ifourinterestistobuildintelligentmachinesthatcanworkwithus,supportourneedsandbeourhelpers,thanitmayberequiredforthesemachinestoknowmoreaboutwhotheyaresupportingandhelping.Ifourcomputersaretodomorethensupportourtext-basedneedslikewritingpapers,spreadsheets,andcommunicatingviaemail;perhapstakeonaroleofbeingapersonalassistant,thentheabilitytoseeapersonisessential.Suchanabilitytoperceivepeopleissomethingthatwetakeforgrantedinoureverydayinteractionswitheachother.Atpresentourmodelofamachineormorespecificallyofacomputerissomethingthatisplacedinthecorneroftheroom.Itisdeaf,dumb,andblind,havingnosenseoftheenvironmentthatitisinorofthepersonthatisnearit.Wecommunicatewiththiscomputerusingacodedsequenceoftappingsonakeyboard.Imagineacomputerthatknowsthatyouarenearit,thatyouarelookingatit,knowswhoyouareandwhatyouaretryingtodo.Imagineamachinethatcaninterpretavideosignalbasedonwhoisinthesceneandwhattheyaredoing.Suchabilitiesinacomputerarehardtoimagine,unlessithasanabilitytoperceivepeople.Speechperceptionhasmademuchprogressintherecentyearsandsomeamazingresultsinwordspottinghaverecentlybeenpresented[28,26,75].Thecomputervisionfieldhasalsotakentothisproblemhasmadesomeamazingprogressintherecentyears.Theimportantproblemsthatcomputervisionaimstoaddressinordertomakemachinesthatseepeopleisthatitneedstofirstdetermineifsomeoneisnearit(where)andhowmanypeopleareinitsfieldofview.1Thenextstepisidentifywhoitis.Afterithasidentifiedthepeople,itcaninterpretfacialexpression,handgesturesandbodylanguagetodeterminewhatthepeoplewantoraredoinginthesceneandwhy.Intheupcomingsections,wewilldetailoutthespecificsoftheapproachestodeterminewhere,howmany,who,what,andwhywithreferencetopeopleinascene.Needlesstostatethatdeterminationofeachoftheaboveissuesinasceneisnotpossiblewithoutentirelyonitsownandthateachonedependsontheotherasdictatedbycontext.Beforewegetintodetailsitisimportanttotalkbrieflyabouttheapplicationsofsuchatechnology.ApplicationsApplicationsofcomputervisionmethodsaimedspecificallyatseeingpeoplearemanyandencompassseveraldifferentareas.Effectivehuman-computerinteraction:Imaginecomputersthatcaninteractwithyouaswecaninter-actwitheachother,usingspeechandgestures.Suchcomputerswillbeawareofwhenyouarelookingatthem,willbeabletodetectwhereyouarepointingandinterpretyourgestures.Thesetypesofgesturalinterfacesareanintegralpartofagrowingtrendtowardsmorehuman-centeredinterfacesinHCIresearch.Specificapplicationsforthistechnologyariseinareaswheretraditionalinterfacessuchasthekeyboardandmousearenoteffective.Suchtechniqueswillallowustomovetowardsmorenon-invasiveandunencumberedinterfacesthatareatpresentprevalentinuserinter-facesallowingforInteractivevisualizationofmulti-dimensionalscientificdataanduser-centereddirectinteractionwithvirtualenvironments.SmartandInteractiveEnvironments:Machinethatcanseewillaidusindevelopingsmartrooms;roomsthatknowwhoiswhereandwhattheyaredoing.Suchroomscanbeusedtomonitorchildren,seniorcitizens,orphysicallychallengedindividualsandprovideassistanceandcareasneeded[68].SurveillanceandSecurity:Amoretraditionalapplicationofthisworkissurveillanceandsecurity.Facerecognitionhasbecomequiteausefultechnologyinthesecurityindustry,whereaccessisallowedbasedonfacialidentity.Similarly,alotofworkisbeingdonetoaidinsearchofcriminalsviaautomatedsearchesofmug-shotdatabases.Recently,therehasbeenworkaimedatrecognitionofpeoplesactionsforactivesurveillanceapplications.Entertainment,Education,andTraining:Twoareasofmostrapidgrowththesedaysareeducationandentertainment.ComputerVisionmethodsfornoninvasivetrackingandinterpretationofhumanac-tivitieswillaidtorevolutionizevariousaspectsoftheseareastoo.Anintelligenttutorthatcanseewillbefarmoreresponsivetotheneedsofitspupilifthetutorcanjudgebytheactionsandmoodsofthestudentwhethershe/heisconfused,frustrated,o