SYSTEMS FOR VIDEO TECHNOLOGY 1 1 IEEE TRANSACTIONS

IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY11IEEETRANSACTIONSONCIRCUITSANDSYSTEMSFORVIDEOTECHNOLOGY23DTV:3DTime-varyingSceneCaptureTechnologies-ASurveyElenaStoykova,AydinAlatan,PhilipBenzie,NikosGrammalidis,SotirisMalassiotis,JoernOstermann,SergejPiekh,VentseslavSainov,ChristianTheobalt,ThangavelThevar,XenophonZabulisAbstract—Advancesinimagesensorsandevolutionofdigitalcomputationisastrongstimulusfordevelopmentandimple-mentationofsophisticatedmethodsforcapturing,processingandanalysisof3Ddatafromdynamicscenes.Researchonperspectivetime-varying3Dscenecapturetechnologiesisimportantfortheupcoming3DTVdisplays.Methodssuchasshape-from-texture,shape-from-shading,shape-from-focusandshape-from-motionextractioncanrestore3Dshapeinformationfromasinglecameradata.Theexistingtechniquesfor3Dextractionfromsinglecameravideosequencesareespeciallyusefulforconversionofthealreadyavailablevastmono-viewcontenttothe3DTVsystems.Scene-orientedsinglecameramethodsashumanfacereconstructionandfacialmotionanalysis,bodymodelingandbodymotiontrackingandmotionrecognitionsolveefficientlyavarietyoftasks.Intensiveareaofresearchis3Dmulti-cameradynamicacquisitionandreconstructionwithitshardwarespecificsascalibrationandsynchronizationanditssoftwaredemands.Differentclassesofmulti-viewstereoalgorithmsasthosebasedoncostfunctioncomputingandoptimization,fusingofmultipleviewsandfeaturepointreconstructionarepossiblecandidatesfordynamic3Dreconstruction.High-resolutiondigitalholographyandpatternprojectiontechniquessuchascodedlightorfringeprojectionforreal-timeextractionof3Dobjectpositionsandcolorinformationcouldmanifestthemselvesasanalternativetotraditionalcamera-basedmethods.Apartfromalltheseapproaches,therealsoaresomeactiveimagingdevicescapableof3Dextractionsuchasthe3Dtime-of-flightcamerawhichprovides3Dimagedataofitsenvironmentbymeansofamodulatedinfraredlightsource.IndexTerms—3DTV,Three-dimensionaldisplays,3D/stereosceneanalysis,shape,videoanalysis,digitalholography,codedlight,patternprojectionI.INTRODUCTIONTHREE-DIMENSIONALtelevision(3DTV)isexpectedtoevokearevolutioninvisualtechnology.Beyondanydoubt,preciseacquisitionof3Dinformationofdynamicscenesiscrucialfor3DTVimplementation.Extensiveresearchhasbeenconductedforcapturing,processingandanalysisof3Dscenedataaimedatreliable3DreconstructionsformoreElenaStoykovaandVentseslavSainovarewiththeCentralLab.ofOpticalStorageandProcessingofInformation,BulgarianAcademyofSciences,Sofia,Bulgaria;AydinAlataniswithMiddleEastTechnicalUniversity,Ankara,Turkey;PhilipBenzieandThangavelThevararewiththeSchoolofEngineeringandPhysicalSciences,UniversityofAberdeen,UK;NikosGram-malidis,SotirisMalassiotisandXenophonZabulisarewiththeInformaticsandTelematicsInstitute,CentreforResearchandTechnologyHellas,Thermi-Thessaloniki,Greece,ChristianTheobaltiswiththeMax-Planck-InstituteofInformatics,Saarbruecken,Germany;JoernOstermannandSergejPiekharewiththeInstituteofInformationProcessing,LeibnizUniversityofHannover,Germany.ThisworkissupportedbyECwithinFP6underGrant511568withtheacronym3DTV.thantwodecades.Avarietyoftechniqueshaveemergedasaresultoftherecentadvancesinimagesensortechnologiesandevolutionofdigitalcomputation.Theaimofthesurveyistoshedlightonstate-of-the-artinthemajortime-varying3Dscenecapturetechnologieswithanemphasisontheirincorporationinfunctioning3DTVsystems.Havingtheirownpedigree,underlyingprinciples,limitationsandtrendsfordevelopment,theexamined3Dscenecapturingtechniquesco-existwithinthescopeofthesurvey,asseparatetopics.Theyareunited,however,notonlybythesamegoaltheypursue,butalsobytheuseofcommonmathematicaltoolstotacklethe3Dreconstructionproblem,confrontingsimilardifficultiesinmanycases.Morespecifically,thesurveypro-posesconciseanalysisofapproachesfor3Dsceneextractionfromsingleandmultiplecamerasdatastreams.Italsofocusesonholographicandpatternprojectiontechniquesforreal-timecollectionofpointlocationsandcolorinformationof3Dobjects,asanalternativetotraditionalcamera-basedmethods,andbrieflydescribespotentialsoftime-of-flightbasedsystemscapableofintegratingdepthextractionandimaginginrealtime.Beinganill-posedproblem,thechallengesforthe3Dscenereconstructionfrommono-viewvideosequencesarepointedoutinSectionII.Thefirstpartofthesectionintroducesallshape-from-Xapproachesfor3Dshapeextractionbutconcen-tratesmainlyonshape-from-motionapproaches,relyingontherelativemotionbetweenthecameraandthescene,asbeingquiteappropriateformanypracticalsituations.Thesecondpartisdedicatedtothetechniquesforcapturingof3Dstructuresofhumanfaceandbody,whichhavemanypotentialapplications,stillfrommono-viewsequences.Briefdescriptionofmethodsforfacecaptureandfacialmotionanalysis,aswellasforbodymodelingandbodymotiontrackingandrecognition,isalsopresented.SectionIIIisdevotedtohardwareissuesinmulti-cameraset-ups,suchascalibrationandsynchronizationanditaddressestheimplementationofdifferentclassesofmulti-viewreconstructionalgorithmsfordynamic3Dreconstruction.SectionIVbrieflypresentstheprincipleofdigitalholography,asameansfordigitalrecordingofahologramofanobjectwithfurthernumerical