On automatic selection of temporal scales in time-

OnAutomatiSeletionofTemporalSalesinTime-CausalSale-Spae?TonyLindebergComputationalVisionandAtivePereptionLaboratory(CVAP)DepartmentofNumerialAnalysisandComputingSieneKTH,S-10044Stokholm,SwedenAbstrat.Thispaperoutlinesageneralframeworkforautomatise-letionintemporalsale-spaerepresentations,andshowshowthesug-gestedtheoryappliestomotiondetetionandmotionestimation.1IntrodutionAfundamentalonstraintonthedesignofavisionsystemoriginatesfromthefatthatimagestruturesarepereivedasmeaningfulentitiesonlyoverertainrangesofsale.Ingeneralsituations,itishardlyeverpossibletoknowinadvaneatwhatsalesinterestingstruturesanbeexpetedtoappear.Forthisreason,animagerepresentationthatexpliitlyinorporatesthenotionofsaleisaruiallyimportanttoolwhendealingwithsensorydata,suhasimages.Amulti-salerepresentationbyitself,however,ontainsnoexpliitinfor-mationaboutwhatimagestruturesshouldberegardedassigniantorwhatsalesareappropriatefortreatingthose.Earlyworkaddressingtheseproblemsforblob-likeimagestrutureswaspresentedin(Lindeberg1993a),leadingtothenotionofasale-spaeprimalsketh.Then,in(Lindeberg1993b,1996b)anextensiontootheraspetsofimagestrutureswaspresentedbyseletingsalesfordierentialfeaturedetetors(suhasblobs,orners,edgesandridges)frommaximaoversalesofnormalizeddierentialentities.Thesubjetofthisartileistoaddresstheproblemofsaleseletioninthetemporaldomain,inordertodealwithimagedataovertime.Whereas,itisnowrathergenerallyaeptedthatsomekindof\smoothingovertimeisneessarywhenproessingtime-varyingimages,mosturrentworkonmotionanalysisisstillarriedoutatasingletemporalsale(see,e.g.,(Barronetal.1994;BeauheminandBarron1995)).Amainargumentwhihwillbeadvoatedinthisartile,isthatinanalogytoearlieradvanesonspatialdomains,theperformaneandrobustnessofalgo-rithmsoperatingovertimeanbeimprovedsubstantially,ifthespatio-temporalimagedataareonsideredatseveraltemporalsalessimultaneously,andifweinorporateexpliitmehanismsforautomatiseletionoftemporalsales.?TehnialreportISRNKTHNA/P{97/09{SE.DepartmentofNumerialAnalysisandComputingSiene,RoyalInstituteofTehnology,S-10044Stokholm,Sweden,Sep1997.AlsopresentedinPro.AFPAC’97:AlgebraiFramesforthePereption-AtionCyle(G.SommerandJ.J.Koenderink,eds.),vol.1315ofLetureNotesinComputerSiene,(Kiel,Germany),pp.94{113,SpringerVerlag,Berlin,Sept.1997.Toformthebasisofatheoryfortemporalsaleseletion,wewillstartbyshowinghowtime-ausalnormalizedsale-spaederivativesanbedenedfordierenttypesoftime-ausalsale-spaeonepts.Then,anadaptationofapre-viouslyproposedheuristipriniplewillpresented,statingthatintheabseneoffurtherinformation,importantluesforspatio-temporalsaleseletionanbeobtainedfromthesalesatwhih(possiblynon-linear)ombinationsofnormal-izedspatio-temporalderivativesassumemaximaoversales.Speially,itwillbeshownhowthisapproahappliestomotiondetetionandveloityestimation.2Spatialandtemporalsale-spae:OverviewTraditionally,mostworkonsale-spaerepresentationhasbeenonernedwiththespatialdomain,inwhihthevaluesoftheinputsignalareavailableinalloordinatediretions.GivenanyD-dimensionalsignalf:IRD!IR,its(spatial)sale-spaerepresentationL:IRDIR+!IRisdenedbyonvolutionL(;s)=g(;s)f(1)withthe(rotationallysymmetri)Gaussiankernelg(x;s)=1(2s)N=2exTx=2s(2)andsale-spaederivativesaredenedfromthisrepresentationbyLx(;s)=xL(;s)wheres2IR+isthesaleparameterand=(1;:::;D)representstheorderofdierentiation.Ashasbeenshownbyseveralauthors(Witkin1983;Koenderink1984;YuilleandPoggio1986;KoenderinkandvanDoorn1992;Florak1993;Lindeberg1994;Pauwelsetal.1995),thehoieoftheGaussiankernelanditsderivativesisbasiallyauniquehoie,givennaturalassumptionsonavisualfront-end(sale-spaeaxioms).Thissale-spaeonept,however,annotbediretlyappliedtotemporaldata,sineinareal-timesituationitisessentialthatimageoperatorsdonotextendintothefuture.Onesuggestionforhowtodealwiththisproblemwasgivenby(Koenderink1988),whoproposedtotransformthetimeaxissoastomapthepresentmomenttotheunreahableinnity.Inthetransformeddomain,hethenappliedthetraditionalsale-spaeoneptgivenby(1)and(2).Basedonalassiationofsale-spaekernelsintheontinuousanddisretedomains,whihguaranteenon-reationofloalextremaandrespetthetimediretionasausal(Lindeberg1990;LindebergandFagerstrom1996;Lindeberg1997),threeothertypesoftemporalsale-spaeapproahesanbedistinguished:Continuoustimeanddisretesaleparameter:Forontinuoustime,itturnsoutthatalltime-ausalsale-spaekernelsanbedeomposedintoonvolutionwithprimitivetrunatedexponentialkernelshprim(t;)=1et=(t0)(3)having(possiblydierent)timeonstants.Foreahsuhprimitivelter,themeanisandthevariane2.Hene,ifweoupleksuhltersinasade,theequivalentonvolutionkernelwillhaveaLaplaetransformoftheformHomposed(s;)=Z1t=1ki=1hprim(t;i)estdt=kYi=111+is;(4)withmean(timedelay)Pki=1iandvariane(eetiveintegrationtime)Pki=12i.Disretetimewithdisretesaleparameter.Thedisreteorrespondenetothetrunatedexponentialltersarerst-ordergeometrimovingaverageltersor-respondingtothereurrenerelationfout(t)fout(t1)=11+(fin(t)fout(t1)):(5)Suhaprimitivelterhasmeanandvariane2+.Couplingksuhltersinasade,givesalterwithgener