A performance evaluation of local descriptors

MIKOLAJCZYKANDSCHMID:APERFORMANCEEVALUATIONOFLOCALDESCRIPTORS1AperformanceevaluationoflocaldescriptorsKrystianMikolajczykandCordeliaSchmidDept.ofEngineeringScienceINRIARhˆone-AlpesUniversityofOxford655,av.del'EuropeOxford,[e-mail]@inrialpes.frAbstractInthispaperwecomparetheperformanceofdescriptorscomputedforlocalinterestregions,asforexampleextractedbytheHarris-Afnedetector[32].Manydifferentdescriptorshavebeenproposedintheliterature.However,itisunclearwhichdescriptorsaremoreappropriateandhowtheirperformancedependsontheinterestregiondetector.Thedescriptorsshouldbedistinctiveandatthesametimerobusttochangesinviewingconditionsaswellastoerrorsofthedetector.Ourevaluationusesascriterionrecallwithrespecttoprecisionandiscarriedoutfordifferentimagetransformations.Wecompareshapecontext[3],steerablelters[12],PCA-SIFT[19],differentialinvariants[20],spinimages[21],SIFT[26],complexlters[37],momentinvariants[43],andcross-correlationfordifferenttypesofinterestregions.WealsoproposeanextensionoftheSIFTdescriptor,andshowthatitoutperformstheoriginalmethod.Furthermore,weobservethattherankingofthedescriptorsismostlyindependentofthepointdetectorandthattheSIFTbaseddescriptorsperformbest.Momentsandsteerableltersshowthebestperformanceamongthelowdimensionaldescriptors.IndexTermsLocaldescriptors,interestpoints,interestregions,invariance,matching,recognition.I.INTRODUCTIONLocalphotometricdescriptorscomputedforinterestregionshaveprovedtobeverysuccessfulinapplicationssuchaswidebaselinematching[37,42],objectrecognition[10,25],textureCorrespondingauthorisK.Mikolajczyk,[e-mail]6,2004DRAFTMIKOLAJCZYKANDSCHMID:APERFORMANCEEVALUATIONOFLOCALDESCRIPTORS2recognition[21],imageretrieval[29,38],robotlocalization[40],videodatamining[41],buildingpanoramas[4],andrecognitionofobjectcategories[8,9,22,35].Theyaredistinctive,robusttoocclusionanddonotrequiresegmentation.Recentworkhasconcentratedonmakingthesedescriptorsinvarianttoimagetransformations.Theideaistodetectimageregionscovarianttoaclassoftransformations,whicharethenusedassupportregionstocomputeinvariantdescriptors.Giveninvariantregiondetectors,theremainingquestionsarewhichisthemostappropriatedescriptortocharacterizetheregions,anddoesthechoiceofthedescriptordependontheregiondetector.Thereisalargenumberofpossibledescriptorsandassociateddistancemeasureswhichemphasizedifferentimagepropertieslikepixelintensities,color,texture,edgesetc.Inthisworkwefocusondescriptorscomputedongray-valueimages.Theevaluationofthedescriptorsisperformedinthecontextofmatchingandrecognitionofthesamesceneorobjectobservedunderdifferentviewingconditions.Wehaveselectedanumberdescriptors,whichhavepreviouslyshownagoodperformanceinsuchacontextandcomparethemusingthesameevaluationscenarioandthesametestdata.Theevaluationcriterionisrecall-precision,i.e.thenumberofcorrectandfalsematchesbetweentwoimages.AnotherpossibleevaluationcriterionistheROC(ReceiverOperatingCharacteristics)inthecontextofimageretrievalfromdatabases[6,31].Thedetectionrateisequivalenttorecallbutthefalsepositiverateiscomputedforadatabaseofimagesinsteadofasingleimagepair.Itisthereforedifculttopredicttheactualnumberoffalsematchesforapairofsimilarimages.Localfeatureswerealsosuccessfullyusedforobjectcategoryrecognitionandclassication.Thecomparisonofdescriptorsinthiscontextrequiresadifferentevaluationsetup.However,itisunclearhowtoselectarepresentativesetofimagesforanobjectcategoryandhowtopreparethegroundtruth,sincethereisnolineartransformationrelatingimageswithinacategory.Apossiblesolutionistoselectmanuallyafewcorrespondingpointsandapplylooseconstraintstoverifycorrectmatches,asproposedin[18].Inthispaperthecomparisoniscarriedoutfordifferentdescriptors,differentinterestregionsandfordifferentmatchingapproaches.Comparedtoourpreviouswork[31],thispaperperformsamoreexhaustiveevaluationandintroducesanewdescriptor.Severaldescriptorsanddetectorshavebeenaddedtothecomparisonandthedatasetcontainsalargervarietyofscenestypesandtransformations.Wehavemodiedtheevaluationcriterionandnowuserecall-precisionforimagepairs.TherankingofthetopdescriptorsisthesameasintheROCbasedevaluation[31].October6,2004DRAFTMIKOLAJCZYKANDSCHMID:APERFORMANCEEVALUATIONOFLOCALDESCRIPTORS3Furthermore,ournewdescriptor,gradientlocationandorientationhistogram(GLOH),whichisanextensionoftheSIFTdescriptor,isshowntooutperformSIFTaswellastheotherdescriptors.A.RelatedworkPerformanceevaluationhasgainedmoreandmoreimportanceincomputervision[7].Inthecontextofmatchingandrecognitionseveralauthorshaveevaluatedinterestpointdetectors[14,30,33,39].Theperformanceismeasuredbytherepeatabilityrate,thatisthepercentageofpointssimultaneouslypresentintwoimages.Thehighertherepeatabilityratebetweentwoimages,themorepointscanpotentiallybematchedandthebetterarethematchingandrecognitionresults.Verylittleworkhasbeendoneontheevaluationoflocaldescriptorsinthecontextofmatchingandrecognition.CarneiroandJepson[6]evaluatetheperformanceofpointdescriptorsusingROC(ReceiverOperatingChara