Local features and kernels for classification of t

Localfeaturesandkernelsforclassicationoftextureandobjectcategories:AcomprehensivestudyJ.Zhang1,M.Marszalek1,S.Lazebnik2andC.Schmid1,1INRIA,GRAVIR-CNRS,655,av.del'Europe,38330Montbonnot,France2BeckmanInstitute,UniversityofIllinois,405N.MathewsAve.,Urbana,IL61801,USAAbstractRecently,methodsbasedonlocalimagefeatureshaveshownpromisefortextureandobjectrecog-nitiontasks.Thispaperpresentsalarge-scaleevaluationofanapproachthatrepresentsimagesasdistributions(signaturesorhistograms)offeaturesextractedfromasparsesetofkeypointlocationsandlearnsaSupportVectorMachineclassierwithkernelsbasedontwoeectivemeasuresforcomparingdistributions,theEarthMover'sDistanceandthe2distance.Werstevaluatetheper-formanceofourapproachwithdierentkeypointdetectorsanddescriptors,aswellasdierentkernelsandclassiers.Wethenconductacomparativeevaluationwithseveralstate-of-the-artrecognitionmethodsonfourtextureandveobjectdatabases.Onmostofthesedatabases,ourimplementationexceedsthebestreportedresultsandachievescomparableperformanceontherest.Finally,weinves-tigatetheinuenceofbackgroundcorrelationsonrecognitionperformanceviaextensivetestsonthePASCALdatabase,forwhichground-truthobjectlocalizationinformationisavailable.Ourexperi-mentsdemonstratethatimagerepresentationsbasedondistributionsoflocalfeaturesaresurprisinglyeectiveforclassicationoftextureandobjectimagesunderchallengingreal-worldconditions,in-cludingsignicantintra-classvariationsandsubstantialbackgroundclutter.Keywords:imageclassication,texturerecognition,objectrecognition,scale-andane-invariantkeypoints,supportvectormachines,kernelmethods.1IntroductionTherecognitionoftextureandobjectcategoriesisoneofthemostchallengingproblemsincomputervision,especiallyinthepresenceofintra-classvariation,clutter,occlusion,andposechanges.Histori-cally,textureandobjectrecognitionhavebeentreatedastwoseparateproblemsintheliterature.Itiscustomarytodenetextureasavisualpatterncharacterizedbytherepetitionofafewbasicprimitives,ortextons[27].Accordingly,manyeectivetexturerecognitionapproaches[8,31,33,57,58]obtaintex-tonsbyclusteringlocalimagefeatures(i.e.,appearancedescriptorsofrelativelysmallneighborhoods),andrepresenttextureimagesashistogramsordistributionsoftheresultingtextons.Notethattheseapproachesareorderless,i.e.,theyretainonlythefrequenciesoftheindividualfeatures,anddiscardallinformationabouttheirspatiallayout.Ontheotherhand,theproblemofobjectrecognitionhastypi-callybeenapproachedusingparts-and-shapemodelsthatrepresentnotonlytheappearanceofindividualobjectcomponents,butalsothespatialrelationsbetweenthem[1,17,18,19,60].However,recentliter-aturealsocontainsseveralproposalstorepresentthe\visualtextureofimagescontainingobjectsusingorderlessbag-of-featuresmodels.Suchmodelshaveproventobeeectiveforobjectclassication[7,61],unsuperviseddiscoveryofcategories[16,51,55],andvideoretrieval[56].Thesuccessoforderlessmodelsfortheseobjectrecognitiontasksmaybeexplainedwiththehelpofananalogytobag-of-wordsmodelsfortextdocumentclassication[40,46].Whereasfortexturerecognition,localfeaturesplaytheroleoftextons,orfrequentlyrepeatedelements,forobjectrecognitiontasks,localfeaturesplaytheroleof\visualwordspredictiveofacertain\topic,orobjectclass.Forexample,aneyeishighlypredictiveofafacebeingpresentintheimage.Ifourvisualdictionarycontainswordsthataresucientlydiscrim-inativewhentakenindividually,thenitispossibletoachieveahighdegreeofsuccessforwhole-imageclassication,i.e.,identicationoftheobjectclasscontainedintheimagewithoutattemptingtosegment1orlocalizethatobject,simplybylookingwhichvisualwordsarepresent,regardlessoftheirspatiallayout.Overall,thereisanemergingconsensusinrecentliteraturethatorderlessmethodsareeectiveforbothtextureandobjectdescription,anditcreatestheneedforalarge-scalequantitativeevaluationofasingleapproachtestedonmultipletextureandobjectdatabases.Todate,state-of-the-artresultsinbothtexture[31]andobjectrecognition[18,23,48,61]havebeenobtainedwithlocalfeaturescomputedatasparsesetofscale-orane-invariantkeypointlocationsfoundbyspecializedinterestoperators[34,43].Atthesametime,SupportVectorMachine(SVM)classiers[54]haveshowntheirpromiseforvisualclassicationtasks(see[50]foranearlyexample),andthedevelopmentofkernelssuitableforusewithlocalfeatureshasemergedasafruitfullineofre-search[4,13,23,37,47,59].Mostexistingevaluationsofmethodscombiningkernelsandlocalfeatureshavebeensmall-scaleandlimitedtooneortwodatasets.Moreover,thebackgroundsinmanyofthesedatasets,suchasCOIL-100[44]orETH-80[32]areeither(mostly)uniformorhighlycorrelatedwiththeforegroundobjects,sothattheperformanceofthemethodsonchallengingreal-worldimagerycannotbeassessedaccurately.Thismotivatesustobuildaneectiveimageclassicationapproachcombiningabag-of-keypointsrepresentationwithakernel-basedlearningmethodandtotestthelimitsofitsperfor-manceonthemostchallengingdatabasesavailabletoday.Ourstudyconsistsofthreecomponents:Evaluationofimplementationchoices.Inthispaper,weplaceaparticularemphasisonproducingacarefullyengineeredrecognitionsystem,whereeverycomponen