A statistical approach to material classification

IEEEPAMI,VOL.X,NO.Y,APRIL20061AStatisticalApproachToMaterialClassificationUsingImagePatchExemplarsManikVarmaAndrewZissermanMicrosoftResearchDept.ofEngineeringScienceBangaloreUniversityofOxfordIndia560080Oxford,UKOX13PJMay18,2007DRAFT2IEEEPAMI,VOL.X,NO.Y,APRIL2006AbstractInthispaper,weinvestigatematerialclassificationfromsingleimagesobtainedunderunknownviewpointandillumination.Itisdemonstratedthatmaterialscanbeclassifiedusingthejointdistributionofintensityvaluesoverextremelycompactneighbourhoods(startingfromassmallas3×3pixelssquare),andthatthisoutperformsclassificationusingfilterbankswithlargesupport.Itisalsoshownthattheperformanceoffilterbanksisinferiortothatofimagepatcheswithequivalentneighbourhoods.Wedevelopnoveltextonbasedrepresentationswhicharesuitedtomodellingthisjointneighbour-hooddistributionforMRFs.Therepresentationsarelearntfromtrainingimages,andthenusedtoclassifynovelimages(withunknownviewpointandlighting)intotextureclasses.Threesuchrepresentationsareproposed,andtheirperformanceisassessedandcomparedtothatoffilterbanks.Thepowerofthemethodisdemonstratedbyclassifying2806imagesofall61materialspresentintheColumbia-Utrechtdatabase.TheclassificationperformancesurpassesthatofrecentstateoftheartfilterbankbasedclassifierssuchasLeungandMalik(IJCV01),CulaandDana(IJCV04),andVarmaandZisserman(IJCV05).WealsobenchmarkperformancebyclassifyingallthetexturespresentintheMicrosoftTextiledatabaseaswellastheSanFranciscooutdoordataset.Weconcludewithdiscussionsonwhyfeaturesbasedoncompactneighbourhoodscancorrectlydiscriminatebetweentextureswithlargeglobalstructureandwhytheperformanceoffilterbanksisnotsuperiortothesourceimagepatchesfromwhichtheywerederived.IndexTermsMaterialclassification,3Dtextures,textons,imagepatches,filterbanks.1.INTRODUCTIONOurobjective,inthispaper,istheclassificationofmaterialsfromtheirappearanceinsingleimagestakenunderunknownviewpointandilluminationconditions.Thetaskisdifficultasmaterialstypicallyexhibitlargeintra-class,andsmallinter-class,variability(seeFigure1)andtherearen’tanywidelyapplicableyetmathematicallyrigorousmodelswhichaccountforsuchtransformations.Thetaskismadeevenmorechallengingifnoaprioriknowledgeabouttheimagingconditionsisavailable.Earlyinterestinthetextureclassificationproblemfocusedonthepre-attentivediscriminationoftexturepatternsinbinaryimages[1],[24],[25],[36].Lateron,thisevolvedtotheclassificationoftexturesingreyscaleimageswithsynthetic2Dvariations[18],[20],[45].This,inturn,hasbeensupersededbytheproblemofclassifyingrealworldtextureswith3DvariationsduetoDRAFTMay18,2007VARMAANDZISSERMAN:ASTATISTICALAPPROACHTOMATERIALCLASSIFICATIONUSINGIMAGEPATCHEXEMPLARS3Fig.1.SingleimageclassificationontheColumbia-Utrechtdatabaseisademandingtask.Inthetoprow,thereisaseachangeinappearance(duetovariationinilluminationandpose)eventhoughalltheimagesbelongtothesametextureclass.Thisillustrateslargeintraclassvariation.Inthebottomrow,severaloftheimageslooksimilarandyetbelongtodifferenttextureclasses.Thisillustratesthatthedatabasealsohassmallinterclassvariation.changesincameraposeandillumination[4],[9],[27],[28],[42],[52].Currently,effortsareonextendingtheproblemtotheaccurateclassificationofentiretexturecategoriesratherthanofspecificmaterialinstances[7],[21].Anotherinterestingtrendinvestigateshowregularityinformationcanbeexploitedfortheanalysisofnearregulartextures[22],[31],[33],[34].Acommonthreadthroughthisevolutionhasbeenthesuccessthatfilterbankbasedmethodshavehadintacklingtheproblem.Astheproblemhasbecomemoredifficult,suchmethodshavecopedbybuildingricherrepresentationsoffilterresponses.Theuseoflargesupportfilterbankstoextracttexturefeaturesatmultiplescalesandorientationshasgainedwideacceptance.However,inthispaper,wequestionthedominantrolethatfilterbankshavecometoplayinthefieldoftextureclassification.Insteadofapplyingfilterbanks,wedevelopanalternativeimagepatchrepresentationbasedonthejointdistributionofpixelintensitiesinaneighbourhood.Wefirstinvestigatetheadvantagesofthisimagepatchrepresentationempirically.TheVZalgorithm[52]givesoneofthebest3DtextureclassificationresultsontheColumbia-UtrechtdatabaseusingtheMaximumResponse8(MR8)filterswithsupportaslargeas49×49pixelssquare.WedemonstratethatsubstitutingthenewpatchbasedrepresentationintheVZalgorithmleadstothefollowingtworesults:(i)verygoodclassificationperformancecanbeachievedusingextremelycompactneighbourhoods(startingfromassmallas3×3)andthat(ii)foranyfixedsizeoftheneighbourhood,imagepatchesleadtosuperiorclassificationascomparedtofilterbankswiththesamesupport.Thesuperiorityoftheimagepatchrepresentationisempiricallydemonstratedbyclassifyingall61materialspresentintheColumbia-UtrechtdatabaseandMay18,2007DRAFT4IEEEPAMI,VOL.X,NO.Y,APRIL2006showingthattheresultsoutperformtheVZalgorithmusingtheMR8filterbank.ClassificationresultsarealsopresentedfortheSanFrancisco[27]andMicrosoftTextile[40]databases.Wethendiscusstheoreticalreasonsastowhysmallimagepatchescancorrectlydiscriminatebetweentextureswithlargeglobalstructureandalsochallengethepopularbeliefthatfilterbankfeaturesaresuperiorforclassificationascomparedt