Single-Image-Super-Resolution-Using-Multi-Scale-Co

SingleImageSuper-ResolutionUsingMulti-ScaleConvolutionalNeuralNetworkXiaoyiJia,XiangminXu()?,BolunCai,andKailingGuoSchoolofElectronicandInformationEngineeringSouthChinaUniversityofTechnology,Guangzhou,[e-mail],[e-mail],{caibolun,eecollinguo}@gmail.comAbstract.Methodsbasedonconvolutionalneuralnetwork(CNN)havedemonstratedtremendousimprovementsonsingleimagesuper-resolution.However,thepreviousmethodsmainlyrestoreimagesfromonesingleareainthelowresolution(LR)input,whichlimitstheexibilityofmodelstoinfervariousscalesofdetailsforhighresolution(HR)out-put.Moreover,mostofthemtrainaspecicmodelforeachup-scalefactor.Inthispaper,weproposeamulti-scalesuperresolution(MSSR)network.Ournetworkconsistsofmulti-scalepathstomaketheHRin-ference,whichcanlearntosynthesizefeaturesfromdierentscales.ThispropertyhelpsreconstructvariouskindsofregionsinHRimages.Inaddition,onlyonesinglemodelisneededformultipleup-scalefactors,whichismoreecientwithoutlossofrestorationquality.Experimentsonfourpublicdatasetsdemonstratethattheproposedmethodachievedstate-of-the-artperformancewithfastspeed.Keywords:Super-resolution,convolutionalneuralnetwork,multi-scale1IntroductionThetaskofsingleimagesuper-resolutionaimsatrestoringahigh-resolution(HR)imagefromagivenlow-resolution(LR)one.Super-resolutionhaswideapplicationsinmanyeldswhereimagedetailsareondemand,suchasmedi-cal,remotesensingimaging,videosurveillance,andentertainment.Inthepastdecades,super-resolutionhasattractedmuchattentionfromcomputervisioncommunities.Earlymethodsincludebicubicinterpolation[5],Lanczosresam-pling[9],statisticalpriors[15],neighborembedding[4],andsparsecoding[23].However,super-resolutionishighlyill-posedsincetheprocessfromHRtoLRcontainsnon-invertibleoperationsuchaslow-passlteringandsubsampling.Deepconvolutionalneuralnetworks(CNNs)haveachievedstate-of-the-artperformanceincomputervision,suchasimageclassication[20],objectdetec-tion[10],andimageenhancement[3].Recently,CNNsarewidelyusedtoaddress?ThisworkissupportedbytheNationalNaturalScienceFoundationofChina(61171142,61401163,U1636218),theScienceandtechnologyPlanningProjectofGuangdongProvinceofChina(2014B010111003,2014B010111006),GuangzhouKeyLabofBodyDataScience(201605030011).arXiv:1705.05084v1[cs.CV]15May20172X.Jiaetal.theill-posedinverseproblemofsuper-resolution,andhavedemonstratedsupe-riorityovertraditionalmethods[9,15,4,23]withrespecttobothreconstructionaccuracyandcomputationaleciency.Dongetal.[6,7]successfullydesignasuper-resolutionconvolutionalneuralnetwork(SRCNN)todemonstratethataCNNcanbeappliedtolearnthemappingfromLRtoHRinanend-to-endmanner.Afastsuper-resolutionconvolutionalneuralnetwork(FSRCNN)[8]isproposedtoacceleratethespeedofSRCNN[6,7],whichtakestheoriginalLRimageasinputandadoptsadeconvolutionlayertoreplacethebicubicinterpo-lation.In[19],anecientsub-pixelconvolutionlayerisintroducedtoachieverealtimeperformance.Kimetal.[14]usesaverydeepsuper-resolution(VDSR)networkwith20convolutionallayers,whichgreatlyimprovestheaccuracyofthemodel.ThepreviousmethodsbasedonCNNhasachievedgreatprogressontherestorationqualityaswellaseciency.However,therearesomelimitationsmainlycomingfromthefollowingaspects:{CNNbasedmethodsmakeeortstoenlargethereceptiveeldofthemodelsaswellasstackmorelayers.TheyreconstructanytypeofcontentsfromLRimagesusingonlysingle-scaleregion,thusignorethevariousscalesofdierentdetails.Forinstance,restoringthedetailintheskyprobablyreliesonalagerimageregion,whilethetinytextmayonlyberelevanttoasmallpatch.{Mostpreviousapproacheslearnaspecicmodelforonesingleup-scalefac-tor.Therefore,themodellearnedforoneup-scalefactorcannotworkwellforanother.Thatis,manyscale-specicmodelsshouldbetrainedfordier-entup-scalefactors,whichisinecientbothintermsoftimeandmemory.Though[14]trainsamodelformultipleup-scales,itignoresthefactthatasinglereceptiveeldmaycontaindierentinformationamountinvariousresolutionversions.Inthispaper,weproposeamulti-scalesuperresolution(MSSR)convolu-tionalneuralnetworktoissuetheseproblems{therearetwofoldsofmeaninginthetermmulti-scale.First,theproposednetworkcombinesmulti-pathsubnet-workswithdierentdepth,whichcorrespondtomulti-scaleregionsintheinputimage.Second,themulti-scalenetworkiscapabletoselectaproperreceptiveeldfordierentup-scalestorestoretheHRimage.Onlysinglemodelistrainedformultipleup-scalefactorsbymulti-scaletraining.2Mutli-scaleSuper-ResolutionGivenalow-resolutionimage,super-resolutionaimsatrestoringitshigh-resolutionversion.Forthisill-posedrecoveryproblem,itisprobablyaneectivewaytoes-timateatargetpixelbytakingintoaccountmorecontextinformationintheneighborhood.In[6,7,14],authorsfoundthatlargerreceptiveeldtendstoachievebetterperformanceduetoricherstructuralinformation.However,weSingleImageSuper-ResolutionUsingMulti-ScaleCNN3arguethattherestorationprocessisnotonlydependingonsingle-scaleregionswithlargereceptiveeld.Dierentkindsofcomponentsinanimagemayberelevanttodierentscalesofneighbourhood.In[26],multi-scaleneighborhoodhasbeenproveneectiveforsuper-resolution,whichsimultaneousl