深度学习综述讨论简介deepLearning

IntroductiontoDeepLearningHuihuiLiuMar.1,2017Outline•Conceptionofdeeplearning•Developmenthistory•Deeplearningframeworks•Deepneuralnetworkarchitectures•ConvolutionalneuralnetworksIntroductionNetworkstructureTrainingtricks•ApplicationinAestheticImageEvaluation•IdeaDeepLearning(Hinton,2006)•Deeplearningisabranchofmachinelearningbasedonasetofalgorithmsthatattempttomodelhighlevelabstractionsindata.•Theadvantageofdeeplearningistoextractingfeaturesautomaticallyinsteadofextractingfeaturesmanually.ComputervisionSpeechrecognitionNaturallanguageprocessingDevelopmentHistory194319401950196019701980199020002010MPmodel1958Single-layerPerceptron1969XORproblem1986BPalgorithm1989CNN-LeNet19951997SVMLSTMGradientdisappearanceproblem19912006DBNReLU201120122015DropoutAlexNetBNFasterR-CNNResidualNetGeoffreyHintonW.S.McCullochW.PittsRosenblattMarvinMinskyYannLeCunHintonHintonHintonLeCunBengioDeepLearningFrameworksDeepneuralnetworkarchitectures•DeepBeliefNetworks(DBN)•RecurrentNeuralNetworks(RNN)•GenerativeAdversarialNetworks(GANs)•ConvolutionalNeuralNetworks(CNN)•LongShort-TermMemory(LSTM)DBN(DeepBeliefNetwork,2006)HiddenunitsandvisibleunitsEachunitisbinary(0or1).Everyvisibleunitconnectstoallthehiddenunits.Everyhiddenunitconnectstoallthevisibleunits.Therearenoconnectionsbetweenv-vandh-h.HintonGE.Deepbeliefnetworks[J].Scholarpedia,2009,4(6):5947.Fig1.RBM(restrictedBoltzmannmachine)structure.Fig2.DBN(deepbeliefnetwork)structure.•Idea?ComposedofmultiplelayersofRBM.•Howtowetraintheseadditionallayers?UnsupervisedgreedyapproachRNN(RecurrentNeuralNetwork,2013)What?RNNaimstoprocessthesequencedata.RNNwillrememberthepreviousinformationandapplyittothecalculationofthecurrentoutput.Thatis,thenodesofthehiddenlayerareconnected,andtheinputofthehiddenlayerincludesnotonlytheoutputoftheinputlayerbutalsotheoutputofthehiddenlayer.MarhonSA,CameronCJF,KremerSC.RecurrentNeuralNetworks[M]//HandbookonNeuralInformationProcessing.SpringerBerlinHeidelberg,2013:29-65.Applications?MachineTranslationGeneratingImageDescriptionsSpeechRecognitionHowtotrain?BPTT(Backpropagationthroughtime)GANs(GenerativeAdversarialNetworks,2014)GANsInspiredbyzero-sumGameinGameTheory,whichconsistsofapairofnetworks-ageneratornetworkandadiscriminatornetwork.•Thegeneratornetworkgeneratesasamplefromtherandomvector,thediscriminatornetworkdiscriminateswhetheragivensampleisnaturalorcounterfeit.•Bothnetworkstraintogethertoimprovetheirperformanceuntiltheyreachapointwherecounterfeitandrealsamplescannotbedistinguished.GoodfellowI,Pouget-AbadieJ,MirzaM,etal.Generativeadversarialnets[C]//Advancesinneuralinformationprocessingsystems.2014:2672-2680.Applacations:•Imageediting•Imagetoimagetranslation•Generatetext•Generateimagesbasedontext•Combinedwithreinforcementlearning•Andmore…LongShort-TermMemory(LSTM,1997)NeuralNetworksNeuronNeuralnetworkConvolutionalNeuralNetworks(CNN)Convolutionneuralnetworkisakindoffeedforwardneuralnetwork,whichhasthecharacteristicsofsimplestructure,lesstrainingparametersandstrongadaptability.CNNavoidsthecomplexpre-processingofimage(etc.extracttheartificialfeatures),wecandirectlyinputtheoriginalimage.Basiccomponents:ConvolutionLayers,PoolingLayers,FullyconnectedLayersConvolutionlayerTheconvolutionkerneltranslatesona2-dimensionalplane,andeachelementoftheconvolutionkernelismultipliedbytheelementatthecorrespondingpositionoftheconvolutionimageandthensumalltheproduct.Bymovingtheconvolutionkernel,wehaveanewimage,whichconsistsofthesumoftheproductoftheconvolutionkernelateachposition.localreceptivefieldweightsharingReducedthenumberofparametersPoolinglayerPoolinglayeraimstocompresstheinputfeaturemap,whichcanreducethenumberofparametersintrainingprocessandthedegreeofover-fittingofthemodel.Max-pooling:Selectingthemaximumvalueinthepoolingwindow.Mean-pooling:Calculatingtheaverageofallvaluesinthepoolingwindow.FullyconnectedlayerandSoftmaxlayerEachnodeofthefullyconnectedlayerisconnectedtoallthenodesofthelastlayer,whichisusedtocombinethefeaturesextractedfromthefrontlayers.Fig1.Fullyconnectedlayer.Fig2.CompleteCNNstructure.Fig3.Softmaxlayer.TrainingandTestingForwardpropagation-Takingasample(X,Yp)fromthesamplesetandputtheXintothenetwork;-CalculatingthecorrespondingactualoutputOp.Backpropagation-CalculatingthedifferencebetweentheactualoutputOpandthecorrespondingidealoutputYp;-Adjustingtheweightmatrixbyminimizingtheerror.Trainingstage:Testingstage:Puttingdifferentimagesandlabelsintothetrainedconvolutionneuralnetworkandcomparingtheoutputandtheactualvalueofthesample.Beforethetrainingstage,weshouldusesomedifferentsmallrandomnumberstoinitializeweights.CNNStructureEvolutionHintonBPNeocognitionLeCunLeNetAlexNetHistoricalbreakthroughReLUDropoutGPU+BigDataVGG16VGG19MSRA-NetDeepernetworkNINGoogLeNetInceptionV3InceptionV4R-CNNSPP-NetFastR-CNNFasterR-CNNInceptionV2(BN)FCNFCN+CRFSTNetCNN+RNN/LSTMResNetEnhancedthefunctionalityoftheconvolutionmoduleClassificationtaskDetectiontaskAddnewfunctionalunitintegration19801998198920142015ImageNetILSVRC(ImageNetLargeScaleVisualRecognitionChallenge)201320142