cs231n-2017-lecture5

Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Lecture5:ConvolutionalNeuralNetworks1Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-AdministrativeAssignment1dueThursdayApril20,11:59pmonCanvasAssignment2willbereleasedThursday2Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-3Linearscorefunction:2-layerNeuralNetworkLasttime:NeuralNetworksxhW1sW23072100103Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-4Next:ConvolutionalNeuralNetworksIllustrationofLeCunetal.1998fromCS231n2017Lecture14Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,20175FrankRosenblatt,~1957:PerceptronTheMarkIPerceptronmachinewasthefirstimplementationoftheperceptronalgorithm.Themachinewasconnectedtoacamerathatused20×20cadmiumsulfidephotocellstoproducea400-pixelimage.recognizedlettersofthealphabetupdaterule:Abitofhistory...ThisimagebyRockyAcostaislicensedunderCC-BY3.0Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,20176WidrowandHoff,~1960:Adaline/MadalineAbitofhistory...ThesefiguresarereproducedfromWidrow1960,StanfordElectronicsLaboratoriesTechnicalReportwithpermissionfromStanfordUniversitySpecialCollections.Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,20177Rumelhartetal.,1986:Firsttimeback-propagationbecamepopularrecognizablemathAbitofhistory...IllustrationofRumelhartetal.,1986byLaneMcIntosh,copyrightCS231n2017Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,20178[HintonandSalakhutdinov2006]ReinvigoratedresearchinDeepLearningAbitofhistory...IllustrationofHintonandSalakhutdinov2006byLaneMcIntosh,copyrightCS231n2017Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,20179FirststrongresultsAcousticModelingusingDeepBeliefNetworksAbdel-rahmanMohamed,GeorgeDahl,GeoffreyHinton,2010Context-DependentPre-trainedDeepNeuralNetworksforLargeVocabularySpeechRecognitionGeorgeDahl,DongYu,LiDeng,AlexAcero,2012IllustrationofDahletal.2012byLaneMcIntosh,copyrightCS231n2017FigurescopyrightAlexKrizhevsky,IlyaSutskever,andGeoffreyHinton,2012.Reproducedwithpermission.ImagenetclassificationwithdeepconvolutionalneuralnetworksAlexKrizhevsky,IlyaSutskever,GeoffreyEHinton,2012Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-10Abitofhistory:Hubel&Wiesel,1959RECEPTIVEFIELDSOFSINGLENEURONESINTHECAT'SSTRIATECORTEX1962RECEPTIVEFIELDS,BINOCULARINTERACTIONANDFUNCTIONALARCHITECTUREINTHECAT'SVISUALCORTEX1968...CatimagebyCNXOpenStaxislicensedunderCCBY4.0;changesmade10Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-11AbitofhistoryTopographicalmappinginthecortex:nearbycellsincortexrepresentnearbyregionsinthevisualfieldRetinotopyimagescourtesyofJesseGomezintheStanfordVision&PerceptionNeuroscienceLab.HumanbrainVisualcortex11Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-12HierarchicalorganizationIllustrationofhierarchicalorganizationinearlyvisualpathwaysbyLaneMcIntosh,copyrightCS231n201712Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-13Abitofhistory:Neocognitron[Fukushima1980]“sandwich”architecture(SCSCSC…)simplecells:modifiableparameterscomplexcells:performpooling13Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-14Abitofhistory:Gradient-basedlearningappliedtodocumentrecognition[LeCun,Bottou,Bengio,Haffner1998]LeNet-514Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-FigurecopyrightAlexKrizhevsky,IlyaSutskever,andGeoffreyHinton,2012.Reproducedwithpermission.15Abitofhistory:ImageNetClassificationwithDeepConvolutionalNeuralNetworks[Krizhevsky,Sutskever,Hinton,2012]“AlexNet”15Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,201716Fast-forwardtotoday:ConvNetsareeverywhereClassificationRetrievalFigurescopyrightAlexKrizhevsky,IlyaSutskever,andGeoffreyHinton,2012.Reproducedwithpermission.Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,2017FigurescopyrightShaoqingRen,KaimingHe,RossGirschick,JianSun,2015.Reproducedwithpermission.17Fast-forwardtotoday:ConvNetsareeverywhere[FasterR-CNN:Ren,He,Girshick,Sun2015]DetectionSegmentation[Farabetetal.,2012]FigurescopyrightClementFarabet,2012.Reproducedwithpermission.Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,201718Fast-forwardtotoday:ConvNetsareeverywhereNVIDIATeslaline(thesearetheGPUsonrye01.stanford.edu)NotethatforembeddedsystemsatypicalsetupwouldinvolveNVIDIATegras,withintegratedGPUandARM-basedCPUcores.self-drivingcarsPhotobyLaneMcIntosh.CopyrightCS231n2017.ThisimagebyGBPublic_PRislicensedunderCC-BY2.0Fei-FeiLi&JustinJohnson&SerenaYeungApril18,2017Lecture5-Fei-FeiLi&JustinJohnson&SerenaYeungLecture5-April18,201719Fast-forwardtotoday:ConvNetsareeverywhere[Taigmanetal.2014][Simonyanetal.2014]Figurescop