国科大中科院人工智能与机器学习12-CNN_Beyond42

卷积神经网络CNN及其变种山世光中科院计算所InstituteofComputingTechnology,ChineseAcademyofSciencesCNN的早期历史卷积神经网络CNNK.Fukushima,“Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition,”BiologicalCybernetics,vol.36,pp.193–202,1980Y.LeCun,B.Boser,J.S.Denker,D.Henderson,R.E.Howard,W.Hubbard,andL.D.Jackel,“Backpropagationappliedtohandwrittenzipcoderecognition,”NeuralComputation,vol.1,no.4,pp.541–551,1989Y.LeCun,L.Bottou,Y.Bengio,andP.Haffner,“Gradient-basedlearningappliedtodocumentrecognition,”ProceedingsoftheIEEE,vol.86,no.11,pp.2278–2324,19982InstituteofComputingTechnology,ChineseAcademyofSciencesDL时代的CNN扩展AKrizhevsky,ISutskever,GEHinton.ImageNetclassificationwithdeepconvolutionalneuralnetworks.NIPS2012Y.Jiaetal.Caffe:ConvolutionalArchitectureforFastFeatureEmbedding.ACMMM2014K.Simonyan,A.Zisserman.Verydeepconvolutionalnetworksforlarge-scaleimagerecognition.arXivpreprintarXiv:1409.1556,2014C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,A.Rabinovich.Goingdeeperwithconvolutions.CVPR2015(&arXiv:1409.4842,2014)3InstituteofComputingTechnology,ChineseAcademyofSciences卷积——示例4InstituteofComputingTechnology,ChineseAcademyofSciences卷积——形式化积分形式𝑠𝑡=𝑥𝑎𝑤𝑡−𝑎𝑑𝑎常用表达式𝑠𝑡=(𝑥∗𝑤)(𝑡)离散形式一维情况𝑠𝑡=𝑥∗𝑤𝑡=𝑥𝑎𝑤(𝑡−𝑎)+∞𝑎=−∞二维情况𝑠𝑖,𝑗=𝐼∗𝐾[𝑖,𝑗]=𝐼𝑚,𝑛𝐾[𝑖−𝑚,𝑗−𝑛]𝑛𝑚𝑠𝑖,𝑗=𝐼∗𝐾[𝑖,𝑗]=𝐼𝑖−𝑚,𝑗−𝑛𝐾[𝑚,𝑛]𝑛𝑚K称为kernel5InstituteofComputingTechnology,ChineseAcademyofSciences卷积——why?1.sparseinteractions有限连接，Kernel比输入小连接数少很多，学习难度小，计算复杂度低m个节点与n个节点相连O(mn)限定k(m)个节点与n个节点相连，则为O(kn)6InstituteofComputingTechnology,ChineseAcademyofSciences卷积——why?1.sparseinteractions有限连接，Kernel比输入小连接数少很多，学习难度小，计算复杂度低m个节点与n个节点相连O(mn)限定k(m)个节点与n个节点相连，则为O(kn)7InstituteofComputingTechnology,ChineseAcademyofSciences卷积——why?1.sparseinteractions有限(稀疏)连接Kernel比输入小局部连接连接数少很多学习难度小计算复杂度低层级感受野（生物启发）越高层的神经元，感受野越大8InstituteofComputingTechnology,ChineseAcademyofSciences卷积——why?2.ParameterSharing（参数共享）Tiedweights进一步极大的缩减参数数量3.Equivariantrepresentations等变性配合Pooling可以获得平移不变性对scale和rotation不具有此属性9InstituteofComputingTechnology,ChineseAcademyofSciencesCNN的基本结构三个步骤卷积突触前激活，net非线性激活DetectorPoolingLayer的两种定义复杂定义简单定义有些层没有参数10InstituteofComputingTechnology,ChineseAcademyofSciencesPooling11定义（没有需要学习的参数）replacestheoutputofthenetatacertainlocationwithasummarystatisticofthenearbyoutputs种类maxpooling(weighted)averagepoolingInstituteofComputingTechnology,ChineseAcademyofSciencesWhyPooling?12获取不变性小的平移不变性：有即可，不管在哪里很强的先验假设ThefunctionthelayerlearnsmustbeinvarianttosmalltranslationsInstituteofComputingTechnology,ChineseAcademyofSciencesWhyPooling?13获取不变性小的平移不变性：有即可，不管在哪里旋转不变性？9个不同朝向的kernels（模板）0.20.610.10.50.30.020.050.1InstituteofComputingTechnology,ChineseAcademyofSciencesWhyPooling?14获取不变性小的平移不变性：有即可，不管在哪里旋转不变性？9个不同朝向的kernels（模板）0.50.30.0210.40.30.60.30.1InstituteofComputingTechnology,ChineseAcademyofSciencesPooling与下采样结合更好的获取平移不变性更高的计算效率（减少了神经元数）15InstituteofComputingTechnology,ChineseAcademyofSciences从全连接到有限连接部分链接权重被强制设置为0通常：非邻接神经元，仅保留相邻的神经元全连接网络的特例，大量连接权重为016InstituteofComputingTechnology,ChineseAcademyofSciencesWhyConvolution&Pooling？apriorprobabilitydistributionovertheparametersofamodelthatencodesourbeliefsaboutwhatmodelsarereasonable,beforewehaveseenanydata.模型参数的先验概率分布(Nofreelunch)在见到任何数据之前，我们的信念（经验）告诉我们，什么样的模型参数是合理的Localconnections；对平移的不变性；tiedweigts来自生物神经系统的启发17InstituteofComputingTechnology,ChineseAcademyofSciences源起：Neocognitron(1980)SimplecomplexLowerorderhighorder18K.Fukushima,“Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition,”BiologicalCybernetics,vol.36,pp.193–202,1980LocalConnectionInstituteofComputingTechnology,ChineseAcademyofSciences源起：Neocognitron(1980)19InstituteofComputingTechnology,ChineseAcademyofSciences源起：Neocognitron(1980)训练方法分层自组织competitivelearning无监督输出层独立训练有监督20InstituteofComputingTechnology,ChineseAcademyofSciencesLeCun-CNN1989—用于字符识别简化了Neocognitron的结构训练方法监督训练BP算法正切函数收敛更快，SigmoidLoss，SGD用于邮编识别大量应用21InstituteofComputingTechnology,ChineseAcademyofSciencesLeCun-CNN1989—用于字符识别输入16x16图像L1—H112个5x5kernel8x8个神经元L2--H212个5x5x8kernel4x4个神经元L3—H330个神经元L4—输出层10个神经元总连接数5*5*12*64+5*5*8*12*16+192*30，约66,000个22InstituteofComputingTechnology,ChineseAcademyofSciencesLeCun-CNN1989—用于字符识别Tiedweights对同一个featuremap，kernel对不同位置是相同的！23InstituteofComputingTechnology,ChineseAcademyofSciencesLeCun-CNN1989—用于字符识别24InstituteofComputingTechnology,ChineseAcademyofSciences1998年LeNet——数字/字符识别LeNet-5Featuremapasetofunitswhoseweighsareconstrainedtobeidentical.25InstituteofComputingTechnology,ChineseAcademyofSciences1998年LeNet——数字/字符识别例如：C3层参数个数(3*6+4*9+6*1)*25+16=151626InstituteofComputingTechnology,ChineseAcademyofSciences后续：CNN用于目标检测与识别27InstituteofComputingTechnology,ChineseAcademyofSciencesAlexNetforImageNet(2012)大规模CNN网络650K神经元60M参数使用了各种技巧DropoutDataaugmentReLULocalResponseNormalizationContrastnormalization...28Krizhevsky,Alex,I