搜索
51前馈神经网络(3)52多层感知机(MLP)反向传播(BP)学习算法53多层感知机（MLP）模型FirstLayera1=f1(W1p+b1)a2=f2(W2a1+b2)a3=f3(W3a2+b3) f1 f2 f3Inputsa32n32w3S3,S2w31,1b32b31b3S3a3S3n3S3a31n31111111111p1a12n12p2p3pRw1S1,Rw11,1a1S1n1S1a11n11a22n22w2S2,S1w21,1b12b11b1S1b22b21b2S2a2S2n2S2a21n21 Σ Σ Σ Σ Σ Σ Σ Σ Σ f1 f1 f2 f2 f3 f3a3=f3(W3f2(W2f1(W1p+b1)+b2)+b3)ThirdLayerSecondLayerR–S1–S2–S3Network54基本逻辑运算的实现设计能够实现“与”、“或”和“异或”的感知机网络[][][][]}1,}{0,}{0,}{0,{4114310320121001========tptptptp[][][][]}1,}{1,}{1,}{0,{4114310320121001========tptptptp判定边界：，。•“与”操作运算：•“或”操作运算：05.121=−+pp判定边界：，。05.021=−+pp5.1],1,1[−==bW5.0],1,1[−==bW•“异或”操作运算：[][][][]}0,}{1,}{1,}{0,{4114310320121001========tptptptp判定边界？W=？b=？见《神经网络设计》第199页。55分类例子56多个判定边界第一个边界:a11hardlim1–0p0.5+()=12第二个边界:a21hardlim01–p0.75+()=第一个子网p1a12n12Inputsp2-1a11n110.5a21n2111 Σ Σ Σ 10.75 00-1-1.511IndividualDecisionsANDOperation57多个判定边界34第三个边界:a31hardlim10p1.5–()=34第四个边界:a41hardlim01p0.25–()=第二个子网络p1a14n14Inputsp21a13n13-1.5a22n2211 Σ Σ Σ 1-0.25 001-1.511IndividualDecisionsANDOperation58整个网络W11–001–1001=b10.50.751.5–0.25–=W211000011=b21.5–1.5–=W311=b30.5–=pa1a2 W1 b1 W2 b211n1n2a3n31 W3 b32x42x12x12x11x21x11x11x12x14x24x14x14x1Input2421 InitialDecisionsANDOperationsOROperationa1=hardlim(W1p+b1)a2=hardlim(W2a1+b2)a3=hardlim(W3a2+b3)59函数逼近例子pa12n12Inputw11,1a11n11w21,1b12b11b2a2n2111 Σ Σ Σw12,1w21,2 Log-SigmoidLayer LinearLayera1=logsig(W1p+b1)a2=purelin(W2a1+b2)f1n()11en–+-----------------=f2n()n=参数值w11,110=w21,110=b1110–=b2110=w11,21=w12,21=b20=510网络的响应-2-1012-10123511参数变化对网络响应的影响-2-1012-10123-2-1012-10123-2-1012-10123-2-1012-101231–w11,21≤≤1–w12,21≤≤0b2120≤≤1–b21≤≤512多层网络FirstLayer f1 f2 f3pa1a2 W1 b1 W2 b211n1n2a3n31 W3 b3S2xS1S2x1S2x1S2x1S3xS2S3x1S3x1S3x1Rx1S1xRS1x1S1x1S1x1InputRS1S2S3SecondLayerThirdLayera1=f1(W1p+b1)a2=f2(W2a1+b2)a3=f3(W3a2+b3)a3=f3(W3f2(W2f1(W1p+b1)+b2)+b3)am1+fm1+Wm1+ambm1++()=前一层的输出是后一层的输入m02…M1–,,,=a0p=网络的输入aaM=网络的输出513性能指数训练集p1t1{,}p2t2{,}…pQtQ{,},,,均方误差Fx()Ee2][=Eta–()2][=向量情况Fx()EeTe][=Eta–()Tta–()][=近似均方误差(单个样本)Fˆx()tk()ak()–()Ttk()ak()–()eTk()ek()==wij,mk1+()wij,mk()αFˆ∂wij,m∂------------–=bimk1+()bimk()αFˆ∂bim∂---------–=近似最速下降514链法则fnw()()dwd-----------------------fn()dnd--------------nw()dwd---------------×=dwwdnnnnfdwwdnnnnfdwwnwndf)(),()(),())(),((2221112121×∂∂+×∂∂=121222112111211212211),(),(),(),()),(),,((∂∂×∂∂+∂∂×∂∂=∂∂应用到梯度计算Fˆ∂wij,m∂------------Fˆ∂nim∂---------nim∂wij,m∂------------×=Fˆ∂bim∂---------Fˆ∂nim∂---------nim∂bim∂---------×=515梯度计算nimwij,majm1–j1=Sm1–∑bim+=nim∂wij,m∂------------ajm1–=nim∂bim∂---------1=敏感性：simFˆ∂nim∂---------≡梯度：Fˆ∂bim∂---------sim=Fˆ∂wij,m∂------------simajm1–=516最速下降bimk1+()bimk()αsim–=wij,mk1+()wij,mk()αsimajm1––=Wmk1+()Wmk()αsmam1–()T–=bmk1+()bmk()αsm–=smFˆ∂nm∂----------≡Fˆ∂n1m∂---------Fˆ∂n2m∂---------…Fˆ∂nSmm∂-----------=下一步:计算敏感性(反向传播)517敏感性计算(反向传播)mimSmSmimmmimmmiminnnFnnnFnnnFnFsmm∂∂∂∂++∂∂∂∂+∂∂∂∂=∂∂=++++++++111212111111ˆ...ˆˆˆ求多元复合函数的偏导数FirstLayera1=f1(W1p+b1)a2=f2(W2a1+b2)a3=f3(W3a2+b3) f1 f2 f3Inputsa32n32w3S3,S2w31,1b32b31b3S3a3S3n3S3a31n31111111111p1a12n12p2p3pRw1S1,Rw11,1a1S1n1S1a11n11a22n22w2S2,S1w21,1b12b11b1S1b22b21b2S2a2S2n2S2a21n21 Σ Σ Σ Σ Σ Σ Σ Σ Σ f1 f1 f2 f2 f3 f3a3=f3(W3f2(W2f1(W1p+b1)+b2)+b3)ThirdLayerSecondLayer518Jacobian矩阵nm1+∂nm∂-----------------n1m1+∂n1m∂----------------n1m1+∂n2m∂----------------…n1m1+∂nSmm∂----------------n2m1+∂n1m∂----------------n2m1+∂n2m∂----------------…n2m1+∂nSmm∂----------------………nSm1+m1+∂n1m∂----------------nSm1+m1+∂n2m∂----------------…nSm1+m1+∂nSmm∂----------------≡nim1+∂njm∂----------------wil,m1+alml1=Sm∑bim1++⎝⎠⎜⎟⎛⎞∂njm∂-----------------------------------------------------------wij,m1+ajm∂njm∂---------==nim1+∂njm∂----------------wij,m1+fmnjm()∂njm∂---------------------wij,m1+fÝmnjm()==fÝmnjm()fmnjm()∂njm∂---------------------=FÝmnm()fÝmn1m()0…00fÝmn2m()…0………00…fÝmnSmm()=nm1+∂nm∂-----------------Wm1+FÝmnm()=519反向传播(敏感性)smFˆ∂nm∂----------nm1+∂nm∂-----------------⎝⎠⎜⎟⎛⎞TFˆ∂nm1+∂-----------------FÝmnm()Wm1+()TFˆ∂nm1+∂-----------------===smFÝmnm()Wm1+()Tsm1+=敏感性从最后一层开始计算,然后通过网络反向传播到第一层。sMsM1–…s2s1→→→→520初始化(最后一层)siMFˆ∂niM∂----------ta–()Tta–()∂niM∂---------------------------------------tjaj–()2j1=SM∑∂niM∂-----------------------------------2tiai–()–ai∂niM∂----------====ai∂niM∂----------aiM∂niM∂----------fMniM()∂niM∂-----------------------fÝMniM()===siM2tiai–()–fÝMniM()=sM2FÝMnM()ta–()–=521小结前向传播：a0p=am1+fm1+Wm1+ambm1++()=m02…M1–,,,=aaM=反向传播：sM2FÝMnM()ta–()–=smFÝmnm()Wm1+()Tsm1+=mM1–…21,,,=权修正：Wmk1+()Wmk()αsmam1–()T–=bmk1+()bmk()αsm–=522BP算法的缺点•算法的收敛速度很慢。•可能有多个局部极小点。•BP网络的隐层神经元个数的选取尚无理论上的指导，而是根据经验选取。•BP网络是一个前向网络，具有非线性映射能力，但较之非线性动力学系统，功能上有其局限性。523BP算法的变形•启发式改进–动量–可变的学习速度•标准的数值优化–共轭梯度–牛顿法(Levenberg-Marquardt)524性能曲面例子pa12n12Inputw11,1a11n11w21,1b12b11b2a2n2111 Σ Σ Σw12,1w21,2 Log-SigmoidLayerLog-SigmoidLayera1=logsig(W1p+b1)a2=logsig(W2a1+b2) 网络结构w11,110=w21,110=b115–=b215=w11,21=w12,21=b21–=-2-10