10第十章---神经网络剪枝方法

10Reed[Reed1993]Skeletonization10.17)()()(λ+=)(WE)(WC)(WC∑=2)(iwCWGauss∑=iwC)(WLaplace()()∑=+=NjjwC1221log1ααWCauchyλ[-199]Weigend[WeRu1991]Weigend[WeRu1991]Weight-Elimination10.1.1(1)(2)Weight-EliminationRissanen[Riss1978](|)()Weigend21ErrErrErr+=10.11Err2Err()∑=−=sNjijijtyErr12110.2∑+=jjj()2022021=10.40/wwi10.1()0wwCj0wwj0wwi()0wwCj10wwi()0wwCj00wwi0wwi10.10w1w2w21wwS+=0wSk=Sw1=α()()()()()()2222211111kkkkwCwCαααα−+−++=+10.510.2()()21wCwC+αkk1.1()()21wCwC+5.0=α21ww=k()()21wCwC+0=α1=αS0w0w10.2)(mod)mod|()|(modelpeldatapdataelp∗=10.6)mod|(eldatap)(modelp+−=2022021exp)(−=λ1exp)(jwf0wwi10.7−=λ202exp)(=λ0w010.3λ0.050.20.51.02.0λ0wwi10.3λjw()()211jjjjwwkwkw∆+∆+−=λ10.8jjwErrw∂∂−=∆11η10.9jwErr∂∂1RumelhartBP()2202202212−=∂∂−=∆10.10:(1)(2)(10.1)10.8,(3)10.1.2λz()1−tEz()tA()()()()tEtAtAµµ−+−=11µ1zD0=D()tE()tA8λz()()1−tEtE()DtE()()λλλ∆+−=1ttz()()1−≥tEtE()()tAtE()DtE≥()()λλλ∆−−=1tt()()1−≥tEtE()()tAtEz()()1−≥tEtE()()tAtE≥()DtE≥()()1−=ttρλλρ1λ010.1.3SISO)1(102)1(102)1(4)1(43)1(8)1(16sin4029)(22−+−+−+−+−+−=kykukykukykuky10.11)(ku[]1,1−0)0(=y20010010010.410.522013BP)1(−ku)1(−ky)(ky[]1.0,1.0−50000.00050.50.1092.0=µ95.0=ρ75.0−=∆eλ0.0110.410.52.305971002.206110.410.5,MatlabL10.2MozerSmolensky[MoSm1989]Karnin[Karn1990]E∑−−∆∂∂=10)()(Niijfijfijijijij)fijwEiijwNepochsLeCun[LiDe1990]2/2kkkkwhs=22kkkwEh∂∂=Hassibistork[HaSt1993][-1996][-1996-1991]ki∑∑+=pMjkikjkikiowg21)(kio1+kjkiw1+kjMp1+kMozer-SmolenskySkeletonization[MoSm1989]SkeletonizationSkeletonization10.2.1SkeletonizationiWithUnititWithoutUniiEE−−−=ρ10.12E()npOnpSkeletonizationiρ1iρ10.6iρMozerSmolenskyattentionalstrength,iiαiαi10.6)(∑=iiijijowfoα10.13joiojijiwijfsigmodaliα0=iαi1=iαii10==−=iiEEiααρ10.142iρiαiρ1111lim===→∂∂=−−iiiiEEEααγαγαγ10.150=γ1101===∂∂≈−−iiiiEEEαααα10.16iρ1=∂∂−=iiiEααρ10.17iρBPiρiρ1=iαiαiαiρ∑−=pjpjlotE∑−=2)(pjpjqotElEiρqEBPBPSigmoidalN∑=kkhhkhow)(σρ10.18()∑∑−=khkjjhkhkhhkjxwoow)()()(1σρ10.19hj)(kjxkjjhwjh)(khohkhwhkσk3iρ10.16iρ(1)MozerSmolenskyiiitEttαρρ∂∂+=+)(2.)(ˆ8.)1(ˆ10.20(2)()()∑=iittSρ()tSttii)()(ˆρρ=10.2111(3)()∑=−=LkiiktL1ˆ1ρρ()[]∑=−−−=LkiiktLV12ˆ11ρρiρiV0βρ+V10.22L0β4SkeletonizationBPBPSigmoidal1L0β2BP310.1810.19iρ410.2010.2110.22LiρiV5iiiV+ρ0βρ+iiVi16210.2.210.1.310.11()1−ku()1−ky()2−ku()2−ky()3−ky5-20-15[]1.0,1.0−100000.00054.550=L005.00=β10.710.8epoch=2763329433193349340634663556365239054234600291.55302.134410.7Skeletonization10.85()1−ku()1−ky010.7Skeletonization10.8Skeletonization710()1−ku()1−ky10.8()3−ky()2−ku()2−ky()1−ku10.9SkeletonizationSkeletonization10.16MatlabM10.3[SiDo1988-1993-1993][-1995][-1992]Castellano[CaFa1997].10.3.1BP()()edcbaf∨∨∧∨=10.23∧∨10.10BP515232031248BP5-5-5-110.10SigmoidalBP[]5.0,5.0−2.0=η8.0=α15001500ij}{ipV}{jpViVjV}{ipV}{jpV()()212212−−−=∑∑∑pjjppiipjipjpipijVVVVVVnVVr10.24i()∑−=piipiVVs2210.252410.1110.1410.111310.123410.132510.14510.3.21()1221ε≤−∑∑pkkpkpTOkpTkpOp1ε2}{ipV}{jpVijr2isij1θijr22θis22θjs10.261θ2θi2is22θis10.273}{ipV}{jpVbaVVij+=10.28∑∑==−−=npiipnpjijpipVVnVVVVna12211110.29ijVaVb−=10.30k∑∑∑≠≠≠++++=++++=+++jillklkjkbikjkijillklkbikjikijillklkbjkjikiVwbwwVawwVwwbaVwVwVwwVwVw,,,1*)()(1*)(1*10.31kbwkjkkjkikia→10.32kjkbkbb→10.33iiVV=k∑∑≠≠++=++jillklkiikbjillklkbikiVwwVwVwwVw,,1*)(1*10.34ikkiikbkbwVww+→10.354(1)1ε2ε1θ2θk=1(2)BPdelta(3)1εEk=k+1212εεE42εE(4)(5)i,j22221,θθθjiijssri,j(6)22θisk=k+12510.3.310.239.0=η9.0=αMaxEpoches=20003110−=ε5210*5−=ε8.01=θ01.02=θ[-0.5,0.5]10.15epochs=659660698epochs=9209211028epochs=1028200010.1652118BP813edc∧∧5ba∧1)()(edcba∧∧∨∧10.1510.16MatlabN