神经网络之——BP神经网络

Lecture7Total78pages11Backpropagation(BP)Lecture7Total78pages22OutLineWhyBPXORProblemcannotbesolvedbyperceptronandAdlinenetworksMultilayerPerceptronBPPrincipleApplicationsLecture7Total78pages33WhyBPLMS:基于最速下降法，训练单层线性网络BP:近似最速下降法，训练多层非线性网络，用微积分的链法则多层网络的应用模式分类函数逼近降维处理……Lecture7Total78pages44110,00qt331,10qt441,01qt220,11qt2q1q4q3qXORProblemThesepatentscannotbeclassifiedbysingle-layerpercetron.--------MinskyandPapert,1969Lecture7Total78pages552q1q4q3qXORProblem110,00qt331,10qt441,01qt220,11qtLecture7Total78pages66XORProblem2q1q4q3q012221pp05.121pp5.15.05.15.0110,00qt331,10qt441,01qt220,11qtLecture7Total78pages7722115.11XORProblem2q1q4q3q012221pp05.121pp5.15.05.15.0Lecture7Total78pages882q1q4q3q012221pp05.121pp115.1XORProblemLecture7Total78pages992q1q4q3q012221pp05.121pp115.1XORProblem22115.11+Lecture7Total78pages10102q1q4q3q012221pp05.121pp22115.1111-1.5p1p2XORProblemLecture7Total78pages11112q1q4q3q22115.1111-1.5p1p2XORProblem110,00qt331,10qt441,01qt220,11qtLecture7Total78pages1212R–S1–S2–S3NetworkMultilayerPerceptronLecture7Total78pages1313MultilayerPerceptron1222awfaTpwfaT1112p1pRp1f1f1f2f2f2fLecture7Total78pages1414MultilayerPerceptron1222awfaTpwfaT1112p1pRp1f1f1f2f2f2fpwfwfaTT11222Lecture7Total78pages1515MultilayerPerceptron1222awfaTpwfaT1112p1pRp1f1f1f2f2f2fpwfwfaTT11222p1t1{,}p2t2{,}pQtQ{,}TrainingSetWPfaLecture7Total78pages1616Howtodeveloplearningalgorithmformultilayerperceptron?Prof.PaulWerbos,1974BPLearningAlgorithmLecture7Total78pages1717MultilayerPerceptron1222awfaTpwfaT1112p1pRp1f1f1f2f2f2fpwfwfaTT11222p1t1{,}p2t2{,}pQtQ{,}TrainingSetWPfaiiWPftLecture7Total78pages1818011awn)(111nfa122awn)(222nfaaBPPrinciplepa01f1n2naa22w1a1w2fLecture7Total78pages19192atE),(),(21212nnEwwEatEEqqtptptp,,,,,,2211BPPrinciplepa01f1n2naa22w1a1w2fLecture7Total78pages2020111)()1(wEkwkw222)()1(wEkwkwBPPrinciple2atEpa01f1n2naa22w1a1w2fSteepestDescentW?Lecture7Total78pages21211111wnnEwE2222wnnEwE010111)(awawwn121222)(awawwn11snE22snESensitivityBPPrinciple),(),(2121nnEwwEE011awn122awnpa01f1n2naa22w1a1w2fLecture7Total78pages2222011111aswnnEwE122222aswnnEwEBPPrinciple111)()1(wEkwkw222)()1(wEkwkwpa01f1n2naa22w1a1w2fLecture7Total78pages23230111)()1(askwkw1222)()1(askwkwBPPrinciplepa01f1n2naa22w1a1w2fLecture7Total78pages242422121211nnEEssnnnn)()()(112111211211212nfwnnfwnawnawnn22111)(swnfsBPPrinciple11snE22snESensitivitypa01f1n2naa22w1a1w2fLecture7Total78pages252522222222()()2()2()fnEtaastatannnn)()(2222atnfspa01f1n2naa22w1a1w2fBPPrinciple222()()tafnLecture7Total78pages262611122()sfnws)()(2222atnfsBPPrinciplepa01f1n2naa22w1a1w2fLecture7Total78pages272722111)(swnfs))()(()(2222kaktnfs0111)()1(askwkw1222)()1(askwkw)(0kpa)(0111awfa)(1222awfaBPPrinciplepa01f1n2naa22w1a1w2fLecture7Total78pages282822111)(swnfs))()(()(2222kaktnfs0111)()1(askwkw1222)()1(askwkw)(0kpa)(0111awfa)(1222awfaBPPrincipleForwardPropagationBackpropagationWeightUpdateLecture7Total78pages2929am1+fm1+Wm1+ambm1++=m02M1–=a0p=aaM=MultilayerNetworkLecture7Total78pages3030am1+fm1+Wm1+ambm1++=m02M1–=a0p=aaM=MultilayerNetworkp1t1{,}p2t2{,}pQtQ{,}TrainingSetLecture7Total78pages3131p1t1{,}p2t2{,}pQtQ{,}TrainingSetMeanSquareErrorFxEeTe=Eta–Tta–=Fˆxtkak–Ttkak–eTkek==ApproximateMeanSquareError(SingleSample)wijmk1+wijmkFˆwijm------------–=bimk1+bimkFˆbim---------–=ApproximateSteepestDescentPerformanceIndexLecture7Total78pages32Fˆwijm------------Fˆnim---------nimwijm------------=Fˆbim---------Fˆnim---------nimbim---------=ChainRulewijmk1+wijmkFˆwijm------------–=bimk1+bimkFˆbim---------–=ApproximateSteepestDescentLecture7Total78pages33nimwijmajm1–j1=Sm1–bim+=nimwijm------------ajm1–=nimbim---------1=simFˆnim---------SensitivityFˆwijm------------simajm1–=Fˆbim---------sim=GradientGradientCalculationLecture7Total78pages34wijmk1+wijmksimajm1––=bimk1+bimksim–=Wmk1+Wmksmam1–T–=bmk1+bmksm–=smFˆnm----------Fˆn1m---------Fˆn2m---------FˆnSmm-----------=NextStep:ComputetheSensitivities(Backpropagation)SteepestDescentLecture7Total78pages35nm1+nm-----------------n1m1+n1m----------------n1m1+n2m----------------n1m1+nSmm----------------n2m1+n1m----------------n2m1+n2m----------------n2m1+nSmm----------------nSm1+m1+n1m----------------nSm1+m1+n2m----------------nSm1+m1+nSmm----------------nim1+njm----------------wilm1