神经网络之——递归神经网络

Lecture10Total69pages1RecurrentNeuralNetworksLecture10Total69pages2ClassificationofNNsFeedforwardNNsRecurrentNNsNeuralNetworksLecture10Total69pages33视网膜信息处理的基本系统视网膜分3层神经细胞（自下而上）：外层、中间层、最后层光信息自光感受器经双极细胞传至神经节细胞，神经节细胞的轴突汇聚成视神经离开眼球。水平细胞和无长突细胞通过侧向联系调节双极细胞和神经节细胞的反应。Lecture10Total69pages4FeedforwardNNs神经节细胞层内核层外核层•三层神经网络：神经节细胞层-内核层-外核层•每层内各神经元之间无连接•前一层神经元计算完后传递给下一层神经元进行计算Lecture10Total69pages5FeedforwardNNs1x21w13w12w11wRx2x22w23w1Rw2Rw3Rw12f11sf11sn12n11f11n22sf22sn22f22n21f21n31f31n32f32n33sf33sn11sa12a11a22sa22a21a22sa22a21aWxfaLecture10Total69pages6ContainfeedbackamongneuronsRecurrentNNsLecture10Total69pages7RecurrentNNsLecture10Total69pages8RecurrentNNsHowtoderivemathmodelsofRNNs?Lecture10Total69pages9RecurrentNNs1f1n2n2fLecture10Total69pages10RecurrentNNs1f1n2n2f)(1kx)(2kx)1(2kx)1(1kx)(2kx)(1kxLecture10Total69pages11RecurrentNNs1f1n2n2f)(1kx)(2kx)1(2kx)1(1kx)(2kx)(1kx22w21w12w11wLecture10Total69pages12RecurrentNNs1f1111122()()nwxkwxk)()(2221212kxwkxwn2f)(1kx)(2kx)1(2kx)1(1kx)(2kx)(1kx22w21w12w11wLecture10Total69pages13RecurrentNNs1f)()(121111kxwkxwn)()(2221212kxwkxwn2f)(1kx)(2kx))()(()1(21211111kxwkxwfkx)(2kx)(1kx22w21w12w11w)()()1(22212122kxwkxwfkxLecture10Total69pages14RecurrentNNs)()()1()()()1(2221212221211111kxwkxwfkxkxwkxwfkx1f1n2n2f)(1kx)(2kx)1(2kx)1(1kx)(2kx)(1kx22w21w12w11wLecture10Total69pages15RecurrentNNs1f1n2n2f)(1kx)(2kx)1(2kx)1(1kx)(2kx)(1kx22w21w12w11w)()()1()()()1(2221212221211111kxwkxwfkxkxwkxwfkxnjjijiikxwfkx1)()1(Lecture10Total69pages16RecurrentNNs)()1(kwxfkxLecture10Total69pages17RecurrentNNsbbbbLecture10Total69pages18DiscreteTimeRNNsbkwxfkx)()1(Lecture10Total69pages19DiscreteTimeRNNsbkwxfkx)()1(Networkcomputing?0x1x6x3x5x2x7x4x8xxLecture10Total69pages20DiscreteTimeRNNsbkwxfkx)()1(Networkcomputing0x1x6x3x5x2x7x4x8xxRNNInputOutputx0xLecture10Total69pages21Computing:DiscreteorContinuous?Lecture10Total69pages22DiscretevsContinuousDiscretetimecomputingContinuoustimecomputingLecture10Total69pages23DiscretevsContinuousContinuoustimecomputingHowtoderivecontinuoustimecomputingmathmodelsofRNNs?Lecture10Total69pages24FromDiscreteComputingtoContinuousComputingChangingtimestepsLecture10Total69pages25btwxftxtxtx)()()()1(btwxftx)()1(FromDiscreteComputingtoContinuousComputingLecture10Total69pages26btwxftxtxtx)()(1)()1(btwxftxtxtx)()()()1(FromDiscreteComputingtoContinuousComputingLecture10Total69pages27btwxftxtxtx)()()()(btwxftxtxtx)()(1)()1(FromDiscreteComputingtoContinuousComputingLecture10Total69pages28btwxftxtxtx)()()()(btwxftxtxtx)()()()(FromDiscreteComputingtoContinuousComputingLecture10Total69pages29btwxftxtxtx)()()()(btwxftxdttdx)()()(0FromDiscreteComputingtoContinuousComputingLecture10Total69pages30btwxftxdttdx)()()(ContinuousComputingRNNsLecture10Total69pages31RecurrentNNsttptagdttda),(),()(RNNmodel:NetworkstateNetworkinputNetworktimeLecture10Total69pages32RecurrentNNsWhat’stheoutputofaRNN?NetworkstateNetworkinputNetworktimettptagdttda),(),()(Networkoutputtata)(Lecture10Total69pages33ConvergenceofRNNsNetworkstatettptagdttda),(),()(tata)(Converge?0allfor0),(,tttpagEquilibriumpoint:Lecture10Total69pages34Trajectoriesttptagdttda),(),()())0(,(ytrajectoraisthere),0(conditioninitialanygivenataanR0,00,spaceesTrajectoriatataLecture10Total69pages35Trajectoriesttptagdttda),(),()(0anyfor,,then,If2121tataataaaLecture10Total69pages36Trajectoriesttptagdttda),(),()(0anyfor,,then,If2121tataataaaLecture10Total69pages37ASimpleExampleptadttda)()(ttepeata1)0()(Lecture10Total69pages38EquilibriumPointsttptagdttda),(),()(0allfor0),(,tttpagEquilibriumpoint:Lecture10Total69pages39EquilibriumPointsptadttda)()(ttepeata1)0()(tappaLecture10Total69pages40ConvergenceofRNNsttptagdttda),(),()(tata)(AttractorsLecture10Total69pages41ConvergenceofRNNsttptagdttda),(),()(tata)(DoeseachtrajectoryofaRNNconvergetoanequilibrium?Methods:1.Solvingdifferentialequationdirectly;2.Energymethod.Lecture10Total69pages42MethodOneSolvingDifferentialEquationsLecture10Total69pages43ASimpleExampleptadttda)()(ttepeata1)0()(tapLecture10Total69pages44LinearRNNsptWatadttda)()()(tata)(Lecture10Total69pages45LinearRNNs2122212111)()()()()()(ptatataptatatatata)(Lecture10Total69pages46://hebb.mit.edu/people/seung/index.htmlLecture10Total69pages48LinearRNNsH.S.Seung,Howthebrainkeepstheeyesstill,Proc.Natl.Acad.Sci.USA,vol.93,pp.13339-13344,1996ysensitivitpositionthe-----0gazecentralatratefiringthe-----ratefiringthe-----00iiiiiikEvvEkvvLecture10Total69pages49HowthebrainkeepstheeyesstillH.S.Seung,Howthebrainkeepstheeyesstill,Proc.Natl.Acad.Sci.USA,vol.93,pp.13339-13344,1996ABSTRACTThebraincanholdtheeyesst