Using Recurrent Neural Networks for Time Series Fo

UsingRecurrentNeuralNetworksforTimeSeriesForecastingSandyD.BalkinPennsylvaniaStateUniversityDepartmentofManagementScienceandInformationSystemsUniversityPark,Pennsylvania16802e-mail:[e-mail],articialneuralnetworks(ANNs)havebeeninvestigatedasatoolfortimeseriesanalysisandforecasting.Themostpopulararchitectureisthemultilayerperceptron,afeedforwardnetworkoftentrainedbyback-propagation.TheforecastingperformanceofANNsrelativetotraditionalmethodsisstillopentoquestionalthoughmanyexperimentersseemoptimistic.Oneproblemwiththemultilayerperceptronisthat,initssimplestform,itissimilartoapureautoregressivetypemodel,soitlackstheabilitytoaccountforanymovingaveragestructurethatmayexist.Bymakinganetworkrecurrent,itispossibletoincludesuchstructure.WepresentseveralexamplesshowinghowanANNcanbeusedtorepresentanARMAschemeandcomparetheforecastingabilitiesoffeedforwardandrecurrentneu-ralnetworkswithtraditionalmethods.WorkingPaperSeriesnumber97-11.Thispaperwaspresentedatthe1997InternationalSymposiumonForecasting,Barbados.1UsingRecurrentNeuralNetworksforTimeSeriesForecasting21.IntroductionAnArticialNeuralNetwork(ANN)isaninformationprocessingparadigmthatisinspiredbythewaythebrainprocessesinformation.Thekeyelementofthisparadigmisthenovelstructureoftheinformationprocessingsystem.Itiscomposedofalargenumberofhighlyinterconnectedprocessingelements(neurons)workinginunisontosolvespecicproblems.ANNs,likepeople,learnbyexample.AnANNisconguredforaspecicapplication,suchaspatternrecognitionordataclassication,throughalearningprocess.Learninginbiologicalsystemsinvolvesadjustmentstothesynapticconnectionsthatexistbetweentheneurons.ThisistrueofANNsaswell.ForanintroductiontoANNssee,forexample,Hinton(1988),Lippmann(1987),orChengandTitterington(1994).Recently,ANNshavebeeninvestigatedasatoolfortimeseriesforecasting.ExamplesincludeTang,etal.(1991),Hill,etal.(1996),andFarawayandChateld(1995).Themostpopularclassisthemultilayerperceptron,afeedforwardnetworktrainedbybackpropaga-tion.Thisclassofnetworkconsistsofaninputlayer,anumberofhiddenlayersandanoutputlayer.Figure1isanexampleofanANNofthistypewith3neuronsintheinputlayer,asinglehiddenlayerwith2neurons,and1neuronintheoutputlayer.TherststepinusinganANNconcernsthechoiceofstructure,knownasthearchitectureofthenetwork.NeuralNetworksareknowntobeuniversalfunctionapproximators(Hornik,Stinchcombe,&White,1989),however,foranANNtoperformwell,theinputsandnumberofnodesinthehiddenlayermustbecarefullychosen.Sincetheylearnbyexample,itisvitalthattheinputscharacterizetheimportantrelationshipsandcorrelationsinthetimeseriesbeingmodeled.Forafeedforwardneuralnetworkwithonehiddenlayer,thepredictionequation,givenbyFarawayandChateld(1995),forcomputingforecasts^xtusingselectedpastobservations,xtj1;:::;xtjk,atlags(j1;:::;jk)andhnodesinthehiddenlayerwillbereferredtoasNN[j1;:::;jk;h]:Thus,thenetworkinFigure1isNN[1;12;13;2]:ThefunctionalformUsingRecurrentNeuralNetworksforTimeSeriesForecasting3OutputLayerHiddenLayerInputLayerxt-1xt-12xt-131Figure1:Afeedforwardneuralnetworkmaybewrittenas^xt=o(wco+Xhwhoh(wch+Xiwihxtji))wherefwchgdenotetheweightsfortheconnectionsbetweentheconstantinputandthehiddenneuronsandwcodenotestheweightofthedirectconnectionbetweentheconstantinputandtheoutput.Theweightsfwihgandfwhogdenotetheweightsfortheothercon-nectionsbetweentheinputsandthehiddenneuronsandbetweentheneuronsandtheout-putrespectively.Thetwofunctionshandodenotetheactivationfunctionsthatdenethemappingsfrominputstohiddennodesandfromhiddennodestooutput(s)respec-tively.Thelogisticfunction,y=(x)=11+exiswidelyusedintheANNliterature.FarawayandChateld(1995)recommend,forforecastingapplications,thathbelogisticand0linear.Forecastsaregeneratediterativelybyperformingsuccessiveone-stepaheadforecastsusingpreviousforecastsasestimatesofobservables.Thisapproachrepresentsaneuralnetworkimplementationofanon-linearautoregressive(NLAR)modeloforderkasbxt=ef(xtj1;xtj2;:::;xtjk;t)whereftgisasequenceofIIDrandomvariablesandUsingRecurrentNeuralNetworksforTimeSeriesForecasting4ef:k+1!:ItshouldbenotedthattheliteratureonANNsrarelyspeaksintermsofrandomvariables,orofanyassumptionsaboutthem.However,almostallapplicationsusetheleastsquarescriterionforttingsothat,byimplication,theerrorsareassumedtobeatleastuncorrelatedwithequalvariances.Further,theuseofANNsforforecastingrequiresconstancyofmodelstructureforfutureobservations,implyingtemporalinvarianceofef:ThisNLARnetwork,initssimplestform,willhavezeronodesinthehiddenlayer.Theresultingpredictionequationforcomputingaforecastforxtbecomes^xt=o(wco+Xiwioxtji)whereoisalinearfunction,wcodenotestheweightofthedirectconnectionbetweentheconstantinputandtheoutputandfwiogaretheweightsoftheconnectionsbetweentheinputneuronsandoutputneuron.Thisequationcorrespondstothetraditionalautoregressive(AR)forecastingfunction^xt=c+1xtj1+2xtj2++pxtjk:werekisthenumberofinputnodesinthecorrespondingneuralnetworkatlags(j1;:::;jk).So,initssimplestform,theneuralnetworkarchitecturereducestothestandardARtypemodel