On the Nonparametric Prediction of Conditionally S

OntheNonparametricPredictionofConditionallyStationarySequencesS.CairesKNMI,RoyalNetherlandsMeteorologicalInstituteP.O.Box201,NL-3730AEDeBilt,[e-mail]94079,1090GBAmsterdam,[e-mail]ngconsistencyofestimatorsoftheconditionaldistributionfunctionandconditionalexpectationofafutureobservationofadiscretetimestochasticprocessgivenaxednumberofpastobservations.Theresultsapplytoconditionallystationaryprocesses(aclassofprocessesincludingMarkovandstationaryprocesses)satisfyingastrongmixingcondition,andtheyextendandbringtogethertheworkofseveralauthorsintheareaofnonparametricestimation.Oneofourgoalsistoprovidefurtherjusticationforthegrowingpracticalapplicationofestimatorsinnon-stationarytimeseriesandinother`noni.i.d.'settings.Someargumentsastowhysuchestimatorsshouldworkverygenerallyinpractice,ofteninanearly`optimal'way,aregiven.Twonumericalillustrationsareincluded,onewithsimulateddataandtheotherwithoceanographicdata.2000MathematicsSubjectClassication:62G08,62G30,62G07,62G15,62-07.KeywordsandPhrases:Nonparametricprediction,conditionaldistributionfunction,conditionalexpectation,timeseries,dataanalysis.Note:TheresearchoftherstauthorwasfundedbytheEuropeanCommissionERA-40Project(no.EVK2-CT-1999-00027).TheworkofthesecondauthorwascarriedoutundertheprojectPNA3.3,`StochasticProcessesandApplications',andfundedbyTheFifthFrameworkProgrammeoftheEuropeanCommissionthroughtheDynstochresearchnetwork.1IntroductionLetX=fXi:i2Ngbeasequenceofreal-valuedrandomvariablesdenedonaprobabil-ityspace(;F;P).WeconsidertheproblemofpredictingthevalueofXn+1givenonlytheknowledgeofthepastobservationsXn;:::;X1whenlittleisknownaboutthedistributionofX.SolutionstothisproblemincludeanestimateoftheconditionaldistributionfunctionofXn+1given(Xn;:::;X1)=(xn;:::;x1)2Rn,PXn+1xXn=xn;:::;X1=x1
;x2R;(1.1)anestimateoftheconditionalmeanE[Xn+1jXn=xn;:::;X1=x1](whenthisexists),anestimateofthemedianof(1.1),andapredictionintervalx(n)=2;x(n)1=2
suchthatPXn+12x(n)=2;x(n)1=2
Xn=xn;:::;X1=x111foragiven2(0;1).Sincepredictionistobecarriedoutsolelyonthebasisofthepresentrealizationofthesequence,anyapproachtotheproblemmustconsistoftreating,inonewayoranother,therealizationoftheprocessassamplesoftheprocessitself.Oneverygeneralformofthisapproach,whichisbeingmoreandmorefrequentlyappliedinpracticalproblems,isbasedonasimpleandintuitiveideapervadingallthoseareasandsub-areasofsciencesometimesputundertheheadingof`statisticallearning',suchasneuralnetworksandnearest-neighbourmethods:Inordertopredicttheoutcomeofanevent(a`response'variable,say)underaparticularcontext(acollectionofmeasurements,initialconditions,`explanatoryvariables',`features',etc.),wemaylookbackintoourpasthistory(asampleor`trainingset')forsituationswherethesameorapproximatelythesamecontextwasobserved,and(ifatleastonesuchinstanceisfound)predicttheoutcomeoftheeventinquestiononthebasisofwhatthehomologousoutcomeswereinthepast,forexamplebyaveragingthemorbychoosingthemostfrequentamongthem.InthecontextofourtimeseriesXthemostnaturalwayofimplementingthisideaisperhapstoxapositiveintegermn,constructestimatorsoftheconditionaldistributionfunction,expectationorquantilesofXn+1giventhepreviousmobservationsXn;:::;Xnm+1,andusetheseastoolstomakeinferencesandpredictivestatementsaboutthefutureobservation.Themainobjectiveofthispaperistoshowthataparticularclassofsuchestimatorsisconsistentunderrathergeneralassumptions,aconclusionwhichwillhaveatleastthevirtueofencouragingandjustifyingevenmoretheirapplicationinpracticalproblems.Fortheideaoutlinedabovetoworkitseemsnecessary,atleastfromatechnicalpointofview,toassumethatXisconditionallystationaryinthesensethattheconditionaldistributionfunctionofXn+1given(Xn;:::;Xnm+1)=u2Rmdoesnotdependonn.Accordingly,weshallassumeineverythingthatfollowsthatthereexistsaso-calledprobabilitykernel(u;v)!F(vju)suchthatZ[Ui2B]F(vjUi)dP=Z[Un2B]P(Xn+1vjXn;:::;Xnm+1)dP=Z[Um2B]P(VmvjUm)dPforallv2RandB2Bm,whereUi=(Xi;:::;Xim+1);Vi=Xi+1;im:(1.2)ForsimplicityweshallavoidindicatingthedependenceonminournotationforF(
ju),butsincemwillalwayshavethesamemeaningthroughoutthepaperthisshouldnotbeasourceofconfusion.IfF(
ju)hasarstmomentforuinagivenset,weshallcallu!R(u)=RvdF(vju)E[VijUi=u]theregressionfunction(ofVionUi).WriteS(u;h)=fu02Rm:h=2uiu0ih=2;i=1;:::;mgforthem-dimensionalsquarecentredatuwithsidesoflengthh0paralleltothecoordinateaxes.Givenasequencefhngofstrictlypositivenumbersconvergingto0,wedenetheempiricalconditionaldistributionfunction(ofXn+1given(Xn;:::;Xnm+1)=u)basedon(1.2)byFn(vju)=Pn1i=m1[Viv;Ui2S(u;hn)]Pn1i=m1[Ui2S(u;hn)];v2R;(1.3)2andtheempiricalregressionfunctionbyRn(u)=ZvdFn(vju)=Pn1i=mVi1[Ui2S(u;hn)]Pn1i=m1[Ui2S(u;hn)]:(1.4)ThesefunctionsaredenedonlyonthesetPn1i=m1[Ui2S(u;hn)]0 ;theymaybearbi-trarilydenedelsewhere.Fromamoregeneralstandpointweshallregardthemasregressionestimatorsbasedonanarbitrarysequence(U1;V1);(U2;V2);:::whichinparticularneednothaveanythingtodowiththetimeseriesX1;X2;:::Then