统计学习[The-Elements-of-Statistical-Learning]第六章习题

TheElementofStatisticalLearning{Chapter6oxstar@SJTUJanuary6,2011Ex.6.2ShowthatPNi=1(xix0)li(x0)=0forlocallinearregression.Denebj(x0)=PNi=1(xix0)jli(x0).Showthatb0(x0)=1forlocalpolynomialregressionofanydegree(includinglocalconstants).Showthatbj(x0)=0forallj2f1;2;:::;kgforlocalpolynomialregressionofdegreek.Whataretheimplicationsofthisonthebias?ProofBythedenitionofvector-valuedfunction,b(x)T=(1;x)andB=[1;x],sowehaveb(x0)T=b(x0)T(BTW(x0)B)1BTW(x0)B(1;x0)=b(x0)T(BTW(x0)B)1BTW(x0)[1;x0]1x0=b(x0)T(BTW(x0)B)1BTW(x0)1=PNi=1li(x0)b(x0)T(BTW(x0)B)1BTW(x0)x0=PNi=1li(x0)xi(1)NXi=1(xix0)li(x0)=NXi=1li(x0)xix0NXi=1li(x0)=x0x0
1=0From(1),wehaveb0(x0)=NXi=1(xix0)0li(x0)=NXi=1li(x0)=1Whenj2f1;2;:::;kg,wehavevector-valuedfunctionb(x)T=(1;x;x2;:::;xk)andB=[1;x;x2;:::;xk].From(1),wesimilarlyhavexj0=NXi=1li(x0)xji(2)Expanding(xjx0)jwithoutcombingofsimilarterms,eachtermcanbewrittenas(1)bxaixb0,wherea+b=j.Obviously,thenumberofpositivetermsequalswiththenumberofnegativeterms,i.e.P2jn=1(1)bn=0.Soeachtermofbj(x0)canbewrittenasNXi=1(1)bxaixb0li(x0)=(1)bxb0NXi=1li(x0)xai=(1)bxb0xa0=(1)bxj0//(2)bj(x0)=NXi=1(xix0)jli(x0)=NXi=12jXn=1(1)bnxj0li(x0)=01HencewehavethebiasE^f(x0)f(x0)=NXi=1li(x0)f(xi)f(x0)=f(x0)NXi=1li(x0)f(x0)+f0(x0)NXi=1(xix0)li(x0)+k2f00(x0)NXi=1(xix0)2li(x0)+:::+kj+1f(j+1)(x0)NXi=1(xix0)j+1li(x0)+:::=kj+1f(j+1)(x0)NXi=1(xix0)j+1li(x0)+:::whereknisthecoecientsofseriesexpansionterms.Nowweseethatthebiasdependsonlyon(j+1)th-degreeandhigherordertermsintheexpan-sionoff.Ex.6.3Showthatjjl(x)jj(Section6.1.2)increaseswiththedegreeofthelocalpolynomial.ProofPreliminary:BisaN2regressionmatrix,soBisnotinvertiblewhileBBTisinvert-ible.From^f(xj)=b(xj)T(BTW(xj)B)1BTW(xj)y=NXi=1li(xj)yi=l(xj)Tywehavel(xj)T=b(xj)T(BTW(xj)B)1BTW(xj)l(xj)=W(xj)B(BTW(xj)B)1b(xj)jjl(xj)jj2=l(xj)Tl(xj)=[b(xj)T(BTW(xj)B)1BTW(xj)][W(xj)B(BTW(xj)B)1b(xj)]=b(xj)T(BTW(xj)B)1BTW(xj)[BBT(BBT)1(BBT)1BBT]W(xj)B(BTW(xj)B)1b(xj)=b(xj)TBT(BBT)1(BBT)1Bb(xj)jjl(x)jj2=d+1Xj=1jjl(xj)jj2=d+1Xj=1b(xj)TBT(BBT)1(BBT)1Bb(xj)=trace(BBT(BBT)1(BBT)1BBT)//Prof.Zhanghasprovedit=trace(Id+1)=d+1Hencejjl(x)jj=pd+1whichincreaseswiththedegreeofthelocalpolynomial.Ex.6.4SupposethattheppredictorsXarisefromsamplingrelativelysmoothanalogcurvesatpuniformlyspacedabscissavalues.DenotebyCov(XjY)=theconditionalcovariancematrixofthepredictors,andassumethisdoesnotchangemuchwithY.DiscussthenatureofMahalanobischoiceA=1forthemetricin(6.14).HowdoesthiscomparewithA=I?HowmightyouconstructakernelAthat(a)downweightshigh-frequencycomponentsinthedistancemetric;(b)ignoresthemcompletely?2AnswerD=p(xx0)T1(xx0)iscalledtheMahalanobisdistanceofthepointxtox0.Ittakesthecorrelationsofthedatasetintoaccount.Ifthepredictorsarehighlycorrelated,Maha-lanobisdistanceismuchaccuratethanEuclidiandistance.IfA=I,thend=p(xx0)T(xx0)equalstoEuclidiandistanceofthepointxtox0.Priortosmoothing,weshouldstandardizeeachpredictor,forexamplex0i=xiE(xi)pVar(xi)Whencomparingitwith1andusingthestandardpredictors,wehaveCov(x0i;x0j)=E[(x0iE(x0i))(x0jE(x0j))]=E(x0ix0j)=E(xiE(xi)pVar(xi))(xjE(xj)pVar(xj))=E[(xiE(xi))(xjE(xj))]pVar(xi)pVar(xj)=Cov(xi;xj)pVar(xi)pVar(xj)=(xi;xj)Thenthecovariancematrixwillchangetoitsstandardizedversion(correlationmatrix).HenceA=Imeans8i6=j;(xi;xj)=0,i.e.,alldimensionsofxarenotcorrelated.(a)IfwewanttoconstructakernelAthatdownweightshigh-frequencycomponents(xis)inthedistancemetric,wecanjustdecreaseCov(xi;xj)or(xi;xj)inordertosuppresstheirinuence.(b)IfwewanttoconstructakernelAthatignoresthemcompletely,wecanjustsetCov(xi;xj)or(xi;xj)as0.3