统计学习[The Elements of Statistical Learning]第七章习题

影之恋歌
1 ℃
2020-04-30

整理文档很辛苦，赏杯茶钱您下走！

还剩 ... 页未读，继续阅读 >>

免费阅读已结束，点击下载阅读编辑剩下 ... 页

阅读已结束，您可以下载文档离线阅读编辑

资源描述

TheElementofStatisticalLearning{Chapter7oxstar@SJTUJanuary6,2011Ex.7.1Derivetheestimateofin-sampleerror(7.24).Ey(Errin)=Ey(err)+2dN2AnswerNXi=1Cov(^yi;yi)=trace(Cov(^Y;Y))=trace(Cov(HY;Y))//H=X(XTX)1XT=trace(HCov(Y;Y))=trace(HVar(Y))=2trace(H)=2trace(X(XTX)1XT)=2trace(XTX(XTX)1)=2trace(Id)=d2SowehaveEy(Errin)=Ey(err)+2NNXi=1Cov(^yi;yi)=Ey(err)+2dN2Ex.7.3Let^f=Sybealinearsmoothingofy.1.IfSiiistheithdiagonalelementofS,showthatforSarisingfromleastsquaresprojectionsandcubicsmoothingsplines,thecross-validatedresidualcanbewrittenasyi^fi(xi)=yi^f(xi)1Sii:(7.64)2.Usethisresulttoshowthatjyi^fi(xi)jjyi^f(xi)j.3.FindgeneralconditionsonanysmootherStomakeresult(7.64)hold.Proof1.Sarisesfromleastsquaresprojectionsandcubicsmoothingsplines,sowecanassumeS=X(XTX+)1XT=XA1XTHenceSii=xTi(XTX+)1xi=xTiA1xi^f(xi)=xTi(XTX+)1XTy=xTiA1XTy1whereA=XTX+.DenotebyXiandyithecorrespondingresultswithxiremoved,thenwehave^fi(xi)=xTi(XTiXi+)1XTiyi=xTi(XTXxixTi+)1(XTyxiyi)=xTi(AxixTi)1(XTyxiyi)(1)(7.64)equalsto^fi(xi)=yiyi^f(xi)1Sii=^f(xi)yiSii1Sii(2)Toprove(1)=(2),weshouldproposealemma(AxxT)1=A1+A1xxTA11xTA1x(3)Proof.(AxxT)(A1+A1xxTA11xTA1x)=I+xxTA11xTA1xxxTA1xxTA1xxTA11xTA1x=I+xxTA1xxTA1+xxTA1xTA1xxxTA1xxTA11xTA1x=I+xxTA1(xTA1x)x(xTA1x)xTA11xTA1x=IHerewecancontinuederiving(1)fi(xi)=xTiA1+A1xixTiA11xTiA1xi(XTyxiyi)//from(3)=xTiA1+SiixTiA11Sii(XTyxiyi)(4)=xTiA1XTyxTiA1xiyi+SiixTiA1XTy1SiiSiixTiA1xiyi1Sii=^f(xi)yiSii+Sii^f(xi)1SiiyiS2ii1Sii=(2)2.(XTiXi+)1ispositive-semidenite,sowehavexTi(XTiXi+)1xi=xTiA1+SiixTiA11Siixi//from(1)(4)=Sii+S2ii1Sii=Sii1Sii0)0Sii1yi^fi(xi)=yi^f(xi)1Siiyi^f(xi)3.Iftherecipeforproducing^ffromydoesnotdependonyitselfandSdependsonlyonthexiand,wecanjustreplaceyiwith^fi(xi)iny(y!y0)withoutSbeingchangedinthe2secondprocess.Notethat^fi(xi)hereiscalculatedbytherstprocess.Hencewestillhave^fi(xi)=Siy0=Xi6=jSijy0j+Sii^fi(xi)=^f(xi)Siiyi+Sii^fi(xi)(7.64)canalsobederivedfromthisequation.SothegeneralconditionsonanysmootherStomakeresult(7.64)holdaretherecipeforproducing^ffromydoesnotdependonyitselfandSdependsonlyonthexiand.Ex.7.4Considerthein-samplepredictionerror(7.18)andthetrainingerrorerrinthecaseofsquared-errorloss:Errin=1NNXi=1EY0(Y0i^f(xi))2err=1NNXi=1(yi^f(xi))2:Addandsubtractf(xi)andE^f(xi)ineachexpressionandexpand.Henceestablishthattheaverageoptimisminthetrainingerroris2NNXi=1Cov(^yi;yi);asgivenin(7.21).ProofBecauseY0andyhavethesamedistribution,wehaveEY0(Y0i)=Ey(yi)andobviouslyEY0[(Y0i)2]=Ey(y2i).HenceEy(op)=Ey(Errinerr)=1NNXi=1Ey[EY0(Y0i^f(xi))2(yi^f(xi))2]=1NNXi=1Ey[EY0(Y0i)22^yiEY0(Y0i)+^y2iy2i+2yi^yi^y2i]=1NNXi=1[EY0[(Y0i)2]2Ey(^yi)EY0(Y0i)Ey(y2i)+2Ey(yi^yi)]=1NNXi=1[Ey(y2i)2Ey(^yi)Ey(yi)Ey(y2i)+2Ey(yi^yi)]=2NNXi=1[Ey(yi^yi)Ey(^yi)Ey(yi)]=2NNXi=1Cov(^yi;yi)Ex.7.5Foralinearsmoother^y=Sy,showthatNXi=1Cov(^yi;yi)=trace(S)2;whichjustiesitsuseastheeectivenumberofparameters.3ProofNXi=1Cov(^yi;yi)=trace(Cov(^y;y))=trace(Cov(Sy;y))=trace(SCov(y;y))=trace(SVar(y))=trace(S)2Ex.7.6Showthatforanadditive-errormodel,theeectivedegrees-of-freedomforthek-nearest-neighborsregressiontisN=k.ProofBythedenitionofk-nearest-neighbortfor^Y,wehave^Y(x)=1kXxi2Nk(x)yi=1kINyHencetheeectivedegrees-of-freedomdf(S)=trace(S)=trace(1kIN)=NkEx.7.8ShowthatthesetoffunctionsfI(sin(x)0)gcanshatterthefollowingpointsontheline:z1=101;:::;z`=10`;forany`.HencetheVCdimensionoftheclassfI(sin(x)0)gisinnite.ProofIfweclassifyz`tosin(z`)0,thenwehave2k10`2(k+12)(5)2k10`2(k+12)10`(6)Weshoulddeterminekthatsatisestheinequality(6)aboveforany`.Letk=Xm2L;ml10ml2+122Xm2L10m22Xm2L10m2+12whereListhesetof`.4Nowwecanproveinequality(5)foranyl2L.10l2Xm2L10ml2=2Xm2L;m6=l10ml2+122Xm2L;ml10ml2+12=2k10l2Xm2L10ml2+10l2=2Xm2L;ml10ml2+12+Xm2L;ml10ml2+10l22Xm2L;ml10ml2+12+1Xi=110i2(7)2Xm2L;ml10ml2+12+12=2(k+12)(8)Ifl=2L,wehave10l2Xm2L10ml2=2Xm2L;ml10ml2+Xm2L;ml10ml2=2Xm2L;ml10ml2+1212+Xm2L;ml10ml22Xm2L;ml10ml2+1212=2(k12)10l2Xm2L10ml2+10l2=2Xm2L;ml10ml2+Xm2L;ml10ml2+10l22Xm2L;ml10ml2+12=2k//from(7)(8)Obviously,sin(zl)0,hencewecanshatterallpointsontheline.5