统计学习[The-Elements-of-Statistical-Learning]第三章习题

TheElementofStatisticalLearning{Chapter3oxstar@SJTUJanuary4,2011Ex.3.5Considertheridgeregressionproblem(3.41).Showthatthisproblemisequivalenttotheproblem^c=argminc(NXi=1yic0pXj=1(xijxj)cj2+pXj=1cj2)Givethecorrespondencebetweencandtheoriginalin(3.41).Characterizethesolutiontothismodiedcriterion.Showthatasimilarresultholdsforthelasso.ProofByreplacingxijwithxijxjin(3.41),weget^ridge=argmin(NXi=1yi0pXj=1xjjpXj=1(xijxj)j2+pXj=12j)Comparethesetwoequations,wecandenec0=0+pXj=1xjjcj=jWenoticethatonly0(intercept)ismodied,andalldatapointsarecentered.Ex.3.6Showthattheridgeregressionestimateisthemean(andmode)oftheposteriordistribu-tion,underaGaussianpriorN(0;I),andGaussiansamplingmodelyN(X;2I).Findtherelationshipbetweentheregularizationparameterintheridgeformula,andthevariancesand2.ProofFromBayes'theoremwehaveP(jy)=P(yj)P()P(y)=N(X;2I)N(0;I)P(y)Thenthe(negative)log-posteriordensityofln(P(jy))=ln(P(yj))ln(P())+ln(P(y))=12(yX)T(yX)2+T+ConstantSincethedistributionisGaussian,themodeisalsotheposteriormean.Let=2=,wecangetthemodeandthemeanofthisdistribution^=argmax(ln(P(jy)))=argmin(22ln(P(jy)))=argmin((yX)T(yX)+T)1Compareitwithequation(3.43)intextbook,wehavetheridgeregressionestimateisthemean(andmode)oftheposteriordistribution.Ex.3.7AssumeyiN(0+xTi;2);i=1;2;:::;N,andtheparametersjareeachdistributedasN(0;2),independentlyofoneanother.Assuming2and2areknown,showthatthe(minus)log-posteriordensityofisproportionaltoPNi=1(yi0Pjxijj)2+Ppj=12jwhere=2=2.ProofEveryyiN(0+xTi;2)andeveryjN(0;2),thereforeP(yj)=1(p22)Ne122PNi=1(yi0Pjxijj)2P()=1(p22)pe122Ppj=12jFromBayes'theorem,wehaveP(jy)=P(yj)P()P(y)=1P(y)(p22)N(p22)pe122PNi=1(yi0Pjxijj)2122Ppj=12jThe(minus)log-posteriordensityofln(P(jy))=122NXi=1(yi0Xjxijj)2+122pXj=12j+Constant=1220@NXi=1(yi0Xjxijj)2+pXj=12j1A+Constant//Let=2=2/NXi=1(yi0Xjxijj)2+pXj=12j2