Some Properties Of the Gaussian Distribution Conte

SomePropertiesOftheGaussianDistributionJianxinWuGVUCenterandCollegeofComputingGeorgiaInstituteofTechnologyApril22,2004Contents1Introduction22De…nition22.1UnivariateGaussian.........................22.2MultivariateGaussian........................33NotationandParameterization44LinearOperationandSummation54.1Univariatecase............................54.2MultivariateCase...........................55GeometryandMahalanobisDistance66Conditioning77ProductofGaussians98ApplicationI:ParameterEstimation108.1MaximumLikelihoodEstimation..................108.2BayesianParameterEstimation...................119ApplicationII:KalmanFilter129.1TheModel..............................129.2TheEstimation............................12AGaussianIntegral14BCharacteristicFunctions15CSchurComplementandtheMatrixInversionLemma161DVectorandMatrixDerivatives171IntroductionTheGaussiandistributionisthemostwidelyusedprobabilitydistributioninstatisticalpatternrecognitionandmachinelearning.ThenicepropertiesoftheGaussiandistributionmightbethemainreasonforitspopularity.Inthisshortpaper1,ItrytoorganizethebasicfactsabouttheGaussiandistribution.Thereisnoadvancedtheoryinthispaper.However,inordertounderstandthesefacts,somelinearalgebraandmultivariateanalysisareneeded,whicharenotalwayscoveredsu¢cientlyinundergraduatetexts.Theattemptofthispaperistopoolthesefactstogether,andhopethatitwillbeusefulfornewresearchersenteringthisarea.2De…nition2.1UnivariateGaussianTheprobabilitydensityfunctionofaunivariateGaussiandistributionhasthefollowingform:p(x)=1p2exp(x)222!,(1)inwhichistheexpectedvalueofx,and2isthevariance.Weassumethat0.Wehaveto…rstverifythateq.(1)isavaliddensity.Itisobviousthatp(x)0alwaysholdsforx2R.Fromeq.(96)inAppendixAweknowthatR11expx2tdx=pt.Applyingthisequation,wehaveZ11p(x)dx=1p2Z11exp(x)222!dx(2)=1p2Z11expx222dx(3)=1p2p22=1,(4)whichmeansthatp(x)isavaliddensity.Thedensity1p2expx22iscalledthestandardnormaldensity.InAppen-dixA,itisshowedthatthemeanvalueandstandarddeviationofthestandardnormaldistributionare0and1respectively.Bydoingachangeofvariables,itiseasytoshowthat=Rxp(x)dxand2=R(x)2p(x)dx.1IplannedtowriteashortnotelistingsomepropertiesofGaussian.However,somehowIdecidedtokeepthispaperself-containing.Theresultisthatitbecomesveryfat.22.2MultivariateGaussianTheprobabilitydensityfunctionofamultivariateGaussiandistributionhasthefollowingform:p(x)=1(2)d=2jj1=2exp12(x)T1(x),(5)inwhichxisad-dimensionalvector,isthed-dimensionalmeanvector,andisthed-by-dcovariancematrix.Weassumethatisasymmetric,positivede…nitematrix.Wehaveto…rstverifythateq.(5)isavalidprobabilitydensityfunction.Itisobviousthatp(x)0alwaysholdsforx2Rd.Nextwediagonalizeas=UTUinwhichUisanorthogonalmatrixcontainingtheeigenvectorsof,=[1;:::;d]isadiagonalmatrixcontainingtheeigenvaluesofinitsdiagonalentriesandjj=jj.Let’sde…neanewrandomvectorasy=1=2U(x).(6)Ifwetreateq.(6)asachangeofvariables,thedeterminantoftheJacobianmatrixwillbe1=2=jj1=2.NowwearereadytocalculatetheintegralZp(x)dx=Z1(2)d=2jj1=2exp12(x)T1(x)dx(7)=Z1(2)d=2jj1=2jj1=2exp12yTydy(8)=dYi=1Z1p2expy2i2dyi(9)=dYi=11=1(10)inwhichyiistheithcomponentofy,i.e.y=(y1;:::;yd).ThisequationgivesthevalidityofthemultivariateGaussiandensity.Sinceyisarandomvector,ithasadensity,denotedaspy(y).Usingtheinversetransformmethod,wegetpy(y)=p+UT1=2yUT1=2(11)=UT1=2(2)d=2jj1=2exp12UT1=2yT1UT1=2y(12)=1(2)d=2exp12yTy(13)Thedensityde…nedbypy(y)=1(2)d=2exp12yTy.(14)3iscalledasphericalGaussiandistribution.Letzbearandomvectorformedbyasubsetofthecomponentsofy.Bymarginalizationitisclearthatp(z)=1(2)jzj=2exp12zTz
,andspeci…callyp(yi)=1p2expy2i2.Usingthisfact,itisstraightforwardtoshowthatthemeanvectorandcovariancematrixofasphericalGaussianare0andIrespectively.Usingtheinversetransformofeq.(6),wecaneasilycalculatethemeanvectorandcovariancematrixofthedensityp(x).Ex=E+UT1=2y=+EUT1=2y=(15)E(x)(x)T=EUT1=2yUT1=2yT(16)=UT1=2EyyT
1=2U(17)=UT1=21=2U(18)=(19)3NotationandParameterizationWhenwehaveadensityoftheformineq.(5),itisoftenwrittenasxN(;)(20)or,N(x;;)(21)InmostcaseswewillusethemeanvectorandcovariancematrixtoexpressaGaussiandensity.Thisiscalledthemomentparameterization.ThereisanotherparameterizationofaGaussiandensity,calledthecanonicalparame-terization.Inthecanonicalparameterization,aGaussiandensityisexpressedasp(x)=exp+Tx12xTx,(22)inwhich=12dlog2logjj+T1
isanormalizationconstant.Theparametersinthesetworepresentationsarerelatedby=1(23)=1(24)=1(25)=1.(26)Noticethatthereisaconfusioninournotation:hasdi¤erentmeaningsineq.(22)andeq.(6).Ineq.(22),isaparameterinthecanonicalparameterizationofaGaussiandensity,whichisnotnecessarilydiagonal.Ineq.(6),isadiag-onalmatrixformedbytheeigenvaluesof.ItisstraightforwardtoshowthatthemomentparameterizationandcanonicalparameterizationoftheGaussiandistributionareequivalent.Insomecasesthecanonicalparameterizationismoreconvenienttousethanthemomentpar