Mathematical-Statistics-and-Data-Analysis

MathematicalStatisticsandDataAnalysisJohnA.RiceUniversityofCalifornia,BerkeleyArrangementoftheCourseChapter1SummarizingDataChapter2ComparingTwoSamplesChapter3TheAnalysisofVarianceChapter4LinearLeastSquaresWhatweshouldlearn?Mathematics(Statistics)EnglishComputer(SAS)(StatisticalAnalysisSystem)Chapter1SummarizingDataMethodsBasedontheCumulativeDistributionFunctionHistograms,DensityCurvesandStem-and-LeafPlotsMeasuresofLocationMeasuresofDispersionTheEmpiricalCumulativeDistributionFunction(ecdf)Supposethatisabatchofnumbers.TheecdfisdefinedasDenotetheorderedbatchofnumbersby,thentheecdfcanbeexpressedasnxxx,,,21)(#1)(xxnxFin)()2()1(nxxx)()1()()1(10)(nkknxxxxxnkxxxF)()1()()1()1()1(0)(nkknxxnnxxxnkxxxFPropertiesoftheEmpiricalCumulativeDistributionFunctionTheorem1Theorem2Thatis,tendstosimultaneouslywithprobabilityone.)())((xFxFEn))(1)((1))((xFxFnxFVarn1)0)()(maxlim(xFxFpnxn)(xFn)(xFExamplesPlottheecdfofthisbatchofnumbers:1,14,10,9,11,9SASdataset:beeswax.sasSolutionsAnalysisInteractiveDataAnalysis(Findbeeswax.sasfromWork)AnalyzeDistributionOutputCumulativeDistributionfunctionEmpiricalTheSurvivalFunctionIfdenotestimeuntilfailureordeathwithcdf,thesurvivalfunctionisdefinedaswhichissimplytheprobabilitythatthelifetimewillbelongerthan.Theempiricalsurvivalfunctionisgivenbywhereistheecdfofrandomvariable.TF)(1)()(tFtTptSt)(1)(tFtSnn)(tFnTTheHazardFunctionThehazardfunctionisdefinedaswhichistheinstantaneousrateofmortalityofanindividualaliveat.Thelogoftheempiricalsurvivalfunctionisdefinedas)(log)(1)()(1)()(tsdtdtFtFtFtftht)()1()()1()11log()11log(0)(lognkknttnntttnktttSExampleCalculatethehazardfunctionfortheexponentialdistribution:Letdenotethedensityfunctionandthehazardfunctionofanonnegativerandomvariable.Showthat0001)(ttetFthtdsshethtf0)()()(fQuantile-QuantilePlotsThethquantileofthedistributionisthevalueofsuchthatorpxFp)(ppx)(1pFxp1ppxx)(xFPthquantileTheempiricalquantileofdataForthegivensample,theecdfforor;Let,thusthethquantileofdataisassignedto;nxxx,,,21)1()(nkxFn)1()(kkxxx)1()()(nkxFkn)1()()(nkxFkn)1(nk)(kxAssessingGoodnessofFitbyUsingQ-QPlotIsthesamplefromthedistribution?Theempiricalthquantileis;Thetheoreticalthquantileofis,whichsatisfies;Thedotsontheplanewouldbeapproximatelyastraightlineifthesamplecomesfrom.nxxx,,,21F)1(nk)(kx)1(nkF)1(nkx)1()()1(nkxFnk),()1()(nkkxxFComparingTwoSamplesbyusingQ-QplotAresampleandfromthesamedistribution?Theempiricalthquantileofis;Theempiricalthquantileofis;Thedotsontheplanewouldbeapproximatelyastraightlineifthesamplecomesfromthesamedistribution.nxxx,,,21nyyy,,,21)1(nksx')(kx)1(nksy')(ky),()()(kkyxExampleSASdataset:beeswax.sasSolutionsAnalysisInteractiveDataAnalysis(Findbeeswax.sasfromWork)AnalyzeDistributionOutputNormalQ-QplotHistogramsExample:beeswax.sas5.642.6452.649.6369.636.63156.633.63243.63638637.621xxxxxxfrequency5.642.642825.02.649.63339.09.636.638475.06.633.633559.13.6363452.0637.620565.0)3.0)595(()3.0)596(()3.0)5915(()3.0)5924(()3.0)598()3.0)591((xxxxxxdensityDensityCurves—KernelProbabilityDensityEstimationLetbethestandardnormaldensity,thentherescaledversionof,isdefinedaswhichisthenormaldensitywithstandarddeviation;)(xw)(xw)(xwh2222)(2121211)(1)(hxhxhehehhxwhxwhLetbeasamplefromaprobability,thenisthenormaldensitywithmeanandstandarddeviation;Thekernelprobabilitydensityestimateofisthengivenbywhereisachosenbandwidth.nxxx,,,21f)(ihxxwixhfniihhxxwnxf1)(1)(hExampleBeeswaxSolutionsAnalysisInteractiveDataAnalysis(Findbeeswax.sasfromWork)AnalyzeDistributionOutputDensityEstimateNormal(kerneldensity)Stem-and-LeafPlotsExamplebeeswax.sas02:64422:64302147:642352:64116:6400622223:639511334668:63861788:6372190013689:6367260000113668:6351001335:634523001446669:63391877:63229033:63137358:63034:629015:62811leafstemMeasuresofLocationTheArithmeticMeanForabatchofnumbers,themostcommonlyusedmeasureoflocationisDisadvantage:Themeasureissensitivetooutliersinthedataset.niixnx11nxxx,,,21TheMedianFortheorderobservationsthemedianisdefinedasAdvantage:Medianisrobustandinsensitivetooutliers.)()2()1(nxxxevenoddisisnnifxxifxmediannnn2)()12()2()2)1((TheTrimmedMeantrimmedmeanistheaverageoftheremainingdataleftbydiscardingthelowestandthehighestoftheorderdata.ItcanbeexpressedasIngeneral,willbechosenfrom0.1to0.2.%100%100%100][2])[()2]([)1]([nnxxxxnnnnM-estimatesThesamplemeanminimizesThemedianistheminimumofM-estimatesisdefinedtominimizewhere21)(niixniix1)(1niix5.15.1)(2xifxxifxxComparisonofLocationEstimationThereisnosingleestimatethatisbestforallsymmetricdistributionalthoughtheyallestimatethecenterofsymmetry;Ingeneral,10%trimmedmeanor20%trimmedmeanisoverallquiteaneffectiveestimatesinceitsvarianceisquitesmall.MeasuresofDispersionThesamplestandarddeviation,,whichisthesquarerootofthesamplevariance,Disadvantage:Thesamplestandarddeviationissensitivetooutlyin