language testing (pma)3-4(2)

Session3ReliabilityandValidityWhenmakingatest,therearetwobasicfactorstoconsider--validityandreliability.Inthischapter,wewilldiscusswhattheseareandhowtheymustbeconsidered.1.DefinitionReliabilityisconcernedwithansweringthequestion“Howmuchofanindividual’stestperformanceisduetomeasurementerror,ortofactorsotherthanthelanguageabilitywewanttomeasure?”andwithminimizingtheeffectsofthesefactorsontestscores.Validityisconcernedwiththequestion“Howmuchofanindividual’stestperformanceisduetothelanguageabilitieswewanttomeasure?Andwithmaximizingtheeffectsoftheseabilitiesontestscores.(Bachman,1990:160-161)validityValiditycanbedefinedasthedegreetowhichatestactuallytestswhatitisintendedtotest.IfthepurposeofatestistotestabilitytocommunicateinEnglish,thenitisvalidifitdoesactuallytestabilitytocommunicate.Ifwhatitistestingisactuallyknowledgeofgrammar,thenitisnotavalidtestfortestingabilitytocommunicate.Thisdefinitionhastwoveryimportantaspects.Thefirstisthatvalidityisamatterofdegree.Testsarenoteithervalidornotvalid.Therearedegreesofvalidity,andsometestsaremorevalidthanothers.Asecondimportantaspectofthisdefinitionisthattestsareonlyvalidorinvalidintermsoftheirintendeduse.Ifatestisintendedtotestreadingability,butitalsotestswriting,thenitmaynotbevalidfortestingreading--butitmaytestreadingandwritingtogether.RelationshipbetweenreliabilityandvalidityInorderforatesttobevalid,itfirstneedstobereliable.Investigationofreliabilityandvaliditycanbeviewedascomplementaryaspectsofidentifying,estimating,andinterpretingdifferentsourcesofvarianceintestscores.FactorsaffectingperformanceofexamineesCommunicativelanguageabilityTESTSCORETestmethodfacetsPersonalattributesRandomfactorsMeaningfulvariancce(directlyrelatedtothepurposeofthetest)MeasurementerrorMeasurementerrorMeasurementerror(errorvariance)(Bachman,1990:165)(Brown,1996:189)Varianceameasureofvariability,SD,1)(22NxXVariance1)(2NxxsStandarddeviation:Z-score:sxxzRelationshipbetweenreliabilityandvalidityValidityandreliabilityhaveacomplicatedrelationship.Ifatestisvalid,itmustalsobereliable.Atestthatgivesdifferentresultsatdifferenttimescannotbevalid.However,itispossibleforatesttobereliablewithoutbeingvalid.Thatis,atestcangivethesameresulttimeaftertimebutnotbemeasuringwhatitwasintendedtomeasure.Reliabilityisconcernedwithdetermininghowmuchofthevarianceintestscoresisreliablevariance,whilevalidityisconcernedwithdeterminingwhatabilitiescontributetothisreliablevariance.Reliabilitysourcesoferrorintestscores,thatis,testscoresthemselvesValiditytestperformanceandfactorsoutsidethetestitselfRelationshipbetweenreliabilityandvalidityCampbell(1959:83)Agreementbetweensimilarmeasuresofthesametrait(e.g.correlationbetweenscoresonparalleltests)Agreementbetweendifferentmeasuresofthesametrait(e.g.correlationbetweenscoresonamultiplechoicetestofgrammarandratingsofgrammaronanoralinterview)ReliabilityValidityValidityConditionsReliabilityDBalancebetweenreliabilityandvalidityprescientificperiodValidityConditionsReliabilityDBalancebetweenreliabilityandvaliditypsychometricperiodValidityConditionsReliabilityDBalancebetweenreliabilityandvalidityintegrative-sociolinguisticperiodValidityConditionsReliabilityDBalancebetweenreliabilityandvalidityValidityConditionsReliabilityDBalancebetweenreliabilityandvalidityValidityConditionsReliabilityRelationshipbetweenreliabilityandvalidityAlthoughitisessentialtoconsiderbothreliabilityandvalidityinthedevelopmentanduseoflanguagetests,thedistinctionbetweenthemmaynotalwaysbeclear.2.ReliabilityClassicaltruescoretheoryetxxxObservedscore(x)Truescore(xt)Errorscore(xe)'xxr=.91:thescoresare91%consistent,orreliable,with9%measurementerror.ThreeapproachestoestimatingreliabilitywithintheclassicaltruescoremodelTest-retestreliabilityhowconsistenttestscoresareovertimeNottooshortbetweenthetwotestsNottoolongbetweenthetwotestsEquivalencetheextenttowhichscoresonalternateformsofatestareequivalentTest-RetestReliabilityDeterminingtest-retestreliabilityisnotasimplematter.Therearevariouswaysoftryingtomeasureit,buteachofthemhaspotentialproblems.Test-retest.Onewayofmeasuringreliabilityistogivethestudentsthesametesttwicetothesamegroupofstudents.However,ifatestisgiventwice,particularlyifthereisnotmuchtimebetweenthetwotests,thestudentsmightdobetterthesecondtimeduetoapracticeeffect.Ontheotherhand,ifthereisalongertimebetweenthetwotests,thepracticeeffectisnotaslikelytobeimportant,butitmaybethatwiththepassageoftime,students'Englishproficiencyhasimproved.Parallelgroups.Anotherwaytobedeterminereliabilityistohavetwoparallelgroupstakethesametest.Theproblemisdeterminingwhetherthetwogroupsaretrulyparallel.Paralleltests.Reliabilitycanalsobemeasuredbygivingparalleltests,thatis,twosimilartestswiththesametypeandnumberofitems,thesameinstructions,etc.Theproblemwiththisapproachisdeterminingwhetherthetwotestsareactuallyparallel.ParalleltestsThetruescoreononetestisequaltothetruescoreontheother.Theerrorvariancesforthetwotestsareequal.Inpractice,wevirtuallyneverhavestrictlyparalleltests,wetreattwotestspara