Score generalizability of academic writing tasks D

http://ltj.sagepub.comLanguageTestingDOI:10.1177/02655322093401882009;26;507LanguageTestingAttaGebrilitall?Scoregeneralizabilityofacademicwritingtasks:Doesonetestmethodfithttp://ltj.sagepub.com/cgi/content/abstract/26/4/507Theonlineversionofthisarticlecanbefoundat:Publishedby:http://www.sagepublications.comcanbefoundat:LanguageTestingAdditionalservicesandinformationforhttp://ltj.sagepub.com/cgi/alertsEmailAlerts:http://ltj.sagepub.com/subscriptionsSubscriptions:http://www.sagepub.com/journalsReprints.navReprints:http://www.sagepub.co.uk/journalsPermissions.navPermissions:http://ltj.sagepub.com/cgi/content/refs/26/4/507CitationsatNATIONALTAIWANUNIVLIBonJanuary27,2010http://ltj.sagepub.comDownloadedfromLanguageTesting200926(4)507–531Addressforcorrespondence:AttaGebril,DepartmentofLinguistics,CollegeofHumanitiesandSocialSciences,TheUnitedArabEmiratesUniversity,P.O.Box17771,AlAin,UAE;email:[e-mail]©TheAuthor(s),2009.ReprintsandPermissions:http://www.sagepub.co.uk/JournalsPermissions.navDOI:10.1177/0265532209340188Scoregeneralizabilityofacademicwritingtasks:Doesonetestmethodfititall?AttaGebrilTheUnitedArabEmiratesUniversity,UAEGeneralizabilityofwritingscoreshasalwaysbeenalongstandingconcerninL2writingassessment.Anumberofstudieshavebeenconductedtoinvesti-gatethistopicduringthelasttwodecades.However,withtheintroductionofnewtestmethods,suchasreading-to-writetasks,generalizabilitystud-iesneedtofocusonthescoreaccuracyofdifferenttasktypes.Thecurrentstudyattemptstoexaminehowreading-to-writetasksarecomparabletoindependenttasksintermsofscoregeneralizability.Tolookintothisissue,115Egyptianuniversitystudentswereaskedtowriteontwoindependenttasksandtworeading-to-writetasks.Theessayswereholisticallyscoredbythreeraters,whoreceivedtrainingpriortoscoring.Thestudydesignfollowedafullycrossedunivariatedesign(p×t×r),andconsequentlyGENOVA(Crock&Brennan,1983)wasusedtoanalyzetheresults.Resultsshowedthatthereading-to-writetasksyieldedasreliablescoresasindepend-enttasks.Inaddition,theresultindicatedthatscoregeneralizabilityisverylowwhenusingonewritingtaskduetothelarge(pt)variancecomponent.Implicationsandlimitationsofthestudyarepresented.Keywords:academicwriting,generalizabilitytheory,independenttasks,integratedtasks,scorereliability,testmethod,writingassessmentIIntroductionAssessingtheacademicwritingabilitiesofuniversitystudentshasreceivedconsiderableattentionsincethe1970swiththeinclusionofdirectwritingtasksinanumberofL2tests(Hamp-Lyons,1991).Theacademicwritingliterature(Carson,2001;Haleetal.,1996;Eblen,1983;Moore&Morton,1999;Horowitz,1986;Cansecon&Byrd,1989)hasprovidedevidencethatmostuniversityclassesrequirestudentstoperformdifferenttypesofwritingtasks,suchasatNATIONALTAIWANUNIVLIBonJanuary27,2010http://ltj.sagepub.comDownloadedfrom508Scoregeneralizabilityofacademicwritingtasksresearchpapers,reportsandessayexams.Consequently,universityadministratorsaskstudentstoprovideindicatorsoftheirwritingabilitytohelpinmakingeitheradmissionorplacementdecisions.Weigle(2004)referstotheinstitutionalizationofformalwritingprofi-ciencyrequirementsforallundergraduatestudentsinmanycollegesanduniversitiesintheUSA.Thisaction,asWeigleargues,cameasaresponsetoboththeexpansionofpostsecondaryeducationandcomplaintsbyemployersthatcollegegraduatesdidnothavetherequiredwritingskills.SincetheinclusionofdirectwritingtasksinL2tests,thewritingassessmentfieldhasfrequentlyusedthetimed-essayindependenttaskastheprimarymethodtoassessthewritingskillsofuniversitystudents.Hamp-LyonsandKroll(1996,p.18)callthistestmethoda‘snapshotapproach’towritingassessment.Independenttaskshavebeencriticizedbymanyresearchers(Gebril&Plakans,2009;Plakans,2007;Gebril,2006;Weigle,2002,2004;Cho,2003;Cumming,Kantor,Powers,Santos,&Taylor,2000;Leki&Carson,1997;Hamp-Lyons&Kroll,1996)sincetheyfallshortofadequatelytappingintotheacademicwritingconstruct.Forexample,Cho(2003)maintainsthatconventionalessaytestsincludetasksthatareneithertheoret-icallyvalidnorhaveaplaceinreal-lifecontexts.Anotherinherentprobleminindependenttasksistopicfamiliarity.Ifstudentsareassignedatopicwithoutsufficientbackgroundknowledge,thisvari-ablecouldaffectstudents’performanceonthetest.Consequently,construct-irrelevantvariancemayconfoundtestscores.Giventhiscriticism,integratedtaskshavebeenperceivedasanalternative,oraddition,toindependenttasks.Researchhasshownthatacademicwritingtasksarerarelydonewithoutusingreferencesourcesasabasisforwriting(Gebril,2009;Weigle,2002,2004;Cummingetal.,2000;Leki&Carson,1997;Hamp-Lyons&Kroll,1996).Integratedtasksareintendedtoreplicatethelanguagesitua-tionsthatstudentsoftenencounterinacademiccontexts(Lewkowicz,1997).Inatypicalintegratedtask,studentsreadtexts,analyzethem,reachconclusions,andthensynthesizethisinformationtoproduceapieceofwriting.Addedtothisadvantage,source-basedwritingprovidesbackgroundknowledgefortesttakers(Weigle,2004).Sincetesttakersbringdifferentculturalandindividualcharacteristicstotheexamroom,readingasourcetextbeforewritingwouldprovidethemwithacommonplatformand,asPlakans(2007)indicat