Model Analysis Assessing the dynamics of student l

1ModelAnalysis:AssessingtheDynamicsofStudentLearningLeiBaoDepartmentofPhysics,TheOhioStateUniversity174West18thAve.,Columbus,OH43210-1106EdwardF.RedishDepartmentofPhysicsUniversityofMarylandCollegePark,MD20742-4111AbstractInthispaperwepresentamethodofmodelingandanalysisthatpermitstheextractionandquantitativedisplayofdetailedinformationabouttheeffectsofinstructiononaclass’sknowledge.Themethodreliesonacognitivemodelofthinkingandlearningthatrepresentsstudentthinkingintermsofpatternsofassociationinlong-termmemorystructuresthatwerefertoasschemasormentalmodels.Asshownbypreviousresearch,studentsfrequentlyfailtorecognizerelevantconditionsthatleadtoappropriateusesoftheirmentalmodelsand,asaresult,canusemultiplemodelsinconsistentlytotreatproblemsthatappearequivalenttoanexpert.Oncethemostcommonmentalmodelshavebeendeterminedviaqualitativeresearch,theycanbemappedontoprobinginstrumentssuchasamultiple-choicetest.WehavedevelopedModelAnalysistoanalyzetheresultsoftheseinstrumentsthattreatsthestudentasifhe/shewereinamixedstate–astatewhich,whenprobedwithasetofscenariosunderdiversecontextualsettings,givestheprobabilitythatthestudentwillchooseaparticularmentalmodeltoanalyzethescenario.WeillustratetheuseofourmethodbyanalyzingresultsfromtheForceConceptInventory,aresearch-basedmultiple-choiceinstrumentdevelopedtoprobestudent’sconceptualunderstandingofNewtonianMechanicsinaphysicsclass.ModelAnalysisallowsonetousequalitativeresearchresultstoprovideaframeworkforanalyzingandinterpretingthemeaningofstudents’incorrectresponsesonawell-designedresearch-basedmultiple-choicetest.Theseresultscanthenbeusedtoguideinstruction,eitherforanindividualteacherorfordevelopersofreformcurricula.I.IntroductionTheProblem:Howcanweevaluatewhatourstudentsknowinalargeclassroom?Oneofthemostimportantthingseducationalresearchershavelearnedoverthepastfewdecadesisthatitisessentialforinstructorstounderstandwhatknowledgestudentsbringintotheclassroomandhowtheyrespondtoinstruction.Insmallclasses,thisinformationcanbeobtainedfromcarefulone-on-onedialogsbetweenstudentandteacher.Inlargeclasses,suchasthosetypicallyofferedinintroductoryscienceatcollegesanduniversities,suchdialogsareallbutimpossible.Instructorsinthesevenuesoftenresorttopre-posttestingusingtheirownorresearch-basedclosed-endeddiagnosticinstruments.Theresultsfromtheseinstrumentstendtobeusedinaverylimitedway—throughoverallscoresandaveragepre-postgains.Thisapproachmayignoremuchvaluableinformation,especiallyiftheinstrumenthasbeendesignedonthebasisofstrongqualitativeresearch,containssub-clustersofquestionsprobingsimilarissues,andhasdistractersthatrepresentalternativemodesofstudentreasoning.Inthispaper,wepresentamethodofmodelanalysisthatallowsaninstructortoextractspecificinformationfromawell-designedassessmentinstrument(test)onthestateofaclass’sknowledge.Themethodisespeciallyvaluableincaseswherequalitativeresearchhasdocumentedthatstudentsenteraclasswithasmallnumberofstrongnaïveconceptionsthatconflictwithorencouragemisinterpretationsofthecommunity-consensusview.2TheTheoreticalFrame:Knowingwhatwemeanby“whatourstudentsknow”requiresacognitivemodel.Althoughthedesireto“understandwhatourstudentsknow”isanhonorableone,wecannotmakemuchprogressuntilwebothdevelopagoodunderstandingofthecharacteristicsofthesystemwearetryingtoinfluence(thestudent’sknowledgestructure)andhavealanguageandtheoreticalframewithwhichtotalkaboutit.Fortunately,muchhasbeenlearnedoverthepastfewdecadesabouthowstudentsthinkandlearnandmanytheoreticalmodelsofhumancognitionhavebeendeveloped.1Unfortunately,theyhavenotyetcoalescedintoasinglecoherentmodel.Althoughsignificantdifferencesamongthemremain,thereismuchcommongroundandmuchthatisvaluableforhelpingboththeeducationalresearcherandtheteacher.Despitetheprogressincognitivescience,mosteducationalresearchersanalyzingreal-worldclassroomsmakelittleuseofthisknowledge.Manyofthemathematicaltoolscommonlyusedtoextractinformationfromeducationalobservationsrelyonstatisticalmethodsthat(oftentacitly)assumethatquantitativeprobesofstudentthinkingmeasureasysteminaunique“true”state.Webelievethatthismodelofthinkingandlearningisnotthemostappropriateoneforanalyzingastudent’sprogressthroughgoal-orientedinstructionandisinconsistentwithcurrentmodelsofcognition.(Exampleswillbegiveninthebodyofthepaper.)Asaresult,theanalysisofquantitativeeducationaldatacandrawincompleteorincorrectconclusionsevenfromlargesamples.Inthispaperwehopetomakeasteptowardsamelioratingthissituation.Webeginbysketchingapartofatheoreticalframeworkbasedontheworkofmanycognitiveresearchersineducation,psychology,andneuroscience.Thisframeworkdescribesahierarchyofstructuresincludingelementsofknowledge(bothdeclarativeandprocedural),patternsofassociationamongthem,andmappingsbetweentheseelementsandtheexternalworld.2Thewell-knowncontext-dependenceofthecognitiveresponseisrepresentedbyprobabilitiesintheassociativelinks.Notethattheseprobabilitiesdonotrepresentsamplingprobabilitiesassociatedwithastatisticalanalysisofeducationaldatatakenfrommanystudents.Theseprobabilitiesarefundamentaland