Combining qualitative evaluation and social networ

CombiningqualitativeevaluationandsocialnetworkanalysisforthestudyofclassroomsocialinteractionsA.Mart´ınezaY.DimitriadisbB.RubiacE.G´omezbP.delaFuenteaaDpt.ofComputerScience,UniversityofValladolid,SpainbDpt.ofSignalTheory,CommunicationsandTelematicsEngineering,UniversityofValladolid,SpaincDpt.ofDidacticsandSchoolOrganization,UniversityofValladolid,SpainAbstractStudyingandevaluatingrealexperiencesthatpromoteactiveandcollaborativelearningisacrucialfieldinCSCL.Majorissuesthatremainunsolveddealwiththemergingofqualita-tiveandquantitativemethodsanddata,especiallyineducationalsettingsthatinvolvebothphysicalandcomputer-supportedcollaboration.Inthispaperwepresentamixedevalua-tionmethodthatcombinestraditionalsourcesofdatawithcomputerlogs,andintegratesquantitativestatistics,qualitativedataanalysisandsocialnetworkanalysisinanoverallinterpretativeapproach.Severalcomputertoolshavebeendevelopedtoassistinthispro-cess,integratedwithgenericsoftwareforqualitativeanalysis.Theevaluationmethodandtoolshavebeenincrementallyappliedandvalidatedinthecontextofaneducationalandresearchprojectthathasbeengoingonduringthelastthreeyears.Theuseofthemethodisillustratedinthispaperbyanexampleconsistingoftheevaluationofaparticularcategorywithinthisproject.TheproposedmethodandtoolsaimatgivingananswertotheneedofinnovativetechniquesforthestudyofnewformsofinteractionemerginginCSCL;atPreprintsubmittedtoElsevierScience15April2003increasingtheefficiencyofthetraditionallydemandingqualitativemethods,sothattheycanbeusedbyteachersincurriculum-basedexperiences;andatthedefinitionofasetofguidelinesforbridgingdifferentdatasourcesandanalysisperspectives.Keywords:cooperative/collaborativelearning,evaluationmethodologies,learningcommunities,post-secondaryeducation1IntroductionDesignanddevelopmentofComputerSupportedCollaborativeLearning(CSCL)systemsisverycomplex,duetothediversityofimpliedactorsandthevarietyofissuestoconsider:learningimprovement,schoolorganization,culturalprob-lems,softwaredesign,distributedsystemsmanagement,human-computerinter-action,etc.Thiscomplexitydemandsappropriatemethodsofevaluationthatletresearchersandpractitionerslearnbyapplyinginnovativeexperiencesandreflect-ingonthem(Neale&Carroll,1999).Theapplicationofcomputerstocollaborativelearninghasbeenconsideredasanewresourceforresearchinthefield,duetotheircapabilityoflogginginteractionsandprocessingthemautomatically.However,italsopresentsnewchallenges,mainlyrelatedtotheappearanceofnewcollabora-tivesituationswithnewformsofinteractionandtoproblemsofautomaticdatamanagementandinterpretation(Guribye&Wasson,2002).InordertosupportthedevelopmentofCSCLsituationsweproposedaconceptualframeworknamedDELFOS(“aDescriptionofatele-EducationalLayeredFrame-workorientedtoLearningSituations”)(Osuna&Dimitriadis,1999).ItdefinesanarchitectureforthedesignofCSCLapplicationsandadevelopmentmethodologybasedontheideasofparticipatoryanalysisanddesign(Chin,Rosson,&Carroll,1997),whichemphasizestheroleofformativeevaluationinthedevelopmentof2informationsystems.WearecurrentlyworkingontherefinementofthemethodsandtechniquesdefinedinDELFOSforthisformativeevaluation.Forthispurpose,wedrawonasituatedlearningapproachthatdemandstheadoptionofaninterpre-tativeparadigmforevaluation.Thisperspectivepointsouttheneedofstudyingthelearningprocessesintheirrealcontexts,takingtheparticipants’perspectiveintoaccount,andconsidersbothindividualandsocialaspectsoflearning(Wilson&Myers,2000).InthepreviousversionofDELFOS,evaluationwasmainlyorientedtotheconstructivistaspectsoflearning,focusingontheindividualratherthanonthesocialperspective.Therefore,wearenowcompletingtheevaluationmethodinDELFOSbydefiningamethodologyandtoolsforevaluatingsocialaspectsrelatedtoparticipationinacommunityoflearners.AdisciplinethathasshowedtobeappropriatefortheefficientstudyofthesesocialandparticipatoryaspectsoflearningisSocialNetworkAnalysis(SNA)(Wasser-man&Faust,1994).SNAseekstodescribepatternsofrelationshipsamongactors,toanalyzethestructureofthesepatternsanddiscoverwhattheireffectsareonpeo-pleandorganizations.SeveralstudieshavedemonstrateditsvaluewithintheCSCLfieldforthestudyofstructuralpropertiesofindividualslearningingroupssuchasactors’prominenceornetworkdensity(Nurmela,Lehtinen,&Palonen,1999;Cho,Stefanone,&Gay,2002).Thesestudiesusuallytakecomputerlogsasaninput,andprocessthemwithaSNAsoftwarepackage,suchasUCINET(Borgatti,Everett,&Freeman,1999).However,SNAbyitselfisnotenoughforachievingafullun-derstandingoftheproblems,andneedstobecomplementedwithothermethods,likequalitativedataanalysis.Unfortunately,noguidelineshavebeenprovidedintheCSCLliteratureregardingtheintegrationofqualitativedataandmethodswithSNA.Ontheotherhand,existingSNAtoolsrequireahighlevelofexpertiseandtheyuseproprietarydataformats.AnapproachbasedonXML(W3C,2000)forthe3representationofcollaborativeinteractions,anditslaterprocessingbyintegratedautomatictools,couldthenofferameanstosolvetheproblemsofheterogeneityandintegrationinasystematicway.Weconsiderthattheprinciplesofqualitativecasestudyresearch(Stake,1995)con-stituteagoodframeworkt