Design and implementation of a graphic-haptic disp

Designandimplementationofagraphic-hapticdisplaysystemFarrokhJanabi-Shari¯,a;¤VincentHayward,bQingYuanWangcaDepartmentofMechanicalandIndustrialEngineering,RyersonUniversity,Toronto,Ontario,Canada,M5B2K3bCenterforIntelligentMachines,McGillUniversity,Montr¶eal,Qu¶ebec,Canada,H3A2A7cMotorolaCanadaLtd.AbstractDespitecurrentadvancesinmultimediaenvironments,tracingthegeometricalstruc-turesofgraphicalimagesusingforcefeedbackremainsanimportantresearchissue.Inthispaper,thedevelopmentandimplementationofaMulti-ModalDisplaySys-tem(MMDs)fortracing2Dboundariesingraphicimagesarediscussed.Amethodisproposedthatprovidesatypeofhapticfeedbackdesignedtoassistausertotracethecontoursofobjectsseeninimages.Thismethodisanexampleofafamilyofhapticsynthesismethodswherebytheforce¯eldexploredbytheuserisdynamicinthesensethatitdependsbothonmovementaswellasontheobjectbeinghap-ticallyrepresented.Theproposedperformance-basedmethodprovidesuserswithamovementguidancethroughanactivehapticsenseratherthancommonimpedancetechnique.Thetracinge®ectivenessoftheproposedmethodisveri¯edexperimen-tally.PreprintsubmittedtoElsevier24November2006Thisistheauthors'draft.ThefinalversionispusblishedinDisplays28(2007)118–128Keywords:hapticinterface,boundaryextraction,imagedisplay,userinterface1IntroductionRealworlddecisionsareusuallymadebasedonthecombinationofsensesfromenvironment.Amongpossiblecombinations,hapticandvisioncombi-nationcontributestoalargeclassesofourdailytasks.Whilevisionisthequickestandmoste±cientsensefortheperceptionofspatialevents,touchprovidestactileandkinestheticinformation.Integratingvisualandtactilein-formation,notonlyprovidesvisuallyabsentcharacteristicsofsurfaces,butalso,aspsychophysicalstudiesshow,facilitatesthemanipulationofobjectsintermsofreducingcognitionloads,errors,timeandenergyrequiredtocom-pleteatask.Inhuman-machineandinteractivecomputersystems,visualandhapticinformationareexpectedtobetransferredtotheuserthroughacoor-dinatedgraphic-hapticdisplaysystem.Acoordinatedgraphic-hapticdisplaycanplayimportantrolesinseveralapplicationareas,suchasteleoperation,virtualsurgerysimulation,entertainment,scienti¯cvisualization(Hollerbach,2000),etc.Inthiswork,ane®ectivemethodfordesignandimplementationofahapticdisplayfortracingcontoursinimagesofagraphicdisplayisproposed.Tracingcontoursindigitalimagesconstitutesanimportantclassofourdailytasks.Thisisneeded,forexample,toquantitativelyassessthesizeofstructuresinmedicalimages(e.g.,tomonitorgrowthovertime),ortooutlinetheirshapeforregistration.AgoodexampleisprovidedbyNIH'sVisibleHumanProjectwherethousandsofslicesweremanuallytracedtosegmenttissues.¤E-mail:fshari¯@ryerson.ca.Tel:[phone]x7097.Fax:[phone]Despiteassistancefrommachinevisiontechniques,mostoftheworkwasdonemanually(Crawfored-Hineset.al.,1998).ProfessionalimageprocessingtoolssuchasPhotoshopTM(forgeneralpurposeimages)orManifoldTM(fordomainspeci¯cimages)o®erautomaticassistancetotracingcontours,butalwaysinconjunctionwithmanualinput.Ascanbereadilyappreciatedbygraphicinterfaceusers,manualtracingisstraininganderrorprone.Therefore,thereisanopportunitytotakeadvantageofhapticfeedbacktechniquestoprovideassistancetothetaskoftracingthecontoursoftheimageobjects,tofacilitateinspectionandmanipulation.Therehavebeenanextensiveresearchonhapticdisplays,mostlyrelatedtophysics-basedmodelingandrenderingofdeformableobjects.ExamplesincludeworkofZillesandSalisbury(1994)whereapoint-basedrenderingapproachistakentorepresentagod-object.Otherapproachestophysics-basedmodelinghavebeenalsoreported(SrinnivasanandBasdogan,1997).Themainchallengeistoachieveanoptimalbalancebetweenthecomplexityofphysicalmodelsandtherealismofhapticandgraphicdisplaysinreal-time(Duriezetal.,2006).Thereislittlepriorworkcomparedtoours.RecentworkisthatofFukuiandShimojo(1992)whoemployedaninputdevicetoinvestigatethehumansensitivitytovirtualshapeswhilefollowingtheircontourswhenforcefeedbackisavailable.Inotherwords,almostnoattentionhasbeenpaidtoperformanceoftheuser.Theyusedaninputdevicecomprisingatwo-dimensionalforcesensormountedontheheadofanxyrecorder.Thisdeviceswasconnectedtoacomputercommandedthedevicetomoveinthedirectionofforceappliedtotheuser.OfparticularinteresthereistheworkofKruze(1997)whostudiedmethodstoproduce2D\hapticimagesfrom2Dgraphics,andthatofShiandPai(1997)whodevelopedasystemtodemonstratetheautomaticgeneration3ofhapticinteractionwith3Dobjectsderivedfromstereo2Dimages.Theseworksapproachtheproblemofinformationextractionfromanimageinvariousways,butinvariably,thehapticfeedbackissynthesizedfromastaticforce¯eldwhichisentirelyafunctionofthegraphicsunderconsideration.ThisisdepictedinFigure1.Oneproblemwithtranslatingapictureintoastaticforce¯eld{forexamplebasedontheimagegradient{isthatthesynthesizedforcefeedbackdoesdependonthetaskwhichisexpressedbythemovementsoftheuser.Thatiswhatmakesthepreviousworknon-performance-based.Incontrast,inthispaperweintroduceanapproachwherebytheforce¯eldisafunctionofboththeimageandthemovementsoftheuser.Ourapproachistoautomaticallyextractfeaturesfromanimageandtousethemto