A Middleware System for Distributed Real-Time Mult

AMiddlewareSystemforDistributedReal-TimeMultimediaProcessingNelsonPosseLago1y,FabioKon11DepartmentofComputerScienceInstituteofMathematicsandStatisticsUniversityofSã[e-mail],[e-mail]tedapplicationsforMultimediaprocess-ingwithreal-timeandlow-latencyrequirementsfacesafewkeychallenges.Inthispaper,wecharacterizesuchapplicationsas“rmreal-timeapplications”,arguethattheyshouldbebasedonacallbackarchitecture,anddiscusstheas-sociatednetworklimitations.Wepresentanewmiddlewaresystemaddressingthechallengesdiscussedanddescribeanapplicationfordistributedaudiopro-cessingbuiltontopofthismiddleware.Experimentalresultsobtainedwiththisapplicationdemonstratetheeffectivenessofourapproach.1.IntroductionInseveralcomputersystemsformultimediaprocessing(suchasinteractivesystemsforthecreationandeditionofmultimedia,particularlyaudioandmusic,orsystemsforpat-ternrecognitionincontinuousmedia),itishighlydesirabletobeabletodotheprocessingnotonlyinreal-time,butalsowithlowlatency.Lowlatencyprocessingmeansthatthetimeittakesforachangeintheinputdataofthecomputersystemtoproducethecorre-spondingoutputshouldbeassmallaspossible;howsmallisenough,giventheusualgoalthatthelatencymustnotbeperceptiblebytheuser,variesalotwiththeapplicationandtheuser,butitseemsthatupto5msforthemostcriticalinteractiveapplicationsisrea-sonable(seeadiscussiononthesubjectinSteinmetzandNahrstedt,1995,p.588–599).Ininteractivesystems,forinstance,lowlatencyprocessingservesthepurposeofgivingtheusertheillusionthatthesystemperformsthecomputationsimmediately,whichisveryimportantsincetheusergenerallyadjustshisinputtothecomputersystemaccordingtotheoutputhereceivesfromthesystem.Lowlatencymayalsobeimportantifwewantpartofthedatatobeprocessedinreal-timebyexternaldevices(forinstance,wemaywanttorouteapreviouslycapturedaudiosignalintoananalogeffectsprocessorandrecordtheresultingsoundwithoutlosingthetiminginformationofthesignal).Asimpleexampleofasituationinwhichreal-timelow-latencyprocessingisde-sirableistherecordingofanacousticmusicalinstrumentwithsomeeffectprocessing(forThisresearchissupportedinpartbyagrantfromCNPq,proc.55.2028/02-9.yNelsonPosseLagoispartlysupportedbyCAPES.instance,anelectricguitarprocessedbyacustomdigitaldistorter):whileplayingthein-strument,themusicianneedstohearthesoundbeingproduced;iftheprocessinglatencyistoolarge,themusicianwillhavedifcultiestoperformcorrectly.Systemsformultimediaprocessinginreal-time,includingthosewherelowla-tencyisdesirable,usuallydemandlargeprocessingpowerfromthecomputersystem.Problemsthatdemandlargecomputingpower(asmultimediadoes)areusuallysolvedbyparallelordistributedprocessing.However,thereal-timeandlowlatencyrequirementsofmostmultimediaprocessingsystemscoupledwiththeneedforsometimesstrictsyn-chronizationbetweenseveralmediastreamsaswellasthecostofmultiprocessorsystemshavemademostmultimediaapplicationstobedevelopedforsingleprocessorsystems.Intheaudioandmusicprocessingeld,itisnotuncommonforsuchsystemstobecoupledwithdedicated,specializedhardwareinordertoboostthesystemperformance.Suchhardware,however,isusuallyproprietaryandexpensive:forexample,asinglepro-cessingboardfortheProToolsHDsystemcosts4timesasmuchasacompletemid-rangedesktopPCintheUSA1.Onsmallerstudioswithoutaccesstohigh-endequipment,itisrelativelycommonforthecomputingpowerduringaudioeditingtobeexceeded;whenthishappens,usuallypartoftheprocessingisdoneinnon-realtimemodeandthepro-cessedresultissavedtodisk,whichisinconvenient,sincethepossibilityofinteractiveexperimentationislost.Giventheeconomicadvantageandexibilityofferedbygeneral-purposecom-putersystems,beingabletoprocessmultimediadatainadistributedsystemwouldbeuseful,allowinguserstogobeyondtheperformancelimitsofsingleprocessorsystemsinamorecost-effectiveway.Homeandsmallmusicrecordingstudios,whichusuallycannotaffordtheexpensiveproprietarysolutions,wouldbenetfromtheuseofsmallclustersofolderandinexpensivecomputerstoincreasetheirprocessingpoweratalowcost.Thispaperdescribes(1)therequirementsandlimitationsofmechanismsfordis-tributedmultimediaprocessinginreal-timewithlow-latency,(2)asimplemiddlewaresystemwhichisatestbedfortheideaspresentedhere,and(3)anapplicationgearedto-wardsaudioprocessingdevelopedontopofthismiddleware.Insection2.1.wecharacter-izesystemsformultimediaprocessingasrmreal-timeapplicationsandargue,insection2.2.,thatlow-latency,rmreal-timesystemsshouldbebasedoncallbackfunctions.Insections2.3.and2.4.,wediscussthepossibleapproachesforloaddistribution,addressthelimitspresentedbythenetworkingmedium,andproposeamechanismtosolvethem.Finally,insection3.1.,wedescribethecurrentimplementationofthesystemandpresentexperimentalresultsinsection5.2.ProblemsandmechanismsWhendesigningamethodfordistributedprocessingofmultimediainreal-timewithlowlatency,onemusttakeintoaccountseveralfactors.Thespeciccharacteristicsofmul-timediawithregardtoreal-timeandlowlatency,whichsuggesttheuseofspecializedreal-timeoperatingsystems(suchasRT-Linux–BarabanovandYodaiken,19