In-kernel servers on Mach 3.0 Implementation and p

In-KernelServersonMach3.0:ImplementationandPerformanceJayLepreauMikeHiblerBryanFordJereyLawCenterforSoftwareScienceDepartmentofComputerScienceUniversityofUtahSaltLakeCity,UT84112E-mail:flepreau,mike,baford,[e-mail]inmodularityandpowerofmicrokernel-basedoperatingsystemssuchasMach3.0arewellknown.Theexistingperformanceproblemsofthesesystems,however,aresignicant.Muchoftheperformancedegradationisduetothecostofmaintainingseparateprotectiondomains,traversingsoftwarelayers,andusingasemanticallyrichinter-processcom-municationmechanism.Anapproachthatoptimizesthecommoncaseistopermitmergingofprotectiondomainsinperformancecriticalapplications,whilemaintainingprotectionbound-ariesfordebuggingorinsituationsthatdemandrobustness.Inoursystem,clientcallstotheserverareeectivelyboundeithertoasimplesystemcallinterface,ortoafullRPCmechanism,dependingontheserver’slocation.Theoptimizationreducesargumentcopies,aswellasworkdoneinthecontrolpathtohandlecomplexandinfrequentlyencounteredmessagetypes.InthispaperwepresentageneralmethodofdoingthisforMach3.0andtheresultsofapplyingittotheMachmicrokernelandtheOSF/1singleserver.Wedescribethenecessarymodicationstothekernel,thesingleserver,andtheRPCstubgenerator.Semanticequivalence,backwardscompatibility,andcommonsourceandbinarycodearepreserved.Performanceonmicroandmacrobenchmarksisreported,withRPCperformanceimprovingbyafactorofthree,Unixsystemcallstotheserverimprovingbetween20%andafactoroftwo,and4{13%performancegainonlargebenchmarks.AbreakdownofthetimesontheRPCpathisalsopresented.11IntroductionThemodularityofmicrokernel-basedoperatingsystems(OS’s)providesthemwell-knownben-ets.Thesebenetsincludeimproveddebuggingfacilitiesandsoftwareengineeringimprovementswhichresultfromahigherdegreeofsystemstructuring.Theuseofseparateprotectiondomainstoimplementsystemsontopofamicrokernelprovidesrobustness;theabilitytocomposeserverstoexhibitdierentOS\personalitiesoersexibility.Inaddition,microkernelsexportpowerfulabstractionswhichareusefulforbuildingdistributedsystemsandmulticomputerOS’s.However,thismodularityandpowerhascomeatsomecost.Becausethemodularityisstrictlyatthetasklevelandtaskscommunicateviaapowerfulbut\heavyinterprocesscommunication(IPC)mechanism2,allinter-taskinteractionsareexpensive.Theexpensiveinter-taskoperationsincludethecommoncaseofuntrustedclientscallingtrustedserversonthesamemachine,passingsmallamountsofdatabackandforth.1ThisresearchwassponsoredbytheHewlett-PackardResearchGrantsProgram.2Althoughtaskscancommunicatethroughsharedmemory,muchinter-taskinteractionishandledviaremoteprocedurecalls(RPCs).AfewcommonI/Ooperations,suchasUnixread,areoftenimplementedwithsharedmemoryandcanbereasonablyfast.AlthoughtheMach3.0[1,8]microkernelhasshownisolatedinstancesofperformancecomparabletothatofmacrokernelsystems[5],thishasnotbeenobservedingeneral.Manysoftwarelayersandsomehardwareboundariesmustbetraversedevenforthemosttrivialinteractionsbetweentasks.AllmessagesexercisemuchoftheIPCmechanism,eventhoughcommonservers,suchastheUnixserver,useonlyasmallsubsetofitscapabilities.Theconclusionisthatforinter-taskcalls,Machhasfailedtooptimizethecommoncase.InstrumentationoftheMachIPCpathontheHewlett-PackardPA-RISChasshownthatatleastonthisplatform,thetraditional\contextswitchfromoneaddressspacetoanotherisonlyaminorcomponentofsame-machineMachRPCcosts.Asdetailedlater,mostofthecostisinkernelentryandexitcode,datacopiesintheMIG-generatedstubroutines,andinportshuingandrelatedcode.Wehavelargelyeliminatedthelasttwocostsinthecommoncase.Tooptimizethecommoncase,wehaveconstructedaseparatelylinkedOSF/1serverimage(referredtoasthein-kernelserverorINKS)thatcanbeloadedintothekernelprotectiondomainatruntime.WedynamicallyreplaceheavyweightIPC-basedcommunicationmechanismswithlighter-weighttrapandprocedurecallmechanismswherepossible.WehaveextendedtheRPCstubgeneratortoproducespecialstubswhichenactthesenewmechanisms.Sincethechangeofcommunicationmechanismisessentiallyachangeinprocedurebinding,thepresenceorabsenceofthesemechanismsistransparenttoclients.Intherestofthispaperwediscussthegoals,constraints,anddesigninSection2,giveim-plementationdetailsinSection3,andgivestatusandperformanceresultsinSection4,includingbreakdownoftheRPCpath,microbenchmarks,andmacrobenchmarks.Finally,weexaminerelatedwork,discussplannedandpossiblefuturework,andgiveourconclusions.2Design2.1GoalsOurgoalswereve-fold.Mostimportantly,wewantedtoimprovetheperformanceofsystemsbasedonMach|boththoseimplementingUnixandmorespecializedsystems,suchasnetworkprotocolservers.Nearlyasimportant,wesoughttobeascompatibleaspossible,aselaboratedbelow.Wewantedtomaketherequiredsystemtransformationsacceptable,byimposingminimalconstraintsonMachprogramstructure.Wewantedtoexploretheissuesinvolved,andnally,wewantedasamplesystemasatargetforamoredynamicbindingmechanism,basedonanobjectserverwehaveimplemented[9,10].2.2CompatibilityConstraintsTomaintaincompatibility,weimposedanumberofconstraintsonthedesign:Thesourceforin-andout-of-kernelserversmustbeidentical,withlittleus