OpenDist - incremental software distribution

ThefollowingpaperwasoriginallypresentedattheNinthSystemAdministrationConference(LISA’95)Monterey,California,September18-22,1995FormoreinformationaboutUSENIXAssociationcontact:1.Phone:[phone].FAX:[phone].Email:[e-mail]4.://–IncrementalSoftwareDistributionPeterW.OselandWilfriedGänsheimer–SiemensAG,München,GermanyABSTRACTOpenDistprovidesefficientproceduresandtoolstosynchronizeoursoftwarefileservers.Thissimplegoalbecomeschallengingbecauseofthesizeandcomplexityofsupportedsoftware,thediversityofplatforms,andbecauseofnetworkconstraints.Ourcurrentsolutionisbasedonrdist(1)[1].However,itisnotpossibleanymoretosynchronizefileserversnightly,becauseittakesseveraldaysjusttocomparedistantservers.Wehaveanalyzedtheupdateprocesstofindbottlenecksinthecurrentsolution.Wemeasuredtheeffectsofnetworkbandwidthandlatencyonrdist.Wecreatedstatisticsonthenumberoffilesandfilesizeswithinallsoftwarepackages.Wefoundthatnotonlythelinespeed,butalsothelinedelaycontributessubstantiallytotheoverallupdatetime.Ourmeasurementsrevealedthataddingacompressionmodetordistwouldnothavesolvedourproblem,sowedecidedtolookforanewsolution.Wehavecompiledalistofrequirementsforevaluatingsoftwaredistributionsolutions.Basedontheserequirements,weevaluatedbothcommercialandfreelyavailabletools.Noneofthetoolsfulfilledourmostimportantrequirements,soweimplementedourownsolution.Inthefollowingwewilldescribetheoverallarchitectureofthetoolsetandpresentperformancefiguresforthedistributionenginethatreplacesrdist.Theresultsoftheprototypeimplementationarepromising.WeconcludewithadescriptionofthenextstepsforenhancingtheOpenDisttoolset.OurEnvironmentTheCADSupportGroupoftheSemiconductorDivisionofSiemensAGinstalls,integratesanddis-tributesallsoftwareneededtodevelopIntegratedCircuits.WehavedevelopmentsitesinGermany(MünchenandDüsseldorf),Austria(Villach),theUnitedStates(Cupertino,CA),andSingapore.Thedevelopmentsitesareconnectedbyleasedlineswithaspeedof64to128kBit/s.Ateachsite,acentralfileserverstoresallsoftware.Clientworkstationsmountsoftwarefromtheseservers.SoftwareisinstalledandintegratedinMünchenanddistributedtoallotherdevelopmentsites.Systemadministra-torsofthedevelopmentsitesinitiatethetransferonthemasterserverinMünchen.TheCADSupportGrouptakescareoftheCADsoftwareandtools,only.Aseparatedepartmentisresponsibleforsystemadministration,i.e.,mainte-nanceoftheoperatingsystemandsystemtools,backups,etc.Oursoftwaredistributionproblemdiffersinmanywaysfromtheonesolvedbytraditionalsoftwaredistributiontools.Mostsoftwaredistribu-tiontoolswelookedataredesignedtodistributeamoderatenumberoffairlystaticsoftwarepackagesofmoderatesizetomanyclients.Incontrast,wehavetosynchronizefewfileservers(underadozen),whichstoremany(about200)packagesofsizesrangingfromtiny(acoupleofkilobytes)tohuge(1.8GBytes).Thetotalsizeofthesoftwarewestoreiscurrently25GBytes,10-15GBytesarecurrentlybeingkeptup-to-dateatallsites.Manypackagesarechangedeachday.Achangemightupdateonlyasinglefileofafewbytesorcouldchangeupto50,000filesforatotalof1GBytesperday.Everymonthabout10%ofthesoftwarechange.Mostchangesaresmall,butmanyfilesareconstantlyupdated.Theinstallationofahugepatchoranewsoftwarepackagechangesmanyfilesatonce.Thereisnoseparateinstallation-ortest-server,allchangesareappliedtothesystemswhileourclientsareusingthem.ThechangesaretestedinMünchenand,ideally,copiedtoallslavefileserverswithinoneday.Synchronizingorcloningfileserversisthebestwaytodescribeoursetup.OurCurrentSolutionOurcurrentsoftwaredistributionprocessusesrdist(1)tofindchangedfilesandtoupdateslavesoftwareservers.Itisnolongerpossibletocomparetwosoftwareserversinonenight.AcompletecheckofallsoftwarepackagesontheslavefileserverinSingaporewouldtakeseveraldayswhichisnotacceptablenorfeasible.Duringthattime,softwarepackageswouldbeininconsistentstates,andchangesofthemastersoftwareservercouldtake1995LISAIX–September17-22,1995–Monterey,CA181OpenDist–IncrementalSoftwareDistributionOsel&Gänsheimeruptoaweektobetransferredtotheslavefileserver.Thoughitispossibletoapplydifferentupdateschedules–updatingsmallpackagesdaily,someweekly–thesetupisnotsatisfactory.Withanever-increasingnumberofsoftwarepackagesandanever-growingsizeofeachsoftwarepackage,thedis-tributionprocessusingrdistisnotacceptableanymore.LANMÜNCHENDÜSSELDORFVILLACHCUPERTINOSINGAPORELineTypeEthernetISDNX.25leasedX.25leasedNominalLineSpeed[kBit/s]10,00064641286464Transferrate[kByte/s]90-1006-74-57-122-33-4PingResponseTime[ms]133-88188-37281-311530-1083617-1375rdistfilecreate[s]0.20.21.20.62.14.5rdistfilecheck[s]0.020.060.50.21.2.1rdistfiledelete[s]0.10.130.50.31.2.310kBytestransferrate[kByte/s]-5.92.54.91.51.4RunBenchmark[h]12.5741224rdistcheckSWsubset[h]--168-802OpenDistcheckSWsubset[h]30.5-21.5.75.75rdistcheckallSW[h]23-6927140290OpenDistcheckallSW[h]1.5-511.5231Increasedtime,becausesoftwarepoolsinDüsseldorfareaccessedviaNFSnotUFS.2Estimated.3Thissubsetconsistsoftechnologydataandischangedanddistributeddaily