PREPRINT An Automated Approach for Identifying Pot

PREPRINTAnAutomatedApproachforIdentifyingPotentialVulnerabilitiesinSoftwareAnupK.Ghosh,TomO’Connor,&GaryMcGrawReliableSoftwareTechnologiesCorporation21515RidgetopCircle,#250,Sterling,VA20166faghosh,toconnor,[e-mail]-ologyemployssoftwarefaultinjectiontoforceanoma-lousprogramstatesduringtheexecutionofsoftwareandobservestheircorrespondingeectsonsystemse-curity.Ifinsecurebehaviorisdetected,theperturbedlocationthatresultedintheviolationisisolatedforfurtheranalysisandpossiblyretrottingwithfault-tolerantmechanisms.1AnalyzingthebehaviorofsoftwareItisnowwellunderstoodthatavastmajorityofsecurityintrusionsaremadepossiblebyawsinsoft-ware.OneneedonlylookattheannalsofBugtraqforempiricalevidenceofthisassertion.1Toaddressthisproblem,computersecurityresearchersandpracti-tionershavecreatedmaturesoftwareengineeringpro-cessessuchastheTCSECandtheSystemsSecurityEngineeringCapabilityMaturityModel(SSECMM)[13]toimprovethelikelihoodofproducingmorese-curesystems.Anotherbodyofresearchfocusesonproducingse-cureprotocolsfortransportingandaccessingcon-dentialdataacrossinsecurenetworksandinsharedThisworkissponsoredundertheDefenseAdvancedRe-searchProjectsAgency(DARPA)ContractF30602-95-C-0282.theviewsandconclusionscontainedinthisdocumentarethoseoftheauthorsandshouldnotbeinterpretedasrepresentingtheofficialpolicies,eitherexpressedorimplied,ofthedefenseadvancedresearchprojectsagencyortheu.s.government.1Bugtraqcanbeviewedon-lineatedprotocolsplayanecessaryandimportantroleindevelopingsecuresystems.Itisimportanttonote,however,thateventhemostrigorousprocessescanproducepoorqualitysoftware[17].Likewise,eventhemostrigorouslyandformallyanalyzedprotocolspeci-cationcanbepoorlyimplemented.Inpractice,mar-ketpressurestendtodominatetheengineeringanddevelopmentofsoftware,oftenattheexpenseoffor-malvericationandeventestingactivities.Thisisespeciallytrueofcommercialgradesoftwareforusebyconsumers.Theresultisasoftwareproductem-ployedinsecurity-criticalapplications(suchasInter-netclientsandservers)whosebehavioralattributesinrelationtosecurityarelargelyunknown.Theobjectiveoftheapproachpresentedhereistoprovidethecapabilitytoanalyzesoftwarepro-gramsforpotentialvulnerabilitiesthatcanbelever-agedintosecurityintrusions.Softwaredeveloperscur-rentlyhaveattheirdisposalanumberoftechniquesforaidinginthedevelopmentofqualitysoftware,in-cluding:programdebugging,congurationmanage-ment,memoryleakdetection,performanceproling,loadtesting,analysisofstructuralmetrics,testcasegeneration,andcodecoverageanalysis.Whileallofthesetoolsifproperlyusedcanresultinhigherqualitysoftware,nonearespecicallyorientedtowardanaly-sisofsecurityproperties.Theanalysistechniquepre-sentedinthispaperisspecicallyorientedtowardsidentifyingportionsofsoftwarethatifawedcanresultinsecurityviolations.Thispaperpresentsanauto-matedapproachandtoolforsimulatingprogramawstodeterminetheirpotentialeectonsystemsecurity.Resultsfromapplyingtheautomatedfaultinjectionanalysisonvecommonnetworkservicedaemonsarepresented.Theapproachdescribedherefocusesonthebehav-PREPRINTiorofthesoftwarewhenexecutingunderanomalouscircumstances.Thetoolprovidesthedeveloperoranalystwithsomeassuranceforhowbadlythesoft-warecanbehavewhensubjectedtounexpectedevents.Untilsoftwarehasbeenexercisedthoroughlyunderanomalousormaliciousscenarios,thesecurityofitsbehaviorwillremainunknown.CheswickandBellovinsummeditupnicelyin(page7,[4]):...anyprogram,nomatterhowinnocuousitseems,canharborsecurityholes.(Whowouldhaveguessedthatonsomemachinesintegerdivideexceptionscouldleadtosys-tempenetrations?)Wethushavearmbe-liefthateverythingisguiltyuntilprovenin-nocent.[emphasisadded]Findingawsinsoftwarethatcanbeleveragedintofull-scalesecurityintrusionshasbeenlikenedtond-inganeedleinahaystack.Thisapproachispracticedregularlybycomputer\crackersseekingtoleveragesoftwareawsintohigherlevelsofprivilege.Theap-proachdescribedinthispaperdoesnotpurporttondtheneedleinthehaystack,butrathertoreducethesizeofthehaystacksignicantlybyusingauto-matedsoftwareanalysismethods.Knowingtheloca-tionsofallawsinanynon-trivialpieceofsoftwarerequiresomnipotence.Ratherthansearchingforawsinsoftware,theapproachsimulatestheeectofawsinsoftwarebyusingdataperturbationfunctionsbet-terknownasfaultinjectionfunctions.Ifthesimulatedawviolatesthesecurityofthesystem,thenthelo-cationwheretheawwasintroducedisidentiedforfurtherinvestigationandpossiblyretrottingoffault-tolerantmechanisms.Thisapproachhasbeensuccessfullyappliedinotherareasofcriticalsoftwaresystemswhereawsinsoftwarecanresultincriticallosses[14,15,16].Thecommongroundbetweentheanalysisofsa