ChinaTest2012-T7-陈耿自动化测试的演进和成熟度

[e-mail]se:KnowwhereyouareKnowwhereyougoCompanyOrgKnowwhattolearnKnowwhattoresearchIndividualIndustryWhatmakeAutomationTestEvolveWHATISTHEREALDRIVE?TechHumanneedsInterestingImproveskillManagerRequireFashion•Products/ServicesCurse:maintenanceeffortRealProject•Repeated•RedundancyEfficiencyImprove•Stress•Aging•PerformanceBetterQuality•Stress•PerformanceHumanunableAutomationEvolveGeneralEvolutionAutoExe(4Levels)AutoAnalyze(3Levels)AutoStrategy(Unknown)MostCompaniesStayHereIndustryR&DConcepts。。。→MakeTestExecutionAutomationConformiq…→MakeTestAnalyzeAutomation→MakeTestManagementAutomatedAutoExe(4Levels)MostCompaniesStayHere。。。→MakeTestExecutionAutomationAutoAnalyze(3Levels)AutoStrategy(Unknown)IndustryR&DConceptsConformiq…→MakeTestAnalyzeAutomation→MakeTestManagementAutomatedAutoExelevel1——IndividualInitialBatchedScriptingNonDedicateAutoTesterManualWorkFlowManualTestManagementJustBatchingRepeatedOperation,No/FewResultingCheckingEx.ShellScript,AutoIT※AutoTesterManualTesterKeyMotivation:“Lazy”(TooManyManualOperation)AutoExelevel2——Project/ProductBaseDedicatedAutomationSystemToolTeamStartManualTestInvestigatingROIMostCompanyStaysHereSHALLFocusingonROIBeforelaunchAutomationProject※AutoTesterManualTesterAutoTestSingleProductsLackofresourceLowROIKeyMotivation:TestExecutionEfficiencyImprovementAutoTotalCostAutoExelevel2——Project/ProductBaseKeyofROIOpportunitiesThreats•RepeatwithoutANYChange•PeoplecanNOThandle•ChangeFrequently•ManualCostLowToolDEVScriptDEVScriptmaintainResultAnalyzeReportToolmaintainManualTotalCostManualExecutionReportManagementWholeTestCyclesWholeTestCyclesROIAnalyzeSample(RealProject)0102030405060704.0.2.24…4.0.2.28…4.0.2.34…4.0.2.36…4.0.2.38…4.0.2.40…4.0.2.45…4.0.2.47…4.0.2.49…4.0.2.51…4.0.2.53…4.0.2.55…4.0.2.57…4.0.2.61…AutoTest（单位：天/人）ManualTest（单位：天/人）自动化测试执行趋势AutomationEffortReducingNewRelease:automationeffortbiggerthanmanualNewRelease:automationeffortbiggerthanmanualManualEffortChangeRapidlybasedonCaseNumberEffortAnalyze050001000015000200002500030000350004.0.2.24…4.0.2.34…4.0.2.38…4.0.2.45…4.0.2.49…4.0.2.53…4.0.2.57…AutoTotalCostManualTotalCost自动化测试执行趋势AutomationCostHigherthanManualCostAnalyze•EffortReducedhalf•EffortConsumestableButCalculatewithCost•ManualTester—8300RMB/Month•AutoTester—15000RMB/Month•AutomationTestCostishigherorequalasManualTestTotalProjectROI1526.0170454523.16621.635775051015202530AutoTestManualTestEffortCost•TotalEffort—Automationislow•TotalCost—Automationishigh058186113180AutoTest（单位：天/人）ToolDevScriptDevScriptmaintainResultAnalyzeReportToolmaintain•ToolMaintainalmostconsumehalfautomationeffortAutoExelevel2——Project/ProductBaseTechIntroduce!ArchPurposeOS•Windows•Linux•Embedded•CrossPlatform•Local•Distributed•Common•Batchsystem•SpecificAutoExelevel2——Project/ProductBaseOpenSourceToolCompareToolUsabilityMaintainabilitycomplexityJameleon☆☆☆☆☆☆☆☆☆☆GUITAR★★☆☆☆☆☆☆☆☆★★☆☆☆☆☆☆☆☆★★☆☆☆☆☆☆☆☆SoftwareTestingAutomationFramework(STAF)★★★★★★☆☆☆☆★★★★★★★☆☆☆★★★★★★☆☆☆☆Abbot★★★★☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆Haste★★☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆Jsystem★★★★★★☆☆☆☆★★★★★★☆☆☆☆JOSIT/Marathon/UISpec4J/Slimdog/Jemmy/Testare/JavaTestRunner/Jetif/Pounder★★★★☆☆☆☆☆☆AutoExelevel2——Project/ProductBasePropertyPlatformCharactersOpensource(52)Windowsonly(30)GUIautomatictools(16)Commercial(27)Linuxonly(2)AutomationTestingFramework(6)Free(7)Windows,Linux(12)Automationtoolsforwebapplications(38)OSIndependent(40)Performance/Loadtestingtool(4)Otherplatforms(2)AutomationtoolsforJavaGUI/Applications(10)AutomationtoolsforDatabaseapplications(1)Testingtooltocalculatecodecoverage(2)Automationtestingtoolforgeneratingunittestcase(1)TestingframeworkforXML-basedtestautomation(1)Automatedtestingtoolforserialcommunicationprotocols(1)ApplicationLifecycleManagement(1)Automatedtestingtoolforembeddedapplications(2)AutomatedtesttoolforWindowsclient/serverapplications(2)AutomatedtestingframeworkforSmalltalk(1)TestDataGenerationTool(1)AutomationToolforLinux(1)Detailedmaterial86availableAutomationTestingToolsAutoExelevel2——Project/ProductBaseRecord/PLAY•CreatescriptEasy•EasytoStart•HardtomaintainForstableUIandRegressionTeamTechislow——noscriptdesigningskillAutoExelevel2——Project/ProductBaseOrgChange:ManualTestManualScriptToolSkillDifferentHighCostHighROIRiskBeCarefulonTeamSizeAutoExelevel2——Project/ProductBaseTRAPonAutomation:makeFullAutomation×ScriptmaintainonfrequentchangeSUT×RepeatTimeisfew(5times)×HighResultAnalyzeCostMustmakeBenefitInvestmentOtherwise:STOP!AutoExelevel2.5——ScriptSystemDesignEvolve•KeyWordDrivenInitialtoAutoAnalyzeEvolution•DataDrivenEvolvetolevel3IntegratewithT.M.S.IntegratewithCIAutomationExeFormalTestProcessDev.EfficiencyImproveAutoExelevel3——IntegratewithotherSystemMajorEvolveIntegratewithT.M.SIntegratewithCI※AutoTestDEVTestScriptManualTesterProduct/Project:SurroundingSub-System—NetworkSimlator—SignalingTesterCruiseControlHudsonAutoExelevel3——Integratewithothersystem(T.M.S.)SUTATSystemStimuliResponse3.ScriptCommand2.OrganizeScriptTester4.CheckAnalyzeTMS1.GetTes