and Validation of Large Scale Simulations IEEE Man

1ACriticalLookatDesign,Verication,andValidationofLargeScaleSimulationsIEEEManuscript#107243ToAppearIEEECSED.E.StevensonDepartmentofComputerScienceClemsonUniversityClemson,SC29634-1906NotetotheReader.IseesixconstituenciesinCSE:computer,mathematical,andphysicalsci-entists;engineers;andtechnicalandnon-technicalmanagers.Ihaveadoptedaconversationaltonetomakethisarticleaswidelyaccessibleaspossible.IhavealsoprovidedaBibliography.I.ThereIsADisconnectBetweenUs?Purpose:Toshowwearenotcommunicating.1.Introduction:ThereisadisconnectbetweenthemodelersandthemanagersTheJanuary-March,1998,issueofIEEECom-putationalScienceandEngineeringpresentedtwopapersontheDepartmentofEnergy’sAccel-eratedScienticComputingInitiativeorASCI.Thesetwopaperscomefromtworespectedau-thors.JohnGustafson[23]fromAmesLabatIowaStatepresentedadiscussionofthecom-putationalproblemswithASCI.AlexLarzelere[31],thepastDirectorforStrategicComputingandModelingforDepartmentofEnergyDefensePrograms,discussedsoftwareengineeringandmanagementaspectsofASCI.Thetwopaperscouldnothavebeenmorediametricallyopposedintheircontent.Hereinliestheproblem:man-agementthinksthatthe\techystuisamajorannoyanceandthetechnicalpeopledonotun-derstandthemanagerialneedtokeepcostsandschedulesundercontrol.Bothsidesspeakinthelanguageofmetrics:errorandconvergenceontheonehandandlinesofcode,testcoverage,anddollarsspentontheother.Butthemetricsmeasuredierentthingsfordierentpurposes.Discussionsaboutratesofconvergenceandlinesofcodetesteddonotleadtomeaningfulcom-munications.2.Itcanallgowrong.Thereareconse-quencestoASCIfailures.TheGustafson-Larzelerediscussionisnotjustanacademicdis-cussion.Therearerealconsequencestofailure.ImaginethisCNNnewsreport:[Jan28,2003.]Thisjustin.FromMakkah,SaudiArabia.AnFA-18ghtercarryingatac-ticalnuclearweaponhascrashedinthecenterofthisholiestofallIslamicsites.Althoughtheweaponwasnotarmed,theweapondetonated.USexpertscannotexplainhowthisexplosioncouldhaveoccurred.Thecitywaslledwithpeoplemakingthehajj.Becauseofthepilgrim-age,thereisnoestimatethenumberofdead.Indeed,wemayneverknow.TheentireArabworldisseethinginanger.CallsaboundforajihadagainsttheUnitedStates.....II.IsThereaProblem?Purpose:Togivespecicinstancesofscienticcalculationsgonewrong.3.Historyhorrorstoriesshowthatsoft-wareissuspect.CancurrentsimulationsbeheldtothehighstandardthatASCIdemands?Canweguaranteetheabovestoryfromneverbecomesareality?Issuchanaccidentpossi-ble?Believeit!Someevidenceforcomputer-mediateddisasters:OnOctober5,1960,theNorthAmericanDe-fenseCommandwentto99.9%alertbecausethemooncameup|thedesigners\forgotthatthemoonrisesoverthehorizonandwouldshowonradar.Forgiven.ExcepttwentyyearslaterandtwiceinonemonthNORADthreatenedtoshooteverythingduetocomputerglitches[25].ThesoftwareinApollo11hadthesignwrongonthegravitationalconstant:someprogrammermadegravityrepulsiveinsteadofattractive[36].GeminiVwas100milesocoursebecauseaprogrammerplayedfastandloosewithphysics[25],[36].ThePatriotmissilesmissedaScudoverDhahran,SaudiArabia,duringtheGulfWar.Oneproblemamongmanywasthattwodier-entbinaryversionsofthenumber0.1wereused.ThisledthePatriotstoimproperlycomputetheclosingspeed[42].OnMarch13,1985,aBlackhawkhelicopterfromFortBragg,NorthCarolina,crashed.Blackhawksare\y-by-wire,meaningthatcomputersmediateallcontrolactionsbythepi-lot.ThenalmomentsofChalkThreewereob-servedfromtwootherhelicopters.\...ChalkThreesuddenlypitchessharplyupwards,thenpitchessharplydownwards,plummetinguntilitimpactstheearthatanearverticalangle,upsidedownandbackwards[32].Thesearejustafewoftheproblemsintechnicalsystemsthathavesurfacedintheliterature.4.Werethingsalwaysbetterinthe\goodolddays?Twoshortexamplesofhowtechnologyhasfailedusinthepast.In1940,thefamousGallopingGertie,theTacomaNarrowsBridge,wasdestroyedduetoundampedoscillations.In1959,wingswerefallingoLockheedElec-tras.Thecausewaseventuallytrackedtoasixthharmonicsetupbytheoverly-powerfulengines.Theharmonicwastotallyunsuspected;there-fore,theproblemhadescapedexposurebytest-ing.IntheSectionVIII,IpresentaproblemfromWorldWarIIthatdriveshomethepointthatvalidationisanoldproblem.5.Canwemeetexpectations?Isitpos-siblethatAmericanmanagementpracticesandsoftwareengineeringcanguaranteethatmillionsoflinesofcodewillbewithoutsignicantde-fects?IfHattoniscorrectthatthenumberoffatalerrorsisproportionaltothelogofthenum-beroflinesofcode[25],thenamillionlinecodehasaroundtenfatalerrors.6.Otherareasofengineeringhavemetthecrisis.Wehavesomehintsfromread-ingDeming’sOutoftheCrisis[11].WecanseethattheconceptofqualityheproposeshasyettotakeholdofAmericanmanufacturing.Sincesoftwareengineeringusesthesesamemetaphors,wecanexpectsoftwareengineeringtonotadheretoDeming’sthoughts.Whatisevenmoredis-concertingisthatsoftwareisnowherenearaswell-craftedasanautomobile.7.Thetwochallenges:changingscienceandchangingculture.Sowhatisthepoint?Therearetwo:1.Scienticcomputingissubjecttoanynum-berofproblems:scienticandengineeringjudg-mentssubjecttomodication;numericaltech-niquessubjecttochangeduetonewalgorithms,environments,orprecisionrequireme