Evaluating Federal Research Programs Research and

32EvaluatingFederalResearchPrograms:ResearchandtheGovernmentPerformanceandResultsActCommitteeonScience,Engineering,andPublicPolicy,NationalAcademyofSciences/NationalAcademyofEngineering/InstituteofMedicineExecutiveSummaryTheGovernmentPerformanceandResultsAct(GPRA),enactedin1993,focusesagencyandoversightattentionontheperformanceandresultsofgovernmentactivitiesbyrequiringthatallfederalagenciesmeasureandreportontheresultsoftheiractivitiesannually.Agenciesarerequiredtodevelopastrategicplanthatsetsgoalsandobjectivesforatleasta5-yearperiod,anannualperformanceplanthattranslatesthegoalsofthestrategicplanintoannualtargets,andanannualper-formancereportthatdemonstrateswhetherthetargetsaremet.TheCommitteeonScience,Engineering,andPublicPolicy(COSEPUP)oftheNationalAcademyofSciences,theNationalAcademyofEngineer-ing,andtheInstituteofMedicinehasaddressedtheissueofmeasuringandevaluatingresearchincompliancewiththerequirementsofGPRA.ReprintedwithpermissionfromEvaluatingFederalResearchPrograms:Re-searchandtheGovernmentPerformanceandResultsAct,pp.1–23(Execu-tiveSummaryandStatementoftheProblem).Copyright1999bytheNa-tionalAcademyofSciences.CourtesyoftheNationalAcademyPress,Wash-ington,DC.380NATIONALACADEMYOFSCIENCESCOSEPUPrecognizestheopportunitiesandchallengesthatGPRApresentsforagenciesthatinvestinresearch.GPRAoffersthoseagen-ciestheopportunitytocommunicatetopolicy-makersandthepublictherationaleforandresultsoftheirresearchprograms.Atthesametime,GPRApresentssubstantialchallengestotheagencies.Duringthecourseofthisstudy,COSEPUPheldseveralworkshops.Intheseworkshopsandinotherinputtothecommittee,wehaveheardtwodistinctandconflictingviewpointsonapproachestomeasuringbasicresearch.Oneisthatitshouldbepossibletomeasureresearch,includ-ingbasicresearch,annuallyandprovidequantitativemeasuresoftheusefuloutcomesofbothbasicandappliedresearch.Theotheristhat,giventhelong-rangenatureofbasicresearch,thereisnosensiblewaytorespondtotheGPRAannualmeasurementrequirementandthatthebestthatcanbedoneistoprovidemeasuresthatappeartorespondbutinfactareessentiallymeaningless,suchasalistofanagency’stop100discoveriesoftheprecedingyear.COSEPUP’sview,spelledoutinmoredetailinwhatfollows,isdif-ferentfromboththoseviewpoints.Inessence,ourreporttakestwostrongpositions.First,theusefuloutcomesofbasicresearchcannotbemeasureddirectlyonanannualbasis,becausetheusefulnessofnewbasicknowledgeisinherentlytoounpredictable;sotheusefulnessofbasicresearchmustbemeasuredbyhistoricalreviewsbasedonamuchlong-ertimeframe.Second,thatdoesnotmeanthattherearenomeaningfulmeasuresofperformanceofbasicresearchwhiletheresearchisinprogress;infact,thecommitteebelievesthattherearemeaningfulmea-suresofquality,relevance,andleadershipthataregoodpredictorsofeventualusefulness,thatthesemeasurescanbereportedregularly,andthattheyrepresentasoundwaytoensurethatthecountryisgettingagoodreturnonitsbasicresearchinvestment.Theproblemofreportingonappliedresearchismuchsimpler:itcon-sistsofsystematicallyapplyingmethodswidelyusedinindustryandinsomepartsofgovernment.Forexample,anappliedresearchprogramusuallyincludesaseriesofmilestonesthatshouldbeachievedbypartic-ulartimesandadescriptionoftheintendedfinaloutcomesandtheirsignif-icance.Periodicreportingcanindicateprogresstowardthosemilestones.Theremainderofthisexecutivesummaryprovidesamorein-depthdescriptionofCOSEPUP’sconclusionsandrecommendationsregardinghowtoevaluatefederalresearchprogramsrelativetoGPRA.Italsoaddressescoordinationamongfederalresearchprogramsandhuman-resourceissues.COSEPUPconcludesthatbothbasicresearchandap-EVALUATINGFEDERALRESEARCHPROGRAMS381pliedresearchprograms1canbemeaningfullyevaluatedonaregularbasis.Fortheappliedresearchprogramsofthemissionagencies,spe-cificpracticaloutcomescanbedocumentedandprogresstowardtheirachievementcanbemeasuredannually.Forexample,iftheDepartmentofEnergyadoptedthegoalofproducingcheapersolarenergy,itcouldmeasuretheresultsofresearchdirectedtowarddecreasingthecostofsolarcells;thisappliedresearchprojectwouldbeevaluatedannuallyagainstspecificmeasurablemilestones.However,thepracticaloutcomesofbasicresearchinscienceandengineeringcanseldombeidentifiedwhiletheresearchisinprogress.Basicresearchhasannualresultsthatcanbemeaningfullyevaluated,buttheseevaluationsoftendonotgiveevenahintofultimatepracticaloutcomes.Historytellsusunmistakablythatbyanymeasure,thebenefittotheUnitedStatesforleadershipinbasicresearchisextremelyhigh—livessaved,inventionsfostered,andjobsandwealthcreated.Historyalsoshowsushowoftenbasicresearchinscienceandengineeringleadstooutcomesthatwereunexpectedortookmanyyearsorevendecadestoemerge.COSEPUPstronglybelievesthatmeasuresofthepracticalout-comesofbasicresearchusuallymustberetrospectiveandhistoricalandthattheunpredictablenatureofpracticaloutcomesisaninherentandunalterablefeatureofbasicresearch.Forexample,pre-WorldWarIIbasicresearchonatomicstructurecontributedtotoday’sGlobalPosi-tioningSystem,anoutcomeofgreatpracticalandeconomicvalue,but,attemptstoevaluateayear’sworthofthatearlyresearcheveniftheydemonstratedhighqualityandworld