Robust reinforcement learning control with static

ComputerScienceTechnicalReportRobustReinforcementLearningControlwithStaticandDynamicStabilityaR.MatthewKretchmar,PeterM.Young,CharlesW.Anderson,DouglasC.Hittle,MichaelL.Anderson,ChristopherC.DelneroColoradoStateUniversityJuly20,2000TechnicalReportCS-00-102aFromathesissubmittedtotheAcademicFacultyofColoradoStateUniversityinpartialfulfill-mentoftherequirementsforthedegreeofDoctorofPhilosophyinComputerScience.ThisworkwaspartiallysupportedbytheNationalScienceFoundationthroughgrantsCMS-9804757and9732986.ComputerScienceDepartmentColoradoStateUniversityFortCollins,CO80523-1873Phone:[phone]Fax:[phone]://R.MatthewKretchmar,PeterM.Young,CharlesW.Anderson,DouglasC.Hittle,MichaelL.Anderson,ChristopherC.DelneroColoradoStateUniversityJuly20,2000AbstractRobustcontroltheoryisusedtodesignstablecontrollersinthepresenceofuncertainties.Byreplacingnonlinearandtime-varyingaspectsofaneuralnetworkwithuncertainties,arobustreinforcementlearningprocedureresultsthatisguaranteedtoremainstableevenastheneuralnetworkisbeingtrained.Thebehaviorofthisprocedureisdemonstratedandanalyzedontwosimplecontroltasks.Foronetask,reinforcementlearningwithandwithoutrobustconstraintsresultsinthesamecontrolperformance,butatintermediatestagesthesystemwithoutrobustconstraintsgoesthroughaperiodofunstablebehaviorthatisavoidedwhentherobustconstraintsareincluded.FromathesissubmittedtotheAcademicFacultyofColoradoStateUniversityinpartialfulllmentoftherequirementsforthedegreeofDoctorofPhilosophyinComputerScience.ThisworkwaspartiallysupportedbytheNationalScienceFoundationthroughgrantsCMS-9804757and9732986.11IntroductionThedesignofacontrollerisbasedonamathematicalmodelthatcapturesasmuchaspossibleallthatisknownabouttheplanttobecontrolledandthatisrepresentableinthechosenmathematicalframework.Theobjectiveisnottodesignthebestcontrollerfortheplantmodel,butfortherealplant.Robustcontroltheoryachievesthisgoalbyincludinginthemodelasetofuncertainties.WhenspecifyingthemodelinaLinear-Time-Invariant(LTI)framework,thenominalmodelofthesystemisLTIand\uncertaintiesareaddedwithgainsthatareguaranteedtoboundthetruegainsofunknown,orknownandnonlinear,partsoftheplant.Robustcontroltechniquesareappliedtotheplantmodelaugmentedwithuncertaintiesandcandidatecontrollerstoanalyzethestabilityofthetruesystem.Thisisasignicantadvanceinpracticalcontrol,butdesigningacontrollerthatremainsstableinthepresenceofuncertaintieslimitstheaggressivenessoftheresultingcontroller,resultinginsuboptimalcontrolperformance.Inthisarticle,wedescribeanapproachforcombiningrobustcontroltechniqueswithareinforcementlearningalgorithmtoimprovetheperformanceofarobustcontrollerwhilemaintainingtheguaranteeofstability.Reinforcementlearningisaclassofalgorithmsforsolvingmulti-step,sequentialdecisionproblemsbyndingapolicyforchoosingsequencesofactionsthatoptimizethesumofsomeperformancecriterionovertime[27].Theyavoidtheunrealisticassumptionofknownstate-transitionprobabilitiesthatlimitsthepracticalityofdynamicprogrammingtechniques.Instead,reinforcementlearningalgorithmsadaptbyinteractingwiththeplantitself,takingeachstate,action,andnewstateobservationasasamplefromtheunknownstatetransitionprobabilitydistribution.Aframeworkmustbeestablishedwithenoughexibilitytoallowthereinforcementlearningcontrollertoadapttoagoodcontrolstrategy.Thisexibilityimpliesthattherearenumerousundesirablecontrolstrategiesalsoavailabletothelearningcontroller;theengineermustbewillingtoallowthecontrollertotemporarilyassumemanyofthesepoorercontrolstrategiesasitsearchesforthebetterones.However,manyoftheundesirablestrategiesmayproduceinstabilities.Thus,ourobjectivesfortheapproachdescribedherearetwofold.Themainobjectivethatmustalwaysbesatisedisstablebehavior.Thesecondobjectiveistoaddareinforcementlearningcomponenttothecontrollertooptimizethecontrollerbehavioronthetrueplant,whileneverviolatingthemainobjective.WhilethevastmajorityofcontrollersareLTIduetothetractablemathematicsandextensivebodyofLTIresearch,anon-LTIcontrollerisoftenabletoachievegreaterperformancethananLTIcontroller,becauseitisnotsaddledwiththelimitationsofLTI.Twoclassesofnon-LTIcontrollersareparticularlyusefulforcontrol:nonlinearcontrollersandadaptivecontrollers.However,nonlinearandadaptivecontrollersaredicult,andoftenimpossible,tostudyanalytically.Thus,theguaranteeofstablecontrolinherentinLTIdesignsissacricedfornon-LTIcontrollers.Neuralnetworksascontrollers,orneuro-controllers,constitutemuchoftherecentnon-LTIcontrolre-search.Becauseneuralnetworksarebothnonlinearandadaptive,theycanrealizesuperiorcontrolcomparedtoLTI.However,mostneuro-controllersarestaticinthattheyrespondonlytocurrentinput,sotheymaynotoeranyimprovementoverthedynamicnatureofLTIdesigns.Littleworkhasappearedondynamicneuro-controllers.Stabilityanalysisofneuro-controllershasbeenverylimited,whichgreatlylimitstheiruseinrealapplications.Thestabilityissueforsystemswithneuro-controllersencompassestwoaspects.Staticstabilityisachievedwhenthesystem