搜索
NeuralNetworkIntroduction1.ObjectivesAsyoureadthesewordsyouareusingacomplexbiologicalneuralnetwork.Youhaveahighlyinterconnectedsetofsome1011neuronstofacilitateyourreading,breathing,motionandthinking.Eachofyourbiologicalneurons，arichassemblyoftissueandchemistry,hasthecomplexity,ifnotthespeed,ofamicroprocessor.Someofyourneuralstructurewaswithyouatbirth.Otherpartshavebeenestablishedbyexperience.Scientistshaveonlyjustbeguntounderstandhowbiologicalneuralnetworksoperate.Itisgenerallyunderstoodthatallbiologicalneuralfunctions,includingmemory,arestoredintheneuronsandintheconnectionsbetweenthem.Learningisviewedastheestablishmentofnewconnectionsbetweenneuronsorthemodificationofexistingconnections.Thisleadstothefollowingquestion:Althoughwehaveonlyarudimentaryunderstandingofbiologicalneuralnetworks,isitpossibletoconstructasmallsetofsimpleartificial“neurons”andperhapstrainthemtoserveausefulfunction?Theansweris“yes.”Thisbook,then,isaboutartificialneuralnetworks.Theneuronsthatweconsiderherearenotbiological.Theyareextremelysimpleabstractionsofbiologicalneurons,realizedaselementsinaprogramorperhapsascircuitsmadeofsilicon.Networksoftheseartificialneuronsdonothaveafractionofthepowerofthehumanbrain,buttheycanbetrainedtoperformusefulfunctions.Thisbookisaboutsuchneurons,thenetworksthatcontainthemandtheirtraining.2.HistoryThehistoryofartificialneuralnetworksisfilledwithcolorful,creativeindividualsfrommanydifferentfields,manyofwhomstruggledfordecadestodevelopconceptsthatwenowtakeforgranted.Thishistoryhasbeendocumentedbyvariousauthors.OneparticularlyinterestingbookisNeurocomputing:FoundationsofResearchbyJohnAndersonandEdwardRosenfeld.Theyhavecollectedandeditedasetofsome43papersofspecialhistoricalinterest.Eachpaperisprecededbyanintroductionthatputsthepaperinhistoricalperspective.Historiesofsomeofthemainneuralnetworkcontributorsareincludedatthebeginningofvariouschaptersthroughoutthistextandwillnotberepeatedhere.However,itseemsappropriatetogiveabriefoverview,asampleofthemajordevelopments.Atleasttwoingredientsarenecessaryfortheadvancementofatechnology:conceptandimplementation.First,onemusthaveaconcept,awayofthinkingaboutatopic,someviewofitthatgivesclaritynottherebefore.Thismayinvolveasimpleidea,oritmaybemorespecificandincludeamathematicaldescription.Toillustratethispoint,considerthehistoryoftheheart.Itwasthoughttobe,atvarioustimes,thecenterofthesoulorasourceofheat.Inthe17thcenturymedicalpractitionersfinallybegantoviewtheheartasapump,andtheydesignedexperimentstostudyitspumpingaction.Theseexperimentsrevolutionizedourviewofthecirculatorysystem.Withoutthepumpconcept,anunderstandingoftheheartwasoutofgrasp.Conceptsandtheiraccompanyingmathematicsarenotsufficientforatechnologytomatureunlessthereissomewaytoimplementthesystem.Forinstance,themathematicsnecessaryforthereconstructionofimagesfromcomputer-aidedtopography(CAT)scanswasknownmanyyearsbeforetheavailabilityofhigh-speedcomputersandefficientalgorithmsfinallymadeitpracticaltoimplementausefulCATsystem.Thehistoryofneuralnetworkshasprogressedthroughbothconceptualinnovationsandimplementationdevelopments.Theseadvancements,however,seemtohaveoccurredinfitsandstartsratherthanbysteadyevolution.Someofthebackgroundworkforthefieldofneuralnetworksoccurredinthelate19thandearly20thcenturies.Thisconsistedprimarilyofinterdisciplinaryworkinphysics,psychologyandneurophysiologybysuchscientistsasHermannvonHelmholtz,ErnstMuchandIvanPavlov.Thisearlyworkemphasizedgeneraltheoriesoflearning,vision,conditioning,etc.,anddidnotincludespecificmathematicalmodelsofneuronoperation.Themodernviewofneuralnetworksbeganinthe1940swiththeworkofWarrenMcCullochandWalterPitts[McPi43],whoshowedthatnetworksofartificialneuronscould,inprinciple,computeanyarithmeticorlogicalfunction.Theirworkisoftenacknowledgedastheoriginoftheneuralnetworkfield.McCullochandPittswerefollowedbyDonaldHebb[Hebb49],whoproposedthatclassicalconditioning(asdiscoveredbyPavlov)ispresentbecauseofthepropertiesofindividualneurons.Heproposedamechanismforlearninginbiologicalneurons.Thefirstpracticalapplicationofartificialneuralnetworkscameinthelate1950s,withtheinventionoftheperceptionnetworkandassociatedlearningrulebyFrankRosenblatt[Rose58].Rosenblattandhiscolleaguesbuiltaperceptionnetworkanddemonstrateditsabilitytoperformpatternrecognition.Thisearlysuccessgeneratedagreatdealofinterestinneuralnetworkresearch.Unfortunately,itwaslatershownthatthebasicperceptionnetworkcouldsolveonlyalimitedclassofproblems.(SeeChapter4formoreonRosenblattandtheperceptionlearningrule.)Ataboutthesametime,BernardWidrowandTedHoff[WiHo60]introducedanewlearningalgorithmandusedittotrainadaptivelinearneuralnetworks,whichweresimilarinstructureandcapabilitytoRosenblatt’sperception.TheWidrowHofflearningruleisstillinusetoday.(SeeChapter10formoreonWidrow-Hofflearning.)Unfortunately,bothRosenblatt'sandWidrow'snetworkssufferedfromthesameinherentlimitations,whichwerewidelypublicizedinabookbyMarvinMinskyandSeymourPapert[MiPa69].RosenblattandWidrowwereawareoftheselimitationsandproposednewnetworksthatwouldoverc