Theano Deep Learning Tutorial

DeepLearningTutorialRelease0.1LISAlab,UniversityofMontrealMay29,2013CONTENTS1LICENSE12DeepLearningTutorials33GettingStarted53.1Download............................................53.2Datasets.............................................53.3Notation............................................73.4APrimeronSupervisedOptimizationforDeepLearning...................83.5Theano/PythonTips......................................144ClassifyingMNISTdigitsusingLogisticRegression174.1TheModel...........................................174.2DefiningaLossFunction....................................194.3CreatingaLogisticRegressionclass..............................194.4LearningtheModel......................................214.5Testingthemodel.......................................224.6PuttingitAllTogether.....................................235MultilayerPerceptron315.1TheModel...........................................315.2GoingfromlogisticregressiontoMLP............................325.3PuttingitAllTogether.....................................365.4TipsandTricksfortrainingMLPs...............................436ConvolutionalNeuralNetworks(LeNet)456.1Motivation...........................................456.2SparseConnectivity......................................466.3SharedWeights.........................................466.4DetailsandNotation......................................476.5TheConvOp..........................................486.6MaxPooling..........................................506.7TheFullModel:LeNet.....................................516.8PuttingitAllTogether.....................................526.9RunningtheCode.......................................546.10TipsandTricks.........................................55i7DenoisingAutoencoders(dA)577.1Autoencoders..........................................577.2DenoisingAutoencoders....................................617.3PuttingitAllTogether.....................................657.4RunningtheCode.......................................668StackedDenoisingAutoencoders(SdA)698.1StackedAutoencoders.....................................698.2Puttingitalltogether......................................748.3RunningtheCode.......................................758.4TipsandTricks.........................................759RestrictedBoltzmannMachines(RBM)779.1Energy-BasedModels(EBM).................................779.2RestrictedBoltzmannMachines(RBM)............................799.3SamplinginanRBM......................................809.4Implementation.........................................819.5Results.............................................9010DeepBeliefNetworks9310.1DeepBeliefNetworks.....................................9310.2JustifyingGreedy-LayerWisePre-Training..........................9410.3Implementation.........................................9510.4Puttingitalltogether......................................9910.5RunningtheCode.......................................10010.6TipsandTricks.........................................10111HybridMonte-CarloSampling10311.1Theory.............................................10311.2ImplementingHMCUsingTheano..............................10511.3TestingourSampler......................................11211.4References...........................................11412ModelingandgeneratingsequencesofpolyphonicmusicwiththeRNN-RBM11512.1TheRNN-RBM........................................11512.2Implementation.........................................11612.3Results.............................................12112.4Howtoimprovethiscode...................................12213Miscellaneous12513.1PlottingSamplesandFilters..................................12514References129Bibliography131Index133iiCHAPTERONELICENSECopyright(c)2008–2013,TheanoDevelopmentTeamAllrightsreserved.Redistributionanduseinsourceandbinaryforms,withorwithoutmodification,arepermittedprovidedthatthefollowingconditionsaremet:•Redistributionsofsourcecodemustretaintheabovecopyrightnotice,thislistofconditionsandthefollowingdisclaimer.•Redistributionsinbinaryformmustreproducetheabovecopyrightnotice,thislistofconditionsandthefollowingdisclaimerinthedocumentationand/orothermaterialsprovidedwiththedistribution.•NeitherthenameofTheanonorthenamesofitscontributorsmaybeusedtoendorseorpromoteproductsderivedfromthissoftwarewithoutspecificpriorwrittenpermission.THISSOFTWAREISPROVIDEDBYTHECOPYRIGHTHOLDERS‘’ASIS”ANDANYEXPRESSORIMPLIEDWARRANTIES,INCLUDING,BUTNOTLIMITEDTO,THEIMPLIEDWARRANTIESOFMERCHANTABILITYANDFITNESSFORAPARTICULARPURPOSEAREDISCLAIMED.INNOEVENTSHALLTHECOPYRIGHTHOLDERSBELIABLEFORANYDIRECT,INDIRECT,IN-CIDENTAL,SPECIAL,EXEMPLARY,ORCONSEQUENTIALDAMAGES(INCLUDING,BUTNOTLIMITEDTO,PROCUREMENTOFSUBSTITUTEGOODSORSERVICES;LOSSOFUSE,DATA,ORPROFITS;ORBUSINESSINTERRUPTION)HOWEVERCAUSEDANDONANYTHEORYOFLIA-BILITY,WHETHERINCONTRACT,STRICTLIABILITY,ORTORT(INCLUDINGNEGLIGENCEOROTHERWISE)ARISINGINANYWAYOUTOFTHEUSEOFTHISSOFTWARE,EVENIFADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGE.1DeepLearningTutorial,Release0.12Chapt