新时代里咨询对我们的影响MassagingthePony

MassagingthePony:MessageQueuesandYouShawnRiderPBSEducationDjangoCon2010://://://://•RepeatingTaskscanreplaceCronjobs•KeepallyourcodeinyourprojectAvailableMQSolutionsTherearemanysolutionsoutthere.Tonameafewofthem:•RabbitMQ•AmazonSimpleQueueSystem•ApacheMQ•Gearman•OpenAMQ•Peafowl•Q4M•Starling•SunJavaSystemMessageQueueMessageQueueProtocols•AMQP:AdvancedMessageQueuingProtocol•JMS:JavaMessagingService•STOMP:StreamingTextOrientedMessagingProtocolCriteriaforBrokerSelectionThefollowingcriteriawereimportanttousinselectingasolution.Youmayhavesomedifferentneeds.•Handlingexceptionsandrecovery•Lowlevelofrequiredmaintenance•Easeofdeployment•Persistentqueuing•Communitysupport•Whatlanguageisitwrittenin?Howcompatibleisthatwithourcurrentsystems?•ClustersupportFailsafeMQBrokers•Mustbeabletosurviveaserverfailure.•UponrestartingtheMQBroker,processingshouldresumefromthelastun-processedmessagesinqueuesQueueDurabilityTheMessagesQueueBrokermustbeabletoreconstructthequeueswhenitisrestarted.(taskstobecompleted)mustpersistbetweenshutdownsandrestartsofMQServerOurChoice•Celery(aDjangotaskmanagementapp)•Carrot(Djangomiddleware)•PyAMQP(Pythonmodule)•RabbitMQ(MQBrokersoftware)TheProductionSystemWebappDatabaseWorkerServer(s)CeleryRabbitMQMQinDevelopmentEnvironsItisimportanttobeabletodevelopMessageQueuetasksinDjangowithoutbeingconcernedaboutrunningthebrokerprocess,queuesetup,persistenceetc.MQinDevelopmentEnvirons•Carrotcanbesetuptousedifferentqueuingbackendfordevelopmentpurposes•GhettoQprovidesdatabasebackendforcarrotwhereadatabaseservesasthequeuestorage•ThebrokerisasimpleDjangoapplicationthatmonitorsthequeueinDB•Inefficient,thereforeonlysuitablefordevelopmentenvironmentsCARROT_BACKEND=‘qhettoq.taproot.Database’EaseafutureMQimplementationIsolatefunctionalityintoreusablecomponents.RecycleExistingCodeDjango,HTTPRequests&MQ•MostfunctionalitiesprovidedthroughDjangoapplicationsaretightlycoupledwithHTTPrequestse.g.formdataprocessing•Inordertoasynchronouslyexecuteatask,Celery(anytaskmanagementplatform)mustserialize&storeparameters•HTTPrequestsareusuallylargeandnoteasilyserializable.e.g.WSGIrequestsThePickleableHTTPRequestObjectUsingPickleableHttpRequestifrequest.GET.get(‘download’):try:result=EnrollmentCSVExportTask.delay(PickleableHttpRequest(request,attributes=\[‘user’,‘admin_current_organization’]))MakingMessageQueueTasksItisfastandeasytoaddadecoratortospecificfunctionstocreateMessageQueuetasks.fromcelery.decoratorsimporttask@taskdefadd(x,y):returnx+yExplicitlyInheritfromCelery’sTaskObjectThenWhat?SomeMessageQueuetaskscancompletewithnonotificationrequired.Forothertasks:•Notifybyemail•UseanAJAXlistenertonotifyusers•SomeothermessagingInReview•ChooseyourMQsolutionwisely•Setuparobustsystem•Leverageyourexistingcode•Replacetroublespotsastheycomeup•Profit!MessageQueuesMakeLifeBetter