Asynchronous Python Logging Using MSMQ

Asynchronous logging in Python using MSMQ.

If you have a web application running in Python, there can be a need for the logging to not interfere with the performance. Default file based loggers can lead to a slow down because of the constand disk writes. An alternate solution that can be quite handy is logging into a message based logger, such as MSMQ. This post is built on my earlier post Sending MSMQ Messages Using Python, where I discuss how to setup, send and receive messages using MSMQ.

Here I will show how one can use the MSMQ to build a custom handler, that can be used with the logging module in python. Here is the MSMQHandler class:

    # customhandler.py
import logging

class MSMQHandler(logging.Handler):
    def __init__(self,queue_name,label_name,dest_computer=None):
        logging.Handler.__init__(self)
        import os
        import win32com.client
        self.queue_name = queue_name
        self.label_name = label_name
        self.computer_name = dest_computer if not dest_computer is  None\
            else os.getenv('COMPUTERNAME')
        qinfo=win32com.client.Dispatch("MSMQ.MSMQQueueInfo")
        qinfo.FormatName="direct=os:"+self.computer_name+\
            "\\PRIVATE$\\"+self.queue_name
        try:
            self.queue = qinfo.Open(2,0)
        except Exception as e:
            self.queue = None
            raise RuntimeError(str(e))

    def emit(self,record):
        import win32com.client
        if self.queue :
            msg=win32com.client.Dispatch("MSMQ.MSMQMessage")
            msg.Label=self.label_name
            msg.Body = self.format(record)
            msg.Send(self.queue)

    def close(self)        :
        self.acquire()
        try:
            if self.queue:
                self.queue.close()
        finally:
            self.release()

Once you have the handler in place, and setup a private MSMQ queue, say KaruthQueue, then you can incorporate it into your workflow as shown below:

# example.py
from customhandler import MSMQHandler
import logging

lgr = logging.getLogger("Test")
hnd = MSMQHandler("KaruthQueue","QPyLog") # here KaruthQueue is the private queue name
lgr.addHandler(hnd)
lgr.setLevel(logging.INFO)
lgr.info("Test Message")

And that completes the MSMQ logger using python. Now all your logs will be pushed to the KaruthQueue that we created. You can use the example shown here to read the messages in a seperate application and store any way you chose, files, database etc.

   python   programming  

---

Related Post

• QuantLib Python CookBook Is Almost Done
• Sending MSMQ Messages Using Python
• Optimizing Python Code: Numba vs Cython
• Using SQLAlchemy in Luigi Workflow Pipeline
• QuantLib Python Notebooks On Docker

---