Having fun with asyncio and zmq

Posted on June 26, 2016

A few weeks ago, I gave an adhoc session at the SaarCamp, trying to promote the incredible powerful combination of ZeroMQ and Pythons asyncio module. Sadly the session was a bit too much “ad hoc” and my final example did not worked out well. I promised to fix it and to provide working code. So let’s have a look.

Asyncio

There is quite some hype around Node.js, so I assume you are familiar with the underlying concepts. If not, check the Tornado documentation for a nice summary. Twisted and Tornado provided that kind of functionality for a long time, but they are harder to learn and deploy. But with Python3, there is good news: The asyncio module makes asynchronous i/o much more convenient to use. Here is a minimal asynchronous http server:

Such a server is able to handle thousends of connections with just one singlethreaded process. But only as long as the request handler is not blocking. So if there is some_heavy_work going on, we have to move it out of the request handling process.

ZeroMQ

My tool of choice for building distributed systems is ZeroMQ. It is not the simplest tool to learn, but it is worth the effort. So if you don’t know it yet, read the guide. To give you an impression how simple ZeroMQ can be (or rather: appear) check this code:

If you just have some kind of fire-and-forget messages, this might be all you need. You can even just start multiple workers, let them all connect to the same socket and ZeroMQ will distribute the work. In theory. In reality, this is the first wall you will hit while learning ZeroMQ. At least it was for me. The work is not distributed in a perfect way, so one worker might end up with much more messages than the other ones. The details about this are explained in detail in the ZeroMQ guide. You really have to read it, if you want to build distributed stuff.

In this article, I will use the Simple Pirate Pattern to build basic reliable queuing. In this pattern, the server is not pushing out the work. The workers send a message to the server, signaling that they are ready to do some work. The answer from the server will be the workload to be processed. The worker will send the result back to the server. That is also the implicit signal, that the worker is available again to do more work.

Asyncio + ZeroMQ

We have an asynchronous frontend web server and a ZeroMQ distribution pattern for the backend. Their exchange has to be like this:

Here is the code for the worker:

The tricky part is to close the gap between both worlds. The asyncio request has to be put to sleep in a proper way, so that he can be waked up by an incomming ZeroMQ message. Finding the correct handler is easy and just some book keeping: The request needs a unique id, which we pass to the worker. The worker will send this id back, together with the result. So we know to which handler a result relates to.

But how to put the request handler to sleep? Rephrasing the question leads to the answer. The handler has to wait for something that will happen in the future. And that’s exactly the use case of a Future. Here is the summary of the involved steps in detail:

Here is the code for the handler:

The very last step is to receives messages from the workers and to dispatch them to the correct futures. Here we go:

That’s it! Asyncio and ZeroMQ are now working together smoothly and we can use best of both worlds. Deploy everything in Docker containers and conquer the world!