Author: Tan Yuanhong
Reviewers: Tiu Wee Han, Tejas Bhuwania
ZeroMQ is a high-performance asynchronous messaging library. It provides a message queue (MQ) for all processes communicating using the library, without a dedicated message broker - thus the zero in the name of ZeroMQ.
Asynchronous (or async, in short) communication means the two (or more than two) ends of the communication don't need to be occupied at the same time to send messages. You may consider making a phone-call synchronous and sending an email async. And To learn more about MQ, click heremessage queue is one of the async communication protocols that puts the messages in a queue, stores them temporarily until the recipient retrieves them.
Notice the word "protocol" used in the description above - communication protocol remains the same despite the actual medium of transport. It doesn't matter if your message is sent and received within the process between threads, or between processes, or between different physical machines across the network. It merely specifies how communicators are supposed to encode/send and decode/receive their messages.
Essentially: By adopting ZeroMQ in your concurrent application, you're less likely to have (or at least have less) performance bottleneck on inter-process/thread/machine communication, while still being able to write communication logic in familiar socket APIs.
Most operating systems provides system calls or equivalent support for message queues (e.g. POSIX mq, Win32 winmsg). However, the way they operate is far from high-level and making it directly interact with underlying OS APIs could hurt the scalability of your application as you may, in the future, find yourself in need of distributing the workload to different machines.
ZeroMQ looks like an embeddable networking library (read: easy to use) but acts as a concurrency framework (read: high performance). Using Python's built-in socket library as an example, if we are to write a simple TCP server in It's just a language of my choice. ZeroMQ provides bindings for most mainstream programming languages, which are listed herePython, it looks like this:
import socket
s = socket.socket() # Create the socket
s.bind(('', 12345)) # Bind socket to a port
s.listen(5) # Listen on the socket
# Communicate using the socket
while True:
c, addr = s.accept() # Receive input
c.send('Connected. Now closing connection.') # Send response
c.close()
Implementing the same functionality in ZeroMQ, it looks like this:
import time
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP) # Create the socket
socket.bind("tcp://*:12345") # Bind socket to a port
# Communicate using the socket
while True:
message = socket.recv() # Receive input
socket.send(b"Connected. Now closing connection.") # Send response
The similarity between the two is rather obvious - as long as you have experience with socket programming, using ZeroMQ to make communication between your concurrent processes possible and efficient should be no problem at all.
Unlike system calls, ZeroMQ supports various transports like in-process, inter-process, TCP, and multicast. And you don't need to change your code to switch underlying transport as ZeroMQ abstracted the details away, leaving you a reusable messaging layer. All you need to do is to update the definition (i.e. context.socket(...) and socket.bind(...)) of the socket and you're ready to scale your application from multi-core concurrency to a cluster of machines.
Unlike e.g. the socket module of Python standard librarynetworking libraries, ZeroMQ performs well enough to "be the fabric for clustered products". To give you a rough idea of how lightweight ZeroMQ is, the C++ source code for the ZeroMQ core engine libzmq is only 1.5 megabytes. Moreover, in the benchmark for ZeroMQ, the latency is measured in microseconds and the throughput is measured in million messages per second. Even CERN approved the performance of ZeroMQ in the setting of operating accelerators.
Developing software for concurrent operation is by no means entry-level content. Luckily, the creator(s) of ZeroMQ has written a guide explaining not only the basic use cases of ZeroMQ, but also the ideas behind their design choices and probably most importantly, why they feel necessary to develop a library like this.
If you simply want to get started soon and understand the details as you adopt the library, you can download the library in your favorite language here, then follow the corresponding examples written in the language of your choice.
In addition to the links mentioned above, here are the links used in the writing of this article: