Message Queues for web applications with STOMP

Just like human beings, machines need to communicate with each other – and they do this by passing messages around. The simplest form is to use a protocol like TCP and build some sort of API on top of it. But sometimes this just isn’t enough, especially if you want the connection to be reliable and you can’t afford to loose a single message.

Many people tend to implement it with the use of a database. They just save all messages to a table and let the other system fetch those messages and process them. But there are many pitfalls in such implementations and that’s why it’s considered an anti-pattern to use a database for these tasks. Better use a message queue system, these are particularly built for these types of tasks!

What is a queue?

At the very core, a queue is just a simple data structure that holds messages, where a message itself can hold any arbitrary data. The queue works with the first-in-first-out principle and will always keep the correct sort order of the messages. This and the fact that no message in a queue will ever get lost suits them perfectly for a wide array of use cases. These include bridgeing network gaps, running asynchronous tasks in background, reliably connect systems or even distributing work amongst several machines.

Modern message queue systems mostly consist of a broker, which is a server that manages connected clients and queues, receives messages, persists them (in memory or on the disk) and finally delivers them to other connected clients. Some common broker systems include ActiveMQ, RabbitMQ, MSMQ, Beanstalkd and Resque.

As there are many brokers out there, there are also many different messaging protocols:

• JMS (Java Message Service)
• AMQP (Advanced Message Queuing Protocol)
• STOMP (Streaming Text Oriented Messaging Protocol)
• OpenWire
• XMPP (Jabber)
• Memcached-based protocols

STOMP

As you can see, there is a great variety of messaging protocols, ranging from very simple Memcached-inspired protocols to more complex XML formats like XMPP. STOMP is somewhere in between this range. It’s lightweight yet very powerful and heavily inspired by HTTP, a protocol every web developer knows. There is a variety of client libraries for STOMP out there and every major programming language is supported. I’ll use it now in a simple example, together with the ActiveMQ Apollo broker system.

ActiveMQ Apollo

ActiveMQ Apollo is a rewrite of the ActiveMQ broker and will supersede it in the future. Both systems run on the Java Virtual Machine (JVM), but Apollo’s thread handling has been rewritten in Scala to allow for higher performance results. Also, the default persistence layer is now LevelDB, a file-based key-value store developed by Google.

A simple implementation example

Apollo also supports connections via Websockets, so I am going to develop an example with a client in a web application, talking to the broker via Websocket and a worker on the other side, that runs on NodeJS and connects itself to the broker through TCP. For demonstration purposes, the worker will just reverse the message, which therefor is just a simple string. In a real-world application, XML, JSON or similiar data format would be rather suited.

For the following example code, I suppose you downloaded and installed ActiveMQ Apollo and it runs listening on the standard ports, which are 61613 for TCP and 61623 for Websocket connections.

Libraries

There are already a number of vendor libraries for both Websocket- and TCP-NodeJS-based connections for JavaScript. I am going to use stomp-websocket and node-stomp-client here, but others will do quite as well.

Install both libraries:

git clone git://github.com/jmesnil/stomp-websocket
npm install stomp-client

Producer

Now that we have all required libraries, we first need to implement the producer, which is the Websocket-based client that initiates the asynchronous task on the message queue (“produces messages”). In this case, it’s just a simple HTML page with some embedded JavaScript:

The code is very simple: it instantiates the STOMP client and connects to the broker on port 61623 as the user admin with the password password. Once connected, it subscribes to the queue out and waits for incoming messages on that queue. Our worker will later send all the results to this queue and our client will display it in the output div.

The client sends a single message to the in-queue with the body ‘a sample message’ and some meta information in the header (job & clientId). Remember, this is heavily inspired by HTTP!

consumer

The counterpart to a producer is called a consumer, which in this case is the NodeJS-based worker. Again, this simple example consists of only a few lines of code:

var StompClient = require('stomp-client').StompClient;
var client = new StompClient('localhost', 61613, 'admin', 'password', '1.0');

client.connect(function(sessionId) {
    client.subscribe('/queue/in', function(body, headers) {
		console.log('processing message: ' + body);
		client.publish('/queue/out', body.split('').reverse().join(''));
    });
});

The consumer has a similiar behaviour of the producer: it connects to the server and subscribes to a queue, this time to the in queue. It sends messages to the out queue on the broker containing the reversed body of the incoming message.

Now, if you open the HTML page in your browser and the Apollo broker is running in the background, you will see the reversed string on your page:

egassem elpmas a

That alone may sound a little lousy, but it’s the basic example of what is possible using message queues. But it’s surely not the end here. Apollo and the STOMP protocol support many more features, like ACK’ing every single message to handle errors in producers and consumers, run brokers with queue mirrors for failover-scenarios and a REST management API. But all that is beyond the scope of this article.