We wanted to make sure any given message got consumed once per logical consumer, even when multiple instances of this component are running.

In a clustered environment where you have multiple instances of your consumer component running at the same time, each instance would receive each message produced on the channel.

What Redis offers with Pub/Sub is a listener model, where each subscriber receives each messages when it is listening, but won’t receive them when not connected.

I already see you coming and asking why not using the Pub/Sub semantics supported out-of-the-box by Redis? The reason was two fold:

Each consumer continuously looks out for a new entry in its consumer-specific list and when one comes in, removes the entry, handles the message and passes on to the next message.

Iterates over the set of consumers registered on the queue, and pushes the message ID in a List for each of the registered consumers.

Saves the content of the message in a Redis key.

When a producers publishes a message on a queue, it:.

When each consumer starts up and gets ready to consume messages, it registers by adding itself to a Set representing all consumers registered on a queue.

The high-level strategy consists of the following:

In this post, I will go over the strategy we used with regards to Redis data structures and operations for handling the message publishing and consuming. Lightweight: Using Redis from any language/platform is extremely simple and provisioning it / maintaining it on a production server is dead easy.Persistence: Configured properly, we can ensure persistence of our queues to disk using one of the supported Redis persistence strategies.Concurrency: Because all operations in Redis are atomic, supporting concurrency without too much of a hassle is straightforward.Redis allowed for all of the implementation characteristics we were looking for, namely:.Note: although Redis requires everything you store in it to fit in memory, it supports persistence to disk. Expected volume was low, which meant that our data set could fit in memory.It was already part of our architecture and we had a simple inter-component messaging use case but didn’t want to introduce a new component in our architecture just for this.In a recent project, we ended up choosing Redis mostly because: The solution has advantages and some caveats, but if your problem allows you to live with the possible drawbacks, it’s a nice lightweight solution to a problem that is usually answered using some more traditional (and more complex) tools, like *MQs. One particular combination has recently allowed my team and I to implement a moderately (read: good enough) reliable message delivery mechanism for multiple consumers consuming messages at their own pace. Each data type has its own set of features to manipulate the data it contains in an atomic manner, making it an ideal tool for highly distributed system where concurrency is a potential issue.Ĭombining those features in creative ways allows for novel ways of doing “traditional” things differently.

Instead of just storing values as simple strings, it recognizes multiple specific data types such as Lists, Sets, Hashes (maps), Strings or Numbers. Redis is a high performance key-value datastore that differs from other key-value solutions in the way it handles values. UPDATE : I have open-sourced a Java implementation of the below principles called “ RedisQ“.

Message queue vs pub sub