What are Redis streams?
The big data boom has meant that there's more data at our fingertips than ever before, including real-time streaming data. But there's one problem: how can we efficiently capture and analyze this data, filtering through the noise to find crucial insights?
Redis streams are an exciting new feature that have been added to the Redis project since version 5.0. But what are Redis streams exactly, and how do Redis streams work? Below, we'll go over everything you want to know about Redis streams.
What are Redis streams? How do Redis streams work?
Redis streams are essentially abstract models of a log data structure. You can think of Redis streams as lists that only allow you to append data to the end of the structure. Each entry in the stream consists of a unique ID and a value. The ID is automatically generated and includes a timestamp, while the value is a hash.
One of the most important features of Redis streams is a concept known as the "consumer group," which was originally introduced by the Apache Kafka streaming platform. Consumer groups are groups of clients that can work together when consuming stream entries - for example, distributing many different messages across multiple clients.
What's the difference between Redis streams and pub/sub?
You might notice that Redis streams are similar to the publish/subscribe (i.e. "pub/sub") messaging pattern. In pub/sub architectures, two different entities can exchange messages without necessarily knowing about each other:
- Message senders (known as "publishers") send their messages to a particular topic or channel.
- Receivers (known as "subscribers") express interest in a particular topic by subscribing to it.
Pub/sub functionality has been present since Redis 2.0, so what's the difference between Redis streams and pub/sub? Both Redis streams and Redis pub/sub allow you to build applications that publish and consume messages. However, unlike Redis pub/sub, Redis streams can consume messages that are published even when the client application (i.e. the consumer) isn't running. Redis streams also allow you to consume messages starting only from a specific point - e.g. a certain timestamp in the log file.
To demonstrate the difference between Redis streams and pub/sub, below is an example of publishing messages in Redis:
PUBLISH #foo "hello world"
The message "hello world" is sent to the #foo channel. You can read and/or stop reading messages sent to this topic by subscribing and/or unsubscribing:
SUBSCRIBE #foo UNSUBSCRIBE #foo
Now let's contrast pub/sub with Redis streams. The code below publishes a message with Redis streams to the #foo channel:
XADD #foo * msg "hello world"
The XADD command is used both to create a new stream and to publish new messages to that stream. The * character denotes that Redis should generate a new unique ID for this message that is higher than any previous ID.
The code snippet below demonstrates how to read from an existing Redis stream:
XREAD BLOCK 1000 STREAMS #foo $
The XREAD command indicates a read operation, "BLOCK 1000" indicates that that the client will time out after 1000 milliseconds (1 second), and the $ character indicates that we will only read stream entries that arrive after the command is executed.
As you can see, compared with Redis pub/sub, Redis streams have both a different syntax and extra functionality that makes them an exciting addition to the Redis feature set.
Redis streams with Redisson
One of the greatest strengths of Redis is its compatibility with a wide range of programming languages. Although Redis is compatible with Java, however, it doesn't include support for familiar Java constructs such as Java objects and collections - at least, not right out of the box.
Third-party Redis Java clients such as Redisson help fill in the gaps for Java developers who want an easy learning curve with Redis. The good news is that Redisson includes support for Redis streams via the RStream interface. Below is an example of how to create and use an RStream with Redis and Redisson:
RStream<String, String> stream = redisson.getStream("test");
StreamMessageId sm = stream.add("0", "0");
stream.createGroup("testGroup");
StreamId id1 = stream.add("1", "1");
StreamId id2 = stream.add("2", "2");
Map<StreamId, Map<String, String>> group = stream.readGroup("testGroup", "consumer1");
long amount = stream.ack("testGroup", id1, id2));
The RStream interface also includes support for the asynchronous, reactive, and RxJava2 programming models, so that you can use the paradigm that fits your preferred development method.