Benchmarks should be the first-class citizen​

In the world of software development, benchmarks are often treated as a second-class citizen. They're more of an addition to the codebase, rather than a crucial part of it, which should be the other way around, especially when it comes to the performance-critical systems or infrastructure tools.

Sometimes, the benchmarking results are nothing more than just a cherry-picking of the best-case scenarios, which are not representative of real-world usage. In such a case, they simply serve a sole purpose of either making the project look better than it is or how well it does outperform the competition, under the extremely optimized conditions when comparing with its counterparts.

Trying to reproduce the benchmarks is often a nightmare, as the environment setup is not documented, the code is unavailable, or the instructions are not clear enough. This makes it close to impossible to verify the results, which are then taken for granted.

Or even worse, the benchmarking tool might be so complex, that it's hard to understand how it works, and what are the assumptions behind it. ALl of these, does result in hard to extend or modify the existing benchmarks, which are not covering the particular use case you're interested in. It's just here to tell everyone that we do have benchmarks, but how we do it, and what they measure, is a mystery.

Which is why at Iggy, we've decided to make the benchmarks a first-class citizen.

Our iggy-bench tool, which is used to run the benchmarks and is part of the core open source repository (can be found under the bench directory), has come a long way and has been serving us well.

We use it to do quick performance checks, regression testing, and to see how the changes we introduce affect the performance. We run it on our localhost, as well as on the Virtual Machines in the cloud, to see how it behaves under a variety of environments.

Iggy benchmarking dashboard​

And today, we're proud to present benchmarks.iggy.rs - a benchmarking dashboard, which is available to everyone. It's a website where you can see how Iggy performs under the different conditions, and how it scales with the number of clients, messages, and topics.

This is our community-driven effort, where everyone can contribute, and add their own benchmarks. For all the information on how to run the benchmarks, render them on the dashboard, upload your results or contribute to the project, please check the iggy-bench-dashboard repository. In general, it's as simple as:

• Building the Iggy in the release mode with cargo build --release
• Starting your Iggy server with cargo r --bin iggy-server -r (feel free to adjust the configuration in server.toml or via environment variables)
• Running the iggy-bench tool with the desired parameters, e.g. cargo r --bin iggy-bench -r pinned-producer tcp
• Extending your benchmark with the output (HTML charts, JSON sampling etc.) cargo r --bin iggy-bench -r pinned-producer tcp output -o performance_results --identifier spetz
• Navigating to the specific benchmark directory to browse the charts and/or uploading them to the dashboard.
• And there's always help command e.g. pinned-producer --help to make your life easier :)

And this is just the beginning, as we plan to extend the dashboard, and add more benchmarks, which are covering the different use cases.

Our main goal is to make the benchmarking process (and its results) transparent, reproducible, and easy to understand. We want to make them a first-class citizen, and a crucial part of the Iggy project. We want to make them a tool, which will help us to improve the performance, and to make Iggy the best streaming server out there. We're looking forward to your feedback, and we hope you'll enjoy the benchmarks.

Towards the microsecond latency​

And as a cherry on top, we've recently managed to achieve the sub-millisecond write latency. This is a huge milestone for us, as it's a proof that Iggy can be used in low-latency applications, where speed is crucial. Lately, we've been experimenting a lot with rkyv - zero-copy deserialization framework, which has yielded some great results. Keep in mind that streaming the data within the range of microseconds latency depends on the several factors, such as message size, network conditions, or the hardware you're running on.

And the best part is that we're just getting started. We're looking forward to pushing the limits even further, and to see how far we can go. There's still tons of optimizations coming, including switching the runtime to the monoio which does support io_uring, and we've experienced superb results with this one on our experimental branch. Then, there's the whole concept of shared-nothing & thread-per-core design, and many more. Stay tuned!

We are thrilled to announce that Iggy has officially joined the Apache Incubator! This marks a major milestone in our journey to redefine message streaming — one that is blazingly fast, hyper-efficient, and built for the future. Since the very first day, Iggy was always meant to be a truly FOSS project — not just open-source in name, but deeply rooted in the values of transparency, collaboration, and community-driven innovation.

🚀 A Journey Two Years in the Making​

It’s been almost two years since the very first line of code was written, and we’ve come a long way since the initial release. What started as a bold idea has grown into a high-performance, cost-efficient message streaming solution with a thriving community of contributors, users, and advocates. We’ve received invaluable feedback, contributions, and support, and we’re more excited than ever to take Iggy to the next level.

🔥 Beyond Just Iggy – A Growing Ecosystem​

Iggy is no longer just about the core repository. Over time, we’ve built an entire ecosystem around it, including:

• ✅ SDKs for seamless developer integrations
• ✅ CLIs for powerful command-line control
• ✅ Web UIs for intuitive management
• ✅ And many more…

We’re already approaching 20 repositories under the Iggy organization (see them here), and as we enter the Apache Incubator, we are committed to ensuring that all of them follow ASF guidelines while maintaining our core values of speed, efficiency, and simplicity.

🔗 Why Apache?​

The Apache Software Foundation has nurtured some of the most transformative open-source projects in history — Hadoop, Spark, Lucene, Solr, Cassandra, Kafka, Flink, and Airflow — technologies that have shaped big data, search, and real-time processing. By joining Apache Incubator, we align with this legacy and gain access to a larger community, better governance, and long-term sustainability.

🔗 Dive into the Apache Incubator Details

• Official Proposal
• Discussion Thread
• Voting Results

📈 What’s Next?​

Our vision is clear: Iggy as a future Apache Top-Level Project (TLP). We already have ambitious ideas on how to improve both the project and the community around it:

• Scaling performance benchmarks to push the limits of ultra-low-latency streaming
• Expanding integrations with modern data infrastructure
• Building a vibrant developer ecosystem that makes message streaming frictionless

🦀 Codebase transition​

After the recent discussion, we plan to stick to the monorepo approach, under which, we will have a single repository for all the Iggy-related projects. This will make it easier for the contributors to navigate through the codebase, and to see how the changes in one project affect the others. This should also help us to keep the consistency across the projects, especially, once we release the Rust bindings to be used within the other languages SDKs.

All the repositories under iggy-rs will be eventually moved to the apache/iggy repository, and we will make sure that all the existing links and references are updated accordingly.

We'll also host our website under the iggy.apache.org domain, including the documentation, blog, and the upcoming benchmarks. We've also updated our social media handles to contain the Apache prefix.

There will be most likely some changes related to hosting the Docker images, as well as the other tools we're using, but we'll make sure to keep you updated on that.

💡 Join the Movement​

If you believe in a future where message streaming is lightning-fast, hyper-efficient, and accessible to all, we invite you to be part of this journey. Whether you're a developer, architect, or enterprise innovator, your contributions, feedback, and ideas will shape what comes next.

And this is just the beginning. Welcome to the era of Hyper-Efficient Message Streaming at Laser Speed.

Last but not least, we've got a new logo! Our lovely Italian Greyhound is now a part of the Apache family, and we're proud to have it as our mascot. As you can see, it's so fast, that even the light travelling through the optical fiber can't keep up with it :)

👉 Follow us, contribute, and help to build the future of the message streaming!

• Discord
• X (Twitter)
• Bluesky
• LinkedIn

Technology Radar​

Quite recently (a few days ago), Iggy has been listed on Technology Radar by Thoughtworks - a well-known technology consulting company.

If you're not familiar with the Technology Radar, it's essentially an opinionated set (updated twice a year and subscribed by the thousands of developers worldwide) of the tools, platforms, frameworks, techniques etc. which you may want to try out & explore in your IT projects. Everything is split into the different categories, depending on the maturity or popularity of the particular tool.

As you can see, we were put right into the assess bucket (next to such renowned solutions such as e.g. FoundationDB) - being the projects which are worth exploring & understanding how they might affect your enterprise. Frankly speaking, we weren't expecting this at all, and from our perspective, it's quite of an accomplishment.

Besides gaining an additional amount of trust & recognition, it has led us to another conclusion - someone out there we don't know yet about (maybe even one of their customers) is using/experimenting with Iggy :)

And if you are (or will be) one of such persons, please hop onto our Discord and share your invaluable feedback with us!

Now, given the recent publication and increased activity within our OSS community building the core streaming server & SDKs in multiple programming languages, it's worth mentioning what are the current goals for Iggy.

Current goals​

Replication​

Without a doubt, being able to run your infrastructure (which processes & stores the data) as a cluster, gives much more confidence and greatly impacts the overall reliability.

We've started experimenting with the replication over half a year ago already by implementing the basic, Raft based consensus algorithm for the simple message streaming server.

At the same time, we were researching the other possible solutions, after we've finally decided to move on with Viewstamped Replication (in its revisited form), which was successfully used by e.g. TigerBeetle.

Long story short - the deterministic leader election, allows us to go for ring topology and chain replication of our data - it's excellent for high throughput, which is very important for us.

Moreover, VSR can be run completely in memory, providing us an opportunity to work independently both on the consensus and the storage and how to link these two together, to form a bulletproof storage fault model.

Below is our very first draft for the initial implementation of VSR.

S3 storage​

A few months ago, we did implement an optional archiver for the server state log & streaming data (messages etc.) which supports any S3 compatible storage (just pick up your favorite cloud provider). The configuration is as simple as this example:

[data_maintenance.archiver]
# Enables or disables the archiver process.
enabled = true

# Kind of archiver to use. Available options: "disk", "s3".
kind = "s3"

[data_maintenance.archiver.disk]
# Path for storing the archived data on disk.
path = "local_data/archive"

[data_maintenance.archiver.s3]
# Access key ID for the S3 bucket.
key_id = "123"

# Secret access key for the S3 bucket
key_secret = "secret"

# Name of the S3 bucket.
bucket = "iggy"

# Endpoint of the S3 region.
endpoint = "http://localhost:9000"

# Region of the S3 bucket.
region = "eu-west-1"

# Temporary directory for storing the data before uploading to S3.
tmp_upload_dir = "local_data/s3_tmp"

By making use of S3, you could almost infinitely (and very cheaply) store your data - for the need of additional backups, being compliant with law regulations etc. However, there's one catch - in order to read the data stored with S3, you'd need to download it from the cloud and restart your server. And this is where things will change in the future - we're planning to implement a dedicated S3 storage, for both, writing and reading the data in real-time if needed. You could think of the following analogy to the different kinds of cache storages in your PC.

• L1 - data available directly from the server RAM (super fast writes/reads)
• L2 - data stored on your servers disks (still very, very fast with NVME SSD gen4 or 5)
• L3 - S3 storage, still fast for the typical use-cases which do not require a very stable, microsecond level latencies

Each of these storage layers could be optionally enabled or disabled. You can already decide if and how much memory to use for caching the messages. With S3 tiered storage in place, you could e.g. treat your server's SSD as a sort of ring buffer for keeping the most recent data (easily millions or billions of messages, depending on their size) and only fetch the ones from S3, when you need something very old.

Or, you could just ignore your server's RAM & SSD, and do all the writes and reads directly on S3, and still remain blazingly fast (just like Quickwit).

OpenTelemetry​

Speaking of the Quickwit, we've also implemented a support for OpenTelemetry logs & traces for the server. Since our SDK already uses the logging & tracing libraries, we thought that adding such a feature on the server, could help you gain even better, real-time observability into what's happening under the hood.

# OpenTelemetry configuration
[telemetry]
# Enables or disables telemetry.
enabled = false
# Service name for telemetry.
service_name = "iggy"

# OpenTelemetry logs configuration
[telemetry.logs]
# Transport for sending logs. Options: "grpc", "http".
transport = "grpc"
# Endpoint for sending logs.
endpoint = "http://localhost:7281/v1/logs"

# OpenTelemetry traces configuration
[telemetry.traces]
# Transport for sending traces. Options: "grpc", "http".
transport = "grpc"
# Endpoint for sending traces.
endpoint = "http://localhost:7281/v1/traces"

And just like with S3 storage, it's merely a beginning - one of the members on our Discord had already thought of extending this implementation by propagating the trace context (via existing message headers metadata) between the clients & server in order to get full understanding of the distributed systems and its dependencies, which could be further visualized by tools like Zipkin or Jaeger.

Optimizations​

Improved messages batching, keeping the indexes & time indexes in a single file, making use of mmap or directIO for the data storage processing, rkyv for zero-copy (de)serialization, keeping open the file descriptors and lots of other minor improvements - all these low hanging fruits (or at least some of them), will hopefully build up to making Iggy even more performant & resource effective than it already is.

To start the Iggy server, you just need to wait for a few milliseconds, and the RAM consumption is within a range ~20 MB, which is already over an order of magnitude lower than when compared to Kafka.

io_uring​

This will certainly require to have its own blog post, as there's so much to talk about. We did experiment with Monoio (which, in its basic form without additonal enhancements allowed us to reach over 15 GB/s reads when compared to 10-12 GB/s for Tokio that we currently use), we also might experiment with Glommio, yet, most likely, we might build our own io_uring backend to fully utilize all its features.

Yes, at this point you might call us crazy (io_uring won't happen before we release the first version of the VSR clustering anyway), but if you want to tick all the possible boxes, it's hard to find a generic framework that will meet your demands, especially when mixed altogether with VSR clustering, thread-per-core & shared-nothing design (if will turn out to be suitable), zero-copy deserialization libraries and other things we might even not be aware of yet.

To innovate, one must experiment, and although we do all these things in our spare time, it's been an exciting journey so far (and lots of experience gained in the meantime) for all of our team members building something fresh, from the very ground up, and regardless of the final outcome, we already know it was all worth it :)

Throwback​

It's been a little over a year, since the Iggy.rs was born. The initial idea of building a side project (as a way of studying Rust) — an infrastructure for the message streaming (think of Kafka, RedPanda, Aeron etc.) — eventually turned out to be something much bigger that I could've ever imagined. In the previous post (from almost half a year ago), I did describe what's Iggy.rs all about, how it started, what's the ecosystem around it, what our goals are etc.

This particular article turned out to be a sort of catalyst, as it received a really nice traction on Reddit, and was also mentioned on the main site of Hacker News, which I do believe were the two main reasons for the growing interest & community since then. At this point, I'd like to thank you all very much for such a kind feedback — honestly, during the very first weeks of 2024, there were so many things happening on our Discord, that with the rest of the team, we sometimes had a feeling as if we were providing enterprise premium support — really cool stuff!

And although it may seem as if the project development has recently slowed down a bit, I'd say it's quite the opposite — let me quickly summary, what we've achieved so far during the last few months and what we're focusing on now, as the future looks bright :)

Community​

First and foremost, if it weren't for the community, we wouldn't have seen such an enormous growth of the Iggy's ecosystem — we've received the dozens of pull requests and there's ~250 members on our Discord. Whether we talk about bug fixes, improvements, new features, or just sharing the experiences and discussing potential ideas — it's all equally important.

And it's been even more than that — we've seen our community members take on building the new SDKs in their favorite programming languages, fully on their own. Today, you can find the following list of supported SDKs for Iggy.rs — some of them could be lagging behind, but it's expected, as the project is still evolving, and it's not an easy task, to come up with a great development experience from the very beginning.

• Rust
• C#
• C++
• Node (TS)
• Go
• Python
• Java
• Elixir

Changelog​

Adding the brand new SDKs wasn't the only great thing that has happened during last few months. We've also made quite a lot of improvements for the streaming server itself:

• Increased the streaming server throughput by over 30% for both writes and reads
• Added messages compression for client-side & server-side supporting the different algorithms
• Implemented a new way of message batching with an additional tooling for data migration
• Fixed the possible server-side deadlock that could happen for a specific configuration
• Fixed issues with possible memory leaks when storing too many indices in memory
• Rebuilt our custom benchmarking tool
• Improved TCP connection handling
• Constantly upgrading our CI/CD with lots of testing, different runtimes, artifacts, crates and Docker images releases
• Refactored the existing Rust client SDK to follow the new conventions (without the breaking changes to the previous one)

And at the same time, we've been experimenting a lot with some fancy stuff, which you can read about in the last paragraphs :)

Tooling​

The core message streaming server and multiple SDKs might sound as the most important parts of the whole ecosystem, but let's not forget about the management tools. How to quickly connect to the the server, create new topics, validate if the messages are being sent correctly, change the user permissions or check the node statistics?

This is where our CLI and Web UI come in handy. If you're a fan of working with the terminal and used to the great developer experience, you'll find our CLI a joy to work with.

On the other hand, if you prefer a graphical interface accessible via your browser, Web UI has got you covered. What's even more impressive, is that both of these tools have been developed by the single developers.

Last but not least, in order to run the benchmarks, we have our own bench available as a part of the core repository — you can easily configure the number of producers, consumers, streams, etc. and get an overview of the possible streaming performance on your machine.

Early adopters​

Overall, coding and implementing new features is one side of the story, the other is making an actual use of it. You might have the most sophisticated/performant/reliable (you name it) tooling out there, however, if no one is using it or at least experimenting with it, how could you possibly know whether it's even worth an effort to continue with the further development of the project? Well, I truly wish I had an easy answer how to find users willing to play with your new shiny toy.

In our case, I do believe, that it was a mix of two things — a limited amount of such tooling in the Rust ecosystem (so that the language enthusiasts could to try out something fresh), as well as a much more lightweight and (hopefully) performant message streaming infrastructure than some of the well-established solutions.

I'm fully aware that it's a bold claim, and running the synthetic benchmarks is not a viable proof (e.g. on my 7950X, I was able to hit 3GB/s writes and up to 10 GB/s reads with some additional caching enabled), yet, most of our early adopters were very happy with their results e.g. outnumbering Kafka while utilizing much less memory. For example, Marvin wrote:

20 million msg/sec via tcp is pretty nuts and already blows several commercial systems out of the water.

And he's not the only one who found Iggy.rs to be the right tool for his needs. For example, a few days ago, one of the users on our Discord said that thanks to Iggy he was able to achieve 2ms latency when compared to 300ms with Kafka. Again, just to make it clear, I'm not saying that we're better than X or Y — I'm simply stating that for some specific usages, we might be a better choice than X or Y.

Clustering & replication​

One of the most frequent questions we receive is whether we plan to incorporate some sort of data replication feature. Without any doubts, especially when considering the general system resiliency and reliability, being able to spin up a cluster of the particular piece of infrastructure (database/messaging/streaming/logging etc.) is quite often a critical feature.

And yes, we will certainly implement clustering in Iggy — as a matter of fact, we've already built its basic version in the sandbox repository. In order to achieve that, we've decided to (at least for now) stick to the Raft consensus algorithm. However, adding the data replication feature to the core Iggy project will require a new way of storing the server metadata (most likely in a way of event-sourced messages, to play nicely with the replication between the nodes) and one more "tiny" thing regarding the overall I/O.

io_uring & thread-per-core + share-nothing​

What is a message streaming server, besides some more-or-less complicated logic regarding topics, partitioning, message ordering, consumer groups and a few more features? In its very core, it's mostly I/O (disk + networking) — the more efficient you can make it, the greater it will be. Of course, there's a major difference between throughput and latency, especially when talking about the so-called tail latency (p99 and more). Wouldn't it be great if we could have a very high throughput and very stable/predictable (and low) latency at the same time? This is exactly what we're currently trying to achieve. Originally, we've started with the most popular Tokio runtime, which uses the work-stealing approach, and it's actually quite impressive (based on the benchmarks and the experiences shared by our early adopters).

However, due to the nature of tasks being shared across the different threads, you can't simply avoid lots of context switches and data being shared & synchronized across these threads (therefore even more context switches will occur). While it's probably not an issue for like 90% or maybe even 99% use-cases, there might be some (financial systems and similar), where an unpredictable tail latency is a total no-go.

Certainly, there are already existing solutions dealing with such challenges, such as Aeron, but we do believe, that we can make Iggy something much easier to use — one that can handle the typical workloads, as well as the very demanding ones, without the need of getting a PhD in specific tooling :)

We've decided to experiment with io_uring to maximize the I/O performance (and at the same time vastly reduce the need of context switches), and at the same time utilize thread-per-core architecture, where each thread is pinned to the CPU core, thus keeping the data locally, without the need of sharing it with the other threads (share-nothing). In order to achieve this, we've picked up monoio runtime, and have already managed (as a starting point for future integration) to fully rewrite existing Tokio runtime into monoio on this branch.

And just recently, we've established yet another sandbox repository to tackle the different challenges before deciding on the best solution possible and merging these changes into core Iggy streaming server.

So far, we've got a very simple prototype in place, but there's still lots to be done, especially when thinking of:

• How to evenly split partitions (the unit of data parallelism) between multiple cores?
• How to efficiently rewrite existing server metadata heavily relying on synchronized data with Arc<> and RwLock<>?
• Should we load into memory the same server metadata across all the threads (separately from each other) and notify all of them when something changes?
• Should there be single or multiple threads handling the incoming TCP connections?
• When Thread #1 receives the request, which has to access the partition from Thread #2, should we use an async two-way channel (remember, no explicit locking and data synchronization between the threads) or maybe just send the descriptor using a one-way channel to the second thread to complete the request?
• What and when could become a bottleneck in such architecture?

This is just the tip of an iceberg, and we've already started studying some of the existing solutions out there, including Seastar framework. As you can see, this part has to be done before the clustering, as it involves lots of changes not only regarding the disk I/O, but also networking I/O.

Production readiness​

Is it ready for production deployment? When you look at the versioning, Rust SDK is at 0.4.* and the server is currently at 0.2.*, which may look like a very long way from v1.0.

As mentioned before, some of our users already experiment with Iggy — and simply because of this, we haven't really introduced any significant breaking changes (except one regarding data compression, which was handled by the provided data migration feature).

We can't guarantee that it will always be like this, but at least for now, we do not see anything that would dramatically impact the existing solution. One of such things could be the redesigned storage, I/O, and clustering feature (as described in the previous paragraph), but even then, we'll do our best to make it as seamless upgrade as possible — and once we achieve that, it means, we're getting very close to the version 1.0.

For the time being, if you're fine with a single-node solution that delivers a really good message streaming performance, give Iggy a try, or at least run the benchmarks to see what are its possibilities and please share your results and thoughts on our Discord — your opinion is really important to us and we respect it no matter what.

It's been a very productive year for our core team (we've been and still are doing this in our free time), and once again, huge thanks to all our supporters and contributors!

We've got the fundamentals right, now it's high time to make Iggy blazingly fast!

Origins​

Over half a year ago (in April, to be exact), I eventually decided to learn Rust for good. My previous attempt during the 2022 AoC had failed rather quickly, after a few days of completing the exercises - I finally realized that I needed a real project to work on. For the last few years, I've been dealing with the different kinds of distributed systems (mostly using C#), including the typical microservices architecture or Web3. Regardless of their nature, some sort of the messaging between the independent components was always required. I had a chance to use the tools such as RabbitMQ, ZeroMQ, Kafka or Aeron (just to name a few), as well as implementing the low-level peer-to-peer communication protocols.

After a few months of trying to figure out (or just staying in the limbo I guess), what would be the best project to work on, I decided to build the message streaming platform (keep in mind that streaming is not the same as regular message broker). The other reason (besides getting to know Rust) was to truly understand the internals of the messaging systems and the trade-offs that were made by their developers - some of them being the sole implication of the theory of distributed systems (ordering, consistency, partitioning etc.), while others the result of the implementation details (programming language, OS, hardware and so on).

And this is how the Iggy.rs was born. The name is an abbreviation of the Italian Greyhound (yes, I own two of them), small yet extremely fast dogs, the best in their class.

Therefore, what I want, or actually what we want (since there's a few of us working on it already) for Iggy.rs to be - the best message streaming platform in its class. Lightweight in terms of the resource consumption, fast (and predictable) when it comes to the throughput and latency, and easy to use when speaking of its API, SDK and configuration of the project.

Project​

At the very beginning, Iggy had rather limited functionality, and everything was handled using the QUIC protocol based on Quinn library. You could connect multiple applications into the server, and start exchanging the messages between them, simply by appending the data to the stream (from the producer perspective), and fetching the records on the consumer side, by providing an offset (numeric value specifying from which element in the stream, you'd like to query the data) - that's pretty much the very basics of how the message streaming platform works in terms of the underlying infrastructure.

After spending a few weeks on building the initial version, and then another few weeks on rewriting its core part (yes, prototyping + validation repeated in a continuous loop worked quite well), I managed to implement the persistent streaming server being capable of parallel writes/reads to/from independent streams supporting many distinct apps connected into it. Simply put, one could easily have many applications, and even thousands of the streams (depending on how do you decide to split your data between them e.g. one stream for user related events, another one for the payments events etc.) and start producing & consuming the messages without interfering to each other.

On top of this, the support for TCP and HTTP protocols have been added. Under the hood, the typical architecture of streams, consisting of the topics being split into the partitions, which eventually operate on a raw file data using so-called segments has been implemented as well.

It was one of the "aha" moments, when reimplementing the parallel access to the data with the usage of underlying synchronization mechanism (RwLock etc.), optimized data structures e.g. for dealing with bytes, along with the Tokio work stealing approach, yielded the great improvements for the overall throughput.

I do believe, that somewhere at this point I had realized, that Iggy might actually become something useful - not just a toy project, to be abandoned after reaching its initial goal (which was sort of already achieved).

let polled_messages = client.poll_messages(&PollMessages {
  stream_id: Identifier::numeric(1)?,
  topic_id: Identifier::named("orders")?,
  consumer: Consumer::group(Identifier::named("payments")?),
  partition_id: None,
  strategy: PollingStrategy::offset(0),
  count: 10,
  auto_commit: true,
}).await?;

After running some benchmarks (yes, we have a dedicated app for the benchmarking purposes) and seeing the promising numbers (range of 2-6 GB/s for both, writes & reads when processing millions of messages), I eventually decided to give it a long-term shot. Being fully aware that there's still lots to be done (speaking of many months, or even years), I couldn't be more happy to find out that there's also someone else out there, who would like to contribute to the project and become a part of the team.

Team​

At the time of writing this post, Iggy consists of around 10 members contributing to its different parts. Some of us do work on the core streaming server, while the other ones are focused on SDKs for the different programming languages or tooling such as Web UI or CLI - all these projects are equally important, as they add up to the overall ecosystem. But how do you actually gather a team of open source contributors, who are willing to spend their free time working on it?

Well, I wish I had an answer to that question - honestly, in case of Iggy I wasn't actually looking for anyone, as I didn't think this could be an interesting project to work on (except for myself). Then how did that happen anyway? There were only 2 things in common - all the people that joined the project were part of the same Discord communities, yet more importantly they all shared the passion for programming, and I'm not talking about Rust language specifically. From junior to senior, from embedded to front-end developers - regardless of the years of experience and current occupation, everyone has found a way to implement something meaningful.

For example, when I asked one guy what was the reason behind building an SDK in Go, the reply was the need of playing with and learning a new language. Why C# SDK? Well, the other guy wanted to dive more into the low-level concepts and decided to squeeze out great performance from the managed runtime. Why build Web UI in Svelte? At work, I mostly use React, and I wanted to learn a new framework - another member said.

My point is - as long as you believe in what you're building, and you're consistent about it (it was one of the main reasons why I've been contributing to Iggy every day since its inception, and still doing so), there's a chance that someone out there will notice it and happily join you in your efforts. Lead by example, or whatever you call it.

At the same time, we've started receiving the first, external contributions from all around the world - whether talking about simpler tasks, or the more sophisticated ones, requiring significant amount of time being spent on both, the implementation and the discussions to eventually deliver the code.

It gave us even more confidence, that there are other people (outside our internal bubble), who find this project to be interesting and worth spending their time. And without all these amazing contributors, it'd be much harder (or even impossible) to deliver so many features.

Features​

At first, let me just point out some of the properties and features that are part of the core streaming server:

• Highly performant, persistent append-only log for the message streaming
• Very high throughput for both writes and reads
• Low latency and predictable resource usage thanks to the Rust compiled language (no GC)
• Users authentication and authorization with granular permissions and PAT (Personal Access Tokens)
• Support for multiple streams, topics and partitions
• Support for multiple transport protocols (QUIC, TCP, HTTP)
• Fully operational RESTful API which can be optionally enabled
• Available client SDK in multiple languages
• Works directly with the binary data (lack of enforced schema and serialization/deserialization)
• Configurable server features (e.g. caching, segment size, data flush interval, transport protocols etc.)
• Possibility of storing the consumer offsets on the server
• Multiple ways of polling the messages:
  • By offset (using the indexes)
  • By timestamp (using the time indexes)
  • First/Last N messages
  • Next N messages for the specific consumer
• Possibility of auto committing the offset (e.g. to achieve at-most-once delivery)
• Consumer groups providing the message ordering and horizontal scaling across the connected clients
• Message expiry with auto deletion based on the configurable retention policy
• Additional features such as server side message deduplication
• TLS support for all transport protocols (TCP, QUIC, HTTPS)
• Optional server-side as well as client-side data encryption using AES-256-GCM
• Optional metadata support in the form of message headers
• Built-in CLI to manage the streaming server
• Built-in benchmarking app to test the performance
• Single binary deployment (no external dependencies)
• Running as a single node (no cluster support yet)

And as already mentioned, we've been working on SDKs for the multiple programming languages:

• Rust
• C#
• Go
• Node
• Python
• Java

Please keep in mind, though, that some of them e.g. for Rust or C# are more up to date with the recent server changes, while the other ones might still need to do some catching up with the latest features. However, given the amount of available methods on the server's API and the underlying TCP/UDP stack with custom serialization to be implemented from the scratch (except for HTTP transport, that's the easier one), I'd say we're doing quite ok, and I can't stress enough how grateful I am to all the contributors for their huge amount of work!

But wait, there's even more - what would be a message streaming platform without some additional tooling for managing it? We've also been developing the CLI.

As well as modern Web UI to make it happen :)

Last but not least, we've got a fully-featured CI/CD pipeline responsible for running all the checks and tests on multiple platforms, and finally producing the release artifacts and Docker images.

At first glance, it might look like there's plenty of features already in place, but for anyone who has ever worked with the message streaming infrastructure before, that might be just a tip of an iceberg, thus let's discuss the roadmap.

Roadmap​

After gaining some traction a few months ago (mostly due to landing on the GitHub trending page in July), we've talked to some users potentially interested in making Iggy part of their infrastructure (there's even one company using it already), and discussed what features would be a good addition to the current stack.

Considering what's already there, being worked on or planned for the future releases, such as interactive CLI, modern Web UI, optional data compression and archivization, plugin support or multiple SDKs, there are at least three additonal challenges to overcome:

Clustering - the possibility of having a highly available and fault tolerant distributed message streaming platform in a production environment, is typically one of the most important aspects when considering the particular tool. While it wouldn't be too difficult to implement the extension (think of a simple proxy/load balancer), allowing to monitor and deliver the data either to the primary or secondary replica (treated as a fallback server) and switch between them when one of the nodes goes down, such a solution would still result in SPOF and wouldn't really scale. Instead, we've started experimenting with Raft consensus mechanism (de facto the industry standard) in a dedicated repository, which should allow us in delivering the truly fault tolerant, distributed infrastructure with an additional data replication at the partition level (so-called unit of parallelization).

Low-level I/O - although the current results (based on the benchmarking tool measuring the throughput etc.) are satisfying, we strongly believe that there's still (potentially a huge) room for improvement. We're planning to use io_uring for all I/O operations (disk or network related). The brand new, completion based API (available in the recent Linux kernels) shows a significant boost when compared to the existing solutions such as epoll or kqueue - at the end of the day, the streaming server at its core is all about writing & reading data to/from disk and sending it via the network buffer. We've decided to give a try monoio runtime, as it seems to be the most performant one. Going further, we'd like to incorporate techniques such as zero-copy, kernel bypass and all the other goodies e.g. from DPDK or other relevant frameworks.

Thread-per-core - in order to avoid the rather costly context switches due to the usage of synchronization mechanism when accessing the data from the different threads (e.g. via Tokio's work stealing mechanism), we're planning to explore (or actually, already doing it, in the previously mentioned repository for clustering sandbox) thread-per-core architecture, once again, delivered as part of monoio runtime. The overall idea can be described in two words - share nothing (or as little as possible). For example, the streams could be tied to the particular CPU cores, resulting in no additional overhead (via Mutexes, RwLocks etc.) when writing or reading the data. As good as it might sound, there are always some tradeoffs - what if some specific streams are more frequently accessed than the others? Would the remaining cores remain idle instead of doing something useful? On the other hand, tools such as ScyllaDB or Redpanda seem to be leveraging this model quite effectively (both are using the same Seastar framework). We will be looking for the answers, before deciding which approach (thread-per-core or work stealing) suits Iggy better in the future.

Future​

Why building another message streaming then? A few months ago, I would probably answer - strictly for fun. Yet, after exploring more in-depth the status quo, what we would like to achieve is sort of twofold - on one hand, it'd be great to deliver the general-purpose tool, such as Kafka. On the other hand, why not to try and really push hard the OS and hardware to its limits when speaking of the performance, reliability, throughput and latency, something what e.g. Aeron does? And what if we could put this all together into the easy-to-use, unified platform, supporting the most popular programming languages, with the addition of modern CLI and Web UI for managing it?

Only the time will tell, but we're already excited enough to challenge ourselves. We'd love to hear your thoughts, ideas and feedback - anything that will help us in building the best message streaming platform in Rust that you will enjoy using! Feel free to join our Discord community and let us know what do you think :)

P.S.

This blog uses Rust.

We are happy to announce that Iggy.rs has reached the 0.1.0 release. This is a major milestone for the project, as it's getting closer to the first stable release.

Since the most recent update, Iggy.rs supports personal access tokens, which can be used to authenticate the clients, instead of the username and password. The tokens can be created and deleted using the available APIs.

In the latest update, the Iggy server as well as the clients for all the available transport protocols have been extended with the support for consumer identifier. Whether you poll the messages, store the consumer offsets, or create consumer groups, you can use the well-established identifier type, instead of just u32, which is now a common standard for the resources' identification such as streams, topics, users and consumers.

In the most recent update, the Iggy server as well as the clients for all the available transport protocols have been extended with the support for users and permissions. From now on, you can additionally secure your data using the authentication and authorization by specifying the granular set of permissions for each user.

The latest update introduces the changes to the PollMessages and GetConsumerOffset commands response schema, as well as the Stream, Topic and Partition structs extended with created_at field.