Why I Built VortexMQ: A 167M ops/sec Message Broker in 100% Pure Go
DEV Community

Why I Built VortexMQ: A 167M ops/sec Message Broker in 100% Pure Go

Every engineering team eventually runs into the "message broker tax." You start building a clean, modern microservice architecture in Go or Rust. The services compile to small binaries, boot in 20 milliseconds, and use under 25 MB of memory. Everything feels snappy and predictable. Then your system grows, and you need asynchronous queueing, delayed job retries, and task streaming. So you deploy Kafka or RabbitMQ. Almost immediately, your infrastructure profile shifts: You now manage multi-gigabyte JVM heaps or an Erlang beam runtime. You need coordinator daemons (ZooKeeper, KRaft, or Mnesia clusters). Nodes consume 500 MB to 1 GB of RAM at idle before processing their first message. Stop-the-world garbage collection pauses occasionally turn a 150-microsecond latency into a 200-millisecond p99 spike. You have to wire up external Prometheus exporters, Grafana dashboards, and third-party UIs just to check queue depths and inspect dead messages. When the queue consumes 20 times more resources than the application services producing and consuming the messages, something feels unbalanced. I wanted a message broker that felt like Go itself: a single 5.7 MB static binary, zero runtime dependencies, instant startup, sub-microsecond latency, and enough throughput to saturate a 10GbE network link on commodity hardware. That is why I built VortexMQ.

What is VortexMQ?

VortexMQ is an open-source, ultra-fast message broker and task engine written in pure Go (zero CGO, zero external dependencies). You can run it locally with Docker in a couple of seconds:

docker run -d -p 8379:8379 -p 8380:8380 ianshugarg/vortexmq:latest

Here is the core feature set:

  • 167.1 Million ops/sec lock-free ring buffer throughput (5.98 ns/op on bare metal)
  • 2.33 Million ops/sec full TCP network throughput with pooled memory buffers
  • Drop-in Redis RESP2 and RESP3 compatibility: connect using standard Redis clients in Go (go-redis), Python (redis-py), Node.js (ioredis), Rust, or redis-cli
  • Hierarchical Timing Wheel: O(1) delayed message delivery without sorted set polling hacks
  • Poison-Pill Dead Letter Queues (DLQ): automatic panic recovery that isolates crashing payloads with 1-click GUI replay
  • Embedded Quantum Web Studio: an interactive dashboard served directly from the 5.7 MB binary on port 8380 via Go's embed.FS
  • Under 15 MB RAM idle footprint

The Bottleneck in Go Channels (chan T)

When I started experimenting with the core engine, the first logical choice was native Go channels (chan T). Channels are great for application concurrency, but under heavy multi-core benchmark loads (32 producer goroutines pushing to 16 consumer workers), CPU profiles in

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.