At the edge, the number that matters is memory - not throughput (specially in Ramageddon)

We rebuilt LF Edge eKuiper in Rust and ran it against eKuiper, Telegraf and Redpanda Connect on five real MQTT workloads - one core, 1 GB of memory, and output checked message-by-message. The most important result wasn't speed. I-Dacs Labs Engineering · ~16 min read Most stream-processing benchmarks you'll read optimize for one number: peak throughput on a big server. That number is close to useless for the place these engines actually run - an industrial gateway, an ESPHome hub, a vehicle head-unit, an EV charger. There, you get one or two CPU cores and a few hundred megabytes of free memory, your input arrives over MQTT, and your traffic is bursty in the worst way: fleets reconnect together, chargers start sessions together, devices flush buffered readings all at once after an outage. In that world two questions decide whether your pipeline survives, and neither is peak throughput: - Does the engine keep up on a single core? - Does its memory stay bounded when traffic grows? We built a stream engine called rekuiper to answer "yes" to both, and then we built a benchmark honest enough to tell us if we'd actually managed it. This post is about the benchmark as much as the engine, because the benchmark taught us more than we expected - including a correctness bug in our own code that a throughput-only test would have rewarded as "fast." The headline: across five MQTT workloads shaped like real deployments, rekuiper produced complete, correct output at 100,000 messages per second on one core in every workload - the top of our tested range, so we never found its ceiling. But the result we care about most isn't that. It's that on the windowed workloads, rekuiper's memory stayed between 5 and 10 MB while the Go-based engines climbed to half a gigabyte to a full gigabyte, or failed. That gap is the whole point, and it comes from design, not from the language. What rekuiper is rekuiper is a stream-processing engine written in Rust that reimplements the surface of LF Edge eKuiper: its REST API, its SQL dialect, its stream and rule definitions, and its kuiper command-line interface. The goal was boring on purpose - existing eKuiper rules, the eKuiper Manager web UI, and deployment tooling should keep working - so that "switch the engine" isn't also "rewrite everything." Concretely, the compatibility surface covers eKuiper's REST API (98 paths and 140 operations, checked black-box against eKuiper's own OpenAPI description), the SQL dialect including JSON paths, CASE , array indexing and unnest , and eKuiper's stream option names (DATASOURCE , FORMAT , CONF_KEY , SCHEMAID , TIMESTAMP , and so on). If you know eKuiper, you already know rekuiper. What's different is underneath, and it's built around one principle: memory stays bounded under load. Three design choices carry that, and each one shows up later in the numbers. Three design choices that keep memory flat Bounded queues with real backpressure Sources publish records into an in-process stream bus with bounded per-subscriber queues - 4,096 records each. Admission is reserve-then-commit: a batch first reserves capacity in every subscriber's queue, and only then commits. So a batch is either delivered to all subscribers or rejected outright, never half-delivered, and a slow rule pushes back on its source instead of quietly dropping data. Each rule runs as its own task, and its output drains through a bounded sink queue (default 10,000) served by a dedicated sink worker. The MQTT source uses the rumqttc client, and when a single network read surfaces several publishes, the source admits everything already buffered as one batch of up to 1,024 records. That avoids a per-message wakeup without ever waiting around for more data - you pay one scheduling cost for a burst instead of one per message. This batch-admission trick is a big part of why rekuiper uses roughly half the CPU per message of the Go engines on the simple workloads. Incremental window aggregation: O(groups), not O(messages) This is the important one. When you compute GROUP BY device, TUMBLINGWINDOW(ss, 10) with count , avg , max and friends, the naive way is to buffer every row that falls in the window and aggregate at the trigger. Memory then grows with traffic - messages per window - which is exactly the thing that explodes when a fleet reconnects. rekuiper instead keeps one accumulator per group per aggregate and never stores the rows. Window memory becomes a function of the number of devices, not the number of messages. For the common edge shape - group columns, plain columns, and count /sum /avg /min /max over simple expressions - this incremental evaluator does the whole job. Statements that genuinely need the rows (collect() , joins, some HAVING ) fall back to a buffered evaluator, and a unit test checks the two produce identical output on mixed data. For a fleet, this is the difference between memory that scales with how many vehicles you have and memory that scales with how fast they're all talking at once. Only one of those is safe on a 1 GB box. An offline sink cache that spills instead of dropping For intermittent uplinks - a vehicle in a tunnel, a remote site on flaky cellular - a sink can enable a cache using eKuiper's own options (enableCache , memoryCacheThreshold , maxDiskCache , and the rest). Records whose send fails recoverably are queued FIFO: in memory up to a threshold, then in disk pages, and only when the disk budget is exhausted are the oldest records dropped - and counted, not silently lost. The MQTT sink holds one persistent connection per action and reports disconnection, so an outage is detected and cached rather than quietly discarded. (The cache is covered by an integration test but isn't part of the performance numbers here.) The benchmark that doesn't lie to you Here's the uncomfortable truth about a lot of edge stream-processing comparisons: they measure throughput at the point the engine acknowledges ingest, or they count output records without checking that the records are correct. Both can hide loss and duplication completely. An engine that drops 15% of your data can look fast if you never verify what came out the other end. So we built the benchmark around exact output verification, and gave every engine the same cramped room to work in. Equal, realistic limits. Every engine runs in a container pinned to one CPU core with 1 GB of memory and no swap (--cpuset-cpus=2 --cpus=1 --memory=1g --memory-swap=1g ). A separate Mosquitto broker gets its own cores and generous queue limits, so the broker is never the bottleneck. An open-loop Rust load generator (mqttgen , standard library only, MQTT 3.1.1, QoS 0) feeds every engine from the same schedule, and a step only counts if the generator actually stayed on schedule. Four engines. rekuiper v0.425-beta, eKuiper 2.4.1, Telegraf 1.40.0, and Redpanda Connect 4.109.0 (formerly Benthos). We deliberately excluded Apache Flink: neither Flink 2.x nor Apache Bahir ships an MQTT connector, so testing Flink would have meant a custom source or a Kafka bridge - changing the very ingest path under test. Rather than benchmark a different pipeline and call it Flink, we left it out and said so. Five workloads shaped like real deployments: - W1 - telemetry filter. 1,000 devices, one topic, a simple WHERE temp > 21.0 with a unit conversion. Stateless. - W2 - per-device windows. 1,000 devices, 10-second tumbling windows with count /avg /max . Stateful. - W3 - ESPHome states. 10,000 plain-text topics via wildcard, using FORMAT="binary" andmeta(topic) to carry the topic through. Stateless but wide. - W4 - vehicle windows. 10,000 topics (one per VIN), 10-second tumbling windows. Stateful and wide - the hardest memory test. - W5 - EV charger sessions. 2,000 topics, SESSIONWINDOW(ss, 10, 2) . Neither Telegraf nor Redpanda Connect has a session window, so they can't express it at all. Proofs, not vibes. For each engine, workload and rate (5k, 20k, 50k, 100k msg/s), we warm up until the subscription is provably live, send for 30 seconds on a fixed schedule, drain until the sink file stops growing, then verify the output exactly: - W1: the count of unique message IDs carrying the run tag must equal the closed-form expected filtered count, with no duplicates. - W2, W4, W5: the sum of per-device counts across all output windows must equal the messages sent, and every device must appear. - W3: output rows must equal messages sent, and all 10,000 topics must appear. A step is complete only when its proof holds. "Loss" is the relative shortfall against the proof. This is the part that makes the numbers trustworthy - and, as you'll see, it's the part that caught our own bug. Results Nobody else finished the range Highest tested rate with complete, correct output: | Workload | rekuiper | eKuiper 2.4.1 | Telegraf 1.40.0 | Redpanda Connect 4.109.0 | |---|---|---|---|---| | W1 telemetry filter | ≥ 100,000 | 20,000 | 50,000 (lag 10 s) | 20,000 | | W2 per-device windows | ≥ 100,000 | 20,000 | none | 5,000 | | W3 ESPHome states | ≥ 100,000 | 20,000 | 50,000 (lag 5 s) | 20,000 | | W4 vehicle windows | ≥ 100,000 | 20,000 | 50,000 only | 5,000 | | W5 charger sessions | ≥ 100,000 | 20,000 | not supported | not supported | rekuiper completed all 20 steps. Because 100,000 msg/s was the top of the range, we never reached its limit - at 100k on the wide ESPHome workload it used 96.6% of the core, and the windowed workloads used 78-85%, so there's headroom left. eKuiper was solid and complete up to 20,000 msg/s across the board. Telegraf managed 50,000 on two stateless workloads but never produced complete per-device windows at any rate. Redpanda Connect reached 20,000 on stateless workloads and 5,000 on windows. The memory gap This is the result we'd frame and put on the wall. Peak engine heap (cgroup anonymous memory) at 20,000 msg/s, the highest rate every engine could still be compared at: | Workload | rekuiper | eKuiper | Telegraf | Redpanda Connect | |---|---|---|---|-