Zenoh's put is fire-and-forget, get isn't - a read-after-write race in Elixir

This English version is an AI translation of my original article on Qiita (in Japanese). Background I've been experimenting with Zenoh via its Elixir bindings, Zenohex, not for its usual pub/sub use case but for its put /get storage feature. It mostly worked, except every so the state I picked back up was one step behind. Digging into why turned into a fun rabbit hole, so here's the writeup. Reproducing it To keep things simple, strip out the GenServer part entirely and just loop put immediately followed by get on the same key: {:ok, session_id} = Zenohex.Session.open(config) Enum.each(1..2000, fn i -> payload = Integer.to_string(i) :ok = Zenohex.Session.put(session_id, key, payload) {:ok, replies} = Zenohex.Session.get(session_id, key, 3_000, consolidation: :latest) case Enum.find(replies, &match?(%Zenohex.Sample{}, &1)) do %Zenohex.Sample{payload: ^payload} -> :ok %Zenohex.Sample{payload: other} -> IO.puts("stale! put #{payload} but got #{other}") nil -> IO.puts("no reply at all") end end) Out of 2000 iterations, a small fraction print stale! - about 78 (3.9%) in one run. The interesting part: querying again immediately afterward almost always returns the correct value (the fastest I measured was a single extra get about 1ms later). So it's not that the value disappears - there's just a small window of lag before the write is actually visible. Why Zenohex.Session.put/4 is a thin Rustler wrapper around zenoh-rust's put . Looking at the NIF implementation: fn session_put(...) -> rustler::NifResult { ... publication_builder .apply_opts(opts)? .wait() // = deadline do {:error, :not_confirmed} else Process.sleep(confirm_interval_ms) confirm(session_id, key_expr, payload, query_timeout_ms, confirm_interval_ms, deadline) end end end defp fetch(session_id, key_expr, query_timeout_ms) do case Zenohex.Session.get(session_id, key_expr, query_timeout_ms, consolidation: :latest) do {:ok, replies} -> case Enum.find(replies, &match?(%Zenohex.Sample{}, &1)) do %Zenohex.Sample{payload: found_payload} -> found_payload nil -> nil end {:error, _reason} -> nil end end end Usage: iex> ZenohAckPut.put(session_id, "key/expr", "payload") :ok Three possible return values: - :ok - the put succeeded and the read-after-write confirmation also succeeded - {:error, :not_confirmed} - the put itself succeeded, but confirmation didn't land within the timeout (this does not mean the write failed - it likely just hasn't shown up yet) - {:error, reason} - the underlyingput itself failed Running the same 2000-iteration loop through ZenohAckPut.put instead: zero stale reads, zero unconfirmed timeouts. It's published as a standalone module, along with the reproduction scripts used above and a script that verifies the fix: Not on Hex yet, so pull it in as a git dependency for now: defp deps do [ {:zenohackput, git: "https://github.com/kikuyuta/zenohackput.git"} ] end Takeaway Zenoh's put is fire-and-forget while get is a real request/response, and a get right after a put can occasionally return a stale value - a few percent of the time in my measurements. It's a known, currently-unresolved gap upstream. An application-level "put, then confirm with a get" wrapper is enough to close it in practice for use cases (like state handoff) that need read-your-own-writes. If you're using Zenoh's put /get for anything where you expect a write to be immediately visible - not just eventually - keep this asymmetry in mind. Top comments (0)