DEV Community

I Shipped My First Rust Release, and CI Turned Red Twice in 20 Minutes

I shipped the first public release of my Rust project last night. Within about twenty minutes, CI turned red twice. Here's what broke, how I found it, and the two calls I made under a bit of time pressure.

The project AgentOS is a runtime layer for AI agents - the infrastructure around an agent, not the agent logic itself: supervised lifecycle, a gRPC message bus, a health endpoint, an SSE event stream, a secrets vault, and deterministic time-travel replay (every LLM/tool exchange gets journaled, so a run can be replayed offline with no API cost, and forked into alternate timelines from any checkpoint). It's a 10-crate Rust workspace, MIT/Apache-2.0.

I'd been building it privately for a while. Last night I flipped the repo to public and tagged the first alpha release. That's when things got interesting.

Failure #1: a clippy lint that didn't exist last month

The repo had been private, so CI hadn't run against a fresh clippy in a while. The moment I pushed a commit after going public, this showed up:

error: you seem to want to iterate on a map's values
  --> crates/bus/src/grpc.rs:631:37
   |
631 | for (_agent_id, senders) in subs.iter() {
   |     ^^^^^^^^^^^
   |
   = help: use the corresponding method
631 - for (_agent_id, senders) in subs.iter() {
631 + for senders in subs.values() {
   |
   clippy::for_kv_map , denied by -D warnings in CI.

Fair catch - the code was iterating a HashMap and discarding the key on every iteration, which is exactly what .values() is for. One-line fix:

// before
for (_agent_id, senders) in subs.iter() {
// after
for senders in subs.values() {

Pushed, CI green again. This one was boring, which is the best kind of bug to have.

Failure #2: the release matrix, at 12:33am

The bigger one showed up when I tagged v0.1.0-alpha. The release workflow builds five targets in parallel - Linux x64, Linux arm64, macOS Intel, macOS Apple Silicon, Windows - and packages each into a binary release with checksums. Four went green. Linux arm64 died with:

error: failed to run custom build command for `openssl-sys v0.9.116`
...
Could not find directory of OpenSSL installation
$HOST = x86_64-unknown-linux-gnu
$TARGET = aarch64-unknown-linux-gnu

openssl-sys needs a real OpenSSL installation (headers + libs) for the target architecture to link against, and cross-compiling from an x86_64 GitHub Actions runner to arm64 doesn't have that available without extra setup. Not a code bug - a missing piece of cross-compilation infrastructure.

I had three honest options at that point:

  • Block the whole release until arm64 cross-compiling was solved properly
  • Vendor OpenSSL and wire up a cross-linker for the arm64 target under time pressure, late at night, for a release I hadn't tested that way before
  • Ship the four targets that work, drop arm64 from this release, and track the real fix as its own issue

I went with option 3. Four platforms is still a real alpha release that covers the overwhelming majority of contributors trying it out. Blocking on a cross-compilation edge case for a Linux architecture almost nobody in the alpha audience runs felt like optimizing for the wrong thing at midnight. And silently vendoring a rushed fix into a security-relevant dependency (TLS) is exactly the kind of shortcut that comes back to bite you.

So: pulled the target from the matrix, re-tagged, watched the four remaining builds go green, and filed the real fix as its own issue with three candidate approaches laid out (the best one is probably migrating off openssl-sys to rustls entirely, which would remove the system OpenSSL dependency for every target, not just arm64 - cross-compilation problems like this are usually a sign the dependency choice needs revisiting, not just the CI config).

The one check I'm glad I ran before either of this

Before any of the above, I'd actually run the README's own quickstart end to end - cargo build --workspace, then run the example agent - specifically to catch the gap between "the README claims this works" and "this actually works." It did, and the real terminal output (supervisor spawning the agent, health server on :8080, gRPC bus on :50051, an SSE stream on :8081) is now in the README instead of a hypothetical example.

I also downloaded the actual released Windows binary afterward and ran agentOS.exe --version against it - small thing, but it's the difference between "the release workflow exited 0" and "a stranger who downloads this file gets a working program."

Neither bug was catastrophic. Both were the kind of thing that's mildly stressful in the moment and completely mundane in hindsight - which is most of what real engineering work actually looks like, release-day war stories included.

If you want to poke at it, the repo's here: github.com/WAHIB-EL-KHADIRI/AgentOS. There are a handful of labeled good first issues if anyone wants to dig into a self-contained piece of it - I'm around to answer questions on any of them.

Comments

No comments yet. Start the discussion.