Most Embedded CI Stops One Step Too Early

The Gap in Embedded CI

Software teams have spent years making CI feel normal. A pull request builds automatically. Unit tests run automatically. Static analysis, linting, packaging, and deployment checks all happen before a reviewer ever touches the merge button.

Then embedded firmware hits a wall. The CI job builds the firmware, maybe runs host-side unit tests, maybe runs a simulator, and then quietly stops before the part everyone actually worries about: what happens on the physical device.

For many embedded teams, the next step is still a person at a bench. Someone flashes the board, opens a serial terminal, sends a command, watches logs, maybe power-cycles the device, maybe checks a measurement, and then copies the result into a ticket, release note, spreadsheet, or Slack thread.

That workflow is familiar. It also does not scale. It breaks when the team grows beyond one bench expert. It breaks when hardware is shared across locations. It breaks when QA asks which firmware hash was tested, on which board revision, against which requirement, with which logs. It breaks when a release is delayed because the evidence has to be reconstructed after the fact.

What BenchCI Does

BenchCI exists for that gap. BenchCI is hardware CI for embedded firmware teams. It runs firmware tests on real devices directly from CI, then captures structured results, logs, measurements, failure context, traceability, and release evidence from each run.

The basic idea is simple:

1. CI builds firmware
2. BenchCI schedules a hardware run
3. BenchCI Agent flashes and tests a real device
4. Results and evidence return to CI and the dashboard

This is not simulation, emulation, or a mock device. BenchCI is designed for the actual bench: the board, flasher, serial port, GPIO, relay, protocol bus, power path, instrument, or existing HIL rig your team already uses.

The Bench and the Test Are Separate

One of the design choices behind BenchCI is that the physical bench and the test intent should not be welded together.

bench.yaml describes the hardware setup: the DUT, flashing backend, UART port, CAN interface, GPIO chip, relay, measurement resource, or Agent-connected lab machine.

suite.yaml describes what you want to prove: flash firmware, wait for a boot banner, send a command, expect a response, power-cycle the device, assert a measurement, or link a test case to a requirement.

A small smoke test looks like this:

tests:
  - name: boot_ok
    steps:
      - expect_uart:
          node: dut
          transport: console
          contains: "[BOOT] OK"
          within_ms: 3000

  - name: ping
    steps:
      - send_uart:
          node: dut
          transport: console
          data: "PING\n"
      - expect_uart:
          node: dut
          transport: console
          contains: "PONG"
          within_ms: 1000

That suite stays stable even if the underlying bench changes from one UART adapter to another, from a GPIO relay to a LAN relay, or from a local run to a cloud-scheduled Agent run.

The goal is not to make hardware testing magical. Hardware is still hardware. Cables loosen. Boards behave differently. Instruments need configuration. The goal is to make the workflow repeatable enough that it can live in CI, be reviewed by a team, and produce evidence without someone rebuilding the story by hand.

Evidence Is the Missing Product

Most test tools answer one question: did it pass? That question matters, but it is rarely enough for embedded teams shipping real products.

A release reviewer often needs to know:

• Which firmware artifact was tested, and what was its SHA256?
• Which Git commit and CI job produced it?
• Which physical DUT or board revision ran the test?
• Which requirements, risks, or test cases were covered?
• Which logs, measurements, and artifacts support the result?
• Who reviewed or approved the release bundle?

BenchCI treats that evidence as a first-class output, not an afterthought. A run produces results.json, evidence.json, evidence.html, artifact manifests with hashes, input snapshots, flash logs, transport logs, measurement data, JUnit XML, CTRF, and release evidence bundles.

For teams with QA or regulated workflows, those records can support release review and audit-friendly traceability.

BenchCI does not certify your product. It does not replace your quality system, acceptance criteria, risk process, or final release decision. It records the facts of the run in a way that makes those decisions easier to defend. That distinction matters. The promise is not "click a button and become compliant." The promise is "stop reconstructing evidence from screenshots and terminal logs after release week has already started."

You Do Not Have to Throw Away Existing Tests

Many embedded teams already have useful automation: pytest, labgrid, Robot Framework, a custom harness, CANoe, NI TestStand, dSPACE, Speedgoat, XJTAG, or a homegrown HIL rig. BenchCI is meant to sit alongside that reality.

If your current runner can emit JUnit XML or CTRF, BenchCI can import the result and attach firmware provenance, DUT identity, artifacts, traceability, and release-bundle workflow around it. If you already have a rig that can be triggered by a command or HTTP API, BenchCI can orchestrate it as part of the same evidence flow.

That is deliberate. The embedded world is too varied for one product to replace every test harness, simulator, instrument, and lab controller. But many teams still need the same layer above those tools: scheduling, shared bench access, run history, traceability, review, sign-off, and evidence packages.

Where BenchCI Fits

BenchCI is a good fit when your CI currently stops at the build step and the real-device check still depends on someone walking to a bench. It also fits when your team has scripts, but those scripts are hard to share, hard to audit, or hard to connect to release decisions.

Typical uses:

• Boot and smoke validation on real boards
• UART, CAN, Modbus, I2C, SPI, GPIO, power, and measurement checks
• Firmware regression testing before release
• OTA or fleet-update confidence gates
• Evidence packages for QA and customer acceptance
• Shared access to scarce hardware benches
• Keeping existing test tools while adding traceability and release workflow

It is probably not the right tool if your team is software-only, if simulation is the only validation you need, or if you already have a complete internal system for scheduling hardware tests, preserving evidence, reviewing releases, and maintaining all of it long-term.

Try It Without Hardware

Local and direct-agent runs are free and need no account:

pip install benchci
benchci demo

The demo runs without login or hardware and opens a local dashboard-style report with logs, artifacts, and evidence output.

For a real run:

benchci doctor
benchci init --preset flash-uart
benchci run -b bench.yaml -s suite.yaml -a build/firmware.elf

From there, teams connect BenchCI to GitHub Actions, GitLab CI, or another CI system, add a BenchCI Agent on a Raspberry Pi or Linux lab machine, and move real-device checks into the same workflow as the rest of their engineering process.

The philosophy is straightforward: unit tests prove your logic, static analysis improves your code, simulation explores scenarios, and BenchCI proves what happened on the hardware. Embedded CI should not stop at "the firmware compiled." It should end with a real device, a real result, and evidence your team can use.

• Website: https://benchci.dev
• Docs: https://docs.benchci.dev
• Questions: tech@benchci.dev