Building a physical API

Listening to the bus

In MIT mode, every command a motor receives generates a reply. Each reply frame carries the motor's position, velocity, torque, and temperature, packed in 8 bytes. With six motors commanded 100 times per second, that's around 600 frames per second of live hardware state on the wire, feeding the memory boxes the driver keeps per motor and then being discarded.

CAN is a broadcast medium. Every node on the bus sees every frame, and reading costs the other participants nothing. So the telemetry problem reduces to a program that opens the same bus, decodes the frames that pass by, and ships them somewhere useful, without ever transmitting. The control loop doesn't know it exists.

I wrote that program in Rust and called it sentinel. It runs on the Raspberry Pi in the robot's head, next to the teleop but in a separate process, and its structure is small. One thread reads the CAN socket and decodes status frames, a second thread wakes up once per second, drains what accumulated, and POSTs the batch to my API over HTTP. Between the two sits a bounded channel.

The reader also doesn't keep everything it hears. Each motor reports its state 100 times per second, far more than monitoring needs, so only 10 of those samples per second go through and the rest are discarded, which makes storage ten times cheaper.

The design decision I care most about is that telemetry is best effort. If the API is slow or offline, the channel fills, and the reader starts throwing samples away instead of queueing them, keeping only a count of what was lost. That count travels in the next batch that succeeds, so the hole in the data explains itself. I tested this by taking the API down for forty minutes during a session. Sentinel kept running on stable memory, and when the API came back, the first batch reported around 140 thousand discarded samples, which is what six motors at ten samples per second accumulate over those forty minutes, so the loss accounting checks out. A telemetry agent that grows a queue until it takes down the machine it's supposed to watch would be worse than no agent.

Listening to the replies also means listening to the commands. The command frames carry the target position for each joint, so sentinel remembers the last target it saw and attaches it to every sample. The difference between where a joint was told to be and where it actually is becomes a tracking error signal, and that turned out to be one of the most informative charts. A small constant offset is just gravity working against a finite gain, a transient spike is a fast motion, and a sustained error is a joint fighting an obstacle.

When the robot is quiet

A powered but disengaged robot sends no status frames, and that's precisely when I want to watch the power supply. For that case sentinel has an active mode. It notices the bus has gone silent and starts asking the motors, one question per second, for the bus voltage and each motor's current, rotating through them so every motor gets its turn. The reply comes back on the same bus and the same decoding path picks it up.

Writing to the bus sounds like it contradicts the passive design, but CAN itself resolves this. When two nodes transmit at the same time, the frame with the numerically lower identifier wins arbitration. The parameter read frame has a higher identifier than every control and status frame, so my questions always lose to control traffic. They can be delayed by the robot working, but they can never delay the robot. That property is what let me later keep the voltage readings running even during teleoperation.

The API and the database

The receiving side is deliberately boring. It's a FastAPI service backed by TimescaleDB, a Postgres extension for time series that splits large tables into time-ordered chunks, so queries over a recent window stay fast no matter how much history sits behind them. Sentinel POSTs batches, the API writes rows, and a retention policy deletes anything older than thirty days.

The surface of the API is small. Telemetry comes in through two endpoints, one for the joint samples and one for the voltage readings, each taking a batch per request with a key in the header. A robot doesn't need to be registered anywhere. The first batch that arrives with a new robot id creates it, and every batch after that refreshes its last seen timestamp, which is also how everything downstream decides whether a robot is online.

The reading side has three endpoints:

• One lists the robots
• One returns a robot's current snapshot
• One returns the history of a metric for charts

The snapshot does more work than it sounds, because the pieces arrive at different rhythms. The freshest status sample of each joint, the last voltage reading, the last current measurement, and the collector's own health report all live in different rows with different timestamps, so the endpoint gathers the latest of each and merges everything into one answer.

The history endpoint aggregates on the database side, grouping the raw samples into time buckets, so a window of fifteen minutes or a whole day comes back as roughly two hundred points either way, instead of a dump of raw rows.

Going through an HTTP API instead of letting the collector write straight to the database is the part that makes this a layer instead of a script. The same door, with the same key, works for the next robot I add, and it's the same door the human feedback from the first article comes through, with each verdict landing next to the telemetry of the episode it judges and helping decide what goes into the next dataset. That flow deserves its own article, so I'll leave the details for a next post.

Grafana as the starting point

Before writing any interface of my own, I pointed Grafana at the database. The dashboards are provisioned as code in the repo, so bringing the stack up recreates them, and within a day I had the bus voltage as a big number, circular gauges for each motor's temperature and current, the tracking error over time, and alert rules for the situations I actually fear: a motor past 50 degrees, the supply under 45 volts, or the robot silent for too long.

The panels answer different kinds of questions:

• Temperature and current per motor tell me whether the hardware is suffering. Heat builds slowly over a long session, so the gauges creep instead of jumping, and each joint's current shows how hard it works, with the shoulder carrying most of the load and spikes marking the moments the arm meets resistance.
• The voltage panel watches the power supply react to that same effort.
• Tracking error covers the motion itself, and the position, velocity, and torque charts per joint let me scroll back through a session and see what the arm was doing at the moment something looks off.
• The control loop rate watches the software.
• Two small panels watch the watcher: the count of samples sentinel had to discard and the age of the freshest data, so I know when to distrust the rest of the dashboard.

A joint filter narrows all of it to a single motor when I'm chasing something specific.

With the real robot on, the numbers gained texture. The supply idles at 47.8 volts and dips visibly under load. The shoulder pitch motor holds around 5 newton meters doing nothing but resisting gravity. The control loop reports between 99 and 101 hertz, measured by counting status frames, which would catch the class of bug where some blocking call quietly drags the loop down.

Grafana answered the question of whether I could see the robot at all, cheaply, and it taught me which visualizations matter before I invested in anything custom. It remains my engineering console.

The control plane

An engineering console works for the person who built it. For anyone else, a wall of charts asks too much - you need to know which panel matters, which threshold is normal, and which number is safe to ignore. The control plane is my answer to that. It's a web frontend in the visual language of the project, reading from the same API, where the first glance already answers the questions that matter: Is the robot online, how much power is it drawing, is anything running hot.

The main page shows the fleet, one card per robot. Each card carries the robot's status, the bus voltage, the total current draw, and a table of its motors with the temperature and current of each one. These are the same signals from the Grafana dashboard, reduced to what someone standing next to the robot actually wants to know, and presented with some personality - small hand-drawn icons that come alive while the robot is online. A page that people enjoy looking at gets looked at more often, and for monitoring that's half the job.

Clicking a robot opens its page. The centerpiece is a live 3D model of the arm that moves with the joint positions coming off the bus, so teleoperating the real arm makes its copy move on the page a moment later, fed by nothing but the passive listening described earlier. Around it, the page shows the health of the collector itself - loop rate, staleness and discarded samples - along with charts of voltage and tracking error over the recent window.

The motors table on this page also answers a question I had no way of answering before: which exact firmware each motor is running. Every joint shows its motor model and firmware version next to the live temperature and current, so the page works as an inventory of the arm and not just a monitor. How those version numbers get there is a story of its own, and it's the next section.

There's more planned for this page. The robot already carries three cameras for the model, and streaming them into the control plane would let a remote session show what the robot sees next to what its joints report, in the same place.

The inventory

Everything up to here watches what the robot is doing. There's a second kind of question observability has to answer, which is what the robot is. A robot is an assembly of parts that each carry an identity of their own - motors of different models, boards, and the software running on all of them - and that composition changes over time. A motor gets swapped after a failure, a firmware gets updated, a package gets upgraded on the Pi. When two robots are supposed to be identical, or when one robot is supposed to be the same as it was last month, the inventory is what turns "supposed" into "verified."

In this project it started with the hardest column of the motors table: the firmware version. That column took an afternoon of digging through the motor's manual. It lists version strings among the readable parameters, but on my motors those reads come back flagged as an error. The manual also describes a dedicated version query, a frame of the same communication type as the stop command, distinguished only by a marker byte in the payload. Sending a malformed one could stop a motor, so I tested it with a throwaway probe on an idle bus, one motor at a time. All six answered.

The answer came with a surprise. The reply arrives as a frame of the same type as the status frames, so sentinel, which was running while I probed, decoded those replies as telemetry and recorded a motor running at 1050 degrees. The version reply has to be recognized by its own signature and intercepted before status decoding, and that check is now the first thing the decoder does.

With that settled, sentinel asks each motor for its version once at startup, when the bus is quiet, and the control plane fills in the model and firmware of every joint. The versions revealed that firmware follows the motor model, with same-model joints running identical builds, and having that confirmed in a table instead of assumed is what an inventory is for. It's also the groundwork for updating firmware over the same channel later.

The motors are only the first entry. The robot is also a computer - a Raspberry Pi with an operating system, drivers, and the packages my whole stack depends on - and all of that changes over time just like firmware does. The plan is for the inventory to cover that side too: the OS version, the kernel, the versions of the packages that matter, reported through the same API. When the robot behaves differently from one week to the next, the first question is what changed, and an inventory that covers the machine end to end is what lets me answer it without guessing.

Conclusion

What exists now is a robot that reports its own state while it works. Every joint's state arrives ten times a second, voltage and current keep coming even when it sits idle, the collector reports its own health, the database holds thirty days of history, alerts cover the failures I fear, and two interfaces