The Model Is the Easy Part: What a Real-Time Computer Vision Product Actually Takes

I used to think computer vision was a model problem. It is a pipeline problem, and the model is the easy part.

Here is the pattern I keep seeing. A team trains or fine-tunes a detector, it hits some impressive number on a curated clip, everyone claps, and then it goes into the real world and falls over. The lighting is wrong. The camera moved. Someone wore a color the training set never saw. The response arrives half a second late and the whole thing feels broken. The demo worked. The product did not.

I have shipped enough camera-driven products now to believe the mental model most people start with is wrong. Computer vision is not "a model that recognizes things." It is a real-time loop: sense, decide, act, fast and reliable, in a world you do not control. The model is one node in that loop. Getting the model to work is table stakes. Getting the loop to hold up is the actual engineering.

The model is a node, not the product

If you have built any real-time system, this framing will feel familiar. Think of the vision model the way you think of a single service in a request path. It takes an input, produces an output, and has a latency cost. It is not the system. The system is everything around it: where the frames come from, what happens to the output, how fast the whole chain runs, and what the product does when the model is wrong (because it will be wrong).

Detection is only half a product

A useful vision system closes a loop. It senses something, decides what that means, and then does something visible in the real world, fast enough that the response feels connected to the event. Sensing without a fast, dependable response is a science project, not a product.

The clearest example I can point to is Raqts, a product I handle all the software development for. It is a responsive racquet-sport wall: you hit the wall, and it reacts to how and where you hit it, in real time. The camera and vision pipeline read the play, and the system responds. That only works if every stage of the loop is quick and reliable. A late read does not make the wall "slightly less accurate." It makes the wall feel broken. Real-time is a binary experience, not a gradient.

The four things that decide whether it works

A vision demo is easy. A vision product is not. Four drivers decide which one you end up with, and they are where the real budget and risk live. None of them are "pick a better model."

1. Real-world accuracy, which is a different number
   The accuracy you get on a curated clip and the accuracy you get in production are two different numbers, and only one of them matters. The physical world is adversarial in the same way untrusted user input is adversarial. It will hand you the lighting you did not test, the angle you did not mount for, the background clutter your clean data never had. Treat it like input validation. You do not trust a number from a controlled recording any more than you trust a request body because it parsed once. Real accuracy is bought with representative data, hard-case testing, and honest measurement of where the model fails. The first question to ask is never "how accurate is it" but "accurate under which real conditions, and how do you know."

2. The latency budget is a hard constraint
   For anything interactive, the response has to feel tied to the event. That gives you an end-to-end budget, camera to inference to action, and it is a hard engineering constraint the same way a frame budget is in a game loop or a p99 target is in a service. It shapes every other choice: which model, running where, on what hardware. A system that is accurate but slow fails a real-time product just as surely as one that is fast but wrong. If Raqts read a hit a half-second late, no amount of accuracy would save the feel of it. You are engineering for both at once, and the budget is fixed before you pick the model, not discovered after.

3. Where the model runs is an architecture decision
   A vision model can run in the cloud, where compute is cheap and flexible, or on the edge, on a device next to the camera. This is the same cloud-versus-local trade you already know from anywhere else, with the same shape. Cloud is simpler to build and update but adds network round-trips and a hard dependency on connectivity. Edge keeps latency low and works offline but constrains you to what the on-site hardware can handle and makes updates harder. Real-time and privacy-sensitive products lean edge. Batch or tolerant ones lean cloud. This single decision ripples through cost, speed, and reliability, so make it deliberately and early, against your actual latency and connectivity numbers, not out of habit.

4. The hardware seams are each a small project
   Vision products rarely live in software alone. There are cameras to choose and place, mounts and lighting to control, and often actuators, sensors, or displays the software has to drive in sync. This is the IoT half of the build, and it carries the weight of any hardware seam: timing, failure modes, and physical constraints a pure web app never faces. Raqts is a wall, cameras, and responsive feedback working as one system. That is a genuinely different build from a screen-only app. The lesson I keep relearning: count the hardware seams honestly, because each one is a small project inside the project, the same way every third-party integration is a small project inside a normal build. The failure modes live in the seams, not in the model.

How I scope one now

The discipline is the same one I use for any zero-to-one build: name the risky parts before anyone quotes a number. For computer vision that comes down to five questions.

• Define the one decision the model must make. Detect, classify, count, or track. One clear vision task, done reliably, beats five fuzzy ones. Everything else is downstream software.
• Pin the real-world conditions. Lighting, angles, distances, speeds, and the edge cases that actually occur. This list is the difference between a demo and a product.
• Set the latency budget. Decide how fast the response must feel, because that constraint chooses your model and where it runs.
• Decide edge versus cloud on purpose. Against your latency, connectivity, and privacy needs, not out of habit.
• Count the hardware seams. Cameras, mounts, lighting, and anything the system has to physically drive in response.

Pin those five down and the estimate stops being a leap of faith and becomes a conversation about a defined thing.

What AI actually changed here

This is the part that genuinely shifted, and it is not the part people expect. A few years ago a custom vision capability meant collecting a large dataset and training a model from scratch, slow and expensive enough that only big budgets attempted it. Modern foundation models and mature vision tooling now get a strong first version working far faster. But notice what that does. It does not remove the hard work. It moves it. When the model is cheap to stand up, more of your effort lands on the parts that were always the real work: the real-world accuracy, the latency loop, and the hardware integration. The scarce skill stopped being "can you train a model" and became "do you know which vision task actually solves the problem, and where the real-world failure modes hide."

That is the honest state of computer vision right now. The barrier to a working prototype dropped through the floor. The barrier to a working product did not move at all, because it was never the model. It was always the loop.

A computer vision product is not a model. It is a loop: see, decide, act, fast and reliable, in a world you do not control. Get the four drivers right and you have software that senses the physical world and responds to it. Skip them and you have a demo that will never survive contact with a real room.

If you want the longer buyer-facing version of this, with the checklist I hand clients, I wrote it up on the Null Studio blog at nullstud.io.