Show HN: Open-source, real-time object tracking for live video feeds
Comments
Live, multi-camera tracking with SAM 2, on one GPU or many.
SAM 2 and its smaller sibling EdgeTAM can track anything you point at, but their code was written for one recorded video at a time. tamserve is the serving engine they are missing: one copy of the model tracks every object on every camera, for as long as the cameras run.
Point at what to track, or name it ("dog", "the red mug") and tamserve finds it.
Why tamserve ยท New to video tracking? ยท Using it ยท Tracking by name ยท How it works ยท Results ยท Limits
Six live cameras on one GPU, every one at 30 frames/s. Watch the 32-second demo, or read how it was recorded.
Why tamserve
The models are excellent. The code that comes with them was made for editing one recorded video, and the usual way to put it on several cameras is to run a separate copy of the model for each one. Memory piles up with every copy and with every minute of video.
tamserve does what LLM servers such as vLLM do for chat: every camera shares one copy of the model, and whatever needs the GPU at a given moment goes through it together. That turns out to be faster as well as lighter, and because the model itself is not touched, the results are the same.
| the model's own code | tamserve |
|---|---|
| Several cameras | one copy of the model per camera | one copy for all of them |
| GPU memory, 8 cameras | 8.6 GB | 2.2 GB |
| Speed, 1 camera | 177 frames/s | 219 frames/s |
| Speed, 8 cameras | 280 frames/s | 323 frames/s |
| 64 cameras | would need about 70 GB of GPU memory | 321 frames/s in 5.7 GB |
| GPU memory for an 800-frame video | 9.8 GB | 0.28 GB, however long it runs |
| Starting to track something new mid-stream | not possible | on any frame |
| Finding what to track | you point at each object | you point, or you name it and a Tracker finds and follows every match by itself |
| Several GPUs | you split the cameras between them by hand | a Router places the cameras and gathers every result |
| Accuracy on DAVIS 2017 | 0.8686 | 0.8685 |
These are for EdgeTAM on one RTX 4090, against EdgeTAM's own code at its fastest: compiled, one copy per camera, all sharing the GPU. Speeds are totals across the cameras, with one object each.
With the larger SAM 2.1 models, tamserve runs at the same speed as Meta's current SAM 2 code and uses about a third of the GPU memory. Results has the full tables.
New to video tracking?
Say you have video, and you need to know where one particular thing is in every frame: this person, that forklift, the dog. That is tracking.
SAM 2 and its smaller relatives, EdgeTAM and SAM 2.1, are models from Meta that do it very well. You point at the thing once, on a single frame, and the model draws its exact outline on every frame after that.
tamserve sits between one of those models and your program. Your program hands it the latest frame from each camera and gets back the outline of every tracked object in each one. You can start tracking something new, or stop, at any moment, and cameras can come and go.
It is meant for anyone who has to answer "where exactly is this thing, right now, in each of my video feeds": sports and broadcast graphics, warehouse and retail analytics, robotics, wildlife and lab monitoring, annotation tools.
There are two ways to say what to track. You can point: a person clicking on the picture, or a detector, says where the thing is, and tamserve takes over from there. Or you can name it: a Tracker takes a word or a phrase, finds everything that matches on every camera, and follows it.
Four words come up a lot below:
| word | what it means here |
|---|---|
| frame | one image from a video |
| mask | the answer for one object in one frame: a grid of True and False the size of the frame, True on every pixel that belongs to the object |
| prompt | how you say what to track: a box around it, a click on it, a rough mask of it, or, with a Tracker, a word or a phrase |
| stock | the model's own code: the tracker in Meta's EdgeTAM repository, used exactly as it comes |
Using it
pip install tamserve
You need Python 3.10 or newer and an NVIDIA GPU. It does run on a CPU, which is handy for trying it out but much too slow for live video.
The first time you load a model, tamserve downloads it into ~/.cache/tamserve, so that run needs an internet connection. EdgeTAM is 54 MB. The download is pinned to one version and verified before any of it is loaded.
tamserve does not open cameras, and it does not run a loop of its own. Your program reads frames however it likes and calls engine.step() once per round, with the newest frame from each camera exactly as the decoder gives it.
This example uses OpenCV to read two network cameras:
import threading
import cv2 # pip install opencv-python; any source of frames will do
from tamserve import Box, Engine, build_video_predictor
class Camera:
"""Reads one stream on a thread of its own and keeps its newest frame."""
def __init__(self, url):
self.stream, self.newest, self.arrived = cv2.VideoCapture(url), None, threading.Event()
threading.Thread(target=self.read, daemon=True).start()
def read(self):
while True:
ok, picture = self.stream.read()
if not ok:
break # the stream has ended
self.newest = cv2.cvtColor(picture, cv2.COLOR_BGR2RGB) # OpenCV gives BGR, the engine takes RGB
self.arrived.set()
def take(self):
"""Waits for a frame the loop has not seen yet, and returns it."""
self.arrived.wait()
self.arrived.clear()
return self.newest
engine = Engine(build_video_predictor("cuda")) # loads EdgeTAM onto the GPU
cameras = {"door": Camera("rtsp://door-camera/stream"), "yard": Camera("rtsp://yard-camera/stream")}
# What to track and where it is in the first frame: a box as (left, top, right, bottom), in pixels.
# In a real system this comes from a detector, or from someone drawing on the picture.
start = {"door": {"courier": Box(410, 120, 560, 480)}}
while True:
# 1. Take the newest frame from each camera.
frames = {name: camera.take() for name, camera in cameras.items()}
# 2. One call tracks every object in every one of those frames.
masks = engine.step(frames, start)
start = None # the box was only needed once; from here on the engine tracks the courier by itself
# 3. Use the answer. masks["door"]["courier"] is True on the courier's pixels and False everywhere else.
for camera_name, objects in masks.items():
for object_name, mask in objects.items():
print(camera_name, object_name, "covers", int(mask.sum()), "pixels")
Each call to step() takes one frame per camera and returns when the masks for those frames are ready. The engine remembers what it is tracking between calls, which is why the box is passed only once. Camera names and object names are yours to choose.
The frames go in exactly as the decoder gives them. The engine uploads them and resizes them on the GPU, which takes about half a millisecond a frame where resizing on the CPU takes 16.
The Camera class is there for speed too: with a thread per camera, the next round's frames are decoded while the GPU tracks this one. And because a camera keeps only its newest frame, a loop that falls behind skips frames and does not lag. The engine tracks from whatever frame it is given and does not need to see every one.
How fast can the loop go? On an RTX 4090 with the compiled engine, eight 720p cameras and one object each, a loop like this one tracks 273 frames a second from video to masks, which is 34 for every camera. With 1080p cameras it is 238.
To start tracking something, you give it a name and point at it in the current frame. There are three ways to point. All of them use the frame's own pixel coordinates, with x counted from the left edge and y from the top.
| prompt | what it is |
|---|---|
Box(left, top, right, bottom) |
A rectangle around the thing. This is the most reliable, and it is what a detector gives you. |
Clicks([(x, y), ...]) |
One or more points on the thing, as from a mouse. A second list of 1s and 0s can mark each point as on the thing (1) or deliberately off it (0). |
| a mask | A True/False array the size of the frame, if you already have the outline from somewhere else. |
Start tracking something. Pass its name and where it is, together with the frame it is in. You do this once. Everything already being tracked carries on undisturbed.
engine.step(frames, {"yard": {"dog": Box(300, 380, 420, 470)}})Stop tracking something.
engine.drop("door", "courier") # just the courierengine.drop("door") # everything on the door cameraFix a track that has gone wrong. Now and then an outline drifts. It slides onto the wrong person, or keeps only half of the object. Point at the object again under the same name, exactly as you did to start it. Tracking begins afresh from the new box, and the old, mistaken history is thrown away.
engine.step(frames, {"door": {"courier": Box(455, 130, 600, 500)}}) # same name, new boxStart something you noticed in the frame you just processed.
step()moves every tracked object forward by one frame. Suppose a detector looked at the frame you just passed in and found a dog. Callingstep()again with that same frame would move everything else forward twice.add()starts the dog on that frame and leaves the rest alone.engine.add(frames, {"yard": {"dog": Box(300, 380, 420, 470)}}) # frames: the ones you just gave step()
Tracking by name
Everything above starts from someone pointing at the thing. A Tracker takes words instead. Give it a phrase, or several, and it finds every match on every camera, follows each one, picks up new ones as they appear and lets go of those that leave.
One live camera on one RTX 4090. The words are set while it runs, and nobody points at anything. Watch the 17-second clip, read how it was recorded, or try it in Colab.
from tamserve import Engine, Tracker, build_video_predictor
engine = Engine(build_video_predictor("cuda", "sam2.1-large"), compile=True)
tracker = Tracker(engine, {"door": ["person", "package"], "yard": "dog"}) # loads SAM 3, the model that finds things
while True:
frames = {name: camera.take() for name, camera in cameras.items()} # as in the live loop above
masks = tracker.step(frames) # masks["door"] is {1: mask, 2: mask, 3: mask}: one mask for everything found on the door camera
# tracker.names("door") is {1: "person", 2: "person", 3: "package"}: what each of them is
For the same phrase on every camera, Tracker(engine, "dog") is enough. Each thing gets a number, counted from 1 on each camera, and keeps it for as long as it is followed.
It works by pairing two models. SAM 3 is Meta's model that outlines whatever you name. It is accurate and slow: one look at one frame takes about 65 ms on an RTX 4090. The model inside the engine cannot find anything by itself, but it follows what it is given far more cheaply, a few ms a frame with EdgeTAM and a few tens with SAM 2.1 large. So SAM 3 looks at each camera once every three frames, the engine follows what it found in between, and the looks are spread out so that they never all land in the same round.
SAM 3 can also follow things through a video by itself. Here are the two side by side on one RTX 4090, on 117 videos from Meta's own benchmark for SAM 3, each paired with phrases such as "the white scarf" (224 pairs, none of them used to choose any setting). The security cameras are a second set of videos:
| SAM 3's own tracker | Tracker |
|---|---|
| Accuracy (HOTA) | 59.4 | 56.3, which is 95% of it |
| GPU time per frame | 101 ms | 41 ms |
| Following with EdgeTAM, the smaller model | 54.6 at 26 ms |
| Frames per second, when there is something to follow | 8, one video at a time | 15 on one camera, or 1 on each of 16; twice that with EdgeTAM |
| Several things asked of one camera | a full pass for each | one look for all: 30 ms each at two a camera, 17 ms at eight |
| Security cameras that do not move, five things asked of each | 116 ms for each | 13 ms for each, with still=True |
| GPU memory | grows with the video | 6.8 GB for 16 cameras, the same after 19 minutes |
HOTA is the usual score for tracking, from 0 to 100. It combines how well things are found with how well each one keeps the same number.
So Tracker gives up a few points of accuracy and costs two fifths as much, or a quarter with EdgeTAM. It also does three things SAM 3's tracker cannot: it runs many cameras at once, it asks one camera for several things at little extra cost, and its memory does not grow with the video. Results has the full tables.
| if you want | do this |
|---|---|
| a lower cost, for a little accuracy | follow with EdgeTAM, the engine's default model: Engine(build_video_predictor("cuda"), compile=True) |
| cameras that do not move to cost less | Tracker(engine, prompts, still=True). A camera looks again only when its picture has changed, and is not followed while nothing moves. On security-camera video that cut the cost by two fifths and moved the answers by about a point. |
| new things found sooner | Tracker(engine, prompts, look_every=1). It costs about twice as much and does not raise accuracy. |
| something out of sight to keep its number for longer | Tracker(engine, prompts, hold=900): the number of frames a thing no look has seen is still followed. The default is 300. Everything seen in that time is still being followed, so on a busy camera a shorter hold costs less. |
Tracker and PeopleTracker are add-ons. They need Meta's SAM 3, which does not come with tamserve unless you ask for it:
pip install "tamserve[sam3]"
hf auth login # once: SAM 3's weights are gated
Before the weights will download, accept Meta's licence on the facebook/sam3 page on Hugging Face. They are 3.5 GB and are fetched the first time a tracker starts. SAM 3 has its own licence, which is not Apache-2.0, so read it before you build on it.
People are what cameras are most often asked about, so they have a tracker of their own. A PeopleTracker works as a Tracker does, with rules tuned on people: it looks five times less often, and it is both more accurate and far cheaper than SAM 3's own tracker.
from tamserve import Engine, PeopleTracker, build_vide
Comments
No comments yet. Start the discussion.