Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins

Key takeaways

• A FiftyOne plugin is a folder with two files, fiftyone.yml and __init__.py. FiftyOne discovers it at runtime, so you never fork FiftyOne or wait for a release to add a button, form, panel, or background job.
• FiftyOne reads .mcap, .bag, and .rrd files as media_type="multimodal" episodes and imports LeRobot v3 with fo.types.LeRobotDataset, so a plugin can score every episode in a dataset with the same code.
• FiftyOne plugin code runs on the FiftyOne server in Python so that it can use NumPy, SciPy, PyTorch, a GPU, or a secret API key. Long jobs run in the background through delegated execution.
• Most viewer extensions run in the browser against one recording, and labeling-platform hooks act on one workflow task. FiftyOne plugins run server-side Python with custom UI across a whole dataset.
• Two open-source plugins show the pattern for physical AI: lerobot_data_curation ranks LeRobot episodes worst-first, and demo_quality_scorer does the same for MCAP recordings.

Why physical AI data needs custom tooling

You have 400 teleoperation episodes and a policy to train by Friday. Some of those episodes are bad. The operator hesitated for eight seconds before the grasp. The wrist camera froze halfway through. The gripper opened and closed three times because the first grasp slipped. The task string says "pick up" and nothing else. A policy will learn from all of it.

You could open each episode in a log viewer and scrub through it. At two minutes per episode, that's more than 13 hours, and you'll stop paying attention long before the end. What you want is a ranked list, worst first, with every flag one click away from the moment it happened.

No viewer ships that list, because the checks depend on your robot. A dual-arm rig with multi-joint hands needs different smoothness metrics than a wheeled rover with an inertial measurement unit (IMU) and GPS. The tool that curates your data has to be one you can extend in Python, on your schema, across the whole dataset. That's what the FiftyOne plugin framework is for.

Image datasets are forgiving. Every sample is a picture, every label is a box or a mask, and the same quality checks work on most of them. Robot data has no such standard shape. One MCAP file from an autonomous vehicle might hold six cameras, two LIDARs, an IMU, GPS fixes, and a /tf tree. One LeRobot episode from a tabletop arm holds two camera streams, a 14-dimensional observation.state, a matching action array, and a task string. An egocentric dataset adds audio and hand poses.

The questions you need answered are specific to that shape:

• Is the motion in this episode smooth, or full of corrections?
• Did any sensor drop frames, drift its clock, or clip its values?
• Does the action track the state, or is there lag between video and action?
• Is the gripper behavior consistent with the other episodes of the same task?
• Is the language instruction usable for fine-tuning a vision-language-action (VLA) model?

A generic viewer can't answer those. It can only show you the data and wait for you to notice. FiftyOne gives you the dataset layer: episodes as samples, fields you can filter and sort on, views, tags, and an App with a multimodal viewer that plays MCAP and LeRobot streams in lockstep. Plugins give you the part that's specific to your robot.

What is a FiftyOne plugin?

A plugin is a folder FiftyOne finds. If the folder contains a fiftyone.yml, FiftyOne loads it at runtime and adds whatever it declares to the App, the SDK, and the CLI.

Plugins add two kinds of things:

• Operators are actions. You trigger one, it runs, it finishes. "Compute episode quality" is an operator.
• Panels are UI that stays open next to your data and reacts to what you do. A worst-first ranking that re-sorts when you filter the grid is a panel. Under the hood, a panel is a special kind of operator, so everything below about operators applies to panels too.

What changes when you promote a notebook script to a FiftyOne plugin operator:

• Who can run it: You, in your kernel vs. Anyone with the App open, no code
• Inputs: Variables you edit by hand vs. A typed form, validated before anything runs
• What it acts on: The dataset you loaded vs. The view or selection the user has open
• Long jobs: Your kernel is busy until it finishes vs. Delegated to a worker, so the App stays free
• Reuse: Copy and paste vs. One URI: buttons, panels, pipelines, notebooks, agents

Good to know. Why not keep it in a notebook? Start in a notebook. Promote the code to an operator when someone else needs to run it, or when you catch yourself running it a second time.

The row that matters most for robot data is "what it acts on". A script scores whatever dataset you loaded. An operator scores the view the user has open right now, so the same code works on one task, one robot, one week of collection, or the episodes someone tagged for review.

The anatomy of a FiftyOne plugin

Here's the folder for a plugin with a Python backend and a custom React panel: FiftyOne plugin folder with required fiftyone.yml and __init__.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App.

fiftyone.yml is the manifest. It holds the plugin name in @org/name form, the operators and panels it exposes, and any secrets it needs. __init__.py holds your classes and a register(p) function. requirements.txt lists pip dependencies. For MCAP work, that's usually mcap, mcap-protobuf-support and friends. Everything below the dashed line is optional JavaScript for custom UI. A Python-only plugin is two files.

Teal in every figure in this post means the FiftyOne server, which runs Python. Orange means the App in your browser.

Keep it honest. Every operator name listed in fiftyone.yml must match the name in that operator's config. If you register a class but forget to list it, the operator doesn't show up, and you get no error. It's the most common "where did my operator go" bug.

A complete operator in 25 lines

This operator tags samples whose detections fall below a confidence threshold:

import fiftyone as fo
import fiftyone.operators as foo
import fiftyone.operators.types as types
from fiftyone import ViewField as F

class TagLowConf(foo.Operator):
    @property
    def config(self):
        return foo.OperatorConfig(
            name="tag_low_conf",
            label="Tag low-confidence"
        )

    def resolve_placement(self, ctx):
        return types.Placement(
            types.Places.SAMPLES_GRID_SECONDARY_ACTIONS,
            types.Button(label="Tag low-conf"),
        )

    def resolve_input(self, ctx):
        inputs = types.Object()
        fields = list(ctx.dataset.get_field_schema(embedded_doc_type=fo.Detections))
        inputs.enum("field", fields, required=True, label="Label field")
        inputs.float("thresh", default=0.3, label="Confidence threshold")
        return types.Property(inputs)

    def execute(self, ctx):
        field, thresh = ctx.params["field"], ctx.params["thresh"]
        view = ctx.target_view().filter_labels(field, F("confidence") < thresh)
        view.tag_samples("low_conf")
        ctx.ops.notify(f"Tagged {len(view)} samples")

def register(p):
    p.register(TagLowConf)

Four methods, four jobs:

• config names the operator. name is its ID, and label becomes the form's heading.
• resolve_placement puts a "Tag low-conf" button in the grid's action bar.
• resolve_input declares a dropdown and a number. The App builds the form, with defaults and validation, from those two lines.
• execute does the work on the target view and sends a notification.

You wrote no UI code. You described the inputs and the App built the interface. Swap the body of execute for spectral arc length (SPARC) smoothness over an /odom channel, and you have the skeleton of an MCAP quality scorer.

How an operator call works

Every operator call is two round trips between the App and the FiftyOne server:

1. Resolve inputs. You click the button. The App sends the current context (dataset, view, selection) to the server, which calls your resolve_input and returns a schema. The App turns it into a form. If the operator is dynamic, this loop re-runs on every input change, which is how demo_quality_scorer shows a signal picker only after you choose a telemetry channel.
2. Execute. You hit Execute. The server validates the parameters against your schema, then calls execute. You return a result plus any UI operations, and the App applies them.

Your code runs only on the server. The browser draws what the server tells it to. That's why an operator can decode protobuf MCAP messages with mcap-protobuf-support, run SciPy's Welch power spectral density (PSD) on an IMU stream, or call a model on a GPU, without shipping any of that to the browser.

When something breaks, ask which arrow failed. A wrong form means resolve_input. A fine form that does nothing means execute. An execute that ran but didn't change the App means the UI operations you returned.

Long jobs run in the background with delegated execution

Scoring 400 episodes is not a click-and-wait job. Decoding video alone costs lerobot_data_curation about 0.4 seconds per camera per episode.

Set allow_delegated_execution=True in the operator's config, and the user can choose. Immediate execution runs inside the server while the App waits, with a live progress bar. Delegated execution writes the job to a queue in MongoDB, and a separate worker runs the same execute method. You start a worker with fiftyone delegated launch and track jobs with fiftyone delegated list. Both physical AI plugins below default to delegated execution, because a real corpus takes minutes.

Keep it honest. If no worker is running, delegated jobs sit in the queue forever. And a delegated job has no browser attached, so write results to the dataset rather than trying to change the user's view.

Panels: UI that stays open

A curation tool needs more than a form. It needs a place to see the ranking, click into an episode, and tag it. That's a panel.

A panel's on_load runs when it opens, render returns a layout, and events call your Python handlers with a fresh context. The event that matters most for curation is on_change_view: when the user filters the grid, your panel re-computes for the new view.

Here's a complete panel that shows class counts and follows the grid:

class ClassCounts(foo.Panel):
    @property
    def config(self):
        return foo.PanelConfig(
            name="class_counts",
            label="Class counts"
        )

    def on_load(self, ctx):
        self.refresh(ctx)

    def on_change_view(self, ctx):
        self.refresh(ctx)

    def refresh(self, ctx):
        counts = ctx.view.count_values("ground_truth.detections.label")
        ctx.panel.set_state("n", len(counts))
        ctx.panel.set_data("counts", [{"type": "bar", "x": list(counts), "y": list(counts.values())}])

    def render(self, ctx):
        panel = types.Object()
        panel.md(f"**{ctx.panel.get_state('n')} classes**")
        panel.plot("counts")
        return types.Property(panel)

Small values go in panel state, which your Python handlers can read. Large payloads like chart data go through set_data, which is write-only from Python. Settings you want to survive a reload go in the execution store, ctx.store. Results that belong to the data go in dataset fields, where you can filter and sort on them like any other field.

When the built-in widgets run out, you have two options. A hybrid panel keeps state and compute in Python and renders a React component you write. A JS panel puts React in charge of the UI and calls your Python operators whenever it needs the server. Both plugins below are JS panels backed by Python operators, which suits dense, interactive dashboards. If you build a React UI, VOODOO, the component library the FiftyOne App itself uses, makes it look native.

Operators compose

Every operator has a URI, like @voxel51/brain/compute_similarity, and a typed input schema. Anything that can call an operator can call yours. One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates.

That includes the FiftyOne Model Context Protocol (MCP) server, which exposes list_operators, get_operator_schema, and execute_operator to AI agents. An agent can find your quality scorer, read its inputs, run it on the episodes tagged review, and report back which ones failed sensor health. The schema validates the agent's JSON the same way it validates a person filling in the form.