50 Embodiments, 497 Episodes, One Indexed Dataset: Curating the LeRobot Community Dataset with FiftyOne

A single prompt to an AI agent running FiftyOne Skills pulled 10 episodes from every robot embodiment in the LeRobot community dataset, computed video embeddings with Qwen3-VL, and built a queryable similarity index. Here's the code and the thinking behind it.

The LeRobot community dataset is one of the largest community-built pools of cross-embodiment robot data: 235 contributors, 50 robot types, ~900 GB. Hugging Face's dataset card lists 46 robot types across 791 cleaned datasets. The 50 here come from scanning robot_type in all 1,755 contributor folders, so the two counts cover different sets of datasets.

It's organized as 1,755 contributor datasets, one per folder, so picking a balanced slice starts with reading each one's metadata. If you're training a policy that has to generalize across robots, you need episodes from every embodiment and a way to see which clips are actually worth keeping.

Now that FiftyOne natively supports LeRobot, I figured I'd use the FiftyOne skill to import LeRobot datasets and check it out. All it took was a one-sentence prompt:

"i want to have a little bit from every embodiment, i have enough disk space for the whole dataset... but let's take 10 episodes from each embodiment to start"

That sentence kicked off a session that read every contributor dataset's metadata, found 50 unique robot types, downloaded the right shard files for each, imported 497 episodes into FiftyOne, embedded every clip with Qwen3-VL, computed a similarity index, and saved 75 queryable views. All without writing a data loader, a Parquet parser, or a video decode loop.

One sentence was enough because the agent had FiftyOne Skills installed. Skills are packaged workflows that teach AI assistants how to do complex FiftyOne tasks: importing LeRobot episodes, running zoo models, computing embeddings, finding duplicates. The agent didn't have to guess at the LeRobot v3 layout or the FiftyOne API. It followed the skill.

Install them in Cursor, Claude Code, Codex, or Gemini CLI with one command:

curl -sL skil.sh | sh -s -- voxel51/fiftyone-skills

This is what FiftyOne is for. Not visualization. Dataset curation at the scale that robot learning actually requires.

New here? FiftyOne is an open-source toolkit for loading, querying, visualizing, and curating ML datasets - images, videos, and multimodal data. LeRobot is Hugging Face's framework for robot learning: it defines a standard v3 dataset format (Parquet + MP4 shards with episode metadata) that the community uses to share teleoperation recordings.

Key takeaways

• lerobot/community_dataset_v3 is a collection of 1,755 contributor datasets across 50 robot embodiments, each a complete LeRobot v3 dataset with its own meta/info.json.
• Using the FiftyOne Skills dataset-import workflow, the agent scanned every */*/meta/info.json, grouped by robot_type, picked the smallest source dataset per embodiment with ≥10 episodes, and downloaded only the needed shard-0 files.
• Each FiftyOne sample is one episode - not a frame, not a video file - with fields robot_type, task, duration, fps, dataset_name, and a media_reference pointer into the local LeRobot directory tree.
• Qwen3-VL-Embedding-2B turns each episode (up to 32 frames from its time window) into one 2048-dimensional video embedding. The similarity index, Uniform Manifold Approximation and Projection (UMAP), uniqueness, and representativeness all run from that one field in under a minute.
• The same model embeds text into the same space, so the index answers text queries: type "folding a cloth" and get the same bimanual cloth-folding episodes back, with no labels involved.
• uniqueness scores range from 0.013 to 1.0 across the 497 episodes. 292 clips score below 0.1 because each embodiment's 10 clips come from one recording session. The two highest scores, the only ones above 0.7, both belong to single-frame episodes (0.02-0.03 s long): uniqueness flags broken recordings as readily as rare behavior.
• 75 saved views - 50 by robot type plus analytical cuts by frames per second (FPS), duration, robot category, task keyword, and curation intent - are stored in the dataset and load from the App's Views dropdown with no code.

What you need

To load and query the pre-built dataset (no GPU required):

pip install "fiftyone>=1.22.0" "huggingface_hub[cli]" transformers qwen-vl-utils torch torchvision

To re-run UMAP visualization yourself (Step 3):

pip install umap-learn

FiftyOne 1.22.0 is the first release with fo.types.LeRobotDataset. The dataset loads from the Hub with embeddings, similarity index, UMAP layout, and curation scores already computed.

Steps 1-2, Step 4, and episode-to-episode search in Step 5 run entirely on the pre-built data, with no GPU or model weights. Text search in Step 5 downloads Qwen3-VL-Embedding-2B (about 4 GB) to embed your query; it runs on CPU and is faster on a GPU. Step 3 is for readers building their own dataset who want to replicate the embedding pipeline.

How the LeRobot community dataset is organized

FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values.

lerobot/community_dataset_v3 is a collection of contributor datasets. Each one is a complete LeRobot v3 dataset in its own <contributor>/<dataset>/ folder:

community_dataset_v3/kuehnrobin/g1_stack_cube_left/meta/info.json
data/videos/villekuosmanen/pick_coffee_capsule_under_dome/meta/info.json
...

1,755 directories. 17,617 files. 897.5 GB total.

Each sub-dataset carries its own meta/info.json with its robot type, episode count, and FPS. fo.types.LeRobotDataset reads one LeRobot v3 dataset at a time, so the agent started by downloading all */*/meta/info.json files - a 56-second fetch - and then scanning them to build the plan:

# Counts across all 1,755 sub-datasets
Robot types (50 unique):
so100 899 datasets 34801 total_episodes
so101 312 datasets 7490 total_episodes
so101_follower 121 datasets 4414 total_episodes
so100_follower 120 datasets 3065 total_episodes
arx5 43 datasets 2108 total_episodes
Unitree_G1_inspire 1 dataset 7 total_episodes # only 7 available
so100MovellaDot 75 datasets 157 total_episodes # max 5 per source
...

All 1,755 sub-datasets were already at codebase_version: v3.0. No conversion needed.

Left: the contributor/dataset folder layout, with the per-dataset meta/info.json files the agent scans first. Middle: the four-step plan - scan, group by robot_type, pick one source dataset per embodiment, download. Right: inside one source dataset, only file-000.parquet and file-000.mp4 (the first ~10 episodes) are fetched; the other shards are skipped.

Good to know. What's a shard-0 set? LeRobot v3 packs many episodes into shared MP4 and Parquet shards. file-000.parquet and file-000.mp4 hold the first batch of episodes for each dataset. Downloading only those files makes a 10-episode subset cheap, even from a 50-GB source dataset.

Step 1: load the dataset

The agent session did the heavy lifting - reading every contributor dataset's metadata, picking the right shard files, importing 497 episodes, computing embeddings, and pushing the result to the Hub. The import step is covered in How to Import LeRobot Datasets into FiftyOne Using Agent Skills.

The dataset is at Voxel51/community_v3_10per_embodiment. Load it:

import fiftyone as fo
import fiftyone.utils.huggingface as fouh
ds = fouh.load_from_hub("Voxel51/community_v3_10per_embodiment")
print(len(ds)) # 497
print(ds.distinct("robot_type")) # 50 embodiments
print(ds.get_field_schema()) # episode_index, robot_type, task, duration, fps,
# dataset_name, media_reference,
# qwen3vl_embedding, qwen3vl_embedding_camera,
# uniqueness, representativeness, ...

From here on, the source videos live in the Hub repo alongside the metadata. FiftyOne reads through the media_reference pointer - nothing to download separately. And because the unit is an episode rather than a file, every query you run is a dataset-level question, not a file-browser search.

What "one sample = one episode" actually means

A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference.

Most tools treat robot data as video files or frame sequences. FiftyOne takes a different unit: the episode. Every sample in this dataset carries:

• sample.episode_index # int - 0..9 within the source dataset
• sample.robot_type # str - e.g. "arx5", "so100", "Unitree_G1_Dex3"
• sample.dataset_name # str - e.g. "villekuosmanen/pick_coffee_capsule_under_dome"
• sample.task # str - e.g. "Pick up the coffee capsule"
• sample.duration # float - seconds (0.02 to 148.6 in this dataset)
• sample.fps # float - 10, 15, 20, 25, 30, or 50
• sample.media_reference # LeRobotEpisodeReference - a FiftyOne type with two fields:
  • .data = [chunk_index, file_index, first_row, last_row] (Parquet)
  • .videos = {camera: [chunk_index, file_index, from_ts, to_ts]} (MP4)

Think of a FiftyOne sample as a row in a table: the left side has filterable scalar fields (robot_type, task, duration, fps) and the right side has the media_reference pointer. The pointer's two values - data[2:4] and videos[top][2:4] - map to the actual Parquet rows and MP4 time window in the LeRobot folder on disk.

A FiftyOne sample (left) with filterable scalars above the line and the media_reference pointer below it. Arrows connect the pointer's two values - data[2:4] and videos[top][2:4] - to the actual Parquet rows and MP4 time window in the LeRobot folder on disk.

Three roads lead from a FiftyOne view back to frame data:

• Read in place via sample.media_reference.data - no copy, your code, one episode at a time.
• Torch dataset via view.values("episode_index") passed to LeRobotDataset(..., episodes=...) - no copy, for training loops. episode_index restarts at 0 in every source dataset, so filter the view to one dataset_name first; for a mix of sources, export instead.
• Export via view.export(dataset_type=fo.types.LeRobotDataset) - copies data, produces a standalone v3 repo. Each export must come from a single source dataset, so filter by dataset_name first.

The rule: in-process → don't copy. Out-of-process → copy.

Three ways to get frames from a FiftyOne view of LeRobot episodes, and when each one copies data.

From inspection to curation

Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid.

Timeline viewers are built for inspecting one recording at a time: open a log, scrub the timeline, and watch camera, inertial measurement unit (IMU), and joint streams in sync. That's the right tool when you're debugging one specific run.

FiftyOne starts where that ends. Once a run is recorded and you have hundreds of episodes, the question isn't "what happened in this episode?" It's "across all 497 episodes from 50 different robots, which clips are visually unique, which are near-duplicates, which ones actually belong in my training set?" A timeline scrubber can't answer that. A queryable, indexed dataset can.

The rest of this post shows exactly what that looks like - every step from here on is something you can't do in a timeline scrubber.

Step 2: what's already on the dataset

The Hub dataset ships with a video embedding for every episode, computed with Qwen/Qwen3-VL-Embedding-2B from the qwen3vl_embeddings remote zoo source. Each episode's time window is decoded with PyAV; up to 32 evenly spaced frames go to the model as one video clip, producing a 2048-dimensional qwen3vl_embedding field. qwen3vl_embedding_camera records which camera stream the frames came from, since camera names and layouts differ across the 50 embodiments.

What Qwen3-VL-Embedding actually produces, and why it matters

Qwen3-VL is an open vision-language model family from Qwen/Alibaba. It comes in two variants: Qwen3-VL-Instruct (generates text - captions, bounding boxes, answers) and Qwen3-VL-Embedding (trained for retrieval, so it produces dense vectors directly). The embedding model reads the clip, and the hidden state of the last token becomes the vector, L2-normalized:

# what produced qwen3vl_embedding (inside the qwen3vl_embeddings zoo model)
outputs = model(**inputs) # frames + instruction
embedding = outputs.last_hidden_state[row, last_token_position] # one 2048-d vector per clip
embedding = F.normalize(embedding, p=2, dim=-1)

Top lane: an episode's time window is decoded with PyAV, up to 32 evenly spaced frames are tokenized by the vision encoder and run through the language model, and the last token's hidden state becomes one 2048-d vector stored as qwen3vl_embedding. Bottom lane: at query time, a text query goes through the same model and lands in the same space, which is what makes text-to-video search work.

This model is useful for robotics data because video and text map into the same vector space. The similarity index qwen3vl_embedding_sim keeps the model name, so it accepts text queries as well as episodes:

ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim")

• no labels, no metadata matching, just the model's understanding of what was in the clip. For Vision-Language-Action (VLA) model development, this is specifically useful at the data curation stage. A VLA policy needs to generalize