A Drone Remembers Everything It Saw. Now You Can Watch It Happen Again.

FiftyOne multimodal viewer playing back a UAV MCAP file with 3D LiDAR point cloud, stereo camera, GPS map, and odometry plot tiles synced to one clock.

FiftyOne 1.19 can now open MCAP files natively - camera, LiDAR, GPS, IMU, and logs, all on one shared clock, no ROS install required.

A UAV flies under the canopy of a Virginia forest for four and a half minutes. In that window, it isn't recording one thing - it's recording thirty-one things at once. Stereo cameras. A LiDAR sweeping 64 channels at 1024 points each. GPS fixes. An IMU. Onboard diagnostics. Two competing estimates of its own position, one live, one refined after the fact.

Every one of those streams ticks on its own clock, at its own rate, into a single append-only log file. That file is real: it's VAT-0723U-AUTO-01_episode.mcap, one sample in a dataset we built for this post, and by the time the flight is over, it holds 599,698 messages across 31 channels.

Nothing about that file tells you what a "frame" is. There's no sample, no image, no obvious unit at all - just a firehose of timestamped messages on parallel channels, exactly as the robot's middleware wrote them.

Historically, opening that file meant leaving your dataset tool behind. You reached for rosbag, or Foxglove, or rviz - because whatever held your curated samples had no idea what to do with a heterogeneous channel firehose. As of FiftyOne 1.19, that's no longer true. Point a Sample at a .mcap file, and FiftyOne opens it: cameras, point clouds, GPS tracks, IMU plots, and logs, all synced to one playback clock, inside the same App you already use for images and video.

Key Takeaways

• MCAP is a self-describing log container, not a dataset format - every message's schema is encoded in the file, so a recording remains decodable years after the code that wrote it is gone.
• Point a FiftyOne Sample's filepath at a .mcap file, and dataset.media_type is automatically inferred as "multimodal" - no importer class, no config, no media-type flag.
• The App's multimodal viewer renders six tile types - Image, 3D, Map, Plot, Logs, and Message - all synced to one shared playback clock, so scrubbing the timeline moves every tile at once.
• Built-in decoders support both ROS 1 and ROS 2 message encodings, plus Foxglove's protobuf/CDR schemas, with no ROS installation required on your machine.
• In the 10-episode UAV LiDAR dataset built for this post, one autonomous flight (VAT-0723U-AUTO-01, ~4.5 minutes) alone carries 31 channels and nearly 600K messages across stereo cameras, LiDAR, IMU, GPS, and logs - all playable from a single sample.
• Anything FiftyOne doesn't have a built-in decoder for - raw Ouster LiDAR packets, custom drone-autonomy messages - still opens in the Message tile as a raw, collapsible record tree, so nothing in the file is ever invisible, even unrecognized data.

What is MCAP, actually?

FiftyOne App showing an MCAP recording as 3D point cloud, forest stereo camera, and GPS map tiles side by side.

Strip away the robotics branding, and MCAP is a log container, not a dataset format. It's the modern successor to ROS 1's .bag and the SQLite-backed rosbag2, and it's been the default output of ros2 bag record since the ROS 2 Iron release. Its entire reason for existing is recording timestamped publish/subscribe message streams to disk, fast and durably, in a format that survives a crash mid-write.

Four record types carry the whole mental model:

• Schema - a serialization-agnostic message definition (ROS msg, protobuf, JSON schema, FlatBuffers). This is what makes the file self-describing.
• Channel - a named stream, essentially an ROS topic. Each channel maps to a single schema and specifies its message encoding.
• Message - one record: a channel ID, a sequence number, a log time, a publish time, and an opaque serialized payload.
• Chunk (plus index records) - messages are batched into compressed chunks with a summary section at the end, so a reader can seek by time or by channel without scanning the whole file.

The hierarchy is File → Channels → time-ordered Messages → decoded via Schemas. The primary axis is time, and the data on that axis is heterogeneous by design - camera frames on one channel, LiDAR sweeps on another, IMU, GPS and diagnostics on others, all interleaved by timestamp with no shared "frame" concept tying them together.

Good to know. Why does the schema live inside the file instead of in code? Because then, months or years later, any tool without access to the original ROS packages can still decode every message. That's exactly why a raw ouster_ros/msg/PacketMsg blob and a self-describing sensor_msgs/msg/PointCloud2 message are both fully readable at the byte level in our dataset - even though, as you'll see below, only one of them means anything to a generic viewer without vendor-specific decode logic.

Why your dataset tool and your robot's memory used to be two different tools

A FiftyOne dataset's unit is the sample - a piece of media with fields and labels. An MCAP file has no notion of a sample at all. It has channels, ticking independently, and the question "what even is one data point here?" has no answer baked into the format.

Turning that channel firehose into curated training samples - deciding which channels sync into which moment, and what time tolerance counts as "the same moment" - is a real, separate data-curation problem, and it still is one.

What changed in FiftyOne 1.19 is narrower and more immediately useful: you no longer need a second tool just to look at what's in the file. The multimodal viewer reads the MCAP directly via byte-range reads - no server-side conversion, no pre-processing step - discovers its channels and schemas, and renders whatever it can decode in a tiled, scrubbable viewer.

Keep it honest. Deciding which recorded moments become labeled training samples is still a modeling decision you have to make yourself; FiftyOne's multimodal viewer doesn't make it for you. What it gives you is the ability to see everything a recording contains across every channel before you make that call - without spinning up Foxglove or rviz alongside your dataset tool.

Meet the dataset: 10 UAV flights through a Virginia forest

Multimodal viewer with a GPS Map tile zoomed to the VAT-0723 site in Virginia, alongside LiDAR and stereo camera tiles.

TreeScope is a LiDAR dataset for precision agriculture and forestry, collected by the GRASP Laboratory at the University of Pennsylvania with UAV and mobile-robot platforms across six forest and orchard sites. Its original release ships raw and processed ROS 1 bags, HDF5 semantic labels, and field-measured tree diameters - built for benchmarking segmentation and diameter-estimation algorithms, not for showing off a multimodal viewer.

For this post, we built a 10-episode subset from one TreeScope site, VAT-0723 (Appomattox-Buckingham State Forest, Virginia), where each sample is one continuous UAV flight, merged from that flight's raw and processed ROS 1 bag pair into a single .mcap file.

VAT-0723 was picked deliberately: it's the one TreeScope site we could confirm still has real GPS (NavSatFix) data in its released bags, which is what makes the Map tile demonstrable at all - the TreeScope paper mentions GPS hardware broadly, but it isn't present in every site's public release.

Four episodes are manually piloted flights; six are autonomous. Durations range from 36 seconds to 6.4 minutes. Nine of the ten have every tile type populated with real, decodable data - the tenth (VAT-0723U-01) genuinely lacks camera and IMU topics, and we kept it in rather than drop it, because a dataset where every sample has every sensor isn't realistic.

Keep it honest. This is a 10-episode slice from one site, not the ~2.2 TB, six-site TreeScope corpus, and it does not include TreeScope's own semantic segmentation or diameter ground truth - the timestamps for those labels don't overlap with the specific episodes we chose.

What's attached instead is schema-derived metadata (has_image, has_gps, and so on) that describes what each episode's MCAP actually contains, not annotations.

Step 1: Point FiftyOne at your .mcap files

import fiftyone as fo
dataset = fo.Dataset("vat-0723-uav-flights")
dataset.add_samples([
    fo.Sample(filepath="/data/VAT-0723U-01_episode.mcap"),
    fo.Sample(filepath="/data/VAT-0723U-AUTO-01_episode.mcap"),
    # ...
])
print(dataset.media_type) # multimodal

That's the entire import. No dataset-type argument, no schema mapping. FiftyOne infers media_type: "multimodal" straight from the .mcap extension, the same way it infers image from a .jpg.

Step 2: Let FiftyOne tell you what's actually decodable - don't guess from topic names

FiftyOne Topics sidebar listing an MCAP file's channels and schemas next to a point cloud and stereo camera tiles.

A recording's topic list is tempting to pattern-match on, and it will burn you. While building this dataset, an early version tagged VAT-0723U-01 as having IMU data because its topic list includes /os_node/imu_packets - but that topic is ouster_ros/msg/PacketMsg, a raw undecoded Ouster UDP blob, not a real sensor_msgs/msg/Imu stream. It will never render in the Plot tile, no matter what its topic name suggests.

The fix is checking schema names, not topic names, and storing the result as ordinary sample fields:

IMAGE_SCHEMAS = {"sensor_msgs/msg/CompressedImage", "sensor_msgs/msg/Image"}
IMU_SCHEMAS = {"sensor_msgs/msg/Imu"}
for sample in dataset:
    schemas = set(sample.schemas) # pulled from the MCAP summary at import time
    sample["has_image"] = bool(IMAGE_SCHEMAS & schemas)
    sample["has_imu"] = bool(IMU_SCHEMAS & schemas)
    sample.save()

Every episode in this dataset carries has_image, has_pointcloud, has_gps, has_imu, has_logs, and has_unrecognized_schema fields built this way - which is also what makes it possible to query the dataset for exactly the episodes worth demoing (more on that below).

Step 3: Open the tiled viewer and wire up tiles

Add Tile menu open in the FiftyOne multimodal viewer, listing Image, 3D, Map, Plot, Logs, and Message tile types.

Open any multimodal sample, and the App gives you a mosaic-style viewer built from tiles - Image, 3D, Map, Plot, Logs, and Message - that you add, remove, resize, and rearrange. The Add tile button (grid icon at the top of the viewer) lets you pick a type; the left sidebar's Topics tab lists every channel in the recording, grouped by category, so you can see at a glance which tiles a given topic can feed into.

Bind each tile to a stream, and every tile shares one playback clock. Scrub the timeline once, and the Image tile's camera frame, the 3D tile's point cloud, and the Map tile's GPS marker all advance together - because underneath, they're all being read from the same set of timestamped messages in the same file.

Step 4: Scrub the timeline and watch it stay honest

Plot tile charting IMU linear acceleration and odometry height over a UAV flight, with a playhead tracking the scrubbed timeline.

On VAT-0723U-AUTO-01, scrubbing forward moves through a real ~4.5-minute autonomous flight: the /ublox/fix GPS track moves across a small span of the Appomattox-Buckingham forest (latitude 37.45492-37.45507, longitude −78.61621 to −78.61609, altitude 173-186 m), the stereo /ovc/left and /ovc/right cameras update frame by frame, and the /tree_cloud channel plays back 453 organized 64×1024 point-cloud sweeps - the same resolution as the Ouster OS1-64 that captured them.

Add a Plot tile on /ovc/vectornav/imu's linear_acceleration, and the y-axis sits at a median of about −9.3 m/s² for most of the flight - gravity, projected onto that axis given how the IMU is mounted - then swings sharply during real flight maneuvers.

Bind a second Plot tile to both /Odometry and /refined/Odometry's pose.pose.position.z, and you're watching two independent height estimates of the same flight: the live onboard estimate and the offline, Faster-LIO-refined one agree to within about 3 cm most of the time (median absolute difference), but momentarily pull apart by as much as 0.84 m at points where the offline correction has more to fix.

Step 5: Fall back to the Message tile for anything else

Four synced FiftyOne tiles during playback: LiDAR point cloud, two stereo cameras, and a GPS map advancing on one clock.

Every episode in this dataset also includes channels that FiftyOne has no built-in decoders for: raw ouster_ros/msg/PacketMsg LiDAR packets (undecoded UDP payloads straight off the sensor) and, on the autonomous flights, custom quadrotor-autonomy schemas like kr_mav_msgs/msg/PositionCommand and planning_ros_msgs/msg/VoxelMap.

None of that data is hidden. Bind a Message tile to any of those topics, and you get the most recent message at the current playhead as a collapsible record tree - every field, still inspectable, even though nothing about the schema was known to FiftyOne ahead of time.

Nine things to try in the multimodal viewer

1. The grid, at a glance. Load all 10 episodes, let the grid render its point cloud and camera previews, then use the stream selector to switch which channel each card previews. Follow with applying the full_multimodal_showcase saved view to filter down to the 9 episodes with full tile coverage - a clean before/after of curation by capability, not by eyeballing files.
2. Four tiles, one clock. Open VAT-0723U-AUTO-01, arrange Image ×2 (/ovc/left, /ovc/right), 3D (/tree_cloud or /os_node/segmented_point_cloud_no_destagger), and Map (/ublox/fix) tiles, then hit play. This is the single best "one file, every sensor, one clock" shot - cameras, point cloud, and GPS marker all advancing together over the ~4.5-minute flight.
3. IMU under a maneuver. Add a Plot tile on /ovc/vectornav/imu's linear_acceleration (x, y, z) on VAT-0723U-AUTO-01 and scrub to a section where the value swings well past its ~−9.3 m/s² gravity baseline - a visible flight maneuver on a live chart, with a playhead marker tracking the scrub position.
4. Onboard vs. refined trajectory. A second Plot tile c