Masked depth modeling with sensor-validity masking: reports best RMSE on 7 of 8 masked/sparse depth benchmarks, plus a controlled encoder-init study[R]

Core Idea

The core idea in masked depth modeling is to treat the sensor's own missing regions as the masking signal rather than using random block dropout. Specular highlights, transparent surfaces, and textureless areas where RGB-D cameras return no valid depth become the natural training target. The model therefore learns on exactly the failure distribution it faces at inference.

LingBot-Depth 2.0

Robbyant, an embodied AI company under Ant Group, describes this framing in LingBot-Depth 2.0. Version 2.0 changes nothing in the training recipe except the encoder initialization and data scale.

Encoder-Init Study

The encoder-init study is the clean experiment here: same MDM pipeline, same data curation, only the pretrained backbone swapped. Per the paper, the LingBot-Vision init wins on nearly every benchmark at ViT-L and on most benchmarks at ViT-g, with one concession: DINOv2 keeps an edge on the Hammer captures. The gap widens with data scale rather than washing out, per their scaling figure.

Benchmark Results

They report best RMSE on:

• 7 of 8 block-mask and sparse benchmarks
• 6 of 8 real camera configurations across three capture suites (Hammer D435/L515/ToF, ClearGrasp D415/D435, and their own D415/D435/D455 set)

They report the strongest numbers on the transparent-object ClearGrasp captures, with block-masked DIODE-Indoor RMSE roughly halving versus the 1.0 release.

Visuals and Availability

The attached images are screenshots from their paper (Tables 6, 7, 8 and a qualitative mirror/glass point-cloud figure); interactive point-cloud demos live on the project page.

Depth 2.0 weights are not released, so none of these completion numbers can be independently rerun. Only the four Vision backbones are open under Apache-2.0 and checkable at https://github.com/robbyant/lingbot-vision, which hosts the paper and the open weights. The renders shown come from the vendor's comparison page.

Open Question

Does sensor-validity masking beat random masking for other sensing modalities, say lidar or thermal? That would test how general the framing really is.