Reddit - r/MachineLearning 1h ago

LingBot-Vision: masked boundary modeling for self-supervised pretraining (0.296 NYUv2 linear-probe RMSE at 1.1B vs 0.309 for DINOv3-7B, trails on ImageNet); weights in 4 sizes[R]

Method Overview

The idea: instead of masking random patches and hoping boundary structure emerges, the teacher predicts a dense boundary field online and the boundary-bearing tokens are forced into the student's mask. The student must reconstruct exactly the regions that can't be inferred by copying context. The boundary targets come from the teacher itself rather than labels or an external edge detector.

Key Design Choices

Two design choices that look load-bearing:

Boundary fields are recast as per-pixel categorical distributions so the geometric branch can reuse the centering/sharpening machinery that keeps self-distillation from collapsing (continuous regression targets drift under an EMA teacher).
Decoded segments pass an a-contrario validation test before they're allowed to supervise anything.

Reported Results (All Self-Reported, Images)

They report the best NYUv2 linear-probe RMSE of their comparison (0.296 at 1.1B/patch-16 vs 0.309 for DINOv3-7B), with segmentation on par with the distilled DINOv3 ViT-H+. The distilled ViT-L (0.3B) lands at 0.310 NYUv2, basically the 7B's number.

Data budget per the report: 161M images, less than a third of DINOv3's samples.

Where It Loses

ImageNet classification trails at giant and L scale (their B/S students lead their class on linear probe).
ADE20K trails the DINOv3 family.
KITTI favors the bigger models.

Encoder-Initialization Study

The encoder-initialization study (last image) is the part I find hardest to dismiss: the exact same depth-completion pipeline trained on the same data, only the init swapped. The LingBot init wins across the board at ViT-L and on most benchmarks at ViT-g (they concede DINOv2 keeps an edge on the Hammer captures), and the data-scaling curve shows the gap growing rather than washing out as training data grows.

Caveats and Open Questions

What I'd want before treating the DINOv3 comparison as settled: they do run all baselines under one probe protocol, which helps, but a 0.013 RMSE delta is within what probe LR/resolution choices can produce, and there's no ablation against learned/hard-masking baselines (ADIOS/AttMask-style), which seems like the natural comparison for "mask the hard tokens". Checkpoints are public so the probes are cheap to rerun. Given the eval complaints around Ant's Ling-1T release, I'd treat the numbers as unverified until that happens.

One thing I can't square: DINOv3 needed Gram anchoring to stop dense-feature degradation over long schedules, and this method keeps it, so boundary forcing looks complementary rather than a replacement. Anyone read it differently?