NVIDIA Research Bets on Code, Not Tool Calls, to Fix AI Spatial Reasoning

NVIDIA Research Bets on Code, Not Tool Calls, to Fix AI Spatial Reasoning

NVIDIA's SpatialClaw uses code, not tool calls, to boost AI spatial reasoning by 11.2 points across 20 benchmarks and six model sizes.

NVIDIA Research has released SpatialClaw, an open-source framework that rethinks how AI agents handle one of the hardest problems in computer vision: determining where things are in physical space. The project, published by NVIDIA's research labs and hosted on GitHub under the NVlabs account, targets a long-standing weakness in vision-language models (VLMs). These models are good at describing what they see, but they tend to struggle with the geometry of a scene: how far apart two objects sit, which way something is facing, or how an object moves across multiple video frames.

SpatialClaw doesn't try to retrain a model to fix that. Instead, it changes the interface the agent uses to reason about space.

Code as the Action, Not a Tool Call

Most spatial reasoning agents today take one of two approaches. Some commit to a single pass of code execution before seeing any results, locking in a strategy upfront. Others rely on a fixed set of structured tool calls, which limits how freely the agent can mix and match operations for a given task.

SpatialClaw instead treats code itself as the action interface, allowing a VLM-backed agent to write Python in a persistent Jupyter kernel preloaded with perception primitives and standard scientific libraries. The kernel comes with tools like SAM3 for segmentation and Depth-Anything-3 for 3D reconstruction, along with familiar libraries such as NumPy, SciPy, and Matplotlib.

The agent writes one executable cell at a time, sees the output, and decides what to do next. Each cell can build on the tool outputs from previous steps, inspect the evidence gathered so far, and revise its analysis before committing to a final answer.

That step-by-step, code-native loop is the core bet behind the project: that an agent reasoning about 3D space benefits from the same flexibility a human data scientist has when poking around in a notebook, rather than being boxed into a rigid menu of tool calls.

The Numbers Behind the Claim

NVIDIA tested SpatialClaw across 20 spatial reasoning benchmarks and reported an average accuracy of 59.9%, an 11.2-point improvement over the previous best spatial agent. Notably, the team says this held up when using the same system prompt, tool set, and hyperparameters across all benchmarks and six different VLM backbones, ranging from 26 billion to 397 billion parameters.

That consistency matters. A framework that only works well when hand-tuned to a specific model or dataset is far less useful in production than one that generalizes.

According to coverage from MarkTechPost, SpatialClaw was evaluated across five benchmark categories spanning single-image, multi-view, general, video, and 4D understanding, and it outperformed the no-tool baseline on all six backbones tested. The outlet also noted that the framework outperformed a recent comparison agent, referred to as SpaceTools, by the same 11.2-point margin.

For teams building agentic systems that need to interact with the physical or visual world - robotics, autonomous inspection, AR/VR, and video analytics - that kind of consistency across model sizes is the more interesting headline than the raw accuracy number.

Why This Matters for AI-Native Engineering Teams

Spatial reasoning has quietly become one of the bottlenecks for the next wave of agentic AI. A chatbot doesn't need to know how far apart two warehouse shelves are. A robotics agent, a drone inspection system, or an AR assistant absolutely does.

SpatialClaw is a sign that the fix for that gap might not be a bigger model or more training data; it might just be a better way for the agent to think out loud. It's also a useful data point for engineering teams evaluating how to architect their own agents. The training-free angle means SpatialClaw can, in theory, sit atop an existing model rather than requiring a costly retraining cycle. That's the kind of incremental, bolt-on capability that tends to find its way into production faster than approaches that require rebuilding a model from scratch.

"Where an agent's reasoning happens matters as much as the model behind it," said Mitch Ashley, VP and practice lead for software lifecycle engineering and AI-native software engineering at The Futurum Group. "The gains here come from composing perception tools in code, each step conditioned on the last, which positions the action interface as a primary lever for spatial reasoning."

Ashley added that the implications go beyond this one framework. "For teams building agents that act on the physical or visual world, capability becomes an architecture decision they own rather than a model property they wait on. The interface an agent reasons through gates how far it can go."

The project is now available on GitHub under the NVlabs organization, with setup instructions for running it on standard spatial reasoning datasets or on self-hosted models via vLLM. As with most NVIDIA Research releases, the real test will be whether the broader agentic AI community adopts it and builds on it, or whether it remains a research curiosity. Given how much attention spatial reasoning is getting right now, this one seems likely to get a second look.