NVlabs/SpatialClaw

TL;DR — SpatialClaw (NVIDIA + KAIST) is a training-free agent framework arguing that the right action interface for spatial reasoning is not structured tool-calls but a live Python kernel: the VLM writes one Jupyter cell at a time against pre-loaded perception primitives (SAM3, Depth-Anything-3, geometry utils, NumPy/SciPy/Matplotlib), inspects intermediate text and images, and iterates before committing an answer. The claim: 59.9% average across 20 spatial benchmarks, +11.2 points over the prior best spatial agent, with a single fixed prompt/tool set across six backbones from 26B to 397B.

The idea

The framing is the interesting part. Prior spatial agents come in two flavors: single-pass code generation (commit to a full analysis before seeing any result) or JSON-style tool calling (flexible turn-by-turn, but you can only compose what the schema allows — no arbitrary math on a depth map, no ad-hoc filtering of segmentation masks). SpatialClaw's position is that a stateful REPL is strictly better: the agent gets full expressivity of Python for composing tool outputs, plus turn-by-turn adaptivity because each cell executes in a persistent kernel and its stdout, new variables, and rendered images flow back as the next observation.

Concretely, per sample it runs a five-stage loop: a planner drafts a strategy; the main VLM emits one Python cell; the cell passes an AST safety check; it executes in the kernel (input frames and tools pre-loaded); results come back as observations, with reflection loops, until the agent calls ReturnAnswer(...). This is essentially "code interpreter, but with 3D perception tools mounted" — a natural convergence of the code-agent trend with spatial reasoning, and it's plausible it becomes the default pattern for this task family.

Evidence

The README gives one headline number: 59.9% average accuracy over 20 benchmarks, +11.2 points over "the recent spatial agent" (unnamed in the README — check the paper for the baseline). Benchmarks span single-image (ERQA, OmniSpatial), multi-view (MindCube, MMSI, SPAR-Bench), video/4D (VSI-Bench-U, DSI-Bench, etc.), and general video (Video-MME). The stronger claim is invariance: same system prompt, tool set, and hyperparameters across all 20 benchmarks and six backbones (Qwen3.5/3.6, Gemma4, 26B–397B). If that holds, it's evidence the interface itself carries the gain rather than per-task prompt tuning — the usual failure mode of agent papers. No per-benchmark or per-backbone breakdown appears in the README, so you can't tell where the gains concentrate without reading the paper.

What's actually in the repo

This is the full official implementation, not a stub: LangGraph agent runtime, persistent kernel management, AST safety checking, all 20 benchmark loaders as JSON configs, FastAPI GPU tool server (reconstruction + SAM3), vLLM auto-discovery with load balancing, and SLURM chain-job launch managers to reproduce the paper tables. No weights to release since it's training-free. Docs cover installation, experiment reproduction, monitoring, and architecture.

Caveats

• Infrastructure-heavy. Three coordinated services (vLLM backbone, GPU tool server, agent), submodule installs, two conda envs, ~15–30 min setup. It runs on a single machine without SLURM, but this is a research harness, not a pip-installable library.
• Cost. Multi-step code execution plus SAM3/depth calls per question means latency and compute per sample will be high. The README says nothing about steps-per-question or wall-clock cost — the main open question for practical use.
• Evaluation is benchmark-bound. All evidence is on the curated 20-benchmark suite; no robotics or embodied deployment shown despite the framing.
• License. NVIDIA Source Code License-NC — non-commercial only, plus a pile of third-party licenses (SAM3, Depth-Anything-3).
• Minor oddity: the quickstart example points at a gemini-3-pro.json model config, though the paper's claims are about Qwen/Gemma backbones.

Try it

git clone --recursive https://github.com/NVlabs/SpatialClaw.git
cd SpatialClaw
bash spatial_agent/scripts/setup.sh
cp .env.example .env   # add API keys, or point at self-hosted vLLM

python -m spatial_agent.entrypoints.run \
  --dataset spatial_agent/config/dataset/erqa.json \
  --model spatial_agent/config/model/gemini-3-pro.json \
  --concurrency 4

Project page with the paper PDF: https://spatialclaw.github.io