SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation

TL;DR: SimWeaver trains RGB-only vision-language-action policies purely in simulation — 200 synthesized demos per task, no teleoperation, no real-world fine-tuning, no per-task calibration — and deploys them zero-shot on a real bimanual robot manipulating T-shirts, silk, and plastic bags at 91% average success. The interesting part isn't any single trick; it's the claim, backed by ablations, about where the deformable sim-to-real gap actually lives: contact reliability in the solver, determinism in trajectory synthesis, measured (not tuned) fabric physics, and — most surprisingly — the image signal processor inside a $300 RealSense camera. Turn off the camera-noise-aware photometric augmentation and success collapses from 91% to exactly 0% on all five tasks.

Why this has been stuck

Rigid-object sim-to-real is basically a solved engineering problem: objects have poses, simulators handle rigid contact well, and pipelines like MimicGen synthesize training data by rigid-frame transformation of a few seed demos. Every ingredient breaks for cloth and bags.

A deformable object has no canonical pose. Its configuration space is enormous — folds, wrinkles, self-contact, multiple layers stacked against each other. Widely used simulators fail at exactly the interactions that matter: the paper shows Isaac Sim's PBD cloth cannot even hold a stable bimanual grasp on a T-shirt (0% grasp success in their benchmark), while a Newton VBD solver grasps fine but produces cloth-rigid penetration in 77.5% of trials and outright simulation explosions in another 22.5%. Trajectory synthesis via rigid-frame correspondence has nothing to anchor to. And the vision community largely gave up on RGB for deformables, retreating to depth and point clouds — which themselves fail on the very materials that make deformables interesting (dark, reflective, thin silk returns almost no depth signal).

The concurrent SIM1 paper showed RGB sim-to-real garment folding is possible, but with teleoperated demonstration seeds, a learned trajectory generator with post-hoc filtering, and expert material calibration per task. SimWeaver's pitch is removing all three dependencies simultaneously.

The four gaps, and how each is closed

Gap 1: the solver. SimWeaver builds on the authors' prior RGBench simulator rather than proposing a new one; the contribution is making it contact-reliable for closed-loop grasping. The clever bit is an "active collision region" around each gripper jaw. When a parallel-jaw gripper (with no tactile sensing) clamps down on multiple layers of thin fabric at near-maximal force, resolving self-collisions between layers inside the jaw is nearly impossible and produces jitter. So they simply disable cloth self-collision inside a small collision-forbidden zone around the jaws, and adaptively maintain a surrounding active region where the solver's untangling machinery must resolve all intersections before particles exit. The region contracts as the gripper closes (suppressing oscillation) and expands as it opens (giving room to untangle). It's a hack, but a principled one: it acknowledges that the physics inside a clamped jaw is both unresolvable and irrelevant, and spends solver effort where it matters. Result: 100% task success on repeated bimanual garment grasping where both baselines score 0%, at the lowest per-step cost of the three (4.44 ms vs 7.80/10.38 ms).

Gap 2: trajectory synthesis without teleoperation. Here's the conceptual reframe worth remembering. For deformables, manipulation semantics are determined not by keypoints but by their topological relations — grasping the same two cloth corners can fold or unfold depending on which pair you pick. SimWeaver-Syn encodes this as a labeled graph G=(V,E) on the canonical mesh: nodes V are semantic landmarks (corners, sleeve tips, bag strap tips), edges E are task-feasible bimanual grasp pairs. Selection is just

where \mathcal{F}(\text{obs}_t) is a set of closed-form geometric predicates on the current observation (is this landmark occluded from above? buried under another layer? reachable without crossing arms?) and S is a lightweight scorer. The graph does most of the work by construction — invalid pairs never enter the argmax. Because the predicates are closed-form and the planner (TOPP-RA + Pinocchio IK) is deterministic, the whole synthesizer is deterministic: no learned generative model, no teleoperation seeds, no post-hoc discriminator filtering. A closed-loop wrapper re-observes and retries on grasp failure, since grasp slippage under self-occlusion is the dominant failure mode.

The cost of this design: the graph must be hand-labeled once per asset class (a few minutes; automatic for symmetric assets like rectangular cloth), and per-task predicates/scorers presumably need some engineering. This is the "task-specific structure" that learned generators try to avoid — SimWeaver's bet is that a little structure buys enormous reliability, and the numbers back it up: 97.2% synthesis pass rate on T-shirt fold vs 24.0% for SIM1's learned sampler under matched physics, and a striking replay experiment — one successful trajectory replayed 100× from fresh resets succeeds 100/100 in their simulator vs 13/100 in SIM1's. Since the trajectory is fixed, that 87-point gap is pure contact-resolution nondeterminism: learned samplers produce trajectories sitting at the boundary of stable contact regimes, where numerical noise cascades into failure.

Gap 3: physics parameters. Instead of tuning simulator proxies per task, every asset is bound to a fabric class from an industrially measured library (ASTM cantilever bending, stretch tests, etc.). Because the solver's parameters correspond to real measurable quantities — mass per area, bending stiffness, friction — assigning physics is a one-time table lookup, not a calibration loop. This is what makes "no per-task calibration" possible, and it extends to single-image asset generation: a VLM classifies the fabric category of a generated mesh and samples measured properties for it.

Gap 4: the camera's ISP. This is the most unexpected finding. The team characterized their three RealSense D435i units under fixed manual exposure, gain, and white balance, and found two sensor-internal failure modes: per-unit color bias (identical settings, camera-dependent color casts spanning ~39 units of R−G difference on a 0–255 scale) and gain-loop drift (one camera's red channel drifting ~29 units peak-to-peak within a 5-second window, triggered by scene changes and not calibratable away). Their photometric augmentation ranges are fitted to these measured envelopes — hue ±0.15, per-channel jitter σ≈±13 units — rather than to literature defaults.

The ablation is brutal and clarifying: disable this augmentation and real-world success is 0% on all five tasks. Not degraded — zero. Meanwhile the policy tolerates cloth-state, lighting, and pose variation just fine. The visual sim-to-real gap for these policies is apparently dominated not by rendering realism or scene appearance but by the noise statistics of the camera's internal image signal processor. That's a genuinely useful diagnosis for anyone doing RGB sim-to-real.

The evidence

The main result: five tasks (snack-into-plastic-bag packaging, T-shirt fold, T-shirt flatten, silk flatten, bimanual silk grasp), one \pi_{0.5} policy per task fully fine-tuned on 200 sim demos, n=23 consecutive real trials each.

The generalization study on silk grasping is where sim data most clearly beats real data. Under distribution shifts in table texture, lighting, and object rotation, the sim-trained policy (200 demos + domain randomization) holds 100% while a policy trained on 100 real teleoperated demos with matched 2D image augmentation drops to 13.0%, 69.6%, and 8.7% respectively (all gaps significant, single-tail Fisher, α=0.05).

The honest reading, which the paper itself gives: this comparison isn't quite apples-to-apples. Texture and rotation randomization require programmatic scene control — you can't cheaply randomize table textures during real teleoperation. So the result isn't "sim data is intrinsically better"; it's "sim data gives you randomization axes real collection can't afford," which is the actual argument for simulation. In-distribution, sim also scales better per demo, which the authors attribute to deterministic synthesized trajectories being cleaner supervision than noisy operator-specific teleoperation — a plausible but underexplored claim.

Cost: 4.1 min per usable trajectory, 2824 usable trajectories/day, 0.03 each on an 8×4090 server — 4× SIM1's throughput at a third of its unit cost, ~90× cheaper than real collection (2.71/trajectory). The "per usable trajectory" accounting matters: SIM1's fast rollouts get filtered heavily post-hoc, inflating true unit cost.

What to be skeptical about

Small n, per-task policies. 23 trials per task give Wilson CIs of ±15–20 points; the headline 91.3% pools to [84.7, 95.2]. Five separate single-task policies, each with hand-configured task setups (per-task DR ranges, per-task synthesis skills like the multi-stage bimanual fling) — this is not a general deformable-manipulation policy, and the "no per-task calibration" claim applies to physics parameters, not to task engineering, of which there's plenty (see the appendices).

The generalization frontier is untested where it matters most. The triage question — does success hold on objects far from the asset-generation distribution? — is not answered. All real evaluations use objects matched to trained asset classes. The single-image asset pipeline exists but there's no experiment deploying on a novel bag or garment generated from a photo.

The DP3 point-cloud baseline failure is attributed, not proven. DP3 trained on the same demos fails on all five real tasks; the paper blames D435i depth on matte-black grippers and silk (with a convincing scan figure), but a scene with depth-friendly materials would be needed to separate "point clouds fail here" from "point clouds fail generally." To their credit, they say exactly this.

Cross-method comparisons favor the home team. SIM1's 24% pass rate is measured running SIM1's pipeline under "matched physics," and the 13/100 replay in SIM1's own simulator — fair on its own terms, but SIM1's whole system (its simulator + calibration + filtering) achieved 87% real T-shirt folding, so the head-to-head is about pipeline economics, not final capability.

One embodiment, one lab, single fabric library. Physics "measurements" come from a library the same group built (RGBench); the full industrial collection stays behind licensed access, which limits reproducibility of the measurement-backed claim.

Why it matters anyway

If the recipe holds, the practical implication is large: deformable manipulation data — the most expensive kind to collect — becomes generatable at $0.03/trajectory with better OOD robustness than real teleoperation, using RGB cameras on exactly the materials where depth sensing fails. The paper also reframes the sim-to-real problem usefully: the gap is not one monolithic thing but four separable failures (contact, determinism, physics parameters, camera ISP), each of which can be measured and closed independently. The ISP finding in particular — that a full task-level collapse hides inside per-unit camera color drift no amount of rendering realism would fix — deserves to circulate widely.

Most worth your time: Section 7.3 plus Appendix H (the replay-determinism experiment is the cleanest piece of evidence in the paper) and Appendix B (the D435i characterization and the 0% ablation). Section 4 is the read if you want the topology-graph synthesis idea in enough detail to reimplement.