dominik-schnaus/unpaired-rosetta

TL;DR: This is the official code for a paper (Schnaus, Dagès, Cremers, Wang, Isola — note Isola, of Platonic Representation Hypothesis fame) showing that you can align the embedding spaces of independently trained vision and language models with zero paired examples, using only the shared geometry of the two point clouds. A single orthogonal map, found by Wasserstein Procrustes with a smart initialization, is enough for coarse cross-modal retrieval and even unpaired text-to-image generation. It's the strongest empirical support yet for the idea that unimodal models converge to a common representational geometry.

Why this matters

The standard story is that cross-modal grounding requires massive paired corpora (CLIP: 400M image-text pairs). This work argues the pairing is largely redundant: if two embedding spaces have converged to similar shapes, the correspondence is already in the geometry, and you just need to solve a matching problem to recover it. That reframes paired data from "the source of grounding" to "a refinement signal," and it directly operationalizes the Platonic Representation Hypothesis into a usable tool. The same trick works beyond vision-language: they align text encoders (Natural Questions), two single-cell assays (PBMC), and — relevant to your BCI interests — fMRI responses across different subjects on NSD, i.e., unpaired cross-subject brain alignment.

How it works

The method is deliberately simple. Given two disjoint sets of embeddings (e.g., image embeddings from DINO, caption embeddings from a text model, no correspondence known):

1. Coarse geometric initialization — a Gromov-Wasserstein-style matching on pairwise distance structure gives an initial soft correspondence (Algorithms 1–2; the repo includes their MPOpt QAP solver and low-rank GW ablations).
2. Wasserstein Procrustes refinement — alternate between (a) solving an optimal-transport assignment between the two clouds given the current map, and (b) solving orthogonal Procrustes for the best rotation given the assignment. Multiple random restarts (R=100), batched assignments (b=10,000).
3. The output is a single orthogonal matrix mapping one space into the other — no learned networks, unlike baselines such as vec2vec.

A second useful finding: standard geometry metrics (CKA, mutual k-NN, TSI, QSI) computed between two models predict whether unpaired alignment will succeed — so you can know in advance if two spaces are alignable.

What's actually in the repo

• MIT-licensed Python library (WassersteinProcrustes with sklearn-style fit/transform), a Colab-ready notebook, and one script per paper experiment with run/plot commands.
• Precomputed vision-language embeddings on Hugging Face; code to recompute embeddings for 11+ other modality pairs (including the NSD fMRI benchmark).
• All baselines reimplemented (vec2vec, mini-vec2vec, ASIF, SUE, STRUCTURE, SOTAlign, LocalCKA) with tests that check them against the official code at pinned commits — unusually rigorous reproducibility hygiene.
• Text-to-image: captions mapped into an RAE diffusion model's latent space, scored with CLIPScore/VQAScore/TIFA/CycleReward.

Caveat: the README contains no quantitative results — you'll need the paper (arXiv 2610.09411) for retrieval/classification numbers. The authors' own framing is honest: "coarse" alignment, and metrics predict when it's possible, implying it sometimes isn't. The triage flags to check are the right ones: expect degradation on fine-grained concepts, and watch Fig. 5 for how much heavy lifting a handful of pairs does.

Practicalities

Bleeding-edge stack (Python 3.14, PyTorch 2.14, pixi for env management). The core method is CPU-only and cheap — most experiment jobs request 4 CPU cores and no GPU, though up to 96 GB RAM for vision-language runs; GPUs only needed for text-to-image sampling/scoring (≥24 GB) and neural-net baselines. Day-old repo, so expect rough edges, but the test suite and notebook-library equivalence checks suggest care.

Try it

git clone --recursive https://github.com/dominik-schnaus/unpaired-rosetta.git
cd unpaired-rosetta && pixi install

from unpaired_rosetta import WassersteinProcrustes

aligner = WassersteinProcrustes()
aligner.fit(images, captions)            # two unpaired embedding sets
scores = aligner.similarity(val_images, val_captions)
mapped = aligner.transform(val_images)   # images mapped into caption space

Or skip setup entirely: unpaired_rosetta.ipynb runs on Colab with dropdown-selectable models/datasets and pre-hosted embeddings. Project page: https://dominik-schnaus.github.io/unpaired-rosetta/