EPFL-VILAB/Modus

TL;DR: MODUS is a single decoder-only causal transformer (adapted from BAGEL-7B) that generates in any direction across 16 modalities — text, RGB, depth, normals, segmentation, edges, detection boxes, and VQ-tokenized DINOv2/CLIP/ImageBind features — using exactly two losses and zero modality-specific heads or task pipelines. It's a fully examinable release (code, weights, dataset, demo, ICML 2026 paper), and the interesting question is whether treating dense vision tasks and learned representations as first-class generation targets in one trunk actually holds up against specialists.

What it is and why it matters

Most "any-to-any" models are federations: an LLM trunk plus per-task encoders, decoders, and losses bolted on. MODUS's bet is architectural minimalism — one causal transformer, one shared context, two experts that split by data type rather than task:

• 1D expert: discrete sequences (text, captions, grounding boxes, VQ-tokenized feature vectors) via standard next-token cross-entropy.
• 2D expert: continuous spatial latents (VAE + ViT features for images, depth, normals, seg, edges) via flow matching.

Training is literally the sum of those two losses. No depth head, no detection head, no segmentation decoder. Modalities aren't even hard-coded: each is a ModalitySpec (token range, positional-embedding handling, decode logic) registered from a YAML file, so model, data packing, and inference share one definition. Adding a 17th modality is a config change, not a surgery.

The symmetry enables things federated designs can't do cleanly: chained prediction (caption → depth → image, with the generated depth conditioning the image), cross-modal self-verification (use the model's own grounding/VQA to score its generated candidates), and conditioning on representations — treating DINOv2/CLIP/ImageBind feature vectors as modalities you can generate into or from.

What's released and what the numbers say

Everything: inference and training code (Hydra + FSDP, three-stage curriculum resuming from BAGEL-7B-MoT), model weights on HF, the MODUS-15Modality dataset, and a Gradio demo/Space. Apache-2.0 throughout.

The reported numbers, per benchmark: MMMU 51.1%, GenEval 0.81, DIODE depth AbsRel 0.285, NYUv2 normals 19.92° MAE, RefCOCO val 54.5%, ImageNet-1k 77.9/92.5 top-1/5. Read these honestly: GenEval 0.81 is genuinely strong for a unified model, and the depth/normals numbers are in respectable diffusion-based-estimator territory. But RefCOCO 54.5% is far below specialist grounding models (~90%), and MMMU 51 is mid-tier VLM performance. This is the classic unification tax — the paper's selling point is breadth-per-parameter and architectural simplicity, not per-task SOTA. The README doesn't include baseline comparisons; you'd need the paper for the actual deltas versus 4M, Unified-IO 2, or BAGEL itself.

Practicalities

It's a 7B mixture-of-transformers plus a SigLIP ViT and VAE, so expect a substantial single-GPU footprint for inference (README doesn't state VRAM; the model card may). Needs torch==2.5.1 (flex_attention), transformers==4.49.0; flash-attn optional with an SDPA fallback. Feature-modality targets require Apple's ml-4m VQ tokenizers. Training scripts contain cluster placeholders you'll need to edit. The repo is weeks old (61 stars) — expect rough edges, but the release is unusually complete for a fresh paper drop.

Try it

git clone https://github.com/EPFL-VILAB/Modus.git && cd Modus
conda create -n modus python=3.11 -y && conda activate modus
pip install -r requirements.txt
huggingface-cli download EPFL-VILAB/MODUS --local-dir models/modus

# chained: caption → depth → image
python infer.py --condition caption --target image --intermediate depth \
  checkpoint_path=models/modus prompt="a red double-decker bus in front of a clock tower"

Or skip the setup entirely: there's a hosted demo at huggingface.co/spaces/EPFL-VILAB/MODUS with any-to-any, chained, and representation-analysis tabs — the fastest way to probe whether the unification tax is acceptable for the tasks you care about.