MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

TL;DR: MJEPA takes the V-JEPA recipe — predict masked representations in latent space, no pixels, no contrastive pairs, no augmentations — and asks: can one shared transformer learn both audio and video with that single objective? The naive answer is no: a shared encoder trained with two intra-modal JEPA losses gets worse at both modalities than two separate models. The paper's contribution is the fix, and it's almost embarrassingly small: six 3-layer MLPs that predict the pooled representation of one modality from the other. That one addition flips the interference into synergy — frozen audio features beat fully-finetuned specialists on ESC-50 and FSD50K, and video representations nearly match VJEPA2 with roughly 10× less video data.

Why multimodal SSL keeps ending up complicated

Video is the obvious playground for multimodal self-supervision: every clip comes with a free, temporally aligned audio track that carries information pixels don't (a glass shattering off-screen, the timbre of an engine). Exploiting that correspondence should improve both modalities.

But the field's actual solutions have been Rube Goldberg machines. CAV-MAE, MAViL, EquiAV, XKD and friends combine separate audio and video encoders with fusion layers, then stack contrastive losses (which need careful negative sampling and batch sizes), masked reconstruction losses (which force the model to waste capacity on low-level detail), heavy augmentation pipelines, or cross-modal distillation. Each component has its own failure modes and hyperparameters, and the separate-encoder design means unimodal data — the bulk of what's available at scale — can't easily flow into the same model.

Worse, when people have tried the clean architecture — one shared encoder for both modalities — it reliably backfires. VATT, XKD, and AVSiam all report that unimodal representations degrade when encoders are merged. So the field settled on the folk wisdom that audio and video need their own towers, glued together with a bag of objectives.

Meanwhile, JEPAs (I-JEPA for images, V-JEPA for video, A-JEPA for audio) have quietly established a much simpler paradigm within single modalities: encode a masked view with a context encoder, encode the full input with an EMA copy of the same encoder, and train a predictor to fill in the masked tokens' representations under an L1 loss with a stop-gradient on the target:

Here E_\theta is the context encoder, E_{\bar\theta} its EMA (the target encoder), P_\phi the predictor, \Delta_y learnable mask tokens marking the missing positions, and M the set of masked indices. No negatives, no reconstruction, no augmentations. Since the objective never touches raw pixels or spectrogram bins, nothing about it is modality-specific — which makes the multimodal gap in the JEPA literature conspicuous. MJEPA is the paper that closes it.

The failure mode, and why it makes sense

The paper is structured as a progressive ablation (Section 4, the best part of the paper), and the first two steps set up the puzzle cleanly.

Step one: train an audio-only JEPA and a video-only JEPA separately on AudioSet-2M (~1.8M 10-second clips). Frozen attentive-probe evaluation on AudioSet-20K gives 30.87 mAP (audio) and 19.84 mAP (video). These are the bars to beat.

Step two: merge them. One ViT-L, one shared predictor, trained simultaneously on both modalities with the two intra-modal losses \mathcal{L}_{a\to a} and \mathcal{L}_{v\to v}. Audio and video are encoded independently — the encoder just happens to serve both. Result: audio drops to 28.70, video to 17.90. Both below baseline, replicating exactly what VATT and XKD had reported.

The intuition for the failure: nothing in the training signal tells the encoder that "dog barking" in the spectrogram and "dog" in the pixels are the same concept. The two loss terms pull the shared weights toward two unrelated feature geometries — spectral textures on one side, spatiotemporal appearance on the other — and the encoder ends up a compromise that serves neither. Weight sharing alone is a hope, not an inductive bias; the modalities can happily occupy disjoint subspaces of the same network.

The fix: predict the other modality's summary

The insight is that alignment must be an explicit objective, not an architectural wish — but it can be a remarkably cheap one. MJEPA adds cross-modal predictors C_{\psi}^{m_1\to m_2}: 3-layer MLPs that take the mean-pooled, last-layer features of a masked view of modality m_1 and predict the mean-pooled, last-layer features of the full view of modality m_2:

Two design choices carry the weight here. Last layer only: low-level features are genuinely modality-specific (spectral patterns vs. textures), so forcing alignment there would recreate the tug-of-war; only high-level semantics are shared, so only high-level features are aligned. Global pooling: there is no token-level correspondence between a spectrogram patch and a video tubelet, so the alignment target is a single holistic vector per clip. The intra-modal loss, by contrast, uses multi-level features to preserve a rich hierarchy within each modality.

Adding just \mathcal{L}_{a\to v} and \mathcal{L}_{v\to a} to the failing shared encoder reverses the interference: audio jumps to 33.52 mAP (now +2.65 over the unimodal baseline), video recovers to 19.58, and audio-video probing hits 36.34.

The full MJEPA then adds a third input mode — concatenated audio-video tokens — with its own intra-modal loss \mathcal{L}_{av\to av} and the remaining cross-modal pairs, for nine loss terms total, summed without weights. This is where the biggest jump happens: audio 38.89 (+8.02 over its unimodal baseline), video 25.38 (+5.54), audio-video 42.90. Predicting masked video tokens while attending to audio (and vice versa) forces the encoder to actually use the other modality, not just be compatible with it.

One more consequence of the unified design deserves emphasis: because the same encoder accepts audio-only, video-only, or joint inputs, mixing in unimodal data is trivial — feed video-only clips and compute only \mathcal{L}_{v\to v}. The authors add VideoMix2M (~2M videos, ~136K hours, no audio used) alongside AudioSet's ~5K hours of paired data. The striking result: video-only data improves audio performance (38.89 → 40.00 mAP), because better video representations propagate to audio through the cross-modal predictors. Separate-encoder methods can't get this for free.

The evidence

All headline numbers use frozen features with a small attentive probe — no finetuning of the backbone. The authors re-evaluated all public baselines with the same probe, which matters: attentive probing is much stronger than the linear probes many papers report (it lifts CAV-MAE Sync from a published 8.7 to 21.66 mAP, and SSLAM from 16.9 to 31.4 on AS20K), so their baseline numbers are more generous than the literature's.

On audio, the results are the paper's strongest card. Frozen MJEPA ViT-g doesn't just beat prior frozen features — it beats prior finetuned models on two of three benchmarks:

On video, MJEPA doesn't win — it matches dedicated video models with far less video data. A ViT-L trained on AS+VM2M gets 84.7% on Kinetics-400 and 73.3% on SSv2, versus VJEPA2 ViT-L's 85.1% / 73.7% — but VJEPA2 trained on VM22M, roughly 10× more video. The controlled comparison is also informative: MJEPA trained video-only on VM2M gets 80.6/69.8, essentially identical to VJEPA ViT-L (80.8/69.5), so the unified architecture costs nothing; adding AudioSet's paired data then buys +4.1 points on K400. Audio is acting as a data-efficiency multiplier for video.

The supplementary retrieval result is a nice bonus for anyone who has internalized "you need a contrastive loss for retrieval": using the cross-modal MLPs themselves as retrieval functions (predict, then nearest-neighbor in L1), MJEPA hits 29.3 R@1 audio→video on AudioSet, right at EquiAV's 29.6 — despite never seeing a negative pair. Prior work found retrieval collapses to chance without a contrastive term; here the predictive objective alone carries it.

What to make of it, and what to be skeptical about

The conceptual takeaway is bigger than the benchmarks. The paper turns "shared encoders hurt multimodal SSL" — a repeatedly observed failure that pushed the field toward two-tower designs — into "shared encoders hurt unless you add an explicit high-level alignment objective, which can be nearly free." That's a resolution to a live design debate, and it fits neatly with the Platonic Representation Hypothesis framing: modalities converge to a shared statistical model of the world, but at this scale the convergence needs a nudge. The recipe — pooled, last-layer, predictive, L1 — is simple enough that it will likely be tried immediately in other modality pairs; the authors explicitly gesture at medical imaging and vision+proprioception for robotics, where a single encoder robust to missing modalities at test time is genuinely valuable.

Now the caveats. First, parameter counts: MJEPA's ViT-L is 300M and ViT-g is 1B, while every AV-SSL baseline in the tables is ~170M. Some of the frozen-evaluation margin is surely scale, not method — although the ablation (all at ViT-L on the same data) isolates the mechanism cleanly, and the ViT-L-only rows already beat the baselines.

Second, the gains are lopsided. The dramatic wins are all on audio benchmarks; on video the claim is parity-with-less-data, not superiority, and MJEPA ViT-g still trails VJEPA2 ViT-g (85.0 vs. 86.6 on K400, 73.9 vs. 75.3 on SSv2). "Competitive with 10× less video data" is real and useful, but if you only care about video representations, VJEPA2 remains the stronger model. Note also that AS20K, where the headline +6.8 mAP frozen gain lives, is fundamentally an audio-event benchmark even in its "video-only" setting.

Third, finetuned comparisons are imported from other papers under different protocols, and MJEPA itself is never finetuned — we don't know its ceiling. The "frozen beats finetuned" claims hold on ESC-50 and FSD50K but not AS20K (40.9 frozen vs. EquiAV's 42.4 finetuned). And the design has some unexplained hyperparameter surface after all: a loss-scaling factor of 5.0 on the video-only data, found by search, and a compute-partitioning scheme across GPUs for the mixed data — the simplicity story is mostly true but not perfectly clean.

If you read one section, read Section 4: the progressive ablation is a model of how to present a method, with each architectural decision tied to a number and the central negative result (the naive shared encoder) shown rather than hidden. The retrieval experiment in the supplement (Section 9) is the second-most interesting few paragraphs — predictive-only cross-modal alignment matching contrastive retrieval is the kind of small result that quietly changes defaults.