Agogic: Performance-Timed Music Tokens for LLM-Native Text-to-Symbolic-Music Generation

TL;DR: Every text-to-music language model starts by choosing how to serialize notes into tokens — and until now, that choice has never been tested in isolation, because it's always tangled up with the backbone, the data, and the training recipe. This paper freezes everything else (Qwen3.5 backbones from 0.8B to 27B, one dataset, one budget, one decoding scheme) and swaps only the tokenization across seven representations. The result is stark: a 0.8B model with performance-resolution tokens (10 ms timing, per-note velocity) beats a 27B model with beat-grid tokens on distributional fidelity, and a 34× parameter increase never closes the gap. The lesson generalizes to any LLM being extended to a new modality: how you serialize can dominate how big you scale, and you can find that out before spending the compute.

The choice everyone makes by default

To make an LLM generate music, you need to turn a score — notes with pitch, onset time, duration, loudness, instrument — into a token stream. Three families dominate:

Beat grids (REMI, TSD and descendants) snap every onset to a metrical lattice: bar 3, position 2-of-8. This is robust and popular, but the grid step is tempo-relative and coarse — roughly 63–125 ms at 120 BPM. Everything that makes a performance sound human — swing, rubato, the pianist's slight anticipation of the downbeat — gets rounded away before training begins.

Text notations (ABC, used by ChatMusician, MuPT, NotaGen) serialize the score as plain text, betting that the LLM's web-pretraining prior on ABC-like strings outweighs everything else. The compactness argument is real: ~2.8 tokens per note versus ~5 for REMI.

Performance-event streams (the Oore et al. lineage) keep raw timing as fine time-shift tokens.

Every published system commits to one of these plus its own data and recipe, so published numbers can't tell you whether the representation matters. The field has been arguing from plausibility — "ABC exploits the text prior," "beat grids are more learnable" — with no controlled evidence either way.

The experimental design: hold everything, swap one thing

The setup is almost embarrassingly simple once you see it, which is part of why it's valuable. Take a pretrained Qwen3.5 backbone. Append a small music vocabulary as mean-initialized embeddings (no architecture change, one softmax). Fine-tune on caption→music pairs with the loss masked on the caption. At inference, renormalize logits over the music token range so every sample decodes to playable MIDI. Now run this identical recipe for seven tokenizations: two performance-resolution (their own PMT, plus the pre-existing PerTok), three beat-grid (Beat-TSD, REMI, MIDI-Like — all configured with identical 32-level velocity so the swap isolates timing quantization), one grouped attribute-tuple control, and ABC-as-text.

One subtlety makes the comparison fair rather than self-serving: ceiling anchoring. Different formats can express different amounts of music — a beat grid physically cannot round-trip a sub-beat chord. So for every texture metric m, the paper reports attainment

where the denominator is the metric's value after a model-free encode–decode round trip of real held-out music — the most of m the format can physically carry. This cleanly separates expressiveness (a property of the format) from learnability (the fraction a model realizes). Per-token loss is used only within a representation, since across vocabularies it's meaningless.

PMT: the representation they release

PMT (Performance-timed Music Tokens) is deliberately not a novel encoding — the authors say so — but a clean instantiation of the performance-event class. Every note becomes ~4 tokens drawn from six named families totaling 609 symbols: PITCH (128), DUR (200 bins at 10 ms), VEL (32 levels), TSHIFT (100 bins at 10 ms, encoding the gap since the previous onset), plus TRACK (16) and PROG (129) emitted only when the instrument changes. Chords are runs of pitches at zero time-shift; all instruments interleave in one onset-ordered stream, so attention sees the full vertical texture at every moment. The decoder is exact on the quantization lattice (they prove it), so round-trip error is pure quantization: 2.3 ms median onset error on real music, versus 53 ms for beat grids — before any model touches anything.

The result: representation Pareto-dominates scale

The headline metric is Fréchet Music Distance (FMD) — a Fréchet distance between generated and real music in CLaMP-2 embedding space, the symbolic analogue of FID. Lower is closer to real music's distribution.

Read across, then down. Across representations at 0.8B, the ordering is a clean gradient of the representation class: both performance-resolution encodings (PMT 159, PerTok 188) sit far below all three beat grids (285–286) and the grouped control (287), with ABC worst at 406 despite its web-pretraining prior. Down the scale axis, almost nothing happens: PMT stays at 152–159 from 0.8B to 27B, REMI only drops from 285 to 272 at 27B. Fitting FMD against log-parameters, every arm's slope is within noise of flat — a near-constant ~125-point offset that 1.5 decades of scale never erodes. So a 0.8B PMT model (159) beats a 27B REMI model (272) outright, on both FMD (1.7×) and onset-timing divergence (4.5×).

The PerTok result matters as much as PMT's: a second, independently designed performance-resolution tokenizer lands in the same low-FMD corner, so this is a property of the class, not one lucky vocabulary. (PMT's edge over PerTok is completeness, not timing — PerTok is single-track and spends 6.8 tokens/note; the two overlap within CIs on timing divergence, and the authors explicitly decline to claim PMT > PerTok.)

The ABC result is a small controlled bombshell of its own. The compactness argument for text-native music LLMs — ~2.8 tokens/note, familiar to the pretrained prior — buys nothing for generation: ABC models produce half the polyphony of token models at every scale (attainment collapses from .69–.92 to .46–.51), because ABC has no native fields for micro-timing or velocity and expresses polyphony only through multi-voice syntax LLMs rarely emit. Its model-free ceiling is the lowest tested. As the paper puts it: ABC stores fewer tokens because it stores less music.

Killing the confounds

What elevates this from a nice benchmark to a result you can trust is how systematically the obvious objections are hunted down.

"It's just the pretrained prior." Repeat the swap with a 26M decoder-only Transformer trained from scratch on MAESTRO: PMT 125.1±11.7 (four seeds) vs. REMI 344.4±9.8. It transfers to POP909, and a multi-instrument from-scratch control (PMT 287.9±1.0 vs. Beat-TSD 395.3±1.1) rules out "solo piano only."

"It's just the finer lattice." The sharpest control: take a trained PMT model's generations, snap their onsets onto a fixed grid at the beat grids' own resolution — destroying the sub-grid timing while keeping pitch, duration, velocity — and recompute FMD.

If PMT won only by placing onsets where beat grids physically can't, coarsening would erase the gap. It doesn't: PMT stays 67–129 points ahead at every grid. Meanwhile the raw onset-interval divergence (JSD 0.03 vs. 0.15) mostly does close under common coarse quantization — and the authors say so plainly, refusing to read the timing histogram as a quality claim. The FMD gap is a holistic distributional difference the representation lets the model learn, not a re-measured format ceiling (the model-free round-trip FMD gap is only ~8 points, under 10% of the ~120-point trained gap).

"It's the embedder / the reference set." Recomputing with CLaMP-3 embeddings (a different model family, lossless MIDI serialization) reproduces the ordering at 1.9× vs. 1.8×. Swapping the folk-heavy reference for a source-stratified one widens the gap from 1.8× to 2.4× — the reported numbers are the conservative ones.

Ablations locate the mechanism. Dropping velocity in the from-scratch setup degrades FMD from 106 to 146 (and to 697 in one configuration); coarsening the time-shift from 10 ms to 20 ms in a trained model quintuples timing divergence and inflates FMD from 162 to 221. The performance parameters are load-bearing, and 10 ms is the coarsest grid at which a trained model still reproduces the real inter-onset distribution.

The imprinting diagnostic

A side finding that deserves independent circulation: run published text-to-MIDI systems through the harness on identical captions, and they reproduce their training distribution nearly invariant to what the caption asks. MIDI-LLM emits 72% chord-time on the paper's captions and 71% on MidiCaps — two disjoint domains — against references of 36.7% and different profiles. text2midi emits 17.3 max-polyphony everywhere. The Anticipatory Music Transformer, the strongest prior member of PMT's own fine-timing family, emits 79% chord-time and 36.8 polyphony at perfect validity — evidence that fine timing tokens alone don't buy fidelity; the controlled recipe around them does. The diagnosis is subtle: a "more texture is better" metric would reward this pathology. Only distance-to-each-domain's-own-reference, run bidirectionally, exposes it. Anyone evaluating conditional generative models in any modality should recognize this failure mode.

What the paper does not claim — and what it fixes at decode time

Two honest boundaries. First, audibility. The entire result is distributional. An automatic listener (Qwen3-Omni-30B) prefers micro-timing-preserving renders 60% of the time on expressive solo piano (p=0.07, directional) but is at 42% on the folk-heavy generated contrast — where the source material is already grid-quantized, so there's nothing to hear. A properly powered human study is pre-registered (protocol frozen, ~40–50 sessions targeted) but uncollected. Until it reports, "halves FMD" should be read as "matches the statistics of real performances much better," not "sounds twice as good." Second, chord-time is confounded: PMT's aggregate match (36.2% vs. 36.7% reference) partly comes from over-harmonizing folk while under-filling classical — a cancellation the authors expose against their own headline, resting the claim on FMD and timing divergences instead (which hold per-source: PMT leads Beat-TSD on both the folk and non-folk splits).

Native caption adherence is genuinely weak — Correct-Key 0.10 vs. 0.32 for MIDI-LLM, and dedicated systems like Amadeus reach 0.90 tempo adherence (by over-texturing to 72 max-polyphony against a 6.1 reference). But adherence turns out to be separable from fidelity: a decode-time constraint — restrict PROG tokens to the caption's named instruments, down-weight out-of-key pitches — doubles instrument-F1 (.28→.60) and Correct-Key (.16→.35) at unchanged validity and no FMD cost, with no retraining. Fidelity lives in the representation; adherence is a conditioning problem you can patch at inference. Notably, no system in Table 2 is both adherent and close to real — that corner is still open.

What changes if this holds

For anyone building music LLMs: stop serializing scores, start serializing performances, and stop expecting scale to compensate for a lossy tokenization — the curves are flat and never cross. The releases make this actionable: the harness, 25+ checkpoints, an 86.6k corpus aligned per-piece across caption/MIDI/ABC/audio, and a 6.25M captioned corpus (37× MidiCaps, the largest for music), so "the field's next representation claim can now be measured, not asserted."

The broader lesson is the transferable one. Whenever an LLM is extended to a new tokenized modality — vector graphics, character rigs, motion, 4D scenes — the serialization is usually chosen by convenience and never isolated. This paper shows the effect can exceed a 34× scale gap, and that a controlled swap on a small backbone predicts the ordering at every scale — meaning you can settle representation questions for a few A800-hours before committing to a large run.

Skepticism to carry: the 27B rows are two-seed and the MoE point single-seed; FMD is protocol-sensitive (the same model scores 159 or 114 depending on the reference, and the PMT-vs-grid ratio spans 1.6–2.8× across protocols — the authors flag this themselves); the decode budget interacts with a note-clipped reference in ways Appendix E documents (FMD structurally rewards short generations); and the perceptual question is genuinely open. But the ordering — performance-resolution above beat grids above ABC — survives every control thrown at it, which is more than most representation claims in generative modeling can say.

Where to spend your time: the Method's Component 3 and Appendix F (the ceiling-anchoring idea and the grid-snap/embedder/reference controls) are the intellectual core; Appendix I's imprinting diagnostic (Table 25 and Figure 10) is the most portable idea for evaluators; Appendix B.2's worked token-stream example makes the encoding concrete in two minutes.