Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls

TL;DR: The multiplicative repetition_penalty shipped in essentially every LLM inference engine — HuggingFace, vLLM, llama.cpp, and a dozen more — contains a hidden dependence on a quantity that training never constrains: the zero-point of the model's logits. Adding a constant to all logits is a provable no-op for the model's output distribution, yet at a routine penalty setting of \theta=1.3, this "no-op" flips 58–96% of greedy tokens, because the penalty branches on whether each raw logit is positive or negative. Worse, the penalty demolishes structured output: valid JSON against real schemas drops from 97% to 23%. The fix is one reorder: apply the penalty to normalized log-probabilities instead of raw logits, which zeroes out both effects in every measurement in the paper.

A knob that isn't what it says it is

If you've ever deployed an LLM, you've probably touched repetition_penalty. The mental model everyone carries: it's a scalar knob, set it to 1.1–1.3, get less repetition, and 1.3 means roughly the same thing on any model. The operator itself, inherited from Salesforce's 2019 CTRL paper, looks trivially simple. For each token that has already appeared in the context, transform its logit z_i:

where \theta > 1 is the penalty strength. Positive logits get divided (pushed down toward zero), negative logits get multiplied (pushed further down). Both branches decrease the logit, so both suppress the token. Three lines of PyTorch.

The sign-branch exists for a reason: the naive version just divided every seen logit by \theta, but dividing a negative logit by \theta > 1 moves it toward zero — raising that token's probability, which is backwards. This was caught and reported to HuggingFace in 2019, and the sign-branch became the accepted fix, copied verbatim into every downstream engine.

Here's the paper's observation, which is embarrassingly simple once stated: the fix branches on the sign of a raw logit, but the sign of a raw logit is meaningless.

The gauge freedom you already know about

You know that softmax is invariant to uniform shifts: \text{softmax}(z + c) = \text{softmax}(z) for any constant c added to every coordinate. This is used constantly for numerical stability (subtract the max before exponentiating). Since cross-entropy training only sees the logits through the softmax, the training objective is completely blind to where the logits sit relative to zero. The paper borrows physics language and calls the choice of c a gauge: the model's behavior fixes the probability distribution, but not the zero-point. Two checkpoints with logits shifted by 100 are the same model in every observable sense.

And real checkpoints do sit at wildly different, arbitrary zero-points. Measured at decode time, the fraction of already-seen tokens whose logits are positive — and therefore take the divide branch — ranges from 0.17 on gpt2 to 0.95 on Qwen2.5-Coder-7B (Table 1). The same repetition_penalty=1.3 mostly multiplies on the first model and mostly divides on the second. It is literally a different mathematical operation on each checkpoint. gpt2's median top-1 logit is -161; Qwen's is +25. Neither number was chosen by anyone; training just happened to land there.

Look at the geometry. The two branches have different slopes, so the strength of the suppression a token receives depends on where its logit sits relative to the kink. And the kink sits at z=0 — a location the model can move all its tokens across without changing anything observable. The accepted fix for the 2019 bug branches on a coordinate the training objective leaves completely free. The paper found exactly one prior sighting of this — a 2023 llama.cpp issue noting the conflict with shift-invariance, closed without action.

Consequence 1: the penalty is not well-defined

The gauge freedom hands the author a beautifully clean experiment. Take one model. Decode the same 200 WikiText-103 prefixes twice, greedy, at the same \theta: once with c=+5 added to every logit, once with c=-5 (equivalently, tweak lm_head.bias). These two configurations have identical output distributions — a mathematical fact, not an approximation. Any well-defined intervention on the model's behavior must produce token-for-token identical output. Count the fraction of the 40,000 greedy positions where the runs differ: the flip rate.

The controls behave exactly as they should. At \theta=1 (penalty off), zero flips on every model — the harness is clean. The subtractive presence penalty (z_i - \alpha on seen tokens, the form hosted APIs like OpenAI's use) is shift-invariant by construction: zero flips out of 40,000 on every model. Same for the normalized variant (below).

The CTRL operator at \theta=1.3:

The effect is undiminished at 7B scale and by RLHF (base and instruct Qwen2.5-7B both flip 92%). The probe doesn't even need a synthetic constant: re-centering each model by its own median logit — a gauge it could equally have shipped with — flips 71–97% of greedy tokens.

One important reading note the paper handles carefully: a flip is not automatically a quality failure. A penalty is supposed to change tokens, and once two greedy trajectories diverge, they stay ~99% different simply because any two different texts do. The headline rate reflects early, pervasive seeding of divergence (median first divergence at position 4–10 on most models, reaching 200/200 prefixes on four of five models) rather than per-position disagreement. What the flip rate does establish is that repetition_penalty=1.3 doesn't name one behavior — it names a family of behaviors indexed by an accident of training.

The paper also prices the intervention in log-likelihood under the unpenalized model. On gpt2-large at \theta=1.3, the penalty costs 235 nats over 200 tokens in the shipped gauge and 57 nats in the own-median gauge — the same knob costs 4.1\times more in one gauge than the other, and the two prices straddle the subtractive penalty's 79 nats. The cumulative gap between the c=\pm 5 runs grows roughly linearly, reaching 140–190 nats by position 200 on four of five models. For the controls, the gap is exactly zero at every position.

Consequence 2: it eats structured output alive

Here's where flips stop being neutral. Structured output has repetition that the grammar requires: JSON must re-close }, re-separate with ,, re-quote with "; code must re-indent. Every one of those delimiters has appeared before, so the penalty pushes them down — and it has no way to distinguish "the grammar needs this token" from "the model is looping."

The mechanism is sharp enough to write down. At a position where the correct token is the confident top choice with logit z_{\text{top}} > 0, and the runner-up sits a gap g below (itself unpenalized), the penalty flips the greedy choice exactly when

This closed form isn't just a heuristic: on StarCoder2-7B decoding HumanEval, it predicts the observed penalty-induced flips with balanced accuracy 0.999 over 48,919 positions where its assumptions hold, and all 27,720 flips land on the pre-penalty runner-up. (Replicated on Qwen2.5-Coder-7B.) Note the counterintuitive shape: corruption is gap-driven, not confidence-driven. A high logit with a tight runner-up flips easily; the paper pre-registered the "hits most confident tokens first" hypothesis and its own data refuted it — a nice bit of intellectual honesty they left in.

End to end, the damage is severe. Generating complete JSON objects against 200 real-world schemas from JSONSchemaBench with Qwen2.5-Coder-7B:

A routine \theta=1.3 takes valid output from 97% to 23%, and the survivors are dominated by near-empty schemas with almost no required delimiters. llama.cpp's default 64-token sliding penalty window makes things worse (12% valid), because on multi-hundred-token structured outputs the window concentrates the penalty precisely on the most recent, grammar-obligatory tokens.

How far this reaches, and the fix

Both effects were replicated inside vLLM and llama.cpp through each stack's own sampler on the same inputs — flip rate 0.964 in both at \theta=1.3 on gpt2-large, matching HuggingFace, with the vLLM run token-identical to the HF run. A source survey found the same sign-branch-on-raw-logits operator, with no normalization, in twelve further engines: TGI, SGLang, TensorRT-LLM, ExLlamaV2, mlx-lm, LMDeploy, aphrodite-engine, KoboldCpp, mistral.rs, candle, text-generation-webui, and Ollama — which enables the penalty by default at 1.1. A git-history trace shows the sign-branch was independently derived at least twice (HuggingFace 2019, llama.cpp 2023 from the CTRL paper), and every other engine copied a prior copy; because each keeps its own implementation, no single upstream patch fixes them all. The subtractive presence_penalty/frequency_penalty family in hosted APIs is immune by construction.

The fix is almost anticlimactic. Apply the penalty to \ell = \log\text{softmax}(z) instead of z. Two things happen at once:

1. \log\text{softmax} is shift-invariant, so the gauge dependence vanishes by construction.
2. Every \ell_i \leq 0, so the sign-branch stops branching — every seen token takes the multiply path, \ell_i \mapsto \theta\,\ell_i. Since \theta \log p_i = \log p_i^{\theta}, this is just p_i \mapsto p_i^{\theta}: temper each seen token's probability downward by a common power. A monotone, well-defined suppression reading a coordinate the model's distribution actually determines.

Measured: gauge flips collapse to zero at every \theta on every model, delimiter corruption on code drops ~80× (flip rate 0.139 → 0.0017), JSON validity at \theta=1.3 returns from 23% to 97% — its no-penalty level — and it still breaks degenerate loops (repetition falls monotonically in \theta). The delicious irony: HuggingFace already ships this exact operator as LogitNormalization. It's off by default, and when enabled, it runs after the penalty. The fix is a reorder.

One real caveat: log-probs are O(1) while raw logits are O(10), so a given \theta acts more gently after normalization, and tuned values need re-setting. In exchange, the same \theta finally means the same operation on every model — which is what everyone assumed it meant all along.

What to make of it

This is a small paper with an unusually clean epistemic structure. The gauge probe is a genuine controlled experiment — the two conditions are provably equivalent models, so any divergence is attributable to the operator, and the controls (zero flips in 40,000 positions, zero flips at \theta=1) rule out harness bugs. The delimiter-flip closed form with 0.999 predictive accuracy means the mechanism is understood, not merely observed. The pre-registered claims that failed are reported. Cross-stack replication through each engine's own sampler rules out a HuggingFace-specific quirk. They even found and reported an unrelated llama.cpp bug along the way (--repeat-last-n -1 silently disables the penalty).

Things to stay skeptical about. All models are ≤7B; the flip rate held through Qwen2.5-7B base and instruct, but the flagged concern — does this replicate on large frontier-scale RLHF checkpoints? — is fair, since logit statistics could differ. The paper deliberately makes no quality claims about the normalized variant's open-ended text: it removes the gauge pathology and the JSON corruption, but whether p \mapsto p^\theta tempering produces subjectively better prose than the subtractive family at matched repetition levels is unmeasured. Everything is greedy decoding; with temperature sampling the flips become probability perturbations rather than hard swaps, presumably softening (but not eliminating — the suppression strength is still gauge-dependent) the effect. And the practical blast radius depends on how many production deployments actually combine repetition_penalty > 1 with structured output; anyone using constrained decoding (grammar-masked sampling) is partially shielded, a mitigation the paper doesn't discuss.

Still, the broader lesson generalizes beyond this one knob: any inference-time intervention that reads raw logits — not the distribution — is reading coordinates training never pinned down. Logits have gauge freedom (and, without a softmax temperature convention, scale freedom too). The repetition penalty happens to be the operator where this bit hardest, but it's worth auditing any logit-processor you rely on with the same z + c probe: it costs one bias tweak and a diff.

If you read one section, read Section 2 — the gauge-flip experiment and the "reading the flip rate" discussion are a model of how to interpret an ambiguous metric honestly. Section 5's derivation of why normalize-first collapses the sign-branch into probability tempering is two lines and worth internalizing.