Task-Error Residual Learning for Real-Robot Five-Ball Juggling

TL;DR: Two 4-DOF robot arms learn to juggle a five-ball cascade — a skill that takes dedicated humans years — and they converge after essentially one failed attempt. There is no simulation training, no in-loop vision, and the planner and controller are deliberately crude. The whole result rests on one methodological point, cleanly demonstrated: if you give a residual learner the direction of the task error (a 3-vector: "the ball landed 4 cm too far and 2 cm left") instead of the scalar reward RL would use ("that throw scored 0.3"), and you seed it with an informative analytic prior, learning collapses from a search problem into a root-finding problem. The winning method is almost embarrassingly simple: \mathbf{u}_{n+1} = \mathbf{u}_n - \alpha_n \mathbf{e}_n — subtract the observed error from the command. The paper's claim is that on real robots, the bottleneck isn't your model's accuracy; it's the information content of your supervision signal.

Why five-ball juggling is a good stress test

Toss juggling — actually releasing balls into ballistic flight, not batting them off a paddle — is one of the more punishing dynamic tasks you can put on a robot arm. A five-ball cascade demands fast, precise throws on a rigid schedule: each ball flies for about a second, the arms must catch and re-throw in the gaps, and a single miss ends the run. The pattern's dwell times shrink and the required precision grows as you add balls; that's exactly why humans need years of practice for five. Robust five-ball juggling on robot arms was, per the related-work section, an open problem.

The naïve deep-RL playbook — train in sim, transfer — hits a wall here because the sim-to-real gap on highly dynamic contact tasks (ball seated in a funnel, released at 5+ m/s) is exactly the part simulators get wrong. On-robot RL hits a different wall: every sample is a real attempt, so you can afford dozens of trials, not thousands. The authors' earlier work learned two-ball juggling on one arm via black-box policy search on a scalar survival reward, and that lineage is precisely the baseline this paper argues against.

The insight: minimize a cost, or find a root?

Here's the framing that makes everything click. When a throw misses, the world tells you a vector: the ball landed here, you wanted it there. The displacement \mathbf{e} \in \mathbb{R}^3 is the task error. Standard RL immediately compresses this to a scalar, \|\mathbf{e}\| or \|\mathbf{e}\|^2, and then must recover the lost direction by random perturbation: sample around the current policy, see which perturbations score better, move that way. You paid for a 3-vector of information and kept one number.

Keeping the vector changes the type of problem. Minimizing \|\mathbf{e}(\mathbf{u})\|^2 over the correction \mathbf{u} is local optimization. Driving \mathbf{e}(\mathbf{u}) \to \mathbf{0} is root-finding — and root-finding with a decent local linear model is what Newton's method eats for breakfast, converging in a handful of steps.

This isn't a brand-new observation in isolation — iterative learning control (ILC) has done signed-error root-finding for decades, and Aboaf et al. applied task-level corrections to single-arm juggling in 1989. The paper's contribution is (a) lifting the ILC idea from joint-tracking error (which is blind to contact dynamics and ballistic flight) to task error (where the ball actually lands), (b) scaling it to a genuinely hard multi-ball, two-arm task, and (c) mapping out systematically why it works via a 3×3 method comparison.

The mechanism: you could code this in an afternoon

The stack under the learner is intentionally rough. A kinematic planner solves a constrained jerk/acceleration minimization (CasADi + IPOPT) using a "1g contact-switch" model — during carry the ball is assumed seated, at release it's assumed instantly ballistic — and a parabolic flight model ignoring drag. Tracking is soft PD with inertia-only feedforward straight from CAD data, no system identification, no friction model. Execution is fully open-loop: the OptiTrack ball tracker watches the flights but never enters the control loop, and there is no active catching.

The learned quantity is a 3D takeoff-velocity correction \mathbf{u}_n per throw. The idealized ballistic map

sends takeoff velocity \mathbf{v} from release point \mathbf{p}_0 to a touchdown position after fixed flight time t_{\mathrm{flight}}. Inverting it gives the nominal command \mathbf{v}_{\mathrm{TO}}; the planner executes \mathbf{v}_{\mathrm{TO}} + \mathbf{u}_n. After the throw, the tracker fits a parabola to the late flight, projects to the planned touchdown time, and back-projects through f^{-1} into velocity space, yielding the error label \mathbf{e}_n.

The key modeling assumption: near a fixed throw configuration, the true map is the idealized map plus an unknown constant offset \mathbf{c} absorbing everything the stack ignores — drag, spin, motor miscalibration, contact dynamics. Under that assumption the residual-to-error map is exactly affine with identity Jacobian:

That identity Jacobian is a free analytic prior: velocity errors pass through to velocity-space error labels one-to-one. The Newton update \mathbf{u}_{n+1} = \mathbf{u}_n - \alpha_n \hat{\mathbf{J}}_n^{+}\mathbf{e}_n then collapses, with \hat{\mathbf{J}}_n = \mathbf{I}, to

with an exponential damping schedule \alpha_n to avoid thrashing on noise near the floor. That's the entire learner. Each juggler runs one such learner instance per throw index and arm during the transient (warm-starting successors from predecessors) and one per arm in the cyclic phase.

The 3×3 matrix: which ingredient does the work?

The structured comparison is the paper's scientific core. Two axes, each ternary. Feedback: the full error vector \mathbf{e}, its norm \|\mathbf{e}\|, or the squared norm. Prior specificity: none (pure stochastic search — per-axis (1+1)-ES, CMA-ES, REPS; the scalar cells here are the stand-ins for scalar-reward RL), structural (fit the Jacobian or cost-surface curvature from data — MLE Jacobian, BO with a fitted cone/paraboloid mean), or calibrated (fix \mathbf{J} = \mathbf{I} from the analytic model — Fixed Jacobian, MAP Jacobian, composite BO with affine mean).

Sweeping all nine cells in simulation (5-ball cascade, ten seeds each), sample efficiency degrades almost monotonically along both axes, and the effects compound. The one instructive anomaly: BO on a calibrated paraboloid mean fails on every seed, because the squared cost's gradient vanishes exactly at the optimum, leaving only posterior uncertainty to steer a non-convex search — whereas the cone mean \|\mathbf{J}(\mathbf{u}-\mathbf{u}^*)\| keeps a constant-magnitude gradient all the way in. A nice reminder that the choice between \|\mathbf{e}\| and \|\mathbf{e}\|^2, usually treated as cosmetic, matters when the surrogate does the steering.

The only cell that is both fast and reliable on all ten seeds is directional + calibrated, and within it the three variants (Fixed Jacobian, MAP Jacobian, composite BO) perform comparably — so the cheapest one wins. Neither ingredient suffices alone: directional feedback without a prior (per-axis evolution strategies) is slow; a calibrated prior with scalar feedback is unreliable or fails.

The evidence on hardware

The real-robot headline numbers (Table 2, six seeds of ten attempts per pattern, each attempt up to 120 throws ≈ 30 s of continuous juggling):

Read that again: five-ball juggling, first full success on attempt ~2, a three-in-a-row streak by attempt ~4, and after the first drop the task error decreases monotonically with no further failures. The converged residual is a ~0.23 m/s velocity correction — substantial, meaning the idealized model really is wrong and the learner really is doing work — against a noise floor of 0.022 m/s, which through the 1.00 s flight corresponds to ~22 mm landing scatter at the catch plane. The funnels absorb that.

Two real-robot ablations then probe how much each crutch matters:

Prior quality. Rotate the analytic Jacobian by a random axis, \mathbf{J}' = R\mathbf{J}, from 0° to 90°. Up to 30° of misalignment, convergence speed is unchanged; 60° still converges, slower — but still faster and more reliably than any scalar or stochastic method managed in simulation with a perfect setup. Only 90° fails, and the geometry explains it: as long as the rotated step retains a positive projection onto the true descent direction, every Newton step still shrinks the error. The prior needs to be roughly right, not calibrated.

Stack accuracy. Scale the PD gains down to 25% of nominal. Final task performance is invariant across the whole sweep — same error floor, same reliable juggle — with only the transient (number of attempts to converge) degrading. The mechanism is the paper's second conceptual contribution: a soft controller tracks worse but consistently, making the same error every throw, and a constant error is exactly what a residual absorbs. The authors distill this into a sharp aphorism: stack accuracy makes learning unnecessary; stack repeatability makes it possible. In between, accuracy buys only convergence speed. A practical bonus: soft gains mean lower impact forces if the arm hits something, which matters for a machine whipping around near humans.

What this changes, and what to be skeptical about

If the result holds broadly, the actionable lesson is: before reaching for on-robot RL to fine-tune a behavior, ask whether the task defines a directional error you can measure, and whether your nominal model gives you even a crude sensitivity (Jacobian) from action to outcome. If yes, a damped Newton update on the task error will likely crush any scalar-reward method on sample efficiency — here, convergence in ~2 attempts versus the dozens-to-hundreds typical of episodic policy search (the authors' own prior work needed black-box search for just two balls). The concluding sim-to-real suggestion is also worth noting: instead of transferring a policy from simulation, transfer a cheap adaptation process (the prior \mathbf{J}_\phi = \mathbf{I} is extractable from a simulator automatically) and let it close the gap on hardware — the prior-rotation results show such adaptation tolerates exactly the modest mismatch a sim-to-real gap introduces.

Now the caveats, and they're real:

• The task is unusually well-suited. The learned quantity is a static 3-vector at a fixed throw configuration, the constant-offset assumption makes the problem exactly affine with identity Jacobian, and the flight time is fixed. This is the best possible case for Newton root-finding. Nothing here learns a policy over states; "contextual residual learning" (conditioning the correction on catch outcomes) is explicitly future work. Whether the "directional error + prior" recipe survives high-dimensional, state-dependent residuals is untested.
• The 3×3 matrix was swept only in simulation. Understandable — a full hardware sweep would take prohibitive wall time — but the central comparative claim rests on sim, with a 30%-level calibration check on residual magnitudes. Only the Fixed Jacobian ran on hardware.
• The stack is cruder than it sounds, but not absent. There's a serious constrained-optimization trajectory planner, dual encoders at 1 kHz, custom cooling, motion capture with per-ball Kalman filters and Hungarian association, and funnel end-effectors that make catching passive. "Simple stack" means uncalibrated dynamics, not minimal engineering. The repeatability that makes everything work is itself a hard-won hardware property.
• Open-loop execution cuts both ways. No in-loop perception is an elegant demonstration of repeatability, but it also means the system cannot recover from a bad throw within a run — the residual only helps the next attempt. The invariance results explicitly break "under strongly reduced gains," when executed motion decouples from commands.
• The comparison is fair but favorable. Scalar baselines (CMA-ES, REPS) are competent choices, but no modern model-based RL or ILC-hybrid was included; the "antagonists" section argues why they'd struggle rather than testing them.

None of this undermines the demonstrated capability — five-ball juggling on real anthropomorphic arms, converging in two attempts, is a genuine first — but it bounds the generalization claim to tasks that are periodic, repeatable, and locally well-modeled.

Where to spend your time

Section 4.1 (task-error model and taxonomy) is the intellectual heart — ten minutes there and you understand the whole design space, including why the Jacobian prior is free. Section 5.5 (stack-accuracy ablation) is the most quotable finding: performance invariant to a 4× gain reduction is the cleanest evidence I've seen for the repeatability-over-accuracy thesis. And watch the videos at the project page — a first drop followed by a stable five-ball cascade on attempt two is more persuasive than any table.