ThunderSyncRL: Agent Training That Never Idles Cards and Speeds Without Staleness

A team from Stanford, Yonsei and Korea universities opened an October 5, 2026 study on the bleeding wound of agent trainers: half the trainer cards sit idle waiting for work. The ThunderSyncRL setup computes ready work gradients without waiting and changes nothing in the result. In plain words: do not pile dishes until dinner ends; wash each dirty plate at once but clear the table in one go. Waiting is costly while early math is cheap. The technical report on arXiv proves this loop on two separate training objectives.

The problem is this: agent training runs long and uneven. Some work finishes in 10 seconds, some wrestles tools for 10 minutes. In the old synchronous setup the learner cards wait for the slowest job; measurements show 49.5 percent of step time passing in idle wait. Async setups that overlap work for speed produce stale data instead: jobs from an old policy mix into the new step, and extra machinery is needed to fix it. Either slow and clean, or fast and stale. The study says this dilemma is fake.

The dish analogy: wash at once, clear the table in one go

The fix is called gradient streaming. Each round fixes the policy like a snapshot; every job in that round stays bound to it. As soon as one job reward lands, its backward pass starts without waiting for the rest of the group. Partial results pile up, and a single shared step is taken when the group closes. Since the math is exactly preserved, the result matches the lazy setup that computes everything at the end. Neither the objective nor the agent nor the reward changes; only waiting disappears.

The first objective is GRPO: eight separate solutions are produced per prompt, and rewards are scored relative to the group mean. The obstacle is that one job score depends on the whole group. The study splits the gradient in two: the reward-weighted part and the plain part pile up separately and merge with one scalar when the group closes. So each job heavy math starts as soon as its reward lands. The GRPO trainer in the verl package is the natural home of this setup; the public verl records on GitHub show how the same trainer builds its group logic.

The second objective is OPD: a student produces work, a frozen teacher grades each turn, and the student distills from those grades. Here each turn is its own source: once teacher grades land, that turn backward pass starts while tools keep running in the sandbox. The normalization scalar applies when the round closes. The SkyRL docs explain why off-policy corrections matter on such distillation lines; ThunderSyncRL never needs that fix because staleness is zero.

The numbers: on an 8-B200 node with the Qwen-3.8-27B model, the SWE-bench Verified target was reached in 121.3 card hours instead of 233.5, i.e. 1.9 times faster. Terminal work on the Terminal Bench 4.0 measure shows 1.8 and 1.9 times speed too. At a fixed 128 card-hour budget it leads async setups by 1.27 to 2.47 points, ahead on all four. Step time fell from 1,313 to 687 seconds. The SWEBench records write that the verified set holds 500 real repository jobs; the TBench announcement reports the fourth release arriving with hard command-line work; the ArtificialAnalysis evaluation publishes its independent measure of the same release.

Balanced cards approach two times, imbalanced less: the speed ceiling

The speed ceiling follows card balance: when production and learning take equal time, gains near 2 times; when one dominates, it sets the bound. As job durations spread, GRPO speed climbs from 1.58 to 1.95; per-turn OPD stays between 1.73 and 1.94. On the small GLM-4.7-Flash model production dominates, so speed stays at 1.64; on H200 cards where work binds to learning, 1.63 and 1.67 appear. Trainer efficiency tops every lineup, up to 45.9 percent. The price is memory: holding partial gradients lifts per-card peaks from 117.0 to 122.8 GB at most.

Theory draws the ceiling in advance too: with production and learning in exact balance and zero tail, speed runs toward 2 times; across 588 paired lineups the speed predicted from release traces missed the measured one by 1.6 percent on average, with wall clocks 6.5 percent behind. So model word and field word agree; the rest is runtime cost. This rests the speed claim on hundreds of lineups, not one run.

Measurement discipline is tight too: 40 rounds per seed with the first 4 counted as warmup, rounds 5 to 40 averaged; seeds 42, 43 and 44. Every lineup used the same workload, the same card split, the same seed list; so speed came from melting wait, not changing work. The table points the same way for the LoRA lineup and for full fine-tuning.

The data side is transparent too: 37 thousand training jobs from the SWESmith pool for program work, 4,333 jobs from the SETA-Env set on HuggingFace for terminal work, with measure sets split from training. The SWESmith guide describes building repair jobs from broken repositories. The honest limits are written in the report: early math needs partial-state holding, paid in memory; the benefit follows objective and balance; multi-node work and generalization to more objectives are still missing. Still the lesson is clear: when outside feedback is costly, melt idle waiting, compute what is ready at once, merge in one step, and never mix stale data. Any team with a growing training bill learns the same lesson.