Train on one chip, roll out on any: a field guide to RL on heterogeneous hardware
Reinforcement learning spends most of its compute generating tokens. We moved that generation onto four different chips from two vendors, measured each piece in controlled runs, then fed one trainer from all four at once.
ブログ
Yagiz Calik, Founding Member of Technical Staff (Post-Training)
TL;DR
- One trainer, four chips, two vendors. A two-node B300 trainer took RL rollouts from B300, H200, MI355X and MI300X engines for ten hours over plain HTTP. Ten steps, and every adapter sync landed on every engine.
- The wire barely matters with LoRA. InfiniBand syncs 6.5× faster than HTTP and saves under 2% of a step. This depends on LoRA: a 4 GB adapter, not the 709 GB base.
- More rollout nodes help until the trainer is the limit. In our test, a second MI355X node almost doubled rollout output (1.9×) and made the run trainer-bound instead of rollout-bound.
- The hard part is software. Identical launch settings gave a different engine on each chip, and most failures were LoRA paths few people had run, or at least reported.
Figure 1. The flagship run, 01:55 to 12:05 UTC+3 on November 5. Island size is each engine's KV-cache capacity; the squares in each callout are its KV cache in use; dots flow home at each island's measured tokens per second. Every quantity comes from the run's logs and a 30-second logger on each engine.
01 Compute comes in islands
Accelerators do not arrive as one clean cluster. A few nodes of one chip sit in one rack, a different chip in the next building, spot capacity appears for an afternoon and leaves. NVIDIA supply is tight, and a wave of new silicon is arriving, much of it built for inference first. Each of these is an island: useful on its own, hard to combine.
Training works best on one island. Its gradient all-reduce wants every GPU on one fast fabric, from one vendor, and it stalls on the slowest link. Training can span islands, but it pays for every hop. Rollout generation is different. Each engine serves the model by itself, takes prompts and returns tokens. In LLM RL that is most of the compute.
Every inference server is a rollout server waiting for a trainer.
The idea we set out to test
The bet is that training and generation do not have to live together. Training is happiest on one tight island. Rollouts only need tokens per second, from anywhere. If any machine that can run the inference engine and expose a safe port can serve rollouts, then every spare node, every new chip and, eventually, every datacenter becomes part of the RL fleet.
The timing matters. NVIDIA supply is tight, and the rest of the field is built to serve tokens: AMD’s Instinct line, Google’s TPUs (which SGLang already serves through SGLang-Jax), AWS Trainium (SGLang added a Trainium plugin in October, already serving GLM-5.3 and Miles rollouts), Tenstorrent, and inference-first designs like Etched’s and Rebellions’. RL is a training workload those chips can already carry, because most of RL is inference. As their software matures, teams that can mix islands get compute that others cannot use.
What decides it is software, not silicon. An engine on any chip has to honour the same contract as one on a B300: the same numerics, the RL features, cheap weight updates. This post tests that contract on four chips from two vendors, and says plainly where it holds only because of LoRA.
02 Make the rollout engine a contract
An engine can join the rollout pool if it honours three things. None of them names a vendor, though how much work each takes, numerics most of all, depends on how mature a chip’s software is.
Correct numerics. The trainer re-scores every sampled token itself, so an engine’s probabilities have to match the trainer’s closely. We measure the gap as KL per step.
The RL features. Log-probabilities with every token, abort, and taking new weights while requests are still running.
LoRA-sized updates over plain HTTP. Once per step the trainer posts the new adapter to every engine (miles#3690), one copy per engine (sglang#48241). There is no shared collective between trainer and pool, and no shared filesystem.
- 1
POST /pause_generationengines hold decoding for the swap
- 2
POST /register_lora_adapterfirst sync only · adapter name, rank, alpha
- 3
POST /begin_weight_updateopen the update session
- 4
POST /update_weights_from_tensor × 8512 MB chunk · encoded once · the same bytes to every engine
- 5
POST /end_weight_updateclose the session · then /update_weight_version stamps vN
- 6
POST /continue_generationrollouts resume on the new adapter
03 Why this works today, and where it would not
It works partly because of LoRA
Each sync moves a 4 GB adapter; the 709 GB fp8 base never moves, because every engine loaded its own copy once. On our 25 Gb/s Ethernet the adapter is about a second of wire time; the full model would be about four minutes per engine, every step.
It ran inside one datacenter, but the wire is not the limit
Distance matters less than it sounds. A sync is one 4 GB transfer, not a chatty collective, so latency barely counts. Over a modest 100 MB/s path, like the public internet between sites, the adapter takes about 40 seconds against a 50-minute step, while the 709 GB base would take about two hours, longer than the step itself. Over a dedicated 100–800 Gbps interconnect, the adapter takes well under a second and even the full base a minute or less per engine. At that point the wire stops being the limit, and, as in section 06, serialization and one copy per engine take over.
One HTTP API, two ways to move the adapter
NCCL and RCCL cannot share a communicator, so updates travel outside the GPU collective. HTTP runs over any IP network, so engines in other racks, clouds or datacenters can join. One copy per engine still scales: with a small model, 24 engines across three chip types sync in about 8 seconds. For much larger pools, the same http-lora API has a second mode, path mode: the trainer writes the adapter once to shared storage and sends each engine only its path (miles#3690, sglang#47135). These runs used the default, tensors mode, which streams the adapter to each engine.
| Engines | Chips | Sync (s) | Posting (s) |
|---|---|---|---|
| 8 | MI355X | 5.5 | 3.5 |
| 24 | MI355X · H200 · B300 | 8.1 | 5.9 |
It needs gates
Every engine passes a known-answer check, a log-probability parity check and a diff of the configuration it actually resolved. Without them we would have trained on wrong outputs at least twice.
04 Five arms, one variable at a time
Same model, same recipe (math problems from DAPO-Math-17k, filtered to a pass-rate band, with 12k-token responses), one 8× B300 trainer node on an AMD EPYC host. Each arm changes one thing.
| Arm | Rollout engines | Weight sync | Question |
|---|---|---|---|
| A− | 1× MI355X node | HTTP | baseline at 8k-token responses |
| A12 | 1× MI355X node | HTTP | NVIDIA trains, AMD generates |
| B12 | 1× B300 node | HTTP | the same, all NVIDIA |
| C12 | 1× B300 node | NCCL over InfiniBand | does the wire matter? |
| D12 | 2× MI355X nodes | HTTP | do more rollout nodes help? |
05 Where the step goes
At 8k-token responses the trainer barely waited. At 12k, generation and training take about as long as each other, so the trainer waits whenever rollouts run dry. The flagship, with a two-node trainer, is rollout-bound throughout: after the first step, 3 to 56 minutes of waiting against about 24 of training.
Flagship: band narrowed to 35–65% to stay rollout-bound → 2.4× more generation per step
06 The wire barely matters
B12 and C12 differ only in how weights reach the engine: HTTP between two B300 nodes, or NCCL broadcast over 8×800G InfiniBand. The links differ by about 250×, but the syncs by 6.5×: almost all of a sync is serialization and engine-side work, not time on the wire.
Flagship: one copy to each of four engines on four chips, from a two-node trainer
InfiniBand is 6.5× faster per sync and saves under 2% of a 50-minute step.
With LoRA, moving the policy costs about a minute and generating the next batch costs most of an hour. As frontier RL moves to longer, agentic rollouts, that share only shrinks. The fast fabric earns its keep inside the trainer, where the gradients are.
07 Add rollout nodes until the trainer is the limit
A second MI355X node almost doubled generation, from 2,690 to 5,190 tokens per second, and cut the trainer’s wait at step 1 from 51 minutes to under 2. One fp8 MI355X node falls just short of keeping a fast 8× B300 trainer fed; two leave headroom.
Every engine here uses an fp8 KV cache.
08 In our runs, precision mattered more than the vendor
The trainer re-scores every sampled token, so the train/rollout KL measures how far each engine’s probabilities sit from the trainer’s.
With the same fp8 KV cache, MI355X and B300 land in the same range at step 0: 13.9–15.7 against 16.6–16.7×10⁻³, inside the run-to-run spread. A bf16 cache sits 2.4× closer to the trainer. So in these runs, KV-cache precision moved the gap far more than the vendor did. We measured the gap itself, not its effect: we did not test whether it changes what the model learns over a long run.
Numerics is still the part to be most careful about. AMD and NVIDIA matched here, on mature kernels. TPUs, Intel GPUs, Trainium and inference-first chips run different kernels and number formats, and their RL paths are younger, so matching the trainer may take more work. That is why correct numerics is the contract’s first clause, checked before every run.
09 One trainer, four islands, ten steps
Then everything in one run. The trainer spans two B300 nodes over InfiniBand. The rollout pool is one node each of B300, MI355X, MI300X and H200: two vendors, two generations each, 48 GPUs in total. It is the largest run we have done. All four engines use fp8 KV cache and passed the gates before the trainer started.
| Step | Wait (min) | Train (min) | Sync (s) | KL ×10⁻³ |
|---|---|---|---|---|
| 0 | 109.7 | 25.7 | 160 | 16.6 |
| 1 | 47.9 | 23.5 | 154 | 14.8 |
| 2 | 2.8 | 24.7 | 156 | 17.7 |
| 3 | 36.7 | 22.7 | 156 | 18.8 |
| 4 | 8.6 | 24.9 | 162 | 24.1 |
| 5 | 29.2 | 24.6 | 163 | 24.4 |
| 6 | 29.8 | 24.7 | 161 | 31.2 |
| 7 | 26.7 | 24.6 | 157 | 39.5 |
| 8 | 14.9 | 24.7 | 155 | 34.2 |
| 9 | 56.4 | 23.9 | 154 | 47.4 |
Ten steps on four chips from two vendors. Every adapter sync landed. Zero restarts.
KV cache in use, 0–100% · UTC+3
| Engine | Attention (DSA) | MoE · LoRA | Running | Tok/s, whole run |
|---|---|---|---|---|
| B300 | trtllm | Triton · Triton | ~640 | 1,790 |
| MI355X | triton | Triton · Triton | ~660 | 2,490 |
| MI300X | triton | Triton · Triton | ~200 | 1,180 |
| H200 | flashmla_kv · flashmla_sparse_q8 | Triton · Triton | ~80 | 1,030 |
Every engine: TP8 on one node, fp8 KV cache, CUDA or HIP graphs on, no speculative decoding, no data-parallel attention. Means of a 30-second logger, 01:55 to 12:05 UTC+3.
10 A mixed pool needs a router that knows capacity
The router policy we started with, SGLang’s cache_aware, balances load by counting requests, so with little shared prefix every engine gets about the same number in flight. That is right for identical engines. Ours differ by 7× in KV capacity.
Count-based
cache_aware · equal requests per engineBy KV capacity
sgl-router#30Tokens generated
whole runUnder count-based routing, the H200 had about 870 requests waiting while running about 70, and the B300 sat at its limit with more than half its KV cache free. Capacity-weighted routing (sgl-router#30) gives each engine work in proportion to its KV capacity, and the split now tracks capacity within a point.
The run also showed the limit of that idea. Capacity is not throughput: the B300 holds 37% of requests and produces 28% of tokens, the H200 holds 5% and produces 16%.
We measured it on a five-engine pool replaying the same 3,072 rollout requests: weighting by KV capacity finished the batch 25% faster than round-robin, with 11% higher steady throughput. Weighting by each engine’s current speed did not beat it. The next step is marginal throughput: send each request where one more request adds the most tokens per second.
11 What we had to patch
These are the issues we hit and patched along the way to get the runs working. None needed a new chip feature; most were defaults or kernel paths few people had run with LoRA on this model. Some are hard to reproduce, versions move quickly, and you may hit different ones. The table is here so they don’t surprise you, and so you know roughly where the fix is.
| What we hit | Where | Effect | What we did |
|---|---|---|---|
| fp8 MoE weights pre-shuffled for a kernel the engine wasn't using | MI355X | fluent but wrong outputs | only shuffle for the kernel in use (fixed upstream since) |
| Attention matmul tile size crashes the compiler at some batch sizes | MI355X | crash at 65–256 rows | clamp the kernel’s block size |
| Data-parallel attention with LoRA and graphs | MI355X | wrong outputs on 7 of 8 ranks | run plain TP8, no data-parallel attention |
| New fused activation path reads a field LoRA layers lack | MI300X | crash at startup | skip the fused path on LoRA layers |
| LoRA kernel launch asks for more resources than the GPU allows | MI300X | crash after 30 min of load | cap running requests at 256 (not root-caused) |
| Same flags, different KV precision: fp8 on Blackwell, bf16 elsewhere | all | 2× capacity gap, silent | set the fp8 KV cache explicitly |
| Default fp8-KV attention kernel shifts first-token choices | H200 | wrong greedy answers | use the SGLang cookbook’s Hopper attention backends |
| Load-aware routing refuses to start with external engines | router | launch failure | use a policy that accepts engines joining after startup |
Two patterns. LoRA is the untested path: turning it on cost about 35% throughput on both vendors, because both fall back to generic MoE kernels. And identical launch settings do not give identical engines: each chip quietly picks its own kernels and defaults, so it pays to check what every engine actually resolved.
Hardware diversity is already here. Software maturity is the gap.
12 Limits of these results
Learning. Raw reward over the flagship’s ten steps is flat and noisy, as expected for a short run on a strong model, so these runs make no claim about learning speed. Our first test did learn: GLM-5.2 744B with a LoRA trained on a B300 node and rolled out on an MI300X pool, 30 steps of Countdown over about 10 hours with zero failed syncs. Held-out accuracy rose from 0.820 to 0.910 (64 problems, 4,096-token answers).
Response length. About 60% of responses hit the 12k cap. On one real training prompt the model was still mid-derivation at 12,288 tokens, so the cap cuts real reasoning and caps reward.
Peak speed. Every engine ran a generic LoRA path: Triton MoE kernels, no speculative decoding, no data-parallel attention. These runs measure the contract, not each chip’s best, and the per-engine gaps will move as each vendor’s software matures.
Policy drift. Train/rollout KL in the flagship rose from about 15 to 47 ×10⁻³ over ten steps. Two things add to it that this run cannot separate: fully async training scores some rollouts against a newer adapter than the one that generated them, and the growing LoRA adds a kernel path that differs between engine and trainer. We used the standard GRPO recipe with PPO clipping and no importance-sampling correction for the mismatch. Whether the drift stays small over hundreds of steps, or needs that correction, is the main open question for this setup.
Few steps per arm. Most arms ran three steps (C12 ran five).
Controls. Arms A to D are controlled comparisons (A− on an Intel trainer host). The flagship is a demonstration and changes several things at once.
13 Where this sits
Core Automation’s The Hitchhiker’s Guide to Training on Compute Islands (Rohan Anil, November 2026) takes the training side of the same problem: one MoE model trained across two GPU clusters about 9,000 miles apart, exchanging compressed model fragments every ~50 steps instead of every gradient. This post is the rollout-side counterpart: the trainer stays on one island, and generation spreads across the rest.
INTELLECT-2 (Prime Intellect) trained a 32B model with RL from rollout workers spread across the internet. StreamRL disaggregates generation onto separate GPU clusters, including different GPU types. HetRL (MLSys 2026) schedules RL across mixed NVIDIA GPUs (A100, L40S, L4) and simulated wide-area networks, and lists other vendors as untested. SGLang’s Trainium plugin already serves Miles rollouts from Trainium3 to an H200 trainer. Modal feeds one BF16 trainer from samplers on six NVIDIA GPU types in three weight formats, and finds that off-policy correction is what keeps training stable: without it, GRPO collapsed on mixed samplers soon after starting, and even on matched hardware and precision it only lasted longer. We used no correction in these short runs, and our flagship’s KL was still rising when it ended (section 12), so their result is the main thing to test next. Our piece is complementary: one pool that mixes vendors and generations at one precision, feeding one trainer, on the mainstream open stack, with the changes sent upstream and controlled arms measuring the cost.
14 What we would run next
Marginal-throughput routing: weight engines by what one more request adds, not by their current speed.
A longer run with a held-out evaluation curve, with and without off-policy correction.
Each chip on its fastest path, under the same gates.
Path mode at dozens of engines, and a run across two datacenters.
15 What we sent upstream
Everything here runs on Miles and SGLang with these changes, plus a few local patches.*
| PR | What it does | Status |
|---|---|---|
| miles#3690 | http-lora: LoRA weight sync to external engines over HTTP, tensors or path mode | merged |
| sglang#48241 | one adapter payload shared by every TP rank, so each engine gets one copy | merged |
| sglang#47135 | upsert on /load_lora_adapter: swap an adapter in place | merged |
| miles#3693 | sequence packing for GLM-5.x LoRA on tilelang, about 2.2× faster train steps | in review |
| sgl-router#30 | capacity_aware routing by KV capacity in sgl-router | in review |
* The ROCm and LoRA fixes in section 11 are local patches, not upstream. Results may not reproduce exactly on newer versions, since these code paths change quickly and upstream fixes keep landing.
16 Acknowledgements
We would like to thank Yueming Yuan for reworking, reviewing and merging the transport. Alex Nails for the first conversations, design guidance and getting it reviewed. Zhichen Zeng for the early review and the green light. And Banghua Zhu for the intro that started this work. All four are at RadixArk, whose work on Miles and SGLang this all builds on.
References
Related systems.
Core Automation, The Hitchhiker’s Guide to Training on Compute Islands (2026). Training one model across distant GPU clusters; the training-side counterpart to this post.
INTELLECT-2 (Prime Intellect, 2025). RL with rollout workers spread across the internet.
StreamRL (2025). Generation disaggregated onto separate GPU clusters, including different GPU types.
HetRL (MLSys 2026). RL scheduling across mixed NVIDIA GPUs and simulated wide-area networks.
Announcing the SGLang AWS Trainium Plugin Beta (LMSYS, 2026). Miles rollouts served from Trainium3.
Toward RL on mixed-precision, mixed-accelerator samplers (Modal, 2026). One trainer fed by six NVIDIA GPU types, with off-policy correction.
Stack and our changes.
Miles, the RL framework the trainer runs on.
SGLang, the inference engine on every rollout chip.
sgl-router-for-miles, the router Miles uses by default.
The pull requests we sent upstream are listed in section 15.
Citation
@misc{calik2026heterorl, title = {Train on one chip, roll out on any: a field guide to RL on heterogeneous hardware}, author = {Calik, Yagiz}, year = {2026}, month = nov, note = {ai\& Research blog post}}前の記事