Skip to content
ai& ホーム
ai& ホーム
ai& ホーム

Train on one chip, roll out on any: a field guide to RL on heterogeneous hardware

Reinforcement learning spends most of its compute generating tokens. We moved that generation onto four different chips from two vendors, measured each piece in controlled runs, then fed one trainer from all four at once.

ブログ

Nov 13, 2026

Nov 13, 2026

ai& Research · 11 min read

ai& Research · 11 min read

Yagiz Calik, Founding Member of Technical Staff (Post-Training)

TL;DR

  • One trainer, four chips, two vendors. A two-node B300 trainer took RL rollouts from B300, H200, MI355X and MI300X engines for ten hours over plain HTTP. Ten steps, and every adapter sync landed on every engine.
  • The wire barely matters with LoRA. InfiniBand syncs 6.5× faster than HTTP and saves under 2% of a step. This depends on LoRA: a 4 GB adapter, not the 709 GB base.
  • More rollout nodes help until the trainer is the limit. In our test, a second MI355X node almost doubled rollout output (1.9×) and made the run trainer-bound instead of rollout-bound.
  • The hard part is software. Identical launch settings gave a different engine on each chip, and most failures were LoRA paths few people had run, or at least reported.
ROUTERby KV capacity1MI355XAMD CDNA4 · 3.0M KV2,758 tok/sKV in use2MI300XAMD CDNA3 · 1.2M KV1,102 tok/sKV in use3B300NVIDIA Blackwell · 2.7M KV1,313 tok/sKV in use4H200NVIDIA Hopper · 0.4M KV1,354 tok/sKV in use725 tok/s1,823 tok/s2,920 tok/s2,877 tok/s2,834 tok/s2,896 tok/s2,958 tok/s3,051 tok/s3,143 tok/s2,862 tok/s2,581 tok/s2,420 tok/s2,258 tok/s3,235 tok/s4,212 tok/s3,725 tok/s3,237 tok/s2,751 tok/s2,264 tok/s2,501 tok/s2,737 tok/s2,840 tok/s2,942 tok/s2,545 tok/s2,147 tok/s2,728 tok/s3,309 tok/s3,260 tok/s3,211 tok/s2,777 tok/s2,343 tok/s2,414 tok/s2,484 tok/s2,469 tok/s2,454 tok/s2,333 tok/s2,211 tok/s2,572 tok/s2,933 tok/s2,496 tok/s2,058 tok/s2,974 tok/s3,890 tok/s3,504 tok/s3,118 tok/s2,785 tok/s2,452 tok/s2,507 tok/s2,561 tok/s2,794 tok/s3,027 tok/s2,705 tok/s2,382 tok/s1,546 tok/s710 tok/s1,745 tok/s2,779 tok/s2,841 tok/s2,903 tok/s2,543 tok/s2,182 tok/s2,009 tok/s1,836 tok/s2,631 tok/s3,425 tok/s3,563 tok/s3,700 tok/s3,257 tok/s2,814 tok/s2,787 tok/s2,759 tok/s2,496 tok/s2,233 tok/s2,307 tok/s2,380 tok/s2,815 tok/s3,250 tok/s3,385 tok/s3,520 tok/s3,039 tok/s2,558 tok/s1,966 tok/s1,373 tok/s1,590 tok/s1,807 tok/s2,257 tok/s2,707 tok/s2,820 tok/s2,932 tok/s2,654 tok/s2,376 tok/s2,211 tok/s2,046 tok/s1,473 tok/s900 tok/s1,806 tok/s2,712 tok/s2,763 tok/s2,813 tok/s2,456 tok/s2,098 tok/s2,601 tok/s3,103 tok/s3,172 tok/s3,240 tok/s3,177 tok/s3,114 tok/s3,686 tok/s4,258 tok/s3,714 tok/s3,169 tok/s2,491 tok/s1,812 tok/s2,013 tok/s2,214 tok/s2,372 tok/s2,530 tok/s1,382 tok/s234 tok/s1,587 tok/s2,940 tok/s2,713 tok/s2,485 tok/s2,515 tok/s2,545 tok/s2,906 tok/s3,266 tok/s2,835 tok/s2,403 tok/s1,938 tok/s1,472 tok/s1,267 tok/s1,061 tok/s1,966 tok/s2,870 tok/s2,464 tok/s2,058 tok/s1,908 tok/s1,758 tok/s1,775 tok/s1,791 tok/s2,063 tok/s2,334 tok/s2,814 tok/s3,293 tok/s3,271 tok/s3,248 tok/s2,875 tok/s2,501 tok/s2,526 tok/s2,551 tok/s1,720 tok/s888 tok/s1,240 tok/s1,591 tok/s2,112 tok/s2,633 tok/s2,777 tok/s2,920 tok/s2,582 tok/s2,243 tok/s2,447 tok/s2,651 tok/s3,022 tok/s3,392 tok/s3,129 tok/s2,866 tok/s2,498 tok/s2,129 tok/s2,217 tok/s2,305 tok/s2,366 tok/s2,427 tok/s1,492 tok/s556 tok/s1,391 tok/s2,226 tok/s2,550 tok/s2,874 tok/s2,505 tok/s2,136 tok/s2,887 tok/s3,637 tok/s3,269 tok/s2,901 tok/s2,503 tok/s2,105 tok/s1,983 tok/s1,861 tok/s1,938 tok/s2,014 tok/s1,999 tok/s1,983 tok/s1,550 tok/s1,117 tok/s1,635 tok/s2,152 tok/s2,485 tok/s2,818 tok/s2,571 tok/s2,324 tok/s2,058 tok/s1,792 tok/s1,918 tok/s2,044 tok/s2,745 tok/s3,445 tok/s3,250 tok/s3,055 tok/s2,357 tok/s1,658 tok/s2,132 tok/s2,606 tok/s2,560 tok/s2,513 tok/s2,545 tok/s2,576 tok/s2,305 tok/s2,034 tok/s1,914 tok/s1,794 tok/s2,796 tok/s3,797 tok/s3,395 tok/s2,993 tok/s2,786 tok/s2,578 tok/s2,653 tok/s2,727 tok/s2,663 tok/s2,599 tok/s2,781 tok/s2,962 tok/s2,743 tok/s2,524 tok/s2,273 tok/s2,021 tok/s2,486 tok/s2,950 tok/s3,005 tok/s3,060 tok/s2,100 tok/s1,139 tok/s570 tok/s0 tok/s309 tok/s735 tok/s1,161 tok/s1,201 tok/s1,241 tok/s1,371 tok/s1,500 tok/s1,529 tok/s1,557 tok/s1,477 tok/s1,397 tok/s1,234 tok/s1,071 tok/s1,012 tok/s952 tok/s1,076 tok/s1,199 tok/s1,277 tok/s1,355 tok/s1,433 tok/s1,510 tok/s1,519 tok/s1,527 tok/s1,413 tok/s1,298 tok/s1,172 tok/s1,045 tok/s1,011 tok/s976 tok/s1,144 tok/s1,311 tok/s1,432 tok/s1,553 tok/s1,497 tok/s1,441 tok/s1,479 tok/s1,517 tok/s1,327 tok/s1,137 tok/s1,050 tok/s963 tok/s969 tok/s974 tok/s1,104 tok/s1,233 tok/s1,328 tok/s1,423 tok/s1,430 tok/s1,437 tok/s1,400 tok/s1,363 tok/s1,232 tok/s1,100 tok/s810 tok/s520 tok/s929 tok/s1,338 tok/s1,373 tok/s1,407 tok/s1,383 tok/s1,358 tok/s1,375 tok/s1,391 tok/s1,280 tok/s1,169 tok/s1,065 tok/s960 tok/s793 tok/s626 tok/s697 tok/s767 tok/s1,002 tok/s1,237 tok/s1,271 tok/s1,305 tok/s1,342 tok/s1,378 tok/s1,434 tok/s1,489 tok/s1,366 tok/s1,242 tok/s881 tok/s520 tok/s966 tok/s1,411 tok/s1,404 tok/s1,396 tok/s1,440 tok/s1,484 tok/s1,472 tok/s1,460 tok/s1,301 tok/s1,142 tok/s571 tok/s0 tok/s484 tok/s968 tok/s1,044 tok/s1,119 tok/s1,212 tok/s1,304 tok/s1,289 tok/s1,273 tok/s1,236 tok/s1,198 tok/s1,128 tok/s1,057 tok/s948 tok/s839 tok/s852 tok/s865 tok/s1,000 tok/s1,134 tok/s1,199 tok/s1,263 tok/s1,171 tok/s1,078 tok/s978 tok/s878 tok/s1,173 tok/s1,467 tok/s1,433 tok/s1,399 tok/s1,405 tok/s1,411 tok/s1,305 tok/s1,199 tok/s1,111 tok/s1,023 tok/s747 tok/s470 tok/s719 tok/s968 tok/s1,234 tok/s1,499 tok/s1,453 tok/s1,407 tok/s1,346 tok/s1,285 tok/s1,142 tok/s998 tok/s1,017 tok/s1,035 tok/s853 tok/s670 tok/s975 tok/s1,279 tok/s1,290 tok/s1,300 tok/s1,361 tok/s1,421 tok/s979 tok/s537 tok/s1,000 tok/s1,463 tok/s1,416 tok/s1,369 tok/s1,352 tok/s1,334 tok/s1,304 tok/s1,273 tok/s1,115 tok/s956 tok/s950 tok/s944 tok/s891 tok/s837 tok/s1,127 tok/s1,417 tok/s1,448 tok/s1,478 tok/s1,395 tok/s1,312 tok/s866 tok/s419 tok/s928 tok/s1,436 tok/s1,471 tok/s1,506 tok/s1,493 tok/s1,479 tok/s1,394 tok/s1,308 tok/s1,208 tok/s1,107 tok/s1,055 tok/s1,002 tok/s870 tok/s737 tok/s776 tok/s814 tok/s1,094 tok/s1,373 tok/s1,037 tok/s701 tok/s983 tok/s1,264 tok/s1,315 tok/s1,365 tok/s1,361 tok/s1,357 tok/s1,353 tok/s1,348 tok/s1,232 tok/s1,116 tok/s1,017 tok/s917 tok/s798 tok/s678 tok/s577 tok/s475 tok/s949 tok/s1,422 tok/s1,434 tok/s1,445 tok/s1,357 tok/s1,268 tok/s1,243 tok/s1,217 tok/s1,103 tok/s989 tok/s997 tok/s1,004 tok/s989 tok/s974 tok/s1,153 tok/s1,332 tok/s1,355 tok/s1,378 tok/s1,350 tok/s1,322 tok/s1,389 tok/s1,455 tok/s1,413 tok/s1,371 tok/s1,424 tok/s1,476 tok/s1,417 tok/s1,357 tok/s1,328 tok/s1,298 tok/s924 tok/s549 tok/s488 tok/s426 tok/s277 tok/s1,236 tok/s2,195 tok/s2,220 tok/s2,245 tok/s2,236 tok/s2,227 tok/s2,237 tok/s2,246 tok/s2,239 tok/s2,231 tok/s2,067 tok/s1,902 tok/s1,695 tok/s1,487 tok/s1,618 tok/s1,748 tok/s2,496 tok/s3,244 tok/s2,977 tok/s2,709 tok/s2,224 tok/s1,738 tok/s1,747 tok/s1,756 tok/s1,922 tok/s2,087 tok/s2,054 tok/s2,021 tok/s1,792 tok/s1,562 tok/s1,417 tok/s1,271 tok/s2,304 tok/s3,336 tok/s2,905 tok/s2,474 tok/s1,866 tok/s1,257 tok/s1,398 tok/s1,538 tok/s1,543 tok/s1,547 tok/s1,594 tok/s1,641 tok/s1,609 tok/s1,576 tok/s1,505 tok/s1,433 tok/s1,325 tok/s1,217 tok/s1,259 tok/s1,301 tok/s1,006 tok/s711 tok/s1,351 tok/s1,991 tok/s1,947 tok/s1,903 tok/s1,903 tok/s1,903 tok/s1,757 tok/s1,610 tok/s1,571 tok/s1,531 tok/s2,408 tok/s3,284 tok/s2,813 tok/s2,341 tok/s2,624 tok/s2,906 tok/s2,773 tok/s2,640 tok/s2,023 tok/s1,405 tok/s1,558 tok/s1,710 tok/s1,646 tok/s1,582 tok/s1,539 tok/s1,495 tok/s1,103 tok/s711 tok/s1,007 tok/s1,302 tok/s1,630 tok/s1,957 tok/s1,928 tok/s1,898 tok/s1,714 tok/s1,529 tok/s1,399 tok/s1,268 tok/s705 tok/s141 tok/s961 tok/s1,780 tok/s1,834 tok/s1,888 tok/s1,725 tok/s1,561 tok/s1,648 tok/s1,734 tok/s2,454 tok/s3,174 tok/s2,639 tok/s2,103 tok/s1,973 tok/s1,843 tok/s1,813 tok/s1,782 tok/s1,703 tok/s1,624 tok/s1,510 tok/s1,396 tok/s1,252 tok/s1,107 tok/s882 tok/s656 tok/s1,262 tok/s1,868 tok/s1,913 tok/s1,957 tok/s1,831 tok/s1,704 tok/s1,615 tok/s1,526 tok/s1,454 tok/s1,382 tok/s1,005 tok/s627 tok/s601 tok/s574 tok/s1,308 tok/s2,041 tok/s1,838 tok/s1,634 tok/s1,894 tok/s2,154 tok/s2,547 tok/s2,939 tok/s2,397 tok/s1,855 tok/s2,020 tok/s2,185 tok/s1,702 tok/s1,219 tok/s1,214 tok/s1,208 tok/s2,025 tok/s2,841 tok/s1,840 tok/s838 tok/s1,305 tok/s1,771 tok/s1,749 tok/s1,727 tok/s1,797 tok/s1,866 tok/s1,717 tok/s1,567 tok/s1,432 tok/s1,297 tok/s1,942 tok/s2,586 tok/s2,848 tok/s3,110 tok/s2,749 tok/s2,387 tok/s2,500 tok/s2,612 tok/s2,358 tok/s2,104 tok/s1,295 tok/s485 tok/s1,091 tok/s1,696 tok/s1,834 tok/s1,972 tok/s1,995 tok/s2,018 tok/s1,859 tok/s1,699 tok/s1,543 tok/s1,387 tok/s1,961 tok/s2,535 tok/s2,846 tok/s3,157 tok/s2,982 tok/s2,806 tok/s2,286 tok/s1,765 tok/s1,366 tok/s966 tok/s889 tok/s812 tok/s1,294 tok/s1,776 tok/s1,708 tok/s1,639 tok/s1,778 tok/s1,916 tok/s1,877 tok/s1,838 tok/s1,656 tok/s1,474 tok/s1,501 tok/s1,527 tok/s1,456 tok/s1,385 tok/s1,531 tok/s1,676 tok/s1,728 tok/s1,779 tok/s1,872 tok/s1,965 tok/s1,821 tok/s1,677 tok/s1,566 tok/s1,455 tok/s1,583 tok/s1,710 tok/s2,417 tok/s3,124 tok/s2,850 tok/s2,576 tok/s2,350 tok/s2,123 tok/s2,180 tok/s2,237 tok/s2,047 tok/s1,856 tok/s1,695 tok/s1,534 tok/s1,426 tok/s1,317 tok/s1,551 tok/s1,784 tok/s1,512 tok/s1,239 tok/s906 tok/s573 tok/s305 tok/s36 tok/s525 tok/s1,119 tok/s1,712 tok/s1,352 tok/s991 tok/s869 tok/s746 tok/s977 tok/s1,208 tok/s953 tok/s698 tok/s1,015 tok/s1,332 tok/s1,145 tok/s958 tok/s1,024 tok/s1,090 tok/s1,274 tok/s1,457 tok/s1,201 tok/s945 tok/s1,049 tok/s1,152 tok/s1,051 tok/s949 tok/s868 tok/s786 tok/s971 tok/s1,156 tok/s1,067 tok/s978 tok/s1,068 tok/s1,157 tok/s1,017 tok/s876 tok/s962 tok/s1,048 tok/s1,255 tok/s1,462 tok/s1,192 tok/s921 tok/s1,008 tok/s1,094 tok/s1,051 tok/s1,007 tok/s1,023 tok/s1,038 tok/s1,095 tok/s1,151 tok/s932 tok/s712 tok/s975 tok/s1,237 tok/s893 tok/s549 tok/s842 tok/s1,134 tok/s1,111 tok/s1,088 tok/s1,004 tok/s919 tok/s1,029 tok/s1,138 tok/s1,000 tok/s861 tok/s1,083 tok/s1,304 tok/s1,230 tok/s1,155 tok/s1,075 tok/s994 tok/s1,046 tok/s1,097 tok/s1,083 tok/s1,069 tok/s1,093 tok/s1,116 tok/s1,108 tok/s1,100 tok/s1,016 tok/s931 tok/s700 tok/s468 tok/s812 tok/s1,155 tok/s1,194 tok/s1,232 tok/s1,120 tok/s1,007 tok/s1,073 tok/s1,139 tok/s1,103 tok/s1,066 tok/s803 tok/s539 tok/s809 tok/s1,079 tok/s958 tok/s837 tok/s1,008 tok/s1,179 tok/s1,038 tok/s897 tok/s1,066 tok/s1,235 tok/s1,243 tok/s1,250 tok/s1,064 tok/s877 tok/s1,007 tok/s1,137 tok/s1,025 tok/s912 tok/s1,037 tok/s1,162 tok/s1,058 tok/s954 tok/s840 tok/s725 tok/s780 tok/s835 tok/s876 tok/s917 tok/s1,048 tok/s1,178 tok/s1,078 tok/s977 tok/s1,139 tok/s1,301 tok/s977 tok/s653 tok/s899 tok/s1,144 tok/s959 tok/s774 tok/s785 tok/s795 tok/s948 tok/s1,100 tok/s1,045 tok/s990 tok/s1,091 tok/s1,191 tok/s1,196 tok/s1,200 tok/s975 tok/s749 tok/s968 tok/s1,186 tok/s1,026 tok/s865 tok/s618 tok/s370 tok/s909 tok/s1,448 tok/s1,134 tok/s820 tok/s910 tok/s1,000 tok/s1,073 tok/s1,146 tok/s1,020 tok/s894 tok/s1,133 tok/s1,371 tok/s1,145 tok/s918 tok/s1,028 tok/s1,138 tok/s1,046 tok/s954 tok/s1,019 tok/s1,083 tok/s803 tok/s523 tok/s818 tok/s1,113 tok/s918 tok/s722 tok/s1,006 tok/s1,290 tok/s1,208 tok/s1,126 tok/s1,089 tok/s1,051 tok/s999 tok/s946 tok/s1,181 tok/s1,416 tok/s1,233 tok/s1,050 tok/s1,064 tok/s1,077 tok/s874 tok/s670 tok/s919 tok/s1,167 tok/s1,051 tok/s934 tok/s1,064 tok/s1,193 tok/s1,048 tok/s903 tok/s1,119 tok/s1,334 tok/s1,240 tok/s1,146 tok/s1,083 tok/s1,019 tok/s794 tok/s569 tok/s913 tok/s1,257 tok/s1,239 tok/s1,220 tok/s1,144 tok/s1,067 tok/s1,115 tok/s1,163 tok/s1,091 tok/s1,019 tok/s1,121 tok/s1,223 tok/s1,248 tok/s1,272 tok/s1,067 tok/s861 tok/s1,131 tok/s1,401 tok/s1,191 tok/s980 tok/s943 tok/s906 tok/s1,254 tok/s1,602 tok/s1,248 tok/s893 tok/s1,051 tok/s1,208 tok/s1,144 tok/s1,079 tok/s723 tok/s367 tok/s433 tok/s499 tok/s0TRAINER · 16× B300two nodes over InfiniBandWAITING FOR ROLLOUTSstep 0 · 0 adapter syncs landedWAITING FOR ROLLOUTSstep 0 · 1 adapter syncs landedTRAININGstep 0 · 1 adapter syncs landedWAITING FOR ROLLOUTSstep 1 · 1 adapter syncs landedWAITING FOR ROLLOUTSstep 1 · 2 adapter syncs landedTRAININGstep 1 · 2 adapter syncs landedWAITING FOR ROLLOUTSstep 2 · 2 adapter syncs landedWAITING FOR ROLLOUTSstep 2 · 3 adapter syncs landedTRAININGstep 2 · 3 adapter syncs landedWAITING FOR ROLLOUTSstep 3 · 3 adapter syncs landedWAITING FOR ROLLOUTSstep 3 · 4 adapter syncs landedTRAININGstep 3 · 4 adapter syncs landedWAITING FOR ROLLOUTSstep 4 · 4 adapter syncs landedWAITING FOR ROLLOUTSstep 4 · 5 adapter syncs landedTRAININGstep 4 · 5 adapter syncs landedWAITING FOR ROLLOUTSstep 5 · 5 adapter syncs landedWAITING FOR ROLLOUTSstep 5 · 6 adapter syncs landedTRAININGstep 5 · 6 adapter syncs landedWAITING FOR ROLLOUTSstep 6 · 6 adapter syncs landedWAITING FOR ROLLOUTSstep 6 · 7 adapter syncs landedTRAININGstep 6 · 7 adapter syncs landedWAITING FOR ROLLOUTSstep 7 · 7 adapter syncs landedWAITING FOR ROLLOUTSstep 7 · 8 adapter syncs landedTRAININGstep 7 · 8 adapter syncs landedWAITING FOR ROLLOUTSstep 8 · 8 adapter syncs landedWAITING FOR ROLLOUTSstep 8 · 9 adapter syncs landedTRAININGstep 8 · 9 adapter syncs landedWAITING FOR ROLLOUTSstep 9 · 9 adapter syncs landedWAITING FOR ROLLOUTSstep 9 · 10 adapter syncs landedTRAININGstep 9 · 10 adapter syncs landedWAITING FOR ROLLOUTSstep 10 · 10 adapter syncs landedWAITING FOR ROLLOUTSstep 10 · 11 adapter syncs landedWAITING FOR ROLLOUTSstep 0 · 1 adapter syncs landedFLAGSHIP RUN · NOV 5 · ONE TRAINER · FOUR ROLLOUT ISLANDS · TWO VENDORSGLM-5.3 744B · LoRA · fully async · every sync over plain HTTP012345678950123401:02:03:04:05:06:07:08:09:10:11:12:03:31UTC+3InfiniBand: training trafficHTTP, no shared collectiveadapter (4 GB), every steprollouts, rate ∝ tokens/ss0s1s2s3s4s5s6s7s8s902:0003:0004:0005:0006:0007:0008:0009:0010:0011:0012:00STEP 0 · WAITING FOR ROLLOUTSSTEP 0 · WAITING FOR ROLLOUTSSTEP 0 · TRAININGSTEP 1 · WAITING FOR ROLLOUTSSTEP 1 · WAITING FOR ROLLOUTSSTEP 1 · TRAININGSTEP 2 · WAITING FOR ROLLOUTSSTEP 2 · WAITING FOR ROLLOUTSSTEP 2 · TRAININGSTEP 3 · WAITING FOR ROLLOUTSSTEP 3 · WAITING FOR ROLLOUTSSTEP 3 · TRAININGSTEP 4 · WAITING FOR ROLLOUTSSTEP 4 · WAITING FOR ROLLOUTSSTEP 4 · TRAININGSTEP 5 · WAITING FOR ROLLOUTSSTEP 5 · WAITING FOR ROLLOUTSSTEP 5 · TRAININGSTEP 6 · WAITING FOR ROLLOUTSSTEP 6 · WAITING FOR ROLLOUTSSTEP 6 · TRAININGSTEP 7 · WAITING FOR ROLLOUTSSTEP 7 · WAITING FOR ROLLOUTSSTEP 7 · TRAININGSTEP 8 · WAITING FOR ROLLOUTSSTEP 8 · WAITING FOR ROLLOUTSSTEP 8 · TRAININGSTEP 9 · WAITING FOR ROLLOUTSSTEP 9 · WAITING FOR ROLLOUTSSTEP 9 · TRAININGSTEP 10 · WAITING FOR ROLLOUTSSTEP 10 · WAITING FOR ROLLOUTS
01:55 → 12:05 UTC+3 in 36 seconds · click the timeline to jump
Trainer16× B300 · two nodes over InfiniBand
MI355XAMD CDNA4 · 3.0M KV · 2,490 tok/s
MI300XAMD CDNA3 · 1.2M KV · 1,180 tok/s
B300NVIDIA Blackwell · 2.7M KV · 1,790 tok/s
H200NVIDIA Hopper · 0.4M KV · 1,030 tok/s

Figure 1. The flagship run, 01:55 to 12:05 UTC+3 on November 5. Island size is each engine's KV-cache capacity; the squares in each callout are its KV cache in use; dots flow home at each island's measured tokens per second. Every quantity comes from the run's logs and a 30-second logger on each engine.

Jump target #s01 · invisible on the published site

01 Compute comes in islands

Accelerators do not arrive as one clean cluster. A few nodes of one chip sit in one rack, a different chip in the next building, spot capacity appears for an afternoon and leaves. NVIDIA supply is tight, and a wave of new silicon is arriving, much of it built for inference first. Each of these is an island: useful on its own, hard to combine.

Training works best on one island. Its gradient all-reduce wants every GPU on one fast fabric, from one vendor, and it stalls on the slowest link. Training can span islands, but it pays for every hop. Rollout generation is different. Each engine serves the model by itself, takes prompts and returns tokens. In LLM RL that is most of the compute.

Every inference server is a rollout server waiting for a trainer.

The idea we set out to test

The bet is that training and generation do not have to live together. Training is happiest on one tight island. Rollouts only need tokens per second, from anywhere. If any machine that can run the inference engine and expose a safe port can serve rollouts, then every spare node, every new chip and, eventually, every datacenter becomes part of the RL fleet.

The timing matters. NVIDIA supply is tight, and the rest of the field is built to serve tokens: AMD’s Instinct line, Google’s TPUs (which SGLang already serves through SGLang-Jax), AWS Trainium (SGLang added a Trainium plugin in October, already serving GLM-5.3 and Miles rollouts), Tenstorrent, and inference-first designs like Etched’s and Rebellions’. RL is a training workload those chips can already carry, because most of RL is inference. As their software matures, teams that can mix islands get compute that others cannot use.

What decides it is software, not silicon. An engine on any chip has to honour the same contract as one on a B300: the same numerics, the RL features, cheap weight updates. This post tests that contract on four chips from two vendors, and says plainly where it holds only because of LoRA.

Jump target #s02 · invisible on the published site

02 Make the rollout engine a contract

An engine can join the rollout pool if it honours three things. None of them names a vendor, though how much work each takes, numerics most of all, depends on how mature a chip’s software is.

  • Correct numerics. The trainer re-scores every sampled token itself, so an engine’s probabilities have to match the trainer’s closely. We measure the gap as KL per step.

  • The RL features. Log-probabilities with every token, abort, and taking new weights while requests are still running.

  • LoRA-sized updates over plain HTTP. Once per step the trainer posts the new adapter to every engine (miles#3690), one copy per engine (sglang#48241). There is no shared collective between trainer and pool, and no shared filesystem.

TRAINERrank 0B300MI355XMI300XH200ROLLOUT ENGINES · ANY VENDOR1POST /pause_generationengines hold decoding for the swap2POST /register_lora_adapterfirst sync only · adapter name, rank, alpha3POST /begin_weight_updateopen the update session4POST /update_weights_from_tensor × 8512 MB chunk · encoded once · the same bytes to every engine5POST /end_weight_updateclose the session · then /update_weight_version stamps vN6POST /continue_generationrollouts resume on the new adapter
TRAINERrank 0 · B300
B300MI355XMI300XH200
Rollout engines · any vendor
  1. 1
    POST /pause_generation

    engines hold decoding for the swap

  2. 2
    POST /register_lora_adapter

    first sync only · adapter name, rank, alpha

  3. 3
    POST /begin_weight_update

    open the update session

  4. 4
    POST /update_weights_from_tensor × 8

    512 MB chunk · encoded once · the same bytes to every engine

  5. 5
    POST /end_weight_update

    close the session · then /update_weight_version stamps vN

  6. 6
    POST /continue_generation

    rollouts resume on the new adapter

Figure 2. One adapter sync, as the HTTP calls the trainer makes to every engine. Each 512 MB chunk is encoded once and posted to all engines in parallel; the chunks go one after another, so a sync takes about eight times the slowest engine's time per chunk. The register call happens on the first sync only.
Jump target #s03 · invisible on the published site

03 Why this works today, and where it would not

It works partly because of LoRA

Each sync moves a 4 GB adapter; the 709 GB fp8 base never moves, because every engine loaded its own copy once. On our 25 Gb/s Ethernet the adapter is about a second of wire time; the full model would be about four minutes per engine, every step.

It ran inside one datacenter, but the wire is not the limit

Distance matters less than it sounds. A sync is one 4 GB transfer, not a chatty collective, so latency barely counts. Over a modest 100 MB/s path, like the public internet between sites, the adapter takes about 40 seconds against a 50-minute step, while the 709 GB base would take about two hours, longer than the step itself. Over a dedicated 100–800 Gbps interconnect, the adapter takes well under a second and even the full base a minute or less per engine. At that point the wire stops being the limit, and, as in section 06, serialization and one copy per engine take over.

One HTTP API, two ways to move the adapter

NCCL and RCCL cannot share a communicator, so updates travel outside the GPU collective. HTTP runs over any IP network, so engines in other racks, clouds or datacenters can join. One copy per engine still scales: with a small model, 24 engines across three chip types sync in about 8 seconds. For much larger pools, the same http-lora API has a second mode, path mode: the trainer writes the adapter once to shared storage and sends each engine only its path (miles#3690, sglang#47135). These runs used the default, tensors mode, which streams the adapter to each engine.

EnginesChipsSync (s)Posting (s)
8MI355X5.53.5
24MI355X · H200 · B3008.15.9
Scale-out. One B300 trainer, Qwen3-4B with a rank-64 LoRA (264 MB adapter), one TP1 engine per GPU, tensors mode, a copy per engine. Sync is the full weight update per step, including the pause; posting is the time spent sending. Each extra engine adds about 0.15 s.

It needs gates

Every engine passes a known-answer check, a log-probability parity check and a diff of the configuration it actually resolved. Without them we would have trained on wrong outputs at least twice.

Jump target #s04 · invisible on the published site

04 Five arms, one variable at a time

Same model, same recipe (math problems from DAPO-Math-17k, filtered to a pass-rate band, with 12k-token responses), one 8× B300 trainer node on an AMD EPYC host. Each arm changes one thing.

ArmRollout enginesWeight syncQuestion
A−1× MI355X nodeHTTPbaseline at 8k-token responses
A121× MI355X nodeHTTPNVIDIA trains, AMD generates
B121× B300 nodeHTTPthe same, all NVIDIA
C121× B300 nodeNCCL over InfiniBanddoes the wire matter?
D122× MI355X nodesHTTPdo more rollout nodes help?
Jump target #s05 · invisible on the published site

05 Where the step goes

At 8k-token responses the trainer barely waited. At 12k, generation and training take about as long as each other, so the trainer waits whenever rollouts run dry. The flagship, with a two-node trainer, is rollout-bound throughout: after the first step, 3 to 56 minutes of waiting against about 24 of training.

waiting on rolloutstraining
A− · 1× MI355X · HTTP · 8k
35 min
A12 · 1× MI355X · HTTP
53 min
B12 · 1× B300 · HTTP
51 min
C12 · 1× B300 · IB
46 min
D12 · 2× MI355X · HTTP
46 min

Flagship: band narrowed to 35–65% to stay rollout-bound → 2.4× more generation per step

Flagship · 4 chips · HTTP · 2× trainer
52 min
0 min30 min60 min
Figure 3. Minutes for step 2 (flagship: average step), split into waiting on rollouts (hatched) and training (solid). The flagship trains on two B300 nodes, so training takes about 24 minutes instead of 40 to 50.
Jump target #s06 · invisible on the published site

06 The wire barely matters

B12 and C12 differ only in how weights reach the engine: HTTP between two B300 nodes, or NCCL broadcast over 8×800G InfiniBand. The links differ by about 250×, but the syncs by 6.5×: almost all of a sync is serialization and engine-side work, not time on the wire.

C12 · 1× B300 · IB
9 s · 0.3% of step
B12 · 1× B300 · HTTP
61 s · 2.0% of step
A12 · 1× MI355X · HTTP
91 s · 2.9% of step
D12 · 2× MI355X · HTTP
105 s · 3.8% of step

Flagship: one copy to each of four engines on four chips, from a two-node trainer

Flagship · 4 chips · HTTP · 2× trainer
158 s · 5.0% of step
0 s60 s120 s180 s
Figure 4. Mean weight-update time per step. HTTP sends one copy per engine. The adapter arrives in eight 512 MB chunks: a B300 engine takes about 7.4 s per chunk, an MI355X about 11. The final swap-in takes 2 s on both, so the gap is in receiving and unpacking the tensors, not in loading the adapter.

InfiniBand is 6.5× faster per sync and saves under 2% of a 50-minute step.

With LoRA, moving the policy costs about a minute and generating the next batch costs most of an hour. As frontier RL moves to longer, agentic rollouts, that share only shrinks. The fast fabric earns its keep inside the trainer, where the gradients are.

Jump target #s07 · invisible on the published site

07 Add rollout nodes until the trainer is the limit

Engine output, tok/s1.9×
2,690
5,190
1× MI355X2× MI355X
Step time, min1.2×
53
46
1× MI355X2× MI355X
Figure 5. A12 against D12, both with fp8 KV on an EPYC trainer. Engine output is mean generated tokens per second from the engine logger; step time is step 2.

A second MI355X node almost doubled generation, from 2,690 to 5,190 tokens per second, and cut the trainer’s wait at step 1 from 51 minutes to under 2. One fp8 MI355X node falls just short of keeping a fast 8× B300 trainer fed; two leave headroom.

Every engine here uses an fp8 KV cache.

Jump target #s08 · invisible on the published site

08 In our runs, precision mattered more than the vendor

The trainer re-scores every sampled token, so the train/rollout KL measures how far each engine’s probabilities sit from the trainer’s.

KL ×10−3
010203040
step 0123456789
Flagship · 4 chips · HTTP · 2× trainerD12 · 2× MI355X · HTTPC12 · 1× B300 · IBA12 · 1× MI355X · HTTPB12 · 1× B300 · HTTPA12 · MI355X · bf16 KV (old)
Figure 6. KL per step (arms: steps 0 to 2). Dotted: MI355X with a bf16 KV cache. The flagship mixes all four chips in every batch.

With the same fp8 KV cache, MI355X and B300 land in the same range at step 0: 13.9–15.7 against 16.6–16.7×10⁻³, inside the run-to-run spread. A bf16 cache sits 2.4× closer to the trainer. So in these runs, KV-cache precision moved the gap far more than the vendor did. We measured the gap itself, not its effect: we did not test whether it changes what the model learns over a long run.

Numerics is still the part to be most careful about. AMD and NVIDIA matched here, on mature kernels. TPUs, Intel GPUs, Trainium and inference-first chips run different kernels and number formats, and their RL paths are younger, so matching the trainer may take more work. That is why correct numerics is the contract’s first clause, checked before every run.

Jump target #s09 · invisible on the published site

09 One trainer, four islands, ten steps

Then everything in one run. The trainer spans two B300 nodes over InfiniBand. The rollout pool is one node each of B300, MI355X, MI300X and H200: two vendors, two generations each, 48 GPUs in total. It is the largest run we have done. All four engines use fp8 KV cache and passed the gates before the trainer started.

StepWait (min)Train (min)Sync (s)KL ×10⁻³
0109.725.716016.6
147.923.515414.8
22.824.715617.7
336.722.715618.8
48.624.916224.1
529.224.616324.4
629.824.716131.2
726.724.615739.5
814.924.715534.2
956.423.915447.4

Ten steps on four chips from two vendors. Every adapter sync landed. Zero restarts.

KV cache in use, 0–100% · UTC+3

MI355X3.0M KV
B3002.7M KV
MI300X1.2M KV
H2000.4M KV
02:0003:0004:0005:0006:0007:0008:0009:0010:0011:0012:00
Figure 7. KV cache in use on each engine through the flagship run (5-minute means of a 30-second logger). The H200 has the smallest cache and fills first. The dips are the syncs, when each engine briefly drains.
EngineAttention (DSA)MoE · LoRARunningTok/s, whole run
B300trtllmTriton · Triton~6401,790
MI355XtritonTriton · Triton~6602,490
MI300XtritonTriton · Triton~2001,180
H200flashmla_kv · flashmla_sparse_q8Triton · Triton~801,030

Every engine: TP8 on one node, fp8 KV cache, CUDA or HIP graphs on, no speculative decoding, no data-parallel attention. Means of a 30-second logger, 01:55 to 12:05 UTC+3.

Jump target #s10 · invisible on the published site

10 A mixed pool needs a router that knows capacity

The router policy we started with, SGLang’s cache_aware, balances load by counting requests, so with little shared prefix every engine gets about the same number in flight. That is right for identical engines. Ours differ by 7× in KV capacity.

Count-based
cache_aware · equal requests per engine
B300
26%
MI355X
27%
MI300X
23%
H200
23%
By KV capacity
sgl-router#30
B300
37%
MI355X
40%
MI300X
17%
H200
5%
Tokens generated
whole run
B300
28%
MI355X
38%
MI300X
18%
H200
16%
Figure 8. Share of in-flight requests (running plus queued) per engine under count-based routing (an earlier flagship attempt, averaged over its roughly 30 minutes) and under capacity-weighted routing (averaged over the flagship night), and each engine's share of all tokens generated over the run.

Under count-based routing, the H200 had about 870 requests waiting while running about 70, and the B300 sat at its limit with more than half its KV cache free. Capacity-weighted routing (sgl-router#30) gives each engine work in proportion to its KV capacity, and the split now tracks capacity within a point.

The run also showed the limit of that idea. Capacity is not throughput: the B300 holds 37% of requests and produces 28% of tokens, the H200 holds 5% and produces 16%.

We measured it on a five-engine pool replaying the same 3,072 rollout requests: weighting by KV capacity finished the batch 25% faster than round-robin, with 11% higher steady throughput. Weighting by each engine’s current speed did not beat it. The next step is marginal throughput: send each request where one more request adds the most tokens per second.

Jump target #s11 · invisible on the published site

11 What we had to patch

These are the issues we hit and patched along the way to get the runs working. None needed a new chip feature; most were defaults or kernel paths few people had run with LoRA on this model. Some are hard to reproduce, versions move quickly, and you may hit different ones. The table is here so they don’t surprise you, and so you know roughly where the fix is.

What we hitWhereEffectWhat we did
fp8 MoE weights pre-shuffled for a kernel the engine wasn't usingMI355Xfluent but wrong outputsonly shuffle for the kernel in use (fixed upstream since)
Attention matmul tile size crashes the compiler at some batch sizesMI355Xcrash at 65–256 rowsclamp the kernel’s block size
Data-parallel attention with LoRA and graphsMI355Xwrong outputs on 7 of 8 ranksrun plain TP8, no data-parallel attention
New fused activation path reads a field LoRA layers lackMI300Xcrash at startupskip the fused path on LoRA layers
LoRA kernel launch asks for more resources than the GPU allowsMI300Xcrash after 30 min of loadcap running requests at 256 (not root-caused)
Same flags, different KV precision: fp8 on Blackwell, bf16 elsewhereall2× capacity gap, silentset the fp8 KV cache explicitly
Default fp8-KV attention kernel shifts first-token choicesH200wrong greedy answersuse the SGLang cookbook’s Hopper attention backends
Load-aware routing refuses to start with external enginesrouterlaunch failureuse a policy that accepts engines joining after startup

Two patterns. LoRA is the untested path: turning it on cost about 35% throughput on both vendors, because both fall back to generic MoE kernels. And identical launch settings do not give identical engines: each chip quietly picks its own kernels and defaults, so it pays to check what every engine actually resolved.

Hardware diversity is already here. Software maturity is the gap.

Jump target #s12 · invisible on the published site

12 Limits of these results

  • Learning. Raw reward over the flagship’s ten steps is flat and noisy, as expected for a short run on a strong model, so these runs make no claim about learning speed. Our first test did learn: GLM-5.2 744B with a LoRA trained on a B300 node and rolled out on an MI300X pool, 30 steps of Countdown over about 10 hours with zero failed syncs. Held-out accuracy rose from 0.820 to 0.910 (64 problems, 4,096-token answers).

  • Response length. About 60% of responses hit the 12k cap. On one real training prompt the model was still mid-derivation at 12,288 tokens, so the cap cuts real reasoning and caps reward.

  • Peak speed. Every engine ran a generic LoRA path: Triton MoE kernels, no speculative decoding, no data-parallel attention. These runs measure the contract, not each chip’s best, and the per-engine gaps will move as each vendor’s software matures.

  • Policy drift. Train/rollout KL in the flagship rose from about 15 to 47 ×10⁻³ over ten steps. Two things add to it that this run cannot separate: fully async training scores some rollouts against a newer adapter than the one that generated them, and the growing LoRA adds a kernel path that differs between engine and trainer. We used the standard GRPO recipe with PPO clipping and no importance-sampling correction for the mismatch. Whether the drift stays small over hundreds of steps, or needs that correction, is the main open question for this setup.

  • Few steps per arm. Most arms ran three steps (C12 ran five).

  • Controls. Arms A to D are controlled comparisons (A− on an Intel trainer host). The flagship is a demonstration and changes several things at once.

Jump target #s13 · invisible on the published site

13 Where this sits

Core Automation’s The Hitchhiker’s Guide to Training on Compute Islands (Rohan Anil, November 2026) takes the training side of the same problem: one MoE model trained across two GPU clusters about 9,000 miles apart, exchanging compressed model fragments every ~50 steps instead of every gradient. This post is the rollout-side counterpart: the trainer stays on one island, and generation spreads across the rest.

INTELLECT-2 (Prime Intellect) trained a 32B model with RL from rollout workers spread across the internet. StreamRL disaggregates generation onto separate GPU clusters, including different GPU types. HetRL (MLSys 2026) schedules RL across mixed NVIDIA GPUs (A100, L40S, L4) and simulated wide-area networks, and lists other vendors as untested. SGLang’s Trainium plugin already serves Miles rollouts from Trainium3 to an H200 trainer. Modal feeds one BF16 trainer from samplers on six NVIDIA GPU types in three weight formats, and finds that off-policy correction is what keeps training stable: without it, GRPO collapsed on mixed samplers soon after starting, and even on matched hardware and precision it only lasted longer. We used no correction in these short runs, and our flagship’s KL was still rising when it ended (section 12), so their result is the main thing to test next. Our piece is complementary: one pool that mixes vendors and generations at one precision, feeding one trainer, on the mainstream open stack, with the changes sent upstream and controlled arms measuring the cost.

Jump target #s14 · invisible on the published site

14 What we would run next

  • Marginal-throughput routing: weight engines by what one more request adds, not by their current speed.

  • A longer run with a held-out evaluation curve, with and without off-policy correction.

  • Each chip on its fastest path, under the same gates.

  • Path mode at dozens of engines, and a run across two datacenters.

Jump target #s15 · invisible on the published site

15 What we sent upstream

Everything here runs on Miles and SGLang with these changes, plus a few local patches.*

PRWhat it doesStatus
miles#3690http-lora: LoRA weight sync to external engines over HTTP, tensors or path modemerged
sglang#48241one adapter payload shared by every TP rank, so each engine gets one copymerged
sglang#47135upsert on /load_lora_adapter: swap an adapter in placemerged
miles#3693sequence packing for GLM-5.x LoRA on tilelang, about 2.2× faster train stepsin review
sgl-router#30capacity_aware routing by KV capacity in sgl-routerin review

* The ROCm and LoRA fixes in section 11 are local patches, not upstream. Results may not reproduce exactly on newer versions, since these code paths change quickly and upstream fixes keep landing.

Jump target #s16 · invisible on the published site

16 Acknowledgements

We would like to thank Yueming Yuan for reworking, reviewing and merging the transport. Alex Nails for the first conversations, design guidance and getting it reviewed. Zhichen Zeng for the early review and the green light. And Banghua Zhu for the intro that started this work. All four are at RadixArk, whose work on Miles and SGLang this all builds on.

References

Related systems.

Stack and our changes.

  • Miles, the RL framework the trainer runs on.

  • SGLang, the inference engine on every rollout chip.

  • sgl-router-for-miles, the router Miles uses by default.

  • The pull requests we sent upstream are listed in section 15.

Citation

@misc{calik2026heterorl,  title  = {Train on one chip, roll out on any: a field guide to RL on heterogeneous hardware},  author = {Calik, Yagiz},  year   = {2026},  month  = nov,  note   = {ai\& Research blog post}}