The road to the fastest, cheapest H3

Shimpei Hara, Co-founder and President

Fazliddin Vakhobov (Mike), Member of Technical Staff

01 Why we wanted to share this work

At ai&, our mission is to make powerful models as fast and cheap to use as possible. As agentic applications scale, LLM inference has attracted substantial research and engineering effort, and the inference community has done a ton of work pushing the Pareto frontier between throughput and interactivity.

Much of this work addresses autoregressive LLMs: producing tokens sequentially while repeatedly moving weights and attention state through memory. Diffusion takes a different approach, learning to recover data from noise or masked inputs through repeated refinement. Video models such as H3 use it to generate motion and sound together; masked diffusion language models such as LLaDA predict multiple missing text positions within a pass. These execution patterns create different opportunities for parallelism, caching, and memory reuse. Model architectures will continue to evolve alongside the hardware available to run them.

We believe that more use cases of combining multiple modalities will emerge soon, and that is why we expanded our inference work into video. We want to be good at serving all inference.

As a heterogeneous provider operating AMD, NVIDIA, and Tenstorrent systems, we wanted to understand how far each architecture could take H3. We started with the full model, then worked through the changes that made it faster, the quality tradeoffs they introduced, and the cost of turning those gains into a service.

02 The hardware and the full-model baseline

We started by comparing NVIDIA B300, AMD MI355X, and Tenstorrent Blackhole to establish a baseline for H3 and understand how compute, memory, and communication shape video-diffusion performance.

We compare both one and four Blackhole Galaxies because four systems sit in the same broad purchase-budget range as the eight-GPU systems, giving us a more useful starting point for evaluating total cost of ownership.

Memory prices are rising sharply, putting upward pressure on server costs, so these starting estimates need to be checked against fresh quotes week to week.

On the GPUs, we used SGLang with the model’s native BF16/FP32 precision policy and fifty requested sampling steps, with fixed inputs at 1344×768 and seed 42. Both eight-GPU configurations already used sequence parallelism. On Tenstorrent, we used the native full-H3 implementation on one and four Galaxies.

NVIDIA B300 finished the fifteen-second case about 1.3× faster than MI355X, on August software for the B300 and our current build for the MI355X. Beyond matrix throughput, Blackwell Ultra doubles key softmax throughput relative to Blackwell, a relevant advantage for H3’s long-sequence attention.

AMD MI355X gives us substantial compute and memory headroom, and ROCm’s progress has made it an increasingly capable inference platform. Our full-model H3 baseline takes 133 seconds, leaving room to realize more of its higher theoretical BF16 throughput. Continued work on kernels, scheduling, and communication is how we turn that capacity into faster video generation.

Tenstorrent Blackhole took 229.1 seconds on one Galaxy and 68.7 seconds on four. Each Galaxy distributes compute across a network of cores and pairs GDDR6 with 6.2 GB of aggregate SRAM at a quoted 2.9 PB/s of local bandwidth. Scaling adds compute, local memory, and bandwidth together; communication and synchronization limit how much of that translates into a speedup.

The performance gaps give us a starting point for optimization.

03 Why video changes the inference workload

To understand these results, we first need to look at how video changes the inference workload. LLMs and video models both use transformers, but organize generation differently. LLM decoding extends a sequence one discrete token at a time. H3 repeatedly refines continuous representations of video and audio across an entire clip. That difference changes where parallelism comes from, what can be cached, and how the hardware spends its time.

H3 does this refinement on a compressed representation of the clip, rather than directly on every pixel and sound sample. Its diffusion transformer starts from noise; separate decoders turn the refined representation into frames and audio. This reduces the data involved, but still leaves a long sequence to process repeatedly.

Refining a whole clip

During vanilla LLM decoding, the model adds one token to a growing sequence. Earlier tokens stay fixed, and a KV cache retains their attention state for subsequent steps.

H3 uses flow matching to predict adjustments to the video and audio latents at each update. Positions across the clip are refined together, so later frame positions can influence earlier ones. The updates themselves remain sequential: each needs the previous update’s result.

This distinction explains why the usual KV-cache shortcut does not carry over. H3’s video and audio states change between updates, so their attention keys and values generally change too. Reusing those results across steps means approximating work the model would otherwise recompute.

Parallelism within one video

Batching is central to LLM serving because a single decoding request gives the GPU relatively little work at each step. Processing several requests together lets the hardware reuse loaded weights across more calculations, increasing throughput and lowering the cost per token.

Video diffusion starts from a different position. H3 processes thousands of token positions together within each refinement step. Although this is still one request, it already provides much of the parallel work and weight reuse that batching creates for LLM decoding. We can batch multiple videos, but for a long sequence like this, the opportunity to improve utilization is smaller once one generation is keeping the hardware busy.

That changes how we approach optimization. In LLM decoding, we often improve efficiency by putting more requests through the model together. For H3, we first look at how efficiently the hardware executes the substantial work already inside one request: how quickly it computes attention, how often it moves intermediate data, and where it waits for communication. Adding another video does not remove those costs.

Our focus therefore moves inside the generation. We can distribute its sequence across GPUs, improve the kernels and data movement, or reduce the computation required at each refinement step.

We had memory to spare, but the video still took time to compute. Our AMD MI355X text-to-video runs used roughly 99–113 GiB per GPU, about 40% of its 288 GB. The model and its working data fit comfortably; adding more capacity would not accelerate the repeated attention and matrix operations. That is why our focus shifted to faster kernels and distributing the work within one generation. Free memory capacity does not tell us how much bandwidth or compute a request is using.

04 Making H3 faster to serve

We started with the full model running through the available inference stack on each platform. On NVIDIA and AMD, that meant SGLang with eight-way Ulysses sequence parallelism, the model’s native BF16/FP32 precision, and fifty requested sampling steps. We processed one generation at a time, with caching and online quantization disabled. On Blackhole Galaxy, we used Tenstorrent’s native H3 implementation. These baselines already benefit from each runtime’s kernels and execution strategy; the opportunity was to understand what still limited performance and push further.

That requires looking beyond whether the model fits in memory or how much compute the hardware advertises. A request can spend time compiling a kernel, moving intermediate results, waiting for another device, or repeating calculations that could be prepared or reused. The useful optimization depends on which of those costs dominates—and whether removing it preserves the computation or changes the generated result.

Coordinating computation and data movement

Tenstorrent’s TT-DiT code offers a concrete example of this thinking. After attention, results from different processors need to be gathered, projected, and combined with the residual stream. Its optimized implementation fuses these operations, dedicates cores to communication, and reuses preallocated buffers. Coordinating them allows data transfers and arithmetic to overlap while avoiding separate passes over intermediate tensors.

Our GPU work followed the same principle: identify where the request was spending time, make a targeted change, and measure the complete generation. We began with delays before denoising, then turned to the repeated computation inside it.

Removing delays before generation

On AMD, a patch-embedding convolution in the text encoder changed shape with the geometry of a reference image. An unseen shape could trigger MIOpen kernel compilation, while container restarts discarded the local cache. The request paid for that compilation before denoising could begin.

We replaced the convolution with a mathematically equivalent unfold-plus-matrix-multiply operation, enabled fast kernel selection, and persisted the MIOpen databases on the node.

These changes removed startup and unseen-shape overhead without reducing the model’s requested steps or precision.

Accelerating the denoiser

In our warm fifteen-second AMD trace, denoising accounted for 91–97% of engine time across the three execution modes. We explored four ways to reduce that work: reuse intermediate results, reduce precision, compute fewer attention interactions, and generate in fewer refinement steps.

1. Reusing work with Cache-DiT

Successive refinement steps often produce similar intermediate results. Cache-DiT uses that similarity to reuse earlier work instead of rerunning every transformer block.

This differs from the usual LLM KV cache. Earlier tokens remain fixed during causal decoding, while H3’s video and audio states change between refinement steps. Reusing those changing states introduces an approximation.

We used a Cache-DiT profile with four initial full-compute steps, followed by reuse controlled by a change threshold and a maximum of two consecutive cached steps. In one AMD dialogue trace, a full step took 2.53 seconds, while a cached step took 0.05 seconds. The runtime reused work on 29 of the 50 steps.

Making caching practical also required fixing its interaction with H3’s AdaLN precomputation. About 13 billion parameters in each roughly 33-billion-parameter DiT checkpoint belong to timestep-dependent branches. For a fixed sampling schedule, their modulation values can be prepared ahead of time and the corresponding weights released.

Under Cache-DiT, a shared wrapper lost the layer index and gave every block the same layer’s modulation values, producing noise. We fixed the indexing so each block received its own plan.

That made precomputation usable in the cached serving path and recovered about 45 GiB per GPU in our shorter text-to-video probe. Upstream SGLang serves one checkpoint per server. Our combined-serving fork keeps both H3 checkpoints resident in one pod, sharing the encoder and decoders, at 158–161 GB per GPU on MI355X. The AdaLN fix brings that to 100–117 GB, giving the pod headroom for long and reference-heavy requests, and would make combined serving fit on 192 GB parts.

2. Reducing precision in video attention

Caching removes some repeated computation. Reduced precision makes the remaining computation cheaper.

On AMD, we focused on video attention: MXFP6 queries and keys with FP8 values, while text and audio attention remained in BF16.

At fifteen seconds, adding MXFP6 reduced the cached generation time by another 33%. At thirty seconds, the reduction was about 46%. Longer sequences gave the attention optimization more work to save.

3. Computing fewer attention interactions

Sparse attention reduces how many positions each token attends to. For H3’s long sequences, this offers another way to reduce the cost of each refinement step.

We tested a cube-sparse recipe that kept the first ten updates in BF16. Against our cached BF16 configuration, it reduced the fifteen-second text-to-video case from 59.2 to 39.0 seconds, about 34%. The thirty-second case fell from 202.0 to 107.5 seconds, about 47%.

Its speed was close to our MXFP6 dense recipe across all four workloads. That made quality an important part of choosing between them. In the small held-out evaluation, the sparse candidate recorded no failed task checks, while MXFP6 dense triggered checks involving dialogue, alignment, and identity.

4. Generating in fewer steps

Each refinement step takes another pass through the model to move the noisy video and audio closer to the finished result. Step distillation post-trains the model to reach a similar result with fewer, larger updates. The architecture and parameter count can stay the same; the model learns to accomplish more refinement in each pass.

That training is what helps preserve quality. Simply removing steps gives the original model fewer opportunities to refine the result. Distillation adjusts its weights to approximate that longer process with fewer evaluations, spending additional compute during training to reduce the work required for every subsequent generation.

NVIDIA’s Sol-H3 demonstrates this approach with four denoising forwards. Fewer passes create the opportunity for large speed gains, while the quality retained depends heavily on the post-training recipe.

The four-step distilled model running on Tenstorrent reduced generation time for a fifteen-second clip from 229.1 to 19 seconds on one Galaxy, and from 68.7 to 11 seconds on four Galaxies. Avoiding refinement passes saves both computation and the communication required between processors.

Those are approximately 12× and 6.2× improvements for the complete distilled deployment. The measurements include other implementation changes alongside distillation, so they describe the gain from the full configuration.

What the complete recipes delivered

The results below show the gains from combining these approaches, alongside published work that points to further opportunities.

05 Evaluating the quality tradeoff

While it’s important to keep pushing inference faster and cheaper, every optimization comes with tradeoffs, and we need to measure what changes. We find with LLMs that benchmark scores do not always reflect how useful a model feels in practice. That gap becomes especially apparent with video: the score can change considerably, but does the difference actually matter to someone watching?

What SSIM tells us

SSIM measures structural similarity to a reference image, comparing luminance, contrast, and structure. Applied across video frames, it quantifies differences between renders. It does not directly measure whether the action makes sense, the motion looks natural, or the scene follows the prompt.

A subject moving slightly differently, or an action happening a moment earlier, can lower frame-by-frame similarity even when both clips look convincing at normal playback. The score has detected a real difference. The question is whether the viewer notices it—and whether it makes the result less useful.

The reference matters too. A full-model render is a useful comparison point, but it can still misunderstand the prompt. Matching it more closely does not automatically make the optimized output better.

These scores compare repeated runs within each configuration. They tell us how closely a configuration reproduces its own output. The corresponding video pairs will let us see whether the difference between 0.934 and 0.586 looks consequential at normal playback, or mostly reflects variation in incidental details.

Comparing the same prompt across configurations

The practical comparison is the same prompt rendered through the full model, Cache-DiT, and a step-distilled model, with the generation time beside each result.

Watch whether the action unfolds as requested, the subject stays consistent, and motion and detail hold up throughout the clip. Then listen to the dialogue and sound. Because H3 generates video and audio jointly, a change focused on video computation can still affect the soundtrack.

Some differences will be visible without making the result less useful. Others—a missed action, a changing face, or a repeated word—can mean generating again. That is the tradeoff we care about: how much faster we can deliver a result that does what the user asked.

06 Turning the gains into a service

We’ve brought together what we learned across NVIDIA, AMD, and Tenstorrent to make video inference as fast and cheap as possible. Different hardware and optimization techniques give us different ways to serve the model; our job is to turn those choices into useful options for the people building with it.

H3 Fast uses the step-distilled model on Tenstorrent for text and first-and-last-frame generation. H3 Quality runs on AMD with the Cache-DiT recipe, and Cache-DiT + MXFP6 for requests longer than 16 seconds; it supports text, first-and-last-frame, and reference inputs. Together, they give users a choice based on the generation they need and what they want to spend.

We plan to bring these optimizations to more video models on our platform and push them further, making each generation faster and cheaper while preserving the quality people need.