100 tokens per second from a 27B model on a 24 GB AMD card

A dense 27B model quantized to 4 bits reads about 17 GB of weights for every single token it generates. On an RX 7900 XTX with roughly 960 GB/s of memory bandwidth, that caps single-stream decoding at about 56 tokens per second. I measured 37.6 tok/s at baseline and finished at 98.1 tok/s greedy, 113 to 126 tok/s wall clock on the sampled path my coding agent actually runs. Speculative decoding is the only thing that gets past a per-token bandwidth wall, because it stops paying the wall per token.

My coding agent runs on this card now. Frontier models triage and plan, and the coding is free apart from electricity. On my own workloads the local 27B holds its own against GLM-5.3 for day to day coding. That is a hot take rather than a benchmark, and it is downstream of the thing this post is about: making the model fast enough to be a teammate instead of a demo.

The wall

Every token a dense LLM generates costs one full read of the model’s weights.

That sentence is the whole physics of local inference and it is worth sitting with before you touch a single flag. A dense model uses all of its parameters for every token. Mine is Qwen3.8-27B at 4-bit UD-Q4_K_XL, which is about 17 GB of weights. Single-stream decode is a GEMV workload. For each token, every weight matrix gets streamed out of VRAM once, multiplied against one activation vector, and dropped from cache. Arithmetic intensity is around 2 FLOPs per byte, so the compute units sit idle while the memory controller does all the work. Decode is bandwidth bound, full stop.

The card is an RX 7900 XTX: 24 GB of VRAM, roughly 960 GB/s, gfx1100, driven over Vulkan through Mesa RADV. No CUDA and no tensor cores.

960 GB/s / 17 GB per token = about 56 tokens/sec

That is the autoregressive ceiling. No inference engine, no hand tuned kernel and no Rust rewrite gets past it, because all of them still read the same 17 GB per token. Engines differ in how close they get to the ceiling, not in whether the ceiling exists.

Measured baseline on an honest configuration, meaning 32K context, KV cache at q8_0/q4_0 and no speculation: 37.6 tok/s. That sits below theory because real decode also streams the KV cache and pays for kernel launches and sampling.

So 100 tok/s on this card looked like a joke. I set it as one. Then I spent a weekend on it.

How speculative decoding gets past a per-token wall

The only way past a per-token bandwidth wall is to stop paying it per token.

Start with the intuition. The bus caps you near 56 tokens per second, but the compute is nearly idle while that streaming happens. At 2 FLOPs per byte of weights, a card rated in the tens of TFLOPs is loafing. There is a lot of headroom to think internally and very little to spare on trips to memory. Speculative decoding spends the free compute to buy back the scarce bandwidth. Let a cheap proxy run ahead, then pay for many tokens with one trip.

The asymmetry that makes it work: verifying K tokens costs almost the same as generating one. Hand the target model K drafted tokens and it scores all of them in a single forward pass, because the weight matrices get read once and applied to a K wide batch of activations instead of a single vector. Decode is bandwidth bound and the weights dominate the traffic, so that batch dimension rides along nearly free. Verification turns decode into a tiny prefill.

The mechanism, then. A cheap drafter proposes tokens x1 through xK. The target scores them in one pass. A token by token acceptance rule keeps a prefix of them, and the target’s own next token caps the window. In the worst case, where everything gets rejected, you still emit one token per weight read, so you never go slower than plain decode minus the draft overhead.

The acceptance rule is what makes this lossless, and it is worth being precise because “lossless” gets hand waved a lot. Under greedy decoding it is trivial: accept xi while it equals the target’s argmax. Under sampling it is the rejection scheme from the original papers, Leviathan et al. (ICML 2023) and Chen et al. (2023). Accept xi with probability min(1, p_target(xi) / p_draft(xi)), and on rejection resample from the normalized positive part of (p_target minus p_draft). The emitted sequence is distributed exactly as if the target had sampled on its own. Provably, not approximately. The drafter only ever affects speed.

Two numbers govern the entire design space. Tau is the expected number of tokens emitted per verify window, so tau of 4.7 means one 17 GB read buys about 4.7 tokens. Draft cost is the FLOPs, VRAM and verify width overhead the drafter adds. Throughput is roughly tau divided by (verify time plus draft time). Every lever below attacks that fraction, and every failure is one of its terms biting back.

Lever 1: MTP, where the model drafts for itself

Multi-token prediction adds small auxiliary heads to the trunk, trained jointly with the base model. DeepSeek-V3 popularized the recipe and Qwen3.8 ships heads in the release. The trunk produces its hidden state for position t as usual, and head k maps that same hidden state to a prediction for position t+k. During training these act as an auxiliary loss that densifies the learning signal, and as a side effect the checkpoint ships with a free drafter inside it.

At inference the heads become self-drafts. One trunk pass gives you the next token plus candidate futures, with no second model and no extra weight traffic beyond the tiny heads. In llama.cpp:

--spec-type draft-mtp --spec-draft-n-max 2

61 tok/s. That is 1.6x from one flag.

It saturates fast, and the reason is structural. Head k predicts t+k from the hidden state at t, so it never sees its own intermediate guesses and there is no recursive conditioning. Accuracy decays steeply with depth: acceptance at position 1 is around 0.72, position 2 around 0.47, and roughly zero past that, which matches the published RDNA4 numbers. Past the useful depth, extra draft positions are pure verify width cost on a bus that is already saturated.

Lever 2: DFlash2, a trained drafter

llama.cpp PR #27342. Instead of heads bolted onto the trunk, this is a standalone sidecar of about 2 GB at Q8_0. For this target it is a 5 layer model at the target’s hidden width of 5120, sharing the target’s embedding and lm_head, distilled against the target to emit blocks of continuations. Block size here is 8. It reads the same running context and proposes a whole window per step.

The structural win over MTP is that it is a real model with its own layers and its own KV, so prediction quality holds deeper into the window instead of extrapolating from one frozen hidden state. Distilling against the target rather than pretraining generically is what buys high agreement, and agreement is tau. The cost is about 2 GB of VRAM plus a drafter forward pass per window, and both are cheap next to a 17 GB verify read.

With n-max 5: 79 tok/s, twice the baseline.

The f16 rule, and why it is mechanical

I quantized the drafter’s KV cache to q8_0 to claw back some VRAM. Acceptance collapsed. The issue is filed as ggml-org#25725.

Here is why. Tau is a product of per-position agreements between two nearly identical distributions. Quantization noise in the drafter’s KV perturbs its logits slightly at every position, and a window survives only while every position agrees, so a small per-position degradation compounds geometrically across the window. The tail of long accepts, which is where all the profit lives, dies first.

Lever 3: an n-gram overlay for repetition

Prompt lookup decoding, generalized. You keep an incremental hash map from k-token keys to whatever continuation followed them last time, built over the prompt and over everything generated so far. When the most recent k tokens hit the map, the stored continuation gets proposed as draft tokens. Zero FLOPs, zero VRAM traffic, just a hash probe. Verification is the same target pass either way, so a hit is nearly free tau.

Code is the ideal workload for this. Identifiers, imports, call signatures and boilerplate recur constantly, and any edit style task has the model re-emitting spans of its own input. Stacked on the trained drafter:

--spec-type draft-dflash,ngram-map-k4v

Key length defaults to 12, which means twelve exact tokens have to match before the map fires, so in practice it almost never does. I swept it and 6 won: frequent hits, still specific enough that the proposals survive verification. That gave about 86 tok/s median on the sampled path my agent actually runs at temperature 0.6, with peaks in the low 90s.

That is llama.cpp’s ceiling on this card for this model. Polling modes, thread counts, batch shapes, context lengths, stacked combinations, none of it beat 86. Still short of 100.

The 253 tok/s detour, or: read the numerator

Halfway through the chase someone sent me hipfire, an RDNA native Rust engine claiming 253 tok/s on my exact GPU.

Then I read the model card next to the number: Qwen3.6-35B-A3B. That is a mixture of experts with about 3B active parameters per token. An MoE routes each token through a router-selected handful of expert FFNs, so 35B parameters are resident but only about 2 GB get read per token. Same physics, different numerator. 960 divided by 2 is about 480 theoretical, which makes 253 an unremarkable and entirely real number for that model.

I verified it: 216 tok/s on my box, with hipfire’s Redline path engaged. Redline is retained-replay dispatch, recording the kernel graph once and replaying validated PM4 command buffers through the ROCr queues to shave per-launch driver overhead. It was worth about 1% here, which makes sense, because decode is bus bound rather than launch bound.

And my dense 27B on that same engine, with its generic n-gram speculation? 40 tok/s median. Slower than tuned llama.cpp. The bus does not care what language your engine is written in.

The last mile: a sidecar that did not exist a week earlier

While I was poking around hipfire’s registry, it had grown a DFlash draft sidecar for my dense model. 1.2 GB, same shared-embedding block-draft design.

Getting it to load was its own small saga. The shipped binary predates the sidecar’s tensor format and panics on it. The upstream nix flake was broken at HEAD in two separate ways: crates.io’s API had started returning 403 to the fetcher, so I seeded all 362 crate tarballs from static.crates.io into the nix store by hand, and the daemon binary had moved crates without the packaging following. One small patch to nix/package.nix later, a main-HEAD build loaded it.

Dense Qwen3.8-27B plus the DFlash sidecar on hipfire, benched through the same GPU broker as everything else:

Look at the draft-quant column, because it is a hill rather than a slope. The heavier mq6 draft lowered acceptance. The lighter mq3 collapsed it. Draft quality against draft cost has an interior optimum, and you find it by sweeping, not by reasoning.

The number that actually matters is the sampled path the agent runs, at temperature 0.6 and top-p 0.95, served over the OpenAI compatible API:

• Python coding: 113 to 126 tok/s wall clock, meaning completion tokens divided by total request time with prefill included. Decode-only peaks hit 150.7, tau 6 to 8.
• TypeScript: tau around 2.5, so about 64 tok/s. The drafter was clearly fed more Python than TypeScript. Your language mix is a variable, not a footnote.

Sampling cost nothing against greedy here, with no acceptance cliff. Same 27B, same 24 GB card, over the line.

What failed

Four things, all of which cost me hours.

HIP llama.cpp was slower than Vulkan on this dense model, 50 against 76 at identical settings. Measure both backends and assume nothing.

Two servers on one GPU is poison. The second model spilled weights into GTT, which is system RAM masquerading as VRAM over PCIe, and dragged the live service down to about 4.5 tok/s. One GPU, one resident model, one broker enforcing it.

poll=0 and fewer verifier threads looked free on greedy. On sampled decode, acceptance cratered to 0.40 from 0.60. Under top-p and top-k entropy the verify path needs CPU headroom, so 8 threads and poll 50 stayed.

64K context plus the drafter went OOM. I shipped 32K.

And the fifth, covered above: drafter KV must be f16. The target’s KV can stay at q8_0/q4_0.

Why a hiring company is writing this

Curriculo scores resumes with an LLM, and screening is a batch workload. A role closes and there are 400 resumes to evaluate at once, not one token stream that one person is waiting to read.

That sits on the opposite side of the same physics. In a batch, the 17 GB weight read is amortized across every candidate in the batch, so the bus stops being the constraint and the compute units finally have something to do. Speculative decoding is what you reach for when you cannot batch. Batching is what you reach for when you can. Both are the same move underneath: work out what one token actually costs, find the term that dominates, then attack that term instead of the one you assumed.

The reason I spent a weekend on the single-stream case is that a coding agent is single-stream by nature. You cannot batch a conversation with yourself.

What transfers to your setup

Start from the bandwidth math. Weights read per token divided by memory bandwidth gives you the autoregressive ceiling. Everything above that number is speculation and there is nowhere else for it to come from.

Treat speculation as a tau against cost trade. MTP heads if your checkpoint ships them, which is the cheapest and shallowest option. A distilled block drafter if one exists, which gives the deepest tau. An n-gram overlay for repetitive workloads, where the hits are free. They stack.

Tune on your backend, your quant and your workload. CUDA receipts do not transfer to RDNA3, MoE receipts do not transfer to dense, and Python receipts do not transfer to TypeScript. That covers n-max, key length, draft quant, threads and poll.

Benchmark the sampled path. Your agent does not run greedy, and half the optimizations that win on greedy lose under entropy.

37.6, then 86, then past 100, against a hard 56 ceiling. The joke target is the daily driver now.

FAQ

What limits local LLM speed on a single GPU?

Memory bandwidth, not compute. A dense model reads every one of its weights for each token it generates, so tokens per second is capped at roughly (memory bandwidth) divided by (quantized model size). A 17 GB model on a 960 GB/s card cannot exceed about 56 tok/s in single-stream decode regardless of which inference engine you run.

Does speculative decoding change the model’s output?

No. Under greedy decoding a drafted token is accepted only when it matches the target’s argmax. Under sampling, the rejection scheme from Leviathan et al. and Chen et al. accepts a token with probability min(1, p_target/p_draft) and resamples from the normalized difference on rejection. The output distribution is provably identical to the target sampling alone. The drafter only affects speed.

Why can an MoE model report much higher tokens per second than a dense one?

Because a mixture of experts routes each token through a small subset of its parameters. A 35B MoE with 3B active parameters reads about 2 GB per token instead of 17 GB, so its bandwidth ceiling is roughly eight times higher. Comparing a 253 tok/s MoE number against a dense model is comparing different numerators, not different engines.

Should the draft model’s KV cache be quantized?

No. Keep drafter KV at f16. Tau is a product of per-position agreements, so quantization noise in the drafter’s logits compounds geometrically across the draft window and kills the long accepts where the throughput gain lives. The target model’s KV cache can stay at q8_0 or q4_0 safely.

Do CUDA speculative decoding settings work on AMD?

Not directly. Vulkan carries higher per-token verify overhead than CUDA, so the profitable draft window is narrower. On this setup --spec-draft-n-max 3 measured slower than 2 (56 against 61 tok/s), while CUDA writeups commonly recommend 4 to 7. Sweep every parameter on your own backend.

What is a realistic tokens per second target for a 27B model on a 24 GB card?

Around 37 tok/s with no speculation, around 60 with the model’s own MTP heads, around 86 with a distilled block drafter plus an n-gram overlay in llama.cpp, and past 100 with a matched draft sidecar on an RDNA native engine. On a sampled coding workload the wall clock figure reached 113 to 126 tok/s for Python and about 64 tok/s for TypeScript.

Reproducible: bench scripts, sweep results and the one-file flake patch live in my nix-config under loops/2026-08-24-hermes-qwen-100tps/.