Getting Qwen3.8-Next-Flash to prefill on a Radeon 8060S

The strix-halo branch rebased onto upstream master, with nothing of our own on top except the restored prop.integrated flag. Lazy per-layer-embedding loading uses the upstream --lazy-mode on path, which demand-pages rows through the mmap.

Why This is the honest baseline: everything here is already in mainline or in the UMA ring buffer stack, so every later number is attributable to one commit.

measurement pending

Caveat The depth-40000 column is empty here, and stays empty until the bf16 GEMM commit, for a concrete reason: upstream’s mm_ids_helper sizes its shared memory as n_tokens × 4 bytes, which is 98 KB for a 24576-token batch against 64 KB of LDS on gfx1151. GGML_ASSERT(nbytes_shared <= smpbo) fires while filling the context. Upstream cannot prefill at this ubatch on this part at all.

Four of the thirteen set_input_kq_mask call sites in llama-graph.cpp wrote the mask without checking it was allocated. The other nine already did.

Why A graph that stores K without attending through it never references the mask, so the allocator gives it no buffer and the write aborts. Nothing gets faster here, but nothing later works without it: this is what makes the maskless sparse-attention path reachable at all.

measurement pending

Caveat Neutral on its own, by construction. It is an upstream bug in its own right.

The draft head carries its own hyper-connection mixer. Adds the three tensors, plus --mtp-shared-embd, which leaves the token embeddings, output norm and LM head out of the draft and takes them from the target at load time.

Why A 2.8 GB draft instead of one that repeats the 27.5 GB embedding table.

measurement pending

Caveat Invisible to llama-bench, which runs no draft model. It pays off in the server path.

Two model-side changes that resist separation. The sparse path: the lightning indexer already picks the top-k key blocks per query, so this builds the attention graph that visits only those — block selection, the packed key/value layouts, compact index metadata, causally-trimmed score bounds, and the indexer cache. The reader: --lazy-mode on-direct serves the rows of a lazy tensor with explicit pread()s from a thread pool instead of demand-faulting them through the mmap.

Why The reader is worth it for two reasons, not one. Demand paging serialises 16384 scattered faults per ubatch in the fault handler; and on requires mmap, which keeps the weights resident twice during load — once as page cache, once in the managed copy an integrated GPU makes under unified memory.

measurement pending

Caveat These two started as separate commits and were merged, honestly rather than tidily: the reader’s consumer sits inside the same graph-input code the sparse path rewrites, as if (ple_reader) { … } else { … } pairs whose else-branch is the non-direct path. Splitting them mechanically produced dangling else branches. The reader’s isolated contribution was instead measured directly — a one-variable A/B on a single fixed binary, -lm mmap -lzm on against -lm none -lzm on-direct: 233.71 ± 5.14 → 395.58 ± 1.97, 1.69×. That figure is from that A/B, not from this table.

One workgroup of eight waves per (query group, kv head). Four consecutive queries share a sorted union of their selected blocks with a 16-bit membership mask, so 48 rows stay in registers as three WMMA tiles. Each wave owns 32 head dimensions; the K fragment is reused across all three tiles and the owning wave runs the online softmax.

Why Without it the graph still builds block selection and the packed key/value layouts, and then a dense kernel runs over that layout — paying both costs. That configuration measures 1.71× slower than the finished stack, so the graph and the kernel have to travel together. Anyone porting half of this will make things worse.

measurement pending

Caveat And yet the whole sparse path is break-even. Disabling it entirely — graph, indexer, top-k and kernel — measures 0.98× at depth 0, 1.01× at 40 000 and 1.00× at 140 000. The dense alternative is the head-dim-256 WMMA flash-attention kernel, which is fast enough that the lightning indexer’s cost (twelve layers × eighty strips per prefill) cancels what the sparsity saves, even at 140k context. I expected it to pay off at depth and said so; it does not. Dense is also the numerically exact computation, so at equal throughput it is the better choice — though that quality comparison is reasoned, not yet measured by perplexity, and should be before anyone retires this machinery.

Dequantizes the weights to bf16 once per graph, caches them, and runs the GEMM on the WMMA units: a tall 384×64 tile for the narrow projections, a fused gate/up + SwiGLU for the MoE, and routed down-projections that write bf16 in place.

Why At prefill batch sizes these GEMMs are compute-bound, and MMQ’s integer path leaves the f16/bf16 WMMA units idle. The f32 vector roof on this part is 29.7 TFLOPS against 59 for bf16 WMMA.

prefill 372.18 t/s at depth 0, 348.58 t/s at depth 40 000

Caveat This commit also carries the route-bounded mmid, which is what makes a 24576-token ubatch possible on gfx1151 in the first place — so it is the point where the depth-40000 column starts having numbers at all.

Fuses the combine, the residual add and the rms-norm into one kernel.

Why The combine reads four f32 residual streams and writes the normalized stream plus the next residual — five DRAM passes over the same rows on a part with ~256 GB/s of bandwidth. Fusing leaves one.

prefill 393.91 t/s at depth 0, 372.59 t/s at depth 40 000

The gate projection [320 → 10240] is read only by the stream mix that follows it, so the GEMM, the sigmoid and the mix collapse into one kernel.

Why The gate never reaches memory. This is also the commit whose env gate I initially failed to carry over, which cost a confusing afternoon — see the note on method below.

prefill 413.57 t/s at depth 0, 402.64 t/s at depth 40 000

A marking pass over the scheduled graph: a tensor whose every consumer reads the bf16 copy never needs its f32 form written at all. Covers the normalized stream, the gate, the block output, the 16-bit residual chain, the fused SwiGLU output and the routed down-projection.

Why Halves the traffic on the widest tensors in the graph. Needs the allocation dependency that stops an in-place residual sharing a buffer across a mark boundary.

prefill 417.92 t/s at depth 0, 416.13 t/s at depth 40 000

Three small fusions on the same path. The gated rms-norm folds the gate projection into the norm. The indexer’s relu and per-head sum become one kernel. The MoE weighted reduction absorbs the shared-expert merge that followed it.

Why Each is a kernel launch and a memory round trip removed. Individually small; the indexer pipeline runs 12 layers × 80 strips per prefill.

prefill 438.89 t/s at depth 0, 424.09 t/s at depth 40 000

The per-layer-embedding and delta-net short convolutions arrive as a concat plus a transposed cont chain; matching them lets one kernel write the tail directly. get_rows gains a 16-byte same-type copy and a mean-of-four fusion for the embedding gather.

Why The embedding gather runs once per token per layer, so its constant factor is not negligible.

prefill 499.52 t/s at depth 0, 486.81 t/s at depth 40 000

Tiles the recurrence over tokens so the state stays in registers across a tile instead of round-tripping to memory per token, and replaces the generic warp reduction with a DPP/permlanex16 chain: five dependent ds_bpermute round trips become five DPP ops with the same pairing order, so the sum is bit-identical.

Why This is the largest single step in the series — larger than the bf16 WMMA GEMM, larger than sparse attention, larger than everything else put together. Most layers of this model are delta-net rather than attention, and on RDNA the LDS crossbar dominated the per-token latency of the recurrence.

prefill 1181.77 t/s at depth 0, 1081.79 t/s at depth 40 000

Caveat It was very nearly missed. Earlier work measured a delta-net rewrite at roughly 3% end-to-end and dropped it over a 0.21% perplexity cost — a reasonable call, except that it compared a chunked rewrite against this tiled kernel, both already fast. Nobody had measured tiled against stock upstream at a 24576-token ubatch. Walking the series from the real upstream baseline is what exposed it, and it also explains why the hyper-connection fusions look weaker here than they do on the finished stack: every point before this one was bottlenecked on the stock recurrence, which masked everything downstream of it.

Scores the indexer’s query × key grid on WMMA with the keys held in registers, falling back to the existing kernel for shapes it does not cover.

Why Available, but left off by default.

prefill 1176.60 t/s at depth 0, 1070.57 t/s at depth 40 000

Caveat Measured neutral-to-worse in the reference stack, so the gate defaults to off. Kept because the fallback is clean and the shape coverage may widen.

qwen4exp attends with key and value length 256, where the D=256 tile kernel is FMA-bound for prefill batches. Adds the mma_tile_sizes specialization for DV=256, sizes the VKQ accumulator from the tile element type rather than a DV modulus, and adds the tile_Q reuse barrier.

Why This is the model’s own attention path, so it is load-bearing even though the sparse kernel handles the long-context case.

prefill 1182.30 t/s at depth 0, 1074.24 t/s at depth 40 000

Caveat Resolving this against upstream caught a real bug — see the note on method.

MMQ compaction picks the tile from the routed column count rather than the padded maximum. The transposed concat gets a 16-row tile on RDNA3.5.

Why Both target real inefficiencies in the trace.

prefill 1182.91 t/s at depth 0, 1071.21 t/s at depth 40 000

Caveat Measured inside the noise on this workload. Recorded as neutral rather than claimed as a win.

Three commits in the branch — ggml: widen ggml_permute’s ne/nb locals, ggml-cuda: handle equal values in the TOP_K radix gather, and the test cases. The TOP_K gather assumed distinct values, so a bucket containing ties could emit the wrong count and drop or duplicate indices.

Why Correctness, not speed — and the first thing in this series to make the model measurably slower.

prefill 1160.02 t/s at depth 0, 1072.90 t/s at depth 40 000

Caveat This step costs 1.9% (1182.91 ± 2.59 → 1160.02 ± 1.89 at depth 0, non-overlapping spreads; depth 40 000 unchanged). Bundled into one commit the regression was invisible, which is the clearest argument in this whole document for splitting commits. Two honest caveats: the three commits were separated after this measurement, so permute and topkequal have no individual figures — the before/after pair brackets all three. Attributing the cost to the TOP_K gather is inference, not measurement: test cases cannot affect runtime and ggml_permute is host-side graph construction, so it is the only candidate left. The indexer runs top-k twelve layers × eighty strips per prefill, which fits.