Three layers of a local inference stack, and what is happening in each

The problem

A HIP graph is meant to amortise launch cost, but the runtime still re-encodes the PM4 packets for every node on every launch. On a workload made of many small dispatches — which is what a quantized MoE forward pass looks like — that host-side encode is a measurable share of the wall clock, and on an APU there is no spare CPU to hide it behind.

The change

Let a graph materialise its packets once and replay the retained command list. ROCr gains a vendor API — hsa_ven_amd_graph_command_list_create, …_materialize_packet, …_get_capabilities and friends — and HIP grows retained command lists on top of it, with batch merging for collapsed packet runs.

Where it stands

Five commits on a branch of rocm-systems, rebased on current develop: the command lists themselves, packet-batch merging, a fix for quadratic growth in the update path, gfx1201 register programming, and a build fix for header selection. There is a contribution document and a rocrtst case alongside the implementation.

Status Experimental and enabled by an environment variable. It is not a proposal to AMD yet; the register programming is only validated on the two targets to hand.

Honest caveat

For the Qwen3.8-Next-Flash prefill work below, HIP graphs never actually engage: graph capture needs two consecutive graphs with unchanged properties, and each prefill chunk shape occurs once per request. The retained-PM4 path pays off in decode, where the same graph repeats. It is included here because it is the layer that generalises, not because it carries the prefill numbers.

Mainline — slow and stable

Upstream takes RDNA-specific work, but carefully and with broad-hardware review. Merged recently:

• RDNA3 MMA flash attention, a faster AMD transpose and tuning (#22880)
• RDNA3.5 and RDNA3 split into independently tunable MMQ configs (#26199), then tuned (#26284)
• Routed MoE MMQ N-tiles sized from typical expert width on RDNA3 (#28552)
• Flash-attention tuning for gfx1201 (#28102)
• A ROCm CI runner with gfx1151 fixes (#26544)
• iGPU-aware loading: --load-mode auto avoiding mmap (#26081), lazy tensor loading off by default on iGPUs (#28326), virtual iGPU devices (#26953)

Vulkan is moving at least as fast on this hardware — mat-vec row tuning for Strix Halo (#27909), a dedicated IQ4_XS mat-vec shader (#28426), tiled transpose for permuted CONT (#26585), and fused hyper-connection ops for DeepSeek-V4 (#26578).

The community fork — fast and targeted

halo-box/strix-llama.cpp exists to get Strix Halo changes usable before they are upstreamable, and it is currently doing most of its work on the Vulkan backend:

• Quantized dense coopmat pipelines run at wave32
• An optional f16 B operand for quantized MUL_MAT on coopmat1, with an auto mode
• mul_mat_id tiles derived from the wave64 dense tiles; the Strix mmid tile gates on by default
• Command buffers bounded by memory traffic rather than flops alone
• Four env-gated prefill fixes for delta-net MoE, and the transposed-concat path defaulted on
• A coopmat1 FA wave32 pin narrowed to multi-row dispatches

Several of these are the kind of thing that wants a wider hardware matrix before upstream will take it, which is exactly what a fork is for.

My branches — trying to reconcile the two

The work on pwilkin/llama.cpp aims at the middle: keep the fast path usable locally, and factor the pieces that can stand on their own into shapes mainline can review. Currently in flight:

• sched-uma-ring — a graph-input ring buffer in the backend scheduler, which fixes the aliasing race that made upstream disable prop.integrated on HIP rather than work around it
• topk-rocm-fix — wave32-native TOP_K kernels
• cuda-getrows-vec, cuda-q8-norm-fusion, hip-fattn-tile-hs256-tune, fix-igpu-input-layer — small, self-contained pieces aimed at upstream
• strix-halo — the integration branch these pages build from
• strix-halo-for-halobox — the graph-agnostic half of those kernels rebased onto the community fork, so the pieces that do not depend on this branch's model graph can live where the rest of the Strix work is

Position Where mainline already has a good answer, the branch takes mainline's. The head-size-256 attention tuning in the series below is a case where upstream's broader gate turned out to be the correct half and ours was the wrong one.

Why a fork at all

Some of what makes this machine fast is genuinely narrow: kernels that assume 48 value heads, a 2560-wide residual, or a particular expert count. Those are not upstreamable as written, and pretending otherwise wastes reviewers' time. Keeping them on a branch, behind gates, with the measurements written down, seems more useful than either hiding them or filing them.

Qwen3.8-27B

Calibrated IQ4_XS weights with a Q8 output layer and an IQ4_XS DFlash2 drafter. This is the configuration the installer below builds, and the one that has been stable longest.

Two-pass range 26.23–26.28 t/s across six generations, GPU 37–54 °C. Measured with the retained-PM4 runtime and a fixed-width DFlash2 policy.

Qwen3.8-Next-Flash

177 B parameters, 93 GiB of IQ4_NL, a gated delta-net recurrence in most layers, a lightning indexer selecting 2048 key blocks per query, and a 27.5 GB per-layer-embedding table that cannot be resident. It is the harder problem by some distance, and prefill is where the work went.

Measured with a 16384-token batch and ubatch, which is what the installed launcher defaults to, so the whole prompt goes through as one batch. A 24576 ubatch measures the same within error (1199.63 ± 7.12 against 1204.31 ± 2.31 in the same session); 16384 is the default because it reserves smaller compute buffers, which on a unified memory part is 8 GB less competing with the weights. Prefill holds 90% of its depth-0 rate at 40 000 tokens of context, which is the sparse attention path doing its job. Decode is the weaker half — the journey's paired run put it about 8% short of what the same kernels reach out of tree — and that gap is open and written up rather than smoothed over.

Read the optimization journey →