One buffer.Two writers.

One predicate decides both things

A tensor becomes a split input only when the split's backend cannot use the buffer it already lives in. That single test decides whether a copy is made and whether the split waits, because the waits are driven by walking split->inputs[].

llama.cpp hands the CPU slot a device host buffer type — pinned memory owned by the GPU — so that intermediate results move faster. An integrated GPU reports that it can compute on that buffer type. The two facts combine into: no copy, no entry in inputs[], no wait, and a device reading memory a host thread is still writing.

The host writes are a plain memcpy on the calling thread. They are not ordered against an in-flight ggml_backend_graph_compute_async() by anything.

It needs a second ubatch, not a second thread

The window is the gap between issuing one ubatch and writing the next one's inputs inside the same llama_decode(). Widen the batch so the prompt fits a single ubatch and the race vanishes; generate tokens one at a time and it never appears, because the sampling loop reads logits and therefore synchronises.

That is why it hides. The configuration that triggers it — a long prompt on an integrated GPU — is also the one nobody runs under a race detector.

RACE (write-after-read) on ROCm_Host[0, 2048)
  read  ROCm0#2  @3  split 0   inp_tokens (compute)
  write HOST     @5  split -1  inp_tokens (tensor_set)
  no happens-before edge: HOST knows ROCm0#2@2, needs >=3

[0, 2048) is exactly n_ubatch × 4 bytes — the whole inp_tokens tensor, overwritten at the first ubatch boundary.

A happens-before checker for the backend API

The sanitizer keeps a vector clock for the host thread and for each backend, plus a shadow map recording which actor last read or wrote every byte of every buffer. Synchronisation points — backend synchronize, event record, event wait, async copies — advance those clocks. An access that conflicts with an earlier one, where no chain of edges connects the two, is a race.

It is built into ggml-base and costs nothing when the variable is unset.

Two thirds of the races were invisible

The sanitizer hooks the backend API. But llama.cpp does not always go through it: for host-visible tensors it asserts the buffer is host memory and then writes straight through tensor->data. Those writes were never announced, so every race against them was silently missed.

ggml_backend_tensor_set_direct() announces such a write and does nothing else. It is folded into llama_host_write(), which carries the assertion those sites already had — putting the announcement inside the idiom that already marked them, so the two cannot drift apart.

87 races on one multi-ubatch prefill, 69 of them only visible after the hook was added. The largest single contributor was the attention mask, which no earlier instrumentation could see at all.

The reuse path allocates nothing, so it cannot rotate on its own

Rotation happens in ggml_backend_sched_alloc_graph(). But llama.cpp's fast path checks whether the previous graph can be reused and, when it can, skips splitting and allocation entirely — which is exactly the path taken during steady-state decoding.

ggml_backend_sched_prepare_inputs() is the rotation without the allocation. A caller writing into a reused graph must call it first. After alloc_graph() it is a no-op, because allocating already rotated.

This obligation is easy to get wrong in a way that looks fine: forget it and the inputs simply stay at fixed addresses, which is correct — right up until two computes are in flight.

A cached graph must not trust that addresses stood still

Backends that cache work against a graph — the CUDA graph cache is the one in tree — may treat an unchanged ggml_cgraph::uid as a promise that nothing the cached work captured has moved, and skip re-reading the addresses.

Rotation moves tensor addresses without re-splitting. So the scheduler re-stamps the split uids whenever it re-points inputs. A backend keeping such a cache must key it on the uid or re-check addresses itself.

Generation never pays for it

The scheduler tracks whether a compute has been issued that might still be reading. Anything that waits for the backends clears that state. A sampling loop reads logits every token, which synchronises, so its inputs stay at fixed addresses and the ring costs nothing but the memory.

Measured on one multi-ubatch prefill followed by generation: 12 rotations during prefill, zero during decode. The ring engages precisely where the hazard is.

The memory is not free, though — one extra copy of every graph input per extra slot, and attention masks scale with context and batch. That is the argument for a default depth of 2 rather than the maximum of 4.

The alias must not move

A crash inside the allocator, dereferencing buffers[-1], on a test that has nothing to do with integrated GPUs. The ring did not corrupt anything; it violated an undocumented invariant — that the identity of the tensors in a graph does not change between two allocations the allocator considers equivalent.

Pinning the alias to slot 0 keeps identity fixed while the other slots still rotate, which is all the ring actually needs.

The ring cannot help here, and neither can copies

This one appeared with no layers offloaded at all, and survived the ring completely — the two tensors are not graph inputs, so rotating inputs changes nothing. It is the same root cause wearing different clothes: a backend reads memory in place, and nothing orders a later writer against it.

The fix keeps memory that another backend may still be reading out of the allocator's reuse pool, keyed on the view root — the tensor that actually owns the bytes and the one the allocator frees. Keying it on the accessed tensor instead misses every access that goes through a view, which is most of them; that mistake cost a full measurement cycle before the race counts refused to move.

Found while chasing the ring, unrelated to the ring

Pass 4 already said "views are always on the same backend as the source", but only applied it to nodes still unassigned — passes 2 and 3 can move an aliasing node before it gets there. Enforcing the invariant for nodes that alias a source and are not pure view ops moves 13 nodes on a hybrid recurrent model at partial offload, and removes the copies that fed them.

This one reproduces on unmodified master and is the only bug here that is not about integrated GPUs at all. CUDA escapes it because it reports integrated = false, which changes placement enough to avoid the trigger.

Test the condition, not the device

The scheduler detects the hazard by testing the thing that causes it — a host buffer type in the last slot that some other backend accepts for compute — rather than by naming devices. A device that refuses to compute on pinned host memory is unaffected without anyone maintaining a list.

That was still not sufficient. An early version excluded backends by device type, which missed BLAS: an accelerator device that computes on host memory and therefore matched. The ring engaged inside ggml_opt, which does not meet the caller obligation, and 45 tests started failing on a Windows CI runner.

The real condition is asynchrony. A backend with no synchronize has finished its work by the time graph_compute returns, so nothing of its can still be reading. Requiring caps.async covers CPU, BLAS, and everything else synchronous in one test instead of a growing list of exclusions.

// before — a device-type exclusion, and BLAS is not a CPU device
if (props.type == GGML_BACKEND_DEVICE_TYPE_CPU) { continue; }

// after — the property that actually matters
if (!props.caps.async) { continue; }

Every backend in tree, and why it does or does not qualify

Two conditions have to hold together: the backend must be asynchronous, and it must accept the host buffer type the CPU slot was given. Checking every backend's supports_buft rather than reasoning from device class gives an exact answer.

CUDA hard-codes integrated = false, so in practice the ring engages on HIP integrated GPUs and nowhere else. And only llama_context hands the CPU slot a device host buffer type — clip.cpp and ggml_opt both pass a plain CPU buffer type, which no asynchronous backend accepts.

Environment

Note that GGML_SCHED_UMA_RING can only ever reduce what was detected. There is no way to switch the ring on where the scheduler did not find the hazard, because the caller obligations would not be met.

Two ways to measure nothing at all

Both of these produced clean, confident, meaningless numbers before being caught:

• The reuse path needs a small ubatch. At -ub 512 the graph is rebuilt rather than reused, so prepare_inputs() never runs. Race counts went to zero while the code under test had never executed. -ub 32 exercises it.
• Scheduler debug output is filtered by the tool's verbosity. GGML_SCHED_DEBUG=1 without -v prints nothing, which reads identically to "the condition never fired". The aliasing rule appeared to move zero nodes for an entire investigation.

A third: the aliasing rule only fires on a graph shape that needs warmup and a real multi-ubatch prompt. With --no-warmup -n 1 it moves nothing, on a model where it demonstrably moves 13 nodes.

What it bought

Throughput is at parity with racy master and with the alternative fix proposed upstream. The ring buys coverage, not speed: it removes a class of wrong answers that were previously invisible, and it does so without adding a stall to the path that matters.