One graph.Many backends.

reg → device → backend, buffer type → buffer

A reg is a loaded library (CUDA, Vulkan, Metal, RPC, CPU…). It enumerates devices. A device answers the static questions — can you run this op? do you understand this buffer type? how much memory do you have? — without any execution state. Initialising a device gives you a backend, which is really a command stream: it owns queues, events and the ability to actually compute a graph.

Memory is a parallel pair. A buffer type (buft) describes a kind of memory — alignment, max allocation, whether it is host-visible — and belongs to a device. A buffer is an allocation of that type. Every ggml_tensor that has been allocated carries a pointer to the buffer it lives in, and that pointer is the single most important input to the scheduler: the location of the data is the location of the work.

The array is the policy

There is no cost model, no heuristic ranking, no benchmark. ggml_backend_sched_new takes an array of backends and the index in that array is the entire priority scheme:

ggml_backend_sched_t ggml_backend_sched_new(
        ggml_backend_t *             backends,
        ggml_backend_buffer_type_t * bufts,
        int    n_backends,
        size_t graph_size,
        bool   parallel,     // enable pipeline parallelism (n_copies = 4)
        bool   op_offload);  // allow pulling host-weight ops onto a GPU

GGML_ASSERT(ggml_backend_dev_type(
    ggml_backend_get_device(backends[n_backends - 1])) == GGML_BACKEND_DEVICE_TYPE_CPU);

The last entry must be a CPU device. That assert is load-bearing: the CPU is the universal fallback, the assumed home of graph inputs, and the backend whose id every "skip the lowest priority backend" branch compares against. llama.cpp builds the array as GPU devices in user order → ACCEL devices (BLAS, AMX) → CPU, in llama_context's constructor.

Three compile-time ceilings shape the rest: GGML_SCHED_MAX_BACKENDS 16, GGML_SCHED_MAX_COPIES 4, and GGML_SCHED_MAX_SPLIT_INPUTS 30. The last one is only an initial capacity now — overflow doubles the array and logs a warning rather than aborting — but it still matters, because a full input list is one of the two conditions that forces a new split.

Three stages, one pipeline

Every path through the scheduler is the same three stages, exposed at three granularities so callers can stop early:

1. Split — ggml_backend_sched_split_graph() assigns a backend to every node and cuts the graph into splits[]. Pure bookkeeping; touches no device memory.
2. Allocate — ggml_backend_sched_alloc_splits() hands the rewritten graph to ggml_gallocr, which assigns every tensor an offset in a per-backend arena.
3. Compute — ggml_backend_sched_compute_splits() walks the split list, copies each split's foreign inputs in, and issues the subgraph asynchronously.

ggml_backend_sched_reserve() runs stages 1 and 2 against a worst-case graph so that the compute buffers are sized once, up front, at the largest batch you will ever submit. llama.cpp calls it twice — once with an n_ubatch-wide prompt-processing graph, once with a one-token generation graph — and reports the split and node counts of both. That is where the familiar startup line comes from:

llama_context: graph nodes  = 1478
llama_context: graph splits = 2

A high split count at generation time is the single best cheap signal that something is misplaced.

Only the forced nodes

Pass 1 asks each leaf and each node one question — is your placement already decided? — and leaves everything else at -1. Five ways to be decided, checked in this order:

• 1.dst — the tensor already has a buffer. Take the highest-priority backend that both understands that buffer type and supports the op. This is how KV cache writes get pinned to the GPU that owns the cache.
• 1.vsrc — same, but via view_src. A view lives wherever its source lives.
• Pre-allocated but no backend can run the op there → GGML_ABORT. The tensor cannot be moved, so there is nothing to negotiate.
• 1.inp — the tensor carries GGML_TENSOR_FLAG_INPUT. It goes to the last backend, i.e. the CPU, because that is where the caller will write it.
• 1.wgtN — some source lives in a buffer marked GGML_BACKEND_BUFFER_USAGE_WEIGHTS. Run the op where the weight is. This is the rule that decides almost everything in a transformer.

Two ops are deliberately excluded from the weight rule:

// skip ROPE since the rope freqs tensor is too small to choose a backend based on it
allow = allow && tensor->op != GGML_OP_ROPE;
// skip FLASH_ATTN_EXT since the sinks tensor is too small to choose a based based on it
allow = allow && tensor->op != GGML_OP_FLASH_ATTN_EXT;

Both ops take a tiny model tensor (rope frequencies, attention sinks) alongside their real, large operands. Honouring the weight rule would drag the whole attention block to wherever that few-kilobyte tensor happens to live. So they abstain and let pass 2 decide from their neighbours instead.

1.off — the one place the scheduler moves work away from its data

If the weight the op needs lives on the CPU in a host buffer, and op_offload is enabled, the scheduler scans every higher-priority backend and asks two questions: can you run this op, and do you want to?

if (sched->op_offload && src_backend_id == sched->n_backends - 1 &&
    ggml_backend_buffer_is_host(src->buffer)) {
    for (int b = 0; b < src_backend_id; b++) {
        if (ggml_backend_supports_op(sched->backends[b], tensor) &&
            ggml_backend_offload_op(sched->backends[b], tensor)) {
            SET_CAUSE(tensor, "1.off");
            return b;
        }
    }
}

That second question is the interesting one. offload_op is a per-device veto, and CUDA implements it purely as a batch-size threshold: get_op_batch_size(op) >= 32 by default, tunable with GGML_OP_OFFLOAD_MIN_BATCH. Copying a 500 MB weight matrix over PCIe to multiply it by one token vector is a catastrophic trade; doing it to multiply by a 512-token batch is a large win. So the same model, the same flags, and the same graph shape will place the FFN matmuls differently during prompt processing than during generation — and that is intended, not a bug.

--no-op-offload turns the whole mechanism off.

Four sweeps, and why the CPU never spreads

Pass 2 does the actual work of filling the graph in. It runs four linear sweeps: forward then backward propagating only GPU assignments, then forward then backward propagating anything. Each sweep carries a cur_backend_id along and stamps it onto unassigned nodes.

The asymmetry is the whole design. In the first two sweeps:

if (*node_backend_id != -1) {
    if (*node_backend_id == sched->n_backends - 1) {
        cur_backend_id = -1;   // skip cpu (lowest prio backend)
    } else {
        cur_backend_id = *node_backend_id;
    }
} else if (cur_backend_id != -1) {
    ggml_backend_sched_set_if_supported(sched, node, cur_backend_id, node_backend_id);
}

Hitting a CPU node clears the carried id instead of adopting it. The comment in the source states the resulting invariant exactly: "thus, cpu will never be used unless weights are on cpu, or there are no gpu ops between cpu ops". Unassigned regions are claimed by whichever GPU can reach them first, from either direction, and the CPU only ends up owning a node if no GPU sweep ever reached it. The two "rest" sweeps afterwards mop up whatever is left, and only then can the CPU spread.

Two guards run throughout: view ops (VIEW, RESHAPE, PERMUTE, TRANSPOSE) are skipped entirely — they are metadata, they get their placement in pass 4 — and a node is only stamped if the carried backend actually supports_op it. A node the backend refuses is left unassigned on purpose, so that pass 3 can decide it later with better information.

Upgrade what you can, guess the rest

Pass 3 has two unrelated jobs sharing a loop.

Assigned nodes get upgraded. If a higher-priority backend uses the identical buffer type and supports the op and all sources, move the node there. The typical case is BLAS and CPU both using plain host memory: work seeded onto the CPU migrates up to the accelerator for free, because "free" is literally true — same buffer type means no copy. The source is candid that the test is stricter than necessary:

// (*) the actual requirement is more relaxed, the buffer type of the backend should be
//     supported by all the users of this tensor further down the graph
//     however, this is slow to verify, so we have a more strict requirement
//     that the buffer type is the same

Unassigned nodes get a guess. These are exactly the nodes pass 2 refused to stamp because some backend did not support the op. For each backend that does support it, count how many of the node's sources are already in a buffer type that backend can read, and take the winner. It is a one-node-deep greedy heuristic to minimise copies, tie-broken by priority order since > keeps the first best.

Sources inherit from their consumer

Nodes have been placed; many of their sources — constants, views, small leaves nobody seeded — still have not. Pass 4 closes that: a view takes its view_src's backend (4.vsrc, and the comment is blunt — "views are always on the same backend as the source"), everything else takes the backend of the node consuming it (4.cur). Any node still unassigned falls to the first backend that supports it, then:

GGML_ASSERT(*cur_backend_id != -1);

and a few lines later in pass 5, the same invariant with a diagnosis attached:

GGML_ASSERT(node_backend_id != -1);
// all nodes should be assigned by now, this can happen if there is no CPU fallback

If you have ever hit that assert while building a custom backend list, that comment is the answer: something removed the universal fallback and an op has nowhere to run.

Pinning a node by hand, and the one place llama.cpp does it

Passes 1 and 2 both guard their writes with "do not overwrite user assignments". A backend set by ggml_backend_sched_set_tensor_backend() before the split survives all five passes untouched. The header calls this optional and says it "should not be needed in most cases" — and llama.cpp uses it exactly once, at the end of build_attn:

if (!cparams.offload_kqv) {
    // all nodes between the KV store and the attention output are run on the CPU
    ggml_backend_sched_set_tensor_backend(sched, cur, backend_cpu);
}

That single line is a direct consequence of the expansion asymmetry. With the KV cache in host memory, the cache writes are pinned to the CPU by 1.dst — but the attention output is a plain matmul against a GPU weight, so the upward GPU sweep would happily claim every attention node in between, and then every token would copy the entire KV cache across PCIe to feed it. Pinning the far end of the region stops the sweep at the right place. Turn on --no-kv-offload in the explorer above and you can watch that pin do its work.

The backend changed

The obvious one. View ops are skipped when scanning, so a RESHAPE between two GPU nodes never fragments a split.

The same backend, but a new weight to stage

The second rule is not about correctness at all — it is about peak memory. When a node's weight lives on an incompatible backend, that weight must be copied into a staging tensor in this split's arena. If a single split accumulates twenty such weights, all twenty staging buffers are live simultaneously.

if (src->buffer != NULL && src->buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
    int src_backend_id = tensor_backend_id(src);
    if (src_backend_id != cur_backend_id &&
        !ggml_backend_sched_buffer_supported(sched, src, cur_backend_id)) {
        // by starting a new split, the memory of the previously offloaded weights can be reused
        need_new_split = true;
        break;
    }
}

Cutting the split ends the lifetime of the previous staging tensors, so ggml-alloc can hand the same bytes to the next one. This is why a partially-offloaded model reports a split count in the hundreds and still runs in a bounded compute buffer: each offloaded layer is its own split, reusing one weight-sized scratch region.

The second condition in the same block is the input-list backstop: if the split's input array is already at capacity and this node would add another foreign source, cut rather than grow. The source flags it as imprecise (FIXME: count the number of inputs instead of only checking when full).

What "an input" means to a split

For every source of every node in a split, if the source lives on a different backend and this split's backend cannot read that buffer type, the scheduler manufactures a copy: a tensor with identical layout, allocated in the consuming backend's arena, named <Backend>#<original>#<copy_idx>. Then — and this is the part that surprises people reading the code for the first time — it edits the caller's graph in place:

node->src[j] = tensor_id_copy(src_id, cur_backend_id, sched->cur_copy);

The original tensor is remembered in split->inputs[] so the runtime knows what to copy from, but the graph the caller built now points at the local replica. This is why ggml_backend_sched_reset() exists and why its documentation warns that afterwards you must discard your tensors and rebuild the graph.

Note the guard: the copy is only made when the buffer type is unsupported, not merely different. Two backends sharing host memory — CPU and BLAS, say — pass tensors between splits with no copy at all.

input_dep and input_cpy

After the split list is final, the scheduler builds sched->graph, a flattened copy of every split in execution order. Before each split's own nodes it inserts two synthetic nodes per input:

            // add a dependency to the input source so that it is not freed before the copy is done
struct ggml_tensor * input_dep = ggml_view_tensor(sched->ctx, input);
input_dep->src[0] = input;
sched->node_backend_ids[n++] = hv_tensor_backend_ids[input_id];   // source backend

            // add a dependency to the input copy so that it is allocated at the start of the split
sched->node_backend_ids[n++] = split->backend_id;                  // dest backend

Neither node is ever computed. They exist purely so that ggml-alloc, which frees a tensor as soon as its last consumer has run, sees a consumer at the right point in the timeline: the source stays alive until the copy is issued, and the destination is allocated at the start of the split rather than lazily in the middle of it. Their cost is real and budgeted — the graph is grown by total_inputs * 2 * n_copies slots.

Between the split view and the copy, one backend hook fires: ggml_backend_graph_optimize(), which lets a backend rewrite its own subgraph — op fusion, reordering — before allocation sees it. It must happen here, before graph_copy is built, "so they are in sync".

Liveness by refcount, offsets by free list

ggml_gallocr holds one dynamic allocator per buffer type. It walks the graph once counting, for each tensor, how many children and how many views reference it. Then it walks again in execution order: allocate the node, then decrement each parent's child count and free any parent that reaches zero children and zero views. A freed tensor's bytes go back on a free list and are immediately available to the next node.

It is a simulation. Nothing is allocated on the device during the walk — the output is a per-tensor (chunk, offset) and a required arena size. Only afterwards is one real buffer per backend allocated at that size, and ggml_gallocr_alloc_graph just stamps base + offset into each tensor's data pointer.

Two refinements are worth knowing. Ops that ggml_op_can_inplace will reuse a dying parent's storage outright instead of taking new space. And the arena is chunked — if a single allocation would exceed the buffer type's max_size, a new chunk is opened rather than failing.

Two lists, swapped every graph

At the end of every split, the scheduler swaps node_backend_ids with prev_node_backend_ids. The next allocation compares them — but not for equality:

if (sched->node_backend_ids[i] != sched->prev_node_backend_ids[i] &&
    sched->bufts[sched->node_backend_ids[i]] != sched->bufts[sched->prev_node_backend_ids[i]]) {
    backend_ids_changed = true;
    break;
}

A node moving between two backends that share a buffer type is not a change worth reallocating for. Only a change of memory counts.

If the ids did change, or if ggml_gallocr_alloc_graph reports the existing plan no longer fits, the scheduler escalates. Note the order of operations, and the comment explaining it:

// the re-allocation may cause the split inputs to be moved to a different address
// synchronize without ggml_backend_sched_synchronize to avoid changing cur_copy
for (int i = 0; i < sched->n_backends; i++) {
    ggml_backend_synchronize(sched->backends[i]);
}
ggml_gallocr_reserve_n(sched->galloc, &sched->graph,
                       sched->node_backend_ids, sched->leaf_backend_ids);

Every device must be idle before the arena moves under it, and the raw per-backend synchronize is used deliberately so the pipeline-parallel copy index is not disturbed.

Catching the reallocation you did not intend

A reallocation mid-run is a latency spike and, worse, invalidates captured CUDA graphs. So there is a dedicated tripwire. The scheduler records the graph size it computed this round and last round, and defines "unexpected" as: the plan was thrown away even though nothing about the assignment or the size changed.

const bool unexpected = !backend_ids_changed &&
    sched->debug_prev_graph_size == sched->debug_graph_size;

if (unexpected || sched->debug_realloc > 1) {
    GGML_ABORT("unexpected graph reallocation (graph size = %d, nodes = %d, leafs = %d)");
}

Run a suspect model under GGML_SCHED_DEBUG_REALLOC=1 and it will stop at the first offender with a stack trace, rather than silently costing you 5% throughput. GGML_SCHED_NO_REALLOC at compile time sets the same flag to 1 by default.

Two kinds of input, two disciplines

A tensor flagged GGML_TENSOR_FLAG_INPUT is owned by the caller, who may overwrite it the moment the call returns. So it is copied eagerly, after a full synchronize:

// inputs from the user must be copied immediately to prevent the user
// overwriting the data before the copy is done

Everything else is an intermediate produced by an earlier split. There the scheduler only needs to know that the destination staging tensor is no longer being read by an in-flight graph — so it waits on the recorded event (or synchronizes, if the backend has no events) and then tries cpy_tensor_async, falling back to a synchronous copy. On CUDA that async path is a peer-to-peer cudaMemcpyAsync; the two GPUs never round-trip through host memory.

Copying only the experts this batch selected

The most consequential optimisation in the whole execution path is a special case for mixture-of-experts. When a split's first node is a MUL_MAT_ID whose weight input is a host buffer, copying the whole expert tensor would move hundreds of megabytes to use a handful of experts. Instead:

1. Read the router's id tensor back to the host (using the original if it has not been copied to this backend yet — a real ordering hazard the code handles explicitly).
2. Mark used experts in a bitset.
3. Walk the bitset, grouping consecutive used experts into single transfers.
4. Copy each run, plus up to 512 bytes of tail padding.

// copy a bit extra at the to ensure there are no NaNs in the padding of the last expert
// this is necessary for MMQ in the CUDA backend
const size_t padding_end = last_id < n_expert - 1 ? padding : 0;

The results of the previous batch's id readback are cached in prev_ids_tensor, so a graph that uses the same routing for several consecutive matmuls pays for the readback once. The commented-out ADD_ID case notes that those weights are small enough not to be worth the machinery.

Four copies of every crossing tensor

With parallel set, every staging tensor is allocated GGML_SCHED_MAX_COPIES = 4 times and cur_copy rotates per graph. Split k of batch n can be filling copy 2 while split k+1 of batch n−1 still reads copy 1. Devices stay busy across micro-batch boundaries instead of draining at every layer handoff. The price is four times the staging memory, which is why llama.cpp gates it hard:

bool pipeline_parallel =
    model.n_devices() > 1 &&
    model.n_gpu_layers() > model.hparams.n_layer_all &&
    model.split_mode() == LLAMA_SPLIT_MODE_LAYER &&
    cparams.offload_kqv &&
    !model.has_tensor_overrides();

plus a per-device check that every non-CPU backend reports both caps.async and caps.events. If the resulting reserve fails to allocate, llama.cpp logs "compute buffer allocation failed, retrying without pipeline parallelism" and rebuilds the scheduler with parallel=false.

Why generation always uses copy 0

The rotation has a quiet exception that is easy to miss and expensive to break:

void ggml_backend_sched_synchronize(ggml_backend_sched_t sched) {
    for (int i = 0; i < sched->n_backends; i++) ggml_backend_synchronize(sched->backends[i]);
    if (!sched->is_alloc) {
        // if the graph is not already allocated, always use copy 0 after a synchronization
        // this ensures that during generation the same copy is used every time,
        // which avoids changes in the graph that could cause CUDA or other graphs to be disabled
        sched->next_copy = 0;
    }
}

Token generation synchronizes after every token. If the copy index kept rotating, every token would produce a structurally different graph — different tensor addresses in the same slots — and the CUDA graph capture would be invalidated each time. Pinning to copy 0 whenever the scheduler is idle keeps the generation graph bit-identical from token to token.

The eval callback path (ggml_backend_sched_set_eval_callback) is the other execution mode: instead of issuing the split as one graph, the scheduler batches maximal runs of nodes the caller is not interested in, issues those, synchronizes, and hands the observed tensor over. It is how llama-eval-callback and the perplexity tooling inspect intermediates, and it costs a synchronize per observation point.

One backend that is secretly N

ggml_backend_meta_device(devs, n_devs, get_split_state, ud) builds a device that wraps N real devices. It implements the whole device interface by delegation: supports_op is true only if all members support the op; get_memory sums; caps are the AND of every member's caps. A meta buffer is N real buffers; a meta tensor is N real tensor slices; a meta backend holds N real backends.

From the scheduler's side, none of this is visible. llama.cpp pushes exactly one entry into model.devices for tensor-parallel mode, so ggml_backend_sched_new receives [Meta(CUDA0,CUDA1), CPU] — two backends. Splits, copies, priority and the five passes all work exactly as described above, on a two-element array. Everything in this section happens strictly below graph_compute.

Three ways a tensor can be distributed

Every tensor has a ggml_backend_meta_split_state: an axis plus, for real axes, the exact per-device element counts. The three meaningful values carry the whole tensor-parallel algebra.

For statically allocated tensors the state comes from a user callback. llama.cpp's implementation is a long table of regexes over GGUF tensor names — blk.N.attn_q.weight splits by output rows, blk.N.attn_output.weight splits by input columns, blk.N.attn_norm.weight mirrors — with a rotation term so that rounding a non-divisible head count does not always short-change the same GPU.

Compute tensors derive their state from their sources

Nothing tells the meta backend how an intermediate is distributed — it works it out. ggml_backend_meta_get_split_state recursively resolves every source, then switches on the op. The matmul rule is the one that matters:

MIRRORED × MIRRORED         -> MIRRORED        // replicated work
src0 AXIS_1 × MIRRORED src1 -> AXIS_0          // column-parallel: outputs are split
src0 MIRRORED × AXIS_1 src1 -> src1's state
src0 AXIS_0 × AXIS_0 src1   -> PARTIAL        // row-parallel: needs a sum

That last line is the entire cost model of tensor parallelism in four tokens. A column-parallel layer (attention QKV, FFN up/gate) produces a cleanly split output and costs nothing. A row-parallel layer (attention output projection, FFN down) produces per-device partial sums, and those must be reduced before anyone can use them.

Around thirty more rules handle the rest: handle_per_row for norms and softmax (assert the split is not along the row, then pass through), handle_reshape which recomputes which output dimension a split axis lands on, handle_permute which follows op_params, handle_flash_attn_ext which requires Q/K/V split on axis 2 (heads) and returns axis 1, and handle_bin_bcast for the broadcast cases. Unhandled ops abort by name — the table is a whitelist, deliberately.

The assume_sync parameter is a small piece of cleverness: when resolving a tensor's sources, it is passed as true, which makes a would-be PARTIAL resolve as MIRRORED. That is the state the source will be in by the time this node runs, because the reduce will have happened. The cache is keyed on both the tensor and that flag, and invalidated by memcmp against a snapshot of the tensor struct.

Subgraphs, and the AllReduce between them

ggml_backend_meta_graph_compute maps every node to its per-device slice, then cuts the node list into subgraphs — a subgraph ends at every node whose split state is PARTIAL. Then the loop is trivial: issue subgraph i on all N backends asynchronously, AllReduce the last node of each, repeat. The whole graph is rebuilt only when the incoming graph's uid changes, so the per-token path just re-issues cached subgraphs.

The reduce itself prefers a backend-native implementation, discovered through the registry:

comm_init = ggml_backend_reg_get_proc_address(reg, "ggml_backend_comm_init");
// ...
comm_allreduce = ggml_backend_reg_get_proc_address(reg, "ggml_backend_comm_allreduce_tensor");

On CUDA that resolves to NCCL. When it is absent or declines, the generic fallback runs a butterfly reduction in ceil(log2(N)) steps: at step with stride s, device j pushes its buffer to device j⊕s, which adds it in place. Non-power-of-two device counts are handled by folding the excess devices into the lower ones first and copying the result back at the end. Each step uses its own temporary buffer so the copies of different steps cannot alias.

Delaying the reduce past mirrored work

The reduce point does not have to be the PARTIAL node itself. If the operations immediately following it are elementwise against MIRRORED operands — an ADD_ID bias, a chain of MULs by mirrored gates — then reducing after them is arithmetically identical and moves less data. get_i_delayed walks forward recognising exactly that pattern, plus the specific view-and-add tree that MoE expert combination produces, and returns the last node it can safely absorb.

This interacts with a corner case worth flagging, because the code comments it at length: a device whose slice of a tensor is zero-sized has its GGML_TENSOR_FLAG_COMPUTE cleared, and normally gets correct data by participating in the reduce with zeroed contents. If the reduce is delayed, every node up to the new reduce point must also have its compute flag cleared, or that device would contribute garbage. The fallback reduce explicitly scales such nodes by 0 before summing — with its own honest note, FIXME 0.0f * NaN == NaN.

The file is frank about its remaining rough edges: the split-state cache is unbounded, external views are held in a rotating pair of containers that "works correctly for llama.cpp" rather than in general, and buffer base addresses are placeholder constants. It is a young subsystem, and it says so.

What each code means

The SET_CAUSE macros compile to nothing by default — flipping the #if 0 above them to #if 1 costs a static 128-byte-per-node table and gives you the reason for every assignment in the GGML_SCHED_DEBUG=2 output.

$ GGML_SCHED_DEBUG=2 ./llama-cli -m model.gguf -ngl 99 -p hi -n 1 2>&1 | head -40

## SPLIT #0: CUDA0 # 2 inputs: [inp_tokens (   4K)] [KQ_mask (  32K)]
node #  4 (   MUL_MAT):               Qcur-0 (   1K) [CUDA0    1.wgt1] use=1,c=1: norm-0 ... attn_q.weight ...
node #  7 (      ROPE):              Qcur-0  (   1K) [CUDA0    2.sup] use=1,c=1:
...
## SPLIT #1: CPU # 1 inputs: [ffn_norm-0 (   1K)]

Read it backwards from the split headers: each header names the backend and how many tensors had to be copied in. A split with many inputs, or a long run of tiny splits, is where the graph is fighting your device layout.

Environment and flags

Four symptoms and what they mean

Split count in the hundreds at -ngl 99. Something is still on the CPU. Usually the KV cache (check offload_kqv), an -ot rule that did not match what you thought, or a tensor type the GPU backend refuses so pass 2 leaves holes.

Prompt processing fast, generation slow, same model. Expected if weights are partly in host RAM: at batch ≥ 32 the 1.off path pulls the matmuls onto the GPU, below it they stay on the CPU. Confirm by comparing the two split counts llama.cpp prints at startup.

"failed to allocate graph, reserving" in the middle of a run. The plan went stale. Either the graph shape changed (a different batch size than reserved) or a backend id moved to a different buffer type. Reproduce under GGML_SCHED_DEBUG_REALLOC=1.

Assert: "all nodes should be assigned by now". No CPU fallback in the backend array, or an op that nothing in the array supports. The array's last entry must be a CPU device, and that is checked at construction — so if you got here, some op is unsupported everywhere.