index

A model file loading on two machines proves that two loaders accepted it. It does not prove equal outputs, equivalent device placement, or similar speed. Inference portability crosses several contracts, and each can fail independently.

The best migration artifact is a small verification suite plus a manifest of the model and runtime. Inference engine architectures explains why engines optimize different layers. Here the problem is preserving model behavior while moving those layers between hardware targets.

Decide what is supposed to move

Checkpoint portability concerns weights, tensor names, dtypes, and metadata. Graph portability concerns operators, semantics, shapes, and state transitions. Kernel portability concerns compiled device code and the hardware instructions it expects. Performance portability concerns whether the resulting execution keeps the new device busy without excessive traffic or synchronization.

These properties are separable. Safetensors can carry weights without implementing their model. ONNX carries a graph with operator semantics, but an execution provider still needs kernels or a supported compiled subgraph. GGUF can preserve a supported model’s quantized representation while different llama.cpp backends choose different kernels.

Keep an original higher-precision checkpoint where possible. Treat converted and packed files as derived artifacts. Record the conversion tool revision, model configuration, quantization recipe, tokenizer, calibration data if used, and output hashes. The MLX porting guide gives a concrete example of the layout checks this requires.

Map the available execution path

This is a map of paths to verify, not a support guarantee for any device. llama.cpp documents separate CPU, CUDA, HIP, Metal, Vulkan, and SYCL builds.¹ A single binary only contains the backends and kernels actually built or loaded for it.

For SYCL, separate the language standard from a specific backend. llama.cpp’s SYCL documentation primarily targets Intel GPUs and specifies its compiler and runtime requirements.² “Uses SYCL” does not establish that a released binary runs on every vendor’s accelerator.

For CPU targets, runtime ISA dispatch explains why hardware support, OS state, compilation, and selected kernel are separate checks. A container cannot add AVX-512 to a CPU that lacks it.

Tensor cores are GPU facilities

NVIDIA Tensor Cores and other vendors’ GPU matrix hardware accelerate tile operations under a GPU execution model. They are distinct from CPU vector or matrix instruction extensions. CUDA’s programming guide places Tensor Cores inside the GPU’s streaming multiprocessor hardware.³

Using them requires a suitable kernel, supported operand formats, tile layout, and enough useful work. A matrix unit’s peak assumes particular arithmetic. An FP16 peak says little about a four-bit weight path that spends most of its time loading, unpacking, and scaling data.

Also separate weight storage from arithmetic. A quantized file can feed a floating-point matmul after unpacking; another backend may perform integer dot products and apply scales afterward. Equal file sizes do not prove equal compute paths. For a workload dominated by memory traffic, a smaller stored format may help even without native low-bit multiplication.

Repacking can change the memory budget

Suppose a toy runtime stores 64 four-bit values per group with four bytes of metadata. That is 36 bytes per group. Another kernel expects each group expanded to FP16, which occupies 128 bytes. Holding both representations while repacking needs 164 bytes per group before other allocations.

For $N$ groups, the temporary peak is at least $164N$ bytes under those assumptions, although the final FP16 weights require only $128N$. A model can therefore fit after conversion but fail during conversion. Streaming conversion, freeing source shards, or converting offline can change that peak.

This is illustrative arithmetic, not a claim that every backend expands weights. Read the selected kernel and allocator path. Some kernels consume packed codes directly; others produce transient tiles without creating a full floating-point copy.

ONNX quantization also distinguishes operator-oriented and quantize/dequantize graph representations. Runtime support and fusion determine whether the graph uses an efficient quantized path.⁴ A matching bit count is not a sufficient conversion specification.

Graph fallback can dominate the result

ONNX Runtime asks execution providers which regions they support, partitions the graph, and uses the default provider for remaining operations.⁵ The resulting cost includes boundary synchronization and data preparation. Counting accelerated nodes is a weak metric: one fallback operation inside a repeated decode loop can matter more than many accelerated setup operations.

For an NPU, require the exact device, firmware or driver, compiler, operator set, input shapes, and precision in the deployment record. A supported matrix multiplication does not establish support for a complete decoder with dynamic cache updates. Test the compiled model’s partition and state handling. NPU and DSP inference examines those accelerator-specific constraints.

Use a profile to inspect provider execution. This script accepts a model, an NPZ file containing correctly named model inputs, and an installed provider name:

import collections
import json
import sys
import numpy as np
import onnxruntime as ort

model_path, inputs_path, provider = sys.argv[1:4]
if provider not in ort.get_available_providers():
    raise RuntimeError(f"Provider is unavailable: {provider}")
providers = list(dict.fromkeys([provider, "CPUExecutionProvider"]))
options = ort.SessionOptions()
options.enable_profiling = True
session = ort.InferenceSession(
    model_path, sess_options=options, providers=providers
)
with np.load(inputs_path, allow_pickle=False) as archive:
    inputs = {name: archive[name] for name in archive.files}
for _ in range(3):
    outputs = session.run(None, inputs)
profile_path = session.end_profiling()
with open(profile_path) as source:
    events = json.load(source)
counts = collections.Counter(
    event["args"]["provider"] for event in events
    if event.get("cat") == "Node"
    and "provider" in event.get("args", {})
)
print({"registered": session.get_providers(), "node_events": dict(counts)})

These are verified ONNX Runtime session and profiling APIs.⁶ The inputs must match the graph’s names, dtypes, and shapes. Event counts include repeated runs and fused partitions; they are not a percentage of model work. Inspect durations and placement in the emitted trace. Provider registration alone cannot prove full graph coverage.

Bind memory after establishing correctness

Host-array inputs to a GPU session can introduce copies into its execution time. ONNX Runtime I/O Binding permits inputs and outputs to reside on the intended device and can avoid unnecessary host transfers.⁷ Correct binding still requires compatible buffers, valid lifetimes, and synchronization with their producer.

The same concern appears on integrated GPUs. Shared physical memory does not guarantee that two frameworks share an allocation or agree on tensor strides. A layout conversion may remain necessary. Apple backend placement follows this case in detail.

Reproduce one backend change at a time

Inside a pinned llama.cpp checkout with the required SDK installed, use a separate build directory for each configuration. For example:

cmake -S . -B build-cuda -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-cuda --parallel 8

cmake -S . -B build-vulkan -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-vulkan --parallel 8

The corresponding current switches include GGML_HIP and GGML_SYCL; their toolchain requirements differ. Metal is enabled by default on macOS in the documented build. Explicitly inspect the configuration rather than assuming an old build directory picked up a new option.¹

For a compatible local GGUF file, inspect devices and run separate prompt-processing and generation tests:

./build-cuda/bin/llama-bench --list-devices
./build-cuda/bin/llama-bench \
  -m ./model.gguf -p 512 -n 128 -ngl 99 -r 5 -o json

llama-bench documents prompt length, generation length, offloaded-layer count, repetitions, and machine-readable output.⁸ Confirm actual offload in logs. Change the executable path for the other build and keep model bytes and test sizes fixed. These synthetic tests omit much of a production service.

Before comparing speed, compare fixed-input logits or reference outputs, teacher-forced cached decoding, long-context behavior, and task quality. Then measure cold load, warm prefill, decode latency, concurrency, memory peaks, and host transfers separately. Keep the deployment only when both its behavior and its measured execution path satisfy the workload.

Footnotes

1. llama.cpp build instructions for CPU and accelerator backends. ↩ ↩²

2. llama.cpp SYCL backend targets and toolchain requirements. ↩

3. NVIDIA CUDA guide, GPU hardware and Tensor Cores. ↩

4. ONNX Runtime quantization representations and execution constraints. ↩

5. ONNX Runtime architecture and graph partitioning. ↩

6. ONNX Runtime Python sessions, providers, and profiling. ↩

7. ONNX Runtime I/O Binding and device-resident inputs and outputs. ↩

8. llama.cpp benchmark arguments and output formats. ↩

A successful load addresses only part of the migration. Keep a verification artifact for each contract.

Case	What moves	What still needs proof
Checkpoint	Weights and metadata	Names, layout, dtype, tokenizer, and model identity
Graph	Operators and state updates	Semantics, shapes, cache behavior, and provider coverage
Quantized format	Packed codes and scales	Grouping, signedness, packing, and reconstruction
Compiled kernels	Target-specific executable code	Device architecture, driver, and supported instructions
Service behavior	A workload and latency target	Quality, memory peaks, latency tails, and actual placement

Checkpoint compatibility, graph coverage, and performance need separate evidence.

ONNX Runtime graph partitioning · llama.cpp backend-specific builds

Compare provider execution using a trace. Provider registration alone cannot distinguish these cases.

Long supported partition

• Adjacent operations stay in a compatible device execution path.
• Inputs and outputs can use supported device memory binding.
• Inspect kernel duration and boundary costs.

Repeated fallback

• Unsupported regions execute through another provider.
• Repeated boundaries can add synchronization and tensor preparation.
• Inspect the decode loop, not only the number of accelerated nodes.

The placement and cost of each boundary matter more than an accelerator-enabled status flag.

Fallback may be necessary or inexpensive. Measure its actual partition and frequency before changing the graph.

ONNX Runtime provider assignment · ONNX Runtime I/O Binding