index

The model name does not determine whether it runs on your device. A runnable configuration is a combination of a checkpoint, a numerical representation, a context length, a runtime, a hardware backend, and a memory placement plan.

Two files described as the same model can require different amounts of memory. An installed accelerator can sit idle when the runtime lacks a compatible backend. Disk offload can execute a checkpoint larger than available RAM, with storage reads entering the generation path.

This guide provides a repeatable way to estimate a configuration before downloading it and identify the next constraint when the estimate does not fit.

Weight tensors set the first memory estimate

A language model checkpoint is mostly a collection of tensors. The parameter count tells you how many values those tensors contain. The stored precision tells you how many bits are used for each value.

A parameter is a learned numerical value. A checkpoint is the saved set of those learned values plus the metadata needed to reconstruct the model. Parameter count is useful because it gives the first capacity estimate before runtime-specific details enter the calculation.

The simplest weight-size estimate is:

$$\text{weight bytes} = \frac{\text{parameter count}\times\text{stored bits per parameter}}{8}$$

The division by eight converts bits to bytes. This lower bound assumes uniform encoding. Quantization metadata and mixed-precision tensors increase the final file size.

For an illustrative checkpoint containing exactly eight billion parameters:

$$\frac{8{,}000{,}000{,}000\times16}{8} = 16{,}000{,}000{,}000\text{ bytes}$$

The same arithmetic with four stored bits per parameter gives four billion bytes. These are derived values from the displayed formula. They do not include quantization scales, tensor metadata, mixed-precision layers, tokenizer files, runtime buffers, or the key-value cache.

Real quantized formats make the estimate less exact. A GGUF file stores tensor data and metadata, and its quantization blocks can include scales and other block-level values alongside the low-bit weights.¹ A label such as four-bit therefore describes the main weight encoding, not a guarantee that the whole file uses exactly four bits for every parameter.

Mixture-of-experts model listings can show both total and active parameter counts. Use the total parameters present in the checkpoint when estimating stored weights. The active count describes the subset selected for one token’s computation, not the complete set of expert weights that must be stored. The Mixtral paper makes this distinction explicitly when it reports separate total and active counts.²

Parameters billionWeight precision bitsAvailable memory GiB

parameters×bits per weight÷ 8÷ 2³⁰= weight GiB

Theoretical weight floor3.73 GiB

Memory left before runtime costs12.27 GiB

The displayed weights fit inside the entered memory. The runtime still needs room for the KV cache, activations, scratch buffers, quantization metadata, and its own code.

The calculator separates the weight estimate from the additional allocations discussed below. Treat its result as a screening test. If the tensors alone exceed every available memory tier, the runtime needs quantization, partial placement, or disk-backed execution before it can load the checkpoint.

Stored precision and compute precision are separate

Precision answers more than one question:

• How are weights stored in the checkpoint?
• In which type are weights represented after loading?
• In which type does the hardware execute matrix operations?
• In which type are intermediate activations and cache entries stored?

Those answers can differ. A runtime can store a weight matrix in a low-bit format, unpack blocks inside a kernel, and accumulate partial products in a wider type. The checkpoint is smaller, but the accelerator still needs a kernel that understands that encoding.

NVIDIA documents distinct floating-point and integer formats for TensorRT, including FP32, FP16, BF16, FP8, FP4, INT8, and INT4.³ The labels describe different bit layouts and numerical ranges. They are not interchangeable labels for reduced storage.

Transformers quantization methods also differ in hardware support. The Hugging Face compatibility table lists separate support across CPU, CUDA, ROCm, Metal, Intel GPU, and other backends.⁴ A checkpoint that loads on one backend can fall back to a slower implementation or fail to load on another backend even when both devices have enough memory.

CheckpointQuantized weight blocksintegers plus scales and metadata

→

Kernel inputUnpack or dequantizemethod depends on the selected kernel

→

Matrix operationMultiply and accumulatesupported types depend on the accelerator

→

ActivationRuntime-selected typenot necessarily the checkpoint type

File question

How are weights encoded?

Kernel question

Which operations and data types can this backend execute?

Fit question

How much memory do weights, metadata, caches, and buffers occupy together?

Three practical consequences follow.

First, lower stored precision usually reduces checkpoint storage and weight memory, but does not reduce every allocation by the same ratio. Activations, cache entries, and selected layers can remain in wider types.

Second, a lower-bit format is useful only when the runtime provides kernels for the device. NVIDIA’s TensorRT RTX documentation, for example, ties FP8 support to Ada Lovelace and newer GPUs, while BF16 and INT4 support begin with Ampere in that runtime.⁵ Hardware generation and software backend both matter.

Third, quantization changes model outputs. Post-training quantization maps trained values into a smaller numerical representation. Per-channel, per-group, and per-block methods keep separate quantization parameters for smaller regions of a tensor, which increases metadata and implementation complexity in exchange for finer scaling.⁶ Choose the most compressed format only after evaluating it on the tasks you intend to run.

Storage, RAM, and accelerator memory are different budgets

The word memory is too broad to diagnose a failed load. A local inference system usually crosses several capacity limits:

Resource	What occupies it	What happens when it is insufficient
Storage	Checkpoint files, tokenizer files, runtime binaries, compiled kernels, and caches	The model cannot be downloaded, converted, or retained
System RAM	CPU-resident weights, file-backed pages, runtime state, token buffers, and offloaded tensors	Loading fails, the operating system compresses memory, or pages are repeatedly moved to storage
Discrete accelerator memory	GPU-resident weights, activations, key-value cache, and working buffers	Fewer layers fit on the accelerator or allocation fails
Unified memory	CPU and GPU allocations from one physical pool	CPU and GPU compete for the same capacity even though explicit tensor copies can be avoided

Apple silicon provides a concrete unified-memory design. Apple’s MLX documentation states that the CPU and GPU access the same memory pool and that MLX arrays do not need to be moved when an operation changes devices.⁷ This removes a separate CPU-to-GPU copy step for those arrays. It does not create additional capacity. Model weights, cache entries, the operating system, and other applications still share the pool.

Persistent tierStorage

Checkpoint files, tokenizer files, and runtime assets

Capacity answers whether the files can be kept locally.

→read

Host tierSystem memory

Mapped or loaded weights, CPU workspaces, and offloaded state

Capacity and bandwidth affect CPU inference and offload.

⇄place

Compute tierAccelerator memory

Resident layers, KV cache, activations, and kernel workspaces

A unified-memory device shares a pool, not every cost.

Storage fit is only the first capacity check. The loaded checkpoint may exceed system RAM or GPU memory. A split placement can solve capacity while adding interconnect traffic.

Leave headroom for the operating system and runtime instead of matching a checkpoint file to the last available byte. The exact headroom is runtime and workload dependent, so measure it on the target device rather than applying a universal percentage.

Context length grows the key-value cache

Autoregressive generation reuses attention keys and values from earlier tokens. The key-value cache stores those tensors so the model does not recompute the full prefix for every generated token.

For a standard cache without additional compression, an approximate per-token allocation is:

$$\text{KV bytes per token} = 2 \times \text{layers} \times \text{KV heads} \times \text{head dimension} \times \text{bytes per cache element}$$

The first factor represents one key tensor and one value tensor. Hugging Face derives the same structure in its cache quantization explanation.⁸

Grouped-query attention reduces this allocation because several query heads share a smaller set of key-value heads. The relevant field is the number of key-value heads, not the total number of attention heads.

2×layers×KV heads×head width×bytes×cached tokens

LayersKV headsHead widthBytes per value

4,096 tokens8,192 tokens32,768 tokens131,072 tokens

Per cached token128 KiB

Selected context1.00 GiB

These are derived tensor values for the inputs shown. Sliding-window attention, cache quantization, offload, batching, and runtime allocation can change the memory that is resident on one device.

The estimator reports values derived from the inputs shown in the figure. If the context length is multiplied while every architecture and precision field stays fixed, the cache allocation changes by the same factor.

Runtime policy modifies this result:

• A dynamic cache grows as tokens arrive.
• A static cache allocates its configured maximum capacity before all positions are used.
• A quantized cache stores keys and values at lower precision.
• An offloaded cache keeps the active layer on the accelerator and places other layers of cache data in CPU memory.

The Transformers cache documentation describes those strategies and notes that offloading saves accelerator memory at a throughput cost.⁹ A model that fits for a short prompt can therefore run out of memory at a longer context even though its weight tensors never change.

Context settings also affect generation length. The cache contains prompt tokens plus generated tokens that remain inside the active context. Test with the longest prompt and generation you intend to serve.

Accelerator support is an operation-level property

A GPU or neural processing unit does not accelerate a model merely because it exists in the machine. The runtime must provide a backend for the device, and that backend must implement the operations, data types, tensor shapes, and memory layouts used by the graph.

ONNX Runtime formalizes this through execution providers. Each provider reports the graph nodes or subgraphs it supports. ONNX Runtime partitions the graph, assigns supported regions to prioritized providers, and uses a lower-priority provider such as the CPU for unsupported regions.¹⁰

Priority 1Accelerator providerclaims supported operators and subgraphs

Priority 2Secondary providerreceives work the first provider cannot execute

FallbackCPU providerruns remaining supported graph regions

embed

→

attention

→

unsupported op

→

projection

accelerator region fallback region

Every boundary can add synchronization, format conversion, or data movement.

This graph-level view explains several common outcomes:

• The complete model runs on the accelerator because every required operation has a supported kernel.
• Most layers run on the accelerator while a small unsupported region runs on the CPU.
• Frequent boundaries between providers add transfers and synchronization.
• The runtime rejects the graph because an operation has no valid implementation.
• A low-precision checkpoint loads, but selected operations execute in a wider type.

Core ML exposes a related policy through compute units. An application can request the CPU alone, the CPU and GPU, the CPU and Neural Engine, or all available units; the system then selects compatible hardware for the model.¹¹ The request controls eligible devices, not a guarantee that every operation uses the same accelerator.

Peak throughput figures do not answer this compatibility question. Compare a device only after confirming that your runtime can execute the model’s operations and quantization format on it. Then benchmark the complete prompt-processing and token-generation path.

File format, runtime, and backend form one stack

A model file is not an executable application. The format determines how tensors and metadata are represented. The runtime parses that representation, constructs operations, chooses kernels, and places allocations.

Consider three common paths:

GGUF with llama.cpp. GGUF contains tensors and model metadata for the GGML ecosystem.¹ llama.cpp supports several CPU and accelerator backends, integer quantization levels, memory mapping, and CPU plus GPU hybrid inference.¹²

Safetensors with a framework runtime. Safetensors records tensor names, dtypes, shapes, and byte offsets in a format designed for safe loading and memory mapping.¹³ The file does not decide whether a transformer runs through PyTorch eager execution, a compiled graph, CUDA kernels, Metal kernels, or another backend.

ONNX with execution providers. An ONNX graph describes operations and tensors. ONNX Runtime assigns graph regions to execution providers according to backend support and priority.¹⁰

Conversion between formats can change more than the filename. An exporter can rewrite operations, fold constants, select a deployment target, or choose a default tensor precision. Core ML Tools, for example, uses float16 as the default precision for ML Program conversion unless the conversion options specify another behavior.¹⁴

Before downloading a checkpoint, answer these questions:

1. Which runtime reads the format?
2. Which backend in that runtime supports the target device?
3. Does the backend implement this quantization and architecture?
4. Which tensors remain in system RAM?
5. Which operations fall back to the CPU?

If one answer is unknown, the parameter count alone cannot establish compatibility.

Layer offload combines memory tiers

Transformer blocks are repeated groups of operations. A runtime can place some blocks on an accelerator and keep the rest in system RAM. During inference, each block executes where its weights reside.

llama.cpp calls this CPU plus GPU hybrid inference and exposes controls for the number of layers offloaded to the GPU, target devices, and tensor split behavior.¹² This placement can run a model whose weights exceed accelerator memory as long as the remaining allocations fit elsewhere.

Path AAccelerator resident

storage→ load once →accelerator memory

Weights and working state remain near the compute kernels.

Path BLayer split

system memory⇄ selected layers ⇄accelerator memory

Some layers run on the accelerator while others remain on the host.

Path CDisk offload

mapped storage→ host memory →device

Capacity increases, but each transfer enters the inference path.

Less movement per generated token

More capacity, more data movement

Offload changes the bottleneck. When every layer and the active cache fit in accelerator memory, generation can remain within that high-bandwidth memory domain. When layers are split, hidden states cross the device boundary at placement transitions. When weights are fetched from storage, each needed tensor must first become available through system memory.

The fastest placement is not always the placement with the most offloaded layers. Accelerator memory also holds the key-value cache and working buffers. Giving every remaining byte to weights can leave too little room for the requested context. Measure prompt processing and token generation with the final context setting.

Memory mapping avoids an eager file copy

Memory mapping gives a process a virtual address range backed by a file. The operating system loads file pages as they are accessed instead of requiring the application to copy the complete file into a separate heap allocation at startup.

Safetensors supports lazy loading through memory mapping, and llama.cpp enables memory mapping by default unless its command-line option disables it.¹³¹⁵

This can reduce startup copies and allow the operating system to reclaim clean file-backed pages. It does not make storage equivalent to RAM. A page that is not resident still has to be read from the storage device. If the working set repeatedly exceeds available RAM, page faults and eviction place storage latency on the generation path.

Memory locking makes the opposite trade. llama.cpp provides an option that asks the operating system not to swap model data after it is loaded.¹⁵ Locked pages can stabilize residency, but they consume physical memory that other processes cannot reclaim.

Use memory mapping when the format and runtime support it, then watch resident memory and page-fault behavior during a representative request. A successful load does not establish steady-state performance.

Disk offload exchanges capacity for data movement

Some framework runtimes can place selected weights on disk when neither accelerator memory nor system RAM can hold the full checkpoint. Hugging Face Accelerate device maps support accelerator, CPU, and disk placements.¹⁶

For disk-resident layers, Accelerate describes a staged path: weights move from disk into system RAM, then to the execution device before the layer runs. After execution, the memory used for those weights can be released for the next layer.¹⁶

Disk-resident execution can load a larger checkpoint, but it puts storage reads inside model execution. Accelerate’s documentation warns that hard-drive offload can be slow when the storage and CPU path cannot move data quickly enough.¹⁶

Disk offload increases usable capacity by fetching weights when needed. It does not remove the RAM required to stage those weights or the device memory required for execution. Use it when completing the task matters more than interactive token latency, or when only a small fraction of tensors need disk placement. Benchmark with a warm and a cold file cache so an operating-system cache does not hide the storage dependency.

Storage capacity includes more than one checkpoint

The downloaded file is only one storage consumer. A working local setup can also contain:

• the original checkpoint and one or more quantized conversions
• temporary files created during conversion
• tokenizer and configuration files
• compiled engine plans or kernel caches
• runtime packages and accelerator libraries
• generated logs and application data

Do not infer required free space from the final quantized file alone if conversion happens on the same device. The conversion process can require the source and destination files to coexist.

Storage speed matters when the runtime memory maps weights, loads model shards on demand, or uses disk offload. Once the active working set remains resident in RAM or accelerator memory, storage bandwidth should leave the steady-state token path. Use system I/O counters to verify which regime you are in.

A model-fit procedure

Use the same order for each candidate configuration:

1. Identify the exact checkpoint and format.
2. Calculate the theoretical weight size from parameter count and stored precision.
3. Use the actual file size when the checkpoint is available.
4. Add the key-value cache for the target context and cache precision.
5. Reserve capacity for runtime buffers, the operating system, and other applications.
6. Map those allocations to accelerator memory, system RAM, unified memory, and storage.
7. Confirm runtime support for the format, architecture, quantization method, and device backend.
8. Decide whether full acceleration, layer offload, memory mapping, or disk offload is required.
9. Benchmark the longest intended prompt and generation on the target device.
10. Evaluate output quality on the tasks that matter for the application.

1. 01
   Confirm the runtimeDoes it read this model format?
   →
2. 02
   Confirm the backendDoes it support the required operators and types?
   →
3. 03
   Calculate the weightsParameters × stored bits, then include metadata.
   →
4. 04
   Add working memoryKV cache, activations, scratch space, and runtime overhead.
   →
5. 05
   Choose placementAccelerator, host, hybrid, or disk-backed.
   →
6. 06
   Measure the real workloadPrompt length, generation length, latency, and output quality.

Changing an earlier decision can invalidate every estimate that follows it.

The sequence deliberately places compatibility before performance claims. A device can have enough aggregate memory while lacking a usable kernel. A runtime can load a checkpoint while paging so heavily that interactive use is impractical. A quantized model can execute quickly while producing unacceptable results for a specific task.

Record the complete configuration with every benchmark: checkpoint revision, quantization, runtime version, backend, layer placement, context length, cache type, prompt length, generated length, and device power mode. Without those fields, two local-inference measurements are not directly comparable.

References

Footnotes

1. Hugging Face, “GGUF.” https://huggingface.co/docs/hub/gguf ↩ ↩²

2. Jiang et al., “Mixtral of Experts.” https://arxiv.org/abs/2401.04088 ↩

3. NVIDIA, “Data Format Descriptions.” https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/data-format-desc.html ↩

4. Hugging Face, “Quantization Overview.” https://huggingface.co/docs/transformers/quantization/overview ↩

5. NVIDIA, “TensorRT RTX Best Practices.” https://docs.nvidia.com/deeplearning/tensorrt-rtx/latest/performance/best-practices.html ↩

6. Hugging Face, “Quantization Concepts.” https://huggingface.co/docs/transformers/quantization/concept_guide ↩

7. MLX, “Unified Memory.” https://ml-explore.github.io/mlx/build/html/usage/unified_memory.html ↩

8. Hugging Face, “KV Cache Quantization.” https://huggingface.co/blog/kv-cache-quantization ↩

9. Hugging Face, “Cache Strategies.” https://huggingface.co/docs/transformers/kv_cache ↩

10. ONNX Runtime, “Execution Providers.” https://onnxruntime.ai/docs/execution-providers/ ↩ ↩²

11. Apple, “MLComputeUnits.” https://developer.apple.com/documentation/coreml/mlcomputeunits ↩

12. ggml-org, “llama.cpp.” https://github.com/ggml-org/llama.cpp ↩ ↩²

13. Hugging Face, “Safetensors.” https://github.com/huggingface/safetensors ↩ ↩²

14. Apple, “ML Program Conversion.” https://apple.github.io/coremltools/docs-guides/source/new-conversion-options.html ↩

15. ggml-org, “llama.cpp CLI.” https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md ↩ ↩²

16. Hugging Face, “Big Model Inference.” https://huggingface.co/docs/accelerate/v0.29.0/en/concept_guides/big_model_inference ↩ ↩² ↩³