index

Apple silicon exposes several routes for machine-learning computation: CPU operations, GPU kernels, matrix accelerators within the GPU, and the separate Neural Engine. MLX uses the CPU and GPU over a shared memory pool. On M5, its Metal path also uses the Neural Accelerators introduced inside the GPU cores.¹

This architecture explains the different gains seen in prompt processing and token generation. Prefill provides large matrix operations that benefit from more compute. Small-batch decode repeatedly reads weights and attention state, making memory bandwidth a major limit. Memory capacity determines which models and context lengths can be held in the first place.

The M3-to-M6 progression changes all three quantities, but by different amounts. Comparing them separately gives a useful account of local model performance.

GPU accelerators and the Neural Engine

The Neural Engine is a dedicated processor block. The Neural Accelerators introduced in M5 are inside the GPU cores and are reached through the GPU programming path. Apple describes MLX using Metal 4 Tensor Operations and Metal Performance Primitives to use this matrix hardware.¹

An MLX operation dispatched to the GPU therefore remains GPU execution even when those accelerators perform its matrix work. Core ML and other system frameworks provide a separate deployment path whose hardware placement depends on supported operations and runtime decisions.

The distinction also applies to specifications. More Neural Engine compute does not directly imply the same increase in MLX generation speed. The relevant quantities for an MLX workload are the GPU operations it executes, their efficiency on the selected hardware, and the memory traffic supplying them.

Hardware progression

The specifications come from Apple’s M3 announcement, M5 announcement, M5 Pro/Max announcement, and M6/M5 Ultra announcement.² The M6 entry describes the announced base chip in Mac mini.

M5 adds matrix throughput within each GPU core. M6 extends that GPU design to twelve cores and increases base-chip memory bandwidth. M5 Ultra occupies a different capacity class, with a much larger memory pool and higher bandwidth for models that exceed the base chips’ working sets.

These are separate ways to improve a workload. More capacity admits larger models or more live sequences. More bandwidth supplies weights and cache state faster. More matrix throughput accelerates operations with enough arithmetic to use it.

Unified memory and the working set

MLX arrays can be used by CPU and GPU operations without explicit transfers between separate host and device allocations.³ This makes mixed execution straightforward: a GPU operation can produce an array that a CPU operation subsequently consumes in the same memory system.

The shared pool still has finite capacity and bandwidth. Model weights, attention caches, temporary arrays, runtime allocations, and other applications compete for it. A model’s checkpoint size is therefore only one part of its execution footprint.

For a dense model with 8 billion parameters, raw two-byte weights occupy about 16 GB. Four-bit weights have an idealized storage cost of 4 GB. Quantization metadata and tensors retained at higher precision add to that lower bound. The KV cache grows separately with context length and the number of live sequences.

Mixture-of-experts models make the distinction between storage and computation especially visible. Active parameter count describes the experts used for a token. Total parameter count describes the collection of weights that may need to remain resident. A model with 30 billion total parameters and a few billion active parameters has a larger storage requirement than its active count suggests.

Unified memory makes these larger working sets accessible to the GPU, but fitting them remains an explicit allocation problem. Swapping changes the data path and can dominate the benefit of faster arithmetic.

Prefill uses matrix throughput

Prefill processes the input sequence and constructs the attention state used during generation. Many token positions participate in the layer operations, producing matrix workloads with substantial parallelism and reuse.

Apple’s M5 study compares selected Qwen and GPT-OSS models on similarly configured M4 and M5 MacBook Pros. It uses 4,096-token prompts and reports TTFT separately from generation speed.¹ The GPU matrix hardware has a larger effect on prompt processing than on the subsequent token stream.

This phase matters most when a request contains a long document and produces a short answer, or when the application starts a new context frequently. Prefix caching changes the balance by reusing previously computed state. A warm-prefix request can have a much shorter prefill even on unchanged hardware.

A hardware comparison therefore has two meaningful prefill cases: a new context that requires the full computation, and a repeated prefix that allows reuse. They describe different application behavior.

Decode and memory bandwidth

At batch size one, dense-model decoding often spends much of each step supplying weights from memory. A simplified lower bound is the weight bytes read divided by sustained bandwidth.

For an illustrative 4 GB of weight reads per generated token:

at 120 GB/s: 4 / 120 seconds ≈ 33 ms per token
at 150 GB/s: 4 / 150 seconds ≈ 27 ms per token

Those bounds correspond to about 30 and 37.5 tokens/s before attention traffic, dequantization, synchronization, and other work. They describe a bandwidth model, not measured rates for a particular Mac.

Batching changes the arithmetic because a weight read can serve several token positions. Sparse expert activation changes which weights are used. Cache residency and implementation details change the achieved traffic. These effects explain why aggregate throughput and the speed of one interactive stream are different measurements.

Apple’s reported M5 generation improvements range from 19 to 27 percent across its tested models, with the memory-bandwidth increase from 120 to 153 GB/s identified as the main contribution.¹ M6’s 170 GB/s is roughly 11 percent above base M5. A bandwidth-bound workload can benefit from that increase, with its actual gain determined by the rest of the execution path.

Lazy execution and timing

MLX builds computation lazily. Constructing an array expression records work that may execute only when the result is needed. Evaluation boundaries therefore matter when timing an operation.⁴

An array benchmark can measure graph construction, execution, or both. Materializing the result establishes that the computation has completed. Without that boundary, a short elapsed time may describe only the creation of an expression.

Language-model measurements add their own phases: loading the model, first-run initialization, processing the prompt, and generating output. A cold-start experiment includes the first two. A warm serving experiment begins with the model already available. Keeping those intervals separate explains where a change in storage, runtime, or chip affects the result.

MLX LM and reproducible execution

MLX LM supplies model-loading, generation, and quantization workflows on top of MLX.⁵ Its release history also records changes to batching, caching, model support, and generation behavior. The software version is part of a performance configuration alongside the hardware.

An isolated environment and package record can be created with:

uv venv .venv
uv pip install --python .venv/bin/python mlx-lm
uv run --no-project --python .venv/bin/python python -c 'import mlx.core; import mlx_lm'
uv pip freeze --python .venv/bin/python > requirements-benchmark.txt

The model repository and immutable revision identify the weights separately from the Python environment. Quantization, tokenizer, sampling settings, and prompt content complete the configuration. Apple’s M5 research note also specifies macOS 26.2 or later for the accelerator path it describes.¹

This record makes comparisons interpretable. A cache-policy change can lower TTFT on the same machine, while a different quantization can change both memory use and output quality.

Workload-level comparison

A compact experiment matrix covers the main execution regimes:

TTFT, completion time, output count, and peak memory describe each run. Quality establishes whether the configurations perform the same useful task. Power mode, battery state, competing GPU work, and sustained thermal behavior explain variation between repeated runs.

The practical ordering follows the architecture. Capacity determines whether the workload fits. Prefill and decode measurements determine which resource limits it. The runtime determines how effectively the available hardware is used. This is why a larger previous-generation system can be the better fit for one model while a newer base chip is faster on another.

References

Footnotes

1. Apple Machine Learning Research, Exploring LLMs with MLX and the Neural Accelerators in the M5 GPU. Hardware, models, prompt lengths, and measured results. ↩ ↩² ↩³ ↩⁴ ↩⁵

2. Apple, Apple introduces M6 and M5 Ultra, August 25, 2026. Hardware specifications. ↩

3. MLX documentation, Unified Memory. CPU/GPU array sharing. ↩

4. MLX documentation, Lazy Evaluation. Evaluation behavior and timing implications. ↩

5. Apple ML Research, MLX LM repository. Installation, supported workflows, generation, and quantization. ↩

Prefill and low-batch decode present different shapes to the same memory system and arithmetic units.

Prompt prefill

• Many prompt positions share weight reads
• Larger matrix operations can reuse loaded tiles
• Kernel throughput and attention traffic both matter

Single-sequence decode

• One new position is computed per step
• Most active weights may be read again
• Bandwidth and cache-state size can dominate

More peak arithmetic does not imply the same speedup in both phases.

Worked storage arithmetic for eight billion parameters, before scales, metadata, KV cache, or runtime workspace.

16-bit values

16 GB

8 billion × 2 bytes

4-bit codes

4 GB

8 billion × 0.5 bytes

The 4-bit codes take one quarter of the 16-bit weight bytes, but a complete inference working set contains more than those codes.

Decimal GB and a storage example, not a measured allocation or a prediction of decode speed.