index

A processor package can contain several computers and several machines that barely resemble computers. The CPU executes instructions. A GPU runs scheduled kernels. An NPU compiler places a graph onto specialized execution resources. A video decoder consumes a compressed bitstream through a codec interface. All can shorten an inference pipeline, but they accept different kinds of work.

The useful inventory records four things for each block: its operation, its programming interface, its memory access, and the evidence that your application uses it. The broader AI hardware survey compares accelerator architectures. This article maps the less tidy collection inside a CPU core or SoC package.

Instructions describe work; execution units perform it

A scalar floating-point unit handles individual values. A fused multiply-add computes $ab+c$ with a single final rounding. Vector instructions apply arithmetic to multiple lanes, while shuffle, mask, conversion, and reduction instructions arrange those lanes into a useful algorithm. The ISA defines observable results; it does not promise a particular number of physical arithmetic pipelines.

For an illustrative 512-bit FP32 vector, there are $512/32=16$ lanes. One vector FMA represents 32 floating-point operations under the usual multiply-plus-add counting convention. That says nothing about instructions issued per cycle, frequency, memory stalls, or whether a particular chip implements that instruction. Intel and AMD implement selected x86 extensions; Arm CPUs implement selected Neon, SVE, and related extensions. See AVX-512 mechanics and Arm vector and matrix execution for the actual programming contracts.¹

Dot-product instructions combine several narrow products into wider accumulators. Matrix instructions expose more reuse through tile or matrix state. Intel VNNI, Intel AMX, Arm integer matrix multiplication extensions, and Arm SME occupy different points in that design space. SME adds streaming execution and ZA storage; software must obey its state and calling conventions. Neither the word “Arm” nor an amx_* inventory flag establishes a working specialized kernel.²

Storage precision is another independent property. Four-bit weights might expand into integer lanes, convert into FP16, or feed a fused mixed-precision kernel. “INT4 model” alone does not identify the multiply or accumulator.

A package inventory by operation

This table lists representative interfaces, not a promise that every member of a processor family includes each block.

Integrated GPUs still require driver and runtime support. A shared physical DRAM pool does not make every allocation immediately accessible in every API. Layouts, visibility rules, synchronization, and cache ownership can require work even when no PCIe transfer occurs. XMX is a particular matrix implementation, not a synonym for all Intel graphics. Intel’s joint-matrix guide explicitly requires hardware with XMX.³

On Apple systems, distinguish GPU compute from the Neural Engine. Core ML permits combinations of compute devices and chooses a compatible execution plan. Ordinary MLX GPU execution does not establish Neural Engine use. Apple inference backends explains the choices; Mac Studio tuning follows the MLX execution path.⁴

DSPs, NPUs, and media share a package, not an API

DSPs often combine scalar control with wide vector execution and explicitly managed local memory. Qualcomm describes Hexagon HVX vector resources and HMX matrix resources within its NPU architecture. That makes “DSP versus NPU” an incomplete classification: a marketed NPU can contain programmable vector machinery as well as matrix hardware. The exposed SDK and compiler determine how much of it applications can program.⁵

An NPU generally accepts a compiled graph or supported subgraph. Operator coverage includes attributes, tensor shapes, layout, and precision. A supported convolution does not imply every convolution variant works. A model with unsupported normalization or dynamic control flow can split across devices, creating intermediate conversions and synchronization. NPU, DSP, and media deployment examines those boundaries.

A media decoder implements codec-specific parsing, prediction, reconstruction, and related stages. It can save CPU time before a vision encoder sees a frame. It cannot accept an arbitrary transformer GEMM merely because some internal arithmetic resembles matrix operations. Apple’s VideoToolbox exposes encode/decode services. Similar distinctions apply to camera ISPs, image scaling hardware, display engines, and audio pipelines.⁶

The optimization opportunity is often buffer reuse. Decode a supported video format into a buffer that the next stage can consume, then avoid an unnecessary readback and color conversion. Verify the actual pixel format and ownership rules. “Hardware decoded” does not prove the frame stayed in a useful memory representation.

Graphics, security, and network blocks have narrower contracts

GPU compute cores share a device with resources built for graphics. Texture units sample and filter images; raster and depth hardware determine pixel coverage and visibility. Ray-tracing interfaces organize geometry into acceleration structures and query intersections. Hardware support varies by GPU family, and an API exposing ray queries does not by itself establish a dedicated intersection unit. Apple’s Metal capability tables distinguish software-facing features by family.⁷ These facilities can help rendering, simulation, or image preprocessing around a model. They do not accept a general matrix multiplication through the ray-tracing API.

Display engines compose and scan out surfaces. Media blocks encode and decode supported formats; image pipelines scale, convert, and process camera data. Audio processing can run on DSP resources with its own real-time firmware and interfaces. Classify the operation and public API rather than counting every marketing name as another programmable neural processor. A device may combine several operations in one engine or distribute them across multiple blocks.

Security processors combine isolation with dedicated cryptographic hardware. Apple’s Secure Enclave has its own processor, protected memory machinery, a true random-number generator, AES hardware, and a public-key accelerator for asymmetric operations.⁸ The acceleration stays behind the security interface and key-access rules. A secure key store is not spare tensor memory. Hardware entropy also serves a different purpose from a model’s reproducible pseudorandom sampling stream; applications should use the platform’s random service for cryptographic randomness.

Network controllers can calculate supported checksums and segment a large transmitted buffer into packets. Linux distinguishes hardware TCP segmentation offload from generic segmentation performed in software, and documents which checksum state the driver must provide.⁹ Such a controller can be integrated into a server SoC or attached as a separate device. It reduces packet-processing work in distributed serving, rather than executing attention. Treat RDMA, protocol offload, encryption offload, and ordinary DMA as separate capabilities; their presence and access depend on the particular controller and driver.

Crypto, copies, and compressed bytes

AES rounds, carry-less polynomial multiplication, SHA transformations, and CRC instructions operate within the CPU instruction stream. Dedicated QAT, DSA, and IAA devices have a different model. Software submits work through drivers and queues, then observes completion. Intel DML targets DSA data movement; QPL targets IAA compression and analytics. Neither library makes arbitrary tensor multiplication faster.¹⁰

These blocks can still matter to inference. TLS transport, checkpoint validation, compressed input, retrieval filters, host copies, and cache migration consume real time. Crypto, compression, and copy offload follows descriptors, batching, and the size at which offload overhead becomes worthwhile.

DMA changes who moves bytes. It does not eliminate those bytes or their contention with model reads. Device DMA addresses can also differ from CPU physical and virtual addresses. Drivers use DMA mapping interfaces and, where supported, IOMMU translation rather than assuming a userspace pointer is a universally usable address.¹¹

Infrastructure sets the useful ceiling

Caches, prefetchers, memory controllers, DRAM channels, translation caches, and coherent fabrics feed arithmetic. They are not independent tensor accelerators. Their effect can dominate a workload nonetheless.

For a toy decoder that must read 4 GB of uncached weights per step at an assumed effective 100 GB/s, the weight-read floor is 40 ms. Doubling an idle matrix unit’s peak rate cannot lower that floor. Increasing reuse, reducing bytes, or improving effective bandwidth can. The CPU and NUMA guide works through the bound and its limits.

Shared infrastructure also explains why engine rates cannot be added. The CPU, GPU, NPU, and decoder can compete for DRAM, power, and thermal headroom. The fastest isolated engine configuration may lose when all stages run together.

Turn the map into evidence

Start with a read-only Linux inventory:

lscpu
ls /sys/class/drm
ls /sys/class/accel
python3 - <<'PY'
import importlib.metadata
for name in ('torch', 'onnxruntime', 'openvino', 'mlx'):
    try:
        print(name, importlib.metadata.version(name))
    except importlib.metadata.PackageNotFoundError:
        pass
PY

Some directories will not exist. A missing entry is unresolved evidence, particularly inside containers. Next, ask the installed runtime for devices, compile the actual model, inspect placement, and profile a representative request. Keep each result attached to its OS, driver, runtime, model revision, and shape. The downloadable hardware-aware agent workflow provides a concrete record format. An inventory nominates a candidate. A trace and a correctness check establish what it did.

Footnotes

1. Intel Intrinsics Guide, including the distinct FMA, VNNI, and AMX families and their instruction requirements. ↩

2. Arm, Scalable Matrix Extension introduction, on streaming mode and ZA storage. ↩

3. Intel, Programming XMX using the SYCL joint-matrix extension. ↩

4. Apple, MLComputeUnits, which defines the compute-device combinations a model configuration permits. ↩

5. Qualcomm, Hexagon-MLIR FAQ, describing HVX, HMX, and kernel-library access. ↩

6. Apple, VideoToolbox, the codec service interface. ↩

7. Apple, Metal feature-set tables and ray tracing with acceleration structures, for the graphics and intersection interfaces and per-family support. ↩

8. Apple, The Secure Enclave, for isolated execution, entropy, AES, and public-key blocks. ↩

9. Linux kernel, Checksum offloads and segmentation offloads, for the hardware and software processing contracts. ↩

10. Intel, QPL and DML data manipulation interfaces. ↩

11. Linux kernel, Dynamic DMA mapping guide, on CPU, physical, and DMA addresses. ↩

Classify the interface before asking whether an engine can run a model.

Case	Work accepted	Software route	Evidence needed
CPU vector or tile unit	ISA operations on registers/state	Compiler and runtime dispatch	Kernel path and OS state
GPU or NPU	Kernels or supported graphs	Device runtime and compiler	Actual graph placement
Codec or copy engine	Bitstream or buffer descriptors	Media API or device queues	Engine completion and transfers
Memory infrastructure	Feeds every execution path	Allocation and placement policy	Residency and traffic counters

An instruction, a queued engine, and a cache have different programming contracts.

Representative classes. Optional engines and interfaces vary by processor and OS.

Intel intrinsics · Linux DMA mapping

Each step establishes a narrower claim about the deployed workload.

1. Inventory

   Find exposed CPU features and device identities.

2. Access

   Verify OS state, driver access, and runtime support.

3. Compile

   Check operators, shapes, layout, precision, and partitions.

4. Observe

   Trace actual kernels and transfers during a representative request.

5. Measure

   Compare correct outputs, latency, and resource usage.

Usable acceleration is established by execution and measurement of the actual model.

A successful inventory or compilation does not by itself establish a speedup.

Core ML compute units