index

Author: Aadit Agrawal

Compression succeeds only when the deployment runtime benefits from the changed model. Removing isolated weights does not guarantee faster dense matrix multiplication. A low-rank factorization does not help if the two smaller operators launch inefficiently. Distillation can produce a smaller model, but its quality depends on the teacher signal and training distribution.

MethodChanges graphNeeds retrainingRuntime requirementUnstructured pruningmask or sparse weightsusually recovery tuningsparse kernel supportStructured pruningremoves heads, channels, layersrecovery tuningdense kernels can applyDistillationstudent architecturestudent trainingordinary student runtimeLow-rank factorizationW becomes ABoptional recovery tuningtwo smaller operators

Define the deployment objective

Choose measurable constraints before modifying the model:

• resident weight memory;
• peak runtime memory;
• serialized artifact size;
• first-token latency;
• decode latency;
• throughput at the serving batch;
• energy or power under sustained load;
• task quality and safety slices.

Parameter count is an intermediate metric. Hardware latency depends on shapes, sparsity support, memory traffic, and kernel launch behavior.

Unstructured pruning removes individual weights

Magnitude pruning sets low-magnitude weights to zero. A binary mask $M$ produces:

$$\widetilde W = M \odot W$$

SparseGPT instead formulates layer-wise pruning with approximate second-order information and reconstructs outputs while removing weights.¹ The algorithmic goal is to preserve layer behavior under sparsity, but speed still requires a sparse storage format and kernels that exploit the pattern.

An unstructured sparse matrix stores nonzero values plus indices. At modest sparsity, index overhead and irregular access can outweigh skipped arithmetic. Benchmark the exact matrix shapes on the target runtime.

Pruning should be followed by recovery tuning when quality loss exceeds the budget. Keep the mask fixed during recovery unless the method explicitly supports regrowth.

Structured pruning changes dense dimensions

Structured pruning removes complete channels, attention heads, feed-forward neurons, or layers. The resulting matrices remain dense and can use ordinary kernels. This makes latency benefits easier to realize, but each removal changes more computation than one isolated weight.

Head pruning must update:

• query, key, value, and output projection shapes;
• grouped-query head mapping;
• rotary or positional dimensions;
• tensor-parallel divisibility;
• checkpoint metadata and runtime configuration.

Depth pruning removes whole transformer blocks. Width pruning changes hidden or feed-forward dimensions. NVIDIA’s Minitron work combines depth, width, attention, and feed-forward pruning with distillation-based retraining to derive smaller language models from a pretrained model.²

Measure every candidate architecture instead of assuming proportional latency. A narrower matrix can land on a less efficient hardware tile and produce less speedup than its parameter reduction suggests.

Distillation trains a separate student

Knowledge distillation trains a student against teacher outputs, often using softened class probabilities. Hinton and collaborators described transferring the ensemble or teacher function into a deployable model through this training signal.³

For language models, token-level distillation can minimize:

$$\mathcal{L}_{KD} = T^2 \operatorname{KL} \left( p_T(\cdot \mid x) \parallel p_S(\cdot \mid x) \right)$$

where $T$ is temperature. The student can also train on hard target tokens, hidden-state matches, attention relations, or generated teacher data.

Teacher logits expose more information than one sampled token, but full-vocabulary logits are expensive to store. Options include online teacher inference, top-logit storage with a residual bucket, or sequence-level distillation from teacher-generated outputs. Each changes the target.

Distillation risks copying teacher errors and narrowing behavior to the transfer distribution. Evaluate the student on real held-out data and safety tests that were not generated by the teacher.

Low-rank factorization replaces one operator with two

For $W \in \mathbb{R}^{m \times n}$, a rank-$r$ approximation writes:

$$W \approx AB, \qquad A \in \mathbb{R}^{m \times r}, \quad B \in \mathbb{R}^{r \times n}$$

The parameter count changes from $mn$ to $r(m+n)$. This is smaller only when $r$ is below the break-even rank:

$$r < \frac{mn}{m+n}$$

Truncated singular value decomposition provides a reconstruction-optimal low-rank approximation under the Frobenius norm, but minimizing weight reconstruction error does not directly minimize model loss. Recovery tuning can adapt the factors to the data distribution.

Runtime performance depends on both factor matrices. Very small intermediate rank can become launch-bound. A fused implementation or batching may be needed to realize latency gains.

Low-rank adapters such as LoRA are parameter-efficient fine-tuning mechanisms, not automatic base-model compression. Keeping the frozen base matrix plus adapters adds parameters. Compression occurs only if the deployed representation replaces or restructures the base computation.

Combine methods in a deliberate order

Pruning, factorization, and distillation interact. A useful sequence is:

1. identify deployable structured shapes;
2. prune the teacher or initialize the student architecture;
3. reconstruct or factorize eligible matrices;
4. recover with hard labels and teacher targets;
5. apply quantization after the graph is stable;
6. benchmark the exported artifact.

This is not universal. Quantization-aware training may need to participate in recovery. Unstructured sparse kernels may require pruning to a hardware-specific pattern before export.

Change one compression axis at a time during diagnosis. If pruning, factorization, and quantization are applied together, a quality regression cannot be localized.

Preserve tied and shared parameters

Language models can tie input embeddings to the output projection or reuse parameters across layers. A generic traversal that replaces modules independently can break aliasing and duplicate storage.

Before compression, build an alias map:

owners = {}
for name, parameter in model.named_parameters(remove_duplicate=False):
    owners.setdefault(id(parameter), []).append(name)

shared = [names for names in owners.values() if len(names) > 1]

After rewriting the graph, confirm that intended aliases remain tied and the optimizer does not update duplicate copies.

Validate exported graphs

The in-memory training model is not the deployment artifact. Exporters can fold constants, transpose weights, reject sparse operators, or lower factorized layers into unexpected subgraphs.

For each target runtime:

• inspect operator types and tensor shapes;
• confirm sparse metadata survives;
• compare serialized size;
• run fixed input fixtures against the source model;
• measure peak memory and latency after warm-up;
• profile the kernels actually dispatched;
• test all supported batch and sequence buckets.

A compressed checkpoint that falls back to CPU for one unsupported sparse operator can be slower than the dense baseline.

Quality evaluation needs slices

Compression error is not uniform. Pruning can damage rare features. Distillation can suppress low-probability alternatives. Low-rank approximation can remove directions that matter for a small task slice.

Evaluate:

• perplexity or task loss on real held-out text;
• downstream accuracy by domain and language;
• long-context behavior;
• calibration and uncertainty;
• tool-call schema validity;
• safety and refusal behavior;
• generation repetition and degeneration.

Compare against an uncompressed checkpoint under identical decoding settings. If the compressed model needs a different sampler to recover quality, report both the model and sampler change.

Recovery failures

Mask drift: Pruned weights become nonzero during tuning because the optimizer updates them. Reapply the mask after updates or use masked parameterization.

Shape mismatch: Structured pruning updates weights but not configuration, rotary tables, or tensor-parallel metadata. Load the exported model in a fresh process before declaring success.

Teacher leakage: Evaluation prompts or their derivatives enter distillation data. Split source groups before teacher generation.

Rank collapse: A factorization uses one rank for every layer despite different spectra and sensitivity. Allocate rank by measured reconstruction and downstream loss.

No runtime gain: The artifact is smaller but latency is unchanged. Inspect kernels, launch count, memory bandwidth, and sparse support.

Compression is a contract between an algorithm and a runtime. Unstructured pruning needs sparse execution, structured pruning needs valid dense shapes, distillation needs independent evaluation, and factorization needs efficient paired operators. The final evidence is an exported model that meets memory, latency, and quality budgets on its target hardware.

References

Footnotes

1. SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. ↩

2. Compact Language Models via Pruning and Knowledge Distillation. ↩

3. Distilling the Knowledge in a Neural Network. ↩