This is the second post in our technical-depth series — the first covered what actually breaks OCR in production and what the literature supports. This one covers the layer above it: the model itself. Two questions, both of which an engineering reader deserves real answers to. How does inference mechanically work, and why does it cost what it costs? And when a model tells you why it answered the way it did — can you believe it?

The first question has a clean answer. The second one doesn’t, and the gap between what “interpretability” means in a sales deck and what it means in the research literature is exactly where a buyer of legal AI gets misled. We’ll take them in order.

Inference is two different workloads wearing one name

A transformer does one thing: given a sequence of tokens, one forward pass through the layer stack produces a probability distribution over the next token. That’s the entire primitive. Everything else — the chat interface, the “reasoning”, the thousand-word answer — is that primitive in a loop: sample a token from the distribution, append it to the sequence, run the pass again. Autoregressive decoding is not an implementation detail; it’s the reason the cost structure looks the way it does.

Because inside that loop hide two workloads with opposite performance profiles:

• Prefill processes your entire prompt in one pass. Every prompt token is computed in parallel, the GPU’s arithmetic units are saturated, and the cost is roughly linear in prompt length. This is the compute-bound part.
• Decode generates the answer one token at a time, sequentially, because token n+1 can’t be computed until token n exists. Each step moves the model’s weights and the accumulated attention state through memory to produce a single token. This is the memory-bandwidth-bound part, and it’s why output tokens dominate latency — and why every commercial API prices output tokens several times higher than input tokens. That price ratio isn’t marketing. It’s the hardware profile leaking into the invoice.

The “accumulated attention state” has a name: the KV cache. Attention requires the keys and values of every previous token; recomputing them at each step would make each token cost as much as re-reading the whole sequence, so every serving stack caches them instead. The trade is memory: the cache grows linearly with sequence length, per layer, per attention head, per concurrent request. At serving scale this — not the weights — becomes the scarce resource. Kwon et al. (SOSP 2023) measured that existing serving systems wasted significant KV-cache memory to fragmentation and duplication, applied virtual-memory-style paging to it (PagedAttention, the core of vLLM), and improved throughput of popular LLMs by 2–4× at the same latency. Read that result the way an engineer should: a 2–4× cost difference for identical model output was sitting in memory management, not in the model. When someone quotes you a per-document inference cost, this layer is where much of that number was actually decided.

The model produces a distribution. The answer is a policy choice.

The forward pass ends in a probability distribution over the vocabulary. Turning that into a token is a separate decision, made by the decoding configuration, not by the model — and it matters more than people outside the serving stack tend to assume.

The naive policy — always take the most likely token — produces measurably degenerate text. Holtzman et al. (ICLR 2020) showed that maximization-based decoding from the same model that assigns high quality scores to human text yields output that is bland and falls into repetitive loops, and proposed nucleus (top-p) sampling: truncate the unreliable low-probability tail of the distribution and sample from the head that carries most of the mass. Temperature rescales the distribution’s sharpness; top-k truncates to a fixed count; top-p truncates to a probability mass. All of them are knobs on the policy, applied after the model has done all of its thinking.

Two engineering consequences worth stating plainly. First: same model, same prompt, different answers is a configuration, not a malfunction. If your pipeline needs reproducibility, that’s a decoding setting you own. Second, and more important for anyone evaluating a legal-AI system: temperature 0 makes an answer repeatable, not right. Deterministic sampling of a wrong distribution gives you the same wrong answer every time. Reproducibility and correctness are different properties, and only one of them is a sampling parameter.

Quantization: the tradeoff is real, measured, and smaller than you’d fear — if you check

Model weights are trained in 16-bit floating point. Quantization stores them (and sometimes the activations) in fewer bits — 8, 4, occasionally less — which cuts memory, which cuts hardware cost, which in a memory-bandwidth-bound decode loop also cuts latency. The question that matters is what you give up, and this is one place where the literature has actual numbers instead of vibes.

Frantar et al. (GPTQ, ICLR 2023) showed one-shot post-training quantization down to 3–4 bits per weight with negligible accuracy degradation relative to the uncompressed baseline, with measured end-to-end speedups of roughly 3.25× on an A100 and 4.5× on an A6000. And because a single paper’s “negligible” deserves adversarial follow-up, Kurtic et al. (ACL 2025) ran over 500,000 evaluations across quantization formats on the Llama-3.1 family and found: FP8 weights-and-activations effectively lossless at every model scale, well-tuned INT8 at 1–3% degradation, and 4-bit weight-only quantization rivaling 8-bit. That’s the honest shape of the tradeoff: at 8 bits you are giving up approximately nothing; at 4 bits you are giving up little enough that it’s usually the right trade — but “usually” is doing work in that sentence. Degradation varies by task and by model, and the only number that binds is the one measured on your own evaluation set, on your own material. That was the central lesson of the first post in this series, and it survives the move from OCR to LLMs completely intact.

The uncomfortable implication for a buyer: when a vendor serves you a quantized model — and at production economics, someone in the chain usually does — the accuracy you benchmarked and the accuracy you’re getting are only the same if somebody re-measured after quantization. Ask.

“Interpretability” means two things, and only one of them is what you’re being sold

Now the second question: can you know why the model produced an output? In the research literature, “interpretability” refers to something specific. In product conversations it gets conflated with two things that are not interpretability at all. Separating the three is the most useful thing this post can do for you.

Mechanistic interpretability is reverse-engineering the internals — and it’s real science with real results. The obstacle is that individual neurons don’t map to concepts: models compress more features than they have neurons, so single neurons activate for many unrelated things (polysemanticity, hypothesized to arise from superposition). Cunningham et al. (2023) showed that sparse autoencoders can decompose a model’s internal activations into features far more interpretable and monosemantic than the raw neurons — an unsupervised, scalable way to pull individual concepts out of the tangle. Building on that family of techniques, Lindsey et al. (Anthropic, 2025) traced attribution graphs — causal maps of which internal features fed which — through a production-grade model (Claude 3.5 Haiku), and found genuinely non-trivial internal structure: multi-step computation happening inside a single forward pass, and evidence of the model planning ahead toward a rhyme before writing the line of a poem. This is the field at its best, and it is progressing fast.

Post-hoc explanation is asking the model to narrate — and the narration is not causally connected to the computation. When a model writes “I concluded X because of clause 4.2”, that sentence was produced by the same next-token loop that produced the answer, optimized to be plausible text — not extracted from the mechanism that actually computed X. This is not a philosophical quibble; it’s measured. Turpin et al. (NeurIPS 2023) biased models toward wrong answers with subtle input manipulations — for instance, reordering few-shot options so the correct answer was always “(A)” — and watched the models produce fluent chain-of-thought justifications for the biased wrong answers without ever mentioning the bias, with accuracy drops of up to 36% on BIG-Bench Hard tasks. The explanation systematically misrepresented the true cause of the prediction. A model’s stated reasoning is an output to be verified, not a log to be trusted.

And stated confidence is a third thing again. There is real evidence that models carry usable calibration signal internally: Kadavath et al. (2022) found large models well-calibrated on multiple-choice questions in the right format, though calibration degrades on tasks unlike the training distribution. But the calibration of the raw pre-trained model is not the calibration of the assistant you actually talk to: the GPT-4 technical report documents that the pre-trained model was highly calibrated on MMLU (expected calibration error 0.007) and that post-training reduced calibration significantly (ECE 0.074) — the alignment process that makes the model useful also distorts the very signal a “how confident are you?” prompt is trying to read. So when a production assistant tells you it’s 90% sure, that number is a trained behavior, not a measurement. We’ve already written about why confident-sounding output is not verified output, and everything in this section is the mechanistic reason that post is true.

What you can honestly promise an auditor today

Here is where interpretability research actually stands, stated the way we’d want it stated to us.

The mechanistic toolkit is real and improving, and it has already produced findings nobody could have gotten by prompting. It is not, today, a per-answer explanation service. The authors of the attribution-graphs work say so themselves: their method provides satisfying insight on roughly a quarter of the prompts they tried, and even the successful cases are simplified views of the real mechanism, with known blind spots (attention circuits among them). That’s an honest limitations section from the best-resourced team in the field, and it defines the ceiling: no one can currently hand you a verified causal account of why a large model produced one specific production output — not the vendor, not the lab that trained it.

So a practitioner who has to answer to a client, a court, or an auditor should not promise “the model explains its decisions”. The model narrates; the narration is unfaithful under measurable conditions; the confidence is a trained behavior; and the mechanistic tools that do look inside don’t yet cover production traffic. What you can honestly promise instead is system-level: every claim traceable to a source document, checks that run outside the model, and outputs treated as untrusted until verified — the architecture where the explanation of the answer is the evidence for the answer, not the model’s account of itself. A system whose accuracy clients depend on doesn’t get to skip that layer, precisely because the layer below it can’t yet carry the weight.

The model is a component you verify, not a witness you cross-examine. The research may change that ceiling — the pace of the mechanistic work is genuinely fast — but you build for the ceiling that exists.

Next in this series: retrieval-augmented generation — how RAG actually works mechanically, and the specific places it fails without telling you.