Tokens, Context, Attention, and Inference

Four boundaries explain much of an LLM application's behavior: encoding, representation, finite context, and sequential generation.

Encoding is a system concern

Token boundaries affect cost, truncation, retrieval chunking, and multilingual behavior. Measure tokens using the deployed model's tokenizer rather than assuming words or characters.

Attention transforms representations

Embeddings supply vectors and attention mixes information across positions. Feed-forward blocks transform each position. Attention makes interaction possible; it does not guarantee that every supplied fact is used reliably.

Context is a budget

Instructions, requests, history, retrieved evidence, tool results, and output compete for a finite capacity. Select the smallest sufficient set, preserve output headroom, isolate untrusted text, and evaluate position and distractor effects.

Inference is sequential

At each step the model produces a token distribution and decoding selects one token. Output length affects latency directly. Temperature reshapes the distribution while top-p truncates it; neither control adds knowledge or factuality.

Engineering consequence

Separate problems by layer. Token overflow needs budgeting, missing current facts may need retrieval, malformed output needs schemas, and factual risk needs evidence and evaluation. A larger model is not the default diagnosis.

Decision checklist

1. Name the user outcome and unacceptable failures.
2. Identify the layer where the observed problem originates.
3. Choose the smallest mechanism that directly addresses that problem.
4. Define representative evaluation cases and operational budgets.
5. Preserve source, model, prompt, data, and release versions.
6. Require explicit approval before changing public behavior.