Beyond the Prompt: The LLM Mechanics Architects Actually Need

Prompt engineering is UX for a function you don't control. It matters, but it's the surface. To architect with an LLM rather than merely talk to one, you need the mechanics underneath: how text becomes tokens, why the context window is a budget and not a memory, and why the same prompt can give you a different answer every time.

The context window is a byte budget, not a memory

The most expensive misconception in AI engineering is treating the context window like RAM the model "remembers." It is a fixed-width budget you allocate every single call: system instructions, retrieved context, conversation history, and the question all compete for the same finite token count. Overflow doesn't error — it silently truncates, and the thing that gets dropped is usually the context you most needed.

Determinism is a parameter, not a property

A model doesn't "decide" an answer — it produces a probability distribution over the next token, then samples from it. Temperature reshapes that distribution: low temperature sharpens it toward the single most likely token (near-deterministic), high temperature flattens it (more variance). When stakeholders ask why the system gave two different answers to the same question, the honest answer is: because you asked it to roll dice.

// determinism is a knob you set, not a trait you assume
const answer = await model.generate({
  prompt,
  temperature: 0,   // sharpen toward the argmax token
  top_p: 1,         // (no nucleus truncation needed at temp 0)
  max_tokens: 512,  // reserve budget for the ANSWER, not just input
});

Tokenization is where the bugs live

Cost, latency, and truncation are all denominated in tokens, not characters — and tokens don't map cleanly to words. A JSON blob, a non-English language, or a long ID can cost far more tokens than its length suggests. Budgeting in characters is how you get a payload that fits in your head but overflows the window in production. Measure in tokens, reserve headroom for the output, and treat the count as a first-class constraint.

You can't architect a system whose core you model as magic. Tokens, budgets, and sampling are the three mechanics that turn "prompting" into engineering.

These mechanics underwrite everything downstream: the token budget is why retrieval has to be selective, and sampling variance is why guardrails verify outputs instead of trusting them. Full series on the roadmap.