Your LLM isn't thinking. It's guessing. Repeatedly.

Here's the thing that broke my brain when I finally understood it: the model doesn't know what it's going to say. Not even one token ahead. It picks the next word, then forgets it made a choice, then picks the next word again, completely from scratch. Every single time.

That's it. That's the whole trick. A 175-billion-parameter system that shook the world is, at its core, a very expensive next-word guesser running in a loop.

The token loop nobody draws correctly

When you send a prompt to GPT-4, it doesn't read your whole message and then think for a while. It reads your message, outputs one token, appends that token to the input, reads the whole thing again, outputs the next token, and so on. The input grows by one token on every single forward pass.

This is why long generations are slow. Not because the model is thinking harder. Because the sequence it has to process is literally getting longer every step.

Each pass through the full transformer stack costs compute proportional to sequence length squared. That quadratic scaling is the reason context windows were stuck at 4k tokens for years and why extending them to 128k required some genuinely clever engineering — both in positional encoding (RoPE, ALiBi) and KV caching.

What attention is actually doing

Forget the heatmap visualizations. Here's what attention means in plain language: for every token the model is trying to predict, it gets to look back at every previous token and decide how much each one matters right now.

"Paris" matters a lot when the model is trying to complete "The capital of France is ___". "France" is the most relevant prior token. Attention is the mechanism that lets the model say that explicitly — it's a learned routing system that figures out which earlier context is relevant to the current prediction.

The "multi-head" part just means this happens in parallel with different learned notions of relevance. One head might track grammatical agreement. Another might track entity references. Another might track whether we're inside a code block or prose. Nobody explicitly programmed this — it emerged from training.

The war story that made this real for me

In 2023 I was contracting with a Series B startup running about 2M completions per day through the OpenAI API. We were spending $60k/month and the founding team wanted to cut costs by switching to a self-hosted Llama 2 70B on 4x A100s. The math looked great on paper.

What we didn't account for: the KV cache. At our average prompt length of ~3k tokens, the KV cache for a single 70B inference was eating 18GB of VRAM. With batching turned on to improve throughput, we were constantly blowing past memory limits and the server was silently degrading to batch size 1. Our "4x cheaper" setup was running at 15% the throughput we modeled. We found it in the vLLM logs — specifically the gpu_cache_usage_perc metric hitting 1.0 and staying there.

The fix was PagedAttention, which is the core innovation in vLLM. Instead of allocating a contiguous KV cache block per sequence, it pages the cache like virtual memory in an OS. Sequences share physical memory blocks. GPU utilization went from 34% to 81% overnight. That story is all in the vLLM paper (Kwon et al., 2023) and the implementation is readable in vllm/core/block_manager.py if you want the full picture.

Temperature, top-p, and why your prompts feel nondeterministic

After the model runs a forward pass, it has a probability distribution over its entire vocabulary — roughly 50k tokens for GPT-style models. Temperature scales that distribution before sampling. Low temperature (0.1) makes the distribution spiky — the highest-probability token almost always wins. High temperature (1.5) flattens it — weird tokens get a real shot.

Top-p (nucleus sampling) truncates the distribution to the smallest set of tokens whose cumulative probability exceeds p. At p=0.9, the model ignores everything outside the top 90% of probability mass before sampling. This is why your outputs vary even at temp=1.0 — you're sampling, not argmax-ing.

Setting temp=0 forces greedy decoding: always pick the highest-probability token. Deterministic output, but often repetitive and boring. The classic trap is setting temp=0 for production because you want consistent outputs, then wondering why the model keeps getting stuck in loops.

What I'd do differently

Most people learn transformers top-down: attention mechanism, then training, then inference. I'd teach it bottom-up starting from the inference loop. The "guess one token, loop" mental model unlocks everything else — why context length is expensive, why KV caching matters, why you can't just ask the model to reconsider its last word. Get that loop in your head first and the rest clicks into place fast.