LLM Fundamentals

I would describe the Transformer as an encoder-decoder or decoder-only stack where every token can attend to every other token through self-attention. The attention mechanism computes queries, keys, and values from the input, then uses scaled dot-product attention to aggregate information across positions. Multi-head attention runs several attention copies in parallel so the model can learn different relationship types. Feed-forward layers add per-token nonlinear transformations, while residual connections and layer norm keep training stable. Positional encoding — either learned or sinusoidal — gives the model information about token order since attention itself is permutation invariant.

• Omitting positional encoding as a necessary component since attention alone has no inherent notion of order.
• Describing only the decoder half and ignoring the encoder when discussing the original Transformer.

I think of Q, K, V as a content-based lookup. The query represents what the current token is looking for, the key represents what each token offers, and the value is the information that will be passed along if a match is found. The attention score is the dot product of query and key, scaled by the inverse square root of the dimension to prevent vanishing gradients in the softmax. The weighted sum of values lets each token incorporate context from positions that have high attention scores, making the representation context-aware.

• Stating that queries and keys come from different sequences — in self-attention all three come from the same sequence.
• Skipping the scaling factor (√dₖ) and why it is needed for stable training.

Tokenisation bridges raw strings and model embeddings. BPE starts with individual bytes or characters as the base vocabulary, then counts adjacent token pairs in the corpus. The most frequent pair is merged into a new token, and the process repeats until the desired vocabulary size is reached. This lets the model handle any input via subword composition while keeping common words as single tokens. In practice, I verify that important domain terms are not split into meaningless pieces and consider adding domain-specific tokens to reduce sequence length and inference cost.

• Assuming the tokenizer splits on whitespace like traditional NLP, when BPE produces subword units that can break words arbitrarily.
• Ignoring tokenization during prompt engineering — a prompt that looks short in characters may consume many tokens if it contains rare subword sequences.

In autoregressive generation, each new token's attention step still needs keys and values from every prior position. Without KV cache, the model would recompute those tensors for every prefix position on every step. By caching them in memory after the first computation, each subsequent step only computes Q, K, V for the new token, then uses the full KV from the cache for attention. The trade-off is increased memory usage — the cache grows linearly with sequence length — which is why techniques like Paged Attention, sliding-window cache, or quantised caching are important for long sequences.

• Assuming KV cache is lossless and always applicable — it consumes significant GPU memory for long sequences.
• Forgetting to clear the cache between independent generation requests, which leaks context across users.

Temperature divides logits by the temperature value before applying softmax. At low temperatures the probability mass concentrates on the top token, making output deterministic and repetitive. At high temperatures the distribution becomes more uniform, increasing diversity but risking incoherence. Top-k sets a fixed cutoff so long-tail tokens never get sampled, while top-p adapts the cutoff based on the distribution shape — narrow distributions keep fewer candidates, broad distributions keep more. In production I use top-p with a moderate temperature and tune the combination against the specific task, measuring both quality and diversity metrics.

• Setting temperature to zero thinking it guarantees determinism — floating-point non-determinism across GPUs can still cause variation.
• Applying top-k with a small k value without considering that some distributions may have many plausible tokens above that cutoff.