Most developers use LLMs like magic boxes. They copy-paste prompts from Twitter and pray it works.You’ll be different. Understanding how LLMs work lets you:
Debug when outputs are wrong
Optimize costs (save 10x on API bills)
Design prompts that actually work
Know when to use which model
Real Talk: Companies waste thousands on AI because developers don’t understand token economics. After this module, you won’t.
LLMs are next-token predictors. That’s it. Everything else is a consequence of this simple idea.
Input: "The capital of France is"Model thinks: What token is most likely next?Output: " Paris" (with high probability)
They don’t “know” facts. They predict what text is likely to follow based on patterns in training data.
This explains hallucinations: If “The CEO of Apple is Steve Jobs” appeared often in training data, the model might predict “Steve Jobs” even though Tim Cook is the current CEO. It predicts likely text, not true text.
Pro Tip: Start with gpt-4o-mini for development and testing. It’s 15x cheaper than gpt-4o and fast enough to iterate quickly. Switch to a more capable model only when needed.
Notice how the relationship direction is consistent:
King - Man + Woman ≈ QueenThis vector arithmetic allows the model to understand analogies and relationships.
from openai import OpenAIimport numpy as npfrom typing import Listimport jsonclient = OpenAI()class EmbeddingCache: """Cache embeddings to avoid repeated API calls""" def __init__(self, cache_file: str = "embeddings_cache.json"): self.cache_file = cache_file self.cache = self._load_cache() def _load_cache(self) -> dict: try: with open(self.cache_file, 'r') as f: return json.load(f) except FileNotFoundError: return {} def _save_cache(self): with open(self.cache_file, 'w') as f: json.dump(self.cache, f) def get_embedding(self, text: str, model: str = "text-embedding-3-small") -> List[float]: cache_key = f"{model}:{text[:100]}" # Use prefix for key if cache_key in self.cache: return self.cache[cache_key] response = client.embeddings.create(model=model, input=text) embedding = response.data[0].embedding self.cache[cache_key] = embedding self._save_cache() return embedding def get_embeddings_batch(self, texts: List[str], model: str = "text-embedding-3-small") -> List[List[float]]: """Batch embed for efficiency (up to 2048 texts per call)""" # Check cache first uncached = [(i, t) for i, t in enumerate(texts) if f"{model}:{t[:100]}" not in self.cache] if uncached: indices, uncached_texts = zip(*uncached) response = client.embeddings.create(model=model, input=list(uncached_texts)) for i, emb_data in zip(indices, response.data): cache_key = f"{model}:{texts[i][:100]}" self.cache[cache_key] = emb_data.embedding self._save_cache() return [self.cache[f"{model}:{t[:100]}"] for t in texts]def cosine_similarity(a: List[float], b: List[float]) -> float: """Compute cosine similarity between two vectors""" # Cosine similarity measures the angle between two vectors # Returns 1.0 for identical, 0.0 for unrelated, -1.0 for opposite a, b = np.array(a), np.array(b) return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))def find_most_similar(query: str, documents: List[str], top_k: int = 3) -> List[tuple]: """Find most similar documents to query""" cache = EmbeddingCache() query_emb = cache.get_embedding(query) doc_embs = cache.get_embeddings_batch(documents) similarities = [ (doc, cosine_similarity(query_emb, doc_emb)) for doc, doc_emb in zip(documents, doc_embs) ] return sorted(similarities, key=lambda x: x[1], reverse=True)[:top_k]# Example usagedocuments = [ "Python is a programming language", "JavaScript runs in browsers", "Machine learning uses neural networks", "Snakes are reptiles that slither", "The stock market closed higher today"]results = find_most_similar("coding languages", documents)for doc, score in results: print(f"{score:.3f}: {doc}")# 0.847: Python is a programming language# 0.812: JavaScript runs in browsers# 0.623: Machine learning uses neural networks
Why cache embeddings? Each embedding API call costs money and takes time. Since the same text always produces the same embedding, caching can save you 90%+ on costs for repeated queries.
When to use embeddings vs. keyword search:
Use embeddings: When meaning matters (“cheap flights” = “affordable airfare”)
Use keywords: When exact matches matter (error codes, product SKUs)
Remember: LLMs predict the most likely next token. But if they always picked the #1 choice, outputs would be boring and repetitive.Sampling means randomly choosing from the top candidates based on their probabilities. This adds variety while still favoring likely tokens.Think of it like a weighted lottery:
“Paris” has 85 tickets
“Lyon” has 5 tickets
“a” has 3 tickets
“the” has 2 tickets
Most of the time you’ll draw “Paris”, but occasionally you’ll get something else.
When the model predicts the next token, it outputs probabilities for all possible tokens:
"The capital of France is" → " Paris": 0.85 " Lyon": 0.05 " a": 0.03 " the": 0.02 ...
Temperature controls how these probabilities are used. Think of it as a “creativity dial”:
Low temperature (0-0.3): Conservative, picks the most likely tokens → Predictable output
Medium temperature (0.7-1.0): Balanced, some variety → Natural conversation
High temperature (1.5+): Wild, picks unlikely tokens → Creative but potentially nonsensical
What are logits? Raw scores before converting to probabilities. Temperature scales these scores before the conversion.
import numpy as npdef sample_with_temperature(logits: np.ndarray, temperature: float) -> int: """Demonstrate temperature sampling""" # Apply temperature (lower = more confident, higher = more random) scaled_logits = logits / temperature # Convert to probabilities probs = np.exp(scaled_logits) / np.sum(np.exp(scaled_logits)) # Sample return np.random.choice(len(probs), p=probs)# Temperature 0: Always pick highest probability (deterministic)# Temperature 0.5: Slightly random, but still favors likely tokens# Temperature 1.0: Sample according to original probabilities# Temperature 2.0: More random, even unlikely tokens have a chance
Imagine reading a long document and trying to remember every single word equally. Impossible, right? You naturally focus on important parts and skim over others.LLMs had the same problem. Early models (RNNs) processed text sequentially, treating all words equally. They struggled with:
Long-range dependencies (“The cat, which was sitting on the mat that my grandmother gave me, was hungry”)
Understanding which words relate to each other
Attention lets the model decide which words to focus on when processing each token. It’s like highlighting the important parts of a document.Attention is what makes Transformers special. It allows the model to focus on different parts of the input sentence when processing a specific word.Consider the sentence:
“The animal didn’t cross the street because it was too tired.”
To understand what “it” refers to, the model must pay attention to “animal” and ignore “street”.Without attention, the model wouldn’t know if “it” referred to the animal or the street, making translation and comprehension impossible.
Quadratic complexity means the computational cost grows with the square of the input length.
# GPT-4o has 128K context ≈ 96K words ≈ 300 pages# But attention has quadratic complexity: O(n²)# This means every token must "look at" every other token# 128K tokens = 128K × 128K = 16 billion attention computations# This is why:# 1. Long contexts are slower# 2. Long contexts cost more# 3. Information at the start/end is remembered better than middle
Research finding (Liu et al., 2023): LLMs are best at recalling information at the start and end of their context window. Information in the middle gets “lost”.This is similar to human memory - you remember the first and last items in a list better than the middle ones.Practical implication: When feeding multiple documents to an LLM, put the most important ones at the beginning and end:
def structure_for_attention(docs: list[str], query: str) -> str: """Structure documents to avoid 'lost in the middle' problem""" # Put most relevant docs at START and END # Put less relevant docs in MIDDLE # Note: rank_by_relevance would use embeddings to score relevance # (implementation depends on your use case) ranked = rank_by_relevance(docs, query) # Interleave: most relevant at edges n = len(ranked) reordered = [] for i in range(n): if i % 2 == 0: reordered.insert(0, ranked[i]) # Add to start else: reordered.append(ranked[i]) # Add to end return "\n\n".join(reordered)
You wouldn’t send an email that just says “help” and expect a useful response. Same with LLMs - structure and context dramatically improve output quality.Bad prompt: “Write a blog post about AI”
Good prompt: Specific context, clear objective, defined formatWhy structured prompts work better:
Reduces ambiguity - LLM doesn’t have to guess what you want
Provides context - Like briefing a colleague before asking for help
Sets expectations - Defines tone, style, and format upfront
Improves consistency - Same structure = similar quality outputs
def build_prompt_costar( context: str, objective: str, style: str, tone: str, audience: str, response_format: str) -> str: """ COSTAR framework for structured prompts C - Context: Background information O - Objective: What you want to achieve S - Style: Writing style (formal, casual, technical) T - Tone: Emotional tone (professional, friendly) A - Audience: Who will read this R - Response: Format of output """ return f"""# Context{context}# Objective{objective}# StyleWrite in a {style} style.# ToneMaintain a {tone} tone.# AudienceThis is for {audience}.# Response Format{response_format}"""# Exampleprompt = build_prompt_costar( context="We're launching a new AI code review tool for developers.", objective="Write a product announcement for our blog.", style="technical but accessible", tone="excited but professional", audience="software developers who use GitHub", response_format="Blog post with headline, 3-4 paragraphs, and a call to action.")
Research finding (Wei et al., 2022): LLMs perform significantly better on complex tasks when asked to “think step by step”.Why it works: Breaking down reasoning into steps helps the model:
Avoid jumping to conclusions
Show its work (so you can debug)
Handle multi-step logic
Reduce errors on math and reasoning tasks
When to use CoT:
Math problems
Logic puzzles
Multi-step reasoning
Code debugging
Planning tasks
def cot_prompt(question: str) -> str: """Force step-by-step reasoning""" return f"""{question}Let's solve this step by step:1. First, I'll identify what we know2. Then, I'll figure out what we need to find3. Next, I'll work through the logic4. Finally, I'll state the answerStep 1:"""# CoT is especially powerful for:# - Math problems# - Logic puzzles# - Multi-step reasoning# - Code debugging
The problem: APIs rate-limit you (e.g., 3,500 requests/minute for GPT-4). If you hit the limit, requests fail.The solution: Wait and retry, with increasing delays (1s, 2s, 4s, 8s…). This gives the rate limit time to reset.
The problem: If 100 users ask “What is Python?”, you’re paying for 100 identical API calls.The solution: Cache responses by request. First user pays, next 99 get instant free responses.When to cache:
Explain how the attention mechanism works in Transformers and why it replaced RNNs.
Strong Answer:
The core idea behind attention is that when processing any given token, the model should be able to “look at” every other token in the input and decide how much weight to assign each one. In a Transformer, we compute this through three learned projections — Query, Key, and Value matrices. Each token’s Query is dot-producted against every other token’s Key to produce an attention score, then those scores are normalized via softmax and used to create a weighted sum of the Values. That weighted sum becomes the representation of that token.
RNNs processed text sequentially, left-to-right, compressing everything into a fixed-size hidden state. This meant that by the time the model reached token 500, it had largely forgotten token 10. Attention removes that bottleneck entirely — token 500 can directly attend to token 10 with a single matrix multiplication, no sequential chain required.
The trade-off is computational complexity. Self-attention is O(n^2) with respect to sequence length because every token attends to every other token. For a 128K context window, that is 128K times 128K = roughly 16 billion attention computations per layer. This is why long-context models are slower and more expensive, and why approaches like FlashAttention, sliding-window attention, and ring attention exist to make this tractable.
In practice, this also explains the “lost in the middle” phenomenon. While attention theoretically allows equal access to all positions, the learned attention patterns tend to favor tokens near the beginning and end of the context. If you are building a RAG pipeline, this means you should place your most relevant retrieved chunks at the start and end of the prompt, not buried in the middle.
Follow-up: You mentioned O(n^2) complexity. What are the practical approaches used to bring this down in production systems?There are several families of solutions. FlashAttention is probably the most impactful — it does not change the mathematical computation but reorganizes the memory access patterns to be IO-aware, reducing the number of reads and writes to GPU high-bandwidth memory by tiling the attention computation. This gives you a 2-4x speedup without any approximation loss. Sliding-window attention, used in models like Mistral, limits each token to attending only to a local window of, say, 4096 tokens, which brings complexity down to O(n times w) where w is the window size. Multi-query attention and grouped-query attention reduce the number of Key-Value heads, which shrinks KV cache memory and speeds up inference. For extremely long contexts like Gemini’s 1M token window, architectures combine these techniques with ring attention that distributes the sequence across multiple devices. The key insight is that most of these are not approximations — they compute the exact same attention but with better hardware utilization.
A client is seeing unexpectedly high API costs. Walk me through how you would diagnose and reduce their spend.
Strong Answer:
First thing I would do is instrument their pipeline to log token counts per request, broken down by input tokens and output tokens. Most cost blowups come from the input side — someone is stuffing their entire database into the context on every call. I have seen a RAG system where the retrieval step was returning 50 chunks of 500 tokens each, consuming 25K input tokens per query. Cutting that to the top 5 most relevant chunks dropped costs by 80% with no quality degradation.
Second, I would audit their model selection. A common pattern is using gpt-4o for everything including simple classification or extraction tasks that gpt-4o-mini handles just as well. The pricing difference is roughly 17x on input and 17x on output. In one project I worked on, we routed simple intent-classification queries to gpt-4o-mini and only escalated complex reasoning tasks to gpt-4o, which cut the monthly bill from around 8,000to1,200.
Third, response caching. If the same question or a semantically similar question gets asked frequently, cache the response. Semantic caching with an embedding similarity threshold of 0.95 or higher can safely serve cached results for about 30-40% of traffic in many customer-support-style applications.
Fourth, I would look at output token waste. Are they asking for verbose explanations when a JSON object with three fields would suffice? Setting max_tokens appropriately and using structured output formats can cut output tokens by 50-70%.
Finally, check for non-English text. Arabic, Chinese, Japanese, and Korean tokenize at roughly 2-4x the token count of English for the same semantic content. If your user base is multilingual, this can be a hidden cost multiplier.
Follow-up: How would you implement a model routing system that automatically picks the cheapest model capable of handling each request?The approach I would take is a lightweight classifier that runs before the main LLM call. You train it on a few hundred labeled examples where each example is a user query tagged with the minimum model tier that produces acceptable quality — “simple” for gpt-4o-mini, “standard” for gpt-4o, “complex” for o1. The classifier itself can be a small fine-tuned model or even a rule-based system using query length, presence of code blocks, and explicit reasoning keywords. You route based on the classifier output. The key is to measure quality continuously — you run a sample of “downgraded” queries through the higher-tier model in shadow mode and compare outputs. If the quality delta exceeds your threshold, you adjust the routing rules. This is essentially an A/B test running continuously, where the “treatment” is using a cheaper model.
What is the 'lost in the middle' problem, and how does it affect real-world LLM application design?
Strong Answer:
The “lost in the middle” finding, from Liu et al. in 2023, showed that LLMs have a U-shaped recall curve across their context window. They are best at remembering and using information placed at the very beginning and very end of the context, while information buried in the middle gets significantly less attention. In their experiments, accuracy dropped by as much as 20% for information placed in the middle versus the edges.
This has direct implications for RAG pipeline design. If you retrieve 10 document chunks and concatenate them in arbitrary order, the chunks in positions 4-7 are essentially second-class citizens. The practical fix is to re-rank your chunks by relevance and then interleave them: place the most relevant at position 1, the second-most relevant at the last position, the third-most relevant at position 2, and so on. This “edges-first” ordering consistently improves answer quality in my experience.
It also affects system prompt design. If your system prompt is very long — say 2,000 tokens of instructions — the instructions in the middle tend to be followed less reliably. I have seen teams restructure their system prompts to put the most critical behavioral instructions at the very beginning and repeat the single most important constraint at the very end, right before the user message.
The deeper reason this happens is related to how positional encodings and attention patterns are learned during training. The model sees the beginning of sequences and the most recent tokens very frequently, so attention heads learn strong patterns for those positions. The middle positions are more variable during training and thus get weaker learned attention.
Follow-up: If you had to design a system that processes 100-page legal contracts with an LLM, how would you handle the lost-in-the-middle problem at that scale?At that scale, you cannot fit the entire contract in context even with a 128K window, and even if you could, the middle sections would be unreliable. The architecture I would use is a two-stage pipeline. Stage one: chunk the contract into semantically meaningful sections — by clause, by article, by heading — and create embeddings for each chunk. Stage two: for any user question, retrieve the top 5-8 most relevant chunks via hybrid search (keyword plus semantic), then arrange those chunks with the edges-first interleaving pattern I described. I would also add a “citation check” step where the model must quote the exact contract language supporting its answer, which acts as a built-in hallucination detector. For questions that require understanding the contract holistically — like “are there any contradictions between sections?” — you would need a map-reduce approach: summarize each section independently, then have the LLM reason over the summaries. It is more expensive but the only reliable approach for cross-document reasoning at that length.
Compare temperature, top-p, and top-k sampling. When would you use each, and what are the failure modes?
Strong Answer:
Temperature scales the logits before the softmax operation. A temperature of 0 makes the distribution extremely peaked — essentially greedy decoding where the highest-probability token always wins. A temperature of 1.0 preserves the original distribution. Above 1.0, the distribution flattens and unlikely tokens get a meaningful chance of being selected. The failure mode of high temperature is incoherent output — at temperature 2.0, the model can produce grammatically valid but semantically nonsensical text because it is sampling from the tail of the distribution too often.
Top-p (nucleus sampling) takes a different approach: instead of scaling probabilities, it dynamically selects the smallest set of tokens whose cumulative probability exceeds p. So top-p of 0.9 means “consider only the tokens that make up the top 90% of the probability mass.” This adapts naturally to the model’s confidence — when the model is very certain (one token has 95% probability), top-p 0.9 gives you essentially just that one token. When the model is uncertain (many tokens with similar probabilities), top-p 0.9 gives you a wider selection. The failure mode is that very low top-p values (like 0.1) can create repetitive, degenerate text because you are restricting the vocabulary too aggressively.
Top-k is the simplest: only consider the top k tokens by probability, regardless of their actual probability values. Top-k of 50 means “pick from the 50 most likely tokens.” The weakness is that it is not adaptive. Sometimes the top 50 tokens are all reasonable continuations, sometimes only 3 are. Top-k treats both situations identically.
In production, I typically use temperature 0 for any deterministic task — code generation, structured extraction, classification. For conversational applications, I use temperature 0.7 with top-p 0.9, which gives natural variation without risking incoherence. I almost never use top-k alone because top-p is strictly more adaptive. The one exception is when combining top-k with temperature for creative applications — top-k 100 with temperature 1.2 gives wide but not unbounded creativity.
Follow-up: You set temperature to 0 for deterministic tasks, but OpenAI’s API documentation says temperature 0 is not truly deterministic. Why, and how do you handle this in production?Correct — even at temperature 0, you can get different outputs for the same input across API calls. This happens because of floating-point non-determinism in GPU computations, model serving infrastructure that may route to different GPU instances with slightly different numerical behavior, and potential model version updates behind the API. OpenAI introduced the seed parameter to address this — when you set both temperature 0 and a fixed seed, you get a system_fingerprint in the response that tells you which model version served the request. If the fingerprint changes, your output may differ. In production, when I need true reproducibility — for example, in a test suite or an audit log — I use temperature 0, a fixed seed, and I cache the response keyed on the input plus the system fingerprint. If the fingerprint changes, I invalidate the cache and accept that the output may have shifted. For compliance-sensitive applications, I also log the full request and response so that any output can be reproduced or at least explained.
