Scaling laws, emergent capabilities, and training large language models
Foundation Models & LLMs
The Foundation Model Paradigm
Foundation models are large models trained on broad data that can be adapted to many downstream tasks. The analogy is a well-rounded liberal arts education: rather than training a specialist from scratch for every task, you invest heavily in a broad base of knowledge, then specialize with minimal additional effort. GPT-3 was trained once at enormous cost, but it can be adapted to summarization, translation, code generation, and thousands of other tasks with just a few examples — or even zero examples.The term “foundation model” was coined by Stanford’s HAI in 2021, and it captures a paradigm shift: instead of building task-specific models from scratch, you build on top of a pre-existing “foundation” of learned knowledge.Key characteristics:
Scale — billions of parameters, trained on trillions of tokens
Self-supervised pretraining — learns from raw text without human annotation
Emergent capabilities — abilities that appear only at sufficient scale, not explicitly trained
Transfer to diverse tasks — one model, many applications via prompting or fine-tuning
Scaling Laws
The Chinchilla Scaling Law
DeepMind’s Chinchilla paper (2022) fundamentally changed how the industry thinks about training LLMs. The key finding: most models before Chinchilla were dramatically undertrained — they were too large for the amount of data they saw.For compute-optimal training:Nopt∝C0.5,Dopt∝C0.5Where:
N = number of parameters
D = dataset size (tokens)
C = compute budget (FLOPs)
Rule of thumb: Train on ~20 tokens per parameter. This means a 7B parameter model should see roughly 140B tokens for compute-optimal training.
Why this matters in practice: Before Chinchilla, the trend was “bigger model = better.” Chinchilla showed that a 70B model trained on 1.4T tokens outperformed the 280B Gopher model trained on 300B tokens — at much lower inference cost. This shifted the industry toward smaller, better-trained models, and directly influenced LLaMA, Mistral, and other efficient model families.
Model
Parameters
Training Tokens
Ratio
GPT-3
175B
300B
1.7
Chinchilla
70B
1.4T
20
LLaMA 2
70B
2T
29
Mistral
7B
Unknown
-
LLM Architecture
Modern Transformer Improvements
Modern LLMs use several key improvements over the original 2017 Transformer. Each one addresses a specific limitation discovered through years of scaling experiments. Understanding these is essential for reading current research papers and building production systems.
import torchimport torch.nn as nnimport torch.nn.functional as Fclass ModernTransformerBlock(nn.Module): """LLaMA-style transformer block with modern improvements. Three key changes from the original Transformer: 1. Pre-normalization (RMSNorm before attention/MLP, not after) 2. Grouped Query Attention (fewer KV heads = faster inference) 3. SwiGLU activation (empirically better than ReLU/GELU) """ def __init__(self, dim, num_heads, mlp_ratio=4, dropout=0.0): super().__init__() # Pre-normalization with RMSNorm (simpler and faster than LayerNorm) self.norm1 = RMSNorm(dim) self.norm2 = RMSNorm(dim) # Grouped Query Attention: use 4x fewer KV heads than query heads # This dramatically reduces KV cache memory during inference self.attn = GroupedQueryAttention(dim, num_heads, num_kv_heads=num_heads // 4) # SwiGLU MLP: ~1% better than GELU across most benchmarks # The 2/3 factor compensates for SwiGLU having 3 weight matrices instead of 2 self.mlp = SwiGLU(dim, int(dim * mlp_ratio * 2/3)) def forward(self, x, freqs_cis=None): # Pre-norm + residual x = x + self.attn(self.norm1(x), freqs_cis) x = x + self.mlp(self.norm2(x)) return xclass RMSNorm(nn.Module): """Root Mean Square Normalization. Simpler than LayerNorm: skips the mean-centering step. Empirically equivalent in quality but ~10% faster because it avoids computing the mean across the feature dimension. Used in LLaMA, Mistral, and most modern LLMs. """ def __init__(self, dim, eps=1e-6): super().__init__() self.eps = eps self.weight = nn.Parameter(torch.ones(dim)) def forward(self, x): # Compute root mean square (no mean subtraction, unlike LayerNorm) rms = torch.sqrt(torch.mean(x ** 2, dim=-1, keepdim=True) + self.eps) return x / rms * self.weightclass SwiGLU(nn.Module): """SwiGLU activation (better than ReLU/GELU for LLMs). SwiGLU uses a gating mechanism: one linear projection creates a "gate" that controls how much information flows through another projection. Think of it as the network learning to selectively amplify or suppress different features at each position. Formula: SwiGLU(x) = (Swish(xW1)) * (xW3) then projected by W2 """ def __init__(self, dim, hidden_dim): super().__init__() self.w1 = nn.Linear(dim, hidden_dim, bias=False) # Gate projection self.w2 = nn.Linear(hidden_dim, dim, bias=False) # Down projection self.w3 = nn.Linear(dim, hidden_dim, bias=False) # Up projection def forward(self, x): # F.silu = x * sigmoid(x), also called "Swish" # The gate (silu(w1(x))) controls the flow of the value (w3(x)) return self.w2(F.silu(self.w1(x)) * self.w3(x))
def causal_lm_loss(logits, labels): """Next token prediction loss. The fundamental training objective: at every position, predict the NEXT token. If the sequence is [The, cat, sat], we want: - Position 0 (The) -> predict "cat" - Position 1 (cat) -> predict "sat" This is why we shift logits and labels by one position. """ # Shift: logits at position t predict the token at position t+1 shift_logits = logits[..., :-1, :].contiguous() shift_labels = labels[..., 1:].contiguous() loss = F.cross_entropy( shift_logits.view(-1, shift_logits.size(-1)), # Flatten to (batch*seq, vocab) shift_labels.view(-1), # Flatten to (batch*seq,) ignore_index=-100 # Ignore padding tokens in loss computation ) return loss
One of the most fascinating (and debated) phenomena in foundation models: as models scale, new abilities appear that were not present at smaller scales. This is not a gradual improvement — it is more like phase transitions in physics, where water suddenly becomes ice at a critical temperature.
Scale
Emergent Capability
~1B
Basic language understanding
~10B
Few-shot learning
~100B
Complex reasoning, code generation
~500B+
Multi-step reasoning, tool use
Important nuance: The concept of “emergence” in LLMs is actively debated. Some researchers argue that emergence is partly an artifact of how we measure performance (e.g., using exact-match accuracy that jumps from 0% to near-100%). When using continuous metrics like log-likelihood, the improvement often looks gradual. Regardless, the practical reality is clear: larger models can do things smaller models cannot, and predicting exactly which capabilities will appear at which scale remains an open problem.
Training LLMs
Distributed Training Setup
import torch.distributed as distfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDPdef train_llm(): # Initialize distributed dist.init_process_group("nccl") rank = dist.get_rank() # Create model and wrap with FSDP model = LLM(config) model = FSDP( model, mixed_precision=MixedPrecision( param_dtype=torch.bfloat16, reduce_dtype=torch.float32, ), sharding_strategy=ShardingStrategy.FULL_SHARD, ) optimizer = torch.optim.AdamW( model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1, ) # Training loop for step, batch in enumerate(dataloader): loss = model(batch).loss loss.backward() # Gradient clipping torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() optimizer.zero_grad() if rank == 0 and step % 100 == 0: print(f"Step {step}: loss = {loss.item():.4f}")
Instruction Tuning
INSTRUCTION_TEMPLATE = """<|system|>{system_message}<|user|>{instruction}<|assistant|>{response}"""def format_instruction_data(example): return INSTRUCTION_TEMPLATE.format( system_message="You are a helpful assistant.", instruction=example["instruction"], response=example["response"], )# Fine-tune on instruction datasetinstruction_dataset = load_dataset("instruction_data")formatted = instruction_dataset.map(format_instruction_data)
RLHF (Reinforcement Learning from Human Feedback)
RLHF is how raw language models become useful assistants. The base model can generate fluent text, but it might also be toxic, unhelpful, or dishonest. RLHF aligns the model with human preferences through a three-step process: (1) supervised fine-tuning on demonstrations, (2) training a reward model on human preference rankings, and (3) using RL (typically PPO) to optimize the policy against the reward model while staying close to the original model.The analogy: the base model is a talented but unsocialized intern. SFT teaches them basic professional behavior from examples. RLHF then refines their judgment by showing them pairs of responses and learning which ones humans prefer.
class RewardModel(nn.Module): """Reward model trained on human preferences.""" def __init__(self, base_model): super().__init__() self.base = base_model self.reward_head = nn.Linear(base_model.config.hidden_size, 1) def forward(self, input_ids, attention_mask): outputs = self.base(input_ids, attention_mask=attention_mask) last_hidden = outputs.last_hidden_state[:, -1, :] # Last token return self.reward_head(last_hidden)def compute_preference_loss(reward_model, chosen, rejected): """Bradley-Terry preference model loss.""" reward_chosen = reward_model(**chosen) reward_rejected = reward_model(**rejected) loss = -F.logsigmoid(reward_chosen - reward_rejected).mean() return loss
PPO Training
def ppo_step(policy, ref_policy, reward_model, prompts, kl_coef=0.1): """Single PPO update step. The KL penalty is crucial: without it, the model would quickly learn to "hack" the reward model by generating degenerate text that scores highly but is nonsensical. The KL term acts as a leash, keeping the policy close to the well-behaved reference model. """ # Step 1: Generate responses with current policy responses = policy.generate(prompts) # Step 2: Score responses with the reward model rewards = reward_model(prompts, responses) # Step 3: Compute KL divergence from reference model (the "leash") with torch.no_grad(): ref_logprobs = ref_policy.log_prob(prompts, responses) policy_logprobs = policy.log_prob(prompts, responses) kl = policy_logprobs - ref_logprobs # Per-token KL approximation # Total reward = RM reward minus KL penalty # Higher kl_coef = stronger pull toward reference model = more conservative total_reward = rewards - kl_coef * kl # PPO clipped objective: prevent catastrophically large policy updates ratio = torch.exp(policy_logprobs - old_logprobs) clipped = torch.clamp(ratio, 1 - eps, 1 + eps) loss = -torch.min(ratio * total_reward, clipped * total_reward).mean() return loss
Plot loss vs compute for different model sizes. Verify the Chinchilla scaling law.
Exercise 2: Build a Mini-LLM
Train a small (10M parameter) causal language model on a text corpus.
Exercise 3: Instruction Tuning
Fine-tune a small LLM on instruction data using LoRA. Compare before/after.
What’s Next
Module 26: Capstone Project
Build a complete deep learning system from scratch.
Interview Deep-Dive
Explain the Chinchilla scaling law and how it changed the industry's approach to training LLMs. Where does the law break down?
Strong Answer:The Chinchilla scaling law, from DeepMind’s 2022 paper, established that for a given compute budget, the optimal strategy is to scale model size and training data equally. The rule of thumb is roughly 20 tokens per parameter for compute-optimal training. This was a paradigm shift because prior to Chinchilla, the dominant approach (exemplified by GPT-3’s 175B parameters trained on only 300B tokens, a ratio of 1.7) was to build the biggest model you could afford and train it on whatever data you had.Chinchilla demonstrated that their 70B model trained on 1.4T tokens (ratio of 20) outperformed the 280B Gopher model on most benchmarks — at roughly 4x lower inference cost. This directly spawned the “smaller but better-trained” movement: LLaMA (7B-65B trained on 1-1.4T tokens), Mistral, and Gemma all follow Chinchilla-optimal or even over-trained ratios.Where it breaks down: the law assumes you care about training compute optimality. In practice, inference cost often dominates total cost of ownership. If you are serving a model to millions of users, a smaller model trained far beyond Chinchilla-optimal (say 7B parameters on 2T+ tokens, ratio of 285) is cheaper to serve even though you “wasted” training compute. LLaMA 3 trained 8B parameters on 15T tokens — a ratio of 1,875 — precisely because inference savings at Meta’s scale vastly outweigh extra training cost. The scaling law also assumes fixed data quality; in practice, data curation and deduplication can shift the optimal ratio significantly.Follow-up: If you were given a fixed GPU budget of 1000 H100-hours to train a model for a specific production task, how would you allocate between model size and training data?I would first estimate the total compute in FLOPs: roughly 1000 hours * 990 TFLOPS (H100 BF16) = about 3.6e21 FLOPs. Using the Chinchilla formula, that suggests roughly a 1.3B parameter model trained on 26B tokens for compute-optimal training. But since this is for production deployment, I would skew toward a smaller model (say 400M parameters) trained on the same token budget, over-training by about 3x. The reasoning: at serving time, a 400M model is 3x cheaper and faster, and the over-training penalty on final loss is modest — maybe 2-5% worse perplexity. I would also invest in high-quality domain-specific data rather than more generic web text, since data quality has been shown to shift the effective training ratio favorably.
Walk me through the key architectural differences between the original 2017 Transformer and a modern LLM like LLaMA. Why was each change made?
Strong Answer:There are four major changes, each addressing a specific limitation discovered through years of scaling.First, pre-normalization with RMSNorm instead of post-normalization with LayerNorm. The original Transformer applied LayerNorm after the residual addition (post-norm). This creates training instability at scale because the residual stream’s magnitude can grow unpredictably. Pre-norm applies normalization before the attention and MLP sublayers, which stabilizes the residual stream and makes training more robust. RMSNorm specifically drops the mean-centering step of LayerNorm, saving about 10% compute with empirically equivalent quality.Second, Grouped Query Attention (GQA) instead of Multi-Head Attention. Standard MHA uses the same number of key-value heads as query heads. GQA uses fewer KV heads (typically 4-8x fewer). This dramatically reduces the KV cache memory during autoregressive inference. For a 70B model serving long sequences, the KV cache can consume 20+ GB; GQA cuts this proportionally. The quality impact is minimal because keys and values are shared across groups of query heads, and empirically the model learns to use this shared structure effectively.Third, SwiGLU activation instead of ReLU or GELU in the feedforward network. SwiGLU uses a gating mechanism where one linear projection acts as a learned gate that modulates another projection. This consistently beats ReLU and GELU by about 1% on downstream benchmarks. The hidden dimension is adjusted by a factor of 2/3 to compensate for SwiGLU having three weight matrices instead of two, keeping the parameter count constant.Fourth, Rotary Position Embeddings (RoPE) instead of absolute sinusoidal or learned position embeddings. RoPE encodes position information by rotating query and key vectors in 2D subspaces. This has a critical advantage: the attention score between two tokens depends only on their relative distance, not their absolute positions. This enables better length generalization — a model trained on 4K context can often extrapolate to 8K or beyond with minimal degradation, which absolute embeddings cannot do.Follow-up: If you had to drop one of these improvements to simplify your implementation, which would you sacrifice and why?I would drop SwiGLU and revert to GELU. The accuracy gain from SwiGLU (roughly 1%) is the smallest of the four changes, while the implementation complexity is non-trivial (three weight matrices instead of two, adjusted hidden dimension). Pre-norm and RoPE are essential for training stability and length generalization respectively, and GQA is essential for practical inference at scale. GELU with a standard two-matrix FFN is simpler, well-understood, and the 1% quality gap can often be closed with slightly more training data.
How does RLHF work at a high level, and what are the known failure modes of the reward model?
Strong Answer:RLHF has three phases. Phase one: supervised fine-tuning (SFT) on high-quality demonstration data teaches the model basic instruction-following behavior. Phase two: a reward model is trained on human preference data — annotators see pairs of model outputs and indicate which they prefer, and the reward model learns to predict these preferences using a Bradley-Terry loss. Phase three: the policy model (the actual LLM) is optimized via PPO to maximize the reward model’s score while staying close to the SFT model via a KL divergence penalty.The reward model is the weakest link, and its failure modes are well-documented. First, reward hacking: the policy discovers patterns that score highly with the reward model but are not genuinely good responses. A classic example is the model learning to produce longer, more verbose answers because the reward model was trained on data where humans tended to prefer longer responses. The model exploits this spurious correlation rather than actually being more helpful. The KL penalty mitigates this but does not eliminate it.Second, distribution shift: the reward model was trained on outputs from the SFT model, but during PPO training, the policy drifts into regions of output space the reward model has never seen. The reward model’s predictions in these out-of-distribution regions are unreliable, which can amplify reward hacking.Third, annotation inconsistency: different human annotators disagree on preferences, sometimes substantially. The reward model learns an average of these inconsistent signals, which can produce a reward landscape that does not match any individual human’s preferences well. This is particularly problematic for subjective or culturally dependent preferences.In practice, DPO (Direct Preference Optimization) has emerged as a simpler alternative that skips the reward model entirely and directly optimizes the policy on preference data. It avoids the reward hacking failure mode but has its own limitations — it cannot easily incorporate real-time reward signals or non-pairwise preference data.Follow-up: You notice that after RLHF, your model has become sycophantic — it agrees with the user even when the user is factually wrong. How do you diagnose and fix this?Sycophancy is one of the most common RLHF failure modes. The diagnosis starts with targeted evaluation: create a benchmark of prompts where the user states an incorrect fact and asks the model to agree. Measure how often the post-RLHF model agrees versus the pre-RLHF (SFT) model. If agreement rate increased, RLHF is the cause.The root cause is usually in the preference data: annotators rated agreeable responses higher than disagreeable ones, even when disagreement was factually correct. The fix involves multiple interventions: (1) curate “constitutional” preference pairs where the correct ranking explicitly rewards polite disagreement over sycophantic agreement, (2) add a factuality reward signal (using a separate fact-checking model) alongside the preference-based reward, (3) increase the KL penalty to keep the model closer to the SFT baseline where sycophancy was less pronounced. Anthropic’s Constitutional AI approach addresses this by having the model critique its own outputs against explicit principles before the reward model is trained.
What are emergent capabilities in foundation models, and why is there active debate about whether 'emergence' is real or an artifact of measurement?
Strong Answer:Emergent capabilities are abilities that appear in larger models but are absent in smaller ones — not as a gradual improvement, but as a sharp transition. The canonical examples include in-context few-shot learning (appearing around 10B parameters), chain-of-thought reasoning (around 100B), and multi-step tool use (around 500B+). The term draws an analogy to phase transitions in physics, like water freezing at a critical temperature.The debate centers on a 2023 paper by Schaeffer et al. that argued emergence is largely an artifact of the metrics used to measure it. Their key insight: when you measure performance using exact-match accuracy (binary: the answer is either exactly right or completely wrong), you see a sharp jump from 0% to near-100% as scale increases. But when you measure using continuous metrics like token-level log-likelihood, the improvement is smooth and predictable across all scales. The “emergence” is in the metric, not the model.Their argument is compelling for many benchmarks: if a model’s probability of generating the correct answer gradually increases from 1% to 30% to 70%, exact-match accuracy stays at 0% until the model consistently gets every token right, then jumps to near-100%. The underlying capability was improving all along, but the discontinuous metric hid the gradual progress.However, there are genuine counterexamples where the capability itself seems qualitatively new, not just quantitatively better. Multi-step arithmetic, compositional generalization to novel combinations, and self-correction behaviors do appear to require a minimum scale threshold even under continuous metrics. The honest answer is that “emergence” is partly real and partly measurement artifact, and the field has not fully disentangled the two.For practitioners, the practical takeaway is: do not assume your model will gain a specific capability by scaling up. Test explicitly at your target scale, and use continuous evaluation metrics whenever possible to get a more accurate picture of where your model actually is on the capability curve.Follow-up: How would you design an evaluation to determine whether a specific capability in your model is truly emergent versus gradually improving?I would test the model at multiple scales (1B, 3B, 7B, 13B, 70B) using both discrete and continuous metrics for the same underlying task. For example, if testing arithmetic: exact-match accuracy on the full equation (discrete), character-level edit distance from the correct answer (continuous), and per-digit accuracy (semi-continuous). If the discrete metric shows a sharp jump but the continuous metrics show smooth improvement, the “emergence” is a measurement artifact. If both metrics show a sharp phase transition at the same scale, the emergence is more likely real. I would also control for confounds like training data composition and tokenization differences across model sizes, since these can create false emergence signals.
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Dev Weekends","colors":{"primary":"#000000","light":"#6B7280","dark":"#000000"},"favicon":"https://mintcdn.com/devweeekends/uFspW_ITEafc8XbV/images/logo.png?fit=max\u0026auto=format\u0026n=uFspW_ITEafc8XbV\u0026q=85\u0026s=24941b53d3f0793eb83974749e82df4f","navbar":{"links":[{"href":"https://devweekends.com","label":"Back to Site"}]},"footer":{"socials":{"youtube":"https://www.youtube.com/@devweekends","github":"https://github.com/DevWeekends","linkedin":"https://www.linkedin.com/company/dev-weekends"}},"contextual":{"options":["copy","view","chatgpt","claude","perplexity","mcp","cursor","vscode"],"display":"header"},"styling":{"latex":true},"navigation":{"global":{"anchors":[{"anchor":"DevWeekends","icon":"globe","href":"https://devweekends.com"}]},"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"devweeekends","hasAudioNarration":false,"actualSubdomain":"devweeekends","customDomains":["resources.devweekends.com"],"gitSource":{"type":"github","owner":"devweekends","repo":"docs","deployBranch":"main","contentDirectory":"","isPrivate":false,"mountPath":"","label":""},"entitlements":{},"buildId":"6ad5cb977e3343e90aaa7f7f","clientVersion":"0.0.4205","searchFilterCounts":[{"language":"default","tag":"Courses","count":785},{"language":"default","tag":"Crash Courses","count":145},{"language":"default","tag":"Interview Prep","count":69},{"language":"default","tag":"AI Engineering","count":60},{"language":"default","tag":"Distributed System Tools","count":60},{"language":"default","tag":"Deep Learning Mastery","count":48},{"language":"default","tag":"DevOps Tools","count":38},{"language":"default","tag":"Think Like an Engineer","count":38},{"language":"default","tag":"Competitive Programming","count":37},{"language":"default","tag":"Angular Master Course","count":34}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"treatment","enabled":true,"payload":{"type":"string","value":"treatment"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[],"page":{"pagePath":"courses/deep-learning-mastery/25-foundation-models","slug":"/courses/deep-learning-mastery/25-foundation-models","pageMetadata":{"title":"Foundation Models \u0026 LLMs","sidebarTitle":"Foundation Models","description":"Scaling laws, emergent capabilities, and training large language models","icon":"brain","href":"/courses/deep-learning-mastery/25-foundation-models"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Scaling laws, emergent capabilities, and training large language models\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={annotation:`annotation`,code:`code`,hr:`hr`,li:`li`,math:`math`,mi:`mi`,mn:`mn`,mo:`mo`,mrow:`mrow`,mspace:`mspace`,msub:`msub`,msup:`msup`,mtext:`mtext`,munderover:`munderover`,p:`p`,pre:`pre`,semantics:`semantics`,span:`span`,strong:`strong`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,ul:`ul`,..._provideComponents(),...props.components},{Accordion,AccordionGroup,Card,CardGroup,CodeBlock,Frame,Heading,OptimizedImage,Table,Tip,Warning}=_components;if(!Accordion)_missingMdxReference(`Accordion`,true);if(!AccordionGroup)_missingMdxReference(`AccordionGroup`,true);if(!Card)_missingMdxReference(`Card`,true);if(!CardGroup)_missingMdxReference(`CardGroup`,true);if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Frame)_missingMdxReference(`Frame`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!OptimizedImage)_missingMdxReference(`OptimizedImage`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Tip)_missingMdxReference(`Tip`,true);if(!Warning)_missingMdxReference(`Warning`,true);return _jsxs(_Fragment,{children:[_jsx(Frame,{children:_jsx(OptimizedImage,{src:`https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/foundation-models-concept.svg?fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=2fe5eeb8dcc43452c9b0a0a6251b6fdf`,alt:`Foundation Models`,\"data-og-width\":`1080`,width:`1080`,\"data-og-height\":`1080`,height:`1080`,\"data-path\":`images/courses/deep-learning-mastery/foundation-models-concept.svg`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/foundation-models-concept.svg?w=280\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=be0bc66115228c4bedf23c3cd7c116a5 280w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/foundation-models-concept.svg?w=560\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=a7cc5d7f3e8417b6232e18f0c2a7e98f 560w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/foundation-models-concept.svg?w=840\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=4466ae9cc2da79b7c8070fe637c78de8 840w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/foundation-models-concept.svg?w=1100\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=6af1b78add859635b5c6764ec79e8596 1100w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/foundation-models-concept.svg?w=1650\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=25a28d0ece0dbdeb8cb56e530acfe220 1650w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/foundation-models-concept.svg?w=2500\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=a8ee8fe79405b69640e22450d3b23135 2500w`})}),`\n`,_jsx(Heading,{level:`1`,id:`foundation-models-\u0026-llms`,children:`Foundation Models \u0026 LLMs`}),`\n`,_jsx(Heading,{level:`2`,id:`the-foundation-model-paradigm`,children:`The Foundation Model Paradigm`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Foundation models`}),` are large models trained on broad data that can be adapted to many downstream tasks. The analogy is a well-rounded liberal arts education: rather than training a specialist from scratch for every task, you invest heavily in a broad base of knowledge, then specialize with minimal additional effort. GPT-3 was trained once at enormous cost, but it can be adapted to summarization, translation, code generation, and thousands of other tasks with just a few examples — or even zero examples.`]}),`\n`,_jsx(_components.p,{children:`The term “foundation model” was coined by Stanford’s HAI in 2021, and it captures a paradigm shift: instead of building task-specific models from scratch, you build on top of a pre-existing “foundation” of learned knowledge.`}),`\n`,_jsx(_components.p,{children:`Key characteristics:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Scale`}),` — billions of parameters, trained on trillions of tokens`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Self-supervised pretraining`}),` — learns from raw text without human annotation`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Emergent capabilities`}),` — abilities that appear only at sufficient scale, not explicitly trained`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Transfer to diverse tasks`}),` — one model, many applications via prompting or fine-tuning`]}),`\n`]}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`scaling-laws`,children:`Scaling Laws`}),`\n`,_jsx(Heading,{level:`3`,id:`the-chinchilla-scaling-law`,children:`The Chinchilla Scaling Law`}),`\n`,_jsx(_components.p,{children:`DeepMind’s Chinchilla paper (2022) fundamentally changed how the industry thinks about training LLMs. The key finding: most models before Chinchilla were dramatically undertrained — they were too large for the amount of data they saw.`}),`\n`,_jsx(_components.p,{children:`For compute-optimal training:`}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`N`}),_jsx(_components.mtext,{children:`opt`})]}),_jsx(_components.mo,{children:`∝`}),_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`C`}),_jsx(_components.mn,{children:`0.5`})]}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mspace,{width:`1em`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`D`}),_jsx(_components.mtext,{children:`opt`})]}),_jsx(_components.mo,{children:`∝`}),_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`C`}),_jsx(_components.mn,{children:`0.5`})]})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`N_{\\\\text{opt}} \\\\propto C^{0.5}, \\\\quad D_{\\\\text{opt}} \\\\propto C^{0.5}`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.9694em`,verticalAlign:`-0.2861em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.109em`},children:`N`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.2806em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`-0.109em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord text mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`opt`})})})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2861em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`∝`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.1502em`,verticalAlign:`-0.2861em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`C`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8641em`},children:_jsxs(_components.span,{style:{top:`-3.113em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`0.5`})})})]})})})})})]}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`1em`}}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0278em`},children:`D`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.2806em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`-0.0278em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord text mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`opt`})})})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2861em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`∝`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8641em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`C`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8641em`},children:_jsxs(_components.span,{style:{top:`-3.113em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`0.5`})})})]})})})})})]})]})]})]})}),`\n`,_jsx(_components.p,{children:`Where:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`N`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`N`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6833em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.109em`},children:`N`})]})})]}),` = number of parameters`]}),`\n`,_jsxs(_components.li,{children:[_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`D`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`D`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6833em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0278em`},children:`D`})]})})]}),` = dataset size (tokens)`]}),`\n`,_jsxs(_components.li,{children:[_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`C`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`C`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6833em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`C`})]})})]}),` = compute budget (FLOPs)`]}),`\n`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Rule of thumb`}),`: Train on ~20 tokens per parameter. This means a 7B parameter model should see roughly 140B tokens for compute-optimal training.`]}),`\n`,_jsx(Tip,{children:_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Why this matters in practice`}),`: Before Chinchilla, the trend was “bigger model = better.” Chinchilla showed that a 70B model trained on 1.4T tokens outperformed the 280B Gopher model trained on 300B tokens — at much lower inference cost. This shifted the industry toward smaller, better-trained models, and directly influenced LLaMA, Mistral, and other efficient model families.`]})}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Model`}),_jsx(_components.th,{children:`Parameters`}),_jsx(_components.th,{children:`Training Tokens`}),_jsx(_components.th,{children:`Ratio`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`GPT-3`}),_jsx(_components.td,{children:`175B`}),_jsx(_components.td,{children:`300B`}),_jsx(_components.td,{\"data-numeric\":`true`,children:`1.7`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Chinchilla`}),_jsx(_components.td,{children:`70B`}),_jsx(_components.td,{\"data-numeric\":`true`,children:`1.4T`}),_jsx(_components.td,{\"data-numeric\":`true`,children:`20`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`LLaMA 2`}),_jsx(_components.td,{children:`70B`}),_jsx(_components.td,{\"data-numeric\":`true`,children:`2T`}),_jsx(_components.td,{\"data-numeric\":`true`,children:`29`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Mistral`}),_jsx(_components.td,{children:`7B`}),_jsx(_components.td,{children:`Unknown`}),_jsx(_components.td,{\"data-numeric\":`true`,children:`-`})]})]})]}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`llm-architecture`,children:`LLM Architecture`}),`\n`,_jsx(Heading,{level:`3`,id:`modern-transformer-improvements`,children:`Modern Transformer Improvements`}),`\n`,_jsx(_components.p,{children:`Modern LLMs use several key improvements over the original 2017 Transformer. Each one addresses a specific limitation discovered through years of scaling experiments. Understanding these is essential for reading current research papers and building production systems.`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`76`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`76`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn.functional `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` ModernTransformerBlock`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"LLaMA-style transformer block with modern improvements.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Three key changes from the original Transformer:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 1. Pre-normalization (RMSNorm before attention/MLP, not after)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 2. Grouped Query Attention (fewer KV heads = faster inference)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 3. SwiGLU activation (empirically better than ReLU/GELU)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`num_heads`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`mlp_ratio`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Pre-normalization with RMSNorm (simpler and faster than LayerNorm)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` RMSNorm(dim)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm2 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` RMSNorm(dim)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Grouped Query Attention: use 4x fewer KV heads than query heads`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # This dramatically reduces KV cache memory during inference`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` GroupedQueryAttention(dim, num_heads, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`num_kv_heads`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`num_heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`//`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # SwiGLU MLP: ~1% better than GELU across most benchmarks`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # The 2/3 factor compensates for SwiGLU having 3 weight matrices instead of 2`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.mlp `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` SwiGLU(dim, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mlp_ratio `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`freqs_cis`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Pre-norm + residual`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.attn(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm1(x), freqs_cis)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.mlp(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm2(x))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` RMSNorm`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Root Mean Square Normalization.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Simpler than LayerNorm: skips the mean-centering step.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Empirically equivalent in quality but ~10% faster because`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` it avoids computing the mean across the feature dimension.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Used in LLaMA, Mistral, and most modern LLMs.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`eps`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1e-6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.eps `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` eps`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.weight `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Parameter(torch.ones(dim))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Compute root mean square (no mean subtraction, unlike LayerNorm)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` rms `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.sqrt(torch.mean(x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`keepdim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.eps)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` rms `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.weight`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` SwiGLU`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"SwiGLU activation (better than ReLU/GELU for LLMs).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` SwiGLU uses a gating mechanism: one linear projection creates a \"gate\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` that controls how much information flows through another projection.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Think of it as the network learning to selectively amplify or suppress`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` different features at each position.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Formula: SwiGLU(x) = (Swish(xW1)) * (xW3) then projected by W2`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`hidden_dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.w1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(dim, hidden_dim, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Gate projection`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.w2 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(hidden_dim, dim, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Down projection`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.w3 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(dim, hidden_dim, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Up projection`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # F.silu = x * sigmoid(x), also called \"Swish\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # The gate (silu(w1(x))) controls the flow of the value (w3(x))`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.w2(F.silu(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.w1(x)) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.w3(x))`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`rotary-position-embeddings-rope`,children:`Rotary Position Embeddings (RoPE)`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`19`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`19`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` precompute_freqs_cis`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`max_seq_len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`base`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Precompute rotary embedding frequencies.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` freqs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1.0`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` /`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (base `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (torch.arange(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, dim, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).float() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` dim))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` t `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.arange(max_seq_len)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` freqs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.outer(t, freqs)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.polar(torch.ones_like(freqs), freqs) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Complex exponentials`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` apply_rotary_emb`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`xq`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`xk`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`freqs_cis`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Apply rotary embeddings to queries and keys.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Reshape to complex`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` xq_ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.view_as_complex(xq.float().reshape(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`xq.shape[:`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` xk_ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.view_as_complex(xk.float().reshape(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`xk.shape[:`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Apply rotation`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` xq_out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.view_as_real(xq_ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` freqs_cis).flatten(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` xk_out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.view_as_real(xk_ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` freqs_cis).flatten(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` xq_out.type_as(xq), xk_out.type_as(xk)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`pretraining-objectives`,children:`Pretraining Objectives`}),`\n`,_jsx(Heading,{level:`3`,id:`causal-language-modeling-gpt-style`,children:`Causal Language Modeling (GPT-style)`}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{mathvariant:`script`,children:`L`}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mo,{children:`−`}),_jsxs(_components.munderover,{children:[_jsx(_components.mo,{children:`∑`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`t`}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mn,{children:`1`})]}),_jsx(_components.mi,{children:`T`})]}),_jsx(_components.mi,{children:`log`}),_jsx(_components.mo,{children:``}),_jsx(_components.mi,{children:`P`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`x`}),_jsx(_components.mi,{children:`t`})]}),_jsx(_components.mi,{mathvariant:`normal`,children:`∣`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`x`}),_jsxs(_components.mrow,{children:[_jsx(_components.mo,{children:`\u003c`}),_jsx(_components.mi,{children:`t`})]})]}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\mathcal{L} = -\\\\sum_{t=1}^{T} \\\\log P(x_t | x_{\u003ct})`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6833em`}}),_jsx(_components.span,{className:`mord mathcal`,children:`L`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`3.0954em`,verticalAlign:`-1.2671em`}}),_jsx(_components.span,{className:`mord`,children:`−`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mop op-limits`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`1.8283em`},children:[_jsxs(_components.span,{style:{top:`-1.8829em`,marginLeft:`0em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3.05em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mathnormal mtight`,children:`t`}),_jsx(_components.span,{className:`mrel mtight`,children:`=`}),_jsx(_components.span,{className:`mord mtight`,children:`1`})]})})]}),_jsxs(_components.span,{style:{top:`-3.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3.05em`}}),_jsx(_components.span,{children:_jsx(_components.span,{className:`mop op-symbol large-op`,children:`∑`})})]}),_jsxs(_components.span,{style:{top:`-4.3em`,marginLeft:`0em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3.05em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.1389em`},children:`T`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`1.2671em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsxs(_components.span,{className:`mop`,children:[`lo`,_jsx(_components.span,{style:{marginRight:`0.0139em`},children:`g`})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`P`}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`x`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.2806em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`t`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mord`,children:`∣`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`x`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.2806em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mrel mtight`,children:`\u003c`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`t`})]})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1774em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mclose`,children:`)`})]})]})]})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`19`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`19`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` causal_lm_loss`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`logits`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`labels`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Next token prediction loss.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` The fundamental training objective: at every position, predict the NEXT token.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` If the sequence is [The, cat, sat], we want:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Position 0 (The) -> predict \"cat\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Position 1 (cat) -> predict \"sat\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` This is why we shift logits and labels by one position.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Shift: logits at position t predict the token at position t+1`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` shift_logits `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` logits[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#D4D4D4`},children:`...`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, :`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, :].contiguous()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` shift_labels `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` labels[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#D4D4D4`},children:`...`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:].contiguous()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F.cross_entropy(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` shift_logits.view(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, shift_logits.size(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)), `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Flatten to (batch*seq, vocab)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` shift_labels.view(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`), `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Flatten to (batch*seq,)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` ignore_index`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`100`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Ignore padding tokens in loss computation`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`masked-language-modeling-bert-style`,children:`Masked Language Modeling (BERT-style)`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`19`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`19`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` create_mlm_inputs`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`tokens`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`mask_prob`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.15`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`vocab_size`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`32000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Create masked inputs for MLM training.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` labels `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tokens.clone()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Random mask selection`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` probability_matrix `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.full(tokens.shape, mask_prob)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` masked_indices `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.bernoulli(probability_matrix).bool()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Don't mask special tokens`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` labels[`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`~`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`masked_indices] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`100`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # 80% [MASK], 10% random, 10% unchanged`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` indices_replaced `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` masked_indices `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`\u0026`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (torch.rand(tokens.shape) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`\u003c`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tokens[indices_replaced] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#D4D4D4`},children:` MASK_TOKEN_ID`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` indices_random `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` masked_indices `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`\u0026`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` ~`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`indices_replaced `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`\u0026`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (torch.rand(tokens.shape) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`\u003c`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tokens[indices_random] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randint(vocab_size, tokens.shape)[indices_random]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tokens, labels`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`emergent-capabilities`,children:`Emergent Capabilities`}),`\n`,_jsx(_components.p,{children:`One of the most fascinating (and debated) phenomena in foundation models: as models scale, new abilities appear that were not present at smaller scales. This is not a gradual improvement — it is more like phase transitions in physics, where water suddenly becomes ice at a critical temperature.`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Scale`}),_jsx(_components.th,{children:`Emergent Capability`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`~1B`}),_jsx(_components.td,{children:`Basic language understanding`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`~10B`}),_jsx(_components.td,{children:`Few-shot learning`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`~100B`}),_jsx(_components.td,{children:`Complex reasoning, code generation`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`~500B+`}),_jsx(_components.td,{children:`Multi-step reasoning, tool use`})]})]})]}),`\n`,_jsx(Warning,{children:_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Important nuance`}),`: The concept of “emergence” in LLMs is actively debated. Some researchers argue that emergence is partly an artifact of how we measure performance (e.g., using exact-match accuracy that jumps from 0% to near-100%). When using continuous metrics like log-likelihood, the improvement often looks gradual. Regardless, the practical reality is clear: larger models can do things smaller models cannot, and predicting exactly which capabilities will appear at which scale remains an open problem.`]})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`training-llms`,children:`Training LLMs`}),`\n`,_jsx(Heading,{level:`3`,id:`distributed-training-setup`,children:`Distributed Training Setup`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`39`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`39`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.distributed `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` dist`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`from`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.distributed.fsdp `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` FullyShardedDataParallel `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#D4D4D4`},children:` FSDP`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` train_llm`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Initialize distributed`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` dist.init_process_group(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"nccl\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` rank `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` dist.get_rank()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Create model and wrap with FSDP`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` LLM(config)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` FSDP(`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model,`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` mixed_precision`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`MixedPrecision(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` param_dtype`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`torch.bfloat16,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` reduce_dtype`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`torch.float32,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ),`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` sharding_strategy`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`ShardingStrategy.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#D4D4D4`},children:`FULL_SHARD`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.optim.AdamW(`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.parameters(),`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` lr`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3e-4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` betas`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.9`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.95`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` weight_decay`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Training loop`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` step, batch `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` enumerate`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(dataloader):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model(batch).loss`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss.backward()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Gradient clipping`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn.utils.clip_grad_norm_(model.parameters(), `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer.step()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer.zero_grad()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` rank `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` and`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` step `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`%`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 100`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` ==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Step `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`step`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`: loss = `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`loss.item()`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:.4f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`instruction-tuning`,children:`Instruction Tuning`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`17`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`17`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#D4D4D4`},children:`INSTRUCTION_TEMPLATE`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"\u003c|system|>`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{system_message}`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\u003c|user|>`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{instruction}`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\u003c|assistant|>`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{response}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"\"\"`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` format_instruction_data`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`example`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#D4D4D4`},children:` INSTRUCTION_TEMPLATE`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.format(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` system_message`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"You are a helpful assistant.\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` instruction`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`example[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"instruction\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`],`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` response`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`example[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"response\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`],`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Fine-tune on instruction dataset`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`instruction_dataset `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` load_dataset(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"instruction_data\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`formatted `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` instruction_dataset.map(format_instruction_data)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`rlhf-reinforcement-learning-from-human-feedback`,children:`RLHF (Reinforcement Learning from Human Feedback)`}),`\n`,_jsx(_components.p,{children:`RLHF is how raw language models become useful assistants. The base model can generate fluent text, but it might also be toxic, unhelpful, or dishonest. RLHF aligns the model with human preferences through a three-step process: (1) supervised fine-tuning on demonstrations, (2) training a reward model on human preference rankings, and (3) using RL (typically PPO) to optimize the policy against the reward model while staying close to the original model.`}),`\n`,_jsx(_components.p,{children:`The analogy: the base model is a talented but unsocialized intern. SFT teaches them basic professional behavior from examples. RLHF then refines their judgment by showing them pairs of responses and learning which ones humans prefer.`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`21`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`21`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` RewardModel`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Reward model trained on human preferences.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`base_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.base `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` base_model`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.reward_head `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(base_model.config.hidden_size, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`input_ids`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`attention_mask`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.base(input_ids, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`attention_mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`attention_mask)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` last_hidden `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs.last_hidden_state[:, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, :] `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Last token`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.reward_head(last_hidden)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` compute_preference_loss`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`reward_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`chosen`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`rejected`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Bradley-Terry preference model loss.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` reward_chosen `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` reward_model(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`chosen)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` reward_rejected `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` reward_model(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`rejected)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`F.logsigmoid(reward_chosen `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` reward_rejected).mean()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`ppo-training`,children:`PPO Training`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`30`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`30`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` ppo_step`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`policy`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`ref_policy`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`reward_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`prompts`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`kl_coef`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Single PPO update step.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` The KL penalty is crucial: without it, the model would quickly learn to`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"hack\" the reward model by generating degenerate text that scores highly`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` but is nonsensical. The KL term acts as a leash, keeping the policy`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` close to the well-behaved reference model.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Step 1: Generate responses with current policy`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` responses `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` policy.generate(prompts)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Step 2: Score responses with the reward model`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` rewards `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` reward_model(prompts, responses)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Step 3: Compute KL divergence from reference model (the \"leash\")`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` with`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.no_grad():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ref_logprobs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ref_policy.log_prob(prompts, responses)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` policy_logprobs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` policy.log_prob(prompts, responses)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` kl `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` policy_logprobs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ref_logprobs `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Per-token KL approximation`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Total reward = RM reward minus KL penalty`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Higher kl_coef = stronger pull toward reference model = more conservative`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` total_reward `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` rewards `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` kl_coef `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` kl`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # PPO clipped objective: prevent catastrophically large policy updates`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ratio `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.exp(policy_logprobs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` old_logprobs)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` clipped `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.clamp(ratio, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` eps, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` +`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` eps)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`torch.min(ratio `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` total_reward, clipped `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` total_reward).mean()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`using-foundation-models`,children:`Using Foundation Models`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`22`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`22`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`from`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` transformers `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` AutoModelForCausalLM, AutoTokenizer`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Load pretrained LLM`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` AutoModelForCausalLM.from_pretrained(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"mistralai/Mistral-7B-v0.1\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` torch_dtype`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`torch.bfloat16,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` device_map`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"auto\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`tokenizer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` AutoTokenizer.from_pretrained(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"mistralai/Mistral-7B-v0.1\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` generate`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`prompt`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`max_tokens`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`100`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` inputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tokenizer(prompt, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`return_tensors`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"pt\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).to(model.device)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.generate(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` **`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`inputs,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` max_new_tokens`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`max_tokens,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` temperature`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.7`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` do_sample`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` top_p`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.9`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tokenizer.decode(outputs[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`skip_special_tokens`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`model-comparison`,children:`Model Comparison`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Model`}),_jsx(_components.th,{children:`Size`}),_jsx(_components.th,{children:`Open`}),_jsx(_components.th,{children:`Strengths`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`GPT-4`}),_jsx(_components.td,{children:`~1T?`}),_jsx(_components.td,{children:`No`}),_jsx(_components.td,{children:`Multimodal, reasoning`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Claude 3`}),_jsx(_components.td,{children:`Unknown`}),_jsx(_components.td,{children:`No`}),_jsx(_components.td,{children:`Safety, long context`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`LLaMA 3`}),_jsx(_components.td,{children:`8B-70B`}),_jsx(_components.td,{children:`Yes`}),_jsx(_components.td,{children:`Open, efficient`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Mistral`}),_jsx(_components.td,{children:`7B`}),_jsx(_components.td,{children:`Yes`}),_jsx(_components.td,{children:`Quality/size ratio`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Gemma`}),_jsx(_components.td,{children:`2B-7B`}),_jsx(_components.td,{children:`Yes`}),_jsx(_components.td,{children:`Small, efficient`})]})]})]}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`exercises`,children:`Exercises`}),`\n`,_jsxs(AccordionGroup,{children:[_jsx(Accordion,{title:`Exercise 1: Scaling Analysis`,id:`exercise-1-scaling-analysis`,children:_jsx(_components.p,{children:`Plot loss vs compute for different model sizes. Verify the Chinchilla scaling law.`})}),_jsx(Accordion,{title:`Exercise 2: Build a Mini-LLM`,id:`exercise-2-build-a-mini-llm`,children:_jsx(_components.p,{children:`Train a small (10M parameter) causal language model on a text corpus.`})}),_jsx(Accordion,{title:`Exercise 3: Instruction Tuning`,id:`exercise-3-instruction-tuning`,children:_jsx(_components.p,{children:`Fine-tune a small LLM on instruction data using LoRA. Compare before/after.`})})]}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`what’s-next`,children:`What’s Next`}),`\n`,_jsx(CardGroup,{cols:1,children:_jsx(Card,{title:`Module 26: Capstone Project`,icon:`flag-checkered`,href:`/courses/deep-learning-mastery/26-capstone`,children:_jsx(_components.p,{children:`Build a complete deep learning system from scratch.`})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`interview-deep-dive`,children:`Interview Deep-Dive`}),`\n`,_jsxs(AccordionGroup,{children:[_jsxs(Accordion,{title:`Explain the Chinchilla scaling law and how it changed the industry's approach to training LLMs. Where does the law break down?`,id:`explain-the-chinchilla-scaling-law-and-how-it-changed-the-industrys-approach-to-training-llms-where-does-the-law-break-down`,children:[_jsx(_components.p,{children:_jsx(_components.strong,{children:`Strong Answer:`})}),_jsx(_components.p,{children:`The Chinchilla scaling law, from DeepMind’s 2022 paper, established that for a given compute budget, the optimal strategy is to scale model size and training data equally. The rule of thumb is roughly 20 tokens per parameter for compute-optimal training. This was a paradigm shift because prior to Chinchilla, the dominant approach (exemplified by GPT-3’s 175B parameters trained on only 300B tokens, a ratio of 1.7) was to build the biggest model you could afford and train it on whatever data you had.`}),_jsx(_components.p,{children:`Chinchilla demonstrated that their 70B model trained on 1.4T tokens (ratio of 20) outperformed the 280B Gopher model on most benchmarks — at roughly 4x lower inference cost. This directly spawned the “smaller but better-trained” movement: LLaMA (7B-65B trained on 1-1.4T tokens), Mistral, and Gemma all follow Chinchilla-optimal or even over-trained ratios.`}),_jsx(_components.p,{children:`Where it breaks down: the law assumes you care about training compute optimality. In practice, inference cost often dominates total cost of ownership. If you are serving a model to millions of users, a smaller model trained far beyond Chinchilla-optimal (say 7B parameters on 2T+ tokens, ratio of 285) is cheaper to serve even though you “wasted” training compute. LLaMA 3 trained 8B parameters on 15T tokens — a ratio of 1,875 — precisely because inference savings at Meta’s scale vastly outweigh extra training cost. The scaling law also assumes fixed data quality; in practice, data curation and deduplication can shift the optimal ratio significantly.`}),_jsx(_components.p,{children:_jsx(_components.strong,{children:`Follow-up: If you were given a fixed GPU budget of 1000 H100-hours to train a model for a specific production task, how would you allocate between model size and training data?`})}),_jsx(_components.p,{children:`I would first estimate the total compute in FLOPs: roughly 1000 hours * 990 TFLOPS (H100 BF16) = about 3.6e21 FLOPs. Using the Chinchilla formula, that suggests roughly a 1.3B parameter model trained on 26B tokens for compute-optimal training. But since this is for production deployment, I would skew toward a smaller model (say 400M parameters) trained on the same token budget, over-training by about 3x. The reasoning: at serving time, a 400M model is 3x cheaper and faster, and the over-training penalty on final loss is modest — maybe 2-5% worse perplexity. I would also invest in high-quality domain-specific data rather than more generic web text, since data quality has been shown to shift the effective training ratio favorably.`})]}),_jsxs(Accordion,{title:`Walk me through the key architectural differences between the original 2017 Transformer and a modern LLM like LLaMA. Why was each change made?`,id:`walk-me-through-the-key-architectural-differences-between-the-original-2017-transformer-and-a-modern-llm-like-llama-why-was-each-change-made`,children:[_jsx(_components.p,{children:_jsx(_components.strong,{children:`Strong Answer:`})}),_jsx(_components.p,{children:`There are four major changes, each addressing a specific limitation discovered through years of scaling.`}),_jsx(_components.p,{children:`First, pre-normalization with RMSNorm instead of post-normalization with LayerNorm. The original Transformer applied LayerNorm after the residual addition (post-norm). This creates training instability at scale because the residual stream’s magnitude can grow unpredictably. Pre-norm applies normalization before the attention and MLP sublayers, which stabilizes the residual stream and makes training more robust. RMSNorm specifically drops the mean-centering step of LayerNorm, saving about 10% compute with empirically equivalent quality.`}),_jsx(_components.p,{children:`Second, Grouped Query Attention (GQA) instead of Multi-Head Attention. Standard MHA uses the same number of key-value heads as query heads. GQA uses fewer KV heads (typically 4-8x fewer). This dramatically reduces the KV cache memory during autoregressive inference. For a 70B model serving long sequences, the KV cache can consume 20+ GB; GQA cuts this proportionally. The quality impact is minimal because keys and values are shared across groups of query heads, and empirically the model learns to use this shared structure effectively.`}),_jsx(_components.p,{children:`Third, SwiGLU activation instead of ReLU or GELU in the feedforward network. SwiGLU uses a gating mechanism where one linear projection acts as a learned gate that modulates another projection. This consistently beats ReLU and GELU by about 1% on downstream benchmarks. The hidden dimension is adjusted by a factor of 2/3 to compensate for SwiGLU having three weight matrices instead of two, keeping the parameter count constant.`}),_jsx(_components.p,{children:`Fourth, Rotary Position Embeddings (RoPE) instead of absolute sinusoidal or learned position embeddings. RoPE encodes position information by rotating query and key vectors in 2D subspaces. This has a critical advantage: the attention score between two tokens depends only on their relative distance, not their absolute positions. This enables better length generalization — a model trained on 4K context can often extrapolate to 8K or beyond with minimal degradation, which absolute embeddings cannot do.`}),_jsx(_components.p,{children:_jsx(_components.strong,{children:`Follow-up: If you had to drop one of these improvements to simplify your implementation, which would you sacrifice and why?`})}),_jsx(_components.p,{children:`I would drop SwiGLU and revert to GELU. The accuracy gain from SwiGLU (roughly 1%) is the smallest of the four changes, while the implementation complexity is non-trivial (three weight matrices instead of two, adjusted hidden dimension). Pre-norm and RoPE are essential for training stability and length generalization respectively, and GQA is essential for practical inference at scale. GELU with a standard two-matrix FFN is simpler, well-understood, and the 1% quality gap can often be closed with slightly more training data.`})]}),_jsxs(Accordion,{title:`How does RLHF work at a high level, and what are the known failure modes of the reward model?`,id:`how-does-rlhf-work-at-a-high-level-and-what-are-the-known-failure-modes-of-the-reward-model`,children:[_jsx(_components.p,{children:_jsx(_components.strong,{children:`Strong Answer:`})}),_jsx(_components.p,{children:`RLHF has three phases. Phase one: supervised fine-tuning (SFT) on high-quality demonstration data teaches the model basic instruction-following behavior. Phase two: a reward model is trained on human preference data — annotators see pairs of model outputs and indicate which they prefer, and the reward model learns to predict these preferences using a Bradley-Terry loss. Phase three: the policy model (the actual LLM) is optimized via PPO to maximize the reward model’s score while staying close to the SFT model via a KL divergence penalty.`}),_jsx(_components.p,{children:`The reward model is the weakest link, and its failure modes are well-documented. First, reward hacking: the policy discovers patterns that score highly with the reward model but are not genuinely good responses. A classic example is the model learning to produce longer, more verbose answers because the reward model was trained on data where humans tended to prefer longer responses. The model exploits this spurious correlation rather than actually being more helpful. The KL penalty mitigates this but does not eliminate it.`}),_jsx(_components.p,{children:`Second, distribution shift: the reward model was trained on outputs from the SFT model, but during PPO training, the policy drifts into regions of output space the reward model has never seen. The reward model’s predictions in these out-of-distribution regions are unreliable, which can amplify reward hacking.`}),_jsx(_components.p,{children:`Third, annotation inconsistency: different human annotators disagree on preferences, sometimes substantially. The reward model learns an average of these inconsistent signals, which can produce a reward landscape that does not match any individual human’s preferences well. This is particularly problematic for subjective or culturally dependent preferences.`}),_jsx(_components.p,{children:`In practice, DPO (Direct Preference Optimization) has emerged as a simpler alternative that skips the reward model entirely and directly optimizes the policy on preference data. It avoids the reward hacking failure mode but has its own limitations — it cannot easily incorporate real-time reward signals or non-pairwise preference data.`}),_jsx(_components.p,{children:_jsx(_components.strong,{children:`Follow-up: You notice that after RLHF, your model has become sycophantic — it agrees with the user even when the user is factually wrong. How do you diagnose and fix this?`})}),_jsx(_components.p,{children:`Sycophancy is one of the most common RLHF failure modes. The diagnosis starts with targeted evaluation: create a benchmark of prompts where the user states an incorrect fact and asks the model to agree. Measure how often the post-RLHF model agrees versus the pre-RLHF (SFT) model. If agreement rate increased, RLHF is the cause.`}),_jsx(_components.p,{children:`The root cause is usually in the preference data: annotators rated agreeable responses higher than disagreeable ones, even when disagreement was factually correct. The fix involves multiple interventions: (1) curate “constitutional” preference pairs where the correct ranking explicitly rewards polite disagreement over sycophantic agreement, (2) add a factuality reward signal (using a separate fact-checking model) alongside the preference-based reward, (3) increase the KL penalty to keep the model closer to the SFT baseline where sycophancy was less pronounced. Anthropic’s Constitutional AI approach addresses this by having the model critique its own outputs against explicit principles before the reward model is trained.`})]}),_jsxs(Accordion,{title:`What are emergent capabilities in foundation models, and why is there active debate about whether 'emergence' is real or an artifact of measurement?`,id:`what-are-emergent-capabilities-in-foundation-models-and-why-is-there-active-debate-about-whether-emergence-is-real-or-an-artifact-of-measurement`,children:[_jsx(_components.p,{children:_jsx(_components.strong,{children:`Strong Answer:`})}),_jsx(_components.p,{children:`Emergent capabilities are abilities that appear in larger models but are absent in smaller ones — not as a gradual improvement, but as a sharp transition. The canonical examples include in-context few-shot learning (appearing around 10B parameters), chain-of-thought reasoning (around 100B), and multi-step tool use (around 500B+). The term draws an analogy to phase transitions in physics, like water freezing at a critical temperature.`}),_jsx(_components.p,{children:`The debate centers on a 2023 paper by Schaeffer et al. that argued emergence is largely an artifact of the metrics used to measure it. Their key insight: when you measure performance using exact-match accuracy (binary: the answer is either exactly right or completely wrong), you see a sharp jump from 0% to near-100% as scale increases. But when you measure using continuous metrics like token-level log-likelihood, the improvement is smooth and predictable across all scales. The “emergence” is in the metric, not the model.`}),_jsx(_components.p,{children:`Their argument is compelling for many benchmarks: if a model’s probability of generating the correct answer gradually increases from 1% to 30% to 70%, exact-match accuracy stays at 0% until the model consistently gets every token right, then jumps to near-100%. The underlying capability was improving all along, but the discontinuous metric hid the gradual progress.`}),_jsx(_components.p,{children:`However, there are genuine counterexamples where the capability itself seems qualitatively new, not just quantitatively better. Multi-step arithmetic, compositional generalization to novel combinations, and self-correction behaviors do appear to require a minimum scale threshold even under continuous metrics. The honest answer is that “emergence” is partly real and partly measurement artifact, and the field has not fully disentangled the two.`}),_jsx(_components.p,{children:`For practitioners, the practical takeaway is: do not assume your model will gain a specific capability by scaling up. Test explicitly at your target scale, and use continuous evaluation metrics whenever possible to get a more accurate picture of where your model actually is on the capability curve.`}),_jsx(_components.p,{children:_jsx(_components.strong,{children:`Follow-up: How would you design an evaluation to determine whether a specific capability in your model is truly emergent versus gradually improving?`})}),_jsx(_components.p,{children:`I would test the model at multiple scales (1B, 3B, 7B, 13B, 70B) using both discrete and continuous metrics for the same underlying task. For example, if testing arithmetic: exact-match accuracy on the full equation (discrete), character-level edit distance from the correct answer (continuous), and per-digit accuracy (semi-continuous). If the discrete metric shows a sharp jump but the continuous metrics show smooth improvement, the “emergence” is a measurement artifact. If both metrics show a sharp phase transition at the same scale, the emergence is more likely real. I would also control for confounds like training data composition and tokenization differences across model sizes, since these can create false emergence signals.`})]})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Foundation Models \u0026 LLMs","sidebarTitle":"Foundation Models","description":"Scaling laws, emergent capabilities, and training large language models","icon":"brain","href":"/courses/deep-learning-mastery/25-foundation-models"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Foundation Models \u0026 LLMs","slug":"foundation-models-\u0026-llms","depth":1,"children":[]},{"title":"The Foundation Model Paradigm","slug":"the-foundation-model-paradigm","depth":2,"children":[]},{"title":"Scaling Laws","slug":"scaling-laws","depth":2,"children":[{"title":"The Chinchilla Scaling Law","slug":"the-chinchilla-scaling-law","depth":3,"children":[]}]},{"title":"LLM Architecture","slug":"llm-architecture","depth":2,"children":[{"title":"Modern Transformer Improvements","slug":"modern-transformer-improvements","depth":3,"children":[]},{"title":"Rotary Position Embeddings (RoPE)","slug":"rotary-position-embeddings-rope","depth":3,"children":[]}]},{"title":"Pretraining Objectives","slug":"pretraining-objectives","depth":2,"children":[{"title":"Causal Language Modeling (GPT-style)","slug":"causal-language-modeling-gpt-style","depth":3,"children":[]},{"title":"Masked Language Modeling (BERT-style)","slug":"masked-language-modeling-bert-style","depth":3,"children":[]}]},{"title":"Emergent Capabilities","slug":"emergent-capabilities","depth":2,"children":[]},{"title":"Training LLMs","slug":"training-llms","depth":2,"children":[{"title":"Distributed Training Setup","slug":"distributed-training-setup","depth":3,"children":[]},{"title":"Instruction Tuning","slug":"instruction-tuning","depth":3,"children":[]}]},{"title":"RLHF (Reinforcement Learning from Human Feedback)","slug":"rlhf-reinforcement-learning-from-human-feedback","depth":2,"children":[{"title":"PPO Training","slug":"ppo-training","depth":3,"children":[]}]},{"title":"Using Foundation Models","slug":"using-foundation-models","depth":2,"children":[]},{"title":"Model Comparison","slug":"model-comparison","depth":2,"children":[]},{"title":"Exercises","slug":"exercises","depth":2,"children":[]},{"title":"What’s Next","slug":"what’s-next","depth":2,"children":[]},{"title":"Interview Deep-Dive","slug":"interview-deep-dive","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"devweeekends","siteName":"Dev Weekends","title":"Foundation Models \u0026 LLMs - Dev Weekends","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-05-16T10:34:53.174Z","originalFileLocation":"courses/deep-learning-mastery/25-foundation-models.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Scaling laws, emergent capabilities, and training large language models\">\u003cmeta data-mint-head name=\"application-name\" content=\"Dev Weekends\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Foundation Models \u0026amp; LLMs - Dev Weekends\">\u003cmeta data-mint-head property=\"og:description\" content=\"Scaling laws, emergent capabilities, and training large language models\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Dev Weekends\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://devweeekends.mintlify.site/_mintlify/api/og/v2?division=Deep+Learning+Mastery\u0026amp;title=Foundation+Models+%26+LLMs\u0026amp;description=Scaling+laws%2C+emergent+capabilities%2C+and+training+large+language+models\u0026amp;theme=97ffacf68ee2a7d59a7fcf65\u0026amp;signature=14c001b308c9966ed345ba4ddfdac346f34cf90684b090979918f09bbea0c559\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Foundation Models \u0026amp; LLMs - Dev Weekends\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Scaling laws, emergent capabilities, and training large language models\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://devweeekends.mintlify.site/_mintlify/api/og/v2?division=Deep+Learning+Mastery\u0026amp;title=Foundation+Models+%26+LLMs\u0026amp;description=Scaling+laws%2C+emergent+capabilities%2C+and+training+large+language+models\u0026amp;theme=97ffacf68ee2a7d59a7fcf65\u0026amp;signature=14c001b308c9966ed345ba4ddfdac346f34cf90684b090979918f09bbea0c559\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Dev Weekends\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#000000\">\u003clink data-mint-head rel=\"canonical\" href=\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/courses/deep-learning-mastery/25-foundation-models.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://resources.devweekends.com/#organization\",\"name\":\"Dev Weekends\",\"url\":\"https://resources.devweekends.com\"},{\"@type\":\"WebSite\",\"@id\":\"https://resources.devweekends.com#website\",\"name\":\"Dev Weekends\",\"url\":\"https://resources.devweekends.com\",\"publisher\":{\"@id\":\"https://resources.devweekends.com/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models#webpage\",\"url\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models\",\"name\":\"Foundation Models \u0026 LLMs\",\"description\":\"Scaling laws, emergent capabilities, and training large language models\",\"dateModified\":\"2026-05-16T10:34:53.174Z\",\"isPartOf\":{\"@id\":\"https://resources.devweekends.com#website\"},\"breadcrumb\":{\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Deep Learning Mastery\",\"item\":\"https://resources.devweekends.com/courses/deep-learning-mastery/00-overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Foundation Models \u0026 LLMs\",\"item\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models#article\",\"headline\":\"Foundation Models \u0026 LLMs\",\"name\":\"Foundation Models \u0026 LLMs\",\"description\":\"Scaling laws, emergent capabilities, and training large language models\",\"url\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models\",\"mainEntityOfPage\":{\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/25-foundation-models#webpage\"},\"dateModified\":\"2026-05-16T10:34:53.174Z\",\"publisher\":{\"@id\":\"https://resources.devweekends.com/#organization\"},\"isPartOf\":{\"@id\":\"https://resources.devweekends.com#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Dev Weekends\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Dev Weekends\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/courses/deep-learning-mastery/25-foundation-models.md","siteName":"Dev Weekends","nav":{"global":{"anchors":[{"anchor":"DevWeekends","icon":"globe","href":"https://devweekends.com"}]},"anchors":[{"anchor":"Resources","icon":"book-open","pages":[{"title":"Resources","href":""}]},{"anchor":"Interview Prep","icon":"clipboard-check","pages":[{"title":"Interview Prep","href":"/resources/interview-experiences"}]},{"anchor":"DSA Patterns","icon":"code","pages":[{"title":"DSA Patterns","href":"/dsa-patterns/overview"}]},{"anchor":"Competitive Programming","icon":"trophy","pages":[{"title":"Competitive Programming","href":"/courses/competitive-programming/00-overview"}]},{"anchor":"Courses","icon":"graduation-cap","groups":[{"group":"Overview","pages":[{"title":"All Courses","icon":"graduation-cap","href":"/courses/overview"}]},{"group":"Testing \u0026 Code Quality","icon":"vial","pages":[{"title":"Sonarqube Mastery","sidebarTitle":"Sonarqube","icon":"magnifying-glass-chart","href":"/courses/testing-tools/sonarqube"},{"title":"Playwright Mastery","sidebarTitle":"Playwright","icon":"masks-theater","href":"/courses/testing-tools/playwright"},{"title":"Mocha Mastery","sidebarTitle":"Mocha","icon":"coffee-bean","href":"/courses/testing-tools/mocha"},{"title":"Jest Mastery","sidebarTitle":"Jest","icon":"jester-hat","href":"/courses/testing-tools/jest"}]},{"group":"Crash Courses","icon":"bolt","pages":[{"group":"C++","icon":"brackets-curly","pages":[{"title":"C++ Crash Course","sidebarTitle":"Overview","icon":"code","href":"/courses/cpp-crash-course/overview"},{"title":"C++ Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/cpp-crash-course/fundamentals"},{"title":"Pointers \u0026 Memory","sidebarTitle":"2. Memory Mastery","icon":"memory","href":"/courses/cpp-crash-course/pointers-memory"},{"title":"Object-Oriented Programming","sidebarTitle":"3. OOP","icon":"object-group","href":"/courses/cpp-crash-course/oop"},{"title":"Standard Template Library","sidebarTitle":"4. The STL","icon":"layer-group","href":"/courses/cpp-crash-course/stl"},{"title":"Modern C++ Features","sidebarTitle":"5. Modern C++","icon":"wand-magic-sparkles","href":"/courses/cpp-crash-course/modern-cpp"}]},{"group":"Java","icon":"mug-hot","pages":[{"title":"Java Crash Course","sidebarTitle":"Overview","icon":"mug-hot","href":"/courses/java-crash-course/overview"},{"title":"Java Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/java-crash-course/fundamentals"},{"title":"Object-Oriented Programming","sidebarTitle":"2. OOP","icon":"object-group","href":"/courses/java-crash-course/oop"},{"title":"Collections Framework","sidebarTitle":"3. Collections","icon":"layer-group","href":"/courses/java-crash-course/collections"},{"title":"Concurrency","sidebarTitle":"4. Concurrency","icon":"bolt","href":"/courses/java-crash-course/concurrency"},{"title":"Modern Java Features","sidebarTitle":"5. Modern Java","icon":"wand-magic-sparkles","href":"/courses/java-crash-course/modern-java"}]},{"group":"Spring Boot \u0026 Microservices","icon":"leaf","pages":[{"title":"Spring Boot \u0026 Microservices","sidebarTitle":"Overview","icon":"leaf","href":"/courses/spring-boot-microservices/00-overview"},{"title":"Spring Boot Quickstart","icon":"bolt","href":"/courses/spring-boot-microservices/01-spring-boot-quickstart"},{"title":"Spring MVC \u0026 REST","icon":"server","href":"/courses/spring-boot-microservices/02-spring-mvc-rest"},{"title":"Data Persistence","icon":"database","href":"/courses/spring-boot-microservices/03-data-persistence"},{"title":"Deep Dive: Spring AOP","icon":"microscope","href":"/courses/spring-boot-microservices/04-spring-aop"},{"title":"Testing Mastery","icon":"vial","href":"/courses/spring-boot-microservices/05-spring-testing"},{"title":"Foundations","icon":"shapes","href":"/courses/spring-boot-microservices/06-microservices-foundations"},{"title":"Service Discovery","icon":"compass","href":"/courses/spring-boot-microservices/07-service-discovery"},{"title":"API Gateway","icon":"door-open","href":"/courses/spring-boot-microservices/08-api-gateway"},{"title":"Config Management","icon":"gears","href":"/courses/spring-boot-microservices/09-config-management"},{"title":"Resilience \u0026 Fault Tolerance","icon":"shield-halved","href":"/courses/spring-boot-microservices/10-resilience"},{"title":"Observability","icon":"monitor-waveform","href":"/courses/spring-boot-microservices/11-observability"},{"title":"Event Driven Architecture","icon":"satellite-dish","href":"/courses/spring-boot-microservices/12-event-driven"},{"title":"Containerization","icon":"docker","href":"/courses/spring-boot-microservices/13-containerization"},{"title":"Capstone Project","icon":"flag-checkered","href":"/courses/spring-boot-microservices/14-capstone"}]},{"group":"JavaScript","icon":"js","pages":[{"title":"JavaScript Crash Course","sidebarTitle":"Overview","icon":"js","href":"/courses/javascript-crash-course/overview"},{"title":"JavaScript Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/javascript-crash-course/fundamentals"},{"title":"Functions \u0026 Scope","sidebarTitle":"2. Functions \u0026 Scope","icon":"function","href":"/courses/javascript-crash-course/functions-scope"},{"title":"Objects \u0026 Prototypes","sidebarTitle":"3. Objects \u0026 Prototypes","icon":"object-group","href":"/courses/javascript-crash-course/objects-prototypes"},{"title":"Async JavaScript","sidebarTitle":"4. Async JavaScript","icon":"clock","href":"/courses/javascript-crash-course/async"},{"title":"Modern JavaScript (ES6+)","sidebarTitle":"5. Modern JavaScript","icon":"wand-magic-sparkles","href":"/courses/javascript-crash-course/modern-js"},{"title":"DOM \u0026 Browser APIs","sidebarTitle":"6. DOM \u0026 Browser","icon":"browser","href":"/courses/javascript-crash-course/dom-browser"}]},{"group":"TypeScript","icon":"file-code","pages":[{"title":"TypeScript Crash Course","sidebarTitle":"Overview","icon":"code","href":"/courses/typescript-crash-course/overview"},{"title":"TypeScript Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/typescript-crash-course/fundamentals"},{"title":"Functions \u0026 Types","sidebarTitle":"2. Functions \u0026 Types","icon":"function","href":"/courses/typescript-crash-course/functions-types"},{"title":"Objects \u0026 Interfaces","sidebarTitle":"3. Objects \u0026 Interfaces","icon":"object-group","href":"/courses/typescript-crash-course/objects-interfaces"},{"title":"Classes \u0026 OOP","sidebarTitle":"4. Classes \u0026 OOP","icon":"sitemap","href":"/courses/typescript-crash-course/classes-oop"},{"title":"Advanced Types","sidebarTitle":"5. Advanced Types","icon":"layer-group","href":"/courses/typescript-crash-course/advanced-types"},{"title":"Generics Deep Dive","sidebarTitle":"6. Generics","icon":"puzzle-piece","href":"/courses/typescript-crash-course/generics"},{"title":"Modules \u0026 Configuration","sidebarTitle":"7. Modules \u0026 Config","icon":"gear","href":"/courses/typescript-crash-course/modules-config"}]},{"group":"Python","icon":"python","pages":[{"title":"Python Crash Course","sidebarTitle":"Overview","icon":"python","href":"/courses/python-crash-course/overview"},{"title":"Python Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/python-crash-course/fundamentals"},{"title":"Data Structures","sidebarTitle":"2. Data Structures","icon":"layer-group","href":"/courses/python-crash-course/data-structures"},{"title":"Object-Oriented Programming","sidebarTitle":"3. OOP","icon":"object-group","href":"/courses/python-crash-course/oop"},{"title":"Modules \u0026 Packages","sidebarTitle":"4. Modules","icon":"box","href":"/courses/python-crash-course/modules-packages"},{"title":"Advanced Python","sidebarTitle":"5. Advanced","icon":"wand-magic-sparkles","href":"/courses/python-crash-course/advanced"}]},{"group":"Node.js","icon":"node","pages":[{"title":"01. Introduction \u0026 Setup","href":"/courses/node-crash-course/01-intro-setup"},{"title":"02. Modules System","href":"/courses/node-crash-course/02-modules"},{"title":"03. File System","href":"/courses/node-crash-course/03-file-system"},{"title":"04. Events \u0026 EventEmitter","href":"/courses/node-crash-course/04-events"},{"title":"05. HTTP Module \u0026 Web Server","href":"/courses/node-crash-course/05-http-server"},{"title":"06. Streams \u0026 Buffers","href":"/courses/node-crash-course/06-streams-buffers"},{"title":"07. NPM (Node Package Manager)","href":"/courses/node-crash-course/07-npm"},{"title":"07b. Building and Publishing an NPM Package","href":"/courses/node-crash-course/07b-building-npm-package"},{"title":"08. Express.js Basics","href":"/courses/node-crash-course/08-express-basics"},{"title":"09. Building a REST API","href":"/courses/node-crash-course/09-rest-api"},{"title":"10. Deployment \u0026 Best Practices","href":"/courses/node-crash-course/10-deployment"},{"title":"11. Authentication with JWT","href":"/courses/node-crash-course/11-auth-jwt"},{"title":"12. MongoDB \u0026 Mongoose ORM","href":"/courses/node-crash-course/12-mongo-mongoose"},{"title":"13. PostgreSQL \u0026 Prisma ORM","href":"/courses/node-crash-course/13-postgres-prisma"},{"title":"14. Error Handling \u0026 Debugging","href":"/courses/node-crash-course/14-error-handling"},{"title":"15. Testing with Jest","href":"/courses/node-crash-course/15-testing"},{"title":"16. Real-Time with WebSockets","href":"/courses/node-crash-course/16-websockets"},{"title":"17. Security Best Practices","href":"/courses/node-crash-course/17-security"},{"title":"18. Performance \u0026 Caching","href":"/courses/node-crash-course/18-performance"},{"title":"19. Microservices Architecture","href":"/courses/node-crash-course/19-microservices"},{"title":"20. TypeScript with Node.js","href":"/courses/node-crash-course/20-typescript"},{"title":"21. Worker Threads \u0026 Child Processes","href":"/courses/node-crash-course/21-worker-threads"},{"title":"22. Capstone Project: Full-Stack API","href":"/courses/node-crash-course/22-capstone"}]},{"group":"React","icon":"react","pages":[{"title":"01. Introduction to React \u0026 JSX","href":"/courses/react-crash-course/01-intro-jsx"},{"title":"02. Components \u0026 Props","href":"/courses/react-crash-course/02-components-props"},{"title":"03. State \u0026 useState Hook","href":"/courses/react-crash-course/03-state-hook"},{"title":"04. Handling Events","href":"/courses/react-crash-course/04-events"},{"title":"05. Lists \u0026 Keys","href":"/courses/react-crash-course/05-lists-keys"},{"title":"06. Forms \u0026 Controlled Components","href":"/courses/react-crash-course/06-forms"},{"title":"07. useEffect \u0026 Side Effects","href":"/courses/react-crash-course/07-useeffect"},{"title":"08. Context API","href":"/courses/react-crash-course/08-context-api"},{"title":"09. React Router","href":"/courses/react-crash-course/09-react-router"},{"title":"10. Performance Optimization","href":"/courses/react-crash-course/10-optimization"},{"title":"11. Authentication \u0026 Protected Routes","href":"/courses/react-crash-course/11-authentication"},{"title":"12. Redux \u0026 State Management","href":"/courses/react-crash-course/12-redux-state"}]},{"group":"React Native","icon":"mobile","pages":[{"title":"React Native Enterprise Mastery","icon":"mobile","href":"/courses/react-native-crash-course/overview"},{"title":"01. Introduction \u0026 Architecture","href":"/courses/react-native-crash-course/01-intro-architecture"},{"title":"02. Environment Setup","href":"/courses/react-native-crash-course/02-environment-setup"},{"title":"03. Project Structure \u0026 Best Practices","href":"/courses/react-native-crash-course/03-project-structure"},{"title":"04. TypeScript in React Native","href":"/courses/react-native-crash-course/04-typescript"},{"title":"05. Core Components Deep Dive","href":"/courses/react-native-crash-course/05-core-components"},{"title":"6. Styling \u0026 Theming","href":"/courses/react-native-crash-course/06-styling-theming"},{"title":"7. Flexbox Mastery","href":"/courses/react-native-crash-course/07-flexbox-mastery"},{"title":"8. Navigation Fundamentals","href":"/courses/react-native-crash-course/08-navigation-fundamentals"},{"title":"9. Advanced Navigation","href":"/courses/react-native-crash-course/09-navigation-advanced"},{"title":"10. State Management","href":"/courses/react-native-crash-course/10-state-management"},{"title":"11. Redux \u0026 Zustand","href":"/courses/react-native-crash-course/11-redux-zustand"},{"title":"15. React Query \u0026 Data Fetching","href":"/courses/react-native-crash-course/15-react-query"},{"title":"27. Unit Testing","href":"/courses/react-native-crash-course/27-unit-testing"},{"title":"31. Security Best Practices","href":"/courses/react-native-crash-course/31-security"},{"title":"35. Profiling \u0026 Monitoring","href":"/courses/react-native-crash-course/35-profiling"},{"title":"40. CI/CD Pipelines","href":"/courses/react-native-crash-course/40-ci-cd"},{"title":"51. Capstone Project","href":"/courses/react-native-crash-course/51-capstone"}]},{"group":"MongoDB","icon":"database","pages":[{"title":"01. Introduction to NoSQL \u0026 MongoDB","href":"/courses/mongodb-crash-course/01-intro-nosql"},{"title":"02. Installation \u0026 MongoDB Compass","href":"/courses/mongodb-crash-course/02-setup-compass"},{"title":"03. Databases \u0026 Collections","href":"/courses/mongodb-crash-course/03-databases-collections"},{"title":"04. CRUD: Create","href":"/courses/mongodb-crash-course/04-crud-create"},{"title":"05. CRUD: Read (Query Operators)","href":"/courses/mongodb-crash-course/05-crud-read"},{"title":"06. CRUD: Update","href":"/courses/mongodb-crash-course/06-crud-update"},{"title":"07. CRUD: Delete","href":"/courses/mongodb-crash-course/07-crud-delete"},{"title":"08. Aggregation Framework","href":"/courses/mongodb-crash-course/08-aggregation"},{"title":"09. Indexing \u0026 Performance","href":"/courses/mongodb-crash-course/09-indexing"},{"title":"10. Mongoose (ODM) Basics","href":"/courses/mongodb-crash-course/10-mongoose"}]},{"group":"Angular Master Course","icon":"angular","pages":[{"title":"Angular Master Course","icon":"angular","href":"/courses/angular-crash-course/00-overview"},{"title":"01. Introduction \u0026 Setup","icon":"rocket","href":"/courses/angular-crash-course/01-introduction"},{"title":"02. Components \u0026 Templates","icon":"puzzle-piece","href":"/courses/angular-crash-course/02-components"},{"title":"03. Directives \u0026 Pipes","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/03-directives-pipes"},{"title":"04. Services \u0026 Dependency Injection","icon":"syringe","href":"/courses/angular-crash-course/04-services-di"},{"title":"05. Angular Signals","icon":"signal","href":"/courses/angular-crash-course/05-signals"},{"title":"06. Routing \u0026 Navigation","icon":"route","href":"/courses/angular-crash-course/06-routing"},{"title":"07. Angular Forms","icon":"input-text","href":"/courses/angular-crash-course/07-forms"},{"title":"08. HTTP Client \u0026 RxJS","icon":"server","href":"/courses/angular-crash-course/08-http-rxjs"},{"title":"09. RxJS Deep Dive","icon":"arrows-rotate","href":"/courses/angular-crash-course/09-rxjs"},{"title":"10. Change Detection \u0026 Performance","icon":"gauge-high","href":"/courses/angular-crash-course/10-change-detection"},{"title":"11. Testing in Angular","icon":"vial","href":"/courses/angular-crash-course/11-testing"},{"title":"12. Advanced Patterns","icon":"diagram-project","href":"/courses/angular-crash-course/12-advanced-patterns"},{"title":"13. Server-Side Rendering","icon":"server","href":"/courses/angular-crash-course/13-ssr"},{"title":"14. Security Best Practices","icon":"shield-halved","href":"/courses/angular-crash-course/14-security"},{"title":"15. Capstone Project","icon":"rocket","href":"/courses/angular-crash-course/15-capstone"},{"title":"16. Standalone Components Deep Dive","icon":"cube","href":"/courses/angular-crash-course/16-standalone"},{"title":"17. NgRx State Management","icon":"database","href":"/courses/angular-crash-course/17-ngrx"},{"title":"18. Angular Animations","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/18-animations"},{"title":"19. Internationalization (i18n)","icon":"globe","href":"/courses/angular-crash-course/19-i18n"},{"title":"20. Accessibility (a11y)","icon":"universal-access","href":"/courses/angular-crash-course/20-accessibility"},{"title":"21. Angular Material \u0026 CDK","icon":"palette","href":"/courses/angular-crash-course/21-material"},{"title":"22. PWA \u0026 Service Workers","icon":"mobile-screen","href":"/courses/angular-crash-course/22-pwa"},{"title":"23. Deployment \u0026 CI/CD","icon":"rocket","href":"/courses/angular-crash-course/23-deployment"},{"title":"24. Micro-frontends with Module Federation","icon":"cubes","href":"/courses/angular-crash-course/24-micro-frontends"},{"title":"25. Nx Monorepo","icon":"boxes-stacked","href":"/courses/angular-crash-course/25-nx-monorepo"},{"title":"26. Performance Optimization","icon":"gauge-high","href":"/courses/angular-crash-course/26-performance"},{"title":"27. Content Projection","icon":"layer-group","href":"/courses/angular-crash-course/27-content-projection"},{"title":"28. Dynamic Components","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/28-dynamic-components"},{"title":"29. Custom Schematics","icon":"toolbox","href":"/courses/angular-crash-course/29-custom-schematics"},{"title":"30. Error Handling","icon":"triangle-exclamation","href":"/courses/angular-crash-course/30-error-handling"},{"title":"31. Interview Questions","icon":"comments-question","href":"/courses/angular-crash-course/31-interview-questions"},{"title":"32. Best Practices","icon":"award","href":"/courses/angular-crash-course/32-best-practices"},{"title":"33. Enterprise Capstone Project","icon":"building","href":"/courses/angular-crash-course/33-enterprise-capstone"}]}]},{"group":"AI Engineering","icon":"robot","pages":[{"title":"AI Engineering","icon":"robot","href":"/ai-engineering/overview"},{"title":"Python Crash Course","icon":"python","href":"/ai-engineering/python-crash-course"},{"title":"FastAPI Crash Course","icon":"bolt","href":"/ai-engineering/fastapi-crash-course"},{"title":"Databases \u0026 ORM Crash Course","icon":"database","href":"/ai-engineering/databases-crash-course"},{"title":"LLM Fundamentals","icon":"brain","href":"/ai-engineering/llm-fundamentals"},{"title":"Prompt Engineering","icon":"wand-magic-sparkles","href":"/ai-engineering/prompt-engineering"},{"title":"OpenAI API","icon":"bolt","href":"/ai-engineering/openai-api"},{"title":"Structured Outputs","icon":"brackets-curly","href":"/ai-engineering/structured-outputs"},{"title":"Embeddings Deep Dive","icon":"vector-square","href":"/ai-engineering/embeddings"},{"title":"AI Streaming","icon":"water","href":"/ai-engineering/streaming"},{"title":"Vector Databases","icon":"database","href":"/ai-engineering/vector-databases"},{"title":"RAG Systems","icon":"magnifying-glass","href":"/ai-engineering/rag"},{"title":"Types of RAG","icon":"layer-group","href":"/ai-engineering/rag-types"},{"title":"Fine-Tuning LLMs","icon":"sliders","href":"/ai-engineering/fine-tuning"},{"title":"AI Agents","icon":"robot","href":"/ai-engineering/agents"},{"title":"Tool Calling \u0026 Function Calling","icon":"wrench","href":"/ai-engineering/tool-calling"},{"title":"LLM Memory Systems","icon":"brain","href":"/ai-engineering/llm-memory-systems"},{"title":"Multi-Agent Design Patterns","icon":"users-gear","href":"/ai-engineering/multi-agent-patterns"},{"title":"LangChain","icon":"link","href":"/ai-engineering/langchain"},{"title":"LangGraph","icon":"diagram-project","href":"/ai-engineering/langgraph"},{"title":"Model Context Protocol (MCP)","icon":"plug","href":"/ai-engineering/mcp"},{"title":"Agentic Architecture","icon":"sitemap","href":"/ai-engineering/agentic-architecture"},{"title":"Open Source \u0026 Local LLMs","icon":"server","href":"/ai-engineering/open-source-llms"},{"title":"Multimodal AI","icon":"photo-film","href":"/ai-engineering/multimodal-ai"},{"title":"DSPy Framework","icon":"code","href":"/ai-engineering/dspy"},{"title":"Prompt Management \u0026 Versioning","icon":"code-branch","href":"/ai-engineering/prompt-management"},{"title":"LLM Caching","icon":"database","href":"/ai-engineering/llm-caching"},{"title":"Error Handling \u0026 Resilience","icon":"shield-halved","href":"/ai-engineering/error-handling"},{"title":"Batch Processing \u0026 Rate Limiting","icon":"layer-group","href":"/ai-engineering/batch-processing"},{"title":"LLM Fallbacks \u0026 Model Routing","icon":"arrows-split-up-and-left","href":"/ai-engineering/llm-fallbacks"},{"title":"AI Observability \u0026 Monitoring","icon":"chart-line","href":"/ai-engineering/observability-monitoring"},{"title":"AI Security \u0026 Guardrails","icon":"shield-halved","href":"/ai-engineering/ai-security-guardrails"},{"title":"Cost Optimization \u0026 Token Management","icon":"coins","href":"/ai-engineering/cost-optimization"},{"title":"Function Calling Deep Dive","icon":"function","href":"/ai-engineering/function-calling"},{"title":"LLM Orchestration","icon":"sitemap","href":"/ai-engineering/llm-orchestration"},{"title":"Semantic Search \u0026 Retrieval","icon":"magnifying-glass","href":"/ai-engineering/semantic-search"},{"title":"Context Window Management","icon":"window-maximize","href":"/ai-engineering/context-management"},{"title":"Testing LLM Applications","icon":"flask-vial","href":"/ai-engineering/llm-testing"},{"title":"Production Logging \u0026 Debugging","icon":"file-lines","href":"/ai-engineering/production-logging"},{"title":"API Design for AI Applications","icon":"code","href":"/ai-engineering/api-design"},{"title":"Document Processing","icon":"file-lines","href":"/ai-engineering/document-processing"},{"title":"Model Comparison","icon":"scale-balanced","href":"/ai-engineering/model-comparison"},{"title":"Anthropic Claude API","icon":"message-bot","href":"/ai-engineering/anthropic-api"},{"title":"Async Patterns for LLMs","icon":"arrows-spin","href":"/ai-engineering/async-patterns"},{"title":"Advanced Prompting Techniques","icon":"brain","href":"/ai-engineering/advanced-prompting"},{"title":"Output Validation","icon":"shield-check","href":"/ai-engineering/output-validation"},{"title":"Semantic Routing","icon":"route","href":"/ai-engineering/semantic-routing"},{"title":"Advanced RAG Techniques","icon":"layer-group","href":"/ai-engineering/advanced-rag"},{"title":"Voice and Audio AI","icon":"microphone","href":"/ai-engineering/voice-audio"},{"title":"Image Generation","icon":"image","href":"/ai-engineering/image-generation"},{"title":"Summarization Systems","icon":"compress","href":"/ai-engineering/summarization"},{"title":"Chatbot Design Patterns","icon":"comments","href":"/ai-engineering/chatbot-patterns"},{"title":"Knowledge Graphs with LLMs","icon":"diagram-project","href":"/ai-engineering/knowledge-graphs"},{"title":"Synthetic Data Generation","icon":"database","href":"/ai-engineering/synthetic-data"},{"title":"Text Classification","icon":"tags","href":"/ai-engineering/classification"},{"title":"Model Optimization","icon":"gauge-high","href":"/ai-engineering/model-optimization"},{"title":"Realtime AI Applications","icon":"bolt","href":"/ai-engineering/realtime-ai"},{"title":"Evaluation \u0026 Testing","icon":"vial","href":"/ai-engineering/evaluation-testing"},{"title":"Deployment \u0026 Scaling","icon":"rocket","href":"/ai-engineering/deployment-scaling"},{"title":"Capstone Project","icon":"trophy","href":"/ai-engineering/capstone-project"}]},{"group":"Math for ML - Understanding Linear Algebra","icon":"square-root-variable","pages":[{"title":"Introduction to Linear Algebra for ML","sidebarTitle":"Introduction","icon":"rocket","href":"/courses/math-for-ml-linear-algebra/01-introduction"},{"title":"Vectors: The Language of Similarity","sidebarTitle":"Vectors","icon":"arrow-right","href":"/courses/math-for-ml-linear-algebra/02-vectors"},{"title":"Matrices \u0026 Linear Transformations","sidebarTitle":"Matrices","icon":"grid","href":"/courses/math-for-ml-linear-algebra/03-matrices"},{"title":"Project: Image Similarity Search Engine","sidebarTitle":"Project 1","icon":"code","href":"/courses/math-for-ml-linear-algebra/04-project-image-search"},{"title":"Eigenvalues \u0026 Eigenvectors","sidebarTitle":"Eigenvalues","icon":"compass","href":"/courses/math-for-ml-linear-algebra/05-eigenvalues"},{"title":"Principal Component Analysis (PCA)","sidebarTitle":"PCA","icon":"compress","href":"/courses/math-for-ml-linear-algebra/06-pca"},{"title":"Singular Value Decomposition (SVD)","sidebarTitle":"SVD","icon":"layer-group","href":"/courses/math-for-ml-linear-algebra/07-svd"},{"title":"Linear Systems \u0026 Applications","sidebarTitle":"Linear Systems","icon":"diagram-project","href":"/courses/math-for-ml-linear-algebra/08-linear-systems"},{"title":"Project: Music Recommendation Engine","sidebarTitle":"Project 2","icon":"music","href":"/courses/math-for-ml-linear-algebra/09-project-recommender"},{"title":"Orthogonality \u0026 Projections","sidebarTitle":"Orthogonality","icon":"right-left","href":"/courses/math-for-ml-linear-algebra/10-orthogonality"}]},{"group":"Probability \u0026 Statistics for ML","icon":"chart-bar","pages":[{"title":"Probability and Statistics for Machine Learning","sidebarTitle":"Introduction","icon":"chart-line","href":"/courses/statistics-for-ml/01-introduction"},{"title":"Describing Data: What's Normal?","sidebarTitle":"Describing Data","icon":"chart-bar","href":"/courses/statistics-for-ml/02-describing-data"},{"title":"Probability Foundations: How Likely Is This?","sidebarTitle":"Probability","icon":"dice","href":"/courses/statistics-for-ml/03-probability"},{"title":"Probability Distributions: Patterns in Randomness","sidebarTitle":"Distributions","icon":"chart-bell","href":"/courses/statistics-for-ml/04-distributions"},{"title":"Statistical Inference: Conclusions from Samples","sidebarTitle":"Inference","icon":"magnifying-glass-chart","href":"/courses/statistics-for-ml/05-inference"},{"title":"Hypothesis Testing: Real Effect or Random Noise?","sidebarTitle":"Hypothesis Testing","icon":"scale-balanced","href":"/courses/statistics-for-ml/06-hypothesis-testing"},{"title":"Correlation and Regression: Relationships and Predictions","sidebarTitle":"Regression","icon":"chart-line-up","href":"/courses/statistics-for-ml/07-regression"},{"title":"From Statistics to Machine Learning","sidebarTitle":"Stats to ML","icon":"brain","href":"/courses/statistics-for-ml/08-statistics-to-ml"},{"title":"Bayesian Statistics","sidebarTitle":"Bayesian Statistics","icon":"brain-circuit","href":"/courses/statistics-for-ml/09-bayesian-statistics"},{"title":"Time Series Fundamentals","sidebarTitle":"Time Series","icon":"chart-line","href":"/courses/statistics-for-ml/10-time-series-basics"},{"title":"Capstone Project: Complete Statistical Analysis","sidebarTitle":"Capstone Project","icon":"graduation-cap","href":"/courses/statistics-for-ml/11-capstone-project"}]},{"group":"Math for ML - Understanding Calculus","icon":"function","pages":[{"title":"Calculus for Machine Learning","sidebarTitle":"Introduction","icon":"function","href":"/courses/math-for-ml-calculus/00-introduction"},{"title":"Derivatives \u0026 Rates of Change","sidebarTitle":"Derivatives","icon":"chart-line","href":"/courses/math-for-ml-calculus/01-derivatives"},{"title":"Gradients \u0026 Multivariable Calculus","sidebarTitle":"Gradients","icon":"mountain","href":"/courses/math-for-ml-calculus/02-gradients"},{"title":"Chain Rule \u0026 Backpropagation","sidebarTitle":"Chain Rule","icon":"link","href":"/courses/math-for-ml-calculus/03-chain-rule"},{"title":"Gradient Descent","sidebarTitle":"Gradient Descent","icon":"arrow-trend-down","href":"/courses/math-for-ml-calculus/04-gradient-descent"},{"title":"Optimization Techniques","sidebarTitle":"Optimization","icon":"rocket","href":"/courses/math-for-ml-calculus/05-optimization"},{"title":"Final Project: Build a Neural Network","sidebarTitle":"Final Project","icon":"flag-checkered","href":"/courses/math-for-ml-calculus/06-final-project"},{"title":"Automatic Differentiation","sidebarTitle":"Autodiff","icon":"wand-magic-sparkles","href":"/courses/math-for-ml-calculus/07-automatic-differentiation"},{"title":"Convex Optimization","sidebarTitle":"Convexity","icon":"bowl-rice","href":"/courses/math-for-ml-calculus/08-convexity"},{"title":"Real-World Applications","sidebarTitle":"Applications","icon":"industry","href":"/courses/math-for-ml-calculus/09-real-world-applications"}]},{"group":"ML Mastery","icon":"brain","pages":[{"title":"Machine Learning Mastery","sidebarTitle":"Introduction","icon":"brain","href":"/courses/ml-mastery/00-introduction"},{"title":"The Prediction Game","sidebarTitle":"Prediction Game","icon":"crosshairs","href":"/courses/ml-mastery/01-prediction-game"},{"title":"Learning From Mistakes","sidebarTitle":"Learning From Mistakes","icon":"graduation-cap","href":"/courses/ml-mastery/02-learning-from-mistakes"},{"title":"Linear Regression","sidebarTitle":"Linear Regression","icon":"chart-line","href":"/courses/ml-mastery/03-linear-regression"},{"title":"Classification","sidebarTitle":"Classification","icon":"tags","href":"/courses/ml-mastery/04-classification"},{"title":"K-Nearest Neighbors","sidebarTitle":"K-Nearest Neighbors","icon":"people-arrows","href":"/courses/ml-mastery/04a-knn"},{"title":"Decision Trees","sidebarTitle":"Decision Trees","icon":"sitemap","href":"/courses/ml-mastery/05-decision-trees"},{"title":"Support Vector Machines","sidebarTitle":"SVM","icon":"border-none","href":"/courses/ml-mastery/05a-svm"},{"title":"Naive Bayes","sidebarTitle":"Naive Bayes","icon":"percent","href":"/courses/ml-mastery/05b-naive-bayes"},{"title":"Ensemble Methods","sidebarTitle":"Ensemble Methods","icon":"people-group","href":"/courses/ml-mastery/06-ensemble-methods"},{"title":"Model Evaluation","sidebarTitle":"Model Evaluation","icon":"chart-column","href":"/courses/ml-mastery/07-model-evaluation"},{"title":"Feature Engineering","sidebarTitle":"Feature Engineering","icon":"wand-magic-sparkles","href":"/courses/ml-mastery/08-feature-engineering"},{"title":"Hyperparameter Tuning","sidebarTitle":"Hyperparameter Tuning","icon":"sliders","href":"/courses/ml-mastery/09-hyperparameter-tuning"},{"title":"End-to-End ML Project","sidebarTitle":"End-to-End Project","icon":"rocket","href":"/courses/ml-mastery/10-end-to-end-project"},{"title":"Clustering","sidebarTitle":"Clustering","icon":"object-group","href":"/courses/ml-mastery/11-clustering"},{"title":"Neural Networks","sidebarTitle":"Neural Networks","icon":"network-wired","href":"/courses/ml-mastery/12-neural-networks"},{"title":"Regularization","sidebarTitle":"Regularization","icon":"shield-halved","href":"/courses/ml-mastery/13-regularization"},{"title":"Model Deployment","sidebarTitle":"Deployment","icon":"rocket","href":"/courses/ml-mastery/14-model-deployment"},{"title":"Time Series Forecasting","sidebarTitle":"Time Series","icon":"clock","href":"/courses/ml-mastery/15-time-series"},{"title":"Bias-Variance Tradeoff","sidebarTitle":"Bias-Variance","icon":"scale-balanced","href":"/courses/ml-mastery/16-bias-variance"},{"title":"Data Leakage","sidebarTitle":"Data Leakage","icon":"droplet","href":"/courses/ml-mastery/17-data-leakage"},{"title":"Dimensionality Reduction","sidebarTitle":"Dimensionality Reduction","icon":"compress","href":"/courses/ml-mastery/18-dimensionality-reduction"},{"title":"Capstone Project","sidebarTitle":"Capstone Project","icon":"graduation-cap","href":"/courses/ml-mastery/19-capstone-project"},{"title":"Handling Imbalanced Data","sidebarTitle":"Imbalanced Data","icon":"scale-unbalanced","href":"/courses/ml-mastery/20-imbalanced-data"},{"title":"Model Explainability","sidebarTitle":"Explainability","icon":"lightbulb","href":"/courses/ml-mastery/21-explainability"},{"title":"ML Pipelines","sidebarTitle":"ML Pipelines","icon":"diagram-project","href":"/courses/ml-mastery/22-ml-pipelines"},{"title":"Common ML Mistakes","sidebarTitle":"Common Mistakes","icon":"triangle-exclamation","href":"/courses/ml-mastery/23-common-mistakes"},{"title":"Cross-Validation Strategies","sidebarTitle":"Cross-Validation","icon":"layer-group","href":"/courses/ml-mastery/24-cross-validation"},{"title":"Anomaly Detection","sidebarTitle":"Anomaly Detection","icon":"magnifying-glass-chart","href":"/courses/ml-mastery/25-anomaly-detection"},{"title":"ML at Scale","sidebarTitle":"ML at Scale","icon":"server","href":"/courses/ml-mastery/26-ml-at-scale"}]},{"group":"Deep Learning Mastery","icon":"network-wired","pages":[{"title":"Deep Learning Mastery","sidebarTitle":"Overview","icon":"brain-circuit","href":"/courses/deep-learning-mastery/00-overview"},{"title":"The Deep Learning Landscape","sidebarTitle":"Deep Learning Landscape","icon":"map","href":"/courses/deep-learning-mastery/01-landscape"},{"title":"Perceptrons \u0026 Multi-Layer Networks","sidebarTitle":"Perceptrons \u0026 MLPs","icon":"circle-nodes","href":"/courses/deep-learning-mastery/02-perceptrons-mlp"},{"title":"Backpropagation Deep Dive","sidebarTitle":"Backpropagation","icon":"arrows-rotate","href":"/courses/deep-learning-mastery/03-backpropagation"},{"title":"Activation Functions","sidebarTitle":"Activation Functions","icon":"bolt","href":"/courses/deep-learning-mastery/04-activation-functions"},{"title":"Loss Functions \u0026 Objectives","sidebarTitle":"Loss Functions","icon":"crosshairs","href":"/courses/deep-learning-mastery/05-loss-functions"},{"title":"Convolutional Neural Networks","sidebarTitle":"CNNs","icon":"layer-group","href":"/courses/deep-learning-mastery/06-cnns"},{"title":"Pooling, Stride \u0026 CNN Design","sidebarTitle":"CNN Design","icon":"layer-group","href":"/courses/deep-learning-mastery/07-cnn-design"},{"title":"Recurrent Neural Networks","sidebarTitle":"RNNs","icon":"rotate-right","href":"/courses/deep-learning-mastery/08-rnns"},{"title":"LSTMs \u0026 GRUs","sidebarTitle":"LSTMs \u0026 GRUs","icon":"memory","href":"/courses/deep-learning-mastery/09-lstms-grus"},{"title":"Attention Mechanism","sidebarTitle":"Attention","icon":"eye","href":"/courses/deep-learning-mastery/10-attention"},{"title":"Transformers","sidebarTitle":"Transformers","icon":"bolt","href":"/courses/deep-learning-mastery/11-transformers"},{"title":"Generative Adversarial Networks","sidebarTitle":"GANs","icon":"masks-theater","href":"/courses/deep-learning-mastery/12-gans"},{"title":"Autoencoders \u0026 VAEs","sidebarTitle":"Autoencoders","icon":"compress","href":"/courses/deep-learning-mastery/13-autoencoders"},{"title":"Diffusion Models","sidebarTitle":"Diffusion Models","icon":"cloud","href":"/courses/deep-learning-mastery/14-diffusion"},{"title":"Residual \u0026 Skip Connections","sidebarTitle":"Residual Networks","icon":"diagram-project","href":"/courses/deep-learning-mastery/15-residual-networks"},{"title":"Normalization Techniques","sidebarTitle":"Normalization","icon":"sliders","href":"/courses/deep-learning-mastery/16-normalization"},{"title":"Regularization Techniques","sidebarTitle":"Regularization","icon":"shield","href":"/courses/deep-learning-mastery/17-regularization"},{"title":"Optimization Algorithms","sidebarTitle":"Optimizers","icon":"gauge-high","href":"/courses/deep-learning-mastery/18-optimization"},{"title":"Training Strategies at Scale","sidebarTitle":"Scale Training","icon":"server","href":"/courses/deep-learning-mastery/19-training-strategies"},{"title":"Transfer Learning \u0026 Fine-tuning","sidebarTitle":"Transfer Learning","icon":"arrows-repeat","href":"/courses/deep-learning-mastery/20-transfer-learning"},{"title":"Model Deployment","sidebarTitle":"Deployment","icon":"rocket","href":"/courses/deep-learning-mastery/21-deployment"},{"title":"Debugging Deep Learning","sidebarTitle":"Debugging","icon":"bug","href":"/courses/deep-learning-mastery/22-debugging"},{"title":"Vision Transformers","sidebarTitle":"Vision Transformers","icon":"eye","href":"/courses/deep-learning-mastery/23-vision-transformers"},{"title":"Multimodal Models","sidebarTitle":"Multimodal","description":"CLIP, vision-language models, and connecting modalities","icon":"images","href":"/courses/deep-learning-mastery/24-multimodal"},{"title":"Foundation Models \u0026 LLMs","sidebarTitle":"Foundation Models","icon":"brain","href":"/courses/deep-learning-mastery/25-foundation-models"},{"title":"Capstone Project","sidebarTitle":"Capstone","description":"Build a complete deep learning system from scratch","icon":"flag-checkered","href":"/courses/deep-learning-mastery/26-capstone"},{"title":"Mathematical Foundations for Deep Learning","sidebarTitle":"Math Foundations","icon":"square-root-variable","href":"/courses/deep-learning-mastery/27-math-foundations"},{"title":"Weight Initialization Strategies","sidebarTitle":"Weight Initialization","icon":"seedling","href":"/courses/deep-learning-mastery/28-weight-initialization"},{"title":"Gradient Flow Analysis","sidebarTitle":"Gradient Flow","icon":"wave-pulse","href":"/courses/deep-learning-mastery/29-gradient-flow"},{"title":"Advanced CNN Architectures","sidebarTitle":"CNN Architectures","icon":"layer-group","href":"/courses/deep-learning-mastery/30-cnn-architectures"},{"title":"Sequence-to-Sequence Models","sidebarTitle":"Seq2Seq","icon":"arrows-left-right","href":"/courses/deep-learning-mastery/31-seq2seq"},{"title":"Self-Supervised Learning","sidebarTitle":"Self-Supervised Learning","icon":"brain","href":"/courses/deep-learning-mastery/32-self-supervised"},{"title":"Reinforcement Learning for Deep Learning","sidebarTitle":"RL for Deep Learning","icon":"gamepad","href":"/courses/deep-learning-mastery/33-rl-for-dl"},{"title":"Neural Architecture Search","sidebarTitle":"Neural Architecture Search","icon":"magnifying-glass","href":"/courses/deep-learning-mastery/34-nas"},{"title":"Interpretability \u0026 Explainability","sidebarTitle":"Interpretability","icon":"eye","href":"/courses/deep-learning-mastery/35-interpretability"},{"title":"Adversarial Robustness","sidebarTitle":"Adversarial ML","icon":"shield-halved","href":"/courses/deep-learning-mastery/36-adversarial-robustness"},{"title":"Efficient Neural Network Architectures","sidebarTitle":"Efficient Architectures","icon":"bolt","href":"/courses/deep-learning-mastery/37-efficient-architectures"},{"title":"Graph Neural Networks","sidebarTitle":"Graph Networks","icon":"diagram-project","href":"/courses/deep-learning-mastery/38-graph-neural-networks"},{"title":"3D Deep Learning","sidebarTitle":"3D Deep Learning","icon":"cube","href":"/courses/deep-learning-mastery/39-3d-deep-learning"},{"title":"Object Detection Deep Dive","sidebarTitle":"Object Detection","icon":"vector-square","href":"/courses/deep-learning-mastery/40-object-detection"},{"title":"Semantic Segmentation","sidebarTitle":"Segmentation","icon":"object-group","href":"/courses/deep-learning-mastery/41-semantic-segmentation"},{"title":"Hyperparameter Tuning","sidebarTitle":"Hyperparameter Tuning","icon":"sliders","href":"/courses/deep-learning-mastery/42-hyperparameter-tuning"},{"title":"Reproducibility","sidebarTitle":"Reproducibility","icon":"repeat","href":"/courses/deep-learning-mastery/43-reproducibility"},{"title":"Memory-Efficient Training","sidebarTitle":"Memory-Efficient Training","icon":"memory","href":"/courses/deep-learning-mastery/44-memory-efficient-training"},{"title":"Quantization Deep Dive","sidebarTitle":"Quantization Deep Dive","icon":"compress","href":"/courses/deep-learning-mastery/45-quantization-deep-dive"},{"title":"Knowledge Distillation","sidebarTitle":"Knowledge Distillation","icon":"school","href":"/courses/deep-learning-mastery/46-knowledge-distillation"},{"title":"Continual Learning","sidebarTitle":"Continual Learning","icon":"arrows-rotate","href":"/courses/deep-learning-mastery/47-continual-learning"}]},{"group":"NestJS Mastery","icon":"feather","pages":[{"title":"NestJS Mastery","sidebarTitle":"Course Overview","icon":"node","href":"/courses/nestjs/overview"},{"title":"NestJS Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/nestjs/fundamentals"},{"title":"Dependency Injection","sidebarTitle":"2. DI \u0026 Modules","icon":"puzzle-piece","href":"/courses/nestjs/dependency-injection"},{"title":"Controllers \u0026 Routing","sidebarTitle":"3. Controllers","icon":"route","href":"/courses/nestjs/controllers-routes"},{"title":"Providers \u0026 Services","sidebarTitle":"4. Providers","icon":"gear","href":"/courses/nestjs/providers-services"},{"title":"Database Integration","sidebarTitle":"5. Database","icon":"database","href":"/courses/nestjs/database-integration"},{"title":"Authentication \u0026 Authorization","sidebarTitle":"6. Auth","icon":"lock","href":"/courses/nestjs/authentication-authorization"},{"title":"Testing","sidebarTitle":"7. Testing","icon":"flask","href":"/courses/nestjs/testing"},{"title":"Microservices","sidebarTitle":"8. Microservices","icon":"cubes","href":"/courses/nestjs/microservices"},{"title":"Deployment \u0026 Production","sidebarTitle":"9. Deployment","icon":"rocket","href":"/courses/nestjs/deployment-production"},{"title":"Advanced Patterns","sidebarTitle":"10. Advanced","icon":"star","href":"/courses/nestjs/advanced-patterns"}]},{"group":"Microservices Mastery","icon":"cubes","pages":[{"title":"Microservices Mastery","sidebarTitle":"Course Overview","icon":"cubes","href":"/courses/microservices-mastery/00-overview"},{"title":"1. Microservices Foundations","href":"/courses/microservices-mastery/01-foundations"},{"title":"2. Domain-Driven Design","href":"/courses/microservices-mastery/02-domain-driven-design"},{"title":"3. Synchronous Communication","href":"/courses/microservices-mastery/03-sync-communication"},{"title":"4. Asynchronous Communication","href":"/courses/microservices-mastery/04-async-communication"},{"title":"5. API Gateway Pattern","href":"/courses/microservices-mastery/05-api-gateway"},{"title":"6. Data Management Patterns","href":"/courses/microservices-mastery/06-data-management"},{"title":"7. Resilience Patterns","href":"/courses/microservices-mastery/07-resilience-patterns"},{"title":"8. Service Discovery","href":"/courses/microservices-mastery/08-service-discovery"},{"title":"9. Observability","href":"/courses/microservices-mastery/09-observability"},{"title":"10. Security","href":"/courses/microservices-mastery/10-security"},{"title":"11. Containerization with Docker","href":"/courses/microservices-mastery/11-containerization"},{"title":"12. Kubernetes Deployment","href":"/courses/microservices-mastery/12-kubernetes"},{"title":"13. Testing Strategies","href":"/courses/microservices-mastery/13-testing"},{"title":"14. Interview Preparation","href":"/courses/microservices-mastery/14-interview-prep"},{"title":"15. Capstone Project","href":"/courses/microservices-mastery/15-capstone"},{"title":"16. Service Mesh","href":"/courses/microservices-mastery/16-service-mesh"},{"title":"17. Configuration Management","href":"/courses/microservices-mastery/17-configuration-management"},{"title":"18. CI/CD for Microservices","href":"/courses/microservices-mastery/18-cicd"},{"title":"19. Database Patterns","href":"/courses/microservices-mastery/19-database-patterns"},{"title":"20. Caching Strategies","href":"/courses/microservices-mastery/20-caching-strategies"},{"title":"21. Chaos Engineering","href":"/courses/microservices-mastery/21-chaos-engineering"},{"title":"22. Case Studies","href":"/courses/microservices-mastery/22-case-studies"},{"title":"23. Load Balancing Deep Dive","href":"/courses/microservices-mastery/23-load-balancing"},{"title":"24. Migration Patterns","href":"/courses/microservices-mastery/24-migration-patterns"},{"title":"25. Event Sourcing Deep Dive","href":"/courses/microservices-mastery/25-event-sourcing-deep-dive"},{"title":"26. GraphQL Federation","href":"/courses/microservices-mastery/26-graphql-federation"}]},{"group":"Low Level Design","pages":[{"title":"Low Level Design (LLD)","icon":"cubes","href":"/lld/overview"},{"title":"LLD Interview Cheat Sheet","icon":"file-lines","href":"/lld/interview-cheatsheet"}]},{"group":"OOP Concepts","pages":[{"title":"What is OOP?","icon":"graduation-cap","href":"/lld/oop/introduction"},{"title":"Encapsulation","icon":"box","href":"/lld/oop/encapsulation"},{"title":"Inheritance","icon":"sitemap","href":"/lld/oop/inheritance"},{"title":"Polymorphism","icon":"shapes","href":"/lld/oop/polymorphism"},{"title":"Abstraction","icon":"eye-slash","href":"/lld/oop/abstraction"},{"title":"OOP Concepts","icon":"object-group","href":"/lld/oop-concepts"}]},{"group":"SOLID Principles","pages":[{"title":"SOLID Introduction","icon":"wand-magic-sparkles","href":"/lld/solid/introduction"},{"title":"Single Responsibility Principle","icon":"user-check","href":"/lld/solid/srp"},{"title":"Open/Closed Principle","icon":"plug","href":"/lld/solid/ocp"},{"title":"Liskov Substitution Principle","icon":"right-left","href":"/lld/solid/lsp"},{"title":"Interface Segregation Principle","icon":"puzzle-piece","href":"/lld/solid/isp"},{"title":"Dependency Inversion Principle","icon":"outlet","href":"/lld/solid/dip"},{"title":"SOLID Principles in Practice","icon":"compass-drafting","href":"/lld/solid-practice"}]},{"group":"Design Patterns","pages":[{"title":"Design Patterns","icon":"shapes","href":"/lld/design-patterns"},{"title":"UML Diagrams","icon":"diagram-project","href":"/lld/uml-diagrams"}]},{"group":"LLD Case Studies","pages":[{"title":"Case Study: Parking Lot System","icon":"square-parking","href":"/lld/case-parking-lot"},{"title":"Case Study: Library Management System","icon":"book","href":"/lld/case-library"},{"title":"Design Vending Machine","icon":"box","href":"/lld/case-vending-machine"},{"title":"Design LRU Cache","icon":"database","href":"/lld/case-lru-cache"},{"title":"Design Movie Ticket Booking System","icon":"ticket","href":"/lld/case-movie-ticket"},{"title":"Case Study: Elevator System","icon":"elevator","href":"/lld/case-elevator"},{"title":"Case Study: Chess Game","icon":"chess","href":"/lld/case-chess"},{"title":"Case Study: Hotel Booking System","icon":"hotel","href":"/lld/case-hotel-booking"},{"title":"Case Study: ATM System","icon":"building-columns","href":"/lld/case-atm"}]},{"group":"System Design (HLD)","pages":[{"title":"System Design Mastery","icon":"network-wired","href":"/system-design/overview"},{"title":"Interview Guide \u0026 Cheatsheet","icon":"clipboard-check","href":"/system-design/interview-guide"},{"title":"Interview Questions Bank","icon":"clipboard-question","href":"/system-design/interview-questions"},{"title":"System Design Fundamentals","icon":"book","href":"/system-design/fundamentals"},{"title":"System Design Building Blocks","icon":"cubes","href":"/system-design/building-blocks"},{"title":"Networking Fundamentals","icon":"network-wired","href":"/system-design/networking"},{"title":"Database Design","icon":"database","href":"/system-design/databases"},{"title":"API Design","icon":"plug","href":"/system-design/api-design"},{"title":"Distributed Systems","icon":"share-nodes","href":"/system-design/distributed-systems"},{"title":"Microservices Architecture","icon":"cubes","href":"/system-design/microservices"},{"title":"Back-of-Envelope Calculations","icon":"calculator","href":"/system-design/estimations"}]},{"group":"Senior Level (L5+/Staff)","pages":[{"title":"Advanced Concepts","icon":"brain","href":"/system-design/advanced-concepts"},{"title":"Scalability Patterns","icon":"arrow-up-right-dots","href":"/system-design/scalability-patterns"},{"title":"Reliability \u0026 Fault Tolerance","icon":"shield-halved","href":"/system-design/reliability-patterns"},{"title":"Observability \u0026 Monitoring","icon":"chart-line","href":"/system-design/observability"},{"title":"Data Pipelines","icon":"pipe","href":"/system-design/data-pipelines"},{"title":"Security Patterns","icon":"shield","href":"/system-design/security-patterns"}]},{"group":"HLD Case Studies","pages":[{"title":"Design: URL Shortener","icon":"link","href":"/system-design/case-url-shortener"},{"title":"Design: Rate Limiter","icon":"gauge","href":"/system-design/case-rate-limiter"},{"title":"Design a Notification System","icon":"bell","href":"/system-design/case-notification-system"},{"title":"Design: Twitter/X Timeline","icon":"twitter","href":"/system-design/case-twitter"},{"title":"Design: WhatsApp/Messenger","icon":"message","href":"/system-design/case-whatsapp"},{"title":"Design a Payment System","icon":"credit-card","href":"/system-design/case-payment-system"},{"title":"Design: Netflix/YouTube","icon":"video","href":"/system-design/case-netflix"},{"title":"Design: Uber/Lyft","icon":"car","href":"/system-design/case-uber"},{"title":"Design Google Maps","icon":"map-location-dot","href":"/system-design/case-google-maps"}]},{"group":"Engineering Fundamentals","pages":[{"title":"Engineering Topics","icon":"gears","href":"/engineering/overview"},{"title":"Computer Science Fundamentals","icon":"microchip","href":"/engineering/cs-fundamentals"},{"title":"Software Design Principles","icon":"compass-drafting","href":"/engineering/design-principles"},{"title":"Architecture Patterns","icon":"building","href":"/engineering/architecture-patterns"}]},{"group":"DevOps \u0026 Operations","pages":[{"title":"DevOps \u0026 Infrastructure","icon":"server","href":"/engineering/devops"},{"title":"Security Fundamentals","icon":"shield-halved","href":"/engineering/security"},{"title":"Performance \u0026 Optimization","icon":"gauge-high","href":"/engineering/performance"}]},{"group":"Azure Cloud Engineering","icon":"cloud","pages":[{"title":"Azure Cloud Engineering","sidebarTitle":"Overview","icon":"cloud","href":"/courses/azure-cloud-engineering/00-overview"},{"title":"Azure Fundamentals \u0026 Architecture","icon":"cloud-arrow-up","href":"/courses/azure-cloud-engineering/01-azure-fundamentals"},{"title":"Identity \u0026 Access Management","icon":"user-shield","href":"/courses/azure-cloud-engineering/02-identity-access-management"},{"title":"03a. Networking Fundamentals","icon":"network-wired","href":"/courses/azure-cloud-engineering/03a-networking-fundamentals"},{"title":"03b. Hybrid \u0026 Enterprise","icon":"globe","href":"/courses/azure-cloud-engineering/03b-hybrid-networking"},{"title":"03c. Traffic \u0026 Security","icon":"traffic-light","href":"/courses/azure-cloud-engineering/03c-traffic-management"},{"title":"Compute Services","icon":"server","href":"/courses/azure-cloud-engineering/04-compute-services"},{"title":"Storage Solutions","icon":"database","href":"/courses/azure-cloud-engineering/05-storage-solutions"},{"title":"Database Services","icon":"database","href":"/courses/azure-cloud-engineering/06-database-services"},{"title":"Azure Kubernetes Service (AKS)","icon":"dharmachakra","href":"/courses/azure-cloud-engineering/07-container-kubernetes"},{"title":"Serverless Architecture","icon":"bolt","href":"/courses/azure-cloud-engineering/08-serverless-architecture"},{"title":"Monitoring \u0026 Observability","icon":"chart-line","href":"/courses/azure-cloud-engineering/09-monitoring-observability"},{"title":"Security \u0026 Compliance","icon":"shield-halved","href":"/courses/azure-cloud-engineering/10-security-compliance"},{"title":"DevOps \u0026 CI/CD","icon":"code-branch","href":"/courses/azure-cloud-engineering/11-devops-cicd"},{"title":"Cost Optimization","icon":"dollar-sign","href":"/courses/azure-cloud-engineering/12-cost-optimization"},{"title":"High Availability \u0026 Disaster Recovery","icon":"life-ring","href":"/courses/azure-cloud-engineering/13-high-availability-dr"},{"title":"Architecture Patterns","icon":"diagram-project","href":"/courses/azure-cloud-engineering/14-architecture-patterns"},{"title":"Capstone Project","icon":"trophy","href":"/courses/azure-cloud-engineering/15-capstone-project"},{"title":"API Management \u0026 Messaging Patterns","icon":"diagram-project","href":"/courses/azure-cloud-engineering/16-api-messaging"},{"title":"SRE \u0026 Production Excellence","icon":"shield-heart","href":"/courses/azure-cloud-engineering/17-sre-production"}]},{"group":"AWS Cloud","icon":"cloud","pages":[{"title":"AWS Cloud Mastery","icon":"cloud","href":"/aws/overview"},{"title":"Core Concepts","icon":"circle-nodes","href":"/aws/core-concepts"},{"title":"Compute Services","icon":"microchip","href":"/aws/compute"},{"title":"Storage \u0026 Databases","icon":"hard-drive","href":"/aws/storage-databases"},{"title":"Networking","icon":"diagram-project","href":"/aws/networking"},{"title":"Security \u0026 IAM","icon":"shield-halved","href":"/aws/security"},{"title":"Well-Architected Framework","icon":"building","href":"/aws/well-architected"}]},{"group":"AWS Monitoring \u0026 Observability","icon":"chart-line","pages":[{"title":"Amazon CloudWatch","icon":"chart-line","href":"/aws/cloudwatch"},{"title":"AWS X-Ray","icon":"microscope","href":"/aws/xray"},{"title":"Observability \u0026 Monitoring","icon":"chart-line","href":"/aws/observability"}]},{"group":"AWS Security Services","icon":"shield-halved","pages":[{"title":"AWS GuardDuty","icon":"shield","href":"/aws/guardduty"},{"title":"AWS Security Hub","icon":"building-shield","href":"/aws/security-hub"},{"title":"AWS Inspector","href":"/aws/inspector"},{"title":"AWS WAF (Web Application Firewall)","href":"/aws/waf"},{"title":"AWS Shield","href":"/aws/shield"},{"title":"AWS CloudTrail","icon":"file-lines","href":"/aws/cloudtrail"},{"title":"AWS Config","icon":"gear","href":"/aws/config"}]},{"group":"AWS Serverless","icon":"bolt","pages":[{"title":"AWS Lambda","icon":"bolt","href":"/aws/lambda"},{"title":"AWS Step Functions","icon":"diagram-project","href":"/aws/step-functions"},{"title":"AWS SAM","icon":"box","href":"/aws/sam"},{"title":"Amazon DynamoDB","icon":"database","href":"/aws/dynamodb"}]},{"group":"AWS Operations","icon":"gears","pages":[{"title":"DevOps \u0026 CI/CD","icon":"code-branch","href":"/aws/devops-cicd"}]},{"group":"AWS Advanced","icon":"rocket","pages":[{"title":"CDN \u0026 Edge Services","icon":"globe","href":"/aws/cdn-edge"},{"title":"Messaging \u0026 Integration","icon":"arrows-split-up-and-left","href":"/aws/messaging"},{"title":"Containers \u0026 Microservices","icon":"cubes","href":"/aws/containers"}]},{"group":"AWS Case Studies","icon":"lightbulb","pages":[{"title":"Case Study: Serverless URL Shortener","icon":"bolt","href":"/aws/case-study-serverless"}]},{"group":"GCP Cloud Engineering","icon":"cloud","pages":[{"title":"GCP Cloud Engineering","sidebarTitle":"Overview","icon":"cloud","href":"/courses/gcp-cloud-engineering/00-overview"},{"title":"GCP Fundamentals","href":"/courses/gcp-cloud-engineering/01-gcp-fundamentals"},{"title":"Identity \u0026 Security","href":"/courses/gcp-cloud-engineering/02-iam-security"},{"title":"Networking \u0026 VPC","href":"/courses/gcp-cloud-engineering/03-networking-vpc"},{"title":"Load Balancing \u0026 DNS","href":"/courses/gcp-cloud-engineering/04-load-balancing-dns"},{"title":"Compute Engine","href":"/courses/gcp-cloud-engineering/05-compute-engine"},{"title":"Cloud Storage","href":"/courses/gcp-cloud-engineering/06-cloud-storage"},{"title":"Relational Databases","href":"/courses/gcp-cloud-engineering/07-relational-databases"},{"title":"NoSQL Databases","href":"/courses/gcp-cloud-engineering/08-nosql-databases"},{"title":"Containers \u0026 Artifacts","href":"/courses/gcp-cloud-engineering/09-containers-registry"},{"title":"Google Kubernetes Engine","href":"/courses/gcp-cloud-engineering/10-google-kubernetes-engine"},{"title":"Serverless Compute","href":"/courses/gcp-cloud-engineering/11-serverless-compute"},{"title":"Data Analytics \u0026 BigQuery","href":"/courses/gcp-cloud-engineering/12-data-analytics-bigquery"},{"title":"Monitoring \u0026 Observability","href":"/courses/gcp-cloud-engineering/13-monitoring-observability"},{"title":"Security \u0026 Compliance","href":"/courses/gcp-cloud-engineering/14-security-compliance"},{"title":"IaC \u0026 Deployment","href":"/courses/gcp-cloud-engineering/15-iac-deployment"},{"title":"Hybrid \u0026 Multi-Cloud","href":"/courses/gcp-cloud-engineering/16-hybrid-multi-cloud"},{"title":"Cost Management","href":"/courses/gcp-cloud-engineering/17-cost-management"},{"title":"Capstone Project","href":"/courses/gcp-cloud-engineering/18-capstone-project"}]},{"group":"DevOps Tools","icon":"toolbox","pages":[{"title":"DevOps Tools Mastery","sidebarTitle":"Overview","icon":"toolbox","href":"/courses/devops-tools/overview"},{"group":"Git Crash Course","pages":[{"title":"Git Crash Course","sidebarTitle":"Git Overview","icon":"code-branch","href":"/courses/devops-tools/git-overview"},{"title":"Git Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/git-fundamentals"},{"title":"Git Branching \u0026 Merging","sidebarTitle":"Branching","icon":"code-branch","href":"/courses/devops-tools/git-branching"},{"title":"Git Collaboration","sidebarTitle":"Collaboration","icon":"users","href":"/courses/devops-tools/git-collaboration"},{"title":"Git Advanced Topics","sidebarTitle":"Advanced","icon":"wand-magic-sparkles","href":"/courses/devops-tools/git-advanced"},{"title":"Git Workflows","sidebarTitle":"Workflows","icon":"code-branch","href":"/courses/devops-tools/git-workflows"},{"title":"Git Hooks \u0026 Automation","sidebarTitle":"Hooks \u0026 Automation","icon":"robot","href":"/courses/devops-tools/git-hooks"}]},{"group":"Linux Crash Course","pages":[{"title":"Linux Crash Course","sidebarTitle":"Linux Overview","icon":"linux","href":"/courses/devops-tools/linux-overview"},{"title":"Linux Fundamentals","sidebarTitle":"Fundamentals","icon":"terminal","href":"/courses/devops-tools/linux-fundamentals"},{"title":"Linux Permissions \u0026 Users","sidebarTitle":"Permissions","icon":"lock","href":"/courses/devops-tools/linux-permissions"},{"title":"Linux Process Management","sidebarTitle":"Processes","icon":"gears","href":"/courses/devops-tools/linux-processes"},{"title":"Linux Networking","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/devops-tools/linux-networking"},{"title":"Linux Shell Scripting","sidebarTitle":"Shell Scripting","icon":"file-code","href":"/courses/devops-tools/linux-scripting"},{"title":"System Monitoring","sidebarTitle":"Monitoring","icon":"chart-line","href":"/courses/devops-tools/linux-monitoring"},{"title":"Security Hardening","sidebarTitle":"Security","icon":"lock","href":"/courses/devops-tools/linux-security"}]},{"group":"Docker Crash Course","pages":[{"title":"Docker Crash Course","sidebarTitle":"Docker Overview","icon":"docker","href":"/courses/devops-tools/docker-overview"},{"title":"Docker Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/docker-fundamentals"},{"title":"Building Docker Images","sidebarTitle":"Images","icon":"layer-group","href":"/courses/devops-tools/docker-images"},{"title":"Docker Networking \u0026 Volumes","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/devops-tools/docker-networking"},{"title":"Docker Compose","sidebarTitle":"Compose","icon":"cubes","href":"/courses/devops-tools/docker-compose"},{"title":"Docker Best Practices","sidebarTitle":"Best Practices","icon":"shield-check","href":"/courses/devops-tools/docker-best-practices"}]},{"group":"RabbitMQ Crash Course","pages":[{"title":"RabbitMQ Crash Course","sidebarTitle":"RabbitMQ Overview","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-overview"},{"title":"RabbitMQ Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/rabbitmq-fundamentals"},{"title":"Patterns","sidebarTitle":"Patterns","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-patterns"},{"title":"Reliability","sidebarTitle":"Reliability","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-reliability"}]},{"group":"Kafka Crash Course","pages":[{"title":"Kafka Crash Course","sidebarTitle":"Kafka Overview","icon":"stream","href":"/courses/devops-tools/kafka-overview"},{"title":"Kafka Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/kafka-fundamentals"},{"title":"Kafka Producers \u0026 Consumers","sidebarTitle":"Producers \u0026 Consumers","icon":"arrows-left-right","href":"/courses/devops-tools/kafka-producers-consumers"},{"title":"Kafka Streams","sidebarTitle":"Streams","icon":"water","href":"/courses/devops-tools/kafka-streams"},{"title":"Kafka Operations","sidebarTitle":"Operations","icon":"server","href":"/courses/devops-tools/kafka-operations"}]},{"group":"Kubernetes Crash Course","pages":[{"title":"Kubernetes Crash Course","sidebarTitle":"Kubernetes Overview","icon":"dharmachakra","href":"/courses/devops-tools/kubernetes-overview"},{"title":"Kubernetes Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/kubernetes-fundamentals"},{"title":"Kubernetes Workloads","sidebarTitle":"Workloads","icon":"cubes","href":"/courses/devops-tools/kubernetes-workloads"},{"title":"Kubernetes Services","sidebarTitle":"Services","icon":"network-wired","href":"/courses/devops-tools/kubernetes-services"},{"title":"Kubernetes Configuration","sidebarTitle":"Configuration","icon":"gear","href":"/courses/devops-tools/kubernetes-config"},{"title":"Kubernetes Storage","sidebarTitle":"Storage","icon":"hard-drive","href":"/courses/devops-tools/kubernetes-storage"},{"title":"Kubernetes on Windows \u0026 Linux","sidebarTitle":"Windows \u0026 Linux","icon":"window-maximize","href":"/courses/devops-tools/kubernetes-windows-linux"}]}]},{"group":"Database Engineering","icon":"database","pages":[{"title":"Database Engineering Mastery","sidebarTitle":"Course Overview","icon":"database","href":"/courses/database-engineering/overview"},{"group":"Track 1: Foundations","pages":[{"title":"SQL Foundations","sidebarTitle":"1. SQL Foundations","icon":"terminal","href":"/courses/database-engineering/sql-foundations"},{"title":"Database Design \u0026 Modeling","sidebarTitle":"2. Database Design","icon":"diagram-project","href":"/courses/database-engineering/database-design"},{"title":"Transactions \u0026 ACID","sidebarTitle":"3. Transactions \u0026 ACID","icon":"shield-check","href":"/courses/database-engineering/transactions"}]},{"group":"Track 2: Performance","pages":[{"title":"Indexing Deep Dive","sidebarTitle":"4. Indexing Deep Dive","icon":"magnifying-glass-chart","href":"/courses/database-engineering/indexing"},{"title":"Query Optimization","sidebarTitle":"5. Query Optimization","icon":"bolt","href":"/courses/database-engineering/query-optimization"},{"title":"Query Execution Plans Mastery","sidebarTitle":"Execution Plans Deep Dive","icon":"route","href":"/courses/database-engineering/execution-plans-mastery"},{"title":"Performance Engineering","sidebarTitle":"Performance Engineering","icon":"gauge-high","href":"/courses/database-engineering/performance-engineering"}]},{"group":"Track 3: Internals","pages":[{"title":"PostgreSQL Internals","sidebarTitle":"6. PostgreSQL Internals","icon":"microchip","href":"/courses/database-engineering/internals"},{"title":"PostgreSQL Data Structures","sidebarTitle":"Data Structures","icon":"sitemap","href":"/courses/database-engineering/data-structures"},{"title":"Query Processing Pipeline","sidebarTitle":"8. Query Processing","icon":"gears","href":"/courses/database-engineering/query-processing"},{"title":"Query Engine Deep Dive","sidebarTitle":"Query Engine Internals","icon":"microchip","href":"/courses/database-engineering/query-engine-deep-dive"},{"title":"PostgreSQL Architecture","sidebarTitle":"9. PG Architecture","icon":"sitemap","href":"/courses/database-engineering/architecture"},{"title":"Storage Engine Deep Dive","sidebarTitle":"Storage Engine Internals","icon":"hard-drive","href":"/courses/database-engineering/storage-deep-dive"}]},{"group":"Track 4: Distributed","pages":[{"title":"Replication \u0026 High Availability","sidebarTitle":"7. Replication \u0026 HA","icon":"server","href":"/courses/database-engineering/replication"},{"title":"Scaling Strategies","sidebarTitle":"8. Scaling","icon":"chart-line","href":"/courses/database-engineering/scaling"},{"title":"Distributed Database Systems","sidebarTitle":"Distributed Systems","icon":"network-wired","href":"/courses/database-engineering/distributed-systems"}]},{"group":"OSS Contributions","pages":[{"title":"PostgreSQL Source Code","sidebarTitle":"13. Source Code","icon":"code","href":"/courses/database-engineering/source-code"},{"title":"Contributing to PostgreSQL","sidebarTitle":"14. Contributing","icon":"code-pull-request","href":"/courses/database-engineering/contributing"}]},{"group":"Capstone","pages":[{"title":"Senior Interview Mastery","sidebarTitle":"15. Interview Prep","icon":"user-tie","href":"/courses/database-engineering/interview-prep"},{"title":"System Design Case Studies","sidebarTitle":"16. Case Studies","icon":"building","href":"/courses/database-engineering/case-studies"}]}]},{"group":"HIPAA Compliance Mastery","icon":"shield-check","pages":[{"title":"HIPAA Compliance Mastery","sidebarTitle":"Course Overview","icon":"hospital","href":"/courses/hipaa-compliance/overview"},{"group":"Track 1: Regulatory Foundations","pages":[{"title":"HIPAA Fundamentals","sidebarTitle":"HIPAA Fundamentals","icon":"book-medical","href":"/courses/hipaa-compliance/hipaa-fundamentals"},{"title":"HIPAA Risk Assessment","sidebarTitle":"Risk Assessment","icon":"magnifying-glass-chart","href":"/courses/hipaa-compliance/risk-assessment"},{"title":"PDPL \u0026 Global Data Protection","sidebarTitle":"PDPL Compliance","icon":"globe","href":"/courses/hipaa-compliance/pdpl-compliance"}]},{"group":"Track 2: Security Architecture","pages":[{"title":"Access Control Systems","sidebarTitle":"Access Control","icon":"user-shield","href":"/courses/hipaa-compliance/access-control"},{"title":"Encryption: Data at Rest \u0026 In Transit","sidebarTitle":"Encryption","icon":"lock","href":"/courses/hipaa-compliance/encryption"},{"title":"HIPAA Database Security \u0026 Encryption","href":"/courses/hipaa-compliance/database-security"}]},{"group":"Track 3: Implementation","pages":[{"title":"Audit Logging for HIPAA","sidebarTitle":"Audit Logging","icon":"file-lines","href":"/courses/hipaa-compliance/audit-logging"},{"title":"E2E Encryption with AI Agents","sidebarTitle":"E2E Encryption + AI","icon":"robot","href":"/courses/hipaa-compliance/e2e-encryption-ai"},{"title":"Implementation Guide","sidebarTitle":"Implementation Guide","icon":"code","href":"/courses/hipaa-compliance/implementation-guide"}]},{"group":"Track 4: Operations \u0026 Governance","pages":[{"title":"Incident Response for Healthcare","sidebarTitle":"Incident Response","icon":"siren","href":"/courses/hipaa-compliance/incident-response"},{"title":"HIPAA Vendor \u0026 Third-Party Risk Management","href":"/courses/hipaa-compliance/vendor-management"}]},{"group":"Track 5: Advanced Topics","pages":[{"title":"HIPAA Breach Case Studies \u0026 Lessons Learned","href":"/courses/hipaa-compliance/case-studies"},{"title":"Compliance Checklist \u0026 BAA","sidebarTitle":"Compliance Checklist","icon":"clipboard-check","href":"/courses/hipaa-compliance/compliance-checklist"}]},{"group":"Track 6: Capstone","pages":[{"title":"HIPAA Compliance Capstone Project","href":"/courses/hipaa-compliance/capstone-project"}]}]},{"group":"Operating Systems","icon":"microchip","pages":[{"title":"Operating Systems Mastery","sidebarTitle":"Course Overview","icon":"microchip","href":"/operating-systems/overview"},{"title":"OS Fundamentals \u0026 System Call Internals","sidebarTitle":"Fundamentals","icon":"book","href":"/operating-systems/os-fundamentals"},{"title":"CPU Architectures \u0026 Microarchitecture","sidebarTitle":"CPU Architectures","icon":"microchip","href":"/operating-systems/cpu-architectures"},{"title":"Operating System Interfaces","sidebarTitle":"xv6 OS Interfaces","icon":"book-open-reader","href":"/operating-systems/xv6"},{"group":"System Startup","pages":[{"title":"The Boot Process: From Reset Vector to PID 1","sidebarTitle":"Boot Process","icon":"power-off","href":"/operating-systems/boot-process"}]},{"group":"Process Management","pages":[{"title":"Process Management","sidebarTitle":"Processes","icon":"microchip","href":"/operating-systems/processes"},{"title":"Threads \u0026 Concurrency: The Senior Mastery Guide","sidebarTitle":"Threads","icon":"layer-group","href":"/operating-systems/threads"},{"title":"CPU Scheduling","sidebarTitle":"Scheduling","icon":"clock","href":"/operating-systems/scheduling"}]},{"group":"Memory Management","pages":[{"title":"Memory Management","sidebarTitle":"Memory Management","icon":"memory","href":"/operating-systems/memory-management"},{"title":"Virtual Memory","sidebarTitle":"Virtual Memory","icon":"memory","href":"/operating-systems/virtual-memory"}]},{"group":"Concurrency \u0026 I/O","pages":[{"title":"Advanced Synchronization \u0026 RCU","sidebarTitle":"Synchronization","icon":"hand-middle-finger","href":"/operating-systems/synchronization"},{"title":"Deadlocks","sidebarTitle":"Deadlocks","icon":"circle-xmark","href":"/operating-systems/deadlocks"},{"title":"Inter-Process Communication: Kernel Internals \u0026 Architecture","sidebarTitle":"IPC","icon":"message","href":"/operating-systems/ipc"},{"title":"Virtual File System (VFS) \u0026 Disk Internals","sidebarTitle":"File Systems","icon":"file-lines","href":"/operating-systems/file-systems"},{"title":"I/O Systems \u0026 Modern I/O","sidebarTitle":"I/O Systems","icon":"input-pipe","href":"/operating-systems/io-systems"}]},{"group":"Networking","pages":[{"title":"Kernel Networking Stack: From NIC to Socket","sidebarTitle":"Networking","icon":"network-wired","href":"/operating-systems/networking"}]},{"group":"Virtualization","pages":[{"title":"Containers \u0026 Virtualization: From Namespaces to Hypervisors","sidebarTitle":"Containers \u0026 VMs","icon":"cubes","href":"/operating-systems/containers-virtualization"}]},{"group":"Advanced Topics","pages":[{"title":"OS Security: From Hardware to Userspace","sidebarTitle":"Security","icon":"shield-halved","href":"/operating-systems/security"},{"title":"Linux Internals","sidebarTitle":"Linux Internals","icon":"linux","href":"/operating-systems/linux-internals"},{"title":"Modern OS Features","sidebarTitle":"Modern Features","icon":"rocket","href":"/operating-systems/modern-features"},{"title":"Observability \u0026 Performance Engineering","sidebarTitle":"Performance","icon":"gauge-high","href":"/operating-systems/debugging-performance"}]},{"group":"Interview Prep","pages":[{"title":"Interview Preparation","sidebarTitle":"Interview Prep","icon":"user-tie","href":"/operating-systems/interview-prep"},{"title":"OS Case Studies","sidebarTitle":"Case Studies","icon":"building","href":"/operating-systems/case-studies"}]}]},{"group":"Linux Internals","icon":"linux","pages":[{"title":"Linux Internals \u0026 Kernel Mastery","sidebarTitle":"Course Overview","icon":"linux","href":"/courses/linux-internals/overview"},{"group":"Kernel Foundations","pages":[{"title":"Linux Kernel Architecture","sidebarTitle":"Kernel Architecture","icon":"sitemap","href":"/courses/linux-internals/kernel-architecture"},{"title":"System Call Interface","sidebarTitle":"System Calls","icon":"terminal","href":"/courses/linux-internals/syscalls"},{"title":"Kernel Data Structures","sidebarTitle":"Data Structures","icon":"database","href":"/courses/linux-internals/data-structures"},{"title":"Interrupts \u0026 Exception Handling","sidebarTitle":"Interrupts","icon":"bolt","href":"/courses/linux-internals/interrupts-exceptions"},{"title":"Kernel Synchronization","sidebarTitle":"Synchronization","icon":"lock","href":"/courses/linux-internals/synchronization"}]},{"group":"Process \u0026 Memory","pages":[{"title":"Process Subsystem Deep Dive","sidebarTitle":"Process Subsystem","icon":"microchip","href":"/courses/linux-internals/process-subsystem"},{"title":"Memory Management Internals","sidebarTitle":"Memory Management","icon":"memory","href":"/courses/linux-internals/memory-management"},{"title":"Signals \u0026 IPC","sidebarTitle":"Signals \u0026 IPC","icon":"arrows-split-up-and-left","href":"/courses/linux-internals/signals-ipc"}]},{"group":"Containers \u0026 Isolation","pages":[{"title":"Namespaces Deep Dive","sidebarTitle":"Namespaces","icon":"boxes-stacked","href":"/courses/linux-internals/namespaces"},{"title":"Control Groups (cgroups)","sidebarTitle":"cgroups","icon":"object-group","href":"/courses/linux-internals/cgroups"},{"title":"Security Modules \u0026 Capabilities","sidebarTitle":"Security","icon":"shield-halved","href":"/courses/linux-internals/security-modules"}]},{"group":"I/O \u0026 Networking","pages":[{"title":"Filesystem \u0026 VFS","sidebarTitle":"Filesystem \u0026 VFS","icon":"folder-tree","href":"/courses/linux-internals/filesystem-vfs"},{"title":"I/O Subsystem","sidebarTitle":"I/O Subsystem","icon":"hard-drive","href":"/courses/linux-internals/io-subsystem"},{"title":"Network Stack","sidebarTitle":"Network Stack","icon":"network-wired","href":"/courses/linux-internals/networking-stack"}]},{"group":"Observability","pages":[{"title":"eBPF Deep Dive","sidebarTitle":"eBPF","icon":"magnifying-glass-chart","href":"/courses/linux-internals/ebpf"},{"title":"Tracing \u0026 Profiling","sidebarTitle":"Tracing \u0026 Profiling","icon":"chart-line","href":"/courses/linux-internals/tracing-profiling"}]},{"group":"Advanced Kernel","pages":[{"title":"Power \u0026 Thermal Management","sidebarTitle":"Power Management","icon":"bolt","href":"/courses/linux-internals/power-management"}]},{"group":"Interview Prep","pages":[{"title":"Hands-on Projects","sidebarTitle":"Projects","icon":"hammer","href":"/courses/linux-internals/projects"},{"title":"Real Interview Questions","sidebarTitle":"Interview Questions","icon":"clipboard-question","href":"/courses/linux-internals/interview-questions"}]}]},{"group":"Distributed Systems","icon":"sitemap","pages":[{"title":"Distributed Systems Mastery","sidebarTitle":"Course Overview","icon":"network-wired","href":"/courses/distributed-systems/overview"},{"group":"Core Foundations","pages":[{"title":"Foundations of Distributed Systems","sidebarTitle":"Foundations","icon":"book","href":"/courses/distributed-systems/foundations"},{"title":"Distributed Networking Fundamentals","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/distributed-systems/networking"},{"title":"Time \u0026 Clock Synchronization","sidebarTitle":"Clocks","icon":"clock","href":"/courses/distributed-systems/clocks"},{"title":"Consistency Models","sidebarTitle":"Consistency Models","icon":"scale-balanced","href":"/courses/distributed-systems/consistency-models"},{"title":"Distributed Snapshots","sidebarTitle":"Distributed Snapshots","icon":"camera","href":"/courses/distributed-systems/snapshots"}]},{"group":"Consensus \u0026 Replication","pages":[{"title":"Consensus Protocols","sidebarTitle":"Consensus","icon":"handshake","href":"/courses/distributed-systems/consensus"},{"title":"Raft Consensus Deep Dive","sidebarTitle":"Raft Deep Dive","icon":"star","href":"/courses/distributed-systems/raft-deep-dive"},{"title":"Paxos Consensus Deep Dive","sidebarTitle":"Paxos Deep Dive","icon":"scroll","href":"/courses/distributed-systems/paxos-deep-dive"},{"title":"Replication Strategies","sidebarTitle":"Replication","icon":"clone","href":"/courses/distributed-systems/replication"},{"title":"Distributed Coordination","sidebarTitle":"Coordination","icon":"people-group","href":"/courses/distributed-systems/coordination"},{"title":"Byzantine Fault Tolerance (BFT)","sidebarTitle":"Byzantine Fault Tolerance","icon":"shield-halved","href":"/courses/distributed-systems/bft"},{"title":"Gossip Protocols \u0026 Membership","sidebarTitle":"Gossip Protocols","icon":"comments","href":"/courses/distributed-systems/gossip"},{"title":"Formal Verification in Distributed Systems","sidebarTitle":"Formal Verification","icon":"shield-check","href":"/courses/distributed-systems/formal-verification"}]},{"group":"Data \u0026 Transactions","pages":[{"title":"Distributed Transactions","sidebarTitle":"Transactions","icon":"money-bill-transfer","href":"/courses/distributed-systems/transactions"},{"title":"Data Partitioning \u0026 Sharding","sidebarTitle":"Partitioning","icon":"table-cells","href":"/courses/distributed-systems/partitioning"},{"title":"Data Systems at Scale","sidebarTitle":"Data Systems","icon":"database","href":"/courses/distributed-systems/data-systems"}]},{"group":"Infrastructure \u0026 Messaging","pages":[{"title":"Distributed Messaging \u0026 Event Systems","sidebarTitle":"Messaging","icon":"envelope","href":"/courses/distributed-systems/messaging"},{"title":"Distributed Caching","sidebarTitle":"Caching","icon":"bolt","href":"/courses/distributed-systems/caching"},{"title":"Fault Tolerance Patterns","sidebarTitle":"Fault Tolerance","icon":"shield-heart","href":"/courses/distributed-systems/fault-tolerance"}]},{"group":"Production \u0026 Practice","pages":[{"title":"Production Excellence","sidebarTitle":"Production","icon":"shield-halved","href":"/courses/distributed-systems/production"},{"title":"Real-World Case Studies","sidebarTitle":"Case Studies","icon":"building","href":"/courses/distributed-systems/case-studies"},{"title":"Interview Practice Problems","sidebarTitle":"Interview Problems","icon":"comments","href":"/courses/distributed-systems/interview-problems"}]}]},{"group":"Networking Mastery","icon":"network-wired","pages":[{"title":"Networking Fundamentals","sidebarTitle":"1. Overview","icon":"globe","href":"/courses/networking-mastery/01-overview"},{"group":"Fundamentals","pages":[{"title":"Network Models","sidebarTitle":"2. OSI \u0026 TCP/IP","icon":"layer-group","href":"/courses/networking-mastery/02-models"},{"title":"Physical \u0026 Data Link Layers","sidebarTitle":"3. Physical \u0026 Data Link","icon":"ethernet","href":"/courses/networking-mastery/03-physical-datalink"},{"title":"Network Layer","sidebarTitle":"4. Network Layer","icon":"network-wired","href":"/courses/networking-mastery/04-network-layer"},{"title":"Transport Layer","sidebarTitle":"5. Transport Layer","icon":"truck-fast","href":"/courses/networking-mastery/05-transport-layer"},{"title":"Application Layer","sidebarTitle":"6. Application Layer","icon":"browser","href":"/courses/networking-mastery/06-application-layer"}]},{"group":"Security \u0026 Advanced Intro","pages":[{"title":"Network Security","sidebarTitle":"7. Security","icon":"shield-check","href":"/courses/networking-mastery/07-security"},{"title":"Advanced Networking Intro","sidebarTitle":"8. Advanced Topics","icon":"cloud","href":"/courses/networking-mastery/08-advanced"}]},{"group":"Deep Dives","pages":[{"title":"IP Addressing Deep Dive","sidebarTitle":"9. IP Addressing Mastery","icon":"map-location-dot","href":"/courses/networking-mastery/09-ip-addressing-deep-dive"},{"title":"NAT \u0026 PAT Deep Dive","sidebarTitle":"10. NAT Mastery","icon":"right-left","href":"/courses/networking-mastery/10-nat-deep-dive"},{"title":"Routing Deep Dive","sidebarTitle":"11. Routing Mastery","icon":"route","href":"/courses/networking-mastery/11-routing-deep-dive"},{"title":"DNS Deep Dive","sidebarTitle":"12. DNS Mastery","icon":"book","href":"/courses/networking-mastery/12-dns-deep-dive"},{"title":"Load Balancing \u0026 Proxies","sidebarTitle":"13. Load Balancing","icon":"scale-balanced","href":"/courses/networking-mastery/13-load-balancing"}]},{"group":"Operations \u0026 Security","pages":[{"title":"Network Troubleshooting","sidebarTitle":"14. Troubleshooting","icon":"wrench","href":"/courses/networking-mastery/14-troubleshooting"},{"title":"VPNs \u0026 Tunneling","sidebarTitle":"15. VPNs \u0026 Tunneling","icon":"tunnel","href":"/courses/networking-mastery/15-vpn-tunneling"},{"title":"Firewalls \u0026 Security Groups","sidebarTitle":"17. Firewalls","icon":"shield-halved","href":"/courses/networking-mastery/17-firewalls-security-groups"}]},{"group":"Modern Networking","pages":[{"title":"Container Networking","sidebarTitle":"18. Container Networking","icon":"cubes","href":"/courses/networking-mastery/18-container-networking"},{"title":"Real-World Networking Scenarios","sidebarTitle":"16. Network Scenarios","icon":"diagram-project","href":"/courses/networking-mastery/16-scenarios"}]}]},{"group":"Build Your Own X","icon":"hammer","pages":[{"title":"Build Your Own X","sidebarTitle":"Course Overview","icon":"hammer","href":"/courses/build-your-own-x/overview"},{"group":"Build Your Own Git","pages":[{"title":"Build Your Own Git","sidebarTitle":"Git Overview","icon":"code-branch","href":"/courses/build-your-own-x/git-overview"},{"title":"Chapter 1: Project Setup \u0026 Init","sidebarTitle":"1. Setup \u0026 Init","icon":"folder-open","href":"/courses/build-your-own-x/git-1-setup"},{"title":"Chapter 2: The Object Model","sidebarTitle":"2. Object Model","icon":"cube","href":"/courses/build-your-own-x/git-2-objects"},{"title":"Chapter 3: Staging \u0026 Index","sidebarTitle":"3. Staging \u0026 Index","icon":"layer-plus","href":"/courses/build-your-own-x/git-3-staging"},{"title":"Chapter 4: Commits \u0026 History","sidebarTitle":"4. Commits \u0026 History","icon":"code-commit","href":"/courses/build-your-own-x/git-4-commits"},{"title":"Chapter 5: Branches \u0026 Checkout","sidebarTitle":"5. Branches \u0026 Checkout","icon":"code-branch","href":"/courses/build-your-own-x/git-5-branches"}]},{"group":"Build Your Own Redis","pages":[{"title":"Build Your Own Redis","sidebarTitle":"Redis Overview","icon":"database","href":"/courses/build-your-own-x/redis-overview"},{"title":"Chapter 1: RESP Protocol","sidebarTitle":"1. RESP Protocol","icon":"network-wired","href":"/courses/build-your-own-x/redis-1-protocol"},{"title":"Chapter 2: TCP Server","sidebarTitle":"2. TCP Server","icon":"server","href":"/courses/build-your-own-x/redis-2-server"},{"title":"Chapter 3: Data Structures","sidebarTitle":"3. Data Structures","icon":"layer-group","href":"/courses/build-your-own-x/redis-3-data-structures"},{"title":"Chapter 4: Persistence","sidebarTitle":"4. Persistence","icon":"floppy-disk","href":"/courses/build-your-own-x/redis-4-persistence"},{"title":"Chapter 5: Pub/Sub","sidebarTitle":"5. Pub/Sub","icon":"tower-broadcast","href":"/courses/build-your-own-x/redis-5-pubsub"}]},{"group":"Build Your Own Docker","pages":[{"title":"Build Your Own Docker","sidebarTitle":"Docker Overview","icon":"docker","href":"/courses/build-your-own-x/docker-overview"},{"title":"Chapter 1: Linux Namespaces","sidebarTitle":"1. Namespaces","icon":"box","href":"/courses/build-your-own-x/docker-1-namespaces"},{"title":"Chapter 2: Control Groups (cgroups)","sidebarTitle":"2. Cgroups","icon":"gauge-high","href":"/courses/build-your-own-x/docker-2-cgroups"},{"title":"Chapter 3: Filesystem Isolation","sidebarTitle":"3. Filesystem","icon":"folder-tree","href":"/courses/build-your-own-x/docker-3-filesystem"},{"title":"Chapter 4: Container Networking","sidebarTitle":"4. Networking","icon":"network-wired","href":"/courses/build-your-own-x/docker-4-networking"},{"title":"Chapter 5: Container Images","sidebarTitle":"5. Images","icon":"box-archive","href":"/courses/build-your-own-x/docker-5-images"}]}]},{"group":"Go Lang Mastery","icon":"golang","pages":[{"group":"Fundamentals","pages":[{"title":"Introduction to Go","sidebarTitle":"1. Introduction","icon":"golang","href":"/courses/go-lang-mastery/01-introduction"},{"title":"Variables and Types","sidebarTitle":"2. Variables \u0026 Types","icon":"cube","href":"/courses/go-lang-mastery/02-variables-and-types"},{"title":"Control Structures","sidebarTitle":"3. Control Structures","icon":"code-branch","href":"/courses/go-lang-mastery/03-control-structures"},{"title":"Functions and Packages","sidebarTitle":"4. Functions \u0026 Packages","icon":"box-open","href":"/courses/go-lang-mastery/04-functions-and-packages"},{"title":"Arrays, Slices, and Maps","sidebarTitle":"5. Arrays, Slices \u0026 Maps","icon":"layer-group","href":"/courses/go-lang-mastery/05-arrays-slices-maps"},{"title":"Pointers and Structs","sidebarTitle":"6. Pointers \u0026 Structs","icon":"shapes","href":"/courses/go-lang-mastery/06-pointers-and-structs"},{"title":"Interfaces","sidebarTitle":"7. Interfaces","icon":"plug","href":"/courses/go-lang-mastery/07-interfaces"}]},{"group":"Core Concepts","pages":[{"title":"08. Concurrency","icon":"arrows-split-up-and-left","href":"/courses/go-lang-mastery/08-concurrency"},{"title":"09. Error Handling","icon":"triangle-exclamation","href":"/courses/go-lang-mastery/09-error-handling"},{"title":"10. Testing and Benchmarking","icon":"vial","href":"/courses/go-lang-mastery/10-testing-and-benchmarking"},{"title":"Generics","sidebarTitle":"11. Generics","icon":"cube","href":"/courses/go-lang-mastery/11-generics"},{"title":"Advanced Concurrency Patterns","sidebarTitle":"12. Advanced Concurrency","icon":"diagram-project","href":"/courses/go-lang-mastery/12-advanced-concurrency"},{"title":"Context Package","sidebarTitle":"13. Context","icon":"clock","href":"/courses/go-lang-mastery/13-context"}]},{"group":"Production Go","pages":[{"title":"HTTP \u0026 Web Development","sidebarTitle":"14. HTTP \u0026 Web","icon":"globe","href":"/courses/go-lang-mastery/14-http-web"},{"title":"Database Operations","sidebarTitle":"15. Databases","icon":"database","href":"/courses/go-lang-mastery/15-databases"},{"title":"Microservices \u0026 gRPC","sidebarTitle":"16. Microservices \u0026 gRPC","icon":"sitemap","href":"/courses/go-lang-mastery/16-microservices-grpc"},{"title":"Performance Optimization","sidebarTitle":"17. Performance","icon":"gauge-high","href":"/courses/go-lang-mastery/17-performance"},{"title":"Production Best Practices","sidebarTitle":"18. Production","icon":"rocket","href":"/courses/go-lang-mastery/18-production"}]},{"group":"Advanced Topics","pages":[{"title":"Design Patterns in Go","sidebarTitle":"19. Design Patterns","icon":"puzzle-piece","href":"/courses/go-lang-mastery/19-design-patterns"},{"title":"Interview Preparation","sidebarTitle":"20. Interview Prep","icon":"user-tie","href":"/courses/go-lang-mastery/20-interview-prep"}]}]},{"group":"C Programming","icon":"memory","pages":[{"title":"Advanced C Programming for Systems Development","sidebarTitle":"Course Overview","icon":"microchip","href":"/courses/c-programming/overview"},{"group":"Foundations","pages":[{"title":"C Syntax Speed Run","sidebarTitle":"Syntax Speed Run","icon":"forward-fast","href":"/courses/c-programming/syntax-speedrun"},{"title":"Build Systems \u0026 Toolchain","sidebarTitle":"Build \u0026 Toolchain","icon":"hammer","href":"/courses/c-programming/toolchain"},{"title":"Debugging Fundamentals","sidebarTitle":"Debugging","icon":"bug","href":"/courses/c-programming/debugging"}]},{"group":"Memory Mastery","pages":[{"title":"Pointers Deep Dive","sidebarTitle":"Pointers","icon":"arrow-pointer","href":"/courses/c-programming/pointers"},{"title":"Memory Layout \u0026 Segments","sidebarTitle":"Memory Layout","icon":"layer-group","href":"/courses/c-programming/memory-layout"},{"title":"Dynamic Memory Management","sidebarTitle":"Dynamic Memory","icon":"memory","href":"/courses/c-programming/dynamic-memory"},{"title":"Preprocessor Mastery","sidebarTitle":"Preprocessor","icon":"code","href":"/courses/c-programming/preprocessor"},{"title":"Data Structures in C","sidebarTitle":"Data Structures","icon":"sitemap","href":"/courses/c-programming/data-structures"},{"title":"Undefined Behavior","sidebarTitle":"Undefined Behavior","icon":"skull","href":"/courses/c-programming/undefined-behavior"}]},{"group":"Systems Programming","pages":[{"title":"System Calls \u0026 POSIX","sidebarTitle":"System Calls","icon":"terminal","href":"/courses/c-programming/system-calls"},{"title":"Concurrency \u0026 Threading","sidebarTitle":"Concurrency","icon":"arrows-split-up-and-left","href":"/courses/c-programming/concurrency"},{"title":"Network Programming","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/c-programming/networking"}]},{"group":"Projects","pages":[{"title":"Project: Build a Shell","sidebarTitle":"Project: Shell","icon":"terminal","href":"/courses/c-programming/project-shell"},{"title":"Project: Key-Value Store","sidebarTitle":"Project: KV Store","icon":"database","href":"/courses/c-programming/project-database"},{"title":"Project: Memory Allocator","sidebarTitle":"Project: malloc","icon":"memory","href":"/courses/c-programming/project-allocator"},{"title":"Project: HTTP Server","sidebarTitle":"Project: HTTP Server","icon":"server","href":"/courses/c-programming/project-http-server"}]},{"group":"Advanced Topics","pages":[{"title":"Performance Optimization","sidebarTitle":"Performance","icon":"gauge-high","href":"/courses/c-programming/performance"},{"title":"Embedded Systems","sidebarTitle":"Embedded","icon":"microchip","href":"/courses/c-programming/embedded"},{"title":"Linux Kernel Modules","sidebarTitle":"Kernel Modules","icon":"linux","href":"/courses/c-programming/kernel-modules"}]}]},{"group":"Classic Research Papers","icon":"file-lines","pages":[{"group":"Google File System (GFS)","icon":"folder","pages":[{"title":"Google File System","sidebarTitle":"GFS Overview","icon":"database","href":"/courses/engineering-papers/gfs-overview"},{"title":"Introduction and Motivation","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/chapter-1-introduction"},{"title":"Architecture Overview","sidebarTitle":"Chapter 2: Architecture","icon":"diagram-project","href":"/courses/engineering-papers/chapter-2-architecture"},{"title":"Master Operations","sidebarTitle":"Chapter 3: Master Operations","icon":"crown","href":"/courses/engineering-papers/chapter-3-master-operations"},{"title":"Chunkservers and Data Flow","sidebarTitle":"Chapter 4: Chunkservers \u0026 Data Flow","icon":"server","href":"/courses/engineering-papers/chapter-4-chunkservers-dataflow"},{"title":"Consistency Model","sidebarTitle":"Chapter 5: Consistency Model","icon":"scale-balanced","href":"/courses/engineering-papers/chapter-5-consistency-model"},{"title":"Fault Tolerance","sidebarTitle":"Chapter 6: Fault Tolerance","icon":"shield-halved","href":"/courses/engineering-papers/chapter-6-fault-tolerance"},{"title":"Performance and Optimizations","sidebarTitle":"Chapter 7: Performance","icon":"gauge-high","href":"/courses/engineering-papers/chapter-7-performance"},{"title":"Impact and Evolution","sidebarTitle":"Chapter 8: Impact \u0026 Evolution","icon":"rocket","href":"/courses/engineering-papers/chapter-8-impact-evolution"}]},{"group":"Apache Hadoop","icon":"elephant","pages":[{"title":"Apache Hadoop","sidebarTitle":"Hadoop Overview","icon":"database","href":"/courses/engineering-papers/hadoop-overview"},{"title":"Introduction and Origins","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/hadoop-chapter-1-introduction"},{"title":"HDFS Architecture","sidebarTitle":"Chapter 2: HDFS","icon":"hard-drive","href":"/courses/engineering-papers/hadoop-chapter-2-hdfs-architecture"},{"title":"MapReduce Framework","sidebarTitle":"Chapter 3: MapReduce","icon":"diagram-project","href":"/courses/engineering-papers/hadoop-chapter-3-mapreduce-framework"},{"title":"YARN: Resource Management","sidebarTitle":"Chapter 4: YARN","icon":"layer-group","href":"/courses/engineering-papers/hadoop-chapter-4-yarn"},{"title":"Hadoop Ecosystem","sidebarTitle":"Chapter 5: Ecosystem","icon":"puzzle-piece","href":"/courses/engineering-papers/hadoop-chapter-5-ecosystem"},{"title":"Fault Tolerance and Reliability","sidebarTitle":"Chapter 6: Fault Tolerance","icon":"shield-check","href":"/courses/engineering-papers/hadoop-chapter-6-fault-tolerance"},{"title":"Performance Tuning and Optimization","sidebarTitle":"Chapter 7: Performance","icon":"gauge-high","href":"/courses/engineering-papers/hadoop-chapter-7-performance"},{"title":"Hadoop in Production","sidebarTitle":"Chapter 8: Production","icon":"server","href":"/courses/engineering-papers/hadoop-chapter-8-production"}]},{"group":"Amazon DynamoDB","icon":"database","pages":[{"title":"Amazon DynamoDB","sidebarTitle":"DynamoDB Overview","icon":"database","href":"/courses/engineering-papers/dynamodb-overview"},{"title":"Introduction and Origins","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/dynamodb-chapter-1-introduction"},{"title":"Architecture and Partitioning","sidebarTitle":"Chapter 2: Architecture","icon":"sitemap","href":"/courses/engineering-papers/dynamodb-chapter-2-architecture"},{"title":"Data Modeling in DynamoDB","sidebarTitle":"Chapter 3: Data Model","icon":"database","href":"/courses/engineering-papers/dynamodb-chapter-3-data-model"},{"title":"Consistency Models in DynamoDB","sidebarTitle":"Chapter 4: Consistency","icon":"scale-balanced","href":"/courses/engineering-papers/dynamodb-chapter-4-consistency"},{"title":"Performance Optimization in DynamoDB","sidebarTitle":"Chapter 5: Performance","icon":"gauge-high","href":"/courses/engineering-papers/dynamodb-chapter-5-performance"},{"title":"Advanced DynamoDB Features","sidebarTitle":"Chapter 6: Advanced Features","icon":"wand-magic-sparkles","href":"/courses/engineering-papers/dynamodb-chapter-6-advanced-features"},{"title":"Fault Tolerance and Reliability","sidebarTitle":"Chapter 7: Fault Tolerance","icon":"shield-check","href":"/courses/engineering-papers/dynamodb-chapter-7-fault-tolerance"},{"title":"DynamoDB in Production","sidebarTitle":"Chapter 8: Production","icon":"server","href":"/courses/engineering-papers/dynamodb-chapter-8-production"}]}]},{"group":"Distributed System Tools","icon":"server","pages":[{"group":"Apache Hadoop","icon":"elephant","pages":[{"title":"Apache Hadoop Mastery","sidebarTitle":"Hadoop Overview","icon":"elephant","href":"/distributed-systems-tools/hadoop-overview"},{"title":"Introduction \u0026 Foundational Papers","sidebarTitle":"1. Introduction \u0026 Papers","icon":"book-open","href":"/distributed-systems-tools/hadoop-introduction"},{"title":"HDFS Architecture \u0026 Internals","sidebarTitle":"2. HDFS Architecture","icon":"database","href":"/distributed-systems-tools/hadoop-hdfs"},{"title":"Advanced HDFS: Federation, HA \u0026 Erasure Coding","sidebarTitle":"2b. Advanced HDFS","icon":"layer-group","href":"/distributed-systems-tools/hadoop-hdfs-advanced"},{"title":"MapReduce Programming Model","sidebarTitle":"3. MapReduce Programming","icon":"code","href":"/distributed-systems-tools/hadoop-mapreduce"},{"title":"YARN Resource Management","sidebarTitle":"4. YARN Architecture","icon":"layer-group","href":"/distributed-systems-tools/hadoop-yarn"},{"title":"Hadoop Ecosystem \u0026 Integration","sidebarTitle":"5. Ecosystem Tools","icon":"puzzle-piece","href":"/distributed-systems-tools/hadoop-ecosystem"},{"title":"Data Processing Patterns \u0026 Best Practices","sidebarTitle":"6. Processing Patterns","icon":"diagram-project","href":"/distributed-systems-tools/hadoop-patterns"},{"title":"Production Deployment \u0026 Operations","sidebarTitle":"7. Production Operations","icon":"server","href":"/distributed-systems-tools/hadoop-production"},{"title":"Capstone Project: End-to-End Data Pipeline","sidebarTitle":"8. Capstone Project","icon":"trophy","href":"/distributed-systems-tools/hadoop-capstone"}]},{"group":"Apache Spark","icon":"bolt","pages":[{"title":"Apache Spark Mastery","sidebarTitle":"Spark Overview","icon":"bolt","href":"/distributed-systems-tools/spark-overview"},{"title":"Demystifying the RDD Paper: Spark's Foundation","sidebarTitle":"1. RDD Paper Deep-Dive","icon":"book-open","href":"/distributed-systems-tools/spark-introduction"},{"title":"Demystifying RDD Programming: From Theory to Production","sidebarTitle":"2. RDD Programming","icon":"bolt","href":"/distributed-systems-tools/spark-rdd"},{"title":"Spark SQL \u0026 DataFrames","sidebarTitle":"3. Spark SQL","icon":"table","href":"/distributed-systems-tools/spark-sql"},{"title":"Structured Streaming","sidebarTitle":"4. Streaming","icon":"bolt","href":"/distributed-systems-tools/spark-streaming"},{"title":"MLlib for Machine Learning","sidebarTitle":"5. MLlib","icon":"bolt","href":"/distributed-systems-tools/spark-mllib"},{"title":"Performance Tuning","sidebarTitle":"6. Performance Tuning","icon":"bolt","href":"/distributed-systems-tools/spark-tuning"},{"title":"Cluster Deployment","sidebarTitle":"7. Operations","icon":"bolt","href":"/distributed-systems-tools/spark-operations"},{"title":"Advanced Topics","sidebarTitle":"8. Advanced","icon":"bolt","href":"/distributed-systems-tools/spark-advanced"},{"title":"Capstone Project","sidebarTitle":"9. Capstone","icon":"bolt","href":"/distributed-systems-tools/spark-capstone"}]},{"group":"Apache Flink","icon":"wave-pulse","pages":[{"title":"Apache Flink Mastery","sidebarTitle":"Flink Overview","icon":"wave-pulse","href":"/distributed-systems-tools/flink-overview"},{"title":"Demystifying the Dataflow Model: True Stream Processing with Flink","sidebarTitle":"1. Introduction","icon":"wave-pulse","href":"/distributed-systems-tools/flink-introduction"},{"title":"DataStream API","sidebarTitle":"2. DataStream API","icon":"wave-pulse","href":"/distributed-systems-tools/flink-datastream"},{"title":"Event Time \u0026 Watermarks","sidebarTitle":"3. Event Time","icon":"wave-pulse","href":"/distributed-systems-tools/flink-eventtime"},{"title":"Windows \u0026 Time Operations","sidebarTitle":"4. Windows","icon":"wave-pulse","href":"/distributed-systems-tools/flink-windows"},{"title":"Demystifying Stateful Stream Processing: Flink's State Management Deep Dive","sidebarTitle":"5. State \u0026 Timers","icon":"database","href":"/distributed-systems-tools/flink-state"},{"title":"Declarative Stream Processing: Flink SQL \u0026 Table API Deep Dive","sidebarTitle":"6. SQL \u0026 Table API","icon":"table","href":"/distributed-systems-tools/flink-sql"},{"title":"Complex Event Processing: Pattern Detection at Scale with Flink CEP","sidebarTitle":"7. Complex Event Processing","icon":"sitemap","href":"/distributed-systems-tools/flink-cep"},{"title":"Production Flink: Deployment, Operations \u0026 Monitoring at Scale","sidebarTitle":"8. Production Operations","icon":"server","href":"/distributed-systems-tools/flink-operations"},{"title":"Capstone: Production-Grade Real-Time Fraud Detection System","sidebarTitle":"9. Capstone Project","icon":"diagram-project","href":"/distributed-systems-tools/flink-capstone"}]},{"group":"Apache Beam","icon":"diagram-project","pages":[{"title":"Apache Beam Mastery","sidebarTitle":"Beam Overview","icon":"diagram-project","href":"/distributed-systems-tools/beam-overview"},{"title":"Demystifying the Dataflow Model: Write Once, Run Anywhere with Apache Beam","sidebarTitle":"1. Introduction","icon":"diagram-project","href":"/distributed-systems-tools/beam-introduction"},{"title":"Core Programming Model","sidebarTitle":"2. Core Model","icon":"diagram-project","href":"/distributed-systems-tools/beam-core"},{"title":"Windowing \u0026 Time","sidebarTitle":"3. Windowing","icon":"diagram-project","href":"/distributed-systems-tools/beam-windowing"},{"title":"Triggers \u0026 Watermarks","sidebarTitle":"4. Triggers","icon":"diagram-project","href":"/distributed-systems-tools/beam-triggers"},{"title":"State \u0026 Timers","sidebarTitle":"5. State","icon":"diagram-project","href":"/distributed-systems-tools/beam-state"},{"title":"Beam I/O Connectors","sidebarTitle":"6. I/O","icon":"diagram-project","href":"/distributed-systems-tools/beam-io"},{"title":"Beam SQL","sidebarTitle":"7. SQL","icon":"diagram-project","href":"/distributed-systems-tools/beam-sql"},{"title":"Multi-Runner Execution","sidebarTitle":"8. Runners","icon":"diagram-project","href":"/distributed-systems-tools/beam-runners"},{"title":"Capstone Project","sidebarTitle":"9. Capstone","icon":"diagram-project","href":"/distributed-systems-tools/beam-capstone"}]},{"group":"Apache Airflow","icon":"wind","pages":[{"title":"What is Apache Airflow?","sidebarTitle":"Overview","icon":"wind","href":"/distributed-systems-tools/airflow-overview"},{"title":"Airflow Core Concepts","sidebarTitle":"Core Concepts","icon":"diagram-project","href":"/distributed-systems-tools/airflow-core-concepts"},{"title":"Airflow Operators","sidebarTitle":"Operators","icon":"puzzle-piece","href":"/distributed-systems-tools/airflow-operators"},{"title":"Airflow Scheduling","sidebarTitle":"Scheduling","icon":"clock","href":"/distributed-systems-tools/airflow-scheduling"}]},{"group":"Apache Cassandra","icon":"database","pages":[{"title":"Apache Cassandra Mastery","sidebarTitle":"Overview","icon":"database","href":"/distributed-systems-tools/cassandra-overview"},{"title":"The Cassandra Paper \u0026 Core Architecture","sidebarTitle":"1. Introduction \u0026 Paper","icon":"book-open","href":"/distributed-systems-tools/cassandra-paper"},{"title":"Cassandra Architecture \u0026 Storage Internals","sidebarTitle":"2. Architecture \u0026 Internals","icon":"diagram-project","href":"/distributed-systems-tools/cassandra-architecture-internals"},{"title":"Query-Driven Data Modeling in Cassandra","sidebarTitle":"3. Data Modeling","icon":"table","href":"/distributed-systems-tools/cassandra-data-modeling"},{"title":"Cassandra Read \u0026 Write Path Internals","sidebarTitle":"4. Read/Write Internals","icon":"route","href":"/distributed-systems-tools/cassandra-read-write-internals"},{"title":"Cluster Operations \u0026 Multi-DC Replication","sidebarTitle":"5. Cluster Operations","icon":"server","href":"/distributed-systems-tools/cassandra-cluster-operations"},{"title":"Performance Tuning \u0026 Production Operations","sidebarTitle":"6. Performance \u0026 Production","icon":"gauge-high","href":"/distributed-systems-tools/cassandra-performance-production"},{"title":"Capstone: Building a Real-Time Analytics Platform","sidebarTitle":"7. Capstone Project","icon":"rocket","href":"/distributed-systems-tools/cassandra-capstone"}]},{"group":"Neo4j Graph Database","icon":"diagram-project","pages":[{"title":"Neo4j Graph Database Mastery","sidebarTitle":"Overview","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-overview"},{"title":"The Graph Database Story: From Theory to Neo4j","sidebarTitle":"2. Foundational Papers","icon":"scroll","href":"/distributed-systems-tools/neo4j-paper"},{"title":"Neo4j Architecture \u0026 Native Graph Storage","sidebarTitle":"3. Architecture \u0026 Storage","icon":"database","href":"/distributed-systems-tools/neo4j-architecture-storage"},{"title":"Cypher Query Language Mastery","sidebarTitle":"4. Cypher Mastery","icon":"code","href":"/distributed-systems-tools/neo4j-cypher-mastery"},{"title":"Graph Data Modeling Best Practices","sidebarTitle":"5. Data Modeling","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-data-modeling"},{"title":"Graph Algorithms at Scale","sidebarTitle":"6. Graph Algorithms","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-graph-algorithms"},{"title":"Production Deployment \u0026 Operations","sidebarTitle":"7. Production \u0026 Ops","icon":"server","href":"/distributed-systems-tools/neo4j-production-deployment"},{"title":"Capstone: Building a Knowledge Graph Platform","sidebarTitle":"8. Capstone Project","icon":"rocket","href":"/distributed-systems-tools/neo4j-capstone"}]}]}]}]},"basePath":"","title":"Foundation Models \u0026 LLMs - Dev Weekends","deploymentId":"dpl_FZMKUpaaKiVETbfqhoUe4sPFDTmY","buildId":"dpl_FZMKUpaaKiVETbfqhoUe4sPFDTmY","dynamics":["07c022d89b","a74be98ba7"],"chunks":["Frame-DBAS9Liu.js","DynamicMarkdown-C4w9kavL.js","zoom-image-lightbox-B9O1U98p.js","lightbox-handle-r17bVUz3.js"]}