Gradient checkpointing, mixed precision, and memory optimization
Memory-Efficient Training
Understanding GPU Memory
GPU memory during training is like a hotel with a fixed number of rooms: your model parameters check in, then their gradients need rooms, then the optimizer’s bookkeeping (momentum, variance) needs even more rooms, and finally every intermediate computation (activation) holds a room open until the backward pass checks it out. When the hotel is full, training crashes with the dreaded CUDA out of memory error. Understanding who occupies which rooms — and for how long — is the key to training larger models on the hardware you actually have.Where does memory go during training?
Component
Memory Usage
Analogy
Model parameters
4×params (fp32)
The permanent residents
Gradients
4×params (fp32)
Their shadows (same size)
Optimizer states
8×params (Adam)
Adam keeps two running averages per parameter
Activations
Depends on batch size, model depth
The guests — they check in during forward, check out during backward
Temporary buffers
Variable
Room service supplies
For a 7B parameter model:
Parameters: 28 GB
Gradients: 28 GB
Adam states: 56 GB
Total: ~112 GB (without activations!)
The single most impactful optimization for most teams is mixed precision training (FP16/BF16). It nearly halves memory for parameters and activations with minimal accuracy impact, and it is a one-line change in most frameworks. Start there before reaching for more complex techniques like gradient checkpointing or model parallelism.
import torchimport torch.nn as nnimport torch.optim as optimfrom torch.utils.checkpoint import checkpoint, checkpoint_sequentialfrom typing import Optional, Tuple, List, Callableimport gcimport functoolstorch.manual_seed(42)device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
Memory Profiling
class MemoryProfiler: """ Profile GPU memory usage during training. """ def __init__(self, device: torch.device = device): self.device = device self.snapshots = [] def get_memory_stats(self) -> dict: """Get current memory statistics.""" if not torch.cuda.is_available(): return {'allocated': 0, 'cached': 0, 'max_allocated': 0} return { 'allocated_mb': torch.cuda.memory_allocated(self.device) / 1e6, 'cached_mb': torch.cuda.memory_reserved(self.device) / 1e6, 'max_allocated_mb': torch.cuda.max_memory_allocated(self.device) / 1e6, } def snapshot(self, label: str = ""): """Take a memory snapshot.""" stats = self.get_memory_stats() stats['label'] = label self.snapshots.append(stats) return stats def reset_peak_stats(self): """Reset peak memory tracking.""" if torch.cuda.is_available(): torch.cuda.reset_peak_memory_stats() def print_summary(self): """Print memory usage summary.""" print("\n" + "="*60) print("MEMORY USAGE SUMMARY") print("="*60) for snap in self.snapshots: print(f"{snap['label']:30} | " f"Allocated: {snap['allocated_mb']:8.1f} MB | " f"Peak: {snap['max_allocated_mb']:8.1f} MB") @staticmethod def estimate_model_memory(model: nn.Module) -> dict: """Estimate memory requirements for a model.""" param_mem = sum(p.numel() * p.element_size() for p in model.parameters()) buffer_mem = sum(b.numel() * b.element_size() for b in model.buffers()) # Gradient memory (same as params for most cases) grad_mem = param_mem # Optimizer states (Adam: 2x for m and v) adam_mem = 2 * param_mem return { 'parameters_mb': param_mem / 1e6, 'buffers_mb': buffer_mem / 1e6, 'gradients_mb': grad_mem / 1e6, 'adam_states_mb': adam_mem / 1e6, 'total_mb': (param_mem + buffer_mem + grad_mem + adam_mem) / 1e6 }# Memory-efficient model analysisdef analyze_memory_per_layer(model: nn.Module, input_shape: tuple): """Analyze memory usage per layer.""" profiler = MemoryProfiler() if not torch.cuda.is_available(): print("CUDA not available for memory analysis") return model = model.to(device) x = torch.randn(*input_shape).to(device) profiler.reset_peak_stats() profiler.snapshot("After model loading") # Forward with hooks activations = {} def save_activation(name): def hook(module, input, output): if isinstance(output, torch.Tensor): activations[name] = output.numel() * output.element_size() / 1e6 return hook for name, module in model.named_modules(): module.register_forward_hook(save_activation(name)) # Forward pass with torch.no_grad(): _ = model(x) profiler.snapshot("After forward pass") # Print activation memory print("\nActivation memory per layer:") for name, mem in sorted(activations.items(), key=lambda x: -x[1])[:10]: print(f" {name}: {mem:.2f} MB")
Gradient Checkpointing
During the forward pass, PyTorch saves every intermediate activation because it needs them during the backward pass to compute gradients. For a 24-layer transformer, this means storing 24 layers worth of intermediate tensors simultaneously. Gradient checkpointing (also called activation checkpointing or rematerialization) offers a simple trade: do not save intermediate activations — instead, recompute them on-the-fly during the backward pass. You save memory at the cost of running parts of the forward pass twice.The math is elegant: for n layers, standard training stores O(n) activations. With checkpointing every n layers, you store only O(n) activations and recompute the rest, at the cost of about 33% extra compute. For a 24-layer model, that is storing 5 checkpoints instead of 24 activation sets — a nearly 5x memory reduction for activations.
class GradientCheckpointing: """ Trade compute for memory by recomputing activations. Instead of storing all activations for backward pass, we store only at checkpoint boundaries and recompute intermediate activations during backward. Memory savings: O(sqrt(n)) vs O(n) for n layers Compute overhead: ~30-40% more forward passes When to use: When activation memory (not parameter/optimizer memory) is your bottleneck. This is common for models with many layers (deep transformers), large spatial dimensions (segmentation), or long sequences (NLP with 4K+ tokens). """ @staticmethod def checkpoint_sequential( functions: List[nn.Module], segments: int, input: torch.Tensor ) -> torch.Tensor: """ Apply checkpointing to sequential modules. Args: functions: List of modules to apply sequentially segments: Number of checkpoint segments input: Input tensor """ return checkpoint_sequential(functions, segments, input) @staticmethod def checkpoint_function( function: Callable, *args, use_reentrant: bool = False ) -> torch.Tensor: """ Checkpoint a single function. Args: function: Function to checkpoint *args: Function arguments use_reentrant: Use reentrant version (False recommended) """ return checkpoint(function, *args, use_reentrant=use_reentrant)# Transformer with gradient checkpointingclass CheckpointedTransformerBlock(nn.Module): """ Transformer block with optional gradient checkpointing. """ def __init__( self, d_model: int, nhead: int, dim_feedforward: int = 2048, dropout: float = 0.1, use_checkpoint: bool = False ): super().__init__() self.use_checkpoint = use_checkpoint # Self-attention self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True) self.norm1 = nn.LayerNorm(d_model) # Feedforward self.ff = nn.Sequential( nn.Linear(d_model, dim_feedforward), nn.GELU(), nn.Dropout(dropout), nn.Linear(dim_feedforward, d_model), nn.Dropout(dropout) ) self.norm2 = nn.LayerNorm(d_model) self.dropout = nn.Dropout(dropout) def _attention_block(self, x: torch.Tensor) -> torch.Tensor: """Self-attention sub-block.""" attn_out, _ = self.self_attn(x, x, x) return self.dropout(attn_out) def _ff_block(self, x: torch.Tensor) -> torch.Tensor: """Feedforward sub-block.""" return self.ff(x) def forward(self, x: torch.Tensor) -> torch.Tensor: if self.use_checkpoint and self.training: # Checkpoint attention attn_out = checkpoint( self._attention_block, x, use_reentrant=False ) x = self.norm1(x + attn_out) # Checkpoint feedforward ff_out = checkpoint( self._ff_block, x, use_reentrant=False ) x = self.norm2(x + ff_out) else: # Standard forward x = self.norm1(x + self._attention_block(x)) x = self.norm2(x + self._ff_block(x)) return xclass CheckpointedTransformer(nn.Module): """ Full transformer with gradient checkpointing. """ def __init__( self, d_model: int = 512, nhead: int = 8, num_layers: int = 6, vocab_size: int = 10000, use_checkpoint: bool = True ): super().__init__() self.embedding = nn.Embedding(vocab_size, d_model) self.layers = nn.ModuleList([ CheckpointedTransformerBlock( d_model, nhead, use_checkpoint=use_checkpoint ) for _ in range(num_layers) ]) self.output = nn.Linear(d_model, vocab_size) def forward(self, x: torch.Tensor) -> torch.Tensor: x = self.embedding(x) for layer in self.layers: x = layer(x) return self.output(x)# Compare memory with/without checkpointingdef compare_checkpointing_memory(): """Compare memory usage with and without checkpointing.""" if not torch.cuda.is_available(): print("CUDA not available") return batch_size, seq_len = 32, 512 d_model, num_layers = 512, 12 # Without checkpointing model_no_ckpt = CheckpointedTransformer( d_model=d_model, num_layers=num_layers, use_checkpoint=False ).to(device) torch.cuda.reset_peak_memory_stats() x = torch.randint(0, 10000, (batch_size, seq_len)).to(device) out = model_no_ckpt(x) loss = out.sum() loss.backward() mem_no_ckpt = torch.cuda.max_memory_allocated() / 1e9 del model_no_ckpt, out, loss torch.cuda.empty_cache() # With checkpointing model_ckpt = CheckpointedTransformer( d_model=d_model, num_layers=num_layers, use_checkpoint=True ).to(device) torch.cuda.reset_peak_memory_stats() out = model_ckpt(x) loss = out.sum() loss.backward() mem_ckpt = torch.cuda.max_memory_allocated() / 1e9 print(f"Without checkpointing: {mem_no_ckpt:.2f} GB") print(f"With checkpointing: {mem_ckpt:.2f} GB") print(f"Memory saved: {(1 - mem_ckpt/mem_no_ckpt)*100:.1f}%")
Mixed Precision Training
Mixed precision training is the single highest-impact memory optimization for most practitioners. The idea: use 16-bit floating point (FP16 or BF16) for the computationally intensive operations (matrix multiplies, convolutions) while keeping 32-bit precision where it matters (loss computation, gradient accumulation, optimizer states). Modern GPUs have dedicated hardware (Tensor Cores) that run FP16 matrix multiplies at 2-8x the speed of FP32, so you get both memory and speed benefits.The tricky part is avoiding numerical issues. FP16 has a very narrow dynamic range — gradients can underflow to zero (too small to represent) or overflow to infinity. The solution is gradient scaling: multiply the loss by a large number before backward, then divide the gradients by the same number before the optimizer step. This shifts the gradient values into FP16’s representable range. BFloat16 (BF16) avoids this entirely because it has the same exponent range as FP32, at the cost of less precision in the mantissa.
Do not use FP16 mixed precision for fine-tuning large language models. BF16 is strongly preferred because FP16’s limited range causes gradient underflow on very small learning rates (1e-5 range) that are typical for LLM fine-tuning. If your GPU supports BF16 (Ampere or newer), always prefer it.
class MixedPrecisionTraining: """ Train with mixed FP16/FP32 precision. Benefits: - 2x memory reduction for activations and model weights - 2-8x faster matrix multiplications on Tensor Core GPUs - Minimal accuracy loss with proper gradient scaling Key components: - Automatic casting (autocast): PyTorch decides which ops run in FP16 - Gradient scaling (GradScaler): prevents FP16 gradient underflow Typical setup: 3 lines of code change (autocast context, scale loss, scaler.step instead of optimizer.step). """ def __init__(self, enabled: bool = True): self.enabled = enabled self.scaler = torch.amp.GradScaler('cuda') if enabled else None def training_step( self, model: nn.Module, batch: tuple, optimizer: optim.Optimizer, criterion: nn.Module ) -> float: """ Perform one training step with mixed precision. """ inputs, targets = batch optimizer.zero_grad() if self.enabled: # Mixed precision forward with torch.amp.autocast('cuda'): outputs = model(inputs) loss = criterion(outputs, targets) # Scaled backward self.scaler.scale(loss).backward() # Unscale gradients for clipping self.scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # Optimizer step with scaling self.scaler.step(optimizer) self.scaler.update() else: # Standard precision outputs = model(inputs) loss = criterion(outputs, targets) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() return loss.item()# BFloat16 for better stabilityclass BFloat16Training: """ BFloat16 training (available on Ampere+ GPUs). Advantages over FP16: - Same exponent range as FP32 - No gradient scaling needed - Better numerical stability """ @staticmethod def is_available() -> bool: """Check if BF16 is available.""" if not torch.cuda.is_available(): return False return torch.cuda.get_device_capability()[0] >= 8 @staticmethod def training_step( model: nn.Module, batch: tuple, optimizer: optim.Optimizer, criterion: nn.Module ) -> float: """Training step with BF16.""" inputs, targets = batch optimizer.zero_grad() with torch.amp.autocast('cuda', dtype=torch.bfloat16): outputs = model(inputs) loss = criterion(outputs, targets) # No scaling needed for BF16! loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() return loss.item()# Precision comparisondef demonstrate_precision_formats(): """Show memory usage for different precisions.""" num_params = 1_000_000 # 1M parameters formats = { 'float32': (4, "Standard precision"), 'float16': (2, "Half precision"), 'bfloat16': (2, "Brain float"), 'int8': (1, "Quantized"), } print("Memory per million parameters:") print("-" * 45) for name, (bytes_per, desc) in formats.items(): mem = num_params * bytes_per / 1e6 print(f"{name:10} ({desc:20}): {mem:.1f} MB")demonstrate_precision_formats()
CPU Offloading
CPU offloading exploits the fact that your machine typically has 10-50x more CPU RAM than GPU memory. The idea is simple: keep only what the GPU needs right now on the GPU, and store everything else on the CPU (or even NVMe). This is the same principle behind ZeRO-Offload in DeepSpeed and FSDP’s CPU offloading in PyTorch. The trade-off is straightforward: you trade training speed (because of PCIe data transfers) for the ability to train models that otherwise would not fit at all.
CPU offloading is most effective for optimizer states (which are only needed during the parameter update step, not during forward/backward). Offloading activations is less effective because they are accessed frequently during the backward pass, making the PCIe bandwidth a bottleneck. If you are using DeepSpeed, start with ZeRO Stage 2 + CPU offloading of optimizer states before trying full activation offloading.
class CPUOffloadOptimizer: """ Offload optimizer states to CPU to save GPU memory. Trade-off: Slower training (CPU<->GPU transfers over PCIe) but can train models 2-4x larger than what fits in GPU memory alone. When to use: When your model fits on the GPU for forward/backward but OOMs during the optimizer step (because Adam doubles the memory footprint with its momentum and variance states). """ def __init__( self, model: nn.Module, lr: float = 1e-4, offload: bool = True ): self.model = model self.offload = offload if offload: # Keep parameters on GPU but states on CPU self.param_groups = [ { 'params': list(model.parameters()), 'lr': lr } ] # Initialize optimizer states on CPU self.state = {} for p in model.parameters(): if p.requires_grad: # Adam states self.state[p] = { 'm': torch.zeros_like(p.data, device='cpu'), 'v': torch.zeros_like(p.data, device='cpu'), 't': 0 } else: self.optimizer = optim.AdamW(model.parameters(), lr=lr) def zero_grad(self): """Zero gradients.""" if self.offload: for p in self.model.parameters(): if p.grad is not None: p.grad.zero_() else: self.optimizer.zero_grad() def step(self): """Perform optimizer step.""" if not self.offload: self.optimizer.step() return lr = self.param_groups[0]['lr'] beta1, beta2 = 0.9, 0.999 eps = 1e-8 for p in self.model.parameters(): if not p.requires_grad or p.grad is None: continue state = self.state[p] state['t'] += 1 t = state['t'] # Move gradient to CPU grad_cpu = p.grad.data.cpu() # Update moments on CPU state['m'].mul_(beta1).add_(grad_cpu, alpha=1 - beta1) state['v'].mul_(beta2).addcmul_(grad_cpu, grad_cpu, value=1 - beta2) # Bias correction m_hat = state['m'] / (1 - beta1 ** t) v_hat = state['v'] / (1 - beta2 ** t) # Compute update on CPU update = m_hat / (v_hat.sqrt() + eps) # Apply update to GPU parameter p.data.add_(update.to(p.device), alpha=-lr)class GradientOffloading: """ Offload gradients to CPU during backward pass. Useful for very large models where even gradients don't fit. """ def __init__(self, model: nn.Module): self.model = model self.cpu_gradients = {} self._register_hooks() def _register_hooks(self): """Register hooks to offload gradients.""" for name, param in self.model.named_parameters(): if param.requires_grad: param.register_post_accumulate_grad_hook( self._make_hook(name) ) def _make_hook(self, name: str): def hook(param): if param.grad is not None: # Move gradient to CPU self.cpu_gradients[name] = param.grad.cpu() # Free GPU memory param.grad = None return hook def restore_gradients(self): """Move gradients back to GPU for optimizer.""" for name, param in self.model.named_parameters(): if name in self.cpu_gradients: param.grad = self.cpu_gradients[name].to(param.device)
Activation Recomputation Strategies
class SelectiveCheckpointing: """ Smart checkpointing strategies. Not all layers benefit equally from checkpointing. Focus on high-memory layers. """ @staticmethod def should_checkpoint(module: nn.Module) -> bool: """ Decide if a module should be checkpointed. Rules: - Checkpoint attention (quadratic memory) - Checkpoint large FFN layers - Don't checkpoint small layers (overhead not worth it) """ if isinstance(module, nn.MultiheadAttention): return True # Check for large linear layers if isinstance(module, nn.Linear): return module.in_features * module.out_features > 1e6 return False @staticmethod def estimate_activation_memory( module: nn.Module, input_shape: tuple ) -> float: """ Estimate activation memory for a module. Returns memory in MB. """ # This is a simplified estimate batch_size = input_shape[0] if isinstance(module, nn.Linear): # Output activations return batch_size * module.out_features * 4 / 1e6 if isinstance(module, nn.MultiheadAttention): # Attention weights (quadratic in sequence length) seq_len = input_shape[1] num_heads = module.num_heads return batch_size * num_heads * seq_len * seq_len * 4 / 1e6 return 0.0class LayerWiseCheckpointing(nn.Module): """ Apply checkpointing at layer granularity. """ def __init__( self, layers: nn.ModuleList, checkpoint_ratio: float = 0.5 ): super().__init__() self.layers = layers # Checkpoint every other layer (or based on ratio) self.checkpoint_mask = [ i % int(1 / checkpoint_ratio) == 0 for i in range(len(layers)) ] def forward(self, x: torch.Tensor) -> torch.Tensor: for i, layer in enumerate(self.layers): if self.checkpoint_mask[i] and self.training: x = checkpoint(layer, x, use_reentrant=False) else: x = layer(x) return x
Memory-Efficient Attention
Attention is often the single largest memory consumer in transformer models. Standard self-attention materializes an N×N attention matrix, where N is the sequence length. For a 4096-token sequence with 32 attention heads in float16, that single matrix consumes 32×4096×4096×2 bytes = 1 GB. Double the sequence length and the memory quadruples. The methods below tackle this bottleneck from different angles.
If you are using PyTorch 2.0+ or later, use torch.nn.functional.scaled_dot_product_attention with is_causal=True or attn_mask. PyTorch will automatically select the most efficient backend (FlashAttention, memory-efficient attention, or math fallback) based on your hardware, input shapes, and whether you need dropout. This is almost always the right choice — only implement custom attention when you need something the fused kernel does not support.
class MemoryEfficientAttention(nn.Module): """ Memory-efficient attention implementations. Standard attention: O(n^2) memory for the full attention matrix. Chunked attention: O(n * chunk_size) memory -- process queries in chunks, computing attention scores against all keys/values for each chunk. Flash Attention: O(n) memory -- never materializes the full matrix at all, using a tiling algorithm that stays in GPU SRAM. """ def __init__( self, d_model: int, num_heads: int, chunk_size: int = 1024 ): super().__init__() self.d_model = d_model self.num_heads = num_heads self.head_dim = d_model // num_heads self.chunk_size = chunk_size self.qkv = nn.Linear(d_model, 3 * d_model) self.out_proj = nn.Linear(d_model, d_model) def forward(self, x: torch.Tensor) -> torch.Tensor: B, N, C = x.shape # Compute Q, K, V qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim) qkv = qkv.permute(2, 0, 3, 1, 4) # (3, B, H, N, D) q, k, v = qkv[0], qkv[1], qkv[2] # Chunked attention to reduce peak memory output = self.chunked_attention(q, k, v) output = output.transpose(1, 2).reshape(B, N, C) return self.out_proj(output) def chunked_attention( self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor ) -> torch.Tensor: """ Compute attention in chunks. Instead of computing full NxN attention matrix, process in chunks to reduce memory. """ B, H, N, D = q.shape scale = D ** -0.5 output = torch.zeros_like(v) for i in range(0, N, self.chunk_size): end_i = min(i + self.chunk_size, N) q_chunk = q[:, :, i:end_i] # Compute attention for this query chunk attn = torch.matmul(q_chunk, k.transpose(-2, -1)) * scale attn = attn.softmax(dim=-1) output[:, :, i:end_i] = torch.matmul(attn, v) return outputclass FlashAttention(nn.Module): """ Flash Attention wrapper (requires flash-attn package). Key innovations: - Tiling to fit in SRAM - No materialization of attention matrix - IO-aware algorithm design Memory: O(n) instead of O(n²) Speed: 2-4x faster than standard attention """ def __init__(self, d_model: int, num_heads: int, dropout: float = 0.0): super().__init__() self.d_model = d_model self.num_heads = num_heads self.head_dim = d_model // num_heads self.dropout = dropout self.qkv = nn.Linear(d_model, 3 * d_model) self.out_proj = nn.Linear(d_model, d_model) self._check_flash_available() def _check_flash_available(self): """Check if Flash Attention is available.""" try: from flash_attn import flash_attn_func self.flash_attn = flash_attn_func self.use_flash = True except ImportError: print("Flash Attention not available. Using standard attention.") self.use_flash = False def forward(self, x: torch.Tensor) -> torch.Tensor: B, N, C = x.shape qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim) if self.use_flash: # Flash attention format: (B, N, H, D) q = qkv[:, :, 0].transpose(1, 2) k = qkv[:, :, 1].transpose(1, 2) v = qkv[:, :, 2].transpose(1, 2) out = self.flash_attn(q, k, v, dropout_p=self.dropout if self.training else 0.0) out = out.transpose(1, 2).reshape(B, N, C) else: # Standard attention qkv = qkv.permute(2, 0, 3, 1, 4) q, k, v = qkv[0], qkv[1], qkv[2] scale = self.head_dim ** -0.5 attn = (q @ k.transpose(-2, -1)) * scale attn = attn.softmax(dim=-1) out = (attn @ v).transpose(1, 2).reshape(B, N, C) return self.out_proj(out)
Efficient Batch Processing
class GradientAccumulation: """ Accumulate gradients over multiple mini-batches. Simulates larger batch sizes without memory increase. """ def __init__( self, model: nn.Module, optimizer: optim.Optimizer, accumulation_steps: int = 4 ): self.model = model self.optimizer = optimizer self.accumulation_steps = accumulation_steps self.current_step = 0 def backward(self, loss: torch.Tensor): """ Backward with gradient accumulation. """ # Scale loss by accumulation steps scaled_loss = loss / self.accumulation_steps scaled_loss.backward() self.current_step += 1 if self.current_step >= self.accumulation_steps: # Perform optimizer step self.optimizer.step() self.optimizer.zero_grad() self.current_step = 0 return True # Step was performed return False # Step not performed yetclass MicroBatching: """ Split batches into micro-batches for forward pass. Useful when single batch doesn't fit but you need batch normalization statistics from full batch. """ @staticmethod def micro_batch_forward( model: nn.Module, batch: torch.Tensor, micro_batch_size: int ) -> torch.Tensor: """ Forward in micro-batches. """ outputs = [] for i in range(0, batch.size(0), micro_batch_size): micro_batch = batch[i:i + micro_batch_size] with torch.no_grad() if not model.training else torch.enable_grad(): output = model(micro_batch) outputs.append(output) return torch.cat(outputs, dim=0)class InPlaceOperations: """ In-place operations to reduce memory allocations. Use with caution - can break gradient computation! """ @staticmethod def inplace_relu(x: torch.Tensor) -> torch.Tensor: """In-place ReLU.""" return torch.relu_(x) @staticmethod def inplace_add(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: """In-place addition.""" return x.add_(y) @staticmethod def check_safe_inplace(tensor: torch.Tensor) -> bool: """ Check if in-place operation is safe. Not safe if: - Tensor requires grad and is a leaf - Tensor is used elsewhere in computation graph """ if tensor.requires_grad and tensor.is_leaf: return False return True
Memory-Efficient Data Loading
class MemoryEfficientDataLoader: """ Reduce data loading memory overhead. """ @staticmethod def create_efficient_loader( dataset, batch_size: int, num_workers: int = 4, pin_memory: bool = True ): """ Create memory-efficient data loader. Tips: - pin_memory for faster GPU transfer - Appropriate num_workers (usually 4 per GPU) - prefetch_factor controls memory usage """ return torch.utils.data.DataLoader( dataset, batch_size=batch_size, shuffle=True, num_workers=num_workers, pin_memory=pin_memory, # Limit prefetching to reduce memory prefetch_factor=2 if num_workers > 0 else None, # Don't copy tensors persistent_workers=True if num_workers > 0 else False, ) @staticmethod def use_memory_mapping(file_path: str): """ Use memory-mapped files for large datasets. Data stays on disk, loaded on demand. """ return np.memmap(file_path, dtype='float32', mode='r')class LowMemoryDataset(torch.utils.data.Dataset): """ Dataset that loads data on-demand. """ def __init__(self, file_paths: List[str]): self.file_paths = file_paths def __len__(self): return len(self.file_paths) def __getitem__(self, idx): # Load only when needed data = torch.load(self.file_paths[idx]) return data
Memory Debugging Tools
class MemoryDebugger: """ Tools for debugging memory issues. """ @staticmethod def find_memory_leaks(): """Find potential memory leaks.""" if not torch.cuda.is_available(): return [] gc.collect() torch.cuda.empty_cache() leaks = [] for obj in gc.get_objects(): if isinstance(obj, torch.Tensor): if obj.is_cuda: leaks.append({ 'shape': obj.shape, 'dtype': obj.dtype, 'size_mb': obj.numel() * obj.element_size() / 1e6, 'requires_grad': obj.requires_grad }) return sorted(leaks, key=lambda x: -x['size_mb']) @staticmethod def memory_summary(): """Print detailed memory summary.""" if not torch.cuda.is_available(): print("CUDA not available") return print(torch.cuda.memory_summary()) @staticmethod def clear_memory(): """Aggressively clear GPU memory.""" gc.collect() if torch.cuda.is_available(): torch.cuda.empty_cache() torch.cuda.synchronize()# Usage demonstrationdef memory_optimization_demo(): """Demonstrate memory optimization techniques.""" summary = """ ╔════════════════════════════════════════════════════════════════╗ ║ MEMORY OPTIMIZATION TECHNIQUES ║ ╠════════════════════════════════════════════════════════════════╣ ║ ║ ║ TECHNIQUE MEMORY SAVING COMPUTE COST ║ ║ ───────────────────────────────────────────────────────────── ║ ║ Mixed Precision ~50% ~0% (often faster!) ║ ║ Gradient Checkpointing ~60-70% ~30-40% ║ ║ Gradient Accumulation Linear in steps ~0% ║ ║ Flash Attention ~50% for attn ~0% (often faster!) ║ ║ CPU Offloading ~50-70% ~20-50% ║ ║ ║ ╠════════════════════════════════════════════════════════════════╣ ║ RECOMMENDATION ORDER: ║ ║ 1. Enable mixed precision (always) ║ ║ 2. Use Flash Attention (if available) ║ ║ 3. Add gradient checkpointing ║ ║ 4. Gradient accumulation for larger effective batch ║ ║ 5. CPU offloading as last resort ║ ╚════════════════════════════════════════════════════════════════╝ """ print(summary)memory_optimization_demo()
Exercises
Exercise 1: Profile Your Model
Use the MemoryProfiler to analyze a model’s memory usage at each layer.
Identify the biggest memory consumers and optimize them.
Exercise 2: Implement Selective Checkpointing
Create a system that automatically decides which layers to checkpoint
based on their memory usage vs. compute cost.
Exercise 3: Memory Budget Training
Implement training that stays within a fixed memory budget by
dynamically adjusting batch size and checkpointing.
What’s Next?
Quantization Deep Dive
Post-training and quantization-aware training
Knowledge Distillation
Transfer knowledge to smaller models
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Dev Weekends","colors":{"primary":"#000000","light":"#6B7280","dark":"#000000"},"favicon":"https://mintcdn.com/devweeekends/uFspW_ITEafc8XbV/images/logo.png?fit=max\u0026auto=format\u0026n=uFspW_ITEafc8XbV\u0026q=85\u0026s=24941b53d3f0793eb83974749e82df4f","navbar":{"links":[{"href":"https://devweekends.com","label":"Back to Site"}]},"footer":{"socials":{"youtube":"https://www.youtube.com/@devweekends","github":"https://github.com/DevWeekends","linkedin":"https://www.linkedin.com/company/dev-weekends"}},"contextual":{"options":["copy","view","chatgpt","claude","perplexity","mcp","cursor","vscode"],"display":"header"},"styling":{"latex":true},"navigation":{"global":{"anchors":[{"anchor":"DevWeekends","icon":"globe","href":"https://devweekends.com"}]},"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"devweeekends","hasAudioNarration":false,"actualSubdomain":"devweeekends","customDomains":["resources.devweekends.com"],"gitSource":{"type":"github","owner":"devweekends","repo":"docs","deployBranch":"main","contentDirectory":"","isPrivate":false,"mountPath":"","label":""},"entitlements":{},"buildId":"6ad5cb977e3343e90aaa7f7f","clientVersion":"0.0.4205","searchFilterCounts":[{"language":"default","tag":"Courses","count":785},{"language":"default","tag":"Crash Courses","count":145},{"language":"default","tag":"Interview Prep","count":69},{"language":"default","tag":"AI Engineering","count":60},{"language":"default","tag":"Distributed System Tools","count":60},{"language":"default","tag":"Deep Learning Mastery","count":48},{"language":"default","tag":"DevOps Tools","count":38},{"language":"default","tag":"Think Like an Engineer","count":38},{"language":"default","tag":"Competitive Programming","count":37},{"language":"default","tag":"Angular Master Course","count":34}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"treatment","enabled":true,"payload":{"type":"string","value":"treatment"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[],"page":{"pagePath":"courses/deep-learning-mastery/44-memory-efficient-training","slug":"/courses/deep-learning-mastery/44-memory-efficient-training","pageMetadata":{"title":"Memory-Efficient Training","sidebarTitle":"Memory-Efficient Training","description":"Gradient checkpointing, mixed precision, and memory optimization","icon":"memory","href":"/courses/deep-learning-mastery/44-memory-efficient-training"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Gradient checkpointing, mixed precision, and memory optimization\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={annotation:`annotation`,code:`code`,hr:`hr`,li:`li`,math:`math`,mi:`mi`,mn:`mn`,mo:`mo`,mrow:`mrow`,msqrt:`msqrt`,mtext:`mtext`,p:`p`,path:`path`,pre:`pre`,semantics:`semantics`,span:`span`,strong:`strong`,svg:`svg`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,ul:`ul`,..._provideComponents(),...props.components},{Accordion,AccordionGroup,Card,CardGroup,CodeBlock,Frame,Heading,OptimizedImage,Table,Tip,Warning}=_components;if(!Accordion)_missingMdxReference(`Accordion`,true);if(!AccordionGroup)_missingMdxReference(`AccordionGroup`,true);if(!Card)_missingMdxReference(`Card`,true);if(!CardGroup)_missingMdxReference(`CardGroup`,true);if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Frame)_missingMdxReference(`Frame`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!OptimizedImage)_missingMdxReference(`OptimizedImage`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Tip)_missingMdxReference(`Tip`,true);if(!Warning)_missingMdxReference(`Warning`,true);return _jsxs(_Fragment,{children:[_jsx(Frame,{children:_jsx(OptimizedImage,{src:`https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/memory-efficient-training-concept.svg?fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=7e25b6bedd5a25c8e08b1a0fad35a62d`,alt:`Memory-Efficient Training`,\"data-og-width\":`1080`,width:`1080`,\"data-og-height\":`1080`,height:`1080`,\"data-path\":`images/courses/deep-learning-mastery/memory-efficient-training-concept.svg`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/memory-efficient-training-concept.svg?w=280\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=992a471c4f8f52a48f07b618395ceb4a 280w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/memory-efficient-training-concept.svg?w=560\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=a3daf2737df96ab1c61e04dcd223130c 560w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/memory-efficient-training-concept.svg?w=840\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=f24f1a7cd3b0de1ffa13a3993d645721 840w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/memory-efficient-training-concept.svg?w=1100\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=2614650603adce0e104f1819b4c4e223 1100w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/memory-efficient-training-concept.svg?w=1650\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=946283548785fc46f0451371279a013e 1650w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/memory-efficient-training-concept.svg?w=2500\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=acd7f2818c043679b08163431219de29 2500w`})}),`\n`,_jsx(Heading,{level:`1`,id:`memory-efficient-training`,children:`Memory-Efficient Training`}),`\n`,_jsx(Heading,{level:`2`,id:`understanding-gpu-memory`,children:`Understanding GPU Memory`}),`\n`,_jsxs(_components.p,{children:[`GPU memory during training is like a hotel with a fixed number of rooms: your model parameters check in, then their gradients need rooms, then the optimizer’s bookkeeping (momentum, variance) needs even more rooms, and finally every intermediate computation (activation) holds a room open until the backward pass checks it out. When the hotel is full, training crashes with the dreaded `,_jsx(_components.code,{children:`CUDA out of memory`}),` error. Understanding who occupies which rooms — and for how long — is the key to training larger models on the hardware you actually have.`]}),`\n`,_jsx(_components.p,{children:`Where does memory go during training?`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Component`}),_jsx(_components.th,{children:`Memory Usage`}),_jsx(_components.th,{children:`Analogy`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Model parameters`})}),_jsxs(_components.td,{children:[_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mn,{children:`4`}),_jsx(_components.mo,{children:`×`}),_jsx(_components.mtext,{children:`params`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`4 \\\\times \\\\text{params}`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7278em`,verticalAlign:`-0.0833em`}}),_jsx(_components.span,{className:`mord`,children:`4`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`×`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.625em`,verticalAlign:`-0.1944em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`params`})})]})]})]}),` (fp32)`]}),_jsx(_components.td,{children:`The permanent residents`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Gradients`})}),_jsxs(_components.td,{children:[_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mn,{children:`4`}),_jsx(_components.mo,{children:`×`}),_jsx(_components.mtext,{children:`params`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`4 \\\\times \\\\text{params}`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7278em`,verticalAlign:`-0.0833em`}}),_jsx(_components.span,{className:`mord`,children:`4`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`×`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.625em`,verticalAlign:`-0.1944em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`params`})})]})]})]}),` (fp32)`]}),_jsx(_components.td,{children:`Their shadows (same size)`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Optimizer states`})}),_jsxs(_components.td,{children:[_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mn,{children:`8`}),_jsx(_components.mo,{children:`×`}),_jsx(_components.mtext,{children:`params`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`8 \\\\times \\\\text{params}`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7278em`,verticalAlign:`-0.0833em`}}),_jsx(_components.span,{className:`mord`,children:`8`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`×`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.625em`,verticalAlign:`-0.1944em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`params`})})]})]})]}),` (Adam)`]}),_jsx(_components.td,{children:`Adam keeps two running averages per parameter`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Activations`})}),_jsx(_components.td,{children:`Depends on batch size, model depth`}),_jsx(_components.td,{children:`The guests — they check in during forward, check out during backward`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Temporary buffers`})}),_jsx(_components.td,{children:`Variable`}),_jsx(_components.td,{children:`Room service supplies`})]})]})]}),`\n`,_jsx(_components.p,{children:`For a 7B parameter model:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Parameters: 28 GB`}),`\n`,_jsx(_components.li,{children:`Gradients: 28 GB`}),`\n`,_jsx(_components.li,{children:`Adam states: 56 GB`}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Total: ~112 GB`}),` (without activations!)`]}),`\n`]}),`\n`,_jsx(Tip,{children:_jsx(_components.p,{children:`The single most impactful optimization for most teams is mixed precision training (FP16/BF16). It nearly halves memory for parameters and activations with minimal accuracy impact, and it is a one-line change in most frameworks. Start there before reaching for more complex techniques like gradient checkpointing or model parallelism.`})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`10`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`10`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.optim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optim`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`from`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.utils.checkpoint `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` checkpoint, checkpoint_sequential`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`from`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` typing `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` Optional, Tuple, List, Callable`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` gc`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` functools`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`torch.manual_seed(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`42`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`device `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.device(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"cuda\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`else`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"cpu\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`memory-profiling`,children:`Memory Profiling`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`101`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`101`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MemoryProfiler`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Profile GPU memory usage during training.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`device`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.device `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` device):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.device `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` device`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.snapshots `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` []`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` get_memory_stats`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`dict`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Get current memory statistics.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'allocated'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'cached'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'max_allocated'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'allocated_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.cuda.memory_allocated(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.device) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'cached_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.cuda.memory_reserved(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.device) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'max_allocated_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.cuda.max_memory_allocated(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.device) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` snapshot`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`label`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`str`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Take a memory snapshot.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` stats `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.get_memory_stats()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` stats[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'label'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` label`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.snapshots.append(stats)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` stats`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` reset_peak_stats`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Reset peak memory tracking.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.reset_peak_memory_stats()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` print_summary`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Print memory usage summary.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:`\\\\n`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` +`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"=\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`60`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"MEMORY USAGE SUMMARY\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"=\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`60`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` snap `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.snapshots:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`snap[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'label'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:30}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` | \"`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Allocated: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`snap[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'allocated_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:8.1f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` MB | \"`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Peak: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`snap[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'max_allocated_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:8.1f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` MB\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` estimate_model_memory`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module) -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`dict`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Estimate memory requirements for a model.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` sum`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(p.numel() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.element_size() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.parameters())`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` buffer_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` sum`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(b.numel() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` b.element_size() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` b `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.buffers())`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Gradient memory (same as params for most cases)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` grad_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param_mem`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Optimizer states (Adam: 2x for m and v)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` adam_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param_mem`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'parameters_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: param_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'buffers_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: buffer_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'gradients_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: grad_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'adam_states_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: adam_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'total_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: (param_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` buffer_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` grad_mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` adam_mem) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Memory-efficient model analysis`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` analyze_memory_per_layer`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`input_shape`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`tuple`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Analyze memory usage per layer.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` profiler `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` MemoryProfiler()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"CUDA not available for memory analysis\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.to(device)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`input_shape).to(device)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` profiler.reset_peak_stats()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` profiler.snapshot(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"After model loading\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Forward with hooks`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` activations `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {}`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` save_activation`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`name`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` hook`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`input`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`output`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` isinstance`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(output, torch.Tensor):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` activations[name] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output.numel() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output.element_size() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` hook`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` name, module `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.named_modules():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` module.register_forward_hook(save_activation(name))`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Forward pass`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` with`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.no_grad():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` _ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` profiler.snapshot(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"After forward pass\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Print activation memory`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:`\\\\n`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`Activation memory per layer:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` name, mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` sorted`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(activations.items(), `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`key`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`lambda`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`x[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`])[:`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`name`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mem`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:.2f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` MB\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`gradient-checkpointing`,children:`Gradient Checkpointing`}),`\n`,_jsx(_components.p,{children:`During the forward pass, PyTorch saves every intermediate activation because it needs them during the backward pass to compute gradients. For a 24-layer transformer, this means storing 24 layers worth of intermediate tensors simultaneously. Gradient checkpointing (also called activation checkpointing or rematerialization) offers a simple trade: do not save intermediate activations — instead, recompute them on-the-fly during the backward pass. You save memory at the cost of running parts of the forward pass twice.`}),`\n`,_jsxs(_components.p,{children:[`The math is elegant: for `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`n`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`n`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.4306em`}}),_jsx(_components.span,{className:`mord mathnormal`,children:`n`})]})})]}),` layers, standard training stores `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`O`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.mi,{children:`n`}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`O(n)`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0278em`},children:`O`}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsx(_components.span,{className:`mord mathnormal`,children:`n`}),_jsx(_components.span,{className:`mclose`,children:`)`})]})})]}),` activations. With checkpointing every `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.msqrt,{children:_jsx(_components.mi,{children:`n`})})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\sqrt{n}`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.04em`,verticalAlign:`-0.2397em`}}),_jsx(_components.span,{className:`mord sqrt`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8003em`},children:[_jsxs(_components.span,{className:`svg-align`,style:{top:`-3em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,style:{paddingLeft:`0.833em`},children:_jsx(_components.span,{className:`mord mathnormal`,children:`n`})})]}),_jsxs(_components.span,{style:{top:`-2.7603em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`hide-tail`,style:{minWidth:`0.853em`,height:`1.08em`},children:_jsx(_components.svg,{xmlns:`http://www.w3.org/2000/svg`,width:`400em`,height:`1.08em`,viewBox:`0 0 400000 1080`,preserveAspectRatio:`xMinYMin slice`,children:_jsx(_components.path,{d:`M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2397em`},children:_jsx(_components.span,{})})})]})})]})})]}),` layers, you store only `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`O`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.msqrt,{children:_jsx(_components.mi,{children:`n`})}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`O(\\\\sqrt{n})`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.0503em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0278em`},children:`O`}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsx(_components.span,{className:`mord sqrt`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8003em`},children:[_jsxs(_components.span,{className:`svg-align`,style:{top:`-3em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,style:{paddingLeft:`0.833em`},children:_jsx(_components.span,{className:`mord mathnormal`,children:`n`})})]}),_jsxs(_components.span,{style:{top:`-2.7603em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`hide-tail`,style:{minWidth:`0.853em`,height:`1.08em`},children:_jsx(_components.svg,{xmlns:`http://www.w3.org/2000/svg`,width:`400em`,height:`1.08em`,viewBox:`0 0 400000 1080`,preserveAspectRatio:`xMinYMin slice`,children:_jsx(_components.path,{d:`M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2397em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mclose`,children:`)`})]})})]}),` activations and recompute the rest, at the cost of about 33% extra compute. For a 24-layer model, that is storing 5 checkpoints instead of 24 activation sets — a nearly 5x memory reduction for activations.`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`200`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`200`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` GradientCheckpointing`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Trade compute for memory by recomputing activations.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Instead of storing all activations for backward pass,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` we store only at checkpoint boundaries and recompute`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` intermediate activations during backward.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Memory savings: O(sqrt(n)) vs O(n) for n layers`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Compute overhead: ~30-40% more forward passes`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` When to use: When activation memory (not parameter/optimizer memory)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` is your bottleneck. This is common for models with many layers`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` (deep transformers), large spatial dimensions (segmentation), or`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` long sequences (NLP with 4K+ tokens).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` checkpoint_sequential`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` functions`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: List[nn.Module],`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` segments`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` input`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ) -> torch.Tensor:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Apply checkpointing to sequential modules.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` functions: List of modules to apply sequentially`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` segments: Number of checkpoint segments`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` input: Input tensor`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` checkpoint_sequential(functions, segments, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`input`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` checkpoint_function`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` function`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: Callable,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`args`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` use_reentrant`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ) -> torch.Tensor:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Checkpoint a single function.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` function: Function to checkpoint`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` *args: Function arguments`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` use_reentrant: Use reentrant version (False recommended)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` checkpoint(function, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`args, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`use_reentrant`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`use_reentrant)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Transformer with gradient checkpointing`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` CheckpointedTransformerBlock`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Transformer block with optional gradient checkpointing.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` nhead`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` dim_feedforward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2048`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` dropout`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` use_checkpoint`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ):`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.use_checkpoint `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` use_checkpoint`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Self-attention`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.self_attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.MultiheadAttention(d_model, nhead, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dropout, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`batch_first`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.LayerNorm(d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Feedforward`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.ff `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Sequential(`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, dim_feedforward),`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.GELU(),`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Dropout(dropout),`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(dim_feedforward, d_model),`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Dropout(dropout)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm2 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.LayerNorm(d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Dropout(dropout)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` _attention_block`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Self-attention sub-block.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn_out, _ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.self_attn(x, x, x)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout(attn_out)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` _ff_block`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Feedforward sub-block.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.ff(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.use_checkpoint `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`and`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.training:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Checkpoint attention`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn_out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` checkpoint(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`._attention_block, `})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x,`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` use_reentrant`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm1(x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn_out)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Checkpoint feedforward`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ff_out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` checkpoint(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`._ff_block,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x,`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` use_reentrant`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm2(x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ff_out)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Standard forward`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm1(x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`._attention_block(x))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.norm2(x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`._ff_block(x))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` CheckpointedTransformer`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Full transformer with gradient checkpointing.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` nhead`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` num_layers`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` vocab_size`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 10000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` use_checkpoint`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ):`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.embedding `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Embedding(vocab_size, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.layers `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.ModuleList([`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` CheckpointedTransformerBlock(`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model, nhead, `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` use_checkpoint`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`use_checkpoint`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` _ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(num_layers)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ])`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, vocab_size)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.embedding(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` layer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.layers:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` layer(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.output(x)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Compare memory with/without checkpointing`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` compare_checkpointing_memory`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Compare memory usage with and without checkpointing.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"CUDA not available\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch_size, seq_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 32`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model, num_layers `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`12`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Without checkpointing`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model_no_ckpt `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` CheckpointedTransformer(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` d_model`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`d_model, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`num_layers`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`num_layers,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` use_checkpoint`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ).to(device)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.reset_peak_memory_stats()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randint(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, (batch_size, seq_len)).to(device)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model_no_ckpt(x)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out.sum()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss.backward()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mem_no_ckpt `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.max_memory_allocated() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e9`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` del`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model_no_ckpt, out, loss`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.empty_cache()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # With checkpointing`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model_ckpt `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` CheckpointedTransformer(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` d_model`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`d_model, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`num_layers`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`num_layers,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` use_checkpoint`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ).to(device)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.reset_peak_memory_stats()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model_ckpt(x)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out.sum()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss.backward()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mem_ckpt `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.max_memory_allocated() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e9`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Without checkpointing: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mem_no_ckpt`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:.2f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` GB\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"With checkpointing: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mem_ckpt`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:.2f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` GB\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Memory saved: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mem_ckpt`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mem_no_ckpt)`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`100`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:.1f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`%\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`mixed-precision-training`,children:`Mixed Precision Training`}),`\n`,_jsx(_components.p,{children:`Mixed precision training is the single highest-impact memory optimization for most practitioners. The idea: use 16-bit floating point (FP16 or BF16) for the computationally intensive operations (matrix multiplies, convolutions) while keeping 32-bit precision where it matters (loss computation, gradient accumulation, optimizer states). Modern GPUs have dedicated hardware (Tensor Cores) that run FP16 matrix multiplies at 2-8x the speed of FP32, so you get both memory and speed benefits.`}),`\n`,_jsx(_components.p,{children:`The tricky part is avoiding numerical issues. FP16 has a very narrow dynamic range — gradients can underflow to zero (too small to represent) or overflow to infinity. The solution is gradient scaling: multiply the loss by a large number before backward, then divide the gradients by the same number before the optimizer step. This shifts the gradient values into FP16’s representable range. BFloat16 (BF16) avoids this entirely because it has the same exponent range as FP32, at the cost of less precision in the mantissa.`}),`\n`,_jsx(Warning,{children:_jsx(_components.p,{children:`Do not use FP16 mixed precision for fine-tuning large language models. BF16 is strongly preferred because FP16’s limited range causes gradient underflow on very small learning rates (1e-5 range) that are typical for LLM fine-tuning. If your GPU supports BF16 (Ampere or newer), always prefer it.`})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`124`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`124`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MixedPrecisionTraining`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Train with mixed FP16/FP32 precision.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Benefits:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - 2x memory reduction for activations and model weights`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - 2-8x faster matrix multiplications on Tensor Core GPUs`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Minimal accuracy loss with proper gradient scaling`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Key components:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Automatic casting (autocast): PyTorch decides which ops run in FP16`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Gradient scaling (GradScaler): prevents FP16 gradient underflow`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Typical setup: 3 lines of code change (autocast context, scale loss,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` scaler.step instead of optimizer.step).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`enabled`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.enabled `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` enabled`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.scaler `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.amp.GradScaler(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'cuda'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` enabled `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`else`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` training_step`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` batch`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`tuple`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` optimizer`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: optim.Optimizer,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` criterion`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ) -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Perform one training step with mixed precision.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` inputs, targets `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer.zero_grad()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.enabled:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Mixed precision forward`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` with`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.amp.autocast(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'cuda'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model(inputs)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` criterion(outputs, targets)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Scaled backward`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.scaler.scale(loss).backward()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Unscale gradients for clipping`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.scaler.unscale_(optimizer)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn.utils.clip_grad_norm_(model.parameters(), `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Optimizer step with scaling`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.scaler.step(optimizer)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.scaler.update()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Standard precision`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model(inputs)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` criterion(outputs, targets)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss.backward()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn.utils.clip_grad_norm_(model.parameters(), `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer.step()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss.item()`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# BFloat16 for better stability`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` BFloat16Training`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` BFloat16 training (available on Ampere+ GPUs).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Advantages over FP16:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Same exponent range as FP32`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - No gradient scaling needed`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Better numerical stability`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` is_available`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`() -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Check if BF16 is available.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.get_device_capability()[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`>=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 8`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` training_step`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` batch`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`tuple`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` optimizer`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: optim.Optimizer,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` criterion`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ) -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Training step with BF16.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` inputs, targets `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer.zero_grad()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` with`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.amp.autocast(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'cuda'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dtype`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`torch.bfloat16):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model(inputs)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` criterion(outputs, targets)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # No scaling needed for BF16!`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss.backward()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn.utils.clip_grad_norm_(model.parameters(), `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer.step()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss.item()`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Precision comparison`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` demonstrate_precision_formats`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Show memory usage for different precisions.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_params `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1_000_000`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # 1M parameters`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` formats `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'float32'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Standard precision\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'float16'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Half precision\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'bfloat16'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Brain float\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'int8'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Quantized\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Memory per million parameters:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"-\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 45`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` name, (bytes_per, desc) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` formats.items():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mem `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_params `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` bytes_per `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`name`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:10}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` (`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`desc`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:20}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`): `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mem`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:.1f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` MB\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`demonstrate_precision_formats()`})}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`cpu-offloading`,children:`CPU Offloading`}),`\n`,_jsx(_components.p,{children:`CPU offloading exploits the fact that your machine typically has 10-50x more CPU RAM than GPU memory. The idea is simple: keep only what the GPU needs right now on the GPU, and store everything else on the CPU (or even NVMe). This is the same principle behind ZeRO-Offload in DeepSpeed and FSDP’s CPU offloading in PyTorch. The trade-off is straightforward: you trade training speed (because of PCIe data transfers) for the ability to train models that otherwise would not fit at all.`}),`\n`,_jsx(Tip,{children:_jsx(_components.p,{children:`CPU offloading is most effective for optimizer states (which are only needed during the parameter update step, not during forward/backward). Offloading activations is less effective because they are accessed frequently during the backward pass, making the PCIe bandwidth a bottleneck. If you are using DeepSpeed, start with ZeRO Stage 2 + CPU offloading of optimizer states before trying full activation offloading.`})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`122`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`122`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` CPUOffloadOptimizer`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Offload optimizer states to CPU to save GPU memory.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Trade-off: Slower training (CPU\u003c->GPU transfers over PCIe) but can`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` train models 2-4x larger than what fits in GPU memory alone.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` When to use: When your model fits on the GPU for forward/backward`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` but OOMs during the optimizer step (because Adam doubles the memory`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` footprint with its momentum and variance states).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` lr`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e-4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` offload`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ):`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.offload `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` offload`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` offload:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Keep parameters on GPU but states on CPU`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.param_groups `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'params'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`list`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(model.parameters()),`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'lr'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: lr`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Initialize optimizer states on CPU`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.state `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {}`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.parameters():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.requires_grad:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Adam states`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.state[p] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'm'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.zeros_like(p.data, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`device`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'cpu'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'v'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.zeros_like(p.data, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`device`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'cpu'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 't'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.optimizer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optim.AdamW(model.parameters(), `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`lr`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`lr)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` zero_grad`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Zero gradients.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.offload:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.model.parameters():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.grad `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`is`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.grad.zero_()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.optimizer.zero_grad()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` step`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Perform optimizer step.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.offload:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.optimizer.step()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` lr `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.param_groups[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`][`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'lr'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` beta1, beta2 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.9`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.999`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` eps `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e-8`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.model.parameters():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.requires_grad `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`or`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.grad `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`is`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` continue`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` state `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.state[p]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` state[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'t'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` t `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` state[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'t'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Move gradient to CPU`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` grad_cpu `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.grad.data.cpu()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Update moments on CPU`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` state[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'m'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`].mul_(beta1).add_(grad_cpu, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`alpha`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` beta1)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` state[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'v'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`].mul_(beta2).addcmul_(grad_cpu, grad_cpu, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`value`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` beta2)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Bias correction`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` m_hat `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` state[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'m'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` beta1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` t)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` v_hat `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` state[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'v'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` beta2 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` t)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Compute update on CPU`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` update `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` m_hat `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (v_hat.sqrt() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` eps)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Apply update to GPU parameter`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p.data.add_(update.to(p.device), `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`alpha`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`lr)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` GradientOffloading`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Offload gradients to CPU during backward pass.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Useful for very large models where even gradients don't fit.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cpu_gradients `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {}`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`._register_hooks()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` _register_hooks`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Register hooks to offload gradients.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` name, param `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.model.named_parameters():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param.requires_grad:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param.register_post_accumulate_grad_hook(`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`._make_hook(name)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` _make_hook`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`name`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`str`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` hook`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`param`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param.grad `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`is`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Move gradient to CPU`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cpu_gradients[name] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param.grad.cpu()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Free GPU memory`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param.grad `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` hook`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` restore_gradients`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Move gradients back to GPU for optimizer.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` name, param `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.model.named_parameters():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` name `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cpu_gradients:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` param.grad `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cpu_gradients[name].to(param.device)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`activation-recomputation-strategies`,children:`Activation Recomputation Strategies`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`80`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`80`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` SelectiveCheckpointing`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Smart checkpointing strategies.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Not all layers benefit equally from checkpointing.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Focus on high-memory layers.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` should_checkpoint`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module) -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Decide if a module should be checkpointed.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Rules:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Checkpoint attention (quadratic memory)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Checkpoint large FFN layers`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Don't checkpoint small layers (overhead not worth it)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` isinstance`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(module, nn.MultiheadAttention):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Check for large linear layers`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` isinstance`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(module, nn.Linear):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` module.in_features `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` module.out_features `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`>`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` estimate_activation_memory`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` input_shape`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`tuple`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ) -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Estimate activation memory for a module.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Returns memory in MB.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # This is a simplified estimate`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch_size `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` input_shape[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` isinstance`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(module, nn.Linear):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Output activations`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch_size `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` module.out_features `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 4`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` /`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` isinstance`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(module, nn.MultiheadAttention):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Attention weights (quadratic in sequence length)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` seq_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` input_shape[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` module.num_heads`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch_size `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` seq_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` seq_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 4`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` /`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.0`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` LayerWiseCheckpointing`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Apply checkpointing at layer granularity.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` layers`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.ModuleList,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` checkpoint_ratio`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.5`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ):`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.layers `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` layers`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Checkpoint every other layer (or based on ratio)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.checkpoint_mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`%`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` /`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` checkpoint_ratio) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(layers))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i, layer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` enumerate`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.layers):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.checkpoint_mask[i] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`and`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.training:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` checkpoint(layer, x, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`use_reentrant`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` layer(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`memory-efficient-attention`,children:`Memory-Efficient Attention`}),`\n`,_jsxs(_components.p,{children:[`Attention is often the single largest memory consumer in transformer models. Standard self-attention materializes an `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`N`}),_jsx(_components.mo,{children:`×`}),_jsx(_components.mi,{children:`N`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`N \\\\times N`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7667em`,verticalAlign:`-0.0833em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.109em`},children:`N`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`×`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6833em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.109em`},children:`N`})]})]})]}),` attention matrix, where `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`N`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`N`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6833em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.109em`},children:`N`})]})})]}),` is the sequence length. For a 4096-token sequence with 32 attention heads in float16, that single matrix consumes `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mn,{children:`32`}),_jsx(_components.mo,{children:`×`}),_jsx(_components.mn,{children:`4096`}),_jsx(_components.mo,{children:`×`}),_jsx(_components.mn,{children:`4096`}),_jsx(_components.mo,{children:`×`}),_jsx(_components.mn,{children:`2`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`32 \\\\times 4096 \\\\times 4096 \\\\times 2`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7278em`,verticalAlign:`-0.0833em`}}),_jsx(_components.span,{className:`mord`,children:`32`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`×`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7278em`,verticalAlign:`-0.0833em`}}),_jsx(_components.span,{className:`mord`,children:`4096`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`×`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7278em`,verticalAlign:`-0.0833em`}}),_jsx(_components.span,{className:`mord`,children:`4096`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`×`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6444em`}}),_jsx(_components.span,{className:`mord`,children:`2`})]})]})]}),` bytes = 1 GB. Double the sequence length and the memory quadruples. The methods below tackle this bottleneck from different angles.`]}),`\n`,_jsx(Tip,{children:_jsxs(_components.p,{children:[`If you are using PyTorch 2.0+ or later, use `,_jsx(_components.code,{children:`torch.nn.functional.scaled_dot_product_attention`}),` with `,_jsx(_components.code,{children:`is_causal=True`}),` or `,_jsx(_components.code,{children:`attn_mask`}),`. PyTorch will automatically select the most efficient backend (FlashAttention, memory-efficient attention, or math fallback) based on your hardware, input shapes, and whether you need dropout. This is almost always the right choice — only implement custom attention when you need something the fused kernel does not support.`]})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`132`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`132`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MemoryEfficientAttention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Memory-efficient attention implementations.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Standard attention: O(n^2) memory for the full attention matrix.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Chunked attention: O(n * chunk_size) memory -- process queries in chunks,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` computing attention scores against all keys/values for each chunk.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Flash Attention: O(n) memory -- never materializes the full matrix at all,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` using a tiling algorithm that stays in GPU SRAM.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` num_heads`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` chunk_size`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1024`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ):`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.head_dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`//`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.chunk_size `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` chunk_size`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.qkv `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.out_proj `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` B, N, C `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x.shape`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Compute Q, K, V`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.qkv(x).reshape(B, N, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.head_dim)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv.permute(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (3, B, H, N, D)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` q, k, v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], qkv[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], qkv[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Chunked attention to reduce peak memory`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.chunked_attention(q, k, v)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output.transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).reshape(B, N, C)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.out_proj(output)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` chunked_attention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` q`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` k`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` v`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ) -> torch.Tensor:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Compute attention in chunks.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Instead of computing full NxN attention matrix,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` process in chunks to reduce memory.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` B, H, N, D `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` q.shape`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scale `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` D `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.5`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.zeros_like(v)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, N, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.chunk_size):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` end_i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` min`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.chunk_size, N)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` q_chunk `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` q[:, :, i:end_i]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Compute attention for this query chunk`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(q_chunk, k.transpose(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scale`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn.softmax(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output[:, :, i:end_i] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(attn, v)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` FlashAttention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Flash Attention wrapper (requires flash-attn package).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Key innovations:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Tiling to fit in SRAM`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - No materialization of attention matrix`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - IO-aware algorithm design`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Memory: O(n) instead of O(n²)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Speed: 2-4x faster than standard attention`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`num_heads`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.head_dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`//`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` dropout`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.qkv `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.out_proj `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`._check_flash_available()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` _check_flash_available`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Check if Flash Attention is available.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` try`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` from`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` flash_attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` flash_attn_func`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.flash_attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` flash_attn_func`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.use_flash `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` except`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` ImportError`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Flash Attention not available. Using standard attention.\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.use_flash `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` B, N, C `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x.shape`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.qkv(x).reshape(B, N, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.head_dim)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.use_flash:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Flash attention format: (B, N, H, D)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` q `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv[:, :, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`].transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` k `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv[:, :, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`].transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv[:, :, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`].transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.flash_attn(q, k, v, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dropout_p`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.training `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`else`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out.transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).reshape(B, N, C)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Standard attention`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv.permute(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` q, k, v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` qkv[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], qkv[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], qkv[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scale `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.head_dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`**`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` -`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.5`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (q `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`@`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` k.transpose(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scale`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn.softmax(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` out `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`@`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` v).transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).reshape(B, N, C)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.out_proj(out)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`efficient-batch-processing`,children:`Efficient Batch Processing`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`97`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`97`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` GradientAccumulation`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Accumulate gradients over multiple mini-batches.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Simulates larger batch sizes without memory increase.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` optimizer`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: optim.Optimizer,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` accumulation_steps`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 4`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ):`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.optimizer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` optimizer`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.accumulation_steps `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` accumulation_steps`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.current_step `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` backward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`loss`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Backward with gradient accumulation.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Scale loss by accumulation steps`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scaled_loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` loss `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.accumulation_steps`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scaled_loss.backward()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.current_step `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.current_step `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`>=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.accumulation_steps:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Perform optimizer step`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.optimizer.step()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.optimizer.zero_grad()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.current_step `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Step was performed`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Step not performed yet`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MicroBatching`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Split batches into micro-batches for forward pass.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Useful when single batch doesn't fit but you need`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` batch normalization statistics from full batch.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` micro_batch_forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: nn.Module,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` batch`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` micro_batch_size`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ) -> torch.Tensor:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Forward in micro-batches.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` []`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, batch.size(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`), micro_batch_size):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` micro_batch `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch[i:i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` micro_batch_size]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` with`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.no_grad() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model.training `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.enable_grad():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model(micro_batch)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` outputs.append(output)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cat(outputs, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` InPlaceOperations`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` In-place operations to reduce memory allocations.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Use with caution - can break gradient computation!`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` inplace_relu`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"In-place ReLU.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.relu_(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` inplace_add`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`y`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> torch.Tensor:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"In-place addition.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x.add_(y)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` check_safe_inplace`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`tensor`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: torch.Tensor) -> `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Check if in-place operation is safe.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Not safe if:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Tensor requires grad and is a leaf`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Tensor is used elsewhere in computation graph`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tensor.requires_grad `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`and`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tensor.is_leaf:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`memory-efficient-data-loading`,children:`Memory-Efficient Data Loading`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`57`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`57`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MemoryEfficientDataLoader`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Reduce data loading memory overhead.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` create_efficient_loader`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` dataset`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` batch_size`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` num_workers`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`int`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` pin_memory`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`bool`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` True`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ):`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Create memory-efficient data loader.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Tips:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - pin_memory for faster GPU transfer`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Appropriate num_workers (usually 4 per GPU)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - prefetch_factor controls memory usage`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.utils.data.DataLoader(`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` dataset,`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` batch_size`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`batch_size,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` shuffle`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` num_workers`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`num_workers,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` pin_memory`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`pin_memory,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Limit prefetching to reduce memory`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` prefetch_factor`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_workers `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`>`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Don't copy tensors`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` persistent_workers`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_workers `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`>`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` use_memory_mapping`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`file_path`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`str`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Use memory-mapped files for large datasets.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Data stays on disk, loaded on demand.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.memmap(file_path, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dtype`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'float32'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`mode`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'r'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` LowMemoryDataset`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`torch`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`utils`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`data`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Dataset`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Dataset that loads data on-demand.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`file_paths`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: List[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`str`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.file_paths `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` file_paths`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __len__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.file_paths)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __getitem__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`idx`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Load only when needed`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` data `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.load(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.file_paths[idx])`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` data`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`memory-debugging-tools`,children:`Memory Debugging Tools`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`76`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`76`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MemoryDebugger`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Tools for debugging memory issues.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` find_memory_leaks`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Find potential memory leaks.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` []`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` gc.collect()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.empty_cache()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` leaks `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` []`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` obj `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` gc.get_objects():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` isinstance`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(obj, torch.Tensor):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` obj.is_cuda:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` leaks.append({`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'shape'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: obj.shape,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'dtype'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: obj.dtype,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'size_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: obj.numel() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` obj.element_size() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1e6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'requires_grad'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: obj.requires_grad`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` })`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` sorted`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(leaks, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`key`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`lambda`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`x[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'size_mb'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`])`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` memory_summary`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Print detailed memory summary.\"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"CUDA not available\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(torch.cuda.memory_summary())`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` @`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`staticmethod`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` clear_memory`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Aggressively clear GPU memory.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` gc.collect()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.is_available():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.empty_cache()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cuda.synchronize()`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Usage demonstration`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` memory_optimization_demo`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Demonstrate memory optimization techniques.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` summary `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ╔════════════════════════════════════════════════════════════════╗`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ MEMORY OPTIMIZATION TECHNIQUES ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ╠════════════════════════════════════════════════════════════════╣`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ TECHNIQUE MEMORY SAVING COMPUTE COST ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ ───────────────────────────────────────────────────────────── ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ Mixed Precision ~50% ~0% (often faster!) ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ Gradient Checkpointing ~60-70% ~30-40% ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ Gradient Accumulation Linear in steps ~0% ║`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ Flash Attention ~50`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`% f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`or attn ~0% (often faster!) ║`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ CPU Offloading ~50-70% ~20-50% ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ╠════════════════════════════════════════════════════════════════╣`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ RECOMMENDATION ORDER: ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ 1. Enable mixed precision (always) ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ 2. Use Flash Attention (if available) ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ 3. Add gradient checkpointing ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ 4. Gradient accumulation for larger effective batch ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ║ 5. CPU offloading as last resort ║`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ╚════════════════════════════════════════════════════════════════╝`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(summary)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`memory_optimization_demo()`})}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`exercises`,children:`Exercises`}),`\n`,_jsxs(AccordionGroup,{children:[_jsx(Accordion,{title:`Exercise 1: Profile Your Model`,id:`exercise-1-profile-your-model`,children:_jsx(_components.p,{children:`Use the MemoryProfiler to analyze a model’s memory usage at each layer.\nIdentify the biggest memory consumers and optimize them.`})}),_jsx(Accordion,{title:`Exercise 2: Implement Selective Checkpointing`,id:`exercise-2-implement-selective-checkpointing`,children:_jsx(_components.p,{children:`Create a system that automatically decides which layers to checkpoint\nbased on their memory usage vs. compute cost.`})}),_jsx(Accordion,{title:`Exercise 3: Memory Budget Training`,id:`exercise-3-memory-budget-training`,children:_jsx(_components.p,{children:`Implement training that stays within a fixed memory budget by\ndynamically adjusting batch size and checkpointing.`})})]}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`what’s-next`,children:`What’s Next?`}),`\n`,_jsxs(CardGroup,{cols:2,children:[_jsx(Card,{title:`Quantization Deep Dive`,icon:`compress`,href:`/courses/deep-learning-mastery/45-quantization-deep-dive`,children:_jsx(_components.p,{children:`Post-training and quantization-aware training`})}),_jsx(Card,{title:`Knowledge Distillation`,icon:`school`,href:`/courses/deep-learning-mastery/46-knowledge-distillation`,children:_jsx(_components.p,{children:`Transfer knowledge to smaller models`})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Memory-Efficient Training","sidebarTitle":"Memory-Efficient Training","description":"Gradient checkpointing, mixed precision, and memory optimization","icon":"memory","href":"/courses/deep-learning-mastery/44-memory-efficient-training"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Memory-Efficient Training","slug":"memory-efficient-training","depth":1,"children":[]},{"title":"Understanding GPU Memory","slug":"understanding-gpu-memory","depth":2,"children":[]},{"title":"Memory Profiling","slug":"memory-profiling","depth":2,"children":[]},{"title":"Gradient Checkpointing","slug":"gradient-checkpointing","depth":2,"children":[]},{"title":"Mixed Precision Training","slug":"mixed-precision-training","depth":2,"children":[]},{"title":"CPU Offloading","slug":"cpu-offloading","depth":2,"children":[]},{"title":"Activation Recomputation Strategies","slug":"activation-recomputation-strategies","depth":2,"children":[]},{"title":"Memory-Efficient Attention","slug":"memory-efficient-attention","depth":2,"children":[]},{"title":"Efficient Batch Processing","slug":"efficient-batch-processing","depth":2,"children":[]},{"title":"Memory-Efficient Data Loading","slug":"memory-efficient-data-loading","depth":2,"children":[]},{"title":"Memory Debugging Tools","slug":"memory-debugging-tools","depth":2,"children":[]},{"title":"Exercises","slug":"exercises","depth":2,"children":[]},{"title":"What’s Next?","slug":"what’s-next","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"devweeekends","siteName":"Dev Weekends","title":"Memory-Efficient Training - Dev Weekends","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-05-16T10:35:00.457Z","originalFileLocation":"courses/deep-learning-mastery/44-memory-efficient-training.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Gradient checkpointing, mixed precision, and memory optimization\">\u003cmeta data-mint-head name=\"application-name\" content=\"Dev Weekends\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Memory-Efficient Training - Dev Weekends\">\u003cmeta data-mint-head property=\"og:description\" content=\"Gradient checkpointing, mixed precision, and memory optimization\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Dev Weekends\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://devweeekends.mintlify.site/_mintlify/api/og/v2?division=Deep+Learning+Mastery\u0026amp;title=Memory-Efficient+Training\u0026amp;description=Gradient+checkpointing%2C+mixed+precision%2C+and+memory+optimization\u0026amp;theme=97ffacf68ee2a7d59a7fcf65\u0026amp;signature=9b4c08849aeeb4a017bc84fb7d2a87be9693e1118a7931304d99271829132a9c\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Memory-Efficient Training - Dev Weekends\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Gradient checkpointing, mixed precision, and memory optimization\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://devweeekends.mintlify.site/_mintlify/api/og/v2?division=Deep+Learning+Mastery\u0026amp;title=Memory-Efficient+Training\u0026amp;description=Gradient+checkpointing%2C+mixed+precision%2C+and+memory+optimization\u0026amp;theme=97ffacf68ee2a7d59a7fcf65\u0026amp;signature=9b4c08849aeeb4a017bc84fb7d2a87be9693e1118a7931304d99271829132a9c\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Dev Weekends\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#000000\">\u003clink data-mint-head rel=\"canonical\" href=\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/courses/deep-learning-mastery/44-memory-efficient-training.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://resources.devweekends.com/#organization\",\"name\":\"Dev Weekends\",\"url\":\"https://resources.devweekends.com\"},{\"@type\":\"WebSite\",\"@id\":\"https://resources.devweekends.com#website\",\"name\":\"Dev Weekends\",\"url\":\"https://resources.devweekends.com\",\"publisher\":{\"@id\":\"https://resources.devweekends.com/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training#webpage\",\"url\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training\",\"name\":\"Memory-Efficient Training\",\"description\":\"Gradient checkpointing, mixed precision, and memory optimization\",\"dateModified\":\"2026-05-16T10:35:00.457Z\",\"isPartOf\":{\"@id\":\"https://resources.devweekends.com#website\"},\"breadcrumb\":{\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Deep Learning Mastery\",\"item\":\"https://resources.devweekends.com/courses/deep-learning-mastery/00-overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Memory-Efficient Training\",\"item\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training#article\",\"headline\":\"Memory-Efficient Training\",\"name\":\"Memory-Efficient Training\",\"description\":\"Gradient checkpointing, mixed precision, and memory optimization\",\"url\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training\",\"mainEntityOfPage\":{\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/44-memory-efficient-training#webpage\"},\"dateModified\":\"2026-05-16T10:35:00.457Z\",\"publisher\":{\"@id\":\"https://resources.devweekends.com/#organization\"},\"isPartOf\":{\"@id\":\"https://resources.devweekends.com#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Dev Weekends\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Dev Weekends\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/courses/deep-learning-mastery/44-memory-efficient-training.md","siteName":"Dev Weekends","nav":{"global":{"anchors":[{"anchor":"DevWeekends","icon":"globe","href":"https://devweekends.com"}]},"anchors":[{"anchor":"Resources","icon":"book-open","pages":[{"title":"Resources","href":""}]},{"anchor":"Interview Prep","icon":"clipboard-check","pages":[{"title":"Interview Prep","href":"/resources/interview-experiences"}]},{"anchor":"DSA Patterns","icon":"code","pages":[{"title":"DSA Patterns","href":"/dsa-patterns/overview"}]},{"anchor":"Competitive Programming","icon":"trophy","pages":[{"title":"Competitive Programming","href":"/courses/competitive-programming/00-overview"}]},{"anchor":"Courses","icon":"graduation-cap","groups":[{"group":"Overview","pages":[{"title":"All Courses","icon":"graduation-cap","href":"/courses/overview"}]},{"group":"Testing \u0026 Code Quality","icon":"vial","pages":[{"title":"Sonarqube Mastery","sidebarTitle":"Sonarqube","icon":"magnifying-glass-chart","href":"/courses/testing-tools/sonarqube"},{"title":"Playwright Mastery","sidebarTitle":"Playwright","icon":"masks-theater","href":"/courses/testing-tools/playwright"},{"title":"Mocha Mastery","sidebarTitle":"Mocha","icon":"coffee-bean","href":"/courses/testing-tools/mocha"},{"title":"Jest Mastery","sidebarTitle":"Jest","icon":"jester-hat","href":"/courses/testing-tools/jest"}]},{"group":"Crash Courses","icon":"bolt","pages":[{"group":"C++","icon":"brackets-curly","pages":[{"title":"C++ Crash Course","sidebarTitle":"Overview","icon":"code","href":"/courses/cpp-crash-course/overview"},{"title":"C++ Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/cpp-crash-course/fundamentals"},{"title":"Pointers \u0026 Memory","sidebarTitle":"2. Memory Mastery","icon":"memory","href":"/courses/cpp-crash-course/pointers-memory"},{"title":"Object-Oriented Programming","sidebarTitle":"3. OOP","icon":"object-group","href":"/courses/cpp-crash-course/oop"},{"title":"Standard Template Library","sidebarTitle":"4. The STL","icon":"layer-group","href":"/courses/cpp-crash-course/stl"},{"title":"Modern C++ Features","sidebarTitle":"5. Modern C++","icon":"wand-magic-sparkles","href":"/courses/cpp-crash-course/modern-cpp"}]},{"group":"Java","icon":"mug-hot","pages":[{"title":"Java Crash Course","sidebarTitle":"Overview","icon":"mug-hot","href":"/courses/java-crash-course/overview"},{"title":"Java Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/java-crash-course/fundamentals"},{"title":"Object-Oriented Programming","sidebarTitle":"2. OOP","icon":"object-group","href":"/courses/java-crash-course/oop"},{"title":"Collections Framework","sidebarTitle":"3. Collections","icon":"layer-group","href":"/courses/java-crash-course/collections"},{"title":"Concurrency","sidebarTitle":"4. Concurrency","icon":"bolt","href":"/courses/java-crash-course/concurrency"},{"title":"Modern Java Features","sidebarTitle":"5. Modern Java","icon":"wand-magic-sparkles","href":"/courses/java-crash-course/modern-java"}]},{"group":"Spring Boot \u0026 Microservices","icon":"leaf","pages":[{"title":"Spring Boot \u0026 Microservices","sidebarTitle":"Overview","icon":"leaf","href":"/courses/spring-boot-microservices/00-overview"},{"title":"Spring Boot Quickstart","icon":"bolt","href":"/courses/spring-boot-microservices/01-spring-boot-quickstart"},{"title":"Spring MVC \u0026 REST","icon":"server","href":"/courses/spring-boot-microservices/02-spring-mvc-rest"},{"title":"Data Persistence","icon":"database","href":"/courses/spring-boot-microservices/03-data-persistence"},{"title":"Deep Dive: Spring AOP","icon":"microscope","href":"/courses/spring-boot-microservices/04-spring-aop"},{"title":"Testing Mastery","icon":"vial","href":"/courses/spring-boot-microservices/05-spring-testing"},{"title":"Foundations","icon":"shapes","href":"/courses/spring-boot-microservices/06-microservices-foundations"},{"title":"Service Discovery","icon":"compass","href":"/courses/spring-boot-microservices/07-service-discovery"},{"title":"API Gateway","icon":"door-open","href":"/courses/spring-boot-microservices/08-api-gateway"},{"title":"Config Management","icon":"gears","href":"/courses/spring-boot-microservices/09-config-management"},{"title":"Resilience \u0026 Fault Tolerance","icon":"shield-halved","href":"/courses/spring-boot-microservices/10-resilience"},{"title":"Observability","icon":"monitor-waveform","href":"/courses/spring-boot-microservices/11-observability"},{"title":"Event Driven Architecture","icon":"satellite-dish","href":"/courses/spring-boot-microservices/12-event-driven"},{"title":"Containerization","icon":"docker","href":"/courses/spring-boot-microservices/13-containerization"},{"title":"Capstone Project","icon":"flag-checkered","href":"/courses/spring-boot-microservices/14-capstone"}]},{"group":"JavaScript","icon":"js","pages":[{"title":"JavaScript Crash Course","sidebarTitle":"Overview","icon":"js","href":"/courses/javascript-crash-course/overview"},{"title":"JavaScript Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/javascript-crash-course/fundamentals"},{"title":"Functions \u0026 Scope","sidebarTitle":"2. Functions \u0026 Scope","icon":"function","href":"/courses/javascript-crash-course/functions-scope"},{"title":"Objects \u0026 Prototypes","sidebarTitle":"3. Objects \u0026 Prototypes","icon":"object-group","href":"/courses/javascript-crash-course/objects-prototypes"},{"title":"Async JavaScript","sidebarTitle":"4. Async JavaScript","icon":"clock","href":"/courses/javascript-crash-course/async"},{"title":"Modern JavaScript (ES6+)","sidebarTitle":"5. Modern JavaScript","icon":"wand-magic-sparkles","href":"/courses/javascript-crash-course/modern-js"},{"title":"DOM \u0026 Browser APIs","sidebarTitle":"6. DOM \u0026 Browser","icon":"browser","href":"/courses/javascript-crash-course/dom-browser"}]},{"group":"TypeScript","icon":"file-code","pages":[{"title":"TypeScript Crash Course","sidebarTitle":"Overview","icon":"code","href":"/courses/typescript-crash-course/overview"},{"title":"TypeScript Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/typescript-crash-course/fundamentals"},{"title":"Functions \u0026 Types","sidebarTitle":"2. Functions \u0026 Types","icon":"function","href":"/courses/typescript-crash-course/functions-types"},{"title":"Objects \u0026 Interfaces","sidebarTitle":"3. Objects \u0026 Interfaces","icon":"object-group","href":"/courses/typescript-crash-course/objects-interfaces"},{"title":"Classes \u0026 OOP","sidebarTitle":"4. Classes \u0026 OOP","icon":"sitemap","href":"/courses/typescript-crash-course/classes-oop"},{"title":"Advanced Types","sidebarTitle":"5. Advanced Types","icon":"layer-group","href":"/courses/typescript-crash-course/advanced-types"},{"title":"Generics Deep Dive","sidebarTitle":"6. Generics","icon":"puzzle-piece","href":"/courses/typescript-crash-course/generics"},{"title":"Modules \u0026 Configuration","sidebarTitle":"7. Modules \u0026 Config","icon":"gear","href":"/courses/typescript-crash-course/modules-config"}]},{"group":"Python","icon":"python","pages":[{"title":"Python Crash Course","sidebarTitle":"Overview","icon":"python","href":"/courses/python-crash-course/overview"},{"title":"Python Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/python-crash-course/fundamentals"},{"title":"Data Structures","sidebarTitle":"2. Data Structures","icon":"layer-group","href":"/courses/python-crash-course/data-structures"},{"title":"Object-Oriented Programming","sidebarTitle":"3. OOP","icon":"object-group","href":"/courses/python-crash-course/oop"},{"title":"Modules \u0026 Packages","sidebarTitle":"4. Modules","icon":"box","href":"/courses/python-crash-course/modules-packages"},{"title":"Advanced Python","sidebarTitle":"5. Advanced","icon":"wand-magic-sparkles","href":"/courses/python-crash-course/advanced"}]},{"group":"Node.js","icon":"node","pages":[{"title":"01. Introduction \u0026 Setup","href":"/courses/node-crash-course/01-intro-setup"},{"title":"02. Modules System","href":"/courses/node-crash-course/02-modules"},{"title":"03. File System","href":"/courses/node-crash-course/03-file-system"},{"title":"04. Events \u0026 EventEmitter","href":"/courses/node-crash-course/04-events"},{"title":"05. HTTP Module \u0026 Web Server","href":"/courses/node-crash-course/05-http-server"},{"title":"06. Streams \u0026 Buffers","href":"/courses/node-crash-course/06-streams-buffers"},{"title":"07. NPM (Node Package Manager)","href":"/courses/node-crash-course/07-npm"},{"title":"07b. Building and Publishing an NPM Package","href":"/courses/node-crash-course/07b-building-npm-package"},{"title":"08. Express.js Basics","href":"/courses/node-crash-course/08-express-basics"},{"title":"09. Building a REST API","href":"/courses/node-crash-course/09-rest-api"},{"title":"10. Deployment \u0026 Best Practices","href":"/courses/node-crash-course/10-deployment"},{"title":"11. Authentication with JWT","href":"/courses/node-crash-course/11-auth-jwt"},{"title":"12. MongoDB \u0026 Mongoose ORM","href":"/courses/node-crash-course/12-mongo-mongoose"},{"title":"13. PostgreSQL \u0026 Prisma ORM","href":"/courses/node-crash-course/13-postgres-prisma"},{"title":"14. Error Handling \u0026 Debugging","href":"/courses/node-crash-course/14-error-handling"},{"title":"15. Testing with Jest","href":"/courses/node-crash-course/15-testing"},{"title":"16. Real-Time with WebSockets","href":"/courses/node-crash-course/16-websockets"},{"title":"17. Security Best Practices","href":"/courses/node-crash-course/17-security"},{"title":"18. Performance \u0026 Caching","href":"/courses/node-crash-course/18-performance"},{"title":"19. Microservices Architecture","href":"/courses/node-crash-course/19-microservices"},{"title":"20. TypeScript with Node.js","href":"/courses/node-crash-course/20-typescript"},{"title":"21. Worker Threads \u0026 Child Processes","href":"/courses/node-crash-course/21-worker-threads"},{"title":"22. Capstone Project: Full-Stack API","href":"/courses/node-crash-course/22-capstone"}]},{"group":"React","icon":"react","pages":[{"title":"01. Introduction to React \u0026 JSX","href":"/courses/react-crash-course/01-intro-jsx"},{"title":"02. Components \u0026 Props","href":"/courses/react-crash-course/02-components-props"},{"title":"03. State \u0026 useState Hook","href":"/courses/react-crash-course/03-state-hook"},{"title":"04. Handling Events","href":"/courses/react-crash-course/04-events"},{"title":"05. Lists \u0026 Keys","href":"/courses/react-crash-course/05-lists-keys"},{"title":"06. Forms \u0026 Controlled Components","href":"/courses/react-crash-course/06-forms"},{"title":"07. useEffect \u0026 Side Effects","href":"/courses/react-crash-course/07-useeffect"},{"title":"08. Context API","href":"/courses/react-crash-course/08-context-api"},{"title":"09. React Router","href":"/courses/react-crash-course/09-react-router"},{"title":"10. Performance Optimization","href":"/courses/react-crash-course/10-optimization"},{"title":"11. Authentication \u0026 Protected Routes","href":"/courses/react-crash-course/11-authentication"},{"title":"12. Redux \u0026 State Management","href":"/courses/react-crash-course/12-redux-state"}]},{"group":"React Native","icon":"mobile","pages":[{"title":"React Native Enterprise Mastery","icon":"mobile","href":"/courses/react-native-crash-course/overview"},{"title":"01. Introduction \u0026 Architecture","href":"/courses/react-native-crash-course/01-intro-architecture"},{"title":"02. Environment Setup","href":"/courses/react-native-crash-course/02-environment-setup"},{"title":"03. Project Structure \u0026 Best Practices","href":"/courses/react-native-crash-course/03-project-structure"},{"title":"04. TypeScript in React Native","href":"/courses/react-native-crash-course/04-typescript"},{"title":"05. Core Components Deep Dive","href":"/courses/react-native-crash-course/05-core-components"},{"title":"6. Styling \u0026 Theming","href":"/courses/react-native-crash-course/06-styling-theming"},{"title":"7. Flexbox Mastery","href":"/courses/react-native-crash-course/07-flexbox-mastery"},{"title":"8. Navigation Fundamentals","href":"/courses/react-native-crash-course/08-navigation-fundamentals"},{"title":"9. Advanced Navigation","href":"/courses/react-native-crash-course/09-navigation-advanced"},{"title":"10. State Management","href":"/courses/react-native-crash-course/10-state-management"},{"title":"11. Redux \u0026 Zustand","href":"/courses/react-native-crash-course/11-redux-zustand"},{"title":"15. React Query \u0026 Data Fetching","href":"/courses/react-native-crash-course/15-react-query"},{"title":"27. Unit Testing","href":"/courses/react-native-crash-course/27-unit-testing"},{"title":"31. Security Best Practices","href":"/courses/react-native-crash-course/31-security"},{"title":"35. Profiling \u0026 Monitoring","href":"/courses/react-native-crash-course/35-profiling"},{"title":"40. CI/CD Pipelines","href":"/courses/react-native-crash-course/40-ci-cd"},{"title":"51. Capstone Project","href":"/courses/react-native-crash-course/51-capstone"}]},{"group":"MongoDB","icon":"database","pages":[{"title":"01. Introduction to NoSQL \u0026 MongoDB","href":"/courses/mongodb-crash-course/01-intro-nosql"},{"title":"02. Installation \u0026 MongoDB Compass","href":"/courses/mongodb-crash-course/02-setup-compass"},{"title":"03. Databases \u0026 Collections","href":"/courses/mongodb-crash-course/03-databases-collections"},{"title":"04. CRUD: Create","href":"/courses/mongodb-crash-course/04-crud-create"},{"title":"05. CRUD: Read (Query Operators)","href":"/courses/mongodb-crash-course/05-crud-read"},{"title":"06. CRUD: Update","href":"/courses/mongodb-crash-course/06-crud-update"},{"title":"07. CRUD: Delete","href":"/courses/mongodb-crash-course/07-crud-delete"},{"title":"08. Aggregation Framework","href":"/courses/mongodb-crash-course/08-aggregation"},{"title":"09. Indexing \u0026 Performance","href":"/courses/mongodb-crash-course/09-indexing"},{"title":"10. Mongoose (ODM) Basics","href":"/courses/mongodb-crash-course/10-mongoose"}]},{"group":"Angular Master Course","icon":"angular","pages":[{"title":"Angular Master Course","icon":"angular","href":"/courses/angular-crash-course/00-overview"},{"title":"01. Introduction \u0026 Setup","icon":"rocket","href":"/courses/angular-crash-course/01-introduction"},{"title":"02. Components \u0026 Templates","icon":"puzzle-piece","href":"/courses/angular-crash-course/02-components"},{"title":"03. Directives \u0026 Pipes","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/03-directives-pipes"},{"title":"04. Services \u0026 Dependency Injection","icon":"syringe","href":"/courses/angular-crash-course/04-services-di"},{"title":"05. Angular Signals","icon":"signal","href":"/courses/angular-crash-course/05-signals"},{"title":"06. Routing \u0026 Navigation","icon":"route","href":"/courses/angular-crash-course/06-routing"},{"title":"07. Angular Forms","icon":"input-text","href":"/courses/angular-crash-course/07-forms"},{"title":"08. HTTP Client \u0026 RxJS","icon":"server","href":"/courses/angular-crash-course/08-http-rxjs"},{"title":"09. RxJS Deep Dive","icon":"arrows-rotate","href":"/courses/angular-crash-course/09-rxjs"},{"title":"10. Change Detection \u0026 Performance","icon":"gauge-high","href":"/courses/angular-crash-course/10-change-detection"},{"title":"11. Testing in Angular","icon":"vial","href":"/courses/angular-crash-course/11-testing"},{"title":"12. Advanced Patterns","icon":"diagram-project","href":"/courses/angular-crash-course/12-advanced-patterns"},{"title":"13. Server-Side Rendering","icon":"server","href":"/courses/angular-crash-course/13-ssr"},{"title":"14. Security Best Practices","icon":"shield-halved","href":"/courses/angular-crash-course/14-security"},{"title":"15. Capstone Project","icon":"rocket","href":"/courses/angular-crash-course/15-capstone"},{"title":"16. Standalone Components Deep Dive","icon":"cube","href":"/courses/angular-crash-course/16-standalone"},{"title":"17. NgRx State Management","icon":"database","href":"/courses/angular-crash-course/17-ngrx"},{"title":"18. Angular Animations","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/18-animations"},{"title":"19. Internationalization (i18n)","icon":"globe","href":"/courses/angular-crash-course/19-i18n"},{"title":"20. Accessibility (a11y)","icon":"universal-access","href":"/courses/angular-crash-course/20-accessibility"},{"title":"21. Angular Material \u0026 CDK","icon":"palette","href":"/courses/angular-crash-course/21-material"},{"title":"22. PWA \u0026 Service Workers","icon":"mobile-screen","href":"/courses/angular-crash-course/22-pwa"},{"title":"23. Deployment \u0026 CI/CD","icon":"rocket","href":"/courses/angular-crash-course/23-deployment"},{"title":"24. Micro-frontends with Module Federation","icon":"cubes","href":"/courses/angular-crash-course/24-micro-frontends"},{"title":"25. Nx Monorepo","icon":"boxes-stacked","href":"/courses/angular-crash-course/25-nx-monorepo"},{"title":"26. Performance Optimization","icon":"gauge-high","href":"/courses/angular-crash-course/26-performance"},{"title":"27. Content Projection","icon":"layer-group","href":"/courses/angular-crash-course/27-content-projection"},{"title":"28. Dynamic Components","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/28-dynamic-components"},{"title":"29. Custom Schematics","icon":"toolbox","href":"/courses/angular-crash-course/29-custom-schematics"},{"title":"30. Error Handling","icon":"triangle-exclamation","href":"/courses/angular-crash-course/30-error-handling"},{"title":"31. Interview Questions","icon":"comments-question","href":"/courses/angular-crash-course/31-interview-questions"},{"title":"32. Best Practices","icon":"award","href":"/courses/angular-crash-course/32-best-practices"},{"title":"33. Enterprise Capstone Project","icon":"building","href":"/courses/angular-crash-course/33-enterprise-capstone"}]}]},{"group":"AI Engineering","icon":"robot","pages":[{"title":"AI Engineering","icon":"robot","href":"/ai-engineering/overview"},{"title":"Python Crash Course","icon":"python","href":"/ai-engineering/python-crash-course"},{"title":"FastAPI Crash Course","icon":"bolt","href":"/ai-engineering/fastapi-crash-course"},{"title":"Databases \u0026 ORM Crash Course","icon":"database","href":"/ai-engineering/databases-crash-course"},{"title":"LLM Fundamentals","icon":"brain","href":"/ai-engineering/llm-fundamentals"},{"title":"Prompt Engineering","icon":"wand-magic-sparkles","href":"/ai-engineering/prompt-engineering"},{"title":"OpenAI API","icon":"bolt","href":"/ai-engineering/openai-api"},{"title":"Structured Outputs","icon":"brackets-curly","href":"/ai-engineering/structured-outputs"},{"title":"Embeddings Deep Dive","icon":"vector-square","href":"/ai-engineering/embeddings"},{"title":"AI Streaming","icon":"water","href":"/ai-engineering/streaming"},{"title":"Vector Databases","icon":"database","href":"/ai-engineering/vector-databases"},{"title":"RAG Systems","icon":"magnifying-glass","href":"/ai-engineering/rag"},{"title":"Types of RAG","icon":"layer-group","href":"/ai-engineering/rag-types"},{"title":"Fine-Tuning LLMs","icon":"sliders","href":"/ai-engineering/fine-tuning"},{"title":"AI Agents","icon":"robot","href":"/ai-engineering/agents"},{"title":"Tool Calling \u0026 Function Calling","icon":"wrench","href":"/ai-engineering/tool-calling"},{"title":"LLM Memory Systems","icon":"brain","href":"/ai-engineering/llm-memory-systems"},{"title":"Multi-Agent Design Patterns","icon":"users-gear","href":"/ai-engineering/multi-agent-patterns"},{"title":"LangChain","icon":"link","href":"/ai-engineering/langchain"},{"title":"LangGraph","icon":"diagram-project","href":"/ai-engineering/langgraph"},{"title":"Model Context Protocol (MCP)","icon":"plug","href":"/ai-engineering/mcp"},{"title":"Agentic Architecture","icon":"sitemap","href":"/ai-engineering/agentic-architecture"},{"title":"Open Source \u0026 Local LLMs","icon":"server","href":"/ai-engineering/open-source-llms"},{"title":"Multimodal AI","icon":"photo-film","href":"/ai-engineering/multimodal-ai"},{"title":"DSPy Framework","icon":"code","href":"/ai-engineering/dspy"},{"title":"Prompt Management \u0026 Versioning","icon":"code-branch","href":"/ai-engineering/prompt-management"},{"title":"LLM Caching","icon":"database","href":"/ai-engineering/llm-caching"},{"title":"Error Handling \u0026 Resilience","icon":"shield-halved","href":"/ai-engineering/error-handling"},{"title":"Batch Processing \u0026 Rate Limiting","icon":"layer-group","href":"/ai-engineering/batch-processing"},{"title":"LLM Fallbacks \u0026 Model Routing","icon":"arrows-split-up-and-left","href":"/ai-engineering/llm-fallbacks"},{"title":"AI Observability \u0026 Monitoring","icon":"chart-line","href":"/ai-engineering/observability-monitoring"},{"title":"AI Security \u0026 Guardrails","icon":"shield-halved","href":"/ai-engineering/ai-security-guardrails"},{"title":"Cost Optimization \u0026 Token Management","icon":"coins","href":"/ai-engineering/cost-optimization"},{"title":"Function Calling Deep Dive","icon":"function","href":"/ai-engineering/function-calling"},{"title":"LLM Orchestration","icon":"sitemap","href":"/ai-engineering/llm-orchestration"},{"title":"Semantic Search \u0026 Retrieval","icon":"magnifying-glass","href":"/ai-engineering/semantic-search"},{"title":"Context Window Management","icon":"window-maximize","href":"/ai-engineering/context-management"},{"title":"Testing LLM Applications","icon":"flask-vial","href":"/ai-engineering/llm-testing"},{"title":"Production Logging \u0026 Debugging","icon":"file-lines","href":"/ai-engineering/production-logging"},{"title":"API Design for AI Applications","icon":"code","href":"/ai-engineering/api-design"},{"title":"Document Processing","icon":"file-lines","href":"/ai-engineering/document-processing"},{"title":"Model Comparison","icon":"scale-balanced","href":"/ai-engineering/model-comparison"},{"title":"Anthropic Claude API","icon":"message-bot","href":"/ai-engineering/anthropic-api"},{"title":"Async Patterns for LLMs","icon":"arrows-spin","href":"/ai-engineering/async-patterns"},{"title":"Advanced Prompting Techniques","icon":"brain","href":"/ai-engineering/advanced-prompting"},{"title":"Output Validation","icon":"shield-check","href":"/ai-engineering/output-validation"},{"title":"Semantic Routing","icon":"route","href":"/ai-engineering/semantic-routing"},{"title":"Advanced RAG Techniques","icon":"layer-group","href":"/ai-engineering/advanced-rag"},{"title":"Voice and Audio AI","icon":"microphone","href":"/ai-engineering/voice-audio"},{"title":"Image Generation","icon":"image","href":"/ai-engineering/image-generation"},{"title":"Summarization Systems","icon":"compress","href":"/ai-engineering/summarization"},{"title":"Chatbot Design Patterns","icon":"comments","href":"/ai-engineering/chatbot-patterns"},{"title":"Knowledge Graphs with LLMs","icon":"diagram-project","href":"/ai-engineering/knowledge-graphs"},{"title":"Synthetic Data Generation","icon":"database","href":"/ai-engineering/synthetic-data"},{"title":"Text Classification","icon":"tags","href":"/ai-engineering/classification"},{"title":"Model Optimization","icon":"gauge-high","href":"/ai-engineering/model-optimization"},{"title":"Realtime AI Applications","icon":"bolt","href":"/ai-engineering/realtime-ai"},{"title":"Evaluation \u0026 Testing","icon":"vial","href":"/ai-engineering/evaluation-testing"},{"title":"Deployment \u0026 Scaling","icon":"rocket","href":"/ai-engineering/deployment-scaling"},{"title":"Capstone Project","icon":"trophy","href":"/ai-engineering/capstone-project"}]},{"group":"Math for ML - Understanding Linear Algebra","icon":"square-root-variable","pages":[{"title":"Introduction to Linear Algebra for ML","sidebarTitle":"Introduction","icon":"rocket","href":"/courses/math-for-ml-linear-algebra/01-introduction"},{"title":"Vectors: The Language of Similarity","sidebarTitle":"Vectors","icon":"arrow-right","href":"/courses/math-for-ml-linear-algebra/02-vectors"},{"title":"Matrices \u0026 Linear Transformations","sidebarTitle":"Matrices","icon":"grid","href":"/courses/math-for-ml-linear-algebra/03-matrices"},{"title":"Project: Image Similarity Search Engine","sidebarTitle":"Project 1","icon":"code","href":"/courses/math-for-ml-linear-algebra/04-project-image-search"},{"title":"Eigenvalues \u0026 Eigenvectors","sidebarTitle":"Eigenvalues","icon":"compass","href":"/courses/math-for-ml-linear-algebra/05-eigenvalues"},{"title":"Principal Component Analysis (PCA)","sidebarTitle":"PCA","icon":"compress","href":"/courses/math-for-ml-linear-algebra/06-pca"},{"title":"Singular Value Decomposition (SVD)","sidebarTitle":"SVD","icon":"layer-group","href":"/courses/math-for-ml-linear-algebra/07-svd"},{"title":"Linear Systems \u0026 Applications","sidebarTitle":"Linear Systems","icon":"diagram-project","href":"/courses/math-for-ml-linear-algebra/08-linear-systems"},{"title":"Project: Music Recommendation Engine","sidebarTitle":"Project 2","icon":"music","href":"/courses/math-for-ml-linear-algebra/09-project-recommender"},{"title":"Orthogonality \u0026 Projections","sidebarTitle":"Orthogonality","icon":"right-left","href":"/courses/math-for-ml-linear-algebra/10-orthogonality"}]},{"group":"Probability \u0026 Statistics for ML","icon":"chart-bar","pages":[{"title":"Probability and Statistics for Machine Learning","sidebarTitle":"Introduction","icon":"chart-line","href":"/courses/statistics-for-ml/01-introduction"},{"title":"Describing Data: What's Normal?","sidebarTitle":"Describing Data","icon":"chart-bar","href":"/courses/statistics-for-ml/02-describing-data"},{"title":"Probability Foundations: How Likely Is This?","sidebarTitle":"Probability","icon":"dice","href":"/courses/statistics-for-ml/03-probability"},{"title":"Probability Distributions: Patterns in Randomness","sidebarTitle":"Distributions","icon":"chart-bell","href":"/courses/statistics-for-ml/04-distributions"},{"title":"Statistical Inference: Conclusions from Samples","sidebarTitle":"Inference","icon":"magnifying-glass-chart","href":"/courses/statistics-for-ml/05-inference"},{"title":"Hypothesis Testing: Real Effect or Random Noise?","sidebarTitle":"Hypothesis Testing","icon":"scale-balanced","href":"/courses/statistics-for-ml/06-hypothesis-testing"},{"title":"Correlation and Regression: Relationships and Predictions","sidebarTitle":"Regression","icon":"chart-line-up","href":"/courses/statistics-for-ml/07-regression"},{"title":"From Statistics to Machine Learning","sidebarTitle":"Stats to ML","icon":"brain","href":"/courses/statistics-for-ml/08-statistics-to-ml"},{"title":"Bayesian Statistics","sidebarTitle":"Bayesian Statistics","icon":"brain-circuit","href":"/courses/statistics-for-ml/09-bayesian-statistics"},{"title":"Time Series Fundamentals","sidebarTitle":"Time Series","icon":"chart-line","href":"/courses/statistics-for-ml/10-time-series-basics"},{"title":"Capstone Project: Complete Statistical Analysis","sidebarTitle":"Capstone Project","icon":"graduation-cap","href":"/courses/statistics-for-ml/11-capstone-project"}]},{"group":"Math for ML - Understanding Calculus","icon":"function","pages":[{"title":"Calculus for Machine Learning","sidebarTitle":"Introduction","icon":"function","href":"/courses/math-for-ml-calculus/00-introduction"},{"title":"Derivatives \u0026 Rates of Change","sidebarTitle":"Derivatives","icon":"chart-line","href":"/courses/math-for-ml-calculus/01-derivatives"},{"title":"Gradients \u0026 Multivariable Calculus","sidebarTitle":"Gradients","icon":"mountain","href":"/courses/math-for-ml-calculus/02-gradients"},{"title":"Chain Rule \u0026 Backpropagation","sidebarTitle":"Chain Rule","icon":"link","href":"/courses/math-for-ml-calculus/03-chain-rule"},{"title":"Gradient Descent","sidebarTitle":"Gradient Descent","icon":"arrow-trend-down","href":"/courses/math-for-ml-calculus/04-gradient-descent"},{"title":"Optimization Techniques","sidebarTitle":"Optimization","icon":"rocket","href":"/courses/math-for-ml-calculus/05-optimization"},{"title":"Final Project: Build a Neural Network","sidebarTitle":"Final Project","icon":"flag-checkered","href":"/courses/math-for-ml-calculus/06-final-project"},{"title":"Automatic Differentiation","sidebarTitle":"Autodiff","icon":"wand-magic-sparkles","href":"/courses/math-for-ml-calculus/07-automatic-differentiation"},{"title":"Convex Optimization","sidebarTitle":"Convexity","icon":"bowl-rice","href":"/courses/math-for-ml-calculus/08-convexity"},{"title":"Real-World Applications","sidebarTitle":"Applications","icon":"industry","href":"/courses/math-for-ml-calculus/09-real-world-applications"}]},{"group":"ML Mastery","icon":"brain","pages":[{"title":"Machine Learning Mastery","sidebarTitle":"Introduction","icon":"brain","href":"/courses/ml-mastery/00-introduction"},{"title":"The Prediction Game","sidebarTitle":"Prediction Game","icon":"crosshairs","href":"/courses/ml-mastery/01-prediction-game"},{"title":"Learning From Mistakes","sidebarTitle":"Learning From Mistakes","icon":"graduation-cap","href":"/courses/ml-mastery/02-learning-from-mistakes"},{"title":"Linear Regression","sidebarTitle":"Linear Regression","icon":"chart-line","href":"/courses/ml-mastery/03-linear-regression"},{"title":"Classification","sidebarTitle":"Classification","icon":"tags","href":"/courses/ml-mastery/04-classification"},{"title":"K-Nearest Neighbors","sidebarTitle":"K-Nearest Neighbors","icon":"people-arrows","href":"/courses/ml-mastery/04a-knn"},{"title":"Decision Trees","sidebarTitle":"Decision Trees","icon":"sitemap","href":"/courses/ml-mastery/05-decision-trees"},{"title":"Support Vector Machines","sidebarTitle":"SVM","icon":"border-none","href":"/courses/ml-mastery/05a-svm"},{"title":"Naive Bayes","sidebarTitle":"Naive Bayes","icon":"percent","href":"/courses/ml-mastery/05b-naive-bayes"},{"title":"Ensemble Methods","sidebarTitle":"Ensemble Methods","icon":"people-group","href":"/courses/ml-mastery/06-ensemble-methods"},{"title":"Model Evaluation","sidebarTitle":"Model Evaluation","icon":"chart-column","href":"/courses/ml-mastery/07-model-evaluation"},{"title":"Feature Engineering","sidebarTitle":"Feature Engineering","icon":"wand-magic-sparkles","href":"/courses/ml-mastery/08-feature-engineering"},{"title":"Hyperparameter Tuning","sidebarTitle":"Hyperparameter Tuning","icon":"sliders","href":"/courses/ml-mastery/09-hyperparameter-tuning"},{"title":"End-to-End ML Project","sidebarTitle":"End-to-End Project","icon":"rocket","href":"/courses/ml-mastery/10-end-to-end-project"},{"title":"Clustering","sidebarTitle":"Clustering","icon":"object-group","href":"/courses/ml-mastery/11-clustering"},{"title":"Neural Networks","sidebarTitle":"Neural Networks","icon":"network-wired","href":"/courses/ml-mastery/12-neural-networks"},{"title":"Regularization","sidebarTitle":"Regularization","icon":"shield-halved","href":"/courses/ml-mastery/13-regularization"},{"title":"Model Deployment","sidebarTitle":"Deployment","icon":"rocket","href":"/courses/ml-mastery/14-model-deployment"},{"title":"Time Series Forecasting","sidebarTitle":"Time Series","icon":"clock","href":"/courses/ml-mastery/15-time-series"},{"title":"Bias-Variance Tradeoff","sidebarTitle":"Bias-Variance","icon":"scale-balanced","href":"/courses/ml-mastery/16-bias-variance"},{"title":"Data Leakage","sidebarTitle":"Data Leakage","icon":"droplet","href":"/courses/ml-mastery/17-data-leakage"},{"title":"Dimensionality Reduction","sidebarTitle":"Dimensionality Reduction","icon":"compress","href":"/courses/ml-mastery/18-dimensionality-reduction"},{"title":"Capstone Project","sidebarTitle":"Capstone Project","icon":"graduation-cap","href":"/courses/ml-mastery/19-capstone-project"},{"title":"Handling Imbalanced Data","sidebarTitle":"Imbalanced Data","icon":"scale-unbalanced","href":"/courses/ml-mastery/20-imbalanced-data"},{"title":"Model Explainability","sidebarTitle":"Explainability","icon":"lightbulb","href":"/courses/ml-mastery/21-explainability"},{"title":"ML Pipelines","sidebarTitle":"ML Pipelines","icon":"diagram-project","href":"/courses/ml-mastery/22-ml-pipelines"},{"title":"Common ML Mistakes","sidebarTitle":"Common Mistakes","icon":"triangle-exclamation","href":"/courses/ml-mastery/23-common-mistakes"},{"title":"Cross-Validation Strategies","sidebarTitle":"Cross-Validation","icon":"layer-group","href":"/courses/ml-mastery/24-cross-validation"},{"title":"Anomaly Detection","sidebarTitle":"Anomaly Detection","icon":"magnifying-glass-chart","href":"/courses/ml-mastery/25-anomaly-detection"},{"title":"ML at Scale","sidebarTitle":"ML at Scale","icon":"server","href":"/courses/ml-mastery/26-ml-at-scale"}]},{"group":"Deep Learning Mastery","icon":"network-wired","pages":[{"title":"Deep Learning Mastery","sidebarTitle":"Overview","icon":"brain-circuit","href":"/courses/deep-learning-mastery/00-overview"},{"title":"The Deep Learning Landscape","sidebarTitle":"Deep Learning Landscape","icon":"map","href":"/courses/deep-learning-mastery/01-landscape"},{"title":"Perceptrons \u0026 Multi-Layer Networks","sidebarTitle":"Perceptrons \u0026 MLPs","icon":"circle-nodes","href":"/courses/deep-learning-mastery/02-perceptrons-mlp"},{"title":"Backpropagation Deep Dive","sidebarTitle":"Backpropagation","icon":"arrows-rotate","href":"/courses/deep-learning-mastery/03-backpropagation"},{"title":"Activation Functions","sidebarTitle":"Activation Functions","icon":"bolt","href":"/courses/deep-learning-mastery/04-activation-functions"},{"title":"Loss Functions \u0026 Objectives","sidebarTitle":"Loss Functions","icon":"crosshairs","href":"/courses/deep-learning-mastery/05-loss-functions"},{"title":"Convolutional Neural Networks","sidebarTitle":"CNNs","icon":"layer-group","href":"/courses/deep-learning-mastery/06-cnns"},{"title":"Pooling, Stride \u0026 CNN Design","sidebarTitle":"CNN Design","icon":"layer-group","href":"/courses/deep-learning-mastery/07-cnn-design"},{"title":"Recurrent Neural Networks","sidebarTitle":"RNNs","icon":"rotate-right","href":"/courses/deep-learning-mastery/08-rnns"},{"title":"LSTMs \u0026 GRUs","sidebarTitle":"LSTMs \u0026 GRUs","icon":"memory","href":"/courses/deep-learning-mastery/09-lstms-grus"},{"title":"Attention Mechanism","sidebarTitle":"Attention","icon":"eye","href":"/courses/deep-learning-mastery/10-attention"},{"title":"Transformers","sidebarTitle":"Transformers","icon":"bolt","href":"/courses/deep-learning-mastery/11-transformers"},{"title":"Generative Adversarial Networks","sidebarTitle":"GANs","icon":"masks-theater","href":"/courses/deep-learning-mastery/12-gans"},{"title":"Autoencoders \u0026 VAEs","sidebarTitle":"Autoencoders","icon":"compress","href":"/courses/deep-learning-mastery/13-autoencoders"},{"title":"Diffusion Models","sidebarTitle":"Diffusion Models","icon":"cloud","href":"/courses/deep-learning-mastery/14-diffusion"},{"title":"Residual \u0026 Skip Connections","sidebarTitle":"Residual Networks","icon":"diagram-project","href":"/courses/deep-learning-mastery/15-residual-networks"},{"title":"Normalization Techniques","sidebarTitle":"Normalization","icon":"sliders","href":"/courses/deep-learning-mastery/16-normalization"},{"title":"Regularization Techniques","sidebarTitle":"Regularization","icon":"shield","href":"/courses/deep-learning-mastery/17-regularization"},{"title":"Optimization Algorithms","sidebarTitle":"Optimizers","icon":"gauge-high","href":"/courses/deep-learning-mastery/18-optimization"},{"title":"Training Strategies at Scale","sidebarTitle":"Scale Training","icon":"server","href":"/courses/deep-learning-mastery/19-training-strategies"},{"title":"Transfer Learning \u0026 Fine-tuning","sidebarTitle":"Transfer Learning","icon":"arrows-repeat","href":"/courses/deep-learning-mastery/20-transfer-learning"},{"title":"Model Deployment","sidebarTitle":"Deployment","icon":"rocket","href":"/courses/deep-learning-mastery/21-deployment"},{"title":"Debugging Deep Learning","sidebarTitle":"Debugging","icon":"bug","href":"/courses/deep-learning-mastery/22-debugging"},{"title":"Vision Transformers","sidebarTitle":"Vision Transformers","icon":"eye","href":"/courses/deep-learning-mastery/23-vision-transformers"},{"title":"Multimodal Models","sidebarTitle":"Multimodal","icon":"images","href":"/courses/deep-learning-mastery/24-multimodal"},{"title":"Foundation Models \u0026 LLMs","sidebarTitle":"Foundation Models","icon":"brain","href":"/courses/deep-learning-mastery/25-foundation-models"},{"title":"Capstone Project","sidebarTitle":"Capstone","icon":"flag-checkered","href":"/courses/deep-learning-mastery/26-capstone"},{"title":"Mathematical Foundations for Deep Learning","sidebarTitle":"Math Foundations","icon":"square-root-variable","href":"/courses/deep-learning-mastery/27-math-foundations"},{"title":"Weight Initialization Strategies","sidebarTitle":"Weight Initialization","icon":"seedling","href":"/courses/deep-learning-mastery/28-weight-initialization"},{"title":"Gradient Flow Analysis","sidebarTitle":"Gradient Flow","icon":"wave-pulse","href":"/courses/deep-learning-mastery/29-gradient-flow"},{"title":"Advanced CNN Architectures","sidebarTitle":"CNN Architectures","icon":"layer-group","href":"/courses/deep-learning-mastery/30-cnn-architectures"},{"title":"Sequence-to-Sequence Models","sidebarTitle":"Seq2Seq","icon":"arrows-left-right","href":"/courses/deep-learning-mastery/31-seq2seq"},{"title":"Self-Supervised Learning","sidebarTitle":"Self-Supervised Learning","icon":"brain","href":"/courses/deep-learning-mastery/32-self-supervised"},{"title":"Reinforcement Learning for Deep Learning","sidebarTitle":"RL for Deep Learning","icon":"gamepad","href":"/courses/deep-learning-mastery/33-rl-for-dl"},{"title":"Neural Architecture Search","sidebarTitle":"Neural Architecture Search","icon":"magnifying-glass","href":"/courses/deep-learning-mastery/34-nas"},{"title":"Interpretability \u0026 Explainability","sidebarTitle":"Interpretability","icon":"eye","href":"/courses/deep-learning-mastery/35-interpretability"},{"title":"Adversarial Robustness","sidebarTitle":"Adversarial ML","icon":"shield-halved","href":"/courses/deep-learning-mastery/36-adversarial-robustness"},{"title":"Efficient Neural Network Architectures","sidebarTitle":"Efficient Architectures","icon":"bolt","href":"/courses/deep-learning-mastery/37-efficient-architectures"},{"title":"Graph Neural Networks","sidebarTitle":"Graph Networks","icon":"diagram-project","href":"/courses/deep-learning-mastery/38-graph-neural-networks"},{"title":"3D Deep Learning","sidebarTitle":"3D Deep Learning","icon":"cube","href":"/courses/deep-learning-mastery/39-3d-deep-learning"},{"title":"Object Detection Deep Dive","sidebarTitle":"Object Detection","icon":"vector-square","href":"/courses/deep-learning-mastery/40-object-detection"},{"title":"Semantic Segmentation","sidebarTitle":"Segmentation","icon":"object-group","href":"/courses/deep-learning-mastery/41-semantic-segmentation"},{"title":"Hyperparameter Tuning","sidebarTitle":"Hyperparameter Tuning","icon":"sliders","href":"/courses/deep-learning-mastery/42-hyperparameter-tuning"},{"title":"Reproducibility","sidebarTitle":"Reproducibility","description":"Experiment tracking, versioning, and reproducible ML workflows","icon":"repeat","href":"/courses/deep-learning-mastery/43-reproducibility"},{"title":"Memory-Efficient Training","sidebarTitle":"Memory-Efficient Training","icon":"memory","href":"/courses/deep-learning-mastery/44-memory-efficient-training"},{"title":"Quantization Deep Dive","sidebarTitle":"Quantization Deep Dive","description":"INT8, FP8, PTQ, QAT, and mixed-precision quantization","icon":"compress","href":"/courses/deep-learning-mastery/45-quantization-deep-dive"},{"title":"Knowledge Distillation","sidebarTitle":"Knowledge Distillation","icon":"school","href":"/courses/deep-learning-mastery/46-knowledge-distillation"},{"title":"Continual Learning","sidebarTitle":"Continual Learning","icon":"arrows-rotate","href":"/courses/deep-learning-mastery/47-continual-learning"}]},{"group":"NestJS Mastery","icon":"feather","pages":[{"title":"NestJS Mastery","sidebarTitle":"Course Overview","icon":"node","href":"/courses/nestjs/overview"},{"title":"NestJS Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/nestjs/fundamentals"},{"title":"Dependency Injection","sidebarTitle":"2. DI \u0026 Modules","icon":"puzzle-piece","href":"/courses/nestjs/dependency-injection"},{"title":"Controllers \u0026 Routing","sidebarTitle":"3. Controllers","icon":"route","href":"/courses/nestjs/controllers-routes"},{"title":"Providers \u0026 Services","sidebarTitle":"4. Providers","icon":"gear","href":"/courses/nestjs/providers-services"},{"title":"Database Integration","sidebarTitle":"5. Database","icon":"database","href":"/courses/nestjs/database-integration"},{"title":"Authentication \u0026 Authorization","sidebarTitle":"6. Auth","icon":"lock","href":"/courses/nestjs/authentication-authorization"},{"title":"Testing","sidebarTitle":"7. Testing","icon":"flask","href":"/courses/nestjs/testing"},{"title":"Microservices","sidebarTitle":"8. Microservices","icon":"cubes","href":"/courses/nestjs/microservices"},{"title":"Deployment \u0026 Production","sidebarTitle":"9. Deployment","icon":"rocket","href":"/courses/nestjs/deployment-production"},{"title":"Advanced Patterns","sidebarTitle":"10. Advanced","icon":"star","href":"/courses/nestjs/advanced-patterns"}]},{"group":"Microservices Mastery","icon":"cubes","pages":[{"title":"Microservices Mastery","sidebarTitle":"Course Overview","icon":"cubes","href":"/courses/microservices-mastery/00-overview"},{"title":"1. Microservices Foundations","href":"/courses/microservices-mastery/01-foundations"},{"title":"2. Domain-Driven Design","href":"/courses/microservices-mastery/02-domain-driven-design"},{"title":"3. Synchronous Communication","href":"/courses/microservices-mastery/03-sync-communication"},{"title":"4. Asynchronous Communication","href":"/courses/microservices-mastery/04-async-communication"},{"title":"5. API Gateway Pattern","href":"/courses/microservices-mastery/05-api-gateway"},{"title":"6. Data Management Patterns","href":"/courses/microservices-mastery/06-data-management"},{"title":"7. Resilience Patterns","href":"/courses/microservices-mastery/07-resilience-patterns"},{"title":"8. Service Discovery","href":"/courses/microservices-mastery/08-service-discovery"},{"title":"9. Observability","href":"/courses/microservices-mastery/09-observability"},{"title":"10. Security","href":"/courses/microservices-mastery/10-security"},{"title":"11. Containerization with Docker","href":"/courses/microservices-mastery/11-containerization"},{"title":"12. Kubernetes Deployment","href":"/courses/microservices-mastery/12-kubernetes"},{"title":"13. Testing Strategies","href":"/courses/microservices-mastery/13-testing"},{"title":"14. Interview Preparation","href":"/courses/microservices-mastery/14-interview-prep"},{"title":"15. Capstone Project","href":"/courses/microservices-mastery/15-capstone"},{"title":"16. Service Mesh","href":"/courses/microservices-mastery/16-service-mesh"},{"title":"17. Configuration Management","href":"/courses/microservices-mastery/17-configuration-management"},{"title":"18. CI/CD for Microservices","href":"/courses/microservices-mastery/18-cicd"},{"title":"19. Database Patterns","href":"/courses/microservices-mastery/19-database-patterns"},{"title":"20. Caching Strategies","href":"/courses/microservices-mastery/20-caching-strategies"},{"title":"21. Chaos Engineering","href":"/courses/microservices-mastery/21-chaos-engineering"},{"title":"22. Case Studies","href":"/courses/microservices-mastery/22-case-studies"},{"title":"23. Load Balancing Deep Dive","href":"/courses/microservices-mastery/23-load-balancing"},{"title":"24. Migration Patterns","href":"/courses/microservices-mastery/24-migration-patterns"},{"title":"25. Event Sourcing Deep Dive","href":"/courses/microservices-mastery/25-event-sourcing-deep-dive"},{"title":"26. GraphQL Federation","href":"/courses/microservices-mastery/26-graphql-federation"}]},{"group":"Low Level Design","pages":[{"title":"Low Level Design (LLD)","icon":"cubes","href":"/lld/overview"},{"title":"LLD Interview Cheat Sheet","icon":"file-lines","href":"/lld/interview-cheatsheet"}]},{"group":"OOP Concepts","pages":[{"title":"What is OOP?","icon":"graduation-cap","href":"/lld/oop/introduction"},{"title":"Encapsulation","icon":"box","href":"/lld/oop/encapsulation"},{"title":"Inheritance","icon":"sitemap","href":"/lld/oop/inheritance"},{"title":"Polymorphism","icon":"shapes","href":"/lld/oop/polymorphism"},{"title":"Abstraction","icon":"eye-slash","href":"/lld/oop/abstraction"},{"title":"OOP Concepts","icon":"object-group","href":"/lld/oop-concepts"}]},{"group":"SOLID Principles","pages":[{"title":"SOLID Introduction","icon":"wand-magic-sparkles","href":"/lld/solid/introduction"},{"title":"Single Responsibility Principle","icon":"user-check","href":"/lld/solid/srp"},{"title":"Open/Closed Principle","icon":"plug","href":"/lld/solid/ocp"},{"title":"Liskov Substitution Principle","icon":"right-left","href":"/lld/solid/lsp"},{"title":"Interface Segregation Principle","icon":"puzzle-piece","href":"/lld/solid/isp"},{"title":"Dependency Inversion Principle","icon":"outlet","href":"/lld/solid/dip"},{"title":"SOLID Principles in Practice","icon":"compass-drafting","href":"/lld/solid-practice"}]},{"group":"Design Patterns","pages":[{"title":"Design Patterns","icon":"shapes","href":"/lld/design-patterns"},{"title":"UML Diagrams","icon":"diagram-project","href":"/lld/uml-diagrams"}]},{"group":"LLD Case Studies","pages":[{"title":"Case Study: Parking Lot System","icon":"square-parking","href":"/lld/case-parking-lot"},{"title":"Case Study: Library Management System","icon":"book","href":"/lld/case-library"},{"title":"Design Vending Machine","icon":"box","href":"/lld/case-vending-machine"},{"title":"Design LRU Cache","icon":"database","href":"/lld/case-lru-cache"},{"title":"Design Movie Ticket Booking System","icon":"ticket","href":"/lld/case-movie-ticket"},{"title":"Case Study: Elevator System","icon":"elevator","href":"/lld/case-elevator"},{"title":"Case Study: Chess Game","icon":"chess","href":"/lld/case-chess"},{"title":"Case Study: Hotel Booking System","icon":"hotel","href":"/lld/case-hotel-booking"},{"title":"Case Study: ATM System","icon":"building-columns","href":"/lld/case-atm"}]},{"group":"System Design (HLD)","pages":[{"title":"System Design Mastery","icon":"network-wired","href":"/system-design/overview"},{"title":"Interview Guide \u0026 Cheatsheet","icon":"clipboard-check","href":"/system-design/interview-guide"},{"title":"Interview Questions Bank","icon":"clipboard-question","href":"/system-design/interview-questions"},{"title":"System Design Fundamentals","icon":"book","href":"/system-design/fundamentals"},{"title":"System Design Building Blocks","icon":"cubes","href":"/system-design/building-blocks"},{"title":"Networking Fundamentals","icon":"network-wired","href":"/system-design/networking"},{"title":"Database Design","icon":"database","href":"/system-design/databases"},{"title":"API Design","icon":"plug","href":"/system-design/api-design"},{"title":"Distributed Systems","icon":"share-nodes","href":"/system-design/distributed-systems"},{"title":"Microservices Architecture","icon":"cubes","href":"/system-design/microservices"},{"title":"Back-of-Envelope Calculations","icon":"calculator","href":"/system-design/estimations"}]},{"group":"Senior Level (L5+/Staff)","pages":[{"title":"Advanced Concepts","icon":"brain","href":"/system-design/advanced-concepts"},{"title":"Scalability Patterns","icon":"arrow-up-right-dots","href":"/system-design/scalability-patterns"},{"title":"Reliability \u0026 Fault Tolerance","icon":"shield-halved","href":"/system-design/reliability-patterns"},{"title":"Observability \u0026 Monitoring","icon":"chart-line","href":"/system-design/observability"},{"title":"Data Pipelines","icon":"pipe","href":"/system-design/data-pipelines"},{"title":"Security Patterns","icon":"shield","href":"/system-design/security-patterns"}]},{"group":"HLD Case Studies","pages":[{"title":"Design: URL Shortener","icon":"link","href":"/system-design/case-url-shortener"},{"title":"Design: Rate Limiter","icon":"gauge","href":"/system-design/case-rate-limiter"},{"title":"Design a Notification System","icon":"bell","href":"/system-design/case-notification-system"},{"title":"Design: Twitter/X Timeline","icon":"twitter","href":"/system-design/case-twitter"},{"title":"Design: WhatsApp/Messenger","icon":"message","href":"/system-design/case-whatsapp"},{"title":"Design a Payment System","icon":"credit-card","href":"/system-design/case-payment-system"},{"title":"Design: Netflix/YouTube","icon":"video","href":"/system-design/case-netflix"},{"title":"Design: Uber/Lyft","icon":"car","href":"/system-design/case-uber"},{"title":"Design Google Maps","icon":"map-location-dot","href":"/system-design/case-google-maps"}]},{"group":"Engineering Fundamentals","pages":[{"title":"Engineering Topics","icon":"gears","href":"/engineering/overview"},{"title":"Computer Science Fundamentals","icon":"microchip","href":"/engineering/cs-fundamentals"},{"title":"Software Design Principles","icon":"compass-drafting","href":"/engineering/design-principles"},{"title":"Architecture Patterns","icon":"building","href":"/engineering/architecture-patterns"}]},{"group":"DevOps \u0026 Operations","pages":[{"title":"DevOps \u0026 Infrastructure","icon":"server","href":"/engineering/devops"},{"title":"Security Fundamentals","icon":"shield-halved","href":"/engineering/security"},{"title":"Performance \u0026 Optimization","icon":"gauge-high","href":"/engineering/performance"}]},{"group":"Azure Cloud Engineering","icon":"cloud","pages":[{"title":"Azure Cloud Engineering","sidebarTitle":"Overview","icon":"cloud","href":"/courses/azure-cloud-engineering/00-overview"},{"title":"Azure Fundamentals \u0026 Architecture","icon":"cloud-arrow-up","href":"/courses/azure-cloud-engineering/01-azure-fundamentals"},{"title":"Identity \u0026 Access Management","icon":"user-shield","href":"/courses/azure-cloud-engineering/02-identity-access-management"},{"title":"03a. Networking Fundamentals","icon":"network-wired","href":"/courses/azure-cloud-engineering/03a-networking-fundamentals"},{"title":"03b. Hybrid \u0026 Enterprise","icon":"globe","href":"/courses/azure-cloud-engineering/03b-hybrid-networking"},{"title":"03c. Traffic \u0026 Security","icon":"traffic-light","href":"/courses/azure-cloud-engineering/03c-traffic-management"},{"title":"Compute Services","icon":"server","href":"/courses/azure-cloud-engineering/04-compute-services"},{"title":"Storage Solutions","icon":"database","href":"/courses/azure-cloud-engineering/05-storage-solutions"},{"title":"Database Services","icon":"database","href":"/courses/azure-cloud-engineering/06-database-services"},{"title":"Azure Kubernetes Service (AKS)","icon":"dharmachakra","href":"/courses/azure-cloud-engineering/07-container-kubernetes"},{"title":"Serverless Architecture","icon":"bolt","href":"/courses/azure-cloud-engineering/08-serverless-architecture"},{"title":"Monitoring \u0026 Observability","icon":"chart-line","href":"/courses/azure-cloud-engineering/09-monitoring-observability"},{"title":"Security \u0026 Compliance","icon":"shield-halved","href":"/courses/azure-cloud-engineering/10-security-compliance"},{"title":"DevOps \u0026 CI/CD","icon":"code-branch","href":"/courses/azure-cloud-engineering/11-devops-cicd"},{"title":"Cost Optimization","icon":"dollar-sign","href":"/courses/azure-cloud-engineering/12-cost-optimization"},{"title":"High Availability \u0026 Disaster Recovery","icon":"life-ring","href":"/courses/azure-cloud-engineering/13-high-availability-dr"},{"title":"Architecture Patterns","icon":"diagram-project","href":"/courses/azure-cloud-engineering/14-architecture-patterns"},{"title":"Capstone Project","icon":"trophy","href":"/courses/azure-cloud-engineering/15-capstone-project"},{"title":"API Management \u0026 Messaging Patterns","icon":"diagram-project","href":"/courses/azure-cloud-engineering/16-api-messaging"},{"title":"SRE \u0026 Production Excellence","icon":"shield-heart","href":"/courses/azure-cloud-engineering/17-sre-production"}]},{"group":"AWS Cloud","icon":"cloud","pages":[{"title":"AWS Cloud Mastery","icon":"cloud","href":"/aws/overview"},{"title":"Core Concepts","icon":"circle-nodes","href":"/aws/core-concepts"},{"title":"Compute Services","icon":"microchip","href":"/aws/compute"},{"title":"Storage \u0026 Databases","icon":"hard-drive","href":"/aws/storage-databases"},{"title":"Networking","icon":"diagram-project","href":"/aws/networking"},{"title":"Security \u0026 IAM","icon":"shield-halved","href":"/aws/security"},{"title":"Well-Architected Framework","icon":"building","href":"/aws/well-architected"}]},{"group":"AWS Monitoring \u0026 Observability","icon":"chart-line","pages":[{"title":"Amazon CloudWatch","icon":"chart-line","href":"/aws/cloudwatch"},{"title":"AWS X-Ray","icon":"microscope","href":"/aws/xray"},{"title":"Observability \u0026 Monitoring","icon":"chart-line","href":"/aws/observability"}]},{"group":"AWS Security Services","icon":"shield-halved","pages":[{"title":"AWS GuardDuty","icon":"shield","href":"/aws/guardduty"},{"title":"AWS Security Hub","icon":"building-shield","href":"/aws/security-hub"},{"title":"AWS Inspector","href":"/aws/inspector"},{"title":"AWS WAF (Web Application Firewall)","href":"/aws/waf"},{"title":"AWS Shield","href":"/aws/shield"},{"title":"AWS CloudTrail","icon":"file-lines","href":"/aws/cloudtrail"},{"title":"AWS Config","icon":"gear","href":"/aws/config"}]},{"group":"AWS Serverless","icon":"bolt","pages":[{"title":"AWS Lambda","icon":"bolt","href":"/aws/lambda"},{"title":"AWS Step Functions","icon":"diagram-project","href":"/aws/step-functions"},{"title":"AWS SAM","icon":"box","href":"/aws/sam"},{"title":"Amazon DynamoDB","icon":"database","href":"/aws/dynamodb"}]},{"group":"AWS Operations","icon":"gears","pages":[{"title":"DevOps \u0026 CI/CD","icon":"code-branch","href":"/aws/devops-cicd"}]},{"group":"AWS Advanced","icon":"rocket","pages":[{"title":"CDN \u0026 Edge Services","icon":"globe","href":"/aws/cdn-edge"},{"title":"Messaging \u0026 Integration","icon":"arrows-split-up-and-left","href":"/aws/messaging"},{"title":"Containers \u0026 Microservices","icon":"cubes","href":"/aws/containers"}]},{"group":"AWS Case Studies","icon":"lightbulb","pages":[{"title":"Case Study: Serverless URL Shortener","icon":"bolt","href":"/aws/case-study-serverless"}]},{"group":"GCP Cloud Engineering","icon":"cloud","pages":[{"title":"GCP Cloud Engineering","sidebarTitle":"Overview","icon":"cloud","href":"/courses/gcp-cloud-engineering/00-overview"},{"title":"GCP Fundamentals","href":"/courses/gcp-cloud-engineering/01-gcp-fundamentals"},{"title":"Identity \u0026 Security","href":"/courses/gcp-cloud-engineering/02-iam-security"},{"title":"Networking \u0026 VPC","href":"/courses/gcp-cloud-engineering/03-networking-vpc"},{"title":"Load Balancing \u0026 DNS","href":"/courses/gcp-cloud-engineering/04-load-balancing-dns"},{"title":"Compute Engine","href":"/courses/gcp-cloud-engineering/05-compute-engine"},{"title":"Cloud Storage","href":"/courses/gcp-cloud-engineering/06-cloud-storage"},{"title":"Relational Databases","href":"/courses/gcp-cloud-engineering/07-relational-databases"},{"title":"NoSQL Databases","href":"/courses/gcp-cloud-engineering/08-nosql-databases"},{"title":"Containers \u0026 Artifacts","href":"/courses/gcp-cloud-engineering/09-containers-registry"},{"title":"Google Kubernetes Engine","href":"/courses/gcp-cloud-engineering/10-google-kubernetes-engine"},{"title":"Serverless Compute","href":"/courses/gcp-cloud-engineering/11-serverless-compute"},{"title":"Data Analytics \u0026 BigQuery","href":"/courses/gcp-cloud-engineering/12-data-analytics-bigquery"},{"title":"Monitoring \u0026 Observability","href":"/courses/gcp-cloud-engineering/13-monitoring-observability"},{"title":"Security \u0026 Compliance","href":"/courses/gcp-cloud-engineering/14-security-compliance"},{"title":"IaC \u0026 Deployment","href":"/courses/gcp-cloud-engineering/15-iac-deployment"},{"title":"Hybrid \u0026 Multi-Cloud","href":"/courses/gcp-cloud-engineering/16-hybrid-multi-cloud"},{"title":"Cost Management","href":"/courses/gcp-cloud-engineering/17-cost-management"},{"title":"Capstone Project","href":"/courses/gcp-cloud-engineering/18-capstone-project"}]},{"group":"DevOps Tools","icon":"toolbox","pages":[{"title":"DevOps Tools Mastery","sidebarTitle":"Overview","icon":"toolbox","href":"/courses/devops-tools/overview"},{"group":"Git Crash Course","pages":[{"title":"Git Crash Course","sidebarTitle":"Git Overview","icon":"code-branch","href":"/courses/devops-tools/git-overview"},{"title":"Git Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/git-fundamentals"},{"title":"Git Branching \u0026 Merging","sidebarTitle":"Branching","icon":"code-branch","href":"/courses/devops-tools/git-branching"},{"title":"Git Collaboration","sidebarTitle":"Collaboration","icon":"users","href":"/courses/devops-tools/git-collaboration"},{"title":"Git Advanced Topics","sidebarTitle":"Advanced","icon":"wand-magic-sparkles","href":"/courses/devops-tools/git-advanced"},{"title":"Git Workflows","sidebarTitle":"Workflows","icon":"code-branch","href":"/courses/devops-tools/git-workflows"},{"title":"Git Hooks \u0026 Automation","sidebarTitle":"Hooks \u0026 Automation","icon":"robot","href":"/courses/devops-tools/git-hooks"}]},{"group":"Linux Crash Course","pages":[{"title":"Linux Crash Course","sidebarTitle":"Linux Overview","icon":"linux","href":"/courses/devops-tools/linux-overview"},{"title":"Linux Fundamentals","sidebarTitle":"Fundamentals","icon":"terminal","href":"/courses/devops-tools/linux-fundamentals"},{"title":"Linux Permissions \u0026 Users","sidebarTitle":"Permissions","icon":"lock","href":"/courses/devops-tools/linux-permissions"},{"title":"Linux Process Management","sidebarTitle":"Processes","icon":"gears","href":"/courses/devops-tools/linux-processes"},{"title":"Linux Networking","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/devops-tools/linux-networking"},{"title":"Linux Shell Scripting","sidebarTitle":"Shell Scripting","icon":"file-code","href":"/courses/devops-tools/linux-scripting"},{"title":"System Monitoring","sidebarTitle":"Monitoring","icon":"chart-line","href":"/courses/devops-tools/linux-monitoring"},{"title":"Security Hardening","sidebarTitle":"Security","icon":"lock","href":"/courses/devops-tools/linux-security"}]},{"group":"Docker Crash Course","pages":[{"title":"Docker Crash Course","sidebarTitle":"Docker Overview","icon":"docker","href":"/courses/devops-tools/docker-overview"},{"title":"Docker Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/docker-fundamentals"},{"title":"Building Docker Images","sidebarTitle":"Images","icon":"layer-group","href":"/courses/devops-tools/docker-images"},{"title":"Docker Networking \u0026 Volumes","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/devops-tools/docker-networking"},{"title":"Docker Compose","sidebarTitle":"Compose","icon":"cubes","href":"/courses/devops-tools/docker-compose"},{"title":"Docker Best Practices","sidebarTitle":"Best Practices","icon":"shield-check","href":"/courses/devops-tools/docker-best-practices"}]},{"group":"RabbitMQ Crash Course","pages":[{"title":"RabbitMQ Crash Course","sidebarTitle":"RabbitMQ Overview","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-overview"},{"title":"RabbitMQ Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/rabbitmq-fundamentals"},{"title":"Patterns","sidebarTitle":"Patterns","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-patterns"},{"title":"Reliability","sidebarTitle":"Reliability","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-reliability"}]},{"group":"Kafka Crash Course","pages":[{"title":"Kafka Crash Course","sidebarTitle":"Kafka Overview","icon":"stream","href":"/courses/devops-tools/kafka-overview"},{"title":"Kafka Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/kafka-fundamentals"},{"title":"Kafka Producers \u0026 Consumers","sidebarTitle":"Producers \u0026 Consumers","icon":"arrows-left-right","href":"/courses/devops-tools/kafka-producers-consumers"},{"title":"Kafka Streams","sidebarTitle":"Streams","icon":"water","href":"/courses/devops-tools/kafka-streams"},{"title":"Kafka Operations","sidebarTitle":"Operations","icon":"server","href":"/courses/devops-tools/kafka-operations"}]},{"group":"Kubernetes Crash Course","pages":[{"title":"Kubernetes Crash Course","sidebarTitle":"Kubernetes Overview","icon":"dharmachakra","href":"/courses/devops-tools/kubernetes-overview"},{"title":"Kubernetes Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/kubernetes-fundamentals"},{"title":"Kubernetes Workloads","sidebarTitle":"Workloads","icon":"cubes","href":"/courses/devops-tools/kubernetes-workloads"},{"title":"Kubernetes Services","sidebarTitle":"Services","icon":"network-wired","href":"/courses/devops-tools/kubernetes-services"},{"title":"Kubernetes Configuration","sidebarTitle":"Configuration","icon":"gear","href":"/courses/devops-tools/kubernetes-config"},{"title":"Kubernetes Storage","sidebarTitle":"Storage","icon":"hard-drive","href":"/courses/devops-tools/kubernetes-storage"},{"title":"Kubernetes on Windows \u0026 Linux","sidebarTitle":"Windows \u0026 Linux","icon":"window-maximize","href":"/courses/devops-tools/kubernetes-windows-linux"}]}]},{"group":"Database Engineering","icon":"database","pages":[{"title":"Database Engineering Mastery","sidebarTitle":"Course Overview","icon":"database","href":"/courses/database-engineering/overview"},{"group":"Track 1: Foundations","pages":[{"title":"SQL Foundations","sidebarTitle":"1. SQL Foundations","icon":"terminal","href":"/courses/database-engineering/sql-foundations"},{"title":"Database Design \u0026 Modeling","sidebarTitle":"2. Database Design","icon":"diagram-project","href":"/courses/database-engineering/database-design"},{"title":"Transactions \u0026 ACID","sidebarTitle":"3. Transactions \u0026 ACID","icon":"shield-check","href":"/courses/database-engineering/transactions"}]},{"group":"Track 2: Performance","pages":[{"title":"Indexing Deep Dive","sidebarTitle":"4. Indexing Deep Dive","icon":"magnifying-glass-chart","href":"/courses/database-engineering/indexing"},{"title":"Query Optimization","sidebarTitle":"5. Query Optimization","icon":"bolt","href":"/courses/database-engineering/query-optimization"},{"title":"Query Execution Plans Mastery","sidebarTitle":"Execution Plans Deep Dive","icon":"route","href":"/courses/database-engineering/execution-plans-mastery"},{"title":"Performance Engineering","sidebarTitle":"Performance Engineering","icon":"gauge-high","href":"/courses/database-engineering/performance-engineering"}]},{"group":"Track 3: Internals","pages":[{"title":"PostgreSQL Internals","sidebarTitle":"6. PostgreSQL Internals","icon":"microchip","href":"/courses/database-engineering/internals"},{"title":"PostgreSQL Data Structures","sidebarTitle":"Data Structures","icon":"sitemap","href":"/courses/database-engineering/data-structures"},{"title":"Query Processing Pipeline","sidebarTitle":"8. Query Processing","icon":"gears","href":"/courses/database-engineering/query-processing"},{"title":"Query Engine Deep Dive","sidebarTitle":"Query Engine Internals","icon":"microchip","href":"/courses/database-engineering/query-engine-deep-dive"},{"title":"PostgreSQL Architecture","sidebarTitle":"9. PG Architecture","icon":"sitemap","href":"/courses/database-engineering/architecture"},{"title":"Storage Engine Deep Dive","sidebarTitle":"Storage Engine Internals","icon":"hard-drive","href":"/courses/database-engineering/storage-deep-dive"}]},{"group":"Track 4: Distributed","pages":[{"title":"Replication \u0026 High Availability","sidebarTitle":"7. Replication \u0026 HA","icon":"server","href":"/courses/database-engineering/replication"},{"title":"Scaling Strategies","sidebarTitle":"8. Scaling","icon":"chart-line","href":"/courses/database-engineering/scaling"},{"title":"Distributed Database Systems","sidebarTitle":"Distributed Systems","icon":"network-wired","href":"/courses/database-engineering/distributed-systems"}]},{"group":"OSS Contributions","pages":[{"title":"PostgreSQL Source Code","sidebarTitle":"13. Source Code","icon":"code","href":"/courses/database-engineering/source-code"},{"title":"Contributing to PostgreSQL","sidebarTitle":"14. Contributing","icon":"code-pull-request","href":"/courses/database-engineering/contributing"}]},{"group":"Capstone","pages":[{"title":"Senior Interview Mastery","sidebarTitle":"15. Interview Prep","icon":"user-tie","href":"/courses/database-engineering/interview-prep"},{"title":"System Design Case Studies","sidebarTitle":"16. Case Studies","icon":"building","href":"/courses/database-engineering/case-studies"}]}]},{"group":"HIPAA Compliance Mastery","icon":"shield-check","pages":[{"title":"HIPAA Compliance Mastery","sidebarTitle":"Course Overview","icon":"hospital","href":"/courses/hipaa-compliance/overview"},{"group":"Track 1: Regulatory Foundations","pages":[{"title":"HIPAA Fundamentals","sidebarTitle":"HIPAA Fundamentals","icon":"book-medical","href":"/courses/hipaa-compliance/hipaa-fundamentals"},{"title":"HIPAA Risk Assessment","sidebarTitle":"Risk Assessment","icon":"magnifying-glass-chart","href":"/courses/hipaa-compliance/risk-assessment"},{"title":"PDPL \u0026 Global Data Protection","sidebarTitle":"PDPL Compliance","icon":"globe","href":"/courses/hipaa-compliance/pdpl-compliance"}]},{"group":"Track 2: Security Architecture","pages":[{"title":"Access Control Systems","sidebarTitle":"Access Control","icon":"user-shield","href":"/courses/hipaa-compliance/access-control"},{"title":"Encryption: Data at Rest \u0026 In Transit","sidebarTitle":"Encryption","icon":"lock","href":"/courses/hipaa-compliance/encryption"},{"title":"HIPAA Database Security \u0026 Encryption","href":"/courses/hipaa-compliance/database-security"}]},{"group":"Track 3: Implementation","pages":[{"title":"Audit Logging for HIPAA","sidebarTitle":"Audit Logging","icon":"file-lines","href":"/courses/hipaa-compliance/audit-logging"},{"title":"E2E Encryption with AI Agents","sidebarTitle":"E2E Encryption + AI","icon":"robot","href":"/courses/hipaa-compliance/e2e-encryption-ai"},{"title":"Implementation Guide","sidebarTitle":"Implementation Guide","icon":"code","href":"/courses/hipaa-compliance/implementation-guide"}]},{"group":"Track 4: Operations \u0026 Governance","pages":[{"title":"Incident Response for Healthcare","sidebarTitle":"Incident Response","icon":"siren","href":"/courses/hipaa-compliance/incident-response"},{"title":"HIPAA Vendor \u0026 Third-Party Risk Management","href":"/courses/hipaa-compliance/vendor-management"}]},{"group":"Track 5: Advanced Topics","pages":[{"title":"HIPAA Breach Case Studies \u0026 Lessons Learned","href":"/courses/hipaa-compliance/case-studies"},{"title":"Compliance Checklist \u0026 BAA","sidebarTitle":"Compliance Checklist","icon":"clipboard-check","href":"/courses/hipaa-compliance/compliance-checklist"}]},{"group":"Track 6: Capstone","pages":[{"title":"HIPAA Compliance Capstone Project","href":"/courses/hipaa-compliance/capstone-project"}]}]},{"group":"Operating Systems","icon":"microchip","pages":[{"title":"Operating Systems Mastery","sidebarTitle":"Course Overview","icon":"microchip","href":"/operating-systems/overview"},{"title":"OS Fundamentals \u0026 System Call Internals","sidebarTitle":"Fundamentals","icon":"book","href":"/operating-systems/os-fundamentals"},{"title":"CPU Architectures \u0026 Microarchitecture","sidebarTitle":"CPU Architectures","icon":"microchip","href":"/operating-systems/cpu-architectures"},{"title":"Operating System Interfaces","sidebarTitle":"xv6 OS Interfaces","icon":"book-open-reader","href":"/operating-systems/xv6"},{"group":"System Startup","pages":[{"title":"The Boot Process: From Reset Vector to PID 1","sidebarTitle":"Boot Process","icon":"power-off","href":"/operating-systems/boot-process"}]},{"group":"Process Management","pages":[{"title":"Process Management","sidebarTitle":"Processes","icon":"microchip","href":"/operating-systems/processes"},{"title":"Threads \u0026 Concurrency: The Senior Mastery Guide","sidebarTitle":"Threads","icon":"layer-group","href":"/operating-systems/threads"},{"title":"CPU Scheduling","sidebarTitle":"Scheduling","icon":"clock","href":"/operating-systems/scheduling"}]},{"group":"Memory Management","pages":[{"title":"Memory Management","sidebarTitle":"Memory Management","icon":"memory","href":"/operating-systems/memory-management"},{"title":"Virtual Memory","sidebarTitle":"Virtual Memory","icon":"memory","href":"/operating-systems/virtual-memory"}]},{"group":"Concurrency \u0026 I/O","pages":[{"title":"Advanced Synchronization \u0026 RCU","sidebarTitle":"Synchronization","icon":"hand-middle-finger","href":"/operating-systems/synchronization"},{"title":"Deadlocks","sidebarTitle":"Deadlocks","icon":"circle-xmark","href":"/operating-systems/deadlocks"},{"title":"Inter-Process Communication: Kernel Internals \u0026 Architecture","sidebarTitle":"IPC","icon":"message","href":"/operating-systems/ipc"},{"title":"Virtual File System (VFS) \u0026 Disk Internals","sidebarTitle":"File Systems","icon":"file-lines","href":"/operating-systems/file-systems"},{"title":"I/O Systems \u0026 Modern I/O","sidebarTitle":"I/O Systems","icon":"input-pipe","href":"/operating-systems/io-systems"}]},{"group":"Networking","pages":[{"title":"Kernel Networking Stack: From NIC to Socket","sidebarTitle":"Networking","icon":"network-wired","href":"/operating-systems/networking"}]},{"group":"Virtualization","pages":[{"title":"Containers \u0026 Virtualization: From Namespaces to Hypervisors","sidebarTitle":"Containers \u0026 VMs","icon":"cubes","href":"/operating-systems/containers-virtualization"}]},{"group":"Advanced Topics","pages":[{"title":"OS Security: From Hardware to Userspace","sidebarTitle":"Security","icon":"shield-halved","href":"/operating-systems/security"},{"title":"Linux Internals","sidebarTitle":"Linux Internals","icon":"linux","href":"/operating-systems/linux-internals"},{"title":"Modern OS Features","sidebarTitle":"Modern Features","icon":"rocket","href":"/operating-systems/modern-features"},{"title":"Observability \u0026 Performance Engineering","sidebarTitle":"Performance","icon":"gauge-high","href":"/operating-systems/debugging-performance"}]},{"group":"Interview Prep","pages":[{"title":"Interview Preparation","sidebarTitle":"Interview Prep","icon":"user-tie","href":"/operating-systems/interview-prep"},{"title":"OS Case Studies","sidebarTitle":"Case Studies","icon":"building","href":"/operating-systems/case-studies"}]}]},{"group":"Linux Internals","icon":"linux","pages":[{"title":"Linux Internals \u0026 Kernel Mastery","sidebarTitle":"Course Overview","icon":"linux","href":"/courses/linux-internals/overview"},{"group":"Kernel Foundations","pages":[{"title":"Linux Kernel Architecture","sidebarTitle":"Kernel Architecture","icon":"sitemap","href":"/courses/linux-internals/kernel-architecture"},{"title":"System Call Interface","sidebarTitle":"System Calls","icon":"terminal","href":"/courses/linux-internals/syscalls"},{"title":"Kernel Data Structures","sidebarTitle":"Data Structures","icon":"database","href":"/courses/linux-internals/data-structures"},{"title":"Interrupts \u0026 Exception Handling","sidebarTitle":"Interrupts","icon":"bolt","href":"/courses/linux-internals/interrupts-exceptions"},{"title":"Kernel Synchronization","sidebarTitle":"Synchronization","icon":"lock","href":"/courses/linux-internals/synchronization"}]},{"group":"Process \u0026 Memory","pages":[{"title":"Process Subsystem Deep Dive","sidebarTitle":"Process Subsystem","icon":"microchip","href":"/courses/linux-internals/process-subsystem"},{"title":"Memory Management Internals","sidebarTitle":"Memory Management","icon":"memory","href":"/courses/linux-internals/memory-management"},{"title":"Signals \u0026 IPC","sidebarTitle":"Signals \u0026 IPC","icon":"arrows-split-up-and-left","href":"/courses/linux-internals/signals-ipc"}]},{"group":"Containers \u0026 Isolation","pages":[{"title":"Namespaces Deep Dive","sidebarTitle":"Namespaces","icon":"boxes-stacked","href":"/courses/linux-internals/namespaces"},{"title":"Control Groups (cgroups)","sidebarTitle":"cgroups","icon":"object-group","href":"/courses/linux-internals/cgroups"},{"title":"Security Modules \u0026 Capabilities","sidebarTitle":"Security","icon":"shield-halved","href":"/courses/linux-internals/security-modules"}]},{"group":"I/O \u0026 Networking","pages":[{"title":"Filesystem \u0026 VFS","sidebarTitle":"Filesystem \u0026 VFS","icon":"folder-tree","href":"/courses/linux-internals/filesystem-vfs"},{"title":"I/O Subsystem","sidebarTitle":"I/O Subsystem","icon":"hard-drive","href":"/courses/linux-internals/io-subsystem"},{"title":"Network Stack","sidebarTitle":"Network Stack","icon":"network-wired","href":"/courses/linux-internals/networking-stack"}]},{"group":"Observability","pages":[{"title":"eBPF Deep Dive","sidebarTitle":"eBPF","icon":"magnifying-glass-chart","href":"/courses/linux-internals/ebpf"},{"title":"Tracing \u0026 Profiling","sidebarTitle":"Tracing \u0026 Profiling","icon":"chart-line","href":"/courses/linux-internals/tracing-profiling"}]},{"group":"Advanced Kernel","pages":[{"title":"Power \u0026 Thermal Management","sidebarTitle":"Power Management","icon":"bolt","href":"/courses/linux-internals/power-management"}]},{"group":"Interview Prep","pages":[{"title":"Hands-on Projects","sidebarTitle":"Projects","icon":"hammer","href":"/courses/linux-internals/projects"},{"title":"Real Interview Questions","sidebarTitle":"Interview Questions","icon":"clipboard-question","href":"/courses/linux-internals/interview-questions"}]}]},{"group":"Distributed Systems","icon":"sitemap","pages":[{"title":"Distributed Systems Mastery","sidebarTitle":"Course Overview","icon":"network-wired","href":"/courses/distributed-systems/overview"},{"group":"Core Foundations","pages":[{"title":"Foundations of Distributed Systems","sidebarTitle":"Foundations","icon":"book","href":"/courses/distributed-systems/foundations"},{"title":"Distributed Networking Fundamentals","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/distributed-systems/networking"},{"title":"Time \u0026 Clock Synchronization","sidebarTitle":"Clocks","icon":"clock","href":"/courses/distributed-systems/clocks"},{"title":"Consistency Models","sidebarTitle":"Consistency Models","icon":"scale-balanced","href":"/courses/distributed-systems/consistency-models"},{"title":"Distributed Snapshots","sidebarTitle":"Distributed Snapshots","icon":"camera","href":"/courses/distributed-systems/snapshots"}]},{"group":"Consensus \u0026 Replication","pages":[{"title":"Consensus Protocols","sidebarTitle":"Consensus","icon":"handshake","href":"/courses/distributed-systems/consensus"},{"title":"Raft Consensus Deep Dive","sidebarTitle":"Raft Deep Dive","icon":"star","href":"/courses/distributed-systems/raft-deep-dive"},{"title":"Paxos Consensus Deep Dive","sidebarTitle":"Paxos Deep Dive","icon":"scroll","href":"/courses/distributed-systems/paxos-deep-dive"},{"title":"Replication Strategies","sidebarTitle":"Replication","icon":"clone","href":"/courses/distributed-systems/replication"},{"title":"Distributed Coordination","sidebarTitle":"Coordination","icon":"people-group","href":"/courses/distributed-systems/coordination"},{"title":"Byzantine Fault Tolerance (BFT)","sidebarTitle":"Byzantine Fault Tolerance","icon":"shield-halved","href":"/courses/distributed-systems/bft"},{"title":"Gossip Protocols \u0026 Membership","sidebarTitle":"Gossip Protocols","icon":"comments","href":"/courses/distributed-systems/gossip"},{"title":"Formal Verification in Distributed Systems","sidebarTitle":"Formal Verification","icon":"shield-check","href":"/courses/distributed-systems/formal-verification"}]},{"group":"Data \u0026 Transactions","pages":[{"title":"Distributed Transactions","sidebarTitle":"Transactions","icon":"money-bill-transfer","href":"/courses/distributed-systems/transactions"},{"title":"Data Partitioning \u0026 Sharding","sidebarTitle":"Partitioning","icon":"table-cells","href":"/courses/distributed-systems/partitioning"},{"title":"Data Systems at Scale","sidebarTitle":"Data Systems","icon":"database","href":"/courses/distributed-systems/data-systems"}]},{"group":"Infrastructure \u0026 Messaging","pages":[{"title":"Distributed Messaging \u0026 Event Systems","sidebarTitle":"Messaging","icon":"envelope","href":"/courses/distributed-systems/messaging"},{"title":"Distributed Caching","sidebarTitle":"Caching","icon":"bolt","href":"/courses/distributed-systems/caching"},{"title":"Fault Tolerance Patterns","sidebarTitle":"Fault Tolerance","icon":"shield-heart","href":"/courses/distributed-systems/fault-tolerance"}]},{"group":"Production \u0026 Practice","pages":[{"title":"Production Excellence","sidebarTitle":"Production","icon":"shield-halved","href":"/courses/distributed-systems/production"},{"title":"Real-World Case Studies","sidebarTitle":"Case Studies","icon":"building","href":"/courses/distributed-systems/case-studies"},{"title":"Interview Practice Problems","sidebarTitle":"Interview Problems","icon":"comments","href":"/courses/distributed-systems/interview-problems"}]}]},{"group":"Networking Mastery","icon":"network-wired","pages":[{"title":"Networking Fundamentals","sidebarTitle":"1. Overview","icon":"globe","href":"/courses/networking-mastery/01-overview"},{"group":"Fundamentals","pages":[{"title":"Network Models","sidebarTitle":"2. OSI \u0026 TCP/IP","icon":"layer-group","href":"/courses/networking-mastery/02-models"},{"title":"Physical \u0026 Data Link Layers","sidebarTitle":"3. Physical \u0026 Data Link","icon":"ethernet","href":"/courses/networking-mastery/03-physical-datalink"},{"title":"Network Layer","sidebarTitle":"4. Network Layer","icon":"network-wired","href":"/courses/networking-mastery/04-network-layer"},{"title":"Transport Layer","sidebarTitle":"5. Transport Layer","icon":"truck-fast","href":"/courses/networking-mastery/05-transport-layer"},{"title":"Application Layer","sidebarTitle":"6. Application Layer","icon":"browser","href":"/courses/networking-mastery/06-application-layer"}]},{"group":"Security \u0026 Advanced Intro","pages":[{"title":"Network Security","sidebarTitle":"7. Security","icon":"shield-check","href":"/courses/networking-mastery/07-security"},{"title":"Advanced Networking Intro","sidebarTitle":"8. Advanced Topics","icon":"cloud","href":"/courses/networking-mastery/08-advanced"}]},{"group":"Deep Dives","pages":[{"title":"IP Addressing Deep Dive","sidebarTitle":"9. IP Addressing Mastery","icon":"map-location-dot","href":"/courses/networking-mastery/09-ip-addressing-deep-dive"},{"title":"NAT \u0026 PAT Deep Dive","sidebarTitle":"10. NAT Mastery","icon":"right-left","href":"/courses/networking-mastery/10-nat-deep-dive"},{"title":"Routing Deep Dive","sidebarTitle":"11. Routing Mastery","icon":"route","href":"/courses/networking-mastery/11-routing-deep-dive"},{"title":"DNS Deep Dive","sidebarTitle":"12. DNS Mastery","icon":"book","href":"/courses/networking-mastery/12-dns-deep-dive"},{"title":"Load Balancing \u0026 Proxies","sidebarTitle":"13. Load Balancing","icon":"scale-balanced","href":"/courses/networking-mastery/13-load-balancing"}]},{"group":"Operations \u0026 Security","pages":[{"title":"Network Troubleshooting","sidebarTitle":"14. Troubleshooting","icon":"wrench","href":"/courses/networking-mastery/14-troubleshooting"},{"title":"VPNs \u0026 Tunneling","sidebarTitle":"15. VPNs \u0026 Tunneling","icon":"tunnel","href":"/courses/networking-mastery/15-vpn-tunneling"},{"title":"Firewalls \u0026 Security Groups","sidebarTitle":"17. Firewalls","icon":"shield-halved","href":"/courses/networking-mastery/17-firewalls-security-groups"}]},{"group":"Modern Networking","pages":[{"title":"Container Networking","sidebarTitle":"18. Container Networking","icon":"cubes","href":"/courses/networking-mastery/18-container-networking"},{"title":"Real-World Networking Scenarios","sidebarTitle":"16. Network Scenarios","icon":"diagram-project","href":"/courses/networking-mastery/16-scenarios"}]}]},{"group":"Build Your Own X","icon":"hammer","pages":[{"title":"Build Your Own X","sidebarTitle":"Course Overview","icon":"hammer","href":"/courses/build-your-own-x/overview"},{"group":"Build Your Own Git","pages":[{"title":"Build Your Own Git","sidebarTitle":"Git Overview","icon":"code-branch","href":"/courses/build-your-own-x/git-overview"},{"title":"Chapter 1: Project Setup \u0026 Init","sidebarTitle":"1. Setup \u0026 Init","icon":"folder-open","href":"/courses/build-your-own-x/git-1-setup"},{"title":"Chapter 2: The Object Model","sidebarTitle":"2. Object Model","icon":"cube","href":"/courses/build-your-own-x/git-2-objects"},{"title":"Chapter 3: Staging \u0026 Index","sidebarTitle":"3. Staging \u0026 Index","icon":"layer-plus","href":"/courses/build-your-own-x/git-3-staging"},{"title":"Chapter 4: Commits \u0026 History","sidebarTitle":"4. Commits \u0026 History","icon":"code-commit","href":"/courses/build-your-own-x/git-4-commits"},{"title":"Chapter 5: Branches \u0026 Checkout","sidebarTitle":"5. Branches \u0026 Checkout","icon":"code-branch","href":"/courses/build-your-own-x/git-5-branches"}]},{"group":"Build Your Own Redis","pages":[{"title":"Build Your Own Redis","sidebarTitle":"Redis Overview","icon":"database","href":"/courses/build-your-own-x/redis-overview"},{"title":"Chapter 1: RESP Protocol","sidebarTitle":"1. RESP Protocol","icon":"network-wired","href":"/courses/build-your-own-x/redis-1-protocol"},{"title":"Chapter 2: TCP Server","sidebarTitle":"2. TCP Server","icon":"server","href":"/courses/build-your-own-x/redis-2-server"},{"title":"Chapter 3: Data Structures","sidebarTitle":"3. Data Structures","icon":"layer-group","href":"/courses/build-your-own-x/redis-3-data-structures"},{"title":"Chapter 4: Persistence","sidebarTitle":"4. Persistence","icon":"floppy-disk","href":"/courses/build-your-own-x/redis-4-persistence"},{"title":"Chapter 5: Pub/Sub","sidebarTitle":"5. Pub/Sub","icon":"tower-broadcast","href":"/courses/build-your-own-x/redis-5-pubsub"}]},{"group":"Build Your Own Docker","pages":[{"title":"Build Your Own Docker","sidebarTitle":"Docker Overview","icon":"docker","href":"/courses/build-your-own-x/docker-overview"},{"title":"Chapter 1: Linux Namespaces","sidebarTitle":"1. Namespaces","icon":"box","href":"/courses/build-your-own-x/docker-1-namespaces"},{"title":"Chapter 2: Control Groups (cgroups)","sidebarTitle":"2. Cgroups","icon":"gauge-high","href":"/courses/build-your-own-x/docker-2-cgroups"},{"title":"Chapter 3: Filesystem Isolation","sidebarTitle":"3. Filesystem","icon":"folder-tree","href":"/courses/build-your-own-x/docker-3-filesystem"},{"title":"Chapter 4: Container Networking","sidebarTitle":"4. Networking","icon":"network-wired","href":"/courses/build-your-own-x/docker-4-networking"},{"title":"Chapter 5: Container Images","sidebarTitle":"5. Images","icon":"box-archive","href":"/courses/build-your-own-x/docker-5-images"}]}]},{"group":"Go Lang Mastery","icon":"golang","pages":[{"group":"Fundamentals","pages":[{"title":"Introduction to Go","sidebarTitle":"1. Introduction","icon":"golang","href":"/courses/go-lang-mastery/01-introduction"},{"title":"Variables and Types","sidebarTitle":"2. Variables \u0026 Types","icon":"cube","href":"/courses/go-lang-mastery/02-variables-and-types"},{"title":"Control Structures","sidebarTitle":"3. Control Structures","icon":"code-branch","href":"/courses/go-lang-mastery/03-control-structures"},{"title":"Functions and Packages","sidebarTitle":"4. Functions \u0026 Packages","icon":"box-open","href":"/courses/go-lang-mastery/04-functions-and-packages"},{"title":"Arrays, Slices, and Maps","sidebarTitle":"5. Arrays, Slices \u0026 Maps","icon":"layer-group","href":"/courses/go-lang-mastery/05-arrays-slices-maps"},{"title":"Pointers and Structs","sidebarTitle":"6. Pointers \u0026 Structs","icon":"shapes","href":"/courses/go-lang-mastery/06-pointers-and-structs"},{"title":"Interfaces","sidebarTitle":"7. Interfaces","icon":"plug","href":"/courses/go-lang-mastery/07-interfaces"}]},{"group":"Core Concepts","pages":[{"title":"08. Concurrency","icon":"arrows-split-up-and-left","href":"/courses/go-lang-mastery/08-concurrency"},{"title":"09. Error Handling","icon":"triangle-exclamation","href":"/courses/go-lang-mastery/09-error-handling"},{"title":"10. Testing and Benchmarking","icon":"vial","href":"/courses/go-lang-mastery/10-testing-and-benchmarking"},{"title":"Generics","sidebarTitle":"11. Generics","icon":"cube","href":"/courses/go-lang-mastery/11-generics"},{"title":"Advanced Concurrency Patterns","sidebarTitle":"12. Advanced Concurrency","icon":"diagram-project","href":"/courses/go-lang-mastery/12-advanced-concurrency"},{"title":"Context Package","sidebarTitle":"13. Context","icon":"clock","href":"/courses/go-lang-mastery/13-context"}]},{"group":"Production Go","pages":[{"title":"HTTP \u0026 Web Development","sidebarTitle":"14. HTTP \u0026 Web","icon":"globe","href":"/courses/go-lang-mastery/14-http-web"},{"title":"Database Operations","sidebarTitle":"15. Databases","icon":"database","href":"/courses/go-lang-mastery/15-databases"},{"title":"Microservices \u0026 gRPC","sidebarTitle":"16. Microservices \u0026 gRPC","icon":"sitemap","href":"/courses/go-lang-mastery/16-microservices-grpc"},{"title":"Performance Optimization","sidebarTitle":"17. Performance","icon":"gauge-high","href":"/courses/go-lang-mastery/17-performance"},{"title":"Production Best Practices","sidebarTitle":"18. Production","icon":"rocket","href":"/courses/go-lang-mastery/18-production"}]},{"group":"Advanced Topics","pages":[{"title":"Design Patterns in Go","sidebarTitle":"19. Design Patterns","icon":"puzzle-piece","href":"/courses/go-lang-mastery/19-design-patterns"},{"title":"Interview Preparation","sidebarTitle":"20. Interview Prep","icon":"user-tie","href":"/courses/go-lang-mastery/20-interview-prep"}]}]},{"group":"C Programming","icon":"memory","pages":[{"title":"Advanced C Programming for Systems Development","sidebarTitle":"Course Overview","icon":"microchip","href":"/courses/c-programming/overview"},{"group":"Foundations","pages":[{"title":"C Syntax Speed Run","sidebarTitle":"Syntax Speed Run","icon":"forward-fast","href":"/courses/c-programming/syntax-speedrun"},{"title":"Build Systems \u0026 Toolchain","sidebarTitle":"Build \u0026 Toolchain","icon":"hammer","href":"/courses/c-programming/toolchain"},{"title":"Debugging Fundamentals","sidebarTitle":"Debugging","icon":"bug","href":"/courses/c-programming/debugging"}]},{"group":"Memory Mastery","pages":[{"title":"Pointers Deep Dive","sidebarTitle":"Pointers","icon":"arrow-pointer","href":"/courses/c-programming/pointers"},{"title":"Memory Layout \u0026 Segments","sidebarTitle":"Memory Layout","icon":"layer-group","href":"/courses/c-programming/memory-layout"},{"title":"Dynamic Memory Management","sidebarTitle":"Dynamic Memory","icon":"memory","href":"/courses/c-programming/dynamic-memory"},{"title":"Preprocessor Mastery","sidebarTitle":"Preprocessor","icon":"code","href":"/courses/c-programming/preprocessor"},{"title":"Data Structures in C","sidebarTitle":"Data Structures","icon":"sitemap","href":"/courses/c-programming/data-structures"},{"title":"Undefined Behavior","sidebarTitle":"Undefined Behavior","icon":"skull","href":"/courses/c-programming/undefined-behavior"}]},{"group":"Systems Programming","pages":[{"title":"System Calls \u0026 POSIX","sidebarTitle":"System Calls","icon":"terminal","href":"/courses/c-programming/system-calls"},{"title":"Concurrency \u0026 Threading","sidebarTitle":"Concurrency","icon":"arrows-split-up-and-left","href":"/courses/c-programming/concurrency"},{"title":"Network Programming","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/c-programming/networking"}]},{"group":"Projects","pages":[{"title":"Project: Build a Shell","sidebarTitle":"Project: Shell","icon":"terminal","href":"/courses/c-programming/project-shell"},{"title":"Project: Key-Value Store","sidebarTitle":"Project: KV Store","icon":"database","href":"/courses/c-programming/project-database"},{"title":"Project: Memory Allocator","sidebarTitle":"Project: malloc","icon":"memory","href":"/courses/c-programming/project-allocator"},{"title":"Project: HTTP Server","sidebarTitle":"Project: HTTP Server","icon":"server","href":"/courses/c-programming/project-http-server"}]},{"group":"Advanced Topics","pages":[{"title":"Performance Optimization","sidebarTitle":"Performance","icon":"gauge-high","href":"/courses/c-programming/performance"},{"title":"Embedded Systems","sidebarTitle":"Embedded","icon":"microchip","href":"/courses/c-programming/embedded"},{"title":"Linux Kernel Modules","sidebarTitle":"Kernel Modules","icon":"linux","href":"/courses/c-programming/kernel-modules"}]}]},{"group":"Classic Research Papers","icon":"file-lines","pages":[{"group":"Google File System (GFS)","icon":"folder","pages":[{"title":"Google File System","sidebarTitle":"GFS Overview","icon":"database","href":"/courses/engineering-papers/gfs-overview"},{"title":"Introduction and Motivation","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/chapter-1-introduction"},{"title":"Architecture Overview","sidebarTitle":"Chapter 2: Architecture","icon":"diagram-project","href":"/courses/engineering-papers/chapter-2-architecture"},{"title":"Master Operations","sidebarTitle":"Chapter 3: Master Operations","icon":"crown","href":"/courses/engineering-papers/chapter-3-master-operations"},{"title":"Chunkservers and Data Flow","sidebarTitle":"Chapter 4: Chunkservers \u0026 Data Flow","icon":"server","href":"/courses/engineering-papers/chapter-4-chunkservers-dataflow"},{"title":"Consistency Model","sidebarTitle":"Chapter 5: Consistency Model","icon":"scale-balanced","href":"/courses/engineering-papers/chapter-5-consistency-model"},{"title":"Fault Tolerance","sidebarTitle":"Chapter 6: Fault Tolerance","icon":"shield-halved","href":"/courses/engineering-papers/chapter-6-fault-tolerance"},{"title":"Performance and Optimizations","sidebarTitle":"Chapter 7: Performance","icon":"gauge-high","href":"/courses/engineering-papers/chapter-7-performance"},{"title":"Impact and Evolution","sidebarTitle":"Chapter 8: Impact \u0026 Evolution","icon":"rocket","href":"/courses/engineering-papers/chapter-8-impact-evolution"}]},{"group":"Apache Hadoop","icon":"elephant","pages":[{"title":"Apache Hadoop","sidebarTitle":"Hadoop Overview","icon":"database","href":"/courses/engineering-papers/hadoop-overview"},{"title":"Introduction and Origins","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/hadoop-chapter-1-introduction"},{"title":"HDFS Architecture","sidebarTitle":"Chapter 2: HDFS","icon":"hard-drive","href":"/courses/engineering-papers/hadoop-chapter-2-hdfs-architecture"},{"title":"MapReduce Framework","sidebarTitle":"Chapter 3: MapReduce","icon":"diagram-project","href":"/courses/engineering-papers/hadoop-chapter-3-mapreduce-framework"},{"title":"YARN: Resource Management","sidebarTitle":"Chapter 4: YARN","icon":"layer-group","href":"/courses/engineering-papers/hadoop-chapter-4-yarn"},{"title":"Hadoop Ecosystem","sidebarTitle":"Chapter 5: Ecosystem","icon":"puzzle-piece","href":"/courses/engineering-papers/hadoop-chapter-5-ecosystem"},{"title":"Fault Tolerance and Reliability","sidebarTitle":"Chapter 6: Fault Tolerance","icon":"shield-check","href":"/courses/engineering-papers/hadoop-chapter-6-fault-tolerance"},{"title":"Performance Tuning and Optimization","sidebarTitle":"Chapter 7: Performance","icon":"gauge-high","href":"/courses/engineering-papers/hadoop-chapter-7-performance"},{"title":"Hadoop in Production","sidebarTitle":"Chapter 8: Production","icon":"server","href":"/courses/engineering-papers/hadoop-chapter-8-production"}]},{"group":"Amazon DynamoDB","icon":"database","pages":[{"title":"Amazon DynamoDB","sidebarTitle":"DynamoDB Overview","icon":"database","href":"/courses/engineering-papers/dynamodb-overview"},{"title":"Introduction and Origins","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/dynamodb-chapter-1-introduction"},{"title":"Architecture and Partitioning","sidebarTitle":"Chapter 2: Architecture","icon":"sitemap","href":"/courses/engineering-papers/dynamodb-chapter-2-architecture"},{"title":"Data Modeling in DynamoDB","sidebarTitle":"Chapter 3: Data Model","icon":"database","href":"/courses/engineering-papers/dynamodb-chapter-3-data-model"},{"title":"Consistency Models in DynamoDB","sidebarTitle":"Chapter 4: Consistency","icon":"scale-balanced","href":"/courses/engineering-papers/dynamodb-chapter-4-consistency"},{"title":"Performance Optimization in DynamoDB","sidebarTitle":"Chapter 5: Performance","icon":"gauge-high","href":"/courses/engineering-papers/dynamodb-chapter-5-performance"},{"title":"Advanced DynamoDB Features","sidebarTitle":"Chapter 6: Advanced Features","icon":"wand-magic-sparkles","href":"/courses/engineering-papers/dynamodb-chapter-6-advanced-features"},{"title":"Fault Tolerance and Reliability","sidebarTitle":"Chapter 7: Fault Tolerance","icon":"shield-check","href":"/courses/engineering-papers/dynamodb-chapter-7-fault-tolerance"},{"title":"DynamoDB in Production","sidebarTitle":"Chapter 8: Production","icon":"server","href":"/courses/engineering-papers/dynamodb-chapter-8-production"}]}]},{"group":"Distributed System Tools","icon":"server","pages":[{"group":"Apache Hadoop","icon":"elephant","pages":[{"title":"Apache Hadoop Mastery","sidebarTitle":"Hadoop Overview","icon":"elephant","href":"/distributed-systems-tools/hadoop-overview"},{"title":"Introduction \u0026 Foundational Papers","sidebarTitle":"1. Introduction \u0026 Papers","icon":"book-open","href":"/distributed-systems-tools/hadoop-introduction"},{"title":"HDFS Architecture \u0026 Internals","sidebarTitle":"2. HDFS Architecture","icon":"database","href":"/distributed-systems-tools/hadoop-hdfs"},{"title":"Advanced HDFS: Federation, HA \u0026 Erasure Coding","sidebarTitle":"2b. Advanced HDFS","icon":"layer-group","href":"/distributed-systems-tools/hadoop-hdfs-advanced"},{"title":"MapReduce Programming Model","sidebarTitle":"3. MapReduce Programming","icon":"code","href":"/distributed-systems-tools/hadoop-mapreduce"},{"title":"YARN Resource Management","sidebarTitle":"4. YARN Architecture","icon":"layer-group","href":"/distributed-systems-tools/hadoop-yarn"},{"title":"Hadoop Ecosystem \u0026 Integration","sidebarTitle":"5. Ecosystem Tools","icon":"puzzle-piece","href":"/distributed-systems-tools/hadoop-ecosystem"},{"title":"Data Processing Patterns \u0026 Best Practices","sidebarTitle":"6. Processing Patterns","icon":"diagram-project","href":"/distributed-systems-tools/hadoop-patterns"},{"title":"Production Deployment \u0026 Operations","sidebarTitle":"7. Production Operations","icon":"server","href":"/distributed-systems-tools/hadoop-production"},{"title":"Capstone Project: End-to-End Data Pipeline","sidebarTitle":"8. Capstone Project","icon":"trophy","href":"/distributed-systems-tools/hadoop-capstone"}]},{"group":"Apache Spark","icon":"bolt","pages":[{"title":"Apache Spark Mastery","sidebarTitle":"Spark Overview","icon":"bolt","href":"/distributed-systems-tools/spark-overview"},{"title":"Demystifying the RDD Paper: Spark's Foundation","sidebarTitle":"1. RDD Paper Deep-Dive","icon":"book-open","href":"/distributed-systems-tools/spark-introduction"},{"title":"Demystifying RDD Programming: From Theory to Production","sidebarTitle":"2. RDD Programming","icon":"bolt","href":"/distributed-systems-tools/spark-rdd"},{"title":"Spark SQL \u0026 DataFrames","sidebarTitle":"3. Spark SQL","icon":"table","href":"/distributed-systems-tools/spark-sql"},{"title":"Structured Streaming","sidebarTitle":"4. Streaming","icon":"bolt","href":"/distributed-systems-tools/spark-streaming"},{"title":"MLlib for Machine Learning","sidebarTitle":"5. MLlib","icon":"bolt","href":"/distributed-systems-tools/spark-mllib"},{"title":"Performance Tuning","sidebarTitle":"6. Performance Tuning","icon":"bolt","href":"/distributed-systems-tools/spark-tuning"},{"title":"Cluster Deployment","sidebarTitle":"7. Operations","icon":"bolt","href":"/distributed-systems-tools/spark-operations"},{"title":"Advanced Topics","sidebarTitle":"8. Advanced","icon":"bolt","href":"/distributed-systems-tools/spark-advanced"},{"title":"Capstone Project","sidebarTitle":"9. Capstone","icon":"bolt","href":"/distributed-systems-tools/spark-capstone"}]},{"group":"Apache Flink","icon":"wave-pulse","pages":[{"title":"Apache Flink Mastery","sidebarTitle":"Flink Overview","icon":"wave-pulse","href":"/distributed-systems-tools/flink-overview"},{"title":"Demystifying the Dataflow Model: True Stream Processing with Flink","sidebarTitle":"1. Introduction","icon":"wave-pulse","href":"/distributed-systems-tools/flink-introduction"},{"title":"DataStream API","sidebarTitle":"2. DataStream API","icon":"wave-pulse","href":"/distributed-systems-tools/flink-datastream"},{"title":"Event Time \u0026 Watermarks","sidebarTitle":"3. Event Time","icon":"wave-pulse","href":"/distributed-systems-tools/flink-eventtime"},{"title":"Windows \u0026 Time Operations","sidebarTitle":"4. Windows","icon":"wave-pulse","href":"/distributed-systems-tools/flink-windows"},{"title":"Demystifying Stateful Stream Processing: Flink's State Management Deep Dive","sidebarTitle":"5. State \u0026 Timers","icon":"database","href":"/distributed-systems-tools/flink-state"},{"title":"Declarative Stream Processing: Flink SQL \u0026 Table API Deep Dive","sidebarTitle":"6. SQL \u0026 Table API","icon":"table","href":"/distributed-systems-tools/flink-sql"},{"title":"Complex Event Processing: Pattern Detection at Scale with Flink CEP","sidebarTitle":"7. Complex Event Processing","icon":"sitemap","href":"/distributed-systems-tools/flink-cep"},{"title":"Production Flink: Deployment, Operations \u0026 Monitoring at Scale","sidebarTitle":"8. Production Operations","icon":"server","href":"/distributed-systems-tools/flink-operations"},{"title":"Capstone: Production-Grade Real-Time Fraud Detection System","sidebarTitle":"9. Capstone Project","icon":"diagram-project","href":"/distributed-systems-tools/flink-capstone"}]},{"group":"Apache Beam","icon":"diagram-project","pages":[{"title":"Apache Beam Mastery","sidebarTitle":"Beam Overview","icon":"diagram-project","href":"/distributed-systems-tools/beam-overview"},{"title":"Demystifying the Dataflow Model: Write Once, Run Anywhere with Apache Beam","sidebarTitle":"1. Introduction","icon":"diagram-project","href":"/distributed-systems-tools/beam-introduction"},{"title":"Core Programming Model","sidebarTitle":"2. Core Model","icon":"diagram-project","href":"/distributed-systems-tools/beam-core"},{"title":"Windowing \u0026 Time","sidebarTitle":"3. Windowing","icon":"diagram-project","href":"/distributed-systems-tools/beam-windowing"},{"title":"Triggers \u0026 Watermarks","sidebarTitle":"4. Triggers","icon":"diagram-project","href":"/distributed-systems-tools/beam-triggers"},{"title":"State \u0026 Timers","sidebarTitle":"5. State","icon":"diagram-project","href":"/distributed-systems-tools/beam-state"},{"title":"Beam I/O Connectors","sidebarTitle":"6. I/O","icon":"diagram-project","href":"/distributed-systems-tools/beam-io"},{"title":"Beam SQL","sidebarTitle":"7. SQL","icon":"diagram-project","href":"/distributed-systems-tools/beam-sql"},{"title":"Multi-Runner Execution","sidebarTitle":"8. Runners","icon":"diagram-project","href":"/distributed-systems-tools/beam-runners"},{"title":"Capstone Project","sidebarTitle":"9. Capstone","icon":"diagram-project","href":"/distributed-systems-tools/beam-capstone"}]},{"group":"Apache Airflow","icon":"wind","pages":[{"title":"What is Apache Airflow?","sidebarTitle":"Overview","icon":"wind","href":"/distributed-systems-tools/airflow-overview"},{"title":"Airflow Core Concepts","sidebarTitle":"Core Concepts","icon":"diagram-project","href":"/distributed-systems-tools/airflow-core-concepts"},{"title":"Airflow Operators","sidebarTitle":"Operators","icon":"puzzle-piece","href":"/distributed-systems-tools/airflow-operators"},{"title":"Airflow Scheduling","sidebarTitle":"Scheduling","icon":"clock","href":"/distributed-systems-tools/airflow-scheduling"}]},{"group":"Apache Cassandra","icon":"database","pages":[{"title":"Apache Cassandra Mastery","sidebarTitle":"Overview","icon":"database","href":"/distributed-systems-tools/cassandra-overview"},{"title":"The Cassandra Paper \u0026 Core Architecture","sidebarTitle":"1. Introduction \u0026 Paper","icon":"book-open","href":"/distributed-systems-tools/cassandra-paper"},{"title":"Cassandra Architecture \u0026 Storage Internals","sidebarTitle":"2. Architecture \u0026 Internals","icon":"diagram-project","href":"/distributed-systems-tools/cassandra-architecture-internals"},{"title":"Query-Driven Data Modeling in Cassandra","sidebarTitle":"3. Data Modeling","icon":"table","href":"/distributed-systems-tools/cassandra-data-modeling"},{"title":"Cassandra Read \u0026 Write Path Internals","sidebarTitle":"4. Read/Write Internals","icon":"route","href":"/distributed-systems-tools/cassandra-read-write-internals"},{"title":"Cluster Operations \u0026 Multi-DC Replication","sidebarTitle":"5. Cluster Operations","icon":"server","href":"/distributed-systems-tools/cassandra-cluster-operations"},{"title":"Performance Tuning \u0026 Production Operations","sidebarTitle":"6. Performance \u0026 Production","icon":"gauge-high","href":"/distributed-systems-tools/cassandra-performance-production"},{"title":"Capstone: Building a Real-Time Analytics Platform","sidebarTitle":"7. Capstone Project","icon":"rocket","href":"/distributed-systems-tools/cassandra-capstone"}]},{"group":"Neo4j Graph Database","icon":"diagram-project","pages":[{"title":"Neo4j Graph Database Mastery","sidebarTitle":"Overview","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-overview"},{"title":"The Graph Database Story: From Theory to Neo4j","sidebarTitle":"2. Foundational Papers","icon":"scroll","href":"/distributed-systems-tools/neo4j-paper"},{"title":"Neo4j Architecture \u0026 Native Graph Storage","sidebarTitle":"3. Architecture \u0026 Storage","icon":"database","href":"/distributed-systems-tools/neo4j-architecture-storage"},{"title":"Cypher Query Language Mastery","sidebarTitle":"4. Cypher Mastery","icon":"code","href":"/distributed-systems-tools/neo4j-cypher-mastery"},{"title":"Graph Data Modeling Best Practices","sidebarTitle":"5. Data Modeling","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-data-modeling"},{"title":"Graph Algorithms at Scale","sidebarTitle":"6. Graph Algorithms","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-graph-algorithms"},{"title":"Production Deployment \u0026 Operations","sidebarTitle":"7. Production \u0026 Ops","icon":"server","href":"/distributed-systems-tools/neo4j-production-deployment"},{"title":"Capstone: Building a Knowledge Graph Platform","sidebarTitle":"8. Capstone Project","icon":"rocket","href":"/distributed-systems-tools/neo4j-capstone"}]}]}]}]},"basePath":"","title":"Memory-Efficient Training - Dev Weekends","deploymentId":"dpl_FZMKUpaaKiVETbfqhoUe4sPFDTmY","buildId":"dpl_FZMKUpaaKiVETbfqhoUe4sPFDTmY","dynamics":["07c022d89b","a74be98ba7"],"chunks":["Frame-DBAS9Liu.js","DynamicMarkdown-C4w9kavL.js","zoom-image-lightbox-B9O1U98p.js","lightbox-handle-r17bVUz3.js"]}