Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers
Attention Mechanism
The Bottleneck Problem
In the previous chapters, we built sequence-to-sequence models using encoder-decoder LSTMs. But there’s a fundamental problem:The entire source sequence is compressed into a single fixed-size vector. This is like asking someone to summarize a 300-page novel in a single tweet, and then expecting another person to reconstruct the entire plot from that tweet.
import torchimport torch.nn as nnimport torch.nn.functional as Fimport numpy as npimport matplotlib.pyplot as pltdef visualize_bottleneck(): """ The Encoder-Decoder Bottleneck Problem Source: "The quick brown fox jumps over the lazy dog" ↓ LSTM Encoder → Single vector (e.g., 512 dimensions) ↓ LSTM Decoder → "Le rapide renard brun saute par-dessus le chien paresseux" Problem: ALL information about a 9-word sentence must fit in 512 numbers! For longer sentences, this becomes impossible. """ print("The Bottleneck Problem:") print("=" * 60) print() print("Source sentence (English):") print(" 'The quick brown fox jumps over the lazy dog'") print() print("Compressed to: vector of shape (512,)") print() print("Then decoded to (French):") print(" 'Le rapide renard brun saute par-dessus le chien paresseux'") print() print("❌ Problems:") print(" - Information loss for long sequences") print(" - Decoder must 'guess' what encoder meant") print(" - Early encoder states forgotten by final state")visualize_bottleneck()
Evidence of the Problem
def show_translation_quality_vs_length(): """ Real observation: Translation quality degrades with sentence length when using fixed-size encoding. """ # Approximate BLEU scores from research papers lengths = [10, 20, 30, 40, 50, 60] bleu_without_attention = [35, 32, 28, 22, 16, 12] bleu_with_attention = [38, 37, 36, 35, 34, 33] plt.figure(figsize=(10, 5)) plt.plot(lengths, bleu_without_attention, 'o-', label='Without Attention', linewidth=2) plt.plot(lengths, bleu_with_attention, 's-', label='With Attention', linewidth=2) plt.xlabel('Source Sentence Length') plt.ylabel('BLEU Score') plt.title('Translation Quality vs Sentence Length') plt.legend() plt.grid(True, alpha=0.3) plt.show() print("Key insight: Attention maintains quality even for long sentences!")# show_translation_quality_vs_length()
The Attention Solution
Intuition: Looking Back at the Source
Instead of forcing the decoder to use only the final encoder state, let it look at all encoder states and focus on relevant ones:When translating “dog” to “chien”:
Look at all encoder states
Focus attention on the state corresponding to “dog”
Use that information to generate “chien”
Key Insight: Attention computes a weighted combination of all encoder states, where weights indicate relevance to the current decoding step.
def attention_intuition(): """ Attention Intuition: The Query-Key-Value Framework Imagine you're at a library: - Query: "I need information about neural networks" - Keys: Book titles on the shelves - Values: The actual content of each book Process: 1. Compare your query with all keys (book titles) 2. Compute relevance scores 3. Retrieve values (content) weighted by relevance """ print("Attention as Information Retrieval:") print("=" * 60) print() print("Query (what we're looking for):") print(" → Current decoder state: 'trying to translate word X'") print() print("Keys (what we're comparing against):") print(" → All encoder hidden states") print() print("Values (what we retrieve):") print(" → Same encoder hidden states (or transformed versions)") print() print("Output:") print(" → Weighted sum of values, weights from query-key similarity")attention_intuition()
Attention Mechanisms in Detail
Dot-Product Attention
The simplest form of attention:Attention(Q,K,V)=softmax(dkQKT)VWhy divide by dk? This is not an arbitrary choice — it is essential for numerical stability. When dk is large (e.g., 512), each dot product q⋅k is the sum of 512 terms. If entries of q and k have unit variance, the dot product has variance dk (by the central limit theorem, the sum of 512 independent unit-variance terms has variance 512). So the raw scores have standard deviation dk≈22.6. Softmax applied to values this large pushes almost all the probability mass onto a single key — the attention becomes a hard argmax, and its gradient effectively vanishes.Dividing by dk rescales the scores to unit variance, keeping the softmax in its “useful” regime where gradients flow to multiple keys. Without this scaling, training larger models would be dramatically harder. This is one of those small details that separates a paper implementation from a working one.
def scaled_dot_product_attention(query, key, value, mask=None): """ Scaled Dot-Product Attention -- the core computation inside every transformer. Args: query: (batch, num_queries, d_k) -- "what am I looking for?" key: (batch, num_keys, d_k) -- "what do I contain?" value: (batch, num_keys, d_v) -- "what information do I provide?" mask: Optional mask to prevent attention to certain positions Returns: output: (batch, num_queries, d_v) -- weighted combination of values attention_weights: (batch, num_queries, num_keys) -- where we looked """ d_k = query.size(-1) # Step 1: Compute attention scores via dot product # High dot product = query and key point in similar directions = "relevant" # (batch, num_queries, d_k) @ (batch, d_k, num_keys) -> (batch, num_queries, num_keys) scores = torch.matmul(query, key.transpose(-2, -1)) # Step 2: Scale by sqrt(d_k) -- this is critical and often misunderstood # Without scaling, dot products grow proportionally to d_k, pushing softmax # into saturation where gradients are near-zero. Dividing by sqrt(d_k) # keeps the variance of scores at ~1 regardless of dimension. scores = scores / np.sqrt(d_k) # Apply mask if provided if mask is not None: scores = scores.masked_fill(mask == 0, float('-inf')) # Softmax to get attention weights attention_weights = F.softmax(scores, dim=-1) # Apply attention weights to values output = torch.matmul(attention_weights, value) return output, attention_weights# Examplebatch_size = 2num_queries = 4 # e.g., decoder positionsnum_keys = 6 # e.g., encoder positionsd_k = 64 # dimension of queries and keysd_v = 64 # dimension of valuesquery = torch.randn(batch_size, num_queries, d_k)key = torch.randn(batch_size, num_keys, d_k)value = torch.randn(batch_size, num_keys, d_v)output, weights = scaled_dot_product_attention(query, key, value)print(f"Query shape: {query.shape}")print(f"Key shape: {key.shape}")print(f"Value shape: {value.shape}")print(f"Output shape: {output.shape}")print(f"Weights shape: {weights.shape}")print(f"\nAttention weights sum to 1: {weights.sum(dim=-1)}")
Visualizing Attention Weights
def visualize_attention(): """Visualize what attention 'looks at'.""" # Simulated attention weights for translation source = ["The", "cat", "sat", "on", "the", "mat", "."] target = ["Le", "chat", "était", "assis", "sur", "le", "tapis", "."] # Attention matrix (which source words each target word attends to) attention = np.array([ [0.8, 0.1, 0.0, 0.0, 0.1, 0.0, 0.0], # Le → The [0.1, 0.8, 0.0, 0.0, 0.0, 0.1, 0.0], # chat → cat [0.0, 0.1, 0.7, 0.1, 0.0, 0.0, 0.1], # était → sat [0.0, 0.1, 0.7, 0.1, 0.0, 0.0, 0.1], # assis → sat [0.0, 0.0, 0.0, 0.8, 0.1, 0.0, 0.1], # sur → on [0.1, 0.0, 0.0, 0.0, 0.8, 0.0, 0.1], # le → the [0.0, 0.1, 0.0, 0.0, 0.0, 0.8, 0.1], # tapis → mat [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0], # . → . ]) fig, ax = plt.subplots(figsize=(10, 8)) im = ax.imshow(attention, cmap='Blues') ax.set_xticks(np.arange(len(source))) ax.set_yticks(np.arange(len(target))) ax.set_xticklabels(source, fontsize=12) ax.set_yticklabels(target, fontsize=12) ax.set_xlabel("Source (English)", fontsize=14) ax.set_ylabel("Target (French)", fontsize=14) ax.set_title("Attention Weights in Translation", fontsize=16) # Add attention values in cells for i in range(len(target)): for j in range(len(source)): text = ax.text(j, i, f'{attention[i, j]:.1f}', ha='center', va='center', color='black' if attention[i,j] < 0.5 else 'white') plt.colorbar(im) plt.tight_layout() plt.show()visualize_attention()
Types of Attention
Additive (Bahdanau) Attention
The original attention mechanism from the 2014 paper:eij=vTtanh(W1hi+W2sj)αij=∑kexp(ekj)exp(eij)
class AdditiveAttention(nn.Module): """ Bahdanau (Additive) Attention (2014). Uses a learned alignment model (a small neural network) to compute attention scores. More expressive but slower than dot-product attention because it requires an extra matrix multiplication and tanh activation. Why "additive"? The score adds two projections (W1*encoder + W2*decoder) then passes through tanh. This allows non-linear interactions between query and key, unlike the dot product which is strictly linear. """ def __init__(self, encoder_dim, decoder_dim, attention_dim): super().__init__() # Project encoder states self.W1 = nn.Linear(encoder_dim, attention_dim, bias=False) # Project decoder state self.W2 = nn.Linear(decoder_dim, attention_dim, bias=False) # Compute scalar attention score self.v = nn.Linear(attention_dim, 1, bias=False) def forward(self, encoder_outputs, decoder_hidden): """ Args: encoder_outputs: (batch, src_len, encoder_dim) decoder_hidden: (batch, decoder_dim) Returns: context: (batch, encoder_dim) attention_weights: (batch, src_len) """ # Project encoder outputs encoder_proj = self.W1(encoder_outputs) # (batch, src_len, attention_dim) # Project decoder hidden (add dimension for broadcasting) decoder_proj = self.W2(decoder_hidden).unsqueeze(1) # (batch, 1, attention_dim) # Compute alignment scores scores = self.v(torch.tanh(encoder_proj + decoder_proj)) # (batch, src_len, 1) scores = scores.squeeze(-1) # (batch, src_len) # Softmax to get attention weights attention_weights = F.softmax(scores, dim=-1) # Compute context vector context = torch.bmm(attention_weights.unsqueeze(1), encoder_outputs) context = context.squeeze(1) # (batch, encoder_dim) return context, attention_weights# Testattention = AdditiveAttention(encoder_dim=512, decoder_dim=512, attention_dim=256)encoder_outputs = torch.randn(8, 20, 512) # 8 sequences, 20 timestepsdecoder_hidden = torch.randn(8, 512)context, weights = attention(encoder_outputs, decoder_hidden)print(f"Encoder outputs: {encoder_outputs.shape}")print(f"Decoder hidden: {decoder_hidden.shape}")print(f"Context vector: {context.shape}")print(f"Attention weights: {weights.shape}")
Self-attention allows each position in a sequence to attend to all other positions:
class SelfAttention(nn.Module): """ Self-Attention: Sequence attends to itself. Each position can gather information from all other positions, allowing for direct modeling of dependencies regardless of distance. """ def __init__(self, d_model): super().__init__() self.d_model = d_model # Project to query, key, value self.W_q = nn.Linear(d_model, d_model) self.W_k = nn.Linear(d_model, d_model) self.W_v = nn.Linear(d_model, d_model) def forward(self, x, mask=None): """ Args: x: Input sequence (batch, seq_len, d_model) mask: Optional attention mask Returns: output: (batch, seq_len, d_model) attention_weights: (batch, seq_len, seq_len) """ # Project to Q, K, V Q = self.W_q(x) K = self.W_k(x) V = self.W_v(x) # Scaled dot-product attention d_k = Q.size(-1) scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(d_k) if mask is not None: scores = scores.masked_fill(mask == 0, float('-inf')) attention_weights = F.softmax(scores, dim=-1) output = torch.matmul(attention_weights, V) return output, attention_weights# Example: Self-attention on a sentenceself_attn = SelfAttention(d_model=256)# 4 sentences, 10 words each, 256-dim embeddingsx = torch.randn(4, 10, 256)output, weights = self_attn(x)print(f"Input: {x.shape}")print(f"Output: {output.shape}")print(f"Attention weights: {weights.shape}")print(f"\nEach word attends to all words (including itself)")
Why Self-Attention Matters
def compare_rnn_vs_self_attention(): """ Self-attention vs RNN for capturing dependencies. """ print("Self-Attention Advantages over RNNs:") print("=" * 60) print() print("1. PATH LENGTH for long-range dependencies:") print(" RNN: O(n) - information must flow through n steps") print(" Attn: O(1) - direct connection between any two positions") print() print("2. PARALLELIZATION:") print(" RNN: Sequential - must process one step at a time") print(" Attn: Fully parallel - all positions computed simultaneously") print() print("3. INTERPRETABILITY:") print(" RNN: Hidden state is opaque") print(" Attn: Attention weights show what the model focuses on") print() # Computational complexity comparison n = 1000 # sequence length d = 512 # hidden dimension rnn_complexity = n * d * d # O(n·d²) sequential steps attn_complexity = n * n * d # O(n²·d) but parallelizable print(f"For sequence length {n}, hidden dim {d}:") print(f" RNN: ~{rnn_complexity:,} ops (sequential)") print(f" Attn: ~{attn_complexity:,} ops (parallel)")compare_rnn_vs_self_attention()
Multi-Head Attention
Multiple Attention “Perspectives”
Instead of one attention function, use multiple “heads” that each learn different relationships:MultiHead(Q,K,V)=Concat(head1,...,headh)WOwhere headi=Attention(QWiQ,KWiK,VWiV)Why multiple heads instead of one big attention? A single attention head can only compute one weighted average per query position — it assigns a single scalar relevance to each key. But language has multiple simultaneous relationships: “it” relates to “animal” syntactically (coreference), to “tired” semantically (attribute), and to “cross” structurally (subject-verb). Multiple heads let the model maintain all these relationships simultaneously, each head specializing in a different type of dependency.The mathematical cost is negligible: if dmodel=512 and you use 8 heads, each head operates in a dk=64 dimensional subspace. The total computation is the same as a single head with dk=512, but you get 8 independent attention patterns instead of 1. The output projection WO then learns how to combine these perspectives. This is one of the best “free lunches” in deep learning architecture design.
class MultiHeadAttention(nn.Module): """ Multi-Head Attention. Multiple attention heads allow the model to jointly attend to information from different representation subspaces. Analogy: Imagine a team of analysts reading the same document. One analyst focuses on who did what (subject-verb relationships). Another tracks pronoun references (coreference). A third watches for temporal ordering (before/after). Each analyst sees the same text but extracts different relationships. For example, in "The animal didn't cross the street because it was too tired": - One head might focus on "it" -> "animal" (coreference resolution) - Another head might focus on "tired" -> "animal" (attribute binding) - A third might focus on "cross" -> "street" (action-object pairing) The multi-head output combines all these perspectives. """ def __init__(self, d_model, num_heads, dropout=0.1): super().__init__() assert d_model % num_heads == 0, "d_model must be divisible by num_heads" self.d_model = d_model self.num_heads = num_heads self.d_k = d_model // num_heads # Linear projections for Q, K, V self.W_q = nn.Linear(d_model, d_model) self.W_k = nn.Linear(d_model, d_model) self.W_v = nn.Linear(d_model, d_model) # Output projection self.W_o = nn.Linear(d_model, d_model) self.dropout = nn.Dropout(dropout) def forward(self, query, key, value, mask=None): """ Args: query: (batch, seq_len_q, d_model) key: (batch, seq_len_k, d_model) value: (batch, seq_len_v, d_model) mask: Optional mask (batch, 1, seq_len_q, seq_len_k) or (batch, 1, 1, seq_len_k) Returns: output: (batch, seq_len_q, d_model) attention_weights: (batch, num_heads, seq_len_q, seq_len_k) """ batch_size = query.size(0) # 1. Linear projections Q = self.W_q(query) K = self.W_k(key) V = self.W_v(value) # 2. Split into multiple heads # (batch, seq_len, d_model) → (batch, num_heads, seq_len, d_k) Q = Q.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2) K = K.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2) V = V.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2) # 3. Scaled dot-product attention for each head scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(self.d_k) if mask is not None: scores = scores.masked_fill(mask == 0, float('-inf')) attention_weights = F.softmax(scores, dim=-1) attention_weights = self.dropout(attention_weights) context = torch.matmul(attention_weights, V) # 4. Concatenate heads # (batch, num_heads, seq_len, d_k) → (batch, seq_len, d_model) context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model) # 5. Final linear projection output = self.W_o(context) return output, attention_weights# Test multi-head attentionmha = MultiHeadAttention(d_model=512, num_heads=8)# Self-attention: Q=K=Vx = torch.randn(2, 10, 512)output, weights = mha(x, x, x)print(f"Input: {x.shape}")print(f"Output: {output.shape}")print(f"Attention weights: {weights.shape}")print(f" → {weights.shape[1]} heads, each with {weights.shape[2]}×{weights.shape[3]} attention matrix")
Visualizing Multi-Head Attention
def visualize_multi_head_attention(): """ Visualize what different attention heads learn. """ sentence = ["The", "cat", "sat", "on", "the", "mat", ".", "<pad>"] n = len(sentence) # Simulated attention patterns for different heads # In practice, these are learned # Head 1: Focuses on adjacent words head1 = np.eye(n) * 0.4 + np.eye(n, k=1) * 0.3 + np.eye(n, k=-1) * 0.3 head1 = head1 / head1.sum(axis=1, keepdims=True) # Head 2: Focuses on nouns (positions 1, 5 = cat, mat) head2 = np.zeros((n, n)) for i in range(n): head2[i, [1, 5]] = [0.5, 0.5] # Head 3: Focuses on structure (The...the, cat...mat) head3 = np.eye(n) * 0.5 head3[0, 4] = 0.5 head3[4, 0] = 0.5 head3[1, 5] = 0.25 head3[5, 1] = 0.25 head3 = head3 / head3.sum(axis=1, keepdims=True) # Head 4: Focuses on end of sentence head4 = np.zeros((n, n)) head4[:, -2] = 1.0 # Everyone attends to "." heads = [head1, head2, head3, head4] titles = ["Adjacent Words", "Nouns", "Structural", "Punctuation"] fig, axes = plt.subplots(1, 4, figsize=(16, 4)) for ax, head, title in zip(axes, heads, titles): im = ax.imshow(head, cmap='Blues', vmin=0, vmax=1) ax.set_xticks(range(n)) ax.set_yticks(range(n)) ax.set_xticklabels(sentence, rotation=45, ha='right') ax.set_yticklabels(sentence) ax.set_title(f"Head: {title}") plt.tight_layout() plt.show() print("Different heads learn to attend to different types of relationships!")visualize_multi_head_attention()
Positional Encoding
The Problem: Attention is Permutation-Invariant
Unlike RNNs, self-attention has no inherent notion of position:
def demonstrate_position_invariance(): """ Self-attention treats "The cat sat" and "sat cat The" the same way! We need to explicitly add position information. """ print("Problem: Attention is Position-Blind") print("=" * 60) print() print("Self-attention on 'The cat sat':") print(" Q_cat @ K_The → score (same regardless of position)") print() print("Self-attention on 'sat cat The':") print(" Q_cat @ K_The → SAME score!") print() print("Solution: Add positional encoding to embeddings") print(" embedding + position_encoding → position-aware representation")demonstrate_position_invariance()
Sinusoidal Positional Encoding
The original Transformer uses sinusoidal functions:PE(pos,2i)=sin(100002i/dmodelpos)PE(pos,2i+1)=cos(100002i/dmodelpos)
class SinusoidalPositionalEncoding(nn.Module): """ Sinusoidal Positional Encoding from "Attention Is All You Need". Properties: - Deterministic (no learned parameters) - Can extrapolate to longer sequences than seen during training - Relative positions can be represented as linear functions """ def __init__(self, d_model, max_len=5000, dropout=0.1): super().__init__() self.dropout = nn.Dropout(dropout) # Create positional encoding matrix pe = torch.zeros(max_len, d_model) position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1) div_term = torch.exp( torch.arange(0, d_model, 2).float() * (-np.log(10000.0) / d_model) ) pe[:, 0::2] = torch.sin(position * div_term) pe[:, 1::2] = torch.cos(position * div_term) pe = pe.unsqueeze(0) # (1, max_len, d_model) # Register as buffer (not a parameter, but saved with model) self.register_buffer('pe', pe) def forward(self, x): """ Args: x: (batch, seq_len, d_model) """ x = x + self.pe[:, :x.size(1)] return self.dropout(x)# Visualize positional encodingdef visualize_positional_encoding(): d_model = 128 max_len = 100 pe_layer = SinusoidalPositionalEncoding(d_model, max_len, dropout=0) # Get positional encodings x = torch.zeros(1, max_len, d_model) pe = pe_layer.pe[0, :max_len, :].numpy() plt.figure(figsize=(12, 6)) plt.imshow(pe.T, aspect='auto', cmap='RdBu') plt.xlabel('Position') plt.ylabel('Dimension') plt.title('Sinusoidal Positional Encoding') plt.colorbar(label='Encoding Value') plt.show() # Show specific dimensions fig, axes = plt.subplots(2, 2, figsize=(12, 8)) for i, ax in enumerate(axes.flat): dim = i * 20 ax.plot(pe[:, dim], label=f'dim {dim} (sin)') ax.plot(pe[:, dim+1], label=f'dim {dim+1} (cos)') ax.set_xlabel('Position') ax.set_ylabel('Value') ax.legend() ax.set_title(f'Dimensions {dim}, {dim+1}') ax.grid(True, alpha=0.3) plt.tight_layout() plt.show()visualize_positional_encoding()
Learned Positional Embeddings
An alternative: learn position embeddings like word embeddings:
class LearnedPositionalEncoding(nn.Module): """ Learned positional embeddings. Properties: - Can learn task-specific position patterns - Cannot extrapolate beyond max_len - More parameters to learn """ def __init__(self, d_model, max_len=512, dropout=0.1): super().__init__() self.position_embeddings = nn.Embedding(max_len, d_model) self.dropout = nn.Dropout(dropout) # Initialize with small values nn.init.normal_(self.position_embeddings.weight, std=0.02) def forward(self, x): """ Args: x: (batch, seq_len, d_model) """ seq_len = x.size(1) positions = torch.arange(seq_len, device=x.device) position_embeddings = self.position_embeddings(positions) x = x + position_embeddings return self.dropout(x)# Compare parameter countssinusoidal = SinusoidalPositionalEncoding(512, 1000)learned = LearnedPositionalEncoding(512, 1000)print(f"Sinusoidal PE params: {sum(p.numel() for p in sinusoidal.parameters()):,}")print(f"Learned PE params: {sum(p.numel() for p in learned.parameters()):,}")
Attention with Masking
Causal (Autoregressive) Mask
For language models, we need to prevent attending to future positions:
def create_causal_mask(seq_len): """ Create a causal mask to prevent attending to future positions. For position i, can only attend to positions 0, 1, ..., i """ mask = torch.triu(torch.ones(seq_len, seq_len), diagonal=1) mask = mask.masked_fill(mask == 1, float('-inf')) return maskdef visualize_causal_mask(): seq_len = 6 mask = create_causal_mask(seq_len) # Convert to attention-friendly format (0 = attend, -inf = don't attend) viz_mask = torch.zeros_like(mask) viz_mask[mask == float('-inf')] = 1 plt.figure(figsize=(6, 6)) plt.imshow(viz_mask.numpy(), cmap='Greys') plt.xlabel('Key Position (can attend to)') plt.ylabel('Query Position') plt.title('Causal Mask\n(white = masked, black = can attend)') for i in range(seq_len): for j in range(seq_len): color = 'white' if viz_mask[i, j] == 1 else 'black' plt.text(j, i, '✓' if viz_mask[i,j] == 0 else '✗', ha='center', va='center', color=color, fontsize=16) plt.show()visualize_causal_mask()class CausalSelfAttention(nn.Module): """Self-attention with causal masking for autoregressive models.""" def __init__(self, d_model, num_heads, max_len=512, dropout=0.1): super().__init__() self.mha = MultiHeadAttention(d_model, num_heads, dropout) # Register causal mask as buffer mask = torch.triu(torch.ones(max_len, max_len), diagonal=1).bool() self.register_buffer('mask', mask) def forward(self, x): seq_len = x.size(1) # Get appropriate size mask and expand for batch and heads mask = self.mask[:seq_len, :seq_len] mask = mask.unsqueeze(0).unsqueeze(0) # (1, 1, seq_len, seq_len) mask = ~mask # Invert: True = can attend, False = masked return self.mha(x, x, x, mask=mask)
Padding Mask
For variable-length sequences with padding:
def create_padding_mask(seq, pad_idx=0): """ Create mask to prevent attending to padding tokens. Args: seq: Token indices (batch, seq_len) pad_idx: Index of padding token Returns: mask: (batch, 1, 1, seq_len) - True where valid, False where padding """ mask = (seq != pad_idx).unsqueeze(1).unsqueeze(2) return mask# Examplebatch = torch.tensor([ [5, 3, 7, 2, 0, 0], # Padded with 2 zeros [8, 1, 4, 6, 9, 0], # Padded with 1 zero])padding_mask = create_padding_mask(batch, pad_idx=0)print("Sequence:")print(batch)print("\nPadding mask:")print(padding_mask.squeeze())
Implement scaled dot-product attention without using any PyTorch attention functions:
Implement the forward pass with proper scaling
Add masking support (padding and causal)
Verify gradients flow correctly
Test on a simple sequence copying task
Exercise 2: Visualize Real Attention Patterns
Using a pre-trained model (or train your own):
Extract attention weights for various inputs
Create visualizations showing what each head attends to
Identify heads with interpretable patterns
Compare attention patterns for different input types
Exercise 3: Compare Attention Variants
Implement and compare:
Dot-product attention
Additive (Bahdanau) attention
Multiplicative (Luong) attention
Train each on a translation task and compare:
Training speed
Final BLEU score
Attention patterns
Exercise 4: Relative Position Encoding
Implement relative positional encoding:
Instead of absolute positions, encode relative distances
Modify attention scores to include position bias
Compare with sinusoidal on long sequences
Test extrapolation to longer sequences
Exercise 5: Efficient Attention
Implement a more efficient attention mechanism:
Implement local attention (attend only to nearby positions)
Implement sparse attention patterns
Compare memory usage and speed with full attention
Evaluate impact on model quality
Training Pitfalls and Debugging Hints
Attention weights sum to 1 but that does not mean they are correct. A common mistake is interpreting attention weights as “explanation” for model behavior. Attention weights tell you where the model looked, not why it made its decision. Two models with identical outputs can have completely different attention patterns. Use attention visualizations for debugging intuition, not for mechanistic explanations.Uniform attention is a red flag. If your attention weights look roughly uniform (every position gets about 1/n attention), the model has not learned meaningful attention patterns. Common causes: (1) learning rate too low, (2) embedding quality is poor (the model cannot distinguish queries from keys), (3) the task does not actually require attention (try a simpler baseline).Attention memory scales quadratically with sequence length. For sequence length n and h heads, the attention weight matrix consumes O(n2⋅h) memory. At n=4096 with 12 heads in float32, that is about 750 MB per layer just for attention scores. If you run out of memory during training, sequence length is almost always the bottleneck — reduce it before reducing batch size.Mask shape mismatches: The most frustrating attention bug. PyTorch broadcasting rules mean a wrong mask shape often produces no error but silently computes the wrong thing. Always verify: padding masks should broadcast across heads and queries, causal masks should be square and lower-triangular, and the combination should use logical AND. Print mask.shape and scores.shape during debugging and verify they broadcast correctly.Multi-head attention head collapse: Sometimes all heads learn nearly identical attention patterns, wasting capacity. This happens with high dropout on attention weights or very small dk (dimension per head). Monitor head diversity by computing the cosine similarity between attention patterns of different heads — if it is consistently above 0.9, you have collapse. Fix: reduce attention dropout, increase dk, or add auxiliary losses that encourage head diversity.
Key Takeaways
Concept
Key Insight
Attention
Weighted combination of values based on query-key similarity
Dot-Product
softmax(QKT/d)V — simple and efficient
Multi-Head
Multiple attention functions for different relationship types
Self-Attention
Sequence attends to itself — O(1) dependency paths
Positional Encoding
Add position information since attention is permutation-invariant
Masking
Causal for autoregressive, padding for variable lengths
The attention mechanism is the foundation of modern NLP. Understanding it deeply will help you grasp Transformers, BERT, GPT, and virtually all state-of-the-art language models.
What’s Next
Module 11: Transformers
Build the complete Transformer architecture — combine attention with feed-forward networks, layer normalization, and residual connections to create the model that revolutionized NLP.
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Dev Weekends","colors":{"primary":"#000000","light":"#6B7280","dark":"#000000"},"favicon":"https://mintcdn.com/devweeekends/uFspW_ITEafc8XbV/images/logo.png?fit=max\u0026auto=format\u0026n=uFspW_ITEafc8XbV\u0026q=85\u0026s=24941b53d3f0793eb83974749e82df4f","navbar":{"links":[{"href":"https://devweekends.com","label":"Back to Site"}]},"footer":{"socials":{"youtube":"https://www.youtube.com/@devweekends","github":"https://github.com/DevWeekends","linkedin":"https://www.linkedin.com/company/dev-weekends"}},"contextual":{"options":["copy","view","chatgpt","claude","perplexity","mcp","cursor","vscode"],"display":"header"},"styling":{"latex":true},"navigation":{"global":{"anchors":[{"anchor":"DevWeekends","icon":"globe","href":"https://devweekends.com"}]},"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"devweeekends","hasAudioNarration":false,"actualSubdomain":"devweeekends","customDomains":["resources.devweekends.com"],"gitSource":{"type":"github","owner":"devweekends","repo":"docs","deployBranch":"main","contentDirectory":"","isPrivate":false,"mountPath":"","label":""},"entitlements":{},"buildId":"6ad5cb977e3343e90aaa7f7f","clientVersion":"0.0.4227","searchFilterCounts":[{"language":"default","tag":"Courses","count":785},{"language":"default","tag":"Crash Courses","count":145},{"language":"default","tag":"Interview Prep","count":69},{"language":"default","tag":"AI Engineering","count":60},{"language":"default","tag":"Distributed System Tools","count":60},{"language":"default","tag":"Deep Learning Mastery","count":48},{"language":"default","tag":"DevOps Tools","count":38},{"language":"default","tag":"Think Like an Engineer","count":38},{"language":"default","tag":"Competitive Programming","count":37},{"language":"default","tag":"Angular Master Course","count":34}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[]}
{"page":{"pagePath":"courses/deep-learning-mastery/10-attention","slug":"/courses/deep-learning-mastery/10-attention","pageMetadata":{"title":"Attention Mechanism","sidebarTitle":"Attention","description":"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers","icon":"eye","href":"/courses/deep-learning-mastery/10-attention"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={annotation:`annotation`,code:`code`,em:`em`,hr:`hr`,li:`li`,math:`math`,mfrac:`mfrac`,mi:`mi`,mn:`mn`,mo:`mo`,mrow:`mrow`,msqrt:`msqrt`,msub:`msub`,msubsup:`msubsup`,msup:`msup`,mtext:`mtext`,munder:`munder`,ol:`ol`,p:`p`,path:`path`,pre:`pre`,semantics:`semantics`,span:`span`,strong:`strong`,svg:`svg`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,ul:`ul`,..._provideComponents(),...props.components},{Accordion,AccordionGroup,Card,CardGroup,CodeBlock,Frame,Heading,Note,OptimizedImage,Table,Tip}=_components;if(!Accordion)_missingMdxReference(`Accordion`,true);if(!AccordionGroup)_missingMdxReference(`AccordionGroup`,true);if(!Card)_missingMdxReference(`Card`,true);if(!CardGroup)_missingMdxReference(`CardGroup`,true);if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Frame)_missingMdxReference(`Frame`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!OptimizedImage)_missingMdxReference(`OptimizedImage`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Tip)_missingMdxReference(`Tip`,true);return _jsxs(_Fragment,{children:[_jsx(Frame,{children:_jsx(OptimizedImage,{src:`https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/attention-concept.svg?fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=61d466edaab08013ce87f4be351e509e`,alt:`Attention Mechanism`,\"data-og-width\":`1080`,width:`1080`,\"data-og-height\":`1080`,height:`1080`,\"data-path\":`images/courses/deep-learning-mastery/attention-concept.svg`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/attention-concept.svg?w=280\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=daa5e88412584757495474380cfc6401 280w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/attention-concept.svg?w=560\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=3abc4d41a183f1334de2cf9cc04a5045 560w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/attention-concept.svg?w=840\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=b669cb7c27ef76da61e288266ded985d 840w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/attention-concept.svg?w=1100\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=763e8f5d683ee8e09d7f2e918dd39bf0 1100w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/attention-concept.svg?w=1650\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=891f512decd394350372685d8c84a403 1650w, https://mintcdn.com/devweeekends/0kwJwOL2KCwg2YYu/images/courses/deep-learning-mastery/attention-concept.svg?w=2500\u0026fit=max\u0026auto=format\u0026n=0kwJwOL2KCwg2YYu\u0026q=85\u0026s=ad89b582de88d257294cde5c34934b67 2500w`})}),`\n`,_jsx(Heading,{level:`1`,id:`attention-mechanism`,children:`Attention Mechanism`}),`\n`,_jsx(Heading,{level:`2`,id:`the-bottleneck-problem`,children:`The Bottleneck Problem`}),`\n`,_jsx(_components.p,{children:`In the previous chapters, we built sequence-to-sequence models using encoder-decoder LSTMs. But there’s a fundamental problem:`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`The entire source sequence is compressed into a single fixed-size vector.`}),` This is like asking someone to summarize a 300-page novel in a single tweet, and then expecting another person to reconstruct the entire plot from that tweet.`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`37`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`37`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.nn.functional `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` numpy `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` matplotlib.pyplot `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` visualize_bottleneck`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` The Encoder-Decoder Bottleneck Problem`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Source: \"The quick brown fox jumps over the lazy dog\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ↓`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` LSTM Encoder → Single vector (e.g., 512 dimensions)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ↓`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` LSTM Decoder → \"Le rapide renard brun saute par-dessus le chien paresseux\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Problem: ALL information about a 9-word sentence must fit in 512 numbers!`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` For longer sentences, this becomes impossible.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"The Bottleneck Problem:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"=\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 60`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Source sentence (English):\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" 'The quick brown fox jumps over the lazy dog'\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Compressed to: vector of shape (512,)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Then decoded to (French):\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" 'Le rapide renard brun saute par-dessus le chien paresseux'\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"❌ Problems:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" - Information loss for long sequences\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" - Decoder must 'guess' what encoder meant\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" - Early encoder states forgotten by final state\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`visualize_bottleneck()`})}),`\n`]})})}),`\n`,_jsx(Frame,{children:_jsx(OptimizedImage,{src:`https://mintlify.s3.us-west-1.amazonaws.com/devweeekends/images/courses/deep-learning-mastery/encoder-bottleneck.svg`,alt:`Encoder Bottleneck Problem`})}),`\n`,_jsx(Heading,{level:`3`,id:`evidence-of-the-problem`,children:`Evidence of the Problem`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`25`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`25`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` show_translation_quality_vs_length`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Real observation: Translation quality degrades with sentence length`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` when using fixed-size encoding.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Approximate BLEU scores from research papers`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` lengths `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`20`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`30`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`40`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`50`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`60`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` bleu_without_attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`35`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`32`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`28`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`22`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`16`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`12`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` bleu_with_attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`38`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`37`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`36`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`35`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`34`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`33`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.figure(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`figsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.plot(lengths, bleu_without_attention, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'o-'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`label`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Without Attention'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`linewidth`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.plot(lengths, bleu_with_attention, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'s-'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`label`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'With Attention'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`linewidth`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.xlabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Source Sentence Length'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.ylabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'BLEU Score'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.title(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Translation Quality vs Sentence Length'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.legend()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.grid(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`alpha`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.show()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Key insight: Attention maintains quality even for long sentences!\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# show_translation_quality_vs_length()`})}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`the-attention-solution`,children:`The Attention Solution`}),`\n`,_jsx(Heading,{level:`3`,id:`intuition-looking-back-at-the-source`,children:`Intuition: Looking Back at the Source`}),`\n`,_jsxs(_components.p,{children:[`Instead of forcing the decoder to use only the final encoder state, let it `,_jsx(_components.strong,{children:`look at all encoder states`}),` and focus on relevant ones:`]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`When translating “dog” to “chien”:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Look at all encoder states`}),`\n`,_jsx(_components.li,{children:`Focus attention on the state corresponding to “dog”`}),`\n`,_jsx(_components.li,{children:`Use that information to generate “chien”`}),`\n`]}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Key Insight`}),`: Attention computes a weighted combination of all encoder states, where weights indicate `,_jsx(_components.strong,{children:`relevance`}),` to the current decoding step.`]})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`31`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`31`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` attention_intuition`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Attention Intuition: The Query-Key-Value Framework`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Imagine you're at a library:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Query: \"I need information about neural networks\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Keys: Book titles on the shelves`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Values: The actual content of each book`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Process:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 1. Compare your query with all keys (book titles)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 2. Compute relevance scores`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 3. Retrieve values (content) weighted by relevance`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Attention as Information Retrieval:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"=\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 60`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Query (what we're looking for):\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" → Current decoder state: 'trying to translate word X'\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Keys (what we're comparing against):\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" → All encoder hidden states\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Values (what we retrieve):\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" → Same encoder hidden states (or transformed versions)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Output:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" → Weighted sum of values, weights from query-key similarity\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`attention_intuition()`})}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`attention-mechanisms-in-detail`,children:`Attention Mechanisms in Detail`}),`\n`,_jsx(Heading,{level:`3`,id:`dot-product-attention`,children:`Dot-Product Attention`}),`\n`,_jsx(_components.p,{children:`The simplest form of attention:`}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mtext,{children:`Attention`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.mi,{children:`Q`}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mi,{children:`K`}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mi,{children:`V`}),_jsx(_components.mo,{stretchy:`false`,children:`)`}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mtext,{children:`softmax`}),_jsxs(_components.mrow,{children:[_jsx(_components.mo,{fence:`true`,children:`(`}),_jsxs(_components.mfrac,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`Q`}),_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`K`}),_jsx(_components.mi,{children:`T`})]})]}),_jsx(_components.msqrt,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})})]}),_jsx(_components.mo,{fence:`true`,children:`)`})]}),_jsx(_components.mi,{children:`V`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\text{Attention}(Q, K, V) = \\\\text{softmax}\\\\left(\\\\frac{QK^T}{\\\\sqrt{d_k}}\\\\right)V`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`Attention`})}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsx(_components.span,{className:`mord mathnormal`,children:`Q`}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`K`}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.2222em`},children:`V`}),_jsx(_components.span,{className:`mclose`,children:`)`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`2.4684em`,verticalAlign:`-0.95em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`softmax`})}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsxs(_components.span,{className:`minner`,children:[_jsx(_components.span,{className:`mopen delimcenter`,style:{top:`0em`},children:_jsx(_components.span,{className:`delimsizing size3`,children:`(`})}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mopen nulldelimiter`}),_jsx(_components.span,{className:`mfrac`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`1.5183em`},children:[_jsxs(_components.span,{style:{top:`-2.2528em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,children:_jsx(_components.span,{className:`mord sqrt`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8572em`},children:[_jsxs(_components.span,{className:`svg-align`,style:{top:`-3em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,style:{paddingLeft:`0.833em`},children:_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})})]}),_jsxs(_components.span,{style:{top:`-2.8172em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`hide-tail`,style:{minWidth:`0.853em`,height:`1.08em`},children:_jsx(_components.svg,{xmlns:`http://www.w3.org/2000/svg`,width:`400em`,height:`1.08em`,viewBox:`0 0 400000 1080`,preserveAspectRatio:`xMinYMin slice`,children:_jsx(_components.path,{d:`M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1828em`},children:_jsx(_components.span,{})})})]})})})]}),_jsxs(_components.span,{style:{top:`-3.23em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`frac-line`,style:{borderBottomWidth:`0.04em`}})]}),_jsxs(_components.span,{style:{top:`-3.677em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`Q`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`K`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8413em`},children:_jsxs(_components.span,{style:{top:`-3.063em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.1389em`},children:`T`})})]})})})})})]})]})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.93em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mclose nulldelimiter`})]}),_jsx(_components.span,{className:`mclose delimcenter`,style:{top:`0em`},children:_jsx(_components.span,{className:`delimsizing size3`,children:`)`})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.2222em`},children:`V`})]})]})]})}),`\n`,_jsxs(_components.p,{children:[_jsxs(_components.strong,{children:[`Why divide by `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.msqrt,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\sqrt{d_k}`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.04em`,verticalAlign:`-0.1828em`}}),_jsx(_components.span,{className:`mord sqrt`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8572em`},children:[_jsxs(_components.span,{className:`svg-align`,style:{top:`-3em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,style:{paddingLeft:`0.833em`},children:_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})})]}),_jsxs(_components.span,{style:{top:`-2.8172em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`hide-tail`,style:{minWidth:`0.853em`,height:`1.08em`},children:_jsx(_components.svg,{xmlns:`http://www.w3.org/2000/svg`,width:`400em`,height:`1.08em`,viewBox:`0 0 400000 1080`,preserveAspectRatio:`xMinYMin slice`,children:_jsx(_components.path,{d:`M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1828em`},children:_jsx(_components.span,{})})})]})})]})})]}),`?`]}),` This is not an arbitrary choice — it is essential for numerical stability. When `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`d_k`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})]})})]}),` is large (e.g., 512), each dot product `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`q`}),_jsx(_components.mo,{children:`⋅`}),_jsx(_components.mi,{children:`k`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`q \\\\cdot k`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6389em`,verticalAlign:`-0.1944em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0359em`},children:`q`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`⋅`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6944em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0315em`},children:`k`})]})]})]}),` is the sum of 512 terms. If entries of `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`q`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`q`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.625em`,verticalAlign:`-0.1944em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0359em`},children:`q`})]})})]}),` and `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`k`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`k`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6944em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0315em`},children:`k`})]})})]}),` have unit variance, the dot product has variance `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`d_k`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})]})})]}),` (by the central limit theorem, the sum of 512 independent unit-variance terms has variance 512). So the raw scores have standard deviation `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.msqrt,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})}),_jsx(_components.mo,{children:`≈`}),_jsx(_components.mn,{children:`22.6`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\sqrt{d_k} \\\\approx 22.6`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.04em`,verticalAlign:`-0.1828em`}}),_jsx(_components.span,{className:`mord sqrt`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8572em`},children:[_jsxs(_components.span,{className:`svg-align`,style:{top:`-3em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,style:{paddingLeft:`0.833em`},children:_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})})]}),_jsxs(_components.span,{style:{top:`-2.8172em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`hide-tail`,style:{minWidth:`0.853em`,height:`1.08em`},children:_jsx(_components.svg,{xmlns:`http://www.w3.org/2000/svg`,width:`400em`,height:`1.08em`,viewBox:`0 0 400000 1080`,preserveAspectRatio:`xMinYMin slice`,children:_jsx(_components.path,{d:`M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1828em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`≈`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6444em`}}),_jsx(_components.span,{className:`mord`,children:`22.6`})]})]})]}),`. Softmax applied to values this large pushes almost all the probability mass onto a single key — the attention becomes a hard argmax, and its gradient effectively vanishes.`]}),`\n`,_jsxs(_components.p,{children:[`Dividing by `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.msqrt,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\sqrt{d_k}`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.04em`,verticalAlign:`-0.1828em`}}),_jsx(_components.span,{className:`mord sqrt`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8572em`},children:[_jsxs(_components.span,{className:`svg-align`,style:{top:`-3em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,style:{paddingLeft:`0.833em`},children:_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})})]}),_jsxs(_components.span,{style:{top:`-2.8172em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`hide-tail`,style:{minWidth:`0.853em`,height:`1.08em`},children:_jsx(_components.svg,{xmlns:`http://www.w3.org/2000/svg`,width:`400em`,height:`1.08em`,viewBox:`0 0 400000 1080`,preserveAspectRatio:`xMinYMin slice`,children:_jsx(_components.path,{d:`M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1828em`},children:_jsx(_components.span,{})})})]})})]})})]}),` rescales the scores to unit variance, keeping the softmax in its “useful” regime where gradients flow to multiple keys. Without this scaling, training larger models would be dramatically harder. This is one of those small details that separates a paper implementation from a working one.`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`59`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`59`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` scaled_dot_product_attention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`query`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`key`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`value`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Scaled Dot-Product Attention -- the core computation inside every transformer.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` query: (batch, num_queries, d_k) -- \"what am I looking for?\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` key: (batch, num_keys, d_k) -- \"what do I contain?\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` value: (batch, num_keys, d_v) -- \"what information do I provide?\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` mask: Optional mask to prevent attention to certain positions`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Returns:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` output: (batch, num_queries, d_v) -- weighted combination of values`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` attention_weights: (batch, num_queries, num_keys) -- where we looked`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_k `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` query.size(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Step 1: Compute attention scores via dot product`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # High dot product = query and key point in similar directions = \"relevant\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # (batch, num_queries, d_k) @ (batch, d_k, num_keys) -> (batch, num_queries, num_keys)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(query, key.transpose(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Step 2: Scale by sqrt(d_k) -- this is critical and often misunderstood`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Without scaling, dot products grow proportionally to d_k, pushing softmax`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # into saturation where gradients are near-zero. Dividing by sqrt(d_k)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # keeps the variance of scores at ~1 regardless of dimension.`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.sqrt(d_k)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Apply mask if provided`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`is`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores.masked_fill(mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'-inf'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Softmax to get attention weights`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention_weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F.softmax(scores, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Apply attention weights to values`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(attention_weights, value)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output, attention_weights`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Example`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`batch_size `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`num_queries `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 4`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # e.g., decoder positions`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`num_keys `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 6`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # e.g., encoder positions`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`d_k `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 64`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # dimension of queries and keys`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`d_v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 64`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # dimension of values`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`query `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(batch_size, num_queries, d_k)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`key `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(batch_size, num_keys, d_k)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`value `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(batch_size, num_keys, d_v)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`output, weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scaled_dot_product_attention(query, key, value)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Query shape: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`query.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Key shape: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`key.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Value shape: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`value.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Output shape: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`output.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Weights shape: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:`\\\\n`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`Attention weights sum to 1: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.sum(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`visualizing-attention-weights`,children:`Visualizing Attention Weights`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`42`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`42`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` visualize_attention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Visualize what attention 'looks at'.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Simulated attention weights for translation`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` source `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"The\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"cat\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"sat\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"on\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"the\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"mat\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\".\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` target `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Le\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"chat\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"était\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"assis\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"sur\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"le\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"tapis\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\".\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Attention matrix (which source words each target word attends to)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.array([`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Le → The`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# chat → cat`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.7`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# était → sat`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.7`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# assis → sat`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# sur → on`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# le → the`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# tapis → mat`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# . → .`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ])`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` fig, ax `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.subplots(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`figsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` im `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.imshow(attention, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`cmap`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Blues'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_xticks(np.arange(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(source)))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_yticks(np.arange(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(target)))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_xticklabels(source, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`fontsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`12`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_yticklabels(target, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`fontsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`12`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_xlabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Source (English)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`fontsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`14`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_ylabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Target (French)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`fontsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`14`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_title(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Attention Weights in Translation\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`fontsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`16`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Add attention values in cells`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(target)):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` j `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(source)):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` text `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.text(j, i, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`attention[i, j]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:.1f}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` ha`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'center'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`va`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'center'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`color`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'black'`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention[i,j] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`\u003c`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.5`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'white'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.colorbar(im)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.tight_layout()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.show()`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`visualize_attention()`})}),`\n`]})})}),`\n`,_jsx(Frame,{children:_jsx(OptimizedImage,{src:`https://mintlify.s3.us-west-1.amazonaws.com/devweeekends/images/courses/deep-learning-mastery/attention-visualization.svg`,alt:`Attention Weight Visualization`})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`types-of-attention`,children:`Types of Attention`}),`\n`,_jsx(Heading,{level:`3`,id:`additive-bahdanau-attention`,children:`Additive (Bahdanau) Attention`}),`\n`,_jsx(_components.p,{children:`The original attention mechanism from the 2014 paper:`}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`e`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{children:`j`})]})]}),_jsx(_components.mo,{children:`=`}),_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`v`}),_jsx(_components.mi,{children:`T`})]}),_jsx(_components.mi,{children:`tanh`}),_jsx(_components.mo,{children:``}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`W`}),_jsx(_components.mn,{children:`1`})]}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`h`}),_jsx(_components.mi,{children:`i`})]}),_jsx(_components.mo,{children:`+`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`W`}),_jsx(_components.mn,{children:`2`})]}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`s`}),_jsx(_components.mi,{children:`j`})]}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`e_{ij} = v^T \\\\tanh(W_1 h_i + W_2 s_j)`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7167em`,verticalAlign:`-0.2861em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`e`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3117em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0572em`},children:`ij`})})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2861em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.1413em`,verticalAlign:`-0.25em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0359em`},children:`v`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8913em`},children:_jsxs(_components.span,{style:{top:`-3.113em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.1389em`},children:`T`})})]})})})})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mop`,children:`tanh`}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`W`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3011em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`-0.1389em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`1`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`h`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3117em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`+`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.0361em`,verticalAlign:`-0.2861em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`W`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3011em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`-0.1389em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`2`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`s`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3117em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0572em`},children:`j`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2861em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mclose`,children:`)`})]})]})]})}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`α`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{children:`j`})]})]}),_jsx(_components.mo,{children:`=`}),_jsxs(_components.mfrac,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`exp`}),_jsx(_components.mo,{children:``}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`e`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{children:`j`})]})]}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]}),_jsxs(_components.mrow,{children:[_jsxs(_components.munder,{children:[_jsx(_components.mo,{children:`∑`}),_jsx(_components.mi,{children:`k`})]}),_jsx(_components.mi,{children:`exp`}),_jsx(_components.mo,{children:``}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`e`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`k`}),_jsx(_components.mi,{children:`j`})]})]}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]})]})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\alpha_{ij} = \\\\frac{\\\\exp(e_{ij})}{\\\\sum_k \\\\exp(e_{kj})}`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.7167em`,verticalAlign:`-0.2861em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0037em`},children:`α`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3117em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`-0.0037em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0572em`},children:`ij`})})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2861em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`2.4127em`,verticalAlign:`-0.9857em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mopen nulldelimiter`}),_jsx(_components.span,{className:`mfrac`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`1.427em`},children:[_jsxs(_components.span,{style:{top:`-2.314em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsxs(_components.span,{className:`mop`,children:[_jsx(_components.span,{className:`mop op-symbol small-op`,style:{position:`relative`,top:`0em`},children:`∑`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.1864em`},children:_jsxs(_components.span,{style:{top:`-2.4003em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2997em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mop`,children:`exp`}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`e`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`}),_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0572em`},children:`j`})]})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2861em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mclose`,children:`)`})]})]}),_jsxs(_components.span,{style:{top:`-3.23em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`frac-line`,style:{borderBottomWidth:`0.04em`}})]}),_jsxs(_components.span,{style:{top:`-3.677em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mop`,children:`exp`}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`e`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3117em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0572em`},children:`ij`})})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2861em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mclose`,children:`)`})]})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.9857em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mclose nulldelimiter`})]})]})]})]})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`67`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`67`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` AdditiveAttention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Bahdanau (Additive) Attention (2014).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Uses a learned alignment model (a small neural network) to compute`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` attention scores. More expressive but slower than dot-product attention`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` because it requires an extra matrix multiplication and tanh activation.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Why \"additive\"? The score adds two projections (W1*encoder + W2*decoder)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` then passes through tanh. This allows non-linear interactions between`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` query and key, unlike the dot product which is strictly linear.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`encoder_dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`decoder_dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`attention_dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Project encoder states`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(encoder_dim, attention_dim, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Project decoder state`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W2 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(decoder_dim, attention_dim, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Compute scalar attention score`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(attention_dim, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`encoder_outputs`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`decoder_hidden`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` encoder_outputs: (batch, src_len, encoder_dim)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` decoder_hidden: (batch, decoder_dim)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Returns:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` context: (batch, encoder_dim)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` attention_weights: (batch, src_len)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Project encoder outputs`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` encoder_proj `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W1(encoder_outputs) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (batch, src_len, attention_dim)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Project decoder hidden (add dimension for broadcasting)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` decoder_proj `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W2(decoder_hidden).unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (batch, 1, attention_dim)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Compute alignment scores`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.v(torch.tanh(encoder_proj `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` decoder_proj)) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (batch, src_len, 1)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores.squeeze(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (batch, src_len)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Softmax to get attention weights`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention_weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F.softmax(scores, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Compute context vector`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.bmm(attention_weights.unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`), encoder_outputs)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context.squeeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (batch, encoder_dim)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context, attention_weights`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Test`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` AdditiveAttention(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`encoder_dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`decoder_dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`attention_dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`256`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`encoder_outputs `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`20`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# 8 sequences, 20 timesteps`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`decoder_hidden `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`context, weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention(encoder_outputs, decoder_hidden)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Encoder outputs: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`encoder_outputs.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Decoder hidden: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`decoder_hidden.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Context vector: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`context.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Attention weights: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`multiplicative-luong-attention`,children:`Multiplicative (Luong) Attention`}),`\n`,_jsx(_components.p,{children:`Simpler and faster variants:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`56`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`56`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MultiplicativeAttention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Luong (Multiplicative) Attention variants.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Variants:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - dot: score = h_s · h_t`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - general: score = h_s · W · h_t`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - concat: score = v · tanh(W · [h_s; h_t])`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`encoder_dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`decoder_dim`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`method`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'general'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.method `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` method`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` method `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'general'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(decoder_dim, encoder_dim, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` elif`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` method `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'concat'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(encoder_dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` decoder_dim, decoder_dim, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(decoder_dim, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bias`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`False`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`encoder_outputs`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`decoder_hidden`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` encoder_outputs: (batch, src_len, encoder_dim)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` decoder_hidden: (batch, decoder_dim)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.method `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'dot'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Simple dot product`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.bmm(encoder_outputs, decoder_hidden.unsqueeze(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores.squeeze(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` elif`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.method `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'general'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Transform then dot product`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` decoder_proj `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W(decoder_hidden) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (batch, encoder_dim)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.bmm(encoder_outputs, decoder_proj.unsqueeze(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores.squeeze(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` elif`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.method `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'concat'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Concatenate and project`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` src_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` encoder_outputs.size(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` decoder_expanded `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` decoder_hidden.unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).expand(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, src_len, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` concat `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cat([encoder_outputs, decoder_expanded], `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.v(torch.tanh(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W(concat))).squeeze(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention_weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F.softmax(scores, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.bmm(attention_weights.unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`), encoder_outputs).squeeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context, attention_weights`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Compare methods`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` method `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'dot'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'general'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'concat'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` MultiplicativeAttention(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`method`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`method)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` params `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` sum`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(p.numel() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn.parameters())`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`method`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:8s}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`params`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:,}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` parameters\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`self-attention`,children:`Self-Attention`}),`\n`,_jsx(Heading,{level:`3`,id:`attending-to-the-same-sequence`,children:`Attending to the Same Sequence`}),`\n`,_jsx(_components.p,{children:`Self-attention allows each position in a sequence to attend to all other positions:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`58`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`58`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` SelfAttention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Self-Attention: Sequence attends to itself.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Each position can gather information from all other positions,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` allowing for direct modeling of dependencies regardless of distance.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Project to query, key, value`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_q `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_k `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` x: Input sequence (batch, seq_len, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` mask: Optional attention mask`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Returns:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` output: (batch, seq_len, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` attention_weights: (batch, seq_len, seq_len)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Project to Q, K, V`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` Q `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_q(x)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` K `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_k(x)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` V `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_v(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Scaled dot-product attention`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_k `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` Q.size(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(Q, K.transpose(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.sqrt(d_k)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`is`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores.masked_fill(mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'-inf'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention_weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F.softmax(scores, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(attention_weights, V)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output, attention_weights`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Example: Self-attention on a sentence`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`self_attn `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` SelfAttention(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`256`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# 4 sentences, 10 words each, 256-dim embeddings`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`256`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`output, weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` self_attn(x)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Input: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`x.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Output: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`output.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Attention weights: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:`\\\\n`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`Each word attends to all words (including itself)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`why-self-attention-matters`,children:`Why Self-Attention Matters`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`33`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`33`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` compare_rnn_vs_self_attention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Self-attention vs RNN for capturing dependencies.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Self-Attention Advantages over RNNs:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"=\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 60`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"1. PATH LENGTH for long-range dependencies:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" RNN: O(n) - information must flow through n steps\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" Attn: O(1) - direct connection between any two positions\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"2. PARALLELIZATION:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" RNN: Sequential - must process one step at a time\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" Attn: Fully parallel - all positions computed simultaneously\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"3. INTERPRETABILITY:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" RNN: Hidden state is opaque\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" Attn: Attention weights show what the model focuses on\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Computational complexity comparison`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` n `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1000`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # sequence length`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 512`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # hidden dimension`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` rnn_complexity `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` n `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# O(n·d²) sequential steps`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn_complexity `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` n `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` n `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# O(n²·d) but parallelizable`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"For sequence length `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`n`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`, hidden dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`d`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" RNN: ~`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`rnn_complexity`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:,}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ops (sequential)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" Attn: ~`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`attn_complexity`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:,}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ops (parallel)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`compare_rnn_vs_self_attention()`})}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`multi-head-attention`,children:`Multi-Head Attention`}),`\n`,_jsx(Heading,{level:`3`,id:`multiple-attention-“perspectives”`,children:`Multiple Attention “Perspectives”`}),`\n`,_jsx(_components.p,{children:`Instead of one attention function, use multiple “heads” that each learn different relationships:`}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mtext,{children:`MultiHead`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.mi,{children:`Q`}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mi,{children:`K`}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mi,{children:`V`}),_jsx(_components.mo,{stretchy:`false`,children:`)`}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mtext,{children:`Concat`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsxs(_components.msub,{children:[_jsx(_components.mtext,{children:`head`}),_jsx(_components.mn,{children:`1`})]}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mi,{mathvariant:`normal`,children:`.`}),_jsx(_components.mi,{mathvariant:`normal`,children:`.`}),_jsx(_components.mi,{mathvariant:`normal`,children:`.`}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsxs(_components.msub,{children:[_jsx(_components.mtext,{children:`head`}),_jsx(_components.mi,{children:`h`})]}),_jsx(_components.mo,{stretchy:`false`,children:`)`}),_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`W`}),_jsx(_components.mi,{children:`O`})]})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\text{MultiHead}(Q, K, V) = \\\\text{Concat}(\\\\text{head}_1, ..., \\\\text{head}_h) W^O`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`MultiHead`})}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsx(_components.span,{className:`mord mathnormal`,children:`Q`}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`K`}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.2222em`},children:`V`}),_jsx(_components.span,{className:`mclose`,children:`)`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.1413em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`Concat`})}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`head`})}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3011em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`1`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord`,children:`...`}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`head`})}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`h`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mclose`,children:`)`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`W`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8913em`},children:_jsxs(_components.span,{style:{top:`-3.113em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0278em`},children:`O`})})]})})})})})]})]})]})]})}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsxs(_components.msub,{children:[_jsx(_components.mtext,{children:`where\\xA0head`}),_jsx(_components.mi,{children:`i`})]}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mtext,{children:`Attention`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.mi,{children:`Q`}),_jsxs(_components.msubsup,{children:[_jsx(_components.mi,{children:`W`}),_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{children:`Q`})]}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mi,{children:`K`}),_jsxs(_components.msubsup,{children:[_jsx(_components.mi,{children:`W`}),_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{children:`K`})]}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mi,{children:`V`}),_jsxs(_components.msubsup,{children:[_jsx(_components.mi,{children:`W`}),_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{children:`V`})]}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\text{where head}_i = \\\\text{Attention}(QW_i^Q, KW_i^K, VW_i^V)`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`where\\xA0head`})}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3117em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.2361em`,verticalAlign:`-0.2769em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`Attention`})}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsx(_components.span,{className:`mord mathnormal`,children:`Q`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`W`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.9592em`},children:[_jsxs(_components.span,{style:{top:`-2.4231em`,marginLeft:`-0.1389em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`})})]}),_jsxs(_components.span,{style:{top:`-3.1809em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`Q`})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.2769em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`K`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`W`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8913em`},children:[_jsxs(_components.span,{style:{top:`-2.453em`,marginLeft:`-0.1389em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`})})]}),_jsxs(_components.span,{style:{top:`-3.113em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0715em`},children:`K`})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.247em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mpunct`,children:`,`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.2222em`},children:`V`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`W`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.8913em`},children:[_jsxs(_components.span,{style:{top:`-2.453em`,marginLeft:`-0.1389em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`})})]}),_jsxs(_components.span,{style:{top:`-3.113em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.2222em`},children:`V`})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.247em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mclose`,children:`)`})]})]})]})}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Why multiple heads instead of one big attention?`}),` A single attention head can only compute one weighted average per query position — it assigns a single scalar relevance to each key. But language has multiple simultaneous relationships: “it” relates to “animal” syntactically (coreference), to “tired” semantically (attribute), and to “cross” structurally (subject-verb). Multiple heads let the model maintain all these relationships simultaneously, each head specializing in a different type of dependency.`]}),`\n`,_jsxs(_components.p,{children:[`The mathematical cost is negligible: if `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`m`}),_jsx(_components.mi,{children:`o`}),_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`e`}),_jsx(_components.mi,{children:`l`})]})]}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mn,{children:`512`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`d_{model} = 512`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mathnormal mtight`,children:`m`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`o`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`d`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`e`}),_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0197em`},children:`l`})]})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6444em`}}),_jsx(_components.span,{className:`mord`,children:`512`})]})]})]}),` and you use 8 heads, each head operates in a `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mn,{children:`64`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`d_k = 64`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6444em`}}),_jsx(_components.span,{className:`mord`,children:`64`})]})]})]}),` dimensional subspace. The total computation is the same as a single head with `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mn,{children:`512`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`d_k = 512`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6444em`}}),_jsx(_components.span,{className:`mord`,children:`512`})]})]})]}),`, but you get 8 independent attention patterns instead of 1. The output projection `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`W`}),_jsx(_components.mi,{children:`O`})]})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`W^O`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8413em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`W`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8413em`},children:_jsxs(_components.span,{style:{top:`-3.063em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0278em`},children:`O`})})]})})})})})]})]})})]}),` then learns how to combine these perspectives. This is one of the best “free lunches” in deep learning architecture design.`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`97`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`97`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` MultiHeadAttention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Multi-Head Attention.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Multiple attention heads allow the model to jointly attend to`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` information from different representation subspaces.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Analogy: Imagine a team of analysts reading the same document.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` One analyst focuses on who did what (subject-verb relationships).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Another tracks pronoun references (coreference).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` A third watches for temporal ordering (before/after).`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Each analyst sees the same text but extracts different relationships.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` For example, in \"The animal didn't cross the street because it was too tired\":`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - One head might focus on \"it\" -> \"animal\" (coreference resolution)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Another head might focus on \"tired\" -> \"animal\" (attribute binding)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - A third might focus on \"cross\" -> \"street\" (action-object pairing)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` The multi-head output combines all these perspectives.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`num_heads`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` assert`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`%`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"d_model must be divisible by num_heads\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_k `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`//`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` num_heads`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Linear projections for Q, K, V`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_q `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_k `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_v `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Output projection`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_o `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Dropout(dropout)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`query`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`key`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`value`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` query: (batch, seq_len_q, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` key: (batch, seq_len_k, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` value: (batch, seq_len_v, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` mask: Optional mask (batch, 1, seq_len_q, seq_len_k) or (batch, 1, 1, seq_len_k)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Returns:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` output: (batch, seq_len_q, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` attention_weights: (batch, num_heads, seq_len_q, seq_len_k)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` batch_size `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` query.size(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # 1. Linear projections`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` Q `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_q(query)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` K `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_k(key)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` V `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_v(value)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # 2. Split into multiple heads`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # (batch, seq_len, d_model) → (batch, num_heads, seq_len, d_k)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` Q `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` Q.view(batch_size, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_k).transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` K `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` K.view(batch_size, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_k).transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` V `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` V.view(batch_size, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.num_heads, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_k).transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # 3. Scaled dot-product attention for each head`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(Q, K.transpose(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.sqrt(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_k)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`is`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` not`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` scores.masked_fill(mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'-inf'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention_weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` F.softmax(scores, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attention_weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout(attention_weights)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.matmul(attention_weights, V)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # 4. Concatenate heads`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # (batch, num_heads, seq_len, d_k) → (batch, seq_len, d_model)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` context.transpose(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).contiguous().view(batch_size, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # 5. Final linear projection`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.W_o(context)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` output, attention_weights`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Test multi-head attention`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mha `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` MultiHeadAttention(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`num_heads`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Self-attention: Q=K=V`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randn(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`output, weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mha(x, x, x)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Input: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`x.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Output: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`output.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Attention weights: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" → `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.shape[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` heads, each with `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.shape[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`×`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`weights.shape[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` attention matrix\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`visualizing-multi-head-attention`,children:`Visualizing Multi-Head Attention`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`51`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`51`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` visualize_multi_head_attention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Visualize what different attention heads learn.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` sentence `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"The\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"cat\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"sat\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"on\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"the\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"mat\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\".\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"\u003cpad>\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` n `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(sentence)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Simulated attention patterns for different heads`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # In practice, these are learned`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Head 1: Focuses on adjacent words`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.eye(n) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.4`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` +`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.eye(n, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`k`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.3`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` +`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.eye(n, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`k`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.3`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head1 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head1.sum(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`axis`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`keepdims`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Head 2: Focuses on nouns (positions 1, 5 = cat, mat)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head2 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.zeros((n, n))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(n):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head2[i, [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Head 3: Focuses on structure (The...the, cat...mat)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.eye(n) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.5`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.5`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.5`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.25`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0.25`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head3.sum(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`axis`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`keepdims`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Head 4: Focuses on end of sentence`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head4 `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.zeros((n, n))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` head4[:, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1.0`}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Everyone attends to \".\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` heads `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [head1, head2, head3, head4]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` titles `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Adjacent Words\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Nouns\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Structural\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Punctuation\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` fig, axes `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.subplots(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`figsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`16`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax, head, title `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` zip`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(axes, heads, titles):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` im `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.imshow(head, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`cmap`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Blues'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`vmin`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`vmax`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_xticks(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(n))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_yticks(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(n))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_xticklabels(sentence, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`rotation`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`45`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`ha`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'right'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_yticklabels(sentence)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_title(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Head: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`title`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.tight_layout()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.show()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Different heads learn to attend to different types of relationships!\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`visualize_multi_head_attention()`})}),`\n`]})})}),`\n`,_jsx(Frame,{children:_jsx(OptimizedImage,{src:`https://mintlify.s3.us-west-1.amazonaws.com/devweeekends/images/courses/deep-learning-mastery/multi-head-patterns.svg`,alt:`Multi-Head Attention Patterns`})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`positional-encoding`,children:`Positional Encoding`}),`\n`,_jsx(Heading,{level:`3`,id:`the-problem-attention-is-permutation-invariant`,children:`The Problem: Attention is Permutation-Invariant`}),`\n`,_jsx(_components.p,{children:`Unlike RNNs, self-attention has no inherent notion of position:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`19`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`19`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` demonstrate_position_invariance`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Self-attention treats \"The cat sat\" and \"sat cat The\" the same way!`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` We need to explicitly add position information.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Problem: Attention is Position-Blind\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"=\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` *`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 60`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Self-attention on 'The cat sat':\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" Q_cat @ K_The → score (same regardless of position)\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Self-attention on 'sat cat The':\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" Q_cat @ K_The → SAME score!\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Solution: Add positional encoding to embeddings\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\" embedding + position_encoding → position-aware representation\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`demonstrate_position_invariance()`})}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`sinusoidal-positional-encoding`,children:`Sinusoidal Positional Encoding`}),`\n`,_jsx(_components.p,{children:`The original Transformer uses sinusoidal functions:`}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`P`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`E`}),_jsxs(_components.mrow,{children:[_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.mi,{children:`p`}),_jsx(_components.mi,{children:`o`}),_jsx(_components.mi,{children:`s`}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mn,{children:`2`}),_jsx(_components.mi,{children:`i`}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]})]}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mi,{children:`sin`}),_jsx(_components.mo,{children:``}),_jsxs(_components.mrow,{children:[_jsx(_components.mo,{fence:`true`,children:`(`}),_jsxs(_components.mfrac,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`p`}),_jsx(_components.mi,{children:`o`}),_jsx(_components.mi,{children:`s`})]}),_jsxs(_components.msup,{children:[_jsx(_components.mn,{children:`10000`}),_jsxs(_components.mrow,{children:[_jsx(_components.mn,{children:`2`}),_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{mathvariant:`normal`,children:`/`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`m`}),_jsx(_components.mi,{children:`o`}),_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`e`}),_jsx(_components.mi,{children:`l`})]})]})]})]})]}),_jsx(_components.mo,{fence:`true`,children:`)`})]})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`PE_{(pos, 2i)} = \\\\sin\\\\left(\\\\frac{pos}{10000^{2i/d_{model}}}\\\\right)`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.0385em`,verticalAlign:`-0.3552em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`P`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0576em`},children:`E`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3448em`},children:_jsxs(_components.span,{style:{top:`-2.5198em`,marginLeft:`-0.0576em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mopen mtight`,children:`(`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`p`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`os`}),_jsx(_components.span,{className:`mpunct mtight`,children:`,`}),_jsx(_components.span,{className:`mord mtight`,children:`2`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`}),_jsx(_components.span,{className:`mclose mtight`,children:`)`})]})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.3552em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.854em`,verticalAlign:`-0.704em`}}),_jsx(_components.span,{className:`mop`,children:`sin`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsxs(_components.span,{className:`minner`,children:[_jsx(_components.span,{className:`mopen delimcenter`,style:{top:`0em`},children:_jsx(_components.span,{className:`delimsizing size2`,children:`(`})}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mopen nulldelimiter`}),_jsx(_components.span,{className:`mfrac`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`1.1076em`},children:[_jsxs(_components.span,{style:{top:`-2.296em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord`,children:`1000`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord`,children:`0`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.814em`},children:_jsxs(_components.span,{style:{top:`-2.989em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mtight`,children:`2`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`}),_jsx(_components.span,{className:`mord mtight`,children:`/`}),_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mathnormal mtight`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3448em`},children:_jsxs(_components.span,{style:{top:`-2.3488em`,marginLeft:`0em`,marginRight:`0.0714em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.5em`}}),_jsx(_components.span,{className:`sizing reset-size3 size1 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mathnormal mtight`,children:`m`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`o`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`d`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`e`}),_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0197em`},children:`l`})]})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1512em`},children:_jsx(_components.span,{})})})]})})]})]})})]})})})})})]})]})]}),_jsxs(_components.span,{style:{top:`-3.23em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`frac-line`,style:{borderBottomWidth:`0.04em`}})]}),_jsxs(_components.span,{style:{top:`-3.677em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`p`}),_jsx(_components.span,{className:`mord mathnormal`,children:`os`})]})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.704em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mclose nulldelimiter`})]}),_jsx(_components.span,{className:`mclose delimcenter`,style:{top:`0em`},children:_jsx(_components.span,{className:`delimsizing size2`,children:`)`})})]})]})]})]})}),`\n`,_jsx(_components.span,{className:`katex-display`,children:_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,display:`block`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`P`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`E`}),_jsxs(_components.mrow,{children:[_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.mi,{children:`p`}),_jsx(_components.mi,{children:`o`}),_jsx(_components.mi,{children:`s`}),_jsx(_components.mo,{separator:`true`,children:`,`}),_jsx(_components.mn,{children:`2`}),_jsx(_components.mi,{children:`i`}),_jsx(_components.mo,{children:`+`}),_jsx(_components.mn,{children:`1`}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]})]}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mi,{children:`cos`}),_jsx(_components.mo,{children:``}),_jsxs(_components.mrow,{children:[_jsx(_components.mo,{fence:`true`,children:`(`}),_jsxs(_components.mfrac,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`p`}),_jsx(_components.mi,{children:`o`}),_jsx(_components.mi,{children:`s`})]}),_jsxs(_components.msup,{children:[_jsx(_components.mn,{children:`10000`}),_jsxs(_components.mrow,{children:[_jsx(_components.mn,{children:`2`}),_jsx(_components.mi,{children:`i`}),_jsx(_components.mi,{mathvariant:`normal`,children:`/`}),_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`m`}),_jsx(_components.mi,{children:`o`}),_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`e`}),_jsx(_components.mi,{children:`l`})]})]})]})]})]}),_jsx(_components.mo,{fence:`true`,children:`)`})]})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`PE_{(pos, 2i+1)} = \\\\cos\\\\left(\\\\frac{pos}{10000^{2i/d_{model}}}\\\\right)`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.0385em`,verticalAlign:`-0.3552em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.1389em`},children:`P`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0576em`},children:`E`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3448em`},children:_jsxs(_components.span,{style:{top:`-2.5198em`,marginLeft:`-0.0576em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mopen mtight`,children:`(`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`p`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`os`}),_jsx(_components.span,{className:`mpunct mtight`,children:`,`}),_jsx(_components.span,{className:`mord mtight`,children:`2`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`}),_jsx(_components.span,{className:`mbin mtight`,children:`+`}),_jsx(_components.span,{className:`mord mtight`,children:`1`}),_jsx(_components.span,{className:`mclose mtight`,children:`)`})]})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.3552em`},children:_jsx(_components.span,{})})})]})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.854em`,verticalAlign:`-0.704em`}}),_jsx(_components.span,{className:`mop`,children:`cos`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.1667em`}}),_jsxs(_components.span,{className:`minner`,children:[_jsx(_components.span,{className:`mopen delimcenter`,style:{top:`0em`},children:_jsx(_components.span,{className:`delimsizing size2`,children:`(`})}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mopen nulldelimiter`}),_jsx(_components.span,{className:`mfrac`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`1.1076em`},children:[_jsxs(_components.span,{style:{top:`-2.296em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord`,children:`1000`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord`,children:`0`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.814em`},children:_jsxs(_components.span,{style:{top:`-2.989em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mtight`,children:`2`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`i`}),_jsx(_components.span,{className:`mord mtight`,children:`/`}),_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mathnormal mtight`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3448em`},children:_jsxs(_components.span,{style:{top:`-2.3488em`,marginLeft:`0em`,marginRight:`0.0714em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.5em`}}),_jsx(_components.span,{className:`sizing reset-size3 size1 mtight`,children:_jsxs(_components.span,{className:`mord mtight`,children:[_jsx(_components.span,{className:`mord mathnormal mtight`,children:`m`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`o`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`d`}),_jsx(_components.span,{className:`mord mathnormal mtight`,children:`e`}),_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0197em`},children:`l`})]})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1512em`},children:_jsx(_components.span,{})})})]})})]})]})})]})})})})})]})]})]}),_jsxs(_components.span,{style:{top:`-3.23em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`frac-line`,style:{borderBottomWidth:`0.04em`}})]}),_jsxs(_components.span,{style:{top:`-3.677em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`p`}),_jsx(_components.span,{className:`mord mathnormal`,children:`os`})]})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.704em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mclose nulldelimiter`})]}),_jsx(_components.span,{className:`mclose delimcenter`,style:{top:`0em`},children:_jsx(_components.span,{className:`delimsizing size2`,children:`)`})})]})]})]})]})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`76`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`76`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` SinusoidalPositionalEncoding`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Sinusoidal Positional Encoding from \"Attention Is All You Need\".`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Properties:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Deterministic (no learned parameters)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Can extrapolate to longer sequences than seen during training`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Relative positions can be represented as linear functions`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`max_len`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`5000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Dropout(dropout)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Create positional encoding matrix`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.zeros(max_len, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` position `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.arange(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, max_len, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dtype`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`torch.float).unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` div_term `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.exp(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.arange(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, d_model, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).float() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`-`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`np.log(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10000.0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`/`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe[:, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`::`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.sin(position `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` div_term)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe[:, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`::`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.cos(position `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` div_term)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe.unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (1, max_len, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Register as buffer (not a parameter, but saved with model)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.register_buffer(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'pe'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, pe)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` x: (batch, seq_len, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.pe[:, :x.size(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout(x)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Visualize positional encoding`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` visualize_positional_encoding`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 128`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` max_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 100`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe_layer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` SinusoidalPositionalEncoding(d_model, max_len, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Get positional encodings`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.zeros(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, max_len, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pe_layer.pe[`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, :max_len, :].numpy()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.figure(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`figsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`12`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.imshow(pe.T, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`aspect`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'auto'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`cmap`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'RdBu'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.xlabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Position'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.ylabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Dimension'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.title(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Sinusoidal Positional Encoding'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.colorbar(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`label`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Encoding Value'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.show()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Show specific dimensions`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` fig, axes `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.subplots(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`figsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`12`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i, ax `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` enumerate`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(axes.flat):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 20`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.plot(pe[:, dim], `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`label`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` (sin)'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.plot(pe[:, dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`label`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'dim `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` (cos)'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_xlabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Position'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_ylabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Value'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.legend()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.set_title(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Dimensions `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`, `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dim`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ax.grid(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`alpha`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.tight_layout()`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.show()`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`visualize_positional_encoding()`})}),`\n`]})})}),`\n`,_jsx(Frame,{children:_jsx(OptimizedImage,{src:`https://mintlify.s3.us-west-1.amazonaws.com/devweeekends/images/courses/deep-learning-mastery/positional-encoding.svg`,alt:`Positional Encoding Visualization`})}),`\n`,_jsx(Heading,{level:`3`,id:`learned-positional-embeddings`,children:`Learned Positional Embeddings`}),`\n`,_jsx(_components.p,{children:`An alternative: learn position embeddings like word embeddings:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`38`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`38`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` LearnedPositionalEncoding`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Learned positional embeddings.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Properties:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Can learn task-specific position patterns`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - Cannot extrapolate beyond max_len`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` - More parameters to learn`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`max_len`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.position_embeddings `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Embedding(max_len, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Dropout(dropout)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Initialize with small values`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.init.normal_(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.position_embeddings.weight, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`std`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.02`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` x: (batch, seq_len, d_model)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` seq_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x.size(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` positions `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.arange(seq_len, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`device`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`x.device)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` position_embeddings `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.position_embeddings(positions)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` position_embeddings`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout(x)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Compare parameter counts`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`sinusoidal `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` SinusoidalPositionalEncoding(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`learned `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` LearnedPositionalEncoding(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Sinusoidal PE params: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`sum`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(p.numel() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` sinusoidal.parameters())`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:,}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Learned PE params: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`sum`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(p.numel() `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` p `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` learned.parameters())`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`:,}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`attention-with-masking`,children:`Attention with Masking`}),`\n`,_jsx(Heading,{level:`3`,id:`causal-autoregressive-mask`,children:`Causal (Autoregressive) Mask`}),`\n`,_jsx(_components.p,{children:`For language models, we need to prevent attending to future positions:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`57`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`57`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` create_causal_mask`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`seq_len`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Create a causal mask to prevent attending to future positions.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` For position i, can only attend to positions 0, 1, ..., i`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.triu(torch.ones(seq_len, seq_len), `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`diagonal`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask.masked_fill(mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'-inf'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` visualize_causal_mask`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`():`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` seq_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 6`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` create_causal_mask(seq_len)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Convert to attention-friendly format (0 = attend, -inf = don't attend)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` viz_mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.zeros_like(mask)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` viz_mask[mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` float`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'-inf'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.figure(`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`figsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.imshow(viz_mask.numpy(), `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`cmap`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Greys'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.xlabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Key Position (can attend to)'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.ylabel(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Query Position'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.title(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Causal Mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:`\\\\n`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`(white = masked, black = can attend)'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` i `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(seq_len):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` j `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` range`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(seq_len):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` color `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'white'`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` viz_mask[i, j] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 1`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'black'`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.text(j, i, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'✓'`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` viz_mask[i,j] `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 0`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` else`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` '✗'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` ha`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'center'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`va`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'center'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`color`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`color, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`fontsize`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`16`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` plt.show()`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`visualize_causal_mask()`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` CausalSelfAttention`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Self-attention with causal masking for autoregressive models.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`num_heads`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`max_len`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`512`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.mha `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` MultiHeadAttention(d_model, num_heads, dropout)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Register causal mask as buffer`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.triu(torch.ones(max_len, max_len), `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`diagonal`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).bool()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.register_buffer(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'mask'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, mask)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`x`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` seq_len `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x.size(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Get appropriate size mask and expand for batch and heads`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.mask[:seq_len, :seq_len]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask.unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# (1, 1, seq_len, seq_len)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` ~`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mask `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Invert: True = can attend, False = masked`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.mha(x, x, x, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`mask)`})]}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`padding-mask`,children:`Padding Mask`}),`\n`,_jsx(_components.p,{children:`For variable-length sequences with padding:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`26`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`26`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` create_padding_mask`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`seq`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`pad_idx`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Create mask to prevent attending to padding tokens.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Args:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` seq: Token indices (batch, seq_len)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` pad_idx: Index of padding token`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Returns:`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` mask: (batch, 1, 1, seq_len) - True where valid, False where padding`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (seq `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`!=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` pad_idx).unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`).unsqueeze(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` mask`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Example`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`batch `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.tensor([`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`5`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`3`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`7`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Padded with 2 zeros`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`6`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`9`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`], `}),_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Padded with 1 zero`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`])`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`padding_mask `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` create_padding_mask(batch, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`pad_idx`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Sequence:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(batch)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:`\\\\n`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`Padding mask:\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(padding_mask.squeeze())`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`complete-attention-based-seq2seq`,children:`Complete Attention-Based Seq2Seq`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`103`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`103`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`class`}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#4EC9B0`},children:` AttentionSeq2Seq`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`nn`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`Module`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Sequence-to-sequence model with attention.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` Unlike vanilla seq2seq, the decoder can look at all encoder states`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` at each decoding step, selecting relevant information.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` __init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`src_vocab_size`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`tgt_vocab_size`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`d_model`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` num_heads`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`num_layers`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0.1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:` super`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`().`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`__init__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`()`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Embeddings`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.src_embedding `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Embedding(src_vocab_size, d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.tgt_embedding `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Embedding(tgt_vocab_size, d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Positional encoding`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.pos_encoding `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` SinusoidalPositionalEncoding(d_model, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dropout)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Encoder (bidirectional LSTM + self-attention)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.encoder_lstm `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.LSTM(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model, d_model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`//`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 2`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, num_layers,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` batch_first`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`bidirectional`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dropout`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.encoder_attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` MultiHeadAttention(d_model, num_heads, dropout)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.encoder_norm `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.LayerNorm(d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Decoder`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.decoder_lstm `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.LSTM(`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` d_model, d_model, num_layers,`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` batch_first`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`True`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`dropout`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`dropout`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Cross-attention (decoder attends to encoder)`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cross_attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` MultiHeadAttention(d_model, num_heads, dropout)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cross_norm `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.LayerNorm(d_model)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Output projection`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.output_projection `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Linear(d_model, tgt_vocab_size)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` nn.Dropout(dropout)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` encode`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`src`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`src_mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Encode source sequence.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Embed and add positional encoding`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.src_embedding(src) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.sqrt(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.pos_encoding(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # LSTM encoding`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x, _ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.encoder_lstm(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Self-attention`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn_out, _ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.encoder_attention(x, x, x, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`src_mask)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.encoder_norm(x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout(attn_out))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` decode`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`tgt`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`encoder_output`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`src_mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"\"\"Decode target sequence.\"\"\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Embed and add positional encoding`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.tgt_embedding(tgt) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`*`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` np.sqrt(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.d_model)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.pos_encoding(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # LSTM decoding`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x, _ `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.decoder_lstm(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Cross-attention to encoder output`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` attn_out, attention_weights `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cross_attention(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x, encoder_output, encoder_output, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`src_mask`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` )`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.cross_norm(x `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`+`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.dropout(attn_out))`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # Project to vocabulary`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` logits `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.output_projection(x)`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` logits, attention_weights`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` `})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` def`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` forward`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`src`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`tgt`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`src_mask`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`None`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`):`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` encoder_output `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.encode(src, src_mask)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` logits, attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` self`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.decode(tgt, encoder_output, src_mask)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` return`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` logits, attention`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:`# Test`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`model `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` AttentionSeq2Seq(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` src_vocab_size`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` tgt_vocab_size`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` d_model`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`256`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` num_heads`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` num_layers`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`2`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`src `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randint(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`10000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`30`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`tgt `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` torch.randint(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`8000`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, (`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`4`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`25`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`))`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`logits, attention `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` model(src, tgt)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Source: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`src.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Target: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`tgt.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Output logits: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`logits.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:`print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Attention weights: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`attention.shape`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`exercises`,children:`Exercises`}),`\n`,_jsxs(AccordionGroup,{children:[_jsxs(Accordion,{title:`Exercise 1: Implement Attention from Scratch`,id:`exercise-1-implement-attention-from-scratch`,children:[_jsx(_components.p,{children:`Implement scaled dot-product attention without using any PyTorch attention functions:`}),_jsxs(_components.ol,{children:[`\n`,_jsx(_components.li,{children:`Implement the forward pass with proper scaling`}),`\n`,_jsx(_components.li,{children:`Add masking support (padding and causal)`}),`\n`,_jsx(_components.li,{children:`Verify gradients flow correctly`}),`\n`,_jsx(_components.li,{children:`Test on a simple sequence copying task`}),`\n`]})]}),_jsxs(Accordion,{title:`Exercise 2: Visualize Real Attention Patterns`,id:`exercise-2-visualize-real-attention-patterns`,children:[_jsx(_components.p,{children:`Using a pre-trained model (or train your own):`}),_jsxs(_components.ol,{children:[`\n`,_jsx(_components.li,{children:`Extract attention weights for various inputs`}),`\n`,_jsx(_components.li,{children:`Create visualizations showing what each head attends to`}),`\n`,_jsx(_components.li,{children:`Identify heads with interpretable patterns`}),`\n`,_jsx(_components.li,{children:`Compare attention patterns for different input types`}),`\n`]})]}),_jsxs(Accordion,{title:`Exercise 3: Compare Attention Variants`,id:`exercise-3-compare-attention-variants`,children:[_jsx(_components.p,{children:`Implement and compare:`}),_jsxs(_components.ol,{children:[`\n`,_jsx(_components.li,{children:`Dot-product attention`}),`\n`,_jsx(_components.li,{children:`Additive (Bahdanau) attention`}),`\n`,_jsx(_components.li,{children:`Multiplicative (Luong) attention`}),`\n`]}),_jsx(_components.p,{children:`Train each on a translation task and compare:`}),_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Training speed`}),`\n`,_jsx(_components.li,{children:`Final BLEU score`}),`\n`,_jsx(_components.li,{children:`Attention patterns`}),`\n`]})]}),_jsxs(Accordion,{title:`Exercise 4: Relative Position Encoding`,id:`exercise-4-relative-position-encoding`,children:[_jsx(_components.p,{children:`Implement relative positional encoding:`}),_jsxs(_components.ol,{children:[`\n`,_jsx(_components.li,{children:`Instead of absolute positions, encode relative distances`}),`\n`,_jsx(_components.li,{children:`Modify attention scores to include position bias`}),`\n`,_jsx(_components.li,{children:`Compare with sinusoidal on long sequences`}),`\n`,_jsx(_components.li,{children:`Test extrapolation to longer sequences`}),`\n`]})]}),_jsxs(Accordion,{title:`Exercise 5: Efficient Attention`,id:`exercise-5-efficient-attention`,children:[_jsx(_components.p,{children:`Implement a more efficient attention mechanism:`}),_jsxs(_components.ol,{children:[`\n`,_jsx(_components.li,{children:`Implement local attention (attend only to nearby positions)`}),`\n`,_jsx(_components.li,{children:`Implement sparse attention patterns`}),`\n`,_jsx(_components.li,{children:`Compare memory usage and speed with full attention`}),`\n`,_jsx(_components.li,{children:`Evaluate impact on model quality`}),`\n`]})]})]}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`training-pitfalls-and-debugging-hints`,children:`Training Pitfalls and Debugging Hints`}),`\n`,_jsxs(Tip,{children:[_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Attention weights sum to 1 but that does not mean they are correct.`}),` A common mistake is interpreting attention weights as “explanation” for model behavior. Attention weights tell you `,_jsx(_components.em,{children:`where`}),` the model looked, not `,_jsx(_components.em,{children:`why`}),` it made its decision. Two models with identical outputs can have completely different attention patterns. Use attention visualizations for debugging intuition, not for mechanistic explanations.`]}),_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Uniform attention is a red flag.`}),` If your attention weights look roughly uniform (every position gets about 1/n attention), the model has not learned meaningful attention patterns. Common causes: (1) learning rate too low, (2) embedding quality is poor (the model cannot distinguish queries from keys), (3) the task does not actually require attention (try a simpler baseline).`]}),_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Attention memory scales quadratically with sequence length.`}),` For sequence length `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`n`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`n`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.4306em`}}),_jsx(_components.span,{className:`mord mathnormal`,children:`n`})]})})]}),` and `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsx(_components.mi,{children:`h`})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`h`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6944em`}}),_jsx(_components.span,{className:`mord mathnormal`,children:`h`})]})})]}),` heads, the attention weight matrix consumes `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`O`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`n`}),_jsx(_components.mn,{children:`2`})]}),_jsx(_components.mo,{children:`⋅`}),_jsx(_components.mi,{children:`h`}),_jsx(_components.mo,{stretchy:`false`,children:`)`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`O(n^2 \\\\cdot h)`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.0641em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0278em`},children:`O`}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`n`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8141em`},children:_jsxs(_components.span,{style:{top:`-3.063em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mtight`,children:`2`})})]})})})})})]}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}}),_jsx(_components.span,{className:`mbin`,children:`⋅`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2222em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord mathnormal`,children:`h`}),_jsx(_components.span,{className:`mclose`,children:`)`})]})]})]}),` memory. At `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mi,{children:`n`}),_jsx(_components.mo,{children:`=`}),_jsx(_components.mn,{children:`4096`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`n = 4096`})]})})}),_jsxs(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:[_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.4306em`}}),_jsx(_components.span,{className:`mord mathnormal`,children:`n`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}}),_jsx(_components.span,{className:`mrel`,children:`=`}),_jsx(_components.span,{className:`mspace`,style:{marginRight:`0.2778em`}})]}),_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.6444em`}}),_jsx(_components.span,{className:`mord`,children:`4096`})]})]})]}),` with 12 heads in float32, that is about 750 MB per layer just for attention scores. If you run out of memory during training, sequence length is almost always the bottleneck — reduce it before reducing batch size.`]}),_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Mask shape mismatches`}),`: The most frustrating attention bug. PyTorch broadcasting rules mean a wrong mask shape often produces no error but silently computes the wrong thing. Always verify: padding masks should broadcast across heads and queries, causal masks should be square and lower-triangular, and the combination should use logical AND. Print `,_jsx(_components.code,{children:`mask.shape`}),` and `,_jsx(_components.code,{children:`scores.shape`}),` during debugging and verify they broadcast correctly.`]}),_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Multi-head attention head collapse`}),`: Sometimes all heads learn nearly identical attention patterns, wasting capacity. This happens with high dropout on attention weights or very small `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`d_k`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})]})})]}),` (dimension per head). Monitor head diversity by computing the cosine similarity between attention patterns of different heads — if it is consistently above 0.9, you have collapse. Fix: reduce attention dropout, increase `,_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsx(_components.mrow,{children:_jsxs(_components.msub,{children:[_jsx(_components.mi,{children:`d`}),_jsx(_components.mi,{children:`k`})]})}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`d_k`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`0.8444em`,verticalAlign:`-0.15em`}}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,children:`d`}),_jsx(_components.span,{className:`msupsub`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsx(_components.span,{className:`vlist`,style:{height:`0.3361em`},children:_jsxs(_components.span,{style:{top:`-2.55em`,marginLeft:`0em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.0315em`},children:`k`})})]})}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.15em`},children:_jsx(_components.span,{})})})]})})]})]})})]}),`, or add auxiliary losses that encourage head diversity.`]})]}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`key-takeaways`,children:`Key Takeaways`}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Concept`}),_jsx(_components.th,{children:`Key Insight`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Attention`})}),_jsx(_components.td,{children:`Weighted combination of values based on query-key similarity`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Dot-Product`})}),_jsxs(_components.td,{children:[_jsxs(_components.span,{className:`katex`,children:[_jsx(_components.span,{className:`katex-mathml`,children:_jsx(_components.math,{xmlns:`http://www.w3.org/1998/Math/MathML`,children:_jsxs(_components.semantics,{children:[_jsxs(_components.mrow,{children:[_jsx(_components.mtext,{children:`softmax`}),_jsx(_components.mo,{stretchy:`false`,children:`(`}),_jsx(_components.mi,{children:`Q`}),_jsxs(_components.msup,{children:[_jsx(_components.mi,{children:`K`}),_jsx(_components.mi,{children:`T`})]}),_jsx(_components.mi,{mathvariant:`normal`,children:`/`}),_jsx(_components.msqrt,{children:_jsx(_components.mi,{children:`d`})}),_jsx(_components.mo,{stretchy:`false`,children:`)`}),_jsx(_components.mi,{children:`V`})]}),_jsx(_components.annotation,{encoding:`application/x-tex`,children:`\\\\text{softmax}(QK^T/\\\\sqrt{d})V`})]})})}),_jsx(_components.span,{className:`katex-html`,\"aria-hidden\":`true`,children:_jsxs(_components.span,{className:`base`,children:[_jsx(_components.span,{className:`strut`,style:{height:`1.1822em`,verticalAlign:`-0.25em`}}),_jsx(_components.span,{className:`mord text`,children:_jsx(_components.span,{className:`mord`,children:`softmax`})}),_jsx(_components.span,{className:`mopen`,children:`(`}),_jsx(_components.span,{className:`mord mathnormal`,children:`Q`}),_jsxs(_components.span,{className:`mord`,children:[_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.0715em`},children:`K`}),_jsx(_components.span,{className:`msupsub`,children:_jsx(_components.span,{className:`vlist-t`,children:_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.8413em`},children:_jsxs(_components.span,{style:{top:`-3.063em`,marginRight:`0.05em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`2.7em`}}),_jsx(_components.span,{className:`sizing reset-size6 size3 mtight`,children:_jsx(_components.span,{className:`mord mathnormal mtight`,style:{marginRight:`0.1389em`},children:`T`})})]})})})})})]}),_jsx(_components.span,{className:`mord`,children:`/`}),_jsx(_components.span,{className:`mord sqrt`,children:_jsxs(_components.span,{className:`vlist-t vlist-t2`,children:[_jsxs(_components.span,{className:`vlist-r`,children:[_jsxs(_components.span,{className:`vlist`,style:{height:`0.9322em`},children:[_jsxs(_components.span,{className:`svg-align`,style:{top:`-3em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`mord`,style:{paddingLeft:`0.833em`},children:_jsx(_components.span,{className:`mord mathnormal`,children:`d`})})]}),_jsxs(_components.span,{style:{top:`-2.8922em`},children:[_jsx(_components.span,{className:`pstrut`,style:{height:`3em`}}),_jsx(_components.span,{className:`hide-tail`,style:{minWidth:`0.853em`,height:`1.08em`},children:_jsx(_components.svg,{xmlns:`http://www.w3.org/2000/svg`,width:`400em`,height:`1.08em`,viewBox:`0 0 400000 1080`,preserveAspectRatio:`xMinYMin slice`,children:_jsx(_components.path,{d:`M95,702\nc-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14\nc0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54\nc44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10\ns173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429\nc69,-144,104.5,-217.7,106.5,-221\nl0 -0\nc5.3,-9.3,12,-14,20,-14\nH400000v40H845.2724\ns-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7\nc-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z\nM834 80h400000v40h-400000z`})})})]})]}),_jsx(_components.span,{className:`vlist-s`,children:``})]}),_jsx(_components.span,{className:`vlist-r`,children:_jsx(_components.span,{className:`vlist`,style:{height:`0.1078em`},children:_jsx(_components.span,{})})})]})}),_jsx(_components.span,{className:`mclose`,children:`)`}),_jsx(_components.span,{className:`mord mathnormal`,style:{marginRight:`0.2222em`},children:`V`})]})})]}),` — simple and efficient`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Multi-Head`})}),_jsx(_components.td,{children:`Multiple attention functions for different relationship types`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Self-Attention`})}),_jsx(_components.td,{children:`Sequence attends to itself — O(1) dependency paths`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Positional Encoding`})}),_jsx(_components.td,{children:`Add position information since attention is permutation-invariant`})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.strong,{children:`Masking`})}),_jsx(_components.td,{children:`Causal for autoregressive, padding for variable lengths`})]})]})]}),`\n`,_jsx(Tip,{children:_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`The attention mechanism is the foundation of modern NLP.`}),` Understanding it deeply will help you grasp Transformers, BERT, GPT, and virtually all state-of-the-art language models.`]})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`what’s-next`,children:`What’s Next`}),`\n`,_jsx(CardGroup,{cols:1,children:_jsx(Card,{title:`Module 11: Transformers`,icon:`bolt`,href:`/courses/deep-learning-mastery/11-transformers`,children:_jsx(_components.p,{children:`Build the complete Transformer architecture — combine attention with feed-forward networks, layer normalization, and residual connections to create the model that revolutionized NLP.`})})})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Attention Mechanism","sidebarTitle":"Attention","description":"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers","icon":"eye","href":"/courses/deep-learning-mastery/10-attention"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Attention Mechanism","slug":"attention-mechanism","depth":1,"children":[]},{"title":"The Bottleneck Problem","slug":"the-bottleneck-problem","depth":2,"children":[{"title":"Evidence of the Problem","slug":"evidence-of-the-problem","depth":3,"children":[]}]},{"title":"The Attention Solution","slug":"the-attention-solution","depth":2,"children":[{"title":"Intuition: Looking Back at the Source","slug":"intuition-looking-back-at-the-source","depth":3,"children":[]}]},{"title":"Attention Mechanisms in Detail","slug":"attention-mechanisms-in-detail","depth":2,"children":[{"title":"Dot-Product Attention","slug":"dot-product-attention","depth":3,"children":[]},{"title":"Visualizing Attention Weights","slug":"visualizing-attention-weights","depth":3,"children":[]}]},{"title":"Types of Attention","slug":"types-of-attention","depth":2,"children":[{"title":"Additive (Bahdanau) Attention","slug":"additive-bahdanau-attention","depth":3,"children":[]},{"title":"Multiplicative (Luong) Attention","slug":"multiplicative-luong-attention","depth":3,"children":[]}]},{"title":"Self-Attention","slug":"self-attention","depth":2,"children":[{"title":"Attending to the Same Sequence","slug":"attending-to-the-same-sequence","depth":3,"children":[]},{"title":"Why Self-Attention Matters","slug":"why-self-attention-matters","depth":3,"children":[]}]},{"title":"Multi-Head Attention","slug":"multi-head-attention","depth":2,"children":[{"title":"Multiple Attention “Perspectives”","slug":"multiple-attention-“perspectives”","depth":3,"children":[]},{"title":"Visualizing Multi-Head Attention","slug":"visualizing-multi-head-attention","depth":3,"children":[]}]},{"title":"Positional Encoding","slug":"positional-encoding","depth":2,"children":[{"title":"The Problem: Attention is Permutation-Invariant","slug":"the-problem-attention-is-permutation-invariant","depth":3,"children":[]},{"title":"Sinusoidal Positional Encoding","slug":"sinusoidal-positional-encoding","depth":3,"children":[]},{"title":"Learned Positional Embeddings","slug":"learned-positional-embeddings","depth":3,"children":[]}]},{"title":"Attention with Masking","slug":"attention-with-masking","depth":2,"children":[{"title":"Causal (Autoregressive) Mask","slug":"causal-autoregressive-mask","depth":3,"children":[]},{"title":"Padding Mask","slug":"padding-mask","depth":3,"children":[]}]},{"title":"Complete Attention-Based Seq2Seq","slug":"complete-attention-based-seq2seq","depth":2,"children":[]},{"title":"Exercises","slug":"exercises","depth":2,"children":[]},{"title":"Training Pitfalls and Debugging Hints","slug":"training-pitfalls-and-debugging-hints","depth":2,"children":[]},{"title":"Key Takeaways","slug":"key-takeaways","depth":2,"children":[]},{"title":"What’s Next","slug":"what’s-next","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"devweeekends","siteName":"Dev Weekends","title":"Attention Mechanism - Dev Weekends","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-05-16T10:34:49.211Z","originalFileLocation":"courses/deep-learning-mastery/10-attention.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers\">\u003cmeta data-mint-head name=\"application-name\" content=\"Dev Weekends\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Attention Mechanism - Dev Weekends\">\u003cmeta data-mint-head property=\"og:description\" content=\"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Dev Weekends\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://devweeekends.mintlify.site/_mintlify/api/og/v2?division=Deep+Learning+Mastery\u0026amp;title=Attention+Mechanism\u0026amp;description=Learn+how+attention+enables+neural+networks+to+focus+on+relevant+information+-+the+foundation+of+modern+NLP+and+transformers\u0026amp;theme=97ffacf68ee2a7d59a7fcf65\u0026amp;signature=30e15b7115482c914de90cc104ffe21f3d355f14ebe9954ea7e6b35f4117d913\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Attention Mechanism - Dev Weekends\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://devweeekends.mintlify.site/_mintlify/api/og/v2?division=Deep+Learning+Mastery\u0026amp;title=Attention+Mechanism\u0026amp;description=Learn+how+attention+enables+neural+networks+to+focus+on+relevant+information+-+the+foundation+of+modern+NLP+and+transformers\u0026amp;theme=97ffacf68ee2a7d59a7fcf65\u0026amp;signature=30e15b7115482c914de90cc104ffe21f3d355f14ebe9954ea7e6b35f4117d913\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/devweeekends/VN0XII-rWbxEMed4/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Dev Weekends\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#000000\">\u003clink data-mint-head rel=\"canonical\" href=\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/courses/deep-learning-mastery/10-attention.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://resources.devweekends.com/#organization\",\"name\":\"Dev Weekends\",\"url\":\"https://resources.devweekends.com\"},{\"@type\":\"WebSite\",\"@id\":\"https://resources.devweekends.com#website\",\"name\":\"Dev Weekends\",\"url\":\"https://resources.devweekends.com\",\"publisher\":{\"@id\":\"https://resources.devweekends.com/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention#webpage\",\"url\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention\",\"name\":\"Attention Mechanism\",\"description\":\"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers\",\"dateModified\":\"2026-05-16T10:34:49.211Z\",\"isPartOf\":{\"@id\":\"https://resources.devweekends.com#website\"},\"breadcrumb\":{\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Deep Learning Mastery\",\"item\":\"https://resources.devweekends.com/courses/deep-learning-mastery/00-overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Attention Mechanism\",\"item\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention#article\",\"headline\":\"Attention Mechanism\",\"name\":\"Attention Mechanism\",\"description\":\"Learn how attention enables neural networks to focus on relevant information - the foundation of modern NLP and transformers\",\"url\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention\",\"mainEntityOfPage\":{\"@id\":\"https://resources.devweekends.com/courses/deep-learning-mastery/10-attention#webpage\"},\"dateModified\":\"2026-05-16T10:34:49.211Z\",\"publisher\":{\"@id\":\"https://resources.devweekends.com/#organization\"},\"isPartOf\":{\"@id\":\"https://resources.devweekends.com#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Dev Weekends\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Dev Weekends\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/courses/deep-learning-mastery/10-attention.md","siteName":"Dev Weekends","nav":{"global":{"anchors":[{"anchor":"DevWeekends","icon":"globe","href":"https://devweekends.com"}]},"anchors":[{"anchor":"Resources","icon":"book-open","pages":[{"title":"Resources","href":""}]},{"anchor":"Interview Prep","icon":"clipboard-check","pages":[{"title":"Interview Prep","href":"/resources/interview-experiences"}]},{"anchor":"DSA Patterns","icon":"code","pages":[{"title":"DSA Patterns","href":"/dsa-patterns/overview"}]},{"anchor":"Competitive Programming","icon":"trophy","pages":[{"title":"Competitive Programming","href":"/courses/competitive-programming/00-overview"}]},{"anchor":"Courses","icon":"graduation-cap","groups":[{"group":"Overview","pages":[{"title":"All Courses","icon":"graduation-cap","href":"/courses/overview"}]},{"group":"Testing \u0026 Code Quality","icon":"vial","pages":[{"title":"Sonarqube Mastery","sidebarTitle":"Sonarqube","icon":"magnifying-glass-chart","href":"/courses/testing-tools/sonarqube"},{"title":"Playwright Mastery","sidebarTitle":"Playwright","icon":"masks-theater","href":"/courses/testing-tools/playwright"},{"title":"Mocha Mastery","sidebarTitle":"Mocha","icon":"coffee-bean","href":"/courses/testing-tools/mocha"},{"title":"Jest Mastery","sidebarTitle":"Jest","icon":"jester-hat","href":"/courses/testing-tools/jest"}]},{"group":"Crash Courses","icon":"bolt","pages":[{"group":"C++","icon":"brackets-curly","pages":[{"title":"C++ Crash Course","sidebarTitle":"Overview","icon":"code","href":"/courses/cpp-crash-course/overview"},{"title":"C++ Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/cpp-crash-course/fundamentals"},{"title":"Pointers \u0026 Memory","sidebarTitle":"2. Memory Mastery","icon":"memory","href":"/courses/cpp-crash-course/pointers-memory"},{"title":"Object-Oriented Programming","sidebarTitle":"3. OOP","icon":"object-group","href":"/courses/cpp-crash-course/oop"},{"title":"Standard Template Library","sidebarTitle":"4. The STL","icon":"layer-group","href":"/courses/cpp-crash-course/stl"},{"title":"Modern C++ Features","sidebarTitle":"5. Modern C++","icon":"wand-magic-sparkles","href":"/courses/cpp-crash-course/modern-cpp"}]},{"group":"Java","icon":"mug-hot","pages":[{"title":"Java Crash Course","sidebarTitle":"Overview","icon":"mug-hot","href":"/courses/java-crash-course/overview"},{"title":"Java Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/java-crash-course/fundamentals"},{"title":"Object-Oriented Programming","sidebarTitle":"2. OOP","icon":"object-group","href":"/courses/java-crash-course/oop"},{"title":"Collections Framework","sidebarTitle":"3. Collections","icon":"layer-group","href":"/courses/java-crash-course/collections"},{"title":"Concurrency","sidebarTitle":"4. Concurrency","icon":"bolt","href":"/courses/java-crash-course/concurrency"},{"title":"Modern Java Features","sidebarTitle":"5. Modern Java","icon":"wand-magic-sparkles","href":"/courses/java-crash-course/modern-java"}]},{"group":"Spring Boot \u0026 Microservices","icon":"leaf","pages":[{"title":"Spring Boot \u0026 Microservices","sidebarTitle":"Overview","icon":"leaf","href":"/courses/spring-boot-microservices/00-overview"},{"title":"Spring Boot Quickstart","icon":"bolt","href":"/courses/spring-boot-microservices/01-spring-boot-quickstart"},{"title":"Spring MVC \u0026 REST","icon":"server","href":"/courses/spring-boot-microservices/02-spring-mvc-rest"},{"title":"Data Persistence","icon":"database","href":"/courses/spring-boot-microservices/03-data-persistence"},{"title":"Deep Dive: Spring AOP","icon":"microscope","href":"/courses/spring-boot-microservices/04-spring-aop"},{"title":"Testing Mastery","icon":"vial","href":"/courses/spring-boot-microservices/05-spring-testing"},{"title":"Foundations","icon":"shapes","href":"/courses/spring-boot-microservices/06-microservices-foundations"},{"title":"Service Discovery","icon":"compass","href":"/courses/spring-boot-microservices/07-service-discovery"},{"title":"API Gateway","icon":"door-open","href":"/courses/spring-boot-microservices/08-api-gateway"},{"title":"Config Management","icon":"gears","href":"/courses/spring-boot-microservices/09-config-management"},{"title":"Resilience \u0026 Fault Tolerance","icon":"shield-halved","href":"/courses/spring-boot-microservices/10-resilience"},{"title":"Observability","icon":"monitor-waveform","href":"/courses/spring-boot-microservices/11-observability"},{"title":"Event Driven Architecture","icon":"satellite-dish","href":"/courses/spring-boot-microservices/12-event-driven"},{"title":"Containerization","icon":"docker","href":"/courses/spring-boot-microservices/13-containerization"},{"title":"Capstone Project","icon":"flag-checkered","href":"/courses/spring-boot-microservices/14-capstone"}]},{"group":"JavaScript","icon":"js","pages":[{"title":"JavaScript Crash Course","sidebarTitle":"Overview","icon":"js","href":"/courses/javascript-crash-course/overview"},{"title":"JavaScript Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/javascript-crash-course/fundamentals"},{"title":"Functions \u0026 Scope","sidebarTitle":"2. Functions \u0026 Scope","icon":"function","href":"/courses/javascript-crash-course/functions-scope"},{"title":"Objects \u0026 Prototypes","sidebarTitle":"3. Objects \u0026 Prototypes","icon":"object-group","href":"/courses/javascript-crash-course/objects-prototypes"},{"title":"Async JavaScript","sidebarTitle":"4. Async JavaScript","icon":"clock","href":"/courses/javascript-crash-course/async"},{"title":"Modern JavaScript (ES6+)","sidebarTitle":"5. Modern JavaScript","icon":"wand-magic-sparkles","href":"/courses/javascript-crash-course/modern-js"},{"title":"DOM \u0026 Browser APIs","sidebarTitle":"6. DOM \u0026 Browser","icon":"browser","href":"/courses/javascript-crash-course/dom-browser"}]},{"group":"TypeScript","icon":"file-code","pages":[{"title":"TypeScript Crash Course","sidebarTitle":"Overview","icon":"code","href":"/courses/typescript-crash-course/overview"},{"title":"TypeScript Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/typescript-crash-course/fundamentals"},{"title":"Functions \u0026 Types","sidebarTitle":"2. Functions \u0026 Types","icon":"function","href":"/courses/typescript-crash-course/functions-types"},{"title":"Objects \u0026 Interfaces","sidebarTitle":"3. Objects \u0026 Interfaces","icon":"object-group","href":"/courses/typescript-crash-course/objects-interfaces"},{"title":"Classes \u0026 OOP","sidebarTitle":"4. Classes \u0026 OOP","icon":"sitemap","href":"/courses/typescript-crash-course/classes-oop"},{"title":"Advanced Types","sidebarTitle":"5. Advanced Types","icon":"layer-group","href":"/courses/typescript-crash-course/advanced-types"},{"title":"Generics Deep Dive","sidebarTitle":"6. Generics","icon":"puzzle-piece","href":"/courses/typescript-crash-course/generics"},{"title":"Modules \u0026 Configuration","sidebarTitle":"7. Modules \u0026 Config","icon":"gear","href":"/courses/typescript-crash-course/modules-config"}]},{"group":"Python","icon":"python","pages":[{"title":"Python Crash Course","sidebarTitle":"Overview","icon":"python","href":"/courses/python-crash-course/overview"},{"title":"Python Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/python-crash-course/fundamentals"},{"title":"Data Structures","sidebarTitle":"2. Data Structures","icon":"layer-group","href":"/courses/python-crash-course/data-structures"},{"title":"Object-Oriented Programming","sidebarTitle":"3. OOP","icon":"object-group","href":"/courses/python-crash-course/oop"},{"title":"Modules \u0026 Packages","sidebarTitle":"4. Modules","icon":"box","href":"/courses/python-crash-course/modules-packages"},{"title":"Advanced Python","sidebarTitle":"5. Advanced","icon":"wand-magic-sparkles","href":"/courses/python-crash-course/advanced"}]},{"group":"Node.js","icon":"node","pages":[{"title":"01. Introduction \u0026 Setup","href":"/courses/node-crash-course/01-intro-setup"},{"title":"02. Modules System","href":"/courses/node-crash-course/02-modules"},{"title":"03. File System","href":"/courses/node-crash-course/03-file-system"},{"title":"04. Events \u0026 EventEmitter","href":"/courses/node-crash-course/04-events"},{"title":"05. HTTP Module \u0026 Web Server","href":"/courses/node-crash-course/05-http-server"},{"title":"06. Streams \u0026 Buffers","href":"/courses/node-crash-course/06-streams-buffers"},{"title":"07. NPM (Node Package Manager)","href":"/courses/node-crash-course/07-npm"},{"title":"07b. Building and Publishing an NPM Package","href":"/courses/node-crash-course/07b-building-npm-package"},{"title":"08. Express.js Basics","href":"/courses/node-crash-course/08-express-basics"},{"title":"09. Building a REST API","href":"/courses/node-crash-course/09-rest-api"},{"title":"10. Deployment \u0026 Best Practices","href":"/courses/node-crash-course/10-deployment"},{"title":"11. Authentication with JWT","href":"/courses/node-crash-course/11-auth-jwt"},{"title":"12. MongoDB \u0026 Mongoose ORM","href":"/courses/node-crash-course/12-mongo-mongoose"},{"title":"13. PostgreSQL \u0026 Prisma ORM","href":"/courses/node-crash-course/13-postgres-prisma"},{"title":"14. Error Handling \u0026 Debugging","href":"/courses/node-crash-course/14-error-handling"},{"title":"15. Testing with Jest","href":"/courses/node-crash-course/15-testing"},{"title":"16. Real-Time with WebSockets","href":"/courses/node-crash-course/16-websockets"},{"title":"17. Security Best Practices","href":"/courses/node-crash-course/17-security"},{"title":"18. Performance \u0026 Caching","href":"/courses/node-crash-course/18-performance"},{"title":"19. Microservices Architecture","href":"/courses/node-crash-course/19-microservices"},{"title":"20. TypeScript with Node.js","href":"/courses/node-crash-course/20-typescript"},{"title":"21. Worker Threads \u0026 Child Processes","href":"/courses/node-crash-course/21-worker-threads"},{"title":"22. Capstone Project: Full-Stack API","href":"/courses/node-crash-course/22-capstone"}]},{"group":"React","icon":"react","pages":[{"title":"01. Introduction to React \u0026 JSX","href":"/courses/react-crash-course/01-intro-jsx"},{"title":"02. Components \u0026 Props","href":"/courses/react-crash-course/02-components-props"},{"title":"03. State \u0026 useState Hook","href":"/courses/react-crash-course/03-state-hook"},{"title":"04. Handling Events","href":"/courses/react-crash-course/04-events"},{"title":"05. Lists \u0026 Keys","href":"/courses/react-crash-course/05-lists-keys"},{"title":"06. Forms \u0026 Controlled Components","href":"/courses/react-crash-course/06-forms"},{"title":"07. useEffect \u0026 Side Effects","href":"/courses/react-crash-course/07-useeffect"},{"title":"08. Context API","href":"/courses/react-crash-course/08-context-api"},{"title":"09. React Router","href":"/courses/react-crash-course/09-react-router"},{"title":"10. Performance Optimization","href":"/courses/react-crash-course/10-optimization"},{"title":"11. Authentication \u0026 Protected Routes","href":"/courses/react-crash-course/11-authentication"},{"title":"12. Redux \u0026 State Management","href":"/courses/react-crash-course/12-redux-state"}]},{"group":"React Native","icon":"mobile","pages":[{"title":"React Native Enterprise Mastery","icon":"mobile","href":"/courses/react-native-crash-course/overview"},{"title":"01. Introduction \u0026 Architecture","href":"/courses/react-native-crash-course/01-intro-architecture"},{"title":"02. Environment Setup","href":"/courses/react-native-crash-course/02-environment-setup"},{"title":"03. Project Structure \u0026 Best Practices","href":"/courses/react-native-crash-course/03-project-structure"},{"title":"04. TypeScript in React Native","href":"/courses/react-native-crash-course/04-typescript"},{"title":"05. Core Components Deep Dive","href":"/courses/react-native-crash-course/05-core-components"},{"title":"6. Styling \u0026 Theming","href":"/courses/react-native-crash-course/06-styling-theming"},{"title":"7. Flexbox Mastery","href":"/courses/react-native-crash-course/07-flexbox-mastery"},{"title":"8. Navigation Fundamentals","href":"/courses/react-native-crash-course/08-navigation-fundamentals"},{"title":"9. Advanced Navigation","href":"/courses/react-native-crash-course/09-navigation-advanced"},{"title":"10. State Management","href":"/courses/react-native-crash-course/10-state-management"},{"title":"11. Redux \u0026 Zustand","href":"/courses/react-native-crash-course/11-redux-zustand"},{"title":"15. React Query \u0026 Data Fetching","href":"/courses/react-native-crash-course/15-react-query"},{"title":"27. Unit Testing","href":"/courses/react-native-crash-course/27-unit-testing"},{"title":"31. Security Best Practices","href":"/courses/react-native-crash-course/31-security"},{"title":"35. Profiling \u0026 Monitoring","href":"/courses/react-native-crash-course/35-profiling"},{"title":"40. CI/CD Pipelines","href":"/courses/react-native-crash-course/40-ci-cd"},{"title":"51. Capstone Project","href":"/courses/react-native-crash-course/51-capstone"}]},{"group":"MongoDB","icon":"database","pages":[{"title":"01. Introduction to NoSQL \u0026 MongoDB","href":"/courses/mongodb-crash-course/01-intro-nosql"},{"title":"02. Installation \u0026 MongoDB Compass","href":"/courses/mongodb-crash-course/02-setup-compass"},{"title":"03. Databases \u0026 Collections","href":"/courses/mongodb-crash-course/03-databases-collections"},{"title":"04. CRUD: Create","href":"/courses/mongodb-crash-course/04-crud-create"},{"title":"05. CRUD: Read (Query Operators)","href":"/courses/mongodb-crash-course/05-crud-read"},{"title":"06. CRUD: Update","href":"/courses/mongodb-crash-course/06-crud-update"},{"title":"07. CRUD: Delete","href":"/courses/mongodb-crash-course/07-crud-delete"},{"title":"08. Aggregation Framework","href":"/courses/mongodb-crash-course/08-aggregation"},{"title":"09. Indexing \u0026 Performance","href":"/courses/mongodb-crash-course/09-indexing"},{"title":"10. Mongoose (ODM) Basics","href":"/courses/mongodb-crash-course/10-mongoose"}]},{"group":"Angular Master Course","icon":"angular","pages":[{"title":"Angular Master Course","icon":"angular","href":"/courses/angular-crash-course/00-overview"},{"title":"01. Introduction \u0026 Setup","icon":"rocket","href":"/courses/angular-crash-course/01-introduction"},{"title":"02. Components \u0026 Templates","icon":"puzzle-piece","href":"/courses/angular-crash-course/02-components"},{"title":"03. Directives \u0026 Pipes","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/03-directives-pipes"},{"title":"04. Services \u0026 Dependency Injection","icon":"syringe","href":"/courses/angular-crash-course/04-services-di"},{"title":"05. Angular Signals","icon":"signal","href":"/courses/angular-crash-course/05-signals"},{"title":"06. Routing \u0026 Navigation","icon":"route","href":"/courses/angular-crash-course/06-routing"},{"title":"07. Angular Forms","icon":"input-text","href":"/courses/angular-crash-course/07-forms"},{"title":"08. HTTP Client \u0026 RxJS","icon":"server","href":"/courses/angular-crash-course/08-http-rxjs"},{"title":"09. RxJS Deep Dive","icon":"arrows-rotate","href":"/courses/angular-crash-course/09-rxjs"},{"title":"10. Change Detection \u0026 Performance","icon":"gauge-high","href":"/courses/angular-crash-course/10-change-detection"},{"title":"11. Testing in Angular","icon":"vial","href":"/courses/angular-crash-course/11-testing"},{"title":"12. Advanced Patterns","icon":"diagram-project","href":"/courses/angular-crash-course/12-advanced-patterns"},{"title":"13. Server-Side Rendering","icon":"server","href":"/courses/angular-crash-course/13-ssr"},{"title":"14. Security Best Practices","icon":"shield-halved","href":"/courses/angular-crash-course/14-security"},{"title":"15. Capstone Project","icon":"rocket","href":"/courses/angular-crash-course/15-capstone"},{"title":"16. Standalone Components Deep Dive","icon":"cube","href":"/courses/angular-crash-course/16-standalone"},{"title":"17. NgRx State Management","icon":"database","href":"/courses/angular-crash-course/17-ngrx"},{"title":"18. Angular Animations","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/18-animations"},{"title":"19. Internationalization (i18n)","icon":"globe","href":"/courses/angular-crash-course/19-i18n"},{"title":"20. Accessibility (a11y)","icon":"universal-access","href":"/courses/angular-crash-course/20-accessibility"},{"title":"21. Angular Material \u0026 CDK","icon":"palette","href":"/courses/angular-crash-course/21-material"},{"title":"22. PWA \u0026 Service Workers","icon":"mobile-screen","href":"/courses/angular-crash-course/22-pwa"},{"title":"23. Deployment \u0026 CI/CD","icon":"rocket","href":"/courses/angular-crash-course/23-deployment"},{"title":"24. Micro-frontends with Module Federation","icon":"cubes","href":"/courses/angular-crash-course/24-micro-frontends"},{"title":"25. Nx Monorepo","icon":"boxes-stacked","href":"/courses/angular-crash-course/25-nx-monorepo"},{"title":"26. Performance Optimization","icon":"gauge-high","href":"/courses/angular-crash-course/26-performance"},{"title":"27. Content Projection","icon":"layer-group","href":"/courses/angular-crash-course/27-content-projection"},{"title":"28. Dynamic Components","icon":"wand-magic-sparkles","href":"/courses/angular-crash-course/28-dynamic-components"},{"title":"29. Custom Schematics","icon":"toolbox","href":"/courses/angular-crash-course/29-custom-schematics"},{"title":"30. Error Handling","icon":"triangle-exclamation","href":"/courses/angular-crash-course/30-error-handling"},{"title":"31. Interview Questions","icon":"comments-question","href":"/courses/angular-crash-course/31-interview-questions"},{"title":"32. Best Practices","icon":"award","href":"/courses/angular-crash-course/32-best-practices"},{"title":"33. Enterprise Capstone Project","icon":"building","href":"/courses/angular-crash-course/33-enterprise-capstone"}]}]},{"group":"AI Engineering","icon":"robot","pages":[{"title":"AI Engineering","icon":"robot","href":"/ai-engineering/overview"},{"title":"Python Crash Course","icon":"python","href":"/ai-engineering/python-crash-course"},{"title":"FastAPI Crash Course","icon":"bolt","href":"/ai-engineering/fastapi-crash-course"},{"title":"Databases \u0026 ORM Crash Course","icon":"database","href":"/ai-engineering/databases-crash-course"},{"title":"LLM Fundamentals","icon":"brain","href":"/ai-engineering/llm-fundamentals"},{"title":"Prompt Engineering","icon":"wand-magic-sparkles","href":"/ai-engineering/prompt-engineering"},{"title":"OpenAI API","icon":"bolt","href":"/ai-engineering/openai-api"},{"title":"Structured Outputs","icon":"brackets-curly","href":"/ai-engineering/structured-outputs"},{"title":"Embeddings Deep Dive","icon":"vector-square","href":"/ai-engineering/embeddings"},{"title":"AI Streaming","icon":"water","href":"/ai-engineering/streaming"},{"title":"Vector Databases","icon":"database","href":"/ai-engineering/vector-databases"},{"title":"RAG Systems","icon":"magnifying-glass","href":"/ai-engineering/rag"},{"title":"Types of RAG","icon":"layer-group","href":"/ai-engineering/rag-types"},{"title":"Fine-Tuning LLMs","icon":"sliders","href":"/ai-engineering/fine-tuning"},{"title":"AI Agents","icon":"robot","href":"/ai-engineering/agents"},{"title":"Tool Calling \u0026 Function Calling","icon":"wrench","href":"/ai-engineering/tool-calling"},{"title":"LLM Memory Systems","icon":"brain","href":"/ai-engineering/llm-memory-systems"},{"title":"Multi-Agent Design Patterns","icon":"users-gear","href":"/ai-engineering/multi-agent-patterns"},{"title":"LangChain","icon":"link","href":"/ai-engineering/langchain"},{"title":"LangGraph","icon":"diagram-project","href":"/ai-engineering/langgraph"},{"title":"Model Context Protocol (MCP)","icon":"plug","href":"/ai-engineering/mcp"},{"title":"Agentic Architecture","icon":"sitemap","href":"/ai-engineering/agentic-architecture"},{"title":"Open Source \u0026 Local LLMs","icon":"server","href":"/ai-engineering/open-source-llms"},{"title":"Multimodal AI","icon":"photo-film","href":"/ai-engineering/multimodal-ai"},{"title":"DSPy Framework","icon":"code","href":"/ai-engineering/dspy"},{"title":"Prompt Management \u0026 Versioning","icon":"code-branch","href":"/ai-engineering/prompt-management"},{"title":"LLM Caching","icon":"database","href":"/ai-engineering/llm-caching"},{"title":"Error Handling \u0026 Resilience","icon":"shield-halved","href":"/ai-engineering/error-handling"},{"title":"Batch Processing \u0026 Rate Limiting","icon":"layer-group","href":"/ai-engineering/batch-processing"},{"title":"LLM Fallbacks \u0026 Model Routing","icon":"arrows-split-up-and-left","href":"/ai-engineering/llm-fallbacks"},{"title":"AI Observability \u0026 Monitoring","icon":"chart-line","href":"/ai-engineering/observability-monitoring"},{"title":"AI Security \u0026 Guardrails","icon":"shield-halved","href":"/ai-engineering/ai-security-guardrails"},{"title":"Cost Optimization \u0026 Token Management","icon":"coins","href":"/ai-engineering/cost-optimization"},{"title":"Function Calling Deep Dive","icon":"function","href":"/ai-engineering/function-calling"},{"title":"LLM Orchestration","icon":"sitemap","href":"/ai-engineering/llm-orchestration"},{"title":"Semantic Search \u0026 Retrieval","icon":"magnifying-glass","href":"/ai-engineering/semantic-search"},{"title":"Context Window Management","icon":"window-maximize","href":"/ai-engineering/context-management"},{"title":"Testing LLM Applications","icon":"flask-vial","href":"/ai-engineering/llm-testing"},{"title":"Production Logging \u0026 Debugging","icon":"file-lines","href":"/ai-engineering/production-logging"},{"title":"API Design for AI Applications","icon":"code","href":"/ai-engineering/api-design"},{"title":"Document Processing","icon":"file-lines","href":"/ai-engineering/document-processing"},{"title":"Model Comparison","icon":"scale-balanced","href":"/ai-engineering/model-comparison"},{"title":"Anthropic Claude API","icon":"message-bot","href":"/ai-engineering/anthropic-api"},{"title":"Async Patterns for LLMs","icon":"arrows-spin","href":"/ai-engineering/async-patterns"},{"title":"Advanced Prompting Techniques","icon":"brain","href":"/ai-engineering/advanced-prompting"},{"title":"Output Validation","icon":"shield-check","href":"/ai-engineering/output-validation"},{"title":"Semantic Routing","icon":"route","href":"/ai-engineering/semantic-routing"},{"title":"Advanced RAG Techniques","icon":"layer-group","href":"/ai-engineering/advanced-rag"},{"title":"Voice and Audio AI","icon":"microphone","href":"/ai-engineering/voice-audio"},{"title":"Image Generation","icon":"image","href":"/ai-engineering/image-generation"},{"title":"Summarization Systems","icon":"compress","href":"/ai-engineering/summarization"},{"title":"Chatbot Design Patterns","icon":"comments","href":"/ai-engineering/chatbot-patterns"},{"title":"Knowledge Graphs with LLMs","icon":"diagram-project","href":"/ai-engineering/knowledge-graphs"},{"title":"Synthetic Data Generation","icon":"database","href":"/ai-engineering/synthetic-data"},{"title":"Text Classification","icon":"tags","href":"/ai-engineering/classification"},{"title":"Model Optimization","icon":"gauge-high","href":"/ai-engineering/model-optimization"},{"title":"Realtime AI Applications","icon":"bolt","href":"/ai-engineering/realtime-ai"},{"title":"Evaluation \u0026 Testing","icon":"vial","href":"/ai-engineering/evaluation-testing"},{"title":"Deployment \u0026 Scaling","icon":"rocket","href":"/ai-engineering/deployment-scaling"},{"title":"Capstone Project","icon":"trophy","href":"/ai-engineering/capstone-project"}]},{"group":"Math for ML - Understanding Linear Algebra","icon":"square-root-variable","pages":[{"title":"Introduction to Linear Algebra for ML","sidebarTitle":"Introduction","icon":"rocket","href":"/courses/math-for-ml-linear-algebra/01-introduction"},{"title":"Vectors: The Language of Similarity","sidebarTitle":"Vectors","icon":"arrow-right","href":"/courses/math-for-ml-linear-algebra/02-vectors"},{"title":"Matrices \u0026 Linear Transformations","sidebarTitle":"Matrices","icon":"grid","href":"/courses/math-for-ml-linear-algebra/03-matrices"},{"title":"Project: Image Similarity Search Engine","sidebarTitle":"Project 1","icon":"code","href":"/courses/math-for-ml-linear-algebra/04-project-image-search"},{"title":"Eigenvalues \u0026 Eigenvectors","sidebarTitle":"Eigenvalues","icon":"compass","href":"/courses/math-for-ml-linear-algebra/05-eigenvalues"},{"title":"Principal Component Analysis (PCA)","sidebarTitle":"PCA","icon":"compress","href":"/courses/math-for-ml-linear-algebra/06-pca"},{"title":"Singular Value Decomposition (SVD)","sidebarTitle":"SVD","icon":"layer-group","href":"/courses/math-for-ml-linear-algebra/07-svd"},{"title":"Linear Systems \u0026 Applications","sidebarTitle":"Linear Systems","icon":"diagram-project","href":"/courses/math-for-ml-linear-algebra/08-linear-systems"},{"title":"Project: Music Recommendation Engine","sidebarTitle":"Project 2","icon":"music","href":"/courses/math-for-ml-linear-algebra/09-project-recommender"},{"title":"Orthogonality \u0026 Projections","sidebarTitle":"Orthogonality","icon":"right-left","href":"/courses/math-for-ml-linear-algebra/10-orthogonality"}]},{"group":"Probability \u0026 Statistics for ML","icon":"chart-bar","pages":[{"title":"Probability and Statistics for Machine Learning","sidebarTitle":"Introduction","icon":"chart-line","href":"/courses/statistics-for-ml/01-introduction"},{"title":"Describing Data: What's Normal?","sidebarTitle":"Describing Data","icon":"chart-bar","href":"/courses/statistics-for-ml/02-describing-data"},{"title":"Probability Foundations: How Likely Is This?","sidebarTitle":"Probability","icon":"dice","href":"/courses/statistics-for-ml/03-probability"},{"title":"Probability Distributions: Patterns in Randomness","sidebarTitle":"Distributions","icon":"chart-bell","href":"/courses/statistics-for-ml/04-distributions"},{"title":"Statistical Inference: Conclusions from Samples","sidebarTitle":"Inference","icon":"magnifying-glass-chart","href":"/courses/statistics-for-ml/05-inference"},{"title":"Hypothesis Testing: Real Effect or Random Noise?","sidebarTitle":"Hypothesis Testing","icon":"scale-balanced","href":"/courses/statistics-for-ml/06-hypothesis-testing"},{"title":"Correlation and Regression: Relationships and Predictions","sidebarTitle":"Regression","icon":"chart-line-up","href":"/courses/statistics-for-ml/07-regression"},{"title":"From Statistics to Machine Learning","sidebarTitle":"Stats to ML","icon":"brain","href":"/courses/statistics-for-ml/08-statistics-to-ml"},{"title":"Bayesian Statistics","sidebarTitle":"Bayesian Statistics","icon":"brain-circuit","href":"/courses/statistics-for-ml/09-bayesian-statistics"},{"title":"Time Series Fundamentals","sidebarTitle":"Time Series","icon":"chart-line","href":"/courses/statistics-for-ml/10-time-series-basics"},{"title":"Capstone Project: Complete Statistical Analysis","sidebarTitle":"Capstone Project","icon":"graduation-cap","href":"/courses/statistics-for-ml/11-capstone-project"}]},{"group":"Math for ML - Understanding Calculus","icon":"function","pages":[{"title":"Calculus for Machine Learning","sidebarTitle":"Introduction","icon":"function","href":"/courses/math-for-ml-calculus/00-introduction"},{"title":"Derivatives \u0026 Rates of Change","sidebarTitle":"Derivatives","icon":"chart-line","href":"/courses/math-for-ml-calculus/01-derivatives"},{"title":"Gradients \u0026 Multivariable Calculus","sidebarTitle":"Gradients","icon":"mountain","href":"/courses/math-for-ml-calculus/02-gradients"},{"title":"Chain Rule \u0026 Backpropagation","sidebarTitle":"Chain Rule","icon":"link","href":"/courses/math-for-ml-calculus/03-chain-rule"},{"title":"Gradient Descent","sidebarTitle":"Gradient Descent","icon":"arrow-trend-down","href":"/courses/math-for-ml-calculus/04-gradient-descent"},{"title":"Optimization Techniques","sidebarTitle":"Optimization","icon":"rocket","href":"/courses/math-for-ml-calculus/05-optimization"},{"title":"Final Project: Build a Neural Network","sidebarTitle":"Final Project","icon":"flag-checkered","href":"/courses/math-for-ml-calculus/06-final-project"},{"title":"Automatic Differentiation","sidebarTitle":"Autodiff","icon":"wand-magic-sparkles","href":"/courses/math-for-ml-calculus/07-automatic-differentiation"},{"title":"Convex Optimization","sidebarTitle":"Convexity","icon":"bowl-rice","href":"/courses/math-for-ml-calculus/08-convexity"},{"title":"Real-World Applications","sidebarTitle":"Applications","icon":"industry","href":"/courses/math-for-ml-calculus/09-real-world-applications"}]},{"group":"ML Mastery","icon":"brain","pages":[{"title":"Machine Learning Mastery","sidebarTitle":"Introduction","icon":"brain","href":"/courses/ml-mastery/00-introduction"},{"title":"The Prediction Game","sidebarTitle":"Prediction Game","icon":"crosshairs","href":"/courses/ml-mastery/01-prediction-game"},{"title":"Learning From Mistakes","sidebarTitle":"Learning From Mistakes","icon":"graduation-cap","href":"/courses/ml-mastery/02-learning-from-mistakes"},{"title":"Linear Regression","sidebarTitle":"Linear Regression","icon":"chart-line","href":"/courses/ml-mastery/03-linear-regression"},{"title":"Classification","sidebarTitle":"Classification","icon":"tags","href":"/courses/ml-mastery/04-classification"},{"title":"K-Nearest Neighbors","sidebarTitle":"K-Nearest Neighbors","icon":"people-arrows","href":"/courses/ml-mastery/04a-knn"},{"title":"Decision Trees","sidebarTitle":"Decision Trees","icon":"sitemap","href":"/courses/ml-mastery/05-decision-trees"},{"title":"Support Vector Machines","sidebarTitle":"SVM","icon":"border-none","href":"/courses/ml-mastery/05a-svm"},{"title":"Naive Bayes","sidebarTitle":"Naive Bayes","icon":"percent","href":"/courses/ml-mastery/05b-naive-bayes"},{"title":"Ensemble Methods","sidebarTitle":"Ensemble Methods","icon":"people-group","href":"/courses/ml-mastery/06-ensemble-methods"},{"title":"Model Evaluation","sidebarTitle":"Model Evaluation","icon":"chart-column","href":"/courses/ml-mastery/07-model-evaluation"},{"title":"Feature Engineering","sidebarTitle":"Feature Engineering","icon":"wand-magic-sparkles","href":"/courses/ml-mastery/08-feature-engineering"},{"title":"Hyperparameter Tuning","sidebarTitle":"Hyperparameter Tuning","icon":"sliders","href":"/courses/ml-mastery/09-hyperparameter-tuning"},{"title":"End-to-End ML Project","sidebarTitle":"End-to-End Project","icon":"rocket","href":"/courses/ml-mastery/10-end-to-end-project"},{"title":"Clustering","sidebarTitle":"Clustering","icon":"object-group","href":"/courses/ml-mastery/11-clustering"},{"title":"Neural Networks","sidebarTitle":"Neural Networks","icon":"network-wired","href":"/courses/ml-mastery/12-neural-networks"},{"title":"Regularization","sidebarTitle":"Regularization","icon":"shield-halved","href":"/courses/ml-mastery/13-regularization"},{"title":"Model Deployment","sidebarTitle":"Deployment","icon":"rocket","href":"/courses/ml-mastery/14-model-deployment"},{"title":"Time Series Forecasting","sidebarTitle":"Time Series","icon":"clock","href":"/courses/ml-mastery/15-time-series"},{"title":"Bias-Variance Tradeoff","sidebarTitle":"Bias-Variance","icon":"scale-balanced","href":"/courses/ml-mastery/16-bias-variance"},{"title":"Data Leakage","sidebarTitle":"Data Leakage","icon":"droplet","href":"/courses/ml-mastery/17-data-leakage"},{"title":"Dimensionality Reduction","sidebarTitle":"Dimensionality Reduction","icon":"compress","href":"/courses/ml-mastery/18-dimensionality-reduction"},{"title":"Capstone Project","sidebarTitle":"Capstone Project","icon":"graduation-cap","href":"/courses/ml-mastery/19-capstone-project"},{"title":"Handling Imbalanced Data","sidebarTitle":"Imbalanced Data","icon":"scale-unbalanced","href":"/courses/ml-mastery/20-imbalanced-data"},{"title":"Model Explainability","sidebarTitle":"Explainability","icon":"lightbulb","href":"/courses/ml-mastery/21-explainability"},{"title":"ML Pipelines","sidebarTitle":"ML Pipelines","icon":"diagram-project","href":"/courses/ml-mastery/22-ml-pipelines"},{"title":"Common ML Mistakes","sidebarTitle":"Common Mistakes","icon":"triangle-exclamation","href":"/courses/ml-mastery/23-common-mistakes"},{"title":"Cross-Validation Strategies","sidebarTitle":"Cross-Validation","icon":"layer-group","href":"/courses/ml-mastery/24-cross-validation"},{"title":"Anomaly Detection","sidebarTitle":"Anomaly Detection","icon":"magnifying-glass-chart","href":"/courses/ml-mastery/25-anomaly-detection"},{"title":"ML at Scale","sidebarTitle":"ML at Scale","icon":"server","href":"/courses/ml-mastery/26-ml-at-scale"}]},{"group":"Deep Learning Mastery","icon":"network-wired","pages":[{"title":"Deep Learning Mastery","sidebarTitle":"Overview","icon":"brain-circuit","href":"/courses/deep-learning-mastery/00-overview"},{"title":"The Deep Learning Landscape","sidebarTitle":"Deep Learning Landscape","icon":"map","href":"/courses/deep-learning-mastery/01-landscape"},{"title":"Perceptrons \u0026 Multi-Layer Networks","sidebarTitle":"Perceptrons \u0026 MLPs","icon":"circle-nodes","href":"/courses/deep-learning-mastery/02-perceptrons-mlp"},{"title":"Backpropagation Deep Dive","sidebarTitle":"Backpropagation","icon":"arrows-rotate","href":"/courses/deep-learning-mastery/03-backpropagation"},{"title":"Activation Functions","sidebarTitle":"Activation Functions","icon":"bolt","href":"/courses/deep-learning-mastery/04-activation-functions"},{"title":"Loss Functions \u0026 Objectives","sidebarTitle":"Loss Functions","icon":"crosshairs","href":"/courses/deep-learning-mastery/05-loss-functions"},{"title":"Convolutional Neural Networks","sidebarTitle":"CNNs","icon":"layer-group","href":"/courses/deep-learning-mastery/06-cnns"},{"title":"Pooling, Stride \u0026 CNN Design","sidebarTitle":"CNN Design","icon":"layer-group","href":"/courses/deep-learning-mastery/07-cnn-design"},{"title":"Recurrent Neural Networks","sidebarTitle":"RNNs","icon":"rotate-right","href":"/courses/deep-learning-mastery/08-rnns"},{"title":"LSTMs \u0026 GRUs","sidebarTitle":"LSTMs \u0026 GRUs","description":"Master gated recurrent architectures - how LSTM and GRU cells solve the vanishing gradient problem with memory gates","icon":"memory","href":"/courses/deep-learning-mastery/09-lstms-grus"},{"title":"Attention Mechanism","sidebarTitle":"Attention","icon":"eye","href":"/courses/deep-learning-mastery/10-attention"},{"title":"Transformers","sidebarTitle":"Transformers","description":"Build the Transformer architecture from scratch - the model that revolutionized NLP and powers GPT, BERT, and modern AI","icon":"bolt","href":"/courses/deep-learning-mastery/11-transformers"},{"title":"Generative Adversarial Networks","sidebarTitle":"GANs","icon":"masks-theater","href":"/courses/deep-learning-mastery/12-gans"},{"title":"Autoencoders \u0026 VAEs","sidebarTitle":"Autoencoders","icon":"compress","href":"/courses/deep-learning-mastery/13-autoencoders"},{"title":"Diffusion Models","sidebarTitle":"Diffusion Models","icon":"cloud","href":"/courses/deep-learning-mastery/14-diffusion"},{"title":"Residual \u0026 Skip Connections","sidebarTitle":"Residual Networks","icon":"diagram-project","href":"/courses/deep-learning-mastery/15-residual-networks"},{"title":"Normalization Techniques","sidebarTitle":"Normalization","icon":"sliders","href":"/courses/deep-learning-mastery/16-normalization"},{"title":"Regularization Techniques","sidebarTitle":"Regularization","icon":"shield","href":"/courses/deep-learning-mastery/17-regularization"},{"title":"Optimization Algorithms","sidebarTitle":"Optimizers","icon":"gauge-high","href":"/courses/deep-learning-mastery/18-optimization"},{"title":"Training Strategies at Scale","sidebarTitle":"Scale Training","icon":"server","href":"/courses/deep-learning-mastery/19-training-strategies"},{"title":"Transfer Learning \u0026 Fine-tuning","sidebarTitle":"Transfer Learning","icon":"arrows-repeat","href":"/courses/deep-learning-mastery/20-transfer-learning"},{"title":"Model Deployment","sidebarTitle":"Deployment","icon":"rocket","href":"/courses/deep-learning-mastery/21-deployment"},{"title":"Debugging Deep Learning","sidebarTitle":"Debugging","icon":"bug","href":"/courses/deep-learning-mastery/22-debugging"},{"title":"Vision Transformers","sidebarTitle":"Vision Transformers","icon":"eye","href":"/courses/deep-learning-mastery/23-vision-transformers"},{"title":"Multimodal Models","sidebarTitle":"Multimodal","icon":"images","href":"/courses/deep-learning-mastery/24-multimodal"},{"title":"Foundation Models \u0026 LLMs","sidebarTitle":"Foundation Models","icon":"brain","href":"/courses/deep-learning-mastery/25-foundation-models"},{"title":"Capstone Project","sidebarTitle":"Capstone","icon":"flag-checkered","href":"/courses/deep-learning-mastery/26-capstone"},{"title":"Mathematical Foundations for Deep Learning","sidebarTitle":"Math Foundations","icon":"square-root-variable","href":"/courses/deep-learning-mastery/27-math-foundations"},{"title":"Weight Initialization Strategies","sidebarTitle":"Weight Initialization","icon":"seedling","href":"/courses/deep-learning-mastery/28-weight-initialization"},{"title":"Gradient Flow Analysis","sidebarTitle":"Gradient Flow","icon":"wave-pulse","href":"/courses/deep-learning-mastery/29-gradient-flow"},{"title":"Advanced CNN Architectures","sidebarTitle":"CNN Architectures","icon":"layer-group","href":"/courses/deep-learning-mastery/30-cnn-architectures"},{"title":"Sequence-to-Sequence Models","sidebarTitle":"Seq2Seq","icon":"arrows-left-right","href":"/courses/deep-learning-mastery/31-seq2seq"},{"title":"Self-Supervised Learning","sidebarTitle":"Self-Supervised Learning","icon":"brain","href":"/courses/deep-learning-mastery/32-self-supervised"},{"title":"Reinforcement Learning for Deep Learning","sidebarTitle":"RL for Deep Learning","icon":"gamepad","href":"/courses/deep-learning-mastery/33-rl-for-dl"},{"title":"Neural Architecture Search","sidebarTitle":"Neural Architecture Search","icon":"magnifying-glass","href":"/courses/deep-learning-mastery/34-nas"},{"title":"Interpretability \u0026 Explainability","sidebarTitle":"Interpretability","icon":"eye","href":"/courses/deep-learning-mastery/35-interpretability"},{"title":"Adversarial Robustness","sidebarTitle":"Adversarial ML","icon":"shield-halved","href":"/courses/deep-learning-mastery/36-adversarial-robustness"},{"title":"Efficient Neural Network Architectures","sidebarTitle":"Efficient Architectures","icon":"bolt","href":"/courses/deep-learning-mastery/37-efficient-architectures"},{"title":"Graph Neural Networks","sidebarTitle":"Graph Networks","icon":"diagram-project","href":"/courses/deep-learning-mastery/38-graph-neural-networks"},{"title":"3D Deep Learning","sidebarTitle":"3D Deep Learning","icon":"cube","href":"/courses/deep-learning-mastery/39-3d-deep-learning"},{"title":"Object Detection Deep Dive","sidebarTitle":"Object Detection","icon":"vector-square","href":"/courses/deep-learning-mastery/40-object-detection"},{"title":"Semantic Segmentation","sidebarTitle":"Segmentation","icon":"object-group","href":"/courses/deep-learning-mastery/41-semantic-segmentation"},{"title":"Hyperparameter Tuning","sidebarTitle":"Hyperparameter Tuning","icon":"sliders","href":"/courses/deep-learning-mastery/42-hyperparameter-tuning"},{"title":"Reproducibility","sidebarTitle":"Reproducibility","icon":"repeat","href":"/courses/deep-learning-mastery/43-reproducibility"},{"title":"Memory-Efficient Training","sidebarTitle":"Memory-Efficient Training","icon":"memory","href":"/courses/deep-learning-mastery/44-memory-efficient-training"},{"title":"Quantization Deep Dive","sidebarTitle":"Quantization Deep Dive","icon":"compress","href":"/courses/deep-learning-mastery/45-quantization-deep-dive"},{"title":"Knowledge Distillation","sidebarTitle":"Knowledge Distillation","icon":"school","href":"/courses/deep-learning-mastery/46-knowledge-distillation"},{"title":"Continual Learning","sidebarTitle":"Continual Learning","icon":"arrows-rotate","href":"/courses/deep-learning-mastery/47-continual-learning"}]},{"group":"NestJS Mastery","icon":"feather","pages":[{"title":"NestJS Mastery","sidebarTitle":"Course Overview","icon":"node","href":"/courses/nestjs/overview"},{"title":"NestJS Fundamentals","sidebarTitle":"1. Fundamentals","icon":"book","href":"/courses/nestjs/fundamentals"},{"title":"Dependency Injection","sidebarTitle":"2. DI \u0026 Modules","icon":"puzzle-piece","href":"/courses/nestjs/dependency-injection"},{"title":"Controllers \u0026 Routing","sidebarTitle":"3. Controllers","icon":"route","href":"/courses/nestjs/controllers-routes"},{"title":"Providers \u0026 Services","sidebarTitle":"4. Providers","icon":"gear","href":"/courses/nestjs/providers-services"},{"title":"Database Integration","sidebarTitle":"5. Database","icon":"database","href":"/courses/nestjs/database-integration"},{"title":"Authentication \u0026 Authorization","sidebarTitle":"6. Auth","icon":"lock","href":"/courses/nestjs/authentication-authorization"},{"title":"Testing","sidebarTitle":"7. Testing","icon":"flask","href":"/courses/nestjs/testing"},{"title":"Microservices","sidebarTitle":"8. Microservices","icon":"cubes","href":"/courses/nestjs/microservices"},{"title":"Deployment \u0026 Production","sidebarTitle":"9. Deployment","icon":"rocket","href":"/courses/nestjs/deployment-production"},{"title":"Advanced Patterns","sidebarTitle":"10. Advanced","icon":"star","href":"/courses/nestjs/advanced-patterns"}]},{"group":"Microservices Mastery","icon":"cubes","pages":[{"title":"Microservices Mastery","sidebarTitle":"Course Overview","icon":"cubes","href":"/courses/microservices-mastery/00-overview"},{"title":"1. Microservices Foundations","href":"/courses/microservices-mastery/01-foundations"},{"title":"2. Domain-Driven Design","href":"/courses/microservices-mastery/02-domain-driven-design"},{"title":"3. Synchronous Communication","href":"/courses/microservices-mastery/03-sync-communication"},{"title":"4. Asynchronous Communication","href":"/courses/microservices-mastery/04-async-communication"},{"title":"5. API Gateway Pattern","href":"/courses/microservices-mastery/05-api-gateway"},{"title":"6. Data Management Patterns","href":"/courses/microservices-mastery/06-data-management"},{"title":"7. Resilience Patterns","href":"/courses/microservices-mastery/07-resilience-patterns"},{"title":"8. Service Discovery","href":"/courses/microservices-mastery/08-service-discovery"},{"title":"9. Observability","href":"/courses/microservices-mastery/09-observability"},{"title":"10. Security","href":"/courses/microservices-mastery/10-security"},{"title":"11. Containerization with Docker","href":"/courses/microservices-mastery/11-containerization"},{"title":"12. Kubernetes Deployment","href":"/courses/microservices-mastery/12-kubernetes"},{"title":"13. Testing Strategies","href":"/courses/microservices-mastery/13-testing"},{"title":"14. Interview Preparation","href":"/courses/microservices-mastery/14-interview-prep"},{"title":"15. Capstone Project","href":"/courses/microservices-mastery/15-capstone"},{"title":"16. Service Mesh","href":"/courses/microservices-mastery/16-service-mesh"},{"title":"17. Configuration Management","href":"/courses/microservices-mastery/17-configuration-management"},{"title":"18. CI/CD for Microservices","href":"/courses/microservices-mastery/18-cicd"},{"title":"19. Database Patterns","href":"/courses/microservices-mastery/19-database-patterns"},{"title":"20. Caching Strategies","href":"/courses/microservices-mastery/20-caching-strategies"},{"title":"21. Chaos Engineering","href":"/courses/microservices-mastery/21-chaos-engineering"},{"title":"22. Case Studies","href":"/courses/microservices-mastery/22-case-studies"},{"title":"23. Load Balancing Deep Dive","href":"/courses/microservices-mastery/23-load-balancing"},{"title":"24. Migration Patterns","href":"/courses/microservices-mastery/24-migration-patterns"},{"title":"25. Event Sourcing Deep Dive","href":"/courses/microservices-mastery/25-event-sourcing-deep-dive"},{"title":"26. GraphQL Federation","href":"/courses/microservices-mastery/26-graphql-federation"}]},{"group":"Low Level Design","pages":[{"title":"Low Level Design (LLD)","icon":"cubes","href":"/lld/overview"},{"title":"LLD Interview Cheat Sheet","icon":"file-lines","href":"/lld/interview-cheatsheet"}]},{"group":"OOP Concepts","pages":[{"title":"What is OOP?","icon":"graduation-cap","href":"/lld/oop/introduction"},{"title":"Encapsulation","icon":"box","href":"/lld/oop/encapsulation"},{"title":"Inheritance","icon":"sitemap","href":"/lld/oop/inheritance"},{"title":"Polymorphism","icon":"shapes","href":"/lld/oop/polymorphism"},{"title":"Abstraction","icon":"eye-slash","href":"/lld/oop/abstraction"},{"title":"OOP Concepts","icon":"object-group","href":"/lld/oop-concepts"}]},{"group":"SOLID Principles","pages":[{"title":"SOLID Introduction","icon":"wand-magic-sparkles","href":"/lld/solid/introduction"},{"title":"Single Responsibility Principle","icon":"user-check","href":"/lld/solid/srp"},{"title":"Open/Closed Principle","icon":"plug","href":"/lld/solid/ocp"},{"title":"Liskov Substitution Principle","icon":"right-left","href":"/lld/solid/lsp"},{"title":"Interface Segregation Principle","icon":"puzzle-piece","href":"/lld/solid/isp"},{"title":"Dependency Inversion Principle","icon":"outlet","href":"/lld/solid/dip"},{"title":"SOLID Principles in Practice","icon":"compass-drafting","href":"/lld/solid-practice"}]},{"group":"Design Patterns","pages":[{"title":"Design Patterns","icon":"shapes","href":"/lld/design-patterns"},{"title":"UML Diagrams","icon":"diagram-project","href":"/lld/uml-diagrams"}]},{"group":"LLD Case Studies","pages":[{"title":"Case Study: Parking Lot System","icon":"square-parking","href":"/lld/case-parking-lot"},{"title":"Case Study: Library Management System","icon":"book","href":"/lld/case-library"},{"title":"Design Vending Machine","icon":"box","href":"/lld/case-vending-machine"},{"title":"Design LRU Cache","icon":"database","href":"/lld/case-lru-cache"},{"title":"Design Movie Ticket Booking System","icon":"ticket","href":"/lld/case-movie-ticket"},{"title":"Case Study: Elevator System","icon":"elevator","href":"/lld/case-elevator"},{"title":"Case Study: Chess Game","icon":"chess","href":"/lld/case-chess"},{"title":"Case Study: Hotel Booking System","icon":"hotel","href":"/lld/case-hotel-booking"},{"title":"Case Study: ATM System","icon":"building-columns","href":"/lld/case-atm"}]},{"group":"System Design (HLD)","pages":[{"title":"System Design Mastery","icon":"network-wired","href":"/system-design/overview"},{"title":"Interview Guide \u0026 Cheatsheet","icon":"clipboard-check","href":"/system-design/interview-guide"},{"title":"Interview Questions Bank","icon":"clipboard-question","href":"/system-design/interview-questions"},{"title":"System Design Fundamentals","icon":"book","href":"/system-design/fundamentals"},{"title":"System Design Building Blocks","icon":"cubes","href":"/system-design/building-blocks"},{"title":"Networking Fundamentals","icon":"network-wired","href":"/system-design/networking"},{"title":"Database Design","icon":"database","href":"/system-design/databases"},{"title":"API Design","icon":"plug","href":"/system-design/api-design"},{"title":"Distributed Systems","icon":"share-nodes","href":"/system-design/distributed-systems"},{"title":"Microservices Architecture","icon":"cubes","href":"/system-design/microservices"},{"title":"Back-of-Envelope Calculations","icon":"calculator","href":"/system-design/estimations"}]},{"group":"Senior Level (L5+/Staff)","pages":[{"title":"Advanced Concepts","icon":"brain","href":"/system-design/advanced-concepts"},{"title":"Scalability Patterns","icon":"arrow-up-right-dots","href":"/system-design/scalability-patterns"},{"title":"Reliability \u0026 Fault Tolerance","icon":"shield-halved","href":"/system-design/reliability-patterns"},{"title":"Observability \u0026 Monitoring","icon":"chart-line","href":"/system-design/observability"},{"title":"Data Pipelines","icon":"pipe","href":"/system-design/data-pipelines"},{"title":"Security Patterns","icon":"shield","href":"/system-design/security-patterns"}]},{"group":"HLD Case Studies","pages":[{"title":"Design: URL Shortener","icon":"link","href":"/system-design/case-url-shortener"},{"title":"Design: Rate Limiter","icon":"gauge","href":"/system-design/case-rate-limiter"},{"title":"Design a Notification System","icon":"bell","href":"/system-design/case-notification-system"},{"title":"Design: Twitter/X Timeline","icon":"twitter","href":"/system-design/case-twitter"},{"title":"Design: WhatsApp/Messenger","icon":"message","href":"/system-design/case-whatsapp"},{"title":"Design a Payment System","icon":"credit-card","href":"/system-design/case-payment-system"},{"title":"Design: Netflix/YouTube","icon":"video","href":"/system-design/case-netflix"},{"title":"Design: Uber/Lyft","icon":"car","href":"/system-design/case-uber"},{"title":"Design Google Maps","icon":"map-location-dot","href":"/system-design/case-google-maps"}]},{"group":"Engineering Fundamentals","pages":[{"title":"Engineering Topics","icon":"gears","href":"/engineering/overview"},{"title":"Computer Science Fundamentals","icon":"microchip","href":"/engineering/cs-fundamentals"},{"title":"Software Design Principles","icon":"compass-drafting","href":"/engineering/design-principles"},{"title":"Architecture Patterns","icon":"building","href":"/engineering/architecture-patterns"}]},{"group":"DevOps \u0026 Operations","pages":[{"title":"DevOps \u0026 Infrastructure","icon":"server","href":"/engineering/devops"},{"title":"Security Fundamentals","icon":"shield-halved","href":"/engineering/security"},{"title":"Performance \u0026 Optimization","icon":"gauge-high","href":"/engineering/performance"}]},{"group":"Azure Cloud Engineering","icon":"cloud","pages":[{"title":"Azure Cloud Engineering","sidebarTitle":"Overview","icon":"cloud","href":"/courses/azure-cloud-engineering/00-overview"},{"title":"Azure Fundamentals \u0026 Architecture","icon":"cloud-arrow-up","href":"/courses/azure-cloud-engineering/01-azure-fundamentals"},{"title":"Identity \u0026 Access Management","icon":"user-shield","href":"/courses/azure-cloud-engineering/02-identity-access-management"},{"title":"03a. Networking Fundamentals","icon":"network-wired","href":"/courses/azure-cloud-engineering/03a-networking-fundamentals"},{"title":"03b. Hybrid \u0026 Enterprise","icon":"globe","href":"/courses/azure-cloud-engineering/03b-hybrid-networking"},{"title":"03c. Traffic \u0026 Security","icon":"traffic-light","href":"/courses/azure-cloud-engineering/03c-traffic-management"},{"title":"Compute Services","icon":"server","href":"/courses/azure-cloud-engineering/04-compute-services"},{"title":"Storage Solutions","icon":"database","href":"/courses/azure-cloud-engineering/05-storage-solutions"},{"title":"Database Services","icon":"database","href":"/courses/azure-cloud-engineering/06-database-services"},{"title":"Azure Kubernetes Service (AKS)","icon":"dharmachakra","href":"/courses/azure-cloud-engineering/07-container-kubernetes"},{"title":"Serverless Architecture","icon":"bolt","href":"/courses/azure-cloud-engineering/08-serverless-architecture"},{"title":"Monitoring \u0026 Observability","icon":"chart-line","href":"/courses/azure-cloud-engineering/09-monitoring-observability"},{"title":"Security \u0026 Compliance","icon":"shield-halved","href":"/courses/azure-cloud-engineering/10-security-compliance"},{"title":"DevOps \u0026 CI/CD","icon":"code-branch","href":"/courses/azure-cloud-engineering/11-devops-cicd"},{"title":"Cost Optimization","icon":"dollar-sign","href":"/courses/azure-cloud-engineering/12-cost-optimization"},{"title":"High Availability \u0026 Disaster Recovery","icon":"life-ring","href":"/courses/azure-cloud-engineering/13-high-availability-dr"},{"title":"Architecture Patterns","icon":"diagram-project","href":"/courses/azure-cloud-engineering/14-architecture-patterns"},{"title":"Capstone Project","icon":"trophy","href":"/courses/azure-cloud-engineering/15-capstone-project"},{"title":"API Management \u0026 Messaging Patterns","icon":"diagram-project","href":"/courses/azure-cloud-engineering/16-api-messaging"},{"title":"SRE \u0026 Production Excellence","icon":"shield-heart","href":"/courses/azure-cloud-engineering/17-sre-production"}]},{"group":"AWS Cloud","icon":"cloud","pages":[{"title":"AWS Cloud Mastery","icon":"cloud","href":"/aws/overview"},{"title":"Core Concepts","icon":"circle-nodes","href":"/aws/core-concepts"},{"title":"Compute Services","icon":"microchip","href":"/aws/compute"},{"title":"Storage \u0026 Databases","icon":"hard-drive","href":"/aws/storage-databases"},{"title":"Networking","icon":"diagram-project","href":"/aws/networking"},{"title":"Security \u0026 IAM","icon":"shield-halved","href":"/aws/security"},{"title":"Well-Architected Framework","icon":"building","href":"/aws/well-architected"}]},{"group":"AWS Monitoring \u0026 Observability","icon":"chart-line","pages":[{"title":"Amazon CloudWatch","icon":"chart-line","href":"/aws/cloudwatch"},{"title":"AWS X-Ray","icon":"microscope","href":"/aws/xray"},{"title":"Observability \u0026 Monitoring","icon":"chart-line","href":"/aws/observability"}]},{"group":"AWS Security Services","icon":"shield-halved","pages":[{"title":"AWS GuardDuty","icon":"shield","href":"/aws/guardduty"},{"title":"AWS Security Hub","icon":"building-shield","href":"/aws/security-hub"},{"title":"AWS Inspector","href":"/aws/inspector"},{"title":"AWS WAF (Web Application Firewall)","href":"/aws/waf"},{"title":"AWS Shield","href":"/aws/shield"},{"title":"AWS CloudTrail","icon":"file-lines","href":"/aws/cloudtrail"},{"title":"AWS Config","icon":"gear","href":"/aws/config"}]},{"group":"AWS Serverless","icon":"bolt","pages":[{"title":"AWS Lambda","icon":"bolt","href":"/aws/lambda"},{"title":"AWS Step Functions","icon":"diagram-project","href":"/aws/step-functions"},{"title":"AWS SAM","icon":"box","href":"/aws/sam"},{"title":"Amazon DynamoDB","icon":"database","href":"/aws/dynamodb"}]},{"group":"AWS Operations","icon":"gears","pages":[{"title":"DevOps \u0026 CI/CD","icon":"code-branch","href":"/aws/devops-cicd"}]},{"group":"AWS Advanced","icon":"rocket","pages":[{"title":"CDN \u0026 Edge Services","icon":"globe","href":"/aws/cdn-edge"},{"title":"Messaging \u0026 Integration","icon":"arrows-split-up-and-left","href":"/aws/messaging"},{"title":"Containers \u0026 Microservices","icon":"cubes","href":"/aws/containers"}]},{"group":"AWS Case Studies","icon":"lightbulb","pages":[{"title":"Case Study: Serverless URL Shortener","icon":"bolt","href":"/aws/case-study-serverless"}]},{"group":"GCP Cloud Engineering","icon":"cloud","pages":[{"title":"GCP Cloud Engineering","sidebarTitle":"Overview","icon":"cloud","href":"/courses/gcp-cloud-engineering/00-overview"},{"title":"GCP Fundamentals","href":"/courses/gcp-cloud-engineering/01-gcp-fundamentals"},{"title":"Identity \u0026 Security","href":"/courses/gcp-cloud-engineering/02-iam-security"},{"title":"Networking \u0026 VPC","href":"/courses/gcp-cloud-engineering/03-networking-vpc"},{"title":"Load Balancing \u0026 DNS","href":"/courses/gcp-cloud-engineering/04-load-balancing-dns"},{"title":"Compute Engine","href":"/courses/gcp-cloud-engineering/05-compute-engine"},{"title":"Cloud Storage","href":"/courses/gcp-cloud-engineering/06-cloud-storage"},{"title":"Relational Databases","href":"/courses/gcp-cloud-engineering/07-relational-databases"},{"title":"NoSQL Databases","href":"/courses/gcp-cloud-engineering/08-nosql-databases"},{"title":"Containers \u0026 Artifacts","href":"/courses/gcp-cloud-engineering/09-containers-registry"},{"title":"Google Kubernetes Engine","href":"/courses/gcp-cloud-engineering/10-google-kubernetes-engine"},{"title":"Serverless Compute","href":"/courses/gcp-cloud-engineering/11-serverless-compute"},{"title":"Data Analytics \u0026 BigQuery","href":"/courses/gcp-cloud-engineering/12-data-analytics-bigquery"},{"title":"Monitoring \u0026 Observability","href":"/courses/gcp-cloud-engineering/13-monitoring-observability"},{"title":"Security \u0026 Compliance","href":"/courses/gcp-cloud-engineering/14-security-compliance"},{"title":"IaC \u0026 Deployment","href":"/courses/gcp-cloud-engineering/15-iac-deployment"},{"title":"Hybrid \u0026 Multi-Cloud","href":"/courses/gcp-cloud-engineering/16-hybrid-multi-cloud"},{"title":"Cost Management","href":"/courses/gcp-cloud-engineering/17-cost-management"},{"title":"Capstone Project","href":"/courses/gcp-cloud-engineering/18-capstone-project"}]},{"group":"DevOps Tools","icon":"toolbox","pages":[{"title":"DevOps Tools Mastery","sidebarTitle":"Overview","icon":"toolbox","href":"/courses/devops-tools/overview"},{"group":"Git Crash Course","pages":[{"title":"Git Crash Course","sidebarTitle":"Git Overview","icon":"code-branch","href":"/courses/devops-tools/git-overview"},{"title":"Git Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/git-fundamentals"},{"title":"Git Branching \u0026 Merging","sidebarTitle":"Branching","icon":"code-branch","href":"/courses/devops-tools/git-branching"},{"title":"Git Collaboration","sidebarTitle":"Collaboration","icon":"users","href":"/courses/devops-tools/git-collaboration"},{"title":"Git Advanced Topics","sidebarTitle":"Advanced","icon":"wand-magic-sparkles","href":"/courses/devops-tools/git-advanced"},{"title":"Git Workflows","sidebarTitle":"Workflows","icon":"code-branch","href":"/courses/devops-tools/git-workflows"},{"title":"Git Hooks \u0026 Automation","sidebarTitle":"Hooks \u0026 Automation","icon":"robot","href":"/courses/devops-tools/git-hooks"}]},{"group":"Linux Crash Course","pages":[{"title":"Linux Crash Course","sidebarTitle":"Linux Overview","icon":"linux","href":"/courses/devops-tools/linux-overview"},{"title":"Linux Fundamentals","sidebarTitle":"Fundamentals","icon":"terminal","href":"/courses/devops-tools/linux-fundamentals"},{"title":"Linux Permissions \u0026 Users","sidebarTitle":"Permissions","icon":"lock","href":"/courses/devops-tools/linux-permissions"},{"title":"Linux Process Management","sidebarTitle":"Processes","icon":"gears","href":"/courses/devops-tools/linux-processes"},{"title":"Linux Networking","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/devops-tools/linux-networking"},{"title":"Linux Shell Scripting","sidebarTitle":"Shell Scripting","icon":"file-code","href":"/courses/devops-tools/linux-scripting"},{"title":"System Monitoring","sidebarTitle":"Monitoring","icon":"chart-line","href":"/courses/devops-tools/linux-monitoring"},{"title":"Security Hardening","sidebarTitle":"Security","icon":"lock","href":"/courses/devops-tools/linux-security"}]},{"group":"Docker Crash Course","pages":[{"title":"Docker Crash Course","sidebarTitle":"Docker Overview","icon":"docker","href":"/courses/devops-tools/docker-overview"},{"title":"Docker Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/docker-fundamentals"},{"title":"Building Docker Images","sidebarTitle":"Images","icon":"layer-group","href":"/courses/devops-tools/docker-images"},{"title":"Docker Networking \u0026 Volumes","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/devops-tools/docker-networking"},{"title":"Docker Compose","sidebarTitle":"Compose","icon":"cubes","href":"/courses/devops-tools/docker-compose"},{"title":"Docker Best Practices","sidebarTitle":"Best Practices","icon":"shield-check","href":"/courses/devops-tools/docker-best-practices"}]},{"group":"RabbitMQ Crash Course","pages":[{"title":"RabbitMQ Crash Course","sidebarTitle":"RabbitMQ Overview","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-overview"},{"title":"RabbitMQ Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/rabbitmq-fundamentals"},{"title":"Patterns","sidebarTitle":"Patterns","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-patterns"},{"title":"Reliability","sidebarTitle":"Reliability","icon":"rabbit","href":"/courses/devops-tools/rabbitmq-reliability"}]},{"group":"Kafka Crash Course","pages":[{"title":"Kafka Crash Course","sidebarTitle":"Kafka Overview","icon":"stream","href":"/courses/devops-tools/kafka-overview"},{"title":"Kafka Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/kafka-fundamentals"},{"title":"Kafka Producers \u0026 Consumers","sidebarTitle":"Producers \u0026 Consumers","icon":"arrows-left-right","href":"/courses/devops-tools/kafka-producers-consumers"},{"title":"Kafka Streams","sidebarTitle":"Streams","icon":"water","href":"/courses/devops-tools/kafka-streams"},{"title":"Kafka Operations","sidebarTitle":"Operations","icon":"server","href":"/courses/devops-tools/kafka-operations"}]},{"group":"Kubernetes Crash Course","pages":[{"title":"Kubernetes Crash Course","sidebarTitle":"Kubernetes Overview","icon":"dharmachakra","href":"/courses/devops-tools/kubernetes-overview"},{"title":"Kubernetes Fundamentals","sidebarTitle":"Fundamentals","icon":"book","href":"/courses/devops-tools/kubernetes-fundamentals"},{"title":"Kubernetes Workloads","sidebarTitle":"Workloads","icon":"cubes","href":"/courses/devops-tools/kubernetes-workloads"},{"title":"Kubernetes Services","sidebarTitle":"Services","icon":"network-wired","href":"/courses/devops-tools/kubernetes-services"},{"title":"Kubernetes Configuration","sidebarTitle":"Configuration","icon":"gear","href":"/courses/devops-tools/kubernetes-config"},{"title":"Kubernetes Storage","sidebarTitle":"Storage","icon":"hard-drive","href":"/courses/devops-tools/kubernetes-storage"},{"title":"Kubernetes on Windows \u0026 Linux","sidebarTitle":"Windows \u0026 Linux","icon":"window-maximize","href":"/courses/devops-tools/kubernetes-windows-linux"}]}]},{"group":"Database Engineering","icon":"database","pages":[{"title":"Database Engineering Mastery","sidebarTitle":"Course Overview","icon":"database","href":"/courses/database-engineering/overview"},{"group":"Track 1: Foundations","pages":[{"title":"SQL Foundations","sidebarTitle":"1. SQL Foundations","icon":"terminal","href":"/courses/database-engineering/sql-foundations"},{"title":"Database Design \u0026 Modeling","sidebarTitle":"2. Database Design","icon":"diagram-project","href":"/courses/database-engineering/database-design"},{"title":"Transactions \u0026 ACID","sidebarTitle":"3. Transactions \u0026 ACID","icon":"shield-check","href":"/courses/database-engineering/transactions"}]},{"group":"Track 2: Performance","pages":[{"title":"Indexing Deep Dive","sidebarTitle":"4. Indexing Deep Dive","icon":"magnifying-glass-chart","href":"/courses/database-engineering/indexing"},{"title":"Query Optimization","sidebarTitle":"5. Query Optimization","icon":"bolt","href":"/courses/database-engineering/query-optimization"},{"title":"Query Execution Plans Mastery","sidebarTitle":"Execution Plans Deep Dive","icon":"route","href":"/courses/database-engineering/execution-plans-mastery"},{"title":"Performance Engineering","sidebarTitle":"Performance Engineering","icon":"gauge-high","href":"/courses/database-engineering/performance-engineering"}]},{"group":"Track 3: Internals","pages":[{"title":"PostgreSQL Internals","sidebarTitle":"6. PostgreSQL Internals","icon":"microchip","href":"/courses/database-engineering/internals"},{"title":"PostgreSQL Data Structures","sidebarTitle":"Data Structures","icon":"sitemap","href":"/courses/database-engineering/data-structures"},{"title":"Query Processing Pipeline","sidebarTitle":"8. Query Processing","icon":"gears","href":"/courses/database-engineering/query-processing"},{"title":"Query Engine Deep Dive","sidebarTitle":"Query Engine Internals","icon":"microchip","href":"/courses/database-engineering/query-engine-deep-dive"},{"title":"PostgreSQL Architecture","sidebarTitle":"9. PG Architecture","icon":"sitemap","href":"/courses/database-engineering/architecture"},{"title":"Storage Engine Deep Dive","sidebarTitle":"Storage Engine Internals","icon":"hard-drive","href":"/courses/database-engineering/storage-deep-dive"}]},{"group":"Track 4: Distributed","pages":[{"title":"Replication \u0026 High Availability","sidebarTitle":"7. Replication \u0026 HA","icon":"server","href":"/courses/database-engineering/replication"},{"title":"Scaling Strategies","sidebarTitle":"8. Scaling","icon":"chart-line","href":"/courses/database-engineering/scaling"},{"title":"Distributed Database Systems","sidebarTitle":"Distributed Systems","icon":"network-wired","href":"/courses/database-engineering/distributed-systems"}]},{"group":"OSS Contributions","pages":[{"title":"PostgreSQL Source Code","sidebarTitle":"13. Source Code","icon":"code","href":"/courses/database-engineering/source-code"},{"title":"Contributing to PostgreSQL","sidebarTitle":"14. Contributing","icon":"code-pull-request","href":"/courses/database-engineering/contributing"}]},{"group":"Capstone","pages":[{"title":"Senior Interview Mastery","sidebarTitle":"15. Interview Prep","icon":"user-tie","href":"/courses/database-engineering/interview-prep"},{"title":"System Design Case Studies","sidebarTitle":"16. Case Studies","icon":"building","href":"/courses/database-engineering/case-studies"}]}]},{"group":"HIPAA Compliance Mastery","icon":"shield-check","pages":[{"title":"HIPAA Compliance Mastery","sidebarTitle":"Course Overview","icon":"hospital","href":"/courses/hipaa-compliance/overview"},{"group":"Track 1: Regulatory Foundations","pages":[{"title":"HIPAA Fundamentals","sidebarTitle":"HIPAA Fundamentals","icon":"book-medical","href":"/courses/hipaa-compliance/hipaa-fundamentals"},{"title":"HIPAA Risk Assessment","sidebarTitle":"Risk Assessment","icon":"magnifying-glass-chart","href":"/courses/hipaa-compliance/risk-assessment"},{"title":"PDPL \u0026 Global Data Protection","sidebarTitle":"PDPL Compliance","icon":"globe","href":"/courses/hipaa-compliance/pdpl-compliance"}]},{"group":"Track 2: Security Architecture","pages":[{"title":"Access Control Systems","sidebarTitle":"Access Control","icon":"user-shield","href":"/courses/hipaa-compliance/access-control"},{"title":"Encryption: Data at Rest \u0026 In Transit","sidebarTitle":"Encryption","icon":"lock","href":"/courses/hipaa-compliance/encryption"},{"title":"HIPAA Database Security \u0026 Encryption","href":"/courses/hipaa-compliance/database-security"}]},{"group":"Track 3: Implementation","pages":[{"title":"Audit Logging for HIPAA","sidebarTitle":"Audit Logging","icon":"file-lines","href":"/courses/hipaa-compliance/audit-logging"},{"title":"E2E Encryption with AI Agents","sidebarTitle":"E2E Encryption + AI","icon":"robot","href":"/courses/hipaa-compliance/e2e-encryption-ai"},{"title":"Implementation Guide","sidebarTitle":"Implementation Guide","icon":"code","href":"/courses/hipaa-compliance/implementation-guide"}]},{"group":"Track 4: Operations \u0026 Governance","pages":[{"title":"Incident Response for Healthcare","sidebarTitle":"Incident Response","icon":"siren","href":"/courses/hipaa-compliance/incident-response"},{"title":"HIPAA Vendor \u0026 Third-Party Risk Management","href":"/courses/hipaa-compliance/vendor-management"}]},{"group":"Track 5: Advanced Topics","pages":[{"title":"HIPAA Breach Case Studies \u0026 Lessons Learned","href":"/courses/hipaa-compliance/case-studies"},{"title":"Compliance Checklist \u0026 BAA","sidebarTitle":"Compliance Checklist","icon":"clipboard-check","href":"/courses/hipaa-compliance/compliance-checklist"}]},{"group":"Track 6: Capstone","pages":[{"title":"HIPAA Compliance Capstone Project","href":"/courses/hipaa-compliance/capstone-project"}]}]},{"group":"Operating Systems","icon":"microchip","pages":[{"title":"Operating Systems Mastery","sidebarTitle":"Course Overview","icon":"microchip","href":"/operating-systems/overview"},{"title":"OS Fundamentals \u0026 System Call Internals","sidebarTitle":"Fundamentals","icon":"book","href":"/operating-systems/os-fundamentals"},{"title":"CPU Architectures \u0026 Microarchitecture","sidebarTitle":"CPU Architectures","icon":"microchip","href":"/operating-systems/cpu-architectures"},{"title":"Operating System Interfaces","sidebarTitle":"xv6 OS Interfaces","icon":"book-open-reader","href":"/operating-systems/xv6"},{"group":"System Startup","pages":[{"title":"The Boot Process: From Reset Vector to PID 1","sidebarTitle":"Boot Process","icon":"power-off","href":"/operating-systems/boot-process"}]},{"group":"Process Management","pages":[{"title":"Process Management","sidebarTitle":"Processes","icon":"microchip","href":"/operating-systems/processes"},{"title":"Threads \u0026 Concurrency: The Senior Mastery Guide","sidebarTitle":"Threads","icon":"layer-group","href":"/operating-systems/threads"},{"title":"CPU Scheduling","sidebarTitle":"Scheduling","icon":"clock","href":"/operating-systems/scheduling"}]},{"group":"Memory Management","pages":[{"title":"Memory Management","sidebarTitle":"Memory Management","icon":"memory","href":"/operating-systems/memory-management"},{"title":"Virtual Memory","sidebarTitle":"Virtual Memory","icon":"memory","href":"/operating-systems/virtual-memory"}]},{"group":"Concurrency \u0026 I/O","pages":[{"title":"Advanced Synchronization \u0026 RCU","sidebarTitle":"Synchronization","icon":"hand-middle-finger","href":"/operating-systems/synchronization"},{"title":"Deadlocks","sidebarTitle":"Deadlocks","icon":"circle-xmark","href":"/operating-systems/deadlocks"},{"title":"Inter-Process Communication: Kernel Internals \u0026 Architecture","sidebarTitle":"IPC","icon":"message","href":"/operating-systems/ipc"},{"title":"Virtual File System (VFS) \u0026 Disk Internals","sidebarTitle":"File Systems","icon":"file-lines","href":"/operating-systems/file-systems"},{"title":"I/O Systems \u0026 Modern I/O","sidebarTitle":"I/O Systems","icon":"input-pipe","href":"/operating-systems/io-systems"}]},{"group":"Networking","pages":[{"title":"Kernel Networking Stack: From NIC to Socket","sidebarTitle":"Networking","icon":"network-wired","href":"/operating-systems/networking"}]},{"group":"Virtualization","pages":[{"title":"Containers \u0026 Virtualization: From Namespaces to Hypervisors","sidebarTitle":"Containers \u0026 VMs","icon":"cubes","href":"/operating-systems/containers-virtualization"}]},{"group":"Advanced Topics","pages":[{"title":"OS Security: From Hardware to Userspace","sidebarTitle":"Security","icon":"shield-halved","href":"/operating-systems/security"},{"title":"Linux Internals","sidebarTitle":"Linux Internals","icon":"linux","href":"/operating-systems/linux-internals"},{"title":"Modern OS Features","sidebarTitle":"Modern Features","icon":"rocket","href":"/operating-systems/modern-features"},{"title":"Observability \u0026 Performance Engineering","sidebarTitle":"Performance","icon":"gauge-high","href":"/operating-systems/debugging-performance"}]},{"group":"Interview Prep","pages":[{"title":"Interview Preparation","sidebarTitle":"Interview Prep","icon":"user-tie","href":"/operating-systems/interview-prep"},{"title":"OS Case Studies","sidebarTitle":"Case Studies","icon":"building","href":"/operating-systems/case-studies"}]}]},{"group":"Linux Internals","icon":"linux","pages":[{"title":"Linux Internals \u0026 Kernel Mastery","sidebarTitle":"Course Overview","icon":"linux","href":"/courses/linux-internals/overview"},{"group":"Kernel Foundations","pages":[{"title":"Linux Kernel Architecture","sidebarTitle":"Kernel Architecture","icon":"sitemap","href":"/courses/linux-internals/kernel-architecture"},{"title":"System Call Interface","sidebarTitle":"System Calls","icon":"terminal","href":"/courses/linux-internals/syscalls"},{"title":"Kernel Data Structures","sidebarTitle":"Data Structures","icon":"database","href":"/courses/linux-internals/data-structures"},{"title":"Interrupts \u0026 Exception Handling","sidebarTitle":"Interrupts","icon":"bolt","href":"/courses/linux-internals/interrupts-exceptions"},{"title":"Kernel Synchronization","sidebarTitle":"Synchronization","icon":"lock","href":"/courses/linux-internals/synchronization"}]},{"group":"Process \u0026 Memory","pages":[{"title":"Process Subsystem Deep Dive","sidebarTitle":"Process Subsystem","icon":"microchip","href":"/courses/linux-internals/process-subsystem"},{"title":"Memory Management Internals","sidebarTitle":"Memory Management","icon":"memory","href":"/courses/linux-internals/memory-management"},{"title":"Signals \u0026 IPC","sidebarTitle":"Signals \u0026 IPC","icon":"arrows-split-up-and-left","href":"/courses/linux-internals/signals-ipc"}]},{"group":"Containers \u0026 Isolation","pages":[{"title":"Namespaces Deep Dive","sidebarTitle":"Namespaces","icon":"boxes-stacked","href":"/courses/linux-internals/namespaces"},{"title":"Control Groups (cgroups)","sidebarTitle":"cgroups","icon":"object-group","href":"/courses/linux-internals/cgroups"},{"title":"Security Modules \u0026 Capabilities","sidebarTitle":"Security","icon":"shield-halved","href":"/courses/linux-internals/security-modules"}]},{"group":"I/O \u0026 Networking","pages":[{"title":"Filesystem \u0026 VFS","sidebarTitle":"Filesystem \u0026 VFS","icon":"folder-tree","href":"/courses/linux-internals/filesystem-vfs"},{"title":"I/O Subsystem","sidebarTitle":"I/O Subsystem","icon":"hard-drive","href":"/courses/linux-internals/io-subsystem"},{"title":"Network Stack","sidebarTitle":"Network Stack","icon":"network-wired","href":"/courses/linux-internals/networking-stack"}]},{"group":"Observability","pages":[{"title":"eBPF Deep Dive","sidebarTitle":"eBPF","icon":"magnifying-glass-chart","href":"/courses/linux-internals/ebpf"},{"title":"Tracing \u0026 Profiling","sidebarTitle":"Tracing \u0026 Profiling","icon":"chart-line","href":"/courses/linux-internals/tracing-profiling"}]},{"group":"Advanced Kernel","pages":[{"title":"Power \u0026 Thermal Management","sidebarTitle":"Power Management","icon":"bolt","href":"/courses/linux-internals/power-management"}]},{"group":"Interview Prep","pages":[{"title":"Hands-on Projects","sidebarTitle":"Projects","icon":"hammer","href":"/courses/linux-internals/projects"},{"title":"Real Interview Questions","sidebarTitle":"Interview Questions","icon":"clipboard-question","href":"/courses/linux-internals/interview-questions"}]}]},{"group":"Distributed Systems","icon":"sitemap","pages":[{"title":"Distributed Systems Mastery","sidebarTitle":"Course Overview","icon":"network-wired","href":"/courses/distributed-systems/overview"},{"group":"Core Foundations","pages":[{"title":"Foundations of Distributed Systems","sidebarTitle":"Foundations","icon":"book","href":"/courses/distributed-systems/foundations"},{"title":"Distributed Networking Fundamentals","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/distributed-systems/networking"},{"title":"Time \u0026 Clock Synchronization","sidebarTitle":"Clocks","icon":"clock","href":"/courses/distributed-systems/clocks"},{"title":"Consistency Models","sidebarTitle":"Consistency Models","icon":"scale-balanced","href":"/courses/distributed-systems/consistency-models"},{"title":"Distributed Snapshots","sidebarTitle":"Distributed Snapshots","icon":"camera","href":"/courses/distributed-systems/snapshots"}]},{"group":"Consensus \u0026 Replication","pages":[{"title":"Consensus Protocols","sidebarTitle":"Consensus","icon":"handshake","href":"/courses/distributed-systems/consensus"},{"title":"Raft Consensus Deep Dive","sidebarTitle":"Raft Deep Dive","icon":"star","href":"/courses/distributed-systems/raft-deep-dive"},{"title":"Paxos Consensus Deep Dive","sidebarTitle":"Paxos Deep Dive","icon":"scroll","href":"/courses/distributed-systems/paxos-deep-dive"},{"title":"Replication Strategies","sidebarTitle":"Replication","icon":"clone","href":"/courses/distributed-systems/replication"},{"title":"Distributed Coordination","sidebarTitle":"Coordination","icon":"people-group","href":"/courses/distributed-systems/coordination"},{"title":"Byzantine Fault Tolerance (BFT)","sidebarTitle":"Byzantine Fault Tolerance","icon":"shield-halved","href":"/courses/distributed-systems/bft"},{"title":"Gossip Protocols \u0026 Membership","sidebarTitle":"Gossip Protocols","icon":"comments","href":"/courses/distributed-systems/gossip"},{"title":"Formal Verification in Distributed Systems","sidebarTitle":"Formal Verification","icon":"shield-check","href":"/courses/distributed-systems/formal-verification"}]},{"group":"Data \u0026 Transactions","pages":[{"title":"Distributed Transactions","sidebarTitle":"Transactions","icon":"money-bill-transfer","href":"/courses/distributed-systems/transactions"},{"title":"Data Partitioning \u0026 Sharding","sidebarTitle":"Partitioning","icon":"table-cells","href":"/courses/distributed-systems/partitioning"},{"title":"Data Systems at Scale","sidebarTitle":"Data Systems","icon":"database","href":"/courses/distributed-systems/data-systems"}]},{"group":"Infrastructure \u0026 Messaging","pages":[{"title":"Distributed Messaging \u0026 Event Systems","sidebarTitle":"Messaging","icon":"envelope","href":"/courses/distributed-systems/messaging"},{"title":"Distributed Caching","sidebarTitle":"Caching","icon":"bolt","href":"/courses/distributed-systems/caching"},{"title":"Fault Tolerance Patterns","sidebarTitle":"Fault Tolerance","icon":"shield-heart","href":"/courses/distributed-systems/fault-tolerance"}]},{"group":"Production \u0026 Practice","pages":[{"title":"Production Excellence","sidebarTitle":"Production","icon":"shield-halved","href":"/courses/distributed-systems/production"},{"title":"Real-World Case Studies","sidebarTitle":"Case Studies","icon":"building","href":"/courses/distributed-systems/case-studies"},{"title":"Interview Practice Problems","sidebarTitle":"Interview Problems","icon":"comments","href":"/courses/distributed-systems/interview-problems"}]}]},{"group":"Networking Mastery","icon":"network-wired","pages":[{"title":"Networking Fundamentals","sidebarTitle":"1. Overview","icon":"globe","href":"/courses/networking-mastery/01-overview"},{"group":"Fundamentals","pages":[{"title":"Network Models","sidebarTitle":"2. OSI \u0026 TCP/IP","icon":"layer-group","href":"/courses/networking-mastery/02-models"},{"title":"Physical \u0026 Data Link Layers","sidebarTitle":"3. Physical \u0026 Data Link","icon":"ethernet","href":"/courses/networking-mastery/03-physical-datalink"},{"title":"Network Layer","sidebarTitle":"4. Network Layer","icon":"network-wired","href":"/courses/networking-mastery/04-network-layer"},{"title":"Transport Layer","sidebarTitle":"5. Transport Layer","icon":"truck-fast","href":"/courses/networking-mastery/05-transport-layer"},{"title":"Application Layer","sidebarTitle":"6. Application Layer","icon":"browser","href":"/courses/networking-mastery/06-application-layer"}]},{"group":"Security \u0026 Advanced Intro","pages":[{"title":"Network Security","sidebarTitle":"7. Security","icon":"shield-check","href":"/courses/networking-mastery/07-security"},{"title":"Advanced Networking Intro","sidebarTitle":"8. Advanced Topics","icon":"cloud","href":"/courses/networking-mastery/08-advanced"}]},{"group":"Deep Dives","pages":[{"title":"IP Addressing Deep Dive","sidebarTitle":"9. IP Addressing Mastery","icon":"map-location-dot","href":"/courses/networking-mastery/09-ip-addressing-deep-dive"},{"title":"NAT \u0026 PAT Deep Dive","sidebarTitle":"10. NAT Mastery","icon":"right-left","href":"/courses/networking-mastery/10-nat-deep-dive"},{"title":"Routing Deep Dive","sidebarTitle":"11. Routing Mastery","icon":"route","href":"/courses/networking-mastery/11-routing-deep-dive"},{"title":"DNS Deep Dive","sidebarTitle":"12. DNS Mastery","icon":"book","href":"/courses/networking-mastery/12-dns-deep-dive"},{"title":"Load Balancing \u0026 Proxies","sidebarTitle":"13. Load Balancing","icon":"scale-balanced","href":"/courses/networking-mastery/13-load-balancing"}]},{"group":"Operations \u0026 Security","pages":[{"title":"Network Troubleshooting","sidebarTitle":"14. Troubleshooting","icon":"wrench","href":"/courses/networking-mastery/14-troubleshooting"},{"title":"VPNs \u0026 Tunneling","sidebarTitle":"15. VPNs \u0026 Tunneling","icon":"tunnel","href":"/courses/networking-mastery/15-vpn-tunneling"},{"title":"Firewalls \u0026 Security Groups","sidebarTitle":"17. Firewalls","icon":"shield-halved","href":"/courses/networking-mastery/17-firewalls-security-groups"}]},{"group":"Modern Networking","pages":[{"title":"Container Networking","sidebarTitle":"18. Container Networking","icon":"cubes","href":"/courses/networking-mastery/18-container-networking"},{"title":"Real-World Networking Scenarios","sidebarTitle":"16. Network Scenarios","icon":"diagram-project","href":"/courses/networking-mastery/16-scenarios"}]}]},{"group":"Build Your Own X","icon":"hammer","pages":[{"title":"Build Your Own X","sidebarTitle":"Course Overview","icon":"hammer","href":"/courses/build-your-own-x/overview"},{"group":"Build Your Own Git","pages":[{"title":"Build Your Own Git","sidebarTitle":"Git Overview","icon":"code-branch","href":"/courses/build-your-own-x/git-overview"},{"title":"Chapter 1: Project Setup \u0026 Init","sidebarTitle":"1. Setup \u0026 Init","icon":"folder-open","href":"/courses/build-your-own-x/git-1-setup"},{"title":"Chapter 2: The Object Model","sidebarTitle":"2. Object Model","icon":"cube","href":"/courses/build-your-own-x/git-2-objects"},{"title":"Chapter 3: Staging \u0026 Index","sidebarTitle":"3. Staging \u0026 Index","icon":"layer-plus","href":"/courses/build-your-own-x/git-3-staging"},{"title":"Chapter 4: Commits \u0026 History","sidebarTitle":"4. Commits \u0026 History","icon":"code-commit","href":"/courses/build-your-own-x/git-4-commits"},{"title":"Chapter 5: Branches \u0026 Checkout","sidebarTitle":"5. Branches \u0026 Checkout","icon":"code-branch","href":"/courses/build-your-own-x/git-5-branches"}]},{"group":"Build Your Own Redis","pages":[{"title":"Build Your Own Redis","sidebarTitle":"Redis Overview","icon":"database","href":"/courses/build-your-own-x/redis-overview"},{"title":"Chapter 1: RESP Protocol","sidebarTitle":"1. RESP Protocol","icon":"network-wired","href":"/courses/build-your-own-x/redis-1-protocol"},{"title":"Chapter 2: TCP Server","sidebarTitle":"2. TCP Server","icon":"server","href":"/courses/build-your-own-x/redis-2-server"},{"title":"Chapter 3: Data Structures","sidebarTitle":"3. Data Structures","icon":"layer-group","href":"/courses/build-your-own-x/redis-3-data-structures"},{"title":"Chapter 4: Persistence","sidebarTitle":"4. Persistence","icon":"floppy-disk","href":"/courses/build-your-own-x/redis-4-persistence"},{"title":"Chapter 5: Pub/Sub","sidebarTitle":"5. Pub/Sub","icon":"tower-broadcast","href":"/courses/build-your-own-x/redis-5-pubsub"}]},{"group":"Build Your Own Docker","pages":[{"title":"Build Your Own Docker","sidebarTitle":"Docker Overview","icon":"docker","href":"/courses/build-your-own-x/docker-overview"},{"title":"Chapter 1: Linux Namespaces","sidebarTitle":"1. Namespaces","icon":"box","href":"/courses/build-your-own-x/docker-1-namespaces"},{"title":"Chapter 2: Control Groups (cgroups)","sidebarTitle":"2. Cgroups","icon":"gauge-high","href":"/courses/build-your-own-x/docker-2-cgroups"},{"title":"Chapter 3: Filesystem Isolation","sidebarTitle":"3. Filesystem","icon":"folder-tree","href":"/courses/build-your-own-x/docker-3-filesystem"},{"title":"Chapter 4: Container Networking","sidebarTitle":"4. Networking","icon":"network-wired","href":"/courses/build-your-own-x/docker-4-networking"},{"title":"Chapter 5: Container Images","sidebarTitle":"5. Images","icon":"box-archive","href":"/courses/build-your-own-x/docker-5-images"}]}]},{"group":"Go Lang Mastery","icon":"golang","pages":[{"group":"Fundamentals","pages":[{"title":"Introduction to Go","sidebarTitle":"1. Introduction","icon":"golang","href":"/courses/go-lang-mastery/01-introduction"},{"title":"Variables and Types","sidebarTitle":"2. Variables \u0026 Types","icon":"cube","href":"/courses/go-lang-mastery/02-variables-and-types"},{"title":"Control Structures","sidebarTitle":"3. Control Structures","icon":"code-branch","href":"/courses/go-lang-mastery/03-control-structures"},{"title":"Functions and Packages","sidebarTitle":"4. Functions \u0026 Packages","icon":"box-open","href":"/courses/go-lang-mastery/04-functions-and-packages"},{"title":"Arrays, Slices, and Maps","sidebarTitle":"5. Arrays, Slices \u0026 Maps","icon":"layer-group","href":"/courses/go-lang-mastery/05-arrays-slices-maps"},{"title":"Pointers and Structs","sidebarTitle":"6. Pointers \u0026 Structs","icon":"shapes","href":"/courses/go-lang-mastery/06-pointers-and-structs"},{"title":"Interfaces","sidebarTitle":"7. Interfaces","icon":"plug","href":"/courses/go-lang-mastery/07-interfaces"}]},{"group":"Core Concepts","pages":[{"title":"08. Concurrency","icon":"arrows-split-up-and-left","href":"/courses/go-lang-mastery/08-concurrency"},{"title":"09. Error Handling","icon":"triangle-exclamation","href":"/courses/go-lang-mastery/09-error-handling"},{"title":"10. Testing and Benchmarking","icon":"vial","href":"/courses/go-lang-mastery/10-testing-and-benchmarking"},{"title":"Generics","sidebarTitle":"11. Generics","icon":"cube","href":"/courses/go-lang-mastery/11-generics"},{"title":"Advanced Concurrency Patterns","sidebarTitle":"12. Advanced Concurrency","icon":"diagram-project","href":"/courses/go-lang-mastery/12-advanced-concurrency"},{"title":"Context Package","sidebarTitle":"13. Context","icon":"clock","href":"/courses/go-lang-mastery/13-context"}]},{"group":"Production Go","pages":[{"title":"HTTP \u0026 Web Development","sidebarTitle":"14. HTTP \u0026 Web","icon":"globe","href":"/courses/go-lang-mastery/14-http-web"},{"title":"Database Operations","sidebarTitle":"15. Databases","icon":"database","href":"/courses/go-lang-mastery/15-databases"},{"title":"Microservices \u0026 gRPC","sidebarTitle":"16. Microservices \u0026 gRPC","icon":"sitemap","href":"/courses/go-lang-mastery/16-microservices-grpc"},{"title":"Performance Optimization","sidebarTitle":"17. Performance","icon":"gauge-high","href":"/courses/go-lang-mastery/17-performance"},{"title":"Production Best Practices","sidebarTitle":"18. Production","icon":"rocket","href":"/courses/go-lang-mastery/18-production"}]},{"group":"Advanced Topics","pages":[{"title":"Design Patterns in Go","sidebarTitle":"19. Design Patterns","icon":"puzzle-piece","href":"/courses/go-lang-mastery/19-design-patterns"},{"title":"Interview Preparation","sidebarTitle":"20. Interview Prep","icon":"user-tie","href":"/courses/go-lang-mastery/20-interview-prep"}]}]},{"group":"C Programming","icon":"memory","pages":[{"title":"Advanced C Programming for Systems Development","sidebarTitle":"Course Overview","icon":"microchip","href":"/courses/c-programming/overview"},{"group":"Foundations","pages":[{"title":"C Syntax Speed Run","sidebarTitle":"Syntax Speed Run","icon":"forward-fast","href":"/courses/c-programming/syntax-speedrun"},{"title":"Build Systems \u0026 Toolchain","sidebarTitle":"Build \u0026 Toolchain","icon":"hammer","href":"/courses/c-programming/toolchain"},{"title":"Debugging Fundamentals","sidebarTitle":"Debugging","icon":"bug","href":"/courses/c-programming/debugging"}]},{"group":"Memory Mastery","pages":[{"title":"Pointers Deep Dive","sidebarTitle":"Pointers","icon":"arrow-pointer","href":"/courses/c-programming/pointers"},{"title":"Memory Layout \u0026 Segments","sidebarTitle":"Memory Layout","icon":"layer-group","href":"/courses/c-programming/memory-layout"},{"title":"Dynamic Memory Management","sidebarTitle":"Dynamic Memory","icon":"memory","href":"/courses/c-programming/dynamic-memory"},{"title":"Preprocessor Mastery","sidebarTitle":"Preprocessor","icon":"code","href":"/courses/c-programming/preprocessor"},{"title":"Data Structures in C","sidebarTitle":"Data Structures","icon":"sitemap","href":"/courses/c-programming/data-structures"},{"title":"Undefined Behavior","sidebarTitle":"Undefined Behavior","icon":"skull","href":"/courses/c-programming/undefined-behavior"}]},{"group":"Systems Programming","pages":[{"title":"System Calls \u0026 POSIX","sidebarTitle":"System Calls","icon":"terminal","href":"/courses/c-programming/system-calls"},{"title":"Concurrency \u0026 Threading","sidebarTitle":"Concurrency","icon":"arrows-split-up-and-left","href":"/courses/c-programming/concurrency"},{"title":"Network Programming","sidebarTitle":"Networking","icon":"network-wired","href":"/courses/c-programming/networking"}]},{"group":"Projects","pages":[{"title":"Project: Build a Shell","sidebarTitle":"Project: Shell","icon":"terminal","href":"/courses/c-programming/project-shell"},{"title":"Project: Key-Value Store","sidebarTitle":"Project: KV Store","icon":"database","href":"/courses/c-programming/project-database"},{"title":"Project: Memory Allocator","sidebarTitle":"Project: malloc","icon":"memory","href":"/courses/c-programming/project-allocator"},{"title":"Project: HTTP Server","sidebarTitle":"Project: HTTP Server","icon":"server","href":"/courses/c-programming/project-http-server"}]},{"group":"Advanced Topics","pages":[{"title":"Performance Optimization","sidebarTitle":"Performance","icon":"gauge-high","href":"/courses/c-programming/performance"},{"title":"Embedded Systems","sidebarTitle":"Embedded","icon":"microchip","href":"/courses/c-programming/embedded"},{"title":"Linux Kernel Modules","sidebarTitle":"Kernel Modules","icon":"linux","href":"/courses/c-programming/kernel-modules"}]}]},{"group":"Classic Research Papers","icon":"file-lines","pages":[{"group":"Google File System (GFS)","icon":"folder","pages":[{"title":"Google File System","sidebarTitle":"GFS Overview","icon":"database","href":"/courses/engineering-papers/gfs-overview"},{"title":"Introduction and Motivation","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/chapter-1-introduction"},{"title":"Architecture Overview","sidebarTitle":"Chapter 2: Architecture","icon":"diagram-project","href":"/courses/engineering-papers/chapter-2-architecture"},{"title":"Master Operations","sidebarTitle":"Chapter 3: Master Operations","icon":"crown","href":"/courses/engineering-papers/chapter-3-master-operations"},{"title":"Chunkservers and Data Flow","sidebarTitle":"Chapter 4: Chunkservers \u0026 Data Flow","icon":"server","href":"/courses/engineering-papers/chapter-4-chunkservers-dataflow"},{"title":"Consistency Model","sidebarTitle":"Chapter 5: Consistency Model","icon":"scale-balanced","href":"/courses/engineering-papers/chapter-5-consistency-model"},{"title":"Fault Tolerance","sidebarTitle":"Chapter 6: Fault Tolerance","icon":"shield-halved","href":"/courses/engineering-papers/chapter-6-fault-tolerance"},{"title":"Performance and Optimizations","sidebarTitle":"Chapter 7: Performance","icon":"gauge-high","href":"/courses/engineering-papers/chapter-7-performance"},{"title":"Impact and Evolution","sidebarTitle":"Chapter 8: Impact \u0026 Evolution","icon":"rocket","href":"/courses/engineering-papers/chapter-8-impact-evolution"}]},{"group":"Apache Hadoop","icon":"elephant","pages":[{"title":"Apache Hadoop","sidebarTitle":"Hadoop Overview","icon":"database","href":"/courses/engineering-papers/hadoop-overview"},{"title":"Introduction and Origins","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/hadoop-chapter-1-introduction"},{"title":"HDFS Architecture","sidebarTitle":"Chapter 2: HDFS","icon":"hard-drive","href":"/courses/engineering-papers/hadoop-chapter-2-hdfs-architecture"},{"title":"MapReduce Framework","sidebarTitle":"Chapter 3: MapReduce","icon":"diagram-project","href":"/courses/engineering-papers/hadoop-chapter-3-mapreduce-framework"},{"title":"YARN: Resource Management","sidebarTitle":"Chapter 4: YARN","icon":"layer-group","href":"/courses/engineering-papers/hadoop-chapter-4-yarn"},{"title":"Hadoop Ecosystem","sidebarTitle":"Chapter 5: Ecosystem","icon":"puzzle-piece","href":"/courses/engineering-papers/hadoop-chapter-5-ecosystem"},{"title":"Fault Tolerance and Reliability","sidebarTitle":"Chapter 6: Fault Tolerance","icon":"shield-check","href":"/courses/engineering-papers/hadoop-chapter-6-fault-tolerance"},{"title":"Performance Tuning and Optimization","sidebarTitle":"Chapter 7: Performance","icon":"gauge-high","href":"/courses/engineering-papers/hadoop-chapter-7-performance"},{"title":"Hadoop in Production","sidebarTitle":"Chapter 8: Production","icon":"server","href":"/courses/engineering-papers/hadoop-chapter-8-production"}]},{"group":"Amazon DynamoDB","icon":"database","pages":[{"title":"Amazon DynamoDB","sidebarTitle":"DynamoDB Overview","icon":"database","href":"/courses/engineering-papers/dynamodb-overview"},{"title":"Introduction and Origins","sidebarTitle":"Chapter 1: Introduction","icon":"flag","href":"/courses/engineering-papers/dynamodb-chapter-1-introduction"},{"title":"Architecture and Partitioning","sidebarTitle":"Chapter 2: Architecture","icon":"sitemap","href":"/courses/engineering-papers/dynamodb-chapter-2-architecture"},{"title":"Data Modeling in DynamoDB","sidebarTitle":"Chapter 3: Data Model","icon":"database","href":"/courses/engineering-papers/dynamodb-chapter-3-data-model"},{"title":"Consistency Models in DynamoDB","sidebarTitle":"Chapter 4: Consistency","icon":"scale-balanced","href":"/courses/engineering-papers/dynamodb-chapter-4-consistency"},{"title":"Performance Optimization in DynamoDB","sidebarTitle":"Chapter 5: Performance","icon":"gauge-high","href":"/courses/engineering-papers/dynamodb-chapter-5-performance"},{"title":"Advanced DynamoDB Features","sidebarTitle":"Chapter 6: Advanced Features","icon":"wand-magic-sparkles","href":"/courses/engineering-papers/dynamodb-chapter-6-advanced-features"},{"title":"Fault Tolerance and Reliability","sidebarTitle":"Chapter 7: Fault Tolerance","icon":"shield-check","href":"/courses/engineering-papers/dynamodb-chapter-7-fault-tolerance"},{"title":"DynamoDB in Production","sidebarTitle":"Chapter 8: Production","icon":"server","href":"/courses/engineering-papers/dynamodb-chapter-8-production"}]}]},{"group":"Distributed System Tools","icon":"server","pages":[{"group":"Apache Hadoop","icon":"elephant","pages":[{"title":"Apache Hadoop Mastery","sidebarTitle":"Hadoop Overview","icon":"elephant","href":"/distributed-systems-tools/hadoop-overview"},{"title":"Introduction \u0026 Foundational Papers","sidebarTitle":"1. Introduction \u0026 Papers","icon":"book-open","href":"/distributed-systems-tools/hadoop-introduction"},{"title":"HDFS Architecture \u0026 Internals","sidebarTitle":"2. HDFS Architecture","icon":"database","href":"/distributed-systems-tools/hadoop-hdfs"},{"title":"Advanced HDFS: Federation, HA \u0026 Erasure Coding","sidebarTitle":"2b. Advanced HDFS","icon":"layer-group","href":"/distributed-systems-tools/hadoop-hdfs-advanced"},{"title":"MapReduce Programming Model","sidebarTitle":"3. MapReduce Programming","icon":"code","href":"/distributed-systems-tools/hadoop-mapreduce"},{"title":"YARN Resource Management","sidebarTitle":"4. YARN Architecture","icon":"layer-group","href":"/distributed-systems-tools/hadoop-yarn"},{"title":"Hadoop Ecosystem \u0026 Integration","sidebarTitle":"5. Ecosystem Tools","icon":"puzzle-piece","href":"/distributed-systems-tools/hadoop-ecosystem"},{"title":"Data Processing Patterns \u0026 Best Practices","sidebarTitle":"6. Processing Patterns","icon":"diagram-project","href":"/distributed-systems-tools/hadoop-patterns"},{"title":"Production Deployment \u0026 Operations","sidebarTitle":"7. Production Operations","icon":"server","href":"/distributed-systems-tools/hadoop-production"},{"title":"Capstone Project: End-to-End Data Pipeline","sidebarTitle":"8. Capstone Project","icon":"trophy","href":"/distributed-systems-tools/hadoop-capstone"}]},{"group":"Apache Spark","icon":"bolt","pages":[{"title":"Apache Spark Mastery","sidebarTitle":"Spark Overview","icon":"bolt","href":"/distributed-systems-tools/spark-overview"},{"title":"Demystifying the RDD Paper: Spark's Foundation","sidebarTitle":"1. RDD Paper Deep-Dive","icon":"book-open","href":"/distributed-systems-tools/spark-introduction"},{"title":"Demystifying RDD Programming: From Theory to Production","sidebarTitle":"2. RDD Programming","icon":"bolt","href":"/distributed-systems-tools/spark-rdd"},{"title":"Spark SQL \u0026 DataFrames","sidebarTitle":"3. Spark SQL","icon":"table","href":"/distributed-systems-tools/spark-sql"},{"title":"Structured Streaming","sidebarTitle":"4. Streaming","icon":"bolt","href":"/distributed-systems-tools/spark-streaming"},{"title":"MLlib for Machine Learning","sidebarTitle":"5. MLlib","icon":"bolt","href":"/distributed-systems-tools/spark-mllib"},{"title":"Performance Tuning","sidebarTitle":"6. Performance Tuning","icon":"bolt","href":"/distributed-systems-tools/spark-tuning"},{"title":"Cluster Deployment","sidebarTitle":"7. Operations","icon":"bolt","href":"/distributed-systems-tools/spark-operations"},{"title":"Advanced Topics","sidebarTitle":"8. Advanced","icon":"bolt","href":"/distributed-systems-tools/spark-advanced"},{"title":"Capstone Project","sidebarTitle":"9. Capstone","icon":"bolt","href":"/distributed-systems-tools/spark-capstone"}]},{"group":"Apache Flink","icon":"wave-pulse","pages":[{"title":"Apache Flink Mastery","sidebarTitle":"Flink Overview","icon":"wave-pulse","href":"/distributed-systems-tools/flink-overview"},{"title":"Demystifying the Dataflow Model: True Stream Processing with Flink","sidebarTitle":"1. Introduction","icon":"wave-pulse","href":"/distributed-systems-tools/flink-introduction"},{"title":"DataStream API","sidebarTitle":"2. DataStream API","icon":"wave-pulse","href":"/distributed-systems-tools/flink-datastream"},{"title":"Event Time \u0026 Watermarks","sidebarTitle":"3. Event Time","icon":"wave-pulse","href":"/distributed-systems-tools/flink-eventtime"},{"title":"Windows \u0026 Time Operations","sidebarTitle":"4. Windows","icon":"wave-pulse","href":"/distributed-systems-tools/flink-windows"},{"title":"Demystifying Stateful Stream Processing: Flink's State Management Deep Dive","sidebarTitle":"5. State \u0026 Timers","icon":"database","href":"/distributed-systems-tools/flink-state"},{"title":"Declarative Stream Processing: Flink SQL \u0026 Table API Deep Dive","sidebarTitle":"6. SQL \u0026 Table API","icon":"table","href":"/distributed-systems-tools/flink-sql"},{"title":"Complex Event Processing: Pattern Detection at Scale with Flink CEP","sidebarTitle":"7. Complex Event Processing","icon":"sitemap","href":"/distributed-systems-tools/flink-cep"},{"title":"Production Flink: Deployment, Operations \u0026 Monitoring at Scale","sidebarTitle":"8. Production Operations","icon":"server","href":"/distributed-systems-tools/flink-operations"},{"title":"Capstone: Production-Grade Real-Time Fraud Detection System","sidebarTitle":"9. Capstone Project","icon":"diagram-project","href":"/distributed-systems-tools/flink-capstone"}]},{"group":"Apache Beam","icon":"diagram-project","pages":[{"title":"Apache Beam Mastery","sidebarTitle":"Beam Overview","icon":"diagram-project","href":"/distributed-systems-tools/beam-overview"},{"title":"Demystifying the Dataflow Model: Write Once, Run Anywhere with Apache Beam","sidebarTitle":"1. Introduction","icon":"diagram-project","href":"/distributed-systems-tools/beam-introduction"},{"title":"Core Programming Model","sidebarTitle":"2. Core Model","icon":"diagram-project","href":"/distributed-systems-tools/beam-core"},{"title":"Windowing \u0026 Time","sidebarTitle":"3. Windowing","icon":"diagram-project","href":"/distributed-systems-tools/beam-windowing"},{"title":"Triggers \u0026 Watermarks","sidebarTitle":"4. Triggers","icon":"diagram-project","href":"/distributed-systems-tools/beam-triggers"},{"title":"State \u0026 Timers","sidebarTitle":"5. State","icon":"diagram-project","href":"/distributed-systems-tools/beam-state"},{"title":"Beam I/O Connectors","sidebarTitle":"6. I/O","icon":"diagram-project","href":"/distributed-systems-tools/beam-io"},{"title":"Beam SQL","sidebarTitle":"7. SQL","icon":"diagram-project","href":"/distributed-systems-tools/beam-sql"},{"title":"Multi-Runner Execution","sidebarTitle":"8. Runners","icon":"diagram-project","href":"/distributed-systems-tools/beam-runners"},{"title":"Capstone Project","sidebarTitle":"9. Capstone","icon":"diagram-project","href":"/distributed-systems-tools/beam-capstone"}]},{"group":"Apache Airflow","icon":"wind","pages":[{"title":"What is Apache Airflow?","sidebarTitle":"Overview","icon":"wind","href":"/distributed-systems-tools/airflow-overview"},{"title":"Airflow Core Concepts","sidebarTitle":"Core Concepts","icon":"diagram-project","href":"/distributed-systems-tools/airflow-core-concepts"},{"title":"Airflow Operators","sidebarTitle":"Operators","icon":"puzzle-piece","href":"/distributed-systems-tools/airflow-operators"},{"title":"Airflow Scheduling","sidebarTitle":"Scheduling","icon":"clock","href":"/distributed-systems-tools/airflow-scheduling"}]},{"group":"Apache Cassandra","icon":"database","pages":[{"title":"Apache Cassandra Mastery","sidebarTitle":"Overview","icon":"database","href":"/distributed-systems-tools/cassandra-overview"},{"title":"The Cassandra Paper \u0026 Core Architecture","sidebarTitle":"1. Introduction \u0026 Paper","icon":"book-open","href":"/distributed-systems-tools/cassandra-paper"},{"title":"Cassandra Architecture \u0026 Storage Internals","sidebarTitle":"2. Architecture \u0026 Internals","icon":"diagram-project","href":"/distributed-systems-tools/cassandra-architecture-internals"},{"title":"Query-Driven Data Modeling in Cassandra","sidebarTitle":"3. Data Modeling","icon":"table","href":"/distributed-systems-tools/cassandra-data-modeling"},{"title":"Cassandra Read \u0026 Write Path Internals","sidebarTitle":"4. Read/Write Internals","icon":"route","href":"/distributed-systems-tools/cassandra-read-write-internals"},{"title":"Cluster Operations \u0026 Multi-DC Replication","sidebarTitle":"5. Cluster Operations","icon":"server","href":"/distributed-systems-tools/cassandra-cluster-operations"},{"title":"Performance Tuning \u0026 Production Operations","sidebarTitle":"6. Performance \u0026 Production","icon":"gauge-high","href":"/distributed-systems-tools/cassandra-performance-production"},{"title":"Capstone: Building a Real-Time Analytics Platform","sidebarTitle":"7. Capstone Project","icon":"rocket","href":"/distributed-systems-tools/cassandra-capstone"}]},{"group":"Neo4j Graph Database","icon":"diagram-project","pages":[{"title":"Neo4j Graph Database Mastery","sidebarTitle":"Overview","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-overview"},{"title":"The Graph Database Story: From Theory to Neo4j","sidebarTitle":"2. Foundational Papers","icon":"scroll","href":"/distributed-systems-tools/neo4j-paper"},{"title":"Neo4j Architecture \u0026 Native Graph Storage","sidebarTitle":"3. Architecture \u0026 Storage","icon":"database","href":"/distributed-systems-tools/neo4j-architecture-storage"},{"title":"Cypher Query Language Mastery","sidebarTitle":"4. Cypher Mastery","icon":"code","href":"/distributed-systems-tools/neo4j-cypher-mastery"},{"title":"Graph Data Modeling Best Practices","sidebarTitle":"5. Data Modeling","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-data-modeling"},{"title":"Graph Algorithms at Scale","sidebarTitle":"6. Graph Algorithms","icon":"diagram-project","href":"/distributed-systems-tools/neo4j-graph-algorithms"},{"title":"Production Deployment \u0026 Operations","sidebarTitle":"7. Production \u0026 Ops","icon":"server","href":"/distributed-systems-tools/neo4j-production-deployment"},{"title":"Capstone: Building a Knowledge Graph Platform","sidebarTitle":"8. Capstone Project","icon":"rocket","href":"/distributed-systems-tools/neo4j-capstone"}]}]}]}]},"basePath":"","title":"Attention Mechanism - Dev Weekends","deploymentId":"dpl_vSEmv961Ncyvz1psVwgUnTXQMSYk","buildId":"dpl_vSEmv961Ncyvz1psVwgUnTXQMSYk","dynamics":["07c022d89b","a74be98ba7"],"chunks":["Frame-BRsfsIIG.js","DynamicMarkdown-NFqqduU7.js","zoom-image-lightbox-BguUeq2N.js","lightbox-handle-r17bVUz3.js"]}