Skip to main content

Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

Recurrent Neural Networks

​
Recurrent Neural Networks

​
The Problem with Sequences

CNNs transformed computer vision. But they have a fundamental limitation: they assume fixed-size inputs. What about:
  • Text: “The cat sat on the mat” (6 words)
  • Time series: Stock prices over months (varying length)
  • Audio: Speech of different durations
  • Video: Frames over time
These are sequences - data where order matters and length varies.
The Core Insight: Sequences have temporal dependencies. The word “sat” depends on knowing “cat” came before it. Today’s stock price depends on yesterday’s. We need networks that can remember.The fundamental difference between a CNN and an RNN: a CNN asks “what is here?”, while an RNN asks “what just happened, and what does that mean for what comes next?” CNNs are spatial; RNNs are temporal.
Think of a feedforward network as someone with amnesia reading a book word by word. Each word is processed in isolation — when they reach “mat,” they have no memory of “cat” or “sat.” An RNN is like a normal reader: they carry a running mental summary of everything they have read so far. Each new word updates that summary. The summary is lossy — you cannot perfectly reconstruct every previous word from it — but it captures the gist that matters for understanding the next word. This running summary is the hidden state.

​
From Feedforward to Recurrent

​
The Feedforward Limitation

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# A feedforward network processes each input independently
class FeedforwardClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        # Each input processed independently
        # No memory of previous inputs!
        return self.fc2(torch.relu(self.fc1(x)))


# Problem: Processing words one at a time loses context
words = ["The", "cat", "sat", "on", "the", "mat"]
# When processing "mat", the network has no memory of "cat"!

​
The Recurrent Solution

An RNN maintains a hidden state that carries information across time steps: ht=f(ht−1,xt)h_t = f(h_{t-1}, x_t) Think of the hidden state as a person’s “running mental summary” while reading a book. After each sentence (xtx_t), the reader updates their understanding (hth_t) based on what they just read and what they already knew (ht−1h_{t-1}). They cannot go back and re-read (that would be an attention mechanism), so their current understanding must compress everything important from the entire story so far into a fixed-size mental state.
RNN Unrolled Through Time
class SimpleRNNCell(nn.Module):
    """
    A single RNN cell - the building block of RNNs.
    
    At each time step:
    1. Combine current input with previous hidden state
    2. Apply non-linearity
    3. Output new hidden state
    
    The same weights are reused at every time step -- this is
    "weight sharing through time," analogous to how CNNs share
    weights across spatial positions.
    """
    
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.hidden_size = hidden_size
        
        # Weights for input: transform the current observation
        self.W_xh = nn.Linear(input_size, hidden_size, bias=False)
        # Weights for hidden state: transform the memory of the past
        self.W_hh = nn.Linear(hidden_size, hidden_size, bias=True)
    
    def forward(self, x, h_prev):
        """
        Args:
            x: Current input (batch, input_size) -- what we see now
            h_prev: Previous hidden state (batch, hidden_size) -- what we remember
        
        Returns:
            h_new: Updated hidden state -- our new understanding
        """
        # New memory = tanh(transform(what we see) + transform(what we remember))
        # tanh squashes to [-1, 1], preventing the hidden state from growing unbounded
        h_new = torch.tanh(self.W_xh(x) + self.W_hh(h_prev))
        return h_new


# Demonstrate the recurrent connection
cell = SimpleRNNCell(input_size=10, hidden_size=20)

# Process a sequence of length 5
sequence = torch.randn(3, 5, 10)  # (batch=3, seq_len=5, input=10)
batch_size = sequence.size(0)
seq_len = sequence.size(1)

# Initialize hidden state
h = torch.zeros(batch_size, 20)

# Process each time step
hidden_states = []
for t in range(seq_len):
    x_t = sequence[:, t, :]  # Get input at time t
    h = cell(x_t, h)         # Update hidden state
    hidden_states.append(h)
    print(f"Time {t}: h shape = {h.shape}")

print(f"\nFinal hidden state captures the entire sequence!")

​
The RNN Equations

​
Mathematical Formulation

At each time step tt: ht=tanh⁡(Wxhxt+Whhht−1+bh)h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h) yt=Whyht+byy_t = W_{hy} h_t + b_y Where:
  • xt∈Rdx_t \in \mathbb{R}^{d} is the input at time tt
  • ht∈Rhh_t \in \mathbb{R}^{h} is the hidden state
  • yt∈Roy_t \in \mathbb{R}^{o} is the output
  • Wxh∈Rh×dW_{xh} \in \mathbb{R}^{h \times d} transforms input to hidden
  • Whh∈Rh×hW_{hh} \in \mathbb{R}^{h \times h} transforms previous hidden to current
  • Why∈Ro×hW_{hy} \in \mathbb{R}^{o \times h} transforms hidden to output
class VanillaRNN(nn.Module):
    """
    Vanilla (Elman) RNN implemented from scratch.
    
    This is the simplest form of RNN, directly implementing
    the recurrence relation with tanh non-linearity.
    """
    
    def __init__(self, input_size, hidden_size, output_size, num_layers=1):
        super().__init__()
        
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        
        # Input to hidden weights
        self.W_xh = nn.ModuleList([
            nn.Linear(input_size if i == 0 else hidden_size, hidden_size)
            for i in range(num_layers)
        ])
        
        # Hidden to hidden weights
        self.W_hh = nn.ModuleList([
            nn.Linear(hidden_size, hidden_size)
            for _ in range(num_layers)
        ])
        
        # Hidden to output
        self.W_hy = nn.Linear(hidden_size, output_size)
    
    def forward(self, x, h_0=None):
        """
        Process entire sequence.
        
        Args:
            x: Input sequence (batch, seq_len, input_size)
            h_0: Initial hidden state (num_layers, batch, hidden_size)
        
        Returns:
            outputs: Output at each time step (batch, seq_len, output_size)
            h_n: Final hidden state (num_layers, batch, hidden_size)
        """
        batch_size, seq_len, _ = x.size()
        
        # Initialize hidden states
        if h_0 is None:
            h = [torch.zeros(batch_size, self.hidden_size, device=x.device)
                 for _ in range(self.num_layers)]
        else:
            h = [h_0[i] for i in range(self.num_layers)]
        
        outputs = []
        
        for t in range(seq_len):
            x_t = x[:, t, :]
            
            # Process through each layer
            for layer in range(self.num_layers):
                h[layer] = torch.tanh(
                    self.W_xh[layer](x_t) + self.W_hh[layer](h[layer])
                )
                x_t = h[layer]  # Output of this layer is input to next
            
            # Compute output
            y_t = self.W_hy(h[-1])
            outputs.append(y_t)
        
        outputs = torch.stack(outputs, dim=1)
        h_n = torch.stack(h, dim=0)
        
        return outputs, h_n


# Test our implementation
rnn = VanillaRNN(input_size=10, hidden_size=32, output_size=5, num_layers=2)
x = torch.randn(4, 20, 10)  # batch=4, seq_len=20, input=10

outputs, h_n = rnn(x)
print(f"Input shape: {x.shape}")
print(f"Output shape: {outputs.shape}")  # (4, 20, 5)
print(f"Final hidden shape: {h_n.shape}")  # (2, 4, 32)

​
RNN Architectures

​
Many-to-One (Sequence Classification)

Use the final hidden state to classify the entire sequence: 
class SentimentClassifier(nn.Module):
    """
    Many-to-One: Classify entire sequence (e.g., sentiment analysis)
    
    Input: "This movie was absolutely terrible"
    Output: "Negative" (single label)
    """
    
    def __init__(self, vocab_size, embed_dim, hidden_size, num_classes):
        super().__init__()
        
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.RNN(embed_dim, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        # x: (batch, seq_len) - token indices
        embedded = self.embedding(x)  # (batch, seq_len, embed_dim)
        
        # Process through RNN
        output, h_n = self.rnn(embedded)  # h_n: (1, batch, hidden)
        
        # Use final hidden state for classification
        final_hidden = h_n.squeeze(0)  # (batch, hidden)
        logits = self.fc(final_hidden)  # (batch, num_classes)
        
        return logits


# Example
classifier = SentimentClassifier(vocab_size=10000, embed_dim=128, 
                                  hidden_size=256, num_classes=2)

# Simulated tokenized sentence (batch of 8 sentences, max length 50)
sentences = torch.randint(0, 10000, (8, 50))
predictions = classifier(sentences)
print(f"Predictions shape: {predictions.shape}")  # (8, 2)
Many-to-One RNN

​
One-to-Many (Sequence Generation)

Generate a sequence from a single input: 
class CaptionGenerator(nn.Module):
    """
    One-to-Many: Generate sequence from single input (e.g., image captioning)
    
    Input: Image features
    Output: "A cat sitting on a couch"
    """
    
    def __init__(self, image_dim, hidden_size, vocab_size, max_length=20):
        super().__init__()
        
        self.max_length = max_length
        
        # Project image features to hidden state
        self.image_proj = nn.Linear(image_dim, hidden_size)
        
        # Word embedding
        self.embedding = nn.Embedding(vocab_size, hidden_size)
        
        # RNN cell for generation
        self.rnn_cell = nn.RNNCell(hidden_size, hidden_size)
        
        # Output projection
        self.fc = nn.Linear(hidden_size, vocab_size)
    
    def forward(self, image_features, max_length=None):
        """
        Generate caption from image features.
        
        Args:
            image_features: (batch, image_dim)
        
        Returns:
            Generated token indices
        """
        if max_length is None:
            max_length = self.max_length
        
        batch_size = image_features.size(0)
        
        # Initialize hidden state from image
        h = torch.tanh(self.image_proj(image_features))
        
        # Start with <START> token (assume index 0)
        current_token = torch.zeros(batch_size, dtype=torch.long, 
                                    device=image_features.device)
        
        generated = []
        
        for t in range(max_length):
            # Embed current token
            x = self.embedding(current_token)
            
            # RNN step
            h = self.rnn_cell(x, h)
            
            # Predict next token
            logits = self.fc(h)
            current_token = logits.argmax(dim=-1)
            generated.append(current_token)
        
        return torch.stack(generated, dim=1)

​
Many-to-Many (Sequence-to-Sequence)

Transform one sequence into another: 
class Seq2SeqTranslator(nn.Module):
    """
    Many-to-Many: Transform sequence to sequence (e.g., translation)
    
    Input: "Hello, how are you?"
    Output: "Bonjour, comment allez-vous?"
    """
    
    def __init__(self, src_vocab_size, tgt_vocab_size, embed_dim, hidden_size):
        super().__init__()
        
        # Encoder
        self.src_embedding = nn.Embedding(src_vocab_size, embed_dim)
        self.encoder = nn.RNN(embed_dim, hidden_size, batch_first=True)
        
        # Decoder
        self.tgt_embedding = nn.Embedding(tgt_vocab_size, embed_dim)
        self.decoder = nn.RNN(embed_dim, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, tgt_vocab_size)
    
    def forward(self, src, tgt):
        """
        Args:
            src: Source sequence (batch, src_len)
            tgt: Target sequence for teacher forcing (batch, tgt_len)
        """
        # Encode source
        src_embedded = self.src_embedding(src)
        _, encoder_hidden = self.encoder(src_embedded)
        
        # Decode target (teacher forcing during training)
        tgt_embedded = self.tgt_embedding(tgt)
        decoder_output, _ = self.decoder(tgt_embedded, encoder_hidden)
        
        # Project to vocabulary
        logits = self.fc(decoder_output)
        
        return logits


translator = Seq2SeqTranslator(
    src_vocab_size=10000, tgt_vocab_size=8000,
    embed_dim=256, hidden_size=512
)

src = torch.randint(0, 10000, (16, 30))  # Batch of 16, max length 30
tgt = torch.randint(0, 8000, (16, 25))   # Target length 25

output = translator(src, tgt)
print(f"Output shape: {output.shape}")  # (16, 25, 8000)
RNN Architectures

​
Backpropagation Through Time (BPTT)

​
The Challenge of Temporal Gradients

RNNs are trained using BPTT - unrolling the network through time and applying backpropagation: 
def bptt_visualization():
    """
    Demonstrate how gradients flow backward through time.
    """
    # Consider a 4-step sequence
    # Loss at final step L_4
    # 
    # Forward:  x_1 → h_1 → h_2 → h_3 → h_4 → y_4 → L_4
    #              ↗      ↗      ↗      ↗
    #            x_2    x_3    x_4
    #
    # Backward gradient for W_hh:
    # ∂L/∂W_hh = ∂L/∂h_4 · ∂h_4/∂W_hh 
    #          + ∂L/∂h_4 · ∂h_4/∂h_3 · ∂h_3/∂W_hh
    #          + ∂L/∂h_4 · ∂h_4/∂h_3 · ∂h_3/∂h_2 · ∂h_2/∂W_hh
    #          + ...
    
    print("BPTT Gradient Flow:")
    print("=" * 50)
    print("∂L/∂W_hh = Σ_t ∂L/∂h_T · (∏_{k=t+1}^{T} ∂h_k/∂h_{k-1}) · ∂h_t/∂W_hh")
    print()
    print("The product of Jacobians can explode or vanish!")

bptt_visualization()


class RNNWithGradientTracking(nn.Module):
    """RNN that tracks gradient magnitudes through time."""
    
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.gradient_norms = []
    
    def forward(self, x):
        output, h_n = self.rnn(x)
        return output, h_n
    
    def track_gradients(self, x, target):
        """Track how gradients change through time."""
        x.requires_grad_(True)
        output, _ = self.forward(x)
        
        # Compute gradients for output at each time step
        seq_len = output.size(1)
        gradient_norms = []
        
        for t in range(seq_len):
            if x.grad is not None:
                x.grad.zero_()
            
            # Gradient of output at time t w.r.t. input at time 0
            output[:, t, 0].sum().backward(retain_graph=True)
            
            # Gradient norm from first input
            grad_norm = x.grad[:, 0, :].norm().item()
            gradient_norms.append(grad_norm)
        
        return gradient_norms


# Demonstrate gradient flow
torch.manual_seed(42)
model = RNNWithGradientTracking(10, 32)
x = torch.randn(1, 50, 10)

gradient_norms = model.track_gradients(x, None)

plt.figure(figsize=(10, 4))
plt.plot(gradient_norms)
plt.xlabel('Time step (distance from input)')
plt.ylabel('Gradient norm')
plt.title('Gradient Flow in Vanilla RNN')
plt.yscale('log')
plt.grid(True)
plt.show()

print(f"Gradient ratio (step 49 / step 1): {gradient_norms[-1]/gradient_norms[0]:.6f}")

​
The Vanishing/Exploding Gradient Problem

​
Why Gradients Vanish

The gradient through time involves products of the recurrent weight matrix: ∂hT∂h1=∏t=2T∂ht∂ht−1=∏t=2TWhhT⋅diag(tanh⁡′(ht−1))\frac{\partial h_T}{\partial h_1} = \prod_{t=2}^{T} \frac{\partial h_t}{\partial h_{t-1}} = \prod_{t=2}^{T} W_{hh}^T \cdot \text{diag}(\tanh'(h_{t-1})) Mathematical intuition: This product is the crux of the problem. The tanh derivative peaks at 0.25 (since tanh⁡′(z)=1−tanh⁡2(z)\tanh'(z) = 1 - \tanh^2(z) and its maximum is 1 at z=0z=0, but typical values during training are 0.1-0.25). Multiply that by WhhW_{hh} at each step. If the largest singular value of Whh⋅diag(tanh⁡′)<1W_{hh} \cdot \text{diag}(\tanh') < 1 (which is almost always the case), the product shrinks exponentially: 0.2550≈10−300.25^{50} \approx 10^{-30}. That is effectively zero — the network cannot learn from information 50 steps ago because the gradient signal has been annihilated. Conversely, if the singular values exceed 1, the product explodes exponentially. This is why vanilla RNNs are caught in a double bind: they need large weights to maintain gradients, but large weights cause explosions. The only stable regime is the knife-edge where singular values equal exactly 1 — which is unrealistic to maintain during training. 
def analyze_gradient_flow():
    """
    Analyze why gradients vanish or explode in RNNs.
    """
    hidden_size = 100
    seq_length = 100
    
    # Different initialization scales
    scales = [0.5, 1.0, 1.5]
    
    fig, axes = plt.subplots(1, 3, figsize=(15, 4))
    
    for ax, scale in zip(axes, scales):
        # Initialize W_hh with different scales
        W = torch.randn(hidden_size, hidden_size) * scale / np.sqrt(hidden_size)
        
        # Simulate gradient flow (simplified - ignoring tanh derivative)
        gradient = torch.eye(hidden_size)
        gradient_norms = [gradient.norm().item()]
        
        for t in range(seq_length):
            gradient = gradient @ W
            gradient_norms.append(gradient.norm().item())
        
        ax.semilogy(gradient_norms)
        ax.set_xlabel('Time steps back')
        ax.set_ylabel('Gradient norm (log scale)')
        ax.set_title(f'Scale = {scale}')
        ax.axhline(y=1, color='r', linestyle='--', alpha=0.5)
        ax.set_ylim([1e-20, 1e20])
    
    plt.tight_layout()
    plt.show()

analyze_gradient_flow()
Vanishing Gradients

​
The Impact on Learning

def demonstrate_vanishing_gradient_impact():
    """
    Show how vanishing gradients affect learning long-range dependencies.
    """
    
    # Task: Remember the first element of a sequence
    # Output at the end should equal input at the beginning
    
    class CopyFirstTask(nn.Module):
        def __init__(self, hidden_size):
            super().__init__()
            self.rnn = nn.RNN(1, hidden_size, batch_first=True)
            self.fc = nn.Linear(hidden_size, 1)
        
        def forward(self, x):
            output, h_n = self.rnn(x)
            # Use final hidden state
            return self.fc(h_n.squeeze(0))
    
    def train_copy_task(seq_length, hidden_size=32, num_epochs=1000):
        model = CopyFirstTask(hidden_size)
        optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
        criterion = nn.MSELoss()
        
        losses = []
        
        for epoch in range(num_epochs):
            # Generate data: first value is target
            batch_size = 32
            first_value = torch.randn(batch_size, 1)
            
            # Sequence: [first_value, noise, noise, ..., noise]
            sequence = torch.randn(batch_size, seq_length, 1) * 0.1
            sequence[:, 0, :] = first_value
            
            # Forward pass
            prediction = model(sequence)
            loss = criterion(prediction, first_value)
            
            # Backward pass
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            
            losses.append(loss.item())
        
        return losses
    
    # Test with different sequence lengths
    seq_lengths = [10, 30, 50, 100]
    
    plt.figure(figsize=(12, 4))
    
    for seq_len in seq_lengths:
        losses = train_copy_task(seq_len)
        plt.plot(losses, label=f'Length = {seq_len}')
    
    plt.xlabel('Epoch')
    plt.ylabel('MSE Loss')
    plt.title('Copy First Task: Longer Sequences are Harder')
    plt.legend()
    plt.yscale('log')
    plt.grid(True)
    plt.show()

demonstrate_vanishing_gradient_impact()

​
Solutions to Vanishing Gradients

SolutionDescriptionEffectiveness
Gradient ClippingLimit gradient magnitudeHelps exploding, not vanishing
Better InitializationOrthogonal initializationModest improvement
LSTM/GRUGated architecturesVery effective — the standard solution
Skip ConnectionsConnect distant time stepsEffective
AttentionDirect access to all statesVery effective — eventually replaced RNNs entirely
Practical rule of thumb: If your sequence length is under 20 tokens, a vanilla RNN might work. For 20-200 tokens, use LSTM or GRU. For 200+ tokens, you almost certainly need attention or a transformer. The cutoffs are approximate, but the pattern holds: longer sequences need more sophisticated memory mechanisms.
def gradient_clipping_demo():
    """Demonstrate gradient clipping to prevent explosion."""
    
    model = nn.RNN(10, 32, batch_first=True)
    
    x = torch.randn(1, 100, 10, requires_grad=True)
    output, _ = model(x)
    loss = output.sum()
    loss.backward()
    
    # Check gradient norms before clipping
    total_norm = 0
    for p in model.parameters():
        if p.grad is not None:
            total_norm += p.grad.norm().item() ** 2
    total_norm = total_norm ** 0.5
    print(f"Gradient norm before clipping: {total_norm:.4f}")
    
    # Apply gradient clipping
    max_norm = 1.0
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
    
    # Check after clipping
    total_norm_after = 0
    for p in model.parameters():
        if p.grad is not None:
            total_norm_after += p.grad.norm().item() ** 2
    total_norm_after = total_norm_after ** 0.5
    print(f"Gradient norm after clipping:  {total_norm_after:.4f}")

gradient_clipping_demo()

​
Bidirectional RNNs

​
Why Look Both Ways?

In many tasks, future context is as important as past context:
  • “The bank of the river” vs “I went to the bank”
  • In translation, the end of a sentence can clarify the beginning
class BidirectionalRNN(nn.Module):
    """
    Bidirectional RNN - process sequence in both directions.
    
    Forward:  x_1 → x_2 → x_3 → x_4 →
    Backward: ← x_1 ← x_2 ← x_3 ← x_4
    
    At each position, we have context from both directions.
    """
    
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        
        self.hidden_size = hidden_size
        
        # Two separate RNNs
        self.rnn_forward = nn.RNN(input_size, hidden_size, batch_first=True)
        self.rnn_backward = nn.RNN(input_size, hidden_size, batch_first=True)
        
        # Output layer (hidden_size * 2 because we concatenate)
        self.fc = nn.Linear(hidden_size * 2, output_size)
    
    def forward(self, x):
        """
        Args:
            x: (batch, seq_len, input_size)
        
        Returns:
            output: (batch, seq_len, output_size)
        """
        # Forward pass
        forward_out, _ = self.rnn_forward(x)
        
        # Backward pass (flip, process, flip back)
        x_reversed = torch.flip(x, dims=[1])
        backward_out, _ = self.rnn_backward(x_reversed)
        backward_out = torch.flip(backward_out, dims=[1])
        
        # Concatenate
        combined = torch.cat([forward_out, backward_out], dim=-1)
        
        # Output
        output = self.fc(combined)
        
        return output


# PyTorch's built-in bidirectional RNN
birnn = nn.RNN(input_size=10, hidden_size=32, 
               batch_first=True, bidirectional=True)

x = torch.randn(4, 20, 10)
output, h_n = birnn(x)

print(f"Input: {x.shape}")
print(f"Output: {output.shape}")  # (4, 20, 64) - doubled hidden size
print(f"Hidden: {h_n.shape}")     # (2, 4, 32) - 2 directions
Bidirectional RNN

​
Deep RNNs

​
Stacking RNN Layers

class DeepRNN(nn.Module):
    """
    Deep RNN - stack multiple RNN layers.
    
    Layer 1: Processes input sequence
    Layer 2: Processes Layer 1's hidden states
    Layer 3: Processes Layer 2's hidden states
    ...
    
    Deeper = more abstract representations
    """
    
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super().__init__()
        
        # Stack of RNN layers
        self.rnn = nn.RNN(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True,
            dropout=0.2 if num_layers > 1 else 0  # Dropout between layers
        )
        
        self.fc = nn.Linear(hidden_size, output_size)
    
    def forward(self, x, h_0=None):
        output, h_n = self.rnn(x, h_0)
        return self.fc(output), h_n


# Compare shallow vs deep
shallow = nn.RNN(10, 128, num_layers=1, batch_first=True)
deep = nn.RNN(10, 64, num_layers=4, batch_first=True)

# Similar parameter counts but different architectures
shallow_params = sum(p.numel() for p in shallow.parameters())
deep_params = sum(p.numel() for p in deep.parameters())

print(f"Shallow RNN (1 layer, 128 hidden): {shallow_params:,} params")
print(f"Deep RNN (4 layers, 64 hidden):    {deep_params:,} params")

x = torch.randn(8, 50, 10)
print(f"\nShallow output: {shallow(x)[0].shape}")
print(f"Deep output:    {deep(x)[0].shape}")

​
Practical Example: Character-Level Language Model

class CharRNN(nn.Module):
    """
    Character-level language model.
    
    Given: "Hello Worl"
    Predict: "ello World" (next character at each position)
    """
    
    def __init__(self, vocab_size, embed_size, hidden_size, num_layers):
        super().__init__()
        
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        
        self.embedding = nn.Embedding(vocab_size, embed_size)
        self.rnn = nn.RNN(embed_size, hidden_size, num_layers, 
                          batch_first=True, dropout=0.2)
        self.fc = nn.Linear(hidden_size, vocab_size)
    
    def forward(self, x, hidden=None):
        # x: (batch, seq_len) character indices
        embedded = self.embedding(x)  # (batch, seq_len, embed_size)
        output, hidden = self.rnn(embedded, hidden)  # (batch, seq_len, hidden)
        logits = self.fc(output)  # (batch, seq_len, vocab_size)
        return logits, hidden
    
    def generate(self, start_char, char_to_idx, idx_to_char, 
                 length=100, temperature=1.0):
        """Generate text character by character."""
        self.eval()
        
        # Start with initial character
        current = torch.tensor([[char_to_idx[start_char]]])
        hidden = None
        
        generated = [start_char]
        
        with torch.no_grad():
            for _ in range(length):
                logits, hidden = self.forward(current, hidden)
                
                # Apply temperature
                probs = torch.softmax(logits[0, -1] / temperature, dim=0)
                
                # Sample from distribution
                next_idx = torch.multinomial(probs, 1).item()
                next_char = idx_to_char[next_idx]
                
                generated.append(next_char)
                current = torch.tensor([[next_idx]])
        
        return ''.join(generated)


def train_char_rnn():
    """Train a character-level RNN on sample text."""
    
    # Sample text (in practice, use a large corpus)
    text = """
    To be, or not to be, that is the question:
    Whether 'tis nobler in the mind to suffer
    The slings and arrows of outrageous fortune,
    Or to take arms against a sea of troubles
    And by opposing end them.
    """ * 100  # Repeat for more training data
    
    # Build vocabulary
    chars = sorted(set(text))
    char_to_idx = {c: i for i, c in enumerate(chars)}
    idx_to_char = {i: c for c, i in char_to_idx.items()}
    vocab_size = len(chars)
    
    print(f"Vocabulary size: {vocab_size}")
    print(f"Characters: {''.join(chars)}")
    
    # Prepare data
    seq_length = 50
    
    def create_sequences(text, seq_length):
        inputs = []
        targets = []
        
        for i in range(0, len(text) - seq_length):
            inputs.append([char_to_idx[c] for c in text[i:i+seq_length]])
            targets.append([char_to_idx[c] for c in text[i+1:i+seq_length+1]])
        
        return torch.tensor(inputs), torch.tensor(targets)
    
    inputs, targets = create_sequences(text, seq_length)
    
    # Create model
    model = CharRNN(vocab_size, embed_size=64, hidden_size=128, num_layers=2)
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.003)
    
    # Training loop
    batch_size = 64
    num_epochs = 50
    
    for epoch in range(num_epochs):
        # Random batch
        idx = torch.randint(0, len(inputs), (batch_size,))
        x_batch = inputs[idx]
        y_batch = targets[idx]
        
        # Forward
        logits, _ = model(x_batch)
        loss = criterion(logits.view(-1, vocab_size), y_batch.view(-1))
        
        # Backward
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
        optimizer.step()
        
        if (epoch + 1) % 10 == 0:
            print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")
            
            # Generate sample
            sample = model.generate('T', char_to_idx, idx_to_char, 
                                    length=100, temperature=0.8)
            print(f"Sample: {sample[:80]}...")
            print()
    
    return model, char_to_idx, idx_to_char

# Uncomment to train
# model, c2i, i2c = train_char_rnn()

​
RNN Applications

ApplicationArchitectureInputOutput
Sentiment AnalysisMany-to-OneTextPositive/Negative
Language ModelingMany-to-ManyCharacters/WordsNext char/word
Machine TranslationEncoder-DecoderSource sentenceTarget sentence
Speech RecognitionMany-to-ManyAudio framesText
Music GenerationMany-to-ManyNotesNext notes
Time Series ForecastingMany-to-OnePast valuesFuture value
Named Entity RecognitionMany-to-ManyWordsEntity labels

​
Exercises

Implement a complete RNN without using nn.RNN:
class MyRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        # Initialize W_xh, W_hh, bias
        # Implement forward pass with proper state management
        pass
    
    def forward(self, x, h_0=None):
        # Process sequence step by step
        # Return outputs and final hidden state
        pass
Verify it gives similar results to nn.RNN.
Implement the adding problem to test long-range dependencies:
  • Input: Two sequences - numbers and a mask
  • Output: Sum of numbers where mask is 1
  • Example: Numbers [0.3, 0.1, 0.8, 0.2], Mask [1, 0, 0, 1] → Output: 0.5
Test with sequence lengths 10, 50, 100, 200.
Build a model to classify names by nationality:
  • Input: Character sequence (name)
  • Output: Nationality (English, French, German, etc.)
Use the names dataset from PyTorch tutorials.
Build a simple date format converter:
  • Input: “January 15, 2023”
  • Output: “2023-01-15”
Generate synthetic training data and train encoder-decoder.
For a trained sentiment analysis model:
  1. Process sentences with known sentiment
  2. Extract hidden states at each position
  3. Visualize with t-SNE or PCA
  4. Color by sentiment and position
What patterns do you observe?

​
Training Pitfalls and Debugging Hints

Gradient clipping is not optional for RNNs. Unlike feedforward networks where gradient explosion is rare, RNNs will explode without clipping on any non-trivial sequence length. Always use torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5.0) as a default. If training is unstable, lower to 1.0. If loss plateaus, you might be clipping too aggressively — try raising to 10.0.Hidden state detachment between batches: When training on long documents split into chunks, you must detach() the hidden state between chunks. Without this, PyTorch tries to backpropagate through the entire document, consuming unbounded memory. Use hidden = hidden.detach() or hidden = tuple(h.detach() for h in hidden) for LSTMs.Pack padded sequences for variable lengths: If your batch contains sequences of different lengths, padding them and ignoring padding in the loss is not enough — the RNN still processes the padding tokens, which corrupts the hidden state. Use nn.utils.rnn.pack_padded_sequence and pad_packed_sequence to make the RNN skip padding entirely. This is one of the most common performance bugs in RNN training.Loss is NaN after a few epochs: Almost always caused by exploding gradients. Check: (1) Are you clipping gradients? (2) Is your learning rate too high? (3) Are there any sequences with extreme values in the input? Start with lr=0.001 for Adam and lr=0.1 for SGD with RNNs.The hidden state initialization trap: Initializing hidden states to zero is standard, but for bidirectional RNNs the backward direction’s “initial” state is the state at the end of the sequence. If your sequences are padded, the backward RNN starts from a padding position. Use packed sequences to avoid this.

​
Key Takeaways

ConceptKey Insight
Hidden StateMemory that carries information across time steps
BPTTBackprop through unrolled network - gradients flow through time
Vanishing GradientLong sequences lead to gradients shrinking and early inputs being forgotten
Exploding GradientGradients grow and training becomes unstable so use clipping
BidirectionalContext from both past and future
Deep RNNsStack layers for hierarchical representations
Vanilla RNNs are rarely used in practice! The vanishing gradient problem makes them unable to learn long-range dependencies. In the next chapter, we’ll learn about LSTMs and GRUs - architectures specifically designed to solve this problem.

​
What’s Next

Module 9: LSTMs & GRUs

Solve the vanishing gradient problem with gated architectures — learn how LSTM and GRU cells maintain long-term memory through gates.

​
Interview Deep-Dive

Strong Answer:
  • In an RNN, the hidden state update is ht=tanh⁡(Whhht−1+Wxhxt+b)h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b). During BPTT, the gradient of loss with respect to early hidden states requires multiplying the Jacobian ∂ht/∂ht−1\partial h_t / \partial h_{t-1} across all time steps: ∏t=1T∂ht/∂ht−1\prod_{t=1}^{T} \partial h_t / \partial h_{t-1}.
  • Each Jacobian includes WhhT⋅diag(tanh⁡′(zt))W_{hh}^T \cdot \text{diag}(\tanh'(z_t)). The tanh derivative is at most 1 and typically 0.1-0.5. The product of TT such terms approaches zero exponentially. After 50 time steps, gradients can be 10−1010^{-10} or smaller.
  • The consequence is selective amnesia: the RNN can learn short-range patterns (2-5 steps) but cannot learn dependencies from 20+ steps ago. This is fundamentally different from the vanishing gradient in feedforward networks because the SAME weight matrix WhhW_{hh} is applied at every step, making the problem a function of the spectral radius of WhhW_{hh}.
  • If the largest singular value of WhhW_{hh} is less than 1, gradients vanish. If greater than 1, they explode. There is no stable middle ground for vanilla RNNs, which is why gated architectures (LSTM, GRU) were necessary.
Follow-up: Gradient clipping handles exploding gradients but not vanishing gradients. Why the asymmetry?Gradient clipping rescales the gradient norm when it exceeds a threshold, preventing catastrophically large updates. But for vanishing gradients, the gradients are not wrong in direction — they are just too small. You cannot amplify them because the gradient direction itself becomes unreliable when the signal-to-noise ratio is near zero. The solution must be architectural (LSTM gates, skip connections) rather than optimization-level (clipping). Clipping handles the exploding case; gating handles the vanishing case. They are complementary.
Strong Answer:
  • BPTT unrolls the RNN through time and applies standard backpropagation to the resulting feedforward graph. An RNN processing length TT is equivalent to a TT-layer feedforward network with shared weights. Gradients are computed backward through all TT steps.
  • Full BPTT has two problems: memory grows linearly with TT (all activations cached), and gradients vanish or explode through hundreds of multiplications.
  • Truncated BPTT limits backpropagation to a window of kk steps. Every kk steps, the hidden state is detached from the computation graph. Gradients only flow backward through the most recent kk steps.
  • Trade-off: the model can only LEARN dependencies up to kk steps long. A truncation of 35 means a 50-step dependency is invisible to the optimizer. However, the model can still USE long-range information already encoded in the hidden state — it just cannot learn to encode it better. The practical sweet spot is k=35−256k = 35-256 for language modeling, balancing dependency learning against memory and stability.
Follow-up: How do transformers compare to truncated BPTT in terms of effective context?Transformers avoid gradient flow problems entirely — self-attention creates direct connections between any two positions, so the gradient path length is always 1 (through the attention weights). The limitation is computational: O(n2)O(n^2) attention scales poorly for very long contexts. But within the context window, every position has equal gradient access to every other position, unlike RNNs where gradient strength decays with distance. This is the fundamental reason transformers capture long-range dependencies more effectively.
Strong Answer:
  • Real-time streaming inference: RNNs process one token at a time with constant memory (O(1)O(1) per step), making them ideal for streaming audio transcription, real-time sensor processing, or any setting where you receive data one sample at a time and need immediate outputs. Transformers require buffering the entire context window before processing.
  • Extremely long sequences with limited compute: for sequences of 100,000+ steps (e.g., long-duration biosignals, multi-day time series), transformers’ O(n2)O(n^2) attention becomes prohibitive. RNNs process these in O(n)O(n) time with fixed memory, though they sacrifice long-range dependency quality.
  • Edge deployment with tight memory constraints: an LSTM cell has fixed-size state regardless of sequence length, making memory consumption predictable and small. A transformer’s KV-cache grows linearly with sequence length.
  • State Space Models (SSMs) like Mamba represent a modern compromise: they have RNN-like O(n)O(n) inference with transformer-like training parallelism, and they match transformer quality on many benchmarks. SSMs are increasingly the right answer for the “when would you use recurrence” question — they inherit the efficiency of RNNs without the gradient flow problems.
Follow-up: Why is Mamba considered a breakthrough for sequential modeling?Mamba (Gu and Dao, 2023) achieves linear-time inference like an RNN, parallel training like a transformer, and competitive quality on language modeling benchmarks. The key innovation is selective state spaces: the state transition matrices are input-dependent (selective), allowing the model to dynamically decide what to remember and forget — analogous to LSTM gating but formulated as a continuous-time system. This bridges the recurrence vs. attention divide by providing the efficiency of recurrence with the expressiveness of attention-like selection.