Chain Rule & Backpropagation

Your Challenge: The Butterfly Effect

In complex systems, a small change here can cause a massive result there. Imagine you run a global manufacturing company.

Raw Material Price goes up by $0.10.
Production Cost increases by $1.00.
Product Price increases by $5.00.
Sales Volume drops by 1,000 units.
Total Revenue crashes by $50,000.

Your Question: “How did a 10-cent change cause a $50,000 crash?” To understand this, you need to trace the impact through every link in the chain. This is exactly what the Chain Rule does. And it’s how neural networks “blame” a specific weight in Layer 1 for an error in Layer 50.

The Domino Effect

Think of it as a row of dominoes.

You push the first domino (Input).
It hits the second (Hidden Layer).
Which hits the third (Output).

The Chain Rule says:

“The total impact is the product of all the individual impacts along the chain.”

If Domino A hits Domino B with force 2, and Domino B hits Domino C with force 3… Then Domino A hits Domino C with force 2 × 3 = 6.

The Intuition: Composition of Functions

The Problem

You have a function inside another function:

y = f(g(x))

$x$ = Input (Raw Material Price)
$u = g(x)$ = Intermediate (Production Cost)
$y = f(u)$ = Output (Revenue)

Question: If I change

x

, how much does

y

change?

The Solution

You multiply the rates of change!

\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}

$\frac{du}{dx}$ : How much Cost changes when Material Price changes.
$\frac{dy}{du}$ : How much Revenue changes when Cost changes.

Let’s Code It

# 1. Material Price → Production Cost
def cost(material_price):
    return material_price * 10 + 50  # 10 units per product

# 2. Production Cost → Revenue
def revenue(production_cost):
    return -5 * production_cost + 2000 # Higher cost = Lower revenue

# Composed: Material Price → Revenue
def total_revenue(material_price):
    c = cost(material_price)
    return revenue(c)

# The Derivatives (Impacts)
d_cost_d_material = 10    # Every $1 material increase adds $10 to cost
d_revenue_d_cost = -5     # Every $1 cost increase reduces revenue by $5

# The Chain Rule
d_revenue_d_material = d_revenue_d_cost * d_cost_d_material
# -5 * 10 = -50

print(f"Impact of material price on revenue: {d_revenue_d_material}")
print("Interpretation: A $1 increase in material price kills revenue by $50.")

Key Insight: You can break a complex system into small, simple links. Multiply them together to get the total effect.

The Telephone Game Analogy

The chain rule is like the telephone game (or “Chinese whispers”). A message passes through a chain of people, and each person amplifies or dampens the signal. If person A amplifies by 3x, person B dampens by 0.5x, and person C amplifies by 4x, the total effect is

3 \times 0.5 \times 4 = 6\times

. In a neural network, the “message” is the error signal, the “people” are the layers, and each layer’s derivative is its amplification factor. This is precisely why vanishing gradients happen: if every layer has a derivative less than 1 (as sigmoid layers tend to), the signal decays exponentially. Fifty layers of 0.25 multiplication give you

0.25^{50} \approx 10^{-30}

— the error signal from the output never reaches the early layers. They cannot learn. This single insight — that chain rule multiplication can cause exponential decay — drove the entire shift from sigmoid to ReLU activations and motivated the invention of residual connections (skip connections) in ResNets.

Mathematical Definition

For composed functions

f(g(x))

\frac{d}{dx}f(g(x)) = f'(g(x)) \cdot g'(x)

In words:

Take derivative of outer function (evaluated at inner function)
Multiply by derivative of inner function

Simple Example

h(x) = (x^2 + 1)^3

Think of it as:

h(x) = f(g(x))

where:

Inner function: $g(x) = x^2 + 1$
Outer function: $f(u) = u^3$

Derivatives:

$g'(x) = 2x$
$f'(u) = 3u^2$

Chain rule:

h'(x) = f'(g(x)) \cdot g'(x) = 3(x^2 + 1)^2 \cdot 2x = 6x(x^2 + 1)^2

def h(x):
    return (x**2 + 1)**3

def h_derivative(x):
    # Chain rule
    inner = x**2 + 1
    outer_derivative = 3 * inner**2
    inner_derivative = 2 * x
    return outer_derivative * inner_derivative

x = 2
print(f"h'({x}) = {h_derivative(x)}")  # 150

# Verify numerically
h_val = 0.0001
numerical = (h(x + h_val) - h(x)) / h_val
print(f"Numerical: {numerical:.2f}")  # ≈ 150

Example 1: Multi-Stage Business Process

The Scenario

Supply Chain: Raw materials → Manufacturing → Sales → Revenue

Material Cost → Production Quantity → Sales Volume → Total Revenue

Functions:

\begin{align} \text{Production}(c) &= 1000 - 10c \quad \text{(higher cost → less production)} \\ \text{Sales}(p) &= 0.8p \quad \text{(80% of production sells)} \\ \text{Revenue}(s) &= 50s \quad \text{($50 per unit sold)} \end{align}

Question: If material cost increases by $1, how much does revenue change?

Solution Using Chain Rule

def production(cost):
    return 1000 - 10 * cost

def sales(production_qty):
    return 0.8 * production_qty

def revenue(sales_qty):
    return 50 * sales_qty

# Composed function
def total_revenue(cost):
    prod = production(cost)
    sale = sales(prod)
    return revenue(sale)

# Derivatives
def d_production_d_cost(cost):
    return -10

def d_sales_d_production(prod):
    return 0.8

def d_revenue_d_sales(sales):
    return 50

# Chain rule: multiply all derivatives
cost = 20
prod = production(cost)
sale = sales(prod)

chain_derivative = (d_production_d_cost(cost) * 
                   d_sales_d_production(prod) * 
                   d_revenue_d_sales(sale))

print(f"At cost=${cost}:")
print(f"  Production: {prod} units")
print(f"  Sales: {sale} units")
print(f"  Revenue: ${total_revenue(cost)}")
print(f"\nIf cost increases by $1:")
print(f"  Revenue changes by: ${chain_derivative}")
# -10 × 0.8 × 50 = -$400

Interpretation:

$1 cost increase → 10 fewer units produced
10 fewer units → 8 fewer sales (80% sell rate)
8 fewer sales → $400 less revenue

Real Application: Amazon uses chain rule to optimize their entire supply chain, from warehouses to delivery!

Example 2: Your Learning Chain

The Scenario

Let’s model your own learning process as a chain of functions:

Study Time ( $x$ ) → Understanding ( $u$ )
Understanding ( $u$ ) → Test Score ( $t$ )
Test Score ( $t$ ) → Final Grade ( $G$ )

Your Goal: Find out how much 1 extra hour of study improves your Final Grade.

The Functions

def understanding(study_hours):
    """More study → better understanding (diminishing returns)"""
    return 10 * np.sqrt(study_hours)

def test_score(understanding_level):
    """Understanding → test score"""
    return 5 * understanding_level + 20

def final_grade(test_avg):
    """Test average → final grade (curved)"""
    return 0.9 * test_avg + 5

Applying the Chain Rule

You want to find

\frac{dG}{dx}

(Change in Grade per Hour).

\frac{dG}{dx} = \frac{dG}{dt} \cdot \frac{dt}{du} \cdot \frac{du}{dx}

# 1. Derivative of Understanding w.r.t Hours
def d_understanding_d_hours(h):
    return 5 / np.sqrt(h)  # Derivative of 10√h

# 2. Derivative of Test Score w.r.t Understanding
def d_test_d_understanding(u):
    return 5

# 3. Derivative of Grade w.r.t Test Score
def d_grade_d_test(t):
    return 0.9

# Calculate for 9 hours of study
hours = 9
u = understanding(hours)
t = test_score(u)

# Multiply the links!
grade_improvement = (d_grade_d_test(t) * 
                    d_test_d_understanding(u) * 
                    d_understanding_d_hours(hours))

print(f"At 9 hours/week:")
print(f"1 extra hour adds +{grade_improvement:.2f} points to your Final Grade")

Output:

At 9 hours/week:
1 extra hour adds +7.50 points to your Final Grade

Insight: The chain rule lets you connect your input (effort) directly to your output (grade), even through multiple steps!

Example 3: Your First Neural Network

The Computational Graph

This is how deep learning actually works. We represent the network as a graph of nodes.

The Flow:

Forward Pass (Blue): You calculate the prediction and the error.
Backward Pass (Red): You calculate who is to blame for the error.

Neural Network Backpropagation: Layer by Layer

This is the heart of deep learning. Let’s work through a complete 2-layer network step by step.

Network Architecture

\text{Input } x \xrightarrow{W_1, b_1} z_1 \xrightarrow{\text{ReLU}} a_1 \xrightarrow{W_2, b_2} z_2 \xrightarrow{\text{sigmoid}} \hat{y} \xrightarrow{\text{MSE}} L

Forward Pass Equations

Step	Formula	Description
1	$z_1 = W_1 x + b_1$	Linear combination
2	$a_1 = \text{ReLU}(z_1)$	Activation
3	$z_2 = W_2 a_1 + b_2$	Second linear
4	$\hat{y} = \sigma(z_2)$	Output (sigmoid)
5	$L = (\hat{y} - y)^2$	Loss

Backward Pass: Compute ALL Gradients

We want:

\frac{\partial L}{\partial W_1}

\frac{\partial L}{\partial b_1}

\frac{\partial L}{\partial W_2}

\frac{\partial L}{\partial b_2}

Starting from the end (Layer 2):

\frac{\partial L}{\partial \hat{y}} = 2(\hat{y} - y)

\frac{\partial L}{\partial z_2} = \frac{\partial L}{\partial \hat{y}} \cdot \sigma'(z_2) = 2(\hat{y} - y) \cdot \hat{y}(1-\hat{y})

\frac{\partial L}{\partial W_2} = \frac{\partial L}{\partial z_2} \cdot a_1^T

\frac{\partial L}{\partial b_2} = \frac{\partial L}{\partial z_2}

Propagating to Layer 1:

\frac{\partial L}{\partial a_1} = W_2^T \cdot \frac{\partial L}{\partial z_2}

\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial a_1} \cdot \text{ReLU}'(z_1)

\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial z_1} \cdot x^T

\frac{\partial L}{\partial b_1} = \frac{\partial L}{\partial z_1}

Complete Python Implementation

import numpy as np

class TwoLayerNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        # Initialize weights (small random values)
        self.W1 = np.random.randn(input_size, hidden_size) * 0.1
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * 0.1
        self.b2 = np.zeros((1, output_size))
    
    def sigmoid(self, z):
        return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
    
    def relu(self, z):
        return np.maximum(0, z)
    
    def relu_derivative(self, z):
        return (z > 0).astype(float)
    
    def forward(self, X):
        """Forward pass - compute predictions"""
        # Layer 1
        self.z1 = X @ self.W1 + self.b1
        self.a1 = self.relu(self.z1)
        
        # Layer 2
        self.z2 = self.a1 @ self.W2 + self.b2
        self.y_hat = self.sigmoid(self.z2)
        
        return self.y_hat
    
    def backward(self, X, y):
        """Backward pass - compute gradients using chain rule"""
        m = X.shape[0]
        
        # Output layer gradients
        dL_dy_hat = 2 * (self.y_hat - y)                  # dL/dŷ
        dL_dz2 = dL_dy_hat * self.y_hat * (1 - self.y_hat) # Chain: dL/dz2
        
        self.dW2 = (self.a1.T @ dL_dz2) / m               # dL/dW2
        self.db2 = np.sum(dL_dz2, axis=0, keepdims=True) / m
        
        # Hidden layer gradients (propagate backward!)
        dL_da1 = dL_dz2 @ self.W2.T                       # dL/da1
        dL_dz1 = dL_da1 * self.relu_derivative(self.z1)   # Chain: dL/dz1
        
        self.dW1 = (X.T @ dL_dz1) / m                     # dL/dW1
        self.db1 = np.sum(dL_dz1, axis=0, keepdims=True) / m
    
    def update(self, learning_rate):
        """Gradient descent update"""
        self.W1 -= learning_rate * self.dW1
        self.b1 -= learning_rate * self.db1
        self.W2 -= learning_rate * self.dW2
        self.b2 -= learning_rate * self.db2

# Train on XOR problem (the classic neural network test!)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

net = TwoLayerNetwork(input_size=2, hidden_size=4, output_size=1)

print("Training Neural Network on XOR...")
print("="*50)

for epoch in range(5000):
    # Forward
    predictions = net.forward(X)
    
    # Backward (chain rule!)
    net.backward(X, y)
    
    # Update
    net.update(learning_rate=1.0)
    
    if epoch % 1000 == 0:
        loss = np.mean((predictions - y)**2)
        print(f"Epoch {epoch}: Loss = {loss:.6f}")

print("\nFinal Predictions:")
print(net.forward(X).round(2))
print("\nTrue Values:", y.flatten())

Key Insight: Each weight’s gradient is computed by multiplying all the derivatives along the path from that weight to the loss. This is the chain rule in action!

Numerical Pitfall: Vanishing and Exploding GradientsThe chain rule multiplies derivatives together. In a deep network this multiplication happens across dozens or hundreds of layers. Two failure modes emerge:Vanishing gradients: When derivatives are consistently less than 1, their product shrinks exponentially. A 50-layer network where each layer’s derivative averages 0.25 yields a total multiplier of

0.25^{50} \approx 10^{-30}

. The early layers receive essentially zero gradient and never learn.Exploding gradients: When derivatives are consistently greater than 1, the product grows exponentially. Even modest values like 1.5 across 50 layers give

1.5^{50} \approx 6.4 \times 10^{8}

. Your weight update is billions of times too large, and training diverges instantly — you see loss become NaN.Practical defenses used in production:

Problem	Solution	Why It Works
Vanishing	ReLU activation	Derivative is exactly 1 for positive inputs
Vanishing	Residual connections	Gradient has a “shortcut” path that bypasses multiplication
Vanishing	LSTM/GRU gates	Learned gates control gradient flow
Exploding	Gradient clipping	Caps the gradient norm before updating
Both	Proper initialization (He/Xavier)	Keeps variance stable across layers
Both	Batch normalization	Re-centers activations each layer

When debugging a model that “isn’t learning,” always check gradient magnitudes per layer first. If early-layer gradients are orders of magnitude smaller than late-layer gradients, you have a vanishing gradient problem.

The Code (Backpropagation)

Let’s implement the graph above for a single neuron:

x \to z \to a \to L

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def train_neuron(x, w, b, y_true):
    # --- 1. Forward Pass ---
    z = w * x + b
    a = sigmoid(z)
    loss = (a - y_true)**2
    
    print(f"Prediction: {a:.4f}, Error: {loss:.4f}")
    
    # --- 2. Backward Pass (Chain Rule) ---
    # We want dL/dw (How much is w to blame?)
    
    # Link 1: How Loss changes with Activation
    dL_da = 2 * (a - y_true)
    
    # Link 2: How Activation changes with z
    da_dz = a * (1 - a)
    
    # Link 3: How z changes with weight w
    dz_dw = x
    
    # Total Chain: Multiply them all!
    dL_dw = dL_da * da_dz * dz_dw
    
    return dL_dw

# Test it
x = 2.0      # Input
w = 0.5      # Current Weight
b = 0.1      # Bias
y_true = 1.0 # Target (We want output to be 1)

gradient = train_neuron(x, w, b, y_true)
print(f"\nGradient (dL/dw): {gradient:.4f}")
print("Interpretation: Increasing w will reduce the error!")

This is Backpropagation. It’s just the Chain Rule applied to a graph!

Multi-Layer Networks

For a deep network with 100 layers, you just have a longer chain:

\frac{dL}{dw_1} = \frac{dL}{da_{100}} \cdot \frac{da_{100}}{dz_{100}} \dots \frac{dz_1}{dw_1}

The computer simply multiplies these numbers backward from the end to the start.

Practice Exercises

Exercise 1: Chain Rule Practice

# Given: h(x) = sin(x²)
# Find: h'(x) using chain rule

# TODO:
# 1. Identify inner and outer functions
# 2. Find their derivatives
# 3. Apply chain rule
# 4. Verify numerically at x=2

🎯 Practice Exercises & Real-World Applications

Challenge yourself! The chain rule is the engine of backpropagation - master it here!

Exercise 1: Supply Chain Impact Analysis 🏭

A semiconductor shortage affects the entire supply chain. Trace the impact:

# Supply chain model:
# 1. Chip shortage reduces chip supply: C(s) = 1000 - 50*s (s = shortage severity 0-10)
# 2. Less chips means fewer phones: P(c) = 0.1 * c (c = chips available)
# 3. Fewer phones means less revenue: R(p) = 800 * p - 0.01 * p² (p = phones made)

# TODO:
# 1. Compute dR/ds using the chain rule (how does revenue change with shortage severity?)
# 2. At s=5, how much revenue is lost per unit increase in shortage?
# 3. Which link in the chain has the biggest multiplier effect?

💡 Solution

import numpy as np

def chips(s):
    """Chip supply based on shortage severity"""
    return 1000 - 50 * s

def phones(c):
    """Phones produced based on chips available"""
    return 0.1 * c

def revenue(p):
    """Revenue from phones sold"""
    return 800 * p - 0.01 * p**2

# Derivatives of each link
def dC_ds(s):
    """d(chips)/d(shortage) = -50"""
    return -50

def dP_dC(c):
    """d(phones)/d(chips) = 0.1"""
    return 0.1

def dR_dP(p):
    """d(revenue)/d(phones) = 800 - 0.02*p"""
    return 800 - 0.02 * p

def chain_derivative(s):
    """
    Chain rule: dR/ds = dR/dP × dP/dC × dC/ds
    """
    c = chips(s)
    p = phones(c)
    
    return dR_dP(p) * dP_dC(c) * dC_ds(s)

print("🏭 Supply Chain Impact Analysis")
print("=" * 55)

s = 5  # Moderate shortage
c = chips(s)
p = phones(c)
r = revenue(p)

print(f"\n📊 At shortage severity s = {s}:")
print(f"   Chips available: {c}")
print(f"   Phones produced: {p}")
print(f"   Revenue: ${r:,.2f}")

# Chain rule calculation
print("\n🔗 Chain Rule Breakdown:")
print(f"   dC/ds (chip sensitivity): {dC_ds(s)} chips per severity unit")
print(f"   dP/dC (production rate): {dP_dC(c)} phones per chip")
print(f"   dR/dP (revenue per phone): ${dR_dP(p):.2f}")

total_impact = chain_derivative(s)
print(f"\n📉 Total Impact (dR/ds):")
print(f"   = {dR_dP(p):.2f} × {dP_dC(c)} × {dC_ds(s)}")
print(f"   = ${total_impact:,.2f} revenue per unit shortage increase")

# Sensitivity analysis
print("\n📈 Sensitivity Analysis:")
print("   Shortage | Revenue    | Marginal Impact")
print("   ---------|------------|----------------")
for sev in range(0, 11, 2):
    c = chips(sev)
    p = phones(c)
    r = revenue(p)
    impact = chain_derivative(sev)
    print(f"   {sev:8} | ${r:9,.0f} | ${impact:,.0f}/unit")

print("\n💡 Key Insight:")
print("   Even a small chip shortage has a MULTIPLIED effect on revenue!")
print("   This is why supply chain disruptions are so devastating.")

Real-World Insight: This is exactly what happened during COVID - a small disruption in Taiwan chip factories cascaded through auto, phone, and appliance industries, causing billions in lost revenue!

Exercise 2: Backpropagation by Hand ✋

Compute gradients manually for a 2-layer neural network:

import numpy as np

# Network: Input(1) → Hidden(2) → Output(1)
# x → [w1, w2] → h1, h2 → ReLU → [v1, v2] → y
# Loss = (y - target)²

# Forward pass:
x = 2.0
w = np.array([0.5, -0.3])  # Input to hidden
v = np.array([0.8, 0.4])   # Hidden to output
target = 1.0

# h = w * x
# a = ReLU(h)
# y = v · a

# TODO:
# 1. Compute forward pass
# 2. Compute dL/dv (output layer gradients)
# 3. Compute dL/dw (input layer gradients using chain rule)
# 4. Update weights with lr=0.1

💡 Solution

import numpy as np

def relu(x):
    return np.maximum(0, x)

def relu_derivative(x):
    return (x > 0).astype(float)

# Setup
x = 2.0
w = np.array([0.5, -0.3])
v = np.array([0.8, 0.4])
target = 1.0
lr = 0.1

print("✋ Manual Backpropagation")
print("=" * 55)

# Forward pass
print("\n➡️ Forward Pass:")
h = w * x                    # Pre-activation [1.0, -0.6]
print(f"   h = w × x = {w} × {x} = {h}")

a = relu(h)                  # Activation [1.0, 0.0]
print(f"   a = ReLU(h) = {a}")

y = np.dot(v, a)             # Output 0.8
print(f"   y = v · a = {v} · {a} = {y}")

loss = (y - target) ** 2     # Loss 0.04
print(f"   L = (y - target)² = ({y} - {target})² = {loss}")

# Backward pass
print("\n⬅️ Backward Pass (Chain Rule):")

# Step 1: dL/dy
dL_dy = 2 * (y - target)     # -0.4
print(f"   dL/dy = 2(y - target) = {dL_dy}")

# Step 2: dL/dv (output layer)
dL_dv = dL_dy * a            # [-0.4, 0]
print(f"   dL/dv = dL/dy × a = {dL_dy} × {a} = {dL_dv}")

# Step 3: dL/da
dL_da = dL_dy * v            # [-0.32, -0.16]
print(f"   dL/da = dL/dy × v = {dL_dy} × {v} = {dL_da}")

# Step 4: dL/dh (through ReLU)
dL_dh = dL_da * relu_derivative(h)  # [-0.32, 0] (ReLU kills the negative)
print(f"   dL/dh = dL/da × ReLU'(h) = {dL_da} × {relu_derivative(h)} = {dL_dh}")

# Step 5: dL/dw (input layer)
dL_dw = dL_dh * x            # [-0.64, 0]
print(f"   dL/dw = dL/dh × x = {dL_dh} × {x} = {dL_dw}")

# Weight update
print("\n📝 Weight Updates:")
w_new = w - lr * dL_dw
v_new = v - lr * dL_dv
print(f"   w: {w} → {w_new}")
print(f"   v: {v} → {v_new}")

# Verify improvement
h_new = w_new * x
a_new = relu(h_new)
y_new = np.dot(v_new, a_new)
loss_new = (y_new - target) ** 2

print(f"\n✅ Verification:")
print(f"   Old prediction: {y:.4f}, Loss: {loss:.4f}")
print(f"   New prediction: {y_new:.4f}, Loss: {loss_new:.4f}")
print(f"   Improvement: {(1 - loss_new/loss)*100:.1f}%")

Real-World Insight: This is EXACTLY what PyTorch/TensorFlow’s loss.backward() does! They just do it for millions of weights automatically using computational graphs.

Exercise 3: Weather Cascade Model 🌧️

Predict how atmospheric conditions cascade to affect temperature:

# Cascade: Pressure → Humidity → Cloud Cover → Temperature
# 
# humidity(p) = 100 - 0.1*(p - 1013)²  (p = pressure in hPa)
# clouds(h) = 0.8 * h - 10             (h = humidity %)
# temp(c) = 30 - 0.2 * c               (c = cloud cover %)
#
# Current pressure: p = 1020 hPa

# TODO:
# 1. Compute dT/dp using chain rule
# 2. If pressure increases by 5 hPa, how much does temperature change?
# 3. Visualize the cascade effect

💡 Solution

import numpy as np

def humidity(p):
    """Humidity decreases as pressure deviates from ideal"""
    return 100 - 0.1 * (p - 1013)**2

def clouds(h):
    """Cloud cover increases with humidity"""
    return max(0, 0.8 * h - 10)

def temperature(c):
    """Temperature decreases with cloud cover"""
    return 30 - 0.2 * c

# Derivatives
def dH_dP(p):
    """d(humidity)/d(pressure) = -0.2*(p - 1013)"""
    return -0.2 * (p - 1013)

def dC_dH(h):
    """d(clouds)/d(humidity) = 0.8"""
    return 0.8

def dT_dC(c):
    """d(temp)/d(clouds) = -0.2"""
    return -0.2

def full_chain(p):
    """dT/dP = dT/dC × dC/dH × dH/dP"""
    h = humidity(p)
    c = clouds(h)
    return dT_dC(c) * dC_dH(h) * dH_dP(p)

print("🌧️ Weather Cascade Analysis")
print("=" * 55)

p = 1020  # Current pressure

# Forward calculation
h = humidity(p)
c = clouds(h)
t = temperature(c)

print(f"\n📊 Current Conditions (P = {p} hPa):")
print(f"   Humidity: {h:.1f}%")
print(f"   Cloud Cover: {c:.1f}%")
print(f"   Temperature: {t:.1f}°C")

# Chain rule calculation
print("\n🔗 Chain Rule Breakdown:")
print(f"   dH/dP = -0.2 × (P - 1013) = {dH_dP(p):.3f}")
print(f"   dC/dH = 0.8")
print(f"   dT/dC = -0.2")
print(f"\n   dT/dP = dT/dC × dC/dH × dH/dP")
print(f"        = -0.2 × 0.8 × {dH_dP(p):.3f}")
print(f"        = {full_chain(p):.4f}°C per hPa")

# Prediction for pressure change
delta_p = 5
delta_t = full_chain(p) * delta_p
t_new_predicted = t + delta_t

# Actual calculation
p_new = p + delta_p
h_new = humidity(p_new)
c_new = clouds(h_new)
t_new_actual = temperature(c_new)

print(f"\n🌡️ Effect of {delta_p} hPa Pressure Increase:")
print(f"   Predicted temp change: {delta_t:.2f}°C")
print(f"   New temp (predicted): {t_new_predicted:.2f}°C")
print(f"   New temp (actual): {t_new_actual:.2f}°C")
print(f"   Prediction error: {abs(t_new_actual - t_new_predicted):.4f}°C")

# Cascade visualization
print("\n📈 Cascade Effect Visualization:")
print("   Pressure → Humidity → Clouds → Temperature")
for p_val in range(1005, 1030, 5):
    h_val = humidity(p_val)
    c_val = clouds(h_val)
    t_val = temperature(c_val)
    bars = int(t_val)
    print(f"   P={p_val}: {'█' * bars}{'░' * (30-bars)} {t_val:.1f}°C")

Real-World Insight: Weather models use similar cascade calculations with hundreds of interacting variables. The chain rule helps meteorologists understand how small changes in one variable propagate through the entire system!

Exercise 4: Viral Growth Model 📱

Model how a social media post goes viral:

# Viral cascade model:
# 1. Quality → Initial shares: S(q) = 10 * q²
# 2. Initial shares → Network reach: R(s) = 100 * ln(s + 1)
# 3. Network reach → New followers: F(r) = 0.05 * r * (1 - r/10000)

# A post has quality score q = 8

# TODO:
# 1. Compute the full derivative dF/dq
# 2. What's the marginal value of improving quality by 1 point?
# 3. At what quality level does adding more quality have diminishing returns?

💡 Solution

import numpy as np

def shares(q):
    """Initial shares based on content quality"""
    return 10 * q**2

def reach(s):
    """Network reach from initial shares (logarithmic growth)"""
    return 100 * np.log(s + 1)

def followers(r):
    """New followers (logistic-like, saturates at high reach)"""
    return 0.05 * r * (1 - r/10000)

# Derivatives
def dS_dQ(q):
    """d(shares)/d(quality) = 20*q"""
    return 20 * q

def dR_dS(s):
    """d(reach)/d(shares) = 100/(s+1)"""
    return 100 / (s + 1)

def dF_dR(r):
    """d(followers)/d(reach) = 0.05*(1 - 2r/10000)"""
    return 0.05 * (1 - 2*r/10000)

def full_chain(q):
    """dF/dQ using chain rule"""
    s = shares(q)
    r = reach(s)
    return dF_dR(r) * dR_dS(s) * dS_dQ(q)

print("📱 Viral Growth Analysis")
print("=" * 55)

q = 8  # Current quality

# Forward calculation
s = shares(q)
r = reach(s)
f = followers(r)

print(f"\n📊 Current Post (Quality = {q}):")
print(f"   Initial shares: {s:.0f}")
print(f"   Network reach: {r:.0f}")
print(f"   New followers: {f:.1f}")

# Chain rule breakdown
print("\n🔗 Chain Rule at q = 8:")
print(f"   dS/dQ = 20q = {dS_dQ(q)}")
print(f"   dR/dS = 100/(s+1) = {dR_dS(s):.4f}")
print(f"   dF/dR = 0.05(1 - 2r/10000) = {dF_dR(r):.4f}")
print(f"\n   dF/dQ = {dF_dR(r):.4f} × {dR_dS(s):.4f} × {dS_dQ(q)}")
print(f"        = {full_chain(q):.4f} followers per quality point")

# Marginal analysis
print("\n📈 Marginal Analysis:")
print("   Quality | Shares | Reach  | Followers | dF/dQ")
print("   --------|--------|--------|-----------|-------")
for quality in range(1, 15, 2):
    s_val = shares(quality)
    r_val = reach(s_val)
    f_val = followers(r_val)
    deriv = full_chain(quality)
    print(f"   {quality:7} | {s_val:6.0f} | {r_val:6.0f} | {f_val:9.2f} | {deriv:.4f}")

# Find diminishing returns point
print("\n🎯 Diminishing Returns Analysis:")
qualities = np.linspace(1, 15, 100)
derivatives = [full_chain(q) for q in qualities]
max_deriv_idx = np.argmax(derivatives)
optimal_q = qualities[max_deriv_idx]

print(f"   Maximum marginal impact at quality ≈ {optimal_q:.1f}")
print(f"   After this point, each quality improvement gives less followers")
print(f"\n💡 Insight: It's not always worth perfecting content!")
print(f"   Publishing at quality {optimal_q:.0f} maximizes growth rate per effort unit")

Real-World Insight: Social media algorithms like TikTok’s use similar models to predict viral potential. The chain rule helps identify which factor to improve for maximum impact - that’s why “good enough, post fast” often beats perfection!

Key Takeaways

✅ Chain rule = multiply derivatives along the chain
✅ Backpropagation = chain rule applied backward
✅ Deep learning = chain rule through many layers
✅ Gradients flow backward = from output to input
✅ Every framework uses this = PyTorch, TensorFlow, JAX

Automatic Differentiation: How PyTorch Does It

You just learned what happens inside loss.backward()!

Modern frameworks use Automatic Differentiation (AutoDiff) - they build a computational graph during the forward pass and automatically apply the chain rule during backward pass.

import torch

# PyTorch builds the chain automatically!
x = torch.tensor([2.0], requires_grad=True)

# Forward pass (builds computational graph)
z = x ** 2       # Node 1
a = torch.sin(z) # Node 2  
loss = a.sum()   # Node 3

# Backward pass (applies chain rule automatically)
loss.backward()

# x.grad contains dL/dx (computed via chain rule!)
print(f"Gradient: {x.grad.item():.4f}")

# Manual verification:
# dL/dx = dL/da × da/dz × dz/dx
#       = 1 × cos(4) × 4
#       = 4 × cos(4) = -2.6146
print(f"Manual:   {4 * torch.cos(torch.tensor(4.0)).item():.4f}")

Two Types of AutoDiff

Type	Description	Frameworks
Forward Mode	Compute derivatives alongside forward pass	Good for few inputs, many outputs
Reverse Mode	Build graph forward, compute derivatives backward	PyTorch, TensorFlow (good for many inputs, few outputs)

Neural networks have millions of inputs (weights) and one output (loss), so reverse mode (backprop) is optimal!

Common Debugging Issues & Solutions

Debugging gradient problems is one of the most common challenges in ML. Here’s what to watch for:

🔴 Vanishing Gradients

Symptom: Early layers don’t learn, gradients near zeroCause: Chain rule multiplies small numbers. If each layer’s derivative < 1, the product → 0.Example: Sigmoid derivatives max at 0.25. After 10 layers:

0.25^{10} = 0.00000095

Solutions:

Use ReLU instead of sigmoid (derivative = 0 or 1)
Batch normalization
Skip connections (ResNet)
LSTM/GRU for RNNs

🔴 Exploding Gradients

Symptom: Loss becomes NaN, weights blow upCause: Chain rule multiplies large numbers. If each layer’s derivative > 1, the product → ∞.Solutions:

Gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
Proper weight initialization (Xavier, He)
Lower learning rate

🔴 Dead Neurons (ReLU)

Symptom: Some neurons never activate, gradient = 0 foreverCause: ReLU outputs 0 for negative inputs, gradient = 0, never recovers.Solutions:

Leaky ReLU: small slope for negative inputs
Lower learning rate
Better initialization

🟡 Numerical Instability

Symptom: Loss oscillates wildly or becomes inf/nanSolutions:

# Instead of: log(x)
torch.log(x + 1e-8)  # Add small epsilon

# Instead of: exp(x) 
torch.clamp(x, max=50)  # Clip before exp

# Use numerically stable implementations
F.log_softmax(x, dim=-1)  # Instead of log(softmax(x))

Gradient Checking: Verify Your Implementation

def gradient_check(f, x, analytical_grad, eps=1e-5):
    """Verify analytical gradient with numerical approximation"""
    numerical_grad = (f(x + eps) - f(x - eps)) / (2 * eps)
    
    relative_error = abs(numerical_grad - analytical_grad) / max(abs(numerical_grad), abs(analytical_grad), 1e-8)
    
    if relative_error < 1e-5:
        print(f"✅ Gradient check PASSED (error: {relative_error:.2e})")
    else:
        print(f"❌ Gradient check FAILED (error: {relative_error:.2e})")
        print(f"   Numerical: {numerical_grad:.6f}")
        print(f"   Analytical: {analytical_grad:.6f}")
    
    return relative_error

# Example usage
f = lambda x: x**3 + 2*x**2 + x
analytical = lambda x: 3*x**2 + 4*x + 1  # f'(x)

x = 2.0
gradient_check(f, x, analytical(x))

What’s Next?

You now understand how gradients flow through compositions. But how do we USE these gradients to actually train models? That’s Gradient Descent - the optimization algorithm that powers all of machine learning!

Next: Gradient Descent

Learn the algorithm that makes machines learn

Interview Deep-Dive

Walk me through exactly how backpropagation uses the chain rule to compute gradients in a 3-layer neural network. Do not hand-wave -- be specific about what is multiplied at each step.

Strong Answer:

Let me set up a concrete 3-layer network: input x, hidden layer h1 = ReLU(W1x + b1), hidden layer h2 = ReLU(W2h1 + b2), output y_hat = sigmoid(W3h2 + b3), loss L = -(ylog(y_hat) + (1-y)*log(1-y_hat)).
The forward pass computes and caches all intermediate values: z1, h1, z2, h2, z3, y_hat, L. These cached values are essential for the backward pass.
Backward pass starts at the loss. dL/dy_hat = -(y/y_hat) + (1-y)/(1-y_hat). For the output layer: dL/dz3 = dL/dy_hat * dy_hat/dz3 = dL/dy_hat * sigmoid’(z3) = y_hat - y (this simplifies beautifully for cross-entropy + sigmoid). Then dL/dW3 = dL/dz3 * h2^T (outer product), and dL/db3 = dL/dz3.
To propagate to layer 2: dL/dh2 = W3^T * dL/dz3. Then dL/dz2 = dL/dh2 * ReLU’(z2), where ReLU’(z2) is 1 where z2 > 0 and 0 elsewhere (element-wise). Then dL/dW2 = dL/dz2 * h1^T, dL/db2 = dL/dz2.
Same pattern for layer 1: dL/dh1 = W2^T * dL/dz2, dL/dz1 = dL/dh1 * ReLU’(z1), dL/dW1 = dL/dz1 * x^T, dL/db1 = dL/dz1.
The key pattern at every layer is the same three operations: (1) multiply by the transpose of the weight matrix to propagate the error backward, (2) multiply element-wise by the activation derivative, (3) compute the weight gradient as the outer product of the upstream gradient and the cached input activation. This uniformity is why backpropagation is so elegant and implementable.
The chain rule is doing the heavy lifting: dL/dW1 involves multiplying through sigmoid’ * W3^T * ReLU’ * W2^T * ReLU’ * x^T. Each factor in that chain is a local derivative at one layer.

Follow-up: You mentioned caching intermediate values. What is the memory cost, and how does it scale with batch size and network depth?For each layer, you need to store the pre-activation z and the post-activation h (or at minimum one of them, depending on the activation function). For a batch of B samples with layer dimension d, each activation tensor is B * d floating-point numbers. With float32, a layer with d=4096 and batch size 32 takes 32 * 4096 * 4 bytes = 512KB. For a transformer with 96 layers, that is roughly 50MB just for activations — and that ignores attention matrices which scale quadratically with sequence length. For GPT-3 scale models with 96 layers, 12288 hidden dimension, batch size 8, and sequence length 2048, activation memory can hit hundreds of gigabytes. This is precisely why gradient checkpointing, mixed-precision training, and model parallelism are essential at scale.

What is the vanishing gradient problem, and why does the chain rule make it mathematically inevitable for certain activation functions? How have architectures evolved to address it?

Strong Answer:

The vanishing gradient problem occurs when gradients become exponentially small as they propagate backward through many layers. The chain rule says the gradient for an early layer is a product of all the local derivatives along the path to the loss. If each local derivative has magnitude less than 1, the product shrinks exponentially with depth.
For sigmoid, the maximum derivative is 0.25 (at z=0). For a 20-layer network, even in the best case, the gradient for layer 1 is at most 0.25^19 times the output gradient — that is about 3.6e-12. In practice it is worse because weights and biases shift activations into saturated regions where the sigmoid derivative is much less than 0.25.
The exploding gradient problem is the mirror image: if local derivatives are consistently greater than 1, the product grows exponentially. This happens with poorly initialized weight matrices whose spectral norm exceeds 1.
Architectural solutions have evolved in clear stages. ReLU (2011) fixed the activation derivative issue: its derivative is exactly 1 for positive inputs, so it does not shrink gradients. But dead ReLU neurons (permanently zero output) create gradient “holes.” LSTMs and GRUs (1997/2014) added gating mechanisms that create additive gradient paths, allowing gradients to flow across long time sequences without multiplicative decay. ResNets (2015) added skip connections, creating an identity shortcut: dy/dx = dF/dx + I. That identity term means gradients always have a clear path regardless of how small dF/dx becomes. Transformers (2017) combined residual connections with layer normalization, giving even more stable gradient flow. The modern pattern is clear: every major architectural innovation in the last decade has been partly motivated by improving gradient flow.

Follow-up: Residual connections solve vanishing gradients, but do they introduce any new problems or trade-offs?Yes, a few subtle ones. First, the identity shortcut can make it easy for the network to learn the identity function at each layer, which means some layers may contribute very little — effectively wasting capacity. Research on “lazy” layers in ResNets has shown that some layers can be pruned with minimal accuracy loss. Second, residual connections require that the input and output dimensions match (or you need a projection shortcut, which adds parameters and computation). Third, the additive structure can lead to feature explosion in very deep networks — the representation grows by accumulation, so the effective rank of the feature space can increase uncontrollably. Dense connections (DenseNet) and careful normalization help manage this. Fourth, for optimization, residual connections create a smoother loss landscape (Li et al., 2018 showed this visually), which is great for convergence but can also make the network less sensitive to regularization, potentially hurting generalization on small datasets.

Why is reverse-mode autodiff (backpropagation) preferred over forward-mode autodiff for neural network training? When would forward-mode actually be better?

Strong Answer:

The key distinction is computational cost relative to the number of inputs and outputs. Reverse-mode computes the gradient of one scalar output (the loss) with respect to all N parameters in one backward pass — cost is O(1) times the forward pass cost, regardless of N. Forward-mode computes the derivative of all outputs with respect to one input parameter per pass — so for N parameters, you need N forward passes.
Neural networks have a scalar loss (one output) and millions of parameters (many inputs). Reverse-mode needs 1 pass; forward-mode needs millions. The choice is obvious for training.
Forward-mode is better when you have few inputs and many outputs. For example, computing the Jacobian of a function f: R^2 to R^1000 requires 2 forward-mode passes but 1000 reverse-mode passes. This situation arises in physics simulations, sensitivity analysis, and computing Jacobian-vector products for certain optimization algorithms.
Another case for forward-mode: when memory is extremely constrained. Reverse-mode must store the full computational graph (or checkpoint segments of it). Forward-mode processes one pass without storing the graph, using only O(1) extra memory per operation. For very long computation chains (like unrolled RNNs over thousands of timesteps), forward-mode can be practical when reverse-mode runs out of memory.
In practice, JAX offers both modes via jax.jvp (forward) and jax.vjp (reverse), and sophisticated users mix them. For Hessian-vector products, you can nest a forward-mode pass inside a reverse-mode pass: reverse-mode gives you the gradient, then forward-mode differentiates the gradient computation with respect to parameters. This is cheaper than computing the full Hessian.

Follow-up: Can you explain what a vector-Jacobian product (VJP) is and why it is the core operation of backpropagation?A VJP computes v^T * J where v is a vector (the upstream gradient) and J is the Jacobian of some layer. The result is a vector of the same dimension as the layer’s input. This is exactly what happens at each layer during backpropagation: you receive the gradient of the loss with respect to the layer’s output (that is v), and you need the gradient with respect to the layer’s input (that is v^T * J). The beauty is that you never form the full Jacobian matrix — you compute the VJP directly using the chain rule and the layer’s local derivatives. For a matrix multiplication layer y = Wx, the Jacobian with respect to x is W, and the VJP is v^T * W = W^T * v. For a ReLU layer, the Jacobian is a diagonal matrix of 0s and 1s, and the VJP is simply element-wise multiplication by the ReLU mask. This is why backpropagation is a sequence of VJPs, not a sequence of matrix multiplications by full Jacobians.

You implemented a custom activation function and the model is not learning. Gradient checking passes with relative error below 1e-7. What else could be wrong?

Strong Answer:

Gradient checking passing means the backward computation is correct — the gradient you compute matches the numerical derivative. But correct gradients do not guarantee good training dynamics. Several issues can still prevent learning.
First, check the gradient magnitude. If your custom activation has derivatives that are consistently very small (say below 0.01) or very large, you will get vanishing or exploding gradients through the chain rule even though each individual gradient is correct. Plot the distribution of your activation’s derivative across typical input ranges.
Second, check for dead zones. If your activation outputs zero (or a constant) for a wide range of inputs, those regions have zero gradient and the network cannot learn through them. This is the “dead ReLU” problem generalized. Check what fraction of neurons are in the zero-gradient region during training.
Third, check the initialization compatibility. He initialization assumes ReLU-like activations that preserve variance. Xavier initialization assumes linear-like activations. If your custom activation has a different output variance profile, the standard initializations will produce either exploding or vanishing activations at the start of training, even though the gradients are technically correct.
Fourth, check numerical stability in the forward pass. If your activation involves operations like exp() or log(), intermediate values might overflow or underflow even though the final gradient computation is correct at the test point you checked. Gradient checking typically uses a single well-behaved input; production data may hit edge cases.
Fifth, verify the activation’s output range is compatible with the loss function. If you are using cross-entropy loss which expects probabilities in (0,1), but your activation outputs values in (-1, 1), the loss function will produce garbage even though the gradients are correct.
My debugging sequence: plot activation outputs and their derivatives across the input range, verify output range compatibility with the loss, check gradient norms per layer during the first few training steps, and try the model on a trivially small dataset (2-4 examples) to verify it can overfit.

Follow-up: How would you design a gradient checking procedure that catches the edge-case failures you just described, not just the single-point check?I would extend the standard gradient check in three ways. First, check at multiple input points across the expected range, not just one — include extreme values near the edges of typical activation inputs (like -5, -1, 0, 1, 5 for pre-activations). Second, check with full mini-batches, not single inputs, because batch interactions (like in BatchNorm) can introduce gradient errors that are invisible in single-sample checks. Third, run a “gradient flow check” — do one full forward and backward pass on real data, then verify that gradient norms at each layer are within a reasonable range (say 1e-6 to 1e3). If any layer’s gradient norm is zero or astronomically large, that flags a gradient flow problem even if the pointwise gradient is correct. Tools like PyTorch’s register_backward_hook make this easy to instrument.

Explain the relationship between the chain rule, computational graphs, and how PyTorch's autograd system actually works under the hood.

Strong Answer:

When you execute operations in PyTorch with requires_grad=True, every operation builds a node in a directed acyclic graph (the computational graph). Each node records: what operation was performed, what the inputs were, and a function pointer to compute the local gradient (the VJP function).
During the forward pass, PyTorch records this graph dynamically — this is called “define-by-run” or “eager mode” autograd. Unlike TensorFlow 1.x which built a static graph first, PyTorch constructs the graph on-the-fly as Python executes. This means control flow (if statements, loops) naturally works because the graph simply records whatever path the code actually takes.
When you call loss.backward(), PyTorch performs a topological sort of the graph starting from the loss node. It then visits nodes in reverse topological order (from loss back toward inputs). At each node, it calls the node’s VJP function with the accumulated upstream gradient, producing gradients for the node’s inputs. These gradients are accumulated (summed) into each parameter’s .grad attribute.
The chain rule manifests as the composition of VJPs. Each node only needs to know its own local derivative. The global gradient (loss with respect to any parameter) emerges from the sequence of local VJP applications — exactly the chain rule.
A critical implementation detail: gradient accumulation. If a tensor is used in multiple operations (like a weight matrix used in a skip connection), gradients from both paths are summed. This is mathematically correct because the total derivative when a variable appears in multiple terms is the sum of partial effects. But it also means you MUST zero gradients before each training step (optimizer.zero_grad()), or gradients from previous batches accumulate and corrupt the update.
Memory management: once backward() completes, PyTorch by default frees the computational graph (unless you pass retain_graph=True). This is important because the graph holds references to all intermediate tensors. Forgetting to release it is a common memory leak in custom training loops.

Follow-up: What happens if you try to call backward() twice on the same graph? When would you actually need to do this?By default, calling backward() twice raises a RuntimeError because the graph has been freed after the first backward. You need retain_graph=True to keep it. A legitimate use case is when you have multiple losses that share a computational graph — for example, in GANs, the discriminator loss and the generator loss share some forward computations. You call backward on the discriminator loss (with retain_graph=True), update discriminator weights, then call backward on the generator loss through the same graph. Another case is computing higher-order derivatives: to differentiate the gradient itself (for Hessian-vector products or gradient penalties like in WGAN-GP), you need to backward through the backward computation, which requires retaining the original graph. PyTorch supports this via create_graph=True in the first backward call, which makes the gradient computation itself differentiable.

​Chain Rule & Backpropagation

​Your Challenge: The Butterfly Effect

​The Domino Effect

​The Intuition: Composition of Functions

​The Problem

​The Solution

​Let’s Code It

​The Telephone Game Analogy

​Mathematical Definition

​Simple Example

​Example 1: Multi-Stage Business Process

​The Scenario

​Solution Using Chain Rule

​Example 2: Your Learning Chain

​The Scenario

​The Functions

​Applying the Chain Rule

​Example 3: Your First Neural Network

​The Computational Graph

​Neural Network Backpropagation: Layer by Layer

​Network Architecture

​Forward Pass Equations

​Backward Pass: Compute ALL Gradients

​Complete Python Implementation

​The Code (Backpropagation)

​Multi-Layer Networks

​Practice Exercises

​Exercise 1: Chain Rule Practice

​🎯 Practice Exercises & Real-World Applications

​Exercise 1: Supply Chain Impact Analysis 🏭

​Exercise 2: Backpropagation by Hand ✋

​Exercise 3: Weather Cascade Model 🌧️

​Exercise 4: Viral Growth Model 📱

​Key Takeaways

​Automatic Differentiation: How PyTorch Does It

​Two Types of AutoDiff

​Common Debugging Issues & Solutions

​Gradient Checking: Verify Your Implementation

​What’s Next?

Next: Gradient Descent

​Interview Deep-Dive

Chain Rule & Backpropagation

Your Challenge: The Butterfly Effect

The Domino Effect

The Intuition: Composition of Functions

The Problem

The Solution

Let’s Code It

The Telephone Game Analogy

Mathematical Definition

Simple Example

Example 1: Multi-Stage Business Process

The Scenario

Solution Using Chain Rule

Example 2: Your Learning Chain

The Scenario

The Functions

Applying the Chain Rule

Example 3: Your First Neural Network

The Computational Graph

Neural Network Backpropagation: Layer by Layer

Network Architecture

Forward Pass Equations

Backward Pass: Compute ALL Gradients

Complete Python Implementation

The Code (Backpropagation)

Multi-Layer Networks

Practice Exercises

Exercise 1: Chain Rule Practice

🎯 Practice Exercises & Real-World Applications

Exercise 1: Supply Chain Impact Analysis 🏭

Exercise 2: Backpropagation by Hand ✋

Exercise 3: Weather Cascade Model 🌧️

Exercise 4: Viral Growth Model 📱

Key Takeaways

Automatic Differentiation: How PyTorch Does It

Two Types of AutoDiff

Common Debugging Issues & Solutions

Gradient Checking: Verify Your Implementation

What’s Next?

Interview Deep-Dive