> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Ensemble Methods
> Combine many models for better predictions - the wisdom of crowds
# Ensemble Methods
## The Wisdom of Crowds
**Question**: Who's smarter - one expert or 100 average people?
**Surprisingly**: The crowd often wins!
### A Real Experiment
In 1906, statistician Francis Galton visited a county fair. 787 people guessed the weight of an ox:
* Individual guesses ranged wildly
* Average of all guesses: **1,197 pounds**
* Actual weight: **1,198 pounds**
**The crowd was off by 1 pound!**
This is the core idea behind ensemble learning:
> **Many weak learners combined can outperform a single strong learner**
***
## Why Ensembles Work
Imagine 5 decision trees, each 70% accurate:
```python theme={null}
import numpy as np
# Each model votes
def ensemble_vote(models_correct):
"""
Returns True if majority of models are correct.
"""
return sum(models_correct) > len(models_correct) / 2
# Simulate 1000 predictions
np.random.seed(42)
accuracy = 0.7
n_models = 5
n_simulations = 10000
correct = 0
for _ in range(n_simulations):
# Each model independently right 70% of the time
votes = np.random.random(n_models) < accuracy
if ensemble_vote(votes):
correct += 1
print(f"Single model accuracy: {accuracy:.1%}")
print(f"Ensemble accuracy: {correct/n_simulations:.1%}")
# Output: ~83%!
```
**The Math**: For majority voting with independent 70% accurate models:
$P(\text{majority correct}) = \sum_{k=3}^{5} \binom{5}{k} (0.7)^k (0.3)^{5-k} \approx 83.7\%$
***
## Bagging: Bootstrap Aggregating
**Idea**: Train multiple models on different random samples of data.
### How Bagging Works
1. Create N random samples (with replacement) from training data
2. Train a model on each sample
3. Average predictions (regression) or vote (classification)
```python theme={null}
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np
# Load data
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
iris.data, iris.target, test_size=0.2, random_state=42
)
class SimpleBaggingClassifier:
def __init__(self, base_model, n_estimators=10):
self.n_estimators = n_estimators
self.base_model = base_model
self.models = []
def fit(self, X, y):
n_samples = len(X)
for _ in range(self.n_estimators):
# Bootstrap sample (sample with replacement)
indices = np.random.choice(n_samples, n_samples, replace=True)
X_bootstrap = X[indices]
y_bootstrap = y[indices]
# Train a model
model = DecisionTreeClassifier(max_depth=5)
model.fit(X_bootstrap, y_bootstrap)
self.models.append(model)
def predict(self, X):
# Get predictions from all models
predictions = np.array([model.predict(X) for model in self.models])
# Majority vote
from scipy.stats import mode
return mode(predictions, axis=0)[0].ravel()
# Test
bagging = SimpleBaggingClassifier(DecisionTreeClassifier, n_estimators=10)
bagging.fit(X_train, y_train)
predictions = bagging.predict(X_test)
print(f"Bagging Accuracy: {np.mean(predictions == y_test):.2%}")
```
***
## Random Forest: Bagging + Feature Randomness
Random Forest = Bagging + **Random Feature Selection**
At each split, only consider a random subset of features (typically sqrt(n\_features) for classification, n\_features/3 for regression).
This makes trees more **diverse**, improving ensemble performance. Here's the key insight: if one feature is extremely predictive (like "credit score" for loan approval), regular bagging would make every tree split on that feature first, and all trees would look nearly identical. By forcing each tree to sometimes "ignore" the best feature, we create a diverse committee where each tree brings a different perspective. Diversity is the secret sauce of ensembles -- correlated models don't add value, but diverse models cancel each other's errors.
```python theme={null}
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
# Load data
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
cancer.data, cancer.target, test_size=0.2, random_state=42
)
# Train Random Forest
rf = RandomForestClassifier(
n_estimators=100, # 100 trees -- more trees = better (diminishing returns after ~200)
max_depth=10, # Limit tree depth -- prevents individual trees from memorizing data
max_features='sqrt', # Only consider sqrt(n) features per split -- the key to diversity
random_state=42 # For reproducibility -- remove in production
)
# Practical tip: Random Forest is remarkably forgiving of hyperparameters.
# n_estimators=100 and defaults for everything else is a solid starting point.
# Unlike gradient boosting, more trees in a Random Forest NEVER hurts
# (accuracy plateaus but doesn't decrease) -- it only costs compute time.
rf.fit(X_train, y_train)
# Evaluate
y_pred = rf.predict(X_test)
print("Random Forest Performance:")
print(classification_report(y_test, y_pred, target_names=cancer.target_names))
# Compare to single tree
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=10, random_state=42)
tree.fit(X_train, y_train)
print(f"\nSingle Tree Accuracy: {tree.score(X_test, y_test):.2%}")
print(f"Random Forest Accuracy: {rf.score(X_test, y_test):.2%}")
```
### Feature Importance
Random Forests tell you which features matter most:
```python theme={null}
import matplotlib.pyplot as plt
# Get feature importances
importances = rf.feature_importances_
indices = np.argsort(importances)[::-1][:10] # Top 10
# Plot
plt.figure(figsize=(10, 6))
plt.bar(range(10), importances[indices])
plt.xticks(range(10), [cancer.feature_names[i] for i in indices], rotation=45, ha='right')
plt.title('Top 10 Feature Importances')
plt.tight_layout()
plt.show()
```
***
## Boosting: Learning from Mistakes
**Key Idea**: Train models sequentially, each focusing on what previous models got wrong.
### AdaBoost (Adaptive Boosting)
1. Train a model
2. Increase weights of misclassified samples
3. Train next model (focuses on hard examples)
4. Combine all models with weighted voting
```python theme={null}
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
# Train AdaBoost
ada = AdaBoostClassifier(
estimator=DecisionTreeClassifier(max_depth=1), # "Stumps"
n_estimators=50,
learning_rate=1.0,
random_state=42
)
ada.fit(X_train, y_train)
print(f"AdaBoost Accuracy: {ada.score(X_test, y_test):.2%}")
```
### Gradient Boosting
Instead of reweighting samples, fit each tree to the **residual errors** -- the gap between what we predicted and what actually happened:
$\text{New Model} = \text{Previous Model} + \text{Learning Rate} \times \text{Tree that predicts errors}$
Think of it like a team of editors reviewing a document. The first editor makes a rough draft. The second editor only focuses on fixing the first editor's mistakes. The third editor fixes what the second one missed. Each editor makes the document incrementally better, and the learning rate controls how much you trust each editor's corrections (a small learning rate means "make cautious edits," which usually works better).
```python theme={null}
from sklearn.ensemble import GradientBoostingClassifier
# Train Gradient Boosting
gb = GradientBoostingClassifier(
n_estimators=100,
learning_rate=0.1,
max_depth=3,
random_state=42
)
gb.fit(X_train, y_train)
print(f"Gradient Boosting Accuracy: {gb.score(X_test, y_test):.2%}")
```
***
## XGBoost: The Competition Winner
XGBoost (Extreme Gradient Boosting) is often the best choice for tabular data.
```python theme={null}
# pip install xgboost
from xgboost import XGBClassifier
# Train XGBoost
xgb = XGBClassifier(
n_estimators=100,
learning_rate=0.1,
max_depth=5,
use_label_encoder=False,
eval_metric='logloss',
random_state=42
)
xgb.fit(X_train, y_train)
print(f"XGBoost Accuracy: {xgb.score(X_test, y_test):.2%}")
```
### Why XGBoost Wins
* **Regularization**: Built-in L1/L2 regularization prevents overfitting without you having to think about it
* **Parallel training**: Uses all CPU cores -- tree building is parallelized at the split-finding level
* **Missing values**: Handles them automatically by learning the optimal default direction at each split
* **Optimized**: Carefully engineered for speed with histogram-based splitting and cache-aware computation
**When to reach for XGBoost vs Random Forest**: XGBoost typically achieves 1-3% higher accuracy on tabular data but requires more careful tuning (learning\_rate, max\_depth, n\_estimators interact heavily). Random Forest is more "set and forget." If you have time to tune, use XGBoost. If you need a quick, reliable baseline, use Random Forest.
***
## Comparison: When to Use What?
```python theme={null}
from sklearn.ensemble import (
RandomForestClassifier,
GradientBoostingClassifier,
AdaBoostClassifier
)
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
# Load data
cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
# Compare models
models = {
'Logistic Regression': LogisticRegression(max_iter=5000),
'Decision Tree': DecisionTreeClassifier(max_depth=5),
'Random Forest': RandomForestClassifier(n_estimators=100),
'AdaBoost': AdaBoostClassifier(n_estimators=100),
'Gradient Boosting': GradientBoostingClassifier(n_estimators=100),
}
print("5-Fold Cross-Validation Scores:\n")
for name, model in models.items():
scores = cross_val_score(model, X, y, cv=5)
print(f"{name:22s}: {scores.mean():.4f} (+/- {scores.std():.4f})")
```
***
## Bagging vs Boosting
* Train in parallel
* Reduce variance (overfitting)
* Works with high-variance models
* More robust to outliers
* Harder to overfit
* Train sequentially
* Reduce bias (underfitting)
* Learns from mistakes
* Usually more accurate
* Can overfit if not tuned
***
## Hyperparameter Tuning
```python theme={null}
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
# Define parameter grid
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [5, 10, 15, None],
'min_samples_split': [2, 5, 10],
'max_features': ['sqrt', 'log2']
}
# Grid search
rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(rf, param_grid, cv=5, scoring='accuracy', n_jobs=-1)
grid_search.fit(X_train, y_train)
print("Best parameters:", grid_search.best_params_)
print(f"Best CV score: {grid_search.best_score_:.4f}")
# Use best model
best_model = grid_search.best_estimator_
print(f"Test score: {best_model.score(X_test, y_test):.4f}")
```
***
## Voting Classifier: Mix Different Models
Combine different types of models:
```python theme={null}
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
# Create ensemble of different model types
voting = VotingClassifier(
estimators=[
('lr', LogisticRegression(max_iter=5000)),
('rf', RandomForestClassifier(n_estimators=100)),
('svc', SVC(probability=True))
],
voting='soft' # Use probabilities for voting
)
voting.fit(X_train, y_train)
print(f"Voting Ensemble Accuracy: {voting.score(X_test, y_test):.2%}")
```
***
## Stacking: Models Learn from Models
Train a meta-model on the predictions of base models. The idea: each base model has different strengths and weaknesses. A logistic regression meta-learner can figure out "trust the Random Forest on samples like these, but trust the SVM on samples like those."
```python theme={null}
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
# Define base models -- choose models with DIFFERENT inductive biases.
# If all base models make the same mistakes, stacking won't help.
base_models = [
('rf', RandomForestClassifier(n_estimators=50, random_state=42)), # Tree-based
('gb', GradientBoostingClassifier(n_estimators=50, random_state=42)), # Sequential boosting
('svc', SVC(probability=True, random_state=42)) # Margin-based
]
# The meta-model learns: "given these 3 predictions, what's the best final answer?"
# Use a simple model here to avoid overfitting the stacking layer.
meta_model = LogisticRegression()
# cv=5 means base model predictions are generated via cross-validation
# to prevent data leakage -- the meta-model never sees predictions
# that base models made on their own training data.
stacking = StackingClassifier(
estimators=base_models,
final_estimator=meta_model,
cv=5
)
stacking.fit(X_train, y_train)
print(f"Stacking Accuracy: {stacking.score(X_test, y_test):.2%}")
# Note: Stacking adds complexity and training time. Use it when
# you need that last 0.5-1% improvement (e.g., Kaggle competitions),
# not as your first approach.
```
***
## 🚀 Mini Projects
Build and tune a Random Forest classifier
Gradient Boosting for regression
Ensemble comparison on real dataset
**Project 1: Random Forest Classifier** - Credit card fraud detection
**Objective**: Build a Random Forest to detect fraudulent transactions.
```python theme={null}
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.preprocessing import StandardScaler
# Generate imbalanced fraud data
np.random.seed(42)
n_normal = 9500
n_fraud = 500
# Normal transactions
normal_amount = np.random.exponential(100, n_normal)
normal_time = np.random.uniform(0, 24, n_normal)
normal_frequency = np.random.poisson(3, n_normal)
normal_distance = np.random.exponential(10, n_normal)
# Fraudulent transactions (different patterns)
fraud_amount = np.random.exponential(500, n_fraud) + 200
fraud_time = np.random.choice([2, 3, 4], n_fraud) + np.random.normal(0, 0.5, n_fraud)
fraud_frequency = np.random.poisson(8, n_fraud)
fraud_distance = np.random.exponential(100, n_fraud)
# Combine
X = np.vstack([
np.column_stack([normal_amount, normal_time, normal_frequency, normal_distance]),
np.column_stack([fraud_amount, fraud_time, fraud_frequency, fraud_distance])
])
y = np.array([0]*n_normal + [1]*n_fraud)
feature_names = ['amount', 'hour', 'daily_freq', 'distance_from_home']
print(f"Dataset: {len(y)} transactions, {y.mean():.1%} fraud")
# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
# Train Random Forest
rf = RandomForestClassifier(
n_estimators=100,
max_depth=10,
class_weight='balanced', # Handle imbalance
random_state=42,
n_jobs=-1
)
rf.fit(X_train, y_train)
# Evaluate
y_pred = rf.predict(X_test)
y_prob = rf.predict_proba(X_test)[:, 1]
print("\n=== Classification Report ===")
print(classification_report(y_test, y_pred, target_names=['Normal', 'Fraud']))
print(f"ROC-AUC: {roc_auc_score(y_test, y_prob):.4f}")
# Feature importance
print("\n=== Feature Importance ===")
for name, imp in sorted(zip(feature_names, rf.feature_importances_), key=lambda x: -x[1]):
print(f"{name}: {imp:.3f}")
# Tune n_estimators
print("\n=== Tuning n_estimators ===")
for n_trees in [10, 50, 100, 200, 500]:
rf_temp = RandomForestClassifier(n_estimators=n_trees, max_depth=10,
class_weight='balanced', random_state=42)
cv_scores = cross_val_score(rf_temp, X_train, y_train, cv=5, scoring='roc_auc')
print(f"n={n_trees:3d}: AUC = {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")
```
**Project 2: Gradient Boosting Regressor** - House price prediction
**Objective**: Use XGBoost-style boosting for accurate price predictions.
```python theme={null}
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import mean_squared_error, r2_score
import matplotlib.pyplot as plt
# Generate housing data
np.random.seed(42)
n = 1000
sqft = np.random.normal(2000, 500, n).clip(500, 5000)
bedrooms = np.random.choice([2, 3, 4, 5], n, p=[0.1, 0.4, 0.4, 0.1])
bathrooms = np.random.choice([1, 2, 3], n, p=[0.2, 0.5, 0.3])
age = np.random.exponential(15, n).clip(0, 50)
lot_size = np.random.normal(8000, 2000, n).clip(2000, 20000)
# Non-linear price (boosting should capture this well)
price = (
50000 +
120 * sqft +
20000 * bedrooms +
15000 * bathrooms -
1000 * age -
5 * age**2 +
3 * lot_size +
0.05 * sqft * bedrooms + # Interaction
np.random.normal(0, 20000, n)
)
X = np.column_stack([sqft, bedrooms, bathrooms, age, lot_size])
y = price
feature_names = ['sqft', 'bedrooms', 'bathrooms', 'age', 'lot_size']
# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train Gradient Boosting
gb = GradientBoostingRegressor(
n_estimators=100,
max_depth=4,
learning_rate=0.1,
random_state=42
)
gb.fit(X_train, y_train)
# Evaluate
y_pred = gb.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)
print("=== Model Performance ===")
print(f"RMSE: ${rmse:,.0f}")
print(f"R²: {r2:.4f}")
# Learning curve (staged predictions)
print("\n=== Learning Curve ===")
test_scores = []
for i, y_staged in enumerate(gb.staged_predict(X_test)):
if (i+1) % 20 == 0:
score = r2_score(y_test, y_staged)
test_scores.append((i+1, score))
print(f"n_estimators={i+1:3d}: R² = {score:.4f}")
# Feature importance
print("\n=== Feature Importance ===")
for name, imp in sorted(zip(feature_names, gb.feature_importances_), key=lambda x: -x[1]):
print(f"{name}: {imp:.3f}")
# Hyperparameter tuning
print("\n=== Hyperparameter Search ===")
param_grid = {
'n_estimators': [50, 100],
'max_depth': [3, 5],
'learning_rate': [0.05, 0.1]
}
grid_search = GridSearchCV(
GradientBoostingRegressor(random_state=42),
param_grid,
cv=3,
scoring='neg_mean_squared_error',
n_jobs=-1
)
grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV score (neg MSE): {grid_search.best_score_:.0f}")
```
**Project 3: Ensemble Method Comparison** - Which performs best?
**Objective**: Compare all ensemble methods on the same dataset.
```python theme={null}
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.ensemble import (
RandomForestClassifier,
GradientBoostingClassifier,
AdaBoostClassifier,
BaggingClassifier,
VotingClassifier,
StackingClassifier
)
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
import time
# Create challenging dataset
X, y = make_classification(
n_samples=2000,
n_features=20,
n_informative=10,
n_redundant=5,
n_clusters_per_class=3,
random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Define models
models = {
'Decision Tree': DecisionTreeClassifier(max_depth=10, random_state=42),
'Bagging': BaggingClassifier(n_estimators=50, random_state=42),
'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
'AdaBoost': AdaBoostClassifier(n_estimators=100, random_state=42),
'Gradient Boosting': GradientBoostingClassifier(n_estimators=100, random_state=42),
'Voting (Hard)': VotingClassifier([
('rf', RandomForestClassifier(n_estimators=50)),
('gb', GradientBoostingClassifier(n_estimators=50)),
('lr', LogisticRegression(max_iter=1000))
], voting='hard'),
}
# Compare
print("=== Ensemble Method Comparison ===\n")
print(f"{'Model':<25} {'CV Accuracy':>12} {'Test Acc':>10} {'Time':>8}")
print("-" * 60)
results = []
for name, model in models.items():
start = time.time()
cv_scores = cross_val_score(model, X_train, y_train, cv=5)
model.fit(X_train, y_train)
test_acc = model.score(X_test, y_test)
elapsed = time.time() - start
print(f"{name:<25} {cv_scores.mean():.4f}±{cv_scores.std():.3f} {test_acc:>10.4f} {elapsed:>7.2f}s")
results.append((name, cv_scores.mean(), test_acc))
# Best model
best = max(results, key=lambda x: x[2])
print(f"\n=== Best Model: {best[0]} ===")
print(f"Test Accuracy: {best[2]:.4f}")
# Analysis
print("\n=== Key Insights ===")
print("1. Random Forest and Gradient Boosting typically perform best")
print("2. Voting ensembles can improve stability")
print("3. Single Decision Tree is fastest but least accurate")
print("4. Gradient Boosting often wins on complex datasets")
print("5. Time vs accuracy tradeoff is important in production")
```
***
## Key Takeaways
Many weak models beat one strong model
Train on different data samples
Each model fixes previous mistakes
Best starting point for tabular data
***
## What's Next?
Now that you understand the main ML algorithms, let's learn how to properly evaluate and compare models!
Learn cross-validation, metrics, and how to avoid common mistakes