Skip to main content

Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

Probability Foundations

​
Probability Foundations: How Likely Is This?

​
The Doctor’s Dilemma

Dr. Sarah runs a routine test on a patient. The test comes back positive for a rare disease that affects 1 in 1000 people. The test is 99% accurate:
  • If you HAVE the disease, it correctly says “positive” 99% of the time
  • If you DON’T have it, it correctly says “negative” 99% of the time
Question: What’s the probability this patient actually has the disease? Most people (and many doctors!) say “99%”. The real answer? About 9%. Surprised? This is why understanding probability properly can literally save lives - and it’s the foundation of all machine learning.
Real Talk: Probability is one of the most misunderstood topics. Our intuition is terrible at it. By the end of this module, you’ll understand why our intuition fails and how to calculate correctly.
Estimated Time: 3-4 hours
Difficulty: Beginner
Prerequisites: Basic Python, Module 1 (Describing Data)
What You’ll Build: A spam email detector using Bayes’ theorem

​
What Is Probability?

At its core, probability answers: “How likely is something to happen?” Analogy: Probability is the language your ML model speaks. When a spam classifier says “92% chance this is spam,” it is giving you a probability. When a self-driving car decides to brake, it is reasoning about the probability of a pedestrian stepping into the road. Every prediction an ML model makes is, at its foundation, a probability statement.

​
The Coin Flip

import random

def flip_coin():
    return random.choice(['Heads', 'Tails'])

# Flip 1000 times
results = [flip_coin() for _ in range(1000)]
heads_count = results.count('Heads')

print(f"Heads: {heads_count}/1000 = {heads_count/1000:.2%}")
Output (varies): 
Heads: 497/1000 = 49.70%
The probability of heads is 0.5 (or 50%). This means:
  • Over many flips, about half will be heads
  • Any single flip is unpredictable
  • Probability is about long-run frequency
Probability Formula Visualization
The Formula: P(A)=Number of ways A can happenTotal number of possible outcomesP(A) = \frac{\text{Number of ways A can happen}}{\text{Total number of possible outcomes}}

​
Basic Probability Rules

​
Rule 1: Probability is Between 0 and 1

0≤P(A)≤10 \leq P(A) \leq 1
  • P(A) = 0: Impossible (rolling a 7 on a standard die)
  • P(A) = 1: Certain (rolling 1-6 on a standard die)
  • P(A) = 0.5: Equally likely to happen or not
# Die roll probabilities
die_outcomes = [1, 2, 3, 4, 5, 6]

p_roll_7 = 0 / 6  # Impossible
p_roll_any = 6 / 6  # Certain
p_roll_even = 3 / 6  # 2, 4, or 6

print(f"P(roll 7): {p_roll_7}")
print(f"P(roll 1-6): {p_roll_any}")
print(f"P(roll even): {p_roll_even}")

​
Rule 2: Complement Rule

The probability something doesn’t happen is 1 minus the probability it does. P(not A)=1−P(A)P(\text{not } A) = 1 - P(A)
Complement Rule - Rain Example
# Weather example
p_rain = 0.30  # 30% chance of rain

p_no_rain = 1 - p_rain
print(f"P(no rain): {p_no_rain:.0%}")  # 70%

# Useful for "at least one" problems
p_single_die_not_six = 5/6
p_three_dice_at_least_one_six = 1 - (5/6)**3
print(f"P(at least one 6 in 3 rolls): {p_three_dice_at_least_one_six:.2%}")

​
Rule 3: Addition Rule

For mutually exclusive events (can’t happen together): P(A or B)=P(A)+P(B)P(A \text{ or } B) = P(A) + P(B) 
# Drawing a card
p_ace = 4/52
p_king = 4/52

# P(ace OR king) - can't be both!
p_ace_or_king = p_ace + p_king
print(f"P(ace or king): {p_ace_or_king:.2%}")  # 15.38%
For non-mutually exclusive events (can happen together): P(A or B)=P(A)+P(B)−P(A and B)P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B) 
# P(ace OR spade)
p_ace = 4/52
p_spade = 13/52
p_ace_and_spade = 1/52  # Ace of spades!

p_ace_or_spade = p_ace + p_spade - p_ace_and_spade
print(f"P(ace or spade): {p_ace_or_spade:.2%}")  # 30.77%

​
Rule 4: Multiplication Rule for Independent Events

Events are independent if one doesn’t affect the other. P(A and B)=P(A)×P(B)P(A \text{ and } B) = P(A) \times P(B) 
# Flipping a coin and rolling a die
p_heads = 1/2
p_six = 1/6

p_heads_and_six = p_heads * p_six
print(f"P(heads AND six): {p_heads_and_six:.2%}")  # 8.33%

# Password cracking: 4-digit PIN
digits = 10  # 0-9
combinations = digits ** 4  # 10,000
p_guess_correct = 1 / combinations
print(f"P(guess 4-digit PIN): {p_guess_correct:.4%}")  # 0.01%

​
Conditional Probability: The Game Changer

Here’s where it gets interesting. Conditional probability answers: “What’s the probability of A, GIVEN that B already happened?” Notation: P(A∣B)P(A|B) reads as “probability of A given B”

​
The Job Interview Example

You’re applying for jobs. Here’s some data: 
# 100 applicants total
data = {
    'has_experience': {'hired': 30, 'not_hired': 20},  # 50 total
    'no_experience': {'hired': 10, 'not_hired': 40}    # 50 total
}

# Overall probability of getting hired
total_hired = 30 + 10
total_applicants = 100
p_hired = total_hired / total_applicants
print(f"P(hired): {p_hired:.0%}")  # 40%

# BUT... what if you have experience?
experienced_hired = 30
total_experienced = 50
p_hired_given_experience = experienced_hired / total_experienced
print(f"P(hired | experience): {p_hired_given_experience:.0%}")  # 60%

# What if you don't have experience?
no_exp_hired = 10
total_no_exp = 50
p_hired_given_no_experience = no_exp_hired / total_no_exp
print(f"P(hired | no experience): {p_hired_given_no_experience:.0%}")  # 20%
Key Insight: The “given” information changes everything! Analogy: Conditional probability is like updating your GPS after a wrong turn. The probability of arriving on time given you are on the highway is very different from the probability given you are stuck in a detour. The “given” is the context that reshapes all your estimates.
ML Application — Feature Importance: Every feature in a classification model is essentially providing conditional information. When your model uses “number of support tickets” to predict churn, it is computing something like P(churn | high support tickets). Understanding conditional probability helps you reason about why certain features are predictive and others are not — and helps you spot cases where your model is learning spurious correlations.
The Formula: P(A∣B)=P(A and B)P(B)P(A|B) = \frac{P(A \text{ and } B)}{P(B)} 
# Verify with formula
p_hired_and_experienced = 30 / 100  # 0.30
p_experienced = 50 / 100  # 0.50

p_hired_given_exp_formula = p_hired_and_experienced / p_experienced
print(f"P(hired | exp) via formula: {p_hired_given_exp_formula:.0%}")  # 60% ✓

​
Bayes’ Theorem: The Most Important Formula

Now we can solve that medical test problem from the beginning. Bayes’ Theorem lets you flip conditional probabilities: P(A∣B)=P(B∣A)⋅P(A)P(B)P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)} In words: The probability of A given B equals the probability of B given A, times the probability of A, divided by the probability of B.

​
Solving the Medical Test Problem

Let’s set up what we know:
  • P(Disease) = 0.001 (1 in 1000 people have it)
  • P(Positive | Disease) = 0.99 (test catches 99% of sick people)
  • P(Positive | No Disease) = 0.01 (1% false positive rate)
We want: P(Disease | Positive) - given a positive test, what’s the chance they’re actually sick? 
# Known probabilities
p_disease = 0.001  # Prior: base rate of disease
p_positive_given_disease = 0.99  # Sensitivity
p_positive_given_no_disease = 0.01  # False positive rate

# Calculate P(Positive) using law of total probability
p_no_disease = 1 - p_disease
p_positive = (p_positive_given_disease * p_disease + 
              p_positive_given_no_disease * p_no_disease)

print(f"P(Positive test): {p_positive:.4f}")  # 0.011

# Apply Bayes' theorem
p_disease_given_positive = (p_positive_given_disease * p_disease) / p_positive

print(f"P(Disease | Positive): {p_disease_given_positive:.2%}")
Output: 
P(Positive test): 0.0109
P(Disease | Positive): 9.02%
Only 9% chance of actually having the disease, despite a “99% accurate” test!
Bayes Theorem - Medical Test Visualization

​
Why Is This So Low?

Let’s think about 10,000 people: 
population = 10000

# How many actually have disease?
actually_sick = int(population * 0.001)  # 10 people
actually_healthy = population - actually_sick  # 9990 people

# Of the sick, how many test positive?
sick_positive = int(actually_sick * 0.99)  # ~10 true positives

# Of the healthy, how many test positive (false positives)?
healthy_positive = int(actually_healthy * 0.01)  # ~100 false positives!

# Total positives
total_positive = sick_positive + healthy_positive

print(f"Actually sick: {actually_sick}")
print(f"True positives: {sick_positive}")
print(f"False positives: {healthy_positive}")
print(f"Total positives: {total_positive}")
print(f"P(sick | positive): {sick_positive/total_positive:.1%}")
Output: 
Actually sick: 10
True positives: 10
False positives: 100
Total positives: 110
P(sick | positive): 9.1%
The Key Insight: When the disease is rare, even a small false positive rate creates many false alarms that overwhelm the true cases! Step-by-step reasoning: Here is how to think through Bayes problems without getting lost in formulas:
  1. Start with a concrete population — pick 10,000 people (or 100,000, whatever makes the math clean).
  2. Split by the base rate — how many actually have the condition? At 0.1%, that is 10 out of 10,000.
  3. Apply the test to both groups — of the 10 sick people, 99% test positive = ~10 true positives. Of the 9,990 healthy people, 1% test positive = ~100 false positives.
  4. Look at the positive pile — 10 true positives out of 110 total positives = ~9%.
This “frequency table” approach makes Bayes problems intuitive. No formula needed.
Statistical Mistake in ML — Ignoring Class Imbalance: This exact same base rate problem happens in ML classification. If only 1% of transactions are fraudulent, a model that predicts “not fraud” for everything gets 99% accuracy. The Bayesian insight applies: you must look at precision (of all things you flagged as fraud, how many actually were?) and recall (of all actual frauds, how many did you catch?). Accuracy alone is meaningless with imbalanced classes.

​
Bayes’ Theorem in Machine Learning

This isn’t just medical trivia. Bayes’ theorem is the foundation of:
  • Spam filters (Naive Bayes classifier)
  • Recommendation systems
  • Medical diagnosis AI
  • Text classification
  • Bayesian neural networks
Let’s build a spam detector!

​
🚀 Mini-Project: Spam Email Detector

​
The Problem

You receive an email with the word “FREE” in it. What’s the probability it’s spam?

​
The Data

# Training data: 1000 emails
total_emails = 1000
spam_emails = 400
ham_emails = 600  # "ham" = not spam

# Word frequency
emails_with_free = {
    'spam': 200,  # 200 spam emails contain "FREE"
    'ham': 30     # 30 legitimate emails contain "FREE"
}

emails_with_meeting = {
    'spam': 10,
    'ham': 250
}

emails_with_urgent = {
    'spam': 180,
    'ham': 40
}

​
Step 1: Calculate Prior Probabilities

# P(Spam) and P(Ham)
p_spam = spam_emails / total_emails  # 0.40
p_ham = ham_emails / total_emails    # 0.60

print(f"P(Spam): {p_spam:.0%}")
print(f"P(Ham): {p_ham:.0%}")

​
Step 2: Calculate Likelihoods

# P(word | Spam) - probability of word appearing in spam
p_free_given_spam = emails_with_free['spam'] / spam_emails
p_free_given_ham = emails_with_free['ham'] / ham_emails

print(f"P('FREE' | Spam): {p_free_given_spam:.0%}")  # 50%
print(f"P('FREE' | Ham): {p_free_given_ham:.0%}")    # 5%

p_meeting_given_spam = emails_with_meeting['spam'] / spam_emails
p_meeting_given_ham = emails_with_meeting['ham'] / ham_emails

print(f"P('meeting' | Spam): {p_meeting_given_spam:.1%}")  # 2.5%
print(f"P('meeting' | Ham): {p_meeting_given_ham:.1%}")    # 41.7%

​
Step 3: Apply Bayes’ Theorem

def naive_bayes_spam(word, p_word_spam, p_word_ham, p_spam=0.4, p_ham=0.6):
    """
    Calculate P(Spam | word) using Bayes' theorem.
    """
    # P(word) = P(word|Spam)*P(Spam) + P(word|Ham)*P(Ham)
    p_word = p_word_spam * p_spam + p_word_ham * p_ham
    
    # Bayes' theorem
    p_spam_given_word = (p_word_spam * p_spam) / p_word
    
    return p_spam_given_word

# Email contains "FREE"
p_spam_free = naive_bayes_spam("FREE", 0.50, 0.05)
print(f"P(Spam | 'FREE'): {p_spam_free:.0%}")  # 87%

# Email contains "meeting"
p_spam_meeting = naive_bayes_spam("meeting", 0.025, 0.417)
print(f"P(Spam | 'meeting'): {p_spam_meeting:.0%}")  # 4%

​
Step 4: Multiple Words (Naive Assumption)

The “naive” in Naive Bayes assumes words are independent: 
def naive_bayes_multi_word(words_data, p_spam=0.4, p_ham=0.6):
    """
    Calculate P(Spam | multiple words) assuming independence.
    
    words_data: list of (word, p_word_spam, p_word_ham)
    """
    # Start with prior
    log_spam = np.log(p_spam)
    log_ham = np.log(p_ham)
    
    # Multiply likelihoods (add in log space)
    for word, p_word_spam, p_word_ham in words_data:
        log_spam += np.log(p_word_spam + 1e-10)  # Add small value to avoid log(0)
        log_ham += np.log(p_word_ham + 1e-10)
    
    # Convert back and normalize
    spam_score = np.exp(log_spam)
    ham_score = np.exp(log_ham)
    
    p_spam_given_words = spam_score / (spam_score + ham_score)
    return p_spam_given_words

# Email: "FREE meeting URGENT"
words = [
    ("FREE", 0.50, 0.05),
    ("meeting", 0.025, 0.417),
    ("URGENT", 0.45, 0.067)
]

p_spam_multi = naive_bayes_multi_word(words)
print(f"P(Spam | 'FREE meeting URGENT'): {p_spam_multi:.0%}")
Output: 
P(Spam | 'FREE meeting URGENT'): 71%
Even though “meeting” is a ham indicator, the strong spam signals from “FREE” and “URGENT” push the probability up!

​
Complete Spam Classifier

import numpy as np
from collections import defaultdict

class NaiveBayesSpamFilter:
    def __init__(self):
        self.word_counts = {'spam': defaultdict(int), 'ham': defaultdict(int)}
        self.class_counts = {'spam': 0, 'ham': 0}
        self.vocabulary = set()
    
    def train(self, emails, labels):
        """Train on labeled email data."""
        for email, label in zip(emails, labels):
            self.class_counts[label] += 1
            words = email.lower().split()
            for word in words:
                self.word_counts[label][word] += 1
                self.vocabulary.add(word)
    
    def predict_proba(self, email):
        """Calculate P(Spam | email)."""
        words = email.lower().split()
        
        # Prior probabilities
        total = sum(self.class_counts.values())
        p_spam = self.class_counts['spam'] / total
        p_ham = self.class_counts['ham'] / total
        
        # Calculate log likelihoods
        log_spam = np.log(p_spam)
        log_ham = np.log(p_ham)
        
        vocab_size = len(self.vocabulary)
        spam_total = sum(self.word_counts['spam'].values())
        ham_total = sum(self.word_counts['ham'].values())
        
        for word in words:
            # Laplace smoothing
            p_word_spam = (self.word_counts['spam'][word] + 1) / (spam_total + vocab_size)
            p_word_ham = (self.word_counts['ham'][word] + 1) / (ham_total + vocab_size)
            
            log_spam += np.log(p_word_spam)
            log_ham += np.log(p_word_ham)
        
        # Convert to probabilities
        spam_score = np.exp(log_spam)
        ham_score = np.exp(log_ham)
        
        return spam_score / (spam_score + ham_score)
    
    def predict(self, email, threshold=0.5):
        """Classify as spam or ham."""
        return 'spam' if self.predict_proba(email) > threshold else 'ham'

# Example usage
spam_filter = NaiveBayesSpamFilter()

# Training data
train_emails = [
    "FREE money click here now",
    "Congratulations you won FREE prize",
    "URGENT action required FREE offer",
    "Meeting tomorrow at 3pm",
    "Project deadline reminder",
    "Team lunch on Friday",
    "Your invoice attached",
    "FREE trial subscription ending",
]
train_labels = ['spam', 'spam', 'spam', 'ham', 'ham', 'ham', 'ham', 'spam']

spam_filter.train(train_emails, train_labels)

# Test
test_emails = [
    "FREE discount on all products",
    "Meeting rescheduled to Monday",
    "URGENT free money opportunity",
]

for email in test_emails:
    prob = spam_filter.predict_proba(email)
    pred = spam_filter.predict(email)
    print(f"'{email[:40]}...'")
    print(f"  P(Spam): {prob:.1%} → {pred.upper()}\n")

​
🎯 Practice Exercises

​
Exercise 1: Card Probability

# Standard 52-card deck
# Calculate:
# 1. P(drawing a heart)
# 2. P(drawing a face card) - J, Q, K
# 3. P(drawing a red face card)
# 4. P(drawing a heart OR a face card)

# 1. P(heart)
p_heart = 13/52  # 25%

# 2. P(face card) - 12 face cards (3 per suit × 4 suits)
p_face = 12/52  # 23.08%

# 3. P(red face card) - 6 cards (3 per red suit × 2 red suits)
p_red_face = 6/52  # 11.54%

# 4. P(heart OR face card)
# Hearts = 13, Face cards = 12, but 3 are both (J, Q, K of hearts)
p_heart_or_face = (13 + 12 - 3) / 52  # 22/52 = 42.31%

print(f"P(heart): {p_heart:.2%}")
print(f"P(face): {p_face:.2%}")
print(f"P(red face): {p_red_face:.2%}")
print(f"P(heart OR face): {p_heart_or_face:.2%}")

​
Exercise 2: Weather Prediction

# Historical data:
# - 30% of days are rainy
# - On rainy days, 80% are cloudy in the morning
# - On non-rainy days, 40% are cloudy in the morning

# This morning is cloudy. What's the probability of rain?

# Use Bayes' theorem
p_rain = 0.30
p_no_rain = 0.70
p_cloudy_given_rain = 0.80
p_cloudy_given_no_rain = 0.40

# P(Cloudy)
p_cloudy = p_cloudy_given_rain * p_rain + p_cloudy_given_no_rain * p_no_rain
print(f"P(Cloudy): {p_cloudy:.0%}")  # 52%

# P(Rain | Cloudy)
p_rain_given_cloudy = (p_cloudy_given_rain * p_rain) / p_cloudy
print(f"P(Rain | Cloudy): {p_rain_given_cloudy:.1%}")  # 46.2%

# Even though it's cloudy, there's still less than 50% chance of rain
# because the base rate of rain (30%) is low

​
Exercise 3: Two-Test Diagnosis

# A disease affects 2% of the population
# Test A: 95% sensitivity, 90% specificity
# Test B: 90% sensitivity, 95% specificity

# A patient tests positive on BOTH tests.
# What's the probability they have the disease?

# Hint: Apply Bayes' theorem twice (sequentially)

# Initial state
p_disease = 0.02

# Test A
sensitivity_a = 0.95  # P(+|disease)
specificity_a = 0.90  # P(-|no disease)
false_pos_a = 1 - specificity_a  # 0.10

# After Test A positive
p_pos_a = sensitivity_a * p_disease + false_pos_a * (1 - p_disease)
p_disease_after_a = (sensitivity_a * p_disease) / p_pos_a
print(f"After Test A positive: P(disease) = {p_disease_after_a:.1%}")  # 16.2%

# Test B (now using updated probability as prior)
sensitivity_b = 0.90
specificity_b = 0.95
false_pos_b = 0.05

# After Test B positive (using p_disease_after_a as new prior)
p_pos_b_given_disease = sensitivity_b
p_pos_b_given_no_disease = false_pos_b

p_pos_b = (p_pos_b_given_disease * p_disease_after_a + 
           p_pos_b_given_no_disease * (1 - p_disease_after_a))

p_disease_after_both = (p_pos_b_given_disease * p_disease_after_a) / p_pos_b
print(f"After both tests positive: P(disease) = {p_disease_after_both:.1%}")  # 77.6%

# Two positive tests dramatically increases confidence!

​
Key Takeaways

Basic Rules

  • Probability is between 0 and 1
  • P(not A) = 1 - P(A)
  • For exclusive events: P(A or B) = P(A) + P(B)
  • For independent events: P(A and B) = P(A) × P(B)

Conditional Probability

  • P(A|B) = P(A and B) / P(B)
  • “Given B” means we’re only looking at cases where B happened
  • This changes everything!

Bayes' Theorem

  • Lets you flip P(A|B) to P(B|A)
  • Prior × Likelihood = Posterior (after normalizing)
  • Foundation of spam filters, medical diagnosis, ML

Key Insight

  • Base rates matter enormously
  • “99% accurate” doesn’t mean what you think
  • Always consider: what’s the prior probability?

​
Common Mistakes to Avoid

Mistake 1: Ignoring Base Rates (Base Rate Fallacy)A 99% accurate test for a rare disease (1 in 10,000) will mostly produce false positives. Most people who test positive won’t have the disease. Always consider how common the thing you’re testing for actually is.
Mistake 2: Confusing P(A|B) with P(B|A)P(sick|positive test) is NOT the same as P(positive test|sick). This confusion has led to wrongful convictions and medical misdiagnoses. Bayes’ theorem is the bridge between them.
Mistake 3: Treating Dependent Events as IndependentDrawing cards without replacement means probabilities change. P(2nd card is ace | 1st was ace) = 3/51, not 4/52.

​
Interview Questions

Question: A family has two children. Given that at least one is a boy, what’s the probability that both are boys?
Answer: 1/3, not 1/2!Possible outcomes for two children: BB, BG, GB, GG Given “at least one boy”: BB, BG, GB (3 options) Both boys: BB (1 option)P(both boys | at least one boy) = 1/3This is counterintuitive because we’re not told which child is the boy, so both orders (BG, GB) are possible.
Question: In a room of 23 people, what’s the probability that at least two share a birthday?
Answer: About 50%!It’s easier to calculate the complement: P(no shared birthdays) = (365/365) × (364/365) × (363/365) × … × (343/365)
import numpy as np
p_no_match = np.prod([(365-i)/365 for i in range(23)])
p_at_least_one_match = 1 - p_no_match
print(f"P(shared birthday): {p_at_least_one_match:.1%}")  # 50.7%
This is famously counterintuitive because we’re comparing ALL pairs, not just pairs involving you.
Question: You’re on a game show with 3 doors. Behind one is a car, behind the others are goats. You pick door 1. The host (who knows what’s behind each door) opens door 3, revealing a goat. Should you switch to door 2?
Answer: Yes! Switching gives you 2/3 chance of winning.Initial pick: 1/3 chance of being right Switching: 2/3 chance of winningThe key insight: The host’s action gives you information. He always reveals a goat, so when you switch, you’re essentially betting that your initial choice was wrong (which it probably was, 2/3 of the time).
Question: Your spam filter has 98% sensitivity and 95% specificity. If 5% of emails are spam, what fraction of emails flagged as spam are actually spam?
Answer: About 51%
p_spam = 0.05
sensitivity = 0.98  # P(flagged|spam)
specificity = 0.95  # P(not_flagged|not_spam)
false_positive_rate = 0.05

# P(flagged)
p_flagged = sensitivity * p_spam + false_positive_rate * (1 - p_spam)
# = 0.98 × 0.05 + 0.05 × 0.95 = 0.0965

# P(spam|flagged)
p_spam_given_flagged = (sensitivity * p_spam) / p_flagged
# = 0.049 / 0.0965 = 0.508 or about 51%

print(f"Precision: {p_spam_given_flagged:.1%}")
Half of flagged emails are false positives! This is why spam filters need extremely high specificity.

​
Practice Challenge

Build a simple sentiment classifier using Bayes’ theorem:
import numpy as np
from collections import defaultdict

# Training data
reviews = [
    ("great product love it", "positive"),
    ("terrible waste of money", "negative"),
    ("amazing quality highly recommend", "positive"),
    ("awful experience never again", "negative"),
    ("fantastic works perfectly", "positive"),
    ("horrible customer service", "negative"),
    ("excellent value great buy", "positive"),
    ("disappointing poor quality", "negative"),
]

# Your task: Implement a Naive Bayes classifier
# 1. Count word frequencies in positive vs negative reviews
# 2. Calculate P(word|positive) and P(word|negative) for each word
# 3. Use Bayes to classify new review: "great quality but poor service"

class NaiveBayesClassifier:
    def __init__(self):
        self.word_counts = defaultdict(lambda: defaultdict(int))
        self.class_counts = defaultdict(int)
        self.vocab = set()
    
    def train(self, reviews):
        # Your implementation here
        pass
    
    def predict(self, text):
        # Your implementation here
        pass

# Test your classifier
classifier = NaiveBayesClassifier()
classifier.train(reviews)
print(classifier.predict("great quality but poor service"))
Solution:
class NaiveBayesClassifier:
    def __init__(self):
        self.word_counts = defaultdict(lambda: defaultdict(int))
        self.class_counts = defaultdict(int)
        self.vocab = set()
    
    def train(self, reviews):
        for text, label in reviews:
            self.class_counts[label] += 1
            for word in text.lower().split():
                self.word_counts[label][word] += 1
                self.vocab.add(word)
    
    def predict(self, text):
        words = text.lower().split()
        total_reviews = sum(self.class_counts.values())
        
        scores = {}
        for label in self.class_counts:
            # Start with prior P(class)
            log_prob = np.log(self.class_counts[label] / total_reviews)
            
            # Add log P(word|class) for each word
            total_words = sum(self.word_counts[label].values())
            for word in words:
                # Laplace smoothing
                count = self.word_counts[label].get(word, 0) + 1
                prob = count / (total_words + len(self.vocab))
                log_prob += np.log(prob)
            
            scores[label] = log_prob
        
        return max(scores, key=scores.get)

# Test
classifier = NaiveBayesClassifier()
classifier.train(reviews)
result = classifier.predict("great quality but poor service")
print(f"Prediction: {result}")  # Could go either way!

​
📝 Practice Exercises

Exercise 1

Calculate probabilities for card games

Exercise 2

Apply Bayes’ theorem to medical diagnosis

Exercise 3

Build a spam classifier using Naive Bayes

Exercise 4

Real-world: Customer conversion funnel analysis

​
Connection to Machine Learning

Probability ConceptML Application
Prior probabilityInitial model beliefs
LikelihoodHow well data fits model
PosteriorUpdated beliefs after seeing data
Bayes’ theoremNaive Bayes classifier, Bayesian models
Conditional probabilityClassification, prediction

​
Interview Prep: Common Questions

Q: Two cards are drawn from a deck without replacement. What’s P(both are aces)?
P(1st ace) = 4/52. P(2nd ace | 1st ace) = 3/51. Total = (4/52) × (3/51) = 12/2652 ≈ 0.45%
Q: A test is 95% accurate for a disease affecting 1% of population. If positive, what’s P(have disease)?
Use Bayes: P(D|+) = P(+|D)×P(D) / [P(+|D)×P(D) + P(+|¬D)×P(¬D)] = (0.95×0.01) / (0.95×0.01 + 0.05×0.99) ≈ 16%
Q: What’s the difference between independent and mutually exclusive events?
Independent: Occurrence of one doesn’t affect the other (e.g., two coin flips). Mutually exclusive: Both cannot occur simultaneously (e.g., getting heads AND tails on one flip). Note: Mutually exclusive events are NOT independent!
Q: You flip a fair coin 10 times and get 10 heads. What’s P(heads on flip 11)?
Still 50%! This is the gambler’s fallacy. Each flip is independent; past results don’t affect future outcomes.

​
Common Pitfalls

Probability Mistakes to Avoid:
  1. Base Rate Neglect - Forgetting prior probabilities when interpreting test results
  2. Confusing P(A|B) with P(B|A) - P(positive|disease) ≠ P(disease|positive)
  3. Gambler’s Fallacy - Believing past independent events affect future outcomes
  4. Assuming Independence - Many real-world events are correlated; check before multiplying probabilities
  5. Ignoring Complement - Sometimes P(at least one) = 1 - P(none) is easier

​
Key Takeaways

What You Learned:
  • ✅ Basic Probability - P(A) = favorable outcomes / total outcomes
  • ✅ Addition Rule - P(A or B) = P(A) + P(B) - P(A and B)
  • ✅ Multiplication Rule - P(A and B) = P(A) × P(B|A)
  • ✅ Conditional Probability - P(A|B) = P(A and B) / P(B)
  • ✅ Bayes’ Theorem - Update beliefs with new evidence; essential for ML
  • ✅ Independence - Events where one doesn’t affect the other
Bayes’ Theorem Intuition: Think of it as “updating your beliefs.” You start with a prior (what you knew before), see evidence, and get a posterior (what you know after). This is exactly how many ML algorithms learn!
Coming up next: We’ll learn about probability distributions - the patterns that randomness follows. This is where we discover the famous “bell curve” and understand why it shows up everywhere!

Next: Probability Distributions

Discover the patterns hidden in randomness

​
Interview Deep-Dive

Strong Answer:
  • This is a Bayes’ theorem problem, and the answer shocks most people. Let me work through it with a natural frequency approach. Imagine 1,000,000 people. Of those, 100 actually have the disease (1 in 10,000). Of those 100, the test correctly identifies 99 (99% sensitivity). Of the 999,900 healthy people, the test incorrectly flags 1% as positive, giving 9,999 false positives.
  • Total positive tests: 99 true positives plus 9,999 false positives equals 10,098. So the probability of actually having the disease given a positive test is 99 / 10,098, which is about 0.98% — less than 1%.
  • The intuition: when the disease is very rare, even a highly accurate test produces far more false positives than true positives because the healthy population is so much larger. This is the base rate fallacy in action.
  • This has direct real-world consequences. During COVID, rapid antigen tests had different positive predictive values depending on community prevalence. In a low-prevalence area, a positive test was much less reliable than in a high-prevalence area — same test, same accuracy, but radically different interpretation because of the base rate.
Follow-up: How does this base rate problem apply to fraud detection in machine learning?It is the exact same mathematics. If fraud occurs in 0.1% of transactions, even a model with 99% precision and 99% recall will generate far more false alarms than true catches in absolute terms. For every 1 million transactions, you would have roughly 990 true fraud catches but also about 9,990 false alarms. Your fraud investigators get swamped with false positives. The practical solution is to either increase specificity dramatically (accepting some missed fraud), use a two-stage system where a high-recall model generates candidates and a high-precision model filters them, or rank alerts by probability score and only investigate the top-N. The base rate is the single most important number in any classification system — it determines whether your precision is useful or not.
Strong Answer:
  • Independent events are those where the occurrence of one does not change the probability of the other. Formally, P(A and B) = P(A) x P(B). Example: flipping a coin and rolling a die. The coin result has no effect on the die.
  • Mutually exclusive events are those that cannot occur simultaneously. P(A and B) = 0. Example: rolling a 3 and rolling a 5 on the same die throw.
  • Here is the critical point that trips people up: mutually exclusive events are actually dependent, not independent. If I tell you that event A happened (you rolled a 3), then you know with certainty that event B did not happen (you did not roll a 5). That means P(B given A) = 0, which is different from P(B) = 1/6. Since P(B given A) does not equal P(B), the events are dependent.
  • The only exception is when one of the events has probability zero. In every other case, mutually exclusive implies dependent. This is a common interview gotcha question precisely because the two concepts sound like they should be related but they are actually in tension.
Follow-up: Give me a real-world example where confusing these two concepts would lead to a wrong answer in a data science context.Consider customer segmentation. Suppose “clicked ad” and “made purchase” are not mutually exclusive (a customer can do both). If you incorrectly treat them as mutually exclusive and compute P(click or purchase) = P(click) + P(purchase), you will overcount customers who did both and overestimate your reach. On the other hand, if you incorrectly assume click and purchase are independent and compute P(click and purchase) = P(click) x P(purchase), you might underestimate the joint probability because people who click are much more likely to purchase (they are positively dependent). The correct approach uses P(A or B) = P(A) + P(B) - P(A and B), and the correct joint probability requires knowing P(purchase given click), not just multiplying the marginals.
Strong Answer:
  • The way to evaluate this is to ask: “If the coin truly were fair, how likely is it to see a result this extreme or more extreme?” That is the definition of a p-value.
  • For a fair coin, the number of heads in 20 flips follows a Binomial(n=20, p=0.5) distribution. P(X >= 15) is the sum of P(X=15) + P(X=16) + … + P(X=20). Computing this gives approximately 2.1%. Since we should also consider the other tail (5 or fewer heads would be equally surprising), the two-tailed p-value is about 4.1%.
  • At a standard alpha of 0.05, this is borderline significant. We would reject the null hypothesis that the coin is fair — but barely. At alpha = 0.01, we would not reject. This tells us that 15 out of 20 is unusual but not overwhelmingly so for a fair coin.
  • The key insight: a single run of 20 flips is not very powerful for detecting bias. If the true probability of heads were 0.6, we would need roughly 200 flips to detect that with 80% power. Twenty flips can only reliably detect large biases (like p = 0.8). This is why sample size planning matters before making claims.
Follow-up: If you wanted to be 95% confident in detecting a coin that is 60% biased toward heads, how many flips would you need?This is a power analysis for a one-proportion z-test. The null is p = 0.5, the alternative is p = 0.6, and we want 80% power at alpha = 0.05. Using the formula n = ((z_alpha + z_beta)^2 x p_alt x (1 - p_alt)) / (p_alt - p_null)^2, we get approximately 194 flips. The intuition is that a 10 percentage point shift on a binary outcome requires roughly 200 observations to detect reliably. This scales quadratically: detecting a 5% shift (p=0.55) would require roughly 780 flips. Small effect sizes demand large samples — one of the most important lessons in all of statistics.
Strong Answer:
  • Naive Bayes uses Bayes’ theorem to compute P(class given features) by flipping it to P(features given class) x P(class). The “naive” part is the assumption that all features are conditionally independent given the class. That means P(word1 and word2 given spam) = P(word1 given spam) x P(word2 given spam). In reality, words are correlated — “Nigerian” and “prince” tend to appear together in spam — so the assumption is violated.
  • Despite this, Naive Bayes works surprisingly well in practice for several reasons. First, classification only requires getting the rank ordering of class probabilities correct, not the actual probability values. Even with wrong probability estimates, the argmax (most likely class) is often correct. Second, the independence assumption causes errors that tend to cancel out across many features. Third, with limited training data, Naive Bayes has far fewer parameters to estimate than a model that captures all pairwise dependencies, so it suffers less from overfitting.
  • It excels in text classification (spam filtering, sentiment analysis) where you have high-dimensional sparse features and moderate amounts of labeled data. It is fast to train (single pass through data), fast to predict, and handles missing features naturally.
  • Where it fails: when feature dependencies actually matter for classification. For example, XOR-type problems where the class depends on the interaction of two features (neither feature alone is predictive). In those cases, you need a model that captures interactions, like logistic regression with interaction terms, or a tree-based model.
Follow-up: In a production spam filter, would you use Naive Bayes or a deep learning model, and what factors drive that decision?It depends on the constraints. For an email provider processing billions of messages, Naive Bayes is still the first-stage filter because it is incredibly fast (microseconds per email), uses minimal memory, and can be updated incrementally as new spam patterns emerge. Its simplicity means the system is interpretable and debuggable — you can see exactly which words drove the classification. A deep learning model gives better accuracy (maybe 99.5% versus 98.5%) but requires GPU inference, has higher latency, and is harder to debug when it misclassifies. In practice, most production systems use a cascaded approach: a fast Naive Bayes or logistic regression model as the first filter, then a more expensive neural model for borderline cases. You optimize the expensive model for precision in the ambiguous middle zone where the simple model is uncertain.