> ## Documentation Index
> Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Python Interview Questions
> Complete preparation guide covering Python basics, data structures, OOP, advanced concepts, concurrency, web development, and data science -- with CPython internals and production-level trade-off analysis
Python interviews span an unusually wide range -- from basic syntax and data structure manipulation to CPython internals, concurrency models, and framework-specific architecture decisions. What makes Python interviews distinctive is that the language's simplicity on the surface hides significant depth underneath. Interviewers use this gap to distinguish candidates who have genuine understanding from those who have only written scripts. The GIL, descriptor protocol, metaclasses, and memory management model are the topics where strong candidates separate themselves.
The questions below are organized from fundamentals through advanced topics. For each question, the provided answer goes beyond the textbook definition to include the "why," the trade-offs, and the production context that interviewers want to hear. Practice by explaining each concept out loud before reading -- your ability to articulate clearly matters as much as knowing the right answer.
## 1. Python Basics
Python is a high-level, interpreted, general-purpose programming language created by Guido van Rossum (first released 1991). But the real answer an interviewer wants goes beyond the textbook definition.
**Key features that matter in production:**
* **Interpreted with bytecode compilation**: Python is not purely interpreted. CPython compiles source to `.pyc` bytecode first, then the VM interprets that bytecode. This is why you see `__pycache__` directories. Understanding this distinction shows you know what actually happens when you run `python script.py`.
* **Dynamic typing with strong typing**: Python is dynamically typed (variables do not declare types) but *strongly* typed (you cannot add a string to an integer without explicit conversion). Many candidates confuse "dynamic" with "weak" typing -- JavaScript is weakly typed, Python is not.
* **First-class functions**: Functions are objects. You can assign them to variables, pass them as arguments, return them from other functions. This enables decorators, closures, and functional patterns that are central to idiomatic Python.
* **Extensive standard library ("batteries included")**: `collections`, `itertools`, `functools`, `pathlib`, `dataclasses`, `typing` -- knowing these separates a Python developer from someone who just writes Python syntax.
* **Memory management via reference counting + cyclic GC**: Automatic memory management, but understanding it matters for long-running services where memory leaks from circular references can crash your app at 3 AM.
* **GIL (Global Interpreter Lock)**: The elephant in the room. One thread executes Python bytecode at a time in CPython. This has massive implications for concurrency design.
**What interviewers are really testing:** Whether you understand Python beyond "it is easy to learn." A senior candidate connects features to engineering trade-offs (e.g., dynamic typing speeds development but requires more testing discipline).
**Red flag answer:** "Python is easy to learn and has lots of libraries." This is a junior answer that shows no depth. Another red flag: not knowing the difference between dynamic and weak typing.
**War Story:** At Instagram (one of the largest Python deployments in the world), the team hit GIL-related bottleneck serving 2 billion+ daily active users. They disabled Python's cyclic garbage collector, saving \~10% of CPU, and used Cython for hot paths. Understanding these production-level trade-offs is what separates a Python user from a Python engineer.
**Follow-up:**
* "You mentioned Python compiles to bytecode. What is the difference between CPython, PyPy, and Cython, and when would you choose each?"
* "How does Python's strong dynamic typing affect how you design a large codebase with 50+ engineers? What tooling fills the gap?"
* "What is one feature of Python you think is a design mistake, and how do you work around it?"
* "Python 3.13 introduced a free-threaded build. What does this mean for the GIL and existing C extensions?"
Python's built-in data types, organized by category:
* **Numeric**: `int` (arbitrary precision -- no overflow), `float` (64-bit IEEE 754 double), `complex` (real + imaginary)
* **Sequence**: `list` (mutable, ordered), `tuple` (immutable, ordered), `range` (lazy integer sequence), `str` (immutable Unicode text)
* **Mapping**: `dict` (insertion-ordered since 3.7, hash map under the hood)
* **Set**: `set` (mutable, hash-based unique collection), `frozenset` (immutable, can be dict key)
* **Boolean**: `bool` (subclass of `int` -- `True == 1`, `False == 0`)
* **Binary**: `bytes` (immutable), `bytearray` (mutable), `memoryview` (zero-copy buffer access)
* **None Type**: `NoneType` (singleton -- only one `None` object exists)
**What most people miss:**
* `bool` being a subclass of `int` means `True + True == 2`. This is by design (PEP 285) and occasionally causes bugs: `isinstance(True, int)` returns `True`.
* `dict` ordering is an implementation detail in CPython 3.6 and a language guarantee since 3.7. If you are writing code that must run on older Python, do not rely on it.
* `memoryview` is critical for high-performance work -- it lets you slice bytes without copying data. NumPy uses this concept extensively.
* Python integers have **arbitrary precision**: `2**10000` works fine. There is no `int` overflow. Under the hood, CPython uses a C struct with a variable-length array of digits.
**What interviewers are really testing:** Whether you know the *behavior* of these types, not just their names. Especially: mutability, hashability, and when to choose one over another.
**Red flag answer:** Just listing the types without understanding mutability or when to use each.
**Follow-up:**
* "Why can you use a tuple as a dictionary key but not a list? What is the relationship between hashability and immutability?"
* "What happens when you put a `float('nan')` into a set? Can you have duplicate NaN values? Why does `float('nan') == float('nan')` return `False`?"
* "When would you use `bytearray` over `bytes`? Give a network protocol example."
* "`sys.getsizeof(1)` returns 28 bytes for a single integer. Why is a Python `int` so large compared to a C `int`?"
The surface answer is mutability, but the real differences go much deeper:
**Mutability:** Lists are mutable (`append`, `pop`, `sort` in place). Tuples are immutable (cannot change after creation). But a tuple *can contain mutable objects*: `t = ([1, 2], [3, 4])` -- the tuple itself is immutable, but the lists inside it can change. This is a common gotcha.
**Performance (real numbers):**
* Tuple creation is \~5-10x faster than list creation for small sizes (CPython optimizes tuple allocation with a free list cache)
* Tuples use less memory: a 3-element tuple is \~64 bytes vs \~88 bytes for a 3-element list (CPython 3.11), because lists over-allocate space for future `append()` calls
* Tuple element access is marginally faster (simpler C struct, no indirection for resizing metadata)
* `timeit` benchmark: creating `(1,2,3)` takes \~15ns vs `[1,2,3]` at \~50ns on typical hardware
**Hashability:** Tuples are hashable (if all elements are hashable), so they can be dict keys and set members. Lists cannot. This is the reason you would use `tuple` as a dict key for multi-dimensional lookups: `cache[(x, y, z)]`.
**Semantic meaning:** In idiomatic Python, tuples represent *heterogeneous fixed-structure records* (like a row from a database: `(name, age, email)`), while lists represent *homogeneous variable-length collections* (like a list of users). This is why `namedtuple` exists.
**What interviewers are really testing:** Whether you understand the *why* behind choosing one vs the other, not just the syntax difference.
**Red flag answer:** "Tuples use parentheses and lists use brackets." This shows zero understanding of the engineering implications.
**Follow-up:**
* "If tuples are immutable, why can `t = ([1,2],)` have its inner list modified? What does immutability actually mean here?"
* "When would you use a `namedtuple` vs a `dataclass`? What are the trade-offs in terms of memory, mutability, and inheritance?"
* "You have a function returning multiple values. Should you return a tuple or a list? Why? What about a `dataclass`?"
* "Can you hash `(1, [2, 3])`? Why or why not? What exception is raised?"
**Mutable types** (can be modified in place): `list`, `dict`, `set`, `bytearray`, and most user-defined objects.
**Immutable types** (create new objects on modification): `int`, `float`, `str`, `tuple`, `frozenset`, `bytes`, `bool`, `NoneType`.
**Why this matters in real code:**
1. **The mutable default argument trap** -- the single most common Python bug:
```python theme={null}
def add_item(item, items=[]): # BUG: shared across all calls
items.append(item)
return items
# add_item(1) returns [1]
# add_item(2) returns [1, 2] -- surprise!
```
Fix: use `None` as default, create new list inside function. This bug has caused production incidents at companies processing millions of requests where a shared default list accumulated data across HTTP requests.
2. **Dictionary keys must be immutable** (and hashable). If you mutate an object used as a dict key, the hash changes and the dict breaks silently -- you can no longer find the key.
3. **Pass-by-object-reference implications**: When you pass a list to a function, the function gets a reference to the *same* list object. Modifications inside the function affect the caller. With immutable types, operations create new objects, so the caller is unaffected.
4. **Thread safety**: Immutable objects are inherently thread-safe because they cannot change. Mutable shared state requires locks. This is why `str` concatenation in a loop creates new strings each time (use `''.join()` instead for O(n) vs O(n^2)).
**What interviewers are really testing:** Whether you can explain how mutability causes real bugs, not just recite the definition.
**Red flag answer:** Explaining the concept without mentioning the mutable default argument trap or pass-by-reference implications.
**Follow-up:**
* "Show me a bug caused by mutability that would be hard to catch in code review. How would you write a test for it?"
* "How does string immutability affect performance when building a large string in a loop? What is the fix, and what is the Big-O difference?"
* "Is a tuple always immutable? What about `tuple([mutable_list])`? Can you `hash()` a tuple that contains a list?"
* "You are designing a configuration object for a Flask app. Should its attributes be mutable or immutable? What pattern would you use?"
`is` and `is not` check whether two variables point to the **same object in memory** (identity), not whether they have the same value (equality).
**The critical distinction:**
```python theme={null}
a = [1, 2, 3]
b = [1, 2, 3]
c = a
a == b # True (same value)
a is b # False (different objects in memory)
a is c # True (c points to same object as a)
```
**Where `is` should actually be used:**
* **`None` checks**: Always use `if x is None`, never `if x == None`. Why? Because `==` calls `__eq__`, which a class could override to return `True` for `None` comparison. `is` checks identity directly and cannot be overridden. PEP 8 explicitly mandates this.
* **Sentinel values**: When you create a unique sentinel like `_MISSING = object()`, use `is` to check against it.
**The integer interning gotcha:**
```python theme={null}
a = 256
b = 256
a is b # True -- CPython caches integers -5 to 256
a = 257
b = 257
a is b # False in the REPL, but may be True in a .py file!
```
CPython interns small integers (-5 to 256) and short strings as an optimization. The boundary at 257 has tripped up many developers. In a `.py` file, the compiler may constant-fold and reuse the same object, giving different results than the REPL.
**What interviewers are really testing:** Whether you know *when* to use `is` vs `==` and can explain the interning behavior.
**Red flag answer:** Using `is` for general value comparison, or not knowing about `None` check conventions.
**Follow-up:**
* "Why does PEP 8 require `is None` instead of `== None`? Give me a case where `== None` would give a wrong result."
* "What is integer interning, and why does `257 is 257` give different results in the REPL vs a script?"
* "How would you create your own sentinel object, and why would you use `is` to compare against it?"
The real answer here is Python's **LEGB scope resolution rule**, which determines how variable names are looked up:
1. **L**ocal -- inside the current function
2. **E**nclosing -- in the enclosing function (for nested functions/closures)
3. **G**lobal -- at the module level
4. **B**uilt-in -- in Python's built-in namespace (`len`, `print`, `range`, etc.)
Python searches in this exact order. The first match wins.
**The `global` and `nonlocal` keywords:**
```python theme={null}
x = 10 # Global
def outer():
y = 20 # Enclosing
def inner():
nonlocal y # Modifies enclosing scope's y
global x # Modifies module-level x
x = 99
y = 99
inner()
print(y) # 99 (modified by nonlocal)
outer()
print(x) # 99 (modified by global)
```
**The `UnboundLocalError` trap:**
```python theme={null}
x = 10
def broken():
print(x) # UnboundLocalError!
x = 20 # This assignment makes x local to entire function
```
Python determines scope at *compile time*, not runtime. If there is any assignment to `x` in the function, `x` is treated as local for the *entire* function, even before the assignment. This catches many developers off guard.
**Production advice:** Global mutable state is a code smell. In production systems, you should use dependency injection, configuration objects, or module-level constants (uppercase by convention). Flask's `g` object, Django's `settings` module -- frameworks provide patterns to avoid raw globals.
**What interviewers are really testing:** Whether you know LEGB, understand `UnboundLocalError`, and have opinions about global state in real systems.
**Red flag answer:** Only explaining `global` keyword without mentioning LEGB or the `UnboundLocalError` trap.
**Follow-up:**
* "What is an `UnboundLocalError` and when does it occur? Show me a 3-line function that triggers it. Why does Python decide scope at compile time, not runtime?"
* "When would you use `nonlocal` vs `global`? Can you give a real example where `nonlocal` is genuinely useful?"
* "In a production Flask app, how would you manage configuration instead of using global variables? Compare `app.config`, environment variables, and dependency injection approaches."
* "A developer added `logging = custom_logger()` inside a function and now `logging.info()` fails earlier in the same function. What happened?"
Python provides explicit type conversion functions and also performs implicit type coercion in specific situations.
**Explicit conversion (type constructors):**
* `int('123')` returns `123`. `int('0xff', 16)` returns `255` (base conversion). `int(3.9)` returns `3` (truncates, does not round).
* `float('3.14')` returns `3.14`. `float('inf')` and `float('nan')` are valid.
* `str(42)` returns `'42'`. But `repr(42)` also returns `'42'` -- the difference matters for strings: `str('hello')` returns `hello`, `repr('hello')` returns `'hello'` (with quotes). Use `repr()` for debugging, `str()` for user display.
* `bool()` follows truthiness rules: `bool(0)`, `bool('')`, `bool([])`, `bool(None)` are all `False`. Everything else is `True`.
* `list('abc')` returns `['a', 'b', 'c']` -- iterates the string.
**Implicit coercion (where Python does it for you):**
* `3 + 4.0` returns `7.0` (int promoted to float)
* `True + 2` returns `3` (bool is subclass of int)
* `if my_list:` implicitly calls `bool(my_list)` -- this is the Pythonic way to check for empty collections
**Gotchas that bite in production:**
* `int('3.14')` raises `ValueError` -- you must do `int(float('3.14'))` for string-to-int via float
* `float('nan') == float('nan')` is `False` (IEEE 754 spec). Use `math.isnan()` instead
* `bool([False])` is `True` -- the list is not empty, even though its only element is falsy
* `json.loads()` returns Python `dict`/`list` automatically but all numbers become `float`, losing integer precision for large numbers
**What interviewers are really testing:** Edge cases and gotchas, not basic syntax.
**Red flag answer:** Just listing conversion functions without knowing any edge cases.
**Follow-up:**
* "What is the difference between `str()` and `repr()`? When does it matter?"
* "Why does `int('3.14')` fail? How would you safely convert arbitrary user input to an integer?"
* "Explain Python's truthiness rules. What objects are falsy?"
PEP 8 is Python's official style guide (authored by Guido van Rossum, Barry Warsaw, and Nick Coghlan). But the real interview question is about how you *enforce* consistency in a team.
**Key conventions:**
* **Naming**: `snake_case` for functions/variables/modules, `PascalCase` for classes, `UPPER_SNAKE_CASE` for constants, `_single_leading_underscore` for internal use, `__double_leading` for name mangling
* **Indentation**: 4 spaces (never tabs). This is non-negotiable in the Python community.
* **Line length**: 79 characters (72 for docstrings). Many teams relax this to 88 (Black's default) or 100-120 for modern wide monitors.
* **Imports**: Standard library first, then third-party, then local. One import per line. Tools like `isort` automate this.
**How real teams enforce PEP 8 (the actual answer interviewers want):**
* **`black`**: Opinionated auto-formatter. Zero configuration. "Any color you want, as long as it is black." Eliminates style debates in code review.
* **`ruff`**: Blazing-fast linter (written in Rust), replaces `flake8`, `isort`, `pyupgrade`, and dozens of plugins. Adopted by major projects like FastAPI, pandas, and Airflow.
* **`mypy`/`pyright`**: Not PEP 8 per se, but type checking is now standard in serious Python projects. PEP 484 type hints are checked by these tools.
* **Pre-commit hooks**: Run `black` + `ruff` + `mypy` before every commit. This is the industry-standard setup for Python projects in 2024+.
* **CI/CD enforcement**: Fail the build if linting does not pass. No exceptions.
**What interviewers are really testing:** Not whether you know what PEP 8 says, but whether you have experience with tooling that enforces code quality in a team environment.
**Red flag answer:** "I follow PEP 8 by reading it and being careful." This shows no tooling awareness.
**Follow-up:**
* "Walk me through the linting/formatting setup on your last project. What tools did you use?"
* "Your team disagrees on line length (79 vs 120). How do you resolve this?"
* "What is the difference between a linter (`ruff`/`flake8`) and a formatter (`black`)? Why do you need both?"
The Zen of Python (PEP 20) is a collection of 19 aphorisms by Tim Peters that guide Python's design philosophy. Access with `import this`.
**The principles that actually matter in interviews (and why):**
* **"Explicit is better than implicit"** -- This is why Python does not have implicit type coercion like JavaScript (`"1" + 1` throws TypeError in Python, returns `"11"` in JS). This principle also explains why `import *` is discouraged.
* **"Simple is better than complex"** -- Prefer `for item in items` over `for i in range(len(items))`. Prefer list comprehensions over manual loops when they are readable. But: "Complex is better than complicated" -- sometimes a class is clearer than a nested dict of functions.
* **"Readability counts"** -- This is why Python uses significant whitespace, why `and`/`or`/`not` exist instead of `&&`/`||`/`!`, and why the community obsesses over naming.
* **"Errors should never pass silently"** -- Never use bare `except: pass`. Log it, re-raise it, handle it specifically. This principle directly informs exception handling best practices.
* **"There should be one -- and preferably only one -- obvious way to do it"** -- Contrast with Perl's TMTOWTDI ("There is more than one way to do it"). Python values consistency.
* **"If the implementation is hard to explain, it is a bad idea"** -- If your clever metaprogramming requires a 500-word comment, refactor it.
**Real-world application:** When reviewing a PR, the Zen of Python gives you vocabulary for feedback. "This lambda chain is clever, but 'readability counts' -- can we refactor to a named function?" is much better feedback than "I don't like this."
**What interviewers are really testing:** Whether you have internalized these principles and can apply them to code decisions, not just recite them.
**Red flag answer:** Reciting principles without being able to give a concrete code example where one applies.
**Follow-up:**
* "Give me an example of real code where 'explicit is better than implicit' changed your design decision."
* "When does 'simple is better than complex' conflict with 'practicality beats purity'? How do you decide?"
* "How would you use the Zen of Python to argue for or against using a metaclass in a code review?"
**Deep equality** (`==`) recursively compares values. Python's `==` operator calls `__eq__` on objects, and built-in types implement deep comparison by default:
```python theme={null}
[1, [2, 3]] == [1, [2, 3]] # True -- deep comparison
```
**Structural pattern matching** (Python 3.10+, PEP 634) uses `match`/`case` statements to destructure and match against patterns. It is not just a switch statement -- it is a fundamentally different tool:
```python theme={null}
def handle_command(command):
match command:
case ["quit"]:
return "Exiting"
case ["go", direction]:
return f"Going {direction}"
case ["get", item] if item != "sword":
return f"Picking up {item}"
case _:
return "Unknown command"
```
**Why this matters:** Pattern matching replaces chains of `isinstance` checks and nested `if/elif` blocks. Before 3.10, parsing a JSON API response with variable shapes required ugly conditional logic. With `match/case`, you can destructure the shape directly.
**Guards** (`if` clauses in `case`) add conditional logic to patterns. **Capture variables** bind matched values to names. **OR patterns** (`case "yes" | "y"`) match multiple options.
**Gotcha:** `case name:` where `name` is a variable does NOT match against the variable's value -- it is a *capture pattern* that always matches and binds the value to `name`. Use `case MyEnum.VALUE:` for value matching, or use a guard: `case x if x == name:`.
**What interviewers are really testing:** Whether you know modern Python features and can recognize when pattern matching simplifies code vs when it is overkill.
**Red flag answer:** "It is just Python's version of a switch statement." Pattern matching is far more powerful.
**Follow-up:**
* "When would pattern matching be clearer than if/elif chains? When would it be worse?"
* "What is the capture variable gotcha in `match/case`, and how do you avoid it?"
* "How would you use pattern matching to parse different shapes of JSON API responses?"
The walrus operator (`:=`, PEP 572, Python 3.8+) is an **assignment expression** -- it assigns a value to a variable as part of an expression, rather than as a standalone statement.
**Where it genuinely helps:**
```python theme={null}
# Without walrus -- duplicate call or temp variable
data = get_data()
if data:
process(data)
# With walrus -- single expression
if (data := get_data()):
process(data)
# Especially useful in while loops
while (chunk := file.read(8192)):
process(chunk)
# And in list comprehensions with expensive filtering
results = [clean for item in items if (clean := expensive_transform(item)) is not None]
```
**When NOT to use it:**
* When it hurts readability. `x := expr` inside a complex expression makes code harder to parse visually.
* In simple assignments. `x := 5` is worse than `x = 5` -- it adds noise.
* PEP 572 itself warns against overuse. Guido van Rossum stepped down as BDFL partly because of the contentious debate around this feature.
**The key trade-off:** The walrus operator reduces lines of code and avoids duplicate computation, but it increases the cognitive load per line. Senior engineers use it sparingly and only when the intent is clearer than the alternative.
**What interviewers are really testing:** Judgment about when a feature helps vs when it hurts readability.
**Red flag answer:** Either not knowing it exists (knowledge gap) or using it everywhere to show off (poor judgment).
**Follow-up:**
* "Show me a case where the walrus operator makes code genuinely clearer, and one where it makes it worse."
* "Why was PEP 572 so controversial in the Python community?"
* "How does `:=` differ from `=` in terms of scope inside comprehensions?"
## 2. Data Structures
Lists are Python's workhorse data structure -- a dynamic array that can hold mixed types. But understanding the *performance characteristics* is what separates a strong answer.
**Creation:**
```python theme={null}
my_list = [1, 2, 3] # Literal (fastest)
my_list = list(range(10)) # From iterable
my_list = [0] * 100 # Pre-allocated (useful for performance)
my_list = [x**2 for x in range(10)] # Comprehension (Pythonic)
```
**Common operations and their Big-O complexity:**
* `append(item)` -- O(1) amortized. Uses over-allocation strategy: when the internal array is full, CPython allocates \~12.5% more space. This is why `append` is fast on average but occasionally triggers a resize.
* `insert(0, item)` -- O(n). Every element must shift. If you are doing frequent left-inserts, use `collections.deque` instead (O(1) on both ends).
* `pop()` -- O(1) from end, O(n) from beginning (`pop(0)`). Again, `deque` is better for FIFO patterns.
* `remove(item)` -- O(n). Linear search + shift.
* `in` operator -- O(n). If you need fast membership testing, use a `set` instead.
* `sort()` -- O(n log n). Uses Timsort (hybrid merge sort + insertion sort). Stable sort. `sorted()` returns a new list; `sort()` modifies in place.
* Slice `lst[a:b]` -- O(b-a). Creates a shallow copy of the slice.
**What interviewers are really testing:** Whether you know the time complexity of list operations and can choose the right data structure.
**Red flag answer:** Listing methods without knowing their complexity or when lists are the wrong choice.
**Follow-up:**
* "Your code does `list.insert(0, item)` in a loop processing 1M items. What is the total complexity, and how do you fix it?"
* "What is the difference between `sort()` and `sorted()`? When would you use each?"
* "How does CPython's list over-allocation strategy work, and why does it matter?"
List comprehensions are a concise, Pythonic way to create lists from iterables. But the real depth is in understanding *when they help and when they hurt*.
**Syntax:** `[expression for item in iterable if condition]`
**Why they are faster than equivalent loops:**
```python theme={null}
# Loop version (~30% slower in CPython benchmarks)
squares = []
for x in range(1000):
squares.append(x**2)
# Comprehension (faster -- no repeated attribute lookup for .append)
squares = [x**2 for x in range(1000)]
```
The speed difference comes from CPython's internal optimization: comprehensions use a dedicated `LIST_APPEND` bytecode instruction, avoiding the overhead of looking up and calling the `append` method on each iteration.
**Advanced patterns:**
```python theme={null}
# Nested comprehension (flattening)
flat = [x for row in matrix for x in row]
# Dict comprehension
word_lengths = {word: len(word) for word in words}
# Set comprehension
unique_lengths = {len(word) for word in words}
# Conditional expression (ternary in comprehension)
labels = ["even" if x % 2 == 0 else "odd" for x in range(10)]
```
**When NOT to use comprehensions:**
* When the logic requires multiple statements or side effects
* When the comprehension exceeds \~80 characters or nests more than 2 levels deep -- readability drops fast
* When you do not need the full list in memory -- use a **generator expression** `(x**2 for x in range(10_000_000))` instead to avoid allocating a massive list
**The memory trap:** `[process(x) for x in range(10_000_000)]` creates a 10M-element list in memory. If you are only iterating once, use a generator expression or `map()`.
**What interviewers are really testing:** Whether you know the performance implications and readability boundaries, not just the syntax.
**Red flag answer:** Writing deeply nested comprehensions as a flex. One-line does not mean readable.
**Follow-up:**
* "When should you use a generator expression `(...)` instead of a list comprehension `[...]`?"
* "This 3-level nested comprehension is hard to read. Refactor it to be clearer."
* "What is the bytecode difference between a list comprehension and an equivalent for loop? Why is the comprehension faster?"
Dictionaries are Python's hash map implementation. Since Python 3.7, they maintain **insertion order** as a language guarantee (not just a CPython implementation detail).
**Under the hood (CPython 3.6+):**
Python dicts use a compact hash table with two arrays: a sparse hash index array and a dense key-value entries array. This design reduced memory usage by 20-25% compared to Python 3.5's implementation. Lookups are O(1) average, O(n) worst case (hash collisions).
**Essential methods and when to use them:**
* `get(key, default)` -- Returns `default` (or `None`) if key missing. Avoids `KeyError`. Use this over `dict[key]` when the key might not exist.
* `setdefault(key, default)` -- Returns value if key exists, otherwise sets key to default and returns it. Atomic operation useful for building groupings:
```python theme={null}
groups = {}
for item in items:
groups.setdefault(item.category, []).append(item)
```
But `collections.defaultdict` is usually cleaner for this pattern.
* `update(other_dict)` -- Merges another dict. Since Python 3.9, you can use `|` operator: `merged = dict1 | dict2` (dict2 wins on conflicts).
* `pop(key, default)` -- Removes and returns value. Raises `KeyError` if no default.
* `items()` / `keys()` / `values()` -- Return **view objects** (not copies). Views reflect changes to the dict. Do not modify a dict while iterating its views.
**Performance characteristics:**
* Lookup, insert, delete: O(1) average
* Memory: \~50-70 bytes per key-value pair (CPython 3.11)
* Hash function: CPython uses SipHash for strings (resistant to hash-flooding DoS attacks, changed after CVE-2012-1150)
* Keys must be **hashable** (implement `__hash__` and `__eq__`). Mutable objects like lists cannot be keys.
**Python 3.9+ dict merge operators:**
```python theme={null}
merged = dict1 | dict2 # Creates new dict (dict2 wins on conflicts)
dict1 |= dict2 # In-place merge
```
**What interviewers are really testing:** Whether you understand hash maps, know about ordering guarantees, and can choose between `dict`, `defaultdict`, `OrderedDict`, and `Counter`.
**Red flag answer:** "Dictionaries store key-value pairs." This is the level of a beginner tutorial.
**Follow-up:**
* "What happens if two keys have the same hash? How does Python resolve collisions?"
* "When would you use `collections.defaultdict` vs `dict.setdefault()`?"
* "Your dict lookup is O(1) in theory but slow in practice. What could cause this?"
**Sets** are mutable, unordered collections of unique, hashable elements. **Frozensets** are their immutable counterpart.
**Why frozensets exist -- the real answer:**
Since sets are mutable, they are not hashable, which means you cannot put a set inside another set or use it as a dict key. Frozensets solve this:
```python theme={null}
# This fails -- sets are not hashable
# my_set = { {1, 2}, {3, 4} } # TypeError
# This works -- frozensets are hashable
my_set = { frozenset({1, 2}), frozenset({3, 4}) }
cache_key = frozenset(user_permissions) # Use as dict key
```
**Performance (both types):**
* Membership testing (`in`): O(1) average. This is the primary reason to use sets -- `if item in my_set` is O(1) vs O(n) for lists.
* Add/remove: O(1) average (sets only).
* Union/intersection/difference: O(min(len(s1), len(s2))) for intersection, O(len(s1) + len(s2)) for union.
**Real-world use cases for frozensets:**
* Cache keys representing a set of features or permissions
* Dict keys for memoization where the input is a collection of unique items
* Elements of other sets (e.g., a set of sets for graph algorithms)
* Immutable configuration that should not be modified after initialization
**What interviewers are really testing:** Whether you understand hashability and can articulate why immutability matters for hash-based containers.
**Red flag answer:** "Frozensets are just sets you cannot change." This misses the *why* -- hashability and use as dict keys.
**Follow-up:**
* "Why cannot you use a regular set as a dictionary key?"
* "You need to check if a user has any of 10,000 banned words in their input. What data structure do you use?"
* "What happens to set performance when every element has the same hash?"
Sets support mathematical set operations with both operator and method syntax. The key interview insight is knowing *when each is useful in production code*.
**Operations and their complexity:**
* **Union** (`s1 | s2` or `s1.union(s2)`) -- O(len(s1) + len(s2)). All elements from both sets.
* **Intersection** (`s1 & s2` or `s1.intersection(s2)`) -- O(min(len(s1), len(s2))). Only elements in both.
* **Difference** (`s1 - s2` or `s1.difference(s2)`) -- O(len(s1)). Elements in s1 but not s2.
* **Symmetric difference** (`s1 ^ s2`) -- O(len(s1) + len(s2)). Elements in either but not both.
* **Subset/Superset** (`s1 <= s2`, `s1 >= s2`) -- O(len(s1)). Check containment.
**The method syntax advantage:** Methods accept any iterable, operators require both to be sets:
```python theme={null}
my_set = {1, 2, 3}
my_set.union([4, 5, 6]) # Works -- accepts list
my_set | [4, 5, 6] # TypeError -- operator needs set on both sides
```
**Real-world examples:**
```python theme={null}
# Find users with both permissions (intersection)
admin_users = set(admins) & set(active_users)
# Find new users who signed up but have not verified (difference)
unverified = set(signups) - set(verified_emails)
# Find features unique to either A/B test group (symmetric diff)
unique_features = set(group_a_features) ^ set(group_b_features)
# Check if required permissions are all present (subset)
if required_perms <= user_perms:
grant_access()
```
**What interviewers are really testing:** Whether you can apply set operations to solve real problems efficiently, not just define them.
**Red flag answer:** Reciting operations without real-world examples or complexity awareness.
**Follow-up:**
* "You have two lists of 1M user IDs each. How do you find users in both lists? What is the complexity?"
* "What is the difference between `s1 - s2` and `s2 - s1`? When does order matter?"
* "How would you use set operations to implement a simple permissions system?"
This is the "choose your data structure" question. The strong answer is a decision matrix, not a feature comparison.
| Feature | `list` | `tuple` | `set` |
| --------------- | ------------------------ | ------------------------------ | -------------------------------- |
| Ordered | Yes | Yes | No |
| Mutable | Yes | No | Yes (elements cannot be mutable) |
| Duplicates | Allowed | Allowed | Not allowed |
| Indexable | Yes, O(1) | Yes, O(1) | No |
| Membership test | O(n) | O(n) | **O(1)** |
| Hashable | No | Yes (if elements are hashable) | No (use frozenset) |
| Memory | Highest (over-allocates) | Lowest (fixed) | Medium (hash table) |
**When to use each (the real interview answer):**
* **List**: When you need ordered, mutable, indexed access. Default choice for collections. Example: a list of user records you will sort, filter, paginate.
* **Tuple**: When data is fixed and should not change. Function return values, dict keys, named records. Example: `(latitude, longitude)` coordinate pairs. Also used for *heterogeneous* data (name, age, email) vs lists for *homogeneous* data (list of names).
* **Set**: When you need fast membership testing or deduplication. Example: checking if a URL has been visited, deduplicating a list of emails, computing permission intersections.
**The collections module alternatives:**
* `collections.deque` -- When you need O(1) append/pop from both ends (queue/stack)
* `collections.OrderedDict` -- When you need dict with explicit ordering control (e.g., LRU cache before 3.7)
* `collections.Counter` -- When you need to count occurrences
* `collections.defaultdict` -- When you want automatic default values
**What interviewers are really testing:** Whether you can justify your data structure choice based on access patterns and performance requirements.
**Red flag answer:** Describing features without explaining *when* to use each one.
**Follow-up:**
* "You are processing 10M log entries and need to find unique IPs. Which data structure and why?"
* "Your function returns `(success, data, error)`. Should this be a tuple, list, or dict?"
* "When would you reach for `collections.deque` instead of a list?"
This question tests whether you understand Python's reference model. Objects are not values in Python -- variables hold *references* to objects.
**Shallow copy** (`copy.copy()`, list slicing `lst[:]`, `list(lst)`, `dict.copy()`):
Creates a new container object but fills it with references to the *same* nested objects. Only the top-level container is new.
**Deep copy** (`copy.deepcopy()`):
Recursively creates new objects for everything, including all nested containers. A completely independent clone.
```python theme={null}
import copy
original = [[1, 2], [3, 4]]
shallow = copy.copy(original)
deep = copy.deepcopy(original)
# Shallow copy shares nested objects
shallow[0][0] = 99
print(original[0][0]) # 99 -- original affected!
# Deep copy is fully independent
deep[0][0] = 88
print(original[0][0]) # Still 99 -- original unaffected
```
**The subtle cases that matter:**
1. **Circular references**: `deepcopy` handles them correctly by tracking already-copied objects with a memo dict. A naive recursive copy would infinite-loop.
2. **Custom objects**: You can customize copy behavior by implementing `__copy__()` and `__deepcopy__()` methods. SQLAlchemy models, for example, need special handling because they carry database session state.
3. **Performance**: `deepcopy` is **slow** -- it must traverse the entire object graph. On a deeply nested dict with 100K entries, it can take hundreds of milliseconds. In a hot path, consider whether you actually need a full deep copy or can restructure to avoid it.
4. **Hidden shallow copies**: `list[:]` slicing, `dict.copy()`, and the `list()` constructor all create *shallow* copies. Many developers assume these are deep copies.
**Real-world bug example:**
```python theme={null}
# A config template shared across requests
config_template = {"db": {"host": "localhost", "port": 5432}}
def get_config(user):
config = config_template.copy() # SHALLOW -- shares nested dict
config["db"]["host"] = user.db_host # Mutates template!
return config
# After first call, config_template is corrupted for all future calls
```
Fix: `config = copy.deepcopy(config_template)` or restructure to use immutable data.
**What interviewers are really testing:** Whether you have been burned by shallow copy bugs and can explain the reference model.
**Red flag answer:** Defining the difference without mentioning a real bug scenario or knowing about circular reference handling.
**Follow-up:**
* "Show me a production bug caused by shallow copy. How would you find it?"
* "`deepcopy` is too slow for your hot path. What alternatives exist?"
* "How does `deepcopy` handle circular references internally?"
`defaultdict` (from `collections`) automatically creates a default value when you access a missing key, eliminating the need for existence checks.
**How it works internally:**
```python theme={null}
from collections import defaultdict
# Groups items by category -- no KeyError, no setdefault boilerplate
word_counts = defaultdict(int) # Missing keys get 0
grouped = defaultdict(list) # Missing keys get []
nested = defaultdict(lambda: defaultdict(int)) # Nested defaults
for word in words:
word_counts[word] += 1 # No need to check if key exists
for item in items:
grouped[item.category].append(item) # No need for setdefault
```
`defaultdict` overrides `__missing__()`. When `__getitem__` fails to find a key, it calls the factory function, stores the result, and returns it. This means accessing a missing key *creates* that key -- which is a subtle difference from `dict.get()`.
**`defaultdict` vs alternatives:**
* `dict.setdefault(key, [])` -- similar but more verbose. Creates a new default object on every call even if the key exists (though CPython optimizes common cases).
* `dict.get(key, default)` -- does NOT insert the key into the dict. Read-only fallback.
* `collections.Counter` -- specialized `defaultdict(int)` with extra methods like `most_common()`.
**Gotcha:** Because `defaultdict` creates entries on access, iterating with a `for key in d` loop and accessing keys that do not exist will pollute the dict:
```python theme={null}
d = defaultdict(int)
d["a"] = 1
print(d["b"]) # Returns 0 but ALSO creates key "b" in the dict
print(len(d)) # 2, not 1!
```
If you want read-only default behavior, use `dict.get()` instead.
**What interviewers are really testing:** Whether you know the right tool for grouping/counting patterns and understand the side effects.
**Red flag answer:** Not knowing `defaultdict` exists, or using manual `if key in dict` checks everywhere.
**Follow-up:**
* "What is the difference between `defaultdict(list)` and using `dict.setdefault(key, [])`?"
* "When would `defaultdict` create unexpected entries? How do you prevent it?"
* "How would you implement a `defaultdict` from scratch using `__missing__`?"
## 3. Object-Oriented Programming
The four pillars, but explained the way Python *actually* implements them (not Java-style textbook definitions):
**1. Encapsulation** -- Python takes a radically different approach than Java/C++. There are no `private` or `protected` keywords. Instead:
* Single underscore `_name`: Convention for "internal use." Not enforced. Other code *can* access it but *should not*.
* Double underscore `__name`: Triggers **name mangling** -- Python rewrites it to `_ClassName__name`. This prevents accidental override in subclasses, but it is NOT true privacy (you can still access it if you know the mangled name).
* The philosophy: "We are all consenting adults here." Python trusts developers to respect conventions.
**2. Abstraction** -- Implemented via Abstract Base Classes (`abc.ABC` or `abc.ABCMeta`). You cannot instantiate a class with unimplemented abstract methods. Also achieved through Python 3.8+ `Protocol` classes (structural subtyping -- no inheritance required).
**3. Inheritance** -- Python supports multiple inheritance with C3 linearization MRO. But the community strongly prefers **composition over inheritance**. Deep inheritance hierarchies are a code smell in Python. Use mixins sparingly.
**4. Polymorphism** -- In Python, this is primarily **duck typing**: "If it quacks like a duck..." You do not need inheritance for polymorphism. Any object with a `read()` method works wherever a file-like object is expected. `typing.Protocol` formalizes this.
**What makes Python's OOP unique:** It is opt-in and flexible. You can write purely procedural Python, purely functional Python, or full OOP. Most production Python is a pragmatic mix.
**What interviewers are really testing:** Whether you understand how Python's OOP differs from Java/C++ and can articulate the philosophy, not just the definitions.
**Red flag answer:** Describing the four pillars exactly as a Java textbook would. Python's implementation is fundamentally different.
**Follow-up:**
* "Python has no `private` keyword. How do you prevent other developers from accessing internal state?"
* "When would you use `abc.ABC` vs `typing.Protocol`?"
* "Give me an example where duck typing is more Pythonic than inheritance-based polymorphism."
The basic syntax is the `class` keyword, but a strong answer covers the full lifecycle of Python object creation and modern alternatives.
**Basic class:**
```python theme={null}
class Person:
species = "Human" # Class attribute (shared by all instances)
def __init__(self, name, age):
self.name = name # Instance attribute
self.age = age
def greet(self):
return f"Hello, I am {self.name}"
p = Person("Alice", 30)
```
**What actually happens when you write `Person("Alice", 30)`:**
1. `Person.__new__(Person)` is called to create the instance (allocates memory)
2. `Person.__init__(instance, "Alice", 30)` initializes the instance
3. The instance is returned
Most developers never override `__new__`, but it is essential for immutable types (you cannot modify `self` in `__init__` for immutable objects) and Singleton patterns.
**Modern alternatives (what you should actually use in 2024+):**
```python theme={null}
# dataclasses (Python 3.7+) -- eliminates boilerplate
from dataclasses import dataclass
@dataclass
class Person:
name: str
age: int
email: str = "" # Default value
# Gets __init__, __repr__, __eq__ for free
# Also supports frozen=True for immutability, slots=True for memory
# NamedTuple -- lightweight immutable records
from typing import NamedTuple
class Point(NamedTuple):
x: float
y: float
```
**When to use what:**
* `@dataclass` -- Most data-holding classes. Default choice for DTOs, configs, records.
* `NamedTuple` -- When you want immutability and tuple compatibility (hashable, can be dict key).
* Regular class -- When you need complex behavior, custom `__init__` logic, or non-trivial methods.
* `attrs` (third-party) -- More features than `dataclass` (validators, converters). Used by major projects like `pytest`.
**What interviewers are really testing:** Whether you reach for `@dataclass` instead of writing boilerplate `__init__`/`__repr__`/`__eq__` by hand.
**Red flag answer:** Writing a class with manual `__init__`, `__repr__`, and `__eq__` when `@dataclass` would do it in 3 lines.
**Follow-up:**
* "What is the difference between `__new__` and `__init__`? When would you override `__new__`?"
* "Compare `@dataclass` vs `NamedTuple` vs `attrs`. When would you choose each?"
* "How does `@dataclass(frozen=True, slots=True)` work, and when would you use those options?"
**Instance methods** -- The default. Take `self` as first parameter. Operate on instance data. Can access both instance and class attributes.
**Class methods** -- Decorated with `@classmethod`. Take `cls` as first parameter. Operate on the *class itself*, not instances. The killer use case is **alternative constructors**:
```python theme={null}
class Date:
def __init__(self, year, month, day):
self.year = year
self.month = month
self.day = day
@classmethod
def from_string(cls, date_string):
year, month, day = map(int, date_string.split('-'))
return cls(year, month, day) # cls, not Date -- works with subclasses!
@classmethod
def today(cls):
import datetime
t = datetime.date.today()
return cls(t.year, t.month, t.day)
```
The `cls` parameter is crucial: if `SpecialDate(Date)` calls `Date.from_string()`, it returns a `SpecialDate` instance, not a `Date`. This is why `@classmethod` exists instead of just using regular functions.
**Static methods** -- Decorated with `@staticmethod`. Take neither `self` nor `cls`. Pure utility functions that logically belong to the class namespace:
```python theme={null}
class MathUtils:
@staticmethod
def is_even(n):
return n % 2 == 0
```
Controversial opinion: most `@staticmethod` uses should just be module-level functions. `@staticmethod` only makes sense when the function is conceptually bound to the class.
**What interviewers are really testing:** Whether you know *when* to use `@classmethod` (alternative constructors, factory methods) vs `@staticmethod` (rare) vs instance methods (default).
**Red flag answer:** Defining all three without giving the alternative constructor use case for `@classmethod`.
**Follow-up:**
* "Why does `@classmethod` use `cls` instead of the class name directly? What breaks if you hardcode the class name?"
* "When would a `@staticmethod` be better as a module-level function?"
* "How would you implement `dict.fromkeys()` using `@classmethod`?"
Inheritance allows a class to reuse code from a parent class. Python supports both single and multiple inheritance, which makes it more powerful -- and more dangerous -- than languages like Java.
**Single inheritance:**
```python theme={null}
class Animal:
def speak(self):
raise NotImplementedError
class Dog(Animal):
def speak(self):
return "Woof"
```
**Multiple inheritance and the Diamond Problem:**
```python theme={null}
class A:
def method(self): return "A"
class B(A):
def method(self): return "B"
class C(A):
def method(self): return "C"
class D(B, C):
pass
D().method() # Returns "B" -- MRO is D -> B -> C -> A -> object
```
Python resolves this using C3 linearization (MRO). The order is deterministic and predictable.
**`super()` is NOT "call the parent"** -- it is "call the next class in the MRO chain":
```python theme={null}
class B(A):
def method(self):
return super().method() # Calls C.method(), NOT A.method()!
```
In the diamond above, `super()` in B calls C's method, not A's. This is cooperative multiple inheritance, and misunderstanding it causes bugs.
**When to use inheritance vs composition:**
* Inheritance: True "is-a" relationships. `Dog` IS an `Animal`. Interface implementation via ABCs.
* Composition: "has-a" relationships. `Car` HAS an `Engine`. Prefer this in most cases.
* Mixins: Small, focused classes that add specific behavior. `LoggingMixin`, `SerializableMixin`. Keep them single-purpose.
**What interviewers are really testing:** Whether you understand MRO, `super()` in multiple inheritance, and the preference for composition.
**Red flag answer:** "Inheritance lets you reuse code." Without understanding MRO or the composition alternative.
**Follow-up:**
* "In the diamond problem, what does `super()` in class B actually call? Walk me through the MRO."
* "When would multiple inheritance cause problems that composition would not?"
* "How do mixins work, and what rules should you follow when designing them?"
Magic methods (double underscore, aka "dunder" methods) are Python's protocol system. They let your objects integrate with Python's built-in operations, syntax, and standard library. This is arguably the most powerful feature of Python's object model.
**The essential categories:**
**Object lifecycle:** `__init__` (initializer), `__new__` (constructor), `__del__` (finalizer -- avoid using this, use context managers instead)
**String representations:**
* `__str__` -- called by `str()` and `print()`. Human-readable. Example: `"Alice (age 30)"`.
* `__repr__` -- called by `repr()` and in the REPL. Developer-readable, ideally `eval`-able. Example: `"Person('Alice', 30)"`. Rule: always implement `__repr__`. If you only implement one, implement `__repr__` -- `__str__` falls back to it.
**Comparison:** `__eq__`, `__lt__`, `__le__`, `__gt__`, `__ge__`, `__ne__`. Use `@functools.total_ordering` to implement only `__eq__` and `__lt__`, and get the rest for free.
**Container protocol:** `__len__`, `__getitem__`, `__setitem__`, `__delitem__`, `__contains__`, `__iter__`. Implementing these makes your object work with `len()`, `[]` indexing, `in` operator, and `for` loops.
**Arithmetic:** `__add__`, `__sub__`, `__mul__`, etc. Also reverse versions (`__radd__`) for when your object is on the right side of the operator.
**Context manager:** `__enter__`, `__exit__` -- enable `with` statement support.
**Callable:** `__call__` -- makes instances callable like functions. Useful for stateful callables, decorators implemented as classes.
**Hashing:** `__hash__` -- required for dict keys and set membership. Important rule: if you define `__eq__`, Python sets `__hash__` to `None` (making objects unhashable). You must explicitly define `__hash__` if you want hashability with custom equality.
**What interviewers are really testing:** Whether you can design classes that integrate naturally with Python's ecosystem, not just classes with methods.
**Red flag answer:** Listing dunder methods without explaining the protocols they implement or the `__repr__` vs `__str__` distinction.
**Follow-up:**
* "You define `__eq__` on your class. What happens to `__hash__`? What breaks?"
* "How would you make a custom class work with `for` loops? What methods do you need?"
* "What is the difference between `__str__` and `__repr__`? Which should you always implement?"
`__slots__` is a class-level declaration that tells Python: "These are the only attributes instances of this class will ever have." It replaces the per-instance `__dict__` with a fixed-size struct.
**How it works under the hood:**
```python theme={null}
class Point:
__slots__ = ('x', 'y')
def __init__(self, x, y):
self.x = x
self.y = y
p = Point(1, 2)
p.z = 3 # AttributeError! Cannot add attributes not in __slots__
```
Normally, every Python instance has a `__dict__` (a dict storing its attributes). That dict itself costs \~100 bytes per instance (empty dict overhead) plus \~50-70 bytes per attribute. With `__slots__`, Python stores attributes in a fixed C struct, eliminating the dict entirely.
**Real memory savings (CPython 3.11 benchmarks):**
* Regular class with 3 attributes: \~152 bytes per instance
* `__slots__` class with 3 attributes: \~72 bytes per instance
* At 1 million instances: \~152 MB vs \~72 MB. That is 80 MB saved.
* At 10 million instances (e.g., a graph with 10M nodes): \~800 MB saved. This can be the difference between OOM and running smoothly.
**Trade-offs:**
* Cannot add attributes dynamically -- no monkey patching instance state
* Cannot use `__dict__` -- some serialization libraries expect it
* Inheritance is tricky -- if parent has `__slots__`, child must also declare `__slots__` (can be empty `__slots__ = ()`) or it gets a `__dict__` anyway
* Cannot use `__weakref__` unless you include it in `__slots__`
* `@dataclass(slots=True)` (Python 3.10+) -- the modern way to get slots without manual declaration
**When to use:**
* Classes with millions of instances (nodes in a graph, points in a point cloud, rows in a dataset)
* Internal library classes where the API is fixed
* Performance-critical hot paths
**What interviewers are really testing:** Whether you have optimized memory in production and understand the trade-offs.
**Red flag answer:** "It saves memory." Without knowing how much, or the trade-offs.
**Follow-up:**
* "You add `__slots__` to a class but it still has a `__dict__`. What went wrong?"
* "How does `__slots__` interact with inheritance? What happens in a child class?"
* "When would `__slots__` actually hurt you? Give me a scenario."
`@property` turns method calls into attribute access, giving you controlled getters/setters without changing the API. This is Python's answer to Java's `getName()`/`setName()` pattern.
**The core pattern:**
```python theme={null}
class Temperature:
def __init__(self, celsius):
self._celsius = celsius # Store in private attribute
@property
def celsius(self):
return self._celsius
@celsius.setter
def celsius(self, value):
if value < -273.15:
raise ValueError("Temperature below absolute zero")
self._celsius = value
@property
def fahrenheit(self):
return self._celsius * 9/5 + 32
@fahrenheit.setter
def fahrenheit(self, value):
self.celsius = (value - 32) * 5/9 # Uses celsius setter for validation
t = Temperature(100)
print(t.fahrenheit) # 212.0 -- looks like attribute access, calls method
t.celsius = -300 # ValueError -- validation fires
```
**Why properties matter (the real answer):**
1. **API evolution without breaking changes**: Start with a simple attribute `self.name = name`. Later, when you need validation, replace it with `@property` -- all calling code stays the same. No need to change `obj.name` to `obj.get_name()`.
2. **Computed attributes**: `fahrenheit` above is derived from `celsius`. The caller does not need to know it is calculated.
3. **Lazy loading**: Compute expensive values only when accessed, cache the result.
4. **Validation at the boundary**: Enforce invariants when attributes are set.
**Under the hood**: `@property` is syntactic sugar for the descriptor protocol (`__get__`, `__set__`, `__delete__`). Understanding descriptors explains how properties, methods, and `classmethod`/`staticmethod` all work.
**Gotcha:** `@property` on a method that does I/O or expensive computation can surprise callers who expect attribute access to be cheap. If `obj.data` triggers a database query, it should probably be `obj.get_data()` or `obj.load_data()` to signal the cost.
**What interviewers are really testing:** Whether you understand the Pythonic way to evolve APIs and enforce invariants.
**Red flag answer:** Describing the syntax without explaining *why* you would use properties over plain attributes.
**Follow-up:**
* "You start with `self.name = name` and later need validation. How do you add it without breaking existing code?"
* "What is the relationship between `@property` and the descriptor protocol?"
* "When is `@property` a bad idea? Give me a case where a method is better."
MRO determines the order Python searches for methods in a class hierarchy. This is critical for multiple inheritance and understanding `super()`.
**Python uses C3 linearization (since Python 2.3).** The algorithm guarantees:
1. Children come before parents
2. If a class inherits from multiple parents, they are searched in the order listed
3. A valid linearization exists (otherwise Python raises `TypeError` at class definition time)
**Concrete example:**
```python theme={null}
class A: pass
class B(A): pass
class C(A): pass
class D(B, C): pass
print(D.__mro__)
# (D, B, C, A, object)
```
**Why this matters for `super()`:**
`super()` does NOT mean "call the parent." It means "call the next class in the MRO." This is cooperative multiple inheritance:
```python theme={null}
class A:
def method(self):
print("A")
class B(A):
def method(self):
print("B")
super().method() # Calls C.method(), NOT A.method()!
class C(A):
def method(self):
print("C")
super().method() # Calls A.method()
class D(B, C):
def method(self):
print("D")
super().method() # Calls B.method()
D().method() # Prints: D, B, C, A
```
**When C3 linearization fails:**
```python theme={null}
class X(A, B): pass
class Y(B, A): pass
# class Z(X, Y): pass # TypeError! Cannot create consistent MRO
```
This happens when the ordering constraints are contradictory. Python refuses to guess and raises an error at class definition time.
**Production implications:**
* Always use `super()` consistently in a class hierarchy -- mixing `super()` with direct parent calls breaks cooperative inheritance
* Mixins should always call `super()` in their methods
* Use `ClassName.__mro__` or `ClassName.mro()` to debug method resolution issues
* Consider whether you actually need multiple inheritance or if composition is simpler
**What interviewers are really testing:** Whether you understand that `super()` follows the MRO, not just the parent, and can trace through a diamond hierarchy.
**Red flag answer:** "MRO is the order Python searches for methods." Without being able to trace through a concrete diamond example.
**Follow-up:**
* "Walk me through the MRO for `class D(B, C)` where both B and C inherit from A. What does `super()` in B call?"
* "When does C3 linearization fail? What causes a `TypeError`?"
* "How would you debug a method resolution issue in a codebase with deep inheritance?"
## 4. Functions and Decorators
`*args` collects extra positional arguments into a **tuple**. `**kwargs` collects extra keyword arguments into a **dict**. But the real depth is in understanding how they interact with Python's full argument resolution system.
**The complete argument order (must be in this order):**
```python theme={null}
def func(pos_only, /, normal, *, kw_only, **kwargs):
pass
# pos_only: positional-only (Python 3.8+, before /)
# normal: positional or keyword
# kw_only: keyword-only (after *)
# **kwargs: catches remaining keyword args
```
**Real-world use cases:**
```python theme={null}
# 1. Wrapper/decorator functions -- must pass through all arguments
def timing_decorator(func):
def wrapper(*args, **kwargs): # Accept anything
start = time.time()
result = func(*args, **kwargs) # Pass through everything
print(f"Took {time.time() - start:.3f}s")
return result
return wrapper
# 2. Extending parent class methods
class Child(Parent):
def __init__(self, extra_param, *args, **kwargs):
self.extra = extra_param
super().__init__(*args, **kwargs) # Forward to parent
# 3. Unpacking into function calls
args_tuple = (1, 2, 3)
kwargs_dict = {"name": "Alice", "age": 30}
func(*args_tuple, **kwargs_dict) # Equivalent to func(1, 2, 3, name="Alice", age=30)
```
**The `/` and `*` separators (Python 3.8+):**
```python theme={null}
def example(a, b, /, c, d, *, e, f):
pass
# a, b: positional only (cannot pass as keywords)
# c, d: either positional or keyword
# e, f: keyword only (cannot pass positionally)
```
This matters for API design: `len` takes `obj` as positional-only so you can write `len(obj)` but not `len(obj=mylist)`.
**What interviewers are really testing:** Whether you understand argument resolution order, unpacking, and real-world patterns like decorator forwarding.
**Red flag answer:** Only knowing the basic definition without unpacking (`*`/`**`) or positional-only/keyword-only parameters.
**Follow-up:**
* "What is the difference between `*` and `**` in a function *call* vs a function *definition*?"
* "Why does Python 3.8 add positional-only parameters with `/`? Give a real API where this matters."
* "How would you use `*args` and `**kwargs` to implement a decorator that works with any function?"
Lambda functions are anonymous, single-expression functions. But the real interview question is about *when they are appropriate and when they are an anti-pattern*.
**Syntax:** `lambda arguments: expression` (no statements, no assignments, no multi-line)
**Where lambdas shine:**
```python theme={null}
# Sorting with custom key (most common legitimate use)
users.sort(key=lambda u: u.last_login)
sorted(items, key=lambda x: (x.priority, -x.created_at))
# Quick callbacks
button.on_click(lambda event: print(f"Clicked at {event.x}"))
# Simple transformations with map/filter
names = list(map(lambda u: u.name, users))
```
**Where lambdas are an anti-pattern:**
```python theme={null}
# BAD: Assigning lambda to a variable (use def instead)
square = lambda x: x**2 # flake8 E731 violation
# GOOD:
def square(x):
return x**2
# BAD: Complex logic crammed into a lambda
process = lambda x: x.strip().lower().replace(' ', '_') if x else 'unknown'
# GOOD: Named function with clear intent
def normalize_name(x):
if not x:
return 'unknown'
return x.strip().lower().replace(' ', '_')
```
**Why named functions are usually better:**
* They show up with their name in stack traces (lambdas show as ``)
* They can have docstrings
* They can contain multiple statements and early returns
* `ruff`/`flake8` flag lambda assignments (E731) for a reason
**Alternative: `operator` module for common operations:**
```python theme={null}
from operator import itemgetter, attrgetter
# Instead of: sorted(items, key=lambda x: x['name'])
sorted(items, key=itemgetter('name')) # Faster, clearer
sorted(users, key=attrgetter('last_login')) # Same for attributes
```
**What interviewers are really testing:** Judgment about when to use lambdas vs named functions.
**Red flag answer:** Using lambdas everywhere to show off, or assigning them to variables.
**Follow-up:**
* "When would you use `operator.itemgetter` instead of a lambda?"
* "A lambda appears in a stack trace as ``. Why is that a problem?"
* "Can a lambda contain an assignment? A conditional? Multiple expressions?"
Decorators are functions that take a function and return a modified function. But the real depth is understanding they are just **syntactic sugar for higher-order functions**, and knowing the production patterns.
**What `@decorator` actually does:**
```python theme={null}
@my_decorator
def func():
pass
# Is exactly equivalent to:
func = my_decorator(func)
```
**The proper decorator template (production-grade):**
```python theme={null}
import functools
def my_decorator(func):
@functools.wraps(func) # CRITICAL: preserves __name__, __doc__, __module__
def wrapper(*args, **kwargs):
# Before logic
result = func(*args, **kwargs)
# After logic
return result
return wrapper
```
**Without `@functools.wraps`**, the decorated function loses its name, docstring, and module -- which breaks introspection, `help()`, Sphinx docs, and debugging. This is the #1 mistake in decorator implementations.
**Decorator with arguments (the two-level pattern):**
```python theme={null}
def retry(max_attempts=3, delay=1):
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except Exception:
if attempt == max_attempts - 1:
raise
time.sleep(delay)
return wrapper
return decorator
@retry(max_attempts=5, delay=2)
def fetch_data(url):
return requests.get(url)
```
**Real-world decorator patterns:**
* `@functools.lru_cache(maxsize=128)` -- memoization with LRU eviction. Use for expensive pure functions.
* `@functools.cache` (Python 3.9+) -- unbounded memoization (simpler than `lru_cache`).
* `@app.route('/path')` (Flask) -- URL routing.
* `@login_required` (Django) -- authentication enforcement.
* `@pytest.fixture` -- test dependency injection.
* `@contextmanager` -- turn a generator into a context manager.
**Class-based decorators (for stateful decoration):**
```python theme={null}
class CountCalls:
def __init__(self, func):
functools.update_wrapper(self, func)
self.func = func
self.count = 0
def __call__(self, *args, **kwargs):
self.count += 1
return self.func(*args, **kwargs)
```
**What interviewers are really testing:** Whether you know `@functools.wraps`, can write decorators with arguments, and have used decorators in production.
**Red flag answer:** Writing a decorator without `@functools.wraps` -- this is the litmus test.
**Follow-up:**
* "What happens if you forget `@functools.wraps`? What breaks in production?"
* "Write a decorator that takes arguments (e.g., `@retry(max_attempts=3)`)."
* "How would you implement `@lru_cache` from scratch?"
A closure is a function that captures and remembers variables from its enclosing scope, even after the enclosing function has finished executing. The closure "closes over" those variables.
**How it works internally:**
```python theme={null}
def make_multiplier(factor):
def multiply(x):
return x * factor # 'factor' is a free variable, captured by closure
return multiply
double = make_multiplier(2)
triple = make_multiplier(3)
print(double(5)) # 10
print(triple(5)) # 15
print(double.__closure__[0].cell_contents) # 2 -- the captured value
```
Python stores captured variables in `__closure__` as cell objects. The closure holds a *reference* to the variable, not a copy of the value. This leads to a famous gotcha:
**The late-binding closure bug:**
```python theme={null}
# BUG: All functions return 4 (the final value of i)
functions = []
for i in range(5):
functions.append(lambda: i)
print([f() for f in functions]) # [4, 4, 4, 4, 4]
# FIX 1: Default argument captures current value
functions = []
for i in range(5):
functions.append(lambda i=i: i)
print([f() for f in functions]) # [0, 1, 2, 3, 4]
# FIX 2: Use functools.partial
from functools import partial
functions = [partial(lambda i: i, i) for i in range(5)]
```
This bug happens because the closure captures the *variable* `i`, not its value at closure creation time. When the lambda is called, it reads the current value of `i`, which is 4 after the loop finishes.
**Real-world closure use cases:**
* Factory functions (as shown above)
* Callback registration with context
* Implementing decorators (decorators are closures)
* Data encapsulation (private state without classes)
* Partial application of functions
**What interviewers are really testing:** Whether you understand the late-binding gotcha and can explain what `__closure__` contains.
**Red flag answer:** Defining closures correctly but not knowing about the loop variable capture bug.
**Follow-up:**
* "Why does `[lambda: i for i in range(5)]` produce five functions that all return 4? How do you fix it?"
* "What is stored in `func.__closure__`? How can you inspect it?"
* "How are closures related to decorators?"
Generator functions use `yield` to produce values lazily, one at a time, pausing execution between yields. They are the foundation of Python's memory-efficient iteration.
**How generators work under the hood:**
```python theme={null}
def fibonacci():
a, b = 0, 1
while True:
yield a # Pauses here, returns value
a, b = b, a + b # Resumes here on next() call
gen = fibonacci() # No code executes yet -- returns generator object
next(gen) # 0 -- executes until first yield
next(gen) # 1 -- resumes after yield, executes until next yield
next(gen) # 1
```
The generator's execution frame is frozen on `yield` and thawed on `next()`. Local variables, instruction pointer, and stack are all preserved.
**Why generators matter (real numbers):**
```python theme={null}
# List: allocates ALL 10M items in memory at once
data = [process(x) for x in range(10_000_000)] # ~400MB for ints
# Generator: processes ONE item at a time, ~negligible memory
data = (process(x) for x in range(10_000_000)) # ~120 bytes
```
This difference is the reason `range()` in Python 3 returns a lazy object, and why tools like `csv.reader()` yield rows one at a time.
**`yield` vs `return`:**
* `return` terminates the function and sends one value
* `yield` suspends the function and sends one value, resuming on next call
* A generator can `yield` many times. After the function body is exhausted, `StopIteration` is raised.
**`send()` and bidirectional communication:**
```python theme={null}
def accumulator():
total = 0
while True:
value = yield total # yield sends total, then pauses; send() resumes with value
total += value
acc = accumulator()
next(acc) # Initialize (runs to first yield) -- returns 0
acc.send(10) # Returns 10
acc.send(20) # Returns 30
```
**Generator pipelines (Unix-pipe style):**
```python theme={null}
def read_lines(path):
with open(path) as f:
for line in f:
yield line.strip()
def filter_errors(lines):
for line in lines:
if "ERROR" in line:
yield line
def parse_timestamp(lines):
for line in lines:
yield line.split()[0]
# Pipeline processes one line at a time -- handles files larger than RAM
pipeline = parse_timestamp(filter_errors(read_lines("huge.log")))
```
**What interviewers are really testing:** Memory implications, `send()` for advanced use, and pipeline patterns.
**Red flag answer:** Explaining `yield` without mentioning memory benefits or knowing about `send()`.
**Follow-up:**
* "How would you process a 50GB log file without running out of memory?"
* "What does `generator.send(value)` do? When would you use it?"
* "What is the difference between a generator function and a generator expression?"
Function annotations (PEP 3107) provide metadata about parameters and return types. Type hints (PEP 484+) are the modern evolution, forming Python's gradual typing system. This is one of the biggest shifts in Python culture in the last decade.
**Basic syntax:**
```python theme={null}
def process_user(
name: str,
age: int,
emails: list[str], # Python 3.9+ (lowercase generics)
metadata: dict[str, Any] | None = None # Python 3.10+ (union with |)
) -> User:
...
```
**Key point: Type hints are NOT enforced at runtime.** Python ignores them entirely during execution. They are for:
1. **Static analysis**: `mypy`, `pyright`, `pytype` catch type errors before running code
2. **IDE support**: Autocomplete, refactoring, inline errors
3. **Documentation**: Self-documenting function signatures
4. **Runtime validation**: Libraries like `pydantic` use hints for runtime type checking and serialization
**The `typing` module essentials:**
```python theme={null}
from typing import (
Optional, # Optional[X] == X | None
Union, # Union[X, Y] == X | Y (use | in 3.10+)
Any, # Disables type checking for this value
TypeVar, # Generic type variable
Protocol, # Structural subtyping (duck typing for type checkers)
TypeAlias, # Explicit type alias
Literal, # Literal["read", "write"] -- exact values
TypedDict, # Dict with specific key types
)
```
**`typing.Protocol` -- the game changer (Python 3.8+):**
```python theme={null}
from typing import Protocol
class Readable(Protocol):
def read(self) -> str: ...
def process(source: Readable) -> str:
return source.read()
# ANY object with a read() -> str method satisfies this -- no inheritance needed
```
This formalizes duck typing for static analysis.
**Production adoption:**
* Major projects like FastAPI, SQLAlchemy 2.0, and Django 4.1+ are fully typed
* `mypy --strict` catches entire categories of bugs (null reference, wrong type, missing attributes)
* Some teams report 15-30% fewer production bugs after adopting strict type checking
* `pyright` (Microsoft, used in VS Code/Pylance) is faster and stricter than `mypy`
**What interviewers are really testing:** Whether you use type hints in practice and understand the tooling ecosystem.
**Red flag answer:** "Annotations are just comments for documentation." This misses the entire static analysis ecosystem.
**Follow-up:**
* "What is the difference between `mypy` and `pyright`? Which have you used?"
* "How does `typing.Protocol` differ from `abc.ABC`? When would you use each?"
* "How does Pydantic use type hints differently than mypy?"
Duck typing: "If it walks like a duck and quacks like a duck, it is a duck." Python determines an object's suitability by its *behavior* (methods and attributes), not its *type* (class hierarchy).
**How this differs from Java/C++:**
In Java, you must declare that a class implements an interface. In Python, any object that has the required methods just works:
```python theme={null}
class File:
def read(self): return "file data"
class HttpResponse:
def read(self): return "http data"
class MockData:
def read(self): return "test data"
def process(source):
return source.read() # Works with ALL three -- no shared base class needed
process(File()) # Works
process(HttpResponse()) # Works
process(MockData()) # Works -- great for testing!
```
**The Python standard library is built on duck typing protocols:**
* **Iterable protocol**: Has `__iter__()` -- works with `for` loops
* **Sequence protocol**: Has `__getitem__()` and `__len__()` -- works with indexing and `len()`
* **Context manager protocol**: Has `__enter__()` and `__exit__()` -- works with `with`
* **Callable protocol**: Has `__call__()` -- can be called like a function
* **Hashable protocol**: Has `__hash__()` -- can be dict key or set member
**The evolution: duck typing to structural typing:**
* Python 3.8+ `typing.Protocol` formalizes duck typing for static analysis
* You define the expected interface, and `mypy`/`pyright` verify it at check time -- without requiring inheritance
* This gives you the flexibility of duck typing with the safety of static type checking
**Trade-offs:**
* **Pro**: Extremely flexible, easy testing (mock anything), no rigid hierarchies
* **Con**: Runtime `AttributeError` if object does not have required methods, harder to discover expected interfaces without type hints
**What interviewers are really testing:** Whether you understand that Python's polymorphism model is fundamentally different from interface-based languages, and how `Protocol` bridges the gap.
**Red flag answer:** Explaining duck typing without connecting it to Python's protocol system or `typing.Protocol`.
**Follow-up:**
* "How would you make duck typing safer in a large codebase? What tools help?"
* "What is the difference between `typing.Protocol` and `abc.ABC`? When do you want each?"
* "Give me an example where duck typing makes testing dramatically easier than inheritance-based polymorphism."
## 5. File Handling and I/O
File I/O is fundamental, but production code requires understanding encoding, buffering, and resource management.
**The correct pattern (always use `with`):**
```python theme={null}
# Reading -- always specify encoding explicitly
with open('file.txt', 'r', encoding='utf-8') as f:
content = f.read() # Entire file into string (dangerous for large files)
# Line-by-line (memory-efficient for large files)
with open('huge.log', 'r', encoding='utf-8') as f:
for line in f: # File object is an iterator -- reads one line at a time
process(line)
# Writing
with open('output.txt', 'w', encoding='utf-8') as f:
f.write('Hello World\n')
# Binary mode (images, protobuf, pickle)
with open('image.png', 'rb') as f:
data = f.read() # Returns bytes, not str
```
**Why `with` is non-negotiable:**
Without `with`, if an exception occurs between `open()` and `f.close()`, the file handle leaks. In a long-running server processing thousands of requests, leaked file handles exhaust the OS limit (typically 1024 on Linux by default) and crash the process. The `with` statement guarantees `close()` via `__exit__`.
**The encoding trap:** `open()` without `encoding` uses the system default (often `cp1252` on Windows, `utf-8` on Linux). This causes cross-platform bugs. Always specify `encoding='utf-8'`. Python 3.15 will warn about missing encoding (PEP 686).
**Reading strategies by file size:**
* Small files (\<10MB): `f.read()` into memory
* Medium files (10MB-1GB): Iterate line-by-line with `for line in f`
* Large files (1GB+): Use `f.read(chunk_size)` in a loop, or `mmap` for random access
* Structured data: Use `csv`, `json`, `pandas.read_csv()` with `chunksize` parameter
**What interviewers are really testing:** Whether you handle encoding, large files, and resource cleanup correctly.
**Red flag answer:** Using `f = open(...)` without `with`, or calling `f.read()` on a potentially large file.
**Follow-up:**
* "Your script works on Linux but produces garbled text on Windows. What is likely wrong?"
* "How would you read a 50GB log file without loading it into memory?"
* "What happens if you forget to close a file in a web server handling 10,000 requests/second?"
**`'w'` (write)**: Creates file if it does not exist. **Truncates to zero length** if it does exist -- all previous content is destroyed immediately on `open()`, not on `write()`. This is the #1 way developers accidentally delete data.
**`'a'` (append)**: Creates file if it does not exist. Writes are always appended to end. Atomic on most POSIX systems -- multiple processes can append without corruption (this is why log files use append mode).
**The complete mode table (what most people miss):**
* `'r'` -- Read only. File must exist.
* `'w'` -- Write only. Creates or truncates.
* `'a'` -- Append only. Creates or appends.
* `'x'` -- Exclusive creation. Fails if file exists. **Use this for safe file creation** -- prevents overwriting.
* `'r+'` -- Read and write. File must exist. Does not truncate.
* `'w+'` -- Read and write. Creates or truncates.
* `'b'` suffix -- Binary mode (`'rb'`, `'wb'`). No encoding/decoding, returns `bytes`.
* `'t'` suffix -- Text mode (default). Applies encoding, returns `str`.
**Production safety patterns:**
```python theme={null}
# DANGEROUS: truncates on open, data lost if crash before write completes
with open('config.json', 'w') as f:
json.dump(config, f)
# SAFE: atomic write using temp file
import tempfile, os
with tempfile.NamedTemporaryFile('w', dir='.', delete=False, suffix='.tmp') as tmp:
json.dump(config, tmp)
tmp_path = tmp.name
os.replace(tmp_path, 'config.json') # Atomic rename on POSIX
```
**What interviewers are really testing:** Whether you know about data loss risks with `'w'` mode and safer alternatives.
**Red flag answer:** Only knowing `'w'` and `'a'` without mentioning `'x'` mode or atomic write patterns.
**Follow-up:**
* "Your app writes a config file with `'w'` mode. It crashes mid-write. What state is the file in?"
* "How do you safely overwrite a file without risking data loss?"
* "What is `'x'` mode and when would you use it?"
The `csv` module handles CSV parsing and writing, but production use involves edge cases that trip up most developers.
**Basic usage:**
```python theme={null}
import csv
# Reading
with open('data.csv', 'r', encoding='utf-8') as f:
reader = csv.DictReader(f) # Prefer DictReader -- access by column name
for row in reader:
print(row['name'], row['email'])
# Writing
with open('output.csv', 'w', encoding='utf-8', newline='') as f:
writer = csv.DictWriter(f, fieldnames=['name', 'age'])
writer.writeheader()
writer.writerow({'name': 'Alice', 'age': 30})
```
**The `newline=''` trap:** On Windows, without `newline=''`, you get double line endings in CSV output because the csv module writes `\r\n` and Python's text mode adds another `\r`. Always pass `newline=''` when opening files for csv writing.
**When to use `csv` vs `pandas`:**
* `csv` module: Simple parsing, streaming large files (memory efficient), no dependencies
* `pandas.read_csv()`: Data analysis, type inference, handling missing values, chunksize for large files, much faster for large datasets (C engine)
* `polars.read_csv()`: Even faster than pandas for pure CSV loading, no GIL limitations
**Edge cases that break naive parsers:** Fields containing commas, quotes within quoted fields, newlines within fields, different delimiters (TSV, pipe-separated). The `csv` module handles all of these correctly.
**What interviewers are really testing:** Whether you handle encodings, the newline parameter, and know when to graduate to pandas/polars.
**Red flag answer:** Parsing CSV by splitting on commas (`line.split(',')`) instead of using the csv module.
**Follow-up:**
* "Why does splitting on commas not work for real CSV files?"
* "How would you process a 10GB CSV file that does not fit in memory?"
* "When would you use `csv.DictReader` vs `pandas.read_csv()`?"
The `json` module serializes Python objects to JSON strings and back. But production use requires understanding its limitations and alternatives.
**Core API:**
```python theme={null}
import json
# Serialization (Python -> JSON)
data = {'name': 'Alice', 'age': 30, 'active': True}
json_str = json.dumps(data, indent=2) # To string
json.dump(data, open('data.json', 'w')) # To file
# Deserialization (JSON -> Python)
parsed = json.loads(json_str) # From string
parsed = json.load(open('data.json', 'r')) # From file
```
**Type mapping gotchas:**
* JSON has no integer type -- `json.loads('{"n": 12345678901234567890}')` converts to Python int, but JavaScript loses precision for integers > 2^53.
* JSON has no datetime, bytes, set, or tuple type. You need custom serialization.
* `json.dumps` fails on non-serializable types (datetime, Decimal, numpy arrays). Fix with `default` parameter:
```python theme={null}
import datetime
from decimal import Decimal
def json_serializer(obj):
if isinstance(obj, datetime.datetime):
return obj.isoformat()
if isinstance(obj, Decimal):
return str(obj)
raise TypeError(f"Not serializable: {type(obj)}")
json.dumps(data, default=json_serializer)
```
**Performance and alternatives:**
* `json` (stdlib): \~50MB/s. Fine for most use cases.
* `orjson` (third-party): \~1GB/s. Auto-serializes datetime, UUID, numpy. Used by FastAPI internally.
* `ujson`: \~300MB/s. Drop-in replacement.
* `msgpack`: Binary format, faster and smaller than JSON. Good for internal service communication.
**Security concern:** Never use `eval()` to parse JSON. Always use `json.loads()`. `eval()` executes arbitrary code.
**What interviewers are really testing:** Whether you know the type limitations, custom serialization, and when to use faster alternatives.
**Red flag answer:** Not knowing that `json.dumps` fails on datetime or Decimal objects.
**Follow-up:**
* "Your API returns a `Decimal` and `datetime`. How do you serialize them to JSON?"
* "When would you use `orjson` instead of the stdlib `json` module?"
* "`json.loads` is slow for your 500MB response. What are your options?"
Context managers implement the `with` statement protocol, guaranteeing resource cleanup even when exceptions occur. They are Python's answer to C++'s RAII and Java's try-with-resources.
**The protocol -- two magic methods:**
```python theme={null}
class DatabaseConnection:
def __enter__(self):
self.conn = connect_to_db()
return self.conn # Value bound by 'as' in 'with ... as'
def __exit__(self, exc_type, exc_val, exc_tb):
self.conn.close() # Always called, even if exception occurs
return False # False = re-raise exception, True = suppress it
```
**The simpler way -- `@contextmanager` decorator:**
```python theme={null}
from contextlib import contextmanager
@contextmanager
def timer(label):
start = time.time()
try:
yield # Code inside 'with' block runs here
finally:
elapsed = time.time() - start
print(f"{label}: {elapsed:.3f}s")
with timer("data processing"):
process_data() # Automatically timed
```
**Real-world context managers you should know:**
* `open()` -- file cleanup
* `threading.Lock()` -- lock acquire/release
* `contextlib.suppress(ExceptionType)` -- silently ignore specific exceptions
* `tempfile.TemporaryDirectory()` -- auto-cleanup temp directories
* `unittest.mock.patch()` -- mock scoping in tests
* `decimal.localcontext()` -- temporary decimal precision changes
* `contextlib.ExitStack()` -- manage dynamic number of context managers
**The `__exit__` return value matters:**
If `__exit__` returns `True`, the exception is suppressed (swallowed). If `False` (default), the exception propagates. Almost always return `False` -- suppressing exceptions silently is dangerous.
**What interviewers are really testing:** Whether you can write context managers and understand when to use class-based vs `@contextmanager`.
**Red flag answer:** Knowing `with` works with files but not being able to create your own context manager.
**Follow-up:**
* "Write a context manager that temporarily changes the working directory and restores it."
* "What does the return value of `__exit__` control? When would you return `True`?"
* "How would you manage a dynamic number of resources with `contextlib.ExitStack`?"
## 6. Exception Handling
Exception handling provides structured error recovery. But the real interview topic is the *philosophy* of how and when to use exceptions in Python.
**Python's EAFP vs LBYL philosophy:**
* **EAFP** (Easier to Ask Forgiveness than Permission) -- the Pythonic way. Try the operation, catch the exception if it fails:
```python theme={null}
# EAFP (Pythonic)
try:
value = my_dict[key]
except KeyError:
value = default
# LBYL (non-Pythonic in most cases)
if key in my_dict:
value = my_dict[key]
else:
value = default
```
EAFP is preferred because: (1) it avoids race conditions (the key could be deleted between the check and the access), (2) it is faster in the happy path (no extra lookup), and (3) it is more readable for complex conditions.
**Exception handling is NOT error hiding:**
```python theme={null}
# TERRIBLE -- hides all bugs, makes debugging impossible
try:
complex_operation()
except:
pass
# CORRECT -- catch specific exceptions, handle or log them
try:
user = fetch_user(user_id)
except ConnectionError:
logger.warning(f"DB connection failed for user {user_id}")
user = get_cached_user(user_id)
except UserNotFoundError:
raise HTTPException(status_code=404, detail="User not found")
```
**The Zen of Python says:** "Errors should never pass silently. Unless explicitly silenced." This means every `except` block should either handle the error meaningfully, log it, or re-raise it.
**What interviewers are really testing:** Whether you understand EAFP, catch specific exceptions, and never use bare `except: pass`.
**Red flag answer:** Using `try/except` as a replacement for input validation, or catching `Exception` everywhere.
**Follow-up:**
* "What is EAFP and why does Python prefer it over LBYL?"
* "When is LBYL actually better than EAFP? Give me a concrete case."
* "What is wrong with `except Exception as e: pass`?"
Python's exception hierarchy is a tree rooted at `BaseException`. Understanding the hierarchy matters more than memorizing names.
**The hierarchy (what most people miss):**
```
BaseException
+-- SystemExit # sys.exit() -- do NOT catch this
+-- KeyboardInterrupt # Ctrl+C -- do NOT catch this
+-- GeneratorExit # Generator cleanup
+-- Exception # Base for all "normal" exceptions
+-- StopIteration # Signals iterator exhaustion
+-- ArithmeticError
| +-- ZeroDivisionError
+-- LookupError
| +-- IndexError
| +-- KeyError
+-- OSError
| +-- FileNotFoundError
| +-- PermissionError
| +-- ConnectionError
+-- ValueError
+-- TypeError
+-- AttributeError
+-- RuntimeError
+-- RecursionError
```
**Critical rule:** Catch `Exception`, NEVER `BaseException`. Catching `BaseException` traps `KeyboardInterrupt` and `SystemExit`, making your program impossible to kill with Ctrl+C.
**Exception groups (Python 3.11+):**
`ExceptionGroup` wraps multiple exceptions raised concurrently (e.g., from `asyncio.TaskGroup`). Use `except*` to catch specific types from the group. This is a fundamental change for async error handling.
**What interviewers are really testing:** Whether you know the hierarchy, especially the `BaseException` vs `Exception` distinction.
**Red flag answer:** Listing exceptions without understanding the hierarchy or catching `BaseException`.
**Follow-up:**
* "Why should you never catch `BaseException`? What goes wrong?"
* "What are `ExceptionGroup` and `except*` in Python 3.11+?"
* "When would you catch `OSError` vs `FileNotFoundError` specifically?"
The full `try` statement has four blocks, each with a specific purpose. The `else` clause is the one most developers misunderstand.
```python theme={null}
try:
result = fetch_data(url) # Code that might fail
except ConnectionError as e:
logger.error(f"Connection failed: {e}")
result = get_cached_data() # Recovery logic
except ValueError as e:
raise InvalidDataError(f"Bad response: {e}") from e # Re-raise with chain
else:
save_to_database(result) # Only runs if NO exception in try block
logger.info("Data saved successfully")
finally:
cleanup_temp_files() # ALWAYS runs (even if return/break/continue in try)
```
**Why `else` exists (the misunderstood clause):**
Code in `else` only runs if `try` succeeds with no exceptions. Why not just put it in `try`? Because if `save_to_database` raises an exception, you do not want it caught by the `except ConnectionError` handler -- that would hide a database bug as a connection error.
**`finally` guarantees (and gotchas):**
* `finally` runs even if `try` has a `return` statement
* `finally` runs even if an exception is raised and not caught
* **Gotcha**: If `finally` has a `return`, it silently overrides the `try` block's return value and suppresses any active exception. Never put `return` in `finally`.
**Exception chaining (`from` keyword):**
```python theme={null}
try:
data = json.loads(response)
except json.JSONDecodeError as e:
raise ValidationError("Invalid JSON response") from e
# The original exception is preserved in __cause__ for debugging
```
This preserves the full traceback chain. Without `from e`, the original context is still available in `__context__` but the intent is less clear.
**What interviewers are really testing:** Whether you know what `else` is for and understand exception chaining.
**Red flag answer:** Not knowing what `else` does, or putting all logic in `try` instead of using `else`.
**Follow-up:**
* "What happens if `finally` contains a `return` statement?"
* "Why would you put code in `else` instead of at the end of `try`?"
* "What is exception chaining with `raise ... from ...`? When do you use it?"
Custom exceptions create a domain-specific error hierarchy for your application. This is essential for APIs and libraries.
**Production-grade custom exception pattern:**
```python theme={null}
class AppError(Exception):
"""Base exception for our application."""
def __init__(self, message: str, code: str = "UNKNOWN", details: dict = None):
self.message = message
self.code = code
self.details = details or {}
super().__init__(self.message)
class ValidationError(AppError):
"""Raised when input validation fails."""
def __init__(self, field: str, message: str):
super().__init__(
message=f"Validation failed for '{field}': {message}",
code="VALIDATION_ERROR",
details={"field": field}
)
class NotFoundError(AppError):
"""Raised when a requested resource does not exist."""
def __init__(self, resource: str, resource_id: str):
super().__init__(
message=f"{resource} '{resource_id}' not found",
code="NOT_FOUND",
details={"resource": resource, "id": resource_id}
)
```
**Design principles:**
1. **Create a base exception** for your library/app. Callers can catch `AppError` to handle all your exceptions.
2. **Always inherit from `Exception`**, never `BaseException`.
3. **Add structured data** (error codes, details dict) for machine-readable error handling, especially in APIs.
4. **Map to HTTP status codes** in web apps: `ValidationError` -> 400, `NotFoundError` -> 404, `AuthError` -> 401.
5. **Keep the hierarchy shallow** -- 2-3 levels max. Deep exception hierarchies are as bad as deep inheritance.
**Anti-pattern -- generic exceptions:**
```python theme={null}
# BAD: meaningless exception
raise Exception("Something went wrong")
# GOOD: specific, actionable
raise InsufficientFundsError(account_id=acct, requested=100, available=50)
```
**What interviewers are really testing:** Whether you design exception hierarchies for real applications, not just toy examples.
**Red flag answer:** Creating a custom exception that just inherits `Exception` with no additional context or structure.
**Follow-up:**
* "How would you design an exception hierarchy for a payment processing API?"
* "How do your custom exceptions map to HTTP status codes in a REST API?"
* "When is it better to return an error value (like `Result` type) vs raising an exception?"
`assert` tests conditions that should ALWAYS be true if the code is correct. They are a debugging tool, not an error handling mechanism.
**Syntax and behavior:**
```python theme={null}
assert condition, "Error message"
# Equivalent to:
if __debug__:
if not condition:
raise AssertionError("Error message")
```
**The critical rule: assertions are REMOVED by `python -O` (optimize flag).** Running `python -O script.py` sets `__debug__ = False` and strips all `assert` statements. This means:
```python theme={null}
# DANGEROUS: Using assert for input validation
def withdraw(amount):
assert amount > 0, "Amount must be positive" # Removed by -O flag!
self.balance -= amount # Oops, negative withdrawal goes through
# CORRECT: Use explicit validation
def withdraw(amount):
if amount <= 0:
raise ValueError("Amount must be positive")
self.balance -= amount
```
**When to use `assert`:**
* Internal invariants: `assert len(self._items) == self._count`
* Preconditions in internal functions (not public APIs)
* Sanity checks during development
* Test assertions: `assert result == expected` (pytest uses this)
**When NOT to use `assert`:**
* User input validation (use `ValueError`, `TypeError`)
* Security checks (use explicit conditionals)
* Anything that must run in production (assertions are stripped by `-O`)
**What interviewers are really testing:** Whether you know assertions are removed in optimized mode and should not be used for validation.
**Red flag answer:** Using assertions for input validation or not knowing about the `-O` flag.
**Follow-up:**
* "What happens to assertions when you run `python -O`? What breaks if you rely on them?"
* "When should you use `assert` vs `raise ValueError`?"
* "How does pytest use assertions differently from standard Python?"
## 7. Modules and Packages
**Module**: A single `.py` file. When imported, Python executes the file top-to-bottom and creates a module object. The module's functions, classes, and variables become attributes of that object.
**Package**: A directory containing modules and (typically) an `__init__.py` file. Packages create namespaces for organizing related modules hierarchically.
**What actually happens on import (the import machinery):**
1. Python checks `sys.modules` (import cache). If already imported, returns cached module.
2. Python searches `sys.path` (list of directories) to find the module.
3. Python creates a module object and executes the module's code.
4. The module is cached in `sys.modules`.
**This means:**
* Module code runs **exactly once** on first import, regardless of how many files import it. Subsequent imports return the cached module.
* Modules are effectively singletons -- this is why module-level state is shared across the entire application.
* `sys.path` determines where Python looks. It includes the script's directory, `PYTHONPATH` env var, and installed packages.
**Namespace packages (Python 3.3+):**
Directories without `__init__.py` can be namespace packages, allowing a single logical package to be split across multiple directories on disk. Used by large frameworks with plugin systems.
**What interviewers are really testing:** Whether you understand the import machinery (caching, `sys.modules`, execution order).
**Red flag answer:** "A module is a file, a package is a directory." Without understanding caching or `sys.path`.
**Follow-up:**
* "What happens if two different files import the same module? Does the module code run twice?"
* "How do you add a custom directory to Python's import search path?"
* "What are namespace packages and when would you use them?"
Python offers several import styles, each with different implications for readability, performance, and namespace management.
**Import styles and when to use each:**
```python theme={null}
import os # Full module -- use os.path.join()
from os.path import join, exists # Specific names -- use join() directly
from os.path import join as pjoin # Alias -- when name conflicts exist
import numpy as np # Community-standard aliases (np, pd, plt)
```
**Why `from module import *` is dangerous:**
* Pollutes the namespace with unknown names
* Can silently override existing variables
* Makes it impossible to know where a name came from during code review
* Only acceptable in `__init__.py` to re-export a public API, controlled by `__all__`
**Circular import resolution:**
```python theme={null}
# a.py imports b.py, b.py imports a.py -- ImportError or AttributeError
# Fix 1: Import inside function (lazy import)
def process():
from other_module import helper # Only imported when function is called
return helper()
# Fix 2: Restructure into a third module
# Fix 3: Use TYPE_CHECKING for type hints only
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from other_module import MyType # Only imported by type checkers, not at runtime
```
**Import performance:**
* First import executes the module (can be slow for heavy modules like `pandas` -- \~200ms)
* Subsequent imports are O(1) dict lookups in `sys.modules`
* Lazy imports (`importlib.import_module()` or in-function imports) can improve startup time
**What interviewers are really testing:** Whether you know about circular imports, `import *` dangers, and performance implications.
**Red flag answer:** Using `from module import *` in production code.
**Follow-up:**
* "How do you resolve circular imports? Give me three strategies."
* "What is `TYPE_CHECKING` and when do you use it?"
* "Your application takes 5 seconds to start. You suspect heavy imports. How do you diagnose it?"
`__init__.py` is the initializer for a Python package. It runs when the package is first imported.
**Key responsibilities:**
1. **Marks directory as a package** (required before Python 3.3, recommended after)
2. **Controls the public API** via `__all__`:
```python theme={null}
# mypackage/__init__.py
from .core import Engine
from .config import Config
from .exceptions import AppError
__all__ = ['Engine', 'Config', 'AppError'] # Controls 'from mypackage import *'
```
3. **Re-exports for convenience**: Users write `from mypackage import Engine` instead of `from mypackage.core import Engine`
4. **Package-level initialization**: Setup logging, register plugins, validate environment
**`__all__` does two things:**
* Controls what `from package import *` exports
* Signals to type checkers and IDEs what the public API is
**Common patterns in production:**
* Empty `__init__.py`: Most common. Just marks as package.
* Re-export pattern: Flatten nested imports for cleaner public API.
* Lazy loading: Import submodules on first access to improve startup time (used by large libraries like `numpy`).
**What interviewers are really testing:** Whether you understand `__all__`, re-exporting, and API design via `__init__.py`.
**Red flag answer:** "It makes a directory a package." Without knowing about `__all__` or re-exporting.
**Follow-up:**
* "How does `__all__` affect `from package import *`?"
* "How would you design `__init__.py` for a library with a clean public API?"
* "What are the pros and cons of an empty `__init__.py` vs one that re-exports?"
`__name__` is a special variable that Python sets depending on how the file is executed.
* **Executed directly** (`python script.py`): `__name__ == '__main__'`
* **Imported as module** (`import script`): `__name__ == 'script'`
**The standard guard pattern:**
```python theme={null}
def main():
# Application logic here
print("Running as main program")
if __name__ == '__main__':
main()
```
**Why this pattern matters:**
1. **Testability**: `import mymodule` does not trigger side effects. Tests can import and test individual functions.
2. **Reusability**: The module can be both a script and a library.
3. **Multiprocessing on Windows**: `multiprocessing` spawns new Python processes that import the module. Without the guard, the spawned process re-runs the main code, causing infinite process spawning.
**The `__main__.py` convention:**
For packages, `python -m mypackage` runs `mypackage/__main__.py`. This is how `python -m pytest`, `python -m http.server`, and `python -m venv` work.
**What interviewers are really testing:** Whether you know *why* the guard exists, especially the multiprocessing and testability reasons.
**Red flag answer:** Knowing the pattern without understanding why it prevents side effects on import.
**Follow-up:**
* "What happens on Windows with `multiprocessing` if you forget the `if __name__ == '__main__'` guard?"
* "What is `__main__.py` and how does `python -m package_name` use it?"
* "How does the guard pattern improve testability?"
Virtual environments isolate project dependencies, preventing conflicts between projects. This is non-negotiable for professional Python development.
**The modern tool landscape (2024+):**
**`uv`** (by Astral, the `ruff` creators) -- the new standard:
```bash theme={null}
uv venv # Create venv (10-100x faster than python -m venv)
source .venv/bin/activate # Activate (Linux/Mac)
uv pip install flask # Install (10-100x faster than pip)
uv pip compile requirements.in -o requirements.txt # Lock dependencies
```
`uv` is a drop-in replacement for `pip`, `pip-tools`, `virtualenv`, and `venv`. Written in Rust, it is dramatically faster.
**`poetry`** -- dependency management + packaging:
```bash theme={null}
poetry init # Create pyproject.toml
poetry add flask # Add dependency (auto-resolves versions)
poetry lock # Lock exact versions
poetry install # Install from lock file
```
**`pip` + `requirements.txt`** -- the classic approach:
```bash theme={null}
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip freeze > requirements.txt # Pin current versions
```
**Lock files matter (the real production lesson):**
Without pinned versions, `pip install flask` might install Flask 2.0 today and Flask 3.0 tomorrow, breaking your app. Always use lock files (`poetry.lock`, `requirements.txt` with pinned versions, `uv.lock`).
**`pyproject.toml`** (PEP 621) is the modern standard for project metadata, replacing `setup.py`, `setup.cfg`, and `requirements.txt` in many workflows.
**What interviewers are really testing:** Whether you have opinions about dependency management tooling and understand lock files.
**Red flag answer:** "I use `pip install` without a virtual environment." This is a career-ending answer for a senior role.
**Follow-up:**
* "What is the difference between `poetry` and `uv`? When would you choose each?"
* "Why are lock files important? What breaks without them?"
* "How would you set up a reproducible development environment for a new team member?"
## 8. Advanced Python Concepts
This is one of the most fundamental concepts in Python. Understanding the iterator protocol explains how `for` loops, comprehensions, `map`, `filter`, generators, and `zip` all work.
**The protocol:**
* **Iterable**: Any object with `__iter__()` method that returns an iterator. Lists, strings, dicts, files, generators are all iterables.
* **Iterator**: An object with both `__iter__()` (returns self) and `__next__()` (returns next value or raises `StopIteration`).
**What `for` loop actually does:**
```python theme={null}
# This:
for item in [1, 2, 3]:
print(item)
# Is equivalent to:
it = iter([1, 2, 3]) # Calls list.__iter__(), returns list_iterator
while True:
try:
item = next(it) # Calls iterator.__next__()
print(item)
except StopIteration:
break
```
**Building a custom iterator:**
```python theme={null}
class CountDown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self # Iterator returns itself
def __next__(self):
if self.current <= 0:
raise StopIteration
self.current -= 1
return self.current + 1
```
**Key distinction: iterable vs iterator.**
A list is iterable (you can get fresh iterators from it repeatedly). An iterator is a *one-pass* object -- once exhausted, it is done:
```python theme={null}
nums = [1, 2, 3] # Iterable -- can iterate multiple times
it = iter(nums) # Iterator -- single pass
list(it) # [1, 2, 3]
list(it) # [] -- exhausted!
```
Generators are iterators (single-pass). Files are iterators. This is why you can only loop through a generator once.
**What interviewers are really testing:** Whether you understand the protocol mechanics, not just the vocabulary.
**Red flag answer:** Saying "iterables are things you can iterate over" without explaining `__iter__` and `__next__`.
**Follow-up:**
* "What is the difference between an iterable and an iterator? Can something be both?"
* "Why can you loop over a list multiple times but only over a generator once?"
* "Build me an iterator class that yields Fibonacci numbers."
**Multithreading**: Multiple threads within a single process, sharing the same memory space. The GIL allows only one thread to execute Python bytecode at a time.
**Multiprocessing**: Multiple separate processes, each with its own Python interpreter and memory space. True parallel execution on multiple CPU cores.
**Decision matrix:**
| | Threading | Multiprocessing | asyncio |
| ----------------- | ------------------------- | -------------------------------------- | --------------------------- |
| Best for | I/O-bound (network, disk) | CPU-bound (computation) | High-concurrency I/O |
| Memory | Shared (lightweight) | Separate (heavy \~30-50MB per process) | Shared (very lightweight) |
| GIL impact | Blocked for CPU work | No GIL issue (separate interpreters) | Single thread, no GIL issue |
| Communication | Shared objects + locks | Queues, Pipes (serialization overhead) | Coroutine chaining |
| Overhead per unit | \~100KB per thread | \~30-50MB per process | \~1KB per coroutine |
| Debugging | Hard (race conditions) | Medium (process isolation) | Medium (async stack traces) |
**Real-world examples with numbers:**
```python theme={null}
# I/O-bound: Scraping 100 URLs
# Sequential: ~100 seconds (1s per URL)
# Threading (10 threads): ~10 seconds
# asyncio (100 concurrent): ~1-2 seconds
# CPU-bound: Processing 1M images
# Sequential: ~1000 seconds
# Threading: ~1000 seconds (GIL prevents parallelism)
# Multiprocessing (8 cores): ~125 seconds
```
**The modern API (`concurrent.futures`):**
```python theme={null}
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
# Switch between threading and multiprocessing by changing one word
with ThreadPoolExecutor(max_workers=10) as executor:
results = list(executor.map(fetch_url, urls))
with ProcessPoolExecutor(max_workers=8) as executor:
results = list(executor.map(process_image, images))
```
**What interviewers are really testing:** Whether you can choose the right concurrency model based on the task type.
**Red flag answer:** "Use multithreading for parallelism." This shows fundamental GIL misunderstanding.
**Follow-up:**
* "Your image processing is slow. You add threads but it does not speed up. Why?"
* "What are the risks of shared memory in multithreading?"
* "How does `concurrent.futures` simplify switching between threads and processes?"
The GIL is a mutex in CPython that ensures only one thread executes Python bytecode at a time. It is the most misunderstood and debated feature of Python.
**Why the GIL exists:**
CPython's memory management uses reference counting. Without the GIL, every reference count update would need its own lock, which would be far slower for single-threaded code (the common case). The GIL was a pragmatic trade-off: simplify memory management at the cost of limiting multithreaded CPU parallelism.
**What the GIL does and does not prevent:**
* **Prevents**: True parallel execution of Python bytecode across threads
* **Does NOT prevent**: I/O parallelism (the GIL is released during I/O operations -- network, disk, sleep)
* **Does NOT prevent**: Parallel execution of C extensions (NumPy releases the GIL during array operations)
* **Does NOT prevent**: Multiprocessing (separate processes, separate GILs)
**The free-threaded Python initiative (PEP 703, Python 3.13+):**
Python 3.13 introduced an experimental build with the GIL disabled (`python --disable-gil`). This is the biggest change to CPython in decades. The `t` suffix in version tags (e.g., `python3.13t`) indicates the free-threaded build. As of 3.13, it is experimental; expected to become default in 3-5 years.
**Workarounds used in production today:**
1. `multiprocessing` / `ProcessPoolExecutor` for CPU-bound work
2. `asyncio` for I/O-bound concurrency
3. C extensions (NumPy, Cython) that release the GIL
4. Use a different Python implementation (PyPy has the GIL; Jython and GraalPython do not)
**What interviewers are really testing:** Whether you understand the GIL's actual impact (limited to CPU-bound threading) and know the workarounds.
**Red flag answer:** "The GIL makes Python slow." The GIL only limits CPU-bound threading. Single-threaded Python is not affected.
**Follow-up:**
* "If the GIL limits threading, why does Python even have a `threading` module?"
* "What is PEP 703 (free-threaded Python) and what is its status?"
* "How does NumPy achieve parallelism despite the GIL?"
Metaclasses are "classes of classes." Just as an object is an instance of a class, a class is an instance of a metaclass. The default metaclass is `type`.
**The mind-bending truth:**
```python theme={null}
class Foo:
pass
type(Foo) # -- Foo is an instance of type
type(type) # -- type is an instance of itself
isinstance(Foo, type) # True -- classes are objects too
```
**How class creation works:**
When Python encounters `class Foo(Base, metaclass=MyMeta):`, it calls `MyMeta('Foo', (Base,), namespace_dict)`. The metaclass controls:
* `__new__`: Creates the class object
* `__init__`: Initializes the class object
* `__call__`: Controls what happens when the class is instantiated
**When metaclasses are actually used (rare but powerful):**
1. **ORMs**: Django's `Model` metaclass automatically creates database fields from class attributes
2. **API frameworks**: Abstract base classes use `ABCMeta` to enforce abstract method implementation
3. **Singleton pattern**: Metaclass that caches instances
4. **Automatic registration**: Metaclass that registers all subclasses in a registry
5. **Validation**: Ensure class definitions follow specific rules at definition time
**Tim Peters' quote (creator of Timsort and the Zen):** "Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't."
**Modern alternatives that replaced most metaclass use cases:**
* `__init_subclass__` (Python 3.6+) -- hook called when a class is subclassed. Handles most registration/validation patterns without a metaclass.
* `__set_name__` (Python 3.6+) -- descriptor hook for attribute naming.
* `@dataclass` and `typing.Protocol` -- handle many patterns that previously required metaclasses.
**What interviewers are really testing:** Whether you understand the object model deeply enough to explain metaclasses, AND whether you have the judgment to know when simpler alternatives exist.
**Red flag answer:** Either "I do not know what metaclasses are" (knowledge gap) or immediately reaching for metaclasses when `__init_subclass__` would suffice (poor judgment).
**Follow-up:**
* "When would you use `__init_subclass__` instead of a metaclass?"
* "How does Django's `Model` class use metaclasses to create database tables from class definitions?"
* "Write a metaclass that enforces all subclass method names are lowercase."
Coroutines enable cooperative multitasking -- a single thread handles many concurrent I/O operations by switching between them when one is waiting. This is Python's answer to the C10K problem.
**The mental model:**
```python theme={null}
import asyncio
import aiohttp
async def fetch_url(session, url):
async with session.get(url) as response: # I/O wait -- yield control
return await response.text() # I/O wait -- yield control
async def main():
async with aiohttp.ClientSession() as session:
# All 100 requests run concurrently on a SINGLE thread
tasks = [fetch_url(session, url) for url in urls]
results = await asyncio.gather(*tasks)
asyncio.run(main())
```
**How it works:**
1. `async def` creates a coroutine function (calling it returns a coroutine object, does not execute it)
2. `await` suspends the coroutine and yields control back to the event loop
3. The event loop runs other ready coroutines while this one waits for I/O
4. When I/O completes, the event loop resumes the coroutine
**Key asyncio patterns:**
* `asyncio.gather(*coros)` -- Run multiple coroutines concurrently, return all results
* `asyncio.TaskGroup()` (Python 3.11+) -- Structured concurrency with proper error handling
* `asyncio.Queue` -- Producer-consumer patterns
* `async for` -- Iterate over async iterators
* `async with` -- Async context managers
**When async is worth the complexity:**
* Web servers handling thousands of concurrent connections (FastAPI, Starlette)
* API clients making many concurrent requests
* Chat applications, WebSocket servers
* NOT for CPU-bound work (async is single-threaded)
**The "function color" problem:** Once you use `async`, it infects your entire call stack. You cannot call an `async` function from a regular function without `asyncio.run()`. This leads to "async all the way up" or painful refactoring.
**What interviewers are really testing:** Whether you understand the event loop model and when async is the right choice.
**Red flag answer:** "Async makes things run in parallel." It does not -- it is concurrent, not parallel. One thread, cooperative scheduling.
**Follow-up:**
* "What is the difference between concurrency and parallelism? Where does asyncio fit?"
* "What is the 'function color' problem with async/await?"
* "When would you use `asyncio.TaskGroup` vs `asyncio.gather`?"
All four are syntactic sugar for creating collections from iterables, but they differ in output type and memory behavior.
**Side-by-side comparison:**
```python theme={null}
data = range(10)
# List comprehension -- creates full list in memory
squares_list = [x**2 for x in data] # [0, 1, 4, 9, ...]
# Dict comprehension -- creates full dict in memory
squares_dict = {x: x**2 for x in data} # {0: 0, 1: 1, 2: 4, ...}
# Set comprehension -- creates full set in memory (deduplicates)
squares_set = {x % 3 for x in data} # {0, 1, 2}
# Generator expression -- lazy, produces one value at a time
squares_gen = (x**2 for x in data) #
```
**The critical memory difference:**
```python theme={null}
# This allocates ~800MB for 100M integers
big_list = [x for x in range(100_000_000)]
# This uses ~120 bytes regardless of size
big_gen = (x for x in range(100_000_000))
```
**When to use each:**
* **List comprehension**: When you need the full collection (indexing, slicing, multiple iterations, `len()`)
* **Dict comprehension**: Transforming/filtering dicts, inverting key-value pairs
* **Set comprehension**: Extracting unique values from a collection
* **Generator expression**: When you only iterate once, especially for large/infinite sequences. Also when feeding directly into `sum()`, `min()`, `max()`, `any()`, `all()` -- no intermediate list needed.
**Pro tip:** Generator expressions can be passed directly to functions without extra parentheses:
```python theme={null}
total = sum(x**2 for x in range(1000)) # No need for sum([x**2 for ...])
```
**Scope gotcha (Python 3 only):** Comprehensions have their own scope. The loop variable does not leak:
```python theme={null}
x = "before"
[x for x in range(5)]
print(x) # "before" in Python 3, 4 in Python 2
```
**What interviewers are really testing:** Whether you choose the right comprehension type based on memory and access patterns.
**Red flag answer:** Using list comprehensions when a generator expression would be more memory-efficient.
**Follow-up:**
* "Why does `sum(x**2 for x in range(10))` not need square brackets?"
* "You have a 100GB dataset. How do you process it with comprehensions?"
* "What is the scope of the loop variable in a comprehension? Does it differ between Python 2 and 3?"
Python uses a two-part memory management system: **reference counting** (primary) and **cyclic garbage collector** (secondary, for circular references).
**1. Reference counting (immediate cleanup):**
Every object has a reference count. When it reaches zero, memory is freed immediately:
```python theme={null}
a = [1, 2, 3] # refcount = 1
b = a # refcount = 2
del a # refcount = 1
del b # refcount = 0 -> immediately freed
```
You can inspect reference counts with `sys.getrefcount(obj)` (returns count + 1 because the function call itself creates a temporary reference).
**2. Cyclic garbage collector (for circular references):**
Reference counting cannot handle cycles:
```python theme={null}
a = []
b = []
a.append(b) # a references b
b.append(a) # b references a -- circular reference!
del a, b # refcounts are 1 (not 0) -- memory leaked without cyclic GC
```
The cyclic GC uses a **generational** algorithm with three generations:
* **Gen 0**: Newly created objects. Collected most frequently (threshold: \~700 objects).
* **Gen 1**: Survived one collection. Collected less frequently.
* **Gen 2**: Long-lived objects. Collected rarely.
Objects that survive collection are promoted to the next generation. This is based on the "generational hypothesis": most objects die young.
**Production implications:**
* GC pauses can cause latency spikes in real-time systems. Instagram famously disabled the cyclic GC in their Django servers because it caused 10% of CPU time to be spent on GC.
* `gc.disable()` is safe IF you avoid circular references (use weak references or restructure data).
* `gc.collect()` forces a collection -- useful before memory-critical sections.
* `gc.get_threshold()` returns `(700, 10, 10)` -- the generation thresholds.
* `__del__` finalizers complicate GC (objects with `__del__` in a cycle were uncollectable before Python 3.4).
**What interviewers are really testing:** Whether you understand both mechanisms and the generational approach.
**Red flag answer:** Only knowing reference counting, or thinking Python uses Java-style stop-the-world GC.
**Follow-up:**
* "Why did Instagram disable Python's garbage collector? What did they do instead?"
* "How does the generational GC decide when to run?"
* "What happens if two objects in a reference cycle both have `__del__` methods?"
## 9. Web Development with Python
The three major frameworks represent fundamentally different philosophies:
**Django** -- "The web framework for perfectionists with deadlines":
* Batteries-included: ORM, admin panel, auth, forms, templating, migrations, security middleware
* Opinionated: One way to do things. Enforces patterns (MTV -- Model/Template/View).
* Best for: Content sites, admin-heavy apps, rapid prototyping, teams that want conventions over configuration
* Used by: Instagram (until \~2019 at enormous scale), Mozilla, Pinterest, Disqus
* Trade-off: Monolithic, harder to swap components, ORM can be a bottleneck at extreme scale
**Flask** -- "Micro does not mean limited":
* Minimalist core: routing + WSGI + Jinja2 templates. Everything else is extensions.
* Unopinionated: Choose your own ORM, auth, etc.
* Best for: APIs, microservices, projects where you want full control
* Used by: Netflix (some services), LinkedIn, Reddit (early)
* Trade-off: More decisions to make, less built-in security, can lead to inconsistent patterns in large teams
**FastAPI** -- "Modern, fast, ready for production":
* Built on Starlette (async) + Pydantic (validation). Native async support.
* Automatic OpenAPI/Swagger docs from type hints
* Best for: APIs (especially when performance matters), async workloads, ML model serving
* Performance: \~3-5x faster than Flask for I/O-bound workloads due to async
* Trade-off: Smaller ecosystem than Django, async complexity, less mature
**The decision framework:**
* Need admin panel, auth, ORM out of the box? **Django**
* Need lightweight API with full control? **Flask**
* Need async, auto-docs, type validation? **FastAPI**
* Need maximum performance? **FastAPI** or Starlette
* Existing team knows Django? Probably keep using **Django**
**What interviewers are really testing:** Whether you can justify a framework choice based on requirements, not just personal preference.
**Red flag answer:** Only knowing one framework, or choosing based on "I like it" without trade-off analysis.
**Follow-up:**
* "You are starting a new API project. Walk me through how you choose between Flask, FastAPI, and Django REST Framework."
* "When would you choose Django over FastAPI despite FastAPI being 'faster'?"
* "What is WSGI vs ASGI? Why does it matter?"
A production-grade Flask app looks very different from the "Hello World" tutorial:
**Tutorial version (what everyone shows):**
```python theme={null}
from flask import Flask
app = Flask(__name__)
@app.route('/')
def home():
return 'Hello, World!'
```
**Production version (what you should actually discuss):**
```python theme={null}
# Application factory pattern (testable, configurable)
from flask import Flask
def create_app(config_name='default'):
app = Flask(__name__)
app.config.from_object(config[config_name])
# Register blueprints (modular route organization)
from .api import api_bp
app.register_blueprint(api_bp, url_prefix='/api/v1')
# Error handlers
@app.errorhandler(404)
def not_found(error):
return {"error": "Not found"}, 404
return app
```
**Key production concerns:**
* **Application factory** pattern: Do not create `app` at module level. Use a factory function for testability (create fresh app per test).
* **Blueprints**: Organize routes into modules. Without them, a 50-endpoint API becomes unmaintainable.
* **WSGI server**: Never use `app.run()` in production. Use Gunicorn: `gunicorn -w 4 "myapp:create_app()"`.
* **Request validation**: Use `marshmallow` or `flask-pydantic` for input validation.
* **Configuration**: Environment-based config (dev/staging/prod) via environment variables.
**What interviewers are really testing:** Whether you know the application factory pattern and production deployment.
**Red flag answer:** Showing only the "Hello World" example. In an interview, always discuss the factory pattern.
**Follow-up:**
* "What is the application factory pattern and why is it important for testing?"
* "How do you deploy a Flask app in production? What WSGI server do you use?"
* "How would you structure a Flask app with 50+ endpoints?"
Django ORM maps Python classes to database tables and translates Python operations into SQL. It is both powerful and a common source of performance problems.
**The model-to-SQL mapping:**
```python theme={null}
class User(models.Model):
name = models.CharField(max_length=100)
email = models.EmailField(unique=True)
created_at = models.DateTimeField(auto_now_add=True)
# QuerySet API (lazy -- SQL not executed until evaluated)
active_users = User.objects.filter(is_active=True).order_by('-created_at')[:10]
# Generates: SELECT * FROM user WHERE is_active = true ORDER BY created_at DESC LIMIT 10
```
**The N+1 query problem (most common ORM performance issue):**
```python theme={null}
# BAD: N+1 queries (1 query for posts + N queries for authors)
for post in Post.objects.all():
print(post.author.name) # Each access triggers a separate SQL query
# GOOD: 2 queries total using select_related (JOIN)
for post in Post.objects.select_related('author').all():
print(post.author.name)
# GOOD: 2 queries for many-to-many using prefetch_related
for post in Post.objects.prefetch_related('tags').all():
print(post.tags.all())
```
At Disqus, fixing N+1 queries reduced page load time from 3 seconds to 200ms on some endpoints.
**When the ORM is not enough:**
* Complex aggregations: Use `.annotate()` and `F()` expressions
* Raw SQL: `User.objects.raw('SELECT ...')` for complex queries the ORM cannot express
* Database-specific features: Use `django.db.connection` for raw cursor access
* High-volume writes: Use `bulk_create()` and `bulk_update()` instead of saving one object at a time
**What interviewers are really testing:** Whether you know about N+1 queries, `select_related`/`prefetch_related`, and when to bypass the ORM.
**Red flag answer:** Describing the ORM without mentioning N+1 queries or performance implications.
**Follow-up:**
* "What is the N+1 query problem? How do you detect and fix it in Django?"
* "What is the difference between `select_related` and `prefetch_related`?"
* "When would you use raw SQL instead of the ORM?"
The `requests` library is the standard for synchronous HTTP calls, but production code requires handling timeouts, retries, connection pooling, and async alternatives.
**Basic usage with production-grade error handling:**
```python theme={null}
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
# Session with connection pooling and retries
session = requests.Session()
retry = Retry(total=3, backoff_factor=0.5, status_forcelist=[500, 502, 503])
session.mount('https://', HTTPAdapter(max_retries=retry))
try:
response = session.get(
'https://api.example.com/data',
timeout=(3.05, 27), # (connect_timeout, read_timeout) -- ALWAYS set timeouts
headers={"Authorization": f"Bearer {token}"}
)
response.raise_for_status() # Raises HTTPError for 4xx/5xx
data = response.json()
except requests.ConnectionError:
logger.error("Failed to connect to API")
except requests.Timeout:
logger.error("API request timed out")
except requests.HTTPError as e:
logger.error(f"API returned error: {e.response.status_code}")
```
**The #1 production mistake: forgetting timeouts.** Without `timeout`, `requests.get()` will hang *indefinitely* if the server never responds. This has caused production outages at many companies.
**Async alternative (`httpx` or `aiohttp`):**
```python theme={null}
import httpx
async with httpx.AsyncClient() as client:
response = await client.get('https://api.example.com/data', timeout=10)
```
`httpx` is a modern alternative that supports both sync and async, with HTTP/2 support.
**What interviewers are really testing:** Timeouts, retries, error handling, and connection pooling.
**Red flag answer:** Using `requests.get()` without timeout or error handling.
**Follow-up:**
* "What happens if you do not set a timeout on `requests.get()`?"
* "How would you implement exponential backoff retry logic?"
* "When would you use `httpx` or `aiohttp` instead of `requests`?"
REST (Representational State Transfer) is an architectural style for networked applications. But the real interview depth is in understanding REST constraints and knowing what "RESTful" actually means vs how APIs work in practice.
**The six REST constraints:**
1. **Client-Server**: Separation of concerns
2. **Stateless**: Each request contains all needed information (no server-side session)
3. **Cacheable**: Responses must declare if cacheable
4. **Uniform Interface**: Consistent URL structure, standard HTTP methods
5. **Layered System**: Client cannot tell if connected directly to server
6. **Code on Demand** (optional): Server can extend client with executable code
**HTTP methods and idempotency (what most people get wrong):**
* `GET` -- Read. Safe (no side effects). Idempotent. Cacheable.
* `POST` -- Create. NOT idempotent (calling twice creates two resources).
* `PUT` -- Full replace. Idempotent (calling twice produces same result).
* `PATCH` -- Partial update. NOT necessarily idempotent.
* `DELETE` -- Remove. Idempotent (deleting twice, second is a no-op).
**Good URL design:**
```
GET /api/v1/users -- List users
POST /api/v1/users -- Create user
GET /api/v1/users/123 -- Get specific user
PUT /api/v1/users/123 -- Replace user
PATCH /api/v1/users/123 -- Update user fields
DELETE /api/v1/users/123 -- Delete user
GET /api/v1/users/123/orders -- List user's orders (nested resource)
```
**Status codes that matter:**
* `200` OK, `201` Created, `204` No Content (successful DELETE)
* `400` Bad Request, `401` Unauthorized, `403` Forbidden, `404` Not Found, `409` Conflict
* `422` Unprocessable Entity (validation error -- common with FastAPI)
* `429` Too Many Requests (rate limiting)
* `500` Internal Server Error, `503` Service Unavailable
**What interviewers are really testing:** Idempotency, proper status codes, and URL design -- not just CRUD definitions.
**Red flag answer:** "REST uses GET, POST, PUT, DELETE." Without understanding idempotency or status code semantics.
**Follow-up:**
* "What does idempotent mean? Which HTTP methods are idempotent?"
* "Your API returns 200 for everything, even errors. What is wrong with this?"
* "How would you design a REST API for a file upload that supports resumable uploads?"
## 10. Data Science and Libraries
NumPy provides the multi-dimensional array (`ndarray`) that underlies virtually all numerical computing in Python. It is fast because array operations run in compiled C code, not interpreted Python.
**Why NumPy is 10-100x faster than Python lists:**
```python theme={null}
import numpy as np
# Python list: ~500ms for 10M elements
python_list = list(range(10_000_000))
result = [x * 2 for x in python_list]
# NumPy array: ~15ms for 10M elements (30x faster)
np_array = np.arange(10_000_000)
result = np_array * 2 # Vectorized operation -- no Python loop
```
The speedup comes from: (1) contiguous memory layout (cache-friendly), (2) C-level loops (no Python interpreter overhead per element), (3) SIMD vectorization, (4) no type checking per element (homogeneous dtype).
**Essential concepts:**
* **Broadcasting**: Operations between arrays of different shapes without copying:
```python theme={null}
a = np.array([[1, 2, 3],
[4, 5, 6]]) # Shape (2, 3)
b = np.array([10, 20, 30]) # Shape (3,)
result = a + b # Broadcasting: b is "stretched" to (2, 3)
# [[11, 22, 33], [14, 25, 36]]
```
* **Vectorization**: Replace Python loops with array operations
* **Views vs copies**: Slicing creates a *view* (shared memory), not a copy. Modifying a slice modifies the original. Use `.copy()` for independent data.
* **dtype**: All elements share one type (`float64`, `int32`, etc.). This enables the performance gains.
**What interviewers are really testing:** Whether you understand *why* NumPy is fast (memory layout, vectorization) and can explain broadcasting.
**Red flag answer:** "NumPy is fast because it is written in C." This is one reason but misses memory layout and vectorization.
**Follow-up:**
* "Explain broadcasting. What happens when you add a (3,4) array and a (4,) array?"
* "What is the difference between a NumPy view and a copy? When does it matter?"
* "Why is `np.sum(arr)` faster than `sum(arr)` for a NumPy array?"
pandas provides high-level data manipulation built on NumPy. The two core structures are **Series** (1D) and **DataFrame** (2D).
**DataFrame is the workhorse:**
```python theme={null}
import pandas as pd
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'age': [30, 25, 35],
'city': ['NYC', 'LA', 'NYC']
})
# SQL-like operations
nyc_users = df[df['city'] == 'NYC'] # WHERE
avg_age_by_city = df.groupby('city')['age'].mean() # GROUP BY
merged = pd.merge(df, orders, on='name', how='left') # JOIN
```
**Performance gotchas that matter in production:**
* **Never iterate with `for row in df.iterrows()`** -- it is 100-1000x slower than vectorized operations. Use `.apply()` (better) or vectorized NumPy operations (best).
* **`chunksize` for large files**: `pd.read_csv('huge.csv', chunksize=10_000)` returns an iterator of DataFrames, enabling processing files larger than RAM.
* **`category` dtype for strings with few unique values**: Reduces memory by 90%+ for columns like "status" or "country".
* **`inplace=True` does NOT save memory** -- pandas internally creates a copy anyway. It is deprecated in newer pandas.
**When pandas is not enough:**
* **polars**: 5-10x faster than pandas for most operations. Written in Rust. No GIL limitations. Lazy evaluation. The future of Python dataframes.
* **Dask**: Parallel pandas for datasets larger than RAM. Distributed computing.
* **PySpark**: For truly massive datasets (TB+) on clusters.
**What interviewers are really testing:** Whether you know performance best practices and alternatives to pandas.
**Red flag answer:** Using `iterrows()` for DataFrame operations, or not knowing about `polars`.
**Follow-up:**
* "Why is `df.iterrows()` slow? What should you use instead?"
* "How would you process a 50GB CSV file with pandas?"
* "When would you choose polars over pandas?"
Missing data handling is where data engineering meets data science. The strategy depends entirely on context -- there is no universal correct approach.
**Detection:**
```python theme={null}
df.isnull().sum() # Count NaN per column
df.isnull().sum() / len(df) * 100 # Percentage missing per column
```
**Strategies and when to use each:**
1. **Drop rows** (`df.dropna()`): When missing data is \<5% and random (MCAR). Risky if missing data is systematic.
2. **Fill with constant** (`df.fillna(0)` or `df.fillna('Unknown')`): When a default value makes domain sense.
3. **Forward/backward fill** (`df.fillna(method='ffill')`): For time series data where the last known value is a reasonable proxy.
4. **Fill with mean/median** (`df.fillna(df.mean())`): When the distribution is roughly normal (mean) or skewed (median). Warning: reduces variance and can bias models.
5. **Interpolation** (`df.interpolate()`): For continuous time series.
6. **Model-based imputation** (scikit-learn's `SimpleImputer`, `KNNImputer`): When relationships between features can predict missing values.
**The critical gotcha -- missing data types:**
* `NaN` (NumPy `float('nan')`): The default for missing numeric data. `NaN != NaN` is `True` (IEEE 754).
* `None`: Python's null. Used in object-dtype columns.
* `pd.NA` (pandas 1.0+): The new nullable type. Works with integer columns without converting to float.
* Older pandas: Missing integers become floats because `NaN` is a float. `pd.Int64Dtype()` fixes this.
**What interviewers are really testing:** Whether you choose strategies based on domain context, not just what is easy.
**Red flag answer:** "I use `fillna(0)` for everything." This corrupts data when 0 has semantic meaning.
**Follow-up:**
* "Your temperature column has 20% missing values. `fillna(0)` would be wrong. What do you do?"
* "What is the difference between `NaN`, `None`, and `pd.NA` in pandas?"
* "How would you detect if missing data is random (MCAR) vs systematic (MNAR)?"
matplotlib is Python's foundational plotting library. It is powerful but verbose. The real interview depth is knowing its place in the visualization ecosystem and when to use alternatives.
**The two interfaces:**
```python theme={null}
import matplotlib.pyplot as plt
# 1. pyplot (MATLAB-like, quick and dirty)
plt.plot([1, 2, 3], [1, 4, 9])
plt.title("Quick Plot")
plt.show()
# 2. Object-oriented (recommended for production/reusable code)
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
axes[0].plot(x, y1, label='Revenue')
axes[0].set_title('Revenue Over Time')
axes[0].legend()
axes[1].bar(categories, values)
fig.tight_layout()
fig.savefig('report.png', dpi=150, bbox_inches='tight')
```
**When to use alternatives:**
* **seaborn**: Statistical visualizations (distributions, regressions, heatmaps). Built on matplotlib, much less code for complex plots.
* **plotly**: Interactive web-based charts. Zoom, hover, click. Great for dashboards and Jupyter notebooks.
* **Altair**: Declarative visualization based on Vega-Lite. Concise syntax, excellent for exploratory analysis.
* **bokeh**: Interactive web plots with Python backend. Similar to plotly.
**What interviewers are really testing:** Whether you know the OO interface (not just pyplot) and can choose the right visualization library.
**Red flag answer:** Only knowing the pyplot interface without the object-oriented API.
**Follow-up:**
* "What is the difference between the pyplot and object-oriented matplotlib interfaces?"
* "When would you use plotly vs matplotlib? What are the trade-offs?"
* "How do you create publication-quality figures with matplotlib?"
scikit-learn is Python's standard machine learning library. Its consistent API (`fit`/`predict`/`transform`) and comprehensive algorithm collection make it the go-to for classical ML.
**The universal API pattern:**
```python theme={null}
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train) # Train
predictions = model.predict(X_test) # Predict
print(classification_report(y_test, predictions))
```
Every algorithm follows this pattern. Switch from `RandomForestClassifier` to `LogisticRegression` by changing one line.
**The Pipeline pattern (production-critical):**
```python theme={null}
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipeline = Pipeline([
('scaler', StandardScaler()), # Preprocessing
('model', RandomForestClassifier()) # Model
])
pipeline.fit(X_train, y_train) # Applies all steps in order
pipeline.predict(X_test) # Applies all steps in order
```
Pipelines prevent **data leakage** (fitting the scaler on test data) -- a common and serious mistake in ML workflows.
**When scikit-learn is NOT enough:**
* Deep learning: Use PyTorch or TensorFlow
* Gradient boosting at scale: Use XGBoost, LightGBM, or CatBoost (faster than sklearn's GradientBoosting)
* Very large datasets (millions of rows): Some sklearn algorithms do not scale. Use `partial_fit` for incremental learning, or switch to Spark MLlib.
**What interviewers are really testing:** Whether you know the Pipeline pattern and can discuss trade-offs between algorithms.
**Red flag answer:** Knowing the API without understanding Pipelines or data leakage prevention.
**Follow-up:**
* "What is data leakage and how do Pipelines prevent it?"
* "When would you use XGBoost instead of sklearn's RandomForest?"
* "How would you handle a dataset with 100M rows that does not fit in memory?"
## 11. Additional Important Topics
Slicing extracts a portion of a sequence using `sequence[start:end:step]`. But the real depth is in understanding how slicing interacts with the object model.
**The basics:**
* `start` is inclusive, `end` is exclusive
* Negative indices count from end: `lst[-3:]` gets last 3 elements
* `step` controls stride: `lst[::2]` gets every other element
* Omitted values: `lst[:]` copies entire list, `lst[::-1]` reverses
**What slicing actually does under the hood:**
Slicing calls `__getitem__` with a `slice` object: `lst[1:4]` is equivalent to `lst.__getitem__(slice(1, 4))`. You can implement slicing in custom classes by accepting `slice` objects in `__getitem__`.
**Critical nuance -- shallow copy:**
```python theme={null}
original = [[1, 2], [3, 4]]
sliced = original[:] # Shallow copy -- new list, same nested objects
sliced[0][0] = 99
print(original[0][0]) # 99 -- original modified!
```
**Slice assignment (powerful but underused):**
```python theme={null}
lst = [1, 2, 3, 4, 5]
lst[1:3] = [20, 30, 40] # Replace slice with different-length iterable
# lst is now [1, 20, 30, 40, 4, 5]
lst[::2] = [0, 0, 0] # Replace every other element (must match length)
```
**What interviewers are really testing:** Whether you understand shallow copy behavior and slice objects.
**Red flag answer:** Only knowing basic slicing without mentioning shallow copy or `__getitem__`.
**Follow-up:**
* "What does `lst[:]` actually do? Is it a deep copy or shallow copy?"
* "How would you implement slicing in a custom class?"
* "What happens when you assign to a slice: `lst[1:3] = [10, 20, 30, 40]`?"
`==` calls `__eq__` to compare values. `is` compares memory addresses (object identity). This is covered in the identity operators question above -- the key addition here is **when each goes wrong.**
**The dangerous cases:**
```python theme={null}
# WRONG: Never use 'is' for value comparison
if x is 1: # Unreliable! Works for small ints due to interning, fails for large
if x is "hello": # Unreliable! Works for interned strings only
# CORRECT
if x == 1: # Always reliable for values
if x is None: # Always correct for None (singleton)
if x is True: # Technically correct but usually prefer 'if x:'
```
**Why `is` with strings seems to work (but should not be relied on):**
CPython interns string literals and identifier-like strings. So `"hello" is "hello"` is `True` in practice, but `"hello world" is "hello world"` may or may not be `True` depending on context. **Never rely on this behavior.**
**What interviewers are really testing:** Whether you use `is` correctly (only for `None` and sentinels).
**Red flag answer:** Using `is` for any value comparison besides `None`.
**Follow-up:**
* "Can `==` return `True` while `is` returns `False`? Give an example."
* "Why does `a = 256; b = 256; a is b` return `True` but `a = 257; b = 257; a is b` might return `False`?"
`and`, `or`, and `not` are logical operators, but what most people miss is that `and`/`or` return *values*, not booleans.
**Short-circuit evaluation with value return:**
```python theme={null}
# 'and' returns first falsy value, or last value if all truthy
0 and "hello" # Returns 0 (first falsy)
"hello" and "world" # Returns "world" (last value, all truthy)
# 'or' returns first truthy value, or last value if all falsy
0 or "hello" # Returns "hello" (first truthy)
"" or 0 or None # Returns None (all falsy, returns last)
```
**Real-world patterns using this behavior:**
```python theme={null}
# Default value pattern (before walrus operator existed)
name = user_input or "Anonymous"
# Guard clause pattern
result = data and data.get('key') # Returns None if data is falsy
# Conditional assignment
debug = os.environ.get('DEBUG') or False
```
**Gotcha:** `and`/`or` do NOT always return booleans. `bool(x and y)` returns a boolean, but `x and y` returns the actual value. This matters when you store the result.
**What interviewers are really testing:** Whether you know about value return behavior, not just `True`/`False` logic.
**Red flag answer:** "They return True or False." They return values.
**Follow-up:**
* "What does `[] or 'default'` return? What about `[1,2] or 'default'`?"
* "How does short-circuit evaluation work with `and`?"
`enumerate()` adds a counter to an iterable, yielding `(index, value)` tuples. It is essential for Pythonic code -- any time you need an index in a loop.
**Why it exists:**
```python theme={null}
# Non-Pythonic (C-style indexing)
for i in range(len(fruits)):
print(i, fruits[i])
# Pythonic
for i, fruit in enumerate(fruits):
print(i, fruit)
# With custom start index
for line_num, line in enumerate(file, start=1):
print(f"Line {line_num}: {line}")
```
**Under the hood:** `enumerate` is a lazy iterator. It does not create a list of tuples in memory. It yields one `(count, value)` tuple at a time.
**Advanced pattern -- enumerate with unpacking in comprehensions:**
```python theme={null}
indexed_data = {i: val for i, val in enumerate(data)}
```
**What interviewers are really testing:** Whether you use `enumerate` instead of `range(len(...))`.
**Red flag answer:** Using `for i in range(len(list)):` when `enumerate` would be cleaner.
**Follow-up:**
* "How does `enumerate` interact with generators? Does it consume the entire generator?"
* "What does the `start` parameter do, and when would you use it?"
`zip()` pairs up elements from multiple iterables, creating tuples of corresponding elements. It is fundamental for parallel iteration.
**Core behavior:**
```python theme={null}
names = ['Alice', 'Bob', 'Charlie']
ages = [30, 25, 35]
cities = ['NYC', 'LA', 'SF']
for name, age, city in zip(names, ages, cities):
print(f"{name}, {age}, {city}")
# Creating dicts from parallel lists
user_dict = dict(zip(names, ages)) # Pythonic dict construction
```
**Handling unequal lengths:**
```python theme={null}
# zip() silently truncates to shortest -- DATA LOSS risk
list(zip([1, 2, 3], ['a', 'b'])) # [(1, 'a'), (2, 'b')] -- 3 is lost!
# itertools.zip_longest fills missing values
from itertools import zip_longest
list(zip_longest([1, 2, 3], ['a', 'b'], fillvalue='?'))
# [(1, 'a'), (2, 'b'), (3, '?')]
# Python 3.10+ strict mode raises on length mismatch
list(zip([1, 2, 3], ['a', 'b'], strict=True)) # ValueError!
```
**The unzip pattern:**
```python theme={null}
pairs = [('Alice', 30), ('Bob', 25), ('Charlie', 35)]
names, ages = zip(*pairs) # Unpack with * -- transpose operation
```
**What interviewers are really testing:** Whether you know about the truncation behavior and `strict=True`.
**Red flag answer:** Not knowing that `zip` silently drops elements from longer iterables.
**Follow-up:**
* "You zip two lists of different lengths. What happens to the extra elements?"
* "How do you unzip a list of tuples? Explain the `zip(*)` pattern."
* "When would you use `zip_longest` vs `zip(..., strict=True)`?"
## 12. Core Programming Concepts
The key distinction: `append` treats its argument as a **single element**, `extend` treats its argument as an **iterable of elements**.
```python theme={null}
a = [1, 2, 3]
a.append([4, 5]) # a = [1, 2, 3, [4, 5]] -- list added as one element
a = [1, 2, 3]
a.extend([4, 5]) # a = [1, 2, 3, 4, 5] -- elements added individually
```
**Performance:**
* `append(x)`: O(1) amortized. Single element.
* `extend(iterable)`: O(k) where k is length of iterable. But faster than calling `append` k times because it avoids k separate method lookups and resizes once.
* `+= [items]` is equivalent to `extend`, not `append`.
**The gotcha with `extend` and strings:**
```python theme={null}
lst = [1, 2]
lst.extend("abc") # lst = [1, 2, 'a', 'b', 'c'] -- strings are iterable!
lst.append("abc") # lst = [1, 2, 'a', 'b', 'c', 'abc']
```
**What interviewers are really testing:** Whether you understand the iterable vs element distinction.
**Red flag answer:** "append adds one thing, extend adds many" without the string gotcha.
**Follow-up:**
* "What happens when you `extend` a list with a string?"
* "Is `lst += [1, 2]` the same as `lst.extend([1, 2])` or `lst.append([1, 2])`?"
Default arguments are evaluated **once at function definition time**, not at each call. This leads to Python's most infamous bug.
**The mutable default argument trap:**
```python theme={null}
# BUG: Default list is shared across ALL calls
def add_item(item, items=[]):
items.append(item)
return items
add_item(1) # [1]
add_item(2) # [1, 2] -- surprise! The default list persists.
add_item(3) # [1, 2, 3]
```
**The fix -- use `None` sentinel:**
```python theme={null}
def add_item(item, items=None):
if items is None:
items = []
items.append(item)
return items
```
**Why this happens:** The default value `[]` is evaluated once when `def` is executed (at import time). The list object is stored in `add_item.__defaults__`. Every call that uses the default shares the same list object.
**When mutable defaults are intentional (rare but valid):**
Caching/memoization patterns use mutable defaults deliberately:
```python theme={null}
def expensive_function(n, _cache={}):
if n not in _cache:
_cache[n] = compute(n)
return _cache[n]
```
This is a hack -- prefer `@functools.lru_cache` instead.
**What interviewers are really testing:** Whether you know about the mutable default bug and the `None` sentinel pattern.
**Red flag answer:** Not knowing about this trap, or not using `None` as sentinel.
**Follow-up:**
* "Why are default arguments evaluated at definition time, not call time?"
* "How can you inspect a function's default values?"
* "When would a mutable default argument be intentionally useful?"
Three ways to remove elements, each with different behavior and use cases:
| | `remove(value)` | `pop(index)` | `del lst[index]` |
| -------------------- | --------------------- | ------------------------------ | ------------------------------ |
| Removes by | **Value** | **Index** | **Index** |
| Returns removed item | No | Yes | No |
| On missing | `ValueError` | `IndexError` | `IndexError` |
| Complexity | O(n) -- linear search | O(n) from front, O(1) from end | O(n) from front, O(1) from end |
**When to use each:**
* `remove(x)`: When you know the value but not the index. Only removes first occurrence.
* `pop(i)`: When you need the removed value (e.g., stack operations: `stack.pop()`)
* `del lst[i]`: When you just want deletion by index without needing the value
* `del lst[1:3]`: Slice deletion -- removes a range
**Gotcha with `remove` in a loop:**
```python theme={null}
# BUG: Skips elements because list shrinks during iteration
lst = [1, 2, 2, 3]
for item in lst:
if item == 2:
lst.remove(item)
# lst = [1, 2, 3] -- one 2 remains!
# FIX: Use list comprehension
lst = [item for item in lst if item != 2]
```
**What interviewers are really testing:** Whether you know the loop-removal bug and complexity differences.
**Red flag answer:** Not mentioning the danger of `remove()` during iteration.
**Follow-up:**
* "Why does removing items from a list during iteration skip elements?"
* "What is the time complexity of `remove()` vs `pop()`?"
Python has three string formatting approaches. **f-strings are the standard** for new code (Python 3.6+).
**f-strings (formatted string literals) -- the modern way:**
```python theme={null}
name = "Alice"
age = 30
# Basic
f"{name} is {age} years old"
# Expressions
f"{name.upper()} is {age * 2} in dog years"
# Format specifiers
pi = 3.14159
f"{pi:.2f}" # "3.14" -- 2 decimal places
f"{1000000:,}" # "1,000,000" -- thousands separator
f"{0.75:.1%}" # "75.0%" -- percentage
f"{'hello':>20}" # " hello" -- right-aligned, padded
# Debugging (Python 3.8+)
x = 42
f"{x = }" # "x = 42" -- prints variable name and value!
```
**The `f"{x = }"` trick (Python 3.8+) is a game-changer for debugging.** It automatically shows the variable name alongside its value.
**When to use each approach:**
* **f-strings**: Default choice. Fast (compiled at parse time), readable, flexible.
* **`str.format()`**: When the format string is determined at runtime (e.g., loaded from a config file). f-strings cannot be reused as templates.
* **`%` formatting**: Legacy code only. Still common in `logging` module for performance reasons (string not formatted if log level is filtered out).
* **`string.Template`**: For user-provided format strings (safe -- no arbitrary code execution).
**Security note:** f-strings can execute arbitrary code: `f"{__import__('os').system('rm -rf /')}"`. Never build f-strings from user input. Use `string.Template` for user-provided format strings.
**What interviewers are really testing:** Whether you use f-strings, know the `=` debugging feature, and understand security implications.
**Red flag answer:** Using `%` formatting or `str.format()` when f-strings would be simpler.
**Follow-up:**
* "What is the `f'{x = }'` syntax in Python 3.8+? When would you use it?"
* "When would `str.format()` be better than an f-string?"
* "Why is f-string formatting a security risk with user input?"
`any()` and `all()` are built-in functions for checking conditions across iterables. They short-circuit for efficiency.
**Behavior:**
```python theme={null}
any([False, False, True]) # True -- at least one True
all([True, True, True]) # True -- all True
any([False, False, False]) # False
all([True, True, False]) # False
# Edge cases
any([]) # False (no element is True -- vacuous truth edge)
all([]) # True (no element is False -- vacuous truth)
```
**Real-world patterns:**
```python theme={null}
# Check if any user is admin
if any(user.is_admin for user in users):
grant_admin_access()
# Validate all required fields are present
required = ['name', 'email', 'password']
if all(field in request.data for field in required):
process_registration()
# Check if any file is too large (with early termination)
if any(f.size > MAX_SIZE for f in uploaded_files):
raise ValidationError("File too large")
```
**The short-circuit advantage:** With generator expressions, `any()` and `all()` stop as soon as the result is determined. `any` stops at first `True`, `all` stops at first `False`. This means you can use them with expensive checks and they will not evaluate everything unnecessarily.
**De Morgan's equivalence:**
* `not any(x)` is equivalent to `all(not x)` -- "none are true" == "all are false"
* `not all(x)` is equivalent to `any(not x)` -- "not all are true" == "some are false"
**What interviewers are really testing:** Whether you use these with generator expressions for elegant, efficient condition checking.
**Red flag answer:** Not knowing the empty iterable behavior or not using generator expressions with them.
**Follow-up:**
* "Why does `all([])` return `True`? Is this intuitive?"
* "How does short-circuit evaluation work with `any(expensive_check(x) for x in items)`?"
* "Rewrite this nested if-chain using `any()` or `all()`."
## 13. Python Internals and Memory Management
Python's memory management is a multi-layered system. This was covered in depth in Section 8 (garbage collector), but the internals go deeper.
**CPython's memory allocator hierarchy:**
1. **OS allocator** (`malloc`/`free`): Large allocations (>512 bytes)
2. **Python object allocator** (`pymalloc`): Small object allocations. Uses memory pools of fixed-size blocks (8, 16, 24, ... 512 bytes). This avoids calling `malloc` for every small object, which would be slow.
3. **Object-specific allocators**: `int`, `float`, `list`, `dict` each have optimized allocation strategies. For example, small ints (-5 to 256) are pre-allocated and reused.
**Memory pools and arenas:**
`pymalloc` manages memory in 256KB arenas, divided into 4KB pools, divided into fixed-size blocks. Objects of similar sizes share a pool. When all blocks in a pool are freed, the pool is returned to the arena for reuse.
**Practical memory management:**
* Use `sys.getsizeof(obj)` for shallow size of one object
* Use `pympler.asizeof` for deep/recursive size (includes referenced objects)
* `tracemalloc` module traces memory allocations to their source line
* `memory_profiler` shows line-by-line memory usage
* `gc.get_objects()` lists all tracked objects (useful for leak debugging)
**What interviewers are really testing:** Whether you know about `pymalloc`, memory profiling tools, and can debug memory issues.
**Red flag answer:** Only knowing about reference counting without understanding the allocator layers.
**Follow-up:**
* "How would you find a memory leak in a long-running Python service?"
* "What tools do you use to profile memory usage?"
* "Why does CPython have its own memory allocator instead of just using malloc?"
Python uses neither pass-by-value nor pass-by-reference. It uses **pass-by-object-reference** (also called "pass-by-assignment"). Understanding this eliminates an entire class of confusion.
**The model:** Variables are names that point to objects. Assignment binds a name to an object. Function arguments are new names bound to the same objects.
```python theme={null}
def modify(lst, num, text):
lst.append(4) # Mutates the object -- caller sees this
num += 1 # Rebinds local name to new int object -- caller does NOT see this
text += " world" # Rebinds local name to new string -- caller does NOT see this
my_list = [1, 2, 3]
my_num = 5
my_text = "hello"
modify(my_list, my_num, my_text)
print(my_list) # [1, 2, 3, 4] -- modified (mutable, mutated in place)
print(my_num) # 5 -- unchanged (immutable, rebinding created new object)
print(my_text) # "hello" -- unchanged (immutable, rebinding created new object)
```
**The mental model:** Think of variables as sticky notes on objects, not as boxes containing values. `=` moves the sticky note. Mutation changes the object the sticky note is on. Rebinding moves the sticky note to a different object.
**The augmented assignment gotcha:**
```python theme={null}
def tricky(lst):
lst += [4, 5] # For lists, += calls __iadd__ which MUTATES in place
# This DOES affect the caller because list.__iadd__ modifies self
def also_tricky(tup):
tup += (4, 5) # For tuples, += creates a new tuple (immutable)
# This does NOT affect the caller
```
`+=` behaves differently for mutable vs immutable types. For lists, it mutates. For tuples and strings, it creates new objects.
**What interviewers are really testing:** Whether you can predict what happens when you pass mutable vs immutable objects to functions.
**Red flag answer:** Saying Python is "pass by reference" or "pass by value."
**Follow-up:**
* "Is Python pass-by-value or pass-by-reference? Neither -- explain what it actually is."
* "Why does `lst += [4]` inside a function affect the caller but `num += 1` does not?"
* "How would you write a function that cannot accidentally modify its arguments?"
Object interning is CPython's optimization where frequently-used immutable objects are cached and reused instead of creating new objects.
**What gets interned:**
* **Integers -5 to 256**: Pre-allocated at interpreter startup. `a = 100; b = 100; a is b` is always `True`.
* **Short strings** that look like identifiers (alphanumeric + underscore): Automatically interned. `"hello" is "hello"` is `True`.
* **Strings used as variable names, function names, etc.**: Interned by the compiler.
* `True`, `False`, `None`: Singletons (always interned).
**What does NOT get interned (surprising cases):**
```python theme={null}
a = 257; b = 257
a is b # Often False in REPL, may be True in .py file (compiler optimization)
a = "hello world" # Contains space -- NOT interned
b = "hello world"
a is b # May be True or False -- implementation-dependent
a = "hello"; b = "hello"
a is b # True -- looks like an identifier, so it is interned
```
**Force interning with `sys.intern()`:**
```python theme={null}
import sys
a = sys.intern("some long string that repeats a lot")
b = sys.intern("some long string that repeats a lot")
a is b # True -- forced interning
```
Use `sys.intern()` when you have many duplicate strings in memory (e.g., column names in a data processing pipeline with millions of rows).
**Why this matters:** Interning saves memory and makes `is` comparisons O(1). But **never rely on interning for correctness** -- always use `==` for value comparison. Interning is a CPython optimization detail that may change.
**What interviewers are really testing:** Whether you know the interning boundaries and understand it is implementation-specific.
**Red flag answer:** "Integers are interned" without knowing the -5 to 256 range or the REPL vs script difference.
**Follow-up:**
* "Why does `257 is 257` give different results in the REPL vs a .py file?"
* "When would you use `sys.intern()` in production code?"
* "Why is interning an implementation detail you should never rely on?"
Weak references let you reference an object without preventing its garbage collection. The reference becomes `None` when the object is collected.
**The problem weak references solve:**
```python theme={null}
# Without weak references -- cache prevents garbage collection
cache = {}
def get_data(key):
obj = expensive_computation(key)
cache[key] = obj # Strong reference -- obj can NEVER be garbage collected
return obj
# Memory grows forever if keys are unique
```
**With `weakref.WeakValueDictionary`:**
```python theme={null}
import weakref
cache = weakref.WeakValueDictionary()
def get_data(key):
if key in cache:
return cache[key] # May be garbage collected -- check first
obj = expensive_computation(key)
cache[key] = obj # Weak reference -- GC can collect obj
return obj
```
When no strong references to `obj` remain, the GC collects it and the cache entry automatically disappears.
**Use cases:**
* **Caches**: Allow cached objects to be collected when memory is tight
* **Observer pattern**: Observers hold weak references to subjects to avoid preventing collection
* **Circular reference avoidance**: Break cycles by making one reference weak
* **Object tracking/debugging**: Monitor objects without affecting their lifecycle
**Limitations:** Not all objects support weak references. `int`, `str`, `tuple`, and other built-in immutable types cannot have weak references. Custom classes support them by default (unless `__slots__` is used without `__weakref__`).
**What interviewers are really testing:** Whether you understand the GC implications and can identify real use cases.
**Red flag answer:** Knowing the syntax without understanding *why* -- the garbage collection interaction.
**Follow-up:**
* "How does a `WeakValueDictionary` differ from a regular dict in terms of garbage collection?"
* "What types of objects cannot have weak references? Why?"
* "How would you implement an LRU cache that respects memory pressure using weak references?"
Python is a language specification. CPython, PyPy, Jython, etc. are *implementations* of that specification.
**CPython** (the default):
* Written in C. Reference implementation -- defines what "Python" means.
* Compiles to bytecode, interpreted by CPython VM.
* Has the GIL. Uses reference counting + cyclic GC.
* Best ecosystem compatibility (C extensions, pip packages).
* Performance: baseline. \~10-100x slower than C for CPU-bound work.
**PyPy**:
* Written in Python (RPython). Uses **JIT (Just-In-Time) compilation**.
* Can be **4-10x faster** than CPython for long-running programs (the JIT "warms up").
* Has a GIL (same as CPython).
* Compatibility: Most pure-Python code works. C extensions may not (no `ctypes` API compatibility guarantee). NumPy works via cpyext but is slower than on CPython.
* Best for: Long-running servers, computation-heavy pure Python code.
**GraalPython**:
* Runs on GraalVM (polyglot runtime by Oracle).
* No GIL. Can interop with Java, JavaScript, Ruby, R.
* Relatively new, limited ecosystem support.
**Cython** (not a Python implementation, but relevant):
* Compiles Python-like code to C extensions. Gives C-level performance for annotated code.
* Used by many high-performance libraries (NumPy, pandas, scikit-learn).
**Practical decision matrix:**
* Need C extensions (NumPy, pandas, TensorFlow)? **CPython**
* Need faster pure-Python execution? **PyPy**
* Need Java interop? **GraalPython** or Jython
* Need to write C-speed extensions? **Cython**
**What interviewers are really testing:** Whether you know alternatives exist and can make informed choices.
**Red flag answer:** Not knowing that CPython is just one implementation, or thinking Python == CPython.
**Follow-up:**
* "When would you deploy PyPy instead of CPython? What would you lose?"
* "What is JIT compilation and why does it make PyPy faster?"
* "How does Cython differ from CPython?"
The `dis` module disassembles Python functions into CPython bytecode instructions. This is invaluable for understanding performance and debugging mysterious behavior.
**Example:**
```python theme={null}
import dis
def example(a, b):
return a + b
dis.dis(example)
# 0 LOAD_FAST 0 (a)
# 2 LOAD_FAST 1 (b)
# 4 BINARY_ADD
# 6 RETURN_VALUE
```
**Why this matters for performance:**
```python theme={null}
# List comprehension vs loop -- bytecode explains the speed difference
# Comprehension uses LIST_APPEND (fast, no attribute lookup)
# Loop calls list.append (method lookup each iteration)
dis.dis(compile("[x**2 for x in range(10)]", "", "eval"))
dis.dis(compile("""
result = []
for x in range(10):
result.append(x**2)
""", "", "exec"))
```
**Common bytecodes to know:**
* `LOAD_FAST` / `STORE_FAST`: Local variable access (fastest)
* `LOAD_GLOBAL` / `STORE_GLOBAL`: Global variable access (slower -- dict lookup)
* `LOAD_ATTR`: Attribute access (dict lookup on object)
* `CALL_FUNCTION`: Function call overhead
* `BINARY_ADD`, `BINARY_MULTIPLY`: Arithmetic operations
* `FOR_ITER`: Loop iteration
**Practical use:** If two apparently equivalent code paths have different performance, `dis.dis()` shows you exactly what the interpreter does differently.
**What interviewers are really testing:** Whether you can use low-level tools to understand and debug Python performance.
**Red flag answer:** Never having used `dis` or not understanding why local variables are faster than global ones.
**Follow-up:**
* "Why is accessing a local variable faster than a global variable in CPython?"
* "How would you use `dis` to explain why list comprehensions are faster than equivalent for loops?"
* "What bytecode instruction does a `for` loop compile to?"
## 14. Descriptors and Magic Methods
The descriptor protocol is one of Python's most powerful and least understood features. Properties, methods, `classmethod`, `staticmethod`, and `super()` all use it.
**The protocol -- three optional methods:**
* `__get__(self, obj, objtype=None)` -- Called when attribute is accessed
* `__set__(self, obj, value)` -- Called when attribute is assigned
* `__delete__(self, obj)` -- Called when attribute is deleted
**Data descriptors vs non-data descriptors:**
* **Data descriptor**: Defines `__set__` and/or `__delete__`. Takes priority over instance `__dict__`.
* **Non-data descriptor**: Defines only `__get__`. Instance `__dict__` takes priority.
* `@property` creates a data descriptor. Functions are non-data descriptors (which is why you can override methods on instances).
**How `@property` works internally (simplified):**
```python theme={null}
class Property:
def __init__(self, fget=None, fset=None, fdel=None):
self.fget = fget
self.fset = fset
self.fdel = fdel
def __get__(self, obj, objtype=None):
if obj is None:
return self # Accessed on class, return descriptor itself
return self.fget(obj)
def __set__(self, obj, value):
if self.fset is None:
raise AttributeError("can't set attribute")
self.fset(obj, value)
def setter(self, fset):
return type(self)(self.fget, fset, self.fdel)
```
**How functions become methods (descriptor magic):**
```python theme={null}
class MyClass:
def method(self):
pass
obj = MyClass()
# MyClass.__dict__['method'] is a function (non-data descriptor)
# obj.method triggers function.__get__(obj, MyClass) which returns a bound method
```
This is why `obj.method` and `MyClass.method` behave differently.
**Attribute lookup order (the full picture):**
1. Data descriptors on the class (e.g., properties)
2. Instance `__dict__`
3. Non-data descriptors on the class (e.g., methods)
4. `__getattr__` (fallback)
**What interviewers are really testing:** Whether you understand the lookup order and can explain how `@property` and methods work.
**Red flag answer:** Knowing `@property` syntax without understanding descriptors.
**Follow-up:**
* "What is the difference between a data descriptor and a non-data descriptor?"
* "How does a function become a bound method when accessed on an instance?"
* "Build a descriptor that validates attribute types (e.g., must be `int`)."
These two methods control attribute access at different levels:
**`__getattr__(self, name)`** -- The fallback. Only called when normal attribute lookup fails (not found in `__dict__`, class hierarchy, or data descriptors):
```python theme={null}
class FlexibleObject:
def __init__(self):
self.data = {"name": "Alice"}
def __getattr__(self, name):
# Only called for attributes not found normally
if name in self.data:
return self.data[name]
raise AttributeError(f"No attribute '{name}'")
obj = FlexibleObject()
obj.data # Normal lookup -- __getattr__ NOT called
obj.name # Not in __dict__ -- __getattr__ IS called, returns "Alice"
```
**`__getattribute__(self, name)`** -- Total control. Called for EVERY attribute access:
```python theme={null}
class LoggedAccess:
def __getattribute__(self, name):
print(f"Accessing: {name}")
return object.__getattribute__(self, name) # MUST call super to avoid infinite recursion
```
**The infinite recursion trap:**
Inside `__getattribute__`, any `self.x` access calls `__getattribute__` again. You MUST use `object.__getattribute__(self, name)` to break the recursion.
**Use cases:**
* `__getattr__`: Proxy objects, lazy loading, default values, API wrappers
* `__getattribute__`: Logging, access control, transparent proxies (rare -- use with extreme caution)
**What interviewers are really testing:** Whether you know when each is called and can avoid the recursion trap.
**Red flag answer:** Confusing the two, or not knowing about the infinite recursion danger in `__getattribute__`.
**Follow-up:**
* "How do you avoid infinite recursion in `__getattribute__`?"
* "Build a proxy object using `__getattr__` that delegates all attribute access to a wrapped object."
* "When would you use `__getattribute__` instead of `__getattr__`?"
Using `type()` as a metaclass demonstrates that **classes are objects** created by calling a metaclass.
```python theme={null}
# These are equivalent:
# Using class keyword
class Person:
def __init__(self, name):
self.name = name
def greet(self):
return f"Hello, I'm {self.name}"
# Using type() directly
def __init__(self, name):
self.name = name
def greet(self):
return f"Hello, I'm {self.name}"
Person = type('Person', (object,), {
'__init__': __init__,
'greet': greet,
})
```
**What this reveals about Python's object model:**
* The `class` statement is syntactic sugar for a `type()` call
* `type` is both a function (returning an object's type) and a metaclass (creating new classes)
* Classes are first-class objects that can be created, modified, passed as arguments, and returned from functions
**Dynamic class creation use cases:**
* ORMs that create model classes from database schemas
* Plugin systems that register classes at runtime
* Testing frameworks that generate test classes dynamically
* Serialization libraries that create classes from schemas (Pydantic, marshmallow)
**What interviewers are really testing:** Whether you understand that classes are objects and metaclasses are their factories.
**Red flag answer:** Being surprised that `type()` can create classes.
**Follow-up:**
* "If `type` creates classes, what creates `type`? (Answer: `type` creates itself)"
* "How would you dynamically create a class with attributes determined at runtime?"
* "How does this relate to metaclasses?"
Monkey patching is modifying code at runtime by replacing methods, functions, or attributes of existing classes/modules. It is powerful but dangerous.
**The pattern:**
```python theme={null}
# Original
import requests
# Monkey patch -- replace requests.get with a mock
original_get = requests.get # Save original
def mock_get(url, **kwargs):
return MockResponse(status_code=200, json_data={"mock": True})
requests.get = mock_get # Patch
# All code that calls requests.get() now gets our mock
data = requests.get("https://api.example.com/data")
# data is MockResponse, no network call made
requests.get = original_get # Restore (important!)
```
**Legitimate use cases:**
1. **Testing**: `unittest.mock.patch` is structured monkey patching with automatic cleanup
2. **Hotfixing third-party bugs**: Patch a broken method in a library you cannot update immediately
3. **Feature flags**: Replace behavior at runtime based on configuration
4. **gevent/eventlet**: Monkey-patches the standard library to make blocking I/O non-blocking
**Why it is dangerous:**
* Code behavior depends on import order and patch order
* Breaks IDE navigation and static analysis (the patched function is not the one in the source)
* Library updates may change the patched interface, breaking your patch silently
* Makes debugging extremely difficult ("why is requests.get returning a mock in production?")
**The proper way -- `unittest.mock.patch`:**
```python theme={null}
from unittest.mock import patch
with patch('module.requests.get') as mock_get:
mock_get.return_value.json.return_value = {"data": "test"}
result = function_under_test()
# Original automatically restored when exiting 'with' block
```
**What interviewers are really testing:** Whether you know *when* monkey patching is appropriate (testing, hotfixes) vs when it is reckless (production behavior changes).
**Red flag answer:** Enthusiastically monkey patching in production without discussing the risks.
**Follow-up:**
* "How does `unittest.mock.patch` improve on raw monkey patching?"
* "What is gevent's monkey patching and why is it necessary?"
* "You need to fix a bug in a third-party library. Monkey patch or fork the repo?"
## 15. Advanced Generators and Iterators
`yield from` delegates to a sub-generator, but it is far more than syntactic sugar for a `for` loop. It creates a **bidirectional channel** between the caller and the sub-generator.
**Basic delegation:**
```python theme={null}
def sub_generator():
yield 1
yield 2
return "sub done" # Return value is captured by yield from
def main_generator():
result = yield from sub_generator() # Delegates completely
print(f"Sub returned: {result}") # "sub done"
yield 3
```
**What `yield from` handles that a manual loop does not:**
1. **`send()` forwarding**: Values sent to the outer generator are forwarded to the sub-generator
2. **`throw()` forwarding**: Exceptions thrown into the outer generator are thrown into the sub-generator
3. **`close()` forwarding**: Closing the outer generator closes the sub-generator
4. **Return value capture**: The sub-generator's `return` value becomes the value of the `yield from` expression
**Without `yield from`, you would need \~40 lines of boilerplate** to handle all edge cases correctly. PEP 380 formalized this.
**Real-world use case -- flattening nested structures:**
```python theme={null}
def flatten(items):
for item in items:
if isinstance(item, (list, tuple)):
yield from flatten(item) # Recursive delegation
else:
yield item
list(flatten([1, [2, [3, 4], 5], 6])) # [1, 2, 3, 4, 5, 6]
```
**Connection to asyncio:** `yield from` was the original mechanism for coroutine composition before `async`/`await` was introduced in Python 3.5. `await` is essentially `yield from` for coroutines.
**What interviewers are really testing:** Whether you understand the bidirectional channel, not just the iteration delegation.
**Red flag answer:** "It is the same as `for x in gen: yield x`." It is not -- `send()`, `throw()`, and `close()` forwarding are the whole point.
**Follow-up:**
* "What is the difference between `yield from gen` and `for x in gen: yield x`?"
* "How does `yield from` relate to `await` in asyncio?"
* "How do you capture the return value of a sub-generator?"
Generators are unique in Python because they maintain a **suspended execution frame** -- the function's local variables, instruction pointer, and stack are frozen on `yield` and thawed on `next()`.
**The execution lifecycle:**
```python theme={null}
def counter():
print("Starting") # Runs on first next()
i = 0
while True:
i += 1
print(f"Before yield {i}")
yield i # SUSPEND: frame frozen, value sent to caller
print(f"After yield {i}") # RESUME: runs on next next()
gen = counter() # NO code executes -- generator object created
print(next(gen)) # "Starting", "Before yield 1", prints 1
print(next(gen)) # "After yield 1", "Before yield 2", prints 2
```
**Frame persistence internals (CPython):**
* Generator objects hold a reference to their frame object (`gen.gi_frame`)
* The frame stores: local variables, instruction pointer (`f_lasti`), operand stack, block stack
* `gen.gi_frame` is `None` after the generator is exhausted
* You can inspect the current state: `gen.gi_frame.f_locals`, `gen.gi_frame.f_lineno`
**Generator states:**
```python theme={null}
import inspect
gen = counter()
inspect.getgeneratorstate(gen) # 'GEN_CREATED' -- never started
next(gen)
inspect.getgeneratorstate(gen) # 'GEN_SUSPENDED' -- paused at yield
# After exhaustion: 'GEN_CLOSED'
```
**Why this matters for memory:** The frame is a lightweight object (\~200-400 bytes). Compare this to a thread (\~8MB default stack on Linux) or a process (\~30MB+). This is why you can have millions of generators but only thousands of threads.
**What interviewers are really testing:** Whether you understand that generators are coroutines (suspended functions) not just iterators.
**Red flag answer:** Describing generators as "functions that return multiple values" without understanding frame suspension.
**Follow-up:**
* "How can you inspect the current state of a suspended generator?"
* "Why can you have millions of generators but only thousands of threads?"
* "What happens to the generator frame when it is exhausted?"
The `itertools` module provides composable, lazy building blocks for creating efficient data processing pipelines. It is one of the most underused modules in Python.
**Essential itertools functions:**
```python theme={null}
import itertools
# chain -- concatenate iterables without creating a combined list
all_data = itertools.chain(file1_lines, file2_lines, file3_lines)
# islice -- slice an iterator (cannot use [] on iterators)
first_100 = itertools.islice(huge_generator, 100)
# groupby -- group consecutive elements (MUST be sorted first)
sorted_data = sorted(records, key=lambda r: r.category)
for category, group in itertools.groupby(sorted_data, key=lambda r: r.category):
print(category, list(group))
# product -- cartesian product (replaces nested for loops)
for x, y, z in itertools.product(range(10), range(10), range(10)):
pass # 1000 combinations, O(1) memory
# combinations, permutations
pairs = itertools.combinations(users, 2) # All unique pairs
# accumulate -- running totals
running_sum = itertools.accumulate([1, 2, 3, 4]) # 1, 3, 6, 10
# starmap -- map with argument unpacking
results = itertools.starmap(pow, [(2, 3), (3, 2), (10, 3)]) # 8, 9, 1000
```
**Building Unix-pipe-style processing:**
```python theme={null}
def read_logs(path):
with open(path) as f:
yield from f
def filter_errors(lines):
return (line for line in lines if "ERROR" in line)
def extract_timestamps(lines):
return (line.split()[0] for line in lines)
# Process 100GB log file with constant memory
pipeline = extract_timestamps(filter_errors(read_logs("huge.log")))
for timestamp in itertools.islice(pipeline, 1000): # First 1000 errors
print(timestamp)
```
**What interviewers are really testing:** Whether you can build memory-efficient pipelines using lazy iterators.
**Red flag answer:** Loading everything into lists instead of using lazy iterators for large data.
**Follow-up:**
* "How would you process a 100GB log file using `itertools` and generators?"
* "What is the gotcha with `itertools.groupby`? (Must be sorted first)"
* "When would `itertools.product` be better than nested for loops?"
## 16. Advanced Concurrency and Parallelism
This question trips up many candidates. The GIL prevents CPU-bound parallelism, but threading is essential for **I/O-bound concurrency**.
**The key insight:** The GIL is released during I/O operations (network calls, disk reads, database queries, `time.sleep()`). While one thread waits for I/O, other threads can execute Python code.
**Real-world performance impact:**
```python theme={null}
import concurrent.futures
import requests
urls = ["https://example.com"] * 100
# Sequential: ~100 seconds (1s per request)
for url in urls:
requests.get(url)
# Threaded: ~10 seconds (10 threads, each handles 10 requests)
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
list(executor.map(requests.get, urls))
```
The threaded version is 10x faster despite the GIL because each thread spends 99% of its time waiting for the network, not executing Python bytecode.
**Threading is also needed for:**
* GUI applications (separate thread for UI responsiveness)
* Background tasks (periodic cleanup, monitoring)
* Real-time systems (audio playback while processing)
* Libraries that release the GIL in C extensions (NumPy, PIL, cryptography)
**Thread safety concerns (the real production issue):**
Even though the GIL prevents data races *at the bytecode instruction level*, operations like `counter += 1` are NOT atomic (they compile to multiple bytecodes: LOAD, ADD, STORE). You still need `threading.Lock` for shared mutable state.
**What interviewers are really testing:** Whether you understand that the GIL is released during I/O and that threading is valuable for I/O-bound work.
**Red flag answer:** "Threading is useless in Python because of the GIL."
**Follow-up:**
* "Is `counter += 1` thread-safe in Python? Why or why not?"
* "When is the GIL released? Give me specific examples."
* "How many threads should you use for a web scraper hitting 1000 URLs?"
The decision depends on two factors: (1) Is the work I/O-bound or CPU-bound? (2) How many concurrent tasks do you need?
**Decision matrix:**
```
CPU-bound work?
├── Yes → ProcessPoolExecutor (num_workers = num_cores)
└── No (I/O-bound) →
How many concurrent tasks?
├── < 100 → ThreadPoolExecutor (simpler, sufficient)
└── > 100-1000+ → asyncio (lower overhead per task)
```
**ThreadPoolExecutor** -- the workhorse for I/O-bound work:
```python theme={null}
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=20) as pool:
results = list(pool.map(fetch_url, urls))
```
\~100KB per thread. Good for 10-100 concurrent tasks. Shared memory (easy data sharing). Simple mental model.
**ProcessPoolExecutor** -- for CPU-bound work:
```python theme={null}
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(max_workers=os.cpu_count()) as pool:
results = list(pool.map(process_image, images))
```
\~30-50MB per process. Arguments must be picklable (serialized across process boundaries). Each process has its own GIL.
**asyncio** -- for high-concurrency I/O:
```python theme={null}
async def main():
async with aiohttp.ClientSession() as session:
tasks = [fetch(session, url) for url in urls]
results = await asyncio.gather(*tasks)
```
\~1KB per coroutine. Can handle 10,000+ concurrent connections on a single thread. Requires async-compatible libraries.
**Hybrid approach (common in production):**
```python theme={null}
async def main():
# I/O-bound: use asyncio
data = await fetch_all_data()
# CPU-bound: offload to ProcessPool from within asyncio
loop = asyncio.get_event_loop()
with ProcessPoolExecutor() as pool:
results = await loop.run_in_executor(pool, cpu_heavy_work, data)
```
**What interviewers are really testing:** Whether you can match the concurrency model to the task type with clear reasoning.
**Red flag answer:** Always choosing the same model regardless of task type.
**Follow-up:**
* "Your task involves both API calls (I/O) and data processing (CPU). What architecture do you use?"
* "What happens if you use ThreadPoolExecutor for CPU-bound work? Show me the math."
* "Why is asyncio more efficient than threading for 10,000 concurrent connections?"
Understanding resource costs is critical for production capacity planning.
**Per-unit cost comparison:**
| Approach | Memory per unit | Creation time | Max practical count |
| --------- | ----------------------------------- | ------------- | ------------------- |
| Thread | \~100KB (stack) | \~1ms | \~1,000-5,000 |
| Process | \~30-50MB (full Python interpreter) | \~50-100ms | \~CPU count (4-64) |
| Coroutine | \~1KB | \~0.01ms | \~100,000+ |
**Thread-specific costs:**
* Default stack size: 8MB on Linux (most unused). Set `threading.stack_size(65536)` for low-memory threads.
* Context switching: \~1-10 microseconds (OS-managed, preemptive)
* Shared memory: Pro for data sharing, con for needing locks
* GIL contention: Under heavy CPU load, threads fight for the GIL, adding \~5% overhead per thread
**Process-specific costs:**
* Each process gets a full Python interpreter copy
* IPC (Inter-Process Communication) requires serialization: `pickle` for `multiprocessing`, which is slow for large objects
* Copy-on-write after `fork()`: Memory is shared until modified. Linux is efficient here; macOS less so.
* `initializer` functions in `ProcessPoolExecutor` run once per worker -- use them to load large models/data per worker instead of passing through pickle
**asyncio-specific costs:**
* Coroutines are lightweight Python objects (\~1KB)
* Event loop overhead: \~10 microseconds per coroutine switch
* No parallelism: All coroutines share one CPU core
* Blocking call in any coroutine blocks ALL coroutines (the "foot gun")
**What interviewers are really testing:** Whether you can do back-of-envelope capacity calculations.
**Red flag answer:** Not knowing the approximate memory cost of threads vs processes vs coroutines.
**Follow-up:**
* "You need to handle 50,000 concurrent WebSocket connections. Calculate the memory needed for threads vs asyncio."
* "Your process pool workers each load a 500MB ML model. How much total memory?"
* "How does copy-on-write affect memory usage after forking?"
The fundamental difference is **preemptive** (threading) vs **cooperative** (async) multitasking.
**Threading (preemptive):**
* The OS decides when to switch threads (can happen at any bytecode instruction)
* You need locks to protect shared data (race conditions are possible)
* Blocking I/O is fine -- the OS blocks the thread and switches to another
* Simple mental model: write sequential code, wrap in threads
**async/await (cooperative):**
* Coroutines explicitly yield control at `await` points (you control when switching happens)
* No locks needed for coroutine-local data (no preemptive switching between awaits)
* Blocking I/O is FORBIDDEN -- one blocking call freezes the entire event loop
* Requires async-compatible libraries for everything (aiohttp, asyncpg, aiofiles)
**Side-by-side comparison:**
```python theme={null}
# Threading -- simple, works with any library
def fetch_all_threaded(urls):
with ThreadPoolExecutor(10) as pool:
return list(pool.map(requests.get, urls))
# Async -- more efficient, requires async libraries
async def fetch_all_async(urls):
async with aiohttp.ClientSession() as session:
tasks = [session.get(url) for url in urls]
return await asyncio.gather(*tasks)
```
**The "function color" problem:**
Once you have one `async def`, the entire call stack above it must also be async. You cannot `await` from a regular function. This creates a viral effect that can require rewriting large portions of code.
**When threading wins:** Quick scripts, code that uses sync-only libraries, teams without async experience.
**When asyncio wins:** High-concurrency servers (>1000 connections), WebSocket servers, API gateways.
**What interviewers are really testing:** Whether you understand the trade-offs, especially the cooperative nature of async.
**Red flag answer:** "Async is always faster than threading." It is more efficient for many connections but adds complexity.
**Follow-up:**
* "What happens if you accidentally call `time.sleep(10)` inside an async handler?"
* "How do you run blocking code from within an async function?"
* "What is the 'function color' problem and how does it affect your architecture decisions?"
The golden rule: **measure first, optimize second.** Performance intuition is wrong more often than it is right.
**The profiling toolkit:**
```python theme={null}
# 1. cProfile -- function-level CPU profiling
python -m cProfile -s cumulative your_script.py
# 2. py-spy -- sampling profiler (attach to running process, no code changes)
py-spy top --pid 12345
py-spy record -o profile.svg -- python your_script.py # Flame graph
# 3. line_profiler -- line-by-line profiling
@profile # Add decorator, run with kernprof
def slow_function():
pass
# 4. timeit -- micro-benchmarks
python -m timeit -s "lst = list(range(1000))" "sum(lst)"
# 5. memory_profiler -- line-by-line memory usage
@profile
def memory_hungry():
pass
# 6. tracemalloc -- memory allocation tracking
tracemalloc.start()
# ... code ...
snapshot = tracemalloc.take_snapshot()
```
**The profiling workflow:**
1. **Reproduce the slow behavior** with realistic data
2. **Profile to find the hot path** (usually 5% of code is 95% of runtime)
3. **Optimize the hot path** using appropriate techniques:
* Algorithm improvement (O(n^2) to O(n log n))
* Data structure change (list to set for lookups)
* Caching (`functools.lru_cache`)
* Vectorization (NumPy instead of Python loops)
* Concurrency (appropriate model for task type)
* C extension (Cython, pybind11) as last resort
4. **Measure again** to verify improvement
5. **Add regression benchmarks** to prevent future regressions
**`py-spy` is the most practical tool for production profiling** -- it attaches to a running process with zero overhead, requires no code changes, and produces flame graphs. It has saved hours of debugging in my experience.
**What interviewers are really testing:** Whether you follow a systematic profiling workflow or guess at optimizations.
**Red flag answer:** Optimizing without profiling first, or only knowing `print(time.time())`.
**Follow-up:**
* "How would you profile a production service without restarting it?"
* "Your function is slow. `cProfile` shows it spends 80% of time in one line. What is your next step?"
* "What is a flame graph and how do you read one?"
Matching the right concurrency model to the workload is a critical production skill:
**Multithreading use cases (I/O-bound, moderate concurrency):**
* **Web scraping**: Fetch 100 URLs concurrently. Each thread blocks on network I/O while others proceed. ThreadPoolExecutor with 10-50 workers.
* **File processing**: Read/write multiple files simultaneously. Disk I/O is the bottleneck, not CPU.
* **Database operations**: Execute multiple queries in parallel. Database drivers release the GIL during network waits.
* **GUI applications**: Keep the UI responsive while doing background work.
**Multiprocessing use cases (CPU-bound):**
* **Image/video processing**: Resize 10,000 images using all 8 CPU cores. Each worker process handles a batch.
* **Data transformation**: ETL pipelines with heavy computation (parsing, transforming, aggregating).
* **Scientific computing**: Monte Carlo simulations, numerical integration, matrix operations (when NumPy is insufficient).
* **ML model training**: Parallel hyperparameter search with `ProcessPoolExecutor`.
**asyncio use cases (I/O-bound, high concurrency):**
* **Web servers**: FastAPI/Starlette handling 10,000+ concurrent HTTP connections.
* **WebSocket servers**: Real-time chat, live dashboards, gaming servers.
* **API gateways**: Fan-out requests to multiple backend services concurrently.
* **Message queue consumers**: Process Kafka/RabbitMQ messages with async handlers.
* **Web crawlers**: Crawl thousands of pages with aiohttp, rate-limited.
**Hybrid architectures (production reality):**
Most production systems combine approaches:
```python theme={null}
# FastAPI (asyncio) + ProcessPool for CPU work
@app.post("/process-image")
async def process_image(file: UploadFile):
data = await file.read() # asyncio for I/O
loop = asyncio.get_event_loop()
result = await loop.run_in_executor( # Process pool for CPU
process_pool, heavy_processing, data
)
await save_to_s3(result) # asyncio for I/O
return {"status": "done"}
```
**What interviewers are really testing:** Whether you can design a real system using the right concurrency model for each component.
**Red flag answer:** Only knowing one approach, or using threading for CPU-bound work.
**Follow-up:**
* "Design the concurrency architecture for an image-processing API that receives uploads (I/O), processes them (CPU), and stores results (I/O)."
* "How would you handle a mixed workload that is both I/O-bound and CPU-bound?"
* "What is the maximum number of concurrent WebSocket connections a single asyncio server can handle? What is the bottleneck?"
## 17. Design Patterns and Architecture
SOLID principles apply to Python, but Python's dynamic nature means the implementation looks different than in Java/C#.
**S -- Single Responsibility:**
Each class/module has one reason to change. In Python, this often means splitting a "god class" into smaller classes or using standalone functions:
```python theme={null}
# BAD: UserService does everything
class UserService:
def create_user(self): ...
def send_email(self): ...
def generate_report(self): ...
# GOOD: Separate concerns
class UserRepository: # Data access only
class EmailService: # Communication only
class ReportGenerator: # Reporting only
```
**O -- Open/Closed:**
Open for extension, closed for modification. In Python, this is achieved through composition, protocols, and plugin systems -- not necessarily inheritance:
```python theme={null}
# Instead of modifying existing code, extend through Protocol
from typing import Protocol
class Serializer(Protocol):
def serialize(self, data: dict) -> str: ...
class JSONSerializer:
def serialize(self, data: dict) -> str: return json.dumps(data)
class XMLSerializer:
def serialize(self, data: dict) -> str: ... # New format without changing existing code
```
**L -- Liskov Substitution:**
Subclasses must be usable wherever their parent is expected. The classic violation in Python: overriding a method to raise `NotImplementedError`.
**I -- Interface Segregation:**
Use `typing.Protocol` (Python 3.8+) for narrow, focused interfaces. Do not force classes to implement methods they do not need.
**D -- Dependency Inversion:**
Depend on abstractions (Protocols, ABCs), not concrete implementations. In Python, this is often as simple as passing a function or Protocol-typed parameter.
**What interviewers are really testing:** Whether you can translate SOLID from Java-speak to Pythonic patterns.
**Red flag answer:** Describing SOLID exactly as a Java textbook would without Python-specific adaptations.
**Follow-up:**
* "How does `typing.Protocol` implement the Interface Segregation principle in Python?"
* "Give me a real example of Liskov Substitution violation in Python."
* "How do you implement Dependency Inversion in Python without a DI framework?"
Composition means building objects by combining simpler objects, rather than inheriting behavior from parent classes. The Python community strongly favors composition.
**Why inheritance causes problems:**
```python theme={null}
# Inheritance: tight coupling, diamond problem, rigid hierarchy
class Animal: ...
class FlyingAnimal(Animal): ...
class SwimmingAnimal(Animal): ...
class Duck(FlyingAnimal, SwimmingAnimal): ... # Diamond problem!
```
**Composition: flexible, testable, explicit:**
```python theme={null}
# Behaviors as composable objects
@dataclass
class Animal:
name: str
movement: MovementStrategy # Inject behavior
sound: SoundStrategy
# Behaviors can be mixed freely
duck = Animal("Duck", CanFlyAndSwim(), Quack())
dog = Animal("Dog", CanRun(), Bark())
penguin = Animal("Penguin", CanSwim(), Squawk())
```
**Real-world refactoring -- Django views:**
```python theme={null}
# Before: Deep inheritance chain (common Django anti-pattern)
class MyView(LoginRequiredMixin, PermissionRequiredMixin,
FormView, DetailView): # What methods are available? Who knows.
pass
# After: Composition with explicit delegation
class MyView:
def __init__(self):
self.auth = AuthService()
self.form_handler = FormHandler()
self.data_loader = DataLoader()
def handle_request(self, request):
self.auth.require_login(request)
self.auth.check_permission(request, "edit")
data = self.data_loader.get_detail(request.id)
return self.form_handler.process(request, data)
```
**The rule of thumb:** Use inheritance for "is-a" relationships with shared interfaces (ABCs). Use composition for "has-a" relationships and behavior assembly. When in doubt, compose.
**What interviewers are really testing:** Whether you can articulate *why* composition is better with a concrete example.
**Red flag answer:** "Composition is better" without explaining the specific problems inheritance causes.
**Follow-up:**
* "When IS inheritance the right choice in Python?"
* "How do mixins fit into the composition vs inheritance debate?"
* "Refactor this 5-level inheritance chain into composition."
The Factory pattern creates objects without hardcoding their exact class. Python's dynamic nature makes factories simpler than in Java, but the pattern is still valuable for complex creation logic.
**Simple factory (dict-based):**
```python theme={null}
class JSONParser:
def parse(self, data): return json.loads(data)
class XMLParser:
def parse(self, data): return ET.fromstring(data)
class YAMLParser:
def parse(self, data): return yaml.safe_load(data)
def create_parser(format_type: str):
parsers = {
'json': JSONParser,
'xml': XMLParser,
'yaml': YAMLParser,
}
parser_cls = parsers.get(format_type)
if parser_cls is None:
raise ValueError(f"Unknown format: {format_type}")
return parser_cls()
```
**Registry-based factory (scalable, plugin-friendly):**
```python theme={null}
parser_registry = {}
def register_parser(format_type):
def decorator(cls):
parser_registry[format_type] = cls
return cls
return decorator
@register_parser('json')
class JSONParser:
def parse(self, data): return json.loads(data)
@register_parser('csv')
class CSVParser:
def parse(self, data): ...
def create_parser(format_type: str):
return parser_registry[format_type]()
```
This pattern allows adding new parsers without modifying the factory -- just decorate a new class.
**When to use factories vs direct instantiation:**
* Direct instantiation: When the class to create is known at the call site
* Factory function: When the class depends on runtime configuration (format type, environment)
* Abstract factory: When you need families of related objects (rare in Python)
**What interviewers are really testing:** Whether you can implement a factory using Python's first-class classes and decorators, not Java-style abstract factory hierarchies.
**Red flag answer:** Implementing a Java-style AbstractFactory with 5 classes when a dict would do.
**Follow-up:**
* "How does the registry decorator pattern work?"
* "When is a factory function overkill and you should just use `@classmethod`?"
* "How would you make this factory extensible for plugins?"
Singletons ensure only one instance of a class exists. Python offers several approaches, each with different trade-offs:
**1. Module-level variable (the Pythonic way):**
```python theme={null}
# config.py
_settings = None
def get_settings():
global _settings
if _settings is None:
_settings = Settings()
return _settings
```
Simplest. Modules are cached in `sys.modules` and only executed once. Most Pythonic approach. Used by the standard library.
**2. `__new__` override:**
```python theme={null}
class Singleton:
_instance = None
def __new__(cls, *args, **kwargs):
if cls._instance is None:
cls._instance = super().__new__(cls)
return cls._instance
```
Straightforward. But `__init__` runs on every call, which can re-initialize state.
**3. Decorator:**
```python theme={null}
def singleton(cls):
instances = {}
@functools.wraps(cls)
def get_instance(*args, **kwargs):
if cls not in instances:
instances[cls] = cls(*args, **kwargs)
return instances[cls]
return get_instance
@singleton
class Database:
def __init__(self, url):
self.connection = connect(url)
```
Clean syntax. But changes the class to a function, which can break `isinstance` checks.
**4. Metaclass:**
```python theme={null}
class SingletonMeta(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super().__call__(*args, **kwargs)
return cls._instances[cls]
class Database(metaclass=SingletonMeta):
pass
```
Most robust. Preserves class semantics, `isinstance` works. But complex.
**The honest answer:** In Python, just use a module-level instance or a function that lazily creates one. The other approaches are mostly academic exercises. "The Borg pattern" (sharing state across instances) is another alternative.
**What interviewers are really testing:** Whether you know multiple approaches AND can articulate that the simple one is usually best.
**Red flag answer:** Reaching for the metaclass approach first.
**Follow-up:**
* "Why is a module-level singleton considered the most Pythonic approach?"
* "Are any of these approaches thread-safe? How would you make them safe?"
* "What is the Borg pattern and when is it better than a traditional Singleton?"
In languages like Java, many design patterns exist to work around the lack of first-class functions. In Python, functions are objects: they can be assigned to variables, passed as arguments, returned from functions, and stored in data structures.
**Strategy pattern (no classes needed):**
```python theme={null}
# Java-style: Strategy interface, ConcreteStrategyA, ConcreteStrategyB, Context class...
# Python-style:
def process_data(data, strategy):
return strategy(data)
process_data(data, sorted)
process_data(data, reversed)
process_data(data, lambda x: sorted(x, key=len))
```
**Command pattern (closures replace command objects):**
```python theme={null}
def make_command(action, *args):
def command():
return action(*args)
return command
undo_stack = []
undo_stack.append(make_command(delete_file, "temp.txt"))
undo_stack.append(make_command(rename_file, "a.txt", "b.txt"))
```
**Observer pattern (callables as observers):**
```python theme={null}
class EventBus:
def __init__(self):
self.listeners = defaultdict(list)
def on(self, event, callback): # callback is any callable
self.listeners[event].append(callback)
def emit(self, event, data):
for callback in self.listeners[event]:
callback(data)
```
**Template Method pattern (just pass the varying step as a function):**
```python theme={null}
def etl_pipeline(extract_fn, transform_fn, load_fn):
data = extract_fn()
transformed = transform_fn(data)
load_fn(transformed)
```
**What interviewers are really testing:** Whether you recognize that Python's design patterns are simpler than GoF patterns and can explain why.
**Red flag answer:** Implementing Java-style Strategy with interfaces and concrete classes in Python.
**Follow-up:**
* "Which GoF patterns are unnecessary in Python? Why?"
* "When IS a class-based strategy better than a function in Python?"
* "How does `functools.partial` relate to the Command pattern?"
This was covered in Section 5, but here is the deeper dive on advanced patterns:
**Class-based (full control):**
```python theme={null}
class Timer:
def __init__(self, label):
self.label = label
def __enter__(self):
self.start = time.perf_counter()
return self # Available as the 'as' variable
def __exit__(self, exc_type, exc_val, exc_tb):
elapsed = time.perf_counter() - self.start
print(f"{self.label}: {elapsed:.3f}s")
return False # Do not suppress exceptions
with Timer("database query") as t:
result = db.query("SELECT * FROM users")
```
**`@contextmanager` (concise):**
```python theme={null}
from contextlib import contextmanager
@contextmanager
def temporary_directory():
path = tempfile.mkdtemp()
try:
yield path # Everything before yield is __enter__
finally:
shutil.rmtree(path) # Everything after yield is __exit__
```
**Advanced: `contextlib.ExitStack` for dynamic resource management:**
```python theme={null}
from contextlib import ExitStack
def process_files(file_paths):
with ExitStack() as stack:
files = [stack.enter_context(open(p)) for p in file_paths]
# All files are open. All will be closed when exiting 'with'.
for f in files:
process(f)
```
**Async context managers (`async with`):**
```python theme={null}
class AsyncDBConnection:
async def __aenter__(self):
self.conn = await connect()
return self.conn
async def __aexit__(self, *exc):
await self.conn.close()
```
**What interviewers are really testing:** Whether you can write both forms and know about `ExitStack` for dynamic cases.
**Red flag answer:** Only knowing `with open()` without being able to create custom context managers.
**Follow-up:**
* "When would you use `ExitStack` instead of nested `with` statements?"
* "How do async context managers differ from sync ones?"
* "What happens if code inside a `@contextmanager`'s `try` block raises an exception?"
Circular imports are one of the most common Python frustrations. Understanding *why* they happen reveals the fix.
**Why circular imports fail:**
When module A imports module B, and B imports A, Python's import machinery partially initializes A before B can access its attributes. B gets a half-built module object, causing `AttributeError` or `ImportError`.
**Solution 1 -- Restructure (best):**
Extract shared dependencies into a third module. If A needs something from B and B needs something from A, there is probably a shared concept that deserves its own module:
```python theme={null}
# Before: a.py imports b.py, b.py imports a.py
# After: both import from common.py
```
**Solution 2 -- Lazy imports (pragmatic):**
```python theme={null}
def process():
from other_module import helper # Import when needed, not at module level
return helper()
```
**Solution 3 -- `TYPE_CHECKING` guard (for type hints only):**
```python theme={null}
from __future__ import annotations # Makes all annotations strings (lazy)
from typing import TYPE_CHECKING
if TYPE_CHECKING: # Only True when mypy/pyright runs, False at runtime
from other_module import OtherClass
def process(obj: "OtherClass") -> None: # String annotation, no runtime import
...
```
This is the standard pattern for type hints that would cause circular imports.
**Solution 4 -- Dependency injection:**
```python theme={null}
class Service:
def __init__(self, repository): # Accept dependency as parameter
self.repository = repository
# Caller assembles dependencies, breaking the circular chain
```
**What interviewers are really testing:** Whether you can diagnose circular imports and know the `TYPE_CHECKING` pattern.
**Red flag answer:** "I just move the import into the function" without understanding why the cycle exists or knowing about `TYPE_CHECKING`.
**Follow-up:**
* "How does `from __future__ import annotations` help with circular imports?"
* "What is the `TYPE_CHECKING` constant and how does it work?"
* "When is restructuring better than lazy imports?"
## 18. Testing and Debugging
The rule is simple: **mock at the boundary**. Mock I/O and external services. Do NOT mock the code under test or its internal collaborators.
**What to mock (external boundaries):**
* Network calls (HTTP APIs, database queries, message queues)
* File system operations
* System clock (`datetime.now()`, `time.time()`)
* Environment variables
* Third-party services (payment gateways, email providers)
**What NOT to mock (internal logic):**
* Your own business logic classes
* Data transformations
* Validation functions
* Anything that does not cross a process/network boundary
**The `patch` target rule (most common mistake):**
```python theme={null}
# mymodule.py
from requests import get
def fetch_data():
return get("https://api.example.com")
# test_mymodule.py
# WRONG: patch where it is defined
@patch('requests.get')
# CORRECT: patch where it is USED
@patch('mymodule.get')
def test_fetch_data(mock_get):
mock_get.return_value.json.return_value = {"data": "test"}
result = fetch_data()
assert result.json() == {"data": "test"}
mock_get.assert_called_once_with("https://api.example.com")
```
`patch` replaces the name at the *import location*, not the *definition location*. Getting this wrong is the #1 source of "my mock is not working" bugs.
**What interviewers are really testing:** Whether you know the boundary rule and the `patch` target rule.
**Red flag answer:** Mocking internal methods or not knowing where to target `patch`.
**Follow-up:**
* "Why do you `patch('mymodule.get')` instead of `patch('requests.get')`?"
* "When would you use a real database in tests instead of mocking?"
* "How do you test code that depends on `datetime.now()`?"
**Stubs** provide canned responses. **Mocks** also verify interactions. In Python's `unittest.mock`, `Mock()` does both, but the distinction matters for test design.
**Stubbing -- provides data, does not verify behavior:**
```python theme={null}
# Stub: "When called, return this data"
mock_db = Mock()
mock_db.get_user.return_value = User(name="Alice", age=30)
# The test verifies the output, not how the mock was called
result = process_user(mock_db, user_id=1)
assert result.name == "Alice"
```
**Mocking -- verifies interactions:**
```python theme={null}
# Mock: "Verify it was called correctly"
mock_email = Mock()
register_user(user_data, email_service=mock_email)
mock_email.send.assert_called_once_with(
to="user@example.com",
subject="Welcome!",
body=ANY # We do not care about exact body
)
```
**When to use each:**
* **Stub** when you care about the *output* of the function under test
* **Mock** when you care about *side effects* (was the email sent? was the cache invalidated?)
**Over-asserting on mock calls (anti-pattern):**
```python theme={null}
# BAD: Testing implementation details
mock_db.query.assert_called_with("SELECT * FROM users WHERE id = %s", (1,))
# This breaks if you refactor the SQL, even if the behavior is correct
# GOOD: Test behavior, not implementation
result = get_user(1)
assert result.name == "Alice"
```
**What interviewers are really testing:** Whether you distinguish testing behavior from testing implementation.
**Red flag answer:** Asserting on every mock call, testing implementation details instead of behavior.
**Follow-up:**
* "When does verifying mock calls become an anti-pattern?"
* "How does `unittest.mock.ANY` help you write less brittle mock assertions?"
* "What is `spec=True` on a Mock and why should you use it?"
Over-mocking is one of the most common testing anti-patterns. It produces tests that pass but give **false confidence** -- they test your mocks, not your code.
**Signs of over-mocking:**
1. **Tests break on refactoring but code still works** -- you changed internal structure, not behavior, but 50 tests fail because mocks expected specific method calls
2. **Mocks return mocks return mocks** -- `mock.get.return_value.json.return_value.data` is a code smell
3. **Tests are harder to read than the code** -- 40 lines of mock setup for 5 lines of actual test
4. **Bugs slip through** -- the real database driver returns `Decimal`, your mock returns `float`, causing a production bug that tests never caught
**Real-world horror story:** A team had 2,000 unit tests, all passing, all heavily mocked. They upgraded their database driver. Zero tests failed. But production crashed on the first query because the new driver returned different types. Their mocks were testing a fantasy world.
**The testing pyramid approach:**
* **Unit tests (70%)**: Test pure logic, no I/O. No mocks needed because there is nothing to mock.
* **Integration tests (20%)**: Test with real database, real file system. Use `pytest-docker` or `testcontainers` for reproducible environments.
* **End-to-end tests (10%)**: Test the full system. Slow but catches integration issues.
**Better alternatives to mocking:**
* **Fakes**: In-memory implementations of interfaces (e.g., `dict`-based repository instead of database repository)
* **Test databases**: SQLite for unit tests, Docker-based PostgreSQL for integration tests
* **Dependency injection**: Design code so dependencies are passed in, not imported
**What interviewers are really testing:** Whether you have experienced the pain of over-mocking and can design a balanced test strategy.
**Red flag answer:** "I mock everything for fast tests." Speed without accuracy is worthless.
**Follow-up:**
* "How do you decide between a mock and a fake?"
* "How do you test database interactions without mocking?"
* "Your team has 5,000 unit tests that all pass, but production bugs keep slipping through. What is likely wrong?"
Effective debugging follows a systematic approach. The best debuggers use the right tool for the situation.
**The debugging decision tree:**
1. **Obvious error with traceback**: Read the traceback bottom-up. The last frame shows where the error occurred. The cause is usually 1-3 frames above.
2. **Logic error (wrong result, no crash)**: Use `breakpoint()` (Python 3.7+) or `import pdb; pdb.set_trace()` to inspect state at the problem point.
3. **Intermittent/timing issue**: Add structured logging with context. Use `logging.debug()` with correlation IDs.
4. **Performance issue**: Profile with `py-spy` or `cProfile` before optimizing.
5. **Memory issue**: Use `tracemalloc` to track allocations.
**Modern Python debugging (`breakpoint()`):**
```python theme={null}
def process_order(order):
total = calculate_total(order.items)
breakpoint() # Drops into pdb (or any configured debugger)
# Set PYTHONBREAKPOINT=0 to disable all breakpoints in production
# Set PYTHONBREAKPOINT=ipdb.set_trace for enhanced debugger
apply_discount(total, order.coupon)
```
**Essential pdb commands:**
* `n` (next) -- execute current line, step over function calls
* `s` (step) -- step INTO function calls
* `c` (continue) -- run until next breakpoint
* `l` (list) -- show source code around current line
* `p expr` -- print expression value
* `pp expr` -- pretty-print (great for dicts/lists)
* `w` (where) -- show full stack trace
* `u`/`d` (up/down) -- navigate stack frames
* `!statement` -- execute arbitrary Python code
**VS Code / PyCharm debugging** is generally more productive than pdb for complex issues -- conditional breakpoints, watch expressions, and variable inspection are far easier in a GUI.
**Post-mortem debugging (after a crash):**
```python theme={null}
# Automatically drop into debugger on unhandled exception
python -m pdb script.py
# Or in code:
import pdb
try:
failing_function()
except Exception:
pdb.post_mortem() # Debug the exception state
```
**What interviewers are really testing:** Whether you use `breakpoint()` and have a systematic debugging approach.
**Red flag answer:** "I add print statements everywhere." This works but is the least efficient approach.
**Follow-up:**
* "What is `breakpoint()` and how does `PYTHONBREAKPOINT` environment variable work?"
* "How do you debug an issue that only happens in production?"
* "Walk me through how you would debug a function that returns an incorrect result."
Logging is the primary debugging tool for production systems. But most developers configure it incorrectly.
**Production-grade logging setup:**
```python theme={null}
import logging
import json
# Structured JSON logging (parseable by ELK, Datadog, CloudWatch)
class JSONFormatter(logging.Formatter):
def format(self, record):
return json.dumps({
"timestamp": self.formatTime(record),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
"module": record.module,
"function": record.funcName,
"line": record.lineno,
})
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
```
**Log levels and when to use each:**
* `DEBUG`: Detailed diagnostic info. Disabled in production.
* `INFO`: Normal operation milestones. "User created", "Payment processed".
* `WARNING`: Something unexpected but not an error. "Retry attempt 2/3", "Disk 80% full".
* `ERROR`: Something failed but the service continues. "Failed to send email, will retry".
* `CRITICAL`: Service is about to crash. "Database connection pool exhausted".
**The `%s` vs f-string performance trick:**
```python theme={null}
# GOOD: String not formatted if level is filtered out
logger.debug("Processing user %s with %d items", user_id, len(items))
# BAD: f-string is always formatted, even if debug logging is disabled
logger.debug(f"Processing user {user_id} with {len(items)} items")
```
With `%s` formatting, the string is only built if the log level is active. With f-strings, it is always built. This matters in hot loops.
**Correlation IDs for distributed tracing:**
```python theme={null}
import uuid
def handle_request(request):
request_id = str(uuid.uuid4())
logger = logging.getLogger(__name__)
logger = logging.LoggerAdapter(logger, {"request_id": request_id})
logger.info("Request received", extra={"path": request.path})
# All logs from this request share the same request_id
```
**What interviewers are really testing:** Structured logging, log levels, and the `%s` performance pattern.
**Red flag answer:** Using `print()` in production code, or only using `logging.basicConfig()`.
**Follow-up:**
* "Why use `%s` formatting instead of f-strings in logging calls?"
* "How do you correlate logs across microservices?"
* "How would you set up logging that outputs JSON for a production ELK stack?"
Performance debugging follows a strict methodology: **measure, identify, optimize, measure again.**
**Step 1 -- Identify the type of bottleneck:**
* **CPU-bound**: High CPU usage, slow computation. Profile with `cProfile` or `py-spy`.
* **I/O-bound**: Low CPU, waiting for network/disk. Profile with `asyncio` debug mode or logging timestamps.
* **Memory-bound**: Growing RSS, GC pauses. Profile with `tracemalloc`.
**Step 2 -- Profile with the right tool:**
```python theme={null}
# cProfile -- function-level CPU profiling (built-in)
python -m cProfile -s cumulative script.py
# Output: function call counts and cumulative time
# py-spy -- sampling profiler (zero-overhead, attach to running process)
py-spy record -o profile.svg -- python script.py
# Output: flame graph SVG (visual, intuitive)
# line_profiler -- line-by-line timing (surgical)
@profile # Add this decorator
def slow_function():
data = load_data() # 0.1s
processed = transform(data) # 45.2s <-- found it!
save(processed) # 0.3s
# timeit -- micro-benchmarks (compare approaches)
python -m timeit -s "lst = list(range(10000))" "sum(lst)"
python -m timeit -s "import numpy as np; arr = np.arange(10000)" "np.sum(arr)"
```
**Step 3 -- Common optimizations (in order of impact):**
1. **Algorithm change**: O(n^2) to O(n log n) is the biggest win possible
2. **Data structure change**: `list` `in` check to `set` `in` check (O(n) to O(1))
3. **Caching**: `@functools.lru_cache` for pure functions called repeatedly
4. **Vectorization**: Replace Python loops with NumPy/pandas operations
5. **Concurrency**: Right model for the task (threading/async for I/O, multiprocessing for CPU)
6. **C extension**: Cython, pybind11, or Rust via PyO3 (last resort)
**What interviewers are really testing:** Whether you profile first and optimize based on data.
**Red flag answer:** "I would use Cython" without having profiled first.
**Follow-up:**
* "How do you read a flame graph? What pattern indicates a performance issue?"
* "Your function is slow but `cProfile` shows no single slow function. What could cause this?"
* "How does `py-spy` attach to a running process without restarting it?"
Memory leaks in Python usually fall into three categories: unbounded caches, circular references, and C extension leaks.
**Step 1 -- Detect the leak:**
```python theme={null}
import tracemalloc
tracemalloc.start(25) # Track 25 frames deep
# Take baseline
snapshot1 = tracemalloc.take_snapshot()
# ... run code that you suspect leaks ...
# Compare
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
for stat in top_stats[:10]:
print(stat)
# Shows: which lines allocated the most NEW memory between snapshots
```
**Step 2 -- Identify what objects are accumulating:**
```python theme={null}
import objgraph
objgraph.show_most_common_types(limit=20)
# Output: dict: 50000, list: 30000, str: 25000...
# If a type count keeps growing, you have a leak
objgraph.show_growth(limit=10) # Show types that increased since last call
```
**Step 3 -- Find what is holding references:**
```python theme={null}
# Visualize reference chain for suspicious objects
objgraph.show_backrefs(
objgraph.by_type('MyLeakyClass')[:3],
max_depth=5,
filename='refs.png'
)
```
**Common leak patterns and fixes:**
1. **Unbounded cache**: `@lru_cache(maxsize=None)` or plain dict used as cache. Fix: set `maxsize` or use `cachetools.TTLCache`.
2. **Closures capturing large objects**: Lambda in a loop captures a DataFrame. Fix: extract the value, do not close over the large object.
3. **Global collections**: Module-level lists/dicts that `append` but never `pop`. Fix: use bounded collections or weak references.
4. **C extension leak**: `tracemalloc` cannot see it. Monitor RSS via `/proc/self/status` and compare with `tracemalloc` totals. The difference is C-level allocations.
5. **Circular references with `__del__`**: Before Python 3.4, these were uncollectable. After 3.4, the GC handles them, but `__del__` can still cause issues by resurrecting objects.
**What interviewers are really testing:** Whether you have a systematic approach to memory debugging using specific tools.
**Red flag answer:** "Call `gc.collect()` in a loop" (treats the symptom, not the cause).
**Follow-up:**
* "How do you distinguish a Python-level memory leak from a C extension leak?"
* "Your service RSS grows by 1GB/day. Walk me through your debugging approach."
* "Why might `gc.collect()` not free memory, and why might freed Python memory not reduce RSS?"
## Advanced Scenario-Based Questions
**Scenario:** Your team deployed a data-processing microservice that scores ML models in real time. It spawns 8 worker threads via `ThreadPoolExecutor`, yet `htop` shows a single core pegged at 100% while the other 7 sit idle. P99 latency is climbing toward your SLA. Your manager asks why the 8-core box is basically a single-core machine.
**What weak candidates say:**
* "Just add more threads" or "Increase the thread pool size to 64."
* Vaguely mention the GIL without explaining *why* it applies here or what concrete alternatives exist.
* Suggest rewriting the whole service in Go or Rust as the only option.
**What strong candidates say:**
* "This is the GIL in action. Because the workload is CPU-bound (NumPy scoring that drops back into Python between calls, or pure-Python feature preprocessing), the GIL serializes bytecode execution across all 8 threads. You get concurrency but not parallelism."
* "First, I would profile with `py-spy` or `cProfile` to confirm the hot path is actually CPU-bound Python, not I/O. If it is, the fix depends on context:"
* **Quick win**: Switch from `ThreadPoolExecutor` to `ProcessPoolExecutor`. Each worker process gets its own GIL. Serialization cost for IPC is the trade-off — measure it.
* **Medium effort**: Offload the hot loop to a C extension, Cython, or ensure the NumPy/SciPy calls release the GIL (most do via `Py_BEGIN_ALLOW_THREADS`). Verify with `nogil=True` in Cython.
* **Structural fix**: Move to a multi-process architecture with `gunicorn --workers 8` or Celery workers, one per core, with a Redis/RabbitMQ broker.
* **Nuclear option**: Evaluate the free-threaded CPython 3.13+ build (`python --disable-gil`), but acknowledge it is experimental and many C extensions are not yet safe.
* "In a previous role, we had a similar issue with a feature-engineering pipeline. Switching to `ProcessPoolExecutor` with 4 workers on a 4-core box dropped P99 from 850ms to 210ms. The gotcha was that our model object was 400MB and pickling it for each task killed throughput — we fixed that by loading the model once per worker process using an initializer function."
**Follow-up:**
1. "If your NumPy scoring already releases the GIL, but the feature preprocessing between calls is pure Python, how do you handle the mixed workload?"
2. "What is the memory overhead of `ProcessPoolExecutor` with 8 workers each loading a 500MB model? How would you mitigate it?"
3. "Explain what `Py_BEGIN_ALLOW_THREADS` actually does at the C level and why it is safe for NumPy but dangerous for code that touches Python objects."
**Scenario:** Your ETL pipeline runs as a long-lived Python process consuming Kafka messages. Over 48 hours, RSS grows from 500MB to 12GB until the OOM killer terminates it. Restarting resets the cycle. Your on-call team is tired of 3 AM pages.
**What weak candidates say:**
* "Use `del` on variables" or "Call `gc.collect()` in a loop."
* Blame Python for being a memory-hungry language.
* Suggest just restarting the service on a cron schedule (treating the symptom, not the cause).
**What strong candidates say:**
* "I would approach this systematically with three tools in sequence:"
* **Step 1 — `tracemalloc`**: Enable it in production with `tracemalloc.start(25)` (25 frames deep). Take snapshots every 10 minutes and compare with `snapshot.compare_to(old_snapshot, 'lineno')`. This shows which lines are allocating the most *new* memory between snapshots.
* **Step 2 — `objgraph`**: Once I have a suspect module, use `objgraph.show_most_common_types()` to see which object types are accumulating. Then `objgraph.show_backrefs(objgraph.by_type('SuspectClass')[:5])` to visualize what is holding references.
* **Step 3 — `gc` module**: Check `gc.garbage` for uncollectable objects (those with `__del__` methods in reference cycles). Run `gc.set_debug(gc.DEBUG_SAVEALL)` in a staging environment to catch them.
* "The most common culprits I have seen in production:"
* **Caches without eviction**: An `lru_cache` with `maxsize=None` (unbounded), or a plain dict used as a cache that grows forever. Fix: set a maxsize or use `cachetools.TTLCache`.
* **Closures capturing large objects**: A lambda or callback inside a loop that accidentally closes over a large DataFrame. The DataFrame stays alive as long as any callback references it.
* **Circular references with `__del__`**: Pre-Python-3.4, these were uncollectable. Post-3.4, the GC handles them, but custom `__del__` methods can still prevent collection if they resurrect objects.
* **C extension leaks**: If using libraries like `lxml` or database drivers, memory may leak in C code that `tracemalloc` cannot see. Use `memory_profiler` alongside `/proc/self/smaps` to compare Python-tracked vs OS-tracked RSS.
* "At a previous company, we traced a 10GB/day leak to a logging handler that appended formatted log records to an in-memory list (someone left a `MemoryHandler` with no target). `objgraph` showed 8 million `str` objects. Two-line fix, saved us \$400/month in oversized EC2 instances."
**Follow-up:**
1. "What is the difference between RSS, VMS, and USS, and which one should you actually monitor for a Python memory leak?"
2. "Why might `gc.collect()` not reclaim memory, and why might the process RSS not drop even after Python frees objects?"
3. "How would you detect a memory leak in a C extension that `tracemalloc` cannot track?"
**Scenario:** You have a FastAPI service handling 2,000 concurrent WebSocket connections. Under load testing at 5,000 connections, the entire event loop freezes: health check endpoints stop responding, existing connections time out, and Kubernetes kills the pod. Logs show no exceptions. CPU is at 5%.
**What weak candidates say:**
* "Add more asyncio tasks" or "Increase the connection limit."
* Confuse asyncio with threading and suggest adding locks.
* Cannot explain what "blocking the event loop" actually means.
**What strong candidates say:**
* "Classic event loop starvation. The CPU is at 5% which rules out a CPU-bound bottleneck — something is blocking the single event loop thread, preventing it from servicing other coroutines. The three most common causes:"
* **Synchronous I/O in an async handler**: Someone called `requests.get()`, `time.sleep()`, or a synchronous database driver inside an `async def` handler instead of using `aiohttp`, `asyncio.sleep()`, or an async driver. Even one blocking call stalls *every* coroutine.
* **CPU-bound work on the event loop**: JSON serialization of a 50MB payload, or a regex over a massive string, running directly in a coroutine without offloading.
* **Deadlock from incorrect `await` chains**: Two coroutines waiting on each other's `asyncio.Event` or `asyncio.Lock`, or calling `loop.run_until_complete()` from inside an already-running loop (common when mixing sync and async code).
* "To diagnose, I would:"
* Enable `asyncio` debug mode (`PYTHONASYNCIODEBUG=1` or `loop.set_debug(True)`) which logs coroutines that take longer than 100ms without yielding.
* Use `py-spy` to get a live stack trace of the stuck event loop thread — it will show exactly which blocking call it is sitting on.
* Add `loop.slow_callback_duration = 0.05` and watch warnings.
* "The fix depends on the root cause:"
* Blocking I/O: Replace with async equivalent, or wrap with `asyncio.to_thread()` (Python 3.9+) / `loop.run_in_executor()` to push it to a thread pool.
* CPU-bound work: Offload to `ProcessPoolExecutor` via `loop.run_in_executor()`.
* Deadlock: Restructure the `await` chain. Never call `loop.run_until_complete()` from within a running loop — use `await` directly.
* "I once debugged a production freeze caused by a single `dns.resolver.resolve()` call (synchronous `dnspython`) inside an async handler. Under load, DNS lookups took 2-5 seconds each, completely blocking the event loop. Replacing it with `aiodns` fixed the freeze and dropped P99 latency from timeout to 40ms."
**Follow-up:**
1. "What exactly happens internally when you `await asyncio.sleep(0)`, and why do people sprinkle it in long-running coroutines?"
2. "Explain the difference between `asyncio.to_thread()`, `loop.run_in_executor()`, and just spawning a raw thread. When would you pick each?"
3. "Can you have a deadlock in asyncio without using any explicit locks? Describe a scenario."
**Scenario:** You add a new file `utils/cache.py` to your Django project. Suddenly, an unrelated view that imports from `utils.helpers` crashes with `AttributeError: module 'utils' has no attribute 'helpers'`. The attribute clearly exists — `utils/helpers.py` is right there, unchanged. Reverting `cache.py` fixes it. What is happening?
**What weak candidates say:**
* "There is a typo in the import" or "The file is not saved."
* Cannot explain how Python's import system actually resolves modules.
* Suggest deleting `__pycache__` and restarting (sometimes works, but they cannot explain why).
**What strong candidates say:**
* "This is almost certainly a **circular import**. The import system is one of the trickiest parts of CPython. Here is what is likely happening:"
* When Python imports `utils.cache`, that module's top-level code runs. If `cache.py` imports something from `utils.helpers` at the top level, and `helpers.py` (or something it imports) tries to import from `utils.cache`, you get a partially-initialized module.
* Python's import system uses `sys.modules` as a cache. When a module is *being* imported, a partially-initialized module object is placed in `sys.modules`. If another module tries to import from it before its top-level code finishes, it gets the half-built module — which may be missing attributes that have not been defined yet.
* The `AttributeError` says `'utils' has no attribute 'helpers'` because the `utils` *package* `__init__.py` has not finished executing — it is mid-import and the `helpers` submodule has not been bound to the package namespace yet.
* "To debug: add `print(f'Importing {__name__}')` at the top of each suspect module and read the order. Or use `python -v` to see the import sequence. The cycle will be obvious."
* "Fixes, in order of preference:"
* **Refactor**: Extract the shared dependency into a new module that both `cache.py` and `helpers.py` import. Break the cycle architecturally.
* **Lazy import**: Move the problematic import inside the function that uses it, so it runs at call time, not import time.
* **`importlib.import_module()`**: Defer the import programmatically — same idea as lazy import but sometimes cleaner for dynamic cases.
* **Restructure `__init__.py`**: If `__init__.py` does `from .helpers import *`, that is often the trigger. Keep `__init__.py` minimal.
* "One subtle variant: if you name a file `utils/cache.py` and there is *also* a third-party package called `cache`, you can get shadowing where Python imports your local file instead of the library, or vice versa, depending on `sys.path` order. Always check `module.__file__` to confirm you are importing what you think you are."
**Follow-up:**
1. "Walk me through the exact sequence of steps CPython takes when it encounters `import utils.helpers` — what does it check, and in what order?"
2. "What is the difference between a regular package (with `__init__.py`) and a namespace package (PEP 420)? How does it affect import resolution?"
3. "You mentioned `sys.modules`. What happens if you manually delete an entry from `sys.modules` and re-import the module? What are the dangers?"
**Scenario:** Your team built a lightweight ORM using metaclasses. Developers report that `UserModel.fields` contains fields from `OrderModel` — models are somehow sharing state. The bug is intermittent and only appears when both models are imported in the same module. Test suites pass because each test file imports only one model.
**What weak candidates say:**
* "I would not use metaclasses" (avoids the question entirely).
* Cannot explain what a metaclass `__new__` or `__init__` does during class creation.
* Suggest adding print statements with no systematic approach.
**What strong candidates say:**
* "This is a classic mutable-default-on-the-metaclass bug. The most common cause: the metaclass stores field definitions in a *shared* mutable object (like a class-level list or dict on the metaclass itself) instead of creating a fresh one per class. Here is the typical pattern:"
```python theme={null}
# BUG: shared mutable state
class ModelMeta(type):
fields = [] # This list is shared across ALL classes created by this metaclass
def __new__(mcs, name, bases, namespace):
cls = super().__new__(mcs, name, bases, namespace)
for key, value in namespace.items():
if isinstance(value, Field):
mcs.fields.append(key) # Appends to the SHARED list
cls.fields = mcs.fields # Every class gets the SAME list
return cls
```
* "The fix is to create a new list per class in `__new__`:"
```python theme={null}
# FIXED: per-class field list
class ModelMeta(type):
def __new__(mcs, name, bases, namespace):
cls = super().__new__(mcs, name, bases, namespace)
cls.fields = []
for key, value in namespace.items():
if isinstance(value, Field):
cls.fields.append(key)
return cls
```
* "To debug this in the wild, I would:"
* Check `id(UserModel.fields) == id(OrderModel.fields)` — if True, they are literally the same object in memory.
* Use `UserModel.__class__` to confirm the metaclass, then inspect the metaclass's `__new__` and `__init__` for shared mutable state.
* Check if the metaclass inherits from another metaclass that introduces shared state.
* Look for `__init_subclass__` hooks that might mutate a parent class's attributes.
* "This same pattern bites people with Django `ModelForm.Meta` if you accidentally mutate `fields` at class creation time instead of copying. The broader lesson: any time you are in `__new__` or `__init__` of a metaclass, every mutable object must be explicitly copied or freshly created per class. Never assign a metaclass-level mutable to a class attribute by reference."
**Follow-up:**
1. "What is the difference between `__new__` and `__init__` on a metaclass versus on a regular class? When does each run?"
2. "How does `__init_subclass__` (PEP 487) reduce the need for metaclasses? Give an example where you would refactor a metaclass to use it instead."
3. "What happens if two metaclasses conflict — for example, your ORM metaclass and an ABC metaclass? How does Python resolve metaclass conflicts?"
**Scenario:** Your image-processing service uses Pillow and a custom C extension for fast EXIF parsing. In production (Ubuntu, Python 3.11, 16 workers under gunicorn), you get intermittent SIGSEGV crashes — roughly 2 per day. Core dumps show the crash in your C extension. It never reproduces locally on macOS with a single process. Your team is afraid to touch C code.
**What weak candidates say:**
* "Rewrite the C extension in pure Python."
* "Add try/except around the C call" (SIGSEGV is not a Python exception — it kills the process).
* Have no idea how to read a core dump or use debugging tools for C extensions.
**What strong candidates say:**
* "Intermittent segfaults in C extensions that only appear under multi-process/multi-worker conditions usually point to one of three things:"
* **Thread safety**: If gunicorn uses `gthread` workers, multiple threads may call the C extension concurrently. If the C code uses static/global variables without locks, you get data races and memory corruption.
* **Fork safety**: If gunicorn uses `prefork` workers, the C extension is initialized *before* forking. Some C libraries (e.g., OpenSSL, certain database drivers) are not fork-safe — they hold file descriptors, mutexes, or memory-mapped regions that become invalid after `fork()`. Fix: use gunicorn's `post_fork` hook to reinitialize, or switch to `--preload` with careful init.
* **Buffer overflow / use-after-free**: The EXIF parser reads variable-length metadata. A malformed JPEG with unexpected EXIF data could cause the C code to read past a buffer.
* "My debugging approach:"
* **Step 1**: Analyze the core dump with `gdb python core` followed by `bt` (backtrace). With debug symbols (`python3-dbg` package), this shows the exact C function and line number.
* **Step 2**: Run under Valgrind in staging: `valgrind --tool=memcheck python your_script.py`. This catches invalid reads/writes, use-after-free, and leaks. Expect it to be 20-50x slower.
* **Step 3**: If it is a data-dependent crash, collect the JPEG that triggered it from the request logs and write a regression test.
* **Step 4**: Use `faulthandler` module (`python -X faulthandler`) to get a Python-level traceback even on SIGSEGV — this shows you which Python function was calling into the C code.
* **Step 5**: Compile the C extension with AddressSanitizer (`-fsanitize=address`) and run the test suite. ASAN catches buffer overflows, use-after-free, and stack corruption with minimal overhead compared to Valgrind.
* "In a real incident, we had a C extension that cached parsed results in a `static PyObject*`. It was initialized in the first worker before fork, and after fork all child processes shared the pointer. When one child's GC collected the object, the others segfaulted on access. Fix: move initialization into `post_fork` and remove the static cache."
**Follow-up:**
1. "What is the difference between `faulthandler.enable()` and a regular Python traceback? Why can `faulthandler` print something useful during a segfault when a normal except block cannot?"
2. "Explain the GIL's role in C extension safety. If your C extension calls `Py_BEGIN_ALLOW_THREADS`, what are you now responsible for that the GIL previously protected?"
3. "Your C extension needs to create Python objects. Walk me through the reference counting rules: when do you call `Py_INCREF` vs `Py_DECREF`, and what is a 'borrowed reference' vs a 'new reference'?"
**Scenario:** Your data science team has a monorepo with 3 services. Developers use macOS (ARM), CI runs Ubuntu x86, and production is Amazon Linux 2. The `requirements.txt` has unpinned transitive dependencies. Every week someone's build breaks because `numpy 2.0` dropped and broke `pandas`, or a compiled wheel is not available for one platform. A new hire spent two days just getting the dev environment working.
**What weak candidates say:**
* "Just `pip freeze > requirements.txt`" (captures one platform's resolved versions, breaks on another).
* "Use Docker" as the only answer (does not address the dependency management problem itself).
* Cannot distinguish between `requirements.txt`, `setup.py`, `pyproject.toml`, `Pipfile`, and `poetry.lock`.
**What strong candidates say:**
* "This is a dependency resolution and lockfile problem. The Python packaging ecosystem has historically been terrible at this compared to npm/yarn or Cargo, but modern tools solve it. Here is what I would do:"
* **Step 1 — Single source of truth**: Move to `pyproject.toml` (PEP 621) for declaring dependencies with loose constraints (e.g., `numpy>=1.24,<2.0`). This replaces `setup.py`, `setup.cfg`, and `requirements.txt` for dependency declaration.
* **Step 2 — Lockfile with platform awareness**: Use **uv** (fastest, Rust-based), **pip-tools** (`pip-compile`), or **Poetry** to generate a lockfile with pinned transitive dependencies. `uv lock` generates a universal lockfile that works across platforms. `pip-compile --generate-hashes` pins versions AND validates integrity.
* **Step 3 — Per-platform resolution** (if needed): For packages with compiled wheels (NumPy, scipy), you may need separate lock files per platform: `requirements-linux-x86.txt`, `requirements-macos-arm.txt`. `uv` handles this natively with its universal resolver.
* **Step 4 — CI enforcement**: CI should install from the lockfile (`uv sync --frozen` or `pip install -r requirements.txt --require-hashes`), never resolve fresh. If lockfile is out of date, CI fails.
* **Step 5 — Docker for parity**: Use a multi-stage Dockerfile where the build stage resolves/installs dependencies and the runtime stage copies the installed virtualenv. This ensures production matches CI.
* "The specific tool choice matters:"
* **uv**: My current recommendation. 10-100x faster than pip, universal lockfile, drop-in pip replacement. Handles virtualenvs, resolution, and installation.
* **Poetry**: Mature, good lockfile, but slower and sometimes fights with pip-installed packages.
* **pip-tools**: Minimal, just `pip-compile` and `pip-sync`. Works if you want to stay close to pip.
* "For the monorepo specifically: use workspace support (uv workspaces or Poetry monorepo plugins) so the 3 services share a single lockfile but declare their own dependencies. This prevents version conflicts where service A needs `pandas 1.5` and service B needs `pandas 2.0`."
* "At a previous company, switching from bare `requirements.txt` to `uv lock` with hashes cut our 'broken build' tickets from 5/week to zero and reduced CI install time from 4 minutes to 18 seconds."
**Follow-up:**
1. "What is the difference between a wheel and an sdist? Why does `numpy` install in 2 seconds on some platforms but takes 5 minutes compiling on others?"
2. "Explain what `--require-hashes` protects against. What attack vector does it mitigate that pinned versions alone do not?"
3. "Your data scientist needs to add a package that conflicts with an existing pinned dependency. Walk me through how you would resolve this without breaking other services."
**Scenario:** Your company's main backend is a 200,000-line Python monolith. Management wants type safety after a production incident caused by passing a `str` where an `int` was expected. You enable `mypy --strict` and get 14,000 errors. Three senior engineers say it is a waste of time. The intern says "just add `type: ignore` everywhere." How do you proceed?
**What weak candidates say:**
* "Just fix all 14,000 errors" (unrealistic timeline, team will revolt).
* "Add `type: ignore` to everything" (defeats the purpose).
* Cannot explain the difference between `mypy`, `pyright`, `pytype`, or when to use each.
* Think type hints are enforced at runtime by default.
**What strong candidates say:**
* "Adopting strict typing on a large untyped codebase is a *migration*, not a switch flip. I would use an incremental approach:"
* **Phase 1 — Baseline without breaking CI (Week 1):**
Set up `mypy` in CI in *non-blocking* mode. Use `mypy.ini` with `disallow_untyped_defs = false` globally. Generate a baseline error count. Configure per-module overrides so new modules are strict from day one.
* **Phase 2 — Gradual strictness with per-module config (Weeks 2-4):**
Use mypy's `[[tool.mypy.overrides]]` in `pyproject.toml` to enable strict mode *per module*. Start with the module that caused the production incident. Use `monkeytype` or `pytype` to auto-generate type stubs from runtime traces — this handles 60-70% of annotations automatically.
* **Phase 3 — CI enforcement on new code (Month 2):**
Use `mypy --warn-unused-ignores` so `type: ignore` comments do not linger. Block PRs that add new untyped functions to already-typed modules. Use a `pre-commit` hook with `mypy --incremental` (fast, only checks changed files).
* **Phase 4 — Ratchet mechanism (Ongoing):**
Track total mypy error count in CI. New PRs must not *increase* the count. This naturally drives the number to zero over time without requiring a dedicated sprint. Tools like `mypy-baseline` or a simple `wc -l` on mypy output make this easy.
* "For the pushback from senior engineers, I would address their actual concerns:"
* "It slows me down": `monkeytype run` + `monkeytype apply` auto-annotates from tests. `pyright` is faster than mypy for editor integration. Show them that typed code catches bugs *before* code review.
* "Python is not supposed to be typed": Show PEP 484, 526, 544. Typing is opt-in and gradual by design. Protocols (PEP 544) give you structural typing that respects duck typing.
* "It does not catch real bugs": Share the production incident that triggered this initiative. Show a demo where mypy catches `Optional` misuse (`None` passed where a value is expected) — the #1 production error class in untyped Python.
* "Tool selection matters at scale:"
* **mypy**: Most mature, best ecosystem. Daemon mode (`dmypy`) for fast incremental checks. Use `--install-types` to auto-install type stubs.
* **pyright**: Faster, better IDE integration (VS Code / Pylance), stricter by default. Written in TypeScript, harder to extend.
* **pytype**: Google's tool. Can *infer* types without annotations — great for generating initial annotations on a legacy codebase.
* "At a 150k-line codebase, we went from 0% to 85% typed coverage in 6 months using the ratchet approach. Mypy caught 23 bugs in code review that quarter — 3 of which would have been P1 production incidents based on the code paths involved."
**Follow-up:**
1. "What is the difference between `Protocol` (PEP 544) and `ABC`? When would you choose one over the other for type checking?"
2. "Explain `TypeVar`, `Generic`, and `ParamSpec`. How would you type a decorator that preserves the decorated function's signature?"
3. "Your codebase uses dynamic patterns like `setattr`, `getattr`, and `**kwargs` heavily. How does mypy handle these, and what are your options when it cannot infer types?"
## Conclusion and Interview Tips
This guide covers 150+ essential Python interview questions across all major categories -- basics, data structures, OOP, functions, file handling, exception handling, modules, advanced concepts, Python internals, memory management, descriptors, generators, concurrency, design patterns, architecture, testing, web development, and data science. Together, they represent the full range of what you will encounter in Python-focused technical interviews from startups through FAANG.
### Key Interview Preparation Tips
* **Master Python fundamentals before diving into frameworks.** An interviewer can tell within minutes whether you truly understand Python or just know Django's API. The GIL, mutability vs immutability, and the difference between `is` and `==` are tested at every level.
* **Practice coding problems in Python specifically.** Python's idioms (list comprehensions, generator expressions, `collections` module) let you write cleaner solutions than direct translations from other languages. Using `Counter`, `defaultdict`, and `heapq` naturally signals Python fluency.
* **Build real projects and be ready to discuss the decisions you made.** A deployed Flask/FastAPI service with proper error handling, logging, and tests demonstrates more than 100 solved LeetCode problems. Be prepared to explain your architecture choices and what you would change with hindsight.
* **Understand memory management and the GIL deeply.** These are the two topics that most reliably separate mid-level from senior Python candidates. Know when the GIL matters (CPU-bound work), when it does not (I/O-bound work), and the practical implications for choosing between threading, multiprocessing, and asyncio.
* **Study the standard library thoroughly.** Python's "batteries included" philosophy means the standard library covers an enormous range. Knowing about `functools.lru_cache`, `contextlib.contextmanager`, `itertools`, `pathlib`, and `dataclasses` without needing to look them up signals deep familiarity.
* **Prepare for both theoretical and practical questions.** Some interviews ask "explain the descriptor protocol," others ask "write a function that..." Be ready for both. The strongest candidates connect theory to practice: "The descriptor protocol is what makes `@property` work, and here is how I used it to..."
### During the Interview
* **Ask clarifying questions before coding.** "Should this handle concurrent access?" or "What is the expected input size?" shows you think about context before writing code.
* **Explain your thought process continuously.** "I am choosing a dictionary here because lookups are O(1) and we need to check membership frequently" is far stronger than silently writing code.
* **Write Pythonic code, not Java-in-Python.** Using list comprehensions over manual loops, `with` statements for resource management, and f-strings over string concatenation signals Python fluency. Interviewers notice this.
* **Consider edge cases and error handling.** What happens with empty input? What about None values? What if the file does not exist? Mentioning these proactively demonstrates production-level thinking.
* **Discuss time and space complexity for every solution.** Even when not asked, briefly stating the complexity shows algorithmic maturity. "This is O(n) time and O(n) space due to the hash set" takes 3 seconds and makes a strong impression.
* **Be honest about gaps but show your reasoning.** "I have not used asyncio in production, but I understand it uses an event loop similar to Node.js, and I would choose it over threading for I/O-bound workloads because of the GIL" shows that you can reason from principles even about unfamiliar territory.
### What Separates Good from Great Python Candidates
| Good Candidate | Great Candidate |
| ------------------- | ------------------------------------------------------------------------------------------------------ |
| Knows list vs tuple | Explains when tuple's immutability enables hashability and use as dict keys |
| Can write a class | Understands `__slots__`, descriptors, and when to use `@dataclass` instead |
| Knows about the GIL | Can explain when the GIL does and does not matter, and chooses the right concurrency model accordingly |
| Uses `try/except` | Follows EAFP over LBYL and can explain why, with concrete examples |
| Knows `pip install` | Understands virtual environments, dependency resolution, and why `poetry` or `uv` exist |
Python interviews test not just language knowledge but also problem-solving approach, code quality, and understanding of computer science fundamentals. The strongest candidates write code that is simultaneously correct, readable, and Pythonic -- and can explain the reasoning behind every choice they make.
Good luck with your Python interviews! Remember that consistent practice and building real projects are the best ways to prepare.