Skip to main content

Documentation Index

Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt

Use this file to discover all available pages before exploring further.

Python Decorators

Advanced Python

Ready to move beyond the basics? These features are what distinguish intermediate Python developers from experts. They allow you to write code that is efficient, clean, and “Pythonic”.

1. Decorators

Decorators are a powerful way to modify the behavior of functions or classes without changing their source code. They are essentially wrappers. Think of a decorator like a “middleware” for a function.
def timer(func):
    import time
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs) # Call the original function
        end = time.time()
        print(f"{func.__name__} took {end - start:.4f}s")
        return result
    return wrapper

# Apply the decorator using @
@timer
def slow_function():
    import time
    time.sleep(1)
    print("Done")

slow_function()
# Output:
# Done
# slow_function took 1.0012s

2. Generators

Lists store all their data in memory. If you have a list of 1 billion numbers, you’ll run out of RAM. Generators produce data one item at a time, on demand. They are lazy. They use the yield keyword instead of return.
def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a      # Pause here and return 'a'
        a, b = b, a + b

# Usage
for num in fibonacci(10):
    print(num)

# Generator Expression (Like list comp, but with parenthesis)
# This takes almost 0 memory, even for 1 million items!
squares = (x**2 for x in range(1000000))

3. Context Managers (with)

Managing resources (files, network connections, database locks) is hard. You have to remember to close them, even if errors occur. Context Managers handle this automatically.
# The 'with' statement ensures f.close() is called
# even if an error happens inside the block.
with open("file.txt", "w") as f:
    f.write("Hello")

# Creating a custom Context Manager
from contextlib import contextmanager

@contextmanager
def database_connection():
    print("Connecting...")
    yield "db_conn"
    print("Closing...") # This runs after the 'with' block ends

with database_connection() as db:
    print(f"Using {db}")

4. AsyncIO (Asynchronous Programming)

Python is single-threaded. However, using asyncio, you can write concurrent code that handles I/O (like network requests) efficiently. While one request is waiting for a response, Python can start another. This is essential for modern web frameworks like FastAPI.
import asyncio

# Define a coroutine with 'async def'
async def fetch_data(id):
    print(f"Fetching {id}...")
    await asyncio.sleep(1) # Simulate I/O wait (non-blocking)
    print(f"Done {id}")
    return {"id": id}

async def main():
    # Run tasks concurrently!
    # This will take ~1 second total, not 3.
    results = await asyncio.gather(
        fetch_data(1),
        fetch_data(2),
        fetch_data(3)
    )
    print(results)

if __name__ == "__main__":
    asyncio.run(main())

5. Type Hinting (Advanced)

For complex data structures, basic type hints (int, str) aren’t enough. The typing module provides tools to describe shapes of data.
from typing import List, Dict, Optional, Union, Callable

# A list of integers
def process_items(items: List[int]) -> Dict[str, int]:
    return {"sum": sum(items)}

# Optional: Can be string OR None
def find_user(id: int) -> Optional[str]:
    return "Alice" if id == 1 else None

# Union: Can be int OR float
Number = Union[int, float]

def add(a: Number, b: Number) -> Number:
    return a + b

Summary

  • Decorators: Wrap functions to add logic (logging, timing, auth).
  • Generators: Lazy iteration for memory efficiency.
  • Context Managers: with statement for safe resource management.
  • AsyncIO: Concurrency for I/O-bound tasks.
  • Type Hints: Essential for large codebases.
Congratulations! You’ve completed the Python Crash Course. You now possess the tools to write robust, efficient, and modern Python applications.

Interview Deep-Dive

Strong Answer:
  • A decorator is syntactic sugar for wrapping a function. When you write @timer above def slow_function():, Python executes slow_function = timer(slow_function). The decorator receives the original function, returns a new function (the wrapper), and the original name now points to the wrapper. There is no special magic — it is plain function composition.
  • Stacking decorators applies them bottom-up. If you write @auth then @timer above a function, Python executes func = auth(timer(func)). So timer wraps the function first, then auth wraps the timed version. The order matters: if auth rejects the request early, the timer never runs. If you reverse them, you time the auth check too. This evaluation order trips up even experienced developers.
  • The most common pitfall is that the wrapper replaces the original function’s metadata. After decoration, slow_function.__name__ returns "wrapper" instead of "slow_function", and slow_function.__doc__ returns the wrapper’s docstring (usually None). This breaks introspection, logging, debugging, and any framework that uses function names for routing (Flask, FastAPI). The fix is @functools.wraps(func) on the wrapper, which copies __name__, __doc__, __module__, __qualname__, and __dict__ from the original function to the wrapper.
  • A decorator with arguments requires an extra layer of nesting. @retry(max_attempts=3) means Python calls retry(max_attempts=3) first, which must return a decorator (a function that takes and wraps the target function). So you end up with three layers: the outer function that receives the arguments, the decorator that receives the function, and the wrapper that receives the function’s arguments. This is where many developers get confused, and why functools.wraps and clear naming are essential.
  • In production, decorators are used for cross-cutting concerns: authentication (@login_required), caching (@lru_cache), rate limiting, retry logic, input validation, and metrics collection. They keep business logic clean by moving infrastructure concerns to the decoration layer. The key design principle: a decorator should be transparent — the decorated function should behave identically to the undecorated version except for the added behavior.
Follow-up: How would you write a decorator that can be used both with and without arguments — @decorator and @decorator(arg=value) — without duplicating code?
  • The trick is to detect whether the decorator was called with or without arguments. If called without parentheses (@decorator), the first argument is the function itself. If called with parentheses (@decorator(arg=value)), the first argument is arg.
  • The standard pattern uses a single function with a default sentinel: def decorator(func=None, *, arg=default). If func is provided (no parentheses), apply the decorator directly. If func is None (parentheses used), return a partial application that waits for the function. You can implement this cleanly with functools.partial: if func is None: return functools.partial(decorator, arg=arg), then proceed with the wrapping logic.
  • An alternative is functools.wraps combined with a class-based decorator that implements __call__. The class stores the arguments in __init__ and wraps the function in __call__. This is cleaner for complex decorators with many configuration options but more verbose for simple cases.
Strong Answer:
  • A generator is a function that uses yield instead of return. Calling the function does not execute the body — it returns a generator object. The body executes lazily: each call to next() runs until the next yield, suspends the function’s state (local variables, instruction pointer), and returns the yielded value. The function resumes from exactly where it left off on the next next() call.
  • Every generator is an iterator (it implements __iter__ and __next__), but not every iterator is a generator. You can write an iterator as a class with __iter__ and __next__ methods, but generators are the Pythonic shorthand — they accomplish the same thing with dramatically less boilerplate. A class-based iterator for Fibonacci might be 15 lines; the generator version is 5.
  • The full generator protocol includes three methods beyond __next__: send(value) allows the caller to send a value into the generator, which becomes the return value of the yield expression inside the generator. throw(exception) injects an exception at the point where the generator is suspended. close() raises GeneratorExit inside the generator, allowing cleanup code to run.
  • send() is what makes generators into coroutines. The pattern is: value = yield result. The generator yields result to the caller, suspends, and when the caller calls gen.send(new_value), the generator resumes with value = new_value. This bidirectional communication was the foundation of Python’s async ecosystem before async/await syntax existed (PEP 342). Libraries like Twisted and early asyncio used yield-based coroutines extensively.
  • In production, plain generators (without send/throw) are used constantly: lazy data processing, streaming I/O, memory-efficient ETL pipelines, paginated API consumption. The send/throw protocol is rarer in application code but powers the internals of asyncio and testing frameworks like pytest fixtures (which use generator-based setup/teardown).
Follow-up: What happens to a generator that is abandoned without being fully consumed? Does it leak resources?
  • When a generator object is garbage collected without being fully consumed, Python calls its close() method, which raises GeneratorExit inside the generator. If the generator has a try/finally block, the finally runs, allowing cleanup. So in CPython (with reference counting), cleanup happens promptly when the last reference is lost.
  • However, if the generator holds resources (open files, database connections, locks) and the generator is passed around (stored in a list, referenced by a closure), it may not be garbage collected promptly. In PyPy or other GC implementations without reference counting, the timing is even less predictable.
  • The safe pattern is to use a context manager with the generator: wrap it in a contextlib.closing() call, or better yet, design the generator itself as a context manager using @contextmanager. This ensures cleanup happens deterministically when the with block exits, regardless of whether the generator was fully consumed.
Strong Answer:
  • The context manager protocol defines two methods: __enter__ is called when entering the with block, and its return value is bound to the as variable. __exit__ is called when leaving the with block — whether normally or via an exception. __exit__ receives three arguments: the exception type, value, and traceback (all None if no exception occurred). If __exit__ returns True, the exception is suppressed; otherwise, it propagates.
  • A class-based context manager implements these methods directly. This is best when you need complex state management, reusable context managers, or when the context manager needs its own methods. Example: a database transaction manager that tracks nesting depth and only commits the outermost transaction.
  • A generator-based context manager uses @contextlib.contextmanager on a generator function. Everything before yield is the __enter__ phase, the yielded value is the as target, and everything after yield is the __exit__ phase. If the with block raises an exception, it is re-raised at the yield point, so you can wrap yield in a try/except/finally for cleanup. This is more concise for simple cases and is the preferred approach for one-off context managers.
  • The critical production detail in __exit__: if your cleanup code itself raises an exception, it masks the original exception. For example, if the with block raises ValueError and your __exit__ raises IOError during cleanup, the caller only sees IOError. This is a subtle and dangerous bug. The contextlib.ExitStack class handles this correctly by chaining exceptions, and Python 3.11’s exception groups provide even better tools for this.
  • Real-world use cases go far beyond file handles: database transactions (commit on success, rollback on exception), temporary directory creation and cleanup, acquiring and releasing distributed locks, redirecting stdout/stderr for testing, and timing blocks of code. Anywhere you have paired setup/teardown that must be exception-safe, a context manager is the right tool.
Follow-up: Can you nest context managers, and what is contextlib.ExitStack used for?
  • You can nest context managers with multiple with statements or (since Python 3.10) using parenthesized syntax: with (open("a") as f1, open("b") as f2):. But when the number of context managers is dynamic (opening N files from a list), you cannot hard-code the nesting.
  • ExitStack solves this. It acts as a programmatic stack of context managers. You call stack.enter_context(cm) in a loop, and the stack ensures all of them are properly exited in reverse order, even if one of the exits raises an exception. This is essential for resource management in ETL pipelines, test fixtures, and any code that manages a dynamic set of resources.
  • ExitStack also supports registering arbitrary cleanup callbacks with stack.callback(func), making it a general-purpose cleanup manager. It is one of the most underused tools in the standard library.
Strong Answer:
  • asyncio provides cooperative multitasking for I/O-bound programs. The event loop runs a single thread and schedules coroutines. When a coroutine hits an await (which signals “I am waiting for I/O”), the event loop suspends it and runs another coroutine that is ready. This gives the illusion of parallelism, but only one coroutine executes Python code at any given moment.
  • Concurrency is about dealing with multiple things at once (interleaving execution). Parallelism is about doing multiple things at the same time (simultaneous execution). asyncio provides concurrency without parallelism. This is sufficient for I/O-bound workloads because the bottleneck is waiting for external systems (network, disk, databases), not CPU computation. While one coroutine waits for a database response, another can process an HTTP request.
  • The event loop is the scheduler. It maintains a queue of ready-to-run coroutines and uses OS-level I/O multiplexing (epoll on Linux, kqueue on macOS) to know when a socket or file descriptor is ready. When you await asyncio.sleep(1), the coroutine is removed from the ready queue, a timer is set, and the loop runs other coroutines. After 1 second, the coroutine is put back on the ready queue.
  • asyncio is the wrong choice for CPU-bound work. If a coroutine does heavy computation without any await points, it blocks the entire event loop — no other coroutine can run. A single CPU-bound coroutine taking 500ms blocks all concurrent requests for 500ms. The fix is to offload CPU work to a thread pool (await asyncio.to_thread(cpu_func)) or a process pool (loop.run_in_executor(ProcessPoolExecutor(), func)).
  • It is also the wrong choice when your dependencies are not async-compatible. If your database driver, HTTP client, or file operations are synchronous, wrapping them in asyncio.to_thread() works but eliminates most of the performance benefit of async. You are better off using a threaded server (Gunicorn with sync workers) than shoehorning sync code into an async framework. The full benefit of asyncio requires an async ecosystem: aiohttp or httpx for HTTP, asyncpg for PostgreSQL, aiofiles for file I/O.
Follow-up: What is the difference between asyncio.gather(), asyncio.TaskGroup, and asyncio.create_task()? When do you use each?
  • create_task() schedules a single coroutine to run concurrently. It returns a Task object that you can await later. Use it when you want to start background work and collect the result at a later point.
  • gather() runs multiple coroutines concurrently and waits for all of them. It returns results in the same order as the input, regardless of completion order. The gotcha: if one task raises an exception, gather() by default cancels nothing — other tasks keep running, and you get the exception when you await the result. With return_exceptions=True, exceptions are returned as values instead of being raised, which is useful for “best effort” concurrent operations.
  • TaskGroup (Python 3.11+) is the modern replacement for gather() in most cases. It provides structured concurrency: if any task in the group raises an exception, all other tasks in the group are cancelled, and the exception is propagated as an ExceptionGroup. This prevents orphaned tasks that continue running after a failure. It uses async with syntax, which makes the lifetime of tasks explicit and prevents fire-and-forget bugs.
  • The evolution reflects a broader trend in async programming: from unstructured (create_task + manual cleanup) to structured (TaskGroup with automatic cancellation). For new code, prefer TaskGroup for concurrent operations. Use gather(return_exceptions=True) only when you genuinely want partial results despite failures.
Strong Answer:
  • The first and most critical concern is memory leaks. lru_cache stores the function’s arguments as dictionary keys. For a method, the first argument is self — the model instance. This means the cache holds a strong reference to every instance the method was called on. Those instances can never be garbage collected as long as the cache exists, because the cache keeps them alive. In a Django web server handling thousands of requests, each creating model instances, this causes unbounded memory growth.
  • The second concern is cache invalidation. lru_cache has no awareness of the database. If the method computes something based on database state, the cached result becomes stale as soon as the underlying data changes. Django model instances are not singletons — two different Python objects can represent the same database row. Caching on one instance does not help the other, and neither gets invalidated when the row changes.
  • The third concern is thread safety at the application level. While lru_cache itself is thread-safe (it uses a lock), the combination of caching plus Django’s ORM is not. If the cached method reads related objects via lazy-loaded foreign keys, those database queries happen outside the request’s transaction context when the cache is populated, and the cached result might reflect a different transaction’s view of the data.
  • What I would suggest instead: for per-request caching, use functools.cached_property if the value should be computed once per instance lifetime. For cross-request caching, use Django’s cache framework (django.core.cache) backed by Redis or Memcached, which supports TTLs, explicit invalidation, and does not hold references to Python objects. For expensive queries, use .select_related() or .prefetch_related() to eliminate the N+1 query problem rather than caching the result of a bad query pattern.
  • The broader lesson: lru_cache is designed for pure functions with hashable arguments. It works brilliantly for fibonacci(n) or parse_config(filename). It is a foot-gun for methods on mutable objects, anything involving database state, or functions with unhashable arguments. The right caching strategy depends on the invalidation requirements, and lru_cache has no invalidation strategy beyond size eviction.
Follow-up: How would you implement a per-instance method cache that does not prevent garbage collection?
  • Use functools.cached_property for properties that should be computed once per instance. It stores the result in the instance’s __dict__, so it is automatically garbage collected with the instance.
  • For methods with arguments, the pattern is to store a cache dictionary on the instance itself: if not hasattr(self, '_cache'): self._cache = {}. The cache lives on the instance, so it is garbage collected with the instance. You can wrap this in a custom decorator.
  • For a more sophisticated approach, use weakref in combination with a cache. The methodtools library provides lru_cache for methods that uses weak references to instances, avoiding the memory leak. Alternatively, Python 3.8+ functools.cached_property combined with __slots__ gives you efficient per-instance caching with controlled memory.