Documentation Index
Fetch the complete documentation index at: https://resources.devweekends.com/llms.txt
Use this file to discover all available pages before exploring further.
Advanced Python
Ready to move beyond the basics? These features are what distinguish intermediate Python developers from experts. They allow you to write code that is efficient, clean, and “Pythonic”.1. Decorators
Decorators are a powerful way to modify the behavior of functions or classes without changing their source code. They are essentially wrappers. Think of a decorator like a “middleware” for a function.2. Generators
Lists store all their data in memory. If you have a list of 1 billion numbers, you’ll run out of RAM. Generators produce data one item at a time, on demand. They are lazy. They use theyield keyword instead of return.
3. Context Managers (with)
Managing resources (files, network connections, database locks) is hard. You have to remember to close them, even if errors occur.
Context Managers handle this automatically.
4. AsyncIO (Asynchronous Programming)
Python is single-threaded. However, usingasyncio, you can write concurrent code that handles I/O (like network requests) efficiently. While one request is waiting for a response, Python can start another.
This is essential for modern web frameworks like FastAPI.
5. Type Hinting (Advanced)
For complex data structures, basic type hints (int, str) aren’t enough. The typing module provides tools to describe shapes of data.
Summary
- Decorators: Wrap functions to add logic (logging, timing, auth).
- Generators: Lazy iteration for memory efficiency.
- Context Managers:
withstatement for safe resource management. - AsyncIO: Concurrency for I/O-bound tasks.
- Type Hints: Essential for large codebases.
Interview Deep-Dive
Explain how Python decorators work under the hood. What happens when you stack multiple decorators, and what is a common pitfall with decorated functions?
Explain how Python decorators work under the hood. What happens when you stack multiple decorators, and what is a common pitfall with decorated functions?
- A decorator is syntactic sugar for wrapping a function. When you write
@timerabovedef slow_function():, Python executesslow_function = timer(slow_function). The decorator receives the original function, returns a new function (the wrapper), and the original name now points to the wrapper. There is no special magic — it is plain function composition. - Stacking decorators applies them bottom-up. If you write
@auththen@timerabove a function, Python executesfunc = auth(timer(func)). Sotimerwraps the function first, thenauthwraps the timed version. The order matters: ifauthrejects the request early, the timer never runs. If you reverse them, you time the auth check too. This evaluation order trips up even experienced developers. - The most common pitfall is that the wrapper replaces the original function’s metadata. After decoration,
slow_function.__name__returns"wrapper"instead of"slow_function", andslow_function.__doc__returns the wrapper’s docstring (usually None). This breaks introspection, logging, debugging, and any framework that uses function names for routing (Flask, FastAPI). The fix is@functools.wraps(func)on the wrapper, which copies__name__,__doc__,__module__,__qualname__, and__dict__from the original function to the wrapper. - A decorator with arguments requires an extra layer of nesting.
@retry(max_attempts=3)means Python callsretry(max_attempts=3)first, which must return a decorator (a function that takes and wraps the target function). So you end up with three layers: the outer function that receives the arguments, the decorator that receives the function, and the wrapper that receives the function’s arguments. This is where many developers get confused, and whyfunctools.wrapsand clear naming are essential. - In production, decorators are used for cross-cutting concerns: authentication (
@login_required), caching (@lru_cache), rate limiting, retry logic, input validation, and metrics collection. They keep business logic clean by moving infrastructure concerns to the decoration layer. The key design principle: a decorator should be transparent — the decorated function should behave identically to the undecorated version except for the added behavior.
@decorator and @decorator(arg=value) — without duplicating code?- The trick is to detect whether the decorator was called with or without arguments. If called without parentheses (
@decorator), the first argument is the function itself. If called with parentheses (@decorator(arg=value)), the first argument isarg. - The standard pattern uses a single function with a default sentinel:
def decorator(func=None, *, arg=default). Iffuncis provided (no parentheses), apply the decorator directly. Iffuncis None (parentheses used), return a partial application that waits for the function. You can implement this cleanly withfunctools.partial: iffunc is None: return functools.partial(decorator, arg=arg), then proceed with the wrapping logic. - An alternative is
functools.wrapscombined with a class-based decorator that implements__call__. The class stores the arguments in__init__and wraps the function in__call__. This is cleaner for complex decorators with many configuration options but more verbose for simple cases.
What are generators in Python, and how do they differ from iterators? Explain the generator protocol and `send()`/`throw()`/`close()`.
What are generators in Python, and how do they differ from iterators? Explain the generator protocol and `send()`/`throw()`/`close()`.
- A generator is a function that uses
yieldinstead ofreturn. Calling the function does not execute the body — it returns a generator object. The body executes lazily: each call tonext()runs until the nextyield, suspends the function’s state (local variables, instruction pointer), and returns the yielded value. The function resumes from exactly where it left off on the nextnext()call. - Every generator is an iterator (it implements
__iter__and__next__), but not every iterator is a generator. You can write an iterator as a class with__iter__and__next__methods, but generators are the Pythonic shorthand — they accomplish the same thing with dramatically less boilerplate. A class-based iterator for Fibonacci might be 15 lines; the generator version is 5. - The full generator protocol includes three methods beyond
__next__:send(value)allows the caller to send a value into the generator, which becomes the return value of theyieldexpression inside the generator.throw(exception)injects an exception at the point where the generator is suspended.close()raisesGeneratorExitinside the generator, allowing cleanup code to run. send()is what makes generators into coroutines. The pattern is:value = yield result. The generator yieldsresultto the caller, suspends, and when the caller callsgen.send(new_value), the generator resumes withvalue = new_value. This bidirectional communication was the foundation of Python’s async ecosystem beforeasync/awaitsyntax existed (PEP 342). Libraries like Twisted and early asyncio usedyield-based coroutines extensively.- In production, plain generators (without
send/throw) are used constantly: lazy data processing, streaming I/O, memory-efficient ETL pipelines, paginated API consumption. Thesend/throwprotocol is rarer in application code but powers the internals ofasyncioand testing frameworks likepytestfixtures (which use generator-based setup/teardown).
- When a generator object is garbage collected without being fully consumed, Python calls its
close()method, which raisesGeneratorExitinside the generator. If the generator has atry/finallyblock, thefinallyruns, allowing cleanup. So in CPython (with reference counting), cleanup happens promptly when the last reference is lost. - However, if the generator holds resources (open files, database connections, locks) and the generator is passed around (stored in a list, referenced by a closure), it may not be garbage collected promptly. In PyPy or other GC implementations without reference counting, the timing is even less predictable.
- The safe pattern is to use a context manager with the generator: wrap it in a
contextlib.closing()call, or better yet, design the generator itself as a context manager using@contextmanager. This ensures cleanup happens deterministically when thewithblock exits, regardless of whether the generator was fully consumed.
Explain Python's context manager protocol. How do `__enter__` and `__exit__` work, and what is the difference between a class-based and generator-based context manager?
Explain Python's context manager protocol. How do `__enter__` and `__exit__` work, and what is the difference between a class-based and generator-based context manager?
- The context manager protocol defines two methods:
__enter__is called when entering thewithblock, and its return value is bound to theasvariable.__exit__is called when leaving thewithblock — whether normally or via an exception.__exit__receives three arguments: the exception type, value, and traceback (allNoneif no exception occurred). If__exit__returnsTrue, the exception is suppressed; otherwise, it propagates. - A class-based context manager implements these methods directly. This is best when you need complex state management, reusable context managers, or when the context manager needs its own methods. Example: a database transaction manager that tracks nesting depth and only commits the outermost transaction.
- A generator-based context manager uses
@contextlib.contextmanageron a generator function. Everything beforeyieldis the__enter__phase, the yielded value is theastarget, and everything afteryieldis the__exit__phase. If thewithblock raises an exception, it is re-raised at theyieldpoint, so you can wrapyieldin atry/except/finallyfor cleanup. This is more concise for simple cases and is the preferred approach for one-off context managers. - The critical production detail in
__exit__: if your cleanup code itself raises an exception, it masks the original exception. For example, if thewithblock raisesValueErrorand your__exit__raisesIOErrorduring cleanup, the caller only seesIOError. This is a subtle and dangerous bug. Thecontextlib.ExitStackclass handles this correctly by chaining exceptions, and Python 3.11’s exception groups provide even better tools for this. - Real-world use cases go far beyond file handles: database transactions (commit on success, rollback on exception), temporary directory creation and cleanup, acquiring and releasing distributed locks, redirecting stdout/stderr for testing, and timing blocks of code. Anywhere you have paired setup/teardown that must be exception-safe, a context manager is the right tool.
contextlib.ExitStack used for?- You can nest context managers with multiple
withstatements or (since Python 3.10) using parenthesized syntax:with (open("a") as f1, open("b") as f2):. But when the number of context managers is dynamic (opening N files from a list), you cannot hard-code the nesting. ExitStacksolves this. It acts as a programmatic stack of context managers. You callstack.enter_context(cm)in a loop, and the stack ensures all of them are properly exited in reverse order, even if one of the exits raises an exception. This is essential for resource management in ETL pipelines, test fixtures, and any code that manages a dynamic set of resources.ExitStackalso supports registering arbitrary cleanup callbacks withstack.callback(func), making it a general-purpose cleanup manager. It is one of the most underused tools in the standard library.
Explain `asyncio` and the event loop. What is the difference between concurrency and parallelism in the context of async Python? When is asyncio the wrong choice?
Explain `asyncio` and the event loop. What is the difference between concurrency and parallelism in the context of async Python? When is asyncio the wrong choice?
asyncioprovides cooperative multitasking for I/O-bound programs. The event loop runs a single thread and schedules coroutines. When a coroutine hits anawait(which signals “I am waiting for I/O”), the event loop suspends it and runs another coroutine that is ready. This gives the illusion of parallelism, but only one coroutine executes Python code at any given moment.- Concurrency is about dealing with multiple things at once (interleaving execution). Parallelism is about doing multiple things at the same time (simultaneous execution).
asyncioprovides concurrency without parallelism. This is sufficient for I/O-bound workloads because the bottleneck is waiting for external systems (network, disk, databases), not CPU computation. While one coroutine waits for a database response, another can process an HTTP request. - The event loop is the scheduler. It maintains a queue of ready-to-run coroutines and uses OS-level I/O multiplexing (epoll on Linux, kqueue on macOS) to know when a socket or file descriptor is ready. When you
await asyncio.sleep(1), the coroutine is removed from the ready queue, a timer is set, and the loop runs other coroutines. After 1 second, the coroutine is put back on the ready queue. asynciois the wrong choice for CPU-bound work. If a coroutine does heavy computation without anyawaitpoints, it blocks the entire event loop — no other coroutine can run. A single CPU-bound coroutine taking 500ms blocks all concurrent requests for 500ms. The fix is to offload CPU work to a thread pool (await asyncio.to_thread(cpu_func)) or a process pool (loop.run_in_executor(ProcessPoolExecutor(), func)).- It is also the wrong choice when your dependencies are not async-compatible. If your database driver, HTTP client, or file operations are synchronous, wrapping them in
asyncio.to_thread()works but eliminates most of the performance benefit of async. You are better off using a threaded server (Gunicorn with sync workers) than shoehorning sync code into an async framework. The full benefit ofasynciorequires an async ecosystem:aiohttporhttpxfor HTTP,asyncpgfor PostgreSQL,aiofilesfor file I/O.
asyncio.gather(), asyncio.TaskGroup, and asyncio.create_task()? When do you use each?create_task()schedules a single coroutine to run concurrently. It returns aTaskobject that you canawaitlater. Use it when you want to start background work and collect the result at a later point.gather()runs multiple coroutines concurrently and waits for all of them. It returns results in the same order as the input, regardless of completion order. The gotcha: if one task raises an exception,gather()by default cancels nothing — other tasks keep running, and you get the exception when you await the result. Withreturn_exceptions=True, exceptions are returned as values instead of being raised, which is useful for “best effort” concurrent operations.TaskGroup(Python 3.11+) is the modern replacement forgather()in most cases. It provides structured concurrency: if any task in the group raises an exception, all other tasks in the group are cancelled, and the exception is propagated as anExceptionGroup. This prevents orphaned tasks that continue running after a failure. It usesasync withsyntax, which makes the lifetime of tasks explicit and prevents fire-and-forget bugs.- The evolution reflects a broader trend in async programming: from unstructured (create_task + manual cleanup) to structured (TaskGroup with automatic cancellation). For new code, prefer
TaskGroupfor concurrent operations. Usegather(return_exceptions=True)only when you genuinely want partial results despite failures.
You are reviewing a pull request that adds `@functools.lru_cache` to a method on a Django model class. What concerns would you raise?
You are reviewing a pull request that adds `@functools.lru_cache` to a method on a Django model class. What concerns would you raise?
- The first and most critical concern is memory leaks.
lru_cachestores the function’s arguments as dictionary keys. For a method, the first argument isself— the model instance. This means the cache holds a strong reference to every instance the method was called on. Those instances can never be garbage collected as long as the cache exists, because the cache keeps them alive. In a Django web server handling thousands of requests, each creating model instances, this causes unbounded memory growth. - The second concern is cache invalidation.
lru_cachehas no awareness of the database. If the method computes something based on database state, the cached result becomes stale as soon as the underlying data changes. Django model instances are not singletons — two different Python objects can represent the same database row. Caching on one instance does not help the other, and neither gets invalidated when the row changes. - The third concern is thread safety at the application level. While
lru_cacheitself is thread-safe (it uses a lock), the combination of caching plus Django’s ORM is not. If the cached method reads related objects via lazy-loaded foreign keys, those database queries happen outside the request’s transaction context when the cache is populated, and the cached result might reflect a different transaction’s view of the data. - What I would suggest instead: for per-request caching, use
functools.cached_propertyif the value should be computed once per instance lifetime. For cross-request caching, use Django’s cache framework (django.core.cache) backed by Redis or Memcached, which supports TTLs, explicit invalidation, and does not hold references to Python objects. For expensive queries, use.select_related()or.prefetch_related()to eliminate the N+1 query problem rather than caching the result of a bad query pattern. - The broader lesson:
lru_cacheis designed for pure functions with hashable arguments. It works brilliantly forfibonacci(n)orparse_config(filename). It is a foot-gun for methods on mutable objects, anything involving database state, or functions with unhashable arguments. The right caching strategy depends on the invalidation requirements, andlru_cachehas no invalidation strategy beyond size eviction.
- Use
functools.cached_propertyfor properties that should be computed once per instance. It stores the result in the instance’s__dict__, so it is automatically garbage collected with the instance. - For methods with arguments, the pattern is to store a cache dictionary on the instance itself:
if not hasattr(self, '_cache'): self._cache = {}. The cache lives on the instance, so it is garbage collected with the instance. You can wrap this in a custom decorator. - For a more sophisticated approach, use
weakrefin combination with a cache. Themethodtoolslibrary provideslru_cachefor methods that uses weak references to instances, avoiding the memory leak. Alternatively, Python 3.8+functools.cached_propertycombined with__slots__gives you efficient per-instance caching with controlled memory.