Skip to main content
Reproducibility

Reproducibility

The Reproducibility Crisis

Imagine baking a cake from a recipe that says “some flour, a bit of sugar, bake until done.” You would never get the same cake twice. Yet this is roughly how many ML experiments are documented: “we used a learning rate around 1e-3 and trained for a while.” The result is a crisis where published results cannot be replicated, team members cannot reproduce each other’s experiments, and yesterday’s best model cannot be retrained after a framework upgrade. Reproducibility is not a luxury — it is the difference between engineering and alchemy. Deep learning experiments are notoriously hard to reproduce, and the sources of randomness are more numerous than most practitioners realize:
Even with identical code and seeds, moving from one GPU architecture to another (say A100 to H100) can produce different results. Floating-point operations are not associative, and different hardware may fuse or reorder operations differently. Perfect bitwise reproducibility across hardware is generally impossible — aim for statistical reproducibility instead (same conclusions, not identical numbers).

Seed Management

Comprehensive Seeding

Setting random seeds is the most basic reproducibility measure, but it is also the most commonly done incorrectly. The trap: most people set torch.manual_seed(42) and think they are done. In reality, Python, NumPy, PyTorch CPU, PyTorch CUDA, and cuDNN each have their own random number generators, and if you miss even one, your results will vary between runs.

Deterministic Operations


Experiment Configuration

Configuration Management

YAML Configuration


Environment Tracking

Environment Snapshot


Experiment Tracking

In practice, most reproducibility failures are not caused by random seeds or hardware differences — they are caused by losing track of which code, data, and hyperparameters produced a given result. Experiment tracking is the disciplined practice of recording everything needed to recreate any past result. Think of it like a lab notebook in chemistry: without it, you might produce gold once but never remember how.
For teams just starting with experiment tracking: begin with the simplest possible approach (a JSON config file + git commit hash saved alongside each run). Only move to heavyweight tools like MLflow or Weights and Biases when you have 3+ people running experiments and need a shared dashboard. Over-engineering your tracking system before you have a stable training pipeline is a common time sink.

Simple Logger

Weights & Biases Integration


Data Versioning

Code is version-controlled with git. Models are often tracked with experiment loggers. But data — the most impactful component of any ML system — is frequently the least version-controlled. Data versioning ensures you can always answer the question: “exactly which data did this model train on?” Without it, you will eventually encounter the nightmare scenario where a model regresses, nobody knows what changed, and the answer turns out to be a silent data pipeline change from two weeks ago.
Git is not designed for large binary files. Do not commit datasets to your git repository. Use DVC (Data Version Control), git-lfs, or cloud storage with hash-based versioning. The pattern below computes hashes of your data so you can detect changes, but the actual data should live outside git.

Reproducible Data Loading


Complete Reproducibility Template


Exercises

Test your experiment reproduces across:
  • Different GPUs (NVIDIA vs AMD)
  • Different OS (Linux vs Windows)
  • Different PyTorch versions
Create a simple dashboard to:
  • Compare multiple runs
  • Visualize metrics over time
  • Filter by hyperparameters
Create a Dockerfile that:
  • Pins all dependencies
  • Includes data download script
  • Runs experiments reproducibly

What’s Next?

Memory-Efficient Training

Gradient checkpointing and memory optimization

Hyperparameter Tuning

Systematic optimization strategies