Reproducibility and model packaging

Key Lessons

1. Reproducibility is more than saving a trained model. A reproducible ML experiment preserves the data definition, preprocessing, feature set, target, validation strategy, random seeds, hyperparameters, software environment, evaluation results, and code.

2. Model reproducibility and experiment reproducibility are different. Reloading model.joblib and getting identical predictions proves the saved model is reproducible. Re-running train_model.py and reproducing the same metrics demonstrates that the broader experiment can be reconstructed.

3. A standalone training script strengthens reproducibility. Notebooks are useful for exploration, but a clean script makes the final workflow easier to repeat, inspect, automate, and hand over.

4. Random seeds are part of the experiment specification. If splitting or model training involves randomness, the seed must be recorded so the same experiment can be recreated.

5. The exact data split matters. Recording which specimens belonged to training and validation provides stronger provenance than simply saying that an 80/20 split was used.

6. Environment versions matter. Python and library versions can change model behaviour or even break previously working code. Recording them helps diagnose future discrepancies.

7. Summary metrics are not enough. Saving specimen-level predictions makes it possible to trace an MAE or RMSE back to the actual errors that produced it.

8. Configuration should be stored explicitly. Model hyperparameters, selected features, grouping variables, split strategy, and target should not depend on someone's memory.

9. Model Cards provide technical context around a model. They communicate what a model is, how it was developed, how it was evaluated, what it should be used for, and what it should not be used for.

10. Limitations are part of the model, not an optional disclaimer. Dataset size, weak predictive performance, unstable explanations, lack of external validation, and other constraints belong in the technical documentation.

11. A weak model can still be documented extremely well. Good ML engineering does not require pretending every experiment succeeded.

12. Negative results have value. A reproducible record showing that a Random Forest did not outperform a simple baseline can prevent unnecessary repetition and redirect effort toward better data or experimental design.

13. Baseline performance belongs in model documentation. A model's metrics have little meaning without comparison to what could be achieved using a simpler approach.

14. Deployment status should be explicit. For example, the correct conclusion is not merely “MAE = 9.80 kPa,” but: the model underperforms the baseline and is not suitable for deployment.

15. Explainability findings should also be documented. Because permutation importance and SHAP results were unstable, the Model Card records that explanations should not be interpreted as physical materials relationships.

16. A model repository is more than file storage. The Hugging Face repository combines the model artifact, configuration, documentation, metrics, training code, and version history.

17. Version control preserves the evolution of an experiment. Each commit creates a record of what changed and why.

18. Descriptive commit messages matter. A message such as “Add version history and reproducibility status” is much more useful than “update” because it preserves engineering intent.

19. A model should be referenced by a specific version. As datasets, features, validation strategies, or algorithms change, simply referring to “the model” becomes ambiguous.

20. Reproducibility supports technical handover. A well-packaged experiment allows another engineer—or your future self—to understand and reconstruct the work without relying on undocumented knowledge.

Package structure best practices

21. Organise the package by function, not convenience. Each file should have a clear purpose: data, configuration, training code, model artifact, evaluation output, environment information, or documentation. This makes the package easier to inspect and maintain.

22. Separate configuration from executable code. Parameters such as features, target, random seed, validation strategy, and model hyperparameters should be stored explicitly rather than buried throughout the script. This improves traceability and reduces accidental changes.

23. Keep raw inputs, generated outputs, and documentation distinguishable. A well-structured package should make it obvious which files are inputs to the experiment and which were produced by it.

24. Store both summary results and detailed results. metrics.json provides a concise performance summary, while validation_predictions.csv preserves observation-level evidence. Both are useful and serve different purposes.

25. Record the exact data split separately from the dataset. A file such as data_split.csv allows another person to reconstruct the experiment without guessing which specimens were used for training or validation.

26. Keep the trained model separate from the training instructions. model.joblib stores the fitted model, while train_model.py documents how to recreate it. One preserves the result; the other preserves the process.

27. Use human-readable file formats where practical. Files such as .json, .csv, .txt, and .md can be inspected without specialised software and make technical review easier.

28. Use descriptive and stable filenames. Names such as metrics.json, model_config.json, and validation_predictions.csv communicate purpose immediately. Avoid vague names such as final2.csv or model_latest.pkl.

29. Avoid duplicating the same information inconsistently across files. If a parameter is recorded in several places, they can drift out of sync. Ideally, configuration should have one authoritative source and other files should reference or reproduce it systematically.

30. Keep the package minimal but complete. Include everything needed to understand and reproduce the experiment, but avoid unnecessary temporary files, exploratory outputs, caches, or intermediate artifacts that do not support reproducibility.

31. The README should act as the entry point to the package. A new user should be able to open README.md and quickly understand what the experiment is, what files matter, how the model was evaluated, and whether it is suitable for use.

32. The package should allow reconstruction without relying on the original notebook. The notebook can remain useful for exploration, but the final package should contain enough code, data, configuration, and documentation to stand on its own.

33. Package structure is part of technical communication. A well-organised repository reduces ambiguity, makes review easier, and increases confidence that the experiment was conducted systematically.

Recommended package structure

model_package/
│
├── README.md
├── model_dataset.csv
├── data_split.csv
├── model_config.json
├── train_model.py
├── model.joblib
├── metrics.json
├── validation_predictions.csv
└── environment_versions.txt

A useful way to think about these files is:

A useful conceptual grouping is:

Documentation
├── README.md

Data and provenance
├── model_dataset.csv
├── data_split.csv

Experiment definition
├── model_config.json
├── train_model.py

Model artifact
├── model.joblib

Evaluation evidence
├── metrics.json
├── validation_predictions.csv

Environment
├── environment_versions.txt

This separation makes it easier to answer five questions:

What data were used?
How was the experiment configured?
How can it be reproduced?
What model was produced?
What evidence supports the reported result?

Glossary

Final conclusion

We have now created a complete reproducible ML experiment:

synthetic dataset → documented split → explicit configuration → reproducible training script → trained model → saved metrics → specimen-level predictions → environment record → Model Card → Hugging Face repository → version history

The central principle is:

A professional ML model is not just an algorithm or a saved file. It is a well-structured, traceable technical package containing the data assumptions, configuration, code, model artifact, evaluation evidence, limitations, documentation, software environment, and version history needed to understand and reproduce it.