Question the model: Visualisation, automated validation, and the honest baseline

A number means nothing without something to compare it against. Previous blogs were about interrogating data before modelling it. This one is about interrogating a result before believing it, and the tool for that is always a comparison: against a deliberately stupid model, against measurement uncertainty, against what a chart shows. These are some key principles not to be fooled by fancy numbers.

1. Plot before you model, and plot after

Summary statistics can be identical across radically different distributions. Only a plot shows shape, clustering, gaps, boundaries and outliers.

The minimum set for a modelling dataset:

Label every axis with units. An unlabelled chart in a technical review is an invitation to be misread.

2. The constraint plot is the most commercially useful chart

For any problem with a commercial threshold, minimum strength, maximum cost, regulatory limit, plot the two competing properties against each other and draw the threshold as a line.

ax.scatter(df["property_b"], df["property_a"], c=df["key_driver"])
ax.axvline(THRESHOLD, color="red", linestyle="--")

This single chart shows:

• Which formulations are admissible (one side of the line)
• The best achievable value of the priority property within that region
• How much performance sits just on the wrong side of the line

That last quantity is the one nobody plots and everybody should. It converts an abstract constraint into a visible cost, and it is the natural opening for asking whether the threshold is a hard requirement or a round number that was written down once.

Colour a third variable into the same plot to show what drives the trade-off. Three dimensions of insight, one figure.

3. Encode your findings as automated rules

Manual checks are performed once, by one person, and then forgotten. A validation schema runs every time, on every batch, forever.

The workflow:

1. Interrogate the data manually
2. Convert every finding into a rule
3. Run the schema on every subsequent dataset

What this gives you as a manager is an automated incoming-data inspection. When records arrive from a lab, a supplier, or a plant historian, they either pass or they report exactly which rows and which rules failed. Test the schema by deliberately corrupting data and confirming it fails.

4. Schemas encode what is possible, not what has happened

The most common mistake in writing a schema is deriving its limits from the observed minimum and maximum of the current dataset.

Such a schema will reject the first genuinely novel formulation anyone tries, which is precisely the experiment worth running. Every extension of the design space becomes a validation failure, and the schema is quickly disabled by frustrated users.

Set ranges to the physically or procedurally plausible envelope, deliberately wider than the data. The schema's job is to catch impossible values (negative mass, 500 °C in a 95 °C process, a fraction above 1), not unusual ones.

5. Code that runs on import is a script pretending to be a module

A module defines things. A script does things.

Test: importing a file should produce no output and have no side effects.

Belongs in a module:

• Constants: physical limits, column groupings, thresholds
• Functions that transform data
• Schema and configuration objects

Does not belong:

• File loading, hard-coded paths
• print() outside functions
• Plotting, model training

Two supporting habits:

Functions should not mutate their arguments. Begin with df = df.copy(). Silent mutation of an input produces bugs that are genuinely miserable to trace.

Constants live at the top, named, with a comment on their origin. A commercial threshold and a physical limit look identical in code and are entirely different in kind. When the threshold is questioned later, and it should be, you change one line rather than searching notebooks.

6. Never evaluate a model only on its training data

If a model is trained and evaluated on the same data, the result does not tell us whether it can make useful predictions on new cases. The dataset should therefore be divided into training data and test data.

The key question is: “Can the model predict genuinely new observations?”

7. Always establish a baseline first

baseline_pred = np.full(len(y), y.mean())

Predict the mean for everything. Ignore all inputs. This is the floor, and every subsequent claim is measured against it. For classification the equivalent is predicting the majority class, which on a 93/7 split gives 93% accuracy and zero value.

The only honest headline metric is improvement over baseline:

improvement = (mae_baseline - mae_model) / mae_baseline * 100

A model with R² = 0.78 that beats a constant prediction by 6% is a bad model wearing a nice suit. R² is a correlation-based measure and is easily impressive; percentage improvement over the dumbest possible approach is interpretable by a business audience and much harder to inflate.

8. Report error in physical units

MAE in NRC, RMSE in kPa, error in °C. Not R², not "accuracy", not a normalised score. Physical units let a domain expert immediately judge whether the model is useful:

• "±0.012 NRC", a specifier knows whether that matters
• "R² = 0.83", nobody can act on this

MAE versus RMSE: MAE is the average error magnitude and is the more intuitive figure for a general audience. RMSE penalises large errors more heavily and is the better choice when occasional large misses are disproportionately costly. Report both; lead with MAE.

9. Compare model error against measurement noise

This is the comparison that tells you when to stop, and almost nobody makes it. If a model's MAE is 0.012 NRC and the test method repeats to ±0.02 NRC, the model is already predicting more precisely than you can measure. Further improvement is chasing noise in the measurement, not signal in the process.

Consequences:

• It sets a realistic performance ceiling before work begins
• It prevents indefinite model-improvement projects with no defined end
• It reframes the improvement path: better measurement, not better algorithms

This requires replicates. Repeated measurements of the same sample are the only way to estimate measurement repeatability from data. A dataset without replicates cannot tell you how good a model could possibly be, which is a strong argument for including them in any future experimental campaign.

10. Residual plots are where models confess

Metrics summarise; residuals diagnose. Two plots, always:

Residuals versus predicted. Should scatter randomly around zero with no structure.

• Curvature → the true relationship is nonlinear and the model is missing it
• Funnel shape → error grows with magnitude; consider transforming the target
• Clusters → a subgroup the model handles differently, often a hidden variable

Predicted versus actual, with a 1:1 reference line. Shows bias directly: systematic over-prediction at low values and under-prediction at high values is the classic signature of a model regressing toward the mean.

A poor residual plot with a good R² is a warning. A good residual plot with a modest R² is often a sound model on a noisy problem.

11. A model can perform worse than a baseline

This is completely valid. Our linear-regression model sometimes performed worse than simply predicting the mean. That does not mean the modelling process failed.

It tells us something important about:

• The available data
• The selected features
• The model assumptions
• The sample size
• The amount of predictive signal available

Possible reasons include:

• Features do not explain the target well
• Important variables are missing
• The relationship is nonlinear
• Dataset is too small
• Measurements contain substantial noise
• Model is fitting random variation rather than useful patterns

A negative result is still useful evidence.

12. State what the model cannot do

Every model report needs an explicit limitations section. Minimum content:

• Range of validity: the model interpolates within the observed design space and should not be trusted outside it
• Excluded variables: what is not in the model that affects the outcome
• Data provenance: synthetic, historical, or designed, and what that implies
• Measurement uncertainty: how model error compares to test repeatability
• What decision it does and does not support

Omitting this section is how a model built for screening ends up used for specification.

13. Data leakage produces misleadingly optimistic evaluation

Data leakage occurs when information reaches the model during training that would not genuinely be available when predicting a future observation.

Examples include:

• Exact duplicate in training and testing
• Replicates from the same specimen appearing on both sides
• Using future measurements as input features
• Preprocessing the complete dataset before splitting
• Closely related batches appearing in training and testing when the goal is to predict new batches

Leakage can produce excellent-looking metrics that do not survive real-world deployment.

14. Write for the reader, not the analysis

A model report for a technical director is not a notebook with prose attached.

• One page
• One chart, chosen deliberately
• Error in physical units, in the first paragraph
• Improvement over baseline, stated plainly
• What it cannot do
• A recommendation: continue, stop, or change approach

Test: if a sentence would require explanation in the meeting, rewrite it. The report exists to support a decision, not to demonstrate effort.

15. Quick reference: before reporting any model result

• [ ] Baseline computed and beaten by a stated margin
• [ ] Error reported in physical units
• [ ] Error compared against measurement repeatability
• [ ] Residuals plotted and read, not just metrics
• [ ] Predicted-vs-actual plotted with 1:1 line
• [ ] Validity range stated
• [ ] Limitations section written
• [ ] Every chart axis labelled with units
• [ ] Recommendation stated: continue, stop, or change approach