Before asking what a model learned, establish whether there was anything to learn.
Interpretability tools explain any model given to them, including models that perform worse than predicting the mean. They have no concept of whether the model is any good. A convincing SHAP plot of a worthless model is one of the more expensive ways to mislead an organisation, because it looks exactly like insight.
The gate comes first, the interpretation second.
Glosary
Interpretability methods
Permutation importance: Shuffle one feature's values at random and measure how much worse the model gets. If shuffling barely hurts, the model was not relying on that feature. Practical meaning: a ranking of which inputs the model actually uses. Model-agnostic, computed on held-out data. Caution: correlated features share importance and both appear weak. With three related fibre measures in a model, importance splits three ways and each looks unimportant.
Partial Dependence Plot (PDP): The average predicted outcome as one feature is varied across its range, holding others at their observed values. Practical meaning: the shape of a relationship. Where importance says "this matters", PDP says "and here is how" — revealing thresholds, plateaus, saturation and reversals. Caution: averages over the dataset, so it can conceal opposing effects in different subgroups. For a linear model, it returns a straight line by construction.
SHAP (SHapley Additive exPlanations): Attributes a single prediction to its input features, such that the contributions sum to the difference between that prediction and the average prediction. Derived from cooperative game theory. Practical meaning: the only common method that explains individual predictions rather than overall behaviour. Three plot types:
- Summary plot — every sample, every feature. Position shows contribution, colour shows feature value. Reveals both importance and direction.
- Dependence plot — one feature's contribution against its value. Similar to a PDP but per-sample.
- Waterfall plot — a single prediction decomposed. "This batch was predicted at 0.58 because thickness was high and density low." This is the artefact that makes systems trustworthy to operators, because it can be read, disputed, and corrected.
Caution: SHAP explains the model, not reality. If the model learned a proxy, SHAP faithfully reports the proxy.
Plausibility review A structured meeting where domain experts who did not build the model examine its SHAP and PDP outputs against known physics. Practical meaning: the highest-leverage governance practice available to a technical organisation deploying ML. Catches leakage, spurious correlation and sensor artefacts before deployment, using expertise already on the payroll, at a cost of one hour.
Variance and measurement
Repeatability (noise floor): the standard deviation between repeated measurements of nominally identical samples. Computed by pooling within-group variance across replicated conditions. Practical meaning: the hard ceiling on model performance. No model can predict more precisely than the measurement repeats. If model error already approaches repeatability, further modelling is chasing noise and the improvement path lies in measurement or process control. Requires replicates. Without them, this quantity is unknowable, and nobody can say whether a model is good or the problem is hopeless.
Noise-to-total ratio: repeatability variance as a proportion of total variance. Practical rule of thumb:
- Below ~30% — modelling is viable
- 30–50% — modelling is possible but expect modest performance
- Above ~50% — process variability dominates; control the process before modelling
- Approaching 100% — nothing systematic exists to learn
Variance decomposition: separating observed variation into systematic effects and noise. The elementary version: total σ versus repeatability σ. Practical meaning: answers "is there a signal here at all?" before any modelling effort is committed.
Confounding and causation
Confounding: A third variable influencing both the apparent cause and the effect, creating a correlation with no direct mechanism. Practical meaning: why a correlation matrix is a starting point and never evidence of mechanism.
Control variable: A variable included in a regression specifically to remove its influence, so the remaining coefficients reflect effects independent of it. Practical meaning: the standard means of separating confounded effects. y ~ suspect + confounder reveals whether the suspect survives.
Sign reversal under control: When a coefficient changes sign between simple correlation and controlled regression. Practical meaning: the raw correlation was entirely an artefact of confounding. A strong warning sign, and a demonstration that the unadjusted relationship was misleading rather than merely overstated.
Mediator: A variable lying on the causal path between cause and effect. Formulation → density → NRC makes density a mediator. Practical meaning: controlling for a mediator removes the effect you were trying to measure. Distinguishing mediators from confounders requires domain knowledge, not statistics.
Spurious ratio correlation: Two engineered ratios sharing a variable are correlated by construction, irrespective of any physical relationship. SLSA/water and water/solids necessarily move in opposition. Practical meaning: check engineered ratios for shared terms before interpreting them. This is arithmetic, not materials science, and it generates convincing false findings.
Data structure
Design variable: Something set directly like a formulation fraction, a temperature setpoint, a poured mass. Practical meaning: the only variables an optimiser can act upon.
Tier-2 intermediate: Something measured on the finished article that is a consequence of the design: density, achieved thickness, yield. Practical meaning: excellent for explanation, unusable in a design-stage model. At the moment a prediction is needed, these values do not exist.
Two-stage model: Design variables → intermediate → final property, as two chained models rather than one direct model. Practical meaning: mirrors the physics, is separately checkable at each stage, and is frequently more accurate and more interpretable than modelling direct.
Grouped cross-validation (GroupKFold): Splitting so that all rows sharing a group — batch, formulation, campaign — stay together in the same fold. Practical meaning: mandatory when replicates exist. Random splitting lets the model train on a formulation and be tested on its twin, inflating every score.
Negative control: A variable deliberately included that should have no effect on the target. Practical meaning: a built-in noise detector. If an inert variable appears important, the model is fitting noise. Zinc ricinoleate is inert with respect to acoustics and therefore serves this role for NRC.
Feature engineering
Physics-informed feature: A derived variable encoding a known physical relationship: a ratio, an integral, a dimensionless group. Practical meaning: where domain expertise produces measurable advantage. A ratio can outperform its components individually because it encodes a law the model would otherwise have to rediscover from limited data.
Thermal dose / thermal area: The integral of temperature over time through a process cycle. Practical meaning: closer to what a material actually experiences than peak temperature or duration alone. Equivalents appear as degree of cure, equivalent age in concrete, F₀ in sterilisation, thermal budget in semiconductors.
Control stability feature: The standard deviation of a process variable during its steady-state period. Practical meaning: captures how well a setpoint was held, not merely what it was. Two parts at nominally identical settings differ if one was made while the controller was hunting. Frequently explains otherwise inexplicable variation.
Censoring: A measurement recorded as beyond an instrument's range: >2000, <50. Practical meaning: the true value is unknown. Converting it to the limit fabricates a measurement. Flag it and preserve the flag.
Principles
1. Never interpret a model that has not beaten its baseline
SHAP will explain anything. Gate on performance first:
baseline = np.abs(y - y.mean()).mean()
if cv_scores.mean() >= baseline:
print("STOP - does not beat baseline. Do not interpret.")
A detailed interpretation of a worthless model is worse than no analysis, because it carries the appearance of insight.
2. Compute the noise floor before anything else
This requires replicates and takes four lines of code:
g = repeated.groupby("group_id")[target]
n = g.count()
pooled = np.sqrt((g.var(ddof=1) * (n-1)).sum() / (n-1).sum())
print(f"repeatability σ = {pooled:.4f}, total σ = {df[target].std():.4f}")
It is rarely calculated, and its absence is a common reason modelling projects consume months before failing without explanation.
If repeatability approaches the baseline error, stop. No model can succeed. That determination costs minutes and can save a programme.
3. Write hypotheses before building features
Commit in writing to five statements about how the system works, before any code. Then test them and record the verdict — confirmed, contradicted, or no evidence either way.
"No evidence either way" is the honest answer more often than is usually admitted, particularly at small n. Recording it identifies where the data cannot discriminate, which is an argument for a designed experiment.
A rejected hypothesis is worth more than three confirmations. Keeping both the original and revised lists is what distinguishes analysis from confirmation.
4. Correlation matrices are for generating questions, never for answering them
Every strong correlation requires a controlled regression before it is believed:
smf.ols("target ~ suspect + confounder", data=d).fit().summary().tables[1]
A sign reversal between correlation and controlled coefficient means the simple relationship was an artefact.
5. The eye judges correlation poorly
A correlation of 0.73 looks scattered on a 50-point plot, because a third of the variance remains unexplained. This is a documented perceptual limitation, not a personal one.
Compute the number. Do not read it off the plot. Then plot it anyway, because the number conceals thresholds, clustering, and outliers that the plot reveals.
6. Check engineered ratios for shared variables
Ratios sharing a term are correlated by construction. Before interpreting any engineered feature, ask which variables it contains and which other features contain the same ones.
7. Inspect the range of every engineered feature
for c in ENG:
print(f"{c:22s} min={d[c].min():>10.3f} max={d[c].max():>10.3f}")
Two failure modes, both silent:
Explosive range. A denominator approaching zero produces extreme values that dominate a scaled model. Symptom: cross-validation standard deviation exceeding the mean. Remedy: RobustScaler, or investigate the source rows.
Collapsed range. A feature that barely varies is a constant, not a variable, and any hypothesis about it is untested. Discovering that a supposedly key ratio was held fixed across every batch is a design-of-experiments finding.
8. Include a negative control
Deliberately model a variable known to have no effect. If it appears important, the model is fitting noise.
This is cheap, requires no additional data, and is one of the few direct tests of whether a small-n model is finding signal.
9. Missing targets are rarely missing at random
Examine the rows where a target is absent:
print(d[d[target].isna()][other_cols].describe())
print(d[d[target].notna()][other_cols].describe())
If the two groups differ systematically, the available measurements are unrepresentative and any model built on them extrapolates into a region with no data.
Three distinct causes requiring different responses: selective measurement (bias), testability limits (censoring), chronology (confounded with time). Ask which applies.
10. Match the model to the sample size
At n=50 with six or more features, gradient boosting will fit the noise perfectly and SHAP will then explain a fantasy in convincing detail.
Use Ridge with RepeatedKFold, or GroupKFold where replicates exist. A straight line is usually the most that can honestly be claimed.
11. A variable that barely varies can still dominate
Finding a strong, significant effect from a variable with a narrow range is emphatic evidence that deliberately varying it would produce a much larger effect.
This is a design-of-experiments finding, and it is frequently the most valuable output of an analysis: the campaign under-explored its most important factor.
12. Modelling failure is a legitimate and valuable result
A model that cannot be built is a finding when it is accompanied by a measured explanation and a ranked set of corrective actions.
The alternative — fitting an optimiser to noise, generating confident recommendations, and discovering months later that they perform no differently from anything else — costs materials, laboratory time, and organisational appetite for the next attempt.
An honest negative delivered in week one is worth considerably more than a flattering result that collapses in month eighteen.
Quick reference: before interpreting any model
- [ ] Does it beat the baseline? If not, stop.
- [ ] Noise floor computed from replicates?
- [ ] Model error compared against repeatability?
- [ ] Hypotheses written down before modelling?
- [ ] Grouped cross-validation used where replicates exist?
- [ ] Engineered feature ranges inspected for explosion or collapse?
- [ ] Engineered ratios checked for shared variables?
- [ ] Negative control included and behaving correctly?
- [ ] Strong correlations tested in a controlled regression?
- [ ] Missing-target rows compared against measured ones?
- [ ] Model complexity matched to sample size?