Key lessons
1. Explainability asks why a model made a prediction. It does not automatically explain why the physical material behaves the way it does.
2. Global and local explanations answer different questions. Global explanations describe which features generally influence model predictions; local explanations describe why one particular prediction was produced.
3. Model performance must be checked before model explanations are trusted. Explaining a poor model can accurately describe the model's mistakes without revealing anything reliable about the real system.
4. Built-in Random Forest importance describes how the trees used features during training. It does not necessarily show which features help the model generalise to unseen data.
5. Permutation importance asks whether model performance deteriorates when a feature's information is disrupted. It is therefore useful for assessing whether a fitted model depends on that feature for out-of-sample prediction.
6. Built-in importance and permutation importance can disagree. A feature can be heavily used during training yet provide little—or even harmful—information on unseen data.
7. Negative permutation importance is a warning sign. It means the model sometimes performs better after that feature is shuffled, suggesting unstable or unhelpful use of the feature rather than evidence that the physical feature is harmful.
8. Importance estimates themselves have uncertainty. Large standard deviations across repeated permutations or data splits mean the importance estimate should not be treated as precise.
9. Correlated or compositionally related features complicate feature importance. Fibre and seaweed can carry overlapping information, making it difficult to assign a unique importance to either variable.
10. SHAP decomposes a model prediction into feature contributions. It starts from a model reference value and shows how each feature pushes the prediction higher or lower.
11. SHAP magnitude and SHAP direction are different. A global SHAP bar plot usually shows mean absolute contribution, so a large value means the feature strongly changes predictions—not that it always increases them.
12. A globally important feature can push an individual prediction either up or down. Seaweed, for example, ranked highly globally but reduced the prediction for the specimen we examined.
13. SHAP waterfall plots are local explanations. They show how individual feature contributions combine with the base value to produce one final prediction.
14. SHAP beeswarm plots reveal both importance and direction. They can show whether high or low values tend to push predictions up or down, but small datasets can make apparent trends depend heavily on individual specimens.
15. One extreme observation can dominate an explanation. In small datasets, a single high-value observation can substantially affect global SHAP patterns.
16. SHAP values explain the model, not causation. “Seaweed pushed the model prediction upward” is fundamentally different from “increasing seaweed increases mechanical strength.”
17. Explanations should be tested for stability across valid data splits. If different training datasets produce different feature rankings, confidence in the explanation should decrease.
18. Our SHAP explanations were unstable. Across five splits, feature rankings and magnitudes changed substantially; extraction time became the highest average SHAP feature even though seaweed dominated the seed-42 explanation.
19. Model explanation should follow a hierarchy of trust.
First ask:
- Does the model beat the baseline?
- Is its predictive performance stable?
- Are its explanations stable?
- Are those relationships scientifically plausible?
- Only then consider stronger mechanistic or causal interpretations.
20. Explainability is useful for generating hypotheses, not proving mechanisms. An unexpected SHAP pattern may suggest a variable worth studying experimentally, but controlled experiments or DoE are needed to establish physical causation.
The central principle is:
An explanation is only as trustworthy as the model and validation process behind it. Explain the model first; do not confuse that explanation with an explanation of the material system.
Short Glossary
| Term | Definition |
|---|---|
| Explainability | Methods used to understand how or why a model produces its predictions. |
| Interpretability | How easily a human can understand a model's behaviour or reasoning. |
| Global explanation | Describes model behaviour across many observations, such as overall feature importance. |
| Local explanation | Describes why the model made one particular prediction. |
| Feature importance | A measure of how strongly a fitted model relies on a feature. It is not automatically physical importance. |
| Built-in tree importance | Importance calculated from how much tree splits reduce model error/impurity during training. |
| Permutation importance | Measures how much predictive performance changes after randomly shuffling one feature. |
| Negative permutation importance | Performance improves when a feature is shuffled, suggesting the model may be using that feature unreliably. |
| SHAP | SHapley Additive exPlanations; a framework that attributes portions of a model prediction to individual features. |
| SHAP value | The contribution of one feature to moving a prediction away from the model's reference value. |
| Base / expected value | The reference model output from which SHAP feature contributions begin. It is different from the baseline model used for performance comparison. |
| Mean absolute SHAP value | Average magnitude of a feature's SHAP contributions. It measures strength of influence, not whether the effect is positive or negative. |
| SHAP bar plot | Global plot ranking features by average absolute SHAP contribution. |
| SHAP beeswarm plot | Global plot showing the distribution and direction of SHAP contributions across observations. |
| SHAP waterfall plot | Local plot showing how each feature pushes one prediction away from the base value. |
| Explanation stability | Whether feature rankings and contribution patterns remain similar when training data or model conditions change. |
| Correlation / feature dependence | Situation where features contain related information, making unique feature attribution difficult. |
| Causal effect | A genuine physical change in an outcome caused by changing a variable. Predictive explainability alone cannot establish this. |
| Hypothesis generation | Using model explanations to identify variables or relationships worth testing experimentally. |
Conclusion
For this synthetic mechanical-strength model, the explainability tools are useful for understanding how the Random Forest behaves, but they do not provide reliable materials-science conclusions. The model does not beat the baseline, permutation importance is unstable, and SHAP rankings change materially across different data splits. The correct use of these explanations is therefore diagnostic and hypothesis-generating, not causal or decision-making.