Ten dimensions, each scored 1–5 against defined anchors. Score a named dataset for a named use case, not a dataset in the abstract: the same historian tags may be ready for shift-level yield forecasting and hopelessly unready for per-part defect prediction. These feed into key data management principles for ML.
How to use it
- Fix the scope before scoring. Write down the prediction target, the decision unit (part / batch / shift / line), the decision latency, and the time window under assessment. Every anchor below is judged relative to these.
- Score on evidence, not recollection. Each score needs one line naming what was checked: a table, a query result, a document, an interview. "Probably fine" is a 2.
- Score independently, then reconcile. Two assessors (one closer to the process, one closer to the data) score separately. Where they differ by 2+ points, the disagreement usually reveals an undocumented assumption; record it.
- Report the profile, not just the total. A mean of 3.5 built from four 5s and two 1s is a different project from six straight 3s.
- Re-score at gates: before feasibility work, before build, before production, and after any major process or system change.
The ten questions
1. Availability and access rights
Can we lawfully and repeatably obtain this data, at the cadence the use case needs?
| 1 | Data is locked in a vendor, OEM, or customer system with no export path, or no established right to use it for model development. |
| 2 | A one-off extract is possible with manual effort and case-by-case permission. No confirmed rights for ongoing or production use. |
| 3 | A repeatable extract exists. Usage rights are informally understood but undocumented; personal-data and third-party clauses unreviewed. |
| 4 | Documented, permissioned access (warehouse table, API, scheduled export) plus written confirmation of permitted use. Refresh needs a ticket. |
| 5 | Automated permissioned pipeline. Rights documented for training, derived models, and retention, including third-party and customer-owned data. Deletion obligations known. |
2. Granularity
Is the native resolution at or below the level at which the phenomenon and the decision occur?
| 1 | Only aggregates (shift, day, plant). The decision unit cannot be reconstructed from what is stored. |
| 2 | Grain mismatched to the question, batch-level data for a per-part target, with a one-to-many mapping and no key to resolve it. |
| 3 | Nominal grain is correct, but several important variables exist only at coarser grain and are broadcast down, creating false within-group variation. |
| 4 | Grain matches the decision unit for nearly all features. Where downsampling or aggregation occurs, the rule is documented. |
| 5 | Native grain at or finer than the decision unit across all sources, with sampling rate justified against the timescale of the physical phenomenon. Aggregation is a modelling choice, not a constraint. |
3. Label quality and definition
Is the target precisely defined, consistently applied, and independently verifiable?
| 1 | No labels, or labels are free-text operator notes with no controlled vocabulary. |
| 2 | Labels exist but the definition is implicit. Known disagreement between people, shifts, or sites. Categories such as scrap, rework, and false reject are conflated. |
| 3 | A written definition exists. Single source or single labeller; agreement never measured; boundary cases handled by judgement. |
| 4 | Documented labelling rules with worked examples and edge cases. Agreement measured on a re-labelled sample and found acceptable. Label latency (how long after the event the label appears) is known. |
| 5 | As 4, plus every label traceable to an authoritative measurement or disposition record, versioned when the definition changes, with censoring and unresolved cases explicitly represented rather than dropped. |
4. Timestamp integrity
Can events be ordered and joined across sources without ambiguity?
| 1 | No timestamps, or timestamps entered by hand. |
| 2 | Timestamps present but recorded on local clocks with no timezone convention, no DST handling, and unknown drift. |
| 3 | Machine-generated with a consistent timezone convention, but resolution is coarse relative to the process, or it is unclear whether the field is event time or database insert time. |
| 4 | UTC with offset preserved. Event time distinguished from ingest time. Clock synchronisation in place across sources; ingestion latency characterised. |
| 5 | As 4, plus synchronisation verified to a resolution finer than the shortest signal of interest, drift monitored and alarmed, and a documented rule for late-arriving and out-of-order records. |
5. Traceability to part or batch
Can each record be tied to the physical thing it describes, along the whole route?
| 1 | No identifier. Linkage between process data and outcome is inferred from time windows alone. |
| 2 | Identifiers exist within individual systems but there is no crosswalk. Joins are heuristic, nearest-timestamp matching or similar. |
| 3 | A key carries through most of the chain but breaks at one or more steps: a merge or split, a re-serialisation, a manual station, a rework loop. |
| 4 | Persistent identity through the process including splits and merges. Genealogy reconstructible; the break rate is measured and small. |
| 5 | Genealogy queryable in both directions, forward from raw material lot, backward from a field failure, including rework and reprocessing paths, and validated by audit rather than assumed. |
6. Sensor calibration and measurement history
Do we know whether a change in the numbers reflects a change in the process?
| 1 | Calibration state unknown. Sensors have been replaced, repositioned, or re-ranged without record. |
| 2 | Calibration is performed, but records are local or on paper and cannot be joined to the data. |
| 3 | Calibration records held centrally, but sensor swaps, firmware updates, and scaling changes are not consistently logged. |
| 4 | Calibration events, hardware changes, and measurement uncertainty available as joinable metadata. Drift history retrievable per instrument. |
| 5 | As 4, plus measurement system analysis for key characteristics, and calibration or replacement events flagged in the dataset itself so they can be modelled, adjusted for, or excluded. |
7. Coverage of failure modes
Does the data contain enough of what we are trying to predict, across the conditions we will actually face?
| 1 | Effectively no positive examples, or failures known only as aggregate counts detached from the process data. |
| 2 | A handful of events, dominated by one mode. Other known modes appear nowhere in the data. |
| 3 | Several modes present but severely imbalanced, and it is unknown whether rare modes are genuinely rare or simply unrecorded. |
| 4 | Main modes have enough examples to support validation, not just training. Known gaps documented. Conditions, products, tools, materials, suppliers, lines, seasons, largely spanned. |
| 5 | As 4, plus coverage assessed deliberately against an FMEA or failure taxonomy, with a stated decision for each uncovered mode: out of scope, targeted collection, or physics-based augmentation. |
8. Historical depth
Does the record span enough time and enough variation to learn from, and to validate honestly?
| 1 | Weeks of data. No complete production cycle represented. |
| 2 | Shorter than one cycle of the dominant sources of variation, seasonal, campaign, tool-life, supplier, or maintenance interval. |
| 3 | Enough to train on, but too short to hold out a genuine out-of-time validation period. |
| 4 | Multiple cycles. Supports out-of-time validation. Major regime changes are identifiable within the window. |
| 5 | As 4, and the depth is usable: schema, recipe, and process changes across the window are documented well enough that older data can be reconciled rather than silently pooled with newer data. |
9. Change control
Will these fields mean the same thing next quarter, and will we be told when they do not?
| 1 | Tag names, schemas, and units change without notice. No versioning, no changelog. |
| 2 | Changes are announced informally to some people. Nothing is recorded; the record is whoever remembers. |
| 3 | Source systems are under change control, but data consumers are not a notified stakeholder. The ML pipeline discovers changes by breaking. |
| 4 | Schema versioning and a changelog. The ML team is on the notification path for source system, recipe, and process changes. Key fields covered by an agreed contract. |
| 5 | As 4, plus enforced data contracts with automated validation and quarantine of non-conforming data, and engineering changes (setpoints, tooling, materials, control logic) recorded in a form joinable to the process data. |
10. Ownership
Is a named person accountable, with the mandate and capacity to fix problems?
| 1 | No owner. Nobody can authoritatively answer what a field means or arrange for a break to be fixed. |
| 2 | Ownership sits with "IT" or a shared inbox. Questions get routed; they do not get resolved. |
| 3 | A named individual, but the role is informal, unresourced, undocumented, and a single point of failure. |
| 4 | Named owner with documented responsibilities, a named deputy, a response commitment for issues, and a maintained data dictionary. |
| 5 | As 4, plus the owner is accountable for quality metrics that are monitored and reported, and material changes to the dataset require their sign-off. |
Scoring and interpretation
Total: /50. Record the ten individual scores alongside it; the profile carries more information than the sum.
| Band | Reading |
|---|---|
| 10–20 | Not a modelling project yet. The work in front of you is data acquisition and instrumentation. Modelling now will produce a result you cannot trust or reproduce. |
| 21–30 | Feasibility only. Explore, quantify signal, size the remediation. Do not commit to a production date or a business case built on model performance. |
| 31–40 | Build, with named remediation running in parallel. Every dimension below 3 needs an owner and a date before production. |
| 41–50 | Production candidate. Remaining gaps are manageable within normal engineering. |
Override rules - these outrank the total:
- Any 1 on access rights (Q1), label quality (Q3), or coverage (Q7) is a stop, regardless of the sum. You cannot model a target you cannot define, cannot use data you have no right to use, and cannot learn a failure mode that is not in the record.
- Any 1 on traceability (Q5) caps the project at aggregate-level prediction. Per-part questions are off the table until identity is resolved.
- Two or more 2s means the total is misleading. Report the weak dimensions explicitly rather than the average.
What this scorecard does not measure. It assesses whether the data is trustworthy and workable, not whether it contains predictive signal. A dataset can score 48 and still have no usable relationship to the target. Signal is established by feasibility analysis, not by readiness assessment, keep the two conclusions separate so that "the data is ready" is never mistaken for "the model will work."
Adapting the instrument. If your context makes a dimension irrelevant, substitute rather than drop, so totals stay comparable across assessments. Common substitutions: missingness structure (is data missing at random or systematically absent when the process misbehaves, often the more dangerous case), cross-system join integrity, and class balance stability over time. Record any substitution on the scorecard itself.
Assessment record
| Field | |
|---|---|
| Dataset | |
| Use case / prediction target | |
| Decision unit | |
| Window assessed | |
| Assessors | |
| Date |
| # | Dimension | Score | Evidence checked | Gap and owner |
|---|---|---|---|---|
| 1 | Availability and access rights | |||
| 2 | Granularity | |||
| 3 | Label quality and definition | |||
| 4 | Timestamp integrity | |||
| 5 | Traceability to part or batch | |||
| 6 | Sensor calibration history | |||
| 7 | Coverage of failure modes | |||
| 8 | Historical depth | |||
| 9 | Change control | |||
| 10 | Ownership | |||
| Total | /50 |
Overrides triggered:
Recommendation:
Re-score due: