Data Supplier Performance Scorecards and Quarterly Reviews

How a performance scorecard differs from a selection scorecard

A performance scorecard measures what a supplier actually shipped against what the contract promised, while a selection scorecard predicts which supplier will perform. The data vendor evaluation scorecard weights capability, rights posture and price before signature. After signature, those weights stop mattering and observed results take over: did the March refresh land on time, did it pass your acceptance checks, and how long did fixes take.

The cadence is also different. Selection is a one-time event; performance management runs per delivery, rolls up monthly, and gets a structured review every quarter. It sits inside the wider lifecycle covered in the AI training data procurement hub, between onboarding and renewal.

Metric definitions come from the SLA, not the scorecard

Every scorecard metric should reuse the measurement definition already written into the SLA or statement of work, because a metric the supplier never agreed to is a metric they will dispute. Practitioner guidance on data SLAs stresses that terms such as turnaround, capacity and accuracy only work when the contract also states how each is measured [1]. If your agreement says "95% label accuracy," the scorecard needs the same sample size, the same reviewer, the same adjudication rule and the same definition of an error.

Where the SLA is silent, fix the gap at the next amendment rather than inventing a definition in a spreadsheet. The ongoing data supply agreement is the right place to state delivery windows, acceptance criteria and cure periods. ISO/IEC 5259-3 is useful here as a management frame: it sets requirements for running and continually improving data quality management for ML, but deliberately leaves the specific metrics to you [3].

KPIs for data suppliers: the core set

Six KPIs cover most ongoing data supply relationships, and each needs a numerator, a denominator and a source system. Keep the set small enough that both sides can recompute it from the same evidence: your ingestion logs, your acceptance test results and the ticket history.

Illustrative example: invented to show structure; it does not describe an available dataset.

Thresholds above are placeholders. Set your own from the SLA and from what your training or evaluation pipeline can tolerate. If you are still lining up the supply this scorecard will govern, you can describe the recurring data you need to SourceX before setting thresholds.

On-time delivery rate for datasets needs a precise clock

On-time delivery is only meaningful when you define what "delivered" means: file arrival, a complete manifest, or a delivery that passed acceptance. Pick one and write it down. Many buyers measure arrival of a complete manifest (row counts, file checksums, schema version) in the agreed bucket or Delta Sharing share, because a partial upload that trickles in over three days is not a delivery.

Separate late deliveries from incomplete ones. A refresh that arrives on time but missing a partition should count against acceptance or completeness, not punctuality, or the two metrics blur. If you run incremental drops rather than full refreshes, define the window per increment; the trade-offs are covered in incremental vs. full refresh deliveries.

Tracking data vendor quality trend, not just averages

Quality trend matters more than any single quarter's average, because supplier data tends to degrade gradually before it fails visibly. Commentary on enterprise AI data procurement recommends continuous monitoring of supplier quality rather than relying on the evaluation done at selection [2]. Annotation teams turn over, upstream systems change, and a supplier's export job can silently drop a field.

Plot each KPI per delivery with a rolling three-delivery median and flag any two consecutive deliveries that move in the wrong direction. Watch these leading indicators specifically:

• Schema drift: new, renamed or retyped columns between deliveries. Tools can be set to fail on new columns so drift is reviewed before ingestion [5].
• Distribution shift: changes in category mix, record length, date range or language share that the supplier did not announce.
• Label agreement: falling inter-annotator agreement or rising adjudication rates on your audit sample.
• Residual identifiers: any increase in PII scanner hits after de-identification, even below the defect threshold.
• Duplicate and near-duplicate rate: overlap with prior deliveries, which inflates volume without adding signal.

For a deeper method on judging supplier output, see the guide on evaluating data supplier quality.

Documentation and rights belong on the scorecard

Documentation completeness and rights status are scored metrics, not footnotes, because a technically clean delivery with a broken rights chain is unusable. Require each delivery to carry a manifest plus an updated dataset card covering upstream sources, collection and annotation methods and intended use, the fields the Data Cards framework identifies as essential [4]. Score completeness as a gate: missing fields block acceptance.

Rights can change mid-contract. A supplier's own customer contract may be amended, a subcontractor may be replaced, or a consent basis may lapse. Law-firm commentary notes that vendors are generally expected to secure rights to the data they supply and that flow-down terms extend those obligations to the vendor's own suppliers [6]. Score whether such changes were disclosed before the affected delivery, and cross-check against the chain-of-title documents you collected at onboarding.

If you are a general-purpose model provider, rights drift also touches your own obligations. Article 53 of the EU AI Act requires GPAI providers to maintain a copyright compliance policy and publish a summary of training content; as of October 2026, these duties have applied since 2 August 2025 [7]. A scorecard that records rights notices per delivery gives your compliance team the trail it needs.

Data vendor quarterly business review agenda

A data vendor quarterly business review should cover four topics in a fixed order: performance, upcoming deliveries, rights or sourcing changes, and roadmap. Send the scorecard to the supplier five business days before the meeting so the session is spent on causes and actions, not on reconciling numbers.

Illustrative example: invented to show structure; it does not describe an available dataset.

QBR agenda template (60 minutes)

1. Scorecard review (20 min): each KPI against threshold, trend since last quarter, open defect tickets and their ages.
2. Root causes and corrective actions (10 min): status of any open corrective action plan, owner, due date, evidence of closure.
3. Upcoming deliveries (10 min): next two delivery windows, expected volume, known schema or field changes, planned de-identification method changes.
4. Rights, sourcing and workforce changes (10 min): new or replaced upstream sources, subcontractor changes, consent or contract amendments, any regulatory notices.
5. Roadmap and scope (10 min): new fields or record types your teams want, coverage gaps, changes to your intended use that need a license amendment.

Record decisions and action items in the same system as the scorecard so the next review opens with them. Teams running several suppliers in parallel can find portfolio-level practices in managing many data suppliers and single-source vs. multi-source supplier strategy.

Escalation path from missed threshold to contract remedy

An escalation path turns scorecard results into consequences that both sides agreed in advance, so a bad quarter becomes a managed process rather than a renegotiation. Write the tiers into the agreement and reference them on the scorecard.

Illustrative example: invented to show structure; it does not describe an available dataset.

Keep rights failures on a faster track than quality failures. A residual-identifier finding or an undisclosed source change should skip directly to tier 2 and pause ingestion of the affected delivery until resolved. The broader post-signature checks that feed this, including attestations and re-verification, are covered in ongoing due diligence of data vendors.

Common scorecard failure modes

Most scorecards fail for definitional reasons rather than because suppliers underperform. Watch for these:

• Averaging away drift: a quarterly mean hides a supplier that was excellent in month one and failing by month three.
• Unshared evidence: if the supplier cannot see the validation output behind a defect rate, they will contest it.
• Acceptance done late: checks run weeks after arrival make correction turnaround meaningless and let bad data reach training runs.
• Too many metrics: fifteen KPIs produce no clear signal and no clear escalation trigger.
• Rights as a yearly attestation only: sourcing changes happen between renewals and need per-delivery disclosure.

Ongoing data supply sourced and managed with SourceX

SourceX sources operational datasets from US companies on request and manages the commercial process, including licensing agreements and ongoing purchases. Every dataset is rights-reviewed for ownership and consents, delivered under a license that defines records, uses, term and delivery, and released only with the supplying company's approval through private, access-controlled workflows. If you need a recurring supply of support, sales, engineering, document or workflow data, describe the data you need to SourceX.