Golden Dataset Governance for AI Evaluation Systems

A lot of AI evaluation systems start with a folder of examples. Someone collects prompts, documents, expected answers, model failures, incident cases, and edge cases. The evaluation runner loads the file, produces a score, and the team starts treating that score as release evidence.

That is too weak for production AI engineering. A golden dataset is not just test data. It is a controlled asset. If cases have no owner, no labeling guide, no adjudication path, no leakage controls, no lifecycle state, and no versioning policy, the evaluation gate can look serious while measuring unstable or contaminated behavior.

In this issue, we build a markdown-first governance package for golden datasets used by AI evaluation systems. The running example is invoice extraction and approval routing, but the core topic is dataset governance: case contracts, expected-output rules, candidate versus active cases, disputed labels, dataset versioning, leakage prevention, evaluation slices, and release evidence.

The Artifact

You are building a governance package for an evaluation dataset, not an invoice processing app. The repository contains the files a team needs before a golden dataset becomes part of a prompt gate, model upgrade gate, retrieval regression gate, or release process.

The package contains:

• A golden dataset contract
• A labeling guide
• An adjudication runbook
• A dataset versioning policy
• A leakage and retention policy
• A JSON Schema for governed cases
• A small synthetic invoice golden set
• A release evidence example

This is a valid no-runtime issue because the artifact itself is the production control surface. Before a team needs a better evaluation harness, it needs to know whether the dataset feeding that harness can be trusted.

System Structure

The control flow is intentionally simple. Case proposals do not become golden cases automatically. They pass through labeling rules, review, adjudication, schema validation, leakage checks, and versioning before they can affect a release gate.

The important boundary is that the evaluation runner is downstream of governance. It should not be the first place where label quality, privacy risk, leakage, and case lifecycle are discovered.

Repository Shape

The repository stays markdown-first:

There is no model call in this repository. There is no OCR pipeline. There is no invoice approval workflow. The repository defines the dataset operating model that a real evaluation system would consume.

Golden Cases Are Product Contracts

A golden case is not just an input and an expected answer. It is a small product contract. It says what the system should do for one meaningful situation, who approved that expectation, what risk slice it belongs to, where the case came from, and how it may be used.

The contract requires fields like these:

{
  "case_id": "INV-GOLD-002",
  "dataset_version": "1.0.0",
  "lifecycle_state": "active",
  "task": "invoice_extraction_and_approval_routing",
  "labels": [
    "missing_po",
    "total_mismatch",
    "manual_review_required"
  ],
  "slices": [
    "missing_required_field",
    "numeric_consistency",
    "manual_review"
  ],
  "privacy": {
    "sensitivity": "synthetic",
    "retention_class": "template"
  },
  "allowed_uses": [
    "release_gate",
    "prompt_regression",
    "model_upgrade_gate"
  ]
}

Those fields are not paperwork. They are how a release gate knows whether a case is active, which metric slice it affects, whether it is safe to store, and whether it is allowed to influence a promotion decision.

The Invoice Is Only the Example

Invoice extraction is a useful example because it has structured fields, missing data, numeric consistency, document evidence, and deterministic routing decisions. Those properties make governance visible. They are not special to invoices.

One synthetic case asks the evaluated system to preserve printed values instead of repairing the document:

{
  "input": {
    "text": "Contoso Facilities\nInvoice Number: CF-4401\nInvoice Date: 2026-07-05\nPayment Terms: Due on receipt\nSubtotal: USD 1200.00\nTax: USD 240.00\nAmount Due: USD 1520.00\nNotes: Emergency repair work"
  },
  "expected_output": {
    "fields": {
      "purchase_order": null,
      "subtotal": 1200,
      "tax_total": 240,
      "total_due": 1520
    },
    "approval_route": "manual_review",
    "policy_flags": {
      "has_missing_required_field": true,
      "has_total_mismatch": true,
      "requires_human_approval": true
    }
  }
}

The system is not being taught to love invoice math. The case exists to prove a governance rule: expected outputs should describe correct product behavior. If the printed total conflicts with the computed total, the model should not silently fix the invoice. It should preserve the source, mark the mismatch, and route to review.

Labeling Guide Before Scores

A dataset should not wait for evaluation failures before defining what correct means. The labeling guide comes first. It explains how reviewers normalize dates, preserve invoice numbers, handle missing purchase orders, classify routing decisions, and require evidence spans.

For the running example, the guide has rules like this:

### Purchase Order

- Extract the purchase order only when it is explicitly present.
- Use null for missing purchase orders.
- Do not infer a purchase order from a project code, account code, or memo.

### Currency and Amounts

- Normalize currency to ISO 4217 codes such as USD, GBP, or EUR.
- Normalize amounts as decimal numbers without currency symbols.
- Preserve subtotal, tax_total, and total_due separately.
- If totals conflict, keep the printed values and mark has_total_mismatch: true.
- Do not repair invoice math silently.

This is where many eval sets quietly fail. The model output is judged by reviewer intuition instead of a written contract. That makes scores unstable because the expected answer changes depending on who reviews the failure.

Candidate Is Not Active

New cases should not automatically become release-gating cases. A candidate case is useful for exploration, but it should not block or approve a release until the expected output is reviewed and the case is admitted into the active dataset version.

The sample includes a candidate duplicate-invoice case:

{
  "case_id": "INV-GOLD-003",
  "lifecycle_state": "candidate",
  "labels": [
    "duplicate_invoice",
    "manual_review_required"
  ],
  "slices": [
    "duplicate_detection",
    "manual_review"
  ],
  "adjudication": {
    "state": "pending",
    "rationale": "Candidate case requires domain review because duplicate detection depends on an external payment ledger signal."
  },
  "allowed_uses": [
    "exploration"
  ]
}

That distinction matters. Duplicate detection depends on data outside the invoice text. If the evaluation case does not define the ledger signal, reviewer expectation, and routing policy, it should remain a candidate. It can teach the team what coverage is missing without pretending to be a trustworthy gate.

Adjudication Prevents Silent Drift

Every mature eval dataset eventually hits disagreement. One reviewer says a case should pass. Another says the model guessed. A product owner changes the rule. A document is ambiguous. A case leaks into prompt examples. Without adjudication, these become silent edits to the golden file.

The runbook gives those cases a visible path:

1. Freeze the current case content.
2. Record the reason adjudication is needed.
3. Compare the case against the labeling guide.
4. Identify whether the issue is input ambiguity, expected output ambiguity,
 policy change, privacy risk, or leakage.
5. Decide whether the case should be active, changed, retired, or excluded.
6. Update the adjudication record.
7. Update the dataset version according to the versioning policy.
8. Record the decision in the next release evidence file if the case affects a gate.

The goal is not bureaucracy. The goal is to keep the evaluation target from moving invisibly. If expected outputs change after every failed run, the gate is no longer measuring regression. It is measuring the team's willingness to edit the answer key.

Versioning Makes Evidence Comparable

A model or prompt passed the eval is not enough information. It passed which dataset version? With which active cases? With which candidate cases excluded? With which expected-output semantics?

The versioning policy uses semantic dataset versions:

MAJOR.MINOR.PATCH

Major:
- expected output semantics change
- active cases are removed or retired in a way that changes gate meaning
- labeling rules change materially

Minor:
- active cases are added
- new slices are added
- candidate cases are promoted to active

Patch:
- typos are fixed without changing labels
- non-critical metadata is corrected
- documentation is clarified

This is the same reason software teams version APIs and database migrations. A release decision is a historical fact. If the dataset changes later, the old release evidence still needs to be understandable.

Leakage Is an Eval Integrity Problem

Golden cases should not become prompt examples, few-shot examples, fine-tuning data, synthetic data seeds, or public demos. Once active golden cases leak into the system being evaluated, future scores can improve for the wrong reason.

The sample case records leakage controls directly:

"leakage_controls": {
"training_allowed": false,
"few_shot_allowed": false,
"prompt_debugging_allowed": false
}

This is not only a privacy rule. It is an experimental validity rule. The point of a golden set is to test behavior that the system has not been tuned against directly. If the same cases become model context, the release gate becomes easier to pass and harder to trust.

Slice Metrics Beat One Big Score

Aggregate scores hide the failures that matter most. A dataset can pass overall while failing every missing-data case, every policy exception, or every case where the correct answer is abstention.

The invoice example uses slices like:

• happy_path
• date_normalization
• missing_required_field
• numeric_consistency
• manual_review
• duplicate_detection

Those slices are the beginning of a gate policy. A release might require perfect behavior on manual-review routing while allowing a report-only status for duplicate detection until the candidate cases are adjudicated. That is more useful than a single pass rate with no risk shape.

Walking a Release Decision

The repository includes a release evidence template. It records which dataset version was used, how many cases were active, which candidates were excluded, which slices were required, and which gate results mattered.

## Dataset Boundary

- Active cases used: 2
- Candidate cases excluded: 1
- Retired cases excluded: 0
- Excluded cases excluded: 0

Candidate case INV-GOLD-003 was excluded because adjudication is pending.

## Decision

Decision: promote to staging.

Rationale:

The candidate passed all required active cases in dataset version 1.0.0.
Duplicate detection remains report-only because the only duplicate case is still
candidate state and depends on an external ledger signal.

That evidence is more valuable than a naked score. It explains what the release gate covered, what it did not cover, and which limitations should follow the release into staging.

The Deterministic Boundary

The AI system under evaluation can be probabilistic. The dataset governance layer should not be. Lifecycle states, allowed uses, schema requirements, version changes, slice membership, adjudication status, leakage rules, and release evidence should be explicit.

This is the control boundary. The model may extract an invoice total, classify a security alert, review a contract clause, or rank a candidate document. The golden dataset tells the evaluation system which behavior counts, which cases are allowed to gate a release, and when the evidence is no longer clean.

Why This Architecture Works

• The dataset contract makes each golden case reviewable
• The labeling guide prevents expected outputs from becoming reviewer folklore
• Candidate state keeps unreviewed cases out of release gates
• Adjudication makes disagreement visible instead of silent
• Versioning keeps historical release evidence comparable
• Leakage controls protect evaluation integrity
• Slice reporting exposes risk-specific failures that aggregate scores hide

The evaluation runner still matters, but it is not the center of the system. The dataset contract is the center. Without it, a runner can produce precise numbers from unstable evidence.

Potential Enhancements

• Add a schema validation script that fails when active cases are incomplete
• Add minimum slice coverage requirements before dataset activation
• Add a case registry that records source hashes without storing raw documents
• Add reviewer assignment rules for high-risk or regulated slices
• Add separate public, internal, and confidential dataset manifests
• Add automated checks that block active golden cases from prompt fixtures
• Add drift reports that compare current production failures against existing slices

Those are implementation layers. The first step is still the same: write down the dataset contract before treating evaluation output as release evidence.

Final Notes

A golden dataset is not a pile of examples. It is an engineering control.

The invoice example in this issue is deliberately small because the domain is not the point. The point is the operating model around the cases: label rules, ownership, adjudication, versioning, leakage controls, and release evidence. That is what turns an eval file into a production asset.

Explore the repository at the GitHub repository.

See you in the next issue.

Stay curious.