Data Contracts and Data Quality: Enforcing Reliability in Modern Pipelines

Modern data platforms move records across many independent systems: an application emits an event, an ingestion job lands it in a lake, a transformation framework reshapes it, and a dashboard or a machine-learning model consumes the result. Every one of these handoffs is an implicit agreement about the shape and meaning of the data, and every silent violation of that agreement propagates downstream until a report is wrong or a model degrades. A data contract—a versioned, machine-readable agreement between the team that produces a dataset and the teams that consume it, specifying its schema, semantics, service-level guarantees, and quality expectations—turns that implicit agreement into an explicit, testable artifact. This post examines how data contracts and data quality enforcement work together to make pipelines reliable, and why the two ideas are complementary rather than interchangeable.

The distinction that organizes the discussion is between prevention and detection. A data contract prevents bad data from entering a system by asserting expectations at the boundary where data is produced. Data observability, by contrast, detects problems after data has already landed, by monitoring freshness, volume, and schema drift and raising anomaly alerts. A mature platform uses both. The material below defines the contract as an artifact, presents a taxonomy of six data quality dimensions, surveys the enforcement engines that operate at the record, DataFrame, warehouse, and stream layers, and describes the patterns—shift-left validation, continuous-integration gates, pre-ingestion quarantine, circuit breakers, and dead-letter queues—that put a contract into force.

The Data Contract as a First-Class Artifact

A data contract is a formal, versioned agreement between a data producer and its consumers. The producer is the system or team that emits a dataset—an application service publishing events, a database exposing a table, or an ingestion job writing files. The consumers are the downstream systems and teams that read that dataset: analytics models, dashboards, machine-learning features, and other services. Before contracts, the terms of that relationship lived in tribal knowledge and out-of-date documentation, so a producer could rename a column or change a unit of measurement without warning, and consumers discovered the break only when their output failed. A contract makes the terms explicit and machine-readable, so that a change which would violate them can be caught automatically.

Four elements typically appear in a contract. The schema declares the fields, their types, and their nullability. The semantics describe what each field means, including units, allowed value sets, and business definitions, so that a field named revenue is unambiguous about currency and whether it is gross or net. The service-level agreement (SLA) states operational guarantees such as freshness, expected update frequency, and availability. The quality expectations encode testable rules—for example, that a primary key is unique or that a percentage column falls between zero and one hundred. Figure 1 shows how these four elements sit on the boundary between a producer and its consumers.

A Standard for the Contract: ODCS

Writing a contract in an ad-hoc format leaves every team to invent its own structure, so a shared specification is useful. The Open Data Contract Standard (ODCS) is a vendor-neutral YAML format that describes the agreement between a data producer and its consumers. Its current version is v3.1.0 (Bitol / Linux Foundation AI & Data, as of 2026). ODCS originated as PayPal’s internal data contract template, was open-sourced, and was donated to the Linux Foundation; it is now stewarded by Bitol, a Linux Foundation AI & Data incubation project licensed under Apache 2.0 that was formed on 30 November 2023 when the AIDA User Group and LF AI & Data joined forces (bitol.io, as of 2026). Bitol also stewards the Open Data Product Standard (ODPS), which has reached v1.0.0.

An important point about the current landscape is that the ecosystem is consolidating around a single standard. A separate effort, the Data Contract Specification, is being deprecated and is converging on ODCS v3.1.0. Its maintainers advise users to migrate, and tooling support in the Data Contract CLI and Entropy Data is planned only until the end of 2026 (datacontract-specification.com, as of 2026). A team beginning with contracts in 2026 should therefore standardize on ODCS rather than the older specification. A minimal ODCS-style contract illustrates the artifact concretely.

apiVersion: v3.1.0
kind: DataContract
id: orders-contract
name: Orders
version: 1.2.0
status: active
schema:
  - name: orders
    properties:
      - name: order_id
        logicalType: string
        required: true
        unique: true
      - name: customer_id
        logicalType: string
        required: true
      - name: order_total
        logicalType: number
        required: true
        quality:
          - rule: minimum
            mustBeGreaterThanOrEqualTo: 0
      - name: status
        logicalType: string
        required: true
        quality:
          - rule: enum
            values: [active, shipped, cancelled]
slaProperties:
  - property: freshness
    value: 15
    unit: minute

Treating the contract as versioned code has a direct engineering consequence: a change to the contract is a change to a file in version control, subject to review and to automated compatibility checks. This is what makes the shift-left patterns described later in this post possible, and it links data contracts to the broader practice of treating transformation logic as code, as covered in the guide to dbt for building transformation pipelines.

Six Dimensions of Data Quality

Before a contract can assert quality expectations, quality itself must be defined precisely enough to be measured. A widely used approach decomposes quality into distinct dimensions, each answering a different question about the data. The six dimensions below form a compact and practical taxonomy. Each is defined on first use, because the terms are often used loosely in practice.

Completeness asks whether the expected data is present, with no missing values or absent records where they are required. A completeness check might assert that customer_id is never null, or that a daily table’s row count falls within an expected band. Uniqueness asks whether each real-world entity appears once, with no unintended duplicates; a typical check enforces primary-key uniqueness so that no order_id is repeated. Validity asks whether values conform to a defined format, type, range, or allowed set—the domain of the field. Examples include requiring that status belongs to a fixed set of allowed values, that an email address matches a regular expression, or that an age is not negative.

Accuracy asks whether values correctly describe the real-world entity they represent. This is among the hardest dimensions to test in isolation, because it requires comparison against a trusted system of record rather than an internal rule; a value can be valid and unique yet still wrong. Consistency asks whether values agree across systems, tables, and over time, with no contradictions—for instance, that the sum of an order’s line items equals its recorded total, or that a customer’s country is identical in every table. Timeliness asks whether data is fresh and available within its expected latency or SLA window; a timeliness check verifies that the maximum event timestamp is within the last few minutes or that a partition landed on schedule.

Accuracy and consistency are the two dimensions that resist mechanical enforcement most strongly, because they depend on external reference points and cross-system reconciliation rather than a self-contained rule. A contract can assert them, but verifying them often requires a comparison the pipeline cannot perform on its own. A seventh dimension, integrity—referential integrity, meaning that foreign keys resolve to existing rows—is sometimes added; it can be treated as a specialized form of consistency across tables.

Where Enforcement Happens: Four Layers and Their Engines

A single contract is enforced at several points, because data changes form as it moves. It arrives as individual records at an application boundary, is assembled into in-memory tables (DataFrames) for processing, is materialized as warehouse tables, and travels as messages on streams. Each form has a matching class of validation engine. The essential distinction—already stated for the contract itself—applies here too: the specification declares what must hold, and each engine enforces it in the representation it understands.

Record and DataFrame Boundaries

At the application boundary, data arrives one record at a time—an incoming API request, a single event to be published. Pydantic validates and parses data at this record level. Its current version is v2.13.4 (pydantic/pydantic releases, as of 2026). The validation engine lives in a component called pydantic-core, written in Rust through the pyo3 binding layer, which delivers roughly a fivefold to fiftyfold performance improvement over version 1 depending on the workload (pydantic.dev, as of 2026). Pydantic excels at fast per-record validation at ingestion and API edges, but it operates one object at a time and is not designed for table-scale statistical checks.

Once records are assembled into a DataFrame—an in-memory tabular structure—the relevant unit of validation becomes the column and the table rather than the single object. Pandera provides schema-based, statistical validation for DataFrames. Its current version is 0.32.1 (unionai-oss/pandera releases, as of mid-2026). Pandera 0.19.0 added Polars validation, and by version 0.29 (January 2026) a single schema definition could validate data across pandas, Polars, Dask, Modin, PySpark, and Ibis; a Narwhals-powered backend introduced in 0.32.0 adds lazy validation across Polars, Ibis, and PySpark SQL. A Pandera schema expresses column-level expectations directly.

import pandera as pa
from pandera import Column, Check

schema = pa.DataFrameSchema({
    "order_id":    Column(str, unique=True, nullable=False),
    "customer_id": Column(str, nullable=False),
    "order_total": Column(float, Check.ge(0)),
    "status":      Column(str, Check.isin(["active", "shipped", "cancelled"])),
})

# Raises SchemaError on the first violation, or collects all
# violations when lazy=True.
validated = schema.validate(df, lazy=True)

Warehouse Tables

Inside the warehouse, checks run against materialized tables in SQL. Three engines are common here. dbt tests co-locate assertions with the transformation models that produce a table. dbt Core’s current version is 1.11.12 (dbt-core GitHub releases, as of 2026-07-01), with a Rust-based v2.0.0-alpha in development as the foundation for the dbt Fusion engine. dbt distinguishes data tests—generic and singular assertions that run against materialized data—from native unit tests, which validate SQL logic and were introduced in v1.8; in v1.9 and later, dbt test --resource-type test runs data tests while excluding unit tests. Because dbt tests only cover what dbt materializes, they do not guard the ingestion boundary or streams. The role of dbt in the transformation layer is treated fully in the guide to dbt transformation pipelines.

Great Expectations (GX Core) is a dedicated validation framework built around a reusable library of expressive assertions called Expectations, together with automatic profiling and human-readable validation reports called Data Docs. Its current version is 1.18.2 (Great Expectations changelog, as of 2026-06-26); GX Core 1.0 was a major API redesign that separated the open-source GX Core from the commercial GX Cloud. It supports SQL, Spark, and pandas backends. Its strength is the breadth and reusability of its Expectation library; historically its weakness has been a heavier setup and a steeper learning curve. Figure 5 traces its validation flow.

Soda Core is a second dedicated framework, known for concise checks and a continuous-integration-friendly command-line interface. Its current version is 4.16.0, released 29 June 2026, and it requires Python 3.10 or later (Soda Core release notes, as of 2026-06-29). A notable change is that Soda Core version 4 makes data contracts the default way to define quality rules—a breaking change that moves away from the older SodaCL “checks” syntax toward a contract-based syntax. A concise Soda check reads close to natural language.

checks for orders:
  - row_count > 0
  - missing_count(customer_id) = 0
  - duplicate_count(order_id) = 0
  - invalid_count(status) = 0:
      valid values: [active, shipped, cancelled]
  - freshness(order_time) < 15m

Streams

On a Kafka stream, the contract is the message schema, and the enforcement point is a schema registry—a service that stores schemas and rejects producers whose schema is incompatible with the registered version. Confluent Schema Registry supports Avro, Protobuf, and JSON Schema (referred to as JSON_SR) out of the box on both Confluent Platform and Confluent Cloud (docs.confluent.io, as of 2026). It governs schema evolution through compatibility types: BACKWARD (the default), BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, and NONE. BACKWARD is the default because it allows consumers using the new schema to read data written with the previous schema, which lets a consumer rewind to the start of a topic. Confluent recommends BACKWARD_TRANSITIVE for Protobuf, because adding new message types is not forward compatible, whereas Avro was designed with schema evolution in mind (docs.confluent.io, as of 2026). A schema registry enforces the shape and evolution of messages, not their semantic quality; it will not catch a null in a field that is technically nullable or a value out of an expected range. Streaming schema enforcement pairs naturally with change-data-capture pipelines, as discussed in the guide to change data capture with Debezium and Kafka, and with the consumer-side handling covered in the Kafka consumer implementation guide.

The following table summarizes what each engine does best and where its boundary lies. Every version cited carries its source and date in the prose above.

Enforcement Patterns: Shift-Left, Gates, and Dead-Letter Queues

Having an engine is not the same as having a strategy for where and when it runs. Several established patterns place enforcement at different points in the data’s journey, trading availability against correctness in different ways.

Shift-Left at the Producer Boundary

Shift-left is the practice of moving validation as early as possible—toward the point where data is produced—rather than catching problems late in the pipeline. The term borrows from software testing, where moving tests earlier (“to the left” on a left-to-right timeline) reduces the cost of fixing defects. Applied to data, shift-left means validating at the application or ingestion boundary so that non-conforming data never enters the warehouse at all. This is the cheapest place to catch an error, because the cost of a defect rises as it travels downstream: a bad value caught at the producer affects nothing, whereas the same value caught at the BI or ML layer may have already corrupted reports, retrained a model, or been copied into many derived tables. Figure 3 depicts this rising cost.

Continuous-Integration Gates on Contract Changes

Because a contract is a file under version control, a proposed change to a producer’s schema arrives as a pull request. A continuous-integration (CI) gate is an automated check that runs on that pull request and blocks the merge if the change would break the contract. Two checks matter here: validating that the contract file itself is well formed, and running a backward-compatibility check that determines whether existing consumers can still read data produced under the new schema. This mirrors exactly what a schema registry does at runtime for streams, moved earlier to the moment of code review. A breaking change—removing a required field, narrowing a type—fails the gate and cannot merge until the producer and consumers agree on a migration. Figure 6 shows the contract advancing through versions with a compatibility check at each transition.

Pre-Ingestion Gates and Circuit Breakers

When validation cannot happen strictly at the producer—because the producer is a third party, or the data arrives as files—a pre-ingestion gate places incoming data in a staging area first and requires a validation step to pass before the data is promoted to a production table. The gate fails closed: if the check does not pass, the data does not advance. This quarantines suspect data rather than exposing it to consumers.

A related pattern is the circuit breaker, which halts a running pipeline when a contract check fails beyond a chosen threshold, rather than allowing bad data to propagate. The name is borrowed from electrical engineering, where a breaker interrupts a circuit to prevent damage. In a data pipeline hosted by an orchestrator, a failed check fails the orchestrator task and stops downstream jobs from running. This deliberately trades availability for correctness: a stalled pipeline is often preferable to a fast one that delivers wrong answers, because a visible delay prompts investigation whereas a silent error can persist unnoticed for days. Orchestrators such as those covered in the Apache Airflow orchestration guide are the natural host for both pre-ingestion gates and circuit breakers, because they already model tasks, dependencies, and failure propagation. A validation task placed upstream of a load task allows the orchestrator to skip or fail the load automatically when the check does not pass, so the circuit-breaker behavior falls out of the dependency graph rather than requiring bespoke error handling.

Dead-Letter Queues for Streaming

In a streaming pipeline, halting the entire stream because a single record fails a check would be too blunt, since one malformed message would block every well-formed one behind it. A dead-letter queue (DLQ) solves this by routing records that fail schema or quality checks to a separate queue for inspection and later replay, while valid records continue on the main stream. This isolates failures at the level of the individual record rather than the whole pipeline, and it preserves the malformed records for diagnosis instead of discarding them. Figure 4 shows a stream splitting into a pass branch and a dead-letter branch, with a replay path back into the pipeline once the underlying problem is fixed.

Contracts and Observability: Prevention Versus Detection

Data contracts are frequently discussed alongside data observability, and the two are sometimes conflated. They address the same goal—reliable data—from opposite directions. A contract is a mechanism of prevention: it asserts expectations at the boundary and refuses data that violates them, so a defect is stopped before it enters the system. Observability is a mechanism of detection: it monitors the data that has already landed—tracking freshness, volume, distribution, and schema drift—and raises an alert when reality departs from the norm. Observability finds problems the contract did not anticipate, including gradual distribution shifts and upstream failures that produce technically valid but unusual data. Figure 8 places the two side by side over a shared pipeline.

Neither mechanism substitutes for the other. A contract cannot anticipate every failure mode, particularly slow statistical drift in otherwise valid data; observability catches those. Observability, on its own, only tells a team that something has already gone wrong, often after consumers have been affected; a contract prevents the class of failures it can express. The two belong together in a mature stack. Quality gates also interact with the storage layer, because a table’s schema evolution is itself a contract concern—an area explored in the comparison of Iceberg, Delta Lake, and Hudi table formats and in the discussion of typing at the file level in the guide to Parquet and Apache Arrow internals. An end-to-end pipeline where these gates would apply in practice is described in the walkthrough from InfluxDB to AWS Iceberg with Telegraf.

The Business Case for Enforcement

The motivation for this discipline is that poor data quality carries a measurable cost. A frequently cited estimate holds that poor data quality costs organizations at least 12.9 million US dollars per year on average; this figure comes from Gartner’s 2020 Magic Quadrant for Data Quality Solutions, based on a survey of 154 reference customers across 16 vendors (Gartner, as of 2020). It should be read as a 2020 estimate rather than a current measurement, and organizations vary widely, but it captures the order of magnitude at stake. The cost is not only financial. Erroneous data erodes the trust that consumers place in a dataset, and once a dashboard has been visibly wrong, downstream teams begin to second-guess every figure it produces, which slows decisions and encourages the growth of parallel, unofficial data copies. Enforcement at the boundary is a way to protect that trust as much as it is a way to avoid direct cost. Adoption of data contracts is growing as teams formalize the producer-consumer relationship, although a precise adoption figure is not available from a reliable primary survey. The direction of travel is clear from the standards themselves: the consolidation of the Data Contract Specification into ODCS v3.1.0 under the Linux Foundation signals a maturing field converging on shared conventions.

Frequently Asked Questions

What is the difference between a data contract and a database schema?

A database schema declares field names, types, and nullability—the structural shape of a table. A data contract is broader: it wraps the schema together with semantics (what each field means, including units and allowed value sets), a service-level agreement (freshness and availability guarantees), and testable quality expectations, and it is versioned as an explicit agreement between the producing team and its consumers. In short, a schema is one component of a contract, and the contract adds meaning, guarantees, and ownership on top of structure.

Do I need both a schema registry and a tool like Great Expectations?

They cover different concerns and are often used together. A schema registry, such as Confluent Schema Registry, enforces the shape and compatible evolution of messages on a stream, rejecting a producer whose schema is incompatible with the registered version. It does not check semantic quality such as null values in nullable fields, out-of-range numbers, or freshness. A tool like Great Expectations or Soda Core enforces those quality expectations against tables or DataFrames. A streaming platform that also cares about value-level quality therefore benefits from both.

Which data quality dimensions are hardest to enforce automatically?

Accuracy and consistency are the hardest. Accuracy asks whether a value correctly describes the real-world entity it represents, which requires comparison against a trusted system of record rather than a self-contained rule; a value can be well-formed, unique, and within the allowed set yet still be wrong. Consistency asks whether values agree across systems and over time, which requires cross-system reconciliation. Completeness, uniqueness, validity, and timeliness are comparatively straightforward to express as mechanical checks.

Should a team standardize on ODCS or the Data Contract Specification in 2026?

A team beginning in 2026 should standardize on the Open Data Contract Standard (ODCS), currently at version 3.1.0 under the Linux Foundation’s Bitol project. The alternative Data Contract Specification is being deprecated and is converging on ODCS, with its tooling support in the Data Contract CLI and Entropy Data planned only until the end of 2026 (datacontract-specification.com, as of 2026). Choosing the surviving standard avoids a forced migration later.

References

• Open Data Contract Standard (ODCS) v3.1.0 — Bitol / Linux Foundation AI & Data (as of 2026)
• Great Expectations (GX Core) changelog (as of 2026-06-26)
• Soda Core release notes (as of 2026-06-29)
• Confluent Schema Registry — Schema Evolution and Compatibility (as of 2026)
• dbt data tests and unit tests documentation (as of 2026-07-01)
• Pandera documentation (as of mid-2026)

Conclusion

Data reliability is achieved not by a single tool but by a discipline that combines an explicit artifact with layered enforcement. The data contract turns the implicit producer-consumer agreement into versioned, machine-readable code, and the Open Data Contract Standard v3.1.0 provides a vendor-neutral way to express it; the consolidation of the older Data Contract Specification into ODCS marks a field settling on shared conventions. Quality itself becomes testable once it is decomposed into completeness, uniqueness, validity, accuracy, consistency, and timeliness, with the understanding that accuracy and consistency resist mechanical checks and require external reference points.

The contract is then enforced wherever data changes form: Pydantic and Pandera at the record and DataFrame boundary, dbt tests and Great Expectations and Soda Core in the warehouse, and a schema registry on the stream. The patterns that put enforcement into force—shift-left validation at the producer, continuous-integration gates on contract changes, pre-ingestion quarantine, circuit breakers, and dead-letter queues—each trade some availability for correctness in the way best suited to their layer. Finally, contracts and observability are complementary rather than interchangeable: contracts prevent the failures they can express at the boundary, while observability detects the anomalies they cannot anticipate. A platform that runs both, over a contract treated as a first-class artifact, is one whose data consumers can trust.