Schema Evolution and the Schema Registry: Avro, Protobuf, and Compatibility

In an event-driven system built on Apache Kafka, every message that crosses the network is a sequence of raw bytes with no built-in description of its structure. A consumer that reads those bytes must already know how to interpret them: which fields are present, in what order, and with what types. When the producer and consumer are developed and deployed independently, and when the shape of the data changes over months of feature work, this implicit agreement becomes fragile. A schema registry addresses that fragility by turning the implicit agreement into an explicit, versioned, and centrally governed contract. This article examines how a schema registry works, how the Avro and Protocol Buffers serialization formats encode and evolve data, and how compatibility modes let a schema change without breaking the producers and consumers that already depend on it.

The discussion assumes familiarity with Kafka as a distributed log of immutable records. The registry pattern described here follows the Confluent Schema Registry, the most widely deployed implementation, but the underlying concepts, particularly Avro schema resolution and Protobuf field-number stability, are properties of the serialization formats themselves and apply wherever those formats are used.

Why Schemas Matter in Event-Driven Pipelines

A schema is a formal description of the structure of a record: the names of its fields, their data types, and rules such as whether a field may be absent. In a monolithic application, this description lives in the type system of a single codebase, and the compiler enforces it. In a distributed streaming pipeline, the producer and the consumer are separate programs, often written in different languages and released on different schedules, connected only by a stream of bytes. Nothing in the bytes themselves states what they mean.

This is the defining property of a Kafka message: it is not self-describing. A message is a key and a value, each an opaque byte array as far as the broker is concerned. The broker never parses the payload. Interpretation is entirely the responsibility of the client that deserializes it. If a producer begins writing an extra field, or renames one, or changes an integer to a string, a consumer that was compiled against the older structure has no way to detect the change from the bytes alone. It will either read garbage or fail.

The contrast with a columnar file format such as Apache Parquet is instructive. A Parquet file stores its schema in its own footer, so any reader can open the file and discover its structure without external coordination; the file is self-describing at rest. A detailed treatment of that design appears in the discussion of Parquet and Apache Arrow columnar storage internals. Kafka messages have no such footer. Attaching a full schema to every message would be prohibitively wasteful, because a stream may carry millions of records that share one structure. The registry resolves this tension by storing each schema once, assigning it a compact numeric identifier, and letting every message carry only that identifier.

This mechanism is the technical enforcement layer beneath a broader organizational concern known as a data contract, a documented and testable agreement about the meaning and quality of a dataset shared across teams. The relationship is direct: a data contract states what the data should look like, and the schema registry is one place where that statement is enforced automatically on every message. The organizational and policy dimension is treated separately in the discussion of data contracts and data quality enforcement; the present article concentrates on the wire-level mechanics. The registry also sits within a larger architectural decision about how data moves, examined in the comparison of streaming versus batch processing architectures.

Serialization Formats: Avro, Protobuf, and JSON Schema

Serialization is the process of converting an in-memory object into a byte sequence suitable for transmission or storage; deserialization is the reverse. The Confluent Schema Registry supports three serialization formats out of the box, each paired with its own serializer and deserializer: Apache Avro through KafkaAvroSerializer and KafkaAvroDeserializer, Protocol Buffers through KafkaProtobufSerializer and KafkaProtobufDeserializer, and JSON Schema through KafkaJsonSchemaSerializer and KafkaJsonSchemaDeserializer (Confluent SerDes documentation, as of 2026-07-17). The choice among them affects encoding size, tooling, and the exact rules that govern how a schema may change.

Apache Avro

Apache Avro is a data serialization system in which the schema is expressed as JSON and the data is encoded in a compact binary form that contains no field names or type tags. Because the encoding omits field identifiers, a decoder cannot interpret the bytes without a schema. Avro’s central design idea is that the schema used to write the data (the writer’s schema) and the schema used to read it (the reader’s schema) may differ, and a well-defined resolution procedure reconciles the two. The latest stable release is Avro 1.12.1, published on 2025-10-16 with four security fixes, following 1.12.0 from 2024-08-05 (Apache Avro release announcement, as of 2026-07-17).

A minimal Avro schema for an order event is a JSON object of type record:

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "fields": [
    { "name": "order_id",   "type": "string" },
    { "name": "customer_id","type": "string" },
    { "name": "amount",     "type": "double" },
    { "name": "currency",   "type": "string", "default": "USD" }
  ]
}

The default on the currency field is not merely a convenience. As the compatibility section will show, a default value is the single most important element that determines whether a field can be added or removed without breaking readers. An optional field in Avro is expressed as a union with null together with a default, for example "type": ["null", "string"], "default": null. This is a distinct mechanism from optionality in Protobuf and the two should not be conflated.

Protocol Buffers

Protocol Buffers, commonly abbreviated Protobuf, is a serialization format developed by Google in which the structure is declared in a .proto file and each field is assigned a numeric tag known as a field number. In the binary encoding, the field number, not the field name, identifies each value. The current major syntax is proto3; an emerging direction called editions is intended to supersede the proto2 and proto3 distinction, but there is no single version number that meaningfully anchors the format. The same order event expressed in proto3 is:

syntax = "proto3";
package com.example.orders;

message OrderPlaced {
  string order_id    = 1;
  string customer_id = 2;
  double amount      = 3;
  string currency    = 4;
}

The numbers 1 through 4 are the field numbers. They are the anchor of Protobuf compatibility. Once a field number is assigned and data has been written with it, that number must never be changed and must never be reused for a different field, a rule examined in detail in the compatibility section (protobuf.dev best practices, as of 2026-07-17).

JSON Schema and a comparison

JSON Schema describes the structure of JSON documents. Its principal advantage is human readability and native compatibility with systems that already exchange JSON. Its principal cost is size and speed: JSON is a text encoding that repeats every field name in every message and requires parsing text rather than reading a compact binary layout. The three formats occupy different points on a tradeoff surface.

No format is uniformly superior. Avro is common in data-lake and analytics pipelines where its compact encoding and rich resolution rules fit batch and streaming loads; systems that emit change events, such as those built with Debezium, frequently produce Avro records backed by a registry, as described in the treatment of change data capture with Debezium and Kafka. Protobuf is common where the same message types are also used in remote procedure calls and where code generation across many languages is a priority. JSON Schema suits teams that value readability and already operate on JSON. The registry supports all three uniformly.

Schema Registry Architecture and the Wire Format

The Confluent Schema Registry is a service that stores schemas and assigns each a globally unique numeric identifier. It organizes schemas under subjects. A subject is a named scope under which a sequence of schema versions accumulates; each registration under a subject produces a new version number, and each distinct schema receives a global schema ID that is unique across the whole registry. Compatibility checks are performed within a subject. Producers and consumers interact with the registry over a REST API and cache results locally to avoid a lookup on every message.

Subject naming strategies

The name of the subject under which a schema is registered is determined by a subject naming strategy. Three strategies are provided (Confluent SerDes documentation, as of 2026-07-17):

• TopicNameStrategy (the default): the subject is <topic>-key or <topic>-value. This requires every message in a topic to conform to a single schema, which is the most common arrangement.
• RecordNameStrategy: the subject is the fully qualified record name. This permits several record types to share one topic, and compatibility for that record name is scoped across every topic that carries it.
• TopicRecordNameStrategy: the subject is <topic>-<fully qualified record name>. This also permits multiple record types per topic, but scopes compatibility per record type within each topic.

The 5-byte wire format

When a serializer writes a message, it does not embed the schema. It embeds a reference to the schema in a fixed header prepended to the payload. The Confluent wire format begins each serialized value with a 5-byte header: byte 0 is a magic byte with the value 0x00, and bytes 1 through 4 hold the schema ID as a 4-byte signed 32-bit integer in big-endian order, also called network byte order. The serialized payload follows from byte 5 onward (Confluent SerDes documentation; wire-format community references, as of 2026-07-17). A consumer validates the framing by confirming that the message is longer than 5 bytes and that byte 0 equals 0x00.

The wire format is deliberately stable. Confluent states that it will not change without significant warning across multiple major releases, and that within the version identified by the magic byte it never changes in a way that would break existing readers (Confluent SerDes documentation, as of 2026-07-17). The Protobuf variant is the one exception to the simple layout: because a single .proto file may declare several message types, a Protobuf message inserts a message-index array, a length-prefixed list of variable-length integer indexes that identifies which message type was serialized, between the schema ID and the payload. Avro and JSON Schema have no such segment.

The producer and consumer flow

The registry interaction is straightforward once the framing is understood. On the producing side, the serializer extracts the schema from the record, registers it under the appropriate subject (or looks up its ID if already registered), prepends the 5-byte header, and sends the message. On the consuming side, the deserializer reads the header, extracts the schema ID, fetches the corresponding schema from the registry, and uses it, together with the consumer’s own reader schema, to decode the payload. Both sides cache aggressively, so the registry is consulted at most once per distinct schema rather than once per message.

A Python producer using the confluent-kafka library expresses this flow directly. The AvroSerializer holds a reference to the registry client, and the SerializingProducer applies it to every value before sending:

from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

schema_str = open("order_placed.avsc").read()

registry = SchemaRegistryClient({"url": "http://schema-registry:8081"})
avro_serializer = AvroSerializer(registry, schema_str)

producer = SerializingProducer({
    "bootstrap.servers": "broker:9092",
    "value.serializer": avro_serializer,
})

order = {
    "order_id": "ORD-91",
    "customer_id": "CUST-4471",
    "amount": 42.50,
    "currency": "USD",
}
producer.produce(topic="orders", value=order)
producer.flush()

The consumer side mirrors this arrangement with a DeserializingConsumer and an AvroDeserializer. The deserialization step is where a registry-backed stream connects to ordinary consumer code; the surrounding consumer plumbing, including offset management and the poll loop, is treated in the walkthrough of an Apache Kafka consumer implementation in Python. The registry itself can be queried directly over its REST API, which is useful in scripts and continuous-integration checks. Retrieving the current compatibility mode for a subject, for example, is a single request:

# Read the compatibility mode configured for a subject
curl -s http://schema-registry:8081/config/orders-value

# Set the subject to FULL compatibility
curl -s -X PUT http://schema-registry:8081/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL"}'

# List the versions registered under a subject
curl -s http://schema-registry:8081/subjects/orders-value/versions

Compatibility Modes and the Rules of Safe Evolution

Compatibility is the property that a schema change does not break the programs that already read or write data under the older schema. The registry enforces this by rejecting a schema registration that would violate the configured compatibility mode. This point deserves emphasis: the check happens at registration time, when a producer or a deployment pipeline attempts to register a new schema, not when an individual message is decoded. A rejected registration is the safeguard that prevents an incompatible change from ever being written to the log.

Two directions of compatibility are defined. Backward compatibility means new code can read data written by old code; a new reader accepts old data. Forward compatibility means old code can read data written by new code; an old reader accepts new data. Full compatibility requires both directions. Each of these has a transitive variant. A non-transitive mode checks a candidate schema only against the latest registered version, whereas a transitive mode checks it against every previously registered version (Confluent schema-evolution documentation, as of 2026-07-17).

Source: Confluent “Schema Evolution and Compatibility” documentation, as of 2026-07-17. A useful mnemonic follows directly from the definitions: BACKWARD means a new reader reads old data, so consumers are upgraded first; FORWARD means an old reader reads new data, so producers are upgraded first; FULL supports both directions, so the two sides may be upgraded independently.

Avro schema resolution

Avro’s compatibility behavior follows directly from its schema resolution rules, the procedure by which a reader reconciles its own schema with the writer’s schema field by field. The specification defines the outcomes precisely (Apache Avro Specification, Schema Resolution, as of 2026-07-17):

• If the writer’s record contains a field that the reader’s schema does not declare, the writer’s value for that field is ignored. This is what makes it safe for a producer to add a field that older consumers have not yet learned about.
• If the reader’s schema declares a field with a default value and the writer’s schema lacks that field, the reader supplies the default. This is what makes it safe to add a field to the reader.
• If the reader’s schema declares a field with no default and the writer’s schema lacks it, resolution fails with an error. This is the case a compatibility check is designed to prevent.
• Aliases allow a named type or field to declare alternative names, so a reader can match a writer’s old field or record name to a renamed one, which enables compatible renaming.
• Resolution recurses into arrays, maps, and unions, applying the same rules to item, value, and branch schemas.

Protobuf field-number evolution

Protobuf takes a different path to the same goal. Because the binary encoding identifies each value by its field number rather than its name, a field can be renamed freely without affecting the wire format, and a parser that encounters a field number it does not recognize simply skips it. This skipping behavior is the mechanism that makes additive evolution safe in proto3: adding a new field is compatible because older parsers ignore the unknown number, and reading data written by newer code succeeds because the extra fields are passed over (protobuf.dev, as of 2026-07-17).

The corresponding hazard is field-number reuse. Once a field number has been used, it must never be assigned to a different field, even after the original field is deleted. If a number is reused with a different type, a decoder holding old data will interpret the new field’s bytes according to the old field’s type and silently produce corrupt values. The remedy defined by the format is the reserved declaration, which permanently blocks a field number and name from being reused:

message OrderPlaced {
  reserved 3;                 // amount was removed; its number is retired
  reserved "amount";          // the old name is retired as well

  string order_id    = 1;
  string customer_id = 2;
  string currency    = 4;
  int64  amount_cents = 5;    // new field, new number, never reuse 3
}

Upgrade order in practice

The compatibility mode dictates not only which changes are legal but also the sequence in which the two sides of a stream must be deployed. Under BACKWARD, the new schema is designed so that new consumers can read data still being produced under the old schema; consumers are therefore upgraded first, after which producers can move to the new schema safely. Under FORWARD, the arrangement reverses: producers move first because the guarantee is that old consumers can still read the newly produced data. Under FULL, both guarantees hold, so the two sides can be rolled out in any order. Kafka Streams applications, which both consume and produce, are supported only under BACKWARD and the modes that include it, namely FULL and FULL_TRANSITIVE (Confluent schema-evolution documentation, as of 2026-07-17).

Operational Practices for Schema Evolution

Choosing and operating a compatibility policy is as much an organizational decision as a technical one. The following practices reflect the way schema registries are commonly run in production.

Choosing a compatibility mode

The right mode depends on which side of the stream tends to deploy first and how tightly the two sides are coordinated. When consumers are numerous and slow to upgrade, as with a widely subscribed event, BACKWARD or BACKWARD_TRANSITIVE lets producers evolve without waiting for every consumer. When a single producer feeds many independent consumers that cannot be upgraded in lockstep, FORWARD protects those consumers from a producer change. FULL and FULL_TRANSITIVE impose the strictest discipline and are appropriate for long-lived, high-value topics where either side may change at any time. The transitive variants trade a small amount of flexibility for a strong guarantee: because they validate against the entire version history, they prevent a slow drift that a sequence of latest-only checks would permit.

Testing compatibility in continuous integration

Because the registry enforces compatibility at registration time, a broken schema change is most cheaply caught before deployment. The registry exposes a compatibility-check endpoint that reports whether a candidate schema would be accepted under the subject’s configured mode, without registering it. Wiring this call into a continuous-integration pipeline turns a potential production incident into a failed build:

# Check a candidate schema against the latest version of a subject.
# Returns {"is_compatible": true} or false without registering.
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/orders-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d @candidate_schema_request.json

Treating schema definitions as versioned source code, reviewed and tested like any other artifact, is the connective tissue between the registry and a broader data-contract discipline. The same schema that governs the wire format can drive downstream typing in transformation frameworks, so that a well-typed event flows into modeled tables; the transformation layer is discussed in the overview of building transformation pipelines with dbt.

Handling genuinely breaking changes

Some changes cannot be made compatibly. Changing a field’s type in an incompatible way, or removing a required field that consumers depend on, will and should be rejected under any checked mode. Two disciplined options exist. The first is to publish the incompatible data to a new topic and migrate consumers deliberately, retiring the old topic once no consumer remains. The second, to be used sparingly, is to set the subject to NONE for a controlled window, coordinate the producer and consumer deployment manually, and restore a checked mode afterward. Setting NONE permanently removes the very protection the registry exists to provide and is not a substitute for a migration plan.

Platform context

The registry evolves alongside the broader Kafka platform. Confluent Platform 8.0 reached general availability in June 2025 and bundles Apache Kafka 4.0, which removes the dependency on ZooKeeper in favor of the KRaft consensus protocol; the same release brought general availability of client-side field-level encryption and of passwordless authentication for the Schema Registry (Confluent Platform 8.0 announcement, as of 2026-07-17). These platform-level changes do not alter the wire format or the compatibility semantics described here, which remain stable by design, but they do affect how a registry is secured and operated.

Frequently Asked Questions

Is the default compatibility mode BACKWARD or FULL?

The default compatibility mode in the Confluent Schema Registry is BACKWARD. Under BACKWARD, a candidate schema is validated only against the latest registered version, and it permits deleting fields and adding fields that carry a default value. It does not provide the two-directional guarantee of FULL, nor the whole-history guarantee of the transitive variants. Teams that need protection against drift across the entire version history should explicitly configure a transitive mode.

When are compatibility checks actually performed?

Compatibility is enforced at schema-registration time, not when an individual message is decoded. When a producer or a deployment pipeline attempts to register a new schema under a subject, the registry compares it against the versions required by the configured mode and rejects the registration if it would break compatibility. A rejected registration prevents the incompatible schema, and therefore any message written with it, from entering the log.

How does the wire format differ between Avro and Protobuf?

Both use a 5-byte header consisting of a magic byte set to 0x00 followed by a 4-byte big-endian schema ID. Protobuf inserts an additional segment, a length-prefixed message-index array of variable-length integers, between the schema ID and the payload, because a single .proto file may define several message types and the deserializer must know which one was serialized. Avro and JSON Schema do not include this segment.

Should producers or consumers be upgraded first?

The order follows the compatibility mode. Under BACKWARD, consumers are upgraded first, because the guarantee is that new consumers can read data still produced under the old schema. Under FORWARD, producers are upgraded first, because old consumers must be able to read newly produced data. Under FULL, both guarantees hold and the two sides may be upgraded in either order.

Why must a Protobuf field number never be reused?

In the Protobuf binary encoding, each value is identified by its field number rather than its name. A decoder holding data written with an older schema will interpret bytes according to whatever field the number denoted at the time. If the number is later reassigned to a different field, especially with a different type, the old data is misread and silently corrupted. Marking a removed number as reserved permanently blocks its reuse and prevents this class of error.

References

• Apache Avro Specification — Schema Resolution
• Confluent — Schema Evolution and Compatibility
• Confluent — Formats, Serializers, and Deserializers (wire format, subject naming)
• Protocol Buffers — Best Practices (field numbers, reserved)
• Apache Avro 1.12.1 Release Announcement

Conclusion

The schema registry solves a problem that is easy to overlook until it causes an outage: a Kafka message carries data but not the description needed to interpret it. By storing each schema once, assigning it a compact identifier, and enforcing compatibility rules at registration time, the registry turns an implicit and fragile agreement between independently deployed producers and consumers into an explicit, versioned contract. The 5-byte wire format is the small piece of framing that makes this work on every message, with Protobuf’s message-index array the one notable variation to keep straight.

The two dominant binary formats reach compatibility by different means that are worth internalizing separately. Avro reconciles a reader schema against a writer schema, with field defaults and aliases as the levers that make additive and rename changes safe. Protobuf anchors compatibility on stable field numbers, with the reserved declaration as the guard against the corrupting error of number reuse. Above both sits the compatibility-mode matrix, whose central practical consequence is the upgrade order: BACKWARD upgrades consumers first, FORWARD upgrades producers first, and FULL frees the two sides to move independently. Applied with a default of BACKWARD understood correctly, compatibility checks wired into continuous integration, and a deliberate plan for the changes that cannot be made compatibly, the registry becomes the mechanism that lets a streaming data model evolve for years without breaking the systems that depend on it.