Does conformal prediction require the model to be accurate?

No. The coverage guarantee holds regardless of model quality, because it derives from a symmetry argument over the calibration scores rather than from any assumption about accuracy. Model quality affects the size of the prediction sets or the width of the intervals, not their validity: a weak model produces valid but large, uninformative sets, while a strong model produces valid and small ones.

How is conformal prediction different from probability calibration?

Probability calibration adjusts a model's scalar confidence scores so that, for example, predictions made with 0.8 confidence are correct about 80 percent of the time; it provides no finite-sample guarantee. Conformal prediction instead outputs a set or interval with a proven marginal coverage guarantee. The two are complementary, since a well-calibrated score is often the best base score for conformal classification, but only conformal prediction supplies the formal coverage statement.

What does the 90 percent coverage guarantee actually promise?

It promises marginal coverage: averaged over the randomness in the calibration and test data, the true outcome falls inside the prediction set at least 90 percent of the time, with a matching upper bound of 1 minus alpha plus 1/(n+1). It does not promise conditional coverage, meaning 90 percent protection for every individual input or subpopulation, which is impossible to guarantee distribution-free. Coverage can be uneven across easy and hard inputs even when the overall rate is met.

What breaks the coverage guarantee in production?

The guarantee rests on exchangeability between the calibration and test data. Distribution shift, whether covariate shift or label shift, breaks exchangeability, so a deployed system's true coverage can fall below its target as the data drifts. Coverage should therefore be monitored operationally. Weighted conformal prediction can restore validity under covariate shift when the density ratio between test and training inputs can be estimated.

What is the difference between data drift and concept drift?

Data drift is a change in the distribution of the input features, P(X), while the input-to-target relationship stays fixed. Concept drift is a change in that relationship itself, P(y|X), so the same input maps to a different expected output. Data drift can be detected from inputs alone; concept drift generally requires labels or a proxy for model error.

Does data drift always mean the model has become less accurate?

No. Data drift is a warning, not a verdict. A feature can drift into a region the model already handles well, leaving accuracy unaffected, while a small shift in an influential feature near a decision boundary can cause a larger performance drop. Drift alarms should be weighted by feature importance and confirmed against a performance signal.

How is the Population Stability Index interpreted?

By convention, a PSI below 0.1 indicates a stable distribution, a PSI between 0.1 and 0.25 indicates a moderate shift that should be monitored, and a PSI above 0.25 indicates a significant shift that warrants investigation and often retraining. These are widely used heuristics, and the value depends on the binning scheme.

Which drift detectors require labels and which do not?

Distribution-based tests such as PSI, the Kolmogorov-Smirnov test, and chi-square analyse input features only and need no labels. Error-rate detectors such as DDM, Page-Hinkley, and ADWIN monitor the model's error stream and therefore require ground-truth labels. A common strategy runs label-free input tests continuously and applies label-hungry detectors when ground truth is available.

How can model performance be estimated when labels are delayed?

Performance estimation infers likely metrics from the model's outputs. Confidence-Based Performance Estimation (CBPE) uses calibrated predicted probabilities to estimate the confusion matrix and classification metrics, while Direct Loss Estimation (DLE) trains a secondary model to predict per-observation loss for regression. Both should be reconciled with realised metrics once labels arrive.

How often should a drifting model be retrained?

There is no fixed schedule. Retraining should be triggered by evidence, such as a confirmed performance drop or a significant importance-weighted input shift, rather than by the calendar. Recurring seasonal drift is often better handled by adding features that encode the cycle, and a genuinely new regime may call for domain adaptation.

kongastral - AI Code Invest

Library	Ecosystem	Notes
MAPIE	scikit-learn	A fit/predict wrapper for conformal intervals, classification sets (including APS/RAPS-style methods), and time series; scikit-learn compatible.
TorchCP	PyTorch	PyTorch-native, integrating conformal prediction with deep classifiers, regressors, and online prediction, with GPU-accelerated batch processing.
crepes	NumPy / general	A lightweight library for conformal classifiers, regressors, and predictive systems; CPU-oriented, without GPU or batch acceleration.

Bin	Expected %	Actual %	Actual − Expected	ln(Actual / Expected)	Contribution
1	0.10	0.05	−0.05	−0.6931	0.03466
2	0.20	0.15	−0.05	−0.2877	0.01438
3	0.40	0.35	−0.05	−0.1335	0.00668
4	0.20	0.25	+0.05	+0.2231	0.01116
5	0.10	0.20	+0.10	+0.6931	0.06931
Total PSI	0.13619

Speculative Decoding for LLM Inference: How Draft-and-Verify Accelerates Token Generation

Summary. Speculative decoding accelerates large language model inference by letting a small “draft” model propose several tokens that the large “target” model then verifies in a single parallel forward pass. A modified rejection-sampling rule accepts a prefix of the proposed tokens and preserves the target model’s output distribution within hardware numerics, so the method is lossless rather than an approximation. In the settings reported by the founding papers, the technique yields roughly a two-to-threefold reduction in generation latency. Its benefit is largest in the low-batch, latency-bound regime and shrinks when a server is already saturated with a large batch, which is the central trade-off a practitioner must weigh.

Standard autoregressive text generation produces one token per forward pass through the network, and a language model must repeat that pass once for every token it emits. Speculative decoding replaces this strictly sequential loop with a draft-and-verify scheme: a small, fast model proposes a short run of tokens at once, and the large model checks all of them in a single pass. The two approaches produce the same text, but they spend the expensive model’s time very differently. Understanding why the second approach can be several times faster, and why it is not free, requires looking at what actually limits the speed of a forward pass.

The technique was introduced independently in two 2023 papers and has since become a standard component of production inference stacks. This guide explains the mechanism, the mathematics that make it lossless, the speedups that have been measured, the main variants that avoid a separate draft model, and the regime in which the method pays off.

Why autoregressive decoding is slow

A transformer decoder generates text one token at a time. Each new token is conditioned on all previous tokens, so the model runs a full forward pass, produces a probability distribution over the vocabulary, samples a token, appends it, and repeats. For a response of several hundred tokens, the large model is invoked several hundred times in strict sequence. This dependency is the reason latency scales with output length.

The important detail is that a single-token forward pass does not use the accelerator’s arithmetic units efficiently. Generating one token requires reading the model’s weights from memory but performs relatively little computation per byte read, so the pass is memory-bandwidth-bound: its wall-clock time is dominated by moving weights, not by multiplying numbers. A modern accelerator therefore has spare arithmetic capacity during each decode step. Chen and colleagues at DeepMind framed the consequence precisely: scoring a short continuation of several tokens in parallel has latency comparable to sampling a single token, because the extra tokens ride along in the same memory-bound pass (Chen et al., 2023). Speculative decoding is the technique that turns that spare capacity into useful output.

Key Takeaway: A decode step is limited by memory bandwidth, not arithmetic. Verifying several proposed tokens in one batched pass costs roughly the same as generating one token normally, which is the opening that speculative decoding exploits.

The draft-and-verify loop

Speculative decoding pairs two models. The target model is the large, accurate network whose output is desired. The draft model is a much smaller network that approximates the target and can run many times faster. A single iteration proceeds in three steps.

First, the draft model generates a short candidate continuation autoregressively, proposing a fixed number of tokens, conventionally written as the draft length. Because the draft is small, these steps are cheap. Second, the target model processes the original context together with all proposed tokens in one parallel forward pass, yielding its own probability for each position. Third, a rejection rule compares the two models position by position and accepts the longest prefix of proposed tokens that is consistent with the target distribution. When a proposed token is rejected, it is replaced by a token resampled from an adjusted distribution, and the iteration ends. When every proposed token is accepted, the target contributes one additional “bonus” token for free from the same pass.

The number of tokens produced per iteration is therefore variable: it ranges from one, when the very first proposal is rejected, up to the draft length plus one, when all proposals are accepted. The average over many iterations determines the speedup. Each iteration costs one target pass plus the cheap draft steps, and it advances the sequence by more than one token whenever the draft agrees with the target, which is where the acceleration comes from.

The correctness guarantee

The property that distinguishes speculative decoding from lossy speed tricks is that its output is statistically identical to sampling from the target model alone. This is achieved by a modified rejection-sampling rule rather than by trusting the draft. Let the target model assign probability p(x) to a token and the draft model assign it probability q(x). A proposed token drawn from the draft is accepted with probability min(1, p(x)/q(x)) (Leviathan et al., 2023). In words, if the target likes the token at least as much as the draft did, it is always kept; if the target likes it less, it is kept with a probability equal to the ratio of the two.

When a token is rejected, the algorithm does not simply stop with the draft’s choice. It resamples from an adjusted distribution defined as the normalized positive difference between the two, p′(x) = norm(max(0, p(x) − q(x))) (Leviathan et al., 2023). This correction exactly compensates for the cases the draft over-sampled, so that the overall probability of emitting any given token equals the target’s probability. The result is that speculative decoding preserves the target distribution within hardware numerics; it is not an approximation that trades quality for speed, and it needs no fine-tuning or architectural change to the target model.

Caution: “Lossless” here means the output distribution is preserved within hardware numerics, not that a run is bit-for-bit identical to standard decoding. Floating-point order of operations still differs between a single-token pass and a batched verification pass.

Because the correctness rule depends only on the two probability values at each position, the guarantee holds regardless of how good or bad the draft model is. A poor draft does not corrupt the output; it merely gets rejected more often and delivers less speedup. This separation between correctness and performance is what makes the method safe to deploy: the worst case is slower generation, never wrong generation. Readers interested in how models behave at the level of output probabilities may find the discussion in classifier calibration and reliability diagrams a useful companion, since both topics turn on treating a model’s probabilities as first-class objects.

How much speedup to expect

The performance of speculative decoding is governed by the acceptance rate, written α, which is the expected probability that a proposed token is accepted. A higher acceptance rate means the draft and target agree more often, so more tokens survive each verification pass. The acceptance rate can be written as α = E(min(p, q)), an expectation that increases as the draft distribution moves closer to the target distribution (Leviathan et al., 2023).

Given an acceptance rate and a draft length γ, the expected number of tokens produced per iteration follows a capped geometric expression:

E[tokens per iteration] = (1 - alpha^(gamma + 1)) / (1 - alpha)

This formula counts the accepted prefix plus the one bonus token the target contributes. As an illustration, with an acceptance rate of 0.7 and a draft length of 4, the expected yield is (1 − 0.7⁵)/(1 − 0.7) ≈ 2.8 tokens per iteration, meaning the sequence advances by nearly three tokens for each expensive target pass. Raising the acceptance rate to 0.9 lifts the expectation to roughly 4.1 tokens per iteration. These are expected token counts, not wall-clock speedups: the realized speedup also depends on the ratio between the draft model’s cost and the target model’s cost, because each iteration must pay for the draft steps as well.

Measured end-to-end results are consistent with this picture. On the T5-XXL model, the original method reaches a 2×–3× wall-clock speedup over the standard implementation while producing identical samples (Leviathan et al., 2023). In a separate study, a 2×–2.5× speedup was measured when sampling from Chinchilla, a 70-billion-parameter model, in a distributed setup, again without degrading sample quality (Chen et al., 2023). A useful rule of thumb, stated cautiously, is that speculative decoding delivers roughly a twofold to threefold latency reduction in the latency-bound settings these papers examined, with the exact figure depending on the draft-target pair and the workload.

The variant landscape

The classic formulation needs a separate draft model whose tokenizer and vocabulary match the target. Finding or training a good draft is the main practical friction, and a second model consumes additional memory. A family of variants addresses this by generating draft tokens without a distinct second network, trading setup complexity for either extra trained components or a narrower range of gains.

Medusa, an approach that attaches extra decoding heads to the target model and verifies their proposals with tree-structured attention, records a 2.2×–3.6× speedup across a range of models and needs no separate draft network (Cai et al., 2024). EAGLE instead performs the draft step at the feature level, autoregressing over the target’s second-to-top-layer representations; on the MT-bench evaluation it runs roughly 3× faster than vanilla decoding while preserving the output distribution (Li et al., 2024). Its successor, EAGLE-2, extends this with dynamically constructed draft trees and records speedup ratios of 3.05×–4.26×, about 20%–40% above the first version, and remains lossless (Li et al., 2024). Lookahead decoding, an exact method motivated by Jacobi iteration that requires no draft model or datastore, reaches up to 1.8× on MT-bench and up to roughly 4× with strong scaling across multiple GPUs on code-completion workloads (Fu et al., 2024). A further line of work draws candidate continuations directly from the prompt or prior output through an n-gram lookup table rather than any neural drafter; it adds no model weights and is most effective on input-grounded, repetitive tasks such as summarization or code editing, where the output often echoes the input.

Approach	How drafts are made	Extra components	Reported speedup (source setting)
Separate draft model	Small autoregressive model	A second model in memory	2×–3× on T5-XXL (Leviathan et al., 2023)
Medusa	Extra heads on the target	Trained decoding heads	2.2×–3.6× across models (Cai et al., 2024)
EAGLE / EAGLE-2	Feature-level autoregression	A feature-prediction module	3.05×–4.26× on MT-bench (Li et al., 2024)
Lookahead decoding	Jacobi n-gram trajectories	None (exact algorithm)	up to 1.8×–4× (Fu et al., 2024)
Prompt / n-gram lookup	Copy from prompt or history	None (a lookup table)	Task-dependent; best on repetitive output

The variant landscape reflects one underlying trade. Methods on the left of the map require nothing extra but accept fewer tokens per pass; methods on the right invest in trained heads or feature modules to raise the acceptance rate and, with it, the ceiling on speedup. Feature-level and attention-based drafting build on the same transformer machinery covered in the guide to graph attention networks, and the large targets these methods accelerate are the models produced by pretraining regimes such as those described in the overview of self-supervised learning.

When it helps, and when it does not

The single most important practical caveat concerns batch size. Speculative decoding converts spare arithmetic capacity into extra tokens, and that spare capacity exists only when the target’s forward pass is memory-bandwidth-bound. This condition holds in the low-batch, latency-bound regime, where a server processes one request or a few concurrent requests and the accelerator’s compute units are underused. In that regime, verifying several proposed tokens is nearly free, and the technique delivers its full benefit.

At high batch sizes, the situation inverts. When many requests are processed together, the target’s forward pass already keeps the arithmetic units busy, so it becomes compute-bound rather than memory-bound. The extra verification work now competes with real work instead of filling idle capacity, and rejected tokens represent wasted computation that reduces overall throughput. Consequently, speculative decoding is best understood as a latency-optimization technique for interactive, low-concurrency serving rather than a throughput free lunch for saturated batch workloads.

Two further trade-offs shape a deployment. The acceptance rate depends on how well the draft is aligned with the target, so a mismatched draft wastes target compute on rejected proposals; selecting or training a well-aligned draft is the main tuning lever. And the draft length sets a ceiling on tokens per iteration but also raises the wasted work when proposals are rejected, so there is an optimum draft length that depends on the acceptance rate and on the draft-to-target cost ratio.

Tip: Before adopting speculative decoding, characterize the serving regime. For interactive assistants and single-stream generation it is often a clear win; for high-throughput batch pipelines the gain may be small or negative, and the memory spent on a draft model may be better used to enlarge the batch.

Support in production systems

Speculative decoding is available in the major inference stacks, though the exact configuration options and flag names change between releases and should be checked against current documentation rather than memorized. The mechanism, however, is stable across them.

The vLLM serving engine supports several drafters, including zero-overhead n-gram lookup for repetitive workloads and neural methods such as Medusa and the EAGLE family for higher acceptance. NVIDIA’s TensorRT-LLM similarly exposes the draft-model approach alongside EAGLE, Lookahead, Medusa, and related drafters executed within its compiled engine. In the Hugging Face transformers library, the feature is called assisted generation: passing an assistant model to the generation call causes the assistant to draft tokens that the main model verifies in a single pass, and a universal variant relaxes the requirement that the two models share a tokenizer. The C and C++ project llama.cpp enables the same idea by supplying a draft model to its command-line and server tools, provided the draft shares a compatible vocabulary with the main model. Deploying any of these behind a service boundary raises the usual operational questions covered in the guides to containerizing applications for production and to running stateful workloads on Kubernetes pods.

The common thread across these systems is that speculative decoding is applied at inference time as a drop-in accelerator: it changes how tokens are produced, not what the target model is, and its lossless guarantee means an existing model can be served faster without retraining or quality regression. The main engineering decisions are which drafting strategy to use, how to size the draft, and whether the serving regime is latency-bound enough for the technique to pay off.

Conclusion

Speculative decoding accelerates language model inference by exploiting a structural fact: a single-token decode step leaves the accelerator’s arithmetic units idle, and that idle capacity can verify several draft tokens at almost no extra cost. A modified rejection-sampling rule, accepting each proposed token with probability min(1, p/q) and resampling from an adjusted distribution on rejection, makes the acceleration lossless within hardware numerics rather than an approximation. The expected yield per iteration grows with the acceptance rate through the expression (1 − α^γ+1)/(1 − α), and measured latency reductions in the range of two to three times have been reported for the settings the founding papers studied.

The variant landscape, from n-gram lookup through Lookahead, Medusa, and the EAGLE family, trades setup cost against acceptance rate, letting a practitioner pick a point that fits the deployment. The one caveat that determines whether the technique is worthwhile is the serving regime: it is a strong latency optimization for low-batch, interactive inference and a weak one for saturated high-throughput batches. Read against those constraints, speculative decoding is a well-understood and broadly supported way to serve an existing model faster without changing what it produces.

References

Leviathan, Y., Kalman, M., Matias, Y. “Fast Inference from Transformers via Speculative Decoding.” ICML 2023. arxiv.org/abs/2211.17192
Chen, C., et al. (DeepMind). “Accelerating Large Language Model Decoding with Speculative Sampling.” 2023. arxiv.org/abs/2302.01318
Cai, T., et al. “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.” 2024. arxiv.org/abs/2401.10774
Li, Y., et al. “EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees.” 2024. arxiv.org/abs/2406.16858 (EAGLE: arxiv.org/abs/2401.15077)
Fu, Y., et al. “Break the Sequential Dependency of LLM Inference Using Lookahead Decoding.” ICML 2024. arxiv.org/abs/2402.02057

Frequently Asked Questions

Does speculative decoding change the text a model produces?

No. The modified rejection-sampling rule preserves the target model’s output distribution within hardware numerics, so the generated text is statistically identical to standard decoding from the target. The draft model only affects speed, never correctness; a poor draft slows generation but cannot change what is produced.

How large a speedup is realistic?

The founding papers reported roughly two-to-threefold latency reductions in latency-bound settings, such as 2×–3× on T5-XXL and 2×–2.5× on a 70-billion-parameter model. The realized figure depends on the acceptance rate between draft and target, the draft length, and the cost ratio of the two models, so results vary by workload.

What is the acceptance rate and why does it matter?

The acceptance rate is the expected probability that a proposed draft token is kept by the target. It rises as the draft distribution approaches the target distribution. Because the expected tokens per iteration equal (1 − α^(γ+1))/(1 − α), a higher acceptance rate directly increases how many tokens each expensive target pass yields.

Do all variants require a separate draft model?

No. The classic method uses a separate small model, but Medusa adds decoding heads to the target, EAGLE drafts at the feature level, Lookahead decoding uses Jacobi-style n-gram trajectories with no draft model, and prompt or n-gram lookup copies candidates from the input. These variants avoid a second model at the cost of trained components or a narrower range of gains.

Why does the benefit shrink at high batch sizes?

The technique fills spare arithmetic capacity that exists only when the target’s forward pass is memory-bandwidth-bound, which is the case at low batch sizes. At high batch sizes the pass becomes compute-bound, the spare capacity disappears, and the extra verification work competes with real work. Speculative decoding is therefore most useful for interactive, low-concurrency serving.

August 23, 2026

Classifier Calibration: Expected Calibration Error and Reliability Diagrams

A screening classifier reaches 89% accuracy on a held-out test set, and for a batch of cases it labels positive with 57% confidence a reviewer reasonably expects that about 57 of every 100 are genuine. In the run examined below, 73 of every 100 are. The model is accurate, yet its stated probabilities do not match the outcomes they are supposed to describe. This gap between confidence and correctness is miscalibration, and it is a separate property from accuracy: a model can rank cases well while reporting probabilities that are systematically too high or too low.

Calibration asks whether a classifier’s probability outputs mean what they claim: among predictions made with confidence p, the fraction that turn out correct should be close to p. It is measured with reliability diagrams, the expected calibration error (ECE), and proper scoring rules such as the Brier score. A short reproducible experiment on a random-forest classifier shows an ECE of 0.071 that post-hoc scaling reduces to roughly 0.015 while classification accuracy stays essentially unchanged, because calibration adjusts the probabilities without reordering the predictions.

What calibration means

A binary classifier usually outputs a score between 0 and 1 that is treated as the probability of the positive class. The model is calibrated when those scores are honest frequencies: across all cases assigned a confidence near 0.8, close to 80% should belong to the positive class. Accuracy, by contrast, only asks whether the thresholded decision is right. The two come apart often. A model can achieve high accuracy by ranking positives above negatives while attaching probabilities that are uniformly inflated, and a well-ranked model with poor calibration will mislead any downstream step that consumes the probability rather than the label — expected-cost decisions, risk thresholds, or the fusion of several models’ outputs.

Miscalibration has a direction. When the true accuracy in a confidence band is lower than the stated confidence, the model is overconfident; when it is higher, the model is underconfident. Tree ensembles and margin-based classifiers are known to distort probabilities in characteristic ways, which is why the effect is worth measuring rather than assuming.

Measuring miscalibration

The standard visual tool is the reliability diagram. Predictions are sorted into equal-width confidence bins; for each bin the mean predicted confidence is plotted against the empirical accuracy. Perfect calibration lies on the diagonal, where confidence equals accuracy. Points below the diagonal indicate overconfidence, points above it indicate underconfidence, and the vertical distance from the diagonal is the per-bin calibration gap.

Reliability diagram for the uncalibrated model. Between confidence 0.5 and 0.9 the curve sits above the diagonal: accuracy exceeds stated confidence by 12 to 18 points.

A diagram is informative but not a single number. The expected calibration error collapses it to one value: the bin-count-weighted average of the absolute gap between accuracy and confidence across all bins. Lower is better, and zero means the reliability curve lies exactly on the diagonal. ECE depends on the number of bins and on the binning scheme, so it is a comparison tool within a fixed protocol rather than an absolute constant. It also averages away compensating errors, which is why it is best read alongside the diagram and a proper scoring rule.

The Brier score — the mean squared difference between the predicted probability and the 0/1 outcome — is one such proper scoring rule, minimised only when the reported probabilities are both accurate and well calibrated. Log loss (cross-entropy) is another, and it penalises confident mistakes more sharply. Reporting ECE together with Brier score and log loss gives a fuller picture than any one metric alone.

A measured example

To make the numbers concrete, a random-forest classifier of 200 trees (maximum depth 7) was trained on a synthetic dataset of 20,000 examples with 20 features and a 30% positive rate, using a fixed random seed. The data were split into 12,000 training, 4,000 calibration, and 4,000 test examples; the calibration split was reserved for fitting the post-hoc maps described in the next section, and every figure below is computed on the untouched test split. The reliability diagram above is this model’s, and the summary metrics are as follows.

Method	Accuracy	Brier	Log loss	ECE (15 bins)
Uncalibrated	0.8942	0.0843	0.2901	0.0706
Platt (sigmoid)	0.8960	0.0781	0.2640	0.0169
Isotonic	0.8958	0.0781	0.2674	0.0145

The uncalibrated ECE of 0.0706 confirms what the diagram shows. The departure from the diagonal runs in both directions: the two lowest-confidence bins are mildly overconfident, while the mid-to-high range from roughly 0.5 to 0.9 is markedly underconfident, with empirical accuracy exceeding stated confidence by 12 to 18 points. This mixed pattern is exactly the kind of structure a single accuracy figure cannot reveal.

Expected calibration error before and after post-hoc calibration. Both scaling methods bring ECE to roughly a fifth of its original value.

Fixing it after training

Calibration can be repaired after training by fitting a small function that maps the model’s raw scores to corrected probabilities, using data the model did not train on. Two classical approaches appear in the table. Platt scaling fits a one-parameter logistic (sigmoid) transform of the score, which works well when the reliability curve has a smooth S-shape but cannot represent more irregular distortions. Isotonic regression fits any non-decreasing step function, so it is more flexible; the trade-off is that it needs more calibration data and can overfit on small sets. For neural networks a further special case, temperature scaling, divides the logits by a single learned constant before the softmax; because a positive constant does not change which class scores highest, the predicted labels — and therefore accuracy — are left untouched.

Post-hoc calibration fits a monotonic map on a held-out split and applies it to new scores, leaving the ranking — and the labels — intact.

The measured result is the point worth keeping. Both maps cut ECE to roughly a fifth of its original value — 0.0169 for Platt scaling and 0.0145 for isotonic — and both also improve the Brier score and log loss, since better probabilities lower every proper scoring rule. Meanwhile accuracy barely moves, from 0.8942 to about 0.8960: because a monotonic map preserves the ordering of scores, almost no case crosses the 0.5 decision threshold. Calibration is best understood as a correction to the probabilities, not to the classifier’s ability to separate the classes. The same distinction underlies threshold selection in one-class anomaly detection and the score-versus-decision boundary discussed in the SVM and one-class SVM comparison.

Conclusion

Accuracy and calibration answer different questions. Accuracy asks whether the label is right; calibration asks whether the probability is honest. A model that will feed its probabilities into a cost-sensitive decision, a risk threshold, or an ensemble should be checked with a reliability diagram and summarised with ECE and a proper scoring rule, not accuracy alone. When the probabilities drift, a post-hoc map fitted on held-out data usually restores them at negligible cost to accuracy — a cheap and well-understood step that turns a good ranker into a trustworthy probability estimator. For related evaluation pitfalls with skewed classes, see the discussion of thresholds and metrics in time-series anomaly detection models.

Frequently Asked Questions

Does calibrating a model improve its accuracy?

Not in general. Platt scaling, isotonic regression, and temperature scaling are monotonic transforms of the score, so they preserve the ranking of predictions and therefore leave the thresholded labels — and accuracy — essentially unchanged. In the experiment above accuracy moved only from 0.8942 to about 0.8960. What improves is the quality of the probabilities, reflected in lower ECE, Brier score, and log loss.

When should isotonic regression be preferred over Platt scaling?

Isotonic regression fits an arbitrary non-decreasing function and can correct irregular reliability curves that a single sigmoid cannot, so it tends to win when there is enough held-out calibration data. On small calibration sets its flexibility becomes a liability and it can overfit, in which case the one-parameter Platt sigmoid is the safer choice. Comparing both on a validation split, as in the table above, is the reliable way to decide.

References

Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. “On Calibration of Modern Neural Networks.” ICML 2017. arxiv.org/abs/1706.04599 (accessed 2026-07-28).
scikit-learn developers. “Probability calibration” (User Guide, version 1.9). scikit-learn.org/stable/modules/calibration.html (accessed 2026-07-28).
Niculescu-Mizil, A., Caruana, R. “Predicting Good Probabilities With Supervised Learning.” ICML 2005. cs.cornell.edu (accessed 2026-07-28).
Brier, G. W. “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review, 78(1), 1950. journals.ametsoc.org (accessed 2026-07-28).

Related Reading:

July 29, 2026

Parquet Compression Codecs Benchmarked: zstd, snappy, gzip, and lz4 on 10 Million Rows

Summary

In brief: The same 10,000,000-row table was written to Apache Parquet six ways and measured for on-disk size, write time, and read time, followed by separate studies of row-group size and dictionary encoding. Every figure below comes from one recorded benchmark run; none was hand-edited.

What the numbers show:

Zstandard at level 3 produced the smallest practical file at 68.61 MiB, roughly 45 percent below the 125.06 MiB uncompressed baseline, while writing in 0.9421 s.
Raising Zstandard from level 3 to level 9 saved only 0.10 MiB of file size but increased write time from 0.9421 s to 2.3837 s.
gzip was the weakest tradeoff measured: a 74.90 MiB file that is larger than either Zstandard result, produced in a median 44.3629 s, roughly 47 times slower than Zstandard level 3.
Smaller row groups compressed worse (87.41 MiB at 128k rows versus 68.61 MiB at 1M rows) but skipped data far more finely, cutting a selective read to 0.0070 s.
Disabling dictionary encoding on a single low-cardinality string column inflated the whole file by 18.79 MiB, from 68.61 MiB to 87.40 MiB.

Sections covered: the size-versus-speed tradeoff, the test environment, the measurement methodology, the codec matrix, filtered reads and predicate pushdown, row-group sizing, dictionary encoding, and practical codec guidance.

Why codec choice is a size-versus-speed decision

Storing the same 10,000,000-row table as Apache Parquet produced files ranging from 125.06 MiB with no compression down to 68.52 MiB with Zstandard at level 9 — a difference of roughly 45 percent in on-disk footprint from a single writer setting. The write cost, however, ranged from 0.7139 seconds to more than 44 seconds depending on the codec chosen. Those two numbers, measured on one machine over one dataset, frame the practical question this benchmark addresses: which Parquet compression codec to select, and why the answer is rarely the codec that produces the smallest file.

Apache Parquet is a columnar file format, meaning values from the same column are stored together rather than interleaved row by row. Compression is applied per column chunk after encoding, so the codec operates on runs of similar values, which is why columnar layout compresses far better than row-oriented storage. The format specification defines a fixed set of codecs — among them SNAPPY, GZIP, ZSTD, and LZ4 — and each represents a different point on the curve that trades computation for space. The mechanics of columnar layout, page encoding, and predicate pushdown are covered in a companion article on Apache Parquet and Apache Arrow internals; the focus here is narrower and empirical, measuring what those codecs actually cost and save on one representative dataset.

The reason a benchmark is worth running rather than reasoning from first principles is that the tradeoffs interact. A codec that compresses tightly can be slow enough to write that it stalls an ingestion pipeline. A codec that decompresses quickly may barely shrink the file. Row-group size, which controls how finely a query engine can skip data, changes compression ratio at the same time. Dictionary encoding, applied before the codec runs, can matter more than the codec itself for certain columns. Measuring these effects together, on the same data with the same tooling, is the only way to see which choices dominate.

Key Takeaway: On this dataset, the largest single lever was moving from no compression to any general-purpose codec; the second largest was dictionary encoding on the string column; and the smallest, most expensive lever was raising the Zstandard level. Codec choice is best made by ranking these levers, not by chasing the last megabyte.

Test environment: hardware, software, and dataset

Compression benchmarks are only interpretable when the machine, the library versions, and the data are stated exactly, because decompression speed is CPU-bound and codec implementations differ between library releases. The complete environment capture is recorded in the benchmark repository; the essentials are reproduced below.

Hardware and software

Item	Value
CPU	Apple M2 Pro, 10 physical = 10 logical cores (arm64)
Memory	32 GiB (34,359,738,368 bytes)
Operating system	macOS 26.5.1 (build 25F80)
Python	3.13.1 (CPython, Clang 16)
pyarrow / Arrow C++	25.0.0 / 25.0.0
Supporting libraries	duckdb 1.5.4, zstandard 0.25.0, psutil 7.2.2, uv 0.9.13

All writing and reading was done through the PyArrow Parquet writer, so the results reflect the Arrow C++ implementations of each codec at version 25.0.0. A different language binding or a different Arrow release could shift the absolute times, which is precisely why the versions are pinned and stated. The PyArrow write_table documentation lists the codec names accepted by this writer and confirms that its default codec is snappy.

The dataset

The benchmark reused a single 10,000,000-row table shared with a sibling study on DuckDB and Polars in-process analytics, so that both studies run on identical data. The table has five columns of mixed type, an in-memory Arrow footprint of 312,390,663 bytes (297.9 MiB), and the following schema:

ts — timestamp[us], event time spanning roughly 90 days (November 2023 to February 2024).
user_id — int32, values in the range 1 to 499,997.
category — string, low cardinality with 10 distinct values (books, automotive, electronics, garden, beauty, apparel, home, grocery, and others).
amount — float64, values from 0.09 to 3,714.81.
flag — bool.

Two preparation steps were applied once, before any timing. First, the category column was decoded from its stored dictionary form to plain UTF-8 text, so that the writer’s dictionary flag — and not a carried-over Arrow dictionary — is the only thing deciding whether dictionary encoding is applied at the file level. Second, the table was sorted by ts ascending. Time-series data is realistically stored in time order, and sorting makes each row group hold a tight, non-overlapping range of timestamps, which is a precondition for the predicate-pushdown measurements described later.

Methodology: how every number was produced

Each measurement was repeated three times, and the median of the three runs is quoted throughout; the recorded results file also stores the full list of run times for inspection. Timings use a wall-clock performance counter, and garbage collection is forced before each timed operation to reduce interference.

One property of the setup deserves explicit statement rather than silent assumption: the read-cache state is warm. Reads happen immediately after each file is written, so the file is present in the operating-system page cache. Read timings therefore measure the CPU cost of decoding and parsing the columnar data, not cold disk input. This is the standard basis for comparing codecs because it isolates decompression speed from storage latency, but it means the read numbers should be read as decode cost, not as the end-to-end latency of a query against a cold object store. File sizes, by contrast, are exact on-disk byte counts, and per-column compressed and uncompressed sizes are read directly from the Parquet footer metadata.

Three separate experiments were run. The codec matrix fixed the row-group size at 1,000,000 rows and dictionary encoding on, then wrote and read the table with each of six codecs. The row-group study fixed the codec at Zstandard level 3 and varied the row-group size. The dictionary study fixed the codec at Zstandard level 3 and wrote the table twice, once with dictionary encoding on and once off, comparing the string column. The raw console output of the codec matrix is reproduced verbatim below.

==============================================================================
CODEC MATRIX  (row_group_size=1000000, dictionary encoding ON)
==============================================================================
predicate: ts >= 2024-02-08 10:13:18.975015  (expected ~498,711 rows, 4.99%)

[none    ] size=  125.06 MiB  write_med=0.7139s  read_med=0.0739s  col_med=0.0083s  filt_push_med=0.0176s (rows=498,711)  filt_nopush_med=0.0163s
[snappy  ] size=  102.17 MiB  write_med=0.8760s  read_med=0.0890s  col_med=0.0092s  filt_push_med=0.0170s (rows=498,711)  filt_nopush_med=0.0303s
[zstd_l3 ] size=   68.61 MiB  write_med=0.9421s  read_med=0.0783s  col_med=0.0140s  filt_push_med=0.0173s (rows=498,711)  filt_nopush_med=0.0406s
[zstd_l9 ] size=   68.52 MiB  write_med=2.3837s  read_med=0.0755s  col_med=0.0137s  filt_push_med=0.0171s (rows=498,711)  filt_nopush_med=0.0407s
[gzip    ] size=   74.90 MiB  write_med=44.3629s  read_med=0.1048s  col_med=0.0352s  filt_push_med=0.0230s (rows=498,711)  filt_nopush_med=0.0900s
[lz4     ] size=  102.65 MiB  write_med=0.8580s  read_med=0.0858s  col_med=0.0084s  filt_push_med=0.0172s (rows=498,711)  filt_nopush_med=0.0216s

The codec matrix: file size, write speed, read speed

The full codec matrix is shown in the table below. Write and read medians are in seconds; file size is the exact on-disk footprint in MiB; the single-column read measures reading only the amount column.

Codec	Size (MiB)	Write median (s)	Full read (s)	Single column (s)
none	125.06	0.7139	0.0739	0.0083
snappy	102.17	0.8760	0.0890	0.0092
zstd level 3	68.61	0.9421	0.0783	0.0140
zstd level 9	68.52	2.3837	0.0755	0.0137
gzip	74.90	44.3629	0.1048	0.0352
lz4	102.65	0.8580	0.0858	0.0084

Three findings stand out. First, the two Zstandard settings produced almost the same file — 68.61 MiB at level 3 versus 68.52 MiB at level 9 — a difference of under 0.10 MiB, yet level 9 took 2.3837 s to write against 0.9421 s for level 3, roughly two and a half times longer. On this data, the higher level buys essentially nothing for a substantial write cost. Second, gzip is dominated on every axis that matters: its 74.90 MiB file is larger than either Zstandard result, and its 44.3629 s median write is about 47 times slower than Zstandard level 3. gzip is the only codec whose write time is measured in tens of seconds rather than fractions of a second.

Third, snappy and lz4 behave almost identically here: 102.17 MiB against 102.65 MiB, with writes of 0.8760 s and 0.8580 s respectively. Both are fast, lightly compressing codecs that shrink the file to about 82 percent of the uncompressed size. snappy is the writer’s default, and neither of these two clearly beats the other on this dataset. It is worth noting that the Parquet specification marks the older LZ4 codec as deprecated in favor of a newer LZ4_RAW framing; the codec exercised here is the one PyArrow writes under the lz4 name.

Read performance tells a flatter story. On a warm page cache, decoding the full table took between 0.0739 s (no compression) and 0.1048 s (gzip). Zstandard sat between the extremes at 0.0783 s for level 3 and 0.0755 s for level 9, and snappy and lz4 landed at 0.0890 s and 0.0858 s. In other words, choosing a strong general-purpose codec added only a few milliseconds to a full decode of the whole table relative to no compression at all, while gzip again stood out as the slowest to read. When the read touched only the single amount column, the absolute times dropped by roughly an order of magnitude, but the same ordering held, with gzip slowest at 0.0352 s.

Caution: These read times are warm-cache decode measurements. On a cold read from network storage, transfer time scales with file size, which would favor the smaller Zstandard and gzip files. The read ranking here isolates decompression cost and should not be read as full query latency against a remote object store.

Filtered reads and the price of decompression

Parquet stores per-row-group statistics, including the minimum and maximum of each column, which lets a reader skip entire row groups whose range cannot satisfy a predicate. This is predicate pushdown, and its benefit depends on the data being sorted so that ranges do not overlap. Because the table was sorted by ts, a predicate of ts >= a late cutoff — matching 498,711 rows, or 4.99 percent of the table — allowed the reader to skip most row groups.

The benchmark measured two versions of that filtered read. The pushdown version passed the predicate to the reader, which used row-group statistics to skip groups. The control version read the relevant columns in full and filtered them in memory afterward, doing the work that pushdown avoids. The contrast is clearest for the compressed codecs. For Zstandard level 3, the pushdown read took 0.0173 s while the no-pushdown control took 0.0406 s; for snappy the two were 0.0170 s and 0.0303 s. Skipping row groups avoids decompressing them, so the saving grows with how expensive the codec is to decode.

The uncompressed case is the instructive exception. With no compression, the pushdown read (0.0176 s) was marginally slower than the no-pushdown control (0.0163 s), because there is no decompression to avoid and the pushdown path carries a small bookkeeping overhead. In short, predicate pushdown and compression reinforce each other: the more a codec costs to decode, the more a query gains by skipping data it never needed to read. This interaction is central to why lakehouse query engines lean heavily on statistics-based skipping over columnar files.

Row-group size: compression against skipping granularity

A row group is the horizontal partition of a Parquet file within which columns are chunked and compressed. Its size controls two things at once. Larger row groups give the codec more context and amortize per-group overhead, which improves compression; smaller row groups create more, finer boundaries, which lets predicate pushdown skip data more precisely. These pull in opposite directions, and the benchmark measured the tension directly by fixing the codec at Zstandard level 3 and varying the row-group size across 131,072 rows, 1,000,000 rows, and 5,000,000 rows.

Row-group size	Row groups	Size (MiB)	Full read (s)	Filtered pushdown read (s)
131,072	77	87.41	0.0587	0.0070
1,000,000	10	68.61	0.0793	0.0176
5,000,000	2	66.79	0.1922	0.0853

The compression trend is monotonic: the file shrank from 87.41 MiB at 131,072 rows per group to 68.61 MiB at one million and 66.79 MiB at five million. Small row groups pay a real penalty, here roughly 20 MiB, because encoding structures such as the dictionary reset at every group boundary and the codec sees less data per chunk. The selective-read trend runs the other way. With 77 fine-grained groups, the pushdown filter completed in 0.0070 s, because the reader could discard almost every group; with only two coarse groups it took 0.0853 s, because a matching predicate forces reading a full half of the table. The full-scan read was fastest at the smallest row-group size (0.0587 s) and slowest at the largest (0.1922 s) in this run, though full-scan timing is more sensitive to how decode work parallelizes across groups and should be treated with more caution than the size and pushdown figures.

Tip: For time-sorted data queried by time range, a moderate row-group size such as one million rows is a reasonable default: it captured most of the compression benefit (68.61 MiB, close to the 66.79 MiB of the largest groups) while keeping selective reads fast (0.0176 s). Very large row groups optimize storage at the expense of every filtered query.

Dictionary encoding on a low-cardinality column

Dictionary encoding replaces repeated column values with small integer indices into a per-column dictionary, and it is applied before the compression codec runs. For a column with few distinct values it can shrink the data dramatically on its own, which changes how much work is left for the codec. The category column is a natural test: 10 distinct string values repeated across 10,000,000 rows. Holding the codec at Zstandard level 3, the table was written once with dictionary encoding on and once with it off.

The effect was large. With dictionary encoding on, the category column occupied 3.912 MiB compressed on disk, having been reduced to 4.813 MiB uncompressed before the codec even ran. With dictionary encoding off, the same column expanded to 104.825 MiB uncompressed — the raw repeated strings — and although Zstandard still compressed that down, it landed at 14.172 MiB, about 3.6 times the size of the dictionary-encoded column. The whole-file impact followed: the file grew from 68.61 MiB with the dictionary to 87.40 MiB without it, an increase of 18.79 MiB attributable to a single column.

The practical lesson is that for low-cardinality columns, dictionary encoding can matter more than the choice of codec. A strong general-purpose codec applied to raw repeated strings did not recover what dictionary encoding achieved almost for free. PyArrow enables dictionary encoding by default, and the Parquet specification defines the modern RLE_DICTIONARY encoding for exactly this pattern. The failure mode to watch for is a high-cardinality column where the dictionary grows too large to help; the format falls back to plain encoding in that case, and forcing dictionary encoding on such a column wastes effort. The behavior is well matched to the kind of categorical fields common in batch and streaming data pipelines.

Choosing a codec: practical guidance

The measurements support a short, honest set of recommendations for this class of tabular, analytics-oriented data. They are grounded in one dataset on one machine, so they are guidance rather than universal law, but the gaps between codecs are large enough to be robust to modest variation.

Zstandard at a low level is the strongest default. Level 3 produced the smallest practical file (68.61 MiB) at a write cost (0.9421 s) close to the fast codecs and a read cost (0.0783 s) within a few milliseconds of no compression at all. It is the setting that most cleanly balances the three axes. Raising the level to 9 is not worth it on data like this, where it saved under 0.10 MiB for more than double the write time; higher Zstandard levels earn their cost only when the data compresses much further and storage or transfer dominates.

snappy or lz4 make sense when write throughput and decode latency matter more than storage, and the data is written far more often than it is read. Both wrote in about 0.86 to 0.88 s and produced files near 102 MiB, meaning they leave roughly a third of the compressible space on the table in exchange for speed and simplicity. snappy being the PyArrow default is a reasonable choice for intermediate or short-lived files. gzip has no winning case in these results: it was both larger and far slower to write than Zstandard, so Zstandard should be preferred wherever gzip might have been chosen for ratio. Leaving data uncompressed is defensible only for very short-lived scratch files where even a sub-second write matters and the 125.06 MiB footprint is irrelevant.

Two settings beyond the codec deserve equal attention. Row-group size should be tuned to the query pattern: around one million rows is a sound default for time-sorted data read by range, capturing most of the compression benefit while keeping selective reads fast. Dictionary encoding should be left on for low-cardinality columns, where it delivered a larger saving than the codec choice itself. These file-level decisions carry across the broader storage stack, including the open table formats that manage collections of Parquet files, and they compound with engine-level execution choices such as those examined in the article on Apache Spark internals.

Frequently Asked Questions

Which Parquet compression codec is the best default?

On this benchmark, Zstandard at level 3 was the strongest all-round default: it produced the smallest practical file at 68.61 MiB while writing in 0.9421 s and reading in 0.0783 s, within a few milliseconds of no compression. snappy, which PyArrow uses by default, is a reasonable alternative when write speed matters more than storage, but it left the file about 50 percent larger at 102.17 MiB.

Is a higher Zstandard compression level worth the cost?

Not on data like this. Moving from level 3 to level 9 reduced the file by under 0.10 MiB, from 68.61 MiB to 68.52 MiB, while write time rose from 0.9421 s to 2.3837 s, roughly two and a half times longer. Higher levels earn their cost only when the data compresses substantially further and storage or network transfer, rather than write throughput, is the binding constraint.

Why was gzip so much slower than the other codecs?

gzip recorded a median write time of 44.3629 s, about 47 times the 0.9421 s of Zstandard level 3, and it also produced a larger file (74.90 MiB versus 68.61 MiB) and the slowest full read (0.1048 s). In these results gzip was dominated on every axis, so Zstandard is the better choice wherever gzip might have been selected for compression ratio.

How does row-group size affect compression and query speed?

Smaller row groups compress worse but skip data more finely. At 131,072 rows per group the file was 87.41 MiB but a selective pushdown read took only 0.0070 s; at 5,000,000 rows per group the file shrank to 66.79 MiB but the same read rose to 0.0853 s. Around one million rows balanced the two, giving 68.61 MiB and a 0.0176 s selective read.

References

Benchmark source, methodology, environment capture, and recorded results: jkc4416/analytics-bench (parquet-codecs study).
Apache Parquet format, compression codecs: parquet.apache.org — Compression.
Apache Parquet format, dictionary and RLE encodings: parquet.apache.org — Encodings.
PyArrow writer parameters (compression, compression_level, use_dictionary, row_group_size): arrow.apache.org — pyarrow.parquet.write_table.
Zstandard compression library and levels: facebook/zstd.

Conclusion

Across one 10,000,000-row table measured with a fixed, recorded methodology, the codec that produced the smallest file was not the one worth choosing for most workloads. Zstandard at level 3 sat at the balanced point: 68.61 MiB on disk, a 0.9421 s write, and a read cost within a few milliseconds of no compression. Level 9 spent more than twice the write time to save under a tenth of a megabyte, and gzip was both larger and dramatically slower, with a 44.3629 s median write. Two file-level settings mattered as much as the codec — a moderate row-group size preserved most of the compression benefit while keeping selective reads fast, and dictionary encoding on a single low-cardinality column changed the whole-file size by 18.79 MiB. The durable conclusion is that Parquet compression is a system of interacting choices, and the right decision comes from measuring the levers in order of impact rather than optimizing any one of them in isolation.

July 19, 2026

Schema Evolution and the Schema Registry: Avro, Protobuf, and Compatibility

In an event-driven system built on Apache Kafka, every message that crosses the network is a sequence of raw bytes with no built-in description of its structure. A consumer that reads those bytes must already know how to interpret them: which fields are present, in what order, and with what types. When the producer and consumer are developed and deployed independently, and when the shape of the data changes over months of feature work, this implicit agreement becomes fragile. A schema registry addresses that fragility by turning the implicit agreement into an explicit, versioned, and centrally governed contract. This article examines how a schema registry works, how the Avro and Protocol Buffers serialization formats encode and evolve data, and how compatibility modes let a schema change without breaking the producers and consumers that already depend on it.

The discussion assumes familiarity with Kafka as a distributed log of immutable records. The registry pattern described here follows the Confluent Schema Registry, the most widely deployed implementation, but the underlying concepts, particularly Avro schema resolution and Protobuf field-number stability, are properties of the serialization formats themselves and apply wherever those formats are used.

Summary

What this post covers: How a schema registry enforces a versioned data contract on Kafka messages, how Avro and Protobuf encode and evolve records, and how compatibility modes govern which schema changes are safe and in what order producers and consumers must be upgraded.

Key insights:

Kafka messages are opaque bytes and carry no embedded schema, so a shared registry is required to make them interpretable, unlike a Parquet file that stores its schema in its own footer.
The Confluent wire format prefixes every payload with a 5-byte header: one magic byte set to 0x00 followed by a 4-byte big-endian schema ID; Protobuf inserts an additional message-index array that Avro and JSON Schema do not use.
The default compatibility mode is BACKWARD, not FULL; it checks a new schema only against the latest registered version and requires consumers to be upgraded before producers.
Avro evolves through a reader-schema-versus-writer-schema resolution model driven by field defaults and aliases, while Protobuf evolves through stable field numbers that must never be reused after a field is removed.
Compatibility is enforced by the registry at schema-registration time, not when a message is decoded, so a rejected registration is the mechanism that prevents a breaking change from ever reaching the log.

Main topics: Why Schemas Matter in Event-Driven Pipelines, Serialization Formats, Schema Registry Architecture and the Wire Format, Compatibility Modes and the Rules of Safe Evolution, Operational Practices for Schema Evolution.

Why Schemas Matter in Event-Driven Pipelines

A schema is a formal description of the structure of a record: the names of its fields, their data types, and rules such as whether a field may be absent. In a monolithic application, this description lives in the type system of a single codebase, and the compiler enforces it. In a distributed streaming pipeline, the producer and the consumer are separate programs, often written in different languages and released on different schedules, connected only by a stream of bytes. Nothing in the bytes themselves states what they mean.

This is the defining property of a Kafka message: it is not self-describing. A message is a key and a value, each an opaque byte array as far as the broker is concerned. The broker never parses the payload. Interpretation is entirely the responsibility of the client that deserializes it. If a producer begins writing an extra field, or renames one, or changes an integer to a string, a consumer that was compiled against the older structure has no way to detect the change from the bytes alone. It will either read garbage or fail.

The contrast with a columnar file format such as Apache Parquet is instructive. A Parquet file stores its schema in its own footer, so any reader can open the file and discover its structure without external coordination; the file is self-describing at rest. A detailed treatment of that design appears in the discussion of Parquet and Apache Arrow columnar storage internals. Kafka messages have no such footer. Attaching a full schema to every message would be prohibitively wasteful, because a stream may carry millions of records that share one structure. The registry resolves this tension by storing each schema once, assigning it a compact numeric identifier, and letting every message carry only that identifier.

This mechanism is the technical enforcement layer beneath a broader organizational concern known as a data contract, a documented and testable agreement about the meaning and quality of a dataset shared across teams. The relationship is direct: a data contract states what the data should look like, and the schema registry is one place where that statement is enforced automatically on every message. The organizational and policy dimension is treated separately in the discussion of data contracts and data quality enforcement; the present article concentrates on the wire-level mechanics. The registry also sits within a larger architectural decision about how data moves, examined in the comparison of streaming versus batch processing architectures.

Key Takeaway: A Kafka message carries data but not its meaning. The schema registry supplies the missing description by storing each schema once and letting every message reference it by a small numeric identifier, which keeps messages compact while making them interpretable.

Serialization Formats: Avro, Protobuf, and JSON Schema

Serialization is the process of converting an in-memory object into a byte sequence suitable for transmission or storage; deserialization is the reverse. The Confluent Schema Registry supports three serialization formats out of the box, each paired with its own serializer and deserializer: Apache Avro through KafkaAvroSerializer and KafkaAvroDeserializer, Protocol Buffers through KafkaProtobufSerializer and KafkaProtobufDeserializer, and JSON Schema through KafkaJsonSchemaSerializer and KafkaJsonSchemaDeserializer (Confluent SerDes documentation, as of 2026-07-17). The choice among them affects encoding size, tooling, and the exact rules that govern how a schema may change.

Apache Avro

Apache Avro is a data serialization system in which the schema is expressed as JSON and the data is encoded in a compact binary form that contains no field names or type tags. Because the encoding omits field identifiers, a decoder cannot interpret the bytes without a schema. Avro’s central design idea is that the schema used to write the data (the writer’s schema) and the schema used to read it (the reader’s schema) may differ, and a well-defined resolution procedure reconciles the two. The latest stable release is Avro 1.12.1, published on 2025-10-16 with four security fixes, following 1.12.0 from 2024-08-05 (Apache Avro release announcement, as of 2026-07-17).

A minimal Avro schema for an order event is a JSON object of type record:

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "fields": [
    { "name": "order_id",   "type": "string" },
    { "name": "customer_id","type": "string" },
    { "name": "amount",     "type": "double" },
    { "name": "currency",   "type": "string", "default": "USD" }
  ]
}

The default on the currency field is not merely a convenience. As the compatibility section will show, a default value is the single most important element that determines whether a field can be added or removed without breaking readers. An optional field in Avro is expressed as a union with null together with a default, for example "type": ["null", "string"], "default": null. This is a distinct mechanism from optionality in Protobuf and the two should not be conflated.

Protocol Buffers

Protocol Buffers, commonly abbreviated Protobuf, is a serialization format developed by Google in which the structure is declared in a .proto file and each field is assigned a numeric tag known as a field number. In the binary encoding, the field number, not the field name, identifies each value. The current major syntax is proto3; an emerging direction called editions is intended to supersede the proto2 and proto3 distinction, but there is no single version number that meaningfully anchors the format. The same order event expressed in proto3 is:

syntax = "proto3";
package com.example.orders;

message OrderPlaced {
  string order_id    = 1;
  string customer_id = 2;
  double amount      = 3;
  string currency    = 4;
}

The numbers 1 through 4 are the field numbers. They are the anchor of Protobuf compatibility. Once a field number is assigned and data has been written with it, that number must never be changed and must never be reused for a different field, a rule examined in detail in the compatibility section (protobuf.dev best practices, as of 2026-07-17).

JSON Schema and a comparison

JSON Schema describes the structure of JSON documents. Its principal advantage is human readability and native compatibility with systems that already exchange JSON. Its principal cost is size and speed: JSON is a text encoding that repeats every field name in every message and requires parsing text rather than reading a compact binary layout. The three formats occupy different points on a tradeoff surface.

Property	Avro	Protobuf	JSON Schema
Encoding	Compact binary	Compact binary	Text (JSON)
Field identity	Position and name via resolution	Numeric field number	Field name
Evolution anchor	Field defaults and aliases	Stable field numbers	Required vs. optional keys
Relative size	Smallest	Small	Largest
Human readable on the wire	No	No	Yes
Cross-language code generation	Optional	Central to the model	Optional

No format is uniformly superior. Avro is common in data-lake and analytics pipelines where its compact encoding and rich resolution rules fit batch and streaming loads; systems that emit change events, such as those built with Debezium, frequently produce Avro records backed by a registry, as described in the treatment of change data capture with Debezium and Kafka. Protobuf is common where the same message types are also used in remote procedure calls and where code generation across many languages is a priority. JSON Schema suits teams that value readability and already operate on JSON. The registry supports all three uniformly.

Schema Registry Architecture and the Wire Format

The Confluent Schema Registry is a service that stores schemas and assigns each a globally unique numeric identifier. It organizes schemas under subjects. A subject is a named scope under which a sequence of schema versions accumulates; each registration under a subject produces a new version number, and each distinct schema receives a global schema ID that is unique across the whole registry. Compatibility checks are performed within a subject. Producers and consumers interact with the registry over a REST API and cache results locally to avoid a lookup on every message.

Subject naming strategies

The name of the subject under which a schema is registered is determined by a subject naming strategy. Three strategies are provided (Confluent SerDes documentation, as of 2026-07-17):

TopicNameStrategy (the default): the subject is <topic>-key or <topic>-value. This requires every message in a topic to conform to a single schema, which is the most common arrangement.
RecordNameStrategy: the subject is the fully qualified record name. This permits several record types to share one topic, and compatibility for that record name is scoped across every topic that carries it.
TopicRecordNameStrategy: the subject is <topic>-<fully qualified record name>. This also permits multiple record types per topic, but scopes compatibility per record type within each topic.

The 5-byte wire format

When a serializer writes a message, it does not embed the schema. It embeds a reference to the schema in a fixed header prepended to the payload. The Confluent wire format begins each serialized value with a 5-byte header: byte 0 is a magic byte with the value 0x00, and bytes 1 through 4 hold the schema ID as a 4-byte signed 32-bit integer in big-endian order, also called network byte order. The serialized payload follows from byte 5 onward (Confluent SerDes documentation; wire-format community references, as of 2026-07-17). A consumer validates the framing by confirming that the message is longer than 5 bytes and that byte 0 equals 0x00.

The wire format is deliberately stable. Confluent states that it will not change without significant warning across multiple major releases, and that within the version identified by the magic byte it never changes in a way that would break existing readers (Confluent SerDes documentation, as of 2026-07-17). The Protobuf variant is the one exception to the simple layout: because a single .proto file may declare several message types, a Protobuf message inserts a message-index array, a length-prefixed list of variable-length integer indexes that identifies which message type was serialized, between the schema ID and the payload. Avro and JSON Schema have no such segment.

The producer and consumer flow

The registry interaction is straightforward once the framing is understood. On the producing side, the serializer extracts the schema from the record, registers it under the appropriate subject (or looks up its ID if already registered), prepends the 5-byte header, and sends the message. On the consuming side, the deserializer reads the header, extracts the schema ID, fetches the corresponding schema from the registry, and uses it, together with the consumer’s own reader schema, to decode the payload. Both sides cache aggressively, so the registry is consulted at most once per distinct schema rather than once per message.

A Python producer using the confluent-kafka library expresses this flow directly. The AvroSerializer holds a reference to the registry client, and the SerializingProducer applies it to every value before sending:

from confluent_kafka import SerializingProducer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer

schema_str = open("order_placed.avsc").read()

registry = SchemaRegistryClient({"url": "http://schema-registry:8081"})
avro_serializer = AvroSerializer(registry, schema_str)

producer = SerializingProducer({
    "bootstrap.servers": "broker:9092",
    "value.serializer": avro_serializer,
})

order = {
    "order_id": "ORD-91",
    "customer_id": "CUST-4471",
    "amount": 42.50,
    "currency": "USD",
}
producer.produce(topic="orders", value=order)
producer.flush()

The consumer side mirrors this arrangement with a DeserializingConsumer and an AvroDeserializer. The deserialization step is where a registry-backed stream connects to ordinary consumer code; the surrounding consumer plumbing, including offset management and the poll loop, is treated in the walkthrough of an Apache Kafka consumer implementation in Python. The registry itself can be queried directly over its REST API, which is useful in scripts and continuous-integration checks. Retrieving the current compatibility mode for a subject, for example, is a single request:

# Read the compatibility mode configured for a subject
curl -s http://schema-registry:8081/config/orders-value

# Set the subject to FULL compatibility
curl -s -X PUT http://schema-registry:8081/config/orders-value \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"compatibility": "FULL"}'

# List the versions registered under a subject
curl -s http://schema-registry:8081/subjects/orders-value/versions

Compatibility Modes and the Rules of Safe Evolution

Compatibility is the property that a schema change does not break the programs that already read or write data under the older schema. The registry enforces this by rejecting a schema registration that would violate the configured compatibility mode. This point deserves emphasis: the check happens at registration time, when a producer or a deployment pipeline attempts to register a new schema, not when an individual message is decoded. A rejected registration is the safeguard that prevents an incompatible change from ever being written to the log.

Two directions of compatibility are defined. Backward compatibility means new code can read data written by old code; a new reader accepts old data. Forward compatibility means old code can read data written by new code; an old reader accepts new data. Full compatibility requires both directions. Each of these has a transitive variant. A non-transitive mode checks a candidate schema only against the latest registered version, whereas a transitive mode checks it against every previously registered version (Confluent schema-evolution documentation, as of 2026-07-17).

Caution: The default compatibility mode is BACKWARD, not FULL. Under BACKWARD, a new schema is only compared against the single latest version, so a chain of individually valid changes can drift away from an early version in ways a transitive mode would have blocked. Assuming the default provides full or transitive protection is a common and costly error.

Compatibility type	Changes allowed	Versions checked	Upgrade first
BACKWARD (default)	Delete fields; add fields with a default	Latest version only	Consumers
BACKWARD_TRANSITIVE	Delete fields; add fields with a default	All previous versions	Consumers
FORWARD	Add fields; delete fields that have a default	Latest version only	Producers
FORWARD_TRANSITIVE	Add fields; delete fields that have a default	All previous versions	Producers
FULL	Add or remove optional (default-valued) fields only	Latest version only	Either order
FULL_TRANSITIVE	Add or remove optional (default-valued) fields only	All previous versions	Either order
NONE	Any change (no checking)	Not applicable	Coordinate manually

Source: Confluent “Schema Evolution and Compatibility” documentation, as of 2026-07-17. A useful mnemonic follows directly from the definitions: BACKWARD means a new reader reads old data, so consumers are upgraded first; FORWARD means an old reader reads new data, so producers are upgraded first; FULL supports both directions, so the two sides may be upgraded independently.

Avro schema resolution

Avro’s compatibility behavior follows directly from its schema resolution rules, the procedure by which a reader reconciles its own schema with the writer’s schema field by field. The specification defines the outcomes precisely (Apache Avro Specification, Schema Resolution, as of 2026-07-17):

If the writer’s record contains a field that the reader’s schema does not declare, the writer’s value for that field is ignored. This is what makes it safe for a producer to add a field that older consumers have not yet learned about.
If the reader’s schema declares a field with a default value and the writer’s schema lacks that field, the reader supplies the default. This is what makes it safe to add a field to the reader.
If the reader’s schema declares a field with no default and the writer’s schema lacks it, resolution fails with an error. This is the case a compatibility check is designed to prevent.
Aliases allow a named type or field to declare alternative names, so a reader can match a writer’s old field or record name to a renamed one, which enables compatible renaming.
Resolution recurses into arrays, maps, and unions, applying the same rules to item, value, and branch schemas.

Protobuf field-number evolution

Protobuf takes a different path to the same goal. Because the binary encoding identifies each value by its field number rather than its name, a field can be renamed freely without affecting the wire format, and a parser that encounters a field number it does not recognize simply skips it. This skipping behavior is the mechanism that makes additive evolution safe in proto3: adding a new field is compatible because older parsers ignore the unknown number, and reading data written by newer code succeeds because the extra fields are passed over (protobuf.dev, as of 2026-07-17).

The corresponding hazard is field-number reuse. Once a field number has been used, it must never be assigned to a different field, even after the original field is deleted. If a number is reused with a different type, a decoder holding old data will interpret the new field’s bytes according to the old field’s type and silently produce corrupt values. The remedy defined by the format is the reserved declaration, which permanently blocks a field number and name from being reused:

message OrderPlaced {
  reserved 3;                 // amount was removed; its number is retired
  reserved "amount";          // the old name is retired as well

  string order_id    = 1;
  string customer_id = 2;
  string currency    = 4;
  int64  amount_cents = 5;    // new field, new number, never reuse 3
}

Tip: The two formats reach compatibility through different levers, and the mental model should match the format in use. In Avro, reason about defaults and the reader-versus-writer pair. In Protobuf, reason about field numbers and the reserved list. Mixing the two models is a frequent source of confusion when a team migrates between formats.

Upgrade order in practice

The compatibility mode dictates not only which changes are legal but also the sequence in which the two sides of a stream must be deployed. Under BACKWARD, the new schema is designed so that new consumers can read data still being produced under the old schema; consumers are therefore upgraded first, after which producers can move to the new schema safely. Under FORWARD, the arrangement reverses: producers move first because the guarantee is that old consumers can still read the newly produced data. Under FULL, both guarantees hold, so the two sides can be rolled out in any order. Kafka Streams applications, which both consume and produce, are supported only under BACKWARD and the modes that include it, namely FULL and FULL_TRANSITIVE (Confluent schema-evolution documentation, as of 2026-07-17).

Operational Practices for Schema Evolution

Choosing and operating a compatibility policy is as much an organizational decision as a technical one. The following practices reflect the way schema registries are commonly run in production.

Choosing a compatibility mode

The right mode depends on which side of the stream tends to deploy first and how tightly the two sides are coordinated. When consumers are numerous and slow to upgrade, as with a widely subscribed event, BACKWARD or BACKWARD_TRANSITIVE lets producers evolve without waiting for every consumer. When a single producer feeds many independent consumers that cannot be upgraded in lockstep, FORWARD protects those consumers from a producer change. FULL and FULL_TRANSITIVE impose the strictest discipline and are appropriate for long-lived, high-value topics where either side may change at any time. The transitive variants trade a small amount of flexibility for a strong guarantee: because they validate against the entire version history, they prevent a slow drift that a sequence of latest-only checks would permit.

Testing compatibility in continuous integration

Because the registry enforces compatibility at registration time, a broken schema change is most cheaply caught before deployment. The registry exposes a compatibility-check endpoint that reports whether a candidate schema would be accepted under the subject’s configured mode, without registering it. Wiring this call into a continuous-integration pipeline turns a potential production incident into a failed build:

# Check a candidate schema against the latest version of a subject.
# Returns {"is_compatible": true} or false without registering.
curl -s -X POST \
  http://schema-registry:8081/compatibility/subjects/orders-value/versions/latest \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d @candidate_schema_request.json

Treating schema definitions as versioned source code, reviewed and tested like any other artifact, is the connective tissue between the registry and a broader data-contract discipline. The same schema that governs the wire format can drive downstream typing in transformation frameworks, so that a well-typed event flows into modeled tables; the transformation layer is discussed in the overview of building transformation pipelines with dbt.

Handling genuinely breaking changes

Some changes cannot be made compatibly. Changing a field’s type in an incompatible way, or removing a required field that consumers depend on, will and should be rejected under any checked mode. Two disciplined options exist. The first is to publish the incompatible data to a new topic and migrate consumers deliberately, retiring the old topic once no consumer remains. The second, to be used sparingly, is to set the subject to NONE for a controlled window, coordinate the producer and consumer deployment manually, and restore a checked mode afterward. Setting NONE permanently removes the very protection the registry exists to provide and is not a substitute for a migration plan.

Caution: Registry availability is on the critical path for producers and consumers that have not yet cached a needed schema. A registry outage can stall serialization or deserialization for new schema IDs. Production deployments typically run the registry in a highly available configuration and back up its underlying storage, because losing the schema history makes historical data uninterpretable.

Platform context

The registry evolves alongside the broader Kafka platform. Confluent Platform 8.0 reached general availability in June 2025 and bundles Apache Kafka 4.0, which removes the dependency on ZooKeeper in favor of the KRaft consensus protocol; the same release brought general availability of client-side field-level encryption and of passwordless authentication for the Schema Registry (Confluent Platform 8.0 announcement, as of 2026-07-17). These platform-level changes do not alter the wire format or the compatibility semantics described here, which remain stable by design, but they do affect how a registry is secured and operated.

Frequently Asked Questions

Is the default compatibility mode BACKWARD or FULL?

The default compatibility mode in the Confluent Schema Registry is BACKWARD. Under BACKWARD, a candidate schema is validated only against the latest registered version, and it permits deleting fields and adding fields that carry a default value. It does not provide the two-directional guarantee of FULL, nor the whole-history guarantee of the transitive variants. Teams that need protection against drift across the entire version history should explicitly configure a transitive mode.

When are compatibility checks actually performed?

Compatibility is enforced at schema-registration time, not when an individual message is decoded. When a producer or a deployment pipeline attempts to register a new schema under a subject, the registry compares it against the versions required by the configured mode and rejects the registration if it would break compatibility. A rejected registration prevents the incompatible schema, and therefore any message written with it, from entering the log.

How does the wire format differ between Avro and Protobuf?

Both use a 5-byte header consisting of a magic byte set to 0x00 followed by a 4-byte big-endian schema ID. Protobuf inserts an additional segment, a length-prefixed message-index array of variable-length integers, between the schema ID and the payload, because a single .proto file may define several message types and the deserializer must know which one was serialized. Avro and JSON Schema do not include this segment.

Should producers or consumers be upgraded first?

The order follows the compatibility mode. Under BACKWARD, consumers are upgraded first, because the guarantee is that new consumers can read data still produced under the old schema. Under FORWARD, producers are upgraded first, because old consumers must be able to read newly produced data. Under FULL, both guarantees hold and the two sides may be upgraded in either order.

Why must a Protobuf field number never be reused?

In the Protobuf binary encoding, each value is identified by its field number rather than its name. A decoder holding data written with an older schema will interpret bytes according to whatever field the number denoted at the time. If the number is later reassigned to a different field, especially with a different type, the old data is misread and silently corrupted. Marking a removed number as reserved permanently blocks its reuse and prevents this class of error.

References

Conclusion

The schema registry solves a problem that is easy to overlook until it causes an outage: a Kafka message carries data but not the description needed to interpret it. By storing each schema once, assigning it a compact identifier, and enforcing compatibility rules at registration time, the registry turns an implicit and fragile agreement between independently deployed producers and consumers into an explicit, versioned contract. The 5-byte wire format is the small piece of framing that makes this work on every message, with Protobuf’s message-index array the one notable variation to keep straight.

The two dominant binary formats reach compatibility by different means that are worth internalizing separately. Avro reconciles a reader schema against a writer schema, with field defaults and aliases as the levers that make additive and rename changes safe. Protobuf anchors compatibility on stable field numbers, with the reserved declaration as the guard against the corrupting error of number reuse. Above both sits the compatibility-mode matrix, whose central practical consequence is the upgrade order: BACKWARD upgrades consumers first, FORWARD upgrades producers first, and FULL frees the two sides to move independently. Applied with a default of BACKWARD understood correctly, compatibility checks wired into continuous integration, and a deliberate plan for the changes that cannot be made compatibly, the registry becomes the mechanism that lets a streaming data model evolve for years without breaking the systems that depend on it.

July 17, 2026

Apache Spark Internals: Catalyst, Tungsten, and the Shuffle Explained

Summary

What this post covers: This article examines how Apache Spark actually executes a query — from the logical plan produced by the Catalyst optimizer, through the machine code emitted by the Tungsten engine, down to the shuffle that moves data across the cluster. The aim is to explain why a Spark job is slow and where its cost is spent, rather than to introduce Spark from scratch.

Key insights:

Spark decomposes a job into stages at every wide dependency, and each stage boundary is a shuffle — the dominant source of network, disk, and serialization cost in most jobs.
Catalyst optimizes queries in four phases, of which only physical planning is cost-based; the rest are rule-based tree transformations, and Adaptive Query Execution (AQE) has been enabled by default since Spark 3.2.0 to re-optimize plans using runtime statistics.
Tungsten’s whole-stage code generation, shipped in Spark 2.0, fuses a chain of operators into a single JVM function and stores data in a compact off-heap binary format called UnsafeRow, reducing virtual dispatch and garbage-collection pressure.
Data skew, not raw data volume, is the most common cause of a stalled stage; salting and AQE’s runtime skew-join splitting are the standard mitigations.
Join strategy is decided by data size: a side smaller than 10 MB is broadcast, while two large inputs default to a sort-merge join, and AQE can switch between them at runtime.

Main topics: The Spark Execution Model, The Catalyst Optimizer and Adaptive Query Execution, Tungsten and Whole-Stage Code Generation, The Shuffle and Data Skew, Join Strategies and Practical Performance Tuning.

Apache Spark presents a deceptively simple programming surface. A data engineer writes a few DataFrame transformations or a SQL query, calls an action, and a result appears. Underneath that surface sits a distributed query engine that parses the request into a logical plan, rewrites it through dozens of optimization rules, compiles fragments of it to Java bytecode, and coordinates hundreds of tasks across a cluster. When a job runs slowly, the cause is almost never the API call that was written. It lies in how the engine translated that call into stages, how much data crossed the network during a shuffle, and whether the optimizer chose a physical plan suited to the actual shape of the data.

This article traces that translation end to end. It assumes familiarity with running Spark jobs and concentrates on the execution internals that determine performance: the DAG scheduler and its stage boundaries, the Catalyst optimizer and its adaptive re-planning, the Tungsten engine and its code generation, the sort-based shuffle, and the join strategies that Spark selects. The version referenced throughout is the Spark 4.1.x line, current as of mid-2026, with Spark 4.1.2 released on 21 May 2026 and the 4.0 maintenance line still active through Spark 4.0.3 (Apache Spark news and downloads, as of 2026-07-14).

The Spark Execution Model: Jobs, Stages, and Tasks

A Spark application is coordinated by a single process called the driver, which holds the program logic, and a set of executor processes distributed across the cluster, which perform the actual computation on partitions of data. A partition is the unit of parallelism — a contiguous slice of a dataset that a single task processes. The driver builds the plan; the executors run it.

Spark evaluates transformations lazily. Operations such as map, filter, select, and join do not trigger computation; they extend a logical description of the work. Computation begins only when an action — for example count, collect, or write — is invoked. At that moment the driver submits a job, and the DAG scheduler translates the accumulated transformations into an execution plan structured as a directed acyclic graph (DAG) of stages.

Narrow and Wide Dependencies

The DAG scheduler divides a job into stages by examining the dependencies between partitions. A narrow dependency is one in which each parent partition contributes to at most one child partition. Transformations such as map and filter are narrow: a task can process its input partition and produce its output partition without consulting any other partition. Narrow dependencies are pipelined — chained together and executed within a single stage without materializing intermediate results.

A wide dependency is one in which a child partition depends on multiple parent partitions. Transformations such as groupByKey, reduceByKey, join, distinct, and repartition are wide, because a given output key may be assembled from records scattered across every input partition. A wide dependency cannot be pipelined; it requires data to be redistributed across the cluster so that all records sharing a key land in the same partition. That redistribution is the shuffle, and every shuffle defines a stage boundary (Apache Spark documentation; SparkInternals, as of 2026-07-14).

Figure 1: A job is split into stages at each shuffle. Narrow operators are pipelined inside a stage; each stage runs one task per partition.

Within a stage, the unit of execution is the task. Spark launches one task per partition, and tasks within a stage run in parallel across the available executor cores. The number of tasks in a stage therefore equals the number of partitions, which is why partition count directly governs both parallelism and per-task workload. A stage with too few partitions underuses the cluster; a stage with too many partitions incurs scheduling overhead for tasks that each process a trivial amount of data.

Key Takeaway: The mental model that matters for performance is: an action triggers a job, a job is cut into stages at each shuffle, and a stage runs one task per partition. Everything expensive — network transfer, disk spill, serialization — happens at the stage boundaries.

The Catalyst Optimizer and Adaptive Query Execution

Before any task runs, the query passes through Catalyst, Spark SQL’s query optimizer. Catalyst is an extensible optimizer built on functional tree-transformation constructs in Scala. It represents a query as a tree of nodes and rewrites that tree by applying rules — pattern-matching functions that transform one tree into an equivalent, cheaper tree. The DataFrame and SQL APIs both compile down to the same Catalyst representation, so an SQL query and its DataFrame equivalent are optimized identically.

The Four Phases

Catalyst structures optimization in four phases (Databricks, “Deep Dive into Spark SQL’s Catalyst Optimizer,” 2015-04-13):

Analysis. The parser produces an unresolved logical plan in which column and table names are still symbolic. The analyzer resolves these references against the catalog — the registry of tables, columns, and their types — turning the tree into an analyzed logical plan with fully typed attributes.
Logical optimization. A set of rule-based rewrites is applied to reduce work. Named optimizations include predicate pushdown (moving filters as close to the data source as possible), projection pruning (reading only the columns a query needs), constant folding (evaluating constant expressions once at plan time), and join reordering. The output is an optimized logical plan.
Physical planning. Catalyst generates one or more physical plans that specify concrete operators — which join algorithm to use, how to exchange data — and selects among them by cost. This is the only cost-based phase; all others are purely rule-based.
Code generation. The selected physical plan is compiled, in part, to JVM bytecode for execution, a step handled by the Tungsten engine and discussed in the next section.

The design of Catalyst was first described in the academic literature by Armbrust and colleagues in “Spark SQL: Relational Data Processing in Spark,” presented at SIGMOD 2015. That paper introduced the tree-transformation framework that remains the foundation of Spark SQL a decade later.

Figure 2: Catalyst’s four phases. Predicate pushdown into Parquet scans and projection pruning are rule-based; join-algorithm selection is the cost-based decision.

Predicate pushdown and projection pruning are the phases where storage format matters most. When Spark reads a columnar file, it can skip entire column chunks and row groups that a query does not touch, so the physical layout of the source determines how much of Catalyst’s logical optimization can be realized as reduced I/O. The mechanics of that pushdown are covered in the companion discussion of Apache Parquet and Apache Arrow internals, which explains how row-group statistics and column pruning let the scan read only the bytes a query requires.

Adaptive Query Execution

Catalyst’s cost-based decisions historically relied on statistics gathered before execution — table sizes, column histograms — which are often stale, missing, or wrong for intermediate results. A join planned as a sort-merge because both sides looked large may, after a filter, involve one tiny side. Adaptive Query Execution (AQE) addresses this by re-optimizing the plan at runtime using statistics measured from completed shuffles. AQE has been enabled by default since Spark 3.2.0 through the configuration flag spark.sql.adaptive.enabled (change tracked in SPARK-33679); it existed but was off by default in Spark 3.0 and 3.1 (Spark SQL Performance Tuning documentation, as of 2026-07-14).

AQE has three headline capabilities. It dynamically coalesces shuffle partitions, merging contiguous small partitions after a shuffle so that a fixed partition count no longer produces hundreds of tiny tasks. It switches join strategies, upgrading a planned sort-merge join to a broadcast hash join once a shuffle reveals that one side is small enough. And it handles skew by splitting oversized partitions in a sort-merge join at runtime. Because AQE inserts these decisions between stages — precisely where actual data sizes become known — it corrects the very estimation errors that make static planning brittle.

Figure 3: AQE re-optimizes at each shuffle boundary — coalescing partitions, switching join strategies, and splitting skewed partitions using measured statistics.

Tip: Because AQE decisions depend on real shuffle output, examine a slow job in the Spark UI after it runs. The SQL tab shows the final adapted plan, including any “AdaptiveSparkPlan” node and whether a sort-merge join was converted to a broadcast join at runtime.

Tungsten and Whole-Stage Code Generation

Where Catalyst decides what to execute, Project Tungsten governs how efficiently each task runs on a single core. Tungsten began in Spark 1.6 as an initiative to bring Spark’s execution closer to bare-metal CPU and memory efficiency. Its stated goals were four: explicit off-heap memory management, cache-aware computation, code generation, and reduced virtual function calls (Databricks, “Project Tungsten,” 2015-04-28). The second generation of Tungsten, including whole-stage code generation, shipped in Spark 2.0 (Databricks, “Apache Spark as a Compiler,” 2016-05-23).

The UnsafeRow Binary Format

A conventional JVM program represents each row as a tree of Java objects, each carrying object headers, pointers, and padding, all managed by the garbage collector. For a dataset of billions of rows, that overhead dominates both memory consumption and CPU time spent in garbage collection. Tungsten replaces this with UnsafeRow, a compact binary layout in which a row is stored as a contiguous block of bytes. Because a Dataset has a known schema, Tungsten can lay each field out at a fixed offset and manage the backing memory off-heap — outside the region the garbage collector scans — sidestepping JVM object overhead and garbage-collection pressure entirely (The Internals of Spark SQL; Databricks, “Project Tungsten”).

Figure 4: UnsafeRow stores a row as a contiguous off-heap byte block with a null bitset, a fixed-length region, and a variable-length region, avoiding JVM object overhead.

Whole-Stage Code Generation

The classic way to execute a query plan is the Volcano iterator model, in which each operator implements a next() method and pulls rows one at a time from its child. This model is general but slow: every row crosses a virtual function call at every operator, intermediate values are boxed and materialized, and the CPU cannot keep data in registers. Whole-stage code generation (WSCG), tracked as SPARK-12795, collapses an entire chain of operators within a stage into a single JVM function. Instead of many operators each calling the next, Spark generates one tight loop that applies the whole chain of narrow operations to each row, eliminating virtual dispatch and keeping intermediate values in CPU registers rather than materializing rows. The generated Java source is compiled to bytecode at runtime by the Janino compiler (Databricks, “Apache Spark as a Compiler,” 2016-05-23; The Internals of Spark SQL).

Alongside WSCG, Spark 2.0 introduced vectorized columnar reads for Parquet (SPARK-12992), in which the reader decodes a batch of column values at a time rather than row by row, matching the columnar layout of the file to the batch-oriented execution engine. The scan and the compute path both operate on batches, which keeps the CPU’s instruction and data caches warm.

Figure 5: WSCG replaces a chain of Volcano-model operators, each with a virtual next() call, with a single generated loop that keeps intermediate values in registers.

Caution: Whole-stage code generation applies only to operators Spark can compile within a stage. User-defined functions written in Python break the generated pipeline because rows must cross into a separate Python process, and certain complex expressions fall back to interpreted execution. Reviewing the physical plan for a “WholeStageCodegen” wrapper confirms whether a section of the query is actually fused.

Databricks demonstrated the effect of these techniques with a benchmark headlined as joining “one billion rows per second on a laptop.” That figure is a specific micro-benchmark result rather than a general guarantee, and it should be read as an illustration of what fused execution can achieve on a favorable workload, not a number any arbitrary job will reach.

The Shuffle and Data Skew

The shuffle is the operation that redistributes data across the cluster so that all records sharing a key reside on the same partition. It is triggered by every wide dependency and is, in most jobs, the single largest consumer of time and resources, because it combines serialization, disk writes, network transfer, and disk reads. Spark has used a sort-based shuffle as the default since Spark 1.2, when spark.shuffle.manager=sort replaced the earlier hash-based shuffle. The SortShuffleManager is now the only shuffle manager in vanilla Spark; it dispatches to three internal write paths — BypassMergeSortShuffleHandle, SerializedShuffleHandle, and BaseShuffleHandle — depending on the number of partitions and whether map-side aggregation is needed (apache/spark source, SortShuffleManager.scala, as of 2026-07-14).

Map Side and Reduce Side

A shuffle has two halves. On the map side, each task in the upstream stage processes its input partition and assigns every output record to a target partition, determined by hashing the key. Records accumulate in an in-memory structure — a PartitionedAppendOnlyMap when aggregation is required — grouped by target partition. When the structure exhausts its memory budget, the task spills a sorted run to disk, sorting with TimSort. At the end of the task, the spilled runs and any remaining in-memory records are merged into a single shuffle file with an accompanying index that marks where each partition’s bytes begin.

On the reduce side, each task in the downstream stage fetches the byte ranges destined for its partition from every map output across the cluster, then merges those streams on the fly using a min-heap. This on-the-fly merge distinguishes Spark’s shuffle from Hadoop MapReduce, which materializes a fully merged file on disk before the reduce begins. The volume of data fetched during this phase, multiplied by network latency, is what makes the shuffle expensive.

Figure 6: The sort-based shuffle. Map tasks partition and spill sorted runs; reduce tasks fetch their partition from every map output and merge on the fly.

The number of reduce-side partitions is governed by spark.sql.shuffle.partitions, whose default is 200. This fixed default is frequently wrong for a given data size: for a small result it creates 200 tiny tasks with more scheduling overhead than work, and for a large result it creates 200 oversized tasks that spill repeatedly to disk (Spark 4.1.2 SQL Performance Tuning documentation, as of 2026-07-14). AQE’s partition coalescing exists precisely to correct the small-partition case automatically, merging contiguous partitions toward a target size rather than requiring the value to be tuned by hand for every job.

Data Skew

Data skew occurs when records are distributed unevenly across keys, so that a few partitions hold far more data than the rest. Because a stage completes only when its slowest task finishes, a single oversized partition can hold up an entire job while the cluster sits idle. Skew is the most common reason a stage that processes a moderate dataset runs for an unexpectedly long time. It typically arises from a highly frequent key — a null join key, a default value, or a dominant category.

Two mitigations are standard. The first is salting: a random suffix is appended to the skewed key so that its records spread across many partitions, the join or aggregation is performed on the salted key, and the results are combined afterward. Salting is explicit and always available, but it requires rewriting the query. The second is AQE skew-join handling, enabled by default through spark.sql.adaptive.skewJoin.enabled, which detects an oversized partition at runtime and splits it into sub-partitions automatically, replicating the matching side of the join. A partition is treated as skewed when it exceeds spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes — default 256 MB — and also exceeds a configurable factor times the median partition size.

Figure 7: A single hot key stalls a stage. Salting spreads the key manually; AQE splits the skewed partition automatically at the 256 MB threshold.

Join Strategies and Practical Performance Tuning

The join is where the cost of Catalyst’s physical planning, the shuffle, and AQE all converge. Spark chooses among three principal join algorithms, and the choice is governed largely by data size.

Broadcast, Sort-Merge, and Shuffle-Hash

A broadcast hash join is chosen when one side of the join is small — below spark.sql.autoBroadcastJoinThreshold, whose default is 10485760 bytes (10 MB). The small side is collected to the driver and broadcast to every executor, which builds a hash table from it in memory. The large side is then joined locally, partition by partition, with no shuffle of the large side at all. This is the cheapest join when it applies, because it avoids redistributing the large table.

A sort-merge join is the general default for two large inputs. Both sides are shuffle-partitioned on the join key and sorted, and the sorted partitions are then merged. It requires a full shuffle of both sides, which makes it expensive, but it scales to inputs far larger than memory because it never builds a full hash table.

A shuffle-hash join shuffles both sides on the join key but then builds an in-memory hash table on one side rather than sorting. AQE can convert a sort-merge join into a shuffle-hash join when all post-shuffle partitions are small enough to build hash tables locally, governed by spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold. AQE can also upgrade a planned sort-merge join to a broadcast hash join once the actual shuffle-output size of one side proves to be below the broadcast threshold (Spark 4.1.2 SQL Performance Tuning documentation, as of 2026-07-14).

Figure 8: Join-strategy selection. A side under the 10 MB threshold triggers a broadcast; otherwise Spark shuffles both sides and chooses shuffle-hash or sort-merge, with AQE able to switch at runtime.

The following table summarizes the trade-offs. A broadcast join is preferred whenever a side genuinely fits the threshold, because it eliminates the most expensive part of the join.

Strategy	Shuffles	When chosen	Main cost
Broadcast hash join	None on large side	One side < 10 MB (autoBroadcastJoinThreshold)	Broadcasting the small side; driver memory
Shuffle-hash join	Both sides	Post-shuffle partitions small enough to hash locally	Shuffle plus building a hash table in memory
Sort-merge join	Both sides	Two large inputs (the general default)	Shuffle plus sorting both sides

Configuration Defaults That Govern Performance

A small set of configuration values controls most of the behavior described above. The defaults below are those documented for the Spark 4.1.2 line (Spark SQL Performance Tuning documentation, as of 2026-07-14). Understanding them is more useful than memorizing them, because AQE now adjusts several of these dynamically.

Configuration	Default	Effect
`spark.sql.shuffle.partitions`	200	Number of post-shuffle partitions for joins and aggregations
`spark.sql.autoBroadcastJoinThreshold`	10 MB (10485760 B)	Size below which a side is broadcast instead of shuffled
`spark.sql.adaptive.enabled`	true (since 3.2.0)	Master switch for Adaptive Query Execution
`spark.sql.adaptive.coalescePartitions.enabled`	true	Merge small post-shuffle partitions at runtime
`spark.sql.adaptive.advisoryPartitionSizeInBytes`	64 MB	Target size for a coalesced partition
`spark.sql.adaptive.skewJoin.enabled`	true	Split skewed partitions in sort-merge joins at runtime
`spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes`	256 MB	Size above which a partition is treated as skewed

Practical Tuning Priorities

Several practical conclusions follow directly from the mechanics above. The first is that reducing the amount of data read is the cheapest optimization available, because bytes never read are bytes never shuffled. Storing data in a columnar format with row-group statistics lets Catalyst’s predicate pushdown and projection pruning skip most of the file before any compute begins; the same principle underlies the design of modern open table formats such as Iceberg, Delta Lake, and Hudi, which add partition pruning and file-level statistics on top of the columnar layer.

The second is that the shuffle should be minimized, not merely tuned. Filtering before a join, pre-aggregating where possible, and enabling broadcast joins for small dimension tables all reduce the volume of shuffled data. Where a shuffle is unavoidable, AQE’s partition coalescing removes most of the need to hand-tune spark.sql.shuffle.partitions, though a very large job may still benefit from a higher explicit value so that individual tasks do not spill.

The third is that skew deserves direct attention. Because a stage is bounded by its slowest task, a job that appears to under-use the cluster is often waiting on one hot partition. The Spark UI’s stage view, which shows the distribution of task durations and shuffle read sizes, is the fastest way to confirm skew before applying salting or verifying that AQE skew handling engaged.

Spark occupies a specific position in the broader landscape of data systems. It is a distributed, disk-spilling engine designed for datasets that exceed the memory of any single machine, which is a different design point from the single-node vectorized engines examined in the comparison of lakehouse query engines such as Trino, StarRocks, and DuckDB. It is also predominantly a batch and micro-batch engine rather than a low-latency streaming system, a distinction developed in the discussion of streaming versus batch processing architectures. In practice Spark is frequently the compute layer beneath a SQL-first transformation workflow, and the models produced by tools such as dbt for transformation pipelines often compile down to the very Catalyst plans and shuffles described here.

Tip: When diagnosing a slow job, read the physical plan with df.explain("formatted") and the Spark UI SQL tab together. The plan shows which join strategy and which code-generated stages were chosen; the UI shows where time and shuffled bytes were actually spent. The gap between the two is usually where the tuning opportunity lies.

Frequently Asked Questions

What is the difference between Catalyst and Tungsten in Spark?

Catalyst and Tungsten operate at different levels. Catalyst is the query optimizer: it decides what to execute by transforming a logical plan through analysis, rule-based logical optimization, and cost-based physical planning. Tungsten is the execution engine: it decides how efficiently each task runs on a CPU by generating fused machine-friendly code, storing rows in a compact off-heap binary format called UnsafeRow, and managing memory outside the garbage collector. Catalyst produces the plan; Tungsten executes it fast.

Why is the shuffle so expensive in Spark?

A shuffle redistributes data across the cluster so that all records sharing a key land on the same partition. It combines four costly operations: serializing records, spilling sorted runs to disk when memory is exhausted, transferring bytes across the network, and merging the fetched streams. Because it is triggered by every wide dependency and moves data proportional to dataset size, the shuffle is usually the dominant cost of a Spark job, which is why reducing the amount of shuffled data is the highest-value optimization.

Is Adaptive Query Execution enabled by default?

Yes. Adaptive Query Execution has been enabled by default since Spark 3.2.0 through the configuration flag spark.sql.adaptive.enabled. It was available but off by default in Spark 3.0 and 3.1. AQE re-optimizes a plan at runtime using statistics from completed shuffles, coalescing small partitions, switching a sort-merge join to a broadcast join when a side turns out small, and splitting skewed partitions automatically.

What does spark.sql.shuffle.partitions control, and why is 200 often wrong?

It sets the number of post-shuffle partitions for SQL and DataFrame joins and aggregations, and its default is 200. The value is fixed regardless of data size, so it is frequently a poor fit: for a small result it creates 200 tiny tasks whose scheduling overhead exceeds their work, and for a large result it creates 200 oversized tasks that spill to disk. AQE’s partition coalescing corrects the small-partition case automatically by merging contiguous partitions toward a target size, which reduces the need to tune the value by hand.

When does Spark use a broadcast join instead of a sort-merge join?

Spark chooses a broadcast hash join when one side of the join is smaller than spark.sql.autoBroadcastJoinThreshold, whose default is 10 MB. The small side is broadcast to every executor and the large side is joined locally without being shuffled, which is the cheapest join when it applies. Two large inputs default to a sort-merge join, which shuffles and sorts both sides. With AQE, a planned sort-merge join can be upgraded to a broadcast join at runtime once the actual size of one side is known to fall below the threshold.

Conclusion

Spark’s performance is best understood as the product of three cooperating layers. Catalyst decides the shape of the computation, rewriting a query through rule-based logical optimization and one cost-based physical-planning phase, and since Spark 3.2.0 it re-optimizes that plan at runtime through Adaptive Query Execution. Tungsten then executes each task efficiently, fusing operators into generated code that shipped in Spark 2.0 and storing rows in a compact off-heap format that avoids garbage-collection overhead. Between the stages sits the shuffle, the sort-based data-redistribution mechanism that has been the default since Spark 1.2 and that dominates the cost of most jobs.

For a practicing data engineer, the practical consequences are consistent. The largest gains come from reading less data, shuffling less data, and eliminating skew — in that order — because each addresses a cost that the engine cannot optimize away on its own. The configuration defaults matter less than the model behind them: once the boundary between narrow and wide dependencies, the role of the broadcast threshold, and the behavior of AQE are clear, the Spark UI and the physical plan become sufficient to diagnose almost any slow job. Reasoning about how the engine executes a query, rather than adjusting parameters in isolation, is what turns Spark performance work from guesswork into engineering.

References

Apache Spark, “Spark SQL Performance Tuning” (official documentation, latest / 4.1.x). spark.apache.org/docs/latest/sql-performance-tuning.html
Apache Spark, “News and Releases” (version verification, 4.1.2 / 4.0.3, mid-2026). spark.apache.org/news
Databricks, “Deep Dive into Spark SQL’s Catalyst Optimizer” (2015-04-13). databricks.com/blog
Databricks, “Project Tungsten: Bringing Apache Spark Closer to Bare Metal” (2015-04-28). databricks.com/blog
Databricks, “Apache Spark as a Compiler: Joining a Billion Rows per Second on a Laptop” (2016-05-23). databricks.com/blog
Armbrust et al., “Spark SQL: Relational Data Processing in Spark,” SIGMOD 2015 — the paper that introduced Catalyst.

July 14, 2026

Lakehouse Query Engines: Trino vs StarRocks vs DuckDB

A modern data lakehouse separates the data it stores from the software that computes over it. Table formats such as Apache Iceberg or Delta Lake define how records, partitions, and transactions are laid out as files on object storage, but they do not execute queries. That work belongs to a distinct tier: the query engine. Because storage and compute are decoupled in a lakehouse, an organization can place several different engines over the same files and choose among them per workload. This guide examines three query engines that occupy very different points in that design space — Trino, StarRocks, and DuckDB — and explains how their architectures, deployment models, pushdown behavior, and caching strategies determine which one fits a given use case. The comparison is deliberately confined to the compute tier; the underlying file and table formats are treated as a fixed substrate rather than re-derived here.

The engines were chosen because they represent three archetypes rather than three interchangeable products. Trino is a distributed, federated massively parallel processing (MPP) engine built for ad-hoc SQL across many sources. StarRocks is an MPP online analytical processing (OLAP) engine designed for interactive, high-concurrency serving on the lakehouse. DuckDB is a single-node, in-process engine that queries lakehouse tables directly from inside a host program without any cluster. Understanding why each exists, and where each stops being the right tool, is more durable than memorizing a feature list, because the storage-compute separation that makes the lakehouse attractive also makes engine selection a recurring decision rather than a one-time commitment.

Summary

What this post covers: A vendor-neutral comparison of three lakehouse query engines — Trino, StarRocks, and DuckDB — at the compute tier that reads Iceberg, Delta, and related table formats from object storage. It contrasts their architectures, deployment models, predicate and projection pushdown, caching layers, and the workloads each serves best.

Key insights:

The three engines are not substitutes but archetypes: Trino is a distributed federated MPP engine (coordinator plus workers) for ad-hoc cross-source SQL, StarRocks is an MPP OLAP engine (frontend, backend, and compute nodes) for interactive high-concurrency serving, and DuckDB is a single-node in-process engine for embedded analytics.
Trino decomposes a query into stages, tasks, splits, and operators across a cluster and joins many catalogs at once, which makes it strong at federation but reliant on a file-system or Alluxio cache to approach interactive latency over object storage.
StarRocks accelerates repeated lakehouse queries with a cost-based optimizer, materialized views that transparently rewrite queries, and a block-level Data Cache; its shared-data mode separates storage and compute while retaining a local cache.
DuckDB reads Iceberg, Delta, and DuckLake through native extensions that support filter pushdown, but it is bounded by a single machine’s memory and cores and performs no distributed shuffle.
DuckLake stores lakehouse metadata in a SQL catalog database rather than in scattered files on object storage, illustrating a broader shift in where the engine-catalog boundary is drawn.
All vendor speed figures cited are self-reported and workload-dependent; no neutral head-to-head benchmark fairly ranking all three was available, so engine performance is presented qualitatively.

Main topics: The Query-Engine Tier of the Lakehouse; Trino: Federated Distributed MPP; StarRocks: MPP OLAP Serving on the Lakehouse; DuckDB and DuckLake: In-Process Lakehouse Analytics; Pushdown and Caching: How Engines Avoid Reading Data; Choosing an Engine.

The Query-Engine Tier of the Lakehouse

A lakehouse is a data architecture that places a transactional table layer over inexpensive object storage so that a single copy of the data serves both exploratory analytics and production reporting. Its defining property is the separation of storage from compute. Data files, most often Apache Parquet, sit in an object store such as Amazon S3. A table format layered on top of those files records which files belong to a table, how they are partitioned, and which version of the table is current. Query engines then read that table format and execute SQL. Because no engine owns the data, several can read the same tables concurrently, and an organization can add or replace an engine without migrating a single byte.

This decoupling is what distinguishes a lakehouse from a classical data warehouse, in which storage and compute are fused inside one proprietary system. The trade-off is that reading data now crosses a network boundary to object storage, which has higher and more variable latency than a local disk. Much of what differentiates the engines in this comparison is how they cope with that boundary: how aggressively they avoid reading files at all, and how they cache the files they must read. The table format itself is out of scope here; readers who need the storage-substrate details can consult the companion analysis of Iceberg, Delta Lake, and Hudi table formats, and the treatment of the file layer in the guide to Parquet and Apache Arrow columnar storage.

Two mechanisms recur throughout the comparison and are worth defining once. Predicate pushdown is the practice of pushing a query’s filter conditions down to the storage layer so that files, partitions, or row groups that cannot match are never read. Projection pushdown reads only the columns a query references, which is efficient because Parquet stores data column by column. Both reduce the volume of bytes crossing the network boundary, and the degree to which an engine exploits them is a primary determinant of its latency over object storage.

Key Takeaway: Because a lakehouse separates storage from compute, the query engine is a swappable component chosen per workload rather than a fixed part of the platform. The engines differ most in how they hide the latency of the object-storage boundary, through pushdown and caching, and in whether they scale by adding machines or run inside a single one.

Trino: Federated Distributed MPP

Trino is a distributed SQL query engine designed for interactive and ad-hoc analytics across large and heterogeneous datasets. It follows the massively parallel processing model, meaning a query is divided into parallel units of work that execute simultaneously across many machines. Trino uses date-independent sequential release numbers and ships frequently; the latest release at the time of writing is Trino 482, published on 2026-06-25 (github.com/trinodb/trino/releases; trino.io, as of 2026-07-11). Its defining characteristic is federation: a single Trino cluster can query many different data sources in one statement, joining a lakehouse table against a relational database without moving either.

Coordinator and Workers

A Trino cluster consists of one coordinator and zero or more workers. The coordinator is the brain: it receives SQL statements from clients, parses and analyzes them, builds a distributed execution plan, and schedules and monitors the work assigned to the workers. Workers execute the tasks they are given and process the underlying data. Each node runs as a single Java Virtual Machine (JVM) process and achieves parallelism through many threads within that process. A cluster with more workers can process more data concurrently, which is how Trino scales: an operator adds worker nodes to raise throughput rather than replacing the cluster with a larger one (trino.io/docs/current/overview/concepts.html, as of 2026-07-11).

The internal decomposition of a query is precise and worth following, because it explains both Trino’s parallelism and its memory behavior. A client submits a statement, the text of the SQL. Trino turns it into a query, the running instance of that statement. The query plan is a tree of stages, each of which is realized as a set of tasks distributed across workers. Each task processes splits, which are addressable sections of the input data, using a pipeline of drivers and operators that apply the actual relational logic. Data moves between stages through exchanges. This structure is the classic distributed, federated MPP execution model, and it allows a single large scan to be spread across every worker in the cluster.

Connectors and Catalogs

Trino’s federation rests on two concepts. A connector adapts Trino to a particular kind of data source; connectors exist for Hive, Delta Lake, Iceberg, PostgreSQL, and many other systems. A catalog is a configured instance of a connector pointing at a specific data source. Multiple catalogs coexist in one cluster, and because a query can reference tables from more than one catalog, Trino can join across sources in a single statement. The namespace is three levels deep: catalog, then schema, then table (trino.io/docs/current/overview/concepts.html, as of 2026-07-11). A catalog is defined by a small properties file. The following configures an Iceberg catalog backed by a REST catalog service and S3 storage.

# etc/catalog/lakehouse.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://catalog.example.com/api/catalog
fs.native-s3.enabled=true
s3.region=us-east-1
# turn on the local file-system cache for this catalog
fs.cache.enabled=true
fs.cache.directories=/mnt/trino-cache

With this catalog registered, a query such as SELECT * FROM lakehouse.sales.orders o JOIN postgres.crm.customers c ON o.cust_id = c.id reads the orders table from Iceberg on S3 and joins it against a live PostgreSQL table, with the join executed in the Trino cluster. This is the federation capability that makes Trino a shared query gateway over a mixture of lakehouse tables and operational databases. It also positions Trino as a convenient execution engine beneath a transformation framework; a project that models tables with the dbt transformation tool can compile its SQL against a Trino target and reach every registered catalog through one connection.

Pushdown and Caching in Trino

For lakehouse tables, Trino’s Iceberg connector applies predicate pushdown to drive partition and file pruning, so that a filtered query skips partitions and data files that cannot satisfy the condition, and projection pushdown to read only the referenced columns. Vectorized readers handle both Parquet and ORC file formats (Trino Iceberg connector documentation, trino.io, as of 2026-07-11). Pushdown reduces work but does not remove the cost of repeatedly fetching the same files from object storage. To address that, Trino provides a file-system cache for the Delta Lake, Hive, and Iceberg connectors, with Hudi support noted as forthcoming, and native support for Alluxio as a distributed caching layer in front of object storage, enabled with fs.alluxio.enabled=true. The older Rubix caching path has been superseded by the native file-system cache (trino.io/docs/current/object-storage/file-system-cache.html and file-system-alluxio.html, as of 2026-07-11).

Caution: Without a warm file-system or Alluxio cache, a Trino query over object storage pays the full network cost of every file it reads. Trino excels at federated and exploratory workloads, but treating a cold-cache Trino cluster as an always-on low-latency dashboard backend tends to disappoint. Interactive serving generally requires the caching layer to be provisioned and kept warm, or a purpose-built serving engine.

StarRocks: MPP OLAP Serving on the Lakehouse

StarRocks is an MPP OLAP engine oriented toward interactive analytics: sub-second dashboards, high-concurrency BI, and repeated queries over lakehouse tables. It is a Linux Foundation project (github.com/StarRocks/starrocks, as of 2026-07-11). The latest major release is StarRocks 4.0, released on 2025-10-17. The vendor reports roughly 60 percent faster query execution year over year on the TPC-DS benchmark for 4.0; this is a self-reported figure and should be read alongside the benchmark caveat below (starrocks.io/blog/starrocks-2025-year-in-review, as of 2026-07-11). Where Trino optimizes for breadth of sources, StarRocks optimizes for the speed and concurrency of repeated queries over a smaller set of well-modeled tables.

Frontend, Backend, and Compute Nodes

StarRocks also follows the MPP model but uses two primary node types. The Frontend (FE) manages metadata, handles client connections, parses SQL, and performs query planning and scheduling. The Backend (BE) stores data and executes the query fragments assigned to it against its local storage. A third node type, the Compute Node (CN), is effectively a Backend without attached storage, used to separate compute from storage (docs.starrocks.io/docs/introduction/Architecture/, as of 2026-07-11). The division of labor resembles Trino’s coordinator-and-worker split, but StarRocks integrates a storage role into its Backend nodes, which is central to its two deployment modes.

In shared-nothing mode, Backends hold table data on their own local storage, which yields the lowest read latency but couples storage capacity to compute capacity. In shared-data mode, the Frontend coordinates a fleet of stateless Compute Nodes while the authoritative data lives on object storage or HDFS, with a local cache smoothing access. Shared-data mode is StarRocks’ answer to storage-compute separation: it retains the elasticity of a lakehouse, in which compute can scale independently, while keeping a warm local cache so that repeated queries do not pay full object-storage latency each time (docs.starrocks.io/docs/introduction/Architecture/, as of 2026-07-11).

Optimizer, Materialized Views, and Data Cache

StarRocks pairs a fully vectorized execution engine and columnar storage with a cost-based optimizer (CBO), which estimates the cost of alternative execution plans and chooses the cheapest rather than following the query’s written order (starrocks.io/blog/introduction_to_starrocks, as of 2026-07-11). On top of this sits the feature most responsible for its interactive performance on the lakehouse: materialized views (MVs). A materialized view is a precomputed, stored result of a query that the engine can substitute for the original computation. StarRocks supports multi-column partitioned MVs that align with Iceberg or Hive partitions, which allows incremental refresh — only the changed partitions are recomputed rather than the whole view. The cost-based optimizer performs transparent query rewriting, redirecting an incoming query to an eligible MV without the author referencing it. The vendor reports large MV-driven speedups on eligible queries — by an order of magnitude or more; this is a self-reported figure (starrocks.io/blog/starrocks-2025-year-in-review; docs.starrocks.io async_mv documentation, as of 2026-07-11).

An external catalog connects StarRocks to a lakehouse without ingesting the data, and a materialized view built on that catalog becomes the acceleration layer. The following sketch registers an Iceberg catalog and defines an MV that the optimizer can transparently rewrite queries onto.

-- Register the lakehouse as an external catalog
CREATE EXTERNAL CATALOG iceberg_cat
PROPERTIES (
  "type" = "iceberg",
  "iceberg.catalog.type" = "rest",
  "iceberg.catalog.uri" = "https://catalog.example.com/api/catalog"
);

-- Precompute a daily rollup, partition-aligned for incremental refresh
CREATE MATERIALIZED VIEW sales_daily
PARTITION BY order_date
REFRESH ASYNC
AS
SELECT order_date, region, sum(amount) AS revenue
FROM iceberg_cat.sales.orders
GROUP BY order_date, region;
-- Queries that match this pattern are rewritten onto the MV by the CBO

Beneath the MV layer, StarRocks provides a Data Cache that caches blocks of remote files locally to smooth the latency and jitter of object storage. StarRocks 4.0 added metadata caching, compaction, and file bundling that the vendor reports reduce cloud API calls by up to 90 percent, and strengthened Iceberg support with hidden-partition handling, faster metadata parsing, a new compaction API, and native Iceberg table writes; 4.0 also introduced JSON as a first-class type, reported at 3 to 15 times faster on JSON queries. These are self-reported figures (starrocks.io/blog/starrocks-2025-year-in-review, as of 2026-07-11).

Tip: When a dashboard issues the same aggregate repeatedly, a partition-aligned materialized view with incremental refresh usually delivers a larger and more reliable speedup than tuning raw scans, because it turns a recurring scan-and-aggregate into a lookup. Combine the MV layer with the Data Cache so that the queries that still miss the MV also avoid full object-storage latency.

DuckDB and DuckLake: In-Process Lakehouse Analytics

DuckDB occupies the opposite end of the spectrum from Trino and StarRocks. It is a single-node, in-process, vectorized analytical engine — often described as SQLite for analytics — that runs inside the host process rather than as a separate cluster. There is no coordinator, no worker fleet, and no network protocol between the application and the engine; a query executes in the same process as the program that issued it. The current stable line is DuckDB 1.5.x, with v1.5.3 shipping on 2026-05-20 (duckdb.org, as of 2026-07-11). This design removes distributed-systems overhead entirely, at the cost of being bounded by one machine’s memory and cores, with no distributed shuffle. DuckDB’s role as a general in-process SQL and dataframe engine is examined in the companion comparison of DuckDB and Polars for in-process analytics; the concern here is narrower — its role as a lakehouse engine.

Lakehouse Formats as First-Class Citizens

DuckDB reads and writes lakehouse tables through native extensions rather than third-party libraries. The iceberg extension handles Apache Iceberg, the delta extension handles Delta Lake, the ducklake extension handles DuckLake, and Lance is also supported. Implementing these as native extensions rather than external bindings lets DuckDB apply complex filter pushdowns and manage memory carefully during scans (duckdb.org/docs/current/lakehouse_formats, as of 2026-07-11). The Iceberg extension can also connect to Iceberg REST catalogs, such as AWS Glue, Unity-style, or Lakekeeper services, so that DuckDB resolves tables through the same catalog a cluster engine would use (duckdb.org iceberg REST catalog documentation, as of 2026-07-11). Version 1.5.3 extended the Iceberg support with full MERGE INTO against Iceberg and bucket and truncate partition transforms.

Querying an Iceberg table from DuckDB requires only loading the extension and pointing a scan at the table. The engine then applies projection and predicate pushdown to the underlying Parquet files.

INSTALL iceberg;
LOAD iceberg;

-- Attach a REST catalog, then scan an Iceberg table directly
ATTACH 'warehouse' AS lake (
  TYPE iceberg,
  ENDPOINT 'https://catalog.example.com/api/catalog'
);

SELECT region, sum(amount) AS revenue
FROM lake.sales.orders
WHERE order_date >= DATE '2026-07-01'   -- predicate pushed to file pruning
GROUP BY region;                        -- only 3 columns read (projection)

This model is well matched to a specific set of situations: a developer’s laptop, a notebook exploring a dataset, a continuous-integration job validating a data transformation, a small serverless function, and small-to-medium datasets generally. In each case the value is the absence of a cluster to provision and pay for. DuckDB can query an Iceberg or Delta table sitting in S3 directly, from a script, with no distributed infrastructure in between. It is not a replacement for a distributed engine on terabyte-scale joins, because a single machine cannot shuffle data across a fleet, but for a large share of everyday analytical questions the data fits and the cluster is unnecessary.

DuckLake and the Metadata-in-a-Database Shift

DuckLake is a lakehouse format that relocates where table metadata lives. In the prevailing design, exemplified by Iceberg, a table’s metadata — its snapshots, manifests, and file lists — is stored as a chain of files on the same object storage as the data. DuckLake instead stores all of that lakehouse metadata in a SQL catalog database, which may be SQLite, PostgreSQL, or DuckDB, while the data files remain Parquet on object storage (ducklake.select/2026/04/13/ducklake-10; ducklake.select/faq, as of 2026-07-11). The motivation is that many metadata operations — listing snapshots, resolving the current table version, planning which files to read — are exactly the transactional lookups that relational databases perform efficiently, whereas doing them through a tree of small files on object storage incurs many round trips.

DuckLake v1.0, described as production-ready with a backward-compatibility guarantee, was released on 2026-04-13, with a v1.1 specification expected in September 2026 (ducklake.select/2026/04/13/ducklake-10, as of 2026-07-11). Notably, the interoperability path runs through Iceberg: DuckDB’s Iceberg extension can perform a metadata-only copy of Iceberg metadata into DuckLake, after which the same underlying data files can be queried as DuckLake tables. DuckLake is treated here as a supporting anchor rather than a fourth peer engine, because its significance is architectural: it illustrates a broader movement of the catalog boundary from files on object storage into a database, which reshapes how any engine discovers and plans over lakehouse tables.

Pushdown and Caching: How Engines Avoid Reading Data

All three engines share the same fundamental challenge — object storage is slow relative to local memory — and their answers, though implemented differently, follow the same two principles. The first is to avoid reading data that a query cannot use, through pushdown. The second is to avoid re-reading data that has already been fetched, through caching. Understanding these two mechanisms clarifies why cache-warm and cache-cold performance can differ by a large margin, and why an engine’s steady-state behavior matters more than its first-query latency.

The Pushdown Flow

Pushdown proceeds in layers, and each layer that eliminates data spares every layer below it. A query’s filter first prunes partitions using the table format’s partition metadata, so that whole directories of files are skipped. Within the surviving partitions, file-level statistics eliminate individual data files whose value ranges cannot match the filter. Within a file that must be opened, Parquet’s row-group statistics allow the reader to skip row groups, and projection pushdown ensures that only the referenced columns are decoded. What reaches the engine’s operators is therefore a small fraction of the table. The detail of what these statistics contain, and how row groups and encodings are laid out, belongs to the Parquet and Arrow internals discussion; the point here is that all three engines depend on this same cascade, which is why the table and file format the data lands in — often the output of a pipeline such as the one described in the InfluxDB-to-Iceberg data pipeline — directly governs how effective any engine’s pushdown can be.

Caching Layers Compared

Pushdown reduces how much data a query reads; caching reduces how often that reading crosses the network. The three engines place a cache in front of object storage in structurally similar ways but with different scopes. Trino uses a native file-system cache for its Delta Lake, Hive, and Iceberg connectors, and can front object storage with Alluxio as a distributed cache. StarRocks provides a block-level Data Cache in its Compute Nodes and Backends, complemented in 4.0 by metadata caching that reduces cloud API calls, and its shared-data mode is explicitly designed to keep a warm local cache over remote data. DuckDB, running in a single process, benefits from operating-system page caching and its own buffer management, and reads through native extensions tuned for memory efficiency, but it has no distributed cache tier because it has no distribution.

Key Takeaway: Every engine here relies on the same two levers — pushdown to read less, caching to re-read less — but at different scopes. Trino and StarRocks maintain explicit, configurable cache tiers because they front object storage from a cluster; DuckDB relies on process and operating-system caching because it runs in one process. This is why performance claims must always specify cache state, and why cold-start latency is a poor proxy for steady-state behavior.

Choosing an Engine

The three engines map cleanly onto three families of workload, and the selection is usually determined by the shape of the workload rather than by raw speed. The following table summarizes the architectural properties that drive the decision. All performance characterizations are qualitative, in keeping with the benchmark caveat stated below.

Property	Trino 482	StarRocks 4.0	DuckDB 1.5.x
Model	Distributed MPP	Distributed MPP OLAP	Single-node in-process
Node types	Coordinator + workers	FE + BE + CN	None (embedded)
Scaling	Add workers	Add BE / CN nodes	Bounded by one machine
Federation	Many catalogs, cross-source	External catalogs	Format extensions + REST catalog
Acceleration	FS cache, Alluxio	CBO, MVs, Data Cache	Native pushdown, process cache
Storage-compute split	Fully decoupled	Optional (shared-data)	Reads remote storage directly
Best fit	Ad-hoc federated SQL	Interactive BI serving	Embedded / notebook analytics

When Each Engine Fits

Trino is the appropriate choice when the requirement is heterogeneous, federated ad-hoc SQL across many sources: exploring a data lake, or standing up a shared query gateway over Iceberg, Delta, and Hive tables alongside relational and NoSQL catalogs. Its storage and compute are fully decoupled, and it scales by adding workers. It is weaker as an always-on, low-latency serving layer unless paired with a warm file-system or Alluxio cache, so a team that needs a single engine for wide-ranging exploration across systems is its natural user.

StarRocks is the appropriate choice for interactive, sub-second BI and dashboards on the lakehouse, particularly under high concurrency. Its cost-based optimizer, materialized views, and Data Cache accelerate repeated queries over Iceberg and Hive tables, and its shared-data mode preserves storage-compute separation while keeping a warm local cache. A team serving many concurrent dashboard users from lakehouse tables, where the same aggregates recur, is its natural user. Interactive serving is a different problem from batch pipelines; the boundary between low-latency serving engines and batch processing is examined in the guide to streaming versus batch data-processing architectures.

DuckDB is the appropriate choice for single-node embedded analytics: developer laptops, notebooks, continuous-integration jobs, small-to-medium datasets, and serverless functions that need to query Iceberg, Delta, or DuckLake tables directly without a cluster. It is bounded by one machine’s memory and cores and performs no distributed shuffle, so it is not a candidate for very large distributed joins, but for the large fraction of analytical work that fits on one machine it removes the operational cost of a cluster entirely.

These regions are not mutually exclusive in practice. A single organization commonly runs more than one: DuckDB in development and CI, Trino as a federated exploration gateway, and StarRocks behind the production dashboards. Because all three read the same lakehouse tables, using several is a matter of pointing each at the same object storage rather than maintaining separate copies of the data, which is precisely the flexibility the storage-compute separation was intended to provide.

Caution — benchmark caveat: All cross-engine speed figures cited by vendors, including StarRocks’ TPC-DS and materialized-view numbers, are self-reported and workload-dependent. TPC-H and TPC-DS results vary widely with schema, hardware, caching state, and query mix, and no independent neutral head-to-head benchmark fairly ranking all three engines was available as of writing. Engine performance should be reasoned about qualitatively — distributed scale-out versus single node, cache-warm versus cold, materialized-view-accelerated versus raw scan — and any number attributed to its source and date. Head-to-head latency and throughput figures should not be fabricated.

Frequently Asked Questions

Can Trino, StarRocks, and DuckDB query the same Iceberg tables at once?

Yes. Because a lakehouse separates storage from compute, the Iceberg tables live in object storage independently of any engine, and all three can read them concurrently. Trino uses its Iceberg connector, StarRocks uses an external catalog, and DuckDB uses its native Iceberg extension, optionally resolving tables through a shared REST catalog. No engine owns the data, so adding one is a configuration change rather than a data migration.

Is DuckDB a replacement for Trino or StarRocks?

Not for distributed workloads. DuckDB runs in a single process and is bounded by one machine’s memory and cores, with no distributed shuffle, so it cannot spread a large join across a cluster the way Trino or StarRocks can. For datasets that fit on one machine — a common case in development, notebooks, and continuous integration — it removes the need for a cluster entirely and queries lakehouse tables directly, which is a genuine substitute for a cluster in those situations but not in large distributed ones.

Why does the same query run much faster the second time?

Because of caching. On the first execution, an engine fetches data files from object storage across the network. Trino can retain them in its file-system or Alluxio cache, StarRocks in its block-level Data Cache, and DuckDB in process and operating-system buffers. Subsequent queries that touch the same files read them from the local cache instead of object storage, which is substantially faster. This is why a benchmark must state whether the cache was warm or cold; cold-start latency is not representative of steady-state behavior.

What problem does DuckLake solve compared with Iceberg?

DuckLake stores lakehouse metadata — snapshots, manifests, and file lists — in a SQL catalog database such as SQLite, PostgreSQL, or DuckDB, rather than as a chain of small files on object storage, while keeping the data files as Parquet on object storage. Metadata operations such as resolving the current table version become transactional database lookups instead of multiple object-storage round trips. DuckLake v1.0, released on 2026-04-13, is described as production-ready, and Iceberg metadata can be copied into DuckLake so the same data files are queryable as DuckLake tables.

How do materialized views make StarRocks fast on the lakehouse?

A materialized view is a precomputed, stored result of a query. StarRocks supports partition-aligned materialized views over Iceberg and Hive tables that refresh incrementally, recomputing only the changed partitions, and its cost-based optimizer transparently rewrites an incoming query onto an eligible view without the author referencing it. A recurring scan-and-aggregate becomes a lookup. The vendor reports large speedups on eligible queries, by an order of magnitude or more, which is a self-reported and workload-dependent figure.

References

Trino. Trino Concepts: Architecture, Coordinator, Workers, Connectors, and Catalogs. Trino Software Foundation, as of 2026-07-11.
Trino. File System Caching for Object Storage Connectors and Alluxio file-system support. Trino Software Foundation, as of 2026-07-11.
StarRocks. StarRocks Architecture: Frontend, Backend, and Compute Nodes. Linux Foundation, as of 2026-07-11.
StarRocks. StarRocks 2025 Year in Review (4.0 release, materialized views, Data Cache). StarRocks, 2025 (vendor self-reported figures), as of 2026-07-11.
DuckDB. Lakehouse Formats: Iceberg, Delta, DuckLake Extensions and New Iceberg Features in v1.5.3. DuckDB, as of 2026-07-11.
DuckLake. Announcing DuckLake v1.0. DuckLake, 2026-04-13, as of 2026-07-11.

Conclusion

Trino, StarRocks, and DuckDB are not three answers to the same question but three answers to three different questions, unified only by the lakehouse substrate they read. Trino’s coordinator-and-worker MPP architecture and its many-catalog federation make it a strong general engine for ad-hoc SQL across heterogeneous sources, provided a caching layer is added when low latency matters. StarRocks’ frontend, backend, and compute-node design, combined with a cost-based optimizer, transparently rewritten materialized views, and a block-level Data Cache, targets interactive high-concurrency serving on the lakehouse, with a shared-data mode that separates storage from compute while retaining a warm local cache. DuckDB’s single-node, in-process model, with native Iceberg, Delta, and DuckLake extensions, brings lakehouse analytics to a laptop, a notebook, or a serverless function without any cluster, at the cost of being bounded by one machine.

The most durable guidance is to treat the engine as a component chosen per workload rather than a platform-wide commitment. Because storage and compute are separated, an organization can run all three over the same tables, matching the engine to the shape of each workload — embedded, federated, or interactive — rather than forcing every query through one engine. The emergence of DuckLake, which moves lakehouse metadata into a SQL catalog database, is a reminder that even the boundary between engine and catalog is still being redrawn, and that the compute tier of the lakehouse remains one of the more active areas of data-engineering design. An engineer who understands each engine’s architecture and its cache and pushdown behavior, rather than a single benchmark number, is positioned to make that selection correctly as the tooling continues to evolve.

July 11, 2026

Streaming vs Batch Processing Architectures: A Data Engineer’s Guide

Every data platform eventually confronts a single design question that shapes its cost, its latency, and its operational burden: should a given computation run over a finite dataset on a schedule, or continuously over an endless flow of events? Batch processing computes over bounded data in periodic runs, while stream processing computes incrementally over an unbounded sequence of events as they arrive. The two paradigms are often presented as rivals, but they are better understood as points on a single continuum that trades data freshness against cost, throughput, and correctness machinery. This guide provides a mechanistic, vendor-neutral model of that continuum for engineers who already operate pipelines and now need a rigorous basis for deciding where each new workload belongs.

The discussion covers the batch-versus-streaming trade-off surface, the Lambda and Kappa architecture debate, the execution split between micro-batch and event-at-a-time engines, the correctness semantics that make streaming trustworthy (event time, watermarks, windowing, and exactly-once processing), an honest account of where batch remains the correct choice, and the convergence of the two paradigms observed across 2026 toward unified engines and the streaming lakehouse.

Summary

What this post covers: A vendor-neutral comparison of batch and stream processing architectures for data engineering, including the Lambda-versus-Kappa debate, the micro-batch-versus-event-at-a-time execution split, and the correctness semantics (event time, watermarks, windowing, exactly-once) that streaming systems depend on.

Key insights:

Batch is a special case of streaming: a batch job is a stream computation over a bounded input placed in a single global window, a reframing formalized by the Google Dataflow model.
Lambda architecture pays for low latency with two parallel codebases that must be reconciled, while Kappa removes the batch layer by treating reprocessing as a replay of a durable, retained log such as Apache Kafka.
Micro-batch engines such as Spark Structured Streaming trade a latency floor near 100 milliseconds for simple recovery and high throughput, whereas event-at-a-time engines such as Apache Flink process each record on arrival for lower latency and a richer time and state model.
Correct windowed aggregation requires event time rather than processing time, and watermarks are the mechanism by which a system decides that an event-time window is complete enough to emit.
Exactly-once processing in Flink is achieved through asynchronous barrier snapshotting, and end-to-end exactly-once delivery to external sinks additionally requires a two-phase commit; the choice between at-least-once and exactly-once is a cost decision, not a purely technical one.
Batch remains the correct choice for large historical backfills, full-refresh reproducibility, cost-sensitive periodic reporting, and machine-learning training sets, and the 2026 convergence toward unified engines and streaming lakehouse table formats is collapsing the two-store split rather than eliminating either paradigm.

Main topics: Two Ways to Compute Over Data; Lambda and Kappa: The Architecture Debate; Execution Models: Micro-Batch and Event-at-a-Time; Getting Streaming Correct: Time, Watermarks, Windows, and Semantics; When Batch Remains the Right Choice, and the Convergence.

Two Ways to Compute Over Data

The distinction between batch and streaming begins with the shape of the input. A bounded dataset is finite and complete: yesterday’s transactions, a snapshot of a table, a directory of log files. An unbounded dataset is an endless sequence of events that has no defined end, such as clickstream events, sensor readings, or a change-data-capture feed from a production database. Batch processing operates over bounded data in scheduled runs, producing a result after it has observed the entire input. Stream processing operates over unbounded data continuously, updating results incrementally as each event arrives.

This difference in input shape propagates into every operational property of a pipeline. The most useful way to reason about the choice is as a trade-off surface with three axes: latency, throughput, and cost. Latency is the delay between an event occurring and its effect appearing in a result. Throughput is the volume of records the system processes per unit of time. Cost is the compute and operational expense of running the system. These three cannot be optimized independently. Lowering latency toward the millisecond range generally requires always-on compute and additional correctness machinery, which raises both cost and operational complexity. Tolerating higher latency allows work to be amortized into scheduled bursts, which lowers steady-state cost and simplifies recovery.

A central insight, formalized by the Google Dataflow model (Akidau et al., The Dataflow Model, PVLDB 2015), dissolves the apparent opposition between the two paradigms. In that framework, a batch job is simply a stream computation over a bounded input assigned to a single global window that closes when the input is exhausted. Streaming generalizes batch rather than replacing it: the same logical operations (filtering, joining, aggregating) apply in both cases, and the differences reduce to when results are emitted and how completeness is judged. This reframing is more than a rhetorical convenience, because it underpins the unified engines discussed later, in which one API expresses both bounded and unbounded computation.

The following table summarizes how the paradigm choice affects the operational properties an engineer must plan around.

Dimension	Batch	Streaming
Input	Bounded, complete dataset	Unbounded event sequence
Typical latency	Minutes to hours	Milliseconds to seconds
Throughput profile	High, bursty	High, steady
Cost profile	Amortized, scheduled	Always-on, continuous
Reproducibility	Simple full re-run	Needs replay plus retained state
Operational complexity	Lower	Higher
Representative tools	Airflow, Spark batch, dbt	Flink, Spark Structured Streaming, Kafka

Key Takeaway: The batch-versus-streaming decision is not a choice between two technologies but a position on a latency-throughput-cost surface. Because a batch job is a streaming job over a single global window, the same engine and even the same code can increasingly express both, which shifts the question from “which paradigm” to “how fresh must this result be, and what is that freshness worth.”

Lambda and Kappa: The Architecture Debate

Once an organization needs both historical accuracy and low-latency views, it confronts an architectural question that predates modern unified engines. The two canonical answers are the Lambda architecture and the Kappa architecture, and understanding their motivations clarifies design decisions that remain relevant even where neither is adopted by name.

The Lambda Architecture

The Lambda architecture was described by Nathan Marz, the creator of Apache Storm, around 2011 and later formalized with James Warren in the book Big Data (Manning, 2015). It composes three layers. The batch layer holds an immutable master dataset and precomputes comprehensive batch views over all historical data, prioritizing accuracy and completeness. The speed layer processes only recent data with low latency, producing approximate or incremental real-time views that compensate for the batch layer’s delay. The serving layer indexes and merges the outputs of both layers so that a query sees a combined result: authoritative history from the batch layer plus the most recent events from the speed layer.

The Lambda architecture achieves both accuracy and freshness, but at a well-known cost: the same business logic must be implemented twice, once in a batch engine and once in a stream engine, and the two implementations must be kept semantically equivalent. Any divergence between them produces inconsistent results at the serving layer, and every change to the computation must be applied and validated in both places. This dual-codebase reconciliation burden is the defining pain point of the pattern.

The Kappa Architecture

Jay Kreps, a co-creator of Apache Kafka, proposed an alternative in the 2014 essay Questioning the Lambda Architecture (O’Reilly Radar). The argument is direct: if a stream processor is sufficiently expressive and reliable, the separate batch layer is redundant. The Kappa architecture keeps a single streaming layer and backs it with a durable, replayable log, typically Kafka, that retains the raw event history. There are no longer two code paths. When the computation logic changes, or a bug is fixed, “reprocessing” means starting a new instance of the streaming job from the beginning of the retained log and letting it recompute the output, then switching consumers to the new result. The nightly recompute of the batch layer is replaced by a replay of the same code that serves live traffic.

The Kappa architecture depends on two properties of its log. The log must be durable and retained long enough to replay whatever history a recomputation requires, and it must preserve ordering per partition so that replay is deterministic. Kafka provides both, which is why it sits at the center of most Kappa deployments. Implementing a reliable consumer over that log is itself a substantial task; the mechanics of offset management, consumer groups, and rebalancing are treated in this guide to implementing a Kafka consumer in Python. The events feeding such a log frequently originate from operational databases through change-data-capture, in which row-level changes are streamed as events; the change-data-capture pattern with Debezium and Kafka is a canonical source that makes Kappa practical for database-derived data.

Aspect	Lambda	Kappa
Code paths	Two (batch + speed)	One (streaming)
Reprocessing	Batch recompute	Replay the retained log
Storage	Batch store + speed store	Single replayable log
Reconciliation	Serving-layer merge	None; single path
Operational burden	Higher	Lower
Best fit	Reconciliation- and audit-heavy domains	Freshness-first, replayable sources

The industry has drifted toward Kappa because a single codebase is cheaper to maintain and because modern stream engines are expressive enough to carry the full computation. Lambda has not disappeared, however. It survives in domains where an independent, authoritative batch recomputation over an immutable master dataset serves as a correctness audit and a reconciliation baseline, particularly where regulatory or financial reporting demands a reproducible ground truth computed separately from the live path. In practice, many teams run a hybrid: a Kappa-style streaming path for freshness and a periodic batch job that reconciles and corrects, which is closer to Lambda in spirit than either label admits.

Execution Models: Micro-Batch and Event-at-a-Time

Below the architectural layer sits a second decision that determines a pipeline’s latency floor and recovery behavior: how the engine physically executes the stream. Two designs dominate, and they differ in the unit of work they process at a time.

Micro-batch execution

Spark Structured Streaming uses micro-batch execution by default. The incoming stream is divided into a sequence of small, deterministic batches, and each batch is executed as an ordinary Spark job over the records that accumulated during a short interval. This design inherits the fault-tolerance and exactly-once guarantees of Spark’s batch engine almost for free, because recovery means re-running a deterministic batch. According to the Spark Structured Streaming Programming Guide (version 4.1.x, 2026), the default micro-batch engine achieves end-to-end latencies as low as roughly 100 milliseconds with exactly-once guarantees. The trade-off is that the batch boundary imposes a latency floor: a result cannot appear until its micro-batch has been formed and executed. Micro-batch execution is well suited to high-throughput workloads where a latency of a fraction of a second is acceptable.

Spark has offered lower-latency options over time. A Continuous Processing mode, introduced experimentally in Spark 2.3, processes records with latency near one millisecond but provides only at-least-once guarantees. More recently, Spark 4.1 added a Real-Time Mode for Structured Streaming that targets sub-second and single-digit-millisecond latency for stateless tasks (Spark 4.1.0 release notes and the associated Databricks engineering blog, 2026).

Event-at-a-time execution

Apache Flink processes each record as it arrives, one event at a time, through a long-lived graph of operators. There is no batch boundary to wait for, so latency is bounded by the processing of an individual record rather than by an interval. This event-at-a-time model gives Flink native event-time semantics and a rich model of state and timers, which makes it a natural fit for complex, low-latency stream computations. Pattern-matching over event streams is one such application; the complex event processing pipeline with Flink CEP illustrates how the event-at-a-time model supports detecting temporal patterns across a continuous stream.

Apache Beam occupies a distinct position in this landscape. Rather than being an execution engine, Beam is a programming model that implements the Dataflow abstraction and compiles the same pipeline to different execution engines, called runners, including Flink and Spark. The Beam release line reached version 2.74.0 (2026-06-02), which added a Spark 4 runner for the Java software development kit (Beam downloads page, 2026). Beam lets an engineer write a computation once against the Dataflow model and choose the execution backend separately, which is the clearest embodiment of the “batch is a special case of streaming” principle.

Property	Micro-batch (Spark SS)	Event-at-a-time (Flink)	Continuous / RTM (Spark)
Unit of work	Small deterministic batch	Single record	Single record
Minimum latency	~100 ms	Milliseconds	~1 ms to sub-second
Default semantics	Exactly-once	Exactly-once	At-least-once (legacy CP)
Recovery model	Re-run deterministic batch	Restore from snapshot	Restart from offset
Throughput	High	High	Workload dependent

The engine version landscape as of mid-2026 is worth noting for teams selecting a platform. Apache Flink reached 2.3.0 (2026-06-25), with a long-term-support line at 1.20.x, following the 2.0.0 major release in March 2025. Apache Spark’s current stable releases are 4.1.2 (2026-05-21) and 4.0.3 on the 4.0 line. Apache Kafka reached 4.3.1 (2026-06-25); notably, Kafka 4.0.0 removed ZooKeeper entirely, making the KRaft consensus protocol the only operating mode (Flink, Spark, and Kafka download and release pages, 2026).

Caution: Published latency figures such as “as low as 100 ms” describe favorable conditions and specific engine modes, not a guarantee for an arbitrary workload. Actual latency depends on state size, shuffle behavior, backpressure, and sink characteristics. Treat these numbers as the shape of the trade-off rather than a service-level objective, and measure the specific pipeline before committing to it.

Getting Streaming Correct: Time, Watermarks, Windows, and Semantics

The difficulty of streaming is not moving events quickly; it is producing correct answers over data that arrives late, out of order, and without a natural end. This section covers the machinery that makes streaming results trustworthy, which is the substance that separates a robust pipeline from one that silently produces wrong aggregates.

Event time versus processing time

Event time is the moment an event actually occurred, recorded as a timestamp embedded in the record itself. Processing time is the moment the pipeline observes the event, which depends on network delay, buffering, and system load. In any real distributed system these two clocks diverge, and the gap between them, called event-time skew, is variable. A mobile device that loses connectivity may deliver an event minutes after it occurred. Because business questions are almost always framed in terms of when events happened rather than when a server saw them, correct windowed aggregation must be computed in event time. Aggregating in processing time is simpler but produces results that shift depending on system latency, which makes them non-reproducible.

Watermarks

A watermark is a monotonic assertion about completeness in the event-time domain. A watermark of value T declares that the system expects no further events with a timestamp at or before T. This is precisely the mechanism a streaming system uses to decide when an event-time window is complete enough to close and emit a result. A watermark that advances aggressively produces low-latency results but risks excluding genuinely late events; a conservative watermark waits longer, includes more late data, and increases latency. The watermark is therefore the tuning knob that trades completeness against latency, and it is defined per pipeline based on how late data is expected to arrive.

Events that arrive after the watermark has passed their window are late data. Systems handle late data through configurable policies: an allowed-lateness period keeps a window’s state alive for a grace interval, triggers can fire refined results as more data arrives, and retractions can withdraw and correct a previously emitted result. These options are drawn directly from the Dataflow model’s framing of when results are materialized and how refinements relate to earlier outputs.

Windowing

Windowing divides an unbounded stream into finite chunks over which aggregation is defined. Three window types are canonical. A tumbling window is fixed-size, contiguous, and non-overlapping, so each event belongs to exactly one window; a five-minute tumbling window partitions time into adjacent five-minute blocks. A sliding window has a fixed size and a separate slide interval and therefore overlaps, so a single event can belong to several windows; a ten-minute window that advances every minute is a sliding window. A session window is data-driven and gap-based: it groups events separated by less than a configured inactivity gap and closes after the gap elapses, producing variable-length windows that are not aligned across keys, which is well suited to modeling bursts of user activity.

Window	Overlap	Event membership	Typical use
Tumbling	None	Exactly one window	Periodic totals (per-minute counts)
Sliding	Yes	Several windows	Moving averages, rolling metrics
Session	None; gap-defined	One session per activity burst	User sessions, activity grouping

Delivery semantics and exactly-once

Delivery semantics describe how many times an event’s effect can be reflected in the output when failures and retries occur. At-most-once processing may drop records and never duplicates them; it is fire-and-forget and cheapest. At-least-once processing never loses a record but may apply it more than once after a retry or recovery, producing duplicates. Exactly-once processing guarantees that each record affects the computed state exactly once despite failures, which is what most correctness-sensitive aggregations require.

Semantics	Duplicate risk	Loss risk	Mechanism
At-most-once	None	Possible	Fire-and-forget
At-least-once	Possible	None	Retry on failure
Exactly-once	None	None	Checkpoint + two-phase commit

Flink achieves exactly-once state consistency through asynchronous barrier snapshotting, a variant of the Chandy-Lamport distributed snapshot algorithm (Flink Stateful Stream Processing documentation, 2026). The coordinating JobManager periodically injects special records called checkpoint barriers into the streams at the sources. As a barrier flows downstream, it separates the records that belong to the current snapshot from those that belong to the next one. When an operator has multiple input channels, it performs barrier alignment: it waits until the barrier has arrived on every input channel before taking its snapshot, buffering records that arrive after the barrier on faster channels. Once all operators have snapshotted their state, the checkpoint is complete and can be used to restore the entire job after a failure.

Internal exactly-once state is not sufficient by itself. To make results exactly-once all the way to an external system, the sink must participate in the checkpoint through a two-phase commit: it prepares (writes data in an uncommitted transaction) as part of a checkpoint and commits only after the checkpoint has completed successfully, so that a failure before completion leaves nothing visible to downstream readers. Flink implements this through a transactional committer for sinks that support transactions or idempotent writes. Kafka’s transactional producer and its durable, ordered-per-partition log are what make such end-to-end guarantees possible on the sink side.

Barrier alignment has a cost. Under backpressure, an operator can stall while waiting for a barrier on a slow channel, which delays the checkpoint and raises latency. This motivated unaligned checkpoints, in which barriers overtake buffered records and the in-flight data is included in the snapshot instead, trading a larger checkpoint for shorter alignment delay. The default in Flink remains exactly-once with aligned checkpoints, and the choice to relax it is a deliberate latency optimization.

Tip: The choice between at-least-once and exactly-once should be made per sink, not globally. If a downstream consumer deduplicates by an idempotency key, or the aggregation is itself idempotent, at-least-once may deliver the same correctness at lower cost and latency than full exactly-once with two-phase commit. Reserve exactly-once for cases where duplicates genuinely corrupt the result.

When Batch Remains the Right Choice, and the Convergence

The correctness machinery above should make one point clear: streaming buys freshness at the price of real complexity. For many workloads that price is not justified, and batch remains the correct and often superior choice. Recognizing these cases is as important as knowing how to build a stream.

Batch is the better fit for large historical backfills, where terabytes of accumulated data must be reprocessed and there is no latency requirement at all. It is preferable for full-refresh reproducibility, where the ability to re-run a deterministic job and obtain an identical, auditable result is worth more than freshness. It suits complex multi-source joins that tolerate latency, cost-sensitive periodic reporting where always-on compute would be wasteful, and the construction of machine-learning training sets, which are inherently snapshots over a bounded, versioned dataset. It also remains the natural home for correctness-audit reprocessing, the independent recomputation that underpins the surviving uses of Lambda.

The batch layer has its own mature tooling. Scheduled batch pipelines are commonly orchestrated with a workflow scheduler; the Apache Airflow orchestration guide describes how directed acyclic graphs express batch dependencies and retries. Transformation logic over warehouse tables is frequently expressed with a batch-first tool as covered in the dbt transformation pipeline guide. Batch outputs typically land in columnar files whose layout is examined in the Parquet and Arrow internals guide, and interactive analysis over such files increasingly runs in in-process engines compared in the DuckDB and Polars comparison.

The 2026 convergence

The sharp dichotomy between batch and streaming is eroding along three lines. The first is unified engines and APIs: Beam’s single model runs on multiple runners, and both Spark and Flink execute bounded and unbounded jobs through largely shared machinery, so the same logic serves both modes. The second is incremental processing: rather than fully reloading a dataset, a pipeline recomputes only the portion that changed, which imports a streaming efficiency into what looks like a batch job. The third is the streaming lakehouse, in which a single table serves both batch queries and streaming reads and writes.

Open table formats provide the foundation for the lakehouse by adding transactional guarantees and metadata to files in object storage; the trade-offs among them are examined in the Iceberg, Delta Lake, and Hudi comparison. A streaming-native format, Apache Paimon, an Apache Top-Level Project since 2024, extends this idea further by combining a log-structured merge-tree with a changelog stream, so that the same table can be written and read as a continuous stream while also supporting batch queries and incremental reads (Apache Paimon project, 2026). The practical effect is that Lambda’s two separate stores collapse into one, which removes the reconciliation burden that motivated the Kappa argument in the first place.

Key Takeaway: Convergence does not mean streaming has won and batch is obsolete. It means the two paradigms increasingly share an engine, an API, and a storage layer, so the decision moves from choosing a technology stack to choosing a freshness requirement per dataset and letting the same platform serve both.

A Decision Checklist

The following questions guide a workload toward batch or streaming without prescribing a specific engine. They are ordered so that the strongest determinants come first.

Question	Leans batch	Leans streaming
How fresh must the result be?	Minutes to hours is acceptable	Seconds or less is required
Is the input bounded or unbounded?	A finite, complete dataset	A continuous event feed
Is full-refresh reproducibility essential?	Yes, an auditable re-run is needed	Replay plus state is acceptable
What is the tolerance for operational complexity?	Low; a small team	Higher; state and watermarks are manageable
Does cost favor bursts or steady load?	Scheduled bursts amortize better	Freshness justifies always-on compute

When several answers point in the same direction, the decision is clear. When they conflict, the convergence tooling offers a middle path: build on a unified engine and a lakehouse table so that a workload can begin as batch and gain a streaming read later without a rewrite, deferring the commitment until the freshness requirement is genuinely established.

Frequently Asked Questions

Is streaming always more expensive than batch?

Not in a fixed ratio. Streaming runs always-on compute, which produces a steady cost that pays for freshness, while batch runs in scheduled bursts whose cost is amortized. Whether streaming costs more for a given workload depends on the state size, the shuffle behavior, and how continuously the data actually arrives. It is more accurate to reason about the shape of the trade-off, always-on versus bursty, than to apply a single multiplier.

Why not just use processing time instead of event time?

Processing-time aggregation is simpler because it ignores when events actually occurred, but its results shift with system latency and are therefore non-reproducible. If a network delay causes events to arrive late, a processing-time window attributes them to the wrong interval. Business questions are almost always framed in event time, so correct windowed results require it, which in turn requires watermarks to judge completeness.

What is the practical difference between at-least-once and exactly-once?

At-least-once never loses a record but may apply it more than once after a failure, producing duplicates. Exactly-once guarantees each record affects the result exactly once, at the cost of checkpointing and, for external sinks, a two-phase commit. If the downstream system deduplicates or the operation is idempotent, at-least-once can deliver equivalent correctness more cheaply; exactly-once is warranted when duplicates genuinely corrupt the output.

Does the convergence toward unified engines make Lambda and Kappa obsolete?

It reduces the cost that motivated the debate rather than settling it. Streaming lakehouse table formats collapse Lambda’s two stores into one, which removes much of the reconciliation burden. An independent batch recomputation still has value as a correctness audit in regulated or reconciliation-heavy domains, so Lambda-style patterns persist even as the tooling makes a single code path easier to maintain.

Is micro-batch a form of batch or streaming?

It is streaming implemented by repeatedly running very small batch jobs. Spark Structured Streaming chops an unbounded stream into short, deterministic micro-batches, which inherits batch-style recovery and exactly-once guarantees while continuously producing results. It sits between the two paradigms and illustrates why the batch-streaming boundary is better viewed as a continuum than a hard line.

References

Akidau, T., et al. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. PVLDB 8(12), 2015.
Kreps, J. Questioning the Lambda Architecture. O’Reilly Radar, 2014.
Apache Flink. Stateful Stream Processing: Checkpoints, Barriers, and Exactly-Once. Apache Software Foundation, 2026.
Apache Spark. Structured Streaming Programming Guide. Apache Software Foundation, 2026.
Apache Beam. Beam Programming Guide and Downloads. Apache Software Foundation, 2026.
Apache Paimon. Apache Paimon: Streaming Lakehouse Table Format. Apache Software Foundation, 2026.

Conclusion

Batch and stream processing are not opposing technologies but positions on a single trade-off surface defined by latency, throughput, and cost. The Dataflow model makes this concrete by treating a batch job as a stream computation over a bounded input in a single global window, which is why the same engines and APIs increasingly express both. The architectural debate between Lambda and Kappa reduces to whether an independent batch recomputation is worth its dual-codebase burden, and the industry has drifted toward the single-path Kappa answer while retaining Lambda where reconciliation demands it. Beneath the architecture, the choice of execution model, micro-batch or event-at-a-time, sets a pipeline’s latency floor and recovery behavior, and the correctness machinery of event time, watermarks, windowing, and exactly-once processing is what makes streaming results trustworthy rather than merely fast.

The most durable guidance is to resist treating either paradigm as a default. A workload should be placed on the continuum according to how fresh its result must be and what that freshness is worth, and the 2026 convergence toward unified engines and streaming lakehouse tables increasingly allows that decision to be deferred and revised without a rewrite. An engineer who understands the trade-off surface, rather than memorizing a preferred stack, is equipped to make that judgment correctly for each new dataset.

July 8, 2026

Data Contracts and Data Quality: Enforcing Reliability in Modern Pipelines

Modern data platforms move records across many independent systems: an application emits an event, an ingestion job lands it in a lake, a transformation framework reshapes it, and a dashboard or a machine-learning model consumes the result. Every one of these handoffs is an implicit agreement about the shape and meaning of the data, and every silent violation of that agreement propagates downstream until a report is wrong or a model degrades. A data contract—a versioned, machine-readable agreement between the team that produces a dataset and the teams that consume it, specifying its schema, semantics, service-level guarantees, and quality expectations—turns that implicit agreement into an explicit, testable artifact. This post examines how data contracts and data quality enforcement work together to make pipelines reliable, and why the two ideas are complementary rather than interchangeable.

The distinction that organizes the discussion is between prevention and detection. A data contract prevents bad data from entering a system by asserting expectations at the boundary where data is produced. Data observability, by contrast, detects problems after data has already landed, by monitoring freshness, volume, and schema drift and raising anomaly alerts. A mature platform uses both. The material below defines the contract as an artifact, presents a taxonomy of six data quality dimensions, surveys the enforcement engines that operate at the record, DataFrame, warehouse, and stream layers, and describes the patterns—shift-left validation, continuous-integration gates, pre-ingestion quarantine, circuit breakers, and dead-letter queues—that put a contract into force.

Summary

What this post covers: How data contracts and data quality enforcement combine to make data pipelines reliable, covering the contract as a versioned artifact, a taxonomy of quality dimensions, the enforcement engines available at each layer of a stack, and the patterns that put a contract into force at the boundary where data is produced.

Key insights:

A data contract is a specification, not an engine; the Open Data Contract Standard (ODCS), currently at version 3.1.0 under the Linux Foundation’s Bitol project, describes the agreement in YAML, while separate tools enforce it at each layer.
The two historically competing specifications are consolidating: the older Data Contract Specification is being deprecated in favor of ODCS v3.1.0, with tooling support scheduled only until the end of 2026 (datacontract-specification.com, as of 2026).
Data quality decomposes into six measurable dimensions—completeness, uniqueness, validity, accuracy, consistency, and timeliness—of which accuracy and consistency are the hardest to enforce mechanically.
Enforcement is layered: Pydantic and Pandera validate at the record and DataFrame boundary, dbt tests and Great Expectations and Soda Core validate warehouse tables, and Confluent Schema Registry enforces schema evolution on Kafka streams.
Contracts prevent errors by asserting expectations at the producer boundary, whereas observability detects errors by monitoring reality after the fact; a reliable platform needs both.

Main topics: The Data Contract as a First-Class Artifact, Six Dimensions of Data Quality, Where Enforcement Happens, Enforcement Patterns, Contracts and Observability.

The Data Contract as a First-Class Artifact

A data contract is a formal, versioned agreement between a data producer and its consumers. The producer is the system or team that emits a dataset—an application service publishing events, a database exposing a table, or an ingestion job writing files. The consumers are the downstream systems and teams that read that dataset: analytics models, dashboards, machine-learning features, and other services. Before contracts, the terms of that relationship lived in tribal knowledge and out-of-date documentation, so a producer could rename a column or change a unit of measurement without warning, and consumers discovered the break only when their output failed. A contract makes the terms explicit and machine-readable, so that a change which would violate them can be caught automatically.

Four elements typically appear in a contract. The schema declares the fields, their types, and their nullability. The semantics describe what each field means, including units, allowed value sets, and business definitions, so that a field named revenue is unambiguous about currency and whether it is gross or net. The service-level agreement (SLA) states operational guarantees such as freshness, expected update frequency, and availability. The quality expectations encode testable rules—for example, that a primary key is unique or that a percentage column falls between zero and one hundred. Figure 1 shows how these four elements sit on the boundary between a producer and its consumers.

A Standard for the Contract: ODCS

Writing a contract in an ad-hoc format leaves every team to invent its own structure, so a shared specification is useful. The Open Data Contract Standard (ODCS) is a vendor-neutral YAML format that describes the agreement between a data producer and its consumers. Its current version is v3.1.0 (Bitol / Linux Foundation AI & Data, as of 2026). ODCS originated as PayPal’s internal data contract template, was open-sourced, and was donated to the Linux Foundation; it is now stewarded by Bitol, a Linux Foundation AI & Data incubation project licensed under Apache 2.0 that was formed on 30 November 2023 when the AIDA User Group and LF AI & Data joined forces (bitol.io, as of 2026). Bitol also stewards the Open Data Product Standard (ODPS), which has reached v1.0.0.

An important point about the current landscape is that the ecosystem is consolidating around a single standard. A separate effort, the Data Contract Specification, is being deprecated and is converging on ODCS v3.1.0. Its maintainers advise users to migrate, and tooling support in the Data Contract CLI and Entropy Data is planned only until the end of 2026 (datacontract-specification.com, as of 2026). A team beginning with contracts in 2026 should therefore standardize on ODCS rather than the older specification. A minimal ODCS-style contract illustrates the artifact concretely.

apiVersion: v3.1.0
kind: DataContract
id: orders-contract
name: Orders
version: 1.2.0
status: active
schema:
  - name: orders
    properties:
      - name: order_id
        logicalType: string
        required: true
        unique: true
      - name: customer_id
        logicalType: string
        required: true
      - name: order_total
        logicalType: number
        required: true
        quality:
          - rule: minimum
            mustBeGreaterThanOrEqualTo: 0
      - name: status
        logicalType: string
        required: true
        quality:
          - rule: enum
            values: [active, shipped, cancelled]
slaProperties:
  - property: freshness
    value: 15
    unit: minute

Key Takeaway: A contract is a specification, not an engine. ODCS declares what must hold in a portable YAML document; separate validation tools read that declaration—or an equivalent set of checks—and enforce it wherever data crosses a boundary. Keeping the specification and the enforcement engine distinct is what allows one contract to be enforced consistently across a warehouse, a DataFrame, and a stream.

Treating the contract as versioned code has a direct engineering consequence: a change to the contract is a change to a file in version control, subject to review and to automated compatibility checks. This is what makes the shift-left patterns described later in this post possible, and it links data contracts to the broader practice of treating transformation logic as code, as covered in the guide to dbt for building transformation pipelines.

Six Dimensions of Data Quality

Before a contract can assert quality expectations, quality itself must be defined precisely enough to be measured. A widely used approach decomposes quality into distinct dimensions, each answering a different question about the data. The six dimensions below form a compact and practical taxonomy. Each is defined on first use, because the terms are often used loosely in practice.

Completeness asks whether the expected data is present, with no missing values or absent records where they are required. A completeness check might assert that customer_id is never null, or that a daily table’s row count falls within an expected band. Uniqueness asks whether each real-world entity appears once, with no unintended duplicates; a typical check enforces primary-key uniqueness so that no order_id is repeated. Validity asks whether values conform to a defined format, type, range, or allowed set—the domain of the field. Examples include requiring that status belongs to a fixed set of allowed values, that an email address matches a regular expression, or that an age is not negative.

Accuracy asks whether values correctly describe the real-world entity they represent. This is among the hardest dimensions to test in isolation, because it requires comparison against a trusted system of record rather than an internal rule; a value can be valid and unique yet still wrong. Consistency asks whether values agree across systems, tables, and over time, with no contradictions—for instance, that the sum of an order’s line items equals its recorded total, or that a customer’s country is identical in every table. Timeliness asks whether data is fresh and available within its expected latency or SLA window; a timeliness check verifies that the maximum event timestamp is within the last few minutes or that a partition landed on schedule.

Accuracy and consistency are the two dimensions that resist mechanical enforcement most strongly, because they depend on external reference points and cross-system reconciliation rather than a self-contained rule. A contract can assert them, but verifying them often requires a comparison the pipeline cannot perform on its own. A seventh dimension, integrity—referential integrity, meaning that foreign keys resolve to existing rows—is sometimes added; it can be treated as a specialized form of consistency across tables.

Caution: A record can satisfy completeness, uniqueness, and validity while still failing accuracy. A well-formed, unique, non-null value drawn from the allowed set can nonetheless misrepresent reality. Passing structural checks is necessary but not sufficient for correctness, which is why contracts pair schema rules with reconciliation against a system of record wherever one exists.

Where Enforcement Happens: Four Layers and Their Engines

A single contract is enforced at several points, because data changes form as it moves. It arrives as individual records at an application boundary, is assembled into in-memory tables (DataFrames) for processing, is materialized as warehouse tables, and travels as messages on streams. Each form has a matching class of validation engine. The essential distinction—already stated for the contract itself—applies here too: the specification declares what must hold, and each engine enforces it in the representation it understands.

Record and DataFrame Boundaries

At the application boundary, data arrives one record at a time—an incoming API request, a single event to be published. Pydantic validates and parses data at this record level. Its current version is v2.13.4 (pydantic/pydantic releases, as of 2026). The validation engine lives in a component called pydantic-core, written in Rust through the pyo3 binding layer, which delivers roughly a fivefold to fiftyfold performance improvement over version 1 depending on the workload (pydantic.dev, as of 2026). Pydantic excels at fast per-record validation at ingestion and API edges, but it operates one object at a time and is not designed for table-scale statistical checks.

Once records are assembled into a DataFrame—an in-memory tabular structure—the relevant unit of validation becomes the column and the table rather than the single object. Pandera provides schema-based, statistical validation for DataFrames. Its current version is 0.32.1 (unionai-oss/pandera releases, as of mid-2026). Pandera 0.19.0 added Polars validation, and by version 0.29 (January 2026) a single schema definition could validate data across pandas, Polars, Dask, Modin, PySpark, and Ibis; a Narwhals-powered backend introduced in 0.32.0 adds lazy validation across Polars, Ibis, and PySpark SQL. A Pandera schema expresses column-level expectations directly.

import pandera as pa
from pandera import Column, Check

schema = pa.DataFrameSchema({
    "order_id":    Column(str, unique=True, nullable=False),
    "customer_id": Column(str, nullable=False),
    "order_total": Column(float, Check.ge(0)),
    "status":      Column(str, Check.isin(["active", "shipped", "cancelled"])),
})

# Raises SchemaError on the first violation, or collects all
# violations when lazy=True.
validated = schema.validate(df, lazy=True)

Warehouse Tables

Inside the warehouse, checks run against materialized tables in SQL. Three engines are common here. dbt tests co-locate assertions with the transformation models that produce a table. dbt Core’s current version is 1.11.12 (dbt-core GitHub releases, as of 2026-07-01), with a Rust-based v2.0.0-alpha in development as the foundation for the dbt Fusion engine. dbt distinguishes data tests—generic and singular assertions that run against materialized data—from native unit tests, which validate SQL logic and were introduced in v1.8; in v1.9 and later, dbt test --resource-type test runs data tests while excluding unit tests. Because dbt tests only cover what dbt materializes, they do not guard the ingestion boundary or streams. The role of dbt in the transformation layer is treated fully in the guide to dbt transformation pipelines.

Great Expectations (GX Core) is a dedicated validation framework built around a reusable library of expressive assertions called Expectations, together with automatic profiling and human-readable validation reports called Data Docs. Its current version is 1.18.2 (Great Expectations changelog, as of 2026-06-26); GX Core 1.0 was a major API redesign that separated the open-source GX Core from the commercial GX Cloud. It supports SQL, Spark, and pandas backends. Its strength is the breadth and reusability of its Expectation library; historically its weakness has been a heavier setup and a steeper learning curve. Figure 5 traces its validation flow.

Soda Core is a second dedicated framework, known for concise checks and a continuous-integration-friendly command-line interface. Its current version is 4.16.0, released 29 June 2026, and it requires Python 3.10 or later (Soda Core release notes, as of 2026-06-29). A notable change is that Soda Core version 4 makes data contracts the default way to define quality rules—a breaking change that moves away from the older SodaCL “checks” syntax toward a contract-based syntax. A concise Soda check reads close to natural language.

checks for orders:
  - row_count > 0
  - missing_count(customer_id) = 0
  - duplicate_count(order_id) = 0
  - invalid_count(status) = 0:
      valid values: [active, shipped, cancelled]
  - freshness(order_time) < 15m

Streams

On a Kafka stream, the contract is the message schema, and the enforcement point is a schema registry—a service that stores schemas and rejects producers whose schema is incompatible with the registered version. Confluent Schema Registry supports Avro, Protobuf, and JSON Schema (referred to as JSON_SR) out of the box on both Confluent Platform and Confluent Cloud (docs.confluent.io, as of 2026). It governs schema evolution through compatibility types: BACKWARD (the default), BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, and NONE. BACKWARD is the default because it allows consumers using the new schema to read data written with the previous schema, which lets a consumer rewind to the start of a topic. Confluent recommends BACKWARD_TRANSITIVE for Protobuf, because adding new message types is not forward compatible, whereas Avro was designed with schema evolution in mind (docs.confluent.io, as of 2026). A schema registry enforces the shape and evolution of messages, not their semantic quality; it will not catch a null in a field that is technically nullable or a value out of an expected range. Streaming schema enforcement pairs naturally with change-data-capture pipelines, as discussed in the guide to change data capture with Debezium and Kafka, and with the consumer-side handling covered in the Kafka consumer implementation guide.

The following table summarizes what each engine does best and where its boundary lies. Every version cited carries its source and date in the prose above.

Tool (version)	Layer	Best at	Boundary
Pydantic (v2.13.4)	Application record	Fast per-record validation and parsing at API and ingestion edges	One object at a time; not for table-scale statistical checks
Pandera (0.32.1)	DataFrame	Schema-based statistical validation across pandas, Polars, PySpark, Ibis	In-process; not a warehouse or catalog-level gate
dbt tests (1.11.x)	Transformation (in-warehouse)	Tests co-located with models; data tests plus unit tests since v1.8	Only tests what dbt materializes; not the ingestion boundary or streams
Great Expectations (1.18.2)	Warehouse / batch tables	Rich reusable Expectation library; profiling; Data Docs	Heavier setup; historically steeper learning curve
Soda Core (4.16.0)	Warehouse + contracts	Concise checks; contracts as default in v4; CI-friendly CLI	v4 contract migration is a breaking change from SodaCL checks
Confluent Schema Registry	Streaming (Kafka)	Schema-as-contract; compatibility enforcement for Avro/Protobuf/JSON_SR	Enforces shape and evolution, not semantic quality
ODCS v3.1.0 (Bitol/LF)	Specification / governance	Vendor-neutral YAML linking schema, SLA, quality, and ownership	A specification, not an engine; needs a tool to enforce it

Tip: Match the engine to the representation. Validate individual records with Pydantic at the edge, DataFrames with Pandera in processing jobs, warehouse tables with dbt tests or Great Expectations or Soda Core, and stream messages with a schema registry. A single ODCS contract can act as the shared source of truth that these separate engines each enforce in their own layer.

Enforcement Patterns: Shift-Left, Gates, and Dead-Letter Queues

Having an engine is not the same as having a strategy for where and when it runs. Several established patterns place enforcement at different points in the data’s journey, trading availability against correctness in different ways.

Shift-Left at the Producer Boundary

Shift-left is the practice of moving validation as early as possible—toward the point where data is produced—rather than catching problems late in the pipeline. The term borrows from software testing, where moving tests earlier (“to the left” on a left-to-right timeline) reduces the cost of fixing defects. Applied to data, shift-left means validating at the application or ingestion boundary so that non-conforming data never enters the warehouse at all. This is the cheapest place to catch an error, because the cost of a defect rises as it travels downstream: a bad value caught at the producer affects nothing, whereas the same value caught at the BI or ML layer may have already corrupted reports, retrained a model, or been copied into many derived tables. Figure 3 depicts this rising cost.

Continuous-Integration Gates on Contract Changes

Because a contract is a file under version control, a proposed change to a producer’s schema arrives as a pull request. A continuous-integration (CI) gate is an automated check that runs on that pull request and blocks the merge if the change would break the contract. Two checks matter here: validating that the contract file itself is well formed, and running a backward-compatibility check that determines whether existing consumers can still read data produced under the new schema. This mirrors exactly what a schema registry does at runtime for streams, moved earlier to the moment of code review. A breaking change—removing a required field, narrowing a type—fails the gate and cannot merge until the producer and consumers agree on a migration. Figure 6 shows the contract advancing through versions with a compatibility check at each transition.

Pre-Ingestion Gates and Circuit Breakers

When validation cannot happen strictly at the producer—because the producer is a third party, or the data arrives as files—a pre-ingestion gate places incoming data in a staging area first and requires a validation step to pass before the data is promoted to a production table. The gate fails closed: if the check does not pass, the data does not advance. This quarantines suspect data rather than exposing it to consumers.

A related pattern is the circuit breaker, which halts a running pipeline when a contract check fails beyond a chosen threshold, rather than allowing bad data to propagate. The name is borrowed from electrical engineering, where a breaker interrupts a circuit to prevent damage. In a data pipeline hosted by an orchestrator, a failed check fails the orchestrator task and stops downstream jobs from running. This deliberately trades availability for correctness: a stalled pipeline is often preferable to a fast one that delivers wrong answers, because a visible delay prompts investigation whereas a silent error can persist unnoticed for days. Orchestrators such as those covered in the Apache Airflow orchestration guide are the natural host for both pre-ingestion gates and circuit breakers, because they already model tasks, dependencies, and failure propagation. A validation task placed upstream of a load task allows the orchestrator to skip or fail the load automatically when the check does not pass, so the circuit-breaker behavior falls out of the dependency graph rather than requiring bespoke error handling.

Dead-Letter Queues for Streaming

In a streaming pipeline, halting the entire stream because a single record fails a check would be too blunt, since one malformed message would block every well-formed one behind it. A dead-letter queue (DLQ) solves this by routing records that fail schema or quality checks to a separate queue for inspection and later replay, while valid records continue on the main stream. This isolates failures at the level of the individual record rather than the whole pipeline, and it preserves the malformed records for diagnosis instead of discarding them. Figure 4 shows a stream splitting into a pass branch and a dead-letter branch, with a replay path back into the pipeline once the underlying problem is fixed.

Key Takeaway: The patterns form a graduated response. Shift-left prevents most defects at the source; CI gates block breaking schema changes before they merge; pre-ingestion gates quarantine suspect batches; circuit breakers halt a batch pipeline that has already ingested bad data; and dead-letter queues isolate individual failing records on a stream without stopping the whole flow. Each pattern trades some availability for correctness in a way suited to its layer.

Contracts and Observability: Prevention Versus Detection

Data contracts are frequently discussed alongside data observability, and the two are sometimes conflated. They address the same goal—reliable data—from opposite directions. A contract is a mechanism of prevention: it asserts expectations at the boundary and refuses data that violates them, so a defect is stopped before it enters the system. Observability is a mechanism of detection: it monitors the data that has already landed—tracking freshness, volume, distribution, and schema drift—and raises an alert when reality departs from the norm. Observability finds problems the contract did not anticipate, including gradual distribution shifts and upstream failures that produce technically valid but unusual data. Figure 8 places the two side by side over a shared pipeline.

Neither mechanism substitutes for the other. A contract cannot anticipate every failure mode, particularly slow statistical drift in otherwise valid data; observability catches those. Observability, on its own, only tells a team that something has already gone wrong, often after consumers have been affected; a contract prevents the class of failures it can express. The two belong together in a mature stack. Quality gates also interact with the storage layer, because a table’s schema evolution is itself a contract concern—an area explored in the comparison of Iceberg, Delta Lake, and Hudi table formats and in the discussion of typing at the file level in the guide to Parquet and Apache Arrow internals. An end-to-end pipeline where these gates would apply in practice is described in the walkthrough from InfluxDB to AWS Iceberg with Telegraf.

The Business Case for Enforcement

The motivation for this discipline is that poor data quality carries a measurable cost. A frequently cited estimate holds that poor data quality costs organizations at least 12.9 million US dollars per year on average; this figure comes from Gartner’s 2020 Magic Quadrant for Data Quality Solutions, based on a survey of 154 reference customers across 16 vendors (Gartner, as of 2020). It should be read as a 2020 estimate rather than a current measurement, and organizations vary widely, but it captures the order of magnitude at stake. The cost is not only financial. Erroneous data erodes the trust that consumers place in a dataset, and once a dashboard has been visibly wrong, downstream teams begin to second-guess every figure it produces, which slows decisions and encourages the growth of parallel, unofficial data copies. Enforcement at the boundary is a way to protect that trust as much as it is a way to avoid direct cost. Adoption of data contracts is growing as teams formalize the producer-consumer relationship, although a precise adoption figure is not available from a reliable primary survey. The direction of travel is clear from the standards themselves: the consolidation of the Data Contract Specification into ODCS v3.1.0 under the Linux Foundation signals a maturing field converging on shared conventions.

Frequently Asked Questions

What is the difference between a data contract and a database schema?

A database schema declares field names, types, and nullability—the structural shape of a table. A data contract is broader: it wraps the schema together with semantics (what each field means, including units and allowed value sets), a service-level agreement (freshness and availability guarantees), and testable quality expectations, and it is versioned as an explicit agreement between the producing team and its consumers. In short, a schema is one component of a contract, and the contract adds meaning, guarantees, and ownership on top of structure.

Do I need both a schema registry and a tool like Great Expectations?

They cover different concerns and are often used together. A schema registry, such as Confluent Schema Registry, enforces the shape and compatible evolution of messages on a stream, rejecting a producer whose schema is incompatible with the registered version. It does not check semantic quality such as null values in nullable fields, out-of-range numbers, or freshness. A tool like Great Expectations or Soda Core enforces those quality expectations against tables or DataFrames. A streaming platform that also cares about value-level quality therefore benefits from both.

Which data quality dimensions are hardest to enforce automatically?

Accuracy and consistency are the hardest. Accuracy asks whether a value correctly describes the real-world entity it represents, which requires comparison against a trusted system of record rather than a self-contained rule; a value can be well-formed, unique, and within the allowed set yet still be wrong. Consistency asks whether values agree across systems and over time, which requires cross-system reconciliation. Completeness, uniqueness, validity, and timeliness are comparatively straightforward to express as mechanical checks.

Should a team standardize on ODCS or the Data Contract Specification in 2026?

A team beginning in 2026 should standardize on the Open Data Contract Standard (ODCS), currently at version 3.1.0 under the Linux Foundation’s Bitol project. The alternative Data Contract Specification is being deprecated and is converging on ODCS, with its tooling support in the Data Contract CLI and Entropy Data planned only until the end of 2026 (datacontract-specification.com, as of 2026). Choosing the surviving standard avoids a forced migration later.

References

Open Data Contract Standard (ODCS) v3.1.0 — Bitol / Linux Foundation AI & Data (as of 2026)
Great Expectations (GX Core) changelog (as of 2026-06-26)
Soda Core release notes (as of 2026-06-29)
Confluent Schema Registry — Schema Evolution and Compatibility (as of 2026)
dbt data tests and unit tests documentation (as of 2026-07-01)
Pandera documentation (as of mid-2026)

Conclusion

Data reliability is achieved not by a single tool but by a discipline that combines an explicit artifact with layered enforcement. The data contract turns the implicit producer-consumer agreement into versioned, machine-readable code, and the Open Data Contract Standard v3.1.0 provides a vendor-neutral way to express it; the consolidation of the older Data Contract Specification into ODCS marks a field settling on shared conventions. Quality itself becomes testable once it is decomposed into completeness, uniqueness, validity, accuracy, consistency, and timeliness, with the understanding that accuracy and consistency resist mechanical checks and require external reference points.

The contract is then enforced wherever data changes form: Pydantic and Pandera at the record and DataFrame boundary, dbt tests and Great Expectations and Soda Core in the warehouse, and a schema registry on the stream. The patterns that put enforcement into force—shift-left validation at the producer, continuous-integration gates on contract changes, pre-ingestion quarantine, circuit breakers, and dead-letter queues—each trade some availability for correctness in the way best suited to their layer. Finally, contracts and observability are complementary rather than interchangeable: contracts prevent the failures they can express at the boundary, while observability detects the anomalies they cannot anticipate. A platform that runs both, over a contract treated as a first-class artifact, is one whose data consumers can trust.

July 5, 2026

Author: kongastral

Summary

Split (inductive) conformal prediction

The quantile threshold: a worked example

Marginal versus conditional coverage

Prediction sets for classification: APS and RAPS

Conformalized quantile regression

When exchangeability breaks: distribution shift

Tooling and practical notes

Related Reading

Frequently Asked Questions

Does conformal prediction require the model to be accurate?

How is conformal prediction different from probability calibration?

What does the 90 percent coverage guarantee actually promise?

What breaks the coverage guarantee in production?

Conclusion

References

The Two Ways a Deployed Model Goes Wrong

Why Models Degrade After Deployment

Detecting Data Drift Without Labels

A Worked Population Stability Index Calculation

Detecting Concept Drift From the Error Stream

Monitoring When Labels Arrive Late or Never

Building a Monitoring Loop in Practice

Common Pitfalls in Drift Monitoring

Conclusion

Frequently Asked Questions

What is the difference between data drift and concept drift?

Does data drift always mean the model has become less accurate?

How is the Population Stability Index interpreted?

Which drift detectors require labels and which do not?

How can model performance be estimated when labels are delayed?

How often should a drifting model be retrained?

References

Why autoregressive decoding is slow

The draft-and-verify loop

The correctness guarantee

How much speedup to expect

The variant landscape

When it helps, and when it does not

Support in production systems

Related Reading

Conclusion

References

Frequently Asked Questions

Does speculative decoding change the text a model produces?

How large a speedup is realistic?

What is the acceptance rate and why does it matter?

Do all variants require a separate draft model?

Why does the benefit shrink at high batch sizes?

What calibration means

Measuring miscalibration

A measured example

Fixing it after training

Conclusion

Frequently Asked Questions

Does calibrating a model improve its accuracy?

When should isotonic regression be preferred over Platt scaling?

References

Summary

Why codec choice is a size-versus-speed decision

Test environment: hardware, software, and dataset

Hardware and software

The dataset

Methodology: how every number was produced

The codec matrix: file size, write speed, read speed

Filtered reads and the price of decompression

Row-group size: compression against skipping granularity

Dictionary encoding on a low-cardinality column

Choosing a codec: practical guidance

Frequently Asked Questions

Which Parquet compression codec is the best default?

Is a higher Zstandard compression level worth the cost?

Why was gzip so much slower than the other codecs?

How does row-group size affect compression and query speed?

Related Reading

References

Conclusion

Summary

Why Schemas Matter in Event-Driven Pipelines