JSON, MessagePack, Avro, and Arrow IPC Benchmarked on One Million Event Records

By kongastral

Published September 21, 2026 · 10 min read

One million synthetic event records occupy 192.2 MB as standard-library JSON and 13.1 MB as Parquet with zstd compression, a factor of 14.7 for identical input. Encode times ranged from 0.174 to 1.583 seconds, decode times from under one millisecond to 2.208 seconds. Every figure below comes from one run on the author’s workstation.

How the Benchmark Was Run

Each record is a dictionary of seven fields: two 64-bit integers, a 32-bit user identifier, an event type drawn from eight strings, a double-precision value, a country code from twenty, and a nested structure of device, version and latency. Ten keys therefore appear per record. One million were generated with a fixed seed and held in memory, and every format encoded that list.

Encode is the time to turn the whole Python list into bytes in memory; decode reverses that into the object named in the last column. Each figure is the median of three runs of the whole batch. Arrow and Parquet encode times include pa.Table.from_pylist, the row-wise conversion from dictionaries into Arrow columns, while their decode times exclude the reverse conversion; the row labelled arrow ipc + pylist adds Table.to_pylist and is the fair comparison when Python objects are needed.

The compressed column applies external zstd level 3 afterwards, so every uncompressed format meets the same codec; the two rows named zstd compress internally, and Avro used codec="null" for that reason. The machine is an Apple M2 Pro with 32 GiB running macOS and CPython 3.11.14. The script, raw console output, and methodology are published in the analytics-bench repository.

$ cd analytics-bench/serialization-formats
$ uv run --with orjson,msgpack,fastavro,pyarrow,zstandard python bench.py

machine: arm64 Darwin 25.5.0; python 3.11.14
versions: orjson 3.12.0, msgpack (1, 2, 2), fastavro 1.12.2, pyarrow 25.0.1, zstandard 0.25.0
generated 1,000,000 records in 1.9s
json (stdlib)        size=   192.2 MB  zstd3=    26.5 MB  encode=  1.15s  decode=  1.73s  (decoded as python objects)
orjson               size=   173.2 MB  zstd3=    26.2 MB  encode=  0.17s  decode=  1.62s  (decoded as python objects)
arrow ipc zstd       size=    27.5 MB  zstd3=    27.5 MB  encode=  0.37s  decode=  0.01s  (decoded as arrow table)
parquet zstd         size=    13.1 MB  zstd3=    13.1 MB  encode=  0.54s  decode=  0.03s  (decoded as arrow table)

Size on the Wire

Raw serialized size; only the two shortest bars compress internally.

Standard-library JSON repeats all ten key names as text in every record; orjson writes the same structure without the space after each separator, and the 19.0 MB gap between them is exactly 19 bytes per record, one per separator inside a record plus one between records. MessagePack replaces the punctuation with one-byte type tags, reaching 133.8 MB, but it is schema-less, so keys still travel in every record (MessagePack specification, as of 2026-09-21).

Avro removes them: binary encoded Avro “does not include type information or field names”, and a record is its field values in declaration order (Apache Avro 1.12.0 specification, as of 2026-09-21). The schema travels once in the container header, and the raw payload fell to 43.6 MB, 3.1 times smaller than MessagePack. An uncompressed Arrow IPC stream instead holds each column as a contiguous buffer behind one schema message (Arrow columnar format, as of 2026-09-21), and at 65.6 MB exceeds Avro because integers are stored at full width. Internal zstd brought it to 27.5 MB, and Parquet, which encodes columns before compressing them, produced 13.1 MB.

The same payloads after external zstd level 3.

Compression erases most of that hierarchy. After zstd level 3 the six uncompressed payloads fall between 20.6 MB for Avro and 26.5 MB for standard-library JSON, so a 4.4-fold spread becomes a 1.3-fold spread: repeated keys are precisely the redundancy a general-purpose compressor removes cheaply.

Encode and Decode Cost

Encode time, including the Arrow and Parquet conversion from dictionaries.

orjson encoded the batch in 0.174 seconds against 1.155 for the standard library, with MessagePack at 0.376 and pickle at 0.469. Building an Arrow table from the dictionaries and writing an uncompressed IPC stream took 0.356 seconds, and zstd raised that only to 0.373. The slowest encoder was fastavro at 1.583 seconds, 9.1 times the orjson figure.

These figures do not reproduce the ratio the orjson project publishes for itself: its documentation describes dumps as something like ten times as fast as json and loads as something like twice as fast, with latencies measured on an x86-64-v4 machine under Python 3.11.10 (orjson documentation, as of 2026-09-21). This run measured about 6.6 on encoding and about 1.07 on decoding. The difference is a property of the comparison rather than a contradiction: nested dictionaries with ten short keys, a single batch of a million and an arm64 processor depart from that setup on every axis.

Decode time for the paths that return Python dictionaries.

Decode times cluster far more tightly than encode times. Every path returning Python dictionaries took between 0.850 seconds, for Arrow IPC with to_pylist, and 2.208 seconds for fastavro, with pickle, orjson, JSON and MessagePack in between. Parsing bytes is evidently not the dominant read cost; allocating one million dictionaries of ten objects each is. MessagePack, despite a payload 22.8 percent smaller than orjson’s, decoded 0.223 seconds more slowly.

Choosing a Format

For data crossing an organizational boundary, JSON remains the reasonable default: RFC 8259 defines a language-independent interchange format and requires UTF-8 outside a closed ecosystem (RFC 8259, as of 2026-09-21). Substituting orjson costs nothing in interoperability and removed 0.981 seconds of encode time per million records. MessagePack suits uncompressed payloads and non-Python consumers; zstd narrows its 22.8 percent size advantage over orjson to about 3 percent.

Avro is the strongest row format on bytes and the weakest on Python speed. Dropping field names produces 43.6 MB, and the schema that permits it also governs compatibility between producers and consumers, the subject of schema evolution and the schema registry. A producer sending Avro to a broker pays that cost per message, not per million, so 1.583 seconds describes throughput; the receiving side is covered in implementing an Apache Kafka consumer in Python.

pickle protocol 5 was the fastest row decoder at 1.040 seconds and the second most compact at 65.2 MB, but it is Python-specific and its documentation states that the module “is not secure” and that malicious data can “execute arbitrary code during unpickling” (Python pickle documentation, as of 2026-09-21). It belongs inside a trust boundary.

Arrow IPC suits a columnar consumer: the stream was read back as a table in under one millisecond, and internal zstd reduced it to 27.5 MB for an extra 0.017 seconds of encode time. That layout is described in Parquet and Arrow internals. Parquet with zstd produced 13.1 MB for data at rest, 2.1 times smaller than the compressed stream, at 0.544 seconds of write time and 34 milliseconds to read back; its codec trade-offs are measured in Parquet compression codecs benchmarked.

Limits of This Measurement

These numbers describe one machine, one Python build and one record shape. Protobuf is absent because no protoc compiler was available. The statistic is the median of three runs rather than the minimum the timeit documentation recommends (Python timeit documentation, as of 2026-09-21), so gaps of a few percent between adjacent rows are not meaningful. Everything was measured in memory, so file system, network and broker costs are excluded; the Arrow and Parquet figures would also fall for a producer already holding Arrow data. The records are regular, with no nulls and strings from small vocabularies, so real event data would compress differently.

Conclusion

After compression the row formats differed by less than a third, and every decode producing Python objects landed within a factor of 2.6. Where payloads are compressed in transit, the format decides processing time rather than bytes. Constructing Python objects dominated every read, which is why the one decode differing by orders of magnitude belongs to the format whose consumer never builds them.

Frequently Asked Questions

Why is Avro smaller than MessagePack?

Avro does not store field names in the data. A binary encoded record is its field values in declaration order, with no keys and no type information, and the schema appears once in the container header. MessagePack is schema-less, so each of the ten keys is repeated in all one million records. That difference accounts for most of the gap, 43.6 MB against 133.8 MB.

When does the near-zero decode time of Arrow IPC apply?

Only when the consumer operates on the Arrow table itself, such as a query engine or a dataframe library reading Arrow buffers directly. Reading an uncompressed stream finished in under one millisecond because deserialization examines message metadata without copying the underlying data. Once Python dictionaries are required, the conversion runs and the cost returns: 0.850 seconds.

References

• RFC 8259: The JSON Data Interchange Format (2026-09-21).
• MessagePack specification (2026-09-21).
• Apache Avro 1.12.0 specification (2026-09-21).
• Apache Arrow columnar format and IPC (2026-09-21).
• Python documentation: pickle (2026-09-21).