Direct comparison
HDF5 vs Zarr vs Parquet Compared
HDF5, Zarr, and Parquet compared: N-D arrays vs columnar data, local vs cloud storage, and single- vs parallel-write workloads.
Written and maintained by CASRAI Editorial Board
Last updated
Ask CASRAI · included with Regulatory Radar
Ask about HDF5 vs Zarr vs Parquet Compared
Ask CASRAI answers research-administration questions and cites the passages behind every claim — and says so when the corpus does not cover something, instead of guessing. It comes with a Regulatory Radar subscription at $29 a month, alongside the daily digest of regulatory changes and the dashboard of what changed.
150 questions a day, on this site, over the API, or inside your own tools through the CASRAI MCP server.
Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays free to read, with no account and no card.
How do HDF5, Zarr, Parquet compare side by side?
The table below compares HDF5, Zarr, Parquet across 12 procurement-relevant dimensions, from data model through weakest fit.
Side-by-side comparison
| Dimension | HDF5 | Zarr | Parquet |
|---|---|---|---|
| Data model | Hierarchical, self-describing container: groups (like folders) holding typed N-dimensional datasets and attributes anywhere in the tree | Chunked N-dimensional array format with an optional group hierarchy; no single-file container | Columnar tabular format: rows grouped into row groups, stored column-by-column within each |
| Primary data shape | Large dense N-D numeric arrays and heterogeneous, deeply nested scientific data | Large N-D numeric arrays needing parallel or cloud-streamed read/write | Tabular / dataframe-shaped data -- event logs, transaction records, feature tables |
| Physical storage | Single binary file by default (B-tree indexed internally); can span family/external files | Each chunk is its own object/key -- directory of files locally, or native S3/GCS/Azure Blob objects | A file, or a set of files across a partitioned directory, commonly stored directly in object storage |
| Chunking | Native per-dataset chunked storage with a configurable chunk cache | Core design primitive -- array split into regular chunks up front, each independently compressed | Row groups plus per-column pages act as the internal chunking unit; enables column/row-group pruning |
| Compression | Built-in filters (gzip/deflate, szip) plus third-party filter plugins (e.g. Blosc) via the HDF5 filter pipeline | Pluggable codecs via numcodecs (Blosc, Zstandard, LZ4, gzip), configured per array | Per-column compression and encoding (Snappy, gzip, Zstandard, dictionary/run-length encoding) |
| Best storage target | Local or networked POSIX filesystem; cloud object storage is workable but awkward | Built for cloud/object storage from the ground up; equally usable on a local filesystem | Cloud object storage and distributed filesystems -- the standard data-lake format |
| Parallel / concurrent writes | SWMR mode (added in 1.10, 2016) allows one writer with concurrent readers; true multi-writer needs Parallel HDF5 (MPI-IO) into one shared file | Different processes can write different chunks of the same array concurrently with no shared file lock | Parallel workers typically write separate files/partitions rather than one shared file, sidestepping single-file contention |
| Concurrency / corruption risk | Uncoordinated concurrent writes to a plain, non-SWMR file risk corrupting it | Low, since each chunk is an independent object; two writers to the identical chunk still need external coordination | Low, since files are treated as immutable once written and any number of readers can read concurrently |
| Typical ecosystem | h5py, PyTables, native C/Fortran/Java/MATLAB APIs; NetCDF4 and MATLAB .mat v7.3 are built on it | zarr-python, Xarray, Dask, TensorStore | Apache Arrow, Spark, DuckDB, Pandas, Polars, Impala, cuDF |
| Random access pattern | Efficient hyperslab (sub-array) reads out of a large chunked dataset without loading the whole file | Reads only chunks intersecting the requested slice, over range requests against object storage | Efficient column pruning and row-group skipping via predicate pushdown; not built for N-D slicing |
| Maturity / origin | HDF Group; format dates to 1998, successor to HDF4 -- one of the most mature scientific-data formats | Community format from the mid-2010s scientific Python/bio-imaging world; v3 core spec ratified May 2023 | Originated 2013 at Twitter/Cloudera, drawing on Google’s Dremel paper; now an Apache top-level project |
| Weakest fit | Many workers writing the same array concurrently, or storing it as many small cloud objects | A single small array that fits in memory, where per-chunk object overhead outweighs the parallel-I/O benefit | A large dense N-dimensional numeric array (imaging volume, climate grid) -- loses native multi-D slicing |
Common questions
Common questions about HDF5 vs Zarr vs Parquet
Which format should I use for large N-dimensional scientific arrays?
+
HDF5 or Zarr, not Parquet. Choose HDF5 for a single-writer workflow on local or networked POSIX storage where mature tooling matters. Choose Zarr when the array needs parallel writers or lives in cloud object storage.
Can I use Parquet for array data?
+
Only by flattening the array into columns, which loses the native multi-dimensional chunking and slicing that HDF5 and Zarr provide. Parquet is built for tabular data, not dense N-dimensional arrays.
Is Zarr a replacement for HDF5?
+
Not a strict replacement -- the two overlap in what they can store, but Zarr specifically targets parallel, cloud-native workloads, while HDF5’s strength is a mature, self-describing single-file format for single-writer local workflows.
Which of the three is safest for concurrent writes?
+
Zarr (chunk-per-object) and Parquet (partition-per-file) both avoid the single shared-file contention that makes concurrent HDF5 writes fragile without SWMR mode or a Parallel HDF5 (MPI-IO) build.
Can I convert between these formats?
+
Yes. Xarray can read and write both HDF5- and Zarr-backed arrays through a common interface, and Arrow-based tooling reads and writes Parquet from most dataframe libraries.
Going deeper
