This page describes how features are added to the Parquet format specification and how they affect reader and writer compatibility. See the Implementation status page for which implementations (arrow, parquet-java, arrow-rs, etc.) support e…
Column statistics are the secret to Apache Parquet’s blazing fast performance. By storing compact summaries—like min, max, and null counts—for row groups, column chunks, and pages, readers can easily skip irrelevant data that doesn’t match
The latest version of parquet-java is 1.17.1. For the changes, please check out the Release on Github. To check the validity of this release, use its: The latest version of parquet-java on the previous minor branch is 1.16.0. To check the v…
The Apache Parquet community is excited to announce the addition of the Variant type—a feature that brings native support for semi-structured data to Parquet, significantly improving efficiency compared to less efficient formats such as JSO…
In Parquet, a page index is optional metadata for a ColumnChunk, containing statistics for DataPages that can be used to skip those pages when scanning in ordered and unordered columns. The page index is stored using the OffsetIndex and Col…
The Variant type is designed to store and process semi-structured data efficiently, even with heterogeneous values. Query engines encode each Variant value in a self-describing format, and store it as a group containing value and metadata b…
The extension mechanism of the binary Thrift field-id 32767 has some desirable properties: Because only one field-id is reserved the extension bytes themselves require disambiguation; otherwise readers will not be able to decode extensions
A Variant represents a type that contains one of: A Variant is encoded with 2 binary values, the value and the metadata. There are a fixed number of allowed primitive types, provided in the table below. These represent a commonly supported
Parquet files containing sensitive information can be protected by the modular encryption mechanism that encrypts and authenticates the file data and metadata - while allowing for a regular Parquet functionality (columnar projection, predic…
Logical types are used to extend the types that parquet can be used to store, by specifying how the primitive types should be interpreted. This keeps the set of primitive types to a minimum and reuses parquet’s efficient encodings. For exam…
In their current format, column statistics and dictionaries can be used for predicate pushdown. Statistics include minimum and maximum value, which can be used to filter out values not in the range. Dictionaries are more specific, and reade…
This document contains the specification of geospatial types and statistics. The Geometry and Geography class hierarchy and its Well-Known Text (WKT) and Well-Known Binary (WKB) serializations (ISO variant supporting XY, XYZ, XYM, XYZM) are…