Parquet Basics

The org.apache.juneau.marshall.parquet package provides Juneau serializer and parser implementations for the Apache Parquet columnar storage format.

Parquet support is included in juneau-marshall.

Maven	`<dependency> <groupId>org.apache.juneau</groupId> <artifactId>juneau-marshall</artifactId> <version>${juneau.version}</version> </dependency>`

The ParquetSerializer and ParquetParser classes integrate with the standard Juneau Serializer / Parser APIs and support the @Parquet and @ParquetConfig annotations for format-specific configuration. The Parquet marshaller (in org.apache.juneau.marshall.marshaller) provides the usual static shortcuts.

Media type: application/vnd.apache.parquet (both produced and consumed).

Parquet is collection-oriented: each bean is one row and each bean property is one column. The serializer accepts a Collection, an array, or a single bean (wrapped in a one-element list) at the root. The parser always returns a List<T>, never a bare T.

Quick Start

public class Person {
    public String name;
    public int age;
}

List<Person> people = List.of(/* ... */);

// Serialize rows to a Parquet file image.
byte[] bytes = Parquet.of(people);

// Parse back.  The result is always a List.
List<Person> parsed = Parquet.to(bytes, List.class, Person.class);

// Or with the serializer/parser instances directly.
byte[] bytes2 = ParquetSerializer.DEFAULT.write(people);
List<Person> parsed2 = ParquetParser.DEFAULT.read(bytes2, List.class, Person.class);

Serializer settings

Configured through ParquetSerializer.create() (see ParquetSerializer.Builder).

Builder method	Default	Description
`compressionCodec(CompressionCodec)`	`UNCOMPRESSED`	Page compression on write. `UNCOMPRESSED` or `GZIP`. `null` resets to `UNCOMPRESSED`.
`rowGroupSize(int)`	128 MiB	Target row group size in bytes. Values below 1024 are raised to 1024.
`pageSize(int)`	1 MiB	Target page size in bytes. Values below 1024 are raised to 1024.
`addBeanTypesParquet(boolean)`	`false`	Adds a `_type` column for polymorphic beans.
`writeDatesAsTimestamp(boolean)`	`true`	Dates/temporals as `INT64` `TIMESTAMP_MILLIS`; when `false`, as ISO 8601 strings.
`emitLogicalTypes(boolean)`	`false`	Also writes the Parquet `LogicalType` union into the footer for columns whose physical encoding is ambiguous (currently `UUID` only). Legacy `convertedType` is still written.
`nativeLogicalTypes(boolean)`	`false`	Writes binary-native `DECIMAL`, `DATE`, `TIME` and `TIMESTAMP` (micros) columns. See below.
`cycleHandling(ParquetCycleHandling)`	`NULL`	How to handle cyclic references: `THROW` fails with `SerializeException`; `NULL` writes cycles as null.
`maxRecursionDepth(int)`	`5`	Expansion depth for self-referential bean types (e.g. `Node.next`). Deeper structure is truncated to null. Values below 1 are clamped to 1.
`nullKeyString(String)`	`"<NULL>"`	Placeholder written for null `Map` keys (Parquet MAP keys are UTF-8).

The defaults for emitLogicalTypes, maxRecursionDepth and nullKeyString can be overridden with the ParquetSerializer.emitLogicalTypes, ParquetSerializer.maxRecursionDepth and ParquetSerializer.nullKeyString environment settings.

ParquetSerializer s = ParquetSerializer.create()
    .compressionCodec(CompressionCodec.GZIP)
    .rowGroupSize(32 * 1024 * 1024)
    .pageSize(256 * 1024)
    .writeDatesAsTimestamp(true)
    .cycleHandling(ParquetCycleHandling.THROW)
    .build();

Parser settings

Builder method	Default	Description
`nullKeyString(String)`	`"<NULL>"`	A `Map` key equal to this string is restored as null. Must match the serializer's value for round trips.

The parser detects the compression codec per page from the file itself. Supported codecs are UNCOMPRESSED, GZIP and SNAPPY (decode-only); any other codec is reported as a parse error.

Type mapping

Given a bean property type, the serializer writes the following Parquet column (default settings):

Java type	Parquet physical / converted type
`boolean` / `Boolean`	`BOOLEAN`
`byte`, `short`, `int` and wrappers	`INT32` (`INT_8`, `INT_16`, `INT_32`)
`long` / `Long`	`INT64` (`INT_64`)
`Number` (abstract)	`INT64`, re-typed to the concrete `INT32` / `INT64` / `DOUBLE` type when a runtime sample is available
`float` / `double` and wrappers	`FLOAT` / `DOUBLE`
`String` and other `CharSequence` types	`BYTE_ARRAY` (`UTF8`)
Enums	`BYTE_ARRAY` (`ENUM`)
`byte[]`	`BYTE_ARRAY`
Date / Calendar / `java.time` temporals	`INT64` (`TIMESTAMP_MILLIS`), or ISO 8601 `UTF8` string when `writeDatesAsTimestamp(false)`
`Duration`	`UTF8` string
`UUID`	`FIXED_LEN_BYTE_ARRAY(16)`
Nested bean	Optional group with one child per property
`Collection` / array	`LIST`
`Map`	`MAP` (keys stored as strings; non-string keys use `toString()` / enum name)
`Optional<X>`	Optional group with a `value` child of type `X`
Anything else (including `BigDecimal` / `BigInteger` by default)	`UTF8` string

Properties marked with @ParentProperty are excluded from the schema.

External-read interop

The parser reads spec-compliant Parquet produced by external writers (parquet-mr / Spark / Hive), not just files Juneau itself emits:

Multi-page column chunks — a column chunk split across several data pages is read end-to-end.
Multi-row-group files — every row group is read and concatenated, not just the first.
Dictionary-encoded pages — a DICTIONARY_PAGE followed by PLAIN_DICTIONARY / RLE_DICTIONARY data pages (the parquet-mr default) is decoded back through its dictionary.
Snappy compression — Snappy-compressed pages are decompressed by a built-in, dependency-free Snappy block decoder. Snappy is decode-only; on write use UNCOMPRESSED or GZIP. LZ4, Zstd, Brotli, and LZO remain unsupported and are reported as a hard error rather than silent garbage.

The pageSize and rowGroupSize builder settings control how Juneau splits its own output into pages and row groups (useful for streaming large datasets and for interop testing).

Binary-native logical types

The parser always decodes the binary-native logical types external tools emit:

INT96 — legacy 12-byte timestamps (Impala / Hive / older Spark) decode to instants.
DECIMAL (INT32 / INT64 backed) decodes to BigDecimal with the schema's scale.
DATE, TIME (micros), TIMESTAMP (micros) decode to the matching java.time types, preserving sub-millisecond precision.

On write, these native encodings are opt-in via ParquetSerializer.create().nativeLogicalTypes(true). By default Juneau keeps its portable string / TIMESTAMP_MILLIS normalization, so default output is unchanged. Enable nativeLogicalTypes() for sub-millisecond-precise, parquet-mr-interoperable output.

With nativeLogicalTypes(true), the following mappings replace the defaults in the type-mapping table:

Java type	Native column
`BigDecimal` / `BigInteger`	`DECIMAL` on `INT64`, scale 9, precision 18 (extra fractional digits are rounded `HALF_UP`)
`LocalDate`	`DATE` (`INT32`)
`LocalTime` / `OffsetTime`	`TIME(MICROS)` (`INT64`)
Other date / calendar / temporal types	`TIMESTAMP(MICROS)` (`INT64`)

// Emit binary-native DECIMAL / DATE / TIME / TIMESTAMP-micros columns.
ParquetSerializer s = ParquetSerializer.create().nativeLogicalTypes(true).build();

Annotations

The @ParquetConfig annotation (on classes or methods) applies serializer settings declaratively. Attribute values are strings and support variables.

Attribute	Description
`compressionCodec`	`GZIP` (case-insensitive); any other non-empty value selects `UNCOMPRESSED`.
`rowGroupSize`	Row group size in bytes.
`pageSize`	Page size in bytes.
`addBeanTypes`	`true` to add the `_type` column.

@ParquetConfig(compressionCodec="GZIP", pageSize="262144")
public class MyResource { ... }

Record cursors

Both classes also implement the low-level record cursor interfaces (RecordWritable / RecordReadable and the array variants), exposed on the marshaller as Parquet.ofRecords(...), Parquet.toRecords(...), Parquet.ofArrayRecords(...) and Parquet.toArrayRecords(...). Parquet cursors are buffered, not O(1) streaming (isRecordStreaming() returns false).

Quick Start​

Serializer settings​

Parser settings​

Type mapping​

External-read interop​

Binary-native logical types​

Annotations​

Record cursors​