Skip to main content

Token Streaming

The TokenReader / TokenWriter surface exposes structural events one at a time — START_OBJECT, FIELD_NAME, VALUE_STRING, END_OBJECT, etc. — for formats whose wire encoding is already token-shaped.

Available on formats that implement the TokenReadable / TokenWritable role: json, json5, jsonl, cbor, msgpack (read + write), plus jsonschema and jcs (writer-only).

See the per-format capability matrix for the full breakdown of which formats expose the token cursor for reading vs writing.

Pull side: TokenReader​

try (TokenReader r = JsonParser.DEFAULT.readTokens("{\"a\":1,\"b\":[true,null]}")) {
    while (r.next() != TokenType.END_OF_STREAM) {
        switch (r.getCurrentToken()) {
            case FIELD_NAME   -> handleKey(r.getFieldName());
            case VALUE_STRING -> handleString(r.getString());
            case VALUE_NUMBER -> handleNumber(r.getNumber());
            case VALUE_BOOLEAN -> handleBool(r.getBool());
            case VALUE_NULL   -> handleNull();
            default -> { /* structural */ }
        }
    }
}

Key methods:

  • next() — advances and returns the new current token type.
  • getCurrentToken() — current token without advancing.
  • getDepth() — nesting depth (0 at root).
  • getString() / getNumber() / getNumberLexeme() / getBool() / getBinary() — accessors for the current scalar.
  • getFieldName() — when current token is FIELD_NAME.
  • skipChildren() — fast-forward past the current container's body to its matching end.
  • read(Class) — bridge from token cursor to the format's databind path: consumes one whole value at the cursor's current position and binds it to a POJO.

For numeric values both the original lexeme and the parsed Number are available — use getNumberLexeme() for big-decimal / precision-sensitive consumers.

Push side: TokenWriter​

StringWriter sw = new StringWriter();
try (TokenWriter w = JsonSerializer.DEFAULT.writeTokens(sw)) {
    w.startObject();
    w.fieldName("a");
    w.number(1);
    w.fieldName("b");
    w.startArray();
    w.bool(true);
    w.nil();
    w.endArray();
    w.endObject();
}
// sw is now: {"a":1,"b":[true,null]}

The output target must be a Writer or OutputStream (for byte-based formats). To capture output as a string, use a StringWriter and read sw.toString() afterward.

Method families:

  • startObject() / endObject() / startArray() / endArray() — structural framing.
  • fieldName(String) — emits a field name (must precede the value).
  • string(String) / number(...) / bool(boolean) / nil() / binary(byte[]) — scalar values.
  • object(Object) — bridge from token writer to the format's databind path: emits a whole POJO at the current cursor position via PojoWalker.

It's the caller's responsibility to emit a well-formed event sequence (every startObject() matched by an endObject(), every field name followed by exactly one value or container, etc.). Implementations are not required to validate.

Format-specific cursor types​

Each token-capable format exposes its own concrete subtype for callers that want compile-time specificity:

JsonTokenReader r = (JsonTokenReader) JsonParser.DEFAULT.readTokens(input);
CborTokenWriter w = (CborTokenWriter) CborSerializer.DEFAULT.writeTokens(output);

Or declare them as @RestOp parameters — the framework picks the matching parser automatically (see REST Streaming Integration).

Settings propagation​

Cursor-level settings are derived from the parser/serializer's builder. The following families are honored on token cursors:

  • Reader (from Parser.Builder): autoCloseStreams, unbuffered, trimStrings, streamCharset.
  • Writer (from JsonSerializer.Builder): useWhitespace, maxIndent, quoteChar / quoteCharOverride, escapeSolidus, trimStrings.

These are explicitly NOT honored at the token layer — they're databind-path-only: addBeanTypes*, addRootType, keepNullProperties, trimEmptyMaps, trimEmptyCollections, sortMaps, sortCollections, swap configuration, @Schema annotations.

See also​