mancube/shapeshift repository overview

⁠shapeshift

Stream JSON/JSONL into Apache Parquet & Apache Iceberg — one static, dependency-free CLI.

shapeshift is a small, fast shaper: point it at newline-delimited (or arrayed) JSON and it writes a typed Parquet file or an Iceberg v2 table. It streams one record at a time, so memory stays flat whether the input is 1 MB or 1 TB.

This image is the lean default build — Snappy compression, pure Rust, no cloud SDKs — compiled as a single static musl binary on scratch. No libc, no shell, no package manager: just the binary.

⁠Where it fits

shapeshift is the ingest/shaping stage of a lakehouse: it turns raw JSON coming off a queue, a database export, or an API into the columnar, catalog-ready files your query engines read. It owns the JSON → Parquet/Iceberg hop and nothing else — no cluster, no daemon, no coordinator.

   SOURCES                 SHAPESHIFT                 STORAGE            CATALOG              QUERY / COMPUTE
   (JSON/JSONL)            (this image)               (files)           (table metadata)     (read the tables)

 ┌──────────────┐
 │ Queue        │  Kafka / Kinesis / PubSub ─┐
 │ (stream)     │                            │
 └──────────────┘                            │
 ┌──────────────┐                            │      ┌─────────────┐    ┌──────────────┐    ┌──────────────────┐
 │ Database     │  Postgres/MySQL export ────┼────▶ │  Parquet    │    │ Iceberg      │    │ DuckDB / Trino   │
 │ (CDC/dump)   │  Mongo / DynamoDB JSON     │      │  file(s)    │───▶│ v2 catalog   │──▶ │ Spark / Athena   │
 └──────────────┘                            │      └─────────────┘    │ (manifests,  │    │ Flink / Polars   │
 ┌──────────────┐        ┌───────────────┐   │      ┌─────────────┐    │  snapshots)  │    └──────────────────┘
 │ API / logs   │  ────▶ │  shapeshift   │ ◀─┘      │ Iceberg     │───▶│              │
 │ (files/HTTP) │        │ shape · infer │          │ data + meta │    └──────────────┘
 └──────────────┘        └───────┬───────┘          └─────────────┘
                                 │  local disk  ·  or S3 / GCS / Azure (objstore build)
                                 │
                                 └──▶ <output>.rejects.jsonl  (bad lines, with line numbers)

• Upstream — anything that emits JSON records: a queue drained to JSONL, a DB dump/CDC feed, API responses, app logs. shapeshift reads a stream, not a connection — you pipe or mount the JSON; it does not poll brokers itself.
• shapeshift — infers or applies a schema, coerces types, then writes.
• Downstream — Parquet for ad-hoc/columnar reads; Iceberg v2 (manifests + snapshots) for a real table other engines register in a catalog and query.

⁠Tags

⁠Quick start

Shape a JSONL file to Parquet (mount your data at /data):

docker run --rm -v "$PWD:/data" mancube/shapeshift \
  shape -i /data/events.jsonl -o /data/events.parquet

Infer a schema, or write an Iceberg table:

docker run --rm -v "$PWD:/data" mancube/shapeshift infer -i /data/events.jsonl
docker run --rm -v "$PWD:/data" mancube/shapeshift \
  shape -i /data/events.jsonl --to iceberg -o /data/warehouse/events

Inspect the output, or see the CLI:

docker run --rm -v "$PWD:/data" mancube/shapeshift inspect /data/events.parquet
docker run --rm mancube/shapeshift --help

⁠What it does

• Streaming — flat RAM (single-digit MiB) regardless of input size; the ceiling is disk, not memory.
• Typed output — schema inference or an explicit spec; path selection, transforms, coercion, and required-column enforcement.
• Parquet — one row group per batch (bounded RAM); Snappy by default.
• Iceberg v2 — field-id-carrying data files, Avro manifests, snapshots, and additive schema evolution on --append; hidden/identity partitioning.
• Never aborts on a bad line — malformed records go to a <output>.rejects.jsonl sidecar with their line number.

The optional cloud object-store sink (S3/GCS/Azure) and the zstd codec are extra build features and are not in this image; build from source to enable them.

⁠Source & docs

Apache-2.0. Source, full docs, and the standalone binary release: https://github.com/lucheeseng827/shapeshift⁠