ilum/marquez repository overview

⁠Ilum Marquez

⁠Overview

ilum/marquez is Ilum's production-hardened distribution of Marquez⁠, an open-source metadata service for data lineage collection, aggregation, and visualization. Starting with version 0.54.0, the API backend has been rewritten in Rust for improved performance and resource efficiency. This fork maintains 100% API compatibility with upstream Marquez while delivering critical fixes and additive enhancements like the Search API.

• Rust Backend — rewritten from Java for lower latency and memory footprint
• OpenLineage Native — built-in support for OpenLineage 2.0.2 specification
• 100% Compatible — drop-in replacement for upstream Marquez
• Enhanced Search — advanced Search API backed by OpenSearch for large-scale lineage exploration
• Lineage Statistics — dataset facet with upstream/downstream dependency counts
• Upstream First — contributing improvements back to Marquez project

Ilum Fork Philosophy: We forked Marquez when upstream development slowed, shipped critical fixes, then realigned with upstream from version 0.52.x. All enhancements are non-breaking and intended for community contribution.

⁠TL;DR / Quick Start

⁠Docker (Standalone)

# 1. Start PostgreSQL
docker run -d --name marquez-postgres \
  -e POSTGRES_USER=marquez \
  -e POSTGRES_PASSWORD=marquez \
  -e POSTGRES_DB=marquez \
  -p 5432:5432 \
  postgres:16

# 2. Start Marquez (wait ~5s for PostgreSQL to be ready)
docker run -d --name marquez \
  --network=host \
  -e POSTGRES_HOST=localhost \
  -e POSTGRES_PORT=5432 \
  -e POSTGRES_DB=marquez \
  -e POSTGRES_USER=marquez \
  -e POSTGRES_PASSWORD=marquez \
  ilum/marquez:0.54.0

Endpoints:

• HTTP API: http://localhost:5000
• Admin (healthcheck, metrics): http://localhost:5001

⁠Helm (Kubernetes)

# Clone repository and install with included PostgreSQL
git clone https://github.com/ilum-cloud/marquez.git
cd marquez/chart

helm install ilum-marquez . \
  --dependency-update \
  --set postgresql.enabled=true \
  --set postgresql.auth.password=your-secure-password

⁠Kubernetes Deployment (Helm)

⁠Installation

# Add Ilum Helm repository (if available) or clone directly
git clone https://github.com/ilum-cloud/marquez.git
cd marquez/chart

# Install with included PostgreSQL
helm install ilum-marquez . \
  --dependency-update \
  --set postgresql.enabled=true \
  --set postgresql.auth.password=your-secure-password

# Or install connecting to existing PostgreSQL
helm install ilum-marquez . \
  --set marquez.db.host=your-postgres-host \
  --set marquez.db.name=marquez \
  --set marquez.db.user=marquez \
  --set marquez.db.password=your-password

⁠Chart Configuration

⁠Example: Production Configuration

# values-production.yaml
marquez:
  replicaCount: 2
  resources:
    limits:
      memory: 512Mi
      cpu: 500m
    requests:
      memory: 256Mi
      cpu: 250m
  db:
    host: "postgres.database.svc.cluster.local"
    name: marquez
    user: marquez
    password: "" # Use existingSecretName instead
  existingSecretName: "marquez-db-credentials"
  migrateOnStartup: true

web:
  enabled: true
  resources:
    limits:
      memory: 512Mi
      cpu: 500m

ingress:
  enabled: true
  className: nginx
  hosts:
    - lineage.example.com

helm install ilum-marquez . -f values-production.yaml

⁠Accessing Services

# Port-forward API
kubectl port-forward svc/ilum-marquez 5000:9555

# Port-forward Web UI
kubectl port-forward svc/ilum-marquez-web 3000:9444

⁠Supported Tags

• 0.54.0, latest — Rust backend (default), Marquez 0.54.0 + OpenLineage 2.0.2
• 0.54.0-java — Java backend (deprecated, transitional)
• 0.53.1 — Previous stable release (Java backend)
• 0.53.0 — Previous stable release (Java backend)

⁠Recent Changes (0.54.0)

⁠Rust Backend (Default)

• Backend Rewrite: The API has been fully rewritten in Rust using Axum, SQLx, and Figment. This replaces the Java Dropwizard backend.
• Lower Resource Usage: Significantly reduced memory footprint and startup time compared to the JVM-based image.
• Full Search API: OpenSearch-backed full-text search for datasets and jobs.
• 100% API Compatibility: All existing REST API endpoints are preserved.

⁠Previous: 0.53.1

• Lineage Statistics Facet: New dataset facet providing upstream/downstream dependency visibility
• Full Search API (GET /api/v1/search/full): Returns complete dataset/job objects instead of references
• Dataset Fields Null Type Fix (#3083): Fixed critical bug causing exponential row explosion in column_lineage table

⁠OpenLineage Compatibility

⁠Environment Variables

⁠Database Configuration (Required)

Legacy aliases (MARQUEZ_DB_HOST, MARQUEZ_DB_PORT, MARQUEZ_DB, MARQUEZ_DB_USER, MARQUEZ_DB_PASSWORD) are also supported.

⁠Server Configuration

⁠Logging

⁠Search (Optional — OpenSearch)

⁠Advanced (Figment)

The Rust backend uses Figment⁠ for configuration. Any config field can be overridden via environment variables with the MARQUEZ_ prefix and __ (double underscore) for nesting:

⁠Data Persistence (Volumes)

Marquez stores all data in PostgreSQL. No container volumes required for the API server.

⁠Network Configuration

⁠API Endpoints

• POST /api/v1/lineage — OpenLineage event ingestion
• GET /api/v1/namespaces — List namespaces
• GET /api/v1/namespaces/{namespace}/jobs — List jobs
• GET /api/v1/namespaces/{namespace}/datasets — List datasets
• GET /api/v1/search/full — Full search with complete objects

Full API Documentation: OpenAPI Spec⁠

⁠Security

⁠Non-root Considerations

The image runs as root by default. For production, consider:

# Kubernetes - run as non-root
securityContext:
  runAsNonRoot: true
  runAsUser: 1000

⁠Authentication

⁠Secret Hygiene

Use Kubernetes Secrets:

env:
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: marquez-db-secret
        key: password

⁠Vulnerability Scanning

# Trivy
trivy image ilum/marquez:0.54.0

⁠Building the Image

Run from the repository root:

# Build default (Rust) image
docker build -t ilum/marquez:0.54.0 .

# Build deprecated Java image
docker build -f Dockerfile.java -t ilum/marquez:0.54.0-java .

⁠Image Verification

⁠Layer Analysis (Dive)

dive ilum/marquez:0.54.0

⁠Software Bill of Materials (SBOM)

syft ilum/marquez:0.54.0 -o spdx-json > sbom.json

⁠Image Contents — Details

⁠Base Components

• Builder: rust:slim-bookworm (multi-stage build)
• Runtime: debian:bookworm-slim with OpenSSL
• Database: SQLx with Flyway migrations (reused from Java)

⁠Marquez Components

• marquez-api — Core HTTP API (Axum)
• marquez-search — OpenSearch integration
• marquez-tracing — Sentry error tracking
• Figment — Configuration (YAML + environment variables)
• SQLx — Database access (raw SQL, no ORM)
• Flyway — Database migrations (auto-run on startup)

⁠Ilum Enhancements

• Search API — Advanced lineage search capabilities via OpenSearch
• Lineage Statistics Facet — Upstream/downstream dependency metrics
• Dataset Fields Fix — Null-type handling for column lineage
• Stability fixes — Production-tested bug fixes

⁠Architecture

┌─────────────────────────────────────────────────────┐
│                   Data Sources                      │
│  (Spark, Airflow, dbt, Flink via OpenLineage)       │
└──────────────────────┬──────────────────────────────┘
                       │ OpenLineage Events
                       ▼
┌─────────────────────────────────────────────────────┐
│              ilum/marquez (API)                     │
│  ┌─────────────┐              ┌─────────────┐       │
│  │ Lineage API │              │  Search API │       │
│  └─────────────┘              └─────────────┘       │
│                       │                             │
│              ┌────────┴────────┐                    │
│              ▼                 ▼                    │
│        PostgreSQL         OpenSearch                │
│      (metadata store)    (optional search)          │
└─────────────────────────────────────────────────────┘

⁠Integration with Ilum Platform

When deployed as part of the Ilum data lakehouse platform:

1. Automatic lineage: Spark jobs via Ilum Core emit OpenLineage events
2. Data catalog: Integration with Ilum UI for lineage visualization
3. Kafka transport: OpenLineage events can be sent via Kafka

⁠Support and Feedback

• Issues: Bug Tracker⁠
• Documentation: Ilum Docs⁠
• Slack: Join our Community⁠