ilum/spark repository overview

⁠Ilum Spark

⁠Overview

ilum/spark is Ilum's production-hardened Apache Spark image optimized for Kubernetes deployments. It extends the official Apache Spark image with:

• Multi-Cloud Storage — pre-installed connectors for AWS S3, Google Cloud Storage, and Azure Blob Storage
• Data Lakehouse — Delta Lake 3.3.2 + Spark Connect for lakehouse workloads
• SQL Gateway — Kyuubi 1.10.2 extensions for Spark SQL optimization and lineage
• Data Lineage — OpenLineage 1.40.1 for end-to-end data observability
• Custom Variants — specialized images for Iceberg, Hudi, Sedona, Nessie, Unity Catalog, and more
• Dynamic Dependencies — runtime pip module injection via PIP_MODULES environment variable

⁠TL;DR / Quick Start

# Run Spark shell with cloud storage access
docker run -it --rm \
  -e AWS_ACCESS_KEY_ID=your-key \
  -e AWS_SECRET_ACCESS_KEY=your-secret \
  ilum/spark:3.5.7 \
  /opt/spark/bin/spark-shell

# Run as Kubernetes driver/executor
docker run -d \
  ilum/spark:3.5.7 driver \
  --master k8s://https://k8s-api:6443 \
  --class com.example.MyApp \
  s3a://bucket/app.jar

Note: For Ilum-managed Spark clusters, use the official Ilum Helm Chart⁠.

⁠Supported Tags

⁠Main Images

⁠Custom Variants

⁠🛡️ Security-Hardened Variant (DHI)

For enterprise deployments requiring maximum security, use the -dhi tag:

docker pull ilum/spark:3.5.7-dhi

⁠What is DHI?

DHI (Data & AI Hardened Images) is a Spark distribution built on Wolfi OS — a security-first, distroless Linux designed for containers.

⁠Security Features

⁠Included Ilum Enhancements

The DHI variant includes all standard Ilum customizations:

• ✅ Hadoop 3.4.1 + AWS SDK v2 (latest S3A connector)
• ✅ Kyuubi 1.10.2 extensions
• ✅ OpenLineage 1.40.1
• ✅ Delta Lake 3.3.2
• ✅ GCS and Azure connectors
• ✅ Spark Connect 3.5.7

⁠Usage

# Kubernetes deployment
spec:
  containers:
  - name: spark-driver
    image: ilum/spark:3.5.7-dhi
    securityContext:
      runAsNonRoot: true
      runAsUser: 65532
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true

⁠Ilum Customizations

⁠Pre-installed JARs (Main Image)

⁠Python Packages

grpcio pandas delta-spark==3.3.2 pyspark[sql,connect]==3.5.7

⁠Custom Entrypoint Features

The entrypoint.sh provides:

1. Dynamic pip installation — install modules at runtime:

   docker run -e PIP_MODULES="numpy;scikit-learn" ilum/spark:3.5.7 driver ...
   

   Copy

2. Kubernetes modes — driver and executor commands for K8s deployments

3. tini init — proper signal handling in containers

⁠Environment Variables

⁠Cloud Storage Credentials

⁠Runtime Configuration

⁠Data Persistence (Volumes)

⁠Network Configuration

⁠Security

⁠Non-root Architecture

The image runs as spark user (UID defined by spark_uid). Root is used only during build for JAR installation.

⁠Secret Management

CAUTION: Never hardcode cloud credentials!

Use Kubernetes Secrets:

env:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: access-key

⁠Vulnerability Scanning

trivy image ilum/spark:3.5.7

⁠Building Images

⁠Main Image

cd ilum-ops/docker/ilum-spark-docker
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ilum/spark:3.5.7 --push .

⁠Custom Variants

cd custom

# Iceberg variant
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ilum/spark:3.5.7-iceberg -f Dockerfile-iceberg --push .

# Sedona variant
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ilum/spark:3.5.7-sedona -f Dockerfile-sedona --push .

⁠Image Contents — Details

⁠Base Components

• Spark: 3.5.7 (Scala 2.12, Java 17, Python 3)
• Init: tini for proper signal handling
• User: spark (non-root)

⁠Cloud Connectors

• AWS S3: Full SDK for S3, DynamoDB (for lock table)
• GCS: Shaded connector for Google Cloud Storage
• Azure: Blob Storage with Jetty dependencies

⁠Data Lakehouse

• Delta Lake: Full read/write support with optimization
• Spark Connect: Remote Spark sessions via gRPC

⁠Observability

• Kyuubi Extensions: SQL optimization, lineage tracking
• OpenLineage: Data lineage events for observability platforms

⁠Integration with Ilum Platform

When deployed via Ilum:

1. Automatic Configuration: Ilum Core injects Spark config via environment
2. Lineage Collection: OpenLineage events sent to Marquez
3. SQL Gateway: Kyuubi provides JDBC/ODBC access via ilum-kyuubi
4. Auto-Pause: Idle sessions automatically paused to save resources

⁠Support and Feedback

• Issues: Bug Tracker⁠
• Documentation: Ilum Docs⁠
• Slack: Join our Community⁠