ilum/spark-launcher repository overview

⁠Ilum Spark Launcher

⁠Overview

ilum/spark-launcher is a lightweight Docker image designed for running spark-submit commands from external systems (e.g., CI/CD pipelines, scheduled jobs, Kubernetes CronJobs). Unlike full Spark runtime images, these launcher images contain only the Spark client binaries and essential cloud storage connectors, optimized for job submission rather than execution.

• Minimal Footprint — Java + Spark binaries only, no runtime dependencies
• Multi-Cloud Ready — pre-installed connectors for AWS S3, Google Cloud Storage, and Azure Blob Storage
• Version Flexibility — dedicated images for Spark 3.4.x and 3.5.x
• CI/CD Friendly — ideal for automated job submission pipelines

⁠TL;DR / Quick Start

# Submit a Spark job to a remote cluster
docker run --rm \
  -v $(pwd)/my-app.jar:/app.jar:ro \
  ilum/spark-launcher:3.5.7 \
  ./bin/spark-submit \
    --master k8s://https://kubernetes.default.svc:443 \
    --deploy-mode cluster \
    --class com.example.MyApp \
    /app.jar

Note: For Ilum-managed Spark clusters, use the Ilum Core REST API⁠ instead of direct spark-submit.

⁠Supported Tags

⁠Spark 3.5.x Series (Recommended)

• 3.5.7, latest — Spark 3.5.7 + Java 17 JRE
• 3.5.3 — Spark 3.5.3 + Java 17 JRE
• 3.5.1 — Spark 3.5.1 + Java 17 JRE
• 3.5.0 — Spark 3.5.0 + Java 17 JRE

⁠Spark 3.4.x Series (Legacy)

• 3.4.1 — Spark 3.4.1 + Java 11 Alpine
• 3.4.0 — Spark 3.4.0 + Java 11 Alpine

⁠Use Cases

⁠1. CI/CD Pipelines

Submit compiled Spark applications from GitLab CI, GitHub Actions, or Jenkins:

# .gitlab-ci.yml example
deploy_spark_job:
  image: ilum/spark-launcher:3.5.7
  script:
    - spark-submit --master k8s://... --deploy-mode cluster /builds/my-app.jar

⁠2. Kubernetes CronJobs

Schedule periodic Spark job submissions:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-etl
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: spark-submit
            image: ilum/spark-launcher:3.5.7
            command: ["/usr/local/spark/bin/spark-submit"]
            args: ["--master", "k8s://...", ...]

⁠3. External Job Triggers

Submit Spark jobs from non-JVM systems (Python, Node.js, shell scripts):

docker run --rm \
  -v /path/to/app:/app:ro \
  -e AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY \
  ilum/spark-launcher:3.5.7 \
  spark-submit --master yarn /app/etl.jar

⁠Environment Variables

⁠Cloud Credentials (Optional)

When submitting jobs to cloud storage, provide credentials via environment variables:

CAUTION: Use Kubernetes Secrets or CI/CD secret management, never hardcode credentials!

⁠Data Persistence (Volumes)

docker run --rm \
  -v $(pwd)/target/my-app.jar:/app/my-app.jar:ro \
  -v $(pwd)/spark-defaults.conf:/conf/spark-defaults.conf:ro \
  ilum/spark-launcher:3.5.7 \
  spark-submit --conf spark.config.location=/conf /app/my-app.jar

⁠Security

⁠Non-root Not Required

These images run as root by default since they only execute client commands. For enhanced security in restricted environments, user switching is supported:

# In your CI Dockerfile
FROM ilum/spark-launcher:3.5.7
USER 1000:1000

⁠Minimal Attack Surface

• No SSH server
• No runtime daemons
• No persistent processes

⁠Vulnerability Scanning

# Trivy
trivy image ilum/spark-launcher:3.5.7

⁠Building the Image

Run command from the ilum-ops/docker/ilum-spark-launcher directory:

# Build specific Spark version
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t ilum/spark-launcher:3.5.7 \
  -f Dockerfile-spark-3.5.7 \
  --push .

⁠Image Verification

⁠Layer Analysis (Dive)

dive ilum/spark-launcher:3.5.7

⁠Software Bill of Materials (SBOM)

# Syft
syft ilum/spark-launcher:3.5.7 -o spdx-json > sbom.json

⁠Image Contents — Details

⁠Base Components

• Spark 3.5.7: Java 17 JRE (ilum/java:17.0.11_9-jre)
• Spark 3.4.1: Java 11 Alpine (ilum/java-alpine:4.0)

⁠Apache Spark Binaries

• spark-submit — job submission tool
• spark-shell — interactive Scala shell (not typically used in launcher context)
• pyspark — Python API
• Core Spark libraries

⁠Pre-installed Cloud Connectors

⁠AWS S3

• hadoop-aws-3.3.4.jar
• aws-java-sdk-bundle-1.12.262.jar

⁠Google Cloud Storage

• gcs-connector-hadoop3-2.2.13-shaded.jar

⁠Azure Blob Storage

• hadoop-azure-3.3.4.jar
• azure-storage-7.0.1.jar
• jetty-util-ajax-9.4.43.v20210629.jar
• jetty-util-9.4.43.v20210629.jar

⁠Differences from Runtime Images

⁠Support and Feedback

• Issues: Bug Tracker⁠
• Documentation: Ilum Docs⁠
• Slack: Join our Community⁠