ilum/sparkmagic repository overview

⁠Ilum Sparkmagic

⁠Overview

ilum/sparkmagic is Ilum's production-hardened Jupyter environment built on sparkmagic, enabling interactive Spark development through Apache Livy or Ilum Core. This distribution includes significant enhancements for enterprise use: integrated SSH server for VS Code Remote, Jupyter AI for LLM-powered coding assistance, and seamless Ilum platform integration.

• Sparkmagic Foundation — remote Spark execution via Livy/Ilum compatible REST API
• SSH Server — optional SSH access on port 2222 for VS Code Remote integration
• Jupyter AI — LLM-powered code assistance (Anthropic, AWS, Google, OpenAI)
• Ilum Integration — automatic Spark job tagging, auto-pause support, zero-config endpoints
• Enhanced JupyterLab 4.4.10 — git, execute-time, favorites extensions pre-installed
• iFrame Embeddable — CSP configuration for dashboard integration

⁠TL;DR / Quick Start

# Standard run (SSH disabled by default)
docker run -d \
  --name ilum-jupyter \
  -p 8888:8888 \
  ilum/sparkmagic:<version>

# Access Jupyter at http://localhost:8888

Note: For production deployments, use the official Ilum Jupyter Helm Chart⁠.

⁠Supported Tags

• <version>: Sparkmagic + JupyterLab 4.4.10 + Jupyter AI + SSH server

⁠Key Features

⁠Ilum-Specific Enhancements

⁠1. Integrated SSH Server

Enable SSH access for seamless VS Code Remote development:

docker run -e SSH_ENABLED=true -p 2222:2222 ilum/sparkmagic:<version>

Mount SSH keys via volume or Kubernetes Secret for authentication.

⁠2. Jupyter AI Integration

Pre-installed LLM providers for AI-assisted coding:

• Anthropic (Claude)
• AWS Bedrock
• Google Generative AI
• OpenAI

Access via JupyterLab AI chat interface.

⁠3. Automatic Spark Job Tagging

All Spark sessions automatically include spark.ilum.source=jupyter for origin tracking and analytics.

⁠4. Auto-Pause Configuration

UI checkbox in session creation widget for spark.ilum.autopause — automatically pause idle Spark sessions to save resources.

⁠5. Enhanced JupyterLab Extensions

• jupyterlab-git — Git integration
• jupyterlab-execute-time — Cell execution timing
• jupyterlab_favorites — Quick access to frequently used files

⁠6. iFrame Embedding Support

Custom CSP header (frame-ancestors *) enables embedding in external dashboards and portals.

⁠7. Zero-Configuration Ilum Integration

Default Livy endpoint pre-configured to http://ilum-core:9888 for immediate use in Ilum deployments.

⁠Environment Variables

⁠Jupyter Configuration

⁠SSH Server

⁠Jupyter AI (Optional)

⁠Data Persistence (Volumes)

# docker-compose example
volumes:
  - ./notebooks:/home/jovyan/work
  - ~/.ssh/authorized_keys:/home/jovyan/.ssh/authorized_keys:ro

⁠Network Configuration

⁠Security

⁠Non-root Architecture

The image runs as user jovyan (UID 1000, non-root). Ensure mounted volumes have appropriate permissions:

# On host
sudo chown -R 1000:100 ./notebooks

⁠SSH Key Management

IMPORTANT: For SSH access, mount authorized_keys as read-only:

volumes:
  - ~/.ssh/authorized_keys:/home/jovyan/.ssh/authorized_keys:ro

In Kubernetes, use a Secret:

volumes:
  - name: ssh-keys
    secret:
      secretName: jupyter-ssh-keys
      defaultMode: 0600

⁠Jupyter AI Security

CAUTION: Never pass API keys directly in environment variables in production!

Use Kubernetes Secrets:

env:
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: jupyter-ai-secrets
        key: openai-api-key

⁠Vulnerability Scanning

Before production deployment, scan the image:

# Trivy
trivy image ilum/sparkmagic:<version>

⁠Building the Image

Run command from the root of the sparkmagic repository:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t ilum/sparkmagic:<version> \
  -f docker/Dockerfile-jupyter \
  --push .

⁠Image Verification

⁠Layer Analysis (Dive)

dive ilum/sparkmagic:<version>

⁠Software Bill of Materials (SBOM)

# Syft
syft ilum/sparkmagic:<version> -o spdx-json > sbom.json

⁠Image Contents — Details

⁠Base Components

• Base: Jupyter Project base-notebook:hub-5.2.1 (Ubuntu-based)
• JupyterLab: 4.4.10
• Python: 3.x (from base image)
• User: jovyan (UID 1000)

⁠Sparkmagic Components

• hdijupyterutils — Jupyter utilities
• autovizwidget — Automatic SQL visualization widget
• sparkmagic — Core sparkmagic library with Livy client

⁠Sparkmagic Kernels

• PySpark Kernel — Python + Spark
• Spark Kernel — Scala + Spark
• SparkR Kernel — R + Spark
• Python 3 — Standard IPython kernel (local execution)

⁠Ilum-Added Packages

• jupyter-ai — AI chat interface
• langchain-anthropic, langchain-aws, langchain-google-genai, langchain-openai — LLM providers
• jupyterlab-git — Git version control
• jupyterlab-execute-time — Execution timing
• jupyterlab_favorites — Favorites sidebar
• openssh-server — SSH daemon
• requests-kerberos — Kerberos authentication support

⁠System Tools

• tini — Init process for proper signal handling
• sudo, git, curl — Development utilities

⁠Support and Feedback

• Issues: Bug Tracker⁠
• Documentation: Ilum Docs⁠
• Slack: Join our Community⁠