loum/jupyter-spark-pseudo repository overview

Need a notebook to simulate production loads on Apache Spark on YARN over Pseudo Distributed Hadoop? Otherwise, check an alternate implementation, jupyter-pyspark⁠, which is much smaller in size and ideal for experimentation.

⁠What's new with latest

• Bump to Jupyter Notebook 6.5.1⁠
• Bump to Apache Spark 3.3.0⁠

⁠Supported tags and respective Dockerfile links

Jupyter Notebook 6.5.1⁠ | Apache Spark 3.3.0⁠

• latest, 6.5.1-3.3.0⁠

Jupyter Notebook 6.4.12⁠ | Apache Spark 3.3.0⁠

• 6.4.12-3.3.0⁠

Jupyter Notebook 6.4.10⁠ | Apache Spark 3.2.1⁠

• 6.4.10-3.2.1⁠

Jupyter Notebook 6.4.8⁠ | Apache Spark 3.2.1⁠

• 6.4.8-3.2.1⁠

Jupyter Notebook 6.4.6⁠ | Apache Spark 3.2.0⁠

• 6.4.6-3.2.0⁠

⁠Quick reference

• GitHub: loum⁠/jupyter-spark-pseudo⁠
• Where to file issues: https://github.com/loum/jupyter-spark-pseudo/issues⁠
• Apache Hadoop home: http://hadoop.apache.org/⁠
• Apache Spark home: Apache Spark 3.3.0⁠
• Jupyter Notebook: https://jupyter.org/⁠

⁠Quick start

⁠Launch the Jupyter Notebook Server Container

Place your notebooks under the notebook directory and run:

docker run --rm -d\
 --name jupyter-spark-pseudo\
 --hostname jupyter-spark-pseudo\
 --env JUPYTER_PORT=8889\
 --env YARN_SITE__YARN_NODEMANAGER_RESOURCE_DETECT_HARDWARE_CAPABILITIES=true\
 --volume $PWD/notebooks:/home/hdfs/notebooks\
 --publish 8032:8032\
 --publish 8088:8088\
 --publish 8042:8042\
 --publish 18080:18080\
 --publish 8889:8889\
 loum/jupyter-spark-pseudo:latest

To get the URL of your local server:

docker exec -ti jupyter-spark-pseudo bash -c "jupyter notebook list"

⁠Terminate Jupyter Notebook Server Container

docker stop jupyter-spark-pseudo

⁠So far, so good!

With an active Jupyter Notebook server a PySpark instance should be running behind the scenes. To validate, dump the SparkContext help in a Jupyter Notebook cell:

help(sc)

⁠Things to note when coding in your Notebooks...

Spark context available as sc.

SparkSession available as spark.

⁠Example

Taken from Spark examples⁠:

import sys
from random import random
from operator import add
​
partitions = 2
n = 100000 * partitions
​
def f(_):
    x = random() * 2 - 1
    y = random() * 2 - 1
    return 1 if x ** 2 + y ** 2 <= 1 else 0
​
count = sc.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
print("Pi is roughly %f" % (4.0 * count / n))

[Stage 0:>                                                          (0 + 2) / 2]
Pi is roughly 3.143960