Dockerfile linksApache Hive 3.1.2|Apache Spark 2.4.8|Apache Livy 0.7.1-incubating
See Apache Hive on Spark docs for more information.
Lots happening here, but in short this image allows you to run Hive with Spark as the compute engine. Spark itself uses YARN as the resource manager which we leverage from the underlying Hadoop install.
See documentation on the underlying Hive base Docker image for details on how Hadoop/YARN has been configured.
docker run --rm -d\
--publish 10000:10000\
--publish 10002:10002\
--publish 8032:8032\
--publish 8088:8088\
--publish 8042:8042\
--publish 18080:18080\
--publish 8998:8998\
--name hive-on-spark loum/hive-on-spark:latest
Login to beeline (!q to exit CLI):
docker exec -ti hive-on-spark bash -c "HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000"
Check the Beeline Command Reference for more.
Some other handy commands to run with beeline:
Create a Hive table named test:
docker exec -ti hive-on-spark bash -c\
"HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000\
-e 'CREATE TABLE test (c CHAR(10));'"
To show tables:
docker exec -ti hive-on-spark bash -c\
"HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000\
-e 'SHOW TABLES;'"
To insert a row of data into Hive table test (invoke the Spark execution engine through YARN):
docker exec -ti hive-on-spark bash -c\
"HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000\
-e 'INSERT INTO TABLE test VALUES ('\''test'\'');'"
To select all rows in Hive table test:
docker exec -ti hive-on-spark bash -c\
"HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000\
-e 'SELECT * FROM test;'"
To drop the Hive table test:
docker exec -ti hive-on-spark bash -c\
"HADOOP_HOME=/opt/hadoop /opt/hive/bin/beeline -u jdbc:hive2://localhost:10000\
-e 'DROP TABLE test;'"
Alternatively, port 10000 is exposed to allow connectivity to clients with JDBC.
The Spark computing system is available and can be invoked as per normal.
More information on submitting applications to Spark can be found here.
The sample SparkPi application can be launched with:
docker exec -ti hive-on-spark bash -c\
"HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop /opt/spark/bin/spark-submit\
--class org.apache.spark.examples.SparkPi\
--files /opt/spark/conf/metrics.properties.template\
--master yarn\
--deploy-mode cluster\
--driver-memory 1g\
--executor-memory 1g\
--executor-cores 1\
/opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar"
Apart from verbose logging displayed on the console it may appear that not much has happened here. However, since the Spark application has been deployed in cluster mode you will need to dump the associated application ID's log to see meaningful output.
To get a list of Spark application logs (under YARN):
docker exec -ti hive-on-spark bash -c "/opt/hadoop/bin/yarn application -list -appStates ALL"
Then plug in an Application-Id into:
docker exec -ti hive-on-spark bash -c "/opt/hadoop/bin/yarn application -list -appStates ALL <Application-Id>"
To see something similar to the following:
====================================================================
LogType:stdout
LogLastModifiedTime:Sat Apr 11 21:49:03 +0000 2020
LogLength:33
LogContents:
Pi is roughly 3.1398156990784956
End of LogType:stdout
***********************************************************************
pysparkdocker exec -ti hive-on-spark bash -c "/opt/spark/bin/pyspark"
spark-shelldocker exec -ti hive-on-spark bash -c "/opt/spark/bin/spark-shell"
docker stop hive-on-spark
The following web interfaces are available to view configurations and logs and to track YARN/Spark job submissions:
Content type
Image
Digest
Size
1.4 GB
Last updated
about 5 years ago
docker pull loum/hive-on-spark