Launching apache spark using supervisord

Apache Spark comes bundled with several launch scripts to start the master and worker processes. We can launch a master and two workers processes by executing

./sbin/start-master.sh
./sbin/start-slaves.sh 1 spark://master.mydomain.com:7077
./sbin/start-slaves.sh 2 spark://master.mydomain.com:7077

The issue with launching the cluster this way is that each time the cluster reboots you will need to re-execute the shell scripts above. This process is cumbersome and prone to error.

Supervising master and worker processes

A better approach requires the use of supervisord (a process control system) to manage spark processes. It has the following benefits:

• No need to create shell scripts for managing processes
• Centralized monitoring mechanism for sensitive services
• Accurate up/down statuses, even when a process crashes

The only requirements is that processes must be run in the foreground. Therefore the master and worker(s) processes should be initiated through spark-class instead of the sbin shell scripts as they run the processes as a daemon.

Supervisor configuration files

The master process will need to start prior to any workers.

[program:spark_master]
command=/bin/spark-class org.apache.spark.deploy.master.Master --ip master.mydomain.com --port 7077 --webui-port 18080
stderr_logfile = /var/log/spark_master-stderr.log
stdout_logfile = /var/log/spark_master-stdout.log
priority=1
autostart=true

Supervisor is configured to start retries 3 times. That is usually enough time for the master process to start before the workers can begin connecting to it.

[program:spark_worker]
command=/bin/spark-class org.apache.spark.deploy.worker.Worker spark://master.mydomain.com:7077
stderr_logfile = /var/log/spark_worker-stderr.log
stdout_logfile = /var/log/spark_worker-stdout.log
priority=2
autostart=true

It’s convenient to drop your configuration files into /supvisor/conf.d/ as they will be picked up automatically when you start the Supervisord service. It allows you to easily distribute your configuration files across your cluster.

The priority flag means that the master will start prior to the worker. You can also set the number of retries (default is 3) for the worker in case the master has not completed its startup sequence.