Running Apache Spark Standalone

Fire can be run on Spark Standalone cluster. In this case, Hadoop does not need to be installed.

Installing Spark Standalone

Install Scala

Install Apache Spark

  • Download Spark

  • Extract, create a new directory under the /usr/local called spark and copy the extracted connect into it

    • tar xf spark-2.1.0-bin-hadoop2.7.tgz
    • mkdir /usr/local/spark
    • cp -r spark-2.1.0-bin-hadoop2.7/* /usr/local/spark
  • Setup some Environment variables before you start spark-shell ( in .bash_profile)

    • export SPARK_EXAMPLES_JAR=/usr/local/spark/examples/jars/spark-examples_2.11-2.0.0.jar
    • PATH=$PATH:$HOME/bin:/usr/local/spark/bin
  • Start you Scala Shell and run Spark

    • Go to sparkflows home directory
    • cd /usr/local/spark/bin
    • ./spark-shell
    Standalone spark
  • Start a standalone master server by executing:

    • ./sbin/ ( from spark home directory)
  • Once started, the master will print out a spark://HOST:PORT URL

  • You can also find this URL on the master’s web UI,

    Standalone spark

Setup Spark Slave(Worker) Node

  • Go to SPARK_HOME/conf/ directory.
  • Edit the file – Set SPARK_MASTER_HOST
    • If is not present, would be present. Make a copy of with name and add/edit the field SPARK_MASTER_HOST. Part of the file with SPARK_MASTER_HOST
    • cp ./conf/ ./conf/
  • Add a line in :

Start spark as slave

  • Goto SPARK_HOME/sbin and execute the following command.
    • ./ spark://MASTER_HOST_IP:7077

Installing Fire

Install Fire on the master node.

  • Download Fire Jar from website

  • Go to below directory:

    • cd fire-x.y.z
    • Update the port of Fire-ui & Fire to 8090 & 8082 as default port 8080 & 8081 is used by standalone spark, we can chose any other also.
    • From fire-x.y.z directory, we need to go conf/ and update the port No.
    Standalone spark
  • Create database & run fire & fire-ui server

    • ./
    • ./ start
    • ./ start

Configuring Fire

Below are the configuration for Fire to submit the jobs to the Spark Standalone Cluster.

  • Once The server fire & fire-ui start

configurations in spark

The following configurations have to be set appropriately

  • Go to administration section and open Spark configuration there we need to add Below details in specific setup like below:
    • spark.master: spark://Master_host_ip:7077
    • spark.deploy-mode: client
    • spark.sql-context: SQLContext
    • After above updates save the configurations.
    Standalone spark

Now go to application and try to run any workflows

Standalone spark