install spark stand alone mode in ubuntu 16.04
preRequest:
1- install ssh server:
$ sudo apt-get install openssh-server
then to make your machine access to other machine without password:
this command to genarate key between machines via:
$ ssh-keygen -t rsa
this command to copy the id via:
$ ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@master
this command to change the mod:
$ chmod 0600 ~/.ssh/authorized_keys
then reboot the machine:
$ sudo reboot
#######################
2- install java:
here we install java 8.
first check if Java is not already installed:
$ java -version
if it not exist install it via:
$ sudo apt-get install default-jre
$ sudo apt-get install default-jdk
3- Then download Spark via these steps:
*First thing you must sure that the ip address of master and slaves write in the /etc/hosts via:
$ sudo gedit /etc/hosts
#########################################
install git:
i already installed it
$ sudo apt-get install git
*Then
*Download scala From:
https://scala-lang.org/download/
here we install scala-2.12.3.tgz
Extract it via:
$ sudo tar -xvf scala-2.12.3.tgz
*and then give it permission via:
$ sudo chown ubuntu:root /usr/local/scala-2.12.3
$ sudo chmod 777 /usr/local/scala-2.12.3/*
###########################################
3- install spark via:
https://spark.apache.org/downloads.html
here we install spark-2.2.0
Extract it via:
$ sudo tar -xvf spark-2.2.0-bin-hadoop2.7.tgz
*and then give it permission via:
$ sudo chown ubuntu:root /usr/local/spark-2.2.0-bin-hadoop2.7
$ sudo chmod 777 /usr/local/spark-2.2.0-bin-hadoop2.7/*
############################################
*Then configure these Files:
*Edit the .bashrc file and write:
$ sudo gedit .bashrc
*write:
*the Scala home:
export SCALA_HOME=/usr/local/scala-2.12.3
export PATH=$SCALA_HOME/bin:$PATH
*the spark home:
export SPARK_HOME=/usr/local/spark-2.2.0-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH
***************
*Copy these files can find it in /spark/conf/
fire these command:
$ cp slaves.template slaves
$ sudo gedit slaves
type:
localhost
***********
$ cp spark-defaults.conf.template spark-defaults.conf
$ sudo gedit spark-defaults.conf
type:
spark.master spark://localhost:7077
***********
$ cp spark-env.sh.template spark-env.sh
$ sudo gedit spark-env.sh
type:
export SCALA_HOME=/usr/local/scala-2.12.3
##########################################
Start all Spark daemons:
$ sbin/start-master.sh
$ sbin/start-slaves.sh
if these command not run
soo type this
$ sbin/start-all.sh
########################################
open spark sites:
localhost:8080