Configuring and Upgrading Apache Spark

Add the node where you want Apache Spark 1.4.1 History Server to run. Install the version corresponding to the HDP version you currently have installed.
1. su - root
2. wget -nv http://s3.amazonaws.com/dev.hortonworks.com/ HDP/centos6/2.x/BUILDS/<version>/hdpbn.repo -O /etc/ yum.repos.d/Spark<version>.repo
3. yum install spark_<version>-master -y
4. To use Python: yum install spark_<version>-python
5. conf-select create-conf-dir --package spark --stack-version <version> --conf-version 0
6. cp /etc/spark/<version>/0/* /etc/spark/<version>/0/
7. conf-select set-conf-dir --package spark --stack-version <version> --conf-version 0
8. hdp-select set spark-client <version>
9. hdp-select set spark-historyserver <version>
Validate the Spark installation. As user spark, run SparkPI example:
1. sudo su spark
2. cd /usr/hdp/current/spark-client
3. ./bin/run-example SparkPi 10
Restart Spark on YARN in either yarn-cluster mode or yarn-client mode:
- yarn-cluster mode: ./usr/hdp/current/spark-client/bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] [app options]
- yarn-client mode: ./usr/hdp/current/spark-client/bin/spark-shell -- master yarn-client

​Configuring and Upgrading Apache Spark