Spark in K8S (with Minikube) Part 2
1. Install minikube (K8S cluster on single machine) [https://minikube.sigs.k8s.io/docs/start/]
2. Install helm [https://helm.sh/]
3. Start minikube:
minikube start --cpus=4 --memory=8096
4. Check installation:
minikube status
kubectl cluster-info
kubectl get all
5. Create namespace
kubectl create namespace spark
6. Install helm chart of Spark 3.0.1 into K8S:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install spark-301 --namespace spark bitnami/spark
7. Check status of Spark create (Created or not):
minikube dashboard
8. Run Spark Example Pi calculation application:
export EXAMPLE_JAR=$(kubectl exec -ti --namespace spark spark-301-worker-0 -- find examples/jars/ -name 'spark-example*\.jar' | tr -d '\r')
kubectl exec -ti --namespace spark spark-301-worker-0 -- spark-submit --master spark://spark-301-master-svc:7077 \
--class org.apache.spark.examples.SparkPi \
$EXAMPLE_JAR 5
9. Port-forward of Spark Web UI:
kubectl port-forward --namespace spark svc/spark-301-master-svc 8080:80
10. Check status of execution:
Visit web-browser on http://locahost:8080
11. Delete|Stop cluster:
minikube delete
minikube stop