2

I am using this tutorial spark cluster on yarn mode in docker container to launch zeppelin in spark cluster in yarn mode. However I am stuck at step 4. I can't find conf/zeppelin-env.sh in my docker container to put further configuration. I tried putting these conf folder of zeppelin but yet now successful. Apart from that zeppelin notebook is also not running on localhost:9001.

I am very new to distributed system, it would be great if someone can help me start zeppelin on spark cluster in yarn mode.

Here is my docker-compose file to enable zeppelin talk with spark cluster.

version: '2'
services:
sparkmaster:
  build: .
  container_name: sparkmaster
ports:
  - "8080:8080"
  - "7077:7077" 
  - "8888:8888"
  - "8081:8081"
  - "8082:8082"
  - "5050:5050"
  - "5051:5051"
  - "4040:4040"
zeppelin:
  image: dylanmei/zeppelin
  container_name: zeppelin-notebook
env_file:
  - ./hadoop.env
environment:
  ZEPPELIN_PORT: 9001
  CORE_CONF_fs_defaultFS: "hdfs://namenode:8020"
  HADOOP_CONF_DIR_fs_defaultFS: "hdfs://namenode:8020"
  SPARK_MASTER: "spark://spark-master:7077"
  MASTER: "yarn-client"
  SPARK_HOME: spark-master
  ZEPPELIN_JAVA_OPTS: >-
    -Dspark.driver.memory=1g
    -Dspark.executor.memory=2g
ports:
  - 9001:9001
volumes:
  - ./data:/usr/zeppelin/data
  - ./notebooks:/usr/zeppelin/notebook
Rohit Barnwal
  • 482
  • 6
  • 19

1 Answers1

1

this is the dockerfile you used to launch the standalone spark cluster.

But there is no Zeppelin instance inside the container, so you have to use Zeppelin on your local machine.

Please download and use it.

1ambda
  • 1,145
  • 8
  • 19
  • I have used this dockerfile https://github.com/apache/zeppelin/blob/master/scripts/docker/spark-cluster-managers/spark_yarn_cluster/Dockerfile Though this also doesn't have zeppelin, so I launched another container with zeppelin docker image, but unable to make zeppelin talk with spark cluster. – Rohit Barnwal Sep 12 '17 at 05:27
  • You have to merge them into one network, otherwise, can't connect each other. For example, you can make new docker-compose file include thoese dockerfiles. – 1ambda Sep 12 '17 at 06:09
  • Done that as well, I have added my docker-compose file. Above mentioned dockerfile and docker-compose are in the same folder. But this launches a single container only having zeppelin in that container. – Rohit Barnwal Sep 12 '17 at 06:21
  • Could you share your spark interpreter setting? – 1ambda Sep 12 '17 at 06:30
  • Unable to find it, docker ps gives me this: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1e8b6ea5e009 dylanmei/zeppelin "bin/zeppelin.sh" 16 hours ago Up 25 minutes 0.0.0.0:9001->9001/tcp zeppelin-notebook – Rohit Barnwal Sep 12 '17 at 07:03
  • To use spark cluster in Zeppelin, you need to set proper URL for spark interpreter itself. Then, how did you setup the URL for the spark cluster? – 1ambda Sep 12 '17 at 07:30
  • 2
    We don't need to use dockerized Zeppelin, let's try binary in your local machine first to remove complexities. – 1ambda Sep 12 '17 at 07:31