0

I'm running into an issue with my Spark application. When I try to initialize the SparkContext using the provided configuration, I'm encountering the following error: 'RuntimeError: Java gateway process exited before sending its port number.' You are trying to pass an insecure Py4j gateway to Spark. This is not allowed as it is a security risk."

conf = (SparkConf().setMaster("k8s://http://kubernetes.default.svc.cluster.local:443") 
    .set("spark.kubernetes.container.image", "pyspark:latest") # Spark image name
    .set("spark.driver.port", "2222") # Needs to match svc
    .set("spark.driver.blockManager.port", "7777")
    .set("spark.driver.host", "https://driver-service.jupyterhub.svc.cluster.local") 
    .set("spark.driver.bindAddress", "0.0.0.0")
    .set("spark.kubernetes.namespace", "spark")
    .set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")
    .set("spark.kubernetes.authenticate.serviceAccountName", "spark")
    .set("spark.kubernetes.authenticate.submission.caCertFile","/var/run/secrets/kubernetes.io/serviceaccount/ca.crt")
    .set("spark.kubernetes.authenticate.submission.oauthTokenFile","/var/run/secrets/kubernetes.io/serviceaccount/token")
    .set("spark.executor.instances", "2")
    .set("spark.kubernetes.container.image.pullPolicy", "IfNotPresent")
    .set("spark.authenticate", "false")
    .set("spark.app.name", "spark-test"))

'RuntimeError: Java gateway process exited before sending its port number.'

You are trying to pass an insecure Py4j gateway to Spark. This is not allowed as it is a security risk."

KUNAL DAS
  • 23
  • 6
  • Have you checked this [thread](https://stackoverflow.com/questions/31841509/pyspark-exception-java-gateway-process-exited-before-sending-the-driver-its-po), does it resolve your issue. – Sai Chandra Gadde Jun 13 '23 at 08:25
  • @SaiChandraGadde , print(os.environ.get("JAVA_HOME")) gives "/usr/lib/jvm/java-11-openjdk-amd64".. so it seems properly configured – KUNAL DAS Jun 13 '23 at 11:04

0 Answers0