0

I have an ignite cluster running on Google Kubernetes Engine.

sudo kubectl get pods NAME READY STATUS RESTARTS AGE ignite-cluster-bbb4f56c4-nrftv 1/1 Running 0 6d ignite-cluster-bbb4f56c4-skvf6 1/1 Running 0 6d

Now I am trying to connect use igniteRDD on spark using this scala code.

Here is my configuration file which I use to detect ignite pods on GKE cluster.

import org.apache.ignite.configuration.IgniteConfiguration
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi
import org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder

object igniteConf {

  def ignite_configuration: IgniteConfiguration = {
    val spi = new TcpDiscoverySpi
    val ipFinder = new TcpDiscoveryKubernetesIpFinder
    ipFinder.setMasterUrl("https://35.192.214.68")
    ipFinder.setServiceName("ignite")
    spi.setIpFinder(ipFinder)
    val cfg = new IgniteConfiguration
    cfg.setDiscoverySpi(spi)
    cfg
  }

}

and in the main file I use it like this 

val igniteContext = new IgniteContext(sparkContext, () => igniteConf.ignite_configuration, true)

Now I create a jar and after creating a docker image push to Google Container Registry.

Command to run the jar.

sudo bin/spark-submit --master k8s://https://35.192.214.68 --deploy-mode cluster --name sparkIgnite --class org.blk.igniteSparkResearch.ScalarSharedRDDExample --conf spark.executor.instances=3 --conf spark.app.name=sharedSparkIgnite --conf spark.kubernetes.authenticate.driver.serviceAccountName=ignite --conf spark.kubernetes.container.image=us.gcr.io/nlp-research-198620/ignite-spark:v2 local:///opt/spark/jars/igniteSpark-1.0-SNAPSHOT-jar-with-dependencies.jar

The above command creates 1 driver and 3 executors. Logs of any executors shows that it can't connect to ignite-cluster on kubernetes.

class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses

Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://35.192.214.68/api/v1/namespaces/default/endpoints/ignite

Can anybody let me know where I might be going.

Thanks in advance.

wadhwasahil
  • 468
  • 7
  • 28
  • Possible duplicate of [How to setMasterUrl in Ignite XML config for Kubernetes IPFinder](https://stackoverflow.com/questions/49395481/how-to-setmasterurl-in-ignite-xml-config-for-kubernetes-ipfinder) – Denis Jun 19 '18 at 11:54

1 Answers1

1

This is a known issue. It's permission-related.

Here is a JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-8081

You can find a working configuration in a comment.

Denis
  • 3,573
  • 8
  • 13
  • I think the link you shared is used for setting up ignite cluster. I already have ignite cluster running but my task is to connect spark client/job to existing ignite cluster on kubernetes. – wadhwasahil Jun 19 '18 at 12:00
  • From the error you provided I see, that TcpDiscoveryKubernetesIpFinder fails to retrieve the list of available Ignite pods. This is exactly the issue, that's described in the ticket. – Denis Jun 19 '18 at 12:06
  • So my question is how to access ignite cluster within spark application on Google Kubernetes Engine. All these mentioned issues only provide a partial solution to my problem. – wadhwasahil Jun 19 '18 at 12:13
  • The code, that you provided, seems legit. The only thing, that doesn't let you run the example is a failing IP finder. Or do you have some other problem? – Denis Jun 19 '18 at 13:36
  • It's the failing IP finder problem. I guess its due to RBAC issue since 403 is probably due to it only. – wadhwasahil Jun 19 '18 at 13:54
  • Also a working configuration was provided in this answer: https://stackoverflow.com/a/50979688/6708961 – Denis Jun 27 '18 at 07:52