1

I am trying to create an instance pool using databricks api and I need the photon enabled, in the documentation I could not find the parameter to enable it, does anyone know how to do it?

I used this documentation, https://docs.databricks.com/dev-tools/api/latest/instance-pools.html#create

Here is my payload

payload = {
        "instance_pool_name": f"DRIVER - {instance_type}",
        "node_type_id": instance_type,
        "idle_instance_autotermination_minutes": 4,
        "enable_elastic_disk": True,
        "max_capacity": 300,
        "min_idle_instances":0,
        "preloaded_spark_versions": ["12.0.x-scala2.12"],
        
        # RUNTINME PHOTON TESTS
        "preloaded_runtime_engine": "PHOTON",
        "runtime_engine": "PHOTON",
        
        "aws_attributes":{
            "availability":"SPOT",
            "zone_id": "us-east-1a",
            "spot_bid_price_percent": 100
       },

   "custom_tags":[
     {...}
   ]
}

I tried adding the cluster creation options, but neither runtime engine nor preloaded parameters work

        "preloaded_runtime_engine": "PHOTON",
        "runtime_engine": "PHOTON",
MaxAirheart
  • 33
  • 5

1 Answers1

0

It's not a separate parameter, there are dedicated Photon runtimes like 11.3.x-photon-scala2.12. You can obtain the name by creating an instance pool with a Photon runtime using UI and querying its definition via API.

You can also list all available runtimes using runtime versions API endpoint:

$ curl "$DATABRICKS_API/clusters/spark-versions" | jq -r '.versions[] | .key' | grep photon | sort
10.4.x-photon-scala2.12
11.1.x-photon-scala2.12
11.2.x-photon-scala2.12
11.3.x-photon-scala2.12
12.0.x-photon-scala2.12
12.1.x-photon-scala2.12
9.1.x-photon-scala2.12
Kombajn zbożowy
  • 8,755
  • 3
  • 28
  • 60
  • It worked, thanks I thought it was another parameter instead of a proper ruintime, good to know, thank you very much – MaxAirheart Jan 18 '23 at 12:21