Questions tagged [spark-connect]
6 questions
1
vote
0 answers
The proper way to run Spark Connect in Anaconda - error '$HOME' is not recognized as an internal or external command, operable program or batch file
I try to learn this lesson https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_connect.html
Method 1: from anaconda - window
by download the JP notebook to my Downloads folder, then start the jupyter notebook via…
Tom Tom
- 328
- 4
- 15
0
votes
0 answers
Possible to add own method to the Spark Connect Thin Client API?
we would like to add a method to the Spark Connect Server and expose on the Thin Client API. Here is why:
In our organization, we have built a timeseries database using Spark and HDFS storage for hundreds of thousands of signals and a few Petabyte…
Vito B.
- 1
0
votes
0 answers
Running Spark-Connect Server on kubernetes in cluster mode/high availability mode
I am trying to figure out how to effectively use the new Spark-Connect feature of Spark version >= 3.4.0. Specifically, I want so set up a kubernetes Spark cluster where various applications (mainly pyspark) will connect and submit their…
scienceseba
- 61
- 6
0
votes
0 answers
Pandas API support on Spark Connect
I am trying to use Spark PANDAS API on Spark Connect but I am getting assertion erorr
assert isinstance(spark_frame, SparkDataFrame)
AssertionError
I dont get any error if I use the spark Dataframe API.
Are Pandas-Spark API supported on Spark…
0
votes
1 answer
pyspark-connect can't show all hive databases
I'm using pyspark3.4.0 feature spark-connet module to connect remote hive 3.1.3.
When create sparksession in local mode with hive supported, all data base in hive can be viewed;
spark =…
leon
- 1
0
votes
1 answer
Using Spark Connect with Scala
I would like to use the new Spark Connect feature within a Scala program.
I started the Connect server and I am able to connect to it from Pyspark and also when submitting Python script, e.g., with spark-submit --remote sc://localhost…
hage
- 5,966
- 3
- 32
- 42