0

Whenever I try to print the number of rows from a table of pyspark type. I have the same problem.

import findspark

findspark.init()

from pyspark.sql import *

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.driver.extraClassPath", "C:\\Spark\\Spark\\spark-2.4.3-bin-hadoop2.7\\jars\\mssql-jdbc-7.4.1.jre12"
                                           ".jar") \
    .getOrCreate()

mssql_df = spark.read.format("jdbc") \
    .option("url", "jdbc:sqlserver://MCNAVSQLCLU:1433;databaseName=NADIA;integratedSecurity=true") \
    .option("dbtable", "[SBM$Vendor Inspection Plan]") \
    .option("driver", 'com.microsoft.sqlserver.jdbc.SQLServerDriver').load()

mssql_df.count()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/renos.bardis/PycharmProjects/erp-mdm/SparkConn.py", line 19, in <module>
    mssql_df.count()
  File "C:\Spark\Spark\spark-2.4.3-bin-hadoop2.7\python\pyspark\sql\dataframe.py", line 522, in count
    return int(self._jdf.count())
  File "C:\Spark\Spark\spark-2.4.3-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py", line 1257, in __call__
  File "C:\Spark\Spark\spark-2.4.3-bin-hadoop2.7\python\pyspark\sql\utils.py", line 79, in deco
    raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.IllegalArgumentException: 'Unsupported class file major version 56'
Ben
  • 51,770
  • 36
  • 127
  • 149
Renos Bardhis
  • 325
  • 2
  • 5
  • 11
  • I have this as a result: `pyspark.sql.utils.IllegalArgumentException: 'Unsupported class file major version 56'` – Renos Bardhis Aug 27 '19 at 12:49
  • Possible duplicate of [Pyspark error - Unsupported class file major version 55](https://stackoverflow.com/questions/53583199/pyspark-error-unsupported-class-file-major-version-55). See also for confirmation that the answer is the same: [java.lang.UnsupportedClassVersionError - on MAC OSX - Unsupported class file major version 56](https://stackoverflow.com/q/55797467/458741) – Ben Aug 27 '19 at 12:52
  • I have already read them. But, was not so helpful, to be honest. – Renos Bardhis Aug 27 '19 at 13:56
  • Could you explain why they were unhelpful in your question? If they were unhelpful that means you've already followed the advice and are using Java 8? – Ben Aug 27 '19 at 14:19
  • I am using java version 1.8.0_221 on windows 10. Also, I wanted to ask if I install the newest version is a problem? because I am trying on ubuntu – Renos Bardhis Aug 28 '19 at 12:05
  • Hello @Ben, I use: java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) and Javac -version: javac 1.8.0_201 And when I use this command `df.count()` I have the same problem but with other commands I do not have always the same problem. – Renos Bardhis Aug 29 '19 at 14:19

0 Answers0