how we can upload parquet file in azure blob storage using spark and scala?

Question

 val input_blob_path = "Enter input azure blob path here"
    val output_blob_path = "Enter output azure blob path here"
    val storageAccountName = "Enter_Your_Storage_Account_Name"
    val storageKeyValue = "Enter_Your_Storage_Account_Key"
    val containerName = "Enter_Your_containerName"

    val sparkSession = SparkSession.builder()
      .master("local")
      .appName("WriteDataToAzureBlob")
      .config(
        s"fs.azure.account.key.${storageAccountName}.blob.core.windows.net",
        storageKeyValue)
      .getOrCreate()

    parqDF.write.parquet("wasbs://${containerName}@{storageAccountName}.blob.core.windows.net/${output_blob_path}")

I am using this code to write in blob storage but it's now working for me.

Can you please be more specific? What do you mean by "it's not working for me"? — Nir Hedvat, Feb 23 '22 at 12:36
the code I am using is not helping me to write the parquet file in blob storage — Kush Desai, Feb 23 '22 at 12:53
java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem i am getting this exception — Kush Desai, Feb 23 '22 at 13:24
hey nir thanks for dependency but i am still getting this error scala module 2.13.2 requires jackson databind version >= 2.12.0 and < 2.13.0 in scala — Kush Desai, Feb 24 '22 at 05:12
Hi there, do you have any working example with Scala sbt project for writing, listing, updating blobs in a container in azure? — Rodolfo Velasco, May 12 '22 at 07:37

score 0 · Answer 1 · answered Mar 02 '22 at 14:10

For the error:

scala module 2.13.2 requires jackson databind version >= 2.12.0 and < 2.13.0 in scala

Try to remove old dependencies of Jackson and upgrade the plugin or added 2.12.0 series
All of Jackson packages that Azure SDKs use , you may need to pin each of them to a specific version and their version number set to the same one as of Spark.

OR

Try to add a different com.fasterxml.jackson.core dependencies version as per error and then you may need to override them explicitly.
```
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.12.5"
```

See 1. scala - Jackson conflicts in Apache Spark when used with Azure Java SDK - Stack Overflow

Maven Repository: azure-core-serializer-json-jackson

Other References:

how we can upload parquet file in azure blob storage using spark and scala?

1 Answers1