1

When I try to iterate the DataFrame value via map in Spark 2.0 following error will occur

******error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases. resDf_upd.map(row => {******

Code is 

import org.apache.spark.SparkContext
import org.apache.spark.sql.{SQLContext, SparkSession, SaveMode, Row} 
import com.typesafe.config.ConfigFactory 
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.Encoder

object TestProg {
def main(args: Array[String]){

val sc = new SparkContext( "local", "Test Program", "/usr/local/spark", Nil, Map())     

val sqlContext = new SQLContext(sc)

val config =  ConfigFactory.load("test.conf")

val db1_prop = new java.util.Properties
db1_prop.put("user", config.getString("db1.usr"))
db1_prop.put("password", config.getString("db1.pwd"))
db1_prop.put("driver", config.getString("db1.driver"))

val db1_url = config.getString("db1.url")
val db1_type = "jdbc"

val spark = SparkSession
  .builder()
  .appName("Session Program")
  .config("spark.some.config.option", "some-value")
  .getOrCreate()

var empDf = sqlContext.read.format(db1_type).jdbc(db1_url,"employee",db1_prop)
empDf.createOrReplaceTempView("emp")

**val resDf_upd = sqlContext.sql("select * from emp")
  resDf_upd.map(row => {
    val row1 = row.getAs[String](1)        
    val make = if (row1.toLowerCase == "Usr3") "User3" else row1
    Row(row(0),make,row(2))
  }).collect().foreach(println)**

}

}

NetBeans Hints : [ "not enough arguments for method map: (implicit evidence$7: org.apache.spark.sql.Encoder[org.apache.spark.sql.Row])org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]. Unspecified value parameter evidence$7.

Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases." ]

DataFrames classes are not avail in Spark2.0.0 document : https://spark.apache.org/docs/2.0.0/api/java/index.html

DataFrames classes are used Spark2.0.0 examples document : https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection

Jai
  • 11
  • 4
  • Looks like there's some type in `emp` table that isn't supported - what are the types of _all_ the columns in `emp` table? You can also try selecting the specific columns you're using (assuming their types are OK) instead of `select *`. – Tzach Zohar Sep 15 '16 at 14:42

0 Answers0