How do implicit work in Spark/Scala

Question

Here is a sample Spark code, which converts the Seq to the Dataset:

import spark.implicits._
val s = Seq(1, 2, 3, 4)
val ds = s.toDS()

The Scala Seq does not have the toDS method, it comes from Spark implicits. How is the Dataset created here?

Edit: I did look at other SO answers but could not see an example, which would explain how the implicit could be used in s.toDS. I've referenced to the sample answer in the comment.

@user8371915 I did see that answer https://stackoverflow.com/a/10375941/328989 but didn't notice there a mention of `s.applyPrefix`, only `applyPrefix(s)`, which was not clear to me whether it could cover this case here `s.toDS`. — Datageek, Sep 16 '17 at 10:58

Teimuraz · Accepted Answer · 2017-09-16T10:54:02.250

Scala has a way to add methods to existing classes, like extension method in Kotlin (and C# as I remember), but does it in a different way, through implicits.

To add the method to existing class, you first create implicit class:

object StringImplicits {
  implicit class StringUtils(s: String) {
    def someCoolMethod = println("Yooo")
  }
}

object Application extends App {
    import StringImplicits._
    val s = "Hello"
    s.someCoolMethod
}

You import this StringUtils and can call someCoolMethod on instance of String

Notice that StringUtils class takes String as a constuctor param.

When calling some method on String, scala compiler first looks this method in String class.

If it does not find it, it will look imported implicit classes which take String param.

If found, it calls the method from that class.

If no such class found, it will raise the error.

How do implicit work in Spark/Scala

1 Answers1