2

How can I use a custom transformer written in scala in a pyspark pipeline.

class UpperTransformer(override val uid: String)
    extends UnaryTransformer[String, String, UpperTransformer] {

  def this() = this(Identifiable.randomUID("upper"))

  override protected def validateInputType(inputType: DataType): Unit = {
    require(inputType == StringType)
  }

  protected def createTransformFunc: String => String = {
    _.toUpperCase
  }

  protected def outputDataType: DataType = StringType
}

Use this transformer in pyspark pipeline.

pratyush
  • 36
  • 2
  • Sure, you can. But in order to be able to export that model using MLEAP you need to implement the mleap equivalent of that transformer: for further details see https://github.com/combust/mleap/wiki/Adding-an-MLeap-Spark-Transformer – Elmar Macek Sep 11 '18 at 16:47

0 Answers0