Spark 1.4 Scala Dataframe binary to hex

Question

I have this field in my dataframe: myField: binary (nullable = false) (originally a Java byte[16]).

How do I select it and show it as a hex in scala from a spark shell? Please include necessary imports.

what have you tried? do you need information about how to convert binary to hex? or how to apply this function to your RDD? basically... `mydf.select("myField").rdd.map(binToHex(_))` — Dan Ciborowski - MSFT, Aug 11 '16 at 14:35
do I need to convert it to an rdd first? Is there an equivalent dataframe function? — Marsellus Wallace, Aug 11 '16 at 14:37
something like `mydf.select("myField").map(binToHex($"myField"))`, but my DF psuedocode is a bit rougher then my rdd, might not even need the select. — Dan Ciborowski - MSFT, Aug 11 '16 at 14:38
What's the import? I'm getting: error: not found: value binToHex (NOTE: I'm new to everything here) — Marsellus Wallace, Aug 11 '16 at 14:40
you have to code that function up yourself. That why I asked what you were looking for, I know spark, but not how to convert a bin to hex off the top of my head — Dan Ciborowski - MSFT, Aug 11 '16 at 14:42
http://stackoverflow.com/questions/2817752/java-code-to-convert-byte-to-hexadecimal — Dan Ciborowski - MSFT, Aug 11 '16 at 14:44

score 1 · Answer 1 · answered Aug 11 '16 at 19:09

1

Try

import org.apache.spark.sql.functions.{col, hex}

df.select(hex(col("myField")))

answered Aug 11 '16 at 19:09

score 0 · Answer 2 · answered Feb 04 '21 at 15:44

So I generalized @user6022341 solution to the following function

  def castBinaryToHexString(inputSrcDF: DataFrame):DataFrame={
    var srcDF = inputSrcDF

    // get all column names
    val allCols = srcDF.schema.fields.map(f => f.name.toLowerCase())

    // get all columns with data type = BinaryType
    val dateCols = srcDF.schema.fields.filter(f => f.dataType == BinaryType).map(f => f.name.toLowerCase())

    // cast all binary types to hex string
    srcDF = srcDF.select(allCols.map(x => if (dateCols.contains(x)) concat(lit("0x"),hex(col(x))).alias(x) else col(x) ): _*)

    return srcDF
  }

Spark 1.4 Scala Dataframe binary to hex

2 Answers2