3

The timestamp field is losing precision when querying the same table from Hive Metastore using Spark SQL.

My table description goes like this :

col_name  data_type  comment
id          bigint    null
name        string    null
joined_time timestamp null

Using Hive QL, I get joined_time values in milliseconds precision. Hive QL results:

select * from employees;

1   foo 2016-07-04 02:12:10.0
2   bar 2016-07-04 02:12:10.0

While using spark-sql, I lose precision, upto minutes. e.g :

val result = sqlContext.sql("select * from employees")
result.show()

1  foo 2016-07-04 02:12:...
2  bar 2016-07-04 02:12:...
eliasah
  • 39,588
  • 11
  • 124
  • 154
shriyog
  • 938
  • 1
  • 13
  • 26

1 Answers1

3

It's not loosing precision. It has just truncated the display.

Since Spark 1.6, you can display it with result.show(false)

http://spark.apache.org/docs/latest/api/scala/#org.apache.spark.sql.Dataset

val df = Seq((1,2),(2,4)).toDF("x","y")
df.show(false)
// +---+---+
// |x  |y  |
// +---+---+
// |1  |2  |
// |2  |4  |
// +---+---+

Now with timestamps :

sqlContext.sql("select current_timestamp()").show
// +--------------------+
// |                 _c0|
// +--------------------+
// |2017-02-10 14:40:...|
// +--------------------+

sqlContext.sql("select current_timestamp()").show(false)
// +-----------------------+
// |_c0                    |
// +-----------------------+
// |2017-02-10 14:40:14.038|
// +-----------------------+
eliasah
  • 39,588
  • 11
  • 124
  • 154
  • In the case of Spark DataFrames, see https://stackoverflow.com/questions/33742895/how-to-show-full-column-content-in-a-spark-dataframe – shriyog Jul 18 '19 at 12:16