0

I am building a custom app where I want to achieve the following

if condition x
  Query from Cached Temp View in memory
else 
  Query from Hive directly

I am able to achieve the first case by following code

sql = "select * from x"
tableName = "x"
if blah..
  df.createOrReplaceTempView(tableName)
  val result = hc.bi.sql(sql)
else
  // get data from hive directly

ps: name of the view is same as the table name in hive metastore

mrsrinivas
  • 34,112
  • 13
  • 125
  • 125
ShikharDua
  • 9,411
  • 1
  • 26
  • 22
  • you can query on parquet/CSV/ORC directly with Spark SQL. check http://stackoverflow.com/a/42688281/1592191 – mrsrinivas Mar 16 '17 at 02:54
  • I am actually looking for a more direct way than specifying the file in filesystem – ShikharDua Mar 16 '17 at 05:20
  • what do you mean by _more direct way than specifying the file in filesystem_? – mrsrinivas Mar 16 '17 at 05:27
  • At work work there are millions of tables. There are 1000's of users. I can't ask users to specify the query in such format and can't do it myself cos I have to parse the query then – ShikharDua Mar 16 '17 at 18:08
  • 1
    AFAIK, there are only two ways in Spark to query(SQL) on any table. 1) use metastore (for table name) or 2) use the file directly(not all formats supports it) – mrsrinivas Mar 16 '17 at 18:14
  • So this means that, if name of table is x in metastore and I create a view from DF with name x, then "select * from x" will always hit view x ? – ShikharDua Mar 16 '17 at 18:18
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/138260/discussion-between-mrsrinivas-and-shikhardua). – mrsrinivas Mar 16 '17 at 18:20

0 Answers0