1

I just upgraded from Spark 2.0.2 to Spark 2.1.0 (by downloading the prebuilt version for Hadoop 2.7&later). No Hive is installed.

Upon launch of the spark-shell, the metastore_db/ folder and derby.log file are created at the launch location, together with a bunch of warning logs (which were not printed in the previous version).

Closer inspection of the debug logs shows that Spark 2.1.0 tries to initialise a HiveMetastoreConnection:

17/01/13 09:14:44 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.

Similar debug logs for Spark 2.0.2 do not show any initialisation of HiveMetastoreConnection.

Is this intended behaviour? Could it be related to the fact that spark.sql.warehouse.dir is now a static configuration shared among sessions? How do I avoid this, since I have no Hive installed?

Thanks in advance!

hiryu
  • 1,308
  • 1
  • 14
  • 21

3 Answers3

5

From Spark 2.1.0 documentation pages:

When not configured by the hive-site.xml, the context automatically creates metastore_db in the current directory and creates a directory configured by spark.sql.warehouse.dir, which defaults to the directory spark-warehouse in the current directory that the Spark application is started. Note that the hive.metastore.warehouse.dir property in hive-site.xml is deprecated since Spark 2.0.0. Instead, use spark.sql.warehouse.dir to specify the default location of database in warehouse.

Since you do not have Hive installed, you will not have a hive-site.xml config file, and this must be defaulting to the current directory.

If you are not planning to use HiveContext in Spark, you could reinstall Spark 2.1.0 from source, rebuilding it with Maven and making sure you omit -Phive -Phive-thriftserver flags which enable Hive support.

Alexey Svyatkovskiy
  • 646
  • 5
  • 16
  • Thanks, that must indeed be the case. I hoped I could avoid rebuilding from source just to avoid that these folders are created but I see no other way. – hiryu Jan 14 '17 at 15:17
5

For future googlers: the actual underlying reason for the creation of metastore_db and derby.log in every working directory is the default value of derby.system.home.

This can be changed in spark-defaults.conf, see here.

Community
  • 1
  • 1
hiryu
  • 1,308
  • 1
  • 14
  • 21
  • 1
    See also https://issues.apache.org/jira/browse/SPARK-4758 for a way to avoid creating the `metastore_db` by requesting an embedded, volatile, **in-memory** Derby database. – Samson Scharfrichter May 18 '17 at 13:18
  • See also https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-tips-and-tricks-running-spark-windows.html for how to set also the `hive.exec.scratchdir` property. – Samson Scharfrichter May 18 '17 at 13:19
0

This happen also with Spark 1.6. You can change the path by adding in Spark submit extra options:

-Dderby.system.home=/tmp/derby

(or by derby.properties, there are several ways to change it).

Thomas Decaux
  • 21,738
  • 2
  • 113
  • 124