java.lang.RuntimeException:Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

Question

I have configured my Hive as given on link: http://www.youtube.com/watch?v=Dqo1ahdBK_A, but I am getting the following error while creating a table in Hive. I am using hadoop-1.2.1 and hive-0.12.0.

hive> create table employee(emp_id int,name string,salary double);
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

score 46 · Accepted Answer · answered Mar 28 '14 at 14:08

46

Looks like problem with your metastore. If you are using the default hive metastore embedded derby. Lock file would be there in case of abnormal exit. if you remove that lock file this issue would be solved

rm   metastore_db/*.lck

answered Mar 28 '14 at 14:08

SachinJose

8,462
4
42
63

1

I have to run the rm command every time I use pyspark, any ideas why? – user1532587 Sep 09 '16 at 18:32
Using default derby database as hive back end is not recommended, With derby as Hive backend only one session can established at a time. The best option is fix this issue is to use MySQL/Postgres/Oracle server as Hive backend instead of derby. – SachinJose Sep 12 '16 at 14:46
Thanks Sachin. It worked for me. But it is not good to execute this command when you gave lot of data stored in Hive. What should be an alternative in that case – Kanav Narula May 03 '17 at 06:51
1

@KanavNarula, It's not recommended to use derby for a clustered environment. Recommended options is to migrate from derby to Postgres or MySQL. – SachinJose May 03 '17 at 18:31
Great. Thanks @sachin for the quick response. – Kanav Narula May 05 '17 at 04:14

score 10 · Answer 2 · answered Apr 23 '14 at 10:04

This could be an issue with the metastore like sachinjose described or a connection issue. Run hive console in debug mode like so:

hive -hiveconf hive.root.logger=DEBUG,console

Then execute a simple query like show tables; and see what happens. I ran into this issue after restarting a namenode and it was due to the wrong host being set in the configuration files (ec2 generates new private IP/hostname when restarted).

score 8 · Answer 3 · answered Aug 27 '14 at 03:55

Run this command from the console:

./hive -hiveconf hive.root.logger=DEBUG,console

Now run

show databases;

If you see an exception as below:

java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.

It means there is some permissions issue with the metastore_db. You need to change the permissions of the complete folder. I am running my hive as hdfs user. So, i used the command

chown -R hdfs:hadoop *

in the hive directory. This resolved the issue.

score 2 · Answer 4 · edited Jul 20 '15 at 09:04

2

hive> show databases;
FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

To resolve this issue start the hadoop services first.

$ start-all.sh

Then I run

hive> show database;

It works fine for me.

edited Jul 20 '15 at 09:04

Rohan Kumar

40,431
11
76
106

answered Jul 19 '15 at 13:28

user5132316

21
1

Where can I find the script start-all.sh? – supritshah1289 Jun 26 '19 at 03:05

score 1 · Answer 5 · answered Jul 22 '14 at 13:35

The answer is located in http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/5.0/CDH5-Installation-Guide/cdh5ig_hive_schema_tool.html

To suppress the schema check and allow the metastore to implicitly modify the schema, you need to set the hive.metastore.schema.verification configuration property to false in hive-site.xml.

score 1 · Answer 6 · edited Jul 01 '15 at 03:32

1

rm metastore_db/*.lck

It works for me, too. It can be found in your home/user directory. You can use the locate command to find it: locate metastore_db

After removing the lock files, close the current session. Call hive shell in new session

edited Jul 01 '15 at 03:32

Bond

16,071
6
30
53

answered Jul 01 '15 at 03:07

Manish Agrawal

794
1
9
23

score 1 · Answer 7 · answered Jan 13 '16 at 03:58

1

make hive.metastore.schema.verification=false in hive-site.xml file this error will be removed. In my case it is working well.

answered Jan 13 '16 at 03:58

Sandeep Mishra

19
1

score 1 · Answer 8 · answered Dec 28 '18 at 18:01

I was getting the same error but I could not even start the hive shell. After trying the very useful hive debug command:

hive -hiveconf hive.root.logger=DEBUG,console

I was able to see that Hive could not find a valid Kerberos TGT. You will see in the debug info something about SASL negotiation failed and no valid Kerberos TGT. My solution was to run

kinit

before running the hive CLI.

score 0 · Answer 9 · answered Apr 01 '14 at 07:58

0

Restarting the virtual machine or system should also release the lock.

answered Apr 01 '14 at 07:58

Chalpat

76
1
8

score 0 · Answer 10 · answered May 04 '14 at 07:34

0

this could due to more than one "metastore_db". Remove "metastore_db", restart hadoop cluster and open hive shell from $HIVE_HOME/bin folder

answered May 04 '14 at 07:34

Tamil

201
2
2

score 0 · Answer 11 · answered May 19 '15 at 13:51

Being a newbie i got the same set of errors.
It was found that one of the daemons in my case namenode was not initiated. On installing hadoop it would be wise to make it a habit to hit he following commands :
ps -ef | grep "namenode"
ps -ef | grep "datanode"
ps -ef | grep "tracker"

One needs to check the relevant log if any of the daemon is not working.

score 0 · Answer 12 · answered Jul 16 '15 at 11:49

0

If it is in local machine, looks like you have another terminal opened with hive shell/session. You can have only one session using embedded derby data base. Close all other hive sessions and try.

answered Jul 16 '15 at 11:49

ajesh kumar

96
1
8

score 0 · Answer 13 · answered Feb 03 '17 at 04:10

delete code in hive-site.xml and paste this code in hive-site.xml.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:derby:;databaseName=/home/hadoop/Desktop/apache-hive-2.1.1-bin/metastore_db;create=true</value>
<description>
JDBC connect string for a JDBC metastore.
To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
<property>
<name>hive.metastore.uris</name>
<value/>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.derby.jdbc.EmbeddedDriver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.PersistenceManagerFactoryClass</name>
<value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>
<description>class implementing the jdo persistence</description>
</property>
</configuration>

score 0 · Answer 14 · answered Jul 24 '18 at 12:46

I was facing the same issue there are some important point's that could be resolve this problem.

1.Put the following at the beginning of hive-site.xml

  <property>
    <name>system:java.io.tmpdir</name>
    <value>/tmp/hive/java</value>
  </property>
  <property>
    <name>system:user.name</name>
    <value>${user.name}</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:derby:,databaseName=$HIVE_HOME/metastore_db;create=true</value>
    <description>JDBC connect string for a JDBC metastore </description>
  </property>

this is set relative path in absolute URI and configuring Metastore where the database is stored

2.Remove $HIVE_HOME/metastore_db But Be aware, this will remove your schema completely!

3.Now,You must Initialize Derby database.By default, Hive uses Derby database

$HIVE_HOME/bin/schematool -initSchema -dbType derby

also I suppose that your environment variable has been set correctly ,if not please check them that are looking like below:

export HADOOP_HOME=/usr/local/hadoop 
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HIVE_HOME=/usr/lib/hive
export PATH=$PATH:$HIVE_HOME/bin

and then run hive command and type show databases;

score 0 · Answer 15 · answered Nov 26 '18 at 21:13

0

I'm surprised to see no one has pointed out the actual issue which is most often the case.

System.setSecurityManager(null)

Put this line at the start of your application.

answered Nov 26 '18 at 21:13

Abhinandan Dubey

655
2
9
15

1

I'm using windows and pyspark. I tried your solution but System is not defined ?! @Abhinandan Dubey – Peter Dec 17 '19 at 14:06
Try importing `java.lang.System` – Abhinandan Dubey Dec 17 '19 at 16:48

score 0 · Answer 16 · answered Mar 24 '19 at 04:55

When I used jdk 11, I deploymented hive on my master node and then those exceptions were thrown. Many methods I have tried but useless. Eventually I changed the version of jdk from 11 to 8, which used in master node. Then I started the hive successfully.

score 0 · Answer 17 · answered May 18 '23 at 15:44

remove metastore_db/*.lck
restart your Virtual Machine (CentOS, or some other)
start the mysql sevice: service mysqld start
test: hive -e 'show tables;'

Note that the mysql-connector-xxx-bin.jar has been stored in hive/lib/ and the xml file hive-site.xml has been configurated correctly. Check the username and password about MySQL carefully.

score -1 · Answer 18 · edited Feb 15 '16 at 00:09

I was facing the same issue--used the below steps to resolve it:

Create an file hive-site.xml and input the details (for Local/Prod mode). Make sure the below location exist /home/hadoop/bhishm/warehouse

Example:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/home/hadoop/bhishm/warehouse</value>
    <description>
    Local or HDFS directory where Hive keeps table contents.
    </description>
</property>
<property>
    <name>hive.metastore.local</name>
    <value>true</value>
    <description>
    Use false if a production metastore server is used.
    </description>
</property>
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:derby:;databaseName=/home/hadoop/bhishm/warehouse/metastore_db;create=true</value>
    <description>
    The JDBC connection URL.
    </description>
</property>
</configuration>

Edit the hive-env.sh--> add the java path as the first line after reducing memory usage:

Example:

# Hive Client memory usage can be an issue if a large number of clients
# are running at the same time. The flags below have been useful in 
# reducing memory usage:

# The java implementation to use.  Required.

export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk

Run the hive query.

java.lang.RuntimeException:Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

18 Answers18

Linked