The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- (on Windows)

Question

I am running Spark on Windows 7. When I use Hive, I see the following error

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-

The permissions are set as the following

C:\tmp>ls -la
total 20
drwxr-xr-x    1 ADMIN Administ        0 Dec 10 13:06 .
drwxr-xr-x    1 ADMIN Administ    28672 Dec 10 09:53 ..
drwxr-xr-x    2 ADMIN Administ        0 Dec 10 12:22 hive

I have set "full control" to all users from Windows->properties->security->Advanced.

But I still see the same error. I have checked a bunch of links, some say this is a bug on Spark 1.5?

score 88 · Answer 1 · edited Mar 18 '20 at 15:10

88

First of all, make sure you are using correct Winutils for your OS. Then next step is permissions.
On Windows, you need to run following command on cmd:

D:\winutils\bin\winutils.exe chmod 777 D:\tmp\hive

Hope you have downloaded winutils already and set the HADOOP_HOME variable.

edited Mar 18 '20 at 15:10

logi-kal

7,107
6
31
43

answered Apr 08 '16 at 16:40

Nishu Tayal

20,106
8
49
101

7

@user1384205 : If you think, it helped you, can you accept the answer, so that It will help others to use – Nishu Tayal Dec 21 '16 at 12:03
1

It matters if you are using the 64-bit or 32-bit version of winutils. I ran this repeatedly and it looked like permissions were set correctly (via `winutils ls`) but only when I replaced winutils.exe did it work. – wrschneider Jul 13 '17 at 15:18
4

this solved the issue. Originally, I had the wrong winutils though. I'm using windows 10, 64 bit and the winutils.exe at this location [https://osdn.net/projects/win-hadoop/downloads/62852/hadoop-winutils-2.6.0.zip/] works for me – Partha Mishra Aug 03 '17 at 10:13
@ParthaMishra File not found on the link u provided – hshihab Aug 27 '17 at 13:46
you're right. it seems the file is moved. The only way out now is to search the web. The size is ~106 KB, please check it before using. There are other ones of smaller sizes and those don't work. – Partha Mishra Aug 28 '17 at 11:31
3

The 64 bit version of winutils can be found at https://codeload.github.com/gvreddy1210/64bit/zip/master – Nilav Baran Ghosh Jan 27 '18 at 17:39
I am trying spark on windows and I got same error. I have downloaded zip of spark prebuild for hadoop from [here](https://spark.apache.org/downloads.html) and extracted it in D: drive. Then created `hadoop\bin` directories inside extracted directory. Put winutils.exe in `hadoop\bin`. Ran python-spark hello world example and got this error. Point is where is `\tmp\hive` located \ configured? – Mahesha999 Mar 23 '18 at 09:22
@user1384205 : Please don't forget to accept the answer if it helps you. That make it easy for other people to find the solutions :) – Nishu Tayal Mar 07 '19 at 14:16

score 40 · Answer 2 · answered Jul 18 '16 at 15:15

40

First thing first check your computer domain. Try

c:\work\hadoop-2.2\bin\winutils.exe ls c:/tmp/hive

If this command says access denied or FindFileOwnerAndPermission error (1789): The trust relationship between this workstation and the primary domain failed.

It means your computer domain controller is not reachable , possible reason could be you are not on same VPN as your system domain controller.Connect to VPN and try again.

Now try the solution provided by Viktor or Nishu.

answered Jul 18 '16 at 15:15

Aaditya Raj

1,078
12
19

4

This worked for me, I had to connect to a vpn to get on my domain controllers network. wait for sometime and then ran the chmod 777 command. – Mahesh Jan 13 '17 at 06:34
I was doing the %HADOOP_HOME%\bin\winutils.exe chmod 777 c:\tmp\hive over and over again, and was getting confused why the permissions were not being set properly, this was an absolute lifesaver. – D.S. Mar 05 '17 at 23:35
this error FindFileOwnerAndPermission may be caused by domain controller not reachable if you connected on a given AD Domain. In my case I've connected to the VPN so domain controller was reachable and worked for me. Thank you @Aaditya – zaki benz Jun 17 '20 at 11:41
@Aaditya - Absolute life saver. Curious to know how did you get this answer? – Anand Aug 04 '20 at 11:58

OneCricketeer · Answer 3 · 2017-07-27T08:30:56.577

12

You need to set this directory's permissions on HDFS, not your local filesystem. /tmp doesn't mean C:\tmp unless you set fs.defaultFs in core-site.xml to file://c:/, which is probably a bad idea.

Check it using

hdfs dfs -ls /tmp

Set it using

hdfs dfs -chmod 777 /tmp/hive

edited Jul 27 '17 at 08:30

answered Dec 23 '15 at 13:05

OneCricketeer

179,855
19
132
245

Worked for me in Linux as well. Do not forget to change to the hdfs user first – Antoni May 03 '19 at 05:38
I am trying to run this on windows, when I am running hdfs dfs -ls /tmp I get to see bash: hdfs: command not found – Chandan Gawri Feb 12 '23 at 09:03
@ChandanGawri That simply means your OS `PATH` doesn't have `hdfs` command – OneCricketeer Feb 13 '23 at 17:01
I am using winutils.exe and problem was for my case was that I had to connect to VPN, also below I have shared answer for my problem if someone face can refer. – Chandan Gawri Feb 21 '23 at 08:57

score 11 · Answer 4 · edited May 23 '17 at 10:31

11

Next solution worked on Windows for me:

First, I defined HADOOP_HOME. It described in detail here
Next, I did like Nishu Tayal, but with one difference:C:\temp\hadoop\bin\winutils.exe chmod 777 \tmp\hive

\tmp\hive is not local directory

edited May 23 '17 at 10:31

Community

1
1

answered May 10 '16 at 08:20

L. Viktor

143
1
6

Thanks Viktor. But I get the following error when i try your solution. C:\Programs\winutil\bin>winutils.exe chmod 777 \tmp\hive ChangeFileModeByMask error (5): Access is denied. – user1384205 Jun 07 '16 at 07:23
You need to run the command prompt or cygwin as Administrator for the command to work – Eduardo Sanchez-Ros Jan 20 '17 at 13:11

score 9 · Answer 5 · answered Feb 24 '17 at 19:20

Error while starting the spark-shell on VM running on Windows: Error msg: The root scratch dir: /tmp/hive on HDFS should be writable. Permission denied

Solution: /tmp/hive is temporary directory. Only temporary files are kept in this location. No problem even if we delete this directory, will be created when required with proper permissions.

Step 1) In hdfs, Remove the /tmp/hive directory ==> "hdfs dfs -rm -r /tmp/hive"

2) At OS level too, delete the dir /tmp/hive ==> rm -rf /tmp/hive

After this, started the spark-shell and it worked fine..

This reply can be 4 months late, but this helped me. – Metadata Jul 05 '17 at 13:02 — Metadata, Jul 05 '17 at 13:02

Abhinandan Dubey · Answer 6 · 2018-09-17T21:21:39.197

6

This is a simple 4 step process:

For Spark 2.0+:

Download Hadoop for Windows / Winutils

Add this to your code (before SparkSession initialization):

if(getOS()=="windows"){
    System.setProperty("hadoop.home.dir", "C:/Users//winutils-master/hadoop-2.7.1"); 
}

Add this to your spark-session (You can change it to C:/Temp instead of Desktop).
```
.config("hive.exec.scratchdir","C:/Users//Desktop/tmphive")
```

Open cmd.exe and run:

"path\to\hadoop-2.7.1\bin\winutils.exe" chmod 777 C:\Users\\Desktop\tmphive

edited Sep 17 '18 at 21:21

answered Sep 17 '18 at 21:14

Abhinandan Dubey

655
2
9
15

This helped me. I was just setting up for running some test cases on Intellij – Ashish Mishra Sep 10 '20 at 20:52

score 2 · Answer 7 · answered Jun 30 '17 at 06:25

2

The main reason is you started the spark at wrong directory. please create folders in D://tmp/hive (give full permissions) and start your spark in D: drive D:> spark-shell

now it will work.. :)

answered Jun 30 '17 at 06:25

Satya.pydikondala

31
2

score 1 · Answer 8 · answered Dec 10 '15 at 10:11

1

Can please try giving 777 permission to the folder /tmp/hive because what I think is that spark runs as a anonymous user(which will come in other user category) and this permission should be recursive. I had this same issue with 1.5.1 version of spark for hive, and it worked by giving 777 permission using below command on linux

chmod -r 777 /tmp/hive

answered Dec 10 '15 at 10:11

Reena Upadhyay

1,977
20
35

3

Thanks. I tried that. Didnt work.. Did you so this on Windows? – user1384205 Dec 10 '15 at 12:18
3

The question is about this problem on Windows environment not unix. And even then the chmod will not work all by itself under a root level /tmp. This is the correct command for unix, sudo chmod -R 777 /tmp/hive – nitinr708 Jul 27 '17 at 08:18

score 1 · Answer 9 · answered Feb 12 '16 at 03:55

1

There is a bug in Spark Jira for the same. This has been resolved few days back. Here is the link.

https://issues.apache.org/jira/browse/SPARK-10528

Comments have all options, but no guaranteed solution.

answered Feb 12 '16 at 03:55

sunil

11
2

score 1 · Answer 10 · answered Mar 19 '17 at 15:59

1

Issue resolved in spark version 2.0.2 (Nov 14 2016). Use this version . Version 2.1.0 Dec 28 2016 release has same issues.

answered Mar 19 '17 at 15:59

user7735815

11
1

For me on a windows machine, this was the answer which pushed me in the correct direction. I also got another error but could solve this error with the help of the second answer of the following [post](http://stackoverflow.com/questions/38863003/sparkr-from-rstudio-gives-error-in-invokejavaisstatic-true-classname-meth) – Tobias Apr 05 '17 at 12:48

score 1 · Answer 11 · edited Sep 27 '18 at 13:46

1

I also faced this issue. This issue is related to network. I installed spark on Windows 7 using particular domain.

Domain name can be checked

Start -> computer -> Right click -> Properties -> Computer name, domain and workgroup settings -> click on change -> Computer Name (Tab) -> Click on Change -> Domain name.

When I run spark-shell command, it works fine, without any error.

In other networks I received write permission error. To avoid this error, run spark command on Domain specified in above path.

edited Sep 27 '18 at 13:46

Nishu Tayal

20,106
8
49
101

answered Jul 26 '17 at 00:15

Mayank Gupta

61
5

You said it yourself it worked for a different user / domain. Which means it's a permission issue not network. Thank you for an honest attempt – nitinr708 Jul 27 '17 at 08:20

score 1 · Answer 12 · answered Sep 25 '17 at 11:05

1

Use the latest version of "winutils.exe" and try. https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe

answered Sep 25 '17 at 11:05

Nagaraj Vittal

881
13
26

score 1 · Answer 13 · answered Jan 12 '19 at 07:51

I was getting the same error "The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-" on Windows 7. Here is what I did to fix the issue:

I had installed Spark on C:\Program Files (x86)..., it was looking for /tmp/hive under C: i.e., C:\tmp\hive
I downloaded WinUtils.exe from https://github.com/steveloughran/winutils. I chose a version same as what I chose for hadoop package when I installed Spark. i.e., hadoop-2.7.1 (You can find the under the bin folder i.e., https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin)
Now used the following command to make the c:\tmp\hive folder writable winutils.exe chmod 777 \tmp\hive

Note: With a previous version of winutils too, the chmod command was setting the required permission without error, but spark still complained that the /tmp/hive folder was not writable.

Harsha · Answer 14 · 2018-01-25T01:39:05.837

Using the correct version of winutils.exe did the trick for me. The winutils should be from the version of Hadoop that Spark has been pre built for.

Set HADOOP_HOME environment variable to the bin location of winutils.exe. I have stored winutils.exe along with C:\Spark\bin files. So now my SPARK_HOME and HADOOP_HOME point to the same location C:\Spark.

Now that winultils has been added to path, give permissions for hive folder using winutils.exe chmod 777 C:\tmp\hive

score 0 · Answer 15 · answered Dec 12 '19 at 20:39

You don't have to fix the permission of /tmp/hive directory yourself (like some of the answers suggested). winutils can do that for you. Download the appropriate version of winutils from https://github.com/steveloughran/winutils and move it to spark's bin directory (e. x. C:\opt\spark\spark-2.2.0-bin-hadoop2.6\bin). That will fix it.

score 0 · Answer 16 · answered Mar 17 '20 at 20:19

0

I was running spark test from IDEA, and in my case the issue was wrong winutils.exe version. I think you need to match it with you Hadoop version. You can find winutils.exe here

answered Mar 17 '20 at 20:19

gorros

1,411
1
18
29

score 0 · Answer 17 · answered Feb 12 '23 at 09:40

/*
Spark and hive on windows environment
Error: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-

Pre-requisites: Have winutils.exe placed in c:\winutils\bin\
Resolve as follows:
*/

C:\user>c:\Winutils\bin\winutils.exe ls
FindFileOwnerAndPermission error (1789): The trust relationship between this workstation and the primary domain failed.
// Make sure you are connected to the domain controller, in my case I had to connect using VPN

C:\user>c:\Winutils\bin\winutils.exe ls c:\user\hive
drwx------ 1 BUILTIN\Administrators PANTAIHQ\Domain Users 0 Aug 30 2017 c:\user\hive

C:\user>c:\Winutils\bin\winutils.exe chmod 777 c:\user\hive

C:\user>c:\Winutils\bin\winutils.exe ls c:\user\hive
drwxrwxrwx 1 BUILTIN\Administrators PANTAIHQ\Domain Users 0 Aug 30 2017 c:\user\hive

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- (on Windows)

17 Answers17

Linked

Related