Hudson: returned status code 141: fatal: write error: No space left on device

Question

I copied one of the existing project and created a new project in Hudson. While running build it says "returned status code 141: fatal: write error: No space left on device"

Like suggested in other forums I checked free space and inode used in file system and nothing seems problematic here. Hudson is running as service and Hudons user has been given sudo privilege. Older job can be run so nothing different in new cloned job.

Disk Space

bash-4.1$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dev-lv_root
                       20G   19G   28K 100% /
tmpfs                 1.9G  192K  1.9G   1% /dev/shm
/dev/sda1             485M   83M  377M  19% /boot
/dev/mapper/vg_dev-lv_home
                       73G   26G   44G  38% /home

i-nodes used

bash-4.1$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_dev-lv_root
                     1310720  309294 1001426   24% /
tmpfs                 490645       4  490641    1% /dev/shm
/dev/sda1             128016      46  127970    1% /boot
/dev/mapper/vg_dev-lv_home
                     4833280  117851 4715429    3% /home

Hudson build log

bash-4.1$ cat log
Started by user anonymous
Checkout:workspace / /var/lib/hudson/jobs/Demo/workspace - hudson.remoting.LocalChannel@1d4ab266
Using strategy: Default
Checkout:workspace / /var/lib/hudson/jobs/Demo/workspace - hudson.remoting.LocalChannel@1d4ab266
Fetching changes from the remote Git repository
Fetching upstream changes from ssh://demouser@10.10.10.10:20/home/git-repos/proj.git
ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=ERROR:  (Underlying report) : Error performing command: git fetch -t ssh://demouser@10.10.10.10:20/home/git-repos/proj.git +refs/heads/*:refs/remotes/origin/*
Command "git fetch -t ssh://demouser@10.10.10.10:20/home/git-repos/proj.git +refs/heads/*:refs/remotes/origin/*" returned status code 141: fatal: write error: No space left on device

ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=ERROR: Could not fetch from any repository
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=FATAL: Could not fetch from any repository
ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=hudson.plugins.git.GitException: Could not fetch from any repository
        at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:887)
        at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:845)
        at hudson.FilePath.act(FilePath.java:758)
        at hudson.FilePath.act(FilePath.java:740)
        at hudson.plugins.git.GitSCM.gerRevisionToBuild(GitSCM.java:845)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:622)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1483)
        at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:507)
        at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:424)
        at hudson.model.Run.run(Run.java:1366)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:145)

It's out of space in `/var`, which is on your `/` (with only 28K free). — Anton Kovalenko, Feb 04 '13 at 07:03
I overlooked 100% for /dev/mapper/vg_dev-lv_root because of alignment but I think that't my root cause. Can someone help me understand what this filesystem means and how do I clean this up? — Manish Devraj, Feb 04 '13 at 07:06
For your mounts, it means "almost everything belonging to the system". You free it by removing stuff, like unused packages or something. (Maybe it's better to create a symbolic link from `/var/lib/hudson` to `/home/some/place/for/hudson`) — Anton Kovalenko, Feb 04 '13 at 07:12

score 7 · Accepted Answer · answered Feb 04 '13 at 07:40

Your error message is quite clear: There is no space left on device.

This is verified by your df output:

Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/vg_dev-lv_root     20G   19G   28K 100% /

This tells you, you have a root partition / with a total size of 20GB which is use by 100%. 20GB is probably a bit small in your case. As this "partition" is managed by LVM (/dev/mapper/vg...) it is possible to extend it to create more space for your data.

Otherwise you have to check, if there is some "garbage" laying around which can be removed. You can use something like xdiskusage / to find out, what is occupying your precious disk space.

But if you don't understand the concept of a file system, maybe it is easier to find someone else to do it for you.

score 2 · Answer 2 · answered Mar 15 '13 at 04:27

2

I had a very similar issue, it turned out to be a 40 gig log file from a "neverending" build which had been running for 8 hours

answered Mar 15 '13 at 04:27

golden-contact-computing

21
1

Hudson: returned status code 141: fatal: write error: No space left on device

2 Answers2