6

I have a server setup of an XFS partition on LVM on a Ubuntu 14.04.1 LTS. While copying files to the home partition, "No space left on device" is displayed.

df -h displays sufficient space:

/dev/mapper/prod--vg-home     35G   21G   15G  60% /home

I also used fallocate to fill the disk almost completely in the same home account to ensure neither quota nor other disk space issues are related:

/dev/mapper/prod--vg-home     35G   34G  1.5G  96% /home

df -i also displays sufficient inodes:

/dev/mapper/prod--vg-home   36700160  379390 36320770    2% /home

I also used a large amount of small random files in the same home account, which replicated the issue:

mkdir 1
cd 1
dd if=/dev/zero of=masterfile bs=10000000 count=1
split -b 10 -a 10 masterfile
cd ..
cp -R 1 2

This resulted in the "No space left on device" again with almost no additional space required (about 30MB) and df -i mentions:

/dev/mapper/prod--vg-home   36700160 1310464 35389696    4% /home

I did perform a second test on the same machine (Ubuntu 14.04.1 LTS).

I've created a new logical volume of 5 GB and filled it with small files using the procedure of dd and split above.

I received a "No space left on device" with the following disk space and inodes available:

/dev/mapper/prod--vg-test    5.0G  4.2G  811M  85% /mnt/test
/dev/mapper/prod--vg-test    4257712  937920  3319792   23% /mnt/test

I've performed this test using root privileges, to ensure the reserved space is still irrelevant here.

I did perform a third test on a different machine (Debian 2.6.32-5).

I've created a new logical volume of 5 GB and filled it with small files using the procedure of dd and split above.

The procedure succeeded with the following disk space and inodes available:

/dev/mapper/data-test      5.0G  4.2G  909M  83% /mnt/test
/dev/mapper/data-test      4721904 1000005 3721899   22% /mnt/test

I've performed this test using root privileges as well, to ensure the reserved space is still irrelevant here.

Does this point at a bug in Ubuntu 14.04.1 LTS?

I did verify the impact of changing the maximum percentage of in odes on the Ubuntu 14.04.1 LTS:

xfs_growfs -m 25 /dev/mapper/prod--vg-home

This amount can easily be decreased and increased.

While experimenting with this setting, I noticed that decreasing it to 3% and increasing it again to 25%, and deleting some files, allows me to add a lot more files again, but still causes the error prior to filling up either storage or inodes.

xfs_info displays:

meta-data=/dev/mapper/prod--vg-home isize=256    agcount=14, agsize=655360 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=9175040, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Is there any other setting which could cause the "No space left on device" message? Or should I conclude this is a bug?

Thank you

Purf
  • 63

2 Answers2

6

There is a bug with xfs_growfs which causes inodes to not be properly distributed across a partition. The solution is to simply remount with the inode64 option. For example, if this was the /dev/vda1, you would do the following:

mount -o remount,inode64 /dev/vda1

You can find more information about the bug here.

Andrew Lott
  • 953
  • 1
  • 7
  • 22
Daniel Stewart
  • 76
0

There is a situation where a directory (specially /home) might say the disk is full when it isn't, and that situation implies user's quotas. There might be some limitation on user's available space, check if that is the issue.

Isaac
  • 261