4

Our production container crashed because its LXC storage pool is out of space.

Here is dh -h from inside the container:

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       82G   78G  2.0G  98% /

Here the same mount from the host system:

/dev/loop0       82G   78G  2.0G  98% /var/lib/lxd/storage-pools/default

I remember a year ago when I was setting this up, it asked me if 80GB for something that sounded unfamiliar would be enough, and because it did not say "container disk size" and said something else, I just thought I better trust the default option. Now I know it is the disk space limit. How can I increase this limit now?

At the moment, I just cleaned some space urgently. The host machine actually has terabytes of disk space.

P.S. Sorry for complaining but documentation for LXC is one of the worst I've ever seen. It just does not make sense for a user like me.

P.P.S. I forgot that I actually use LXD, doh. The fact that I type lxc to do anything confused me.

UPDATE: Intuitively, I figured this command could resize the storage pool:

# lxc storage set default size 200GB

But it fails with:

Error: The [size] properties cannot be changed for "btrfs" storage pools

Any ideas?

Deltik
  • 19,971
exebook
  • 372

2 Answers2

10

LXD 5.13, released 15 April 2023, added support for growing existing loop file storage pools.

This is the template of the simple command to grow a storage pool <pool_name> to <new_size>:

lxc storage set <pool_name> size=<new_size>

For the resize in the question, this is what you would run:

lxc storage set default size=200G

Old Answer

When this answer was first written on 17 September 2019, we had LXD 3.17, which didn't have a built-in way to resize Btrfs file-backed storage pools, so I came up with this workaround that should work on LXD versions below 5.13:

Online Grow

One-Liner

Replace default with the name of the storage volume and 100G with the new desired size:

grow_lxd_btrfs_file() { STORAGE_POOL="$1" ; NEW_SIZE="$2" ; STORAGE_POOL_SOURCE="$(lxc storage get "$STORAGE_POOL" source)" ; sudo truncate -s "$NEW_SIZE" "$STORAGE_POOL_SOURCE" ; STORAGE_POOL_LOOP_DEVICE="$(losetup -j "$STORAGE_POOL_SOURCE" | awk -F': ' '{print $1}' | head -1)" ; sudo losetup -c "$STORAGE_POOL_LOOP_DEVICE" ; LXD_PID=$(pgrep lxd | head -1) ; NS_MOUNT_POINT=$(sudo nsenter -t "$LXD_PID" -m -- findmnt -no target "$STORAGE_POOL_LOOP_DEVICE") ; sudo btrfs filesystem resize max "/proc/$LXD_PID/root/$NS_MOUNT_POINT" ; lxd sql global "UPDATE storage_pools_config SET value = '$NEW_SIZE' WHERE key = 'size' AND storage_pool_id IN (SELECT id FROM storage_pools WHERE name = '$STORAGE_POOL')" ; } \
grow_lxd_btrfs_file default 100G

Explanation

  1. Set the variable $STORAGE_POOL to the name of your storage pool. In your case, it is default:

    STORAGE_POOL=default

  2. Set the variable $NEW_SIZE to the size you now want the storage pool to be:

    NEW_SIZE=100G

    The value is an integer and optional unit (example: 10K is 10*1024). Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).

    Important: The new size must be bigger than the current Btrfs file system size. You will lose all data in your storage pool if the size is smaller.

  3. On the host, find the path to the Btrfs storage pool file:

    STORAGE_POOL_SOURCE="$(lxc storage get "$STORAGE_POOL" source)"

    Verify
    

    echo "$STORAGE_POOL_SOURCE"

  4. Grow the Btrfs storage pool file:

    sudo truncate -s "$NEW_SIZE" "$STORAGE_POOL_SOURCE"

  5. Make the loopback device(s) aware of the new size:

    STORAGE_POOL_LOOP_DEVICE="$(losetup -j "$STORAGE_POOL_SOURCE" |
                            awk -F': ' '{print $1}' |
                            head -1)"

    sudo losetup -c "$STORAGE_POOL_LOOP_DEVICE"

  6. Get an LXD process ID so that we can enter its mount namespace later:

    LXD_PID=$(pgrep lxd | head -1)

    Verify
    

    echo "$LXD_PID"

  7. Get the online Btrfs mount point:

    NS_MOUNT_POINT=$(sudo nsenter -t "$LXD_PID" -m -- \
                 findmnt -no target "$STORAGE_POOL_LOOP_DEVICE")

    Verify
    

    echo "$NS_MOUNT_POINT"

  8. Finally, resize the Btrfs file system to the maximum possible size:

    sudo btrfs filesystem resize max "/proc/$LXD_PID/root/$NS_MOUNT_POINT"

  9. For consistency, update the LXD database with the new size of the storage pool:

    lxd sql global "UPDATE storage_pools_config SET value = '$NEW_SIZE'
    WHERE
        key = 'size' AND
        storage_pool_id IN
            (SELECT id FROM storage_pools WHERE name = '$STORAGE_POOL')"

    Verify
    

    lxd sql global "SELECT * FROM storage_pools_config
    WHERE storage_pool_id IN
        (SELECT id FROM storage_pools WHERE name = '$STORAGE_POOL')"
Deltik
  • 19,971
2

This link gives instructions on resizing a btrfs pool, but the last command fails as mentioned by #lcnittl.

As lcnittl stated, you need to change the command to

nsenter --mount=/run/snapd/ns/lxd.mnt /snap/lxd/current/bin/btrfs filesystem resize max /var/snap/lxd/common/lxd/storage-pools/pool1/

where pool1 is the said pool. if you try the "btrfs" command direct, it fails first off as the command cannot be found, then if you install via apt, it states its not a btrfs pool.

The above command seems to point to a brtfs command that comes with lxd rather than as a separate install and you need to use this command instead

Marc Nealer
  • 121