0

I'm trying to resize the /var partition on Debian 11.6.

FYI:

(I replaced the true volume group name with "xxx")

# lsblk
NAME                      MAJ:MIN RM   SIZE RO
TYPE  MOUNTPOINT sda                         8:0    0 447,1G  0 disk  
├─sda1                      8:1    0   512M  0 part  /boot/efi
├─sda2                      8:2    0   488M  0 part  /boot
└─sda3                      8:3    0 446,2G  0 part
  └─sda3_crypt            254:0    0 446,1G  0 crypt
    ├─xxx--vg-root        254:1    0  37,3G  0 lvm   /
    ├─xxx--vg-tmp         254:2    0   7,4G  0 lvm   /tmp
    ├─xxx--vg-var         254:3    0   108G  0 lvm   /var
    ├─xxx--vg-home        254:4    0 287,4G  0 lvm   /home
    └─xxx--vg-varlog      254:5    0     6G  0 lvm
sr0                        11:0    1  1024M  0 rom

cat /proc/mounts |grep vg-var


/dev/mapper/xxx--vg-var /var ext4 rw,relatime 0 0

I terminated all the processes accessing /var:

systemctl stop lightdm
systemctl stop tor
systemctl stop cron

lsof and fuser showed no more open files.

Then I did:

# lvresize -r -L 64G xxx-vg/var

Unfortunately I don't have the exact output anymore. lvresize asked if it should unmount the filesystem, I chose YES and it failed to do so.

Then I tried to manually unmount /var:

# umount -f /var

This also failed.

Edit: apparently this failed because /var/log was still mounted.

After some research I tried lazy umount:

# umount -l /var

This worked.

Edit: /var/log is now also unmounted.

Then I tried again:

# lvresize -r -L 64G xxx-vg/var
Do you want to unmount "/var" ? [Y|n] y
[...]

This question is strange, as the filesystem was already unmounted.

lvresize failed again (rest of the output):

[...]
umount: /var: not mounted.
fsadm: Cannot proceed with mounted filesystem "/var".
  /sbin/fsadm failed: 1
  Filesystem resize failed.

Then I tried to use fsadm directly and got the most absurd result:

# fsadm -y resize /dev/mapper/xxx--vg-var 64G
Do you want to unmount "/var" ? [Y|n] y
umount: /var: not mounted.
fsadm: Cannot proceed with mounted filesystem "/var".

mount and umount do work properly:

# mount /var && cat /proc/mounts |grep var
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/xxx--vg-var /var ext4 rw,relatime 0 0

umount /var && cat /proc/mounts |grep var


efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0

umount tells me:

# umount /var
umount: /var: not mounted.

I tried to check with fsadm:

# fsadm check /dev/mapper/xxx--vg-var
# echo $?
3

From the manual:

A status code of 3 indicates the requested check operation could not be performed because the filesystem is mounted and does not support an online fsck(8).

e2fsck isn't working either:

# e2fsck /dev/mapper/xxx--vg-var
e2fsck 1.46.2 (28-Feb-2021)
/dev/mapper/xxx--vg-var is in use.
e2fsck: Cannot continue, aborting.

Apparently /var is "half mounted".

Any suggestions?

AJM
  • 500
ChristophK
  • 330
  • 1
  • 2
  • 11

1 Answers1

1

Thanks to Nikita Kipriyanov for the comments, which led me to the solution.

I made two mistakes:

  1. I didn't understand the implications of umount -l (lazy unmount).

  2. When checking for open files with fuser I missed the -m option:

fuser -m /var
/var:                    1rce     2rc     3rc     4rc     6rc     7rc     9rc    10rc    11rc    12rc    13rc    14rc    15rc    16rc    17rc    18rc    19rc    20rc    21rc    22rc    23rc    25rc    26rc    27rc    28rc    30rc    31rc    32rc    33rc    35rc    36rc    37rc    38rc    40rc    41rc    42rc    43rc    45rc    46rc    47rc    48rc    50rc    51rc    52rc    53rc    54rc    55rc    56rc    57rc    58rc    60rc    61rc    62rc    63rc    65rc    66rc    67rc    68rc    69rc    70rc    71rc    72rc    73rc    75rc    76rc    77rc    78rc    80rc    81rc    82rc    83rc    85rc    86rc    87rc    88rc    90rc   108rc   109rc   110rc   111rc   112rc   113rc   114rc   115rc   117rc   132rc   136rc   137rc   138rc   139rc   140rc   141rc   142rc   143rc   149rc   150rc   151rc   153rc   154rc   155rc   156rc   159rc   160rc   169rc   172rc   173rc   199rc   225rc   231rc   236rc   245rc   257rc   258rc   259rc   260rc   261rc   280rc   281rc   290rc   291rc   292rc   311rc   312rc   313rc   314rc   315rc   316rc   317rc   318rc   319rc   320rc   321rc   322rc   326rc   331rc   335rc   336rc   362rc   370rc   371rc   372rc   437rc   439rc   440rc   441rc   446rc   447rc   450rc   452rc   455rc   462rc   493rc   499rc   500rc   508rc   552rc   553rce   572rce   573rc   574rc   626rc   627rc   628rc   639rc   668rc   705rc   754rc   755rc   757rc   758rc   759rc   760rc   763rc   764rc   793rc   794rc   819rce   821rce   823rce   830rce   831rce   832rce   833rce   835rce   836rce   837rce   902rce   905rc   906rc   907rc   948rce   950rce   954rc   974rc  1021e  1079rc  1088rce  1092rc  1275rc  1299rc  1329rc  1332rc  1398rc  1404rce  1407rce  1408rce  1422rce  1427rce  1428re  1429re  1448rc  1449rc  1450rc  1451rc  1452rc  1455rc  1485re  1492re  1497re  1571rc  1573rc  1586re  1682re  1683re  1703re  1716rc  1717rc  1721rce  1722rce  1937rc  1944rc  1990rc  1992rc  2037rc  2073rc  3346rc  3361rc  3397rc  3419rc  3486rc  3487rc  3489rc  3490rc  3491rc  3492rc  3493rc

There were lots of open files on /var. After lazy unmount they were still open. So umount -l /var leaves the device in a "half mounted" state. Thus the confusing output from fsadm (filesystem not mounted, cannot proceed with mounted filesystem).

See Lazy umount or Unmounting a busy disk in Linux Again, thanks to Nikita Kipriyanov for pointing this out.

This finally did the job:

First, mount the volumes again:

# mount /var
# mount /var/log

Now stop the services which have files open (I'm on a ssh-connection):

# systemctl isolate rescue-ssh.target

Unmount the volumes (probably unnecesary, as lvresize would do this):

# umount /var/log
# umount /var

Do the actual resize operation (shrink):

# lvresize -r -L 64G xxx-vg/var
fsck from util-linux 2.36.1
/dev/mapper/xxx--vg-var: 9189/7036416 files (0.5% non-contiguous), 
1298192/28311552 blocks
resize2fs 1.46.2 (28-Feb-2021)
Resizing the filesystem on /dev/mapper/x--vg-var to 16777216 (4k) blocks.
The filesystem on /dev/mapper/xxx--vg-var is now 16777216 (4k) blocks long.
  Size of logical volume xxx-vg/var changed from 108,00 GiB (27648 extents) to 64,00 GiB (16384 extents).
  Logical volume xxx-vg/var successfully resized.

This time it worked! Now mount the volumes:

# mount /var
# mount /var/log

Ang get back to normal operation:

# systemctl isolate default.target

That's it.

ChristophK
  • 330
  • 1
  • 2
  • 11