Grub2/Debian10 change root partition on boot

Question

I have a headless NAS system that ran out of disk space on it's system partition. It's running Debian 10, usually managed via OpenMediaVault 5 with a grub2 bootloader on an EFI system. Apart from the HDD data raid it had an SSD with partitions for / and /boot. I've installed an additional SSD with partitions for /, /boot/EFI and /var. I've carefully stopped all service and copied over the data via rsync. I've modified the /etc/fstab of the new partition to reflect the new mounting scheme with the new UUIDs (read from the output of blkid). I did not modify the BIOS or the bootloader. I intended to still use the /boot partition of the original SSD for booting up.

By now, grub has already detected the new system in the /boot/grub/grub.cfg file. So I changed the entry for the default entry in /etc/default/grub, ran grub-mkconfig and rebooted. But it still boots with the first entry into my old system. It's hard to tell on a headless system, but at least according to /proc/cmdline, it looks like the first entry with the old UUID as value for root=. But I know of no way to tell if this is the first attempt or maybe a fallback after a fail.

Also I noticed that /boot/grub/grub.cfg does not change after I run grub-mkconfig. And while I see the UUIDs of the root partitions in the /boot/grub/grub.cfg file, I don't see neither the old nor the new ones anywhere in the /etc/grub.d structure.

I've clearly not fully understood how grub2 works and how it is supposed to be configured. Can somebody give me some guidance on the problem or point me to a documentation that could help me?

Update: Thanks to User @user1686 's answer I re-created my grub.cfg by redirecting the output of grub-mkconfig into a file with the -o option. This took over the GRUB_DEFAULT setting. For some reason, while mentioning the right UUID several times in the entry, in the actual linux = boot command it still showed the UUID of the old system. I manually corrected this and finally booted into the new system. /Update

Some relevant files:

output of blkid (shortened)

/dev/sdb: UUID="c71b542e-8149-827c-e596-32c80ade181d" TYPE="linux_raid_member"
/dev/sda: UUID="c71b542e-8149-827c-e596-32c80ade181d" TYPE="linux_raid_member"
/dev/sdc1: UUID="EAFD-673A" TYPE="vfat" PARTLABEL="boot"
/dev/sdc2: UUID="533dfffd-ebd1-4be6-8e7b-55489f1ffece" TYPE="ext4" PARTLABEL="system" 
/dev/sdd1: LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="97CC-45A4" TYPE="vfat" PARTLABEL="boot" 
/dev/sdd2: LABEL="system" UUID="bd04242b-b64d-46e7-97ba-116d3c987c67" TYPE="ext4" PARTLABEL="system"
/dev/sdd3: LABEL="var" UUID="7142881b-3165-4b17-bf20-445e22b0cb03" TYPE="ext4" PARTLABEL="var" 
/dev/md0: UUID="4a432c95-b1ad-4ff6-b10b-66a343f2bfad" TYPE="crypto_LUKS"

/etc/default/grub (old root partition)

GRUB_DEFAULT=4
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_SUBMENU=y

/boot/grub/grub.cfg

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
BEGIN /etc/grub.d/00_header
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi
if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi
export menuentry_id_option
if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi
function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}
if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
set root='hd2,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
else
  search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
fi
    font="/usr/share/grub/unicode.pf2"
fi
if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=C
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=5
Fallback normal timeout code in case the timeout_style feature is
unavailable.
else
    set timeout=5
  fi
fi
END /etc/grub.d/00_header
BEGIN /etc/grub.d/05_debian_theme
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
END /etc/grub.d/05_debian_theme
BEGIN /etc/grub.d/10_linux
function gfxmode {
        set gfxpayload="${1}"
}
set linux_gfx_mode=
export linux_gfx_mode
menuentry 'Debian GNU/Linux, with Linux 4.19.0-14-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-14-amd64-advanced-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-14-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-14-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-14-amd64
}
menuentry 'Debian GNU/Linux, with Linux 4.19.0-14-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-14-amd64-recovery-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-14-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-14-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-14-amd64
}
menuentry 'Debian GNU/Linux, with Linux 4.19.0-10-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-10-amd64-advanced-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-10-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-10-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-10-amd64
}
menuentry 'Debian GNU/Linux, with Linux 4.19.0-10-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-10-amd64-recovery-533dfffd-ebd1-4be6-8e7b-55489f1ffece' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        set root='hd2,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd2,gpt2 --hint-efi=hd2,gpt2 --hint-baremetal=ahci2,gpt2  533dfffd-ebd1-4be6-8e7b-55489f1ffece
        else
          search --no-floppy --fs-uuid --set=root 533dfffd-ebd1-4be6-8e7b-55489f1ffece
        fi
        echo    'Loading Linux 4.19.0-10-amd64 ...'
        linux   /boot/vmlinuz-4.19.0-10-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.19.0-10-amd64
}
END /etc/grub.d/10_linux
BEGIN /etc/grub.d/20_linux_xen
END /etc/grub.d/20_linux_xen
BEGIN /etc/grub.d/30_os-prober
    menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.9-amd64 (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.9-amd64--bd04242b-b64d-46e7-97ba-116d3c987c67' {
            insmod part_gpt
            insmod ext2
            set root='hd3,gpt2'
            if [ x$feature_platform_search_hint = xy ]; then
              search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
            else
              search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
            fi
            linux /boot/vmlinuz-4.19.0-0.bpo.9-amd64 root=bd04242b-b64d-46e7-97ba-116d3c987c67 ro quiet
            initrd /boot/initrd.img-4.19.0-0.bpo.9-amd64
    }
    menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.9-amd64 (recovery mode) (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.9-amd64-root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single-bd04242b-b64d-46e7-97ba-116d3c987c67' {
            insmod part_gpt
            insmod ext2
            set root='hd3,gpt2'
            if [ x$feature_platform_search_hint = xy ]; then
              search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
            else
              search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
            fi
            linux /boot/vmlinuz-4.19.0-0.bpo.9-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
            initrd /boot/initrd.img-4.19.0-0.bpo.9-amd64
    }
    menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.8-amd64 (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.8-amd64--bd04242b-b64d-46e7-97ba-116d3c987c67' {
            insmod part_gpt
            insmod ext2
            set root='hd3,gpt2'
            if [ x$feature_platform_search_hint = xy ]; then
              search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
            else
              search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
            fi
            linux /boot/vmlinuz-4.19.0-0.bpo.8-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro quiet
            initrd /boot/initrd.img-4.19.0-0.bpo.8-amd64
    }
    menuentry 'Debian GNU/Linux, with Linux 4.19.0-0.bpo.8-amd64 (recovery mode) (on /dev/sdd2)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-4.19.0-0.bpo.8-amd64-root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single-bd04242b-b64d-46e7-97ba-116d3c987c67' {
            insmod part_gpt
            insmod ext2
            set root='hd3,gpt2'
            if [ x$feature_platform_search_hint = xy ]; then
              search --no-floppy --fs-uuid --set=root --hint-bios=hd3,gpt2 --hint-efi=hd3,gpt2 --hint-baremetal=ahci3,gpt2  bd04242b-b64d-46e7-97ba-116d3c987c67
            else
              search --no-floppy --fs-uuid --set=root bd04242b-b64d-46e7-97ba-116d3c987c67
            fi
            linux /boot/vmlinuz-4.19.0-0.bpo.8-amd64 root=UUID=533dfffd-ebd1-4be6-8e7b-55489f1ffece ro single
            initrd /boot/initrd.img-4.19.0-0.bpo.8-amd64
    }


END /etc/grub.d/30_os-prober
BEGIN /etc/grub.d/30_uefi-firmware
menuentry 'System setup' $menuentry_id_option 'uefi-firmware' {
        fwsetup
}
END /etc/grub.d/30_uefi-firmware
BEGIN /etc/grub.d/40_custom
This file provides an easy way to add custom menu entries.  Simply type the
menu entries you want to add after this comment.  Be careful not to change
the 'exec tail' line above.
END /etc/grub.d/40_custom
BEGIN /etc/grub.d/41_custom
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
END /etc/grub.d/41_custom

/etc/fstab (new root partition)

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / on new SDD on /dev/sdd2
UUID=bd04242b-b64d-46e7-97ba-116d3c987c67 /               ext4    noatime,nodiratime,errors=remount-ro 0       1
# /boot/efi on /dev/sdd1
UUID=97CC-45A4  /boot/efi       vfat    umask=0077      0       1
# /var on /dev/sdd3
UUID=7142881b-3165-4b17-bf20-445e22b0cb03 /var            ext4    defaults   0    2
tmpfs           /tmp            tmpfs   defaults        0       0
# >>> [openmediavault]
/dev/disk/by-label/data /srv/dev-disk-by-label-data ext4 defaults,nofail,user_xattr,noexec,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,acl 0 2
# <<< [openmediavault]

Answer 1

grub-mkconfig is mainly built to generate GRUB configs for the system it is currently running from, so in order to determine the root UUID it simply looks at the / filesystem.

So if you need to switch the root filesystem, you'll need to override this by manually specifying a new root= parameter in GRUB_CMDLINE_LINUX.

(I would just manually edit grub.cfg by hand to have the UUIDs I want, get it to boot, and once it's actually booting it's finally time to go back to grub-mkconfig.)

Also I noticed that /boot/grub/grub.cfg does not change after I run grub-mkconfig.

That's expected. You actually have to tell the program to write its output to a file, otherwise it won't. For example, this is what Debian's update-grub does:

grub-mkconfig -o /boot/grub/grub.cfg

And while I see the UUIDs of the root partitions in the /boot/grub/grub.cfg file, I don't see neither the old nor the new ones anywhere in the /etc/grub.d structure.

The /etc/grub.d directory doesn't contain the configuration – it contains scripts which try to automatically determine the configuration on the fly. Their output is what goes into grub.cfg.

For example, /etc/grub.d/10_linux is the script responsible for guessing your current root partition UUID and outputting all of the primary menuitems.