Have been debugging some latency related issues on various servers, and eventually tracked it down to what I thought was NVMe power states (APST). However, I'm still seeing some things I can't figure out how to get past.
On a fresh Debian 12 install, ioping would report ~10ms latency on an NVMe drive. After adding nvme_core.default_ps_max_latency_us=0 to my boot config and rebooting, it clearly doesn't go into super-low power mode, and now reports ~300us latency on ioping.
However, it still should be lower. Through some trial and error, running ioping -i0.07 gives ~300us latency on the drive. However, dropping it to ioping -i0.06 reduces the latency down to ~50us (what I'd expect from the drive).
So clearly there is something still putting the drive to sleep, or something moving the drive to a slower mode (not necessarily on the drive itself, somewhere in the kernel?).
Any suggestions on how to keep the drive in full-speed mode all the time? I'd love it to respond at 50us all the time, and not drop to the 300us version at all.
root@dowest:/home/debian# ioping -i 0.06 /tmp
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=1 time=56.4 us (warmup)
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=2 time=50.6 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=3 time=49.6 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=4 time=49.7 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=5 time=50.0 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=6 time=48.9 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=7 time=49.8 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=8 time=49.8 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=9 time=49.0 us
root@dowest:/home/debian# ioping -i 0.07 /tmp
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=1 time=54.4 us (warmup)
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=2 time=324.5 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=3 time=321.2 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=4 time=320.1 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=5 time=320.2 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=6 time=322.0 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=7 time=319.3 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=8 time=318.7 us
4 KiB <<< /tmp (ext4 /dev/nvme0n1p2 467.9 GiB): request=9 time=321.4 us
