1

I was looking into an issue I had with Mender, where the installation progress (which is about copying a file on a block device) is not reported correctly.

My feeling is that it's about the kernel page cache: the progress bar says "100%" when the code has read the whole image, but that does not mean that the kernel is done writing it.

More specifically, Mender calls n, err := io.Copy(dev, image), which returns after the kernel is done writing. But the progress bar is linked to the "image" Reader, which is fully read tens of seconds before io.Copy returns.

Because the file is opened with flags here, I naively thought that I just had to set flag |= O_SYNC, so that io.Copy(dev, image) would not read image faster than it writes to dev.

But setting O_SYNC does not make a difference.

It is not clear to me if O_SYNC is merely a hint (so I cannot count on it), if it could be that I am missing something on my device (say, I forgot a kernel option on my Raspberry Pi and therefore O_SYNC is useless), or if I just misunderstood what O_SYNC does?

EDIT: I also tried to set O_SYNC | O_DIRECT (though O_DIRECT is apparently not exposed in Go and so I did O_SYNC | 0o40000), but I got the following error when opening the block device:

Opening device: /dev/mmcblk0p2 for writing with flag: 1069058
Failed to open the device: "/dev/mmcblk0p2": open /dev/mmcblk0p2: not a directory
JonasVautherin
  • 7,297
  • 6
  • 49
  • 95
  • 3
    Do you _know_ or _assume_ that `image` is fully read before `io.Copy` returns? In the former case, how did you test that? – Yotam Salmon May 04 '22 at 16:40
  • 2
    I added the [tag:linux] tag since this is really a Linux internals question. I don't think `O_SYNC` does much for you here though. It's meant to make writes synchronous but unless it's passed into the device driver and the driver obeys it and the hardware cooperates, it literally *can't* help you out. `O_DIRECT` is for bypassing the buffer cache. [more here](https://stackoverflow.com/q/41257656/1256452) – torek May 04 '22 at 16:56
  • Thanks @torek! I tried O_DIRECT as well and then thought it was not the right way (also because of this: https://svetlinmladenov.wordpress.com/2013/01/20/on-o_direct/). I edited my question with what I tried. Should O_DIRECT work with a block device? I don't understand why it says: "not a directory", but maybe `0o40000` is wrong...? – JonasVautherin May 04 '22 at 17:18
  • 1
    @YotamSalmon: From reading the code (but I did not test it), the progress bar is some kind of "decorator" (not sure if that's the right word) around the `Reader` of the image. So the progress bar is directly updated when the `Reader` reads. Hence I'm reasonably confident that "100%" means that the `Reader` read all the bytes. Unless there is some kind of weird proxy running somewhere in the code and that I missed, that is :-). – JonasVautherin May 04 '22 at 17:21
  • 1
    `040000` (`0o40000` for Go) is the *default* from linux/bits/fcntl-linux.h but your architecture might have a different (arch-dependent) value. It looks like RPi (arm) has that bit as `O_DIRECTORY` and `O_DIRECT` defined as `0200000`: see https://github.com/raspberrypi/tools/blob/master/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/arm-linux-gnueabihf/libc/usr/include/arm-linux-gnueabihf/bits/fcntl.h – torek May 04 '22 at 17:32
  • 2
    Meanwhile, if the progress bar decorates only the `io.Reader` part, that would likely be part of the problem... – torek May 04 '22 at 17:34
  • @torek: you are right, there was a bug in the termination logic for the progress bar -_-. Would you mind writing your comments about O_SYNC and O_DIRECT (also the raspberry pi part) as an answer that I can accept? I'll edit my question a bit to focus on the O_SYNC part (if that seems fine to you) – JonasVautherin May 04 '22 at 20:29
  • 1
    @YotamSalmon: So you were right, it was on the progress bar side. Thanks to you I had a closer look there! Thanks a lot :) – JonasVautherin May 04 '22 at 20:35
  • 1
    Sync wouldn't matter. If the installer thought the write was done (but it was actually pending), so would the rest of the system, and the install would be 'done'. The `io.Reader` decorator explanation makes a lot more sense. – erik258 May 05 '22 at 00:23

1 Answers1

1

Summarizing the comments:

  1. The main issue is that the progress bar is decorating the reader (as Yotam Salmon noted), not the writer; the delay is on the side of the writer.

  2. On most Linux systems, O_DIRECT is indeed 0o40000, but on ARM (including Raspberry Pi) it is 0o200000, with 0o40000 being O_DIRECTORY. This explains the "not a directory" error.

  3. O_SYNC is in fact the bit you want, or you can simply issue an fsync system call (use Flush if appropriate, and then Sync, as noted in When to flush a file in Go?). The O_SYNC bit implies an fsync system call as part of each write system call.

Fully synchronous I/O is a bit of a minefield: some devices lie about whether they've written data to nonvolatile storage. However, O_SYNC or fsync is the most guarantee you'll get here. O_DIRECT is likely irrelevant since you're going directly to a device partition /dev file. O_SYNC or fsync may be passed through to the device driver, which may do something with it, which may get the device to write to nonvolatile storage. There's more about this in What does O_DIRECT really mean?

torek
  • 448,244
  • 59
  • 642
  • 775