2

I know that when I write files using write() + fsync() (or O_SYNC + write(), I think they are same ref #1 ref #2), it means I am using blocking synchronous I/O, and the if the write()(with O_SYNC) or fsync() returns, it means the data is safely on the device medium (e.g., TLC NAND for SSD) rather than the device cache (e.g., DDRAM in SSD).

While what if I use libaio? (because I want to make sure the write issued by libaio is on the storage medium rather than the device cache. i.e., I suppose when io_getevents() returns, it may not make sure the write is on the storage medium, and it may just on the device cache)

  • Question 1: does fsync() exclusively works for synchronous I/O?
  • Question 2：is fsync() after io_submit() an undefined behavior?
  • Question 3: how to make asynchronous write safely persisted to the device medium rather than device cache (no battery-backed cache).
Tim He
  • 350
  • 2
  • 13
  • It seems that linux does not support aio fsync currently: https://github.com/axboe/fio/blob/fio-3.25/engines/libaio.c#L256-L268 – Tim He Feb 07 '21 at 02:23
  • And if one wants to safely persistent write with libaio, using `RWF_SYNC` flag in `io_submit(2)`: https://man7.org/linux/man-pages/man2/io_submit.2.html – Tim He Feb 07 '21 at 02:28

1 Answers1

2

(This answer is only looking at things from Linux perspective. Other operating systems may have different behaviour/semantics)

So long as you wait for any outstanding asynchronous I/O to be completed you will be able to then send down your fsync() and know all previous I/O was written to stable storage to the best knowledge of the Linux kernel*.

does fsync() exclusively works for synchronous I/O?

This question doesn't make sense. fsync() works on I/O that Linux has agreed has been "sent" and then also flushes the device cache. If you asynchronously send an I/O to the kernel, has the kernel agreed it has been "sent"? Also see the answer to final question...

is fsync() after io_submit() an undefined behavior?

Technically undefined behaviour would mean from then on ANYTHING would be allowed to legally happen (e.g. so called nasal daemons). That's not the case here but I think you are asking your first question so the answer there still holds.

how to make asynchronous write safely persisted to the device medium rather than device cache (no battery-backed cache).

If you're stuck using libaio you could manually build a barrier by waiting for all outstanding I/O to be completed and then sending an fsync() and waiting for that or (as mentioned in a comment). Another libaio technique is to set the RWF_SYNC flag during io_submit(). If you were using io_uring you would have another option because you can use its chaining and it has an asynchronous fsync operation (IORING_OP_FSYNC).

* I'm excluding the cases where errors happen because they are rather more complicated. See Writing programs to cope with I/O errors causing lost writes on Linux for a detailed explaintion.

Anon
  • 6,306
  • 2
  • 38
  • 56
  • In the other answer, you mentioned: "With direct I/O io_submit() does not return until all its requests have been allocated and queued at the block layer level." In that case, is it still necessary to manually build a barrier before calling `fsync()`? – sayap Jun 24 '21 at 14:46
  • 1
    @sayap **Yes** because how else do you know the `fsync()` won't overtake the submitted I/Os and return before they do? Just because you've queued some items you need another guarantee that re-ordering won't take at any level before your `fsync()` completes. If you were somehow guarantee the kernel would never re-order submitted commands (how?) I can just turn around and ask "how do you guarantee the disk controller won't re-order the commands"? – Anon Jun 24 '21 at 15:28