How to avoid buffering in the Python fileinput library

Question

I've seen this question asked here, but the answers given did not work in my case and was marked duplicate.

python -u does not work for stdin in Python 3.
sys.stdin = sys.stdin.detach() throws a ValueError: underlying buffer has been detached.
None of these work for non-stdin inputs and other files being used as stream.
Adding a hook does not work: FileInput(openhook=hook_nobuf) and using open(buffering=0) in the hook.

I dug in the source code (/usr/lib/python3.2/fileinput.py) and saw that readlines(bufsize) was being used internally to load a buffer. No shell or other piping shenanigans.

Actually, I think you may want `python -u` on top of whatever else you need. You want to remove any underlying Python-and/or-stdio buffering on `stdin`, and _also_ remove any higher-level line-reading buffer, right? — abarnert, Feb 21 '13 at 21:30

score 5 · Accepted Answer · answered Feb 21 '13 at 21:13

5

What worked for me was simply setting FileInput(bufsize=1). The file.readlines() documentation does state "The optional size argument, if given, is an approximate bound on the total number of bytes in the lines returned." In practice, I get exactly one new line every time rather than having to fill a buffer.

with fileinput.input(bufsize=1) as f:
    for line in f:
        print("One line in, one line out!")

answered Feb 21 '13 at 21:13

dpyro

1,559
2
13
19

It seems like this is actually guaranteed to work, as long as `fileinput` uses `readlines(self._bufsize)`. Unfortunately, that in itself isn't documented to be true, but if you only care about CPython 3.2 you can be sure it is, and it seems pretty likely to be safe pretty widely beyond that, so if that's good enough, great. – abarnert Feb 21 '13 at 21:34
And if you read through `IOBase.readlines` ([pure Python](http://hg.python.org/cpython/file/3.2/Lib/_pyio.py#l497) and [C](http://hg.python.org/cpython/file/3.2/Modules/_io) implementations), it will call `readline`, which will call `read` 1 byte at a time if there's no buffer or `peek`. So, I think that cinches it, and you should accept your own answer. – abarnert Feb 21 '13 at 21:41
Also, you might want to file a documentation bug on the fact that [`fileinput.input`](http://docs.python.org/3/library/fileinput.html) doesn't mention what `bufsize` does at all, and that the language reference should have enough information to guarantee that `bufsize=1` (together with unbuffered `stdin`, when reading from `stdin`) means unbuffered `fileinput`. – abarnert Feb 21 '13 at 21:44
@abarnert Where is the best place to file a python documentation bug? – dpyro Feb 27 '13 at 21:20
I believe docs bugs go to the same issue tracker (http://bugs.python.org) as code bugs, although you probably want to read the [Python Developer's Guide](http://docs.python.org/devguide/) to make sure I'm remembering right. Also, if you're not _sure_ whether something is a bug, it may be better to bring it up on one of the mailing lists and get wider feedback first. – abarnert Feb 27 '13 at 21:26
More specifically, the [Helping with Documentation](http://docs.python.org/devguide/docquality.html) section gives the details of which mailing list, how to filter on the issue tracker, etc. – abarnert Feb 27 '13 at 21:28
From the fileinput docs: `Changed in version 2.7.12: The bufsize parameter is no longer used.` – Peter Gibson May 24 '18 at 05:14

How to avoid buffering in the Python fileinput library

1 Answers1