What causes a http request to overwrite the internal buffer created by file.write()?

Question

I'm trying to write datetime.datetime.now() to a textfile after each iteration of a for loop, so I can calculate the number of calls made per second to our API. Here's my working code

import requests
import datetime
import config

# create a file with a timestamp starting at runtime.
with open('timelog-' + str(datetime.datetime.now().strftime("%y-%m-%d--%H-%S")) + '.txt', 'a') as log_time:
    for x in range(10):
        req = requests.post('https://' + config.env.lower() + '.website.com/args'
                                            + config.cli + '/args/'
                                            + config.con + '/args/?args='
                                            + config.cel + '&date='
                                            + config.req + '&args', headers=config.head)
        log_time.write(str(datetime.datetime.now().replace(microsecond=0)) + "\n")
        log_time.flush()

Now, the confusing thing to me, if I were to comment out req , I would not need to include log_time.flush(). Likewise, if I were to remove log_time.flush(), log_time.write() would not function correctly, instead I'd end with a blank file.

Is there any particular reason for this behavior?

Hm, the `log_time.close()` that's done implicitly when leaving the `with` context includes a `flush`. Is the file still blank after the last of the 10 requests if you comment out the `flush`? — Jeronimo, Nov 03 '18 at 08:05
Yes, this is what is boggling my mind, to be honest! I had a co-worker take a look and they said the exact same thing, understandably. — Glazbee, Nov 03 '18 at 08:18

Patrick Artner · Accepted Answer · 2018-11-03T11:21:14.493

The short answer: nothing - see below at unable to reproduce.

The fileobject buffers writes internally, flush suggests it would be a good time to provide the accumulated data to your OS. Your OS decides if and when they provide the data to the disc controller, which also have buffers and perform the physical write when they sees fit to do so.

You are not guaranteed to create the file simply by calling flush - its merely a "suggestion" and you have no control over the other "deciders" in the chain.

Your OS/disc controller should handle files "transparently" - if it hasn't written them and you request it, it should provide what it has buffered if that's not yet written physically.

See f.e. After how many seconds are file system write buffers typically flushed? or How often does python flush to a file?

Beside that - unable to reproduce:

import requests
import datetime 

# create a file with a timestamp starting at runtime.
name = 'timelog-' + str(datetime.datetime.now().strftime("%y-%m-%d--%H-%S")) + '.txt'
with open(name, 'a') as log_time:
    for x in range(10):
        req = requests.post('https://www.google.de/search?q=google')
        log_time.write(str(datetime.datetime.now()) + "\n")

with open(name) as f:
    print(f.read())

Output:

2018-11-03 10:46:31.220314
2018-11-03 10:46:31.258467
2018-11-03 10:46:31.296618
2018-11-03 10:46:31.333934
2018-11-03 10:46:31.372513
2018-11-03 10:46:31.409757
2018-11-03 10:46:31.447643
2018-11-03 10:46:31.485454
2018-11-03 10:46:31.523937
2018-11-03 10:46:31.562102

Thank you for the thorough answer. Definitely a strange one. I've reinstalled PyCharm and the code works entirely as intended without modification, for what reason I'm unsure. However, I'm marking your answer as accepted as I've learned some valuable information on `flush`. Thanks again! — Glazbee, Nov 03 '18 at 11:19

What causes a http request to overwrite the internal buffer created by file.write()?

1 Answers1