Prevent concurent writing to a file in python

Question

I have a file that holds a JSON which is read and written when devices log onto a server I have. Each device spawns its own process, and I have run into the issue where it looks like somehow two processes have written to the file concurrently. I assumed that using json.dump(object,open(filename,'w')) would be fast enough to prevent this issue, but clearly not.

Is there a way to block write access to an open file? I didn't see a way to do this in the open() documentation. I could control it with some sort of control file, but this seems like a problem that has already been solved. I just don't know where to look.

`json.dump(open(filename,'w'))` implies that you read in the content first, then closed the file. You're just asking for trouble by doing that. — Mad Physicist, May 05 '22 at 20:09

score 1 · Accepted Answer · answered May 05 '22 at 21:36

1

Allthough below comes with so many other questions, just considering your query please see if below helps:

import json
import os
import psutil


def is_fileopen(full_path):
    for current_process in psutil.process_iter():
        try:
            for all_open_files in current_process.open_files():
                if full_path == all_open_files.path:
                    return True
        except psutil.AccessDenied as ex:
            continue
    return False


file_path = os.path.join(os.getcwd(), "test.txt")
d = {"key": "value"}
if not is_fileopen(file_path):
    fo = open("test.txt", mode="w")
    json.dump(d, fo)
    fo.close()
else:
    print("File is already open")

answered May 05 '22 at 21:36

VMSMani

416
4
7

I like this. This makes it so I don't have to bother writing a file of some type and doesn't rely on making sure a lock file gets deleted at the end. Just curious about what issues there might be? – jeffpkamp May 06 '22 at 03:59
That's great that it helps you. One of the issue is that now you will have to put extra logic to make sure your app does log whatever you are logging if other process is holding it. Second issue would be performance impact but if number of systems are less , then you should be good – VMSMani May 06 '22 at 07:14

score 0 · Answer 2 · answered May 05 '22 at 20:09

0

Since multiple agents are writing the same file, you will need to use something to coordinate the file access. I suggest trying portalocker, which implements an easy API to file locking.

answered May 05 '22 at 20:09

felipecrp

999
10
16

This looked like what I was looking for, but can't test it as it gives an error when I try to import the module, and no one seems to have figured out a way around it. I might need to update from python3.5 to higher. – jeffpkamp May 05 '22 at 20:50
Yes, the library requires python 3.5 – felipecrp May 05 '22 at 20:59

score 0 · Answer 3 · answered May 05 '22 at 20:29

0

While it is not the best way to do it, I personally would create a second file that contains a flag or a single integer or some text and use it as a marker that a process is using the file. Just remember to reset the file when the process is done writing to avoid locking the main text file forever.

But I would highly recommend trying felipecrp answer first.

answered May 05 '22 at 20:29

Mohamed Yasser

641
7
17

I thought about this, but then wondered if I would have the same issue here, as the processes writing to that file might hit it at the same time too. – jeffpkamp May 05 '22 at 20:51
It is possible, but I'd say highly unlikely. Writing a simple integer or a binary flag leaves a very small time window. – Mohamed Yasser May 05 '22 at 22:27

Prevent concurent writing to a file in python

3 Answers3