Shared file access between Python and Matlab

Question

I have a Matlab application that writes in to a .csv file and a Python script that reads from it. These operations happen concurrently and at their own respective periods (not necessarily the same). All of this runs on Windows 7.

I wish to know :

Would the OS inherently provide some sort of locking mechanism so that only one of the two applications - Matlab or Python - have access to the shared file?
In the Python application, how do I check if the file is already "open"ed by Matlab application? What's the loop structure for this so that the Python application is blocked until it gets access to read the file?

if the file is open on windows won't you get an error trying to write to it? — Padraic Cunningham, Mar 16 '15 at 19:43
Well this is actually just one possible use case. Later I may have it the other way round as well i.e. Python writes while Matlab reads. So would _winapi.CreateFile method work in such a scenario as well? — JDD, Mar 17 '15 at 06:16

score 0 · Accepted Answer · answered Mar 16 '15 at 19:29

0

I am not sure about window's API for locking files

Heres a possible solution:

While matlab has the file open, you create an empty file called "data.lock" or something to that effect.
When python tries to read the file, it will check for the lock file, and if it is there, then it will sleep for a given interval.
When matlab is done with the file, it can delete the "data.lock" file.

Its a programmatic solution, but it is simpler than digging through the windows api and finding the right calls in matlab and python.

answered Mar 16 '15 at 19:29

yxre

3,576
21
20

As for checking if the file is opened already, it would be easier just to attempt opening it and catch the error. – Eugene Sh. Mar 16 '15 at 19:31
Sure, this is one way to do things whereby I manage the file access by using the "data.lock" as a lock object. I am looking for solutions which use inherent methods of Python and Matlab to handle this. – JDD Mar 16 '15 at 19:35
I am very familiar with python, and you will not find a built in method for this specific task in python without using an external lib. Here is an external lib [related question](http://stackoverflow.com/questions/489861/locking-a-file-in-python). Hope this helps. – yxre Mar 16 '15 at 19:40
LeBarton : I was considering your solution for the sheer simplicity of it. But just the way you maintain a "data.lock" to prevent Python from reading while Matlab is writing, shouldn't I have another lock to prevent Matlab from writing while Python is reading? – JDD Mar 17 '15 at 06:13
If it possible for the scripts to compete for the resource, then you can have the matlab script check for the lock. You can do two lock files or just one. You just need to code it in such a way that the script is aware that it has a lock on the file, and the script is guaranteed to release that lock whether it ends normally or with a fatal exception. – yxre Mar 18 '15 at 16:14

score 0 · Answer 2 · edited May 23 '17 at 12:28

0

If Python is only reading the file, I believe you have to lock it in MATLAB because a read-only open call from Python may not fail. I am not sure how to accomplish that, you may want to read this question atomically creating a file lock in MATLAB (file mutex)

However, if you are simply consuming the data with python, did you consider using a socket instead of a file?

edited May 23 '17 at 12:28

Community

1
1

answered Mar 16 '15 at 19:39

agomcas

695
5
12

score 0 · Answer 3 · answered Mar 17 '15 at 13:14

In Windows on the Python side, CreateFile can be called (directly or indirectly via the CRT) with a specific sharing mode. For example, if the desired sharing mode is FILE_SHARE_READ, then the open will fail if the file is already open for writing. If the latter call instead succeeds, then a future attempt to open the file for writing will fail (e.g. in Matlab).

The Windows CRT function _wsopen_s allows setting the sharing mode. You can call it with ctypes in a Python 3 opener:

import sys
import os
import ctypes as ctypes
import ctypes.util

__all__ = ['shdeny', 'shdeny_write', 'shdeny_read']

_SH_DENYRW = 0x10  # deny read/write mode
_SH_DENYWR = 0x20  # deny write mode
_SH_DENYRD = 0x30  # deny read
_S_IWRITE  = 0x0080  # for O_CREAT, a new file is not readonly

if sys.version_info[:2] < (3,5):
    _wsopen_s = ctypes.CDLL(ctypes.util.find_library('c'))._wsopen_s
else:
    # find_library('c') may be deprecated on Windows in 3.5, if the 
    # universal CRT removes named exports. The following probably 
    # isn't future proof; I don't know how the '-l1-1-0' suffix 
    # should be handled.
    _wsopen_s = ctypes.CDLL('api-ms-win-crt-stdio-l1-1-0')._wsopen_s

_wsopen_s.argtypes = (ctypes.POINTER(ctypes.c_int), # pfh
                      ctypes.c_wchar_p,             # filename
                      ctypes.c_int,                 # oflag
                      ctypes.c_int,                 # shflag
                      ctypes.c_int)                 # pmode

def shdeny(file, flags):
    fh = ctypes.c_int()
    err = _wsopen_s(ctypes.byref(fh),
                    file, flags, _SH_DENYRW, _S_IWRITE)
    if err:
        raise IOError(err, os.strerror(err), file)
    return fh.value

def shdeny_write(file, flags):
    fh = ctypes.c_int()
    err = _wsopen_s(ctypes.byref(fh),
                    file, flags, _SH_DENYWR, _S_IWRITE)
    if err:
        raise IOError(err, os.strerror(err), file)
    return fh.value

def shdeny_read(file, flags):
    fh = ctypes.c_int()
    err = _wsopen_s(ctypes.byref(fh),
                    file, flags, _SH_DENYRD, _S_IWRITE)
    if err:
        raise IOError(err, os.strerror(err), file)
    return fh.value

For example:

if __name__ == '__main__':
    import tempfile
    filename = tempfile.mktemp()
    fw = open(filename, 'w')
    fw.write('spam')
    fw.flush()
    fr = open(filename)
    assert fr.read() == 'spam'
    try:
        f = open(filename, opener=shdeny_write)
    except PermissionError:
        fw.close()
        with open(filename, opener=shdeny_write) as f:
            assert f.read() == 'spam'
    try:
        f = open(filename, opener=shdeny_read)
    except PermissionError:
        fr.close()
        with open(filename, opener=shdeny_read) as f:
            assert f.read() == 'spam'
    with open(filename, opener=shdeny) as f:
        assert f.read() == 'spam'
    os.remove(filename)

In Python 2 you'll have to combine the above openers with os.fdopen, e.g.:

f = os.fdopen(shdeny_write(filename, os.O_RDONLY|os.O_TEXT), 'r')

Or define an sopen wrapper that lets you pass the share mode explicitly and calls os.fdopen to return a Python 2 file. This will require a bit more work to get the file mode from the passed in flags, or vice versa.

Thanks for all your efforts, but I found the other solution to be simpler! — JDD, Mar 19 '15 at 21:35
Yes, the other solution is simpler given you're only concerned about controlling access between Matlab and your Python script. Setting the OS file sharing mode is a universal solution. — Eryk Sun, Mar 19 '15 at 21:39

Shared file access between Python and Matlab

3 Answers3