1

Query : Is there any way in Python to say:

  • Don't use /dev/shm for shared memory storage and use some other temporary location such as /tmp instead. OR

  • Instead of multiprocessing.SemLock use threading.Lock for synchronization (if it's feasible and not illogical).

Background:

I am running a simple multiprocessing.Pool example on an ESX (hypervisor) which is running into OSError : Function not implemented error.

Other mentions to this problem (here and here) on StackOverflow have pointed out that it happens when /dev/shm is not mounted. Due to which, _multiprocessing.SemLockfails to create synchronization lock(s).

It is actually the case for me as ls -lrt /dev/shm fails for me. However I cannot try the solutions recommended on the above links as

  • mount as mount command is not implemented on the platform (ESX/VMKernel) I am running on.

  • VM is not persistent and /etc/fstab gets washed away on shutdown itself.

So I am looking for ways to either use threading.Lock() for synchronization or ask Python to not to use /dev/shm/ for synchronization.

Community
  • 1
  • 1
ViFI
  • 971
  • 1
  • 11
  • 27
  • I understand like 80% of your question. Where I'm stumbling is that you try to use both threading (`threading.lock`) and `multiprocessing`. Are you still unsure which one to choose? Or did you mix up the two? Because you cannot use `threading.lock` using processes – hansaplast Feb 13 '17 at 15:42
  • 1
    Did you find any workaround? I have the same issue on AWS Lambda, where /dev/shm isn't mounted in the container the lambda is run in. – Atif Jun 08 '17 at 14:38

1 Answers1

2

If you try using a threading.Lock object in two multiprocessing process, the two Lock objects won't be linked. The threading.Lock relies on the fact that 2 Thread will run on the same core and share the file descriptors and GIL. This is not the case with 2 Process.

The module _multiprocessing.semaphore relies on the OS pthread.sem_open to create the SemLock object. This means that the main question is: Is there some OS semaphores implemented on ESX? If there is, you can implement your own SemLock objects using ctypes and inspiration from loky.backend.semlock, which allows you to accesses the internal pthread primitives.

Note that it may require lot of work to make sure SemLock is correctly used with the multiprocessing module. A solution could be to use a modify version of loky, that use this SemLock if you succeed in implementing it on ESX.

Thomas Moreau
  • 4,377
  • 1
  • 20
  • 32