Python shutil module move exception handling

Question

Destination path /tmp/abc in both Process 1 & Process 2 Say there are N number of process running we need to retain the file generated by the latest one

Process1

import shutil
shutil.move(src_path, destination_path)

Process 2

import os
os.remove(destination_path)

Solution 1. Handle the process saying if copy fails with [ErrNo2]No Such File or Directory

Is this the correct solution? Is there a better way to handle this

Useful Link A safe, atomic file-copy operation

What's the purpose of this code? Why do you want to create a file and remove the same file at the same time? — blhsing, Oct 19 '19 at 19:02
@blhsing No there are multiple same process running same time which will produce say same file name abc . I need to retain the latest file alone and need to remove the older ones (pruning) — Shankar, Oct 20 '19 at 17:41
Your question is not clear yet. You have process1 say P1 and Process 2 say P2 and it will generate "abc" timestamp & "abc" timestamp you need to keep latest file , Is it correct — nithin, Oct 21 '19 at 12:42
@nithin Correct. Every time while retaining the latest file , and remove all other files . More relevant to this https://stackoverflow.com/questions/11614815/a-safe-atomic-file-copy-operation/28090883 — Shankar, Oct 21 '19 at 14:08
The `os.remove(destination_path)` makes no sense. Each process should write its data to its own temporary unique file. When finished without an error, that file should be atomically moved (see `os.replace`) to the final destination file. When finished with an error, just remove the temporary file. The last process automatically "wins". Only files to remove are temporary files left after a crash. — VPfB, Oct 26 '19 at 14:55

score 0 · Answer 1 · answered Oct 26 '19 at 10:03

0

You can use FileNotFoundError error

try : 
    shutil.move(src_path, destination_path)
except FileNotFoundError: 
    print ('File Not Found') 
    # Add whatever logic you want to execute
except : 
    print ('Some Other error')

answered Oct 26 '19 at 10:03

Trect

2,759
2
30
35

The user wants to keep the latest file and perform deletion to the previously created file. Preferably, he wants to keep track the files – Angus Oct 26 '19 at 12:09

score -1 · Answer 2 · answered Oct 24 '19 at 22:47

The primary solutions that come to mind are either

Keep partial information in per-process staging file
Rely on the os for atomic moves

or

Keep partial information in per-process memory
Rely on interprocess communication / locks for atomic writes

For the first, e.g.:

import tempfile
import os

FINAL = '/tmp/something'

def do_stuff():
    fd, name = tempfile.mkstemp(suffix="-%s" % os.getpid())
    while keep_doing_stuff():
        os.write(fd, get_output())
    os.close(fd)
    os.rename(name, FINAL)

if __name__ == '__main__':
    do_stuff()

You can choose to invoke individually from a shell (as shown above) or with some process wrappers (subprocess or multiprocessing would be fine), and either way will work.

For interprocess you would probably want to spawn everything from a parent process

from multiprocessing import Process, Lock
from cStringIO import StringIO

def do_stuff(lock):
    output = StringIO()
    while keep_doing_stuff():
        output.write(get_output())
    with lock:
        with open(FINAL, 'w') as f:
            f.write(output.getvalue())
    output.close()

if __name__ == '__main__':
    lock = Lock()
    for num in range(2):
        Process(target=do_stuff, args=(lock,)).start()

Python shutil module move exception handling

2 Answers2