Is there a faster way to find a file than by listing all files in the directory?

Question

I've got a workflow that looks like this:

for i in some_list:
    if i not in os.listdir(a_directory):
        x = do_something(i)
        x.to_pickle(f"{a_directory}/{i}")

The os.listdir is expensive, because the directory is huge, and because it's over a network file system.

I have multiple workers doing this job, so I can't just list the contents of the directory once. If I do, then my workers will duplicate their work, and do_something is more expensive after all than os.listdir.

Is there something that looks for the presence of a specific file, rather than dumping all of them into a python list for me to string match on?

score 2 · Accepted Answer · answered Aug 26 '20 at 15:30

You can directly check to see if a file exists, per this question: Check whether a file exists

From that answer:

import os.path
os.path.isfile(fname)

Or:

from pathlib import Path

my_file = Path("/path/to/file")
if my_file.is_file():
    # file exists

Is there a faster way to find a file than by listing all files in the directory?

1 Answers1