How can I iterate over all files in all folders of one master folder?

Question

So I wrote a Python scrip that would do a certain thing to a certain .txt file:

with open("1.txt") as f:
    for line in f:
        #DoStuff

Now this works for 1 .txt file. I have One master folder, in the master folder I have different other folders, and in each folder I also have several .txt files.

How can I iterate over all this to apply my script to every .txt file in every folder in the master file?.

I'm sorry for not being clear, I'll try again: I have one Folder, in that Folder I have several Folders. In each one of the several Folders I have several .txt files I wanna go over all .txt files — Jeries Haddad, Jun 20 '19 at 12:15
Your question was clear, but when it takes me less than a minute to find multiple duplicates its clear that you put no effort into solving this yourself — Sayse, Jun 20 '19 at 12:21
@Sayse instead of being rude try to actually help. This is my second day using Python, I'm still trying to learn. I tried looking up in a lot of places and all I could find was to iterate over different files in one folder. — Jeries Haddad, Jun 20 '19 at 12:58

score 1 · Accepted Answer · answered Jun 20 '19 at 12:17

You can use os.walk()

import os

path = 'c:\\projects\\hc2\\'

files = []
# r=root, d=directories, f = files
for r, d, f in os.walk(path):
    for file in f:
        if '.txt' in file:
            files.append(os.path.join(r, file))

for f in files:
    print(f)

Output:

c:\projects\hc2\app\readme.txt
c:\projects\hc2\app\release.txt
c:\projects\hc2\web\readme.txt
c:\projects\hc2\whois\download\afrinic.txt
c:\projects\hc2\whois\download\apnic.txt
c:\projects\hc2\whois\download\arin.txt
c:\projects\hc2\whois\download\lacnic.txt
c:\projects\hc2\whois\download\ripe.txt
c:\projects\hc2\whois\out\test\resources\asn\afrinic\3068.txt
c:\projects\hc2\whois\out\test\resources\asn\afrinic\37018.txt

Was just writing the same answer... :) I would move the extension check to the last for loop and get rid of the list. Also would use os methods to extract the extension - `os.path.splitext()[1]` — Tomerikoo, Jun 20 '19 at 12:19

score 0 · Answer 2 · answered Jun 20 '19 at 12:20

You could use glob.iglob() and os.walk() for that. Here a little function for you.

def list_of_files(path, extension, recursive=False):
    '''
    Return a list of filepaths for each file into path with the target extension.
    If recursive, it will loop over subfolders as well.
    '''
    if not recursive:
        for file_path in glob.iglob(path + '/*.' + extension):
            yield file_path
    else:
        for root, dirs, files in os.walk(path):
            for file_path in glob.iglob(root + '/*.' + extension):
                yield file_path

You need to import glob, os to use it.

In your case:

for file in list_of_files(path='master_folder_path_here', extension='txt'):
    ...

prudhvi raj · Answer 3 · 2019-06-20T15:46:11.310

0

You can use glob module from python

from glob import glob

file_list = glob("(folder path)/*/*")

this will give you list of all file paths in sub-sub folder.

and then you can iterate and do your operations.

edited Jun 20 '19 at 15:46

answered Jun 20 '19 at 12:23

prudhvi raj

46
3

1

either change to `from glob import glob` or to `glob.glob("(folder path)/*/*")` – Tomerikoo Jun 20 '19 at 12:31
@Tomerikoo forgot about that, thank you – prudhvi raj Jun 20 '19 at 15:46

How can I iterate over all files in all folders of one master folder?

3 Answers3