how do I read file from two folder with the same order in python

Question

I have two folders with the same file names, but when I try to read all text files from the folders in python, it reads in a different order. but I need to read files from two folders in the same order because they correspond. I used the following code to read all text files in a folder.

 dir_psnr=current_path+'\\'+dir_psnr+'\\'
os.chdir(dir_psnr) #change directory to downloads folder
files_path =[os.path.abspath(x) for x in os.listdir()]
fnames_psnr_tmp = [x for x in files_path if x.endswith(".txt")]

the address of the folders are as follows:

F:\RD_data_from_twitch_system\RD_data_from_twitch_system\psnr
F:\RD_data_from_twitch_system\RD_data_from_twitch_system\bitrate

the name of text files in both two folders are as follows:

asmr_1.txt
asmr_2.txt
Counter_strike_1.txt
Counter_strike_2.txt
dota2_1.txt

what is the problem? and how can I read files in the same order? the full code is :

def reading_file_to_array(dir_psnr,current_path):
    dir_psnr=current_path+'\\'+dir_psnr+'\\'
    os.chdir(dir_psnr) #change directory to downloads folder
    files_path =[os.path.abspath(x) for x in os.listdir()]
    fnames_psnr_tmp = [x for x in files_path if x.endswith(".txt")]
   .
   .
   .         

    return()
current_path='F:/RD_data_from_twitch_system/RD_data_from_twitch_system'
current_dir ='F:/RD_data_from_twitch_system/RD_data_from_twitch_system'
all_sub_dir_paths = glob(str(current_dir) + '/*/') 
all_sub_dir_names = [Path(sub_dir).name for sub_dir in all_sub_dir_paths] 
    for i in range(len(all_sub_dir_names)):
    if all_sub_dir_names[i]=='bitrate':
        bitrate_1080p,bitrate_720p,bitrate_480p,bitrate_360p,bitrate_160p=reading_file_to_array(all_sub_dir_names[i], current_path)

    else:
        psnr_1080p,psnr_720p,psnr_480p,psnr_360p,psnr_160p=reading_file_to_array(all_sub_dir_names[i], current_path)

If the file names are the same in both folders, why not just iterate the file name and append it to each of the folder paths to get the file to open in each folder? — Nick, Oct 16 '22 at 00:50
What is `files_path`? Is it the output of `os.listdir()`? If it is, note that [the docs](https://docs.python.org/3/library/os.html#os.listdir) say "The list is in arbitrary order". — wjandrea, Oct 16 '22 at 00:52
Please post the code so we can offer solutions. Its likely just a matter of rebasing the file names to the other folder. But we need the existing code to figure out how to put that in. — tdelaney, Oct 16 '22 at 00:55
@wjandrea, yes I used the os.listdir. what should I do to have the same order? — david, Oct 16 '22 at 00:59
@david - That's way too much! Its best to write a small example program. We don't care about the numpy stuff when the question is just about the files. — tdelaney, Oct 16 '22 at 01:01
Does this answer your question? [How to sort file names in a particular order using python](https://stackoverflow.com/questions/37796598/how-to-sort-file-names-in-a-particular-order-using-python) — Azhar Khan, Oct 16 '22 at 01:04
"what is the problem?" You first: **What happened** when you tried the code? **How is that different** from what you want? — Karl Knechtel, Oct 16 '22 at 02:07

score 2 · Accepted Answer · answered Oct 16 '22 at 01:07

Since the file names are the same, you could list the files in one directory and then add the bases to both for processing. This could be done in a generator that you can use in a loop. For example

folder1 = r"F:\RD_data_from_twitch_system\RD_data_from_twitch_system\psnr"
folder2 = r"F:\RD_data_from_twitch_system\RD_data_from_twitch_system\bitrate"

def list_directories(primary, secondary):
    primary = os.path.abspath(primary)
    secondary = os.path.abspath(secondary)
    for fn in os.listdir(primary):
        if fn.endswith(".txt"):
            yield (os.path.join(primary, fn), 
                os.path.join(secondary, fn))
                
# print files for test
for f1, f2 in list_directories(folder1, folder2):
    print(f1, f2)

Its usually a bad idea to os.chdir- especially without remembering which directory you came from. As long as your code builds absolute path names, the current working directory doesn't matter.

is there a way instead of using os.listdir. because my folders are more than two? — david, Oct 16 '22 at 01:18

score 0 · Answer 2 · answered Oct 16 '22 at 02:04

The easiest way would be to use listdir and to append the path to the front of every element of the list.

import os

#hardcoded folders
def reading_file_to_array(dir_1, dir_2):
  list_1 = [f"{dir_1}/"+f for f in os.listdir(dir_1)]
  list_2 = [f"{dir_2}/"+f for f in os.listdir(dir_2)]
  # Add more lists
  # Do sorting stuff here if needed
  return zip(list_1, list_2)

for f1, f2 in reading_file_to_array("./f_1", "./f_2"):
  print(f1, f2)

#more dynamic appraoch
def reading_file_to_array_dyn(dirs):
  results = list()
  for directory in dirs:
    results.append([f"{directory}/"+f for f in os.listdir(directory)])
    # Do sorting stuff here if needed

  return zip(*results)
  
for f1, f2 in reading_file_to_array_dyn(["./f_1", "./f_2"]):
  print(f1, f2)

The result of this test code looks like this for me:
./f_1/a.txt ./f_2/a.txt
./f_1/b.txt ./f_2/b.txt
./f_1/c.txt ./f_2/c.txt

If you want to filter the files in the folder based on type, I recommend the package glob.

how do I read file from two folder with the same order in python

2 Answers2