read multiple files automatically no manual file naming

Question

I have a directory contains 50 files I want to read them one by one and compare wit the other files - that is fixed. I am using glob.blob. But it didn't work.

Here how I am reading all files. Instead, path = '*.rbd' if I give the file name like path = run-01.rbd it works.

path = '*.rbd'

path = folder + path
files=sorted(glob.glob(path))

complete code

import glob
from itertools import islice
import linecache

num_lines_nonbram =  1891427

bits_perline = 32

total_bit_flips =  0

num_bit_diff_flip_zero = 0
num_bit_diff_flip_ones = 0
folder = "files/"

path = '*.rbd'

path = folder + path
files=sorted(glob.glob(path))

original=open('files/mull-original-readback.rbd','r')

#source1 = open(file1, "r")

for filename in files:
 del_lines = 101

 with open(filename,'r') as f:
  i=1
  while i <= del_lines:
   line1 = f.readline()
   lineoriginal=original.readline()
   i+=1
  i=0  
  num_bit_diff_flip_zero = 0
  num_bit_diff_flip_ones = 0
  num_lines_diff =0

  i=0
  j=0
  k=0
  a_write2 = ""
  while i < (num_lines_nonbram-del_lines):
        line1 = f.readline() 
        lineoriginal = original.readline() 
        while k < bits_perline:
                if ((lineoriginal[k] == line1[k])):
                     a_write2 += " "
                else:
                     if (lineoriginal[k]=="0"): 
                     #if ((line1[k]=="0" and line1[k]=="1")):

                      num_bit_diff_flip_zero += 1
                     if (lineoriginal[k]=="1"): 
                     #if ((line1[k]=="0" and line1[k]=="1")):

                      num_bit_diff_flip_ones += 1

                     #if ((line1[k]==1 and line1[k]==0)):
                      #a_write_file2 = str(i+1) + " " + str(31-k) + "\n" + a_write_file2
                      #a_write2 += "^"
                      #num_bit_diff_flip_one += 1
                   # else:
                    #    a_write2 += " " 
                k+=1


                total_bit_flips=num_bit_diff_flip_zero+num_bit_diff_flip_ones
        i+=1

        k=0
i = 0
print files
print "Number of bits flip zero= %d" %num_bit_diff_flip_zero +"\n" +"Number of bits flip one= %d" %num_bit_diff_flip_ones +"\n" "Total bit flips = %d " %total_bit_flips


f.close()
original.close()

It gives me following error. that is so weird if I give one file it works no index error but when give all files it gives index error.. Traceback (most recent call last): File "random-ones-zeros.py", line 69, in if ((lineoriginal[k] == line1[k])): IndexError: string index out of range — hassan, Sep 14 '17 at 16:40
`Traceback (most recent call last): File "random-ones-zeros.py", line 69, in if ((lineoriginal[k] == line1[k])): IndexError: string index out of range` — hassan, Sep 14 '17 at 16:41
Either `lineoriginal` or `line1` is shorter than `bits_perline`. — Barmar, Sep 14 '17 at 16:42
You're not going back to the beginning of `original` each time you process a new file in the `for` loop. Is that intentional? — Barmar, Sep 14 '17 at 16:44
But why it works when give file name? That is confusing part so I thought may be the file didnt read properly by glob.glob — hassan, Sep 14 '17 at 16:44
No I need to go back to read original file again for each file. I tried as well to read the file original in the loop. But it also didnt work.... — hassan, Sep 14 '17 at 16:46
Why don't you just read the original file into an array just once before the loop? — Barmar, Sep 14 '17 at 16:47
Can you tell how to read the file into the array I can try this as well. — hassan, Sep 14 '17 at 16:48
https://stackoverflow.com/questions/3277503/how-do-i-read-a-file-line-by-line-into-a-list — Barmar, Sep 14 '17 at 16:51
Doest work do you have any idea if I can write a loop and give the file name of the file via loop. — hassan, Sep 14 '17 at 17:51

Omnomnious · Answer 1 · 2017-09-14T19:02:14.427

1

You could use the os module to first list everything in a directory (both files and modules) then use a python generator to filter out only the files. You could then use a second python generator to filter out files with a specific extension. There is probably a more efficient way of doing it but this works:

import os

def main():

    path = './' # The path to current directory

    # Go through all items in the directory and filter out files
    files = [file for file in os.listdir(path) if 
os.path.isfile(os.path.join(path, file))]

    # Go through all files and filter out files with .txt (for example)
    specificExtensionFiles = [file for file in files if ".txt" in file]

    # Now specificExtensionFiles is a generator for .txt files in current
    # directory which you can use in a for loop
    print (specificExtensionFiles)


if __name__ == '__main__':
    main()

For further reference: How do I list all files of a directory?

edited Sep 14 '17 at 19:02

answered Sep 14 '17 at 16:53

Omnomnious

313
1
6
19

Why is this better than using `glob.glob()` to just list the files he wants? – Barmar Sep 14 '17 at 20:09
It's not, I honestly just didn't know about glob.glob() and wanted to give him a quick solution that I know worked on my machine as opposed to an efficient one. – Omnomnious Sep 14 '17 at 20:41
His problem is clearly with the code in the loop, not getting the filenames. – Barmar Sep 14 '17 at 20:41
Didn't you see my comment about the fact that he doesn't rewind `originalfile` when he starts comparing with the next file in the loop? – Barmar Sep 14 '17 at 20:42
Nope, didn't see that. My bad and you are right. I assumed his issue was with what he claimed in the question so I addressed that without digging deeper. – Omnomnious Sep 14 '17 at 20:47
Should I delete the answer? – Omnomnious Sep 14 '17 at 20:48
Yes, I think you should. – Barmar Sep 14 '17 at 20:59

score 0 · Answer 2 · answered Sep 14 '17 at 20:59

The problem is that you're not going back to the beginning of originalfile whenever you start comparing with the next file in the for filename in files: loop. The simplest solution is to put:

original.seek(0)

at the beginning of that loop.

You could also read the whole file into a list just once before the loop, and use that instead of reading the file repeatedly.

And if you only want to process part of the files, you can read the file into a list, and then use a list slice to get the lines you want.

You also shouldn't be setting num_bit_diff_flip_zero and num_bit_diff_flip_one to 0 each time through the loop, since these are supposed to be the total across all files.

with open('files/mull-original-readback.rbd','r') as original:
    original_lines = list(original)[del_lines:num_lines_nonbram]

for filename in files:
    with open(file, 'r') as f:
        lines = list(f)[del_lines:num_lines_nonbram]
    for lineoriginal, line1 in zip(original_lines, lines):
        for k in range(bits_perline):
            if lineoriginal[k] == line1[k]:
                a_write2 += " "
            elif lineoriginal[k] == "0"
                num_bit_diff_flip_zero += 1
            else:
                num_bit_diff_flip_ones += 1

total_bit_flips = num_bit_diff_flip_zero + num_bit_diff_flip_ones

read multiple files automatically no manual file naming

2 Answers2