0

I am trying to read and process each line from a large file and then write the result into another file. But the memory is used out quickly. Any other good solutions? Please help.

time_start = time.time()
tt_X = []
D_tmp = []
with open(filenameRead, 'r') as fr, open(fileNameWrite,'wa') as fw:
    for line in fr:
        _, _, D_tmp = line.strip().split(',') #      
        #process each line
        D_tmp = np.array( D_tmp.split() ).astype(int).reshape(15,4,101,101)/215.   
        #D_tmp = np.reshape(D_tmp, () )/215.
        for i in range(15):
            tt_X.append(D_tmp[i,1:4,:-1,:-1])
        encoded = encoder.predict(np.array(tt_X)).flatten()
        fw.writelines(' '.join(encoded.astype(str)) +'\n')
        tt_X = []
        D_tmp = []
time_end = time.time()
print 'The elapsed time is ', time_end - time_start
Weiwei Shi
  • 19
  • 3
  • I will try this one Wei – codyc4321 May 24 '17 at 02:14
  • Try this solution first, it's a context manager statement https://stackoverflow.com/questions/8009882/how-to-read-large-file-line-by-line-in-python – codyc4321 May 24 '17 at 02:16
  • 1
    It seems you did that. I would recommend writing 500 or so lines to a temp file and reading those before moving on, and repeating, if the `with open...` didnt work – codyc4321 May 24 '17 at 02:22
  • Welcome to Stack Overflow! You can learn [How to Ask a good question](http://stackoverflow.com/help/how-to-ask) and create a [Minimal, Complete, and Verifiable](http://stackoverflow.com/help/mcve) example. That makes it easier for us to help you. There does not appear to be anything fundamentally wrong with this code that would consume memory. But without an MCVE there is really no way to say. – Stephen Rauch May 24 '17 at 02:39
  • You could try and run the garbage collector manually at the end of each step in the loop, or explicitly delete the current line and other variables. At what line does the MemoryError occur? –  May 24 '17 at 02:46
  • I'm guessing you're having a memory leak somewhere else, possibly in the `encoder.predict(...)` (Keras?) call - the data you send there seems to stay there as everything else here seems accounted for. – zwer May 24 '17 at 02:55

0 Answers0