How to manage large size file?

Question

Disk file size of 600+ MB continuously appended by multiple agents.

Below code open and reads this large size file,

def alert_entries():
    with open('large_size_file') as f:
         ....
          ....
        for entry in entry_re.findall(f.read()):
            yield entry

Machine goes slow after opening this file.

How to manage opening and reading large size file, memory efficiently?

perhaps a better strategy could be to "tail" the file, read it as it grows. — LMC, Jul 19 '18 at 23:59
@LuisMuñoz Does memory mapping a file? use the memory efficiently.. — overexchange, Jul 20 '18 at 00:00
A file rotation strategy with active file on ram disk/image? — LMC, Jul 20 '18 at 00:03
@LuisMuñoz Can u provide some python reference on using this strategy? — overexchange, Jul 20 '18 at 00:13
Unfortunately have no experience with mmap mysef, was just thinking out loud sorry. — LMC, Jul 20 '18 at 00:15
Does your regex require the entire string? Or can it be applied line by line? — juanpa.arrivillaga, Jul 20 '18 at 00:35
@juanpa.arrivillaga regex can be aplied line by line. For performance reasons I said re.MULTILINE — overexchange, Jul 20 '18 at 00:43
Well then iterate line by line. It will sacrifice speed for performance. Alternatively, read large chunks and apply the regex on those chunks. — juanpa.arrivillaga, Jul 20 '18 at 00:47

Solaxun · Answer 1 · 2018-07-20T00:36:40.687

2

Calling f.read() will read the whole thing in memory. You can either iterate over the file directly for line in f (as mentioned in the comment below) or pass an argument to read e.g. read(size) to limit how much you process in one go.

Refer to this post for an example using the latter approach.

edited Jul 20 '18 at 00:36

answered Jul 20 '18 at 00:16

Solaxun

2,732
1
22
41

How to manage large size file?

1 Answers1