Pythonic way to iterate through a file on something other than newlines

Question

To iterate a file by lines, one can do -

for line in f:

(where f is the file iterator).

I want to iterate the file by blocks delimited by commas, instead of blocks delimited by newlines. I can read all lines and then split the string on commas, but whats the pythonic way to do this?

score 3 · Accepted Answer · answered Feb 24 '15 at 15:33

3

Iterate over the split as you go then you don't need to store all the lines:

for line in f: 
    for lines in line.split(","):

answered Feb 24 '15 at 15:33

Padraic Cunningham

176,452
29
245
321

4

Think about the case where a newline character is somehwere in between two commas. – thefourtheye Feb 24 '15 at 15:36
1

@Illusionist, no worries. This will save you storing all lines in and the split lines in memory which for a large file could be a problem. – Padraic Cunningham Feb 24 '15 at 15:36
@thefourtheye, that is an edge case where line.replace or something similar would work or just stripping. – Padraic Cunningham Feb 24 '15 at 15:40
@PadraicCunningham: Can you look my question ? I can not get .. link: http://stackoverflow.com/questions/28697319/can-not-able-to-set-request-header-during-ajax-call-in-sslhttps – Vivek Sable Feb 24 '15 at 15:56
1

@VivekSable, I am afraid I know next to nothing about django! – Padraic Cunningham Feb 24 '15 at 15:58
1

@PadraicCunningham: Ok, No problem. If you found something similar then just added in question. – Vivek Sable Feb 24 '15 at 16:03

Vivek Sable · Answer 2 · 2015-02-24T16:02:21.393

2

Use string split() method to split content by comma.

e.g.

input_file = "/home/vivek/Desktop/Work/stack_over/href_input.html"
#- Read File content.
with open(input_file, "rb") as fp:
    content_list = fp.read().split(",")

Iterate file line by line and iterate every line by spiting comma

>>> with open(input_file, "rb") as fp:
...    for f in fp:
...       for i in f.split(","):
...            i

edited Feb 24 '15 at 16:02

answered Feb 24 '15 at 15:34

Vivek Sable

9,938
3
40
56

1

This kills the whole point of iterating over the file object. Think about a file which is twice or thrice as big as your system's memory. – thefourtheye Feb 24 '15 at 15:35
@thefourtheye Isn't that like the 0.1% case, though? How many times do you think this is going to fail by running out of RAM really? – Two-Bit Alchemist Feb 24 '15 at 15:37
@thefourtheye: yes, but we read file line by line then it will not comma separated . – Vivek Sable Feb 24 '15 at 15:39
@Two-BitAlchemist Okay, why do we prefer iterating over the file object rather than `fp.read().split("\n")`? – thefourtheye Feb 24 '15 at 15:39
@thefourtheye Thought experiment: should we also avoid using `split` here since it returns a list and there might be a file with only one line (no `\n`) that will run the system out of memory? – Two-Bit Alchemist Feb 24 '15 at 15:44
@thefourtheye - any idea *how* python implements readline/readlines? I'm trying to track down how it happens to see if a user could monkeypatch it to split on something other than `\n` but I'm having trouble finding the actual Python source code that does it. – dwanderson Mar 25 '15 at 01:52
@Two-BitAlchemist - that's not a particularly absurd idea - if you read in a (large) file that didn't have any newlines, perhaps because it was processed by a different architecture (eg whichever one only uses `\r` and your architecture includes `\n`). If the OP is asking if there's a way to split on something other than newlines, it's not particularly helpful/relevant to say "just read in the whole file" since that defeats the purpose of the question. – dwanderson Mar 25 '15 at 01:55
@thefourtheye - some follow up: perhaps something like http://stackoverflow.com/a/102202/2272638 would be helpful? Potentially slower, but if you're afraid of memory constraints, that might be acceptable/necessary – dwanderson Mar 25 '15 at 02:02

score 1 · Answer 3 · answered Mar 25 '15 at 02:27

If you really need to scan one giant (e.g. 1TB) single-line file and process items by delimiters, you can read file by blocks, split them, and handle border-effects. Here a generator which may help with it:

def split_file(file, delim, block_size=1024*1024):
    block = True
    last_item = ''
    while block:
        block = file.read(block_size)
        items = block.split(delim)
        for i in xrange(len(items)-1):
            item = items[i]
            if last_item:
                yield last_item + item
                last_item = ''
                continue
            if item:
                yield item
        last_item += items[-1]

You can simply use it like this:

f = open("names.in.txt")
for name in split_file(f, ","):
    print name # process one item there

Pythonic way to iterate through a file on something other than newlines

3 Answers3