Split string by delimiter only if not wrapped in certain pattern

Question

I am trying to split a string into a list by a delimiter (let's say ,) but the delimiter character should be considered the delimiter only if it is not wrapped in a certain pattern, in my particular case <>. IOW, when a comma is nested in <>, it is ignored as a delimiter and becomes just a regular character not to be delimited by.

So if I have the following string:

"first token, <second token part 1, second token part 2>, third token"

it should split into

list[0] = "first token"
list[1] = "second token part 1, second token part 2"
list[2] = "third token"

Needless to say, I cannot just do a simple split by , because that will split the second token into two tokens, second token part 1 and second token part 2, as they have a comma in between them.

How should I define the pattern to do it using Python RegEx?

http://stackoverflow.com/questions/10974932/python-split-string-based-on-regular-expression — Farhadix, Nov 21 '13 at 18:06
@Farhadix, that doesn't tell me how to avoid commas wrapped in `<>` — amphibient, Nov 21 '13 at 18:06
Can those patterns be nested? Are they always correctly balanced? Can they be escaped? As in ` token with \> escaped bracket>`? — Tim Pietzcker, Nov 21 '13 at 18:08
when they are nested, they are ignored as delimiters. i.e. if a `,` is inside `<>`, it stops being a delimiter — amphibient, Nov 21 '13 at 18:09
@amphibient: Sorry, I meant the brackets ("patterns" in your question). — Tim Pietzcker, Nov 21 '13 at 18:10
Do you have any data that looks like this `"part1, >, part 3"`. In other words, can the brackets nest? — Steven Rumbalski, Nov 21 '13 at 18:12
@Steve, yes, but in your case there is no commas nested inside `<>` — amphibient, Nov 21 '13 at 18:12
@amphibient: Then you don't have a regular language anymore, and Python regexes don't support recursion. You'll have to write a parser then (look into [pyparsing](http://pyparsing.wikispaces.com/)) for that). — Tim Pietzcker, Nov 21 '13 at 18:15

Tim Pietzcker · Accepted Answer · 2013-11-21T18:40:10.930

10

Update: Since you mentioned that the brackets may be nested, I regret to inform you that a regex solution is not possible in Python. The following can work only if the angle brackets are always balanced and never nested nor escaped:

>>> import re
>>> s = "first token, <second token part 1, second token part 2>, third token"
>>> regex = re.compile(",(?![^<>]*>)")
>>> regex.split(s)
['first token', ' <second token part 1, second token part 2>', ' third token']
>>> [item.strip(" <>") for item in _]
['first token', 'second token part 1, second token part 2', 'third token']

The regex ,(?![^<>]*>) splits on commas only if the next angle bracket that follows isn't a closing angle bracket.

Nested brackets preclude this or any other regex solution from working in Python. You either need a language that supports recursive regexes (like Perl or .NET languages), or use a parser.

edited Nov 21 '13 at 18:40

answered Nov 21 '13 at 18:13

Tim Pietzcker

328,213
58
503
561

The regex matches the commas in 'first token `,` > `,` third token'. – Nov 21 '13 at 18:36
@sln: Yes, that's what I wrote. No regex solution can handle nested tags (if the nesting is arbitrary). Unfortunately, the information about nesting being possible only came after I had written my answer. – Tim Pietzcker Nov 21 '13 at 18:37
Yes, no Python regex can. As you say, others can and its really trivial. – Nov 21 '13 at 18:40
verified. passes functionality **AND** elegance :) – amphibient Nov 21 '13 at 18:47

score 6 · Answer 2 · answered Nov 21 '13 at 18:12

6

One kludgy way that works for your example is to translate the <>'s into "'s and then treat it as a CSV file:

import csv
import string

s = "first token, <second token part 1, second token part 2>, third token"    
a = s.translate(string.maketrans('<>', '""'))
# first token, "second token part 1, second token part 2", third token
print next(csv.reader([a], skipinitialspace=True))
['first token', 'second token part 1, second token part 2', 'third token']

answered Nov 21 '13 at 18:12

Jon Clements

138,671
33
247
280

1

You may want to mention (or deal with) that this precludes quotes being used elsewhere in the data. – Steven Rumbalski Nov 21 '13 at 18:14
This will not work with nested brackets. Comments above indicate they may be nested. – Steven Rumbalski Nov 21 '13 at 18:26

Split string by delimiter only if not wrapped in certain pattern

2 Answers2