Splitting a string but keeping the delimiter

Question

I'm trying tokenize some code and would like to keep the delimiter when I split a string.

For example, I would like to keep any occurences of ., (, ), ;, ~.

I have been using re.split:

line = 'Keyboard.keyPressed();'
re.split(r'([\.\(\)\;\~])', line)

However, my current implementation of re.split currently creates some unnecessary empty strings in the array:

['Keyboard', '.', 'keyPressed', '(', '', ')', '', ';', '']

How can I fix this to exclude the empty strings?

It is "by design", when matches are consecutive, the empty strings are produced by `re.split`. Just remove empty items from the list. — Wiktor Stribiżew, Jul 15 '18 at 18:39
Well, in your exact case, you may just use `re.findall(r'[^.();~]+', line)` — Wiktor Stribiżew, Jul 15 '18 at 18:52
Ah, yeah, just include them, `re.findall(r'[^.();~]+|[.();~]', line)` — Wiktor Stribiżew, Jul 16 '18 at 05:34
Sup buddy. Do'nt forget to choose one poste as "the answer". ;) — Florian Fasmeyer, Oct 09 '20 at 07:50

score 1 · Answer 1 · answered Sep 29 '20 at 10:32

Keep your split simple for clarity and just remove empty strings.

import re
line = 'Keyboard.keyPressed();'
split = re.split(r'([\.\(\)\;\~])', line)

cleared = list(filter(None, split))   # <- Add this line

print(cleared)

<script type="text/javascript" src="//cdn.datacamp.com/dcl-react.js.gz"></script>

<div data-datacamp-exercise data-lang="python">
  <code data-type="sample-code">
import re
line = 'Keyboard.keyPressed();'
split = re.split(r'([\.\(\)\;\~])', line)
cleared = list(filter(None, split))
print(cleared)
  </code>
</div>

See: how to remove elements from a list

score 0 · Answer 2 · answered Jul 18 '18 at 17:10

You may use

re.findall(r'[^.();~]+|[.();~]', line)

See the regex demo

The re.findall will return a list of all matched non-overlapping substrings in the input string, matching the following:

[^.();~]+ - 1 or more (due to + quantifier) chars other than ., (, ), ; and ~
| - or
[.();~] - a single occurrence of ., (, ), ; or ~.

See Python demo online:

import re
line = 'Keyboard.keyPressed();'
print(re.findall(r'[^.();~]+|[.();~]', line))
# => ['Keyboard', '.', 'keyPressed', '(', ')', ';']

Splitting a string but keeping the delimiter

2 Answers2