Python regex - why does end of string ($ and \Z) not work with group expressions?

Question

In Python 2.6. it seems that markers of the end of string $ and \Z are not compatible with group expressions. Fo example

import re
re.findall("\w+[\s$]", "green pears")

returns

['green ']

(so $ effectively does not work). And using

re.findall("\w+[\s\Z]", "green pears")

results in an error:

/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/re.pyc in findall(pattern, string, flags)
    175 
    176     Empty matches are included in the result."""
--> 177     return _compile(pattern, flags).findall(string)
    178 
    179 if sys.hexversion >= 0x02020000:

/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/re.pyc in _compile(*key)
    243         p = sre_compile.compile(pattern, flags)
    244     except error, v:
--> 245         raise error, v # invalid expression
    246     if len(_cache) >= _MAXCACHE:
    247         _cache.clear()

error: internal: unsupported set operator

Why does it work that way and how to go around?

`so $ effectively does not work` -- What is your expected output?? — Rohit Jain, Oct 06 '12 at 20:28
@RohitJain `['green ', 'pears']` (as from `'\w+\s'` plus `'\w+$'`). — Piotr Migdal, Oct 06 '12 at 20:31

Martijn Pieters · Accepted Answer · 2012-10-07T07:48:46.973

37

A [..] expression is a character group, meaning it'll match any one character contained therein. You are thus matching a literal $ character. A character group always applies to one input character, and thus can never contain an anchor.

If you wanted to match either a whitespace character or the end of the string, use a non-capturing group instead, combined with the | or selector:

r"\w+(?:\s|$)"

Alternatively, look at the \b word boundary anchor. It'll match anywhere a \w group start or ends (so it anchors to points in the text where a \w character is preceded or followed by a \W character, or is at the start or end of the string).

edited Oct 07 '12 at 07:48

answered Oct 06 '12 at 20:27

Martijn Pieters

1,048,767
296
4,058
3,343

@EMS: That's because the `:` in my expression was a typo; corrected. – Martijn Pieters Oct 06 '12 at 20:47
...because of my typo, sorry for that. – Piotr Migdal Oct 06 '12 at 20:51
I think you mean a **non-capturing** group. This stuff is confusing enough as it is; let's at least try and use the terminology correctly. – Alan Moore Oct 06 '12 at 23:59
1

@AlanMoore: Yes, indeed I did mean non-capturing. Corrected, thanks! – Martijn Pieters Oct 07 '12 at 07:49

score 4 · Answer 2 · answered Oct 06 '12 at 20:30

Square brackets don't indicate a group, they indicate a character set, which matches one character (any one of those in the brackets) As documented, "special characters lose their special meaning inside sets" (except where indicated otherwise as with classes like \s).

If you want to match \s or end of string, use something like \s|$.

score 2 · Answer 3 · edited May 23 '17 at 11:47

Martijn Pieters' answer is correct. To elaborate a bit, if you use capturing groups

r"\w+(\s|$)"

you get:

>>> re.findall("\w+(\s|$)", "green pears")
[' ', '']

That's because re.findall() returns the captured group (\s|$) values.

Parentheses () are used for two purposes: character groups and captured groups. To disable captured groups but still act as character groups, use (?:...) syntax:

>>> re.findall("\w+(?:\s|$)", "green pears")
['green ', 'pears']

Python regex - why does end of string ($ and \Z) not work with group expressions?

3 Answers3

Linked

Related