How to replace specific punctuation with new name?

Question

My data sample is:

        comment sarc_majority
0        [?, ?]          sarc
1           [0]      non-sarc
2     [!, !, !]          sarc
3           [0]      non-sarc
4           [?]          sarc

I want to replace the punctuation with a new name. Such as ? = punct1, ! = punct2, ' = punct3. I tried using read from csv file.

replace_df = pd.read_csv('./final/eng-mly-punct.csv', sep=',', quoting=csv.QUOTE_NONE,
                       names=["punct", "replacer"])
replace_df.head()

    punct   replacer
0   ?       punct1
1   !       punct2
2   '       punct3

Then I stucked at replacing:

for punct, replacer in replace_df.itertuples(index=False,name=None):
    df.comment = df.comment.str.replace(r'\b{0}\b'.format(punct),replacer)

The error is: error: nothing to repeat

What have gone wrong? Or is there any possible way to do this? The desired output should be just like:

                       comment sarc_majority
0             [punct1, punct1]          sarc
1                          [0]      non-sarc
2     [punct2, punct2, punct2]          sarc
3                          [0]      non-sarc
4                     [punct1]          sarc

Thanks in advance. Cheers.

Thanks @RohanAmrute for your respond. That's what I'm thinking at first. But lets try the str.replace. — Suhairi Suhaimin, Nov 16 '16 at 10:36

jezrael · Accepted Answer · 2016-11-16T11:17:57.553

1

You can use replace by dict d - but need escape ? to \?:

d = {'\?':'punct1','!':'punct2',"'":'punct3'}
df.comment = df.comment.replace(d, regex=True)
print (df)
                    comment sarc_majority
0          [punct1, punct1]          sarc
1                       [0]      non-sarc
2  [punct2, punct2, punct2]          sarc
3                       [0]      non-sarc
4                  [punct1]          sarc

Also you can create d from replace_df:

df = pd.DataFrame({'comment': {0: '[?, ?]', 1: '[0]', 2: '[!, !, !]', 3: '[0]', 4: '[?]'}, 'sarc_majority': {0: 'sarc', 1: 'non-sarc', 2: 'sarc', 3: 'non-sarc', 4: 'sarc'}})
print (df)
     comment sarc_majority
0     [?, ?]          sarc
1        [0]      non-sarc
2  [!, !, !]          sarc
3        [0]      non-sarc
4        [?]          sarc

replace_df = pd.DataFrame({'replacer': {0: 'punct1', 1: 'punct2', 2: 'punct3'}, 'punct': {0: '?', 1: '!', 2: "'"}})
print (replace_df)
  punct replacer
0     ?   punct1
1     !   punct2
2     '   punct3

replace_df.punct = '\\' + replace_df.punct
d = replace_df.set_index('punct')['replacer'].to_dict()
print (d)
{'\\!': 'punct2', "\\'": 'punct3', '\\?': 'punct1'}

df.comment = df.comment.replace(d, regex=True)
print (df)
                    comment sarc_majority
0          [punct1, punct1]          sarc
1                       [0]      non-sarc
2  [punct2, punct2, punct2]          sarc
3                       [0]      non-sarc
4                  [punct1]          sarc

EDIT by comment:

df = pd.DataFrame({'comment':[['?', '?'],[0], ['!', '!', '!'], [0], ['?']], 'sarc_majority': [ 'sarc','non-sarc', 'sarc', 'non-sarc','sarc']})
print (df)
     comment sarc_majority
0     [?, ?]          sarc
1        [0]      non-sarc
2  [!, !, !]          sarc
3        [0]      non-sarc
4        [?]          sarc

print (type(df.ix[0,'comment']))
<class 'list'>

replace_df = pd.DataFrame({'replacer': {0: 'punct1', 1: 'punct2', 2: 'punct3'}, 'punct': {0: '?', 1: '!', 2: "'"}})
#print (replace_df)

replace_df.punct = '\\' + replace_df.punct.apply(lambda x: x.format())
d = replace_df.set_index('punct')['replacer'].to_dict()
print (d)
{'\\!': 'punct2', "\\'": 'punct3', '\\?': 'punct1'}

df.comment = df.comment.apply(lambda x: pd.Series(x).astype(str).replace(d, regex=True).tolist())
print (df)
                    comment sarc_majority
0          [punct1, punct1]          sarc
1                       [0]      non-sarc
2  [punct2, punct2, punct2]          sarc
3                       [0]      non-sarc
4                  [punct1]          sarc

edited Nov 16 '16 at 11:17

answered Nov 16 '16 at 08:47

jezrael

822,522
95
1,334
1,252

what is your pandas version? – jezrael Nov 16 '16 at 10:13
Or maybe you have list in comment column, what return `print (type(df.ix[0,'comment']))` ? – jezrael Nov 16 '16 at 10:15
Thank you @jezrael for your help. I've tried but it still return the same punctuation symbols, none has been replaced :-( I read the post about re.sub for regex [here] (http://stackoverflow.com/questions/16720541/python-string-replace-regular-expression). Is this related to me now? – Suhairi Suhaimin Nov 16 '16 at 10:17
If use pandas, it use another `replace` function. So I think not. – jezrael Nov 16 '16 at 10:19
What return `print (type(df.ix[0,'comment']))` ? – jezrael Nov 16 '16 at 10:19
My pandas: '0.18.0' – Suhairi Suhaimin Nov 16 '16 at 10:20
I add code for creating `DataFrames`. It works for you? – jezrael Nov 16 '16 at 10:21
"" – Suhairi Suhaimin Nov 16 '16 at 10:23
and you need list or string? – jezrael Nov 16 '16 at 10:26
as desired output from my question above. – Suhairi Suhaimin Nov 16 '16 at 10:28
Please check my solution, sorry I was in dinner – jezrael Nov 16 '16 at 11:18
Yesss It works. I'm so happy. Need to study this. Anyway thanks a lot @jezrael. Have a nice day. – Suhairi Suhaimin Nov 16 '16 at 11:26
Thank you for accepting. First solution doesnt work because if use `[]` in columns it can be `list` or `string` seems as list. ; ) – jezrael Nov 16 '16 at 11:28

spectras · Answer 2 · 2016-11-16T10:37:35.460

1

Most punctuation characters have a special meaning in regular expressions. Here you end up with, eg: \b?\b, which means an optional boundary followed by a boundary. Not what you meant.

For passing arbitrary strings into a regexp, it must be escaped using re.escape:

import re
r'\b{0}\b'.format(re.escape(punct))

This will be \b\?\b, which means a boundary, followed by a ?, followed by another boundary.

edited Nov 16 '16 at 10:37

answered Nov 16 '16 at 08:47

spectras

13,105
2
31
53

1

`\b` is not a blank, it's a boundary between word characters and non-word characters. – cco Nov 16 '16 at 10:02
Thanks @spectras. I've tried and return AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas – Suhairi Suhaimin Nov 16 '16 at 10:38
@SuhairiSuhaimin> I don't know pandas, I would believe that means one of your comments or one of your punctuation replacement is either empty or not a string in your file. As I don't know pandas, I just focused on the "how to build a valid regexp from a string that has punctuation in it" part, which seemed to be the core of your problem. – spectras Nov 16 '16 at 10:49
Thank you @spectras for your explanation. The file is not empty. replace_df.head() show the punct and what I try to replace with. Maybe I should try re.sub later. – Suhairi Suhaimin Nov 16 '16 at 11:03

How to replace specific punctuation with new name?

2 Answers2