How to drop records with special characters in a dataframe

Question

I am trying to drop records/ Rows in a dataframe with special characters. I have tried many things like:

df1['Day'] = df1['Day'].str.replace(r"[\"\'\|\?\=\.\@\#\*\,]", '')
df.drop(df[df.ID.str.contains(r'[^0-9a-zA-Z]')].index

But i am getting the following recurssion error.

Also, is there any way to impute the records with special characters with mean/median/mode

---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/IPython/core/formatters.py in __call__(self, obj)
    697                 type_pprinters=self.type_printers,
    698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
    700             printer.flush()
    701             return stream.getvalue()

18 frames
... last 15 frames repeated, from the frame below ...

/usr/local/lib/python3.7/dist-packages/pandas/core/frame.py in __repr__(self)
   1000             line_width=width,
   1001             max_colwidth=max_colwidth,
-> 1002             show_dimensions=show_dimensions,
   1003         )
   1004 

RecursionError: maximum recursion depth exceeded
---------------------------------------------------------------------------
RecursionError                            Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/IPython/core/formatters.py in __call__(self, obj)
    332                 pass
    333             else:
--> 334                 return printer(obj)
    335             # Finally look for special method names
    336             method = get_real_method(obj, self.print_method)

26 frames
... last 15 frames repeated, from the frame below ...

/usr/local/lib/python3.7/dist-packages/pandas/io/formats/format.py in format_col(self, i)
    823             space=self.col_space.get(frame.columns[i]),
    824             decimal=self.decimal,
--> 825             leading_space=self.index,
    826         )
    827 

RecursionError: maximum recursion depth exceeded while calling a Python object

Example Data: Date Month oct-7 MaRch 7 Oct In this case, since the first row has a special character '-', it should be dropped and my output should only have the row: 7 Oct — Vasudha Pasumarthi, Feb 19 '22 at 20:23
Please *update the question* rather than providing question details in comments. — S3DEV, Feb 19 '22 at 20:33
Could you please [update](https://stackoverflow.com/posts/71188780/edit) your question with a self-contained example that reproduces your problem (see [How to create a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example) and [How to make good reproducible pandas examples](https://stackoverflow.com/q/20109391/14311263))? — Timus, Feb 19 '22 at 21:57

score 0 · Answer 1 · answered Feb 19 '22 at 23:42

0

To keep rows that doesn't contain characters other than those you specified

df[~df.ID.str.contains(r'[^0-9a-zA-Z]')]

answered Feb 19 '22 at 23:42

Raymond Kwok

2,461
2
9
11

This appears to double negate. Perhaps simpler to remove the regex `^` and the `~` and *keep* rows which match the regex. – S3DEV Feb 20 '22 at 08:13
not quite, remove them won't filter away 'a@' which the current version can. :) – Raymond Kwok Feb 20 '22 at 08:17

How to drop records with special characters in a dataframe

1 Answers1