Removing non-alphanumeric characters with sed

Question

I am trying to validate some inputs to remove a set of characters. Only alphanumeric characters plus, period, underscore, hyphen are allowed. I've tested the regex expression [^\w.-] here http://gskinner.com/RegExr/ and it matches what I want removed so I not sure why sed is returning the opposite. What am I missing?

My end goal is to input "Â10.41.89.50 " and get "10.41.89.50".

I've tried:

echo "Â10.41.89.50 " | sed s/[^\w.-]//g returns Â...

echo "Â10.41.89.50 " | sed s/[\w.-]//g and echo "Â10.41.89.50 " | sed s/[\w^.-]//g returns Â10418950

I attempted the answer found here Skip/remove non-ascii character with sed but nothing was removed.

Try adding the `-r` option to `sed` so it will recognize extended regular expressions. — Barmar, Nov 15 '13 at 17:44
`sed` doesn't understand the special character classes like `\w`. Just use `[a-zA-Z0-9_-]`. — Mark Reed, Nov 15 '13 at 17:50
neither `-r` nor using `[a-zA-Z0-9_-]` works. Well `echo "Â10.41.89.50 " | sed s/[a-zA-Z0-9.-]//g` returned `Â` but `echo "Â10.41.89.50 " | sed s/[^a-zA-Z0-9.-]//g` still returned `Â10.41.89.50`. — wanderingandy, Nov 15 '13 at 18:06

iruvar · Accepted Answer · 2019-03-04T18:38:47.173

83

tr's -c (complement) flag may be an option

echo "Â10.41.89.50-._ " | tr -cd '[:alnum:]._-'

edited Mar 04 '19 at 18:38

answered Nov 15 '13 at 17:57

iruvar

22,736
7
53
82

@AlexanderMills, @Herlon, While the `tr` incantation above is POSIX-compliant, I do not have MacOS handy to test – iruvar May 21 '18 at 17:14
1

This answer works fine on macOS, it's just that the locale includes non-English letters in the `:alnum:` character class (as it should). If you want to remove non-English characters, try this: `echo "Â10.41.89.50-._ /" | tr -cd '[a-zA-Z0-9]._-'` – tjmcewan Oct 29 '18 at 22:14
@iruvar, I think you need to drop those extra brackets (the class is '[:alnum:]' not '[[:alnum:]]' and tr isn't sed/perl etc.) as otherwise your expression will allow those non-alphanumerics ('[', ']') through. I've just hit this – elbeardmorez Mar 04 '19 at 07:48
1

using `LANG=C tr -cd '...'` might be a good idea – Fravadona Sep 29 '22 at 18:26

gniourf_gniourf · Answer 2 · 2013-11-15T18:09:55.077

25

You might want to use the [:alpha:] class instead:

echo "Â10.41.89.50 " | sed "s/[[:alpha:].-]//g"

should work. If not, you might need to change your local settings.

On the other hand, if you only want to keep the digits, the hyphens and the period::

echo "Â10.41.89.50 " | sed "s/[^[:digit:].-]//g"

If your string is in a variable, you can use pure bash and parameter expansions for that:

$ dirty="Â10.41.89.50 "
$ clean=${dirty//[^[:digit:].-]/}
$ echo "$clean"
10.41.89.50

or

$ dirty="Â10.41.89.50 "
$ clean=${dirty//[[:alpha:]]/}
$ echo "$clean"
10.41.89.50

You can also have a look at 1_CR's answer.

edited Nov 15 '13 at 18:09

answered Nov 15 '13 at 17:50

gniourf_gniourf

44,650
9
93
104

1

@dw1: No, I don't think so. In the first example, we want to *remove all letters, periods and hyphens,* and that's what the command does (`sed` replaces these symbols by nothing). Last example is the same logic, but with Bash's parameter expansion. – gniourf_gniourf Jan 05 '21 at 12:23

score 13 · Answer 3 · answered Aug 07 '19 at 11:17

13

To remove all characters except of alphanumeric and "-" use this code:

echo "a b-1_2" | sed "s/[^[:alnum:]-]//g"

answered Aug 07 '19 at 11:17

panticz

2,135
25
16

score 7 · Answer 4 · answered Nov 15 '13 at 17:45

7

Well sed won't support unicode characters. Use perl instead:

> s="Â10.41.89.50 "
> perl -pe 's/[^\w.-]+//g' <<< "$s"
10.41.89.50

answered Nov 15 '13 at 17:45

anubhava

761,203
64
569
643

@jthill: It didn't on my OSX (could be due to some locales). – anubhava Nov 15 '13 at 18:00
So looks like sed on my OSX 10.6.8 doesn't have support for unicode but gnu-sed should have unicode support. – anubhava Nov 15 '13 at 18:10
1

@gorideyourbike: Fair enough, feel free to use any of the other answers. – anubhava Nov 15 '13 at 18:16
@anubhava shouldn't be locale, perl honors that. I suppose Apple's got a backlevel sed, what's `sed --version` say? – jthill Nov 15 '13 at 21:19
@jthill: `sed --version` gives error: `sed: illegal option -- -` – anubhava Nov 15 '13 at 21:21

Iwan Plays · Answer 5 · 2021-07-03T13:59:50.917

2

s/[^[:alnum:]+._-]//g

removes anything other than alphanumeric and ".+_-" characters.

echo "Â10.41.89.50 +-_" | sed s/[^[:alnum:]+._-]//g
Â10.41.89.50+-_

edited Jul 03 '21 at 13:59

answered Apr 01 '18 at 15:49

Iwan Plays

29
3

2

Please provide some explanation – user3142695 Apr 01 '18 at 16:07
@user3142695 This isn't a replacement for the manual. This answers the Q. man sed from there. No tricks. No hocus pocus. There's no rocket science there. – RichieHH Mar 18 '21 at 08:39
nope, this will replace the first character of the input with nothing - ^ is the beginning of the string in sed – MaurGi Jul 01 '21 at 19:31

score 0 · Answer 6 · edited Sep 25 '15 at 16:39

0

<`[[:alnum:]_.@]`

This worked just fine for me. It preserved all of the characters I specified for my purposes.

edited Sep 25 '15 at 16:39

spenibus

4,339
11
26
35

answered Sep 25 '15 at 16:17

technerdius

253
3
6
16

Removing non-alphanumeric characters with sed

6 Answers6

Linked