How to remove trailing whitespaces for multiple files?

Question

Are there any tools / UNIX single liners which would remove trailing whitespaces for multiple files in-place.

E.g. one that could be used in the conjunction with find.

possible duplicate of [How to remove trailing whitespace of all files recursively?](http://stackoverflow.com/questions/149057/how-to-remove-trailing-whitespace-of-all-files-recursively) — kenorb, Apr 20 '15 at 12:41
Possible duplicate of [How to remove trailing whitespaces with sed?](https://stackoverflow.com/q/4438306/608639) — jww, Sep 03 '18 at 09:13

Tim Pote · Accepted Answer · 2015-05-24T16:23:35.333

157

You want

sed --in-place 's/[[:space:]]\+$//' file

That will delete all POSIX standard defined whitespace characters, including vertical tab and form feed. Also, it will only do a replacement if the trailing whitespace actually exists, unlike the other answers that use the zero or more matcher (*).

--in-place is simply the long form of -i. I prefer to use the long form in scripts because it tends to be more illustrative of what the flag actually does.

It can be easily integrated with find like so:

find . -type f -name '*.txt' -exec sed --in-place 's/[[:space:]]\+$//' {} \+

If you're on a Mac

As pointed out in the comments, the above doesn't work if you don't have gnu tools installed. If that's the case, you can use the following:

find . -iname '*.txt' -type f -exec sed -i '' 's/[[:space:]]\{1,\}$//' {} \+

edited May 24 '15 at 16:23

answered May 22 '12 at 22:51

Tim Pote

27,191
6
63
65

2

By the way what's the thing with \+ as find exec terminator? – Mikko Ohtamaa May 22 '12 at 23:09
6

There are two variants of the `find -exec command`. The first ends with `;`. It runs `command` once for every file `find` returns. The second ends with `+`. It runs `command` as few times as possible by building up a list of files to run `command` on. Since the `;` variant *requires* a backslash to escape the `;`, I also generally put it on the `+` as well (though I don't think it's strictly necessary for the `+`). – Tim Pote May 23 '12 at 00:10
3

Talking about readability (and it's all a matter of taste) but I never use `-exec` with `find` because all that `{}+` stuff is like line noise. I prefer `find . -type f -name '*.txt' | xargs --replace=FILE sed --in-place 's/foo/baz/' FILE` but YMMV :) – seb Dec 12 '12 at 18:14
It looks like this also converts DOS-style line endings to Unix-style. – David Oliver Jun 18 '13 at 07:34
1

It looks like this also messes file permissions on windows (running from git bash); also the \+ variant doesn't work. – srcspider Jul 17 '13 at 07:55
1

On MacOS X, the stock `sed` does not support long options. I was able to get this recipe working by installing GNU sed with Homebrew (`brew install gnu-sed`). – amacleod Aug 13 '13 at 15:10
On MacOSX, the stock `sed` will work with the following tweaks `find . -type f -name '*.rb' -exec sed -i '' 's/[[:space:]]*$//' {} \+`. Note the `-i ''` and that we've replaced the `+` with `*`. – ZPH Nov 01 '14 at 03:24

jlliagre · Answer 2 · 2013-05-19T19:49:27.940

14

Unlike other solutions which all require GNU sed, this one should work on any Unix system implementing POSIX standard commands.

find . -type f -name "*.txt" -exec sh -c 'for i;do sed 's/[[:space:]]*$//' "$i">/tmp/.$$ && mv /tmp/.$$ "$i";done' arg0 {} +

Edit: this slightly modified version preserves the files permissions:

find . -type f -name "*.txt" -exec sh -c 'for i;do sed 's/[[:space:]]*$//' "$i">/tmp/.$$ && cat /tmp/.$$ > "$i";done' arg0 {} +

edited May 19 '13 at 19:49

answered May 23 '12 at 06:05

jlliagre

29,783
6
61
72

This works almost perfect. The only thing is that it changes files permissions to (the default?) 100644. – nacho4d May 19 '13 at 17:54
Works flawlessly on BSD! – Michael-O Sep 19 '13 at 13:02
Very helpful on HP-UX. – Bob Jarvis - Слава Україні Oct 17 '14 at 18:33

score 4 · Answer 3 · edited Oct 04 '17 at 09:43

4

I've been using this to fix whitespace:

while IFS= read -r -d '' -u 9
do
    if [[ "$(file -bs --mime-type -- "$REPLY")" = text/* ]]
    then
        sed -i -e 's/[ \t]\+\(\r\?\)$/\1/;$a\' -- "$REPLY"
    else
        echo "Skipping $REPLY" >&2
    fi
done 9< <(find . \( -type d -regex '^.*/\.\(git\|svn\|hg\)$' -prune -false \) -o -type f -print0)

Features:

Keeps carriage returns (unlike [:space:]), so it works fine on Windows/DOS-style files.
Only worries about "normal" whitespace - If you have vertical tabs or such in your files it's probably intentional (test code or raw data).
Skips the .git and .svn VCS directories.
Only modifies files which file thinks is a text file.
Reports all paths which were skipped.
Works with any filename.

edited Oct 04 '17 at 09:43

Russ

10,835
12
42
57

answered May 23 '12 at 12:35

l0b0

55,365
30
138
223

1

Just to be on the safe side: you probably want to ignore all . files for automatic processing like this (Eclipse .metadata, .bzr, so on) – Mikko Ohtamaa May 23 '12 at 13:56
1

I regularly use dotfiles which should be cleaned up - .bashrc, .gitignore, etc. There's no authority on which files you should always exclude, so it's up to you and the task at hand. – l0b0 May 23 '12 at 14:15
the sed keeps carriage returns but appeats to eat newlines at the end of files :( – CervEd Apr 29 '21 at 14:04
my bad, the sed adds a newline at EOF – CervEd Apr 29 '21 at 14:13
I'm finding that `sed -i -e 's/[ \t]\+$\r\?$$/\1/'` (same sed wo. adding newline at EOF) isn't preserving DOS-style endings. Using gnu sed 4.8. Example `seq 2 | unix2dos | sed -e 's/[ \t]\+$\r\?$$/\1/' | xxd -p` outputs `310a320a` should be `310d0a320d0a` – CervEd Apr 30 '21 at 17:09
with git for windows I had to add the `-b` option to `sed` to preserve `CLRF`. The regex preserved `CLRF` but `sed` didn't https://stackoverflow.com/a/11508669/1507124 – CervEd Apr 30 '21 at 18:20

Kylo · Answer 4 · 2012-05-23T00:31:08.763

3

How about this:

sed -e -i 's/[ \t]*$//'

Btw, this is a handy site: http://sed.sourceforge.net/sed1line.txt

edited May 23 '12 at 00:31

answered May 22 '12 at 22:33

Kylo

322
1
7

-e on my version of sed is for adding a script, but you aren't specifying a script. I'm using GNU sed 4.7. – Josiah Mar 03 '20 at 14:07

kenorb · Answer 5 · 2017-11-30T16:22:53.153

2

`ex`

Try using Ex editor (part of Vim):

$ ex +'bufdo!%s/\s\+$//e' -cxa *.*

^{Note: For recursion (bash4 & zsh), you can use a new globbing option (**/*.*). Enable by shopt -s globstar.}

`perl`

find . -type f -name "*.java" -exec perl -p -i -e "s/[ \t]$//g" {} \;

as per Spring Framework Code Style.

`sed`

For using sed, check: How to remove trailing whitespaces with sed?

See also: How to remove trailing whitespace of all files recursively?

edited Nov 30 '17 at 16:22

answered Apr 20 '15 at 12:29

kenorb

155,785
88
678
743

I think it is `"s/[ \t]+$//g"` in perl. – Labo Dec 29 '17 at 12:13

mendezcode · Answer 6 · 2014-07-26T03:12:39.003

1

For those that are not sed gurus (myself included) I have created a small script to use JavaScript regular expressions to replace text in files and does the replacement in place:

http://git.io/pofQnQ

To remove trailing whitespace you can use it as such:

$ node sed.js "/^[\t ]*$/gm" "" file

Enjoy

edited Jul 26 '14 at 03:12

answered Jul 26 '14 at 02:51

mendezcode

857
8
7

score 0 · Answer 7 · edited Nov 01 '22 at 16:06

0

For some reason, the sed and perl commands did not work for me. This did:

find ./ -type f | rename 's/ +$//g'

Feels like the most straight forward one to read as well.

edited Nov 01 '22 at 16:06

Nike

1,223
2
19
42

answered Sep 24 '16 at 07:43

Indivision Dev

1,097
11
16

2

Is this removing trailing spaces from filenames instead? – Michael Scott Asato Cuthbert Aug 25 '17 at 02:28

How to remove trailing whitespaces for multiple files?

7 Answers7

If you're on a Mac

`ex`

`perl`

`sed`

Linked