How to remove trailing `\r` in shell?

Question

I have a file that looks like this:

1   0.1951  0.1766  0.1943  0.1488  
0.1594  0.2486  0.2044  0.2013  0.1859  
0.1559  0.1761  0.1666  0.1737  0.1595  
0.1940  1   0.2398  0.1894  0.1532  
0.1749  0.2397  1   0.1654  0.1622  
0.1940  0.1895  0.1659  1   0.1384  
0.1489  0.1547  0.1648  0.1390  1   
0.1840  0.2472  0.2256  0.2281  0.1878

Somehow it's created in windows so it has this irritating \r character at the end. But I am running my shell line in linux.

In python I could have read the file and do a line.strip('\r') while looping through the lines in the file. But i have to use shell to run a loop and somehow the '\r' keep appearing.

Is there any way to remove it while in the while loop? I am trying to do a loop for this script in shell: https://github.com/alvations/meanie/blob/master/amgm.py:

# -*- coding: utf-8 -*-

import math, operator

def arithmetic_mean(x): # x is a list of values.
  """ Returns the arithmetic mean given a list of values. """
  return sum(x)/len(x)

def geometric_mean(x): # x is a list of values
  """ Returns the geometric mean given a list of values. """
  return math.pow(reduce(operator.mul, x, 1), 1/float(len(x)))

def arigeo_mean(x, threshold = 1e-10): # x is a list of values
  arith = arithmetic_mean(x)
  geo = geometric_mean(x)
  while math.fabs(arith - geo) > threshold: 
    [arith,geo] = [(arith + geo) / 2.0, math.sqrt(arith * geo)]
  return arith

def main(means):
  print means
  means = map(float,means)
  print "arithmetic mean = ", arithmetic_mean(means)
  print "geometric mean = ", geometric_mean(means)
  print "arithmetic-geometric mean = ", arigeo_mean(means)

if __name__ == '__main__':
  import sys
  if len(sys.argv) < 2:
    sys.stderr.write('Usage: python %s mean1 mean2 mean3 ... \n' % sys.argv[0])
    sys.exit(1)

so I tried the following shell line to iterate through my textfile:

alvas@ubi:~/git/meanie$ while read line; do python amgm.py $line; done < out.tab

and got these errors:

['1', '0.1951', '0.1766', '0.1943', '0.1488', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float: 
['0.1594', '0.2486', '0.2044', '0.2013', '0.1859', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float: 
['0.1559', '0.1761', '0.1666', '0.1737', '0.1595', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float: 
['0.1940', '1', '0.2398', '0.1894', '0.1532', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float: 
['0.1749', '0.2397', '1', '0.1654', '0.1622', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float: 
['0.1940', '0.1895', '0.1659', '1', '0.1384', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float: 
['0.1489', '0.1547', '0.1648', '0.1390', '1', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float: 
['0.1840', '0.2472', '0.2256', '0.2281', '0.1878', '\r']
Traceback (most recent call last):
  File "amgm.py", line 32, in <module>
    main(sys.argv[1:])
  File "amgm.py", line 22, in main
    means = map(float,means)
ValueError: could not convert string to float:

Note: I'm NOT allowed to change the python script though =(

Consider dos2unix: http://stackoverflow.com/a/19702943/758446 — BlackVegetable, Jan 18 '14 at 20:43
why not write python script `with open(..) as ..: for line in .. map (..)` instead of using bash ? — akaRem, Jan 18 '14 at 20:45
@akarem, users are hard to please, you give them a script that automatically calculate AMGM, they require something to do it multiple times. Altering my script will please 1 user and displease another 10, so it's simpler to give that 1 user a shell line to run. — alvas, Jan 18 '14 at 20:48
This is not a Python question, this is a shell question. If you can't change it, why show it? — Russia Must Remove Putin, Jan 18 '14 at 20:49

score 1 · Accepted Answer · edited May 23 '17 at 11:56

You should be able to strip any trailing whitespace from the line before passing it to your python program.

Consider one of the suggestions here.

If you're running bash, the second answer doesn't involve invoking an additional program, if you're not, you may look at the the sed option if your shell doesn't support parameter expansion.

Another option would be to use the dos2unix (or similar) utility that converts the windows line ending (\r\n) to the unix line ending (\n).

Alternatively, you could get crazy and always trim off the last character irrespective of what it was with a bash parameter expansion like the one described here. But I would consider this the least-favorable option.

thanks for the `dos2unix` hint, i'll just force that user to do just that =). — alvas, Jan 18 '14 at 21:06

Russia Must Remove Putin · Answer 2 · 2014-01-18T21:08:15.800

1

To work in your constraints, I would convert the file from the MS format to Unix format with the dos2unix tool.

alvas@ubi:~/git/meanie$ dos2unix /my/file/path.txt

If you gain the ability to edit the Python script, I would open it in Python using the Universal Newlines approach.

with open('/file/path.txt', 'rU') as f:
    for line in f:
        parse(f)

edited Jan 18 '14 at 21:08

answered Jan 18 '14 at 20:55

Russia Must Remove Putin

374,368
89
403
331

Though it's simple enough to do `isinstance` and `os.path.exist`, I'm still not going to change my code just to suit 1 user ;P . I'll stick with asking him to use dos2unix =) – alvas Jan 18 '14 at 21:08

score 1 · Answer 3 · answered Jan 18 '14 at 21:19

dos2unix is popular but it's not installed everywhere. A more portable solution would be

tr -d '\r'

If you need to preserve carriage returns in the middle of lines (unlikely for text files) you could use

sed 's/\r$//'

These should both work fine for the simple case where you do not need to distinguish between text and binary files.

How to remove trailing `\r` in shell?

3 Answers3