How to read from a file or standard input in Bash

Question

The following Perl script (my.pl) can read from either the file in the command line arguments or from standard input (STDIN):

while (<>) {
   print($_);
}

perl my.pl will read from standard input, while perl my.pl a.txt will read from a.txt. This is very handy.

Is there an equivalent in Bash?

Answer 1

The following solution reads from a file if the script is called with a file name as the first parameter $1 and otherwise from standard input.

while read line
do
  echo "$line"
done < "${1:-/dev/stdin}"

The substitution ${1:-...} takes $1 if defined. Otherwise, the file name of the standard input of the own process is used.

Answer 2

Perhaps the simplest solution is to redirect standard input with a merging redirect operator:

#!/bin/bash
less <&0

Standard input is file descriptor zero. The above sends the input piped to your bash script into less's standard input.

Read more about file descriptor redirection.

Answer 3

Here is the simplest way:

#!/bin/sh
cat -

Usage:

$ echo test | sh my_script.sh
test

To assign stdin to the variable, you may use: STDIN=$(cat -) or just simply STDIN=$(cat) as operator is not necessary (as per @mklement0 comment).

---

To parse each line from the standard input, try the following script:

#!/bin/bash
while IFS= read -r line; do
  printf '%s\n' "$line"
done

---

To read from the file or stdin (if argument is not present), you can extend it to:

#!/bin/bash
file=${1--} # POSIX-compliant; ${1:--} can be used either.
while IFS= read -r line; do
  printf '%s\n' "$line" # Or: env POSIXLY_CORRECT=1 echo "$line"
done < <(cat -- "$file")

Notes:

^{- read -r - Do not treat a backslash character in any special way. Consider each backslash to be part of the input line.}

^{- Without setting IFS, by default the sequences of Space and Tab at the beginning and end of the lines are ignored (trimmed).}

^{- Use printf instead of echo to avoid printing empty lines when the line consists of a single -e, -n or -E. However there is a workaround by using env POSIXLY_CORRECT=1 echo "$line" which executes your external GNU echo which supports it. See: How do I echo "-e"?}

See: How to read stdin when no arguments are passed? at stackoverflow SE

Answer 4

I think this is the straightforward way:

$ cat reader.sh
#!/bin/bash
while read line; do
  echo "reading: ${line}"
done < /dev/stdin

--

$ cat writer.sh
#!/bin/bash
for i in {0..5}; do
  echo "line ${i}"
done

--

$ ./writer.sh | ./reader.sh
reading: line 0
reading: line 1
reading: line 2
reading: line 3
reading: line 4
reading: line 5

Answer 5

The echo solution adds new lines whenever IFS breaks the input stream. @fgm's answer can be modified a bit:

cat "${1:-/dev/stdin}" > "${2:-/dev/stdout}"

Answer 6

The Perl loop in the question reads from all the file name arguments on the command line, or from standard input if no files are specified. The answers I see all seem to process a single file or standard input if there is no file specified.

Although often derided accurately as UUOC (Useless Use of cat), there are times when cat is the best tool for the job, and it is arguable that this is one of them:

cat "$@" |
while read -r line
do
    echo "$line"
done

The only downside to this is that it creates a pipeline running in a sub-shell, so things like variable assignments in the while loop are not accessible outside the pipeline. The bash way around that is Process Substitution:

while read -r line
do
    echo "$line"
done < <(cat "$@")

This leaves the while loop running in the main shell, so variables set in the loop are accessible outside the loop.

Answer 7

Perl's behavior, with the code given in the OP can take none or several arguments, and if an argument is a single hyphen - this is understood as stdin. Moreover, it's always possible to have the filename with $ARGV. None of the answers given so far really mimic Perl's behavior in these respects. Here's a pure Bash possibility. The trick is to use exec appropriately.

#!/bin/bash

(($#)) || set -- -
while (($#)); do
   { [[ $1 = - ]] || exec < "$1"; } &&
   while read -r; do
      printf '%s\n' "$REPLY"
   done
   shift
done

Filename's available in $1.

If no arguments are given, we artificially set - as the first positional parameter. We then loop on the parameters. If a parameter is not -, we redirect standard input from filename with exec. If this redirection succeeds we loop with a while loop. I'm using the standard REPLY variable, and in this case you don't need to reset IFS. If you want another name, you must reset IFS like so (unless, of course, you don't want that and know what you're doing):

while IFS= read -r line; do
    printf '%s\n' "$line"
done

Answer 8

#!/usr/bin/bash

if [ -p /dev/stdin ]; then
       #for FILE in "$@" /dev/stdin
    for FILE in /dev/stdin
    do
        while IFS= read -r LINE
        do
            echo "$@" "$LINE"   #print line argument and stdin
        done < "$FILE"
    done
else
    printf "[ -p /dev/stdin ] is false\n"
     #dosomething
fi

Running:

echo var var2 | bash std.sh

Result:

var var2

Running:

bash std.sh < <(cat /etc/passwd)

Result:

root:x:0:0::/root:/usr/bin/bash
bin:x:1:1::/:/usr/bin/nologin
daemon:x:2:2::/:/usr/bin/nologin
mail:x:8:12::/var/spool/mail:/usr/bin/nologin

Answer 9

Please try the following code:

while IFS= read -r line; do
    echo "$line"
done < file

Answer 10

More accurately...

while IFS= read -r line ; do
    printf "%s\n" "$line"
done < file

Answer 11

The following works with standard sh (tested with Dash on Debian) and is quite readable, but that's a matter of taste:

if [ -n "$1" ]; then
    cat "$1"
else
    cat
fi | commands_and_transformations

Details: If the first parameter is non-empty then cat that file, else cat standard input. Then the output of the whole if statement is processed by the commands_and_transformations.

Answer 12

I combined all of the above answers and created a shell function that would suit my needs. This is from a Cygwin terminal of my two Windows 10 machines where I had a shared folder between them. I need to be able to handle the following:

• cat file.cpp | tx
• tx < file.cpp
• tx file.cpp

Where a specific filename is specified, I need to used the same filename during copy. Where input data stream has been piped through, then I need to generate a temporary filename having the hour minute and seconds. The shared mainfolder has subfolders of the days of the week. This is for organizational purposes.

Behold, the ultimate script for my needs:

tx ()
{
  if [ $# -eq 0 ]; then
    local TMP=/tmp/tx.$(date +'%H%M%S')
    while IFS= read -r line; do
        echo "$line"
    done < /dev/stdin > $TMP
    cp $TMP //$OTHER/stargate/$(date +'%a')/
    rm -f $TMP
  else
    [ -r $1 ] && cp $1 //$OTHER/stargate/$(date +'%a')/ || echo "cannot read file"
  fi
}

If there is any way that you can see to further optimize this, I would like to know.

Answer 13

Two principle ways:

• Either pipe the argument files and stdin into a single stream and process that like stdin (stream approach)
• Or redirect stdin (and argument files) into a named pipe and process that like a file (file approach)

Stream approach

Minor revisions to earlier answers:

• Use cat, not less. It's faster and you don't need pagination.

• Use $1 to read from first argument file (if present) or $* to read from all files (if present). If these variables are empty, read from stdin (like cat does)

  #!/bin/bash
  cat $* | ...
  

File approach

Writing into a named pipe is a bit more complicated, but this allows you to treat stdin (or files) like a single file:

• Create pipe with mkfifo.

• Parallelize the writing process. If the named pipe is not read from, it may block otherwise.

• For redirecting stdin into a subprocess (as necessary in this case), use <&0 (unlike what others have been commenting, this is not optional here).

    #!/bin/bash
    mkfifo /tmp/myStream
    cat $* <&0 > /tmp/myStream &           # separate subprocess (!)
    AddYourCommandHere /tmp/myStream       # process input like a file, 
    rm /tmp/myStream                       # cleaning up
  

File approach: Variation

Create named pipe only if no arguments are given. This may be more stable for reading from files as named pipes can occasionally block.

#!/bin/bash
FILES=$*
if echo $FILES | egrep -v . >&/dev/null; then # if $FILES is empty
   mkfifo /tmp/myStream
   cat <&0 > /tmp/myStream &
   FILES=/tmp/myStream
fi
AddYourCommandHere $FILES     # do something ;)
if [ -e /tmp/myStream ]; then
   rm /tmp/myStream
fi

Also, it allows you to iterate over files and stdin rather than concatenate all into a single stream:

for file in $FILES; do
    AddYourCommandHere $file
done

Answer 14

The code ${1:-/dev/stdin} will just understand the first argument, so you can use this:

ARGS='$*'
if [ -z "$*" ]; then
  ARGS='-'
fi
eval "cat -- $ARGS" | while read line
do
   echo "$line"
done

Answer 15

Reading from stdin into a variable or from a file into a variable.

Most examples in the existing answers use loops that immediately echo each of line as it is read from stdin. This might not be what you really want to do.

In many cases you need to write a script that calls a command which only accepts a file argument. But in your script you may want to support stdin also. In this case you need to first read full stdin and then provide it as a file.

Let's see an example. The script below prints the certificate details of a certificate (in PEM format) that is passed either as a file or via stdin.

# print-cert script

content=""
while read line
do
  content="$content$line\n"
done < "${1:-/dev/stdin}"
# Remove the last newline appended in the above loop
content=${content%\\n}

# Keytool accepts certificate only via a file, but in our script we fix this.
keytool -printcert -v -file <(echo -e $content)

# Read from file

cert-print mycert.crt

# Owner: CN=....
# Issuer: ....
# ....


# Or read from stdin (by pasting)

cert-print
#..paste the cert here and press enter
# Ctl-D

# Owner: CN=....
# Issuer: ....
# ....


# Or read from stdin by piping to another command (which just prints the cert(s) ). In this case we use openssl to fetch directly from a site and then print its info.


echo "" | openssl s_client -connect www.google.com:443 -prexit 2>/dev/null \
| sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' \
| cert-print

# Owner: CN=....
# Issuer: ....
# ....

Answer 16

I don't find any of these answers acceptable. In particular, the accepted answer only handles the first command line parameter and ignores the rest. The Perl program that it is trying to emulate handles all the command line parameters. So the accepted answer doesn't even answer the question.

Other answers use Bash extensions, add unnecessary 'cat' commands, only work for the simple case of echoing input to output, or are just unnecessarily complicated.

However, I have to give them some credit, because they gave me some ideas. Here is the complete answer:

#!/bin/sh

if [ $# = 0 ]
then
        DEFAULT_INPUT_FILE=/dev/stdin
else
        DEFAULT_INPUT_FILE=
fi

# Iterates over all parameters or /dev/stdin
for FILE in "$@" $DEFAULT_INPUT_FILE
do
        while IFS= read -r LINE
        do
                # Do whatever you want with LINE here.
                echo $LINE
        done < "$FILE"
done

Answer 17

This one is easy to use on the terminal:

$ echo '1\n2\n3\n' | while read -r; do echo $REPLY; done
1
2
3

Answer 18

As a workaround, you can use the stdin device in the /dev directory:

....| for item in `cat /dev/stdin` ; do echo $item ;done

Answer 19

With...

while read line
do
    echo "$line"
done < "${1:-/dev/stdin}"

I got the following output:

Ignored 1265 characters from standard input. Use "-stdin" or "-" to tell how to handle piped input.

Then decided with for:

Lnl=$(cat file.txt | wc -l)
echo "Last line: $Lnl"
nl=1

for num in `seq $nl +1 $Lnl`;
do
    echo "Number line: $nl"
    line=$(cat file.txt | head -n $nl | tail -n 1)
    echo "Read line: $line"
    nl=$[$nl+1]
done

Answer 20

Just test the number of arguments to your script, and test whether the first argumetn ($1) is a file. If false, use the stdin -:

#!/bin/bash
[ $# -ge 1 -a -f "$1" ] && input="$1" || input="-"
cat $input

See this qustion.

Answer 21

@gniourf_gniourf's answer is the correct one but uses a lot of bashisms. Since this question is the top Google result, here is a POSIX compliant version:

#!/bin/sh

if [ $# -eq 0 ]; then
    set -- -
fi

for f in "$@"; do
    if [ "$f" = - ] || exec < "$f"; then
        while IFS= read -r line; do
            printf '%s\n' "$line"
    done
done

or if you want to be terse:

#!/bin/sh

[ $# -eq 0 ] || set -- -
for f; do
    { [ "$f" = - ] || exec < "$f"; } &&
    while IFS= read -r line; do
        printf '%s\n' "$line"
    done
done

Answer 22

Use:

for line in `cat`; do
    something($line);
done