IO redirection with mkpipe for logging purposes

Question

I have a bunch of scripts that send output to stdout. I am redirecting the output to files, but these files get large very very quickly. For example:

./script_with_lots_of_outpu.sh 2>&1 mylog.txt &

I would like to send the output to a named pipe instead so something like the following script could switch the file being written to:

#!/bin/bash
if [ $# -ne 2 ]; then
echo "USAGE: ./redir.sh pipename filename" 
fi

pipename=$1
filename=$2
trap filename="`date +%s`$filename" 2
mkfifo $pipename

while [ 1 -eq 1 ]
do
    read input
    echo $input >> $filename
done < $pipename

One could send this script a CTRL-C (or some other signal) and it would effectively cause the output of the pipe to start writing to a different file (prepended with a timestamp).

When I run this script and then echo something to it, it starts writing a ton of empty lines:

   > ./redir.sh testpipe testfile &
   > echo "this is a tesT" > testpipe
   > wc -l testfile
   627915 testfile

How can I make redir.sh only write out to a file when the pipe it reads from is written to?

EDIT

final product seems to be working as follows. I need to test some more to find out if it is production-worthy

#!/bin/bash

if [ $# -ne 2 ]; then
    echo "USAGE: ./redir.sh pipename filename" 
    exit -1
fi

pipename=$1; rm $pipename;
origname=$2.log
filename=$2

rename()
{
    filename="$origname-`date +%s`"
    mv $origname $filename
    nohup nice -n 20 tar -czvf $filename.tar.gz $filename &
    trap rename 2
}

mkfifo $pipename
trap rename 2

while [ 1 -eq 1 ]
do
    read input
    echo $input >> $origname
done <> $pipename

Adam Zalcman · Accepted Answer · 2011-12-15T21:16:56.310

There is a caveat concerning the reading from a named pipe. When you open a named pipe your open() call hangs until a writer shows up. When a writer shows up subsequent read() calls will return whatever data the writer has written to the pipe. However, when the writer closes the pipe (or exits) the read() calls start returning 0 (rather than blocking).

This is the reason that instead of

int fd = open("testpipe", O_RDONLY);

one might want to open a pipe like this

int fd = open("testpipe", O_RDWR);

This way the process is not only the reader, but also the writer. Even though you never actually write anything to the pipe, this ensures that a writer exists and hence read() calls do not return 0, but block instead waiting for some writer to write something to the pipe.

Now, when your script does this:

while [ 1 -eq 1 ]
do
    read input
    ...
done < $pipename

your shell opens the pipe for reading only (O_RDONLY).

The solution to your problem is to make shell open the pipe for both reading and writing like this:

while [ 1 -eq 1 ]
do
    read input
    ...
done <> $pipename

Note the replacement of < $pipename with <> $pipename. This is what makes the shell open the pipe for both reading and writing (O_RDWR).

IO redirection with mkpipe for logging purposes

1 Answers1