Bash: create anonymous fifo

Question

We all know mkfifo and pipelines. The first one creates a named pipe, thus one has to select a name, most likely with mktemp and later remember to unlink. The other creates an anonymous pipe, no hassle with names and removal, but the ends of the pipe get tied to the commands in the pipeline, it isn't really convenient to somehow get a grip of the file descriptors and use them in the rest of the script. In a compiled program, I would just do ret=pipe(filedes); in Bash there is exec 5<>file so one would expect something like "exec 5<> -" or "pipe <5 >6" -is there something like that in Bash?

htamas · Answer 1 · 2013-08-17T23:56:27.193

You can unlink a named pipe immediately after attaching it to the current process, which practically results in an anonymous pipe:

# create a temporary named pipe
PIPE=$(mktemp -u)
mkfifo $PIPE
# attach it to file descriptor 3
exec 3<>$PIPE
# unlink the named pipe
rm $PIPE
...
# anything we write to fd 3 can be read back from it
echo 'Hello world!' >&3
head -n1 <&3
...
# close the file descriptor when we are finished (optional)
exec 3>&-

If you really want to avoid named pipes (e.g. the filesystem is read-only), your "get a grip of the file descriptors" idea also works. Note that this is Linux-specific due to the use of procfs.

# start a background pipeline with two processes running forever
tail -f /dev/null | tail -f /dev/null &
# save the process ids
PID2=$!
PID1=$(jobs -p %+)
# hijack the pipe's file descriptors using procfs
exec 3>/proc/$PID1/fd/1 4</proc/$PID2/fd/0
# kill the background processes we no longer need
# (using disown suppresses the 'Terminated' message)
disown $PID2
kill $PID1 $PID2
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptors when we are finished (optional)
exec 3>&- 4<&-

score 30 · Answer 2 · answered Sep 03 '10 at 19:01

While none of the shells I know can make pipes without forking, some do have better than the basic shell pipeline.

In bash, ksh and zsh, assuming your system supports /dev/fd (most do nowadays), you can tie the input or the output of a command to a file name: <(command) expands to a file name that designates a pipe connected to the output from command, and >(command) expands to a file name that designates a pipe connected to the input of command. This feature is called process substitution. Its primary purpose is to pipe more than one command into or out of another, e.g.,

diff <(transform <file1) <(transform <file2)
tee >(transform1 >out1) >(transform2 >out2)

This is also useful to combat some of the shortcomings of basic shell pipes. For example, command2 < <(command1) is equivalent to command1 | command2, except that its status is that of command2. Another use case is exec > >(postprocessing), which is equivalent to, but more readable than, putting the whole rest of the script inside { ... } | postprocessing.

Dennis Williamson · Answer 3 · 2017-07-26T20:00:51.073

13

Bash 4 has coprocesses.

A coprocess is executed asynchronously in a subshell, as if the command had been terminated with the ‘&’ control operator, with a two-way pipe established between the executing shell and the coprocess.

The format for a coprocess is:

coproc [NAME] command [redirections]

edited Jul 26 '17 at 20:00

answered Sep 03 '10 at 16:00

Dennis Williamson

111,445

score 10 · Answer 4 · answered Mar 09 '19 at 15:22

While @DavidAnderson's answer covers all the bases and offers some nice safeguards, the most important thing it reveals is that getting your hands on an anonymous pipe is as easy as <(:), as long as you stay on Linux.

So the shortest and simplest answer to your question is:

exec 5<> <(:)

On macOS it won't work, then you'll need to create a temporary directory to house the named fifo in until you've redirected to it. I don't know about other BSDs.

David Anderson · Answer 5 · 2019-09-17T14:15:22.740

The following function was tested using GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu). The operating system was Ubuntu 18. This function take a single parameter which is the desired file descriptor for the anonymous FIFO.

MakeFIFO() {
    local "MakeFIFO_upper=$(ulimit -n)" 
    if [[ $# -ne 1 || ${#1} -gt ${#MakeFIFO_upper} || -n ${1%%[0-9]*} || 10#$1 -le 2
        || 10#$1 -ge MakeFIFO_upper ]] || eval ! exec "$1<> " <(:) 2>"/dev/null"; then
        echo "$FUNCNAME: $1: Could not create FIFO" >&2
        return "1"
    fi
}

The following function was tested using GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17). The operating system was macOS High Sierra. This function starts out by created a named FIFO in an temporary directory known only to the process that created it. Next, the file descriptor is redirected to the FIFO. Finally, the FIFO is unlinked from from the filename by deleting the temporary directory. This makes the FIFO anonymous.

MakeFIFO() {
    MakeFIFO.SetStatus() {
        return "${1:-$?}"
    }
    MakeFIFO.CleanUp() {
        local "MakeFIFO_status=$?"
        rm -rf "${MakeFIFO_directory:-}"    
        unset "MakeFIFO_directory"
        MakeFIFO.SetStatus "$MakeFIFO_status" && true
        eval eval "${MakeFIFO_handler:-:}'; true'" 
    }
    local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)" "MakeFIFO_file=" 
    MakeFIFO_handler="$(trap -p EXIT)"
    MakeFIFO_handler="${MakeFIFO_handler#trap -- }"
    MakeFIFO_handler="${MakeFIFO_handler% *}"
    trap -- 'MakeFIFO.CleanUp' EXIT
    until "$MakeFIFO_success"; do
        [[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
        && 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
        MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
        MakeFIFO_file="$MakeFIFO_directory/pipe"
        mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
        ! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
        MakeFIFO_success="true"
    done
    rm -rf "${MakeFIFO_directory:-}"
    unset  "MakeFIFO_directory"
    eval trap -- "$MakeFIFO_handler" EXIT
    unset  "MakeFIFO_handler"
    "$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}

The above functions can be combined to a single function that will work on both operating systems. Below is an example of such a function. Here, an attempt is made to create a truly anonymous FIFO. If unsuccessfully, then a named FIFO is created and converted to an anonymous FIFO.

MakeFIFO() {
    MakeFIFO.SetStatus() {
        return "${1:-$?}"
    }
    MakeFIFO.CleanUp() {
        local "MakeFIFO_status=$?"
        rm -rf "${MakeFIFO_directory:-}"    
        unset "MakeFIFO_directory"
        MakeFIFO.SetStatus "$MakeFIFO_status" && true
        eval eval "${MakeFIFO_handler:-:}'; true'" 
    }
    local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)" "MakeFIFO_file=" 
    MakeFIFO_handler="$(trap -p EXIT)"
    MakeFIFO_handler="${MakeFIFO_handler#trap -- }"
    MakeFIFO_handler="${MakeFIFO_handler% *}"
    trap -- 'MakeFIFO.CleanUp' EXIT
    until "$MakeFIFO_success"; do
        [[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
        && 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
        if eval ! exec "$1<> " <(:) 2>"/dev/null"; then
            MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
            MakeFIFO_file="$MakeFIFO_directory/pipe"
            mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
            ! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
        fi
        MakeFIFO_success="true"
    done
    rm -rf "${MakeFIFO_directory:-}"
    unset  "MakeFIFO_directory"
    eval trap -- "$MakeFIFO_handler" EXIT
    unset  "MakeFIFO_handler"
    "$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}

Here is an example of creating an anonymous FIFO, then writing some text to the same FIFO.

fd="6"
MakeFIFO "$fd"
echo "Now is the" >&"$fd"
echo "time for all" >&"$fd"
echo "good men" >&"$fd"

Below is an example of reading the entire contents of the anonymous FIFO.

echo "EOF" >&"$fd"
while read -u "$fd" message; do
    [[ $message != *EOF ]] || break
    echo "$message"
done

This produces the following output.

Now is the
time for all
good men

The command below closes the anonymous FIFO.

eval exec "$fd>&-"

References:
Creating an anonymous pipe for later use
Files in Publicly Writable Directories Are Dangerous
Shell Script Security

score 3 · Answer 6 · answered Oct 08 '12 at 01:49

As of October 2012 this functionality still doesn't seem to exist in Bash, but coproc can be used if all you need unnamed/anonymous pipes for is to talk to a child process. Problem with coproc at this point is that apparently only one is supported at a time. I can't figure out why coproc got this limitation. They should have been an enhancement of the existing task backgrounding code (the & op), but that's a question for the authors of bash.

score 0 · Answer 7 · answered Jul 05 '16 at 22:00

Using the great and bright answer from htamas, I modified it a little to use it in an one liner, here it is:

# create a temporary named pipe
PIPE=(`(exec 0</dev/null 1</dev/null; (( read -d \  e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null ) | ( read -d \  e < /proc/self/stat ; echo $e  >&2 ; exec tail -f /dev/null 2> /dev/null )) &) 2>&1 | for ((i=0; i<2; i++)); do read e; printf "$e "; done`)
# attach it to file descriptors 3 and 4
exec 3>/proc/${PIPE[0]}/fd/1 4</proc/${PIPE[1]}/fd/0
...
# kill the temporary pids
kill ${PIPE[@]}
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptor when we are finished (optional)
exec 3>&- 4<&-

score 0 · Answer 8 · answered Feb 06 '24 at 10:20

Here's my contribution

# open an anonymous pipe, put the read handle in variable named $1 
# and the write handle in variable named $2
# WARNING: Be careful about closing write handles for read commands/subshells, 
# or the read-handle may never get eof
# e.g. do this:
# good:   mkpipe r w ; ( fd-close $w  ; cat <&${r} ; ) & echo hello >${w}
# better: mkpipe r w ; ( fd-close $w  ; cat <&${r} ; ) & fd-close $r ; echo hello >${w}
mk2pipe() {
  mkfifo $(mktemp -u) && local header_pipe="$_" || return $?
  # the <> keeps it open both ways long enough for us to open a read and a write handle without blocking
  eval exec "{_}<>'$header_pipe' {$1}<'$header_pipe' {$2}>'$header_pipe' {_}<&-"
  set $?
  rm -f "$header_pipe"
  return $?
}
with-fd-close() {
  eval '"${@:2}"' "$1<&-"
}
fd-close() {
  while test $# != 0
  do # if it is not a pure number, treat it as a var name
     if test -n "$1" -a -z "${1//[0-9]/}"
     then eval exec "$1<&-"
     else eval exec "${!1}<&-"
          printf -v "$1" "CLOSED-FD-${!1}"
     fi
     shift
  done
}
test-mk2pipe() {
  local BASH_XTRACEFD=2
  local -
  set -x
local w r
  mk2pipe r w || return $?
  with-fd-close $w exec sed -e 's/^/===/g' <&${r} &
  sleep 1
  fd-close $r
  ls >&${w}
  fd-close $w # should cause sub-process to quit
  wait $!
mk2pipe r w || return $?
  echo hello >&${w}
  fd-close $w
  sleep 1
  with-fd-close $w exec cat <&${r}
  fd-close $r
  wait $!
}

score -2 · Answer 9 · answered Apr 19 '21 at 09:49

-2

htamas answer with tail commands works well, but have a race condition. Here an explanation and a binary that resolve it - https://github.com/Z-Wave-Me/mkpipe .

answered Apr 19 '21 at 09:49

Oleg

1

Bash: create anonymous fifo

9 Answers9

Linked