36

I want to execute a command, have the output of that command get gzip'd on the fly, and also echo/tee out the output of that command.

i.e., something like:

echo "hey hey, we're the monkees" | gzip --stdout > my_log.gz

Except when the line executes, I want to see this on standard out:

hey hey, we're the monkees
Sean Bright
  • 118,630
  • 17
  • 138
  • 146
Ross Rogers
  • 23,523
  • 27
  • 108
  • 164

4 Answers4

65

Another way (assuming a shell like bash or zsh):

echo "hey hey, we're the monkees" | tee >(gzip --stdout > my_log.gz)

The admittedly strange >() syntax basically does the following:

  • Create new FIFO (usually something in /tmp/)
  • Execute command inside () and bind the FIFO to stdin on that subcommand
  • Return FIFO filename to command line.

What tee ends up seeing, then, is something like:

tee /tmp/arjhaiX4

All gzip sees is its standard input.

For Bash, see man bash for details. It's in the section on redirection. For Zsh, see man zshexpn under the heading "Process Substitution."

As far as I can tell, the Korn Shell, variants of the classic Bourne Shell (including ash and dash), and the C Shell don't support this syntax.

greyfade
  • 24,948
  • 7
  • 64
  • 80
  • Cool, I'm learning something myself here. Can you elaborate on what is happening there? – Paul Dixon Feb 20 '09 at 20:08
  • if I read it right, instead of given a file to tee, you've got it sending the copy as input to bracketed expression which writes the gzip output to another file. The uncompressed data leaves tee as normal on stdout – Paul Dixon Feb 20 '09 at 20:14
  • thanks for the edit, appreciate the extra info! Never knew about that FIFO syntax. – Paul Dixon Feb 20 '09 at 20:15
  • 1
    It came as a shock to me when I first learned it. But it lets you get away with very complex redirections, with several programs operating on the same input simultaneously. Of course, you can also do `mkfifo` by hand and run all of these commands in different consoles if you need to. – greyfade Feb 20 '09 at 20:17
  • I've been doing Unix for 20 years, and I haven't wrapped my head around this new syntax yet. – Paul Tomblin Feb 20 '09 at 20:23
  • 2
    I'm using this for logging. I've found gzip buffers a lot and when I ctrl-c I lose everything. [Ignoring SIGINT](https://stackoverflow.com/a/47693330/1888983) with `>(trap '' INT; gzip --stdout > my_log.gz)` seems to save it. – jozxyqk May 16 '19 at 05:21
  • @jozxyqk Nice! Indeed it seems that gzip buffers [16KB](https://unix.stackexchange.com/a/643084/8848) of compressed data before writing – Francois Oct 05 '21 at 15:33
46
echo "hey hey, we're the monkees" | tee /dev/tty | gzip --stdout > my_log.gz

As pointed out in the comments, /dev/stdout might work better than /dev/tty in some circumstances.

Paul Tomblin
  • 179,021
  • 58
  • 319
  • 408
  • 3
    What's /dev/tty doing there? The original question wanted the output on standard output, not necessarily on the terminal. – Gareth Rees Feb 20 '09 at 20:29
  • 4
    /dev/tty is a synonym for the current terminal. The questioner used "standard out" in the customary manner to mean the the current terminal, rather than a more strict definition of the term. – Paul Tomblin Feb 20 '09 at 20:41
  • 3
    If it were indeed customary to use "standard output" to mean "current terminal", then that would be a custom likely to lead to much confusion! For this question, bash has /dev/stdout. – Gareth Rees Feb 20 '09 at 21:11
  • 1
    Under what circumstances would /dev/tty not produce the desired outcome where /dev/stdout would? – Paul Tomblin Feb 20 '09 at 21:13
  • 6
    When stdout goes to a file and not to the terminal. – Gareth Rees Feb 26 '09 at 13:10
  • 1
    This is just plain not what was asked -- what was asked for was a shell pipeline that would send the output to its `stdout` AND gzip it to a file. (What if I wanted to send the output on to yet another program -- say, **less(1)**. – SamB Apr 27 '10 at 00:23
  • /dev/stdout didn't work on Solaris 10 using KSH, but /dev/tty did. – Tom Purl May 27 '11 at 15:58
  • /dev/stdout will not work in any cases as it is applied after the |. – Joshua Apr 26 '14 at 14:30
  • 1
    @GarethRees you are right but practicality beats purity if you python -m this. you could serve the public with pointing it out in an answer - as opposed to editing the question, which latter would hide the fact that the customary is, arguably, commonplace. – n611x007 Oct 27 '15 at 10:26
20

Have a nice cup of tee!

The tee command copies standard input to standard output and also to any files given as arguments. This is useful when you want not only to send some data down a pipe, but also to save a copy

As I'm having a slow afternoon, here's some gloriously illustrative ascii-art...

           +-----+                   +---+                  +-----+  
stdin ->   |cmd 1|    -> stdout ->   |tee|   ->  stdout  -> |cmd 2|
           +-----+                   +---+                  +-----+
                                       |
                                       v
                                     file

As greyfade demonstrates in another answer the 'file' need not be a regular file, but could be FIFO letting you pipe that tee'd output into a third command.

           +-----+                   +---+                  +-----+  
stdin ->   |cmd 1|    -> stdout ->   |tee|   ->  stdout  -> |cmd 2|
           +-----+                   +---+                  +-----+
                                       |
                                       v
                                     FIFO
                                       |
                                       v
                                    +-----+
                                    |cmd 3|
                                    +-----+
Paul Dixon
  • 295,876
  • 54
  • 310
  • 348
  • But I want to gzip the intermediary file on the fly. Is that possible only using tee? – Ross Rogers Feb 20 '09 at 19:55
  • yes, the other paul wrote a nice succinct answer while I messed about with an ascii art diagram :) – Paul Dixon Feb 20 '09 at 19:59
  • :-P I'm familiar with tee'ing to a file like you're describing. I just don't know how to gzip that file. if i do "echo foo | tee bar.log", I don't know how to make tee gzip 'bar.log', aside from the solution Paul Tomblin posted. – Ross Rogers Feb 20 '09 at 20:01
  • man, stackoverflow is awesome. I didn't even have to harass the office linux guru. Thanks all. – Ross Rogers Feb 20 '09 at 20:02
  • 2
    greyfade's excellent answer shows how you can do what you like with the tee'd output. – Paul Dixon Feb 20 '09 at 20:16
  • Yes, grayfade's answer does precisely what this one suggests, but with a very succinct syntax. This ones `cmd 3` is written `>(cmd 3)` there. – SamB Apr 27 '10 at 00:30
6

Just to post a way that doesn't involve touching disk:

echo "hey hey, we're the monkees" | (exec 1>&3 && tee /proc/self/fd/3 | gzip --stdout > my_log.gz)
Joshua
  • 40,822
  • 8
  • 72
  • 132