1

I'm trying to run my docker-compose containers via a systemd service, and have fail2ban read their logs. However, docker-compose adds a prefix like:

container_name_1 | Actual log message

So what I'm trying to do is something like docker-compose up --no-color 2>&1 | sed 's/^[^ ]* *| //' to strip that prefix so that fail2ban can match on the log lines correctly. But when I do that, I only see Started mydockercontainers.service in the journalctl logs. Removing the pipe to sed fixes everything.

How can I remove this prefix and keep the log messages in the journal?

Cole
  • 151

1 Answers1

3

Most programs including sed will use block buffering – not line buffering – when their stdout is something else than an interactive tty device. (For services, the stdout is a pipe and you would see a similar effect with sed | cat.)

This means that output will be delayed while it accumulates in the buffer – it will only be written in chunks of 10-20 lines or so (however many fit in a 4kB buffer).

  • Use the sed -u or --unbuffered option to disable the buffering.

    Grep has grep --line-buffered; awk has fflush(...).

  • In cases where the program doesn't have such an option, wrap it with stdbuf -i0 -o0. (This hooks into glibc internals, so it won't work with e.g. Golang programs.)

  • For future readers: If you're writing your own program, use setlinebuf() (for C) or sys.stdout.reconfigure(...) (Python) or some equivalent to disable the runtime's built-in buffering.

    Alternatively, flush stdout after each message, e.g. print(xxx, flush=True) or fflush(stdout).

    Typically stderr remains line-buffered no matter where it goes, and only stdout needs this treatment. So write error messages to stderr or to syslog() instead of stdout.

grawity
  • 501,077