docker container reading another container logs

Question

I want to create a docker that reads the logs, stdout and stderr from another container 'factorio' on a continuous basis.

I know i should do this as a volume but my best current option is

sudo docker logs -f factorio &> /var/log/factorio/current.logs

I run the above in a screen. I had it as a background command but when you exit the terminal it would exit that command. Screen allows it to run on its own terminal.

which runs in the background and constantly updates the file. My newapp container connect as a volume to this file location and tails every 5 seconds to get any updates and send notifications based on those updates. There is definitely a better way to do this with volumes but I dont know how.

someapp docker command below:

sudo docker run -d \
  -p 34197:34197/udp \
  -p 27015:27015/tcp \
  -v /opt/factorio:/factorio \
  --name factorio \
  --restart=always \
  dtandersen/factorio:0.17.16

The log file is /opt/factorio/factorio-current.log but does not include stdout or stderr. docker logs contains this information

Answer 1

Use a shared volume. You don't need docker-compose for it.

You can read more about it in the official documentation.

But the basic idea is to create a volume, then load this volume to both containers. Then store the desired logs/files in the volume, which you can read from the other container.

You could also use a folder on your filesystem and mount it as a volume if you want to access it outside of the containers as well.

Answer 2

So, the preferred way on how to approach that on docker is to use a volume.

So as explained on this page, your first goal is to create the said volume, mind that volumes are actually just some kind of named mount points.

1. Create your volume

docker volume create factorio_logs

2. (Optional step, but might be useful for later): Verify that your volume exists:

docker volume ls

3. Link your application log producer to your volume

docker run -d \ 
  -p 34197:34197/udp \
  -p 27015:27015/tcp \
  # adapt the line below to wherever your application is configured to store logs
  -v factorio_logs:/var/log/factorio \ 
  --name factorio \
  --restart=always \
  dtandersen/factorio:0.17.16

nota: -v is an advanced option that could either mount volume or bind-mount based on the source given to it

4. Create your consumer container and also mount the volume in (this is just a tail -f for the example)

docker run -ti -v factorio_logs:/var/log/factorio alpine tail -f /var/log/factorio/*.log

Mind that volume are persisted beyond the life of a container. You will have to manually clear the volume and recreate it to clear your logs

docker volume rm factorio_logs
docker volume create factorio_logs

Also note that those kind of orchestration of multiple containers, volumes, etc could be greatly simplified using docker-compose

Answer 3

Funnily enough I'm also wanting to export my Factorio server logs. Here's what I've got working.

1. Create a raw socket server (not WebSocket). You can run this remotely, on the host, or in another container.

   I recommend UDP, as TCP seems to miss some messages during container shutdown, but both work similarly.

2. Change the log driver of the container you want to export the logs from to use syslog (see the Docker docs). Set the syslog-address to your socket server.

   services:
     my-server:
       # ...
       logging:
         driver: syslog
         options:
           syslog-address: "udp://syslog-socket-server:9002"
           syslog-format: "rfc5424"
   

3. Start up both the socket server and your container: et voilà! The socket server will now receive the logs of the container, and you can forward them as required.

There are many benefits to this method:

• No need for mounting a Docker socket into another container, which can be risky
• The logs are actively pushed by the required container, so they will be received in real-time - no need for intermittent scraping
• Many other containers can also forward their logs to the socket server
• The syslog format is simple and easy to parse. I'm planning to use palindromicity/simple-syslog, but I found regex parsers too regex101.com/library/Wgbxn2 stackoverflow.com/questions/49049081

Example output

Here's an example of what my socket server prints

[SyslogSocketServer] started listening: InetSocketAddress(hostname=127.0.0.1, port=9002)
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + FACTORIO_VOL=/factorio
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + LOAD_LATEST_SAVE=true
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + GENERATE_NEW_SAVE=false
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + SAVE_NAME=
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + mkdir -p /factorio
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + mkdir -p /factorio/saves
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + mkdir -p /factorio/config
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + mkdir -p /factorio/mods
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + mkdir -p /factorio/scenarios
[SyslogSocketServer] received msg: <27>1 2022-06-26T17:28:07Z docker-desktop f22cae36b987 926 f22cae36b987 - + mkdir -p /factorio/script-output

Implementation

I'll give a very brief overview of my implementation.

Socket server

I've used Kotlin 1.7.0 and Ktor 2.0.2 for the socket server - but any other language will do just fine.

The socket server is based on the example in the Ktor docs. It

import io.ktor.network.selector.SelectorManager
import io.ktor.network.sockets.*
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.*


class SyslogSocketServer(
  socketHostname: String = "127.0.0.1",
  socketPort: Int = 9002,
) {

  private val selectorManager: SelectorManager = SelectorManager(Dispatchers.IO)

  // create a UDP server socket - it will listen to logs from Docker
  private val serverSocketUdp: BoundDatagramSocket = aSocket(selectorManager)
    .udp()
    .bind(localAddress = InetSocketAddress(socketHostname, socketPort))

  suspend fun start() = coroutineScope {
    println("SyslogSocketServer is listening at ${serverSocketUdp.localAddress}")

    serverSocketUdp.incoming.receiveAsFlow()
      .onEach {
        val msg = it.packet.readText()
        println("[SyslogSocketServer] received msg: $msg")
      }.launchIn(this)
  }
}

suspend fun main() {
  val syslogSocketServer = SyslogSocketServer()

  syslogSocketServer.start()
}

Example logging driver settings

Here's how I configured my Factorio server using Docker Compose.

Note that I used host.docker.internal (read more here) as I'm running my socket server on the host, not in another container.

version: "3.9"

services:

  factorio-server:
    image: factoriotools/factorio
    container_name: "factorio-server"
    ports:
      - "34197:34197/udp" # factorio
      - "27015:27015/tcp" # rcon
    logging:
      driver: syslog
      options:
        syslog-address: "udp://host.docker.internal:9002"
        syslog-format: "rfc5424"