How to configure filebeat to handle log rotation?

Question

My settings

I have a docker cluster in which supervisord is writing logs to output.log on a shared volume, from which Filebeat is reading and shipping logs to ES. In order to prevent disk-full problems, I have configured supervisord to rotate the logs, and I'm worried that Filebeat might miss out logs or send logs twice.

My problem

I would like Filebeat to consume the logs written to output.log, without duplication or missing out log lines.

My question

How do I configure Filebeat to handle supervisord log rotation?

Answer 1

Seems like supervisord rotation works with filebeat out of the box.

For example, in the program section of supervisord.conf, the following configuration rotated the logs, and filebeat did not miss a single line.

stdout_logfile_maxbytes=1024
stdout_logfile_backups=5
stderr_logfile_maxbytes=1024
stderr_logfile_backups=5

It is worth to mention that in real production system I would use larger values (in the dozens or hundreds of MBs), so that Filebeat will not miss the files when all the files are rotated during its sleep interval.