I am encountering a challenge while updating our Elasticsearch/Kibana setup, which involves two separate virtual machines named Kibana01 and Kibana02. Each virtual machine hosts an instance of Ubuntu and the ELK stack. We are transitioning from an older setup on Kibana01 to a newer setup on Kibana02, aiming to utilize the latest features and improvements of the ELK stack. Here are the specifics of each setup:
Kibana01 VM:
- Operating System: Ubuntu 16.04.6 LTS
- ELK Stack: Elasticsearch 7.5.2, with corresponding versions of Logstash and Kibana.
Kibana02 VM:
- Operating System: Ubuntu 22.04.2 LTS
- ELK Stack: Elasticsearch 7.17.10, with corresponding versions of Logstash and Kibana.
Identified Problem:
We are seeing an abnormally high volume of log entries for the service pim_queue_manager.sh on the new Kibana server (Kibana02) compared to the old one (Kibana01). Here are the results obtained after running the following command on both servers:
administrator@par-kibana:~$ time gunzip -c syslog_facilities.txt | sort | uniq -c | sort -n
Kibana01 (Ubuntu 16.04.6 LTS, Elasticsearch 7.5.2):
Kibana02 (Ubuntu 22.04.2 LTS, Elasticsearch 7.17.10):
The comparison revealed: Kibana01 VM logs for pim_queue_manager.sh: 465,589 entries Kibana02 VM logs for pim_queue_manager.sh: 32,786,602 entries
On Kibana02, the pim_queue_manager.sh service records 32,786,602 entries, compared to only 465,589 on Kibana01. This overload causes the /var/log/syslog directory to fill up too quickly, exceeding the management capabilities of logrotate and blocking the recording of new logs.
Hypotheses:
- Process Loop: A loop in a process on
Kibana02could be responsible for this log overload. - Complete Transmission of Log History: During the switchover, it appears that the virtual machines are sending their entire log history to
Kibana02, treating it as if it were new. Normally, only post-switchover logs should be transmitted.
Request for Assistance:
I am seeking advice on how to solve these problems. How can I prevent the transmission of the complete log history to the new Kibana server and ensure a transition without overloading the log system?
