I have a Java application(fat jar) that runs on the Linux Ubuntu instance and fails from time to time. The application has embedded web server and exposes health check URL.
I need to implement some kind of external monitoring via some tool installed on this Linux box that will periodically ping this URL and restart the Java application if needed.
Could you please show me how it can be implemented?
I doubt you'd find as single tool that will do all that.
However, here's a set of things you can do to implement what you want. The idea is that you'll need to make your java app a systemd service, install a tool that can ping your health check URL, and have a simple bash script to run occasionally that runs the tool, checks its exit code, and if it's not what we seek, it will restart your service (systemd takes care of all child processes and etc.).
Make a systemd service out of your Java application:
Create a file /etc/systemd/system/<my-service>.service. Replace <my-service> with whatever name you want your service to have.
Edit your file by configuring the following:
[Unit]
Description=My java application
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/java -jar /path/to/my/jar # or whatever command you use to start your application, make sure the paths are absolute (to find your java executable, run whereis java)
User=someuser # the user with which the application will run, can be root
StandardOutput=journal
StandardError=journal # you can configure where stdout/stderr go, you can use journal for easier logging
[Install]
WantedBy=multi-user.target
You need to edit the following:
Description - set a description that fits your application (not required)
Type - if your application forks and runs child processes, then you need to set type to forking, so systemd will know that you fork child processes and it will manage them as well.
ExecStart - the command (with absolute paths) that you use to start your application. You can make it a bash script as well.
After you configure that file, you need to reload the daemon configs for systemd:
sudo systemctl daemon-reload
And then enable your service (if you want it to start automatically at boot):
sudo systemctl enable <my-service>.service
or just start it:
sudo systemctl start <my-service>.service
Congratulations, now your application is working as a systemd service. You can read a lot more detailed information about creating, modifying, and managing custom systemd services in this Red Hat documentation.
We're not done yet though, next step:
Install the required tool for pinging your URL and make a bash script to restart your service:
A tool that allows you to ping URLs is httping for example. You can install it on your Ubuntu system by running sudo apt install httping.
Then write a bash script that calls httping with the proper arguments and restart your service, if something goes wrong. Something like the following (a very basic example):
#!/bin/bash
httping -c 5 -g <url_for_health_check> -s
if [[ $? -eq 0 ]]; then
echo "Health check is OK."
else
echo "Health check is not OK.. restarting service."
sudo systemctl restart <my_service>.service
fi
This bash script calls httping and pings the Health check URL 5 times for its status code (-c for the number of pings, -g for the URL, -s for the status code). When httping is called with -s, it will never exit with an exit code of 0 unless the status of the URL is OK. So, if it doesn't exit with 0, then there's a problem, so we restart the service (that is the if check).
So, are we done? We have a script now that restarts our service (java application) if it's having problems. However, we still need to run this script automatically over time.
To do that, you can either create a cron job or use a systemd timer.
An example of how to do it with systemd:
First you need to make your bash script a service, just like your Java application. A lot simpler one this time though.
Create the file /etc/systemd/system/health_check.service and add the following (edit path of course):
[Unit]
Description=Health check bash script
[Service]
Type=oneshot
ExecStart=/bin/bash /path/to/bash/script.sh
Then create a systemd timer to run that service:
Create /etc/systemd/system/health_check.timer and fill it with the following content:
[Unit]
Description=Health check script timer
After=<my_service>.service
[Timer]
Unit=health_check.service
OnCalendar=minutely
[Install]
WantedBy=multi-user.target
Here you must specify that the timer must always start after the service you're monitoring (replace <my_service>.service with your java application service). And OnCalendar= specifies when to run the timer. Currently, it will run once every minute. You can change that, see systemd.time for more details.
Aaand that's almost it. Now just enable the timer (not the bash script service) and start it, but also reload the daemon configs first:
sudo systemctl daemon-reload
sudo systemctl enable health_check.timer
sudo systemctl start health_check.timer
Now you should be ready. A timer will start your bash script every 1 minute, that will check whether the health check URL is okay, and if it is not, it will restart your service.
There are a bit more complex ways to do that without using bash scripts, only systemd, however, for starters this should work.
Note: This sets up a completely automatic monitoring. No need for manual intervention, that is why you enable the timer as well, so it can run on startup along with your java application. If you want to do it all manually, do not enable those services and start them only when you need to do so.
References:
systemd.service - the manual page for systemd service units, you can read to understand what are services, how to use them, and how to configure them.
systemd.timer - the manual page for systemd timer units, to understand how timers work and how to configure them.
