2

I wanted to install ElastAlert, an alerting tool written in python2, inside of a pipenv in an attempt to keep applications and their requirements/ dependencies from stepping on one another while running with the lowest possible permissions. To accomplish this, I generated a python2 virtual environment with pipenv --two and proceeded to install ElastAlert within it. I configured it to work with my existing elastic stack and success! ElastAlert successfully alerts when running from the command line.

I want this thing to run on start and survive the odd crash so I put together a systemd module to support this my needs and easy orchestration. The *.service file is as follows:

[Unit]
Description=ElastAlert
Documentation=https://elastalert.readthedocs.io
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
Restart=always
User=easervice
Group=easervice
WorkingDirectory=/home/easervice
ExecStart=/home/easervice/.local/bin/pipenv run /home/easervice/.local/bin/python /home/easervice/.local/share/virtualenvs/easervice-r3wr-03C/bin/elastalert --verbose --config /etc/elastalert/config.yaml

[Install]
WantedBy=multi-user.target

The logs are:

Jun 16 21:56:30 host systemd[27895]: elastalert.service: Failed at step CHDIR spawning /usr/bin/elastalert: No such file or directory
Jun 16 21:56:30 host systemd[1]: Started elastalert.
Jun 16 21:56:30 host systemd[1]: elastalert.service: Main process exited, code=exited, status=200/CHDIR
Jun 16 21:56:30 host systemd[1]: elastalert.service: Unit entered failed state.
Jun 16 21:56:30 host systemd[1]: elastalert.service: Failed with result 'exit-code'.

The executable exists in the same space within the venv as outside.

Launching subshell in virtual environment…
easervice@host:~$  . /home/easervice/.local/share/virtualenvs/easervice-r3wr-03C/bin/activate
(easervice) easervice@host:~$ which elastalert
/home/easervice/.local/share/virtualenvs/easervice-r3wr-03C/bin/elastalert
(easervice) easervice@host:~$

200/CHDIR generally stems from 2 things based on my reading:

  1. Bad permissions
  2. File does not exist

Things I have tried to get ExecStart to complete:

  • define different pythons by absolute path
  • Symlinking my elastalert executable into /usr/bin/elastalert
  • changing user to root
  • confirmed that the ExecStart command works from both easervice and root

Sources I used to get started:

I have run out of ideas on what to weak and have not found great success via my google foo. Any idea how to make this go?

UPDATE 1: Attempted to add an environment variable to direct PYTHONPATH: Environment="PYTHONPATH=/home/elastalert/.local/lib/python2.7/site-packages" Resulted in the same error.

Attempted letting the pipenv handle paths by removing hardcoded paths and received the same log error output. The config used is as follows:

Description=ElastAlert
Documentation=https://elastalert.readthedocs.io
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
Restart=always
User=easervice
Group=easervice
Environment="PYTHONPATH=/home/elastalert/.local/lib/python2.7/site-packages"
WorkingDirectory=/home/easervice
ExecStart=/home/easervice/.local/bin/pipenv run elastalert --verbose --config /etc/elastalert/config.yaml

[Install]
WantedBy=multi-user.target

Out of curiosity, I also installed it through pip straight to the user and got the same log output, despite it ALSO working from the command line. The file used for that attempt is as follows:

[Unit]
Description=ElastAlert
Documentation=https://elastalert.readthedocs.io
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
Restart=always
User=easervice
Group=easervice
ExecStart=/home/easervice/.local/bin/elastalert --verbose --config /etc/elastalert/config.yaml

[Install]
WantedBy=multi-user.target

Update 2: I attempted running:

ExecStart=/bin/sh -c '/home/easervice/.local/bin/elastalert --verbose --config /etc/elastalert/config.yaml'

and

ExecStart=/bin/sh -c '/home/easervice/.local/bin/pipenv run elastalert --verbose --config /etc/elastalert/config.yaml'

I received the exact same error. flummoxed

Near
  • 21

2 Answers2

1

Why are you hardcoding all of the paths? To my knowledge, you only need to hardcode the pipenv path: the point of pipenv is that it'll take care of everything else.

Try the following in your ExecStart:

$HOME/.local/bin/pipenv run elastalert --verbose --config /etc/elastalert/config.yaml

When you use pipenv run, it activates a new virtualenv. That environment should know where elaststart is if you used the pip provided by pipenv to install it. This is assuming that the elastalert executable has a shebang line that directs it to /usr/bin/env python, which will direct it to the appropriate interpreter within your environment. Check this; if not, just add python before 'elastalert'.

Let me know if that works!

Edit: Forgot to mention - replace $HOME with the abspath to your user's root. If you're running systemd as root, its $HOME environment variable will resolve to something else.

Hari Amoor
  • 111
0

Summary There was a preinstalled version of the elastalert.service acting as a stub. I was editing my own file, but it was running a different file altogether.

I cleaned out the old file and loaded it into:

/lib/systemd/system/elastalert.service

And symlinked that to:

/etc/systemd/system/elastalert.service

Story Time It could only have been the silliest possible problem. Confounded by what could possibly have been causing me to not get this to work, I pulled through some old debian bugs. I found https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=856924 : a bug where the same error was returned for a working directory issue. I tested with the same file used:

[Unit]
Description=lala

[Service]
WorkingDirectory=/does/not/exist
ExecStart=/bin/sh -c "echo ciao"

[Install]
WantedBy=multi-user.target

Sure enough it wouldn't run. It wasn't in the correct location. It was, however, in the same location as my last *.service file.

Curious, to see where else my service file might have been, I ran:

easervice@host:~$ sudo find / -type f -iname elastalert.service
[sudo] password for easervice: 
/home/easervice/elastalert.service
/lib/systemd/system/elastalert.service

It was reading not from my symlink, but a totally different file. As usual, it was something silly done due to inexperience. May this be a cautionary tale to someone.

Near
  • 21