Systemd Refuses to Start Redis Server

Question

I have duplicate redis.conf & /etc/systemd/system/redis.service files on my test and prod servers via scp.

Where prod is able to start and run redis with a unix socket. My test server refuses to start redis, with the following error output:

● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-07-31 01:42:29 HKT; 27s ago
       Docs: http://redis.io/documentation,
             man:redis-server(1)
redis-server.service: Scheduled restart job, restart counter is at 5.
Stopped Advanced key-value store.
redis-server.service: Start request repeated too quickly.
redis-server.service: Failed with result 'exit-code'.
Failed to start Advanced key-value store.

I have read many threads related to restart counter is at 5, but none of them have been specific to redis.service.

/etc/systemd/system/redis.service contains:

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)
[Service]
Type=forking
ExecStart=/usr/bin/redis-server --loglevel verbose /etc/redis/redis.conf
PIDFile=/run/redis/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis-socket
RuntimeDirectory=redis
RuntimeDirectoryMode=2755
UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWritePaths=-/var/lib/redis
ReadWritePaths=-/var/log/redis
ReadWritePaths=-/var/run/redis
NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
MemoryDenyWriteExecute=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
redis-server can write to its own config file when in cluster mode so we
permit writing there by default. If you are not using this feature, it is
recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/redis
[Install]
WantedBy=multi-user.target
Alias=redis.service

But when I run systemctl restart redis along with tail -f /var/log/redis/redis-server.log

Nothing is written to the log.... nothing.

This log command journalctl -f -u redis-server gave output journalctl -f -u redis-server ... seems redis doesn't like my custom group redis-socket (members redis and a virtuamin server owner called server-owner, necessary to give server owner access to unix socket, works on Prod...previously worked on Test. Better way to config/define the group? EDIT Problem solved. The core problem was that root had somehow become the owner of /etc/redis correcting ownership on that directory to redis:redis got things working again with my custom group as configured below.

-- Logs begin at Sun 2022-07-31 07:37:12 HKT. --
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:55:09 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5.
Jul 31 08:55:09 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:55:09 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:55:09 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:55:13 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:55:13 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:55:13 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8330]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8330]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 1.
Jul 31 08:56:36 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8333]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8333]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 2.
Jul 31 08:56:36 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8336]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8336]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 3.
Jul 31 08:56:36 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:36 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:36 test.example.test systemd[8339]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:36 test.example.test systemd[8339]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:36 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:36 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 4.
Jul 31 08:56:37 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: Starting Advanced key-value store...
Jul 31 08:56:37 test.example.test systemd[8342]: redis-server.service: Failed to determine group credentials: No such process
Jul 31 08:56:37 test.example.test systemd[8342]: redis-server.service: Failed at step GROUP spawning /usr/bin/redis-server: No such process
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Control process exited, code=exited, status=216/GROUP
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:37 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Scheduled restart job, restart counter is at 5.
Jul 31 08:56:37 test.example.test systemd[1]: Stopped Advanced key-value store.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:56:37 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:37 test.example.test systemd[1]: Failed to start Advanced key-value store.
Jul 31 08:56:43 test.example.test systemd[1]: redis-server.service: Start request repeated too quickly.
Jul 31 08:56:43 test.example.test systemd[1]: redis-server.service: Failed with result 'exit-code'.
Jul 31 08:56:43 test.example.test systemd[1]: Failed to start Advanced key-value store.

lil' help?

Answer 1

Ok problem solved.. thanks to the help of @mashuptwice and a good night cleep...

Cause of problem:
Me. I work in multiple terminals and accidentally opened /etc/redis.conf in nano in 2 different terminal windows. While I caught part of my error and deleted .redis.conf.swp file that was created, I simply didn't notice, that the original /etc/redis.conf had changed its permissions to be root/root instead of redis/redis.

The tail -f /var/log/redis/redis-server.log threw no useful error mesages at all to indicate the problem above.

Solution:
The journalctl -f -u redis-server provided by @mashuptwice enabled to the permissions errors beginning with my redis-server.service config and eventually leading me to to find the permissions error on /etc/redis.conf.

In a nutshell, the native redis log, does not seem to very useful when troubleshooting, use journalctl -f -u redis-server if have issues with redis.