13

I have a laravel 5.4 app deployed via envoyer to a non-forge server. I am running queue workers on the database driver, using supervisor to monitor, setup as described in the docs;

command=php /home/data/app/current/artisan queue:work --sleep=3 --tries=3

and using the envoyer deployment hook 

cd ~/app/current
php artisan queue:restart

Problem is, after each deployment the queue workers are not restarted, the old ones continue to run and then throw errors because they are working on previous releases of the code. Running queue:restart manually from the CLI doesn't work either. 

data@medicone:~/ccpbase/current$ ps -aux | grep queue:work
data      4347  0.0  0.2 292988 34852 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4348  0.0  0.2 292988 34864 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4349  0.0  0.2 292988 34720 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4350  0.0  0.2 292988 34880 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4351  0.0  0.2 292988 34972 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4382  0.0  0.2 292988 34904 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4383  0.0  0.2 292988 34992 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4384  0.0  0.2 292988 34980 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4863  0.0  0.0  14228  1016 pts/0    S+   11:32   0:00 grep queue:work
data@medicone:~/ccpbase/current$
data@medicone:~/ccpbase/current$ php artisan queue:restart
Broadcasting queue restart signal.
data@medicone:~/ccpbase/current$ ps -aux | grep queue:work
data      4347  0.0  0.2 292988 34852 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4348  0.0  0.2 292988 34864 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4349  0.0  0.2 292988 34720 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4350  0.0  0.2 292988 34880 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4351  0.0  0.2 292988 34972 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4382  0.0  0.2 292988 34904 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4383  0.0  0.2 292988 34992 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4384  0.0  0.2 292988 34980 ?        S    11:12   0:00 php /home/data/ccpbase/current/artisan queue:work --sleep=3 --tries=3
data      4869  0.0  0.0  14228   960 pts/0    S+   11:32   0:00 grep queue:work
data@medicone:~/ccpbase/current$

If I find and kill the 8 running processes manually, supervisor does restart them correctly and my queued jobs work again.

Can anybody think of anything that might be preventing these workers from being killed? There is nothing relevant in storage/logs/laravel.log

Mark Salmon
  • 206
  • 1
  • 2
  • 9
  • Try restarting Supervisor. – The Alpha Feb 15 '17 at 14:14
  • 1
    `queue:restart` works by posting something to a cache key, which the workers notice when they finish a job and decide whether to start a new one. it is not an immediate thing, the current jobs need to finish first. if your jobs are long-running then it will appear as if they are not restarting. if you want an immediate restart, use `supervisorctl` instead. – datashaman May 08 '17 at 22:05
  • https://laravel.com/docs/5.4/queues#queue-workers-and-deployment – datashaman May 08 '17 at 22:12
  • which cache backend are you using/ – georgexsh Sep 27 '18 at 09:52
  • @georgexsh I was using redis, but I have since upgraded everything and now using horizon and no longer have this problem – Mark Salmon Sep 28 '18 at 10:44

3 Answers3

12

Its important to understant few points related to following queue:restart command

php artisan queue:restart
  • This command will instruct all queue workers to gracefully "die" after they finish processing their current job so that no existing jobs are lost.
  • The queue uses the cache to store restart signals, so you should verify a cache driver is properly configured for your application before using this feature.
  • Since the queue workers will die when the queue:restart command is executed, you should be running a process manager such as Supervisor to automatically restart the queue workers.

Which means queue:restart will not immediately restart the queue but simply 'Broadcasting queue restart signal' for queue worker to get it die, once it finished the current assigned task.

Ref : https://laravel.com/docs/5.6/queues#queue-workers-and-deployment

Amitesh Bharti
  • 14,264
  • 6
  • 62
  • 62
10

In my case also, php artisan queue:restart did nothing. The queue workers' age confirmed that they were not restarting in response to that. The following will show you when your workers launched. Mine were all different, by several days.

$ ps -eo pid,lstart,cmd | grep queue:work

As part of my deployment script (gitlab, in my case) I am running php artisan queue:restart, so the queue workers restart after any code is pushed live.

As mentioned in a previous answer, the queue:restart command saves to the Laravel cache. I am using the 'file' cache driver, so cache entries are on disk at storage/framework/cache/data by default.

It turned out that my deploy script was running php artisan queue:restart as the gitlab-runner user, and thus any cache entries it created were also owned by that user.

Running the following instantly fixed my problem. The queue workers restarted the moment this was run:

sudo chown -R www-data:www-data storage/framework/cache

After this, commandline invocations of php artisan queue:restart work perfectly.

The long-term fix was to ensure my deploy script ran php artisan queue:restart as the proper user (or chown's the cache files as needed).

tldr; if using the 'file' cache driver, your web server user needs to be able to read/write the cache entry created by the php artisan queue:restart command. Check your permissions!

jeff-h
  • 2,184
  • 1
  • 22
  • 33
  • Also, using the file cache driver means each server will have a different cache, unless they share the cache directory through a network mounted storage or something. That means, if you have several servers, that you'd need to run the queue:restart command for each. – Way Too Simple Mar 23 '23 at 15:58
0

In my case queue:restart didn't do anything. The ps command showed clearly the PID didn't change:

ps -eo pid,lstart,cmd | grep queue:work

It turned out the cache driver configuration was wrong. I fixed that configuration but it kept on failing. I had to manually kill the process with kill PID to make it work again. The problem was that the queue was still using the wrong cache driver configuration. Because queue:restart relies on the cache it doesn't do anything without showing an error.

Martijn Gastkemper
  • 630
  • 7
  • 22