32

I have a job that works flawless locally, but in production I run into issues where it doesn't work. I've encompassed the entire handle() with a try/catch and am not seeing anything logged to Bugsnag, despite many other exceptions elsewhere from being deployed. 

public function handle() {
    try {

        // do stuff

    } catch (\Exception $e) {
        Bugsnag::notifyException($e);

        throw $e;
    }
}

According to Laravel Horizon this queue job runs for 0.0026001930236816406 seconds and I never see it work and never see any other errors in the failed_jobs table as it relates to this job.

config/queue.php

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default',
        'retry_after' => (60 * 10), // 10 minutes
        'block_for' => null,
    ],

config/horizon.php

'environments' => [
    'production' => [
        'supervisor'        => [
            'connection'    => 'redis',
            'queue'         => [
                'default',
            ],
            'balance'       => 'auto',
            'processes'     => 10,
            'tries'         => 3,

            // 10 seconds under the queue's retry_after to avoid overlap
            'timeout'       => (60 * 10) - 10, // Just under 10 mins
        ],

If something is causing this job to retry over and over, how can I find out how? I'm at a loss.

Investigation thus far

  • My expectation is I should be able to run the query:
SELECT DISTINCT exception, COUNT(id) as errors
FROM failed_jobs 
WHERE payload LIKE '%[TAG-JOB-HAS]%' 
GROUP BY exception;

To see more than this error message:

Job has been attempted too many times or run too long

but that's all I see.

  • Laravel Horizon's dashboard shows the job in question is running for < 1 second, so I know it's not actually timing out.
Kirk Beard
  • 9,569
  • 12
  • 43
  • 47
Ben
  • 60,438
  • 111
  • 314
  • 488
  • 1
    Have you tried logging debug statements to prove that the job has actually been run? Can you include more information about what the job actually does, and maybe some code? It could be failing in a way that doesn't generate exceptions – Travis Britz Nov 04 '18 at 01:52
  • 1
    "...but in production I run into issues where it doesn't work" - what doesnt work exactly? Do you see an error in the logs? – Laurence Nov 04 '18 at 08:57
  • Just a thought, is it the `Bugsnag::notifyException($e)` line that is causing an exception, causing Laravel to requeue your job to retry? – Jamesking56 Nov 05 '18 at 17:27
  • What is "`// do stuff`"? Does it do anything weird like call exit() or die()? – Travis Britz Nov 06 '18 at 11:28
  • Did you restart the queue after adding the try catch to the handle function? – Elie Nov 06 '18 at 13:28
  • I got this error too, and my queue setup add `tries` options: `php artisan queue:work --tries=3` and you should check this on document https://laravel.com/docs/5.5/queues#failed-job-events – bangnokia Nov 06 '18 at 18:10
  • Try catching `\Throwable` instead of `\Exception`. `\Exception` won't catch internal PHP errors. Does that get you anything? Also, as mentioned, make sure it isn't Bugsnag causing the problem. – patricus Nov 08 '18 at 02:30

11 Answers11

36

I had the same problem

I fixed it by increasing the 'retry_after' parameter

make sure the retry_after value is greater than the time it takes a job to run

in config/queue.php file

    'connections' => [

    'sync' => [
        'driver' => 'sync',
    ],

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 9000,
    ],
Yasser
  • 1,159
  • 1
  • 14
  • 19
  • 8
    Do you have a clue why the retry_after parameter doesn't effect the queue on my local machine but does effect the queue on the server? – mark DE Jong Dec 18 '19 at 14:28
  • @markDEJong you might be using a different connection on your local - eg. database on server and redis on local – ejntaylor Aug 29 '22 at 08:23
  • I am using redis for both local and production environments and am only witnessing this problem in production – danronmoon Jul 05 '23 at 13:25
  • Only difference was I was using --tries=3 on local for queue:work command and --tries=1 in production. Changing to --tries=3 in production fixes it but I'm not sure exactly what "tries" means; the job doesn't retry itself and has no obvious or discernible reason to fail. – danronmoon Jul 05 '23 at 14:10
22

Try to catch the exception in the failed method given by laravel

/**
* The job failed to process.
*
* @param  Exception  $exception
* @return void
*/
public function failed(Exception $exception)
{
    // Send user notification of failure, etc...
}

and check whether your default queue driver in local is sync then its expected behavior.

Vaibhavraj Roham
  • 1,138
  • 11
  • 26
Anil Kumar
  • 814
  • 6
  • 7
  • 1
    Can you please give more specifics? Like where this function can be created and what documentation are you are referring too? Or do you manually execute it from AppServiceProvider? – Yevgeniy Afanasyev Oct 14 '21 at 02:23
  • 2
    @YevgeniyAfanasyev the method is placed in your job class that uses Queueable. There's no abstract method or stub, just add it. Laravel finds it by reflection. – MightyPork Dec 21 '21 at 15:19
9

According to documentation, you can handle job failing in two common ways:

  • using failed job events
  • using failed() method.

In the first case, you can handle all jobs using Queue::failing() method. You'll receive Illuminate\Queue\Events\JobFailed event as a parameter, and it contains exception.

In another case, you can use failed() method, it should be placed near your handle() method. You can receive Exception $exception as a parameter too.

Example:

public function failed(\Throwable $exception)
{
    // Log failure
}

Hope this helps.

Vaibhavraj Roham
  • 1,138
  • 11
  • 26
a_sarana
  • 512
  • 3
  • 13
2

If you've seen this MaxAttemptsExceededException in your error logs or failed_jobs table and you don't have a clue what happened to the job, let me try to explain what may have happened. It's either:

The job timed out and it can't be attempted again.
The job was released back to the queue and it can't be attempted again.

If your job processing time exceeded the timeout configuration, the worker will check the maximum attempts allowed and the expiration date for the job and decide if it can be attempted again. If that's not possible, the worker will just mark the job as failed and throw that MaxAttemptsExceededException.

Also if the job was released back to the queue and a worker picks it up, it'll first check if the maximum attempts allowed was exceeded or the job has expired and throw MaxAttemptsExceededException in that case.

https://divinglaravel.com/job-has-been-attempted-too-many-times-or-run-too-long

levi
  • 1,566
  • 3
  • 21
  • 37
2

Perhaps this will help someone: don't use dd() in queued tasks.

Arthur Shlain
  • 961
  • 10
  • 23
2

I had the same problem

I resolved the issue by using the below code in Job class.

public $failOnTimeout = false;

It will continue if there is timeout or fail job. I also increased timeout time.

public $timeout = 120000;

Reference : https://laravel.com/docs/9.x/queues#failing-on-timeout

Yasin Patel
  • 5,624
  • 8
  • 31
  • 53
1

I was finally able to fix it. This is the solution:

You have to set retry_after (in config/queue) and timeout (in config/horizon).

The two values ​​work together. Either value throws the exception "has been attempted too many times or run too long. The job may have previously timed out."

config/queue.php:

'redis' => [
    'driver' => 'redis',
    'connection' => 'default',
    'queue' => env('REDIS_QUEUE', 'default'),
    'retry_after' => 60 * 7, // always greater than retry_after
    'after_commit' => true, // check this if you dispatch jobs/events inside of DB transactions.
    'block_for' => null,
],
/* ... */

config/horizon.php:

'defaults' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => ['default'],
        /* ... */
        'timeout' => 60 * 5, // always lower than retry_after
    ],
    /* ... */

Explanation: Think of the retry_after value as something global control, a process that every so often checks if there are any jobs still queued. While timeout is precise and is applied at the time of launching the job (It is precisely the --timeout flag of the horizon work command). Therefore, timeout should always be smaller than retry_after. And retry_after should be the maximum value that any job on that connection takes. (More on Job expiration on official documentation).

Is not mandatory, but if you don't want to give high times but you have particularly long jobs, use an exclusive queue with more time (check this issue).

Also, check if you don't have any infinite loops. For example, related with Model observers. Some times Model1 observer touch a Model2 and fire an observer. That Model 2 observer touch again Model 1, and Model 1 observer is fired again. You never receive specific error log about this situation, only a "has been attempted too many times..."

pablorsk
  • 3,861
  • 1
  • 32
  • 37
1

This is a complex issue to work with. That happens when you have a huge data to handle with or working with a slow API, slow/weak server or else, so you have to tune up several things:

  • webserver (Nginx for sure)
  • PHP-CLI
  • Laravel
  • Supervisor

You need to test execution timeout by yourself. In this case let's make 10000 (seconds)

Nginx config

proxy_read_timeout directive - you can increase this value for a needle timeout one in your nginx config.

nano /etc/nginx/sites-available/mysite.com.conf

proxy_read_timeout 10000

PHP-CLI config

memory_limit directive that works with RAM. You can turn off all limits for PHP-CLI scripts with value of -1

max_execution_time directive, time in seconds allowed for script execute. Optional, but you can test up increasing of this value.

nano /etc/php/8.1/cli/php.ini

memory_limit=-1

max_execution_time=10000

Laravel

Timeouts. Increase timeout time of your Queue Job/Task. Also you can turn off failing on timeout

<?php

namespace App\Jobs;

class MyJobTask implements ShouldQueue
{
    /**
     * The number of seconds the job can run before timing out.
     *
     * @var int
     */
    public $timeout = 10000;

    /**
     * Indicate if the job should be marked as failed on timeout.
     *
     * @var bool
     */
    public $failOnTimeout = false;
}

Supervisor

Supervisor is a service to control queue processes on a server side. Install and tune up supervisor config for your Laravel project. But don't forget to provide --timeout option with a command in your config.

command=/usr/bin/php /var/www/app.com/artisan queue:work database --timeout=10000

Igor
  • 755
  • 1
  • 10
  • 22
0

This solved my issue php artisan queue:work --timeout=600 --tries=30

SAm Vice
  • 1
0

https://laravel.com/docs/9.x/queues#timeout

You can set the number of seconds allowed per each job before an exception

php artisan queue:work --timeout=300

Zahit Rios
  • 111
  • 6
0

I was having the same issue, but the cause was different than most answers here.

I was using a Heroku free tier Redis instance, which supports up to 30 connections, and this was not a problem, until i started using supervisor to run multiple queue workers at once.

It seems that the jobs were failing because the max connections were being reached when connecting to add data to the database and working on it at once.

I upgraded to the second tier with 256 connections and the issue is now gone.

Edit: The error was still occuring, it was because I was using the WithoutOverlapping middleware in my job, and the $key I was giving it wasn't unique due to a mistake in my code, make sure that the $key is unique if you're also using it.

Romek
  • 284
  • 2
  • 10