I have Django app in Production working together with Celery and Amazon SQS. Every day in my celery logs I can see that there was SSL error:
[ERROR/MainProcess] Empty body: SQSError: 599 gnutls_handshake() failed: An unexpected TLS packet was received.
which follows by next error while trying to reconnect to broker:
[2016-12-14 16:06:28,917: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 584, in start
c.loop(*c.loop_args())
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/hub.py", line 284, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/hub.py", line 137, in fire_timers
entry()
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/timer.py", line 68, in __call__
return self.fun(*self.args, **self.kwargs)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/timer.py", line 127, in _reschedules
return fun(*args, **kwargs)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/http/curl.py", line 108, in _timeout_check
self._process_pending_requests()
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/http/curl.py", line 132, in _process_pending_requests
self._process(curl, errno, reason)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/http/curl.py", line 178, in _process
buffer=buffer, effective_url=effective_url, error=error,
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 146, in __call__
svpending(*ca, **ck)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 139, in __call__
return self.throw()
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 136, in __call__
retval = fun(*final_args, **final_kwargs)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/funtools.py", line 100, in _transback
return callback(ret)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 139, in __call__
return self.throw()
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/promises.py", line 136, in __call__
retval = fun(*final_args, **final_kwargs)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/funtools.py", line 98, in _transback
callback.throw()
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/vine/funtools.py", line 96, in _transback
ret = filter_(*args + (ret,), **kwargs)
File "/home/ubuntu/virtualenvs/env/lib/python3.5/site-packages/kombu/async/aws/connection.py", line 269, in _on_list_ready
raise self._for_status(response, body)
boto.exception.SQSError: SQSError: 599 gnutls_handshake() failed: An unexpected TLS packet was received.
Sometimes queues are crashing after this message and I have to restart my Celery workers. In general I am not sure if Celery tasks are running and executed correctly after I get this error.
What is the best way to solve this issue?
