12

I'm using a redis docker container to be used as a message/ broker queue for celery and flaskk-socketio in my python app and lately my redis instance is crashing pretty often, and it doesn't come back.

I'm not a redis expert and I couldn't yet find what the problem is, here are the logs, I appreciate any help!

Thanks

[2019-08-09 03:44:29,930: CRITICAL/MainProcess] Unrecoverable error: ResponseError('UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)',)
Traceback (most recent call last):
  File "/server/env/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/server/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 322, in start
    blueprint.start(self)
  File "/server/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/server/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 598, in start
    c.loop(*c.loop_args())
  File "/server/env/lib/python3.6/site-packages/celery/worker/loops.py", line 121, in synloop
    connection.drain_events(timeout=2.0)
  File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 315, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events
    get(self._deliver, timeout=timeout)
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 370, in get
    ret = self.handle_event(fileno, event)
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 352, in handle_event
    return self.on_readable(fileno), self
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 348, in on_readable
    chan.handlers[type]()
  File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 725, in _brpop_read
    **options)
  File "/server/env/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/server/env/lib/python3.6/site-packages/redis/connection.py", line 629, in read_response
    raise response
redis.exceptions.ResponseError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
[2019-07-15 07:04:53,803: WARNING/MainProcess] Traceback (most recent call last):
[2019-07-15 07:04:53,803: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 109, in wait
    listener.cb(fileno)
[2019-07-15 07:04:53,803: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/celery/worker/pidbox.py", line 122, in loop
    pass
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 798, in __exit__
    self.release()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 372, in release
    self._close()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 338, in _close
    self._do_close_self()
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/connection.py", line 331, in _do_close_self
    self.transport.close_connection(self._connection)
[2019-07-15 07:04:53,804: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 953, in close_connection
    channel.close()
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 851, in close
    self.queue_delete(queue, client=client)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 542, in queue_delete
    self._delete(queue, exchange, *meta, **kwargs)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 810, in _delete
    queue or '']))
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 1661, in srem
    return self.execute_command('SREM', name, *values)
[2019-07-15 07:04:53,805: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 668, in execute_command
    return self.parse_response(connection, command_name, **options)
[2019-07-15 07:04:53,806: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
[2019-07-15 07:04:53,806: WARNING/MainProcess] File "/server/env/lib/python3.6/site-packages/redis/connection.py", line 629, in read_response
    raise response
[2019-07-15 07:04:53,806: WARNING/MainProcess] redis.exceptions.ReadOnlyError: You can't write against a read only replica.
[2019-07-15 07:04:53,806: WARNING/MainProcess] Removing descriptor: 19
magnoz
  • 1,939
  • 5
  • 22
  • 42
  • 2
    Any solution ? I also facing similar issue. Thanks – Zahid Irfan Aug 18 '19 at 16:05
  • Not yet, also tried to scale to 2 replicas in a swarm, still sometimes I get this error from python: `redis.exceptions.ConnectionError: Error 113 connecting to outflink_redis:6379. EHOSTUNREACH.` – magnoz Aug 21 '19 at 07:01
  • i'm facing same issue in django with celery and that is dockerized as well. This happen after error occur in run task run by celery beat. Any luck? are you able to resolve it? – Ammad Khalid Nov 13 '19 at 04:05

1 Answers1

0

Crashes in celery code can happen for various reasons (redis downtime/maintenance, bugs etc), the key is to make sure your celery process recovers from these errors.

If you're using docker-compose you can add restart: unless-stopped as recommended in this doc. You can learn more about docker-compose restart policies here.

With this option docker-compose will restart the container if your celery process crashes for whatever reason.

salmin
  • 457
  • 3
  • 12