asyncio client server does not work inside docker

Question

I have client and server (send json messages over http) works ok on my standart ubuntu 16.04

But then I try to run it client and server inside docker or client outside docker and server inside i got errors.

My docker command:
sudo docker run -p 127.0.0.1:8888:8888 -i -t seo_server

Here is my server and client code and errors:

server

import asyncio
import json
import aiohttp
import re

async def get_cy(domain):           
    return {'result': 'ok','value':10}



async def handle(reader, writer):
    # data = await reader.read(1000)

    data = bytearray()
    while not reader.at_eof():
        chunk = await reader.read(2 ** 12)
        data += chunk

    # print(json.loads(data))
    # https://aiomas.readthedocs.io/en/latest/
    message = data.decode()

    addr = writer.get_extra_info('peername')
    print("Received %r from %r" % (message, addr))
    m = json.loads(message)
    f = m['method']
    del m['method']
    r = await globals()[f](**m)
    r = json.dumps(r)
    # await asyncio.sleep(int(m['time']))
    print("Send: %r" % r)
    writer.write(r.encode())
    await writer.drain()

    print("Close the client socket")
    writer.close()



if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    coro = asyncio.start_server(handle, '127.0.0.1', 8888, loop=loop)
    server = loop.run_until_complete(coro)
    # Serve requests until Ctrl+C is pressed
    print('Serving cy microservice on {}'.format(server.sockets[0].getsockname()))
    try:
        loop.run_forever()
    except KeyboardInterrupt:
        pass

    # Close the server
    server.close()
    loop.run_until_complete(server.wait_closed())
    loop.close()

client

import asyncio
import json
from itertools import zip_longest
import time

def to_json(func):
    def wrap(**kwargs):
        message = kwargs
        message['method'] = func.__name__
        print(message)
        return asyncio.ensure_future(tcp_send(json.dumps(message)))

    return wrap


@to_json
def get_cy(domain):
    pass


async def tcp_send(message):
    loop = asyncio.get_event_loop()
    reader, writer = await asyncio.open_connection('127.0.0.1', 8888,
                                                   loop=loop)

    print('Send: %r' % message)
    writer.write(message.encode())
    writer.write_eof()

    data = await reader.read()
    data = data.decode()
    print('Received: %r' % data)

    print('Close the socket')
    writer.close()
    return json.loads(data)


def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
    args = [iter(iterable)] * n
    return zip_longest(*args, fillvalue=fillvalue)


def async_map(loop, f, iterable, chunk_size=2):
    for chunk in grouper(iterable, chunk_size):
        future = asyncio.gather(*(f(param) for param in chunk if param))
        loop.run_until_complete(future)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()


    async def cy(site):
        cy = await get_cy(domain=site)
        print(site + " cy =", cy)
        #update site here

    while True:
        sites = ('site1.ru', 'site2.ru', 'site3.ru', 'site4.ru', 'site5.ru')
        async_map(loop, cy, sites)
        time.sleep(100) #if not sites

error's then i try client and server inside docker:

client error

root@341fdee56d6d:/seo_server# python client.py 
{'domain': 'site1.ru', 'method': 'get_cy'}
{'domain': 'site2.ru', 'method': 'get_cy'}
Send: '{"domain": "site2.ru", "method": "get_cy"}'
Send: '{"domain": "site1.ru", "method": "get_cy"}'
Received: ''
Close the socket
Traceback (most recent call last):
  File "client.py", line 63, in <module>
    async_map(loop, cy, sites)
  File "client.py", line 49, in async_map
    loop.run_until_complete(future)
  File "/usr/local/lib/python3.5/asyncio/base_events.py", line 341, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 276, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 236, in _step
    result = coro.throw(exc)
  File "client.py", line 57, in cy
    cy = await get_cy(domain=site)
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 387, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 287, in _wakeup
    value = future.result()
  File "/usr/local/lib/python3.5/asyncio/futures.py", line 276, in result
    raise self._exception
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 238, in _step
    result = coro.send(value)
  File "client.py", line 36, in tcp_send
    return json.loads(data)
  File "/usr/local/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Task was destroyed but it is pending!
task: <Task pending coro=<tcp_send() running at client.py:30> wait_for=<Future pending cb=[Task._wakeup()]> cb=[Task._wakeup()]>

server error

     sudo docker run -p 127.0.0.1:8888:8888 -i -t seo_server 
Serving cy microservice on ('127.0.0.1', 8888)
Received '{"domain": "site2.ru", "method": "get_cy"}' from ('127.0.0.1', 47768)
http://bar-navig.yandex.ru/u?ver=2&show=31&url=http://site2.ru
Received '{"domain": "site1.ru", "method": "get_cy"}' from ('127.0.0.1', 47770)
http://bar-navig.yandex.ru/u?ver=2&show=31&url=http://site1.ru
Send: '{"result": "ok", "value": "50"}'
Task exception was never retrieved
future: <Task finished coro=<handle() done, defined at seo_server.py:18> exception=ConnectionResetError('Connection lost',)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 238, in _step
    result = coro.send(value)
  File "seo_server.py", line 40, in handle
    await writer.drain()
  File "/usr/local/lib/python3.5/asyncio/streams.py", line 304, in drain
    yield from self._protocol._drain_helper()
  File "/usr/local/lib/python3.5/asyncio/streams.py", line 195, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
Send: '{"result": "ok", "value": "50"}'
Task exception was never retrieved
future: <Task finished coro=<handle() done, defined at seo_server.py:18> exception=ConnectionResetError('Connection lost',)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/asyncio/tasks.py", line 238, in _step
    result = coro.send(value)
  File "seo_server.py", line 40, in handle
    await writer.drain()
  File "/usr/local/lib/python3.5/asyncio/streams.py", line 304, in drain
    yield from self._protocol._drain_helper()
  File "/usr/local/lib/python3.5/asyncio/streams.py", line 195, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

If i try client outside docker i got this client error and no actions at server:

outside client error

/usr/bin/python3.5 /home/se7en/examples/python_3.5/seo_server/client.py
{'method': 'get_cy', 'domain': 'site1.ru'}
{'method': 'get_cy', 'domain': 'site2.ru'}
Send: '{"method": "get_cy", "domain": "site1.ru"}'
Send: '{"method": "get_cy", "domain": "site2.ru"}'
Traceback (most recent call last):
  File "/home/se7en/examples/python_3.5/seo_server/client.py", line 63, in <module>
    async_map(loop, cy, sites)
  File "/home/se7en/examples/python_3.5/seo_server/client.py", line 49, in async_map
    loop.run_until_complete(future)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 373, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 242, in _step
    result = coro.throw(exc)
  File "/home/se7en/examples/python_3.5/seo_server/client.py", line 57, in cy
    cy = await get_cy(domain=site)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 297, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 240, in _step
    result = coro.send(None)
  File "/home/se7en/examples/python_3.5/seo_server/client.py", line 28, in tcp_send
    writer.write_eof()
  File "/usr/lib/python3.5/asyncio/streams.py", line 294, in write_eof
    return self._transport.write_eof()
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 745, in write_eof
    self._sock.shutdown(socket.SHUT_WR)
OSError: [Errno 107] Transport endpoint is not connected

my docker file:

FROM davidjfelix/python3.5
RUN pip3 install aiohttp
ADD . /seo_server
WORKDIR /seo_server
CMD python seo_server.py

docker version:

$ sudo docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Please help to find a problem and fix

Does this answer your question? [From inside of a Docker container, how do I connect to the localhost of the machine?](https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach) — MisterMiyagi, Mar 26 '20 at 16:20
I will look into that writer.write_eof(), it may be closing the entire connection. This may be a docker/iptables issue. — GetLastError, May 04 '20 at 11:15

score 0 · Answer 1 · answered Mar 26 '20 at 16:16

Because of container isolation python threads in different container don't see each other. So for asyncio loop you need to start server and worker/client in the same container. You can do it via launch .sh script as described here https://docs.docker.com/config/containers/multi-service_container/ or same via Supervisord.

score 0 · Answer 2 · answered Jul 26 '20 at 16:03

0

I had this same issue.

async server and async client outside docker refusing to connect, this solved my problem: https://forums.docker.com/t/python-asyncio-container-is-not-receiving-socket-connections/34018

u need to change:

coro = asyncio.start_server(handle, '127.0.0.1', 8888, loop=loop)

to

coro = asyncio.start_server(handle, '0.0.0.0', 8888, loop=loop)

here quoting "currently you just bind the local interface."

answered Jul 26 '20 at 16:03

Aleks J

1
1

Another (bad) solution is to insert `--network host` to `docker run` command. This will disable network isolation for container and loopback of container will be host loopback. – pyhedgehog Oct 20 '21 at 11:41

asyncio client server does not work inside docker

2 Answers2