Connection failure envoyproxy grpc transcoded HTTP

Question

I am following this tutorial, in order to have a grcp service transcoded to HTTP (it is running on Linux). The envoy API updated to the v3, so I also followed this example. I have updated the envoy_config to the API v3, so it fits the new requirement of this version.

However, when I deploy the docker image, I get an error when I try to access an API endpoint. For example, when I run this command curl http://localhost:51051/greeting (which is the exposed endpoint of the generated API), I am getting the following error : upstream connect error or disconnect/reset before headers. reset reason: connection failure, with the status code of 503.

When I run the following command sudo docker ps, I can see that there is not port displayed in the Port column. So I tried to map/expose them thanks to the command docker run -p ... or docker run --expose ..., but they still doesn't appears.

I am a bit lost now, since I don't know at all docker, I don't even know if my issue is linked to port exposure.

But anyway, I have to say that the grpc service is working fine with the java Server/Client. It is just the Web API (envoy) which is not working.

If you want more material, I have uploaded the whole project here, so you are able to reproduce the problem by yourself.

Otherwise, here is the envoy-config.yml (the exposed API endpoint is the address #2 : 0.0.0.0:51051):

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }         #1

static_resources:
  listeners:
  - name: main-listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 51051 }      #2
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager # Explicit v3
          stat_prefix: grpc_json
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" , grpc: {}}  # 3a grpc:{} means that requests are only forwarded if they are found in the grpc service definition, returning 404 for others
                route: { cluster: grpc-backend-services, timeout: { seconds: 60 } }   #3b
          http_filters:
          - name: envoy.filters.http.grpc_json_transcoder
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_json_transcoder.v3.GrpcJsonTranscoder
              proto_descriptor: "/data/greeting_service_definition.pb"             #4
              services: ["helloworld.HelloService"]                        #5
              print_options:
                add_whitespace: true
                always_print_primitive_fields: true
                always_print_enums_as_ints: false
                preserve_proto_field_names: false                                     #6
          - name: envoy.filters.http.router

  clusters:
  - name: grpc-backend-services                  #7
    connect_timeout: 1.25s
    type: logical_dns
    lb_policy: round_robin
    dns_lookup_family: V4_ONLY
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: grpc-backend-services
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1                #8
                port_value: 53000

And here is the script that run protoc and docker:

#!/usr/bin/env bash

if ! [ -x "$(command -v protoc)" ] ; then
    echo "you do not seem to have the protoc executable on your path"
    echo "we need protoc to generate a service defintion (*.pb file) that envoy can understand"
    echo "download the precompiled protoc executable and place it in somewhere in your systems PATH!"
    echo "goto: https://github.com/protocolbuffers/protobuf/releases/latest"
    echo "choose:"
    echo "       for linux:   protoc-3.6.1-linux-x86_64.zip"
    echo "       for windows: protoc-3.6.1-win32.zip"
    echo "       for mac:     protoc-3.6.1-osx-x86_64.zip"
    exit 1
fi

# generate the greeting_service_definition.pb file that we can pass to envoy so that knows the grpc service
# we want to expose
protoc -I. -Isrc/main/proto --include_imports \
                --include_source_info \
                --descriptor_set_out=greeting_service_definition.pb \
                src/main/proto/HelloService.proto

if ! [ $? -eq 0 ]; then
    echo "protobuf compilation failed"
    exit 1
fi

# now we can start envoy in a docker container and map the configuration and service definition inside
# we use --network="host" so that envoy can access the grpc service at localhost:<port>
# the envoy-config.yml has configured envoy to run at port 51051, so you can access the HTTP/JSON
# api at localhost:51051


if ! [ -x "$(command -v docker)" ] ; then
    echo "docker command is not available, please install docker"
    echo "Install docker: https://store.docker.com/search?offering=community&type=edition"
    exit 1
fi

# check if sudo is required to run docker
if [ "$(groups | grep -c docker)" -gt "0" ]; then
    echo "Envoy will run at port 51051 (see envoy-config.yml)"
    docker run -it --rm --name envoy --network="host" \
             -v "$(pwd)/greeting_service_definition.pb:/data/greeting_service_definition.pb:ro" \
             -v "$(pwd)/envoy-config.yml:/etc/envoy/envoy.yaml:ro" \
             envoyproxy/envoy:v1.18.2
else
    echo "you are not in the docker group, running with sudo"
    echo "Envoy will run at port 51051 (see envoy-config.yml)"
    sudo docker run -it --rm --name envoy --network="host"\
             -v "$(pwd)/greeting_service_definition.pb:/data/greeting_service_definition.pb:ro" \
             -v "$(pwd)/envoy-config.yml:/etc/envoy/envoy.yaml:ro" \
             envoyproxy/envoy:v1.18.2
fi

Answer 1

The 503 status is because the transcode operation failed - it looks like envoy wasn't able to connect to the gRPC service.

You have configured envoy to connect to gRPC at host.docker.internal:53000 - this may be an issue if you are using docker on linux rather than docker-desktop for Mac/Windows.

But also the java server listens on port 8080, not 53000, so I think you just need to change that.

[Edit]

I confirmed that "host.docker.internal" works with your project on Ubuntu 18.04, calling docker like this: docker run -it --rm --name envoy --add-host=host.docker.internal:host-bridge -p 51051:5105. The port within lb_endpoints needed changed from 53000 to 8080 in the version I tested (f9e7c1).

N.B. that this is using the default "bridge" networking mode, hence a port mapping is needed for e.g. curl to access to the transcoded envoy endpoint.

You can run the envoy container in "host" mode networking - this is a bit simpler as you can curl to envoy's transcoded endpoint via localhost (no port mapping), and envoy can access the java server via localhost IP too.

See this answer for a great summary of methods to handle these kinds of inter-container dependency.