Localstack SNS: Unable to send message to ElasticMq

Question

I have 2 applications:

1 application uses ElasticMq queue for listening to the messages. The 2nd application publishes the messages on an SNS topic.

I am able to subscribe to the ElasticMq queue on the SNS topic. But when I publish on the topic local stack is unable to send the message to elasticmq eventhough subscription was successful.

awslocal sns list-subscriptions-by-topic --topic-arn arn:aws:sns:us-east-1:123456789012:classification-details-topic
{
    "Subscriptions": [
        {
            "SubscriptionArn": "arn:aws:sns:us-east-1:123456789012:classification-details-topic:ea470c5a-c352-472e-9ae0-a1386044b750",
            "Owner": "",
            "Protocol": "sqs",
            "Endpoint": "http://elasticmq-service:9324/queue/test",
            "TopicArn": "arn:aws:sns:us-east-1:123456789012:classification-details-topic"
        }
    ]
}

Below is the error message I receive:

awslocal sns publish --topic-arn arn:aws:sns:us-east-1:123456789012:classification-details-topic --message "My message"

An error occurred (InvalidParameter) when calling the Publish operation: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the SendMessage operation: AWS.SimpleQueueService.NonExistentQueue; see the SQS docs.

Am I wrong in having elasticmq subscribed on local stack?

I am running localstack using docker-compose file

version: '2.1'

services:
  localstack:
    image: localstack/localstack
    ports:
      - "4567-4584:4567-4584"
      - "${PORT_WEB_UI-8001}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=${SERVICES- }
      - DEBUG=${DEBUG- }
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
networks:
  default:
    external:
      name: my_network

I have the elasticmq and other services as a part of different docker-compose using the same docker network "my_network"

The below is the complete docker-compose. I tried reproducing it by combining the entries into one docker-compose file.

Steps to reproduce

version: '3'
services:
  elasticmq:
    build: ./elasticmq
    ports:
      - '9324:9324'
    networks:
      - my_network
    dns:
      - 172.16.198.101
  localstack:
    image: localstack/localstack
    ports:
      - "4567-4584:4567-4584"
      - "${PORT_WEB_UI-8001}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=${SERVICES- }
      - DEBUG=${DEBUG- }
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
    links:
      - elasticmq:elasticmq-service
    networks:
      - my_network
    dns:
      - 172.16.198.101
networks:
  my_network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.16.198.0/24

After this one can run the following set of commands

awslocal sqs create-queue --queue-name test --endpoint http://elasticmq:9324/

awslocal sns create-topic --name test-topic

awslocal sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:test-topic --protocol sqs --notification-endpoin http://elasticmq-service:9324/queue/test

Answer 1

Based on your comments, I would hazard a guess that the networks of your 2 docker-compose files are not setup correctly.

For simplicity's sake I would merge the elasticmq service in the above docker-compose and try again (if you post your second docker-compose and exact aws command to create the subscription someone can try it locally).

If you really want to keep 2 separate docker-compose files then if the above works then at least you can pin point your problem. I'm afraid I am not too familiar with setting this up but this answer might help.

EDIT:

Thanks for the additional details. I have a simplified version of a docker-compose that works for me. First of all according to this you will need to create a config file to set the hostname of your elasticmq instance since it will not pick up the container_name from docker-compose (similar to the HOSTNAME environment variable in LocalStack which I set below as you will see). The contents of this file named elasticmq.conf (in a folder named config) are:

include classpath("application.conf")

node-address {
  host = elasticmq
}

queues {
  test-queue {}
}

With that in place, the following docker-compose publishes the message without any errors:

version: '3'

services:

  elasticmq:
    image: s12v/elasticmq
    container_name: elasticmq
    ports:
      - '9324:9324'
    volumes:
      - ./config/elasticmq.conf:/etc/elasticmq/elasticmq.conf

  localstack:
    image: localstack/localstack
    container_name: localstack
    environment:
      - SERVICES=sns
      - DEBUG=1
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - HOSTNAME=localstack
    ports:
      - "4575:4575"
      - "8080:8080"

  awscli:
    image: garland/aws-cli-docker
    container_name: awscli
    depends_on:
      - localstack
      - elasticmq
    environment:
      - AWS_DEFAULT_REGION=eu-west-2
      - AWS_ACCESS_KEY_ID=xxx
      - AWS_SECRET_ACCESS_KEY=xxx
    command:
      - /bin/sh
      - -c
      - |
          sleep 20
          aws --endpoint-url=http://localstack:4575 sns create-topic --name test_topic
          aws --endpoint-url=http://localstack:4575 sns subscribe --topic-arn arn:aws:sns:eu-west-2:123456789012:test_topic --protocol http --notification-endpoint http://elasticmq:9324/queue/test-queue
          aws --endpoint-url=http://localstack:4575 sns publish --topic-arn arn:aws:sns:eu-west-2:123456789012:test_topic --message "My message"

And the output:

Admittedly at this point I did not check elasticmq to see if the message got consumed but I leave that to you.