How can I use docker compose on AWS Batch?

Question

I have a multi-container (docker compose) application. I would like to scale it offline on AWS Batch for processing large volumes of data on S3. My .yml file for docker compose looks something like this:

version: '2'

services:
    container1:
      container_name:
      image:
      ports:

    container2:
      container_name:
      image:
      depends_on: container1
      ports:

I, unfortunately, cannot find any examples or tutorials online dealing with such a case. Can anyone help me understand how should I approach this problem?

score 0 · Answer 1 · answered Jun 11 '18 at 13:42

Date: 11-06-2018:

Currently, it is not possible, if it is a multi-container docker compose, you can't. There's no option for a docker-compose up instruction, just a single docker image is allowed. The definition would be like next:

"Properties" : {
    "Type" : "container",
    "Parameters" : {
      "net": "host"
    },
    "ContainerProperties" : {
      "Command" : [ "command1" ],
      "Memory" : 2000,
      "Environment": [
          {
            "Name" : "NAME",
            "Value" : "true"
          }
      ],
      "Vcpus" : 2,
      "Image" : "redis:latest"
    },
    "JobDefinitionName" : "job-1",
    "RetryStrategy" : {
        "Attempts" : 1
    }
}

In fact in the official docs is specified container as the only valid value for type:

https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#type

Type type When you register a job definition, you specify the type of job. At this time, only container jobs are supported.

Type: String

Valid values: container

Required: Yes

For individual containers there's a cli tool to transform docker-compose to a job definition but actually, that's something you can do manually:

https://github.com/turnerlabs/compose-to-batch

Thanks for this answer. Very unfortunate that batch doesn't support `docker-compose up...` They force us to use docker for this service, but then we have to pass env variables and into the job anyway which IMO eliminates one of the better features of docker (bundling `.env` file and calling `docker-compose`) — Adam Hughes, May 06 '20 at 14:44
Does they support docker compose now? Have u solved this problem? What was your solution — Senthil Kumar Vaithiyanathan, Feb 15 '22 at 11:08

John R · Answer 2 · 2023-04-22T04:30:20.370

I'm not sure what the capabilities were when this was posted but I recently had a project for this issue.

{
  "jobDefinitionName": "batch-job-definition",
  "containerProperties": {
    "image": "public.ecr.aws/docker/library/docker:20-dind",
    "command": ["docker-compose","-f", "/mnt/efs/docker-compose.yml","up"],
    "volumes": [
      {
        "efsVolumeConfiguration": {
          "fileSystemId": "fs-XXXXXXXXXXXXXXXXX",
          "rootDirectory": "/",
          "transitEncryption": "ENABLED",
          "authorizationConfig": {
            "accessPointId": "fsap-XXXXXXXXXXXXXXXXX",
            "iam": "ENABLED"
          }
        }
      }
    ],
    "mountPoints": [
      {
        "containerPath": "/mnt/efs",
        "readOnly": false,
        "sourceVolume": "efs-volume"
      }
    ],
    "user": "root",
    "networkConfiguration": {
      "assignPublicIp": "ENABLED"
    }
  },
  "platformCapabilities": [
    "EC2"
  ],
  "containerOrchestrationType": "ECS"
}

How can I use docker compose on AWS Batch?

2 Answers2

Date: 11-06-2018: