Permissions to access ElasticSearch from Lambda?

Question

I'm trying to use Elasticsearch for data storage for a Lambda function connected to Alexa Skills Kit. The Lambda works alright without Elasticsearch but ES provides much-needed fuzzy matching.

The only way I've been able to access it from Lambda is by enabling Elasticsearch global access but that's a really bad idea. I've also been able to access from my computer via open access policy or IP address policy. Is there a way to do read-only access via Lambda and read-write via IP?

On IAM I granted my Lambda role AmazonESReadOnlyAccess. On the ES side I tried this but it only worked for IP address:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::NUMBER:root",
          "arn:aws:iam::NUMBER:role/lambda_basic_execution"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:NUMBER:domain/NAME/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:NUMBER:domain/NAME/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "MY IP"
        }
      }
    }
  ]
}

This forum post asks the same question but went unanswered.

garnaat · Accepted Answer · 2016-06-08T15:11:26.163

15

The only way I know of to do this is to use a resource-based policy or an IAM-based policy on your ES domain. This would restrict access to a particular IAM user or role. However, to make this work you also need to sign your requests to ES using SigV4.

There are libraries that will do this signing for you, for example this one extends the popular Python requests library to sign ElasticSearch requests via SigV4. I believe similar libraries exist for other languages.

edited Jun 08 '16 at 15:11

answered Jun 08 '16 at 01:52

garnaat

44,310
7
123
103

Is [this](https://github.com/DavidMuller/aws-requests-auth) the module you mean? Also does my setup look right otherwise for trying to do a hybrid of IAM and IP? – Keith Jun 08 '16 at 02:21
I updated the answer with a link to the project I was referring to. – garnaat Jun 08 '16 at 15:11
Ok got things mostly working now. I couldn't figure out a way to do auth with the Lambda role but instead used a dedicated user. The only issue is that I can't access Kibana from my browser anymore but that's not too big a deal. – Keith Jun 08 '16 at 22:41
Here is a guide from AWS describing how to sign requests using different languages: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html – a.b.d Nov 30 '18 at 22:33
Is there a more thorough guide on how to do this? I'm still somewhat new at managing roles. – Berry Blue Sep 18 '19 at 18:17

Yurii Holskyi · Answer 2 · 2019-07-05T19:32:07.997

Now it's possible from your code with elasticsearch.js. Before you try it, you must install http-aws-es module.

const AWS = require('aws-sdk');
const httpAwsEs = require('http-aws-es');
const elasticsearch = require('elasticsearch');

const client = new elasticsearch.Client({
    host: 'YOUR_ES_HOST',
    connectionClass: httpAwsEs,
    amazonES: {
        region: 'YOUR_ES_REGION',
        credentials: new AWS.EnvironmentCredentials('AWS')
    }
});

// client.search({...})

Of course, before using it, configure access to elasticsearch domain:

score 5 · Answer 3 · edited May 23 '17 at 11:55

For external (outside AWS) access to your Elasticsearch cluster, you want to create the cluster with an IP-based access policy. Something like the below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "<<IP/CIDR>>"
          ]
        }
      },
      "Resource": "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/*"
    }
  ]
}

For your Lambda function, create the role that the Lambda function will assume with the below policy snippet.

{
  "Sid": "",
  "Effect": "Allow",
  "Action": [
    "es:DescribeElasticsearchDomain",
    "es:DescribeElasticsearchDomains",
    "es:DescribeElasticsearchDomainConfig",
    "es:ESHttpPost",
    "es:ESHttpPut"
  ],
  "Resource": [
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/*"
  ]
},
{
  "Sid": "",
  "Effect": "Allow",
  "Action": [
    "es:ESHttpGet"
  ],
  "Resource": [
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_all/_settings",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_cluster/stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/<<INDEX>>*/_mapping/<<TYPE>>",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_nodes",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_nodes/stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_nodes/*/stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/_stats",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/<<INDEX>>*/_stats"
  ]
}

I think you could more easily condense the above two policy statements into the following:

{
  "Sid": "",
  "Effect": "Allow",
  "Action": [
    "es:DescribeElasticsearchDomain",
    "es:DescribeElasticsearchDomains",
    "es:DescribeElasticsearchDomainConfig",
    "es:ESHttpPost",
    "es:ESHttpGet",
    "es:ESHttpPut"
  ],
  "Resource": [
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>",
    "arn:aws:es:<<REGION>>:<<ACCOUNTID>>:domain/<<DOMAIN_NAME>>/*"
  ]
}

I managed to piece the above together from the following sources:

https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/

How to access Kibana from Amazon elasticsearch service?

https://forums.aws.amazon.com/thread.jspa?threadID=217149

score 4 · Answer 4 · edited Jul 04 '17 at 08:08

4

You need to go to the access policy of Lambda and provide the AWS ARN to connect

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-s3-lambda-es-authorizations

edited Jul 04 '17 at 08:08

Romano Zumbé

7,893
4
33
55

answered Jun 19 '17 at 19:07

Shef

57
4

2

For queries to the ElasticSearch document API, this will not work "out of the box." HTTP(s) requests made with your language's HTTP client module will not have the AWS V4 request signature, and will be treated as coming from an anonymous user. You need to explicitly sign your HTTP requests with an AWS signature to have the intended role carry through. – GrandOpener Jul 18 '17 at 20:05

Raheel Kazi · Answer 5 · 2016-08-31T04:12:26.080

1

AWS Lambda runs on public EC2 instances. So simply adding a whitelist of IP addresses to the Elasticsearch access policy will not work. One way to do this will be to give the Lambda execution role appropriate permissions to the Elasticsearch domain. Make sure that the Lambda Execution role has permissions to the ES domain and the ES domain access policy has a statement which allows this Lambda Role ARN to do the appropriate actions. Once this is done all you would have to do is sign your request via SigV4 while accessing the ES endpoint

Hope that helps!

edited Aug 31 '16 at 04:12

answered Aug 25 '16 at 03:55

Raheel Kazi

49
3

1

I'm actually trying to implement the strategy you suggested but I'm having trouble creating the ElasticSearch inline policy to allow my lambda function. Do you have a sample policy? – VirtualProdigy Nov 12 '18 at 17:13

Permissions to access ElasticSearch from Lambda?

5 Answers5