Terraform + DynamoDB: All attributes must be indexed

Question

I want to create a Terraform configuration for DynamoDB table with multiple (> 10) attributes. And I have no need to add all attributes as an index to global_secondary_index or local_secondary_index. But when I run terraform plan command I have next error:

All attributes must be indexed. Unused attributes: ...

I found the validation check in the Terraform repository in validateDynamoDbTableAttributes function.

But also as I know the best practice is that each table in DynamoDB is limited to a maximum of five global secondary indexes and five local secondary indexes from General Guidelines for Secondary Indexes in DynamoDB.

And since I have more than 10 attributes it looks like a problem to me.

What I would like to understand why all attributes must be indexed and what to do in case if you have a big number of attributes.

Thanks!

Eric M. Johnson · Accepted Answer · 2018-10-16T17:00:48.690

You do not have to define every attribute you want to use up front when creating your table.

attribute blocks inside aws_dynamodb_table resources are not defining which attributes you can use in your application. They are defining the key schema for the table and indexes.

For example, the following Terraform defines a table with only a hash key:

resource "aws_dynamodb_table" "test" {
  name           = "test-table-name"
  read_capacity  = 10
  write_capacity = 10
  hash_key       = "Attribute1"

  attribute {
    name = "Attribute1"
    type = "S"
  }
}

Every item in this table has Attribute1, but you can create additional attributes with your application

This means that you can have your 10+ attributes as long as you don't need to define them in an AttributeDefinition, and since you say you don't need them to be indexed, you'll be fine.

For some discussion of the confusion (attribute is confusing and doesn't match the DynamoDB API), see this pull request.

Thank you for underscoring the fact that this Terraform code block only defines _key_ attributes. I was running into the same problem and understanding that nuance helped me solve my own problem just now. — Mass Dot Net, Nov 19 '19 at 20:26
Thanks, this was exactly what I needed. I'm super new to DynamoDB and recreating someone else's table with Terraform. While I was writing all the attributes in my terraform code itself I wondered, isn't this supposed to be NoSQL? Then why do I have to basically define the equivalent of a SQL table schema up front? — hyperwiser, Sep 17 '20 at 23:41

JohnOpincar · Answer 2 · 2020-07-29T16:08:39.897

I was adding a projected field to an LSI and put an attribute entry in for the projected field. This was causing the error. You can just list the projected field in the non_key_attributes as shown below, it doesn't have to be defined as an attribute since it's not a key:

local_secondary_index {
  name = "allocated_plus_created-lsi"
  range_key = "allocated_plus_created"
  projection_type = "INCLUDE"
  non_key_attributes = ["allocated"]
}

Allocated shouldn't be defined as an attribute for TF.

score 2 · Answer 3 · answered Jan 18 '23 at 01:56

DynamoDB is a schema-less database, meaning you do not need to define all the attributes on creation, but you do need to be very explicit about the primary attributes and indexes at the beginning. Each declared attribute need an explicit index hence the error All attributes must be indexed.

For example, the below code will require Email and Timestamp on any new entries:

provider "aws" {
  region = var.region
}

module "dynamodb_table_1" {
  source              = "../../"
  name                = "Subscribed_Users"
  hash_key            = "Email"
  range_key           = "Timestamp"
  enable_autoscaler   = false
  dynamodb_attributes = [
    {
      name = "Email"
      type = "S"
    }
  ]

  local_secondary_index_map = [
    {
      name               = "EmailAndTimestampSortIndex"
      hash_key           = "Email"
      range_key          = "Timestamp"
      projection_type    = "INCLUDE"
      non_key_attributes = ["Email", "Timestamp"]
    },
  ]

  context = module.this.context
}

Below are both valid inserts (as it contains both email and timestamp):

First Entry

{
  "Email": {
    "S": "foo@foo.com"
  },
  "Timestamp": {
    "S": "Wed, 18 Jan 2023 01:45:40 GMT"
  },
  "Address": {
    "M": {
      "Country": {
        "S": "Australia"
      }
    }
  },
  "Name": {
    "S": "Melchor"
  }
}

Second Entry

{
      "Email": {
        "S": "foo2@foo.com"
      },
      "Timestamp": {
        "S": "Wed, 18 Jan 2023 01:45:40 GMT"
      },
      "Name": {
        "S": "Bob"
      }
}

Inserted Values:

For an extensive explanation on what hash and range keys are see: https://stackoverflow.com/a/27348364/2023728

Terraform + DynamoDB: All attributes must be indexed

3 Answers3