The correct way to query DynamoDb table with .net SDK

Question

I'm trying to understand how to query a table in dynamo using the DataModel. But, I found two ways that seems to work and I can't find an explanation or documentation of what's happening or if there is any difference between them.

The ways of doing it that I found is using either the Filter or the KeyExpression + FilterExpression. What is the difference and correct way of doing it?

Some examples:

Option 1:

-- With Index and Key

public async Task<List<T>> Find<T>(Guid id)
{
    var query = new QueryOperationConfig
    {
        IndexName = "Table_Id_Index",
        Filter = new QueryFilter("TableId", QueryOperator.Equal, id)
    };

    return await _dynamoDbContext
        .FromQueryAsync<T>(query)
        .GetRemainingAsync();
}

-- With Index, Key and extra filtering

public async Task<List<T>> Find<T>(Guid id)
{
    var query = new QueryOperationConfig
    {
        IndexName = "Table_Id_Index",
        Filter = new QueryFilter("TableId", QueryOperator.Equal, id)
    };

    query.AddCondition("Deleted", ScanOperator.NotEqual, true);

    return await _dynamoDbContext
        .FromQueryAsync<T>(query)
        .GetRemainingAsync();
}

-- With GSI, Key and Partion

public async Task<List<T>> Find<T>(Guid id, string partitionKey)
{
    var query = new QueryOperationConfig
    {
        IndexName = "GSI_Index",
        Filter = new QueryFilter("TableId", QueryOperator.Equal, id)
    };

    query.AddCondition("PartitionKey", QueryOperator.Equal, partitionKey);

    return await _dynamoDbContext
        .FromQueryAsync<T>(query)
        .GetRemainingAsync();
}

Option 2:

-- With Index and Key

public async Task<List<T>> Find<T>(Guid id)
{
    var expressionAttributeValues = new Dictionary<string, DynamoDBEntry>();
    expressionAttributeValues.Add(":v_TableId", id);

    var queryOperationConfig = new QueryOperationConfig
    {
        IndexName = "Table_Id_Index",
        KeyExpression = new Expression
        {
            ExpressionStatement = "TableId = :v_TableId"
            ExpressionAttributeValues = expressionAttributeValues
        }
    };

    var result = await _dynamoDBContext
        .FromQueryAsync<T>(query)
        .GetRemainingAsync();
}

-- With Index, Key and extra filtering

public async Task<List<T>> Find<T>(Guid id)
{
    var expressionAttributeValues = new Dictionary<string, DynamoDBEntry>();
    expressionAttributeValues.Add(":v_TableId", id);

    var filterAttributes = new Dictionary<string, DynamoDBEntry>();
    filterAttributes.Add(":v_Deleted", true);

    var queryOperationConfig = new QueryOperationConfig
    {
        IndexName = "Table_Id_Index",
        KeyExpression = new Expression
        {
            ExpressionStatement = "TableId = :v_TableId"
            ExpressionAttributeValues = expressionAttributeValues
        }
        FilterExpression = new Expression
        {
            ExpressionStatement = "Deleted != :v_Deleted"
            ExpressionAttributeValues = filterAttributes
        };
    };

    var result = await _dynamoDBContext
        .FromQueryAsync<T>(query)
        .GetRemainingAsync();
}

-- With GSI, Key and Partion

public async Task<List<T>> Find<T>(Guid id, string partitionKey)
{
    var expressionAttributeValues = new Dictionary<string, DynamoDBEntry>();
    expressionAttributeValues.Add(":v_TableId", id);
    expressionAttributeValues.Add(":v_PartitionKey", partitionKey);

    var queryOperationConfig = new QueryOperationConfig
    {
        IndexName = "GSI_Index",
        KeyExpression = new Expression
        {
            ExpressionStatement = "TableId = :v_TableId and PartitionKey = :v_PartitionKey"
            ExpressionAttributeValues = expressionAttributeValues
        }
    };

    var result = await _dynamoDBContext
        .FromQueryAsync<T>(query)
        .GetRemainingAsync();
}

Answer 1

Confusingly, it seems there is little if any difference between these options. The comments inside the AWSSDK.DynamoDBv2 assemblies seem to give the best documentation. Even though the term "Query Filter" in DynamoDb specifically refers to post-read processing, it is definitely the case that you can apply a QueryOperationConfig.Filter that targets key attributes (and is checked at runtime). The fact that the comments specify key attributes vs non-key attributes clearly indicate that these are in fact read-time key expression "filters" and not post-read filters in the true sense of the word.

Of course this lines up with what we know about DynamoDb queries in general: That they target partitions. So it follows that a QueryOperationConfig with a single filter condition on a hash key would have to target that partition, and could not possibly be acting as a post-read filter. To interpret it otherwise would be to assume to actually scans the table then applies a filter, which wouldn't make any sense for a Query.

So in summary: The DocumentModel provides two equivalent options - the QueryOperationConfig.KeyExpression option, which is very clear but exposes the lower level API, and the QueryOperationConfig.Filter option, which abstracts it but muddles the meaningful term "filter." It would have been much clearer had they designed a KeyExpression type, potentially with AddHashKeyCondition(...) and AddRangeKeyCondition(...) methods.

#region Assembly AWSSDK.DynamoDBv2, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604
// AWSSDK.DynamoDBv2.dll
#endregion

using System.Collections.Generic;
using Amazon.DynamoDBv2.Model;

namespace Amazon.DynamoDBv2.DocumentModel
{
    //
    // Summary:
    //     Query filter.
    public class QueryFilter : Filter
    {
        //
        // Summary:
        //     Constructs an empty QueryFilter instance
        public QueryFilter();
        //
        // Summary:
        //     Constructs an instance of QueryFilter with a single condition. More conditions
        //     can be added after initialization.
        //
        // Parameters:
        //   attributeName:
        //     Target attribute name
        //
        //   op:
        //     Comparison operator
        //
        //   values:
        //     Attributes to compare
        public QueryFilter(string attributeName, QueryOperator op, List<AttributeValue> values);
        //
        // Summary:
        //     Constructs an instance of QueryFilter with a single condition. More conditions
        //     can be added after initialization.
        //
        // Parameters:
        //   attributeName:
        //     Target attribute name
        //
        //   op:
        //     Comparison operator
        //
        //   values:
        //     Attributes to compare
        public QueryFilter(string attributeName, QueryOperator op, params DynamoDBEntry[] values);

        //
        // Summary:
        //     Adds a condition for a specified key attribute that consists of an operator and
        //     any number of AttributeValues.
        //
        // Parameters:
        //   keyAttributeName:
        //     Target key attribute name
        //
        //   op:
        //     Comparison operator
        //
        //   values:
        //     AttributeValues to compare to
        public void AddCondition(string keyAttributeName, QueryOperator op, List<AttributeValue> values);
        //
        // Summary:
        //     Adds a condition for a specified key attribute that consists of an operator and
        //     any number of values
        //
        // Parameters:
        //   keyAttributeName:
        //     Target key attribute name
        //
        //   op:
        //     Comparison operator
        //
        //   values:
        //     Values to compare to
        public void AddCondition(string keyAttributeName, QueryOperator op, params DynamoDBEntry[] values);
        //
        // Summary:
        //     Adds a condition for a specified non-key attribute that consists of an operator
        //     and any number of AttributeValues.
        //
        // Parameters:
        //   nonKeyAttributeName:
        //     Target non-key attribute name
        //
        //   op:
        //     Comparison operator
        //
        //   values:
        //     AttributeValues to compare to
        public void AddCondition(string nonKeyAttributeName, ScanOperator op, List<AttributeValue> values);
        //
        // Summary:
        //     Adds a condition for a specified non-key attribute that consists of an operator
        //     and any number of values
        //
        // Parameters:
        //   nonKeyAttributeName:
        //     Target non-key attribute name
        //
        //   op:
        //     Comparison operator
        //
        //   values:
        //     Values to compare to
        public void AddCondition(string nonKeyAttributeName, ScanOperator op, params DynamoDBEntry[] values);
    }
}