Formatting DynamoDB data to normal JSON in AWS Lambda

Question

I'm using AWS Lambda to scan data from a DynamoDB table. This is what I get in return:

{
  "videos": [
    {
      "file": {
        "S": "file1.mp4"
      },
      "id": {
        "S": "1"
      },
      "canvas": {
        "S": "This is Canvas1"
      }
    },
    {
      "file": {
        "S": "main.mp4"
      },
      "id": {
        "S": "0"
      },
      "canvas": {
        "S": "this is a canvas"
      }
    }
  ]
}

My front-end application is using Ember Data Rest Adapter which does not accepts such response. Is there any way I can get normal JSON format? There is this NPM module called dynamodb-marshaler to convert DynamoDB data to normal JSON. I'm looking for a native solution if possible.

Look into using something from the AWS SDK DynamoDB library. http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html — adamkonrad, Sep 22 '15 at 23:04
Same : https://stackoverflow.com/questions/44535445/unmarshall-dynamodb-json — Sebastian, Mar 19 '20 at 12:20

David Rissato Cruz · Answer 1 · 2021-02-24T21:57:41.423

125

Node.js Use the unmarshall function from AWSJavaScriptSDK:

const AWS = require("aws-sdk");
    
exports.handler = function( event, context, callback ) {
    const newImages = event.Records.map(
            (record) => AWS.DynamoDB.Converter.unmarshall(record.dynamodb.NewImage)
    );
    console.log('Converted records', newImages);
    callback(null, `Success`);
}

Python Use TypeDeserializer.deserialize from boto3.dynamodb.types:

import json
from boto3.dynamodb.types import TypeDeserializer

def ddb_deserialize(r, type_deserializer = TypeDeserializer()):
    return type_deserializer.deserialize({"M": r})

def lambda_handler(event, context):
    new_images = [ ddb_deserialize(r["dynamodb"]["NewImage"]) for r in event['Records'] ]
    print('Converted records', json.dumps(new_images, indent=2))

edited Feb 24 '21 at 21:57

answered Dec 05 '17 at 23:44

David Rissato Cruz

3,347
2
17
17

for some reason unmarshall wasn't deep enough in my case, I had some array values being unmarshalled as `[Array]`, so I had to `JSON.stringify(AWS.DynamoDB.Converter.unmarshall(record))` – divideByZero Mar 20 '19 at 21:37
5

@divideByZero The `console.log` was just an example, and it prints a simplified version of the record when the depth is more than 2 because internally it uses the `util.inspect` (https://nodejs.org/api/util.html#util_util_inspect_object_options). But it doesn't mean the record returned by `unmarshall` was incomplete, otherwise JSON.stringify wouldn't have worked either. – David Rissato Cruz Dec 18 '19 at 02:37
6

Awesome solution!! Works like a charm. Thank you. – moreirapontocom Jul 29 '20 at 12:57
1

Worked like a charm in NodeJS, thanks! For the record, I am not using nested objects. – Codigo Morsa Nov 26 '21 at 18:37
1

For Python, `TypeDeserializer` converts numbers to `Decimal`s (even if they're just `int`s), resulting in `TypeError: Object of type Decimal is not JSON serializable` – Mike B Jul 16 '22 at 00:00
For Python, FWIW, I resolved the `Decimal` problem (& a later `set` problem) using the method described in the 2nd answer here: https://stackoverflow.com/questions/1960516/python-json-serialize-a-decimal-object – Mike B Jul 25 '22 at 16:52

dolcalmi · Answer 2 · 2017-12-29T16:08:08.970

33

I know is a bit old but I had the same problem processing stream data from dynamoDB in node js lambda function. I used the proposed by @churro

import sdk and output converter

var AWS = require("aws-sdk");
var parse = AWS.DynamoDB.Converter.output;

use the parse function with a small hack

exports.handler = function( event, context, callback ) {
  var docClient = new AWS.DynamoDB.DocumentClient();
  event.Records.forEach((record) => {
        console.log(record.eventID);
        console.log(record.eventName);
        console.log('DynamoDB Record:', parse({ "M": record.dynamodb.NewImage }));
    });
  callback(null, `Successfully processed ${event.Records.length} records.`);
}

Hope it helps

edited Dec 29 '17 at 16:08

answered Mar 11 '17 at 08:27

dolcalmi

796
8
13

4

You just saved my bacon. I was going crazy expecting `AWS.DynamoDB.Converter.output` to work on the `NewImage` record directly. – Ryan McGeary May 24 '17 at 19:50
In the above code : closing brackets are missing. It should be : console.log('DynamoDB Record:', parse({ "M": record.dynamodb.NewImage })); – Vishnu Ranganathan Dec 28 '17 at 15:25

score 30 · Accepted Answer · answered Sep 23 '15 at 10:30

30

AWS JavaScript SDK was recently updated with Document Client which does exactly what you need. Check the announce and usage examples here: http://blogs.aws.amazon.com/javascript/post/Tx1OVH5LUZAFC6T/Announcing-the-Amazon-DynamoDB-Document-Client-in-the-AWS-SDK-for-JavaScript

answered Sep 23 '15 at 10:30

Vasily Sochinsky

3,981
2
23
20

This is a more elegant, forward-looking solution. But it does not seem to be available by default in Lambda just yet -- the AWS SDK version in Lambda is 2.1.50, AWS.DynamoDB.DocumentClient requires 2.2.0. – James Sep 23 '15 at 16:07
This is true, but I think it is a matter of days until they rollout the update. Lambda improves rapidly now. – Vasily Sochinsky Sep 23 '15 at 16:10
Agreed, 2.1.50 is from 2015-09-04, they stay very current. – James Sep 23 '15 at 16:38
Hey, that's exactly what I was looking for. I'll use marshaller for now and will switch to this as soon as they update the SDK available in Lambda. – Chaitanya Sep 24 '15 at 20:36
1

How will this work to convert the input event in Lambda that is in the special DynamoDB JSON to normal json? – Thiyagu Jun 29 '16 at 17:35
@user7 I asked them that question on their blog post on `DocumentClient`. Still waiting on an answer. In the meantime, I'm trying to find a way to access this class directly: https://github.com/aws/aws-sdk-js/blob/master/lib/dynamodb/converter.js – Churro Aug 02 '16 at 23:24
@Churro Have you got your answer on converting dynamo JSON to normal JSON? – Andrej Kaurin Dec 20 '16 at 04:17
3

@AndrejKaurin I think you can use a Node library like [dynamodb-marshaler](https://www.npmjs.com/package/dynamodb-marshaler). I ended up doing this manually when I needed it in the past. Unfortunately it seems the `DocumentClient` only works with direct operations against the tables. – Churro Dec 20 '16 at 19:35
2

@VasilySochinsky any solution for Python? – VB_ May 13 '19 at 15:51

forzagreen · Answer 4 · 2020-08-04T11:06:57.313

21

Javascript: AWS SDK provides the unmarshall function
Python: use TypeDeserializer from boto3.dynamodb.types:

from boto3.dynamodb.types import TypeDeserializer, TypeSerializer

def from_dynamodb_to_json(item):
    d = TypeDeserializer()
    return {k: d.deserialize(value=v) for k, v in item.items()}

## Usage:
from_dynamodb_to_json({
    "Day": {"S": "Monday"},
    "mylist": {"L": [{"S": "Cookies"}, {"S": "Coffee"}, {"N": "3.14159"}]}
})
# {'Day': 'Monday', 'mylist': ['Cookies', 'Coffee', Decimal('3.14159')]}

edited Aug 04 '20 at 11:06

answered Dec 20 '19 at 15:56

forzagreen

2,509
30
38

I haven't tried it yet (not currently using DynamoDB), but it looks like `boto3` has something similar to javascript `unmarshall`. Take a look on `TypeDeserializer` from `boto3.dynamodb.types`: (https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/dynamodb/types.html). – David Rissato Cruz Jan 08 '20 at 22:41

igorzg · Answer 5 · 2017-07-09T17:11:03.853

Here you can find gist which does that:

function mapper(data) {

let S = "S";
let SS = "SS";
let NN = "NN";
let NS = "NS";
let BS = "BS";
let BB = "BB";
let N = "N";
let BOOL = "BOOL";
let NULL = "NULL";
let M = "M";
let L = "L";

if (isObject(data)) {
    let keys = Object.keys(data);
    while (keys.length) {
        let key = keys.shift();
        let types = data[key];

        if (isObject(types) && types.hasOwnProperty(S)) {
            data[key] = types[S];
        } else if (isObject(types) && types.hasOwnProperty(N)) {
            data[key] = parseFloat(types[N]);
        } else if (isObject(types) && types.hasOwnProperty(BOOL)) {
            data[key] = types[BOOL];
        } else if (isObject(types) && types.hasOwnProperty(NULL)) {
            data[key] = null;
        } else if (isObject(types) && types.hasOwnProperty(M)) {
            data[key] = mapper(types[M]);
        } else if (isObject(types) && types.hasOwnProperty(L)) {
            data[key] = mapper(types[L]);
        } else if (isObject(types) && types.hasOwnProperty(SS)) {
            data[key] = types[SS];
        } else if (isObject(types) && types.hasOwnProperty(NN)) {
            data[key] = types[NN];
        } else if (isObject(types) && types.hasOwnProperty(BB)) {
            data[key] = types[BB];
        } else if (isObject(types) && types.hasOwnProperty(NS)) {
            data[key] = types[NS];
        } else if (isObject(types) && types.hasOwnProperty(BS)) {
            data[key] = types[BS];
        }
    }
}


return data;

function isObject(value) {
    return typeof value === "object" && value !== null;
}

}

https://gist.github.com/igorzg/c80c0de4ad5c4028cb26cfec415cc600

I had a dump of my DynamoDB table from Data Pipeline that I had to parse, aside from having to adjust the casing of the data types, this worked great. — Don P, Jul 05 '17 at 21:41
Don't know why this was downvoted. I need to do this in clientside JS, and including a big heap of AWS JavaScript for just a few functions seems excessive. This will _do the job_ to a large extent, and it's not going to go wrong unless DynamoDB gives you back something that's malformatted. Which would mean you have bigger problems. — HughPH, Sep 08 '19 at 10:26

score 5 · Answer 6 · answered Jan 15 '20 at 13:37

5

If you are using python in the lambda you can utilise the dynamodb-json library.

Install library

pip install dynamodb-json

and use the below snippet

from dynamodb_json import json_util as util

def marshall(regular_json):
    dynamodb_json = util.dumps(reular_json)

def unmarshall(dynamodb_json):
    regular_json = util.loads(dynamodb_json)

Reference https://pypi.org/project/dynamodb-json/

answered Jan 15 '20 at 13:37

Chandan Kumar

1,066
10
16

for lambdas maby not the best option – Cristián Vargas Acevedo Oct 02 '21 at 03:28
1

hasn't been updated since 2018. Some will run into dep. issues with pip-compile if using newer versions of boto – Matt Swezey Mar 22 '22 at 20:17

score 1 · Answer 7 · answered Sep 22 '15 at 22:10

I think it's just a custom transformation exercise for each app. A simple conversion from DynamoDB's item format to you application format might look like this:

var response = {...} // your response from DynamoDB
var formattedObjects = response.videos.map(function(video) {
    return {
        "file": video.file.S,
        "id": video.id.S,
        "canvas": video.canvas.S
    };
});

If you want to build a generic system for this, you would have to handle DynamoDB's various AttributeValue types. A function like the one below would do the job, but I've left out the hard work of handling most of DynamoDB's more complex attribute value types:

function dynamoItemToPlainObj(dynamoItem) {
    var plainObj = {};
    for (var attributeName in dynamoItem) {
        var attribute = dynamoItem[attributeName];
        var attributeValue;
        for (var itemType in attribute) {
            switch (itemType) {
            case "S":
                attributeValue = attribute.S.toString();
                break;
            case "N":
                attributeValue = Number(attribute.N);
                break;
                // more attribute types...
            default:
                attributeValue = attribute[itemType].toString();
                break;
            }
        }
        plainObj[attributeName] = attributeValue;
    }
    return plainObj;
}    
var formattedObjects = response.videos.map(dynamoItemToPlainObj);

Another downvoted comment, presumably from someone who thinks you should just include the library. Not everyone wants to do that, and not everyone is in a position to do that. This is _a solution_. It might not be _the best_ by some metrics, but it's _acceptable_ by other metrics. — HughPH, Sep 08 '19 at 10:28

mwag · Answer 8 · 2022-04-23T16:10:13.140

I tried several solutions here but none worked with multi-level data, such as if it includes a list of maps e.g.

{
  "item1": {
    "M": {
      "sub-item1": {
        "L": [
          {
            "M": {
              "sub-item1-list-map": {
                "S": "value"

Below, adapted from @igorzg's answer (which also has that drawback), fixes that.

Example usage:

dynamodb.getItem({...}, function(err, data) {
  if (!err && data && data.Item) {
    var converted = ddb_to_json(data.Item);

Here's the conversion function:

function ddb_to_json(data) {
  
  function isObject(value) {
    return typeof value === "object" && value !== null;
  }

  if(isObject(data))
    return convert_ddb({M:data});

  function convert_ddb(ddbData) {
    if (isObject(ddbData) && ddbData.hasOwnProperty('S'))
      return ddbData.S;
    if (isObject(ddbData) && ddbData.hasOwnProperty('N'))
      return parseFloat(ddbData.N);
    if (isObject(ddbData) && ddbData.hasOwnProperty('BOOL'))
      return ddbData.BOOL;
    if (isObject(ddbData) && ddbData.hasOwnProperty('NULL'))
      return null;
    if (isObject(ddbData) && ddbData.hasOwnProperty('M')) {
      var x = {};
      for(var k in ddbData.M)
        x[k] = convert_ddb(ddbData.M[k])
      return x;
    }
    if (isObject(ddbData) && ddbData.hasOwnProperty('L'))
      return ddbData.L.map(x => convert_ddb(x));
    if (isObject(ddbData) && ddbData.hasOwnProperty('SS'))
      return ddbData.SS;
    if (isObject(ddbData) && ddbData.hasOwnProperty('NN'))
      return ddbData.NN;
    if (isObject(ddbData) && ddbData.hasOwnProperty('BB'))
      return ddbData.BB;
    if (isObject(ddbData) && ddbData.hasOwnProperty('NS'))
      return ddbData.NS;
    if (isObject(ddbData) && ddbData.hasOwnProperty('BS'))
      return ddbData.BS;

    return data;
  }

  return data;
}

score -4 · Answer 9 · answered May 08 '17 at 14:42

-4

If you need online editor try this

https://2json.net/dynamo

answered May 08 '17 at 14:42

Beka Tomashvili

2,171
5
21
27

1

The link is down – forzagreen Dec 18 '19 at 13:08

Formatting DynamoDB data to normal JSON in AWS Lambda

9 Answers9

Linked