AWS IoT-Core how to properly publish a actual new line in a JSON value, not \r\n or \n

Question

I have a lambda (nodeJs) that reads a file (.ref) in a S3 bucket and publishes its content in a topic inside the AWS IoT-Core broker. The file contains something like this (50 lines):

model:type
aa;tp1
bb;tpz
cc;tpf
dd;tp1

The code must remove the first line and retrieve the remains 50 lines. This is the code

async function gerRef (BUCKET_NAME) {
    const refFile = version + '/file.ref'
    const ref = await getObject(FIRMWARE_BUCKET_NAME, refFile)
    //Get the file content
    const refString = ref.toString('utf8')
    //Slip by each line
    let arrRef = refString.split('\n')
    //Remove the file header
    arrRef.shift()
    //join the lines
    let refString = arrRef.join('\n')

    return refString
}

Then I am getting this result and publishing in the AWS IoT-Core Broker like this:

const publishMqtt = (params) =>
  new Promise((resolve, reject) =>
  iotdata.publish(params, (err, res) => resolve(res)))
...
let refData = await gerRef (bucket1)
let JsonPayload = {
    "attr1":"val1",
    "machineConfig":`${refData}` //Maybe here is the issue
}
let params = {
            topic: 'test/1',
            payload: JSON.stringify(JsonPayload) //Maybe here is the issue
            qos: '0'
        };

await publishMqtt(params)
...

Then it publishes in the broker. The issue is that the content is being published without a real new line. When I see in the broker I get the follow JSON:

{
     "attr1":"val1",
     "machineConfig":"aa;tp1\nbb;tpz\ncc;tpf\ndd;tp1"
}

The machine that receives this message is expecting a real new line, something like this:

{
     "attr1":"val1",
     "machineConfig":"aa;tp1
                      bb;tpz
                      cc;tpf
                      dd;tp1"
}

If I just copy and paste this entire JSON in the AWS IoT-Core interface it will complain about the JSON parse but will publish as string and the machine will accept the data- because the new line is there:

In short, the main point here is that:

We can use the JSON.stringify(JsonPayload) - The broker will accept
I don't know how to stringfy and keep the actual new line

I have tried these solutions but none of then worked: s1, s2, s3

Any guess in how to achieve this?

The goal isn't clear here. You read in the file, split it apart, then inexplicably glue it back together exactly as it was. Are you just stripping off the header? You could do that by deleting up to the first `"\n"` without all the splitting joining mess. — tadman, Feb 05 '21 at 00:16
@tadman do you have another way to remove the first line (the header) of a file and then retrieve all the lines? If yes, please post here that I will use it -> https://stackoverflow.com/questions/65798821/nodejs-how-to-remove-the-first-line-of-a-text-file-without-read-all-lines — IgorAlves, Feb 05 '21 at 00:27
For a given `str`, `str.replace(/^.*?\n/, '')` or `str.substr(str.indexOf("\n")+1, str.length)`. — tadman, Feb 05 '21 at 00:45

score 1 · Accepted Answer · answered Feb 05 '21 at 00:18

What that machines is expecting is wrong. In JSON any newline data inside a value must be escaped, and \n in the string is the correct way to do it. This is the fault of the receiver's expectations.

A "real" newline would result in an invalid JSON document and most parsers will flat-out reject it.

On the receiving end JSON deserializer can deal with \n encoded strings. If your receiver requires newlines it's broken and needs repairing. If you can't repair it then you're committed to sending busted up, malformed JSON-ish data that's not actually JSON and your broker is fully justified in trashing it.

AWS IoT-Core how to properly publish a actual new line in a JSON value, not \r\n or \n

1 Answers1