Remove trailing json comma with command line tools

Question

I want to remove trailing comma from json as,

{
  "key1": "value1",
  "object": {
    "key2": "value2", // <- remove comma
  },
  "key3": "value3", // <- remove comma
}

I came up with,

tr -d '\n' | \
sed -E 's:,(\s*}):\1:g' | \
jq .

and it works but I want to get this fully in sed.

I came up with,

sed -E '/,\s*$/ { N; s:,\s*(\n\s*},?):\1: }'

which works for above input but fails for

{
  "key1": "value1",
  "object": {
    "key2": "value2",
  },
  "key3": "value3",
  "key4": "value4", // <- remove comma
}

as N reads the next line and starts over from the line after next.

// output sed -E '/,\s*$/ { N;l }' using l/look command
{
  "key1": "value1",\n  "object": {$
  "key1": "value1",
  "object": {
    "key2": "value2",\n  },$
    "key2": "value2",
  },
  "key3": "value3",\n  "key4": "value4",$
  "key3": "value3",
  "key4": "value4",
}

Update:

Adding another example for testing:

{
  "key1": "value1",
  "object1": {
    "object2": {
      "key2": "value2"
    },
  },
  "key3": "value3",
}

Update:

This is working for whatever I've thrown at it.

sed -E -n 'H; x; s:,(\s*\n\s*}):\1:; P; ${x; p}' | \
    sed '1 d'

Explanation:

sed -E -n 'H; x; P; ${x; p}'

-n 'H; x' to get every line appended to the next line in pattern space (except for the last line which is simply printed with ${x; p})

and

s:,(\s*\n\s*}):\1:;

to remove the trailing comma in the pattern space.

Answer 1

Since the input seems to be some kind of extension of JSON, you could use a command-line tool intended for such extensions. For example:

$ hjson -j < input.txt

or:

$ any-json --input-format=hjson input.txt

Output in both cases

{
  "key1": "value1",
  "object": {
    "key2": "value2"
  },
  "key3": "value3"
}

Answer 2

Using the hold buffer:

sed '/^ *\}/{H;x;s/\([^}]\),\n/\1\n/;b};x;/^ *}/d' input

This is just a sed exercise, I don't think sed is the right tool for this job. It also needs a newline at the end or that the file ends with a }.

Answer 3

Not an answer with sed but a (python) solution:

# load as python dictionary
d = {
  "key1": "value1",
  "object": {
    "key2": "value2",
  },
  "key3": "value3",
}

import json

json.dumps(d) # valid json string

Answer 4

https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json seems to be your answer

# this works
echo '{"a": 1,}' | jq -n -f /dev/stdin

# as well as this
cat <<EOF | jq -n -f /dev/stdin
{
  "key1": "value1",
  "object": {
    "key2": "value2",
  },
  "key3": "value3",
}
EOF

Answer 5

I got it by loading the json as yaml with python pyyaml library and worked fine.

So for this example:

$ echo '{"a": 1,}' | jq
parse error: Expected another key-value pair at line 1, column 9

pyyaml fixes the input:

$ echo '{"a": 1,}' | python3 -c "import sys, json, yaml; print(json.dumps(yaml.safe_load(sys.stdin)))" | jq

{
  "a": 1
}

A more complex example:

$ echo '{"a": [1,],}' | python3 -c "import sys, json, yaml; print(json.dumps(yaml.safe_load(sys.stdin)))" | jq

{
  "a": [
    1
  ]
}

Answer 6

Here is one in GNU awk. It uses " as field separator and removes commas before [ \n]*} from odd fields (outside quotes, will probably fail for "escaped \" inside"). Added "key4": "value4,}", to the file:

$ cat file
{
  "key1": "value1",
  "object": {
    "key2": "value2",
  },
  "key3": "value3",
  "key4": "value4,}",
}

The script processes the whole file as a single record (RS="^$") so it might not work for big files as-is:

$ awk '
BEGIN {
    FS=OFS="\""
    RS="^$"
}
{
    for(i=1;i<=NF;i++) {                         # or i+=2 and remove the if
        if(i%2)
            $i=gensub(/,([ \n]*\})/,"\\1","g",$i)
    }
}1' file

Output:

{
  "key1": "value1",
  "object": {
    "key2": "value2"
  },
  "key3": "value3",
  "key4": "value4,}"
}

Answer 7

sed and awk commands wasn’t working for me, so I ended up writing a small remove-json-trailing-comma.js file

const { readFileSync, writeFileSync } = require("fs");

const regex = /,(?!\s*?[{["'\w])/g;

const files = process.argv.splice(2);
for (let file of files) {
    const input = readFileSync(file).toString();
    let correct = input.replace(regex, '');
    writeFileSync(file, correct);
}

And then I could use it like this node remove-json-trailing-comma.js file.json (and work also with multiples files like node remove-json-trailing-comma.js **/package.json)