How to convert a json response into yaml in bash

Question

I read data from a json file with jq. I wanna append the results into a yaml file, but I dont get it working. I am quite new to shell programming. My goal is to append that "users" to an existing "users"-Array in a yaml file.

This is my json file:

#$DEFAULTS_FILE

{"users":
  [
    {"name":"pi",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "groups":"users,docker,video",
      "plain_text_passwd":"pi",
      "lock_passwd":"false",
      "ssh_pwauth":"true",
      "chpasswd": {"expire": false}
    },
    {"name":"admin",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "primary-group": "users",
      "groups":"users,docker,adm,dialout,audio,plugdev,netdev,video",
      "ssh-import-id":"None",
      "plain_text_passwd":"pi",
      "lock_passwd":"true",
      "ssh_pwauth":"true",
      "chpasswd": "{expire: false}",
      "ssh-authorized-keys": ["ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local"]
    }
  ]
  }

I filter it with that:

cat $DEFAULTS_FILE | jq .users

I have no clue how to convert that json into a yaml.

My expected result should be:

users:
  - name:                pi
    gecos:               "Hypriot Pirate"
    sudo:                ALL=(ALL) NOPASSWD:ALL
    shell:               /bin/bash
    groups:              users,docker,video
    plain_text_passwd:   pi
    lock_passwd:         false
    ssh_pwauth:          true
    chpasswd: { expire:  false }
  - name:                admin
    primary-group:       users
    shell:               /bin/bash
    sudo:                ALL=(ALL) NOPASSWD:ALL
    groups:              users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:       None

I tried to use a second tool called yq which is similar to jq and can write yaml files. But I have no positive progress.

EDIT

I know that I can add content to the yaml with that:

yq w -i "my.yml" "users[+]" "some content"

But I dont know how to merge my json into that.

Any help or hint would be nice, thank you in advance...

You can use `python`/`perl` which have in-built `YAML` and `JSON` modules - https://www.commandlinefu.com/commands/view/12218/convert-yaml-to-json — Inian, Nov 15 '18 at 09:11
Bash itself is not a good platform for this. Find an existing tool, or write one in e.g. Python. Depending on your preferred language, it should not be hard to find an existing question on Stack Overflow with suggestions; [here's a search for Python](https://www.google.com/search?q=python+convert+json+to+yaml+site%3Astackoverflow.com) — tripleee, Nov 15 '18 at 10:04
Please follow the [mcve] guidelines. In particular, a sample my.yml (i.e. input) would be helpful. — peak, Nov 15 '18 at 10:17
Thank you all, I followed @Inian `s approach to write a 'lil script in Ruby. and @Jeff Mercado. I don't know why this is quoted, but this is exactly what the program spit. — Jan, Nov 15 '18 at 19:23
as @AHT pointed out `yq r` works (as JSON is a subset of YAML); the output should be pretty-printed, though to make it "look more like YAML": `yq -P r input.json`; see also https://mikefarah.gitbook.io/yq/usage/convert#json-to-yaml — ssc, Feb 17 '20 at 12:16
[YAML spec](https://yaml.org/spec/1.2.1/#id2759572) says: "every JSON file is also a valid YAML file" — djvg, Sep 28 '21 at 12:40

score 77 · Answer 1 · edited Apr 23 '22 at 09:26

77

yq a yaml wrapper for jq

With yq 4.18.1+

cat "$DEFAULTS_FILE" | yq -P    # or yq --prettyPrint

See: https://mikefarah.gitbook.io/yq/#notice-for-v4.x-versions-prior-to-4.18.1

With yq version 4.8.0:

cat $DEFAULTS_FILE | yq e -P -

e or eval handles file separately. ea or eval-all will merge files first.
-P or --prettyPrint YAML output
- from STDIN

Note: you can go the other way (yaml to json) too yq e -j file.yaml

With yq version 3.3.2:

cat $DEFAULTS_FILE | yq r -P -

r read
-P --prettyPrint
- from STDIN

edited Apr 23 '22 at 09:26

ryenus

15,711
5
56
63

answered Jul 08 '20 at 16:00

ianmunoz

1,823
13
8

3

You're a mad lad! Thanks mate! – Robert J Sep 29 '20 at 15:07
2

This answer needs an edit. Your `yq` example is an implementation of mikefarah/yq, which is NOT a wrapper over `jq`. See my [answer](https://stackoverflow.com/a/64625920/5291015) – Inian Nov 01 '20 at 15:25
3

If using mikefarah/yq V4, you need to use the following command: `yq eval '.. style= ""' sample.json` or `cat sample.json | yq eval '.. style= ""' -`. See https://mikefarah.gitbook.io/yq/usage/convert – ethanabrooks Dec 25 '20 at 14:07
1

Actually, all you need is `cat $DEFAULTS_FILE | yq -y` – Ben Davis Jan 03 '21 at 05:06
4

With [`mikefarah/yq`](https://github.com/mikefarah/yq) version 4, `yq eval -P` is the correct syntax. – ijoseph Feb 28 '21 at 03:58
With [mikefarah/yq](https://github.com/mikefarah/yq) version 4.15, `yq e -P` and `yq eval -P` both works fine. – AATHITH RAJENDRAN Dec 01 '21 at 05:21
Tangentially, you want to avoid the [useless use of `cat`](https://stackoverflow.com/questions/11710552/useless-use-of-cat) – tripleee May 16 '22 at 10:08

Richard Gomes · Answer 2 · 2020-03-11T06:35:24.583

36

function yaml_validate {
  python -c 'import sys, yaml, json; yaml.safe_load(sys.stdin.read())'
}

function yaml2json {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read())))'
}

function yaml2json_pretty {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read()), indent=2, sort_keys=False))'
}

function json_validate {
  python -c 'import sys, yaml, json; json.loads(sys.stdin.read())'
}

function json2yaml {
  python -c 'import sys, yaml, json; print(yaml.dump(json.loads(sys.stdin.read())))'
}

More Bash tricks at http://github.com/frgomes/bash-scripts

edited Mar 11 '20 at 06:35

answered Mar 08 '20 at 08:56

Richard Gomes

5,675
2
44
50

Possibly superfluous in light of `yq` and `jq`, but very nicely done, and probably has some good applications. – abalter Oct 02 '20 at 05:52
Ha! As it happens, I ran into errors installing `yq` just now in a gpc ai notebook. so, I guess these will come in very handy! – abalter Oct 02 '20 at 06:00
2

perhaps the best answer,as it does not make you install `yq` – P.... Jul 05 '21 at 17:53
awesome-bash-scripts. – Akhil Dec 28 '21 at 09:51
1

yq is bloaty. Thank you. – Rondo Jan 28 '22 at 02:24

Jeff Mercado · Answer 3 · 2018-11-16T19:17:20.627

I'm not sure what rules you're using to get to your expected result. It seems like you're randomly applying different rules to how the values are being converted.

As I understand it, scalar values are just output as is (with potential encoding), objects are output as key/value pairs, and array objects are output with a - for every item. The indentation associates what's part of what.

So based on those rules if you're going to use jq:

def yamlify:
    (objects | to_entries[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify)"
        else
            "\(.key):\t\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;

Then to use it, add it to your .jq file and use it:

$ jq -r yamlify input.json
users:
  - name:       pi
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    groups:     users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth: true
    chpasswd:
        expire: false
  - name:       admin
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    primary-group:      users
    groups:     users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:      None
    plain_text_passwd:  pi
    lock_passwd:        true
    ssh_pwauth: true
    chpasswd:   {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local

Here's another variation that aligns the values

def yamlify2:
    (objects | to_entries | (map(.key | length) | max + 2) as $w |
        .[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify2)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify2)"
        else
            "\(.key):\(" " * (.key | $w - length))\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify2] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;

$ jq -r yamlify2 input.json
users:
  - name:               pi
    gecos:              Hypriot Pirate
    sudo:               ALL=(ALL) NOPASSWD:ALL
    shell:              /bin/bash
    groups:             users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth:         true
    chpasswd:
        expire:  false
  - name:                 admin
    gecos:                Hypriot Pirate
    sudo:                 ALL=(ALL) NOPASSWD:ALL
    shell:                /bin/bash
    primary-group:        users
    groups:               users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:        None
    plain_text_passwd:    pi
    lock_passwd:          true
    ssh_pwauth:           true
    chpasswd:             {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local

really cool. love it, unfortunately, it breaks for multiline string values.. — Kaos, Jun 02 '20 at 09:57
Based on the code of @Jeff Mercado I added some code to support multiline strings and escaping of single quotes. See my answer "Solution in jq (without other tools)". — jpseng, Jun 29 '21 at 20:01

ijoseph · Answer 4 · 2021-04-15T19:13:54.553

`yq eval -P`

with mikefarah/yq version 4.0 (released December 2020), installable via most Unix-like OS package managers: via Homebrew for macOS (brew install yq), Debian with apt (apt install yq), Alpine with apk (apk add yq), etc.

See Working with JSON.

To read in json, just pass in a json file instead of yaml, it will just work - as json is a subset of yaml. However, you will probably want to use the Style Operator or --prettyPrint/-P flag to make look more like an idiomatic yaml document.

score 8 · Answer 5 · answered Oct 28 '19 at 09:36

8

I've used ruby to write my json content into yaml.

As for your example, it can be achieved like this:

cat $DEFAULTS_FILE | jq .users | ruby -ryaml -rjson -e 'puts YAML.dump(JSON.parse(STDIN.read))' > my.yml

answered Oct 28 '19 at 09:36

Pavel

1,627
15
12

2

While I can't stand ruby, my machine already had it installed. I was able to create an alias and pipe it to this alias. Thank you, this has been very helpful! – Brandon Dec 23 '21 at 21:05
Exact same sentiment here. – rookie099 Mar 30 '22 at 12:59

score 7 · Answer 6 · answered Dec 04 '20 at 14:31

I suggest using yq with -y option

$ pip3 install yq # requires jq

$ cat in.json | yq -y
users:
  - name: pi
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    groups: users,docker,video
    plain_text_passwd: pi
    lock_passwd: 'false'
    ssh_pwauth: 'true'
    chpasswd:
      expire: false
  - name: admin
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    primary-group: users
    groups: users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id: None
    plain_text_passwd: pi
    lock_passwd: 'true'
    ssh_pwauth: 'true'
    chpasswd: '{expire: false}'
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 YOUR_KEY@YOURHOST.local

Sebastian Wagner · Answer 7 · 2020-03-24T10:00:40.883

5

Another oneliner:

python -c 'import yaml, sys; print(yaml.dump(yaml.load(open(sys.argv[1])), default_flow_style=False))' input.json

(exploiting the fact that valid json is also valid yaml)

And yaml to json:

python -c 'import yaml, json, sys; print(json.dumps(yaml.load(open(sys.argv[1])), indent=2))' input.yaml

edited Mar 24 '20 at 10:00

answered Feb 25 '20 at 12:30

Sebastian Wagner

2,308
2
25
32

Your yaml-to-json oneliner fails with `found character '\t' that cannot start any token`; it's the same error message `yq` reports, which must work similarly. Richard Gomes' one-liner works fine. – Bean Taxi Mar 29 '20 at 15:40

jpseng · Answer 8 · 2021-06-28T18:57:26.233

Solution in jq (without other tools)

Based on the code of @Jeff Mercado in this post I added support for multiline strings and escaping for single quotes.

# purpose: converts Json to Yaml
# remarks:
#   You can use 'yq -y' to convert json to yaml, but ...
#     * this function can be used several times within a single jq program
#     * this function may be faster than using yq
#     * maybe yq is not available in your environment
#
# input: any Json
# output: json converted to yaml
def toYaml:
   def handleMultilineString($level):
      reduce ([match("\n+"; "g")]                       # find groups of '\n'
              | sort_by(-.offset))[] as $match
             (.; .[0:$match.offset + $match.length] +
                 "\n\("    " * $level)" +               # add one extra '\n' for every group of '\n's. Add indention for each new line
                 .[$match.offset + $match.length:]);

   def toYamlString($level):
      if type == "string"
      then handleMultilineString($level)
           | sub("'"; "''"; "g")           # escape single quotes
           | "'\(.)'"                      # wrap in single quotes
      else .
      end;

   def _toYaml($level):
      (objects | to_entries[] |
          if (.value | type) == "array" then
              "\(.key):", (.value | _toYaml($level))
          elif (.value | type) == "object" then
              "\(.key):", "\("    ")\(.value | _toYaml($level))"
          else
              "\(.key): \(.value | toYamlString($level))"
          end
      )
      // (arrays | select(length > 0)[] | [_toYaml($level)] |
          "  - \(.[0])", "\("    ")\(.[1:][])"
      )
      // .;

   _toYaml(1);

Example usage

File 'containsMultilineStrings.json'

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

jq -r 'toYaml' < containsMultilineStrings.json

response:
    code: 200
    message: 'greeting

    that''s all folks



    '

jq -r 'toYaml' containsMultilineStrings.json | yq (roundtrip)

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

Test

You can test the correctness of the function toYaml by converting json to yaml and than back to json using yq.

FILE='containsMultilineStrings.json'; diff <(cat "$FILE") <(jq -r 'toYaml' $FILE | yq)

Performance

A quick benchmark shows a reduced runtime of the function toYaml compared to the use of yq. On my computer, I measured:

time for i in {1..100}; do yq -y > /dev/null < containsMultilineStrings.json; done

8.4 sec

time for i in {1..100}; do jq -r 'toYaml' > /dev/null containsMultilineStrings.json; done

3.4 sec

score 1 · Answer 9 · answered Aug 04 '22 at 05:18

You can use the awesome library yq

# install yq on macOS
brew install yq

Then run the following command

cat file.json | yq -P > file.yml

You can also convert and save a JSON Response into a YAML file, for example:

curl https://raw.githubusercontent.com/typicode/demo/master/db.json | yq -P > example.yml

score 0 · Answer 10 · answered Dec 04 '21 at 12:06

0

Another option is to use gojq. It is a port of jq with support for reading and writing yaml. It can be installed via GitHub releases, homebrew, and zero install. The command for your question would be:

cat test.json | gojq --yaml-output > test.yaml

answered Dec 04 '21 at 12:06

Moritz

2,987
3
21
34

score 0 · Answer 11 · answered Jul 14 '23 at 22:15

cat sample.json | yq -P

As of 2023 - the answer from @ianmunoz is correct

The source file

cat sample.json
[
  {"name": "Abc", "id": "10"},
  {"name": "Def", "id": "11"},
  {"name": "Xyz", "id": "12"}
]

As yaml

cat sample.json | yq -P
- name: Abc
  id: "10"
- name: Def
  id: "11"
- name: Xyz
  id: "12"

As csv

cat sample.json | yq e '.[0, 1] | [key+1, .id, .name] | @csv'
1,10,Abc
2,11,Def

Note: you can extract a range of records with [0, 1] and prepend a key in the output.