0

Say I have a yaml file, example.yaml:

networks:
  net:
    driver: overlay
dockerfiles:
  nodeapp: |
    FROM node:18-alpine
    WORKDIR /$NAME
services:
  registry:
    image: registry:2
    ports:
      - "5000:5000"
      - "337:337"
  pinger:
    build:
      context: .
      dockerfile: nodeapp
    deploy:
      replicas: 3

I can convert to JSON using yq, and then set the top level entries as variables with jq:

eval $(yq e example.yaml -j | jq -r '..? | to_entries | .[] | .key + "=" + (.value|tostring)')

echo $services
{registry:{image:registry:2,ports:[5000:5000]},pinger:{build:{context:.,dockerfile:nodeapp},deploy:{replicas:3}}}

However, I cannot seem to get jq to recursively go into the objects and set variables, so that echo $services_pinger_dockerfile gives "nodeapp" for example.

I've seen in the docs that there is a recursion operator. Can anyone suggest a way of doing this?

There are similar questions. But none that I found dealing with multilevel YAML/JSON to shell variables in this way.

jq: Getting two levels of keys

Extract JSON value to shell variable using jq

https://unix.stackexchange.com/questions/413878/json-array-to-bash-variables-using-jq

https://unix.stackexchange.com/questions/121718/how-to-parse-json-with-shell-scripting-in-linux

Parsing JSON with Unix tools

How can I parse a YAML file from a Linux shell script?

How to read yaml file into bash associative array?

Lee
  • 29,398
  • 28
  • 117
  • 170
  • It's not really clear what your expected result should be. The reason we need these fancier formats is that not everything can be flattened into a simple series of name-value pairs where the names are unique. – tripleee Mar 06 '23 at 13:06
  • 1
    For this approach, there's no need to switch to `jq` if you are already in `yq` - unless you want to use `@sh`, which you aren't currently, although it feels like you should. – pmf Mar 06 '23 at 13:17
  • @tripleee I'm experimenting with defining an entire docker swarm app (all info required for all build and deployment steps) in a single yaml file. I'll have a bash script to read the file and run a series of `docker` commands. Hence the need for bash variables. – Lee Mar 06 '23 at 13:51

1 Answers1

1

Why not process it directly with yq?

eval $(yq e '.. | select(. == "*") | {(path | join("_")): . style="double"} ' example.yaml | sed -E 's/(.*): (.*)/\1=\2/g')
echo $services_pinger_build_dockerfile

yq command was taken from this answer, see there for details.

The appended sed rewrites a_b: "c" into a_b="c", so we can then eval it in bash. I do not use the technique you show because it would cause the values to not be quoted in the output, which is bad especially for multi-line strings.

The code outputs nodeapp.

flyx
  • 35,506
  • 7
  • 89
  • 126
  • Great answer, it does miss the `echo $services_registry_ports` case though (admittedly I didn't directly address that case in the question). – Lee Mar 06 '23 at 14:13
  • 1
    @Lee You'd need to do `echo $services_registrey_ports_0` etc, the index is part of the path. You might be wondering whether you can put them into a bash array and that's definitely possible with enough `yq` magic, but mind that sequences in YAML can contain nested mappings where that wouldn't work. – flyx Mar 06 '23 at 14:31
  • Got it, thanks, and yes I was wondering re. array; `yq` magic seems to be a deep rabbit hole. A great tool! – Lee Mar 06 '23 at 14:34