Using yq in for loop bash

Question

I have a yaml array like below,

identitymappings:
- arn: "arn:aws:iam::12345567:role/AdmRole"
  group: "system:masters"
  user: "user1"
- arn: "arn:aws:iam::12345567:role/TestRole"
  group: "system:masters"
  user: "user2"

I am trying to parse this yaml in a bash script using for loop and yq.

 for identityMapping in $(yq read test.yaml "identitymappings[*]"); do
      roleArn=$identityMapping["arn"]
      group=$identityMapping.group
      user=$identityMapping.user
done

But I am not getting the expected results like not able to fetch the values of roleArn,group,user. Please let me know how to fix this.

This is very closely related to [iterate over json with jq](https://stackoverflow.com/questions/48005870/iterate-over-json-with-jq). — Charles Duffy, Jul 14 '20 at 16:30

mike.f · Answer 1 · 2021-11-29T23:17:46.383

17

The way I would do it is:

# load array into a bash array
# need to output each entry as a single line
readarray identityMappings < <(yq e -o=j -I=0 '.identitymappings[]' test.yml )

for identityMapping in "${identityMappings[@]}"; do
    # identity mapping is a yaml snippet representing a single entry
    roleArn=$(echo "$identityMapping" | yq e '.arn' -)
    echo "roleArn: $roleArn"
done

output:

roleArn: arn:aws:iam::12345567:role/AdmRole
roleArn: arn:aws:iam::12345567:role/TestRole

Disclaimer: I wrote yq

edited Nov 29 '21 at 23:17

answered Nov 29 '21 at 22:58

mike.f

1,586
13
14

1

note that the latest `yq` now also has a '@tsv' operator (so you can do it the way @charles-duffy posted without needing jq). – mike.f Dec 05 '21 at 00:18
1

The disclaimer though xD. – redzack Oct 09 '22 at 10:27

score 6 · Answer 2 · answered Jan 25 '22 at 00:02

6

I wasn't able to comment on Charles Duffy's proper answer, but this works for yq v4 without the use of jq...

while IFS=$'\t' read -r roleArn group user _; do
  echo "Role:  $roleArn"
  echo "Group: $group"
  echo "User:  $user"
done < <(yq e '.identitymappings[] | [.arn, .group, .user] | @tsv' test.yaml)

answered Jan 25 '22 at 00:02

lorendms

67
1
4

Good to know; BTW, TSV values might be escaped (with `\t` `\\ ` `\n` etc...); you can unescape them with for example `printf -v roleArn %b "$roleArn"` – Fravadona Jan 25 '22 at 00:36

score 4 · Answer 3 · answered Oct 10 '21 at 12:42

There is an improvement of @Rad4's answer that worked for me.

You can neatly loop through using latest yq and jq via:

for im in $(yq eval -o=j test.yaml | jq -cr '.identitymappings[]'); do
      arn=$(echo $im | jq -r '.arn' -)
      group=$(echo $im | jq -r '.group' -)
      user=$(echo $im | jq -r '.user' -)
      echo $arn $group $user
done

This loops through valid online jsons, which makes jq still work inside the loop.

score 3 · Answer 4 · answered Aug 05 '22 at 01:03

Get identitymappings length
Using index to access element of identitymappings

array_length=`yq e ". identitymappings | length - 1" test.yaml`

if [ $array_length -le 0 ] ; then
  exit
fi

for element_index in `seq 0 $array_length`;do
    arn=`yq e ".identitymappings[$element_index]. arn" test.yml`
    group=`yq e ".identitymappings[$element_index]. group" test.yml`
    user=`yq e ".identitymappings[$element_index]. user" test.yml`
done

How to get length of array in yq?
- https://mikefarah.gitbook.io/yq/operators/length

I find this approach the most straight forward. It allows to utilise yq for parsing data, instead of worrying about the outputs of yq and reading them in bash. The only thing I would change is replace backticks with $() — Ildar Galikov, May 22 '23 at 11:02

Charles Duffy · Answer 5 · 2020-07-14T17:24:49.970

The easiest way to read from jq or yq into bash is to use a BashFAQ #1 while read loop to handle line-oriented data; in the below, we use @tsv to generate line-oriented output:

while IFS=$'\t' read -r roleArn group user _; do
  echo "Role:  $roleArn"
  echo "Group: $group"
  echo "User:  $user"
done < <(yq -j read test.yaml \
         | jq -r '.identitymappings[] | [.arn, .group, .user] | @tsv')

Note that if you were using the Python yq rather than the Go one, you could remove the yq -j read and just use yq -r '...' in place of jq -r '...'.

score 1 · Answer 6 · answered Mar 14 '22 at 19:48

The answer by @mike.f is a good one. However, it does not work on OSX machines, because readarray is not an available command. You can read more about this here.

Here is the equivalent that would work on a mac:

# load array into a bash array
# need to output each entry as a single line
identitymappings=( $(yq e -o=j -I=0 '.identitymappings[]' test.yml ) )

for identityMapping in "${identityMappings[@]}"; do
    # identity mapping is a yaml snippet representing a single entry
    roleArn=$(echo "$identityMapping" | yq e '.arn' -)
    echo "roleArn: $roleArn"
done

score -1 · Answer 7 · answered Jul 14 '20 at 16:22

-1

I figured out..

for identityMapping in $(yq read test.yaml -j "identitymappings[*]"); do
      echo $identityMapping
      roleArn= echo $identityMapping | jq -r '.arn'
      echo $roleArn
      group= echo $identityMapping | jq -r '.group'
      echo $group
      user= echo $identityMapping | jq -r '.user'
      echo $user

answered Jul 14 '20 at 16:22

Rad4

1,936
8
30
50

No. This isn't assigning values to `roleArn`, `group` or `user` at all. Your output isn't written from the `echo`s, it's written by the `jq` commands. – Charles Duffy Jul 14 '20 at 16:24
That's because `roleArn= echo $identityMapping | jq -r '.arn'` doesn't assign to `roleArn`, it's just setting `roleArn` as an environment variable with an empty value during the execution of `echo`. – Charles Duffy Jul 14 '20 at 16:24
...see [How do I set a variable to the output of a command in bash?](https://stackoverflow.com/questions/4651437/how-do-i-set-a-variable-to-the-output-of-a-command-in-bash) – Charles Duffy Jul 14 '20 at 16:25

Using yq in for loop bash

7 Answers7