jq - can't append objects to array in while loop

Question

I've got this service-accounts-list.txt file with service accounts:

serviceAccount:p12345-001@gcp-sa-logging.iam.gserviceaccount.com
serviceAccount:p12345-002@gcp-sa-logging.iam.gserviceaccount.com
serviceAccount:p12345-003@gcp-sa-logging.iam.gserviceaccount.com
serviceAccount:p12345-004@gcp-sa-logging.iam.gserviceaccount.com
serviceAccount:p12345-005@gcp-sa-logging.iam.gserviceaccount.com

and this results.json file which I'd like to append data to:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    }
  ]
}

Expected result: I'd like to add an object for each line in service-accounts-list.txt, like so:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-001@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-002@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-003@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-004@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-005@gcp-sa-logging.iam.gserviceaccount.com"
    }
  ]
}

My bash script so far looks like this:

while read p;
do
  cat result.json | jq --arg email "$p" '.access += [{"role": "WRITER", "userByEmail": $email}]' > result.json
done < service-accounts-list.txt

The resulting result.json file is actually empty. If I redirect the output to for example > result2.json it correctly adds the last service account, like so:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-005@gcp-sa-logging.iam.gserviceaccount.com"
    }
  ]
}

so it seems the jq syntax is correct. I've tried adding the --unbuffered flag to no avail.

What am I missing?

For your convenience, [ShellCheck](https://www.shellcheck.net) automatically detects [this](https://github.com/koalaman/shellcheck/wiki/SC2094) and other common problems. — that other guy, May 11 '20 at 19:04
Thanks a lot for the tip. That looks useful for a bash scripting newbie like me — LundinCast, May 12 '20 at 18:36

score 3 · Answer 1 · answered May 11 '20 at 19:33

While obvious problem is that you are reading and redirecting to same file which you cannot do in bash and this answer has many useful explanations and work around for that.

But a better solution would be to totally avoid invoking jq multiple times in a loop and get it done in a single execution of jq.

# array to hold new values in json format
narr=()

# loop through input file and append json formatted values to array
while read -r line; do
    narr+=('{"role": "WRITER", "userByEmail": "'$line'"}')
done < service-accounts-list.txt

# call jq only once using -n option
jq -n 'input | .access += [inputs]' results.json <(printf '%s\n' "${narr[@]}")

Output:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-001@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-002@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-003@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-004@gcp-sa-logging.iam.gserviceaccount.com"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:p12345-005@gcp-sa-logging.iam.gserviceaccount.com"
    }
  ]
}

peak · Accepted Answer · 2020-05-12T06:41:16.177

3

Maybe the simplest efficient solution would be:

jq -nR --argfile result result.json '
  $result 
  | .access += [{role: "WRITER", userByEmail: inputs}] 
' service-accounts-list.txt

Notice -- no slurping, no splitting, no shell scripting. If the use of --argfile bothers you, feel free to use --slurpfile but "unslurp" $result by writing $result[0].

edited May 12 '20 at 06:41

answered May 11 '20 at 21:58

peak

105,803
17
152
177

score 2 · Answer 3 · answered May 11 '20 at 20:09

You can use jq itself without any shell processing. Process the plain text file using raw input mode -R over null input and start forming the resultant JSON from scratch.

The reduce() function iteratively runs over each entry int the text file and creates a object with role name and appends it to the access array.

jq -Rn \
   --slurpfile res result.json \
   '[ inputs | split("\n") | add ] as $data 
    | $res[] 
    | reduce $data[] as $d (.; .access += [{ role: "WRITER", userByEmail: $d  }] )' \
service-accounts-list.txt

jq - can't append objects to array in while loop

3 Answers3