Extract JSON value to shell variable using jq

Question

I have a json response as shown below :

[
 {"id":10,
 "list_file":["/var/a.txt",
             "/dev/b.txt"]}
]

I need to extract values of list_file and store it shell variable as an array. I tried doing it looping through and reading the values.

#!/bin/bash
x=()
while read -r value
do
  #echo "$value"
  x+=("$value")
done < <(jq -r '.[] | .list_file' input.json)

But the extracted values in the array contains quotes, brackets and comma too.

[
    "/var/a.txt",
    "/dev/b.txt"
]

Could you please help me modify the code so that the array contains only the entries /var/a.txt and /dev/b.txt. Also, I tried readarray and map, but they won't work on Mac Osx. Any help would be really appreciated.

That's not valid json, could you please fix your example (by [editting](https://stackoverflow.com/posts/44981265/edit) the question) — Eric Renouf, Jul 08 '17 at 00:29

randomir · Answer 1 · 2017-07-08T01:27:18.633

3

You can use the @tsv string formatter in combination with --raw-output/-r option and split the output to bash array on tabs:

$ IFS=$'\t'
$ x=($(jq -r '.[] | .list_file | @tsv' input.json))
$ for xx in "${x[@]}"; do echo "$xx"; done
/var/a.txt
/dev/b.txt

For -r will strip the outer quotes (useful for making jq filters talk to non-JSON-based systems), and @tsv will output an array as a series of tab-separated strings (tabs, newlines, etc. will be escaped).

Alternatively, you use the @sh filter which outputs an array as a series of space-separated strings. However, to interpret such output, you have to evaluate it:

$ eval "x=($(jq -r '.[] | .list_file | @sh' input.json))"

edited Jul 08 '17 at 01:27

answered Jul 08 '17 at 00:35

randomir

17,989
1
40
55

Could you please let me know how to get rid of the quotes as I would have to loop over the specified paths and process the files. – tom Jul 08 '17 at 00:47
2

This isn't entirely correct. The output is quoted, but the shell itself won't treat the quotes as syntactic quotes in this context; try this with file names in `list_file` that contain whitespace. – chepner Jul 08 '17 at 01:06

score 3 · Accepted Answer · answered Jul 08 '17 at 01:24

In order to handle all values properly, you need to use the declare command to incorporate the output of jq into an array assignment.

Here's some input with some corner cases to worry about: whitespace, a newline, and a glob character.

$ cat input.json
[
    {"id":10,
    "list_file":[
        "/var/a b.txt",
        "/dev/c\nd.txt",
        "*"]}
]

A jq command that extracts and outputs properly quoted strings for use by the shell:

$ $ jq -r '.[] | .list_file[] | @sh' input.json
'/var/a b.txt'
'/dev/c
d.txt'
'*'

And a shell command that can make use of the output:

$ declare -a "x=($(jq -r '.[] | .list_file[] | @sh' input.json))"

Proof that it worked properly:

$ printf '==%s==\n' "${x[@]}"
==/var/a b.txt==
==/dev/c
d.txt==
==*==

Thank-you! Nice, subtle and tricky to google for this answer. — xer0x, Feb 18 '18 at 06:44

score 0 · Answer 3 · answered Jul 08 '17 at 08:25

On a Mac, it's easy enough to install a bash with readarray (e.g. using homebrew: brew install bash), so in the following I'll assume it's available, but you could just as well use the while read -r value technique to read values on a line-by-line basis. (One advantage of these line-oriented methods is that there's no need to fiddle with IFS.)
For the problem at hand, there may be situations in which it would be better to retain the JSON representation of special characters, e.g "\n" for NEWLINE, "\u0000" for NUL, etc. At any rate, a generic alternative to using -r is to use the -c command-line option instead, as illustrated below:

$ readarray -t x < <(jq -n -c '("\u0001\nb", "c\td", "e\u0000f")'
  | sed -e 's/^\"//' -e 's/\"$//' )
$ printf ":%s:\n" "${x[@]}"
:\u0001\nb:
:c\td:
:e\u0000f:

Extract JSON value to shell variable using jq

3 Answers3

Linked