1

I'm trying to extract a series of properties (named in an input file) in jq and getting error when I feed those from bash via a loop:

while read line; do echo $line; cat big.json | jq ".$line"; sleep 1; done < big.properties.service

cfg.keyload.service.count
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:

When i try to do it manually it works

$ line=cfg.keyload.service.count
$ echo $line
cfg.keyload.service.count
$ cat big.json | jq ".$line"
1

Is there any way to get it work in loop?

Here is example

cat >big.json <<EOF
{
  "cfg": {
    "keyload": {
      "backend": {
    "app": {
      "shutdown": {
        "timeout": "5s"
                  },
      "jmx": {
        "enable": true
                    }
           }
       }
     }
}
}
EOF

cat >big.properties.service <<EOF
cfg.keyload.backend.app.shutdown.timeout
cfg.keyload.backend.app.jmx.enable
cfg.keyload.backend.app.jmx.nonexistent
cfg.nonexistent
EOF

...output should be:

cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
Vladislav Yasenkov
  • 13
  • 3
  • There's no good reason to do the looping in bash instead of in jq. – Charles Duffy Apr 19 '18 at 12:15
  • Please try to add a [mcve] -- complete code someone else can run themselves to see your problem, just as my answer creates input files needed to test that it works. – Charles Duffy Apr 19 '18 at 12:29
  • It also would be helpful to run your script with `bash -x yourscript` to log every command it runs. If, for instance, your input has DOS newlines instead of Windows newlines, that log will make it obvious. – Charles Duffy Apr 19 '18 at 12:31
  • `cat` does not show nonprintable characters on its own. If you're on a GNU system, use `cat -A` -- or, as I suggested before, amend in logs from `bash -x yourscript` showing the failure. – Charles Duffy Apr 19 '18 at 13:45

2 Answers2

0

Immediate Issue - Invalid Input

The "invalid character" at hand here is almost certainly a carriage return. Use dos2unix to convert your input file to a proper UNIX text file, and your original code will work (albeit very inefficiently, rereading your whole big.json every time it wants to extract a single property).

Performant Implementation - Loop In JQ, Not Bash

Don't use a bash loop for this at all -- it's much more efficient to have jq do the looping.

Note the sub("\r$"; "") used in this code to remove trailing carriage returns so it can accept input in DOS format.

jq -rR --argfile infile big.json '
sub("\r$"; "") as $keyname
| ($keyname | split(".")) as $pieces
| (reduce $pieces[] as $piece ($infile; .[$piece]?)) as $value
| ($keyname, ($value | tojson))
' <big.properties.service

properly emits as output, when given the inputs in the question:

cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • That works. Great thanks to you. But could please you explain syntax you used in jq. And is there any way to "echo" each line in loop? – Vladislav Yasenkov Apr 19 '18 at 14:50
  • What's your real goal, with respect to the `echo`s? If you want to intersperse the keys with the outputs, that's easy enough. (Personally, I'd generate tab-separated output, putting each key and its value on the same line). – Charles Duffy Apr 19 '18 at 14:52
  • ...as for the jq syntax, it's all documented at https://stedolan.github.io/jq/manual/. Feel free to ask a specific question that you still have after reading that documentation. – Charles Duffy Apr 19 '18 at 14:52
  • my goal is to validate params file with json. For example if i do `cat big.json | jq '.cfg.keyload.service.count.foo'` i get null because there is no such key in json. But if is use your syntax and put defunct param i get empty value, not null. Can you explain how to get null on nonexisting key? – Vladislav Yasenkov Apr 20 '18 at 07:48
  • So that way when i will get `cfg.keyload.backend.app.shutdown.timeout 5s` `cfg.keyload.backend.app.jmx.enable true` `cfg.keyload.backend.app.jmx.enabled NULL` `cfg.keyload.backend.pepe.jmx.enable false` I will raise alarm on NULL value assuming there is no such key. – Vladislav Yasenkov Apr 20 '18 at 08:32
  • That said, if you want that behavior, you should edit the question to show it -- ie. have an example with "null" output in the question itself. Otherwise, it's not part of the specification you asked answers to follow, so an answer can be correct/valid even if it doesn't do that. – Charles Duffy Apr 20 '18 at 15:02
0

Your properties file is effectively paths in the json that you want to retrieve values from. Convert them to paths that jq recognizes so you can get those values. Just make an array of keys that would need to be traversed. Be sure to read your properties file as raw input (-R) since it's not json, and use raw output (-r) to be able to output the paths as you want.

$ jq --argfile big big.json '
    ., (split(".") as $p | $big | getpath($p) | tojson)
' -Rr big.properties.service
cfg.keyload.backend.app.shutdown.timeout
"5s"
cfg.keyload.backend.app.jmx.enable
true
cfg.keyload.backend.app.jmx.nonexistent
null
cfg.nonexistent
null
Jeff Mercado
  • 129,526
  • 32
  • 251
  • 272