Processing command substitutions in strings embedded in a .json file with jq

Question

I want to read environment variables from .json-file:

{
    "PASSPHRASE": "$(cat /home/x/secret)",
}

With the following script:

IFS=$'\n'
for s in $(echo $values | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" $1); do
    export $s
    echo $s
done
unset IFS

But the I got $(cat /home/x/secret) in PASSPHRASE, and cat is not executed. When I execute the line export PASSPHRASE=$(cat /home/x/secret), I got the correct result (content of file in environment variable). What do I have to change on my script, to get it working?

What's in `$values`? Probably fix the bugs reported by http://shellcheck.net/ before asking for human assistance. — tripleee, Aug 04 '21 at 08:15
A JSON file is just data; you can't embed a shell command and expect it to be executed when you read the file (think about the security consequences if somebody gave you a JSON file which contained `$(sudo rm -rf /)`...) — tripleee, Aug 04 '21 at 08:17
I rolled back your recent edit. You are welcome to post an answer of your own if you like, but your question should remain strictly a question. Having said that, your solution to evaluate arbitrary code in the shell is quite error-prone. A better solution would be to use a template of some sort and change your loop to replace a particular placeholder with some static text (which can come from a file, that's not a problem). It should not be hard to find similar questions with solutions which do just that. https://stackoverflow.com/a/54184956/874188 might be useful as a starting point. — tripleee, Aug 04 '21 at 10:01
I'm using a json-file for setting environment variables for my node application in a docker container. I also need to set some variables for my docker-compose- and Docker-File. I don't want to use two separate files, so I'm used to have the json-file also for shell. The shell-script itself has execution and all other rights only for user root, so it seems to be no security aspect. — user2625247, Aug 05 '21 at 05:38
"Only for user root" -- you realize root can read and write files owned by -- and otherwise impersonate -- any other user, right? — Charles Duffy, Aug 05 '21 at 05:38
Anyhow, the original title didn't describe anything that was on its face a problem, because it's normal, expected, and _desired_ for content in JSON files to be treated as literal for the reasons triplee described. The new title describes _what you're trying to do_, which is different from what most people using jq are going to inspect or intend. — Charles Duffy, Aug 05 '21 at 05:40

score 1 · Accepted Answer · answered Aug 04 '21 at 08:47

When you do export PASSPHRASE=$(cat /home/x/secret) in the shell, it interpretes the $() expression, executes the command within and puts the output of it inside of the variable PASSPHRASE.

When you place $() in the json file, however, it is read by jq and treated as a normal string, which is the equivalent of doing export PASSPHRASE=\$(cat /home/x/secret) (notice the slash, which causes the dollar sign to be escaped and treated as a literal character, instead of creating a new shell). If you do that instead and try to echo the contents of the variable it will have similar results as running your script.

If you want to force bash to interpret the string as a command you could use sh -c <command> instead, for example like this:

test.json:

{
    "PASSPHRASE": "cat /home/x/secret"
}

test.sh:

IFS=$'\n'
for s in $(echo $values | jq -r "to_entries|map(\"\(.value|tostring)\")|.[]" $1); do
    echo $(sh -c $s)
done
unset IFS

This prints out the contents of /home/x/secret. It does not solve your problem directly but should give you an idea of how you could change your original code to achieve what you need.

I'd strongly suggest turning off globbing before running this code. And `IFS=$'\n'` is helpful, but it's not a full replacement for correct quoting -- it suppresses some string splitting, but it doesn't suppress globbing, so there are still risks of filename contents being parsed as code when you have unquoted expansions used to generate strings passed as arguments to `sh`. — Charles Duffy, Aug 05 '21 at 05:44

score 0 · Answer 2 · answered Aug 05 '21 at 05:33

Thanks to Maciej I changed the script and got it working:

IFS=$'\n'
for line in $(jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" "$1"); do
    lineExecuted=$(sh -c "echo $line")
    export "$lineExecuted"
    echo "$lineExecuted"
done
unset IFS

Processing command substitutions in strings embedded in a .json file with jq

2 Answers2