0

I have a little script that reads a json file:

{
    "path": "$HOME/Projects:$HOME/Github"
}

I want to read path value, split on colon : and then read out the two paths with $HOME expanded.

#!/bin/sh
path_list="$(jq -r '.path' < "$JSON_FILE" | tr ':' '\n')"
echo "$path_list" | while IFS= read -r line; do
    echo "$line"
done;

The output is not what I would expect:

$HOME/Projects
$HOME/Github

Yet when I run echo "$HOME/Projects" the $HOME parameter expands fine.

Initially, I thought that I needed double quotes around the variable so I tried echo "\"${line}\"" and that just prints "$HOME/projects". I am confused. Can anyone please shed some light on this for me or point to a good tutorial on bash parameter expansion?

Regarding another SO question addressing similar issue. I do not think this is the same because that OP was asking about expanding strings loaded from a file. I do not think that is the dominant concern in my question. Other responses to this question involve using eval which I would like to because users will be entering their own inputs. Other solutions rely on external packages like gettext. I believe there should be a straight forward answer here.

Jeff
  • 2,293
  • 4
  • 26
  • 43
  • 3
    Parameter expansions are not recursively expanded; `$HOME` is just a literal string in the result of `"$line"`, not a further expansion to process. – chepner Feb 12 '20 at 18:19
  • How would I expand the vars in the `path_list` declaration above the while loop? – Jeff Feb 12 '20 at 18:25
  • @Jeff: Can other environment variables be possible or only `$HOME` is part of the `path` – Inian Feb 12 '20 at 18:28
  • Do you have control over the JSON file? I would recommend making it a `jq` filter instead of JSON, so that `jq` can do the substitution itself. – chepner Feb 12 '20 at 18:29
  • @Inian any env vars should be allowed not just `$HOME` – Jeff Feb 12 '20 at 18:32
  • @Jeff, ...note that jq supports environment variables. Change it from `{"path": "$HOME/whatever"}` to `{"path": "\($ENV.HOME)/whatever"}` and process it as jq source code (as chepner describes above), and there you are. – Charles Duffy Feb 12 '20 at 18:44
  • @chepner can you expand on what you mean by recursively expanded? Are you saying that a string like "$HOME/Projects" -- if set as a variable can never be expanded? I'm just trying to understand what is going on here. – Jeff Feb 12 '20 at 19:16
  • `$line` expands to `$HOME/...`; that string does not further expand to `/home/jeff/...`. – chepner Feb 12 '20 at 19:39
  • @Jeff, ...insofar as this is still open -- whether a string is loaded from a file of from the stdout of a different process (like a command substitution) is completely immaterial. The important commonality is that that string is *data, not code*. – Charles Duffy Feb 12 '20 at 22:56
  • When you run `echo "$HOME/Projects"`, `$HOME` is code, not data, so it's subject to command substitution, replacing `$HOME` with a the current value of the `HOME` variable. – Charles Duffy Feb 12 '20 at 22:57
  • ...by contrast, when you load a string that contains `$HOME` from a file, *or* read it from the output of `jq`, it's not code, it's data. For data to be implicitly treated like code would be like using `eval` everywhere -- it would make the language impossible to use in any circumstance where you didn't own, control and trust all the data it could possibly be asked to process. Hence the previously-proposed duplicate "smelling" right to people who understand the problem space. :) – Charles Duffy Feb 12 '20 at 22:58
  • ...sure, you're not loading content from a file directly, *but the problem you face is exactly identical to those you'd face if you were*, and consequently, the set of available solutions is identical as well. – Charles Duffy Feb 12 '20 at 23:00
  • That's what chepner was saying earlier about parameter expansion not being recursive -- your code goes through parameter expansion and command substitution during evaluation, but the results of those operations never backtrack to earlier [stages in the parsing process](https://stuff.lhunath.com/parser.png) and run the operations at hand a second time unless that (dangerous!) behavior is explicitly requested. (If the diagram linked above is a little too terse, see also the [BashParser](https://mywiki.wooledge.org/BashParser) wiki page). – Charles Duffy Feb 12 '20 at 23:04
  • @CharlesDuffy @chepner thank you both so much for your insights into bash. I think I have a workaround. I am only going to allow the $HOME env var or absolute paths in the json file. Then once I have the string "$HOME/jeff:$HOME/Github" I will simply do a replacement `paths="${path_list//\$HOME/$HOME}"`. I'll of course check to see that `$HOME` is defined first and will make sure that the resolved path is a directory before I do anything on it -- which is just a read operation anyways (find). – Jeff Feb 13 '20 at 15:01
  • Also, I have closed the issue as a duplicate @CharlesDuffy. – Jeff Feb 13 '20 at 15:02

1 Answers1

-2

Solved by adding 'eval':

#!/bin/sh
path_list="$(jq -r '.path' < "$JSON_FILE" | tr ':' '\n')"
echo "$path_list" | while IFS= read -r line; do
    eval echo "$line"
done;

More info: https://unix.stackexchange.com/questions/23111/what-is-the-eval-command-in-bash

jorgeteixe
  • 1
  • 1
  • 3