JSON file modified with cat and jq ouputs differently when ran from function script

Question

I created a bash script that, amongst other things, modifies a json file with jq.

The line doing that is:

cat <<< $(jq '.eslintConfig.extends="@gbrachetta/eslint-config" | .+ {prettier: "@gbrachetta/prettier-config"}' package.json) > package.json

This works great, and running this bash script (MacOS) produces a perfectly formatted json file:

{
  "name": "test",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start"
  },
  "dependencies": {
    "next": "10.1.3",
    "prop-types": "^15.7.2",
    "react": "17.0.2",
    "react-dom": "17.0.2"
  },
  "devDependencies": {
    "@gbrachetta/eslint-config": "1.3.0",
    "eslint": "^7.24.0",
    "eslint-config-airbnb": "^18.2.1",
    "eslint-config-prettier": "^8.2.0",
    "eslint-plugin-import": "^2.22.1",
    "eslint-plugin-jsx-a11y": "^6.4.1",
    "eslint-plugin-prettier": "^3.4.0",
    "eslint-plugin-react": "^7.23.2",
    "eslint-plugin-react-hooks": "^4.2.0",
    "prettier": "^2.2.1"
  },
  "eslintConfig": {
    "extends": "@gbrachetta/eslint-config"
  },
  "prettier": "@gbrachetta/prettier-config"
}

I wanted to improve my script and created a bash script with functions.

The result is great, and the cat command modifies the json file as expected, but the resulting file isn't formatted and it's outputted in a single line.

This is what I get with my functional script:

{ "name": "test", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start" }, "dependencies": { "next": "10.1.3", "prop-types": "^15.7.2", "react": "17.0.2", "react-dom": "17.0.2" }, "devDependencies": { "@gbrachetta/eslint-config": "1.3.0", "eslint": "^7.24.0", "eslint-config-airbnb": "^18.2.1", "eslint-config-prettier": "^8.2.0", "eslint-plugin-import": "^2.22.1", "eslint-plugin-jsx-a11y": "^6.4.1", "eslint-plugin-prettier": "^3.4.0", "eslint-plugin-react": "^7.23.2", "eslint-plugin-react-hooks": "^4.2.0", "prettier": "^2.2.1" }, "eslintConfig": { "extends": "@gbrachetta/eslint-config" }, "prettier": "@gbrachetta/prettier-config" }

The cat line is identical in both scripts, and the resulting json (except for the formatting) is the same in both cases, but I don't understand why running the cat command from a function outputs the file in one line.

I'm really curious to know what I could be doing wrong!

Some versions of bash do weird things with `<<<` followed by an unquoted variable or command substitution (like this). Why not just skip that and `cat`, and redirect `jq`'s output directly to the file (`jq ... > package.json`)? — Gordon Davisson, Apr 23 '21 at 21:53
Really, Gordon's suggestion above is spot-on. This script is both inefficient and bug-prone compared to just doing the simple thing and not using `<<<` or `$(...)` or `cat` at all. It makes more sense to do things the easy way than to worry about the exact details of where, why and how the hard way goes wrong. — Charles Duffy, Apr 23 '21 at 21:57
Also, AFAICT, you aren't actually _showing_ us your function-based script, so how are we to reproduce the issue or test our fixes (even if we know the exact version of bash in use and can reproduce its bugs)? A [mre] is essential. — Charles Duffy, Apr 23 '21 at 22:00
(BTW, note the "unquoted" caveat -- `cat <<<"$(...)"` with the double quotes added shouldn't have those bugs, but it's also very silly when you could just run the parts in `...` on their own) — Charles Duffy, Apr 23 '21 at 22:02
Mind, there are other things you could be doing that would _also_ compress things down to one line. `var=$(yourfunction); echo $var` is one of those; if that were your mistake, this would be a duplicate of [I just assigned a variable, but `echo $variable` shows something else!](https://stackoverflow.com/questions/29378566/i-just-assigned-a-variable-but-echo-variable-shows-something-else) -- and it's the same fix, quoting correctly (`echo "$var"`). — Charles Duffy, Apr 23 '21 at 22:04
Really appreciate all these comments. I'll try and add a minimal reproducible example. In the meantime, I believe that adding the quotes solved my issue. — Guillermo Brachetta, Apr 23 '21 at 22:08
If that _is_ the problem, this is almost certainly duplicative of something else in the knowledge base (but you should also be able to avoid needing a herestring at all, and thus also needing to quote it). — Charles Duffy, Apr 23 '21 at 22:08
I'm sure I'm doing half of it wrong (this is the very first script I try!). If you're okay with it I will add the full script to my original question. You can certainly roast me! I'll learn from that! — Guillermo Brachetta, Apr 23 '21 at 22:10
Please put your code in http://shellcheck.net/ and fix what it finds before asking for human review. (Maybe delete it temporarily and undelete after it's ready). — Charles Duffy, Apr 23 '21 at 22:13
BTW, this is a stylistic thing, but the function declaration format you're using is something bash borrowed from legacy pre-POSIX ksh for compatibility purposes -- but bash's implementation _isn't_ quite compatible with ksh's, and using it makes your code POSIX-noncompliant altogether. Consider using the POSIX sh function declaration format instead: `name() { ...; }` with no `function` preceding. — Charles Duffy, Apr 23 '21 at 22:15
See https://wiki.bash-hackers.org/scripting/obsolete, particularly the entry in the third table table (`function name {` is better than `function name() {`, as the first table notes; but it's worse than just `name() {`) — Charles Duffy, Apr 23 '21 at 22:16
I'd also suggest `GREEN=$'\033[7;32m'` instead of `GREEN='\033[7;32m'` -- the `$'...'` version means that the literal itself gets escape sequences already expanded, so it works as-intended without `echo -e` or `printf %b` or something else that expands those sequences later. (`echo -e` has its own problems, and quite a lot of them; [Why is printf better than echo?](https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo) on [unix.se] goes into details) — Charles Duffy, Apr 23 '21 at 22:21
I also strongly recommend against trying to use recursion when looping is more fit-to-task. When you're inside the `stage1` function, and you call the `stage1` function again, that means that the the rest of the function (the parts _after_ the point of the recursive call) will be run twice, once when finishing the inner copy of `stage1`, then again as the outer copy of `stage1` finishes. Calling stage1 from stage2 is a questionable idea for similar reasons; just check that the `stage1` value the user entered is a directory _while still in stage1_, instead of deferring that check to stage2. — Charles Duffy, Apr 23 '21 at 22:22
...and while there's a lot of code floating around that flaunts this, note that POSIX specifies all-caps names to be used for variables that change behavior of OS-provided tools, and reserves names with at least one lower-case character to be used by applications. Thus, if you want to be certain that setting a variable won't inadvertently change your shell's behavior (a mistake lots of folks have made writing things like `for PATH in */; do` and being surprised their shell can't execute any external commands at all), that variable should have at least one lower-case character in its name. — Charles Duffy, Apr 23 '21 at 22:30
(Complicating that advice is that one of the many ways zsh chooses not to comply with the POSIX specification is having some lower-case names be special to it; but... well, this is one of the reasons those of us who think standardization is a good thing frown on zsh). — Charles Duffy, Apr 23 '21 at 22:31
Wow, I'm definitely copying all this to dig more into the subject. Wonderful comments! — Guillermo Brachetta, Apr 23 '21 at 22:35

score 0 · Answer 1 · answered Sep 02 '22 at 19:21

Simply use jq directly, without cat and without any indirections:

jq '.eslintConfig.extends="@gbrachetta/eslint-config" | . + {prettier: "@gbrachetta/prettier-config"}' package.json > package.json.tmp
mv package.json.tmp package.json

GNU sponge can help to simplify the script/avoid the second command.

JSON file modified with cat and jq ouputs differently when ran from function script

1 Answers1