Modify a key-value in a json using jq in-place

Question

I have a json in which I want to modify a particular value but the terminal always displays the json with the modified value but it does not actually change the value in the particular file.

Sample json:

{
    "name": "abcd",
    "age": 30,
    "address": "abc"
}

I want to change the value of address in the file itself but so far I've been unable to do so. I tried using:

jq '.address = "abcde"' test.json

but it didn't work. Any suggestions?

Does this answer your question? [Jq to replace text directly on file (like sed -i)](https://stackoverflow.com/questions/36565295/jq-to-replace-text-directly-on-file-like-sed-i) — Andy, Apr 05 '20 at 20:27

chepner · Answer 1 · 2019-12-04T15:37:49.110

160

Use a temporary file; it's what any program that claims to do in-place editing is doing.

tmp=$(mktemp)
jq '.address = "abcde"' test.json > "$tmp" && mv "$tmp" test.json

If the address isn't hard-coded, pass the correct address via a jq argument:

address=abcde
jq --arg a "$address" '.address = $a' test.json > "$tmp" && mv "$tmp" test.json

edited Dec 04 '19 at 15:37

answered Mar 10 '17 at 12:38

chepner

497,756
71
530
681

1

this worked for a hardcoded string. Is there any solution for a variable i.e $address – ahmed_khan_89 Oct 29 '19 at 15:31
1

@ahmed_khan_89 you can use jq '.address = "'${address}'"' – Pujan Dec 04 '19 at 14:25
16

No, do not interpolate strings into a `jq` filter. Use `jq --arg a "$address" '.address = $a'` instead. – chepner Dec 04 '19 at 15:36
3

@chepner How come you don't recommend interpolating the string? It works when I use Pujan's method – Alexander D Jun 09 '20 at 21:49
1

It's an enormous headache to deal with the quoting. Use --arg – hagemt Oct 13 '20 at 20:40
3

@AlexanderD Because interpolation doesn't necessarily pass an argument to your filter; it *builds* a filter, and that filter depends on how the variable expansion is parsed. – chepner Oct 13 '20 at 21:06
Notice the file owner and file permission of the `test.json` will be altered due to use of `mv`. I usually use `cat` instead. – VCD Dec 04 '20 at 03:15
1

`mv` is atomic; `cat` is not. You can adjust the permissions and owner of the temp file before executing `mv`, but preventing another process from accessing `test.json` while `cat` overwrites its contents is more difficult. – chepner Dec 04 '20 at 12:37

score 111 · Accepted Answer · answered Mar 10 '17 at 11:16

111

AFAIK jq does not support in-place editing, so you must redirect to a temporary file first and then replace your original file with it, or use sponge utility from the moreutils package, like that:

jq '.address = "abcde"' test.json|sponge test.json

There are other techniques to "redirect to the same file", like saving your output in a variable e.t.c. "Unix & Linux StackExchange" is a good place to start, if you want to learn more about this.

answered Mar 10 '17 at 11:16

zeppelin

8,947
2
24
30

Unfortunately moreutils / sponge is unavailable on CentOS 8 atm.. – willemdh Oct 19 '19 at 12:40
27

Without `sponge`: `echo "$( jq '.address = "abcde"' test.json )" > test.json` – codekandis Aug 03 '20 at 09:35
2

Be careful! Pass the filename as an argument to `sponge`, as in the answer. This is wrong: `jq . test.json | sponge > test.json` , you must do `jq . test.json | sponge test.json` – Flimm Sep 25 '20 at 17:14
1

An additional nod for getting me to look at sponge(1) – sberder Feb 17 '21 at 04:24
2

do not redirect to the same file as suggested by @codekandis comment. this will not always work. Large files it will cause issues, also with whitespaces, non-printable and escapment sequences. Never redirect a file to itself, it is always a bad idea. See sponge or use a temp file, just don't try to do it differently unless you understand what is happening. – Lynch Jul 18 '21 at 19:22

Andy · Answer 3 · 2022-02-09T14:09:01.417

68

Temp files add more complexity when not needed (unless you are truly dealing with JSON files so large you cannot fit them in memory (GB to 100's of GB or TB, depending on how much RAM/parallelism you have)

The Pure bash way.

contents="$(jq '.address = "abcde"' test.json)" && \
echo -E "${contents}" > test.json

Pros

No temp file to juggle
Pure bash
Don't need an admin to install sponge, which is not installed by default
Simpler

Cons

This works perfectly fine for json because it cannot contain a literal null character. If you were to try this outside the json arena, it would fail when a null is encountered (and you would have to do some encoding/decoding workarounds). Bash variables cannot store literal nulls.

Note: this can not be combined as "one command" (like @codekandis suggested), since redirection sometimes starts before the left hand side (LHS) of an expression is run, and starting redirection before running jq erroneously empties the file, hence two separate commands. It may "seem" to work when you try it, but this is misleading and has a very high probability of failing as soon as the circumstances change.

Update: Added -E option to disable escape characters just in case you are on systems where they are interpreted by default. (Which I've never actually seen)

edited Feb 09 '22 at 14:09

answered Apr 05 '20 at 21:18

Andy

2,982
1
19
23

4

This answer is particularly useful for things such as bumping the version on your package.json (e.g. `contents="$(jq '.version = "$version"' package.json)" && echo "${contents}" > package.json`) – Caleb Faruki Dec 11 '20 at 11:11
This shorter version seems to work: `echo -E $(jq '.address = "abcde"' test.json) > test.json` – Pieter Aug 10 '22 at 18:25
2

@Pieter you should not do that. It is extremely unreliable and will fail once the circumstances change (file size, formatted file system, CPU usage, phase of the moon). As soon the redirection on the RHS opens and clears the `test.json` file the `jq` running is now reading file contents that are most likely missing. It mat seem to work, but this is not safe. – Andy Aug 10 '22 at 18:59
Thanks @Andy, I also noticed that it prints everything in one line, so it doesn't even work. – Pieter Aug 10 '22 at 19:12
1

@Pieter The reason why everything was one line was because you didn't put quotes around the $() expression. You lose **all** preservation of whitespaces when you do that in bash. You technically don't need the quotes in the `contents=` example, because those quote are implicit. But notice I do have quotes in the echo command, that is important, or else all multiple spaces/tabs/newlines are replaced with single spaces. If we want compact output, use `jq -c` – Andy Aug 12 '22 at 12:26
I think `contents` variable also needed to be unset to prevent memory leaks if the script runs forever. – Saber Hayati Jul 22 '23 at 05:55

score 11 · Answer 4 · answered Apr 04 '20 at 16:05

Just to add to chepner answer and if you want it in a shell script.

test.json

{
  "name": "abcd",
  "age": 30,
  "address": "abc"
}

script.sh

#!/bin/bash
address="abcde"
age=40

# Strings:
jq --arg a "${address}" '.address = $a' test.json > "tmp" && mv "tmp" test.json

# Integers:
jq --argjson a "${age}" '.age = $a' test.json > "tmp" && mv "tmp" test.json

mahendra rathod · Answer 5 · 2021-04-06T07:01:33.040

Example for nested json with changing single and multiple values.

config.json

{
  "Parameters": {
    "Environment": "Prod",
    "InstanceType": "t2.micro",
    "AMIID": "ami-02d8e11",
    "ConfigRegion": "eu-west-1"
  }
}

with the below command, you can edit multiple values.

tmp=$(mktemp)
jq '.Parameters.AMIID = "ami-02d8sdfsdf" | .Parameters.Environment = "QA"' config.json > "$tmp" && mv "$tmp" config.json

with the below command, you can edit single value.

tmp=$(mktemp)
jq '.Parameters.AMIID = "ami-02d8sdfsdf"' config.json > "$tmp" && mv "$tmp" config.json

score 7 · Answer 6 · answered Sep 15 '21 at 15:34

7

this should work

address = aaaaa
echo $(jq --arg a "$address" '.address = ($a)' test.json) > test.json

for whatever reason, without the echo, it makes a bin file and my python script was not able to parse it.

answered Sep 15 '21 at 15:34

Delcroip

101
1
3

Streaming the result into the input file is generally risky in shell scripts: `>` normally causes the output file to be truncated before the rest of the command is executed. Possibly the subshell command `$(...)` postpones this until after the execution of the shell script. In general I would always use a command line option to deal with this special case, or lacking that, write to an intermediate file. – user1010997 Nov 17 '21 at 09:10

score 1 · Answer 7 · answered Sep 12 '22 at 20:40

I took the best of a couple answers here and here.

This uses a parameter named actionname as an input to an assignment of the name property at the document level. ACTION_NAME is just an envvar I want to use as the replacement value.

contents="$(jq --arg actionname ${ACTION_NAME} '.name = $actionname' ./${ACTION_NAME}/package.json)" && \
echo -E "${contents}" > ${ACTION_NAME}/package.json;

score 0 · Answer 8 · answered May 23 '23 at 17:00

0

The simple answer is just to store interim JSON to a variable rather than a file.

JSON=$(jq '.address = "abcde"' test.json)
echo "$JSON" > test.json

answered May 23 '23 at 17:00

donatJ

3,105
3
32
51

Danila Vershinin · Answer 9 · 2022-10-25T07:47:34.443

-3

I didn't like any of the solutions and created the sde utility.

pip install sde

Then, e.g. for the following JSON data:

{
  "Parameters": {
    "Environment": "Prod",
    "InstanceType": "t2.micro",
    "AMIID": "ami-02d8e11",
    "ConfigRegion": "eu-west-1"
  }   
}

you can simply do:

sde Parameters.Environment Dev test.json

edited Oct 25 '22 at 07:47

answered Jun 30 '21 at 19:12

Danila Vershinin

8,725
2
29
35

Modify a key-value in a json using jq in-place

9 Answers9

Pros

Cons

Linked

Related