format shell script logs and write to a file

Question

I need to log all outputs / error logs from a shell script into a file with a standard format.

For example for file script.sh:

echo "I want to be a json message!"
echo "I also want to be a json message!"

executed like below:

./script.sh >> jsonmessages.log

should give me a file content like:

{timestamp: '2020-12-31 00:00:00', message: 'I want to be a json message!'}
{timestamp: '2020-12-31 00:00:00', message: 'I also want to be a json message!'}

Léa Gris · Answer 1 · 2021-01-29T14:04:57.573

The best choice to reliably translate a shell string into a valid JSON string is to use a JSON parser/formatter. The most popular one is jq.

Here is an implementation of a timestamped JSON message logger using jq:

#!/usr/bin/env bash

json_logger() {
  while read -r msg || [ -n "$msg" ]; do
    jq \
      -nc \
      --arg msg "$msg" \
      '{ "timestamp": (now | strftime("%Y-%m-%d %H:%M:%S")), "message": $msg }'
  done
}


----------

EDIT:

As KamilCuk wrote:

We can do better without slow bash loop - it's just jq --raw-input '{ "timestamp": (now | strftime("%Y-%m-%d %H:%M:%S")), "message": . }'

So here is an improved json_logger:

json_logger() {
  jq \
    --raw-input \
    --compact-output \
    '{ "timestamp": (now | strftime("%Y-%m-%d %H:%M:%S")), "message": . }'
}

Addendum:

Harshit Kushwaha wrote:

For example, I need to print the method name in json from where the json_logger was called. How can I modify the json_logger and how to use it then in echo?

Here is an implementation with adding the method name if provided as an argument to the json_logger function:

#!/usr/bin/env bash

IFS= read -r -d '' __JQ_LOGGER_SCRIPT <<'JQSCRIPT'
{
  "timestamp": now | strftime("%Y-%m-%d %H:%M:%S"),
  "message": .
} |
  if ($name | length) != 0
  then
    . + { "method": $name }
  else
    .
  end
JQSCRIPT

json_logger() {
  jq \
    --raw-input \
    --compact-output \
    --arg name "$1" \
    "$__JQ_LOGGER_SCRIPT"
}

echo "I want to be a json message!" | json_logger
echo "I also want to be a json message!" | json_logger echo

printf %s $'I am a message with "quoted text" and some special characters: \'\t\7\42\\\'; that can only be properly converted to JSON with a JSON formatter and parser.' | json_logger printf

Produces this JSON outputs:

{"timestamp":"2021-01-29 14:02:46","message":"I want to be a json message!"}
{"timestamp":"2021-01-29 14:02:46","message":"I also want to be a json message!","method":"echo"}
{"timestamp":"2021-01-29 14:02:46","message":"I am a message with \"quoted text\" and some special characters: '\t\u0007\"\\'; that can only be properly converted to JSON with a JSON formatter and parser.","method":"printf"}

Thanks @Léa Gris. But with this I would be able to only log echo or printf messages. Can we somehow configure the function to be used for error messages as well that would be generated while running the scripts? — Harshit Kushwaha, Jan 21 '21 at 14:03
@HarshitKushwaha see: [How can I pipe stderr, and not stdout?](https://stackoverflow.com/q/2342826/7939871) — Léa Gris, Jan 21 '21 at 14:55
We can do better without slow bash loop - it's just `jq --raw-input '{ "timestamp": (now | strftime("%Y-%m-%d %H:%M:%S")), "message": . }'` — KamilCuk, Jan 22 '21 at 09:19
Thanks @KamilCuk and Léa Gris, I am able to print the stderr as well as stdout messages now using 2>&1. Can I also add some parameters in json? For example, I need to print the method name in json from where the json_logger was called. How can I modify the json_logger and how to use it then in echo? — Harshit Kushwaha, Jan 29 '21 at 07:22
@HarshitKushwaha I just added an implementation with an optional method name. Although next time, please don't expand your question after getting answers. JQ is a nice tool but it has its own scripting grammar that can be tough to use, but I encourage you try, again, and again with dummy tests for yourself. It is really worth the effort as this will save you time when processing JSON stuffs from shell. — Léa Gris, Jan 29 '21 at 14:08

Harshit Kushwaha · Accepted Answer · 2021-03-08T06:40:44.463

Thanks for the solutions.

I was able to write logs from script executions to the log file with below implementation:

Creating a json logger

  json_logger() {
         # Values for json
         current_timestamp="$(date +\"%Y-%m-%dT%H:%M:%S\")"
         param1="$param1value"
         param2="$param2value"
         log_level=$1
         script_name=$(basename $2 .sh)

         # Create json and re-direct to log file
         jq --raw-input --compact-output \
         '{
                 "@timestamp": '$current_timestamp',
                 "parameter1": "'$param1'",
                 "parameter2": "'$param2'",
                 "log-level": "'$log_level'",
                 "logger": "'$script_name'",
                 "message": .
         }' >> /var/log/app.log 2>&1 
     }

And then executing scripts as below

   "script.sh" 2>&1 | json_logger "INFO" {job_name or script_name}

Raman Sailopal · Answer 3 · 2021-01-22T09:13:48.690

1

./script.sh | awk -v squot="'" '{ print "{\"timestamp\": \""strftime("%Y-%m-%d %H:%M:%S")"\", \"message\": \""$0"\"}" }' mess | jq >> jsonmessages.log

Using GNU awk, print the timestamp in the required format using awk's strftime function, along with double quotes and the other data in the format required. Pipe through to jq to ensure valid json

edited Jan 22 '21 at 09:13

answered Jan 21 '21 at 12:34

Raman Sailopal

12,320
2
11
18

awk cannot reliably produce valid JSON strings – Léa Gris Jan 21 '21 at 12:38
Then you can pipe through to jq to ensure that you have valid json – Raman Sailopal Jan 22 '21 at 09:14

format shell script logs and write to a file

3 Answers3