Trouble Parsing JSON File and using an API PUSH for selected fields

Question

I am looking for advice on how to parse a file and select certain fields. The selected fields then need to be included in an API push to a separate system using curl.

The challenge is that the amount of fields needed may vary each time the file is populated. I can however use the keys "key" and the values provided after that in "value".

So in the following example I'm wanting to capture these values (though a separate message may have more or less):

{
    "result": {
        "problems": [{
            "id": "8103182743126107297_1579010239089V2",
            "startTime": 1579010239089,
            "endTime": -1,
            "displayName": "297",
            "impactLevel": "INFRASTRUCTURE",
            "status": "OPEN",
            "severityLevel": "AVAILABILITY",
            "commentCount": 0,
            "tagsOfAffectedEntities": [{
                "context": "CONTEXTLESS",
                "key": "JBOSS"
            }, {
                "context": "CONTEXTLESS",
                "key": "AC"
            }, {
                "context": "CONTEXTLESS",
                "key": "Host Name",
                "value": "servernameX.com"
            }, {
                "context": "CONTEXTLESS",
                "key": "Tomcat"
            }],
            "rankedImpacts": [{
                "entityId": "PROCESS_GROUP_INSTANCE-2XXXE58A78A142D7",
                "entityName": "service Tomcat *",
                "severityLevel": "AVAILABILITY",
                "impactLevel": "INFRASTRUCTURE",
                "eventType": "PROCESS_UNAVAILABLE"
            }, {
                "entityId": "PROCESS_GROUP_INSTANCE-A2F65660B538D9CF",
                "entityName": "JBoss EAP",
                "severityLevel": "AVAILABILITY",
                "impactLevel": "INFRASTRUCTURE",
                "eventType": "PROCESS_UNAVAILABLE"
            }],
            "affectedCounts": {
                "INFRASTRUCTURE": 2,
                "SERVICE": 0,
                "APPLICATION": 0,
                "ENVIRONMENT": 0
            },
            "recoveredCounts": {
                "INFRASTRUCTURE": 0,
                "SERVICE": 0,
                "APPLICATION": 0,
                "ENVIRONMENT": 0
            },
            "hasRootCause": true
        }],
        "monitored": {
            "INFRASTRUCTURE": 235,
            "SERVICE": 19,
            "APPLICATION": 1,
            "ENVIRONMENT": 1
        }
    }
}

Then... once those values are captured, convert them into a new JSON string like this (only the value of "Host" will change):

{
    "State": "OPEN",
    "ProblemID": "55555",
    "ImpactedEntity": "Alert from Maintenance Window",
    "ProblemTitle": "Process unavailable",
    "Host": "captured value 1(JBOSS), captured value 2(AC), captured value 3(HostName), captured value 4(servernameX.com),captured value 5(Tomcat), "
}

I need to fit them into an curl POST and have it execute.

curl -v -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '(the JSON above)' 'http://serverA.com/?apiKey=8675309:p'

This is a Linux box and I have python 2.7 available on it

So to summarize:

• We need to parse a flat file for the values of interest (call it file.x The amount of values may vary but there is always a key word/s to identify)

• We need to take those values of interest and place them in a CURL
  statement that does an API push (The format is listed above)

How can this be accomplished? Ultimately I will be scheduling this in a cron task for different times. Please advise and thank you so kindly all !

Answer 1

If jq is available, would you try the following:

i=1
while IFS=$'\n' read -r x; do
    str+=$(printf "captured value %d(%s), " "$i" "$x")
    ((i++))
done < <(jq -r '.. | .key? // empty, .value? // empty' < file.json)

cat <<EOS
{
    "State": "OPEN",
    "ProblemID": "55555",
    "ImpactedEntity": "Alert from Maintenance Window",
    "ProblemTitle": "Process unavailable",
    "Host": "$str"
}
EOS

Output:

{
    "State": "OPEN",
    "ProblemID": "55555",
    "ImpactedEntity": "Alert from Maintenance Window",
    "ProblemTitle": "Process unavailable",
    "Host": "captured value 1(JBOSS), captured value 2(AC), captured value 3(Host Name), captured value 4(servernameX.com), captured value 5(Tomcat), "
}

If you want to assign a variable json to the output above, modify the cat statement as:

json=$(cat <<EOS
{
    "State": "OPEN",
    "ProblemID": "55555",
    "ImpactedEntity": "Alert from Maintenance Window",
    "ProblemTitle": "Process unavailable",
    "Host": "$str"
}
EOS
)

Then you can embed the variable json in the curl command line.

[Explanations]

• The filter .. for the jq command traverses the json tree recursively and get all the values keyed by .key or .value. The null values are removed with // empty.
• The output of the jq command is fed to the while loop and the formatted values are appended to the string str one by one.