Need to modify an existing JQ filter of a GitHub CLI GraphQL response

Question

I am creating a GitHub Action workflow which will call a GitHub CLI API request using GraphQL. This gh api graphql response is --paginate and returns JSON Lines (ndjson).

I created the GraphQL and jq queries, and I am close to the desired output; however, my jq query needs to be modified and I can't figure out what to change.

First, here is the desired output format I want to achieve. Notice the single object that holds all the key-value lineage information.

[
  {
    KEY: VALUE,
    KEY: VALUE,
    ...
  }
]

And here is the actual format of the output that I am getting. Notice that every single key-value information is wrapped within its own object.

[
  {
    KEY: VALUE,
  },
  {
    KEY: VALUE,
  },
  ...
]

Here is my current jq query filter along with a snippet of the GraphQL response in jq play. It contains a snippet of 2 JSON Lines (jsonl, ndjson) entries (pretty printed). Search for data to see each individual response.

I need to --slurp/-s my jq query due to the paginated results.

I want to only include milestones which:

• have 100% progress
• don't include the word "withdrawn" within the title
• have issues associated with them.

Also, if the milestone title contains either or , , then I need to split the title. Each split will be its own key with identical values.

Here is my jq query that needs to be modified:

.[] | .data.repository as {
    nameWithOwner: $name, 
    milestones: { 
        nodes: $milestones
    }
}
| [
    foreach $milestones[] as $milestone (
        null; $milestone ; 
        $milestone
        | select($milestone.progressPercentage == 100)
        | select($milestone.title | contains("withdrawn") | not)
        | select($milestone.issues.nodes[])
        |
        {
            (($milestone.title | gsub(", "; " ") | split(" "))[]) : 
            [
                foreach $milestone.issues.nodes[] as $issue (
                    {}; . + { $issue };
                    $issue as $issue | $issue
                    | (reduce $issue.labels.nodes[] as $item ([]; . + [$item.name])) as $labels
                    |
                    {
                        repo: $name,
                        issue: $issue.number,
                        milestone: $milestone.number,
                        labels: $labels
                    }
                    
                )
            ]
        }
    )
]
| .

Here is a small JSON snippet which needs to be filtered by jq. It has 2 milestones but will output 3 key-value pairs (keys: C.1, EXAMPLE_SPLIT, and B.1.429):

{
  "data": {
    "repository": {
      "nameWithOwner": "cov-lineages/pango-designation",
      "milestones": {
        "pageInfo": {
          "hasNextPage": true,
          "endCursor": "Y3Vyc29yOnYyOpHOAGviZA=="
        },
        "nodes": [
          {
            "number": 1,
            "title": "C.1, EXAMPLE_SPLIT",
            "progressPercentage": 100,
            "issues": {
              "nodes": [
                {
                  "number": 2,
                  "labels": {
                    "nodes": [
                      {
                        "name": "proposed"
                      },
                      {
                        "name": "designated"
                      }
                    ]
                  }
                }
              ]
            }
          },
          {
            "number": 2,
            "title": "B.1.429",
            "progressPercentage": 100,
            "issues": {
              "nodes": [
                {
                  "number": 3,
                  "labels": {
                    "nodes": [
                      {
                        "name": "proposed"
                      },
                      {
                        "name": "designated"
                      }
                    ]
                  }
                }
              ]
            }
          }
        ]
      }
    }
  }
}

Answer 1

Something like this?

.data.repository
| .nameWithOwner as $repo
| .milestones.nodes
| map( # create a new array containing the milestones
select(.progressPercentage == 100) | select(.title | contains("withdrawn") | not) | select(.issues.nodes | length > 0) # filter interesting milestone nodes
  | {
    (.title | splits(",? ")): [ # one object per title part, each object containing an array
        { $repo, milestone: .number } # base milestone data, plus …
        + (.issues.nodes[] | {
          issue: .number,
          labels: [.labels.nodes[].name] # collect all label names in an array
        })
    ]
  })
| add # merge all objects of the array into a single object

Might not be the most efficient solution compared to a reduce-based approach (creates intermediate arrays), but can be easily followed and divided into "logical" parts.

Run with plain jq (no slurping, there's only a single top-level element)

Output with the example data from the question:

{
  "C.1": [
    {
      "repo": "cov-lineages/pango-designation",
      "milestone": 1,
      "issue": 2,
      "labels": [
        "proposed",
        "designated"
      ]
    }
  ],
  "EXAMPLE_SPLIT": [
    {
      "repo": "cov-lineages/pango-designation",
      "milestone": 1,
      "issue": 2,
      "labels": [
        "proposed",
        "designated"
      ]
    }
  ],
  "B.1.429": [
    {
      "repo": "cov-lineages/pango-designation",
      "milestone": 2,
      "issue": 3,
      "labels": [
        "proposed",
        "designated"
      ]
    }
  ]
}

---

If your input contains multiple objects and you want the final output to be a single object, use -s (--slurp) in combination with map(…):

jq -s 'map( # -s reads everything as one big array, `map` transforms the elements of this array
  .data.repository
  | .nameWithOwner as $repo
  | .milestones.nodes
  | map( # create a new array containing the milestones
    select(.progressPercentage == 100) | select(.title | contains("withdrawn") | not) | select(.issues.nodes | length > 0) # filter interesting milestone nodes
    | {
      (.title | splits(",? ")): [ # one object per title part containing an array
          { $repo, milestone: .number } # base milestone data, plus …
          + (.issues.nodes[] | {
            issue: .number,
            labels: [.labels.nodes[].name] # collect all label names in an array
          })
      ]
    }
  )
)
| add # merge all objects of the array into a single object
'

Answer 2

100 milestones: https://jqplay.org/s/hmpl7oz5K2z

[ 
  .data.repository 
  | .nameWithOwner as $repo 
  | .milestones.nodes[] 
  | select(
      .progressPercentage == 100 and 
      (.title | contains("withdrawn") | not) and 
      .issues.nodes[]
    )
  | 
  { 
    (.title | splits(",? ")): (
        { $repo, milestone: .number } + 
        (.issues.nodes[] | { issue: .number, labels: [ .labels.nodes[].name ] }))
  }
] 
| add

{
  "C.1": {
    "repo": "cov-lineages/pango-designation",
    "milestone": 1,
    "issue": 2,
    "labels": [
      "proposed",
      "designated"
    ]
  },
  "EXAMPLE_SPLIT": {
    "repo": "cov-lineages/pango-designation",
    "milestone": 1,
    "issue": 2,
    "labels": [
      "proposed",
      "designated"
    ]
  },
  "B.1.429": {
    "repo": "cov-lineages/pango-designation",
    "milestone": 2,
    "issue": 3,
    "labels": [
      "proposed",
      "designated"
    ]
  }
}