Searching and replacing values recursively in the same time

Question

I have a folder of Ansible playbooks, in which I want to grep specific YAML block and replace one of the Jinja2 variables with its value from the playbook folder.

For example:

grep -R -A 35 -w -h 'test_profiles:' .

test_profiles:
  - name: test_1
  source: '{{ service_name }}'

So I want to interpolate the service_name variable with its value

What I tried is to grep for the service_name but couldn't find a way to recursively replace the variable with its respective search result for each playbook folder

Folder structure:

playbook 1
- main.yml < contains service_name value
- tasks
  - file.yml < contains test_profiles block which I need to replace the service_name within it
playbook 2
- main.yml
- tasks
  - file.yml
... etc

main.yml from playbook 1 sample:

---
service_name: 'service 1'

main.yml from playbook 2 sample:

---
service_name: 'service 2'

file.yml from playbook 1 sample:

---
test_profiles:
  - name: test_1
  source: '{{ service_name }}'

other_header:
   key: value

file.yml from playbook 2 sample:

---
test_profiles:
  - name: test_2
  source: '{{ service_name }}'

another_header:
   another_key: value

Expected result:

test_profiles:
  - name: test_1
  source: 'service 1'
test_profiles:
  - name: test_2
  source: 'service 2'
test_profiles:
  - name: test_3
  source: 'service 3'

grep command I tried to get the service_name variable:

grep -R -oE 'service_name=([a-zA-Z]*-*[a-zA-Z]*)*|service_name:(\s*[a-zA-Z]*-*[a-zA-Z]*)*' .

But I couldn't find a way to combine both commands, to result in the expected result above.

While I'm sure some clever `awk` person will give you something here, quite frankly I'd probably write a small python script :D — michjnich, Sep 11 '20 at 09:30
I will try writing it now, was thinking the same as well, thanks @urbanespaceman — Wazery, Sep 11 '20 at 09:33
Please update your question so that you have 2 rows of sample file main.yml and 2 blocks of file.yml and the exact result for them. — thanasisp, Sep 11 '20 at 09:39
Do you want to edit the file.yml files in place, or you need the output only as you indicated in the expected result? — M. Nejat Aydin, Sep 11 '20 at 10:38
Are there more lines into `file.yml` that you don't want? And want to keep only the `test_profiles` block, which ends with an empty line? — thanasisp, Sep 11 '20 at 11:07

thanasisp · Accepted Answer · 2020-09-11T12:57:13.150

This solution runs one awk per playbook. Parse the main.yml and store the value of service_name. Parse the second file and print only the lines block (starting with test_profiles: and ending with a line empty or with only white-space characters) with the variable replaced. You execute the script below from the root of your directory structure.

#!/bin/bash

for d in playbook*; do
    [[ -f "$d"/main.yml ]] && [[ -f "$d"/tasks/file.yml ]] || continue
    awk -F"'" 'FNR==NR && /^service_name:/{s=$2;next} {FS=OFS}
           /test_profiles:/ {f=1}
           f {gsub("{{ service_name }}",s)}
           f && (NF==0) {f=0}
           f' "$d"/main.yml "$d"/tasks/file.yml
done

Also if there is no value in the main.yml, this prints empty string for {{ service_name }}, if this is not a feature but a bug, we can change it. And any playbook not having both files at place, is being ignored.

Reference:

How to read two files with awk (NR==FNR)

gsub and the other GNU awk string functions

How to print using a flag a block between start and end pattern

Searching and replacing values recursively in the same time

1 Answers1