Python multiple replacement with matching

Question

Work in Progress Json2yaml converter for Ansible (http://docs.ansible.com/ansible/ec2_group_module.html) creation

Link to Project: https://github.com/shdobxr/json2yamlAnsible/blob/master/json2yaml.py

Answered! See above for project.

I have been researching this on stack. And it's just not clicking.

Here is my attempt, please be kind I have dabbled with python in the past. The rust is showing here:

#!/usr/local/bin/python

import re, sys, json, yaml

with open(sys.argv[1]) as f:
#   print yaml.safe_dump(json.load(f), default_flow_style=False)
   a = yaml.safe_dump(json.load(f), default_flow_style=False)

c = sys.argv[1] + ".yml"

text_file = open(c, "w")
text_file.write(a)
text_file.close

replacements = {
    '    ToPort:': '            to_port:',
    '  - FromPort:': '            from_port:',
    'UserIdGroupPairs:': '',
    'Vpc:': '',
    'VpcPeeringConnectionId:': '',
    'UserIdGroupPairs:': '',
    'PrefixListIds: []': '',
    '- Description:': '        description:',
    '  GroupName:': '    - name:',
    '  -     IpProtocol:': '          - proto:',
    '- Description:': '        description:',
    'SecurityGroups:': '- name:',
    '  IpPermissions:': '        rules:',
    '  IpPermissionsEgress:': '        rules_egress:',
    '  GroupId:': '',
    '    - GroupId:': '            group_id:'
}


replacements = dict((re.escape(k), v) for k, v in replacements.iteritems())
#pattern = re.compile('|'.join(replacements.keys()))

remove_pattern = re.compile('|'.join(k for k, v in replacements.iteritems() if v is None))
replace_pattern = re.compile('|'.join(k for k, v in replacements.iteritems() if v is not None))


def rewrite(text):
    if remove_pattern.search(text):
        return ''
    return replace_pattern.sub(lambda m: replacements[re.escape(m.group())], text)


with open(c, 'rt') as fin:
    with open('out.txt', 'wt') as fout:
        for line in fin:
            fout.write(rewrite(line))

Test file (save this as test.json)

{
    "SecurityGroups": [
        {
            "IpPermissionsEgress": [
                {
                    "PrefixListIds": [],
                    "FromPort": 22,
                    "IpRanges": [
                        {
                            "CidrIp": "10.0.0.0/24"
                        }
                    ],
                    "ToPort": 22,
                    "IpProtocol": "tcp",
                    "UserIdGroupPairs": [
                        {
                            "UserId": "XXXXXXXXXXXX",
                            "GroupId": "sg-xxxxxxxx"
                        }
                    ]
                },
                {
                    "PrefixListIds": [],
                    "FromPort": 3389,
                    "IpRanges": [
                        {
                            "CidrIp": "10.0.0.0/24"
                        }
                    ],
                    "ToPort": 3389,
                    "IpProtocol": "tcp",
                    "UserIdGroupPairs": [
                        {
                            "UserId": "XXXXXXXXXXXX",
                            "GroupId": "sg-xxxxxxxx"
                        }
                    ]
                }
            ],
            "Description": "TEST JSON",
            "Tags": [
                {
                    "Value": "Test JSON",
                    "Key": "Name"
                }
            ],
            "IpPermissions": [
                {
                    "PrefixListIds": [],
                    "FromPort": 22,
                    "IpRanges": [
                        {
                            "CidrIp": "10.0.0.0/24"
                        }
                    ],
                    "ToPort": 22,
                    "IpProtocol": "tcp",
                    "UserIdGroupPairs": [
                        {
                            "UserId": "XXXXXXXXXXXX",
                            "GroupId": "sg-xxxxxxxx"
                        }
                    ]
                },
                {
                    "PrefixListIds": [],
                    "FromPort": 3389,
                    "IpRanges": [
                        {
                            "CidrIp": "10.0.0.0/24"
                        }
                    ],
                    "ToPort": 3389,
                    "IpProtocol": "tcp",
                    "UserIdGroupPairs": [
                        {
                            "UserId": "XXXXXXXXXXXX",
                            "GroupId": "sg-xxxxxxxx"
                        }
                    ]
                }
            ],
            "GroupName": "Test JSON",
            "VpcId": "vpc-XXXXXXXX",
            "OwnerId": "XXXXXXXXXXXX",
            "GroupId": "sg-xxxxxxxx"
        }
    ]
}

For what it's worth I am writing a json to yaml converter to make EC2 SG JSON output to Ansible Playbook a snap. http://docs.ansible.com/ansible/ec2_group_module.html — shdobxr, Nov 23 '16 at 13:42
Wouldn't something as simple as `line.replace('ToPort', 'to_port').replace('FromPort', 'from_port')` be sufficient for your case? — wrwrwr, Nov 23 '16 at 13:44
Let me give that a go. I wanted to use a dictionary to keep things neat. And to add more diversity to the matching. Thnx! — shdobxr, Nov 23 '16 at 13:53
Wouldn't the dictionary need to be named something different than "replacements"? If so what changes would need to be made through the script? — shdobxr, Nov 29 '16 at 16:03
That's just a variable, the name can be chosen freely (just replace all occurrences of it). — wrwrwr, Nov 29 '16 at 16:06
Understood, since I am using sysargs as the subject file, would the `def rewrite(text):` need to swap out "c" for "text"? — shdobxr, Nov 29 '16 at 16:15
That name can also be chosen freely. One thing I see is the missing parenthesis in `text_file.close()`. You can likely skip creating the intermediate file completely and process the result of `yaml.safe_dump` directly. Could you please drop an example input file somewhere, so I can check what's happening? — wrwrwr, Nov 29 '16 at 16:20
example json posted in orig question area. Save that locally. As test.json thanks for the help! — shdobxr, Nov 29 '16 at 16:46
Fixing the parentheses and ensuring that `remove_pattern` doesn't end up empty should be enough to make it work. — wrwrwr, Nov 29 '16 at 17:39
Merged your pull request. That got me over the bump. Cheers! — shdobxr, Nov 29 '16 at 18:37

wrwrwr · Accepted Answer · 2016-11-23T19:10:26.780

1

The answer you cited adapted for your use case:

import re

replacements = {
    'ToPort': 'to_port',
    'FromPort': None
}
replacements = dict((re.escape(k), v) for k, v in replacements.iteritems())
remove_pattern = re.compile('|'.join(
                    k for k, v in replacements.iteritems() if v is None))
replace_pattern = re.compile('|'.join(
                    k for k, v in replacements.iteritems() if v is not None))


def rewrite(text):
    if remove_pattern.search(text):
        return ''
    return replace_pattern.sub(
                lambda m: replacements[re.escape(m.group())], text)


with open('in.txt', 'rt') as fin:
    with open('out.txt', 'wt') as fout:
        for line in fin:
            fout.write(rewrite(line))

edited Nov 23 '16 at 19:10

answered Nov 23 '16 at 13:55

wrwrwr

1,008
2
11
19

Giving it a run right now. Yep precisely what I needed. Thanks for the assist. For those that may be creating Ansible Playbooks for Security Group creation from json output I will update this comment with my github link. (if allowed) – shdobxr Nov 23 '16 at 13:56
https://github.com/shdobxr/json2yamlAnsible/blob/master/json2yaml.py Project link as promised. – shdobxr Nov 23 '16 at 15:09
`replacements = { 'ToPort': 'to_port', 'FromPort': }` <--- how would you remove the line entirely upon match – shdobxr Nov 23 '16 at 18:12
Updated the answer (a few lines longer, but still with a single `dict` to declare all changes). – wrwrwr Nov 23 '16 at 19:14
`with open(c, 'rt') as fin: with open(c + ".out", 'wt') as fout: for line in fin: fout.write(rewrite(line))` Getting Zerobyte out file – shdobxr Nov 29 '16 at 15:17

Python multiple replacement with matching

Answered! See above for project.

1 Answers1