Can I insert a line into ruamel.yaml's CommentedMap?

Question

I understand that this is related to this SO question, but what I'm mostly concerned about is whether this might mess with things such as the preserved comments.

import ruamel.yaml as yaml

yaml_str = """\
first_name: Art
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...
"""

data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)

# I'd like to extend CommentedMap so that I can do something like:
data.insert(1, 'last_name', 'Vandelay')

print(yaml.dump(data, Dumper=yaml.RoundTripDumper))

Should output:

first_name: Art
last_name: Vandelay
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...

Should not output:

first_name: Art
last_name: Vandelay    # This is an occupation comment
occupation: Architect
about: Art Vandelay is a fictional character that George invents...

Anthon · Accepted Answer · 2016-05-02T06:54:39.150

On Python 2.7, and on Python 3.X with at least ruamel.yaml 0.11.11, this works fine:

import ruamel.yaml

yaml_str = """\
first_name: Art
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...
"""

data = ruamel.yaml.round_trip_load(yaml_str)
data.insert(1, 'last name', 'Vandelay')

print(ruamel.yaml.round_trip_dump(data))

gives:

first_name: Art
last name: Vandelay
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...

as the end-of-line comments are associated with the key of the line in the CommentedMap. (Python 2.7.11 on Linux Mint with ruamel.yaml 0.11.10.)

This will not work on older versions of ruamel.yaml with Python3 as the .insert() you are using is a feature of the full-fledged ruamel.ordereddict and the OrderedDict in the standard library doesn't have that method. Therefore you need to graft an .insert() function onto the CommentedMap:

import ruamel.yaml
from ruamel.yaml.comments import CommentedMap
from ruamel.yaml.compat import ordereddict

yaml_str = """\
first_name: Art
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...
"""

def com_insert(self, pos, key, value, comment=None):
    od = ordereddict()
    od.update(self)
    for k in od:
        del self[k]
    for index, old_key in enumerate(od):
        if pos == index:
            self[key] = value
        self[old_key] = od[old_key]
    if comment is not None:
        self.yaml_add_eol_comment(comment, key=key)

CommentedMap.insert = com_insert

data = ruamel.yaml.round_trip_load(yaml_str)
data.insert(1, 'last name', 'Vandelay', comment="new key")

print(ruamel.yaml.round_trip_dump(data))

gives on Python3:

first_name: Art
last name: Vandelay    # new key
occupation: Architect  # This is an occupation comment
about: Art Vandelay is a fictional character that George invents...

Please note that there is an optional parameter for insert() that allows you to specify a comment for the newly inserted key-value pair. The above works because deleting a key from a CommentedMap doesn't remove the comment associated with the key. So I temporarily park the old key-value pairs in od delete all key-values, and then copy them back inserting the new stuff at the right moment

The above insert, with comment, has been added in ruamel.yaml 0.11.11 for both Python 2 and 3

The .round_trip_load() is equivalent to .load(...., Loader=ruamel.yaml.RoundTripLoader, ...) and .round_trip_dump() to `.dump(....., Dumper=ruamel.yaml.RoundTripDumper, allow_unicode=True, ...)

wait what?? I had no idea ruamel.ordereddict had `.insert()` :D — demux, May 01 '16 at 18:58
That is excusable, the author of that package is notoriously lazy when it comes to writing accessible documentation. — Anthon, May 01 '16 at 18:59
I'm just happy that it's available at all. Thank you! But can I somehow use `ruamel.ordereddict` even though I'm using `Python 3`? — demux, May 01 '16 at 19:06
Looking at [compat.py, line 12](https://bitbucket.org/ruamel/yaml/src/dd14fefad83336b46af54902d781387776b22789/compat.py?at=default&fileviewer=file-view-default#compat.py-12) it seems all I have to do is `pip install ruamel.ordereddict`. Is this a correct analysis? — demux, May 01 '16 at 19:11
Unfortunately not. `ruamel.ordereddict` only works on Python2. I got most of it working on 3.2 (after that there were rather big changes in the C `dict` source that I tried to follow as close as possible). But then found that for 3.5 a C based OrderedDict was being done *from scratch* and decided not to spent more time on that and focus on other things. — Anthon, May 01 '16 at 19:12
So what you would need to do is create a new CommentedMap, copy the key, value pairs over *with any comments* and insert the new key at the right moment. I can look at extending the answer with that tomorrow. — Anthon, May 01 '16 at 19:16
Yeah, if you have the time, that would be super helpful! Thanks! — demux, May 01 '16 at 19:21
This is now in ruamel.yaml 0.11.11 and [documented](http://yaml.readthedocs.io/en/latest/example.html). Thanks for the inspiration. — Anthon, May 02 '16 at 06:58
What I created with this is a [hook](https://gist.github.com/demux/39594b46b18fa6f76c7d63f5c3ccb762) for [Python Schematics](https://schematics.readthedocs.io/en/latest/) The goal is to reformat the yaml as little as possible when saving back but to also have a basic structure for the data when new data is added. — demux, May 02 '16 at 09:06

Can I insert a line into ruamel.yaml's CommentedMap?

1 Answers1