Parse data line by line in python until EOF

Question

I need to parse a ton of host names into a yaml file.

so far my code looks like this

hostname = input.txt

s = (f"{hostname}: \n"
        f"  hostname: {hostname}\n"
        f"  nodename: {hostname}\n"
        f"  username: rundeck\n"
        f"  tags: 'rundeck'\n")

print(s)

I have a massive txt file that looks something like

hostname1
hostname2
hokjdlaskdsfh4
abcdef8

that kind of thing, I want to insert each line into the {hostname} variable in the text. And each line will result in a new "paragraph" Which I can then > into output.txt

forming a large file with all of the yaml code properly parsed. Each "output" in the output.yml file will look something like

hostname1: 
  hostname: hostname2
  nodename: hostname2
  username: rundeck
  tags: 'rundeck'


abcdef: 
  hostname: abcdef
  nodename: abcdef
  username: rundeck
  tags: 'rundeck'

In your example expected output, why is the first line `hostname1` but the next two lines `hostname2`? — Pranav Hosangadi, Apr 14 '21 at 20:06
Your question is essentially this: [How to read a large file - line by line?](https://stackoverflow.com/questions/8009882/how-to-read-a-large-file-line-by-line) Once you have read each line, you already know what to do with your f-string — Pranav Hosangadi, Apr 14 '21 at 20:07

ti7 · Accepted Answer · 2021-04-14T20:14:40.317

You can open files and work with their lines directly as the file-like is iterable

with open("myfile") as fh:
    for line in fh:  # iterate by-line
        # work with line here

If you have a known structure like you presented, you can key creating blocks from features of it such as

count of lines that form a block (maybe each has exactly 4?)
is each block separated by an empty line?
some structure to the blocks (such as the string "host" being present?)

Most yaml logic (such as pyyaml https://pyyaml.org/wiki/PyYAMLDocumentation ) allows writing Python dict-likes to a file, all you have to do is create an intermediate dictionary for it to write, and it should be written the structure you're looking for

Perhaps a form like this would work for you

data = {}  # start a new dictionary

with open("myfile") as fh:
    for index, line in enumerate(fh):  # iterate by-line
        if index % 4 == 0:
            host = line  # do any needed validation here
            data[host] = {}  # start another dict to write to
        elif index % 4 == 1:
            data[host]["hostname"] = line  # host is still the block header!
        ...  # continue with other fields

# write your dict
with open("destination.yml", "w") as fh:
    yaml.dump(data, fh)

This can be adapted to work with whatever separator you like if you don't have exactly 4 fields (ie. incrementing a counter instead of using the lineno's modulus) .. for example if you have variable-length collection of tags which changes the size of the blocks.

Parse data line by line in python until EOF

1 Answers1