How to convert jenkins job configuration config.xml to YAML format in python to be used jenkins-job-builder?

Question

jenkins-job-builder is a nice tool to help me to maintain jobs in YAML files. see example in configuration chapter.

Now I had lots of old jenkins jobs, it will be nice to have a python script xml2yaml to convert the existing jenkins job config.xml to YAML file format.

Do you any suggestions to had a quick solution in python ?

I don't need it to be used in jenkins-job-builder directly, just can be converted it into YAML for reference.

For the convert, some part can be ignored like namespace.

config.xml segment looks like:

<project>
  <logRotator class="hudson.tasks.LogRotator">
    <daysToKeep>-1</daysToKeep>
    <numToKeep>20</numToKeep>
    <artifactDaysToKeep>-1</artifactDaysToKeep>
    <artifactNumToKeep>-1</artifactNumToKeep>
  </logRotator>
  ...
</project>

The yaml output could be:

- project:
   logrotate:
     daysToKeep: -1
     numToKeep: 20
     artifactDaysToKeep: -1
     artifactNumToKeep: -1

If you are not familiar with config.xml in jenkins, you can check infra_backend-merge-all-repo job in https://ci.jenkins-ci.org

score 3 · Answer 1 · answered Aug 26 '15 at 23:05

3

I'm writing a program that does this conversion from XML to YAML. It can dynamically query a Jenkins server and translate all the jobs to YAML.

https://github.com/ktdreyer/jenkins-job-wrecker

Right now it works for very simple jobs. I've taken a safe/pessimistic approach and the program will bail if it encounters XML that it cannot yet translate.

answered Aug 26 '15 at 23:05

ktdreyer

71
2

I am currently using your Jenkins Job Wrecker but i get the following error: `hudson.plugins.git.extensions.impl.SubmoduleOption not supported`. Any idea why this is the case? – Müller Jun 13 '16 at 05:56
I used `job-wrecker` but it gives all data in `raw`, is there way to use it in `yaml` format? – Nilesh Feb 07 '18 at 22:37
@Nilesh it should translate the job into YAML. If it is not working for you, that is a bug. Would you please open a new ticket on GitHub? Please also paste the full XML of your job in the GitHub ticket, so I can reproduce the issue you're seeing. – ktdreyer Feb 09 '18 at 16:20
@ktdreyer created new issue https://github.com/ktdreyer/jenkins-job-wrecker/issues/68 – Nilesh Feb 09 '18 at 17:53

score 2 · Accepted Answer · answered Jan 26 '14 at 12:18

It's hard to tell from your question exactly what you're looking for here, but assuming you're looking for the basic structure:

Python has good support on most platforms for XML Parsing. Chances are you'll want to use something simple and easy to use like minidom. See the XML Processing Modules in the python docs for your version of python.

Once you've opened the file, looking for project and then parsing down from there and using a simple mapping should work pretty well given the simplicity of the yaml format.

from xml.dom.minidom import parse

def getText(nodelist):
    rc = []
    for node in nodelist:
        if node.nodeType == node.TEXT_NODE:
            rc.append(node.data)
    return ''.join(rc)

def getTextForTag(nodelist,tag):
    elements = nodelist.getElementsByTagName(tag)
    if (elements.length>0):
        return getText( elements[0].childNodes)
    return ''

def printValueForTag(parent, indent, tag, valueName=''):
    value = getTextForTag( parent,tag)
    if (len(value)>0):
        if (valueName==''):
            valueName = tag
        print indent + valueName+": "+value

def emitLogRotate(indent, rotator):
    print indent+"logrotate:"
    indent+='  '
    printValueForTag( rotator,indent, 'daysToKeep')
    printValueForTag( rotator,indent, 'numToKeep')  
def emitProject(project):
    print "- project:"
    # all projects have log rotators, so no need to chec
    emitLogRotate("   ",project.getElementsByTagName('logRotator')[0])
    # next section...

dom = parse('config.xml')
emitProject(dom)

This snippet will print just a few lines of the eventual configuration file, but it puts you in the right direction for a simple translator. Based on what I've seen, there's not much room for an automatic translation scheme due to naming differences. You could streamline the code as you iterate for more options and to be table driven, but that's "just a matter of programming", this will at least get you started with the DOM parsers in python.

this is very useful suggestion, almost fits for my need, just try to reuse the `xml2json` (https://github.com/hay/xml2json) code to load into json and manipulate the json internal format, then dump into yaml. — Larry Cai, Jan 28 '14 at 08:17
I'm not sure what you would get from xml2json that isn't in this example. Since the yaml and XML formats do not use the same keywords in many cases, you can't just walk the tree and do a format conversion, hence my suggestion for how you could implement such a conversion with minimal repetition. — gaige, Jan 28 '14 at 09:37

score 1 · Answer 3 · edited Apr 27 '16 at 10:39

I suggest querying and accessing the xml with xpath expressions using xmlstarlet on the command line and in shell scripts. No trouble with low-level programmatical access to XML. XMLStarlet is an XPath swiss-army knife on the command line.

"xmlstarlet el" shows you the element structure of the entire XML as XPath expressions.

"xmlstarlet sel -t -c XPath-expression" will extract exactly what you want.

Maybe you want to spend an hour (or two) on freshing up your XPath know-how in advance.

You will shed a couple of tears, once you recognize how much time you spent with programming XML access before you used XMLStarlet.

How to convert jenkins job configuration config.xml to YAML format in python to be used jenkins-job-builder?

3 Answers3

Linked