How to return data from a Python SAX parser?

Question

I've been trying to parse some huge XML files that LXML won't grok, so I'm forced to parse them with xml.sax.

class SpamExtractor(sax.ContentHandler):
    def startElement(self, name, attrs):
        if name == "spam":
            print("We found a spam!")
            # now what?

The problem is that I don't understand how to actually return, or better, yield, the things that this handler finds to the caller, without waiting for the entire file to be parsed. So far, I've been messing around with threading.Thread and Queue.Queue, but that leads to all kinds of issues with threads that are really distracting me from the actual problem I'm trying to solve.

I know I could run the SAX parser in a separate process, but I feel there must be a simpler way to get the data out. Is there?

Answer 1

I thought I'd give this as another answer due to it being a completely different approach.

You might want to check out xml.etree.ElementTree.iterparse as it appears to do more what you want:

Parses an XML section into an element tree incrementally, and reports what’s going on to the user. source is a filename or file object containing XML data. events is a list of events to report back. If omitted, only “end” events are reported. parser is an optional parser instance. If not given, the standard XMLParser parser is used. Returns an iterator providing (event, elem) pairs.

You could then write a generator taking that iterator, doing what you want, and yielding the values you need.

e.g:

def find_spam(xml):
    for event, element in xml.etree.ElementTree.iterparse(xml):
        if element.tag == "spam":
            print("We found a spam!")
            # Potentially do something
            yield element

The difference is largely about what you want. ElementTree's iterator approach is more about collecting the data, while the SAX approach is more about acting upon it.

Answer 2

David Beazley demonstrates how to "yield" results from a sax ContentHandler using a coroutine:

cosax.py:

import xml.sax

class EventHandler(xml.sax.ContentHandler):
    def __init__(self,target):
        self.target = target
    def startElement(self,name,attrs):
        self.target.send(('start',(name,attrs._attrs)))
    def characters(self,text):
        self.target.send(('text',text))
    def endElement(self,name):
        self.target.send(('end',name))

def coroutine(func):
    def start(*args,**kwargs):
        cr = func(*args,**kwargs)
        cr.next()
        return cr
    return start

# example use
if __name__ == '__main__':
    @coroutine
    def printer():
        while True:
            event = (yield)
            print event

    xml.sax.parse("allroutes.xml",
                  EventHandler(printer()))

Above, every time self.target.send is called, the code inside printer runs starting from event = (yield). event is assigned to the arguments of self.target.send, and the code in printer is executed till the next (yield) is reached, sort of like how a generator works.

Whereas a generator is typically driven by a for-loop, the coroutine (e.g. printer) is driven by send calls.

Answer 3

My understanding is the SAX parser is meant to do the work, not just pass data back up the food chain.

e.g:

class SpamExtractor(sax.ContentHandler):
    def __init__(self, canning_machine):
        self.canning_machine = canning_machine

    def startElement(self, name, attrs):
        if name == "spam":
            print("We found a spam!")
            self.canning_machine.can(name, attrs)

Answer 4

Basically there are three ways of parsing XML:

1. SAX-approach: it is an implementation of the Visitor pattern , the idea is that the events are pushed to your code.
2. StAX-approach: where you pull next element as long as you are ready (useful for partial parsing, i.e. reading only SOAP header)
3. DOM-approach, where you load everything into a tree in memory

You seem to need the second, but I am not sure that it is somewhere in the standard library.