Extracting Data While Parsing a Document

Credit: Rod Gaither

Problem

You want to process a large XML file without loading it all into memory.

Solution

The method REXML::Document.parse_stream gives you a fast and flexible way to scan a large XML file and process the parts that interest you.

Consider this XML document, the output of a hypothetical program that runs auto mated tasks. We want to parse the document and find the tasks that failed (that is, returned an error code other than zero).

event_xml = %{ }

We can process the document as its being parsed by writing a REXML:: StreamListener subclass that responds to parsing events such as tag_start and tag_end. Heres a subclass that listens for tags with a nonzero value for their error attribute. It prints a message for every failed event it finds.

require exml/document require exml/streamlistener class ErrorListener include REXML::StreamListener def tag_start(name, attrs) if attrs["error"] != nil and attrs["error"] != "0" puts %{Event "#{name}" failed for system "#{attrs["system"]}" } + %{with code #{attrs["error"]}} end end end

To actually parse the XML data, pass it along with the StreamListener into the method REXML::Document.parse_stream:

REXML::Document.parse_stream(event_xml, ErrorListener.new) # Event "clean" failed for system "dev" with code 1 # Event "backup" failed for system "dev" with code 2

Discussion

We could find the failed events in less code by loading the XML into a Document and running an XPath query. That approach would work fine for this example, since the document only contains four events. It wouldn work as well if the document were a file on disk containing a billion events. Building a Document means building an elaborate in-memory data structure representing the entire XML document. If you only care about part of a document (in this case, the failed events), its faster and less memory-intensive to process the document as its being parsed. Once the parser reaches the end of the document, you e done.

The stream-oriented approach to parsing XML can be as simple as shown in this recipe, but it can also handle much more complex scenarios. Your StreamListener subclass can keep arbitrary state in instance variables, letting you track complex combinations of elements and attributes.

See Also

Related

Категории