Split XML into smaller files using DomDocument

Question

I'm trying to handle a large (20mb+) XML file but having problems parsing it due to its size. I want to break it down into smaller segments, say 100 property nodes per file.

Currently I'm separating the files based on other criteria using this code below but I'm a bit unsure how to proceed adapting the code to perform 100 records per file split:

$destination = new DOMDocument;
$destination->preserveWhiteSpace = true;
$destination->loadXML('<?xml version="1.0" encoding="utf-8"?><root></root>');

$source = new DOMDocument;
$source->load('bes1c8ca168f910.xml');

$xp = new DOMXPath($source);
$destRoot = $destination->getElementsByTagName("root")->item(0);

foreach ($xp->query('/root/property[rent]') as $item) {
    $newItem = $destination->importNode($item, true);
    $destRoot->appendChild($newItem);
    $item->parentNode->removeChild($item);
}

$source->save("sales.xml");
$destination->formatOutput = true;
$destination->save("rentals.xml");

Any advice is appreciated thanks.

Answer 1

Consider dynamic XSLT where PHP passes multiples of 100 to parse the range of nodes in the source XML into smaller outputs using XSLT's position(). Specifically, PHP passes a loop variable as a parameter into XSLT binded to $splitnum (very similar to SQL parameterization).

Input (assuming an XML structure like your previous post)

<root>
    <property>
      <rent>
        <term>short</term>
        <freq>week</freq>
        <price_peak>5845</price_peak>
        <price_high>5845</price_high>
        <price_medium>4270</price_medium>
        <price_low>3150</price_low>
      </rent>
    </property>
    <property>
      <rent>
        <term>long</term>
        <freq>week</freq>
        <price_peak>6845</price_peak>
        <price_high>6845</price_high>
        <price_medium>4270</price_medium>
        <price_low>3150</price_low>
      </rent>
    </property>
    ...
</root>

XSLT

(save as .xsl file, a special .xml file; script expects a parameter to be passed in for node range split)

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
   <xsl:output method="xml" omit-xml-declaration="yes" indent="yes" />
   <xsl:strip-space elements="*" />

   <xsl:param name="splitnum" />

   <xsl:template match="/root">
      <xsl:copy>
         <xsl:variable name="currsplit" select="$splitnum - 99"/>
         <xsl:apply-templates select="property[position() &gt;= $currsplit and 
                                               position() &lt;= $splitnum]" />
      </xsl:copy>
   </xsl:template>

   <xsl:template match="property">
      <xsl:copy>
         <xsl:copy-of select="*" />
      </xsl:copy>
   </xsl:template>

</xsl:stylesheet>

PHP

(passes loop iterator variable into XSLT as a parameter; produces 10 XMLs, each with successive 100 property nodes, extend limit of 1000 as needed)

// Load XML and XSL
$xml = new DOMDocument;
$xml->load('Input.xml');

$xsl = new DOMDocument;
$xsl->load($xslstr);

$prop_total = $xml->getElementsByTagName('property')->length + 100;

for($i=1; $i<=$prop_total; $i++){
  if ($i % 100 == 0) {         
    // Configure transformer
    $proc = new XSLTProcessor;
    $proc->importStyleSheet($xsl);

    // Binds loop variable to XSLT parameter
    $proc->setParameter('', 'splitnum', $i);

    // Transform XML source
    $newXML = new DOMDocument;
    $newXML = $proc->transformToXML($xml);

    // Output file
    file_put_contents('rentals_'.$i.'.xml', $newXML);
  }
}

Answer 2

An example of splitting a file using XMLReader. I've tried to make it flexible, so the filename is used as the basis of the files being created and the split count is defined as a variable.

The main part of the code is a loop which reads the <property> elements, you could tune this as needed. I've also used $rootNodeName as a place holder for whatever your root node is called.

$fileName = "data/t1.xml";
$original = new XMLReader;
$original->open($fileName);
$path_parts = pathinfo($fileName);
$filePrefix = $path_parts['dirname'].'/'.$path_parts['filename'].'-';
$nextRecord = 0;
$splitCount = 2;
$rootNodeName = "data";

$doc = new DOMDocument();
$doc->loadXML("<$rootNodeName/>");
while ($original->read() && $original->name !== 'property');
while ($original->name === 'property')
{
    $newNode = $doc->importNode($original->expand(), true);
    $doc->documentElement->appendChild($newNode);
    $nextRecord++;

    if ( $nextRecord % $splitCount == 0 )   {
        $nextFileName = $filePrefix.$nextRecord.".".$path_parts['extension'];
        $doc->save($nextFileName);
        $doc = new DOMDocument();
        $doc->loadXML("<$rootNodeName/>");
    }
    $original->next('property');
}
if ( $nextRecord % $splitCount != 0 )   {
    $nextFileName = $filePrefix.$nextRecord.".".$path_parts['extension'];
    $doc->save($nextFileName);
}

It's not the most elegant code, but it could also form the basis of a program that deals with the elements one by one rather than loading the whole document in at one time.