How can I split large xml into small chunks using VTDGenHuge?

Question

I want split large xml into small chunks. I am using VTDGen to split the xml file into small chunks and it works fine for file size < 2 GB. VTD-xml uses IN-Memory to parse the xml where i don't want to load the xml in to memory . So i am trying to map Memory using VTDGenHuge .

Code works fine with VTDGen but when i us VTDGenHuge it not working.

        String prefix = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"+"\n";
    String suffix = "\n</Employees>\n";
    try {

        VTDGenHuge vg = new VTDGenHuge();
        if (vg.parseFile("C:\\Users\\abc\\Desktop\\latestxml\\Input_1.xml", true,VTDGenHuge.MEM_MAPPED)) {
            int splitBy = ;
           System.out.println("Started time"+ new Date());
            VTDNavHuge vn = vg.getNav();               
            AutoPilotHuge ap = new AutoPilotHuge(vn);
            ap.selectXPath("/Employees/Employee");
            FastLongBuffer flb = new FastLongBuffer(4);
            int i;
            byte[] xml = vn.getXML().getBytes();          
            while ((i = ap.evalXPath()) != -1) {  
                flb.append(vn.getElementFragment());
            }
            int size = flb.size();
            if (size != 0) {
                File fo = null;
                FileOutputStream fos = null;
                for (int k = 0; k < size; k++) {
                   if (k % splitBy == 0) {
                        if (fo != null) {
                            fos.write(suffix.getBytes());
                            fos.close();
                            fo = null;
                        }
                    }
                    if (fo == null) {
                        fo = new File("C:\\Users\\abc\\Desktop\\Test\\xml\\"+"out" + k + ".xml");
                        fos = new FileOutputStream(fo);
                        fos.write(prefix.getBytes());
                    }
                    fos.write("\n".getBytes());                       
                    fos.write(xml, flb.lower32At(k), flb.upper32At(k));
                }
                if (fo != null) {                       
                    fos.write(suffix.getBytes());                  
                    fos.close();
                    fo = null;
                }
            }

        }
    } catch (Exception e) {
        e.printStackTrace();
    }

I get NUll value at "byte[] xml = vn.getXML().getBytes();" when u do syso vn.getXML() you get object value. but with "getBytes()" return null.I don't no why. But if u do "byteAt(x)" x = any long value it return value.

My xml file is:

<?xml version="1.0" encoding="UTF-8"?>
<Employees>
<Employee id="1">
    <age>29</age>
    <name>Pankaj</name>
    <gender>Male</gender>
    <role>Java Developer</role>
</Employee>
<Employee id="2">
    <age>35</age>
    <name>Lisa</name>
    <gender>Female</gender>
    <role>CEO</role>
</Employee>
<Employee id="3">
    <age>40</age>
    <name>Tom</name>
    <gender>Male</gender>
    <role>Manager</role>
</Employee>
    <Employee id="1">
    <age>29</age>
    <name>Pankaj</name>
    <gender>Male</gender>
    <role>Java Developer</role>
</Employee>
<Employee id="2">
    <age>35</age>
    <name>Lisa</name>
    <gender>Female</gender>
    <role>CEO</role>
</Employee>
<Employee id="3">
    <age>40</age>
    <name>Tom</name>
    <gender>Male</gender>
    <role>Manager</role>
</Employee>
<Employees>

I want out put like this.

<?xml version="1.0" encoding="UTF-8"?>
 <Employees>
<Employee id="1">
    <age>29</age>
    <name>Pankaj</name>
    <gender>Male</gender>
    <role>Java Developer</role>
</Employee>
<Employee id="2">
    <age>35</age>
    <name>Lisa</name>
    <gender>Female</gender>
    <role>CEO</role>
</Employee>
<Employee id="3">
    <age>40</age>
    <name>Tom</name>
    <gender>Male</gender>
    <role>Manager</role>    
</Employee>
<Employees> 

<?xml version="1.0" encoding="UTF-8"?>
<Employees> 
    <Employee id="1">
    <age>29</age>
    <name>Pankaj</name>
    <gender>Male</gender>
    <role>Java Developer</role>
</Employee>
<Employee id="2">
    <age>35</age>
    <name>Lisa</name>
    <gender>Female</gender>
    <role>CEO</role>
</Employee>
<Employee id="3">
    <age>40</age>
    <name>Tom</name>
    <gender>Male</gender>
    <role>Manager</role>
</Employee>
<Employees>

Your code looks like C#. You need to use a XMLReader. You can search the web for c# xml huge. Try this one : http://stackoverflow.com/questions/15772031/how-to-parse-very-huge-xml-files-in-c — jdweng, Sep 25 '15 at 13:00
@ Vtd-xml-author it is purely java code. Copy past the code between java main() and do the changes required and run the code. it will run — Naveen, Sep 28 '15 at 07:14
We have updated the document for extended vtd-xml... on the web site or as a download as a result of your question. — vtd-xml-author, Oct 19 '15 at 21:59

vtd-xml-author · Answer 1 · 2015-10-19T03:20:46.023

1

I think that vn.getXML() for extended vtd-xml returns a IbyteBuffer interface object different from standard vtd-xml. You can call the inteface method called writeOutputToFile() and pass to it the offset and value parameters.. sorry the documentation part of it is lacking, but that is the basic lowdown...

edited Oct 19 '15 at 03:20

answered Sep 26 '15 at 03:48

vtd-xml-author

3,319
4
22
30

jdweng · Answer 2 · 2015-09-25T17:05:47.347

Try this

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Linq;

namespace ConsoleApplication1
{
    class Program
    {
        const string FILENAME = @"c:\temp\test.xml";
        const int OUTPUT_ELEMENTS = 3;
        static void Main(string[] args)
        {

            XmlReader reader = XmlTextReader.Create(FILENAME, new XmlReaderSettings() { IgnoreWhitespace = true });

            int count = 0;
            XDocument doc = null;
            XElement employees = null;
            reader.ReadToFollowing("Employee");
            while (!reader.EOF)
            {
                if (reader.Name == "Employee")
                {
                    if (doc == null)
                    {
                        string root = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" +
                                        "<Employees>" +
                                        "</Employees>";
                        doc = XDocument.Parse(root);
                        employees = (XElement)doc.FirstNode;
                    }
                    employees.Add(XElement.Parse(reader.ReadOuterXml()));

                    count += 1;
                    if (count % OUTPUT_ELEMENTS == 0)
                    {
                        doc.Save(string.Format(@"c:\temp\test{0}.xml", (int)(count / OUTPUT_ELEMENTS)));
                        doc = null;
                    }
                }
                else
                {
                    if (reader.Value == "")
                    {
                        break;
                    }
                    else
                    {
                        reader.Read();
                    }
                }

            }
            if (doc != null)
            {
                doc.Save(string.Format(@"c:\temp\test{0}", (int)(count / OUTPUT_ELEMENTS)));
            }
        }
    }
}

The OP is using Java, not C#. – Andreas Veithen Sep 25 '15 at 17:57 — Andreas Veithen, Sep 25 '15 at 17:57

How can I split large xml into small chunks using VTDGenHuge?

2 Answers2