RE: [cinjug-users] Max file size for DOM parser

["Mascolino, Mark R." <mark.mascolino@xxxxxx>]

There really isn't a good mapping between file size and how big its internal DOM representation will be.  A lot depends on how many nodes there will be, how much text is in each node, attributes, etc. A document that is just <a>A whole lot of text</a> will have a vastly different in memory footprint than a typical xml document that has loads of nodes in it.

 

As you are figuring out, if you need to handle arbritrarily large XML documents, you need to investigate a SAX or StAX parser approach.

---

From: Hudson, Loren (GE Infra, Aviation, Non-GE, US) [mailto:loren.hudson@xxxxxx]
Sent: Wednesday, February 14, 2007 12:57 PM
To: Kumar; users@xxxxxxxxxx
Subject: RE: [cinjug-users] Max file size for DOM parser

Kumar,

 

Well, I don't know about the max size of the xml file, but I know I've run the JVM out of memory on a linux box before.  There were something like 20,000 top-level objects, each with 5-10 objects under it.  It ran out before it got to 5000, if I remember right.  I've no clue what that would translate into in terms of actual file size, or how much memory was on that system. 

 

I think it's pretty simple to test, but I'm going to guess that you don't have access to the server on which this application will run.  If you did, you could use a package (like jdom - www.jdom.org) with a certain number of top-level objects.  (Am I using the right xml terminology when I say top-level objects?)  Export that to an xml file, then parse it back into a DOM.  Keep increasing the size until you run  the JVM out of memory.  Java command-line programs are simple enough to create that you can have your results in an hour or so. 

 

An important question back to you is this:  Will this application only do one file at a time, or will it need to do concurrent parsing?

 

I hope that my ramblings have helped.  I'm interested in knowing what testing and other users have to say about it.

 

Mike Hudson

MDW - MESA/Java

GE AE, Evendale, Ohio

loren.hudson@xxxxxxxxx

Desk: 513-243-3663

Cell:  513-546-4166

 

---

From: Kumar [mailto:kumarkakani@xxxxxxxxx]
Sent: Wednesday, February 14, 2007 11:28 AM
To: users@xxxxxxxxxx
Subject: [cinjug-users] Max file size for DOM parser

Hello everyone,

I am designing a Java application that would initially run on a Windows PC with 2GB RAM that processes XML files using DOM parser. I was wondering what would the maximum XML file size for such an application be? I would appreciate your advise.

Thanks in advance.
Kumar