RE: [cinjug-users] Max file size for DOM parser

["Baxter, Frank \(LNG-CON\)" <frank.baxter@xxxxxxxxxxxxxx>]

I've had a good experience with StaX, in particular the Woodstox implementation (http://woodstox.codehaus.org/).  No other implementations came close in terms of speed and stability.  I've parsed files up to 1GB in length using the API, which I find to be much easier to use than SAX.

 

Frank

---

From: Kumar [mailto:kumarkakani@xxxxxxxxx]
Sent: Wednesday, February 14, 2007 2:35 PM
To: users@xxxxxxxxxx
Subject: Re: [cinjug-users] Max file size for DOM parser

Thank you for the information. I used DOM just because it was easier 
to do considering the time I had to build the application. I will test this 
application for increasing file sizes and see how that goes. For a start, this 
will parse one file at a time, so concurrent processing should not be an issue. 

I guess at some point down the lane, we might have to extend this 
application as a web service (on z/OS with a lot more memory), in which case I 
would have to consider SAX or StAX etc, which don't hold an in-memory 
representation of the XML data. 

Kumar

On 2/14/07, Hudson, Loren (GE Infra, Aviation, Non-GE, US) <loren.hudson@xxxxxx> wrote:

Kumar,

 

Well, I don't know about the max size of the xml file, but I know I've run the JVM out of memory on a linux box before.  There were something like 20,000 top-level objects, each with 5-10 objects under it.  It ran out before it got to 5000, if I remember right.  I've no clue what that would translate into in terms of actual file size, or how much memory was on that system. 

 

I think it's pretty simple to test, but I'm going to guess that you don't have access to the server on which this application will run.  If you did, you could use a package (like jdom - www.jdom.org) with a certain number of top-level objects.  (Am I using the right xml terminology when I say top-level objects?)  Export that to an xml file, then parse it back into a DOM.  Keep increasing the size until you run  the JVM out of memory.  Java command-line programs are simple enough to create that you can have your results in an hour or so. 

 

An important question back to you is this:  Will this application only do one file at a time, or will it need to do concurrent parsing?

 

I hope that my ramblings have helped.  I'm interested in knowing what testing and other users have to say about it.

 

Mike Hudson

MDW - MESA/Java

GE AE, Evendale, Ohio

loren.hudson@xxxxxxxxx

Desk: 513-243-3663

Cell:  513-546-4166

 

---

From: Kumar [mailto:kumarkakani@xxxxxxxxx]
Sent: Wednesday, February 14, 2007 11:28 AM
To: users@xxxxxxxxxx
Subject: [cinjug-users] Max file size for DOM parser

Hello everyone,

I am designing a Java application that would initially run on a Windows PC with 2GB RAM that processes XML files using DOM parser. I was wondering what would the maximum XML file size for such an application be? I would appreciate your advise.

Thanks in advance.
Kumar