7 messages in net.java.dev.imageio-ext.devRe: OutOfMemoryError reading with MrS...| From | Sent On | Attachments |
|---|---|---|
| Nidel, Mike | Jul 17, 2008 10:22 am | |
| Daniele Romagnoli | Jul 18, 2008 6:52 am | |
| Nidel, Mike | Jul 18, 2008 7:03 am | |
| Simone Giannecchini | Jul 21, 2008 2:05 am | |
| Nidel, Mike | Jul 21, 2008 7:29 am | |
| Simone Giannecchini | Jul 23, 2008 5:24 pm | |
| Nidel, Mike | Jul 23, 2008 8:17 pm |
| Subject: | Re: OutOfMemoryError reading with MrSIDImageReader | |
|---|---|---|
| From: | Simone Giannecchini (simb...@gmail.com) | |
| Date: | Jul 21, 2008 2:05:23 am | |
| List: | net.java.dev.imageio-ext.dev | |
Ciao Mike, our implementation for the readAsRenderedImage method is the default one, which menas that we fallback on the read method which does return a RenderedImage, by means, as you correctly pointed out,of a BufferedImage. Using bufferedimages to read large raster data is ofcourse impractical since the immediate model is not really suited for that since it tries to load everything in memory. As you might imagine one of our goals is to work with very large images, therefore we are sensble to memory problems like the one you reported, anyway I think that the best way to implement readAsRenderedImage, at least in the various specific plugin is to fallback on the immediate model implementation of the RenderedImage since, IMHO that's the reference model for ImageIO plugins. I think that if we would do something different we might confuse people and deviate from what ImageIO has been built for.
Now, all this is more or less useless for you, but I have to write it for the records of the mailing list :-). I understand your requirements and I have a solution/suggestion to propose. What you basically need is an ImageReaderDecorator (or wrapper? :-) )class which wraps an ImageReader and overrides the calls to readAsRenderedImage reusing the getTile calls in a deferred fashion. While at it, one could also add the capability to do tile caching and multithreaded loading in a very similar fashion to what the ImageRead operation does but in a simple way.
I have been thinking about doing something similar this for a while, so if you have some code that you want to share, we would be glad to add that to the imageio-ext codebase and give a hand.
Ciao, Simone.
On Fri, Jul 18, 2008 at 4:04 PM, Nidel, Mike <mike...@lmco.com> wrote:
I think a reply within 24 hours is pretty good, so no apology needed :)
I hadn't thought about the JAI ImageRead operation, but a solution that uses that would be very similar to implementing a RenderedImage, in the sense that it would require inserting another layer between my existing code and your code. That's OK, it's just not ideal. In the case of ImageRead, I would need a hard dependency on JAI ImageIO tools, which I very much want to avoid (for a few reasons). So given a choice between implementing my OWN RenderedImage wrapper for your ImageReader and implementing a solution that uses ImageRead (which ultimately returns a RenderedImage too), I would choose the first since there's no required dependency.
Essentially a possible GDALRenderedImage could do something like the following:
public Raster getTile(int tileX, int tileY) { Rectangle area = getTileRect(tileX, tileY); // do the math to figure out the tile region ImageReadParam param = new ImageReadParam(); param.setSourceRegion(area); // get only the area for the tile Raster result = myImageReader.readRaster(myImageIndex, param); return result; }
Does this sound right? The rest of the operations would have a default or standard implementation perhaps, just to pass through to the ImageReader.
Mike
------------------------------ *From:* Daniele Romagnoli [mailto:dany...@gmail.com] *Sent:* Friday, July 18, 2008 9:53 AM *To:* de...@imageio-ext.dev.java.net *Subject:* Re: OutOfMemoryError reading with MrSIDImageReader
Mike, sorry for the late reply.
Did you considered the choice of using the JAI-ImageRead operation? It leverages on the Deferred Execution Model which allows to load data only when really needed. In such a case a getTile method will exactly get only the required part of data from the underlying gdal reader.
Anyway I will investigate on the implementation of the readAsRenderedImage method.
Regards, Daniele
On Thu, Jul 17, 2008 at 7:22 PM, Nidel, Mike <mike...@lmco.com> wrote:
I reported earlier that I had been having memory problems when reading MrSID. Here is an example portion of the stack trace:
java.lang.OutOfMemoryError at sun.misc.Unsafe.allocateMemory(Native Method) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) at it.geosolutions.imageio.gdalframework.GDALImageReader.readDatasetRaster( GDALImageReader.java:430) at it.geosolutions.imageio.gdalframework.GDALImageReader.readDatasetRaster( GDALImageReader.java:325) at it.geosolutions.imageio.gdalframework.GDALImageReader.read(GDALImageRead er.java:958) at javax.imageio.ImageReader.readAsRenderedImage(ImageReader.java:1560)
I dug in a bit further and realized that our application assumes that a RenderedImage implementation is available to delay reading of actual pixels. But as the stack trace shows, readAsRenderedImage isn't implemented in GDALImageReader or MrSIDImageReader. So the default implementation in ImageReader gets called which just passes through to ImageReader.read() (which IS implemented in GDALImageReader).
The problem with this is that read() returns a BufferedImage which means that the entire extent of the image is read into memory. For large images this is a problem.
I suppose the GDAL ImageReaders are intended to be used by decoding a single BufferedImage for each subregion/tile/block of the image. This is certainly feasible but it doesn't match the expectation of my existing application, to which the ImageIO-ext library is just a plugin. Right now we use RenderedImage.getTile() for pixel access.
Do you have a recommended solution for this? Is there any alternative to just putting an extra layer in between my app and GDAL/ImageIO-ext to convert getTile(x,y) requests to read() calls?
thanks,
Mike
--
------------------------------------------------------- Eng. Daniele Romagnoli Software Engineer
GeoSolutions S.A.S. Via Carignoni 51 55041 Camaiore (LU) Italy
phone: +39 0584983027 fax: +39 0584983027 mob: +39 328 0559267
-------------------------------------------------------
--
------------------------------------------------------- Eng. Simone Giannecchini President /CEO GeoSolutions S.A.S. Via Carignoni 51 55041 Camaiore (LU) Italy
phone: +39 0584983027 fax: +39 0584983027 mob: +39 333 8128928
-------------------------------------------------------