|kimhorn||Mar 4, 2009 6:33 pm|
|Asankha C. Perera||Mar 4, 2009 6:41 pm|
|kimhorn||Mar 4, 2009 10:21 pm|
|Ruwan Linton||Mar 4, 2009 10:51 pm|
|Andreas Veithen||Mar 4, 2009 11:21 pm|
|Andreas Veithen||Mar 7, 2009 11:42 am|
|Ruwan Linton||Mar 7, 2009 6:47 pm|
|Andreas Veithen||Mar 8, 2009 1:38 am|
|Ruwan Linton||Mar 8, 2009 4:46 am|
|kimhorn||Mar 8, 2009 7:05 pm|
|Andreas Veithen||Mar 9, 2009 4:54 am|
|Ruwan Linton||Mar 9, 2009 8:53 am|
|Andreas Veithen||Mar 9, 2009 9:22 am|
|Hubert, Eric||Mar 9, 2009 9:26 am|
|Kim Horn||Mar 11, 2009 4:05 pm|
|Andreas Veithen||Mar 13, 2009 1:31 pm|
|kimhorn||Mar 18, 2009 11:20 pm|
|Andreas Veithen||Mar 18, 2009 11:48 pm|
|Kim Horn||Mar 19, 2009 3:33 pm|
|Andreas Veithen||Mar 19, 2009 4:51 pm|
|Kim Horn||Mar 19, 2009 5:40 pm|
|Andreas Veithen||Mar 20, 2009 2:05 am|
|Kim Horn||Mar 22, 2009 3:37 pm|
|Kim Horn||Mar 24, 2009 9:01 pm|
|Andreas Veithen||Mar 26, 2009 3:45 am|
|Kim Horn||Mar 31, 2009 7:14 pm||.dat|
|Andreas Veithen||Apr 1, 2009 12:28 am|
|Kim Horn||Apr 1, 2009 3:35 pm|
|Subject:||Re: VFS - Synapse Memory Leak|
|From:||Andreas Veithen (andr...@gmail.com)|
|Date:||Mar 7, 2009 11:42:17 am|
I've done an initial implementation of this feature. It is available in trunk and should be included in the next nightly build. In order to enable this in your configuration, you need to add the following property to the proxy:
You also need to add the following mediators just before the <send> mediator:
<property action="remove" name="transportNonBlocking" scope="axis2"/> <property action="set" name="OUT_ONLY" value="true"/>
With this configuration Synapse will stream the data directly from the incoming to the outgoing transport without storing it in memory or in a temporary file. Note that this has two other side effects: * The incoming file (or connection in case of a remote file) will only be opened on demand. In this case this happens during execution of the <send> mediator. * If during the mediation the content of the file is needed several time (which is not the case in your example), it will be read several times. The reason is of course that the content is not cached.
I tested the solution with a 2GB file and it worked fine. The performance of the implementation is not yet optimal, but at least the memory consumption is constant.
Some additional comments: * The transport.vfs.Streaming property has no impact on XML and SOAP processing: this type of content is processed exactly as before. * With the changes described here, we have now two different policies for plain text and binary content processing: in-memory caching + no streaming (transport.vfs.Streaming=false) and no caching + deferred connection + streaming (transport.vfs.Streaming=true). Probably we should define a wider range of policies in the future, including file system caching + streaming. * It is necessary to remove the transportNonBlocking property (MessageContext.TRANSPORT_NON_BLOCKING) to prevent the <send> mediator (more precisely the OperationClient) from executing the outgoing transport in a separate thread. This property is set by the incoming transport. I think this is a bug since I don't see any valid reason why the transport that handles the incoming request should determine the threading behavior of the transport that sends the outgoing request to the target service. Maybe Asankha can comment on this?
On Thu, Mar 5, 2009 at 07:21, kimhorn <kim....@icsglobal.net> wrote:
Thats good; as this stops us using Synapse.
Asankha C. Perera wrote:
Exception in thread "vfs-Worker-4" java.lang.OutOfMemoryError: Java heap space at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518) at java.lang.StringBuffer.append(StringBuffer.java:307) at java.io.StringWriter.write(StringWriter.java:72) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1104) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1078) at org.apache.commons.io.IOUtils.toString(IOUtils.java:382) at org.apache.synapse.format.PlainTextBuilder.processDocument(PlainTextBuilder.java:68)
Since the content type is text, the plain text formatter is trying to use a String to parse as I see.. which is a problem for large content..
A definite bug we need to fix ..
-- Asankha C. Perera AdroitLogic, http://adroitlogic.org
View this message in context:
http://www.nabble.com/VFS---Synapse-Memory-Leak-tp22344176p22345904.html Sent from the Synapse - Dev mailing list archive at Nabble.com.