34 messages in org.codehaus.groovy.devRe: [groovy-dev] Re: [groovy-user] Me...
FromSent OnAttachments
Alex TkachmanOct 16, 2008 2:30 pm 
Alex TkachmanOct 16, 2008 3:17 pm 
Jochen TheodorouOct 16, 2008 3:49 pm 
Tim QuinnOct 16, 2008 4:15 pm 
Alexander VeitOct 16, 2008 4:34 pm 
Jochen TheodorouOct 16, 2008 4:43 pm 
Tim QuinnOct 16, 2008 5:28 pm 
Jochen TheodorouOct 17, 2008 6:35 am 
Alex TkachmanOct 17, 2008 6:44 am 
Graeme RocherOct 17, 2008 6:54 am 
Jochen TheodorouOct 17, 2008 7:05 am 
Graeme RocherOct 17, 2008 7:14 am 
alex...@gmail.comOct 17, 2008 7:30 am 
Jochen TheodorouOct 17, 2008 7:36 am 
Alex TkachmanOct 17, 2008 9:51 am 
Jochen TheodorouOct 22, 2008 5:27 am 
Alex TkachmanOct 22, 2008 5:37 am 
Jochen TheodorouOct 22, 2008 6:05 am 
Alex TkachmanOct 22, 2008 6:08 am 
Jochen TheodorouOct 22, 2008 6:18 am.zip
Alex TkachmanOct 22, 2008 6:32 am 
Jochen TheodorouOct 22, 2008 7:06 am 
Guillaume LaforgeOct 22, 2008 7:08 am 
Alex TkachmanOct 22, 2008 7:11 am 
Guillaume LaforgeOct 22, 2008 7:21 am 
Jochen TheodorouOct 22, 2008 7:22 am 
Jochen TheodorouOct 22, 2008 7:39 am 
Jochen TheodorouOct 22, 2008 7:41 am 
Alex TkachmanOct 22, 2008 9:16 am 
Jochen TheodorouOct 22, 2008 10:11 am 
Jochen TheodorouOct 22, 2008 11:24 am 
Alex TkachmanOct 22, 2008 11:48 am 
Jochen TheodorouOct 22, 2008 2:07 pm 
Jochen TheodorouDec 22, 2008 12:52 pm 
Actions with this message:
Paste this link in email or IM:
Paste this link in email or IM:
Atom feed for this thread
Paste this URL into your reader:
Subject:Re: [groovy-dev] Re: [groovy-user] Memory leaks in GroovyActions...
From:Alex Tkachman (alex@gmail.com)
Date:Oct 22, 2008 9:16:21 am
List:org.codehaus.groovy.dev

There is something I really don't like in this patch after brief look. What you did is kind of repetition of architecture of GoogleCollection. I mean ReferenceType enum and ManagedReference, which contains real strong, soft, weak of phantom reference.

What we had before was different - elements of internal data structures (especially entries in maps) were directly derived from java references, which allowed to use less memory. It was done with purpose (and I did mesaurments) top make things faster and more memory efficient. Also I totally understand idea behind handler but very often (especially for performance critical data structure) handler is just this, so you just add unneeded operations.

On Wed, Oct 22, 2008 at 4:27 PM, Jochen Theodorou <blac@gmx.org> wrote:

Alex Tkachman schrieb:

and the goal of ref counting is stop thread.

I am aware of that, but the logic is not the right direction. The class becomes unloadable once the thread is stopped, the thread stops once the ref count is 0. We basically have here a pair of reference and referent. The ReferenceQueue will have a hardlink to the reference, while the referent might be collected. So when the GC is running now and freeing all referents, then your Thread is most probably still running, meaning the class and the loader cannot be unloaded. The Thread is still running, because the Thread is not executed as part of the GC process, but in parallel. The GC would at last need a second sweep after the thread is done. And I think that is too much for the GC.

I spend Monday+Tuesday on refactoring many of the classes for a more flexible memory management. What I did was:

Abstract the reference types java gives.I have now an enum for defining the reference type and FinalizableRef became ManagedReference. LayzySoftReference became LazyReference and can now be used with soft, weak, phatom or hard references. Your soft and weak maps got replaced by ManagedConcurrentMap and ManagedDoubleKeyMap. Since they now have a configurable reference type, there is no longer any need to have a special map for soft and weak. I have now a ReferenceManager class, with 4 possible versions, one manager that collects references from the ReferenceQueue by thread as done before. One that is callback based like I 1.5.x, one that does nothing and one that changes to a different manager once a threshold is reached.

Yesterday I spend most of the time finding a memory leak, which I found in my implementation for the strong meta class tracker. That implementation was confusedby GroovyShell#evaluate trying to unset the meta class of the script class in the registry. The tracker recorded that accidently as an action where a string meta class has been unset and build up references for it. It is a very small memory hole, because with even only 7mb ram I could run the test around 20k times before it blew up.

So what is left? Currently the memory manager are set statically in three places, that probably should be only one place. The ideea of not starting to collect references right away seems to work well, but I need to check how many references I need to start a small application. I would like to have it so you can start a small application without any memory management and once a certain count is reached, we start to manage. So fine tuning is required.

What is also missing is the public API to control anything here. I needs to be something static it seems, so maybe something hosted in GroovySystem would be fine. But since there are currently three places and four cases, how much control do we want to give? I think we could make all 4 cases into one, meaning there will be only one manager and one reference queue.

I did not run any speed tests, the only thing I can currently say is, that the build needs about the same time on my machine.. ah well, the weak map test is currently failing, because I removed the assignableMap, because it was not used

anyway... how should I proceed? Simply commit what I have once I fixed the test?

bye blackdrag