atom feed4 messages in com.marklogic.developer.generalRe: [MarkLogic Dev General] Apparent ...
FromSent OnAttachments
Eliot KimberAug 28, 2017 10:11 am 
Christopher HamlinAug 28, 2017 10:37 am 
Eliot KimberAug 28, 2017 11:41 am 
Eliot KimberSep 1, 2017 8:15 am 
Subject:Re: [MarkLogic Dev General] Apparent Memory Leak in Profiler
From:Eliot Kimber (ekim@contrext.com)
Date:Sep 1, 2017 8:15:46 am
List:com.marklogic.developer.general

I can verify that ML 8.07 resolves the memory leak in the profiler. I can now
profile 100s of 1000s of tasks no problem.

Cheers,

E.

-- Eliot Kimber http://contrext.com

On 8/28/17, 1:41 PM, "gene@developer.marklogic.com on behalf of Eliot
Kimber" <gene@developer.marklogic.com on behalf of
ekim@contrext.com> wrote:

Thanks—I should be able to test with latest ML 8 in a couple of days.

Cheers,

E.

-- Eliot Kimber http://contrext.com

On 8/28/17, 12:37 PM, "gene@developer.marklogic.com on behalf of
Christopher Hamlin" <gene@developer.marklogic.com on behalf of
cbha@gmail.com> wrote:

There was a bug where, under certain circumstances, the profiler will result in a query deadlock &/or a resource leak (#45569). It could be that this is what you are seeing.

It was noticed in 8.0-2 and is fixed in the latest release (8.0-7).

On Mon, Aug 28, 2017 at 1:11 PM, Eliot Kimber <ekim@contrext.com>
wrote: > I reported earlier that my profiling application was causing MarkLogic
to restart after handling about 20,000 tasks. Turns out it was an out-of-memory
issue on the server itself (currently configured with 256GB of RAM). We could
see a distinct spike in memory usage, at which point the server restarted
MarkLogic. I tried different input data sets so it doesn’t appear to be an issue
with a particular input document (my data set has a few outliers that are much
larger than typical but only a few). > > Subsequent testing determined that it was the use of the MarkLogic
profiler that was causing the memory spike: if I turned off the profiler then
memory usage was flat and all the tasks completed as expected. > > This is ML 8.03. I’m still working on getting my server upgraded to a
newer version of MarkLogic so I can see if this is an issue that has already
been fixed. > > So it looks like there’s some kind of memory leak related to the
profiler and I’d like to understand what that issue and either understand how to
avoid it or report it formally. > > If it’s a general potential problem with large-scale processing would
like to understand how to avoid it or plan for it. If it’s a problem specific to
the profiler then need to report it formally and provide appropriate
diagnostics. > > So my questions: > > 1. Is this a known issue with profiling? I’m guessing not in that I’m
probably doing something out-of-the-ordinary vis-à-vis profiling and is
something that nobody would see in typical single-instance ad-hoc profiling. > 2. What types of MarkLogic processing would cause this kind of memory
spike that lasts across the execution of multiple tasks? I would expect the
memory required for a given task to be released as soon as the task is complete
so I’m guessing it must be an issue with caches or something? > > Thanks, > > Eliot > -- > Eliot Kimber > http://contrext.com > > > > > _______________________________________________ > General mailing list > Gene@developer.marklogic.com > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list Gene@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general

_______________________________________________ General mailing list Gene@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general