atom feed2 messages in com.marklogic.developer.generalRe: [MarkLogic Dev General] [RESOLVE]...
FromSent OnAttachments
Eliot KimberNov 10, 2017 1:44 pm 
Ron HitchensNov 10, 2017 5:44 pm 
Subject:Re: [MarkLogic Dev General] [RESOLVE] Spawned Task Appears to Block Other Threads
From:Eliot Kimber (
Date:Nov 10, 2017 1:44:33 pm

I moved my response handler to a different web app than the web app that starts
the job submission task, but I was still having the issue where the response
handler didn’t handle responses (or didn’t get responses, hard for me to tell
which it is) until the job submission task is canceled.

I realized that my problem was the initial job submission was updating the job
record for each job, but I was doing the update as part of the main processing,
rather than using eval(), so the commit wasn’t done until the task ended.

The response handler also wants to update the job record, but because there is a
pending commit, it is blocked.

By having the job submission do the update via eval() then the commit is done
immediately and everything works.

So the lesson is: you have to really understand the implications of updates and
commits or you will go astray. Or maybe “concurrency is always hard”.



-- Eliot Kimber

On 11/10/17, 9:39 AM, " on behalf of
Eliot Kimber" < on behalf of> wrote:

Yes, the client process is started from a Web app, so I think your analysis
is correct.

I will move the response handling to a separate Web app—probably should have
done that from the start.


Eliot -- Eliot Kimber

On 11/9/17, 11:46 PM, " on behalf of
Geert Josten" < on behalf of> wrote:

Hi Eliot,

I think you kicked off your watcher job with an HTTP request, and it
keeps the port open until it finishes. Only one thread can use the port at the same time. Use a different port for task response traffic, or consider running your watcher as a scheduled task.

Not super robust, and probably not used in production, but i did write
an alternative queque for MarkLogic. It might give you some ideas..

Cheers, Geert

On 11/10/17, 1:06 AM, " on behalf of Eliot Kimber" < on behalf of> wrote:

>I have a system where I have a ³client² ML server that submits jobs to
a >set of remote ML servers, checking their task queues and keeping each >server¹s queue at a max of 100 queued items (the remote servers could
go >away without notice so the client needs to be able to restart tasks and >not have too many things queued up that would just have to
resubmitted). > >The remote tasks then talk back to the client to report status and
return >their final results. > >My job submission code use recursive functions to iterate over the set
of >tasks to be submitted, checking for free remote queue slots via the ML >REST API and submitting jobs as the queues empty. This code is spawned >into a separate task in the task server. It uses xdmp:sleep(1000) to >pause between checking the job queues. > >This all works fine, in that my jobs are submitted correctly and the >remote queues fill up. > >However, as long as the job-submission task in the task server is >running, the HTTP app that handles the REST calls from the remote
servers >is blocked (which blocks the remote jobs, which are of course waiting
for >responses from the client). > >If I kill the task server task, then the remote responses are handled
as >I would expect. > >My question: Why would the task server task block the other app? There >must be something I¹m doing or not doing but I have no idea what it
might >be. > >Thanks, > >Eliot >-- >Eliot Kimber > > > > > >_______________________________________________ >General mailing list > >Manage your subscription at: >

_______________________________________________ General mailing list Manage your subscription at:

_______________________________________________ General mailing list Manage your subscription at: