On Thu, Oct 09, 2008 at 09:23:08AM -0700, Ezra Zygmuntowicz wrote:

Thinking about it, it's not a trivial problem actually. The load balancer must answer *now* with either a backend address or an error. AFAIK there's no mechanism to defer the decision (and thus dispatching the request).

Even if there was a way to do that (e.g. the balancer returns NGX_AGAIN), it wouldn't still solve the problem because Nginx core wouldn't know when to retry, so would probably have to busy loop. The API might include passing back an interval but it would always be a rough estimate. Tying the retry with more load balancer interaction (e.g. when a request ends with a call to ->peer.free, retry all pending requests) would mostly work (for this case) but wouldn't be 100% bulletproof, as the multiple workers don't communicate (in the usual case; upstream_fair only uses shared memory). Attaching retries to FD actions presents a similar problem and changes the behaviour slightly (more often polling, more accurate timing, more CPU usage).

Regarding multiple workers, I can't see a clean solution currently, but restricting ourselves to the one-worker case, the upstream core could keep a queue (or rbtree or...) of pending requests per-upstream and export a function like

ngx_http_upstream_retry(ngx_http_upstream_srv_conf_t *us);

Which would walk the queue calling ->peer.get again. The balancer could then respond "this guy isn't ready yet, try another one" (NGX_EAGAIN) separately from "everything is still busy, no point in asking again" (NGX_DONE?). A nice touch would be a timer interface to set up a hard timeout (I'm sure it can be done now but don't know the whole API yet ;)

The queue could be also maintained in the load balancer but still Nginx needs to be modified to accept NGX_AGAIN as a balancer verdict.

Details would need some ironing out but I think the overall idea is workable.

@Igor, do you think it could work?

BTW, that's just insane (IMVHO). If a Rails app can process requests sequentially faster than concurrently, then why TF does it accept more requests than it can handle? If the app had a short listen queue and ran as a simple iterative server, overlimit requests would simply error out with ECONNREFUSED, which can be rather gracefully handled inside Nginx.