For our needs this would be best handled by nginx. Here's why... Nginx is the first one to know that it considers a server down and has stopped routing traffic to it until fail_timeout occurs.

So regardless of whether its right and the upstream is really down, or was tripped by a false positive, the bottom line is that it is now ignoring that upstream for fail_timeout duration.

Currently nginx is the only one that knows this. So yes I can use Heartbeat or whatever other monitoring tools are out there. But those tools can say an upstream is up, or down, but nginx could have the upstream's state differently (i.e monitoring could say its up when in fact it missed a condition that nginx considered the upstream to be down - so the monitoring goes on saying the upstream is fine, while nginx is treating it as offline - and all the while we have no idea of this).

Bottom line is that it doesn't make any difference whether a monitoring script says an upstream server is down or not. What matters is whether nginx considers it down or not. And for me to know that, nginx needs to tell me.

The beauty of it is that it seems like quite a trivial yet very useful function to implement. Basically where ever the code is that decides to ignore an upstream for fail_timeout, it just needs to call out to some script to launch it and pass it a param like the name of the upstream entity that went down. Seems like something that could be done in just minutes. Unfortunately I'm not a coder or I would take a crack at it.