For our needs this would be best handled by nginx. Here's why... Nginx is the
first one to know that it considers a server down and has stopped routing
traffic to it until fail_timeout occurs. So regardless of whether its right and
the upstream is really down, or was tripped by a false positive, the bottom line
is that it is now ignoring that upstream for fail_timeout duration.

Currently nginx is the only one that knows this. So yes I can use Heartbeat or
whatever other monitoring tools are out there. But those tools can say an
upstream is up, or down, but nginx could have the upstream's state differently
(i.e monitoring could say its up when in fact it missed a condition that nginx
considered the upstream to be down - so the monitoring goes on saying the
upstream is fine, while nginx is treating it as offline - and all the while we
have no idea of this).

Bottom line is that it doesn't make any difference whether a monitoring script
says an upstream server is down or not. What matters is whether nginx considers
it down or not. And for me to know that, nginx needs to tell me.

The beauty of it is that it seems like quite a trivial yet very useful function
to implement. Basically where ever the code is that decides to ignore an
upstream for fail_timeout, it just needs to call out to some script to launch it
and pass it a param like the name of the upstream entity that went down. Seems
like something that could be done in just minutes. Unfortunately I'm not a
coder or I would take a crack at it.

Yes, absolutely. Before I took a server offline I'd tell my script that nginx
calls to ignore reports from upstream xyz - so when the script was fired off
from nginx it would know not to treat it as an alarm condition.

Thanks for this opportunity to provide feedback. It could be great if someone
from the nginx dev team could comment on whether this is something they would
consider adding. Thank you!!