At a high level, what am I missing / oversimplifying?

Perhaps instead, a simpler and more efficient approach may be for me to write to
a log when nginx stops routing requests to an upstream box. I cannot use the
events it logs current, because it doesn't differentiate between a failed
attempt vs. enough failed attempts based on max_fails and fail_timeout.

For instance it'll write to the log saying a connection or timeout occurred with
an upstream, but does not record an entry saying that this has occurred
max_fails times and it is now taking that upstream out of rotation for
fail_timeout duration. I think at a minimum even if nothing further is done
here it should at least report that event in the logs. Also I may be mistaken
but I believe the current logging it does in this regard is only seen with like
"notices" - I would expect those to get a high level logging priority like error
or critical.

That does not surprise me at all! This is over my head for sure. All I know is
that I disparately need to know when nginx stops routing requests to an
upstream. And the only way to know 100% sure that its doing so is for nginx to
tell us, not to try and determine what nginx must be seeing based on some
external monitoring.

Agree it is lousy! :) At the same time, for an event that (hopefully) happens
very rarely I wouldn't mind the couple of second hit from inefficiency here.
That being said clearly better solutions can be engineered (just not be my
unfortunately).

That is an excellent idea! Basically some sort of event / hook system that nginx
can use to report various events (including "upstream down" and "previously
detected downed ppstream now back up").

Perfect! I just need the hook.

Thank you for your consideration and I hope these discussions help make in
incredible product even more robust.