I had another moment of inspiration regarding the ether_ifdetach() races during my shower this morning. Maybe this will gives us a viable solution this time.

The reason we are having races is that once we call foo_stop(), the driver state is out of sync with the ifnet state because IFF_UP is still set even though the driver effectively has marked the interface down. The obvious solution to that is to have the ifnet code "officially" mark the driver down by clearing IFF_UP. This would result in foo_ioctl() calling foo_stop() rather than foo_detach() calling it explicitly. I think this is actually cleaner in that drivers already rely on the ifnet layer calling foo_init() and don't call foo_init() in foo_attach(). Moving foo_stop() out of foo_detach() and into the ifnet layer would thus be more consistent. foo_detach() would go from:

if (device_is_attached(sc)) { FOO_LOCK(sc); foo_stop(sc); FOO_UNLOCK(sc); callout_drain(&sc->sc_stat_callout); ether_ifdetach(sc->sc_ifp); }

to just:

if (device_is_attached(sc)) { ether_ifdetach(sc->sc_ifp); callout_drain(&sc->sc_stat_callout); }

The only question then is when should ether_ifdetach() mark the interface as down, and this actually ends up nice as it lets us fix all the other races with BPF and userland ioctls. Basically, we should detach the ifnet from userland so no more userland requests can come in, then tear down kernel consumers such as BPF, and mark the interface down resulting in an ioctl() that clears IFF_UP and calls foo_stop().