Except during the window when screen switching, or of course you could just ssh in remotely and gdb or similar a process with it as controlling tty running on your console and issue a vt switch back, then mount it. Ironically the move from a root owned X server has made that much simpler to automate, although it was always possible.

Given you can often guess from the idle data if the victim has gone away from the box it's not ideal. Even better any mess will appear on my display and get hidden when I flip it back.

The only way to stop that is to make use of the display locking facility which takes us back where we began in saying that a usable interface is going to need to lock the display.

At that point the current console owner has to choose to allow the console to be switched which can be limited effectively to physical console access and done synchronously. In turn that means to abuse it I already have physical access to the other users key so could just as easily steal it and the software security is therefore sufficient.

Alan