How do I make this work?

location ~ /~someapp(.*)$ { alias /home/someapp/root$1; try_files $uri /~someapp/err404.htm; }

The try_files never catch the $uri (nor $1) and returns 500 (redirection cycle)

Am I 'forced' to use error_page in this case?

This works:

location ~ /~someapp(.*)$ { alias /home/someapp/root$1; error_page 404 /~someapp/err404.htm; }

and without regex the error_page and try_files seem to be completely ignored: neither    location /~someapp/ { alias /home/someapp/root/; error_page 404 /~someapp/err404.htm; } nor    location /~someapp/ { alias /home/someapp/root/; try_files $uri /~someapp/err404.htm; } works.