How can I use a MySQL table instead of /etc/courier/hosteddomains?
One thing nothing to do with the other. MySQL-based authentication can be
set up with or without listing the domains involved in hosteddomains.
Is there a general rule for lookups? For example, Postfix lets you put
mysql:/path/to/config.file as the argument to any lookup table
Read what INSTALL says about configuration and MySQL authentication, review
the description of hosteddomains in the courier(8) man page, and see the
comments in the default authmysqlrc configuration file.