Michael, first, I wanted to thank you for doing this work, and especially for publishing the results. I'd love to see more such work.

I have some questions about the NFS server used that I didn't see answered in the paper.

First, what kernel was used in Debian? I've been using XFS with 2.4.20 (for something other than our mail store server) for some time, and despite claims to the contrary, its performance is in many ways really quite poor. In particular, it has a number of problems that make it especially bad for Maildir. Initial scans of directories tend to be very slow. This would create a long delay after logging in to a POP3 server, before getting a message list. File system operations also tend to be extremely slow. Deleting files is a very expensive operation, and since that is the normal course of action on a POP3 store (though I didn't get the impression that your clients were doing that), this further degrades the performance of the system. Updating the access time on files will normally be done when retrieving mail, as well, which adds another very expensive filesystem operation to the transaction.

Which leads me to the other question: What mount options did you use for the XFS volume? I've found that the "noatime" option is essential on many access patterns, especially when you have large directories of small files which will be scanned. Using the "noatime" option substantially reduces the amount of disk write activity that is required, which normally will seriously degrade read performance.

If you're willing to share, and still have the materials, I'd like to take some of your scripts and follow up on some of those details to see how performance characteristics change on filesystem based use.