7 messages in org.freebsd.freebsd-stableLong Format Directory Listing 15x Slo...| From | Sent On | Attachments |
|---|---|---|
| Nikolai Schupbach | Sep 12, 2005 5:27 pm | |
| Kris Kennaway | Sep 12, 2005 6:08 pm | |
| Brooks Davis | Sep 12, 2005 6:33 pm | |
| Kris Kennaway | Sep 12, 2005 6:37 pm | |
| Brooks Davis | Sep 12, 2005 6:46 pm | |
| Nikolai Schupbach | Sep 12, 2005 7:20 pm | |
| Nikolai Schupbach | Sep 14, 2005 7:23 pm |
| Subject: | Long Format Directory Listing 15x Slower in FreeBSD 5.x | |
|---|---|---|
| From: | Brooks Davis (bro...@one-eyed-alien.net) | |
| Date: | Sep 12, 2005 6:33:39 pm | |
| List: | org.freebsd.freebsd-stable | |
On Mon, Sep 12, 2005 at 08:51:44PM -0400, Kris Kennaway wrote:
On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote:
Hello,
We have been trying to migrate to FreeBSD 5.4 from an older 4.x release for one of our busy mail servers. But we have encountered problems with directory listings on 5.4.
Our /var/mail directory contains approximately 8,000 files doing a long directory listing (ls -l) takes approximately 5 min and during this time the CPU is running near 100%, on a FSBD 4 box the same directory contents takes less than 20 seconds to list. Even on a directory with 200 files, each file with a different owner, it will still take at least 4-5 seconds to list. The problems only seems to occur when the directory contains files from many different users. (as in /var/mail). If I chown all the files in the /var/mail directory to a single user the directory listing is near instant.
It appears it has something to do with 'ls' looking up the id's in the password database, because if I instruct ls to display numeric IDs (ls -ln), rather than converting to user and group names, the directory with 8,000 files, with 8,000 different owners will list instantly.
The reason this concerns me so much is we are also having a problem with our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while operating on small and even 0 byte mailboxes, when there are approx five or more concurrent POP3 or IMAP sessions. And I can't help thinking that the two problems are related.
Does anyone have any ideas? Has anyone else noticed this problem? If I can't resolve it I'm most likely going to revert to using 4.11, but I'd really like to know what is going on. (and yes I'm using UFS_DIRHASH)
Sounds like the real problem is that you have >8000 users on your machine and lookups are taking a long time. There has been discussion of this problem and how to solve it on freebsd-questions@ several times..I think it involves making a minor change to pwd_mkdb and rebuilding. Please search the archives for that mailing list.
If you are using NIS and have any compat options in /etc/nsswitch.conf your performance will really suck in situations like this. IIRC, the compat code is worse than O(n^2) if you look up each user and the non-compat code is close to O(n). I'd really like to stop generating nsswitch.conf entries that use compat in 7.0.
-- Brooks
-- Any statement of the form "X is the one, true Y" is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4