Hello!

On Thu, Oct 14, 2010 at 12:26:59PM +0000, ipta...@yahoo.com wrote:

Basic tunings you have to apply when serving static which doesn't fit into memory are:

If you use sendfile:

- Make sure your OS uses appropriate read-ahead for sendfile to avoid trashing disks with small requests (and seeks). For FreeBSD 8.1+ it should be enough to set read_ahead directive in nginx config (0.8.18+).

http://sysoev.ru/nginx/docs/http/ngx_http_core_module.html#read_ahead (in Russian)

Using bigger socket buffers (listen ... sndbuf=... in nginx config) may help too.

- If serving large files - make sure you use appropriate sendfile_max_chunk to avoid blocking nginx worker on disk for too long.

- Consider switching sendfile off if you can't persuade it to read large blocks from disk.

If not using sendfile:

- Tune output_buffers (again, to avoid trashing disks with small requests and seeks). Default is 2x32k, which will result in 4 disk requests for 100k file. Changing it to 1x128k would result in 2x memory usage but 4x less disk requests, this is probably good thing to do if you are disk-bound.

In both cases:

- Using aio may help a lot ("aio sendfile" is only available under FreeBSD) by adding more concurency to disk load and generally improving nginx interactivity. Though right now it requires patches to avoid socket leaks, see here:

http://nginx.org/pipermail/nginx-devel/2010-October/000498.html

- Using directio may help to improve disk cache effictiveness by excluding large files (if you have some) from cache. Though keep in mind that it disables sendfile so if you generally tune for sendfile - you may have to apply output_buffers tunings as well.

It's hard to say anything more than this without knowing lots of details.