|Stefanita Rares Dumitrescu||Nov 20, 2009 3:21 pm|
|Maxim Dounin||Nov 20, 2009 4:17 pm|
|Stefanita Rares Dumitrescu||Nov 20, 2009 4:39 pm|
|Maxim Dounin||Nov 20, 2009 5:03 pm|
|Stefanita Rares Dumitrescu||Nov 21, 2009 5:18 am|
|Maxim Dounin||Nov 21, 2009 9:29 am|
|Stefanita Rares Dumitrescu||Nov 21, 2009 10:47 am|
|Gabriel Ramuglia||Nov 21, 2009 12:55 pm|
|Stefanita Rares Dumitrescu||Nov 21, 2009 1:43 pm|
|Maxim Dounin||Nov 21, 2009 2:55 pm|
|Stefanita Rares Dumitrescu||Nov 21, 2009 5:01 pm|
|Subject:||Re: nginx high i/o|
|From:||Stefanita Rares Dumitrescu (kat...@keptprivate.com)|
|Date:||Nov 21, 2009 5:18:53 am|
fastcgi_max_temp_file_size 0; This directive turns off fastcgi buffering according to the source code.
i did this, and the buffer errors are gone. however, the pages load very very slow. take about 5-10 seconds to load the page.
could it be my php-fpm.conf ?
the thing is why i am going nuts about this is because the same exact config (same nginx/php-fpm), i am running on a centos server, and i am able to push over 200 mb/s and everything works perfectly fine.
i switched over to freebsd to squeeze some more performance out of the server though.
also i have noticed that when more and more connections are established to the server, my php-cgi processes are frozen in "sbwait" state:
32514 nobody 1 4 0 60816K 14252K sbwait 4 0:12 0.00% php-cgi 32562 nobody 1 4 0 60816K 14212K sbwait 2 0:11 0.00% php-cgi 32577 nobody 1 4 0 60816K 14240K sbwait 6 0:11 0.00% php-cgi 32479 nobody 1 4 0 60816K 14244K sbwait 0 0:11 0.00% php-cgi 32475 nobody 1 4 0 60816K 14256K sbwait 0 0:11 0.00% php-cgi 32497 nobody 1 4 0 60816K 14228K sbwait 0 0:11 0.00% php-cgi 32527 nobody 1 4 0 60816K 14208K sbwait 1 0:11 0.00% php-cgi 32564 nobody 1 4 0 60816K 14192K sbwait 4 0:10 0.00% php-cgi
posted php-fpm.conf below.
<?xml version="1.0" ?> <configuration>
All relative paths in this config are relative to php's install prefix
Pid file <value name="pid_file">/usr/local/kphp/logs/php-fpm.pid</value>
Error log file <value name="error_log">/usr/local/kphp/logs/php-fpm.log</value>
Log level <value name="log_level">notice</value>
When this amount of php processes exited with SIGSEGV or SIGBUS ... <value name="emergency_restart_threshold">10</value>
... in a less than this interval of time, a graceful restart will be initiated. Useful to work around accidental curruptions in accelerator's shared memory. <value name="emergency_restart_interval">1m</value>
Time limit on waiting child's reaction on signals from master <value name="process_control_timeout">5s</value>
Set to 'no' to debug fpm <value name="daemonize">yes</value>
Name of pool. Used in logs and stats. <value name="name">nginx</value>
Address to accept fastcgi requests on. Valid syntax is 'ip.ad.re.ss:port' or just 'port' or '/path/to/unix/socket' <value name="listen_address">127.0.0.1:8000</value>
Set listen(2) backlog <value name="backlog">-1</value>
Set permissions for unix socket, if one used. In Linux read/write permissions must be set in order to allow connections from web server. Many BSD-derrived systems allow connections regardless of permissions. <value name="owner"></value> <value name="group"></value> <value name="mode">0666</value> </value>
Additional php.ini defines, specific to this pool of workers. <value name="php_defines"> <!-- <value name="sendmail_path">/usr/sbin/sendmail -t -i</value> --> <!-- <value name="display_errors">0</value> --> </value>
Unix user of processes <value name="user">nobody</value>
Unix group of processes <value name="group">nobody</value>
Process manager settings <value name="pm">
Sets style of controling worker process count. Valid values are 'static' and 'apache-like' <value name="style">static</value>
Sets the limit on the number of simultaneous requests that will be served. Equivalent to Apache MaxClients directive. Equivalent to PHP_FCGI_CHILDREN environment in original php.fcgi Used with any pm_style. <value name="max_children">128</value>
Settings group for 'apache-like' pm style <value name="apache_like">
Sets the number of server processes created on startup. Used only when 'apache-like' pm_style is selected <value name="StartServers">32</value>
Sets the desired minimum number of idle server processes. Used only when 'apache-like' pm_style is selected <value name="MinSpareServers">32</value>
Sets the desired maximum number of idle server processes. Used only when 'apache-like' pm_style is selected <value name="MaxSpareServers">64</value>
The timeout (in seconds) for serving a single request after which the worker process will be terminated Should be used when 'max_execution_time' ini option does not stop script execution for some reason '0s' means 'off' <value name="request_terminate_timeout">0s</value>
The timeout (in seconds) for serving of single request after which a php backtrace will be dumped to slow.log file '0s' means 'off' <value name="request_slowlog_timeout">0s</value>
The log file for slow requests <value name="slowlog">logs/slow.log</value>
Set open file desc rlimit <value name="rlimit_files">2048</value>
Set max core size rlimit <value name="rlimit_core">0</value>
Chroot to this directory at the start, absolute path <value name="chroot"></value>
Chdir to this directory at the start, absolute path <value name="chdir"></value>
Redirect workers' stdout and stderr into main error log. If not set, they will be redirected to /dev/null, according to FastCGI specs <value name="catch_workers_output">yes</value>
How much requests each process should execute before respawn. Useful to work around memory leaks in 3rd party libraries. For endless request processing please specify 0 Equivalent to PHP_FCGI_MAX_REQUESTS <value name="max_requests">500</value>
Comma separated list of ipv4 addresses of FastCGI clients that allowed to connect. Equivalent to FCGI_WEB_SERVER_ADDRS environment in original php.fcgi (5.2.2+) Makes sense only with AF_INET listening socket. <value name="allowed_clients">127.0.0.1</value>
Pass environment variables like LD_LIBRARY_PATH All $VARIABLEs are taken from current environment <value name="environment"> <value name="HOSTNAME">$HOSTNAME</value> <value name="PATH">/usr/local/bin:/usr/bin:/bin</value> <value name="TMP">/tmp</value> <value name="TMPDIR">/tmp</value> <value name="TEMP">/tmp</value> <value name="OSTYPE">$OSTYPE</value> <value name="MACHTYPE">$MACHTYPE</value> <value name="MALLOC_CHECK_">2</value> </value>
Maxim Dounin wrote:
On Sat, Nov 21, 2009 at 01:39:38AM +0100, Stefanita Rares Dumitrescu wrote:
actually i got a bunch of errors, in error log, as i enabled it to get some idea on what's going on:
2009/11/21 08:47:44 [warn] 24393#0: *57036 an upstream response is buffered to a temporary file /var/tmp/nginx/fastcgi_temp/1/56/0024741561 while reading upstream, client:
i am so sleepy right now, could you please tell me where do i get the fastcgi buffers optimizations ? i searched on the wiki but i can't think straight.
Start from tuning fastcgi_buffers:
If you haven't had enough memory to deligate it for buffers you may consider using fastcgi_max_temp_file_size 0; - this will disable disk buffering at cost of not reading data from backend connection (i.e. keeping backend busy).
You may also consider investigating why your backend replies are so big (fastcgi_buffers should be 64k in total on your platform, this is much more than typical generated html...) and moving them to nginx itself if possible (e.g. by using X-Accel-Redirect).
Maxim Dounin wrote:
On Sat, Nov 21, 2009 at 12:21:52AM +0100, Stefanita Rares Dumitrescu wrote:
i have posted a question a while ago, and now i am coming back once more.
i have now a dual quad core xeon system, running freebsd 7.2 latest patch level, 64 bit, and 16 gb ram. using the latest nginx-devel.
the OS runs on a ssd, the db is set on a normal drive, and the site files reside on another ssd.
i had a pretty similar config just instead i was using centos.
the i/o on the main drive (os drive) is constantly at 22 mb/s.
i have disabled nginx logging, and everything has been setup from the ground. i have no idea why do i get such high i/o, because i have no other apps that could cause this load.
Do you use proxy for big replies and/or have big client uploads?
If yes, it's probably caused by disk buffering (you should have warnings in logs, but you have no logs...) and you probably want to tune your buffer sizes and/or proxy_temp_file_max_size.
could you please point me to some direction ? i don't know what to do.
ad4 ad5 ad6 KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s 54.56 428 22.79 0.00 0 0.00 32.89 4 0.14 55.05 492 26.46 0.00 0 0.00 26.38 8 0.21 55.24 452 24.40 0.00 0 0.00 36.36 5 0.20 53.20 457 23.76 22.31 19 0.42 19.88 8 0.16 53.33 464 24.15 0.00 0 0.00 48.47 23 1.11 54.14 403 21.32 0.00 0 0.00 16.18 217 3.43 52.75 406 20.90 0.00 0 0.00 17.49 178 3.04 55.75 402 21.90 0.00 0 0.00 13.42 12 0.16