27 messages in ru.sysoev.nginxRe: Weird 0.8.11.1 connections spike| From | Sent On | Attachments |
|---|---|---|
| Jeff Waugh | Aug 29, 2009 5:50 am | |
| Igor Sysoev | Aug 29, 2009 6:48 am | |
| Jeff Waugh | Aug 29, 2009 7:32 am | |
| Igor Sysoev | Aug 29, 2009 8:08 am | |
| Jeff Waugh | Aug 29, 2009 8:45 am | |
| Igor Sysoev | Aug 29, 2009 9:40 am | |
| Jim Ohlstein | Aug 30, 2009 7:59 am | |
| Igor Sysoev | Aug 30, 2009 8:31 am | |
| Jim Ohlstein | Aug 30, 2009 8:52 am | |
| Igor Sysoev | Aug 30, 2009 9:29 am | |
| Jim Ohlstein | Aug 30, 2009 7:55 pm | |
| Jeff Waugh | Aug 30, 2009 8:03 pm | |
| Igor Sysoev | Aug 30, 2009 10:23 pm | |
| Igor Sysoev | Aug 30, 2009 10:46 pm | |
| Jeff Waugh | Aug 30, 2009 11:36 pm | |
| Jeff Waugh | Aug 31, 2009 12:14 am | |
| Igor Sysoev | Aug 31, 2009 12:24 am | |
| Jeff Waugh | Aug 31, 2009 12:55 am | |
| Igor Sysoev | Aug 31, 2009 1:34 am | |
| Jeff Waugh | Aug 31, 2009 1:36 am | |
| Jeff Waugh | Aug 31, 2009 2:03 am | |
| Igor Sysoev | Aug 31, 2009 3:37 am | |
| Igor Sysoev | Aug 31, 2009 4:27 am |  .Other |
| Jim Ohlstein | Aug 31, 2009 5:13 am | |
| Igor Sysoev | Aug 31, 2009 5:27 am | |
| Jeff Waugh | Aug 31, 2009 6:53 am | |
| Igor Sysoev | Aug 31, 2009 7:07 am |
| Subject: | Re: Weird 0.8.11.1 connections spike | |
|---|---|---|
| From: | Jim Ohlstein (ji...@ohlste.in) | |
| Date: | Aug 30, 2009 7:59:47 am | |
| List: | ru.sysoev.nginx | |
Igor Sysoev wrote:
On Sun, Aug 30, 2009 at 01:45:30AM +1000, Jeff Waugh wrote:
<quote who="Igor Sysoev">
2009/08/29 23:15:52 [alert] 8716#0: open socket #152 left in connection 105
This is probably after reconfiguraiton.
Sorry, yes, I should've checked the time -> that was from when I reverted to 0.8.10... there no alerts prior to that.
There are signinficant changes in 0.8.11, but you case is strange: I'am testing 0.8.11 on a host with 400r/s with a lot of proxied SSIed subrequests and static files, and there is no connection leak.
Have you built any third party modules ?
None.
btw, the majority of my non-static traffic is fastcgi over unix socket. I'm going to do the upgrade again and watch the activity.
Could you do the following:
1) enable coredumps 2) set in nginx.conf: debug_points abort; 3) reconfigure nginx, if there are open connections, then nginx creates coredump on exit 4) look in log for alerts: open socket #... left in connection NN 5) run "gdb /path/to/nginx /path/to/core", then
p ((ngx_connection_t *) ngx_cycle->connections[NN]->data)->uri p ((ngx_connection_t *) ngx_cycle->connections[NN]->data)->main->count
do not forget to set NN from log message.
Igor,
On one server I seem to be seeing the same issue. Twice since upgrading to 0.8.11 I have found the site unresponsive. It's a relatively slow vBulletin site on a very small VPS. We serve a fair bit of static content (photos mostly) and PHP via fastcgi.
OS is CentOS 5.3 Linux - 2.6.18-028stab062.3
sample of error log (it grew to 36 GB in a few hours):
2009/08/30 07:50:20 [alert] 2043#0: accept() failed (24: Too many open files) 2009/08/30 07:50:20 [alert] 2043#0: accept() failed (24: Too many open files) 2009/08/30 07:50:20 [alert] 2043#0: accept() failed (24: Too many open files) 2009/08/30 07:50:20 [alert] 2043#0: accept() failed (24: Too many open files) 2009/08/30 07:50:20 [alert] 2043#0: accept() failed (24: Too many open files) 2009/08/30 07:50:20 [alert] 2043#0: accept() failed (24: Too many open files)
...
2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files) 2009/08/30 10:29:00 [alert] 2042#0: accept() failed (24: Too many open files)
When I did shut it down I saw hundreds of lines like:
2009/08/30 10:29:00 [alert] 2042#0: open socket #1010 left in connection 1002 2009/08/30 10:29:00 [alert] 2042#0: open socket #981 left in connection 1003 2009/08/30 10:29:00 [alert] 2042#0: open socket #994 left in connection 1004 2009/08/30 10:29:00 [alert] 2042#0: open socket #1017 left in connection 1005 2009/08/30 10:29:00 [alert] 2042#0: open socket #1012 left in connection 1006 2009/08/30 10:29:00 [alert] 2042#0: open socket #1011 left in connection 1007 2009/08/30 10:29:00 [alert] 2042#0: open socket #1016 left in connection 1008 2009/08/30 10:29:00 [alert] 2042#0: open socket #1014 left in connection 1009 2009/08/30 10:29:00 [alert] 2042#0: open socket #1019 left in connection 1010 2009/08/30 10:29:00 [alert] 2042#0: open socket #1020 left in connection 1011 2009/08/30 10:29:00 [alert] 2042#0: open socket #1013 left in connection 1012 2009/08/30 10:29:00 [alert] 2042#0: open socket #995 left in connection 1013 2009/08/30 10:29:00 [alert] 2042#0: open socket #988 left in connection 1014 2009/08/30 10:29:00 [alert] 2042#0: open socket #1023 left in connection 1015 2009/08/30 10:29:00 [alert] 2042#0: aborting
Other servers seem to be running fine including ones with busy sites. For the moment I have reverted that VPS to 0.8.10.
Jim