 | Start a set with this search |
 | Include this search in one of my sets |
 | Exclude this search from one of my sets |
 | Permalink to these results Paste this link in email or IM: |
 | Atom feed for tracking future search results Paste this URL into your reader: |
6 messages in ru.sysoev.nginxRe: SSL FD Leak| From | Sent On | Attachments |
|---|---|---|
| Ben Maurer | Dec 30, 2007 12:47 pm | |
| Ben Maurer | Dec 30, 2007 2:11 pm | |
| Ben Maurer | Dec 30, 2007 2:55 pm | |
| Ben Maurer | Jan 3, 2008 9:37 am | |
| Igor Sysoev | Jan 4, 2008 12:32 am |  .Other |
| Igor Sysoev | Jan 4, 2008 12:43 am |
| Permalink for this message Paste this link in email or IM: |
| Permalink for this thread Paste this link in email or IM: |
| Atom feed for this thread Paste this URL into your reader: |
| Subject: | Re: SSL FD Leak | Actions... |
|---|---|---|
| From: | Ben Maurer (bmau...@public.gmane.org) | |
| Date: | Dec 30, 2007 2:55:28 pm | |
| List: | ru.sysoev.nginx | |
Ben Maurer wrote:
Ben Maurer wrote:
Hi,
On a server that has quite a few SSL connections, I started to notice that FDs were leaking. I set the load balancer in front of nginx to stop sending new requests to one server for a few minutes (to let the keepalive time expire) and found that there were a few thousand FDs open. netstat says that there are many sockets in the CLOSE_WAIT and ESTABLISHED state for the SSL server. Many of them have data in receive queue.
Any ideas what might cause this? This is an up-to-date 0.5.x install.
Some progress on debugging this -- it may have to to do with the deferred setting.
I've managed to get straces like this:
accept(6, {sa_family=AF_INET, sin_port=htons(35327), sin_addr=inet_addr("127.0.0.1")}, [16]) = 92 ioctl(92, FIONBIO, [1]) = 0 recv(92, 0xbf9c6c2b, 1, MSG_PEEK) = -1 EAGAIN (Resource temporarily unavailable)
by using:
ab -c500 -n2000 https://localhost:8095/
and aborting in the middle. It seems that these straces are the ones that result in leaked FDs. The trace really doesn't make much sense to me. Deferred accept promises that the socket only goes into accept once it has data or if it's ready to be closed. Neither of these should result in an EAGAIN. Regardless, it seems the problem is that the FD never gets added to epoll at this point.
It seems like commenting out the check for HTTP requests on the socket made everything work. There's probably a way to do this more correctly (eg, get the event added back into the epoll structure). With that said, maybe it'd be possible to avoid the MSG_PEEK call completely. Openssl is good at detecting this error:
2007/12/30 17:16:43 [crit] 18303#0: *2 SSL_do_handshake() failed (SSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request) while reading client request line, client: 127.0.0.1, server: warp10
The only disadvantage to doing this is that currently, nginx seems to wait until the header has been fully parsed to send an error. However, the client wouldn't be able to notice a difference unless:
1. The initial request didn't fit in a single write() (it'd have to be a few KB for this to happen) 2. It was super-picky about getting input before output.
Given that this is a rare error condition anyways, it doesn't seem worth the extra system call to handle this case...
- Ben