Hi,

On a production server, I found that nginx appears to leak when using ssl. With some investigation, it seems that this is actually memory fragmentation due to the session cache. I made a very simple configuration for the server:

daemon off; master_process off; pid /tmp/x.pid; error_log /tmp/x.log; events { use epoll; } http { client_body_temp_path /tmp; proxy_temp_path /tmp; fastcgi_temp_path /tmp; access_log /tmp/access.log; server { listen localhost:8666; ssl on; ssl_certificate /home/bmaurer/x.pem; ssl_certificate_key /home/bmaurer/x.pem; root /tmp; } }

Then I did a benchmark with the following command:

ab -c500 -n20000 https://localhost:8666/

After doing this, the server uses ~ 30 MB of RSS. Running it once more, it uses ~ 40 MB of RSS. Valgrind claims that there are no "leaks", it seems that there's just a really bad case of memory fragmentation.

I tried applying this to the SSL configuration:

ssl_session_cache builtin:2;

Doing so resulted in the memory use of the nginx server staying relatively low (it appears the memory was reclaimed from the OS after it was used).

It seems like it might be worth switching to something like the shared memory cache by default. Keeping the long-lived session cache in a different pool of memory avoids the risk of large amounts of memory getting pinned in.

One other thing I noticed while investigating this stuff was that nginx keeps a 16 KB buffer for each SSL connection for the entire duration of the connection. I've attached a patch that keeps this buffer alive only while there's a pending write. Sadly, there are some relatively large buffers internal to openssl as well, which means the overhead for SSL keepalive connections is pretty high.

- Ben

Index: nginx-0.5.34/src/event/ngx_event_openssl.c =================================================================== --- nginx-0.5.34.orig/src/event/ngx_event_openssl.c 2007-12-26
11:24:42.000000000 -0500 +++ nginx-0.5.34/src/event/ngx_event_openssl.c 2007-12-26 11:31:34.000000000
-0500 @@ -348,11 +348,6 @@

if (flags & NGX_SSL_BUFFER) { sc->buffer = 1; - - sc->buf = ngx_create_temp_buf(c->pool, NGX_SSL_BUFSIZE); - if (sc->buf == NULL) { - return NGX_ERROR; - } }

sc->connection = SSL_new(ssl->ctx); @@ -793,6 +788,23 @@ limit = NGX_MAX_UINT32_VALUE - ngx_pagesize; }

+ /* + * try to create c->ssl->buf lazily. It is deallocated when it has + * been fully used. This will reduce the overhead of keepalive + * connections. + */ + if (c->ssl->buf == NULL) { + c->ssl->buf = ngx_create_temp_buf(c->pool, NGX_SSL_BUFSIZE); + if (c->ssl->buf == NULL) { + return NGX_CHAIN_ERROR; + } + } + if (c->ssl->buf->start == NULL) { + c->ssl->buf->start = ngx_palloc(c->pool, NGX_SSL_BUFSIZE); + if (c->ssl->buf->start == NULL) { + return NGX_CHAIN_ERROR; + } + }

buf = c->ssl->buf; send = 0; @@ -874,6 +886,9 @@ c->buffered |= NGX_SSL_BUFFERED;

} else { + if (ngx_pfree(c->pool, buf->start) == NGX_OK) { + buf->start = NULL; + } c->buffered &= ~NGX_SSL_BUFFERED; }