16 messages in ru.sysoev.nginxRe: Surviving Digg?
FromSent OnAttachments
Neil ShethApr 29, 2008 1:37 pm 
Aleksandar LazicApr 29, 2008 2:07 pm 
Rt IbmerApr 29, 2008 2:26 pm 
Neil ShethApr 29, 2008 4:00 pm 
Neil ShethApr 29, 2008 4:11 pm 
Rt IbmerApr 29, 2008 5:05 pm 
Igor SysoevApr 30, 2008 12:08 am 
Sasa UgrenovicApr 30, 2008 2:24 am 
Aleksandar LazicMay 1, 2008 1:24 pm 
Neil ShethMay 1, 2008 5:08 pm 
Aleksandar LazicMay 3, 2008 2:16 am 
Neil ShethMay 5, 2008 7:39 pm 
Grzegorz NosekMay 5, 2008 11:54 pm 
Rt IbmerMay 6, 2008 5:48 am 
Aleksandar LazicMay 6, 2008 9:39 am 
eliottMay 6, 2008 10:26 am 
Actions with this message:
Paste this link in email or IM:
Paste this link in email or IM:
Atom feed for this thread
Paste this URL into your reader:
Subject:Re: Surviving Digg?Actions...
From:Aleksandar Lazic (al-n@public.gmane.org)
Date:Apr 29, 2008 2:07:23 pm
List:ru.sysoev.nginx

Hi Neil,

On Die 29.04.2008 13:38, Neil Sheth wrote:

We hit the front page of digg the other night, and our servers didn't handle it well at all. Here's a little of what happened, and perhaps someone has some suggestions on what to tweak!

Basic setup, nginx 0.5.35, serving up static image content, and then passing php requests to 2 backend servers running apache, all running red hat el4.

What was/is the network settings on the maschines?

Now, we started seeing the following: upstream timed out (110: Connection timed out) while connecting to upstream

What was the load on the backends? What are the settings of apache? Have you take a looke about

netstat -nt

how many FIN* things do you have?

So, perhaps the 2 backend servers couldn't handle the load? We were serving the page mostly out of memcache at this point. In any case, couldn't figure out why that wasn't sufficient, so we replaced the page with a static html one.

This seemed to help, but we were now seeing a lot of these: connect() failed (113: No route to host) while connecting to upstream no live upstreams while connecting to upstream

Have you put names or ip-addresses into the nginx config?

This wasn't on every request, but a significant percentage. This, we couldn't figure out. Why couldn't it connect to the backend servers? We ended up rebooting both of the backend servers, and these errors stopped.

Again load and netstat?!

Cheers

Aleks