 | Start a set with this search |
 | Include this search in one of my sets |
 | Exclude this search from one of my sets |
 | Permalink to these results Paste this link in email or IM: |
 | Atom feed for tracking future search results Paste this URL into your reader: |
16 messages in ru.sysoev.nginxRe: Surviving Digg?| From | Sent On | Attachments |
|---|---|---|
| Neil Sheth | Apr 29, 2008 1:37 pm | |
| Aleksandar Lazic | Apr 29, 2008 2:07 pm | |
| Rt Ibmer | Apr 29, 2008 2:26 pm | |
| Neil Sheth | Apr 29, 2008 4:00 pm | |
| Neil Sheth | Apr 29, 2008 4:11 pm | |
| Rt Ibmer | Apr 29, 2008 5:05 pm | |
| Igor Sysoev | Apr 30, 2008 12:08 am | |
| Sasa Ugrenovic | Apr 30, 2008 2:24 am | |
| Aleksandar Lazic | May 1, 2008 1:24 pm | |
| Neil Sheth | May 1, 2008 5:08 pm | |
| Aleksandar Lazic | May 3, 2008 2:16 am | |
| Neil Sheth | May 5, 2008 7:39 pm | |
| Grzegorz Nosek | May 5, 2008 11:54 pm | |
| Rt Ibmer | May 6, 2008 5:48 am | |
| Aleksandar Lazic | May 6, 2008 9:39 am | |
| eliott | May 6, 2008 10:26 am |
| Permalink for this message Paste this link in email or IM: |
| Permalink for this thread Paste this link in email or IM: |
| Atom feed for this thread Paste this URL into your reader: |
| Subject: | Re: Surviving Digg? | Actions... |
|---|---|---|
| From: | Neil Sheth (nshe...@public.gmane.org) | |
| Date: | Apr 29, 2008 4:00:39 pm | |
| List: | ru.sysoev.nginx | |
On Tue, Apr 29, 2008 at 2:07 PM, Aleksandar Lazic
<al-n...@public.gmane.org> wrote:
Hi Neil,
On Die 29.04.2008 13:38, Neil Sheth wrote:
We hit the front page of digg the other night, and our servers didn't handle it well at all. Here's a little of what happened, and perhaps someone has some suggestions on what to tweak!
Basic setup, nginx 0.5.35, serving up static image content, and then passing php requests to 2 backend servers running apache, all running red hat el4.
What was/is the network settings on the maschines?
What specific settings are you asking about?
Now, we started seeing the following: upstream timed out (110: Connection timed out) while connecting to upstream
What was the load on the backends? What are the settings of apache? Have you take a looke about
netstat -nt
how many FIN* things do you have?
Right now, shows about 60. Not sure what the count of FIN objects was at the time of the digg. I did run the following (found in a forum somewhere, to give connection counts by IP): netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr This showed the number of connections to the backend servers to be almost 1000 each.
So, perhaps the 2 backend servers couldn't handle the load? We were serving the page mostly out of memcache at this point. In any case, couldn't figure out why that wasn't sufficient, so we replaced the page with a static html one.
This seemed to help, but we were now seeing a lot of these: connect() failed (113: No route to host) while connecting to upstream no live upstreams while connecting to upstream
Have you put names or ip-addresses into the nginx config?
IP addresses
This wasn't on every request, but a significant percentage. This, we couldn't figure out. Why couldn't it connect to the backend servers? We ended up rebooting both of the backend servers, and these errors stopped.
Again load and netstat?!
Load didn't actually look that bad, if I recall. Probably peaks around 4 while this was occuring, but generally lower.
Cheers
Aleks
Thanks for the help!