atom feed5 messages in ru.sysoev.nginxSlow down, but not stop, serving page...
FromSent OnAttachments
rastranoAug 11, 2011 8:36 am 
rastranoAug 11, 2011 8:55 am 
Rami EssaidAug 11, 2011 2:07 pm 
rastranoAug 12, 2011 2:33 am 
Rami EssaidAug 12, 2011 4:32 am 
Subject:Slow down, but not stop, serving pages to bot that doesn't respect robots.txt delay
From:rastrano (ngin@nginx.us)
Date:Aug 11, 2011 8:36:39 am
List:ru.sysoev.nginx

Hi all.

I want to differentiate the nginx awesome rate limiting for requests, in order to enforce bots to respect my directives (but not to block them):

so i created these limit_req_zone:

limit_req_zone $binary_remote_addr zone=antiddosspider:1m rate=1r/m; limit_req_zone $binary_remote_addr zone=antiddosphp:1m rate=1r/s; limit_req_zone $binary_remote_addr zone=antiddosstatic:1m rate=10r/s;

Now, i ask, is it possible to configure something like this?

if ( $http_user_agent ~* (?:bot|spider) ) { limit_req zone=antiddosspider burst=1; }

location / { limit_req zone=antiddosphp burst=100; proxy_pass http://localhost:8181; include /etc/nginx/proxy.conf; }

I don't know many spider that are crawling my site, but i don't wanna lose the possibility to be indexed if they are not malware! ...But 1 page for minute please ;)

Best regards, Stefano

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,213704,213704#msg-213704