24 messages in ru.sysoev.nginxRe: Centralized logging for multiple ...
FromSent OnAttachments
Kingsley ForemanApr 12, 2009 8:26 pm 
Anton YuzhaninovApr 13, 2009 11:46 am 
Gena MakhomedApr 13, 2009 12:31 pm 
Igor SysoevApr 15, 2009 6:12 am 
Glen LumanauApr 15, 2009 6:23 am 
Dave CheneyApr 15, 2009 6:32 am 
Kingsley ForemanApr 15, 2009 6:35 am 
Glen LumanauApr 15, 2009 6:35 am 
Dave CheneyApr 15, 2009 7:06 am 
Kon WilmsApr 15, 2009 8:40 am 
Michael ShadleApr 15, 2009 8:43 am 
W. Andrew Loe IIIApr 16, 2009 7:37 pm 
Gabriel RamugliaApr 16, 2009 8:41 pm 
W. Andrew Loe IIIApr 16, 2009 8:54 pm 
Michael ShadleApr 16, 2009 9:08 pm 
Gabriel RamugliaApr 17, 2009 3:32 am 
Michael ShadleApr 17, 2009 8:42 am 
Gabriel RamugliaApr 17, 2009 10:36 am 
Gabriel RamugliaApr 17, 2009 10:52 am 
Gabriel RamugliaApr 17, 2009 5:06 pm 
Kon WilmsApr 17, 2009 6:02 pm 
Gabriel RamugliaApr 18, 2009 2:22 am 
Kon WilmsApr 18, 2009 9:15 am 
Gabriel RamugliaApr 18, 2009 12:02 pm 
Actions with this message:
Paste this link in email or IM:
Paste this link in email or IM:
Atom feed for this thread
Paste this URL into your reader:
Subject:Re: Centralized logging for multiple serversActions...
From:W. Andrew Loe III (and@andrewloe.com)
Date:Apr 16, 2009 7:37:34 pm
List:ru.sysoev.nginx

Its commercial, but Splunk is amazing at this. I think you can process a few hundred MB/day on the free version. http://splunk.com/

You set up a light-weight forwarder on every node you are interested in, and then it slurps the files up and relays them to a central splunk installation. It will queue internally if the master goes away. Tons of support for sending different files different directions etc. We have it setup in the default Puppet payload so every log on every server is always centralized and searchable.

On Wed, Apr 15, 2009 at 8:44 AM, Michael Shadle <mike@gmail.com> wrote:

On Wed, Apr 15, 2009 at 7:06 AM, Dave Cheney <da@cheney.net> wrote:

What about

cat *.log | sort -k 4

or just

cat *whatever.log >today.log

I assume the processing script can handle out-of-order requests. but I guess that might be an arrogant assumption. :)

I do basically the same thing igor does, but would love to simplify it by just having Host: header counts for bytes (sent/received/total amount of bytes used, basically) and how many http requests. Logging just enough of that to a file and parsing it each night seems kinda amateur...