45 messages in com.googlegroups.pylons-discussRe: Reducing pylons app memory usage?
FromSent OnAttachments
Marcin Kasperski16 Oct 2007 07:21 
Max Ischenko16 Oct 2007 10:46 
Marcin Kasperski17 Oct 2007 02:59 
Dalius Dobravolskas17 Oct 2007 03:04 
Christoph Haas17 Oct 2007 07:12 
Max Ischenko17 Oct 2007 11:02 
Ben Bangert17 Oct 2007 13:19 
Christoph Haas17 Oct 2007 14:23 
Ben Bangert17 Oct 2007 16:32 
Graham Dumpleton17 Oct 2007 16:56 
Ian Bicking17 Oct 2007 16:58 
Marcin Kasperski18 Oct 2007 03:18 
Ben Bangert18 Oct 2007 11:34 
Philip Jenvey18 Oct 2007 12:22 
Ben Bangert18 Oct 2007 12:32 
Ian Bicking18 Oct 2007 12:37 
Marcin Kasperski19 Oct 2007 02:35 
Marcin Kasperski19 Oct 2007 02:42 
Graham Dumpleton21 Oct 2007 16:12 
Jon Rosebaugh21 Oct 2007 16:48 
Graham Dumpleton21 Oct 2007 18:20 
Marcin Kasperski22 Oct 2007 05:11 
Bob Ippolito22 Oct 2007 05:28 
Bob Ippolito22 Oct 2007 05:33 
Graham Dumpleton22 Oct 2007 16:23 
Graham Dumpleton22 Oct 2007 16:25 
Cliff Wells22 Oct 2007 17:52 
Graham Dumpleton22 Oct 2007 17:59 
Cliff Wells22 Oct 2007 18:53 
Graham Dumpleton22 Oct 2007 22:58 
Marcin Kasperski31 Oct 2007 08:39 
Ben Bangert31 Oct 2007 08:56 
Marcin Kasperski31 Oct 2007 09:38 
Marcin Kasperski31 Oct 2007 09:40 
Ben Bangert31 Oct 2007 12:08 
Graham Dumpleton31 Oct 2007 15:52 
Graham Dumpleton31 Oct 2007 16:44 
Ben Bangert31 Oct 2007 17:04 
Ben Bangert31 Oct 2007 17:08 
Marcin Kasperski05 Nov 2007 03:53 
Marcin Kasperski05 Nov 2007 04:08 
Marcin Kasperski05 Nov 2007 04:33 
Alberto Valverde05 Nov 2007 04:57 
Marcin Kasperski05 Nov 2007 06:03 
Ian Bicking05 Nov 2007 06:58 
Subject:Re: Reducing pylons app memory usage?
From:Ben Bangert (ben-@public.gmane.org)
Date:10/31/2007 05:04:53 PM
List:com.googlegroups.pylons-discuss

On Oct 31, 2007, at 4:44 PM, Graham Dumpleton wrote:

What figures do you get for the above if you run:

ulimit -s 512

in your shell prior to running the web application?

You will need to exit the shell when done to get it back to default value.

If the VPS Memory Limit (virtual memory size) is found to be an issue for some providers of VPS systems, for Python 2.5 at least, the simple solution would be to allow a configuration option something like:

threadpool_workers_stackize = 524288

when this is present, then the web server could call:

import thread thread.stack_size(value)

This way the user could change it if they needed to because of an overly restrictive VPS configuration.

Thanks Graham! I've looked over that Apache thread, the clarifying post was: http://marc.info/?l=apache-httpd-dev&m=119296450928812&w=2

I did some more testing, with base Python, then paster shell (which loads a Pylons app but *no* threads), then paster serve (which has threads). Sure enough, all the virtual consumption seems to be due to the thread size on linux being rather hefty.

On a linux system, Pylons itself consumes about 15mb of Virtual (not too unreasonable I hope?). The massive increase in Virtual is due to "paster serve" spawning 10 threads (at 8mb virtual each), plus one watch thread that monitors the other threads if they get stuck. Since linux is allocating 8mb in the stack per thread, this is where that extra 80mb of Virtual comes from.

You can run Pylons with FastCGI, directly loading the Pylons app (thus no threadpools). This should work great with your nginx setup as well, and mitigate the Virtual issue from the thread size on linux. Your virtual should then hover around 20mb instead of the 100mb you're seeing.

Cheers, Ben