Re: Multiple Pylons instances, processor affinity, and "threads" - Cliff Wells - com.googlegroups.pylons-discuss

14 messages in com.googlegroups.pylons-discussRe: Multiple Pylons instances, proces...

From	Sent On	Attachments
Devin Torres	23 Apr 2008 10:51
Ian Bicking	23 Apr 2008 10:56
Devin Torres	23 Apr 2008 11:05
Ian Bicking	23 Apr 2008 11:19
climbus	23 Apr 2008 11:42
Cliff Wells	23 Apr 2008 11:53
Christopher Weimann	23 Apr 2008 13:20
Devin Torres	23 Apr 2008 14:28
Graham Dumpleton	23 Apr 2008 16:59
Christopher Weimann	23 Apr 2008 18:04
Cliff Wells	23 Apr 2008 18:36
Marcin Kasperski	24 Apr 2008 02:07
Devin Torres	24 Apr 2008 11:59
Graham Dumpleton	26 Apr 2008 03:33

Subject:	Re: Multiple Pylons instances, processor affinity, and "threads"
From:	Cliff Wells (cli...@develix.com)
Date:	04/23/2008 11:53:33 AM
List:	com.googlegroups.pylons-discuss

On Wed, 2008-04-23 at 12:51 -0500, Devin Torres wrote:

So we're using Pylons and Python in general for our new company platform.

I'm reading this as meaning intranet, not internet?

Given this situation, I believe that despite paste making an effort to be multithreaded, it would still be advantageous to run a cluster of four Pylons instances and proxy to these using nginx.

Using our setup we'd have four pylons instances being proxied to by four nginx worker threads.

Unless you are expecting hundreds of req/s or are serving large static files, I'd suggest just one or maybe two nginx workers. Nginx will not be the bottleneck in this situation. One Nginx worker can easily handle proxying four Pylons apps (or a hundred, for that matter).

In nginx you can set the processor affinity for each worker thread, thus placing each worker on a different core 0..3.

Not true on Linux. This has been broken for some time.

Here's where things get tricky: I've found a Python package that apparently allows Python applications to set their processor affinity (I'm afraid it doesn't work on OS X): http://pypi.python.org/pypi/affinity/0.1.0

Using this, what do you guys thing on my idea to write a custom cluster controller, perhaps using supervisord, that will start nginx and the four worker processes, and then fork()'s my Pylons app into into a cluster of four?

Is this overkill? Is Paste more mulithreaded than I'm giving it credit for? Is there a better way to go about this? Does an alternative to the 'affinity' package exist?

I think it's overkill, but not for the reasons you seem to think. Much easier is to simply run four Pylons processes from the command line, each with a custom .ini file. Just use a shell script.

You can also set the CPU affinity for Pylons (or more specifically, Python) from the command line using a small C program (and I'm sure there are pre-written utilities or recipes you could follow).

You'll want to follow a shared-nothing approach or use something like Memcached to share data between processes. It's probably also possible to use Memcached as a secondary cache for SQLAlchemy (although I haven't tried it). There's a thread about someone doing this here:

http://www.mail-archive.com/sqlalchemy-users...@lists.sourceforge.net/msg02499.html

Regards, Cliff