14 messages in com.googlegroups.pylons-discussRe: Multiple Pylons instances, proces...
FromSent OnAttachments
Devin Torres23 Apr 2008 10:51 
Ian Bicking23 Apr 2008 10:56 
Devin Torres23 Apr 2008 11:05 
Ian Bicking23 Apr 2008 11:19 
climbus23 Apr 2008 11:42 
Cliff Wells23 Apr 2008 11:53 
Christopher Weimann23 Apr 2008 13:20 
Devin Torres23 Apr 2008 14:28 
Graham Dumpleton23 Apr 2008 16:59 
Christopher Weimann23 Apr 2008 18:04 
Cliff Wells23 Apr 2008 18:36 
Marcin Kasperski24 Apr 2008 02:07 
Devin Torres24 Apr 2008 11:59 
Graham Dumpleton26 Apr 2008 03:33 
Subject:Re: Multiple Pylons instances, processor affinity, and "threads"
From:Ian Bicking (ia@colorstudy.com)
Date:04/23/2008 10:56:33 AM
List:com.googlegroups.pylons-discuss

Devin Torres wrote:

So we're using Pylons and Python in general for our new company platform. We just bought a server with 4 cores to help us reach our scalability goals, but there are a few questions I'm interested in asking the Pylons community.

I (mostly) understand the nature of "threads" in Python. From my understanding, the GIL locks the interpreter to executing only one Python thread at a time, but C modules can take advantage of a Python application being multithreaded, because they can operate independant of the GIL. Presumably, this would mean that there is, in fact, a benefit to using threads in Paste, because most network I/O bound stuff happens within a C module.

Given this situation, I believe that despite paste making an effort to be multithreaded, it would still be advantageous to run a cluster of four Pylons instances and proxy to these using nginx.

Separate processes is likely to work better. You might find one of the flup forking servers to be better (using fastcgi), though I don't know for sure. That will run each request in its own process, so you'll get multiple processes without the same infrastructure complications of a cluster of servers.

I don't think affinity should be that important. Doesn't the OS handle that itself?