I don't think it would matter that much, especially as the reason for
load balancing is probably that you are serving not static pages,
which usually need some processing time anyway.
What would be interesting though is if you could add a max
connections per IP address in the upstream server section.