atom feed33 messages in org.freebsd.freebsd-currentRe: dev.bce.X.com_no_buffers increasi...
FromSent OnAttachments
Ian FREISLICHMar 5, 2010 3:20 am 
Pyun YongHyeonMar 5, 2010 9:56 am 
Ian FREISLICHMar 5, 2010 10:16 am 
Pyun YongHyeonMar 5, 2010 10:40 am 
Ian FREISLICHMar 5, 2010 12:19 pm 
Pyun YongHyeonMar 5, 2010 1:04 pm 
Ian FREISLICHMar 5, 2010 1:16 pm 
Pyun YongHyeonMar 5, 2010 1:55 pm 
Ian FREISLICHMar 8, 2010 6:44 am 
Pyun YongHyeonMar 8, 2010 9:49 am 
Ian FREISLICHMar 9, 2010 12:26 am 
Ian FREISLICHMar 9, 2010 5:31 am 
Pyun YongHyeonMar 9, 2010 12:49 pm 
Pyun YongHyeonMar 9, 2010 1:21 pm 
David ChristensenMar 9, 2010 1:31 pm 
Pyun YongHyeonMar 9, 2010 1:39 pm 
Ian FREISLICHMar 9, 2010 1:55 pm 
David ChristensenMar 9, 2010 2:04 pm 
Pyun YongHyeonMar 9, 2010 2:12 pm 
Ryan StoneMar 9, 2010 2:30 pm 
Fabien ThomasMar 9, 2010 2:55 pm 
David ChristensenMar 9, 2010 3:00 pm 
Ryan StoneMar 9, 2010 3:07 pm 
Ian FREISLICHMar 9, 2010 9:47 pm 
Ian FREISLICHMar 10, 2010 1:04 am 
David ChristensenMar 10, 2010 11:10 am 
Pyun YongHyeonMar 10, 2010 11:51 am 
David ChristensenMar 10, 2010 2:45 pm 
Pyun YongHyeonMar 10, 2010 3:01 pm 
Ian FREISLICHMar 10, 2010 10:45 pm 
Ian FREISLICHMar 10, 2010 11:05 pm 
David ChristensenMar 12, 2010 3:58 pm 
Ian FREISLICHMar 13, 2010 9:05 am 
Subject:Re: dev.bce.X.com_no_buffers increasing and packet loss
From:Pyun YongHyeon (pyu@gmail.com)
Date:Mar 10, 2010 3:01:57 pm
List:org.freebsd.freebsd-current

On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote:

I successfully reproduced the issue with netperf on BCM5709. You can use UDP frame size 1 to trigger the issue.

Changing the high level design of bce_rx_intr() and bce_rx_fill_chain() slightly to post a new buffer as each frame is passed to the OS would likely avoid these gaps during bursts of small frames but I'm not sure whether they'll have a negative impact on the more common case of streams of MTU sized frames. I've considered this in the past but never coded the change and tested the resulting performance.

I guess this may slightly increase performance with additional bus_dma(9) overheads but I think one of reason of dropping frames under heavy UDP frames may come from lack of free RX descriptors. Because bce(4) just uses a single RX ring so the number of available RX buffers would be 512. However it seems it's not possible to increase the number of RX buffers per RX ring so the next possible approach would be switching to use multiple RX rings with RSS. Even though FreeBSD does not dynamically adjust loads among CPUs I guess using RSS would be the way to go.

The bce(4) hardware supports a linked list of pages for RX buffer descriptors. The stock build supports 2 pages (RX_PAGES) with a total of 511 BD's per page. The hardware can support a maximum of 64K BD's but that would be an unnecessarily large amount of mbufs for an infrequent problem.

Thanks for the info. I guess 2048 or 4096 BDs would be necessary to get satisfactory Rx performance. I'll have to experiment this.

The middle road would probably involve changing RX_PAGES from a #define to a sysctl variable to allow tuning for specific environments along with a change in bce_rx_intr() to fill the ring after all frames have been processed or when more than 256 BDs have been consumed, whichever comes first.

RSS would be great as well though it wouldn't make a dent in this case since RSS is only supported for TCP, not UDP.

Even though UDP is not supported in RSS, RSS can handle IP. This wouldn't distribute UDP load coming from a single host but if source IP address is different it may help, I guess.

Dave

_______________________________________________ free@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "free@freebsd.org"