|Ian FREISLICH||Mar 5, 2010 3:20 am|
|Pyun YongHyeon||Mar 5, 2010 9:56 am|
|Ian FREISLICH||Mar 5, 2010 10:16 am|
|Pyun YongHyeon||Mar 5, 2010 10:40 am|
|Ian FREISLICH||Mar 5, 2010 12:19 pm|
|Pyun YongHyeon||Mar 5, 2010 1:04 pm|
|Ian FREISLICH||Mar 5, 2010 1:16 pm|
|Pyun YongHyeon||Mar 5, 2010 1:55 pm|
|Ian FREISLICH||Mar 8, 2010 6:44 am|
|Pyun YongHyeon||Mar 8, 2010 9:49 am|
|Ian FREISLICH||Mar 9, 2010 12:26 am|
|Ian FREISLICH||Mar 9, 2010 5:31 am|
|Pyun YongHyeon||Mar 9, 2010 12:49 pm|
|Pyun YongHyeon||Mar 9, 2010 1:21 pm|
|David Christensen||Mar 9, 2010 1:31 pm|
|Pyun YongHyeon||Mar 9, 2010 1:39 pm|
|Ian FREISLICH||Mar 9, 2010 1:55 pm|
|David Christensen||Mar 9, 2010 2:04 pm|
|Pyun YongHyeon||Mar 9, 2010 2:12 pm|
|Ryan Stone||Mar 9, 2010 2:30 pm|
|Fabien Thomas||Mar 9, 2010 2:55 pm|
|David Christensen||Mar 9, 2010 3:00 pm|
|Ryan Stone||Mar 9, 2010 3:07 pm|
|Ian FREISLICH||Mar 9, 2010 9:47 pm|
|Ian FREISLICH||Mar 10, 2010 1:04 am|
|David Christensen||Mar 10, 2010 11:10 am|
|Pyun YongHyeon||Mar 10, 2010 11:51 am|
|David Christensen||Mar 10, 2010 2:45 pm|
|Pyun YongHyeon||Mar 10, 2010 3:01 pm|
|Ian FREISLICH||Mar 10, 2010 10:45 pm|
|Ian FREISLICH||Mar 10, 2010 11:05 pm|
|David Christensen||Mar 12, 2010 3:58 pm|
|Ian FREISLICH||Mar 13, 2010 9:05 am|
|Subject:||Re: dev.bce.X.com_no_buffers increasing and packet loss|
|From:||Pyun YongHyeon (pyu...@gmail.com)|
|Date:||Mar 10, 2010 3:01:57 pm|
On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote:
I successfully reproduced the issue with netperf on BCM5709. You can use UDP frame size 1 to trigger the issue.
Changing the high level design of bce_rx_intr() and bce_rx_fill_chain() slightly to post a new buffer as each frame is passed to the OS would likely avoid these gaps during bursts of small frames but I'm not sure whether they'll have a negative impact on the more common case of streams of MTU sized frames. I've considered this in the past but never coded the change and tested the resulting performance.
I guess this may slightly increase performance with additional bus_dma(9) overheads but I think one of reason of dropping frames under heavy UDP frames may come from lack of free RX descriptors. Because bce(4) just uses a single RX ring so the number of available RX buffers would be 512. However it seems it's not possible to increase the number of RX buffers per RX ring so the next possible approach would be switching to use multiple RX rings with RSS. Even though FreeBSD does not dynamically adjust loads among CPUs I guess using RSS would be the way to go.
The bce(4) hardware supports a linked list of pages for RX buffer descriptors. The stock build supports 2 pages (RX_PAGES) with a total of 511 BD's per page. The hardware can support a maximum of 64K BD's but that would be an unnecessarily large amount of mbufs for an infrequent problem.
Thanks for the info. I guess 2048 or 4096 BDs would be necessary to get satisfactory Rx performance. I'll have to experiment this.
The middle road would probably involve changing RX_PAGES from a #define to a sysctl variable to allow tuning for specific environments along with a change in bce_rx_intr() to fill the ring after all frames have been processed or when more than 256 BDs have been consumed, whichever comes first.
RSS would be great as well though it wouldn't make a dent in this case since RSS is only supported for TCP, not UDP.
Even though UDP is not supported in RSS, RSS can handle IP. This wouldn't distribute UDP load coming from a single host but if source IP address is different it may help, I guess.
_______________________________________________ free...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "free...@freebsd.org"