http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1075

Summary: Dom0 crashes with 40Mbps Iperf traffic eithout saturating the CPU Product: Xen Version: 3.0.0 Platform: x86 OS/Version: Linux Status: NEW Severity: critical Priority: P2 Component: Hypervisor AssignedTo: xen-...@lists.xensource.com ReportedBy: kaus...@cc.gatech.edu

Hello,

I have a 3 node experiment as detailed below to estimate the bridged networking performance in para-virtualized Xen 3.0 on Emulab (https://www.emulab.net/)

Here is how the 3-node topology looks (This topology is specified by means of an NS-2 file)

topology:

_______________________________ | | Node0:eth0 -----|Node1:eth0 Node1:eth1|------ Node2:eth0 |_______________________________|

Node0 and Node2 run some standard 2.6.* kernel whereas Node1 runs para-virtualized Xen 3.0 using LVM-created root & swap partitions for DomU (2.6.12-xenU)

******************************************************************* Traffic flow:

____________________________________ | Node1 | | ______________________________ | | | DomU | | | | | | | | eth0 ---------> eth1 | | | |_____|_______________|________| | | | | | | vif1.0 vif1.1 | | | | | | | | | | xenbr0 xenbr1 | | | Dom0 | | | | | | Node0:eth0-|----> eth0 eth1 ---------|-----> Node2:eth0 |____________________________________|

Node0:eth0, Node1:DomU:eth0 are in 10.1.1/24 subnet whereas Node1:DomU:eth1 and Node2:eth0 are in 10.1.2/24 subnet (Since Node1:eth0 and Node1:eth1 are attached to bridges, they do not have IP addresses on them and are not ping'able)

At Node0:eth0, I set the next hop of all traffic as Node1:DomU:eth0 and at Node2, I set the next hop of all traffic as Node1:DomU:eth1.

Now, I try to run an Iperf server at Node2 for UDP traffic (const 64-byte packets size): iperf -s -i 2 -l 64 -u

and an Iperf client at Node0 for UDP traffic to node2:eth0 (const 64-bit packet size, vary the Bandwidth from 10Mbps till possibly 80Mbps) iperf -c <IP address of Node2:eth0> -u -l 64 -t 90 -i 2 -w 256k -b 10M

Where 10M is the bandwidth of iperf traffic between Node0 and Node2 (in Mbps).

I also monitor the CPU usage at Node1 Dom0 using "vmstat 1" at Dom0.

When B/W is 36.6M (Mbps), the CPU utilization at Dom0 is about 70-80% but when the B/W is increased to 39.4M (Mbps), the Dom0 crashes ('hangs') in typically 5-15 seconds but before it crashes, the CPU utiliation is till 75-85%. At this point, I have to again boot Dom0 and the DomU on top of it.

Does anybody know why this happens ?? and how can I debug this situation (if at all) ? I am new to the Xen source code so please be as detailed.

Thanks, Kaushik