atom feed122 messages in com.xensource.lists.xen-develRe: [Xen-devel] Re: APIC rework
FromSent OnAttachments
56 earlier messages
Konrad Rzeszutek WilkOct 21, 2009 11:31 am 
Pasi KärkkäinenOct 21, 2009 11:51 am 
Jeremy FitzhardingeOct 21, 2009 12:49 pm 
Pasi KärkkäinenOct 21, 2009 1:21 pm 
Pasi KärkkäinenOct 27, 2009 8:46 am 
Konrad Rzeszutek WilkOct 27, 2009 9:59 am.makefile, .c
Pasi KärkkäinenOct 27, 2009 10:29 am 
Konrad Rzeszutek WilkOct 27, 2009 12:40 pm 
Pasi KärkkäinenOct 27, 2009 12:45 pm 
Konrad Rzeszutek WilkOct 27, 2009 1:12 pm 
Pasi KärkkäinenOct 27, 2009 1:17 pm 
Pasi KärkkäinenOct 27, 2009 1:23 pm 
Pasi KärkkäinenOct 27, 2009 1:35 pm 
Jeremy FitzhardingeNov 11, 2009 4:46 pm 
Jeremy FitzhardingeNov 11, 2009 4:59 pm 
Jeremy FitzhardingeNov 12, 2009 3:50 pm 
Zhang, XiantaoNov 12, 2009 9:26 pm 
Keir FraserNov 12, 2009 11:24 pm 
Jeremy FitzhardingeNov 13, 2009 3:56 pm 
Keir FraserNov 14, 2009 12:04 am 
Zhang, XiantaoNov 16, 2009 2:37 am.patch, .patch
Jeremy FitzhardingeNov 16, 2009 10:37 am 
Zhang, XiantaoNov 16, 2009 7:12 pm 
Keir FraserNov 16, 2009 7:44 pm 
Jeremy FitzhardingeNov 16, 2009 9:12 pm 
Jeremy FitzhardingeNov 16, 2009 9:19 pm 
Keir FraserNov 16, 2009 9:43 pm 
Zhang, XiantaoNov 17, 2009 4:45 am.patch
Keir FraserNov 17, 2009 5:04 am 
Zhang, XiantaoNov 17, 2009 6:16 am 
Jeremy FitzhardingeNov 17, 2009 10:50 am 
Keir FraserNov 17, 2009 11:49 am 
Jiang, YunhongNov 17, 2009 7:11 pm 
Zhang, XiantaoNov 17, 2009 7:24 pm 
Zhang, XiantaoNov 17, 2009 7:37 pm 
Keir FraserNov 18, 2009 1:36 am 
Konrad Rzeszutek WilkNov 18, 2009 6:14 am 
Konrad Rzeszutek WilkNov 18, 2009 6:29 am 
Zhang, XiantaoNov 19, 2009 5:45 pm 
Zhang, XiantaoNov 19, 2009 5:47 pm 
Zhang, XiantaoNov 24, 2009 2:04 am.patch, .patch
Jeremy FitzhardingeNov 24, 2009 11:25 am 
Konrad Rzeszutek WilkNov 24, 2009 11:43 am 
Jeremy FitzhardingeNov 24, 2009 3:34 pm 
Zhang, XiantaoNov 24, 2009 5:41 pm 
Zhang, XiantaoNov 24, 2009 6:43 pm 
Konrad Rzeszutek WilkNov 25, 2009 5:41 am 
Konrad Rzeszutek WilkNov 25, 2009 6:09 am 
Zhang, XiantaoNov 25, 2009 7:21 am 
Konrad Rzeszutek WilkNov 25, 2009 10:00 am 
Jeremy FitzhardingeNov 25, 2009 10:58 am 
Jeremy FitzhardingeNov 25, 2009 11:13 am 
Zhang, XiantaoNov 25, 2009 5:11 pm 
Zhang, XiantaoNov 26, 2009 3:52 am 
Konrad Rzeszutek WilkNov 30, 2009 6:26 am 
Konrad Rzeszutek WilkNov 30, 2009 6:33 am 
Zhang, XiantaoDec 2, 2009 6:13 pm 
Konrad Rzeszutek WilkDec 3, 2009 6:37 am 
Stefan KuhneDec 4, 2009 8:07 am 
Pasi KärkkäinenDec 4, 2009 10:57 am 
Jeremy FitzhardingeDec 4, 2009 11:26 am 
Pasi KärkkäinenJan 1, 2010 9:20 am 
Konrad Rzeszutek WilkJan 4, 2010 5:37 am 
Pasi KärkkäinenJan 4, 2010 11:41 am 
Konrad Rzeszutek WilkJan 14, 2010 12:05 pm 
Pasi KärkkäinenJan 14, 2010 11:18 pm 
Subject:Re: [Xen-devel] Re: APIC rework
From:Konrad Rzeszutek Wilk (konr@oracle.com)
Date:Nov 25, 2009 10:00:08 am
List:com.xensource.lists.xen-devel

On Wed, Nov 25, 2009 at 11:21:45PM +0800, Zhang, Xiantao wrote:

Konrad Rzeszutek Wilk wrote:

On Wed, Nov 25, 2009 at 10:43:47AM +0800, Zhang, Xiantao wrote:

Konrad Rzeszutek Wilk wrote:

At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think. And I also agree to add a new physdev_op to handle this case, and it should be better way to go. Based on this idea, I worked out the patch, attached! In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed. BTW, I also tested apic and non-apic cases, they works fine after applying the patches.

But I don't think you tested PCI front and PCI back.

Mainly these lines worry me (can you inline the patch next time too, please):

+ map_irq.domid = DOMID_SELF; + map_irq.type = MAP_PIRQ_TYPE_GSI; + map_irq.index = gsi; + map_irq.pirq = irq; + rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);

For PCI passthrough to work, the domid needs to be for the guest domain, while in this case it is set to Dom0. There is already a method of extracting the domain id for PCI devices passed to the guest. Look in the 'xen_create_msi_irq' function.

Could you detail the concern ? This hypercall is only related to GSI, not MSI, why it has side-effect about pci passthrough ?

This is for PV guests _without_ using QEMU. They are using the PCI backend to "enable" a device (drivers/xen/pciback and drivers/pci/xen-pcifront.c). The front and back-end communicate the IRQ number (GSI) to the guest when enabling a INTx PCI device (not MSI ones).

Then the PV guest can bind the IRQ (GSI) number to its own event channel and have fully working PCI device.

With your change, the privileged domain pins the device to itself, not to other domains.

But I think dom0 should own the device first during boot, and then assign it to
PV guest when this device is required by pcifront? Basically, we don't know
which devices should be reserved for non-previleged domains, right ? So I think
the GSI should be initialized and bind to dom0 when dom0 boots? Once the
devices is assigned to PV guests, it maybe need to do the unmapping operation
about the GSI from dom0 and do mapping for the domU.

During boot the device can be owned by pciback or by the modele for which a PCI entry exist. Look for pciback.hide entry.

There are two modes of execution: 1). First being what you described wherein the device initially belogs to Dom0. The user unbinds it from the PCI device and binds it to the pciback module. At that point, the device is disabled and ready for PV guests. When a PV
guest starts pciback module makes the pci_enable_device call and sets the IRQ,
etc for the device (for MSI, it obviously gets the IDT value from the
hypervisor call). 2). Dom0 boots where the user specified on the command line
pciback.hide=(0000:01:03.1). The pciback owns the device (is binded to it) and the native module that
would load for this PCI device is not called.

It is correct that the unmapping/mapping and the ownership needs to be dynamic.
As user could bind to the pciback module, give it to guest, kill the guest, then map the
PCI device back to dom0, and after that repeat the whole thing.

BTW, I just met a strange issue with the function xen_create_msi_irq you
mentioned, and it blocks initialization of SR-IOV devices' VFs , and I think it
should be a bug.

Hmm.. I am not sure if this is the appropiate place for it. You see this driver is designed for the machines that don't have SR-IOV, VFs, or VT-d to be able to passthrough a PCI device to the PV guest. You can use this to run PV guests on Pentium 4 style machine.

I think that the SR-IOV devices would go through a different call stack to enable the device? Either way, I recently got my hands on a SR-IOV machine and will see how this works.

Has the 2.6.31.5 been working before with SR-IOV cards or is this your first experiment with this.

In the function xen_create_msi_irq, there is one line as following to get the
domid of the specified device, but a strange domid(0xfff0) is returned by this
call, could you help to check whether this strange domid is from? domid = rc = xenbus_walk( "/local/domain/0", get_domid_for_dev, dev);

That is due to casting (domid is short int, rc is int). You need to check if rc is negative, and if so set the domid to DOMID_SELF. The next line below does this:

if (domid <= 0) domid = DOMID_SELF;

Ohh wait, looks like the fix for this never got pushed upstream. That should have been 'if (rc <= 0)'. Yikes.

rc is -16 (EBUSY). That implies that xenstored has not been initialized on Dom0.

^^^^^^^ Xiantao