Re: [Xen-devel] IRQ SMP affinity problems in domU with vcpus > 4 on HP ProLiant G6 with dual Xeon 5540 (Nehalem)

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Thu, Oct 15, 2009 at 08:09:42PM -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, Oct 14, 2009 at 01:54:33PM -0600, Cinco, Dante wrote:
> > I switched over to Xen 3.5-unstable (changeset 20303) and pv_ops dom0 
> > 2.6.31.1 hoping that this would resolve the IRQ SMP affinity problem. I had 
> > to use pci-stub to hide the PCI devices since pciback wasn't working. With 
> > vcpus=16 (APIC routing is physical flat), the interrupts were working in 
> > domU and being routed to CPU0 with the default smp_affinity (ffff) but as 
> > soon as I changed it to any 16-bit one-hot value or even setting it to the  
> > same default value resulted in a complete loss of interrupts (even in the 
> > devices that didn't have any change to smp_affinity). With vcpus=4 (APIC 
> > routing is logical flat), I can see the interrupts being load balanced 
> > across all CPUs but as soon as I changed smp_affinity to any value, the 
> > interrupts stopped. This used to work reliably with the non-pv_ops kernel. 
> > I attached the logs in case anyone wants to take a look.
> > 
> > I did see the MSI message address/data change in both domU and dom0 (using 
> > "lspci -vv"):
> > 
> > vcpus=16:
> > 
> > domU MSI message address/data with default smp_affinity: Address: 
> > 00000000fee00000  Data: 40a9
> > domU MSI message address/data after smp_affinity=0010:   Address: 
> > 00000000fee08000  Data: 40b1 (8 is APIC ID of CPU4)
> 
> What does Xen tell you (hit Ctrl-A three times and then 'z'). Specifically 
> look for vector 169 (a9) and 177 (b1).
> Do those values match with what you see in DomU and Dom0? Mainly that 177 has 
> dest_id of 8.
> Oh, and also check the guest interrupt information, to see if those values 
> match..

N/m. I was thinking that maybe your IOAPIC has those vectors programmed in it. 
But
that would not make any sense.

> > 
> > dom0 MSI message address/data with default smp_affinity: Address: 
> > 00000000fee00000  Data: 4094
> > dom0 MSI message address/data after smp_affinity=0010:   Address: 
> > 00000000fee00000  Data: 409c

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel