This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] IOMMU Interrupt Remapping query

To: Keir Fraser <keir.xen@xxxxxxxxx>
Subject: Re: [Xen-devel] IOMMU Interrupt Remapping query
From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
Date: Mon, 6 Jun 2011 17:37:40 +0100
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, "weidong.han@xxxxxxxxx" <weidong.han@xxxxxxxxx>
Delivery-date: Mon, 06 Jun 2011 09:38:48 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <CA12AEFF.1BAA1%keir.xen@xxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <CA12AEFF.1BAA1%keir.xen@xxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20110424 Lightning/1.0b2 Thunderbird/3.1.10

On 06/06/11 16:21, Keir Fraser wrote:
> On 06/06/2011 15:32, "Andrew Cooper" <andrew.cooper3@xxxxxxxxxx> wrote:
>> I am attempting to fix the kexec interactions with x2apic and iommu
>> functionality.  Part of this involves ensuring that all IOMMU
>> functionality is disabled, as the kdump kernels are not happy at having
>> their interrupts remapped without their knowledge.
>> I have introduced iommu_disable_x2apic_IR() onto the kexec path, but it
>> does not seem to actually disable interrupt remapping on Intel boxes
>> (Specifically the two Intel Nehalem boxes I am testing on).
>> Specifying iommu=no-intremap on the commandline causes everything to
>> work correctly, but leaving it out causes the kdump kernel to hang and
>> eventually reboot, as can be seen on the attached serial log.
>> The lines starting DBG: are extra debugging I have put in which shows
>> that the disable_IR() function is being called and writing to the registers.
> Should have attached your patch as well. Noone else can know with certainty
> where you put your debugging, and noone else is going to want to help debug
> your code if they can't even see it. :-)
> Also a good idea to Cc a likely person who can help (i.e., someone who wrote
> the code that you are querying). 'hg annotate' is useful for this -- in this
> case I am adding Weidong Han to the cc list.
> On the bright side, this must have been got working for S3 suspend/resume to
> work properly (indeed that's what the disable code was originally added
> for). So it can't be an insurmountable problem.
>  -- Keir
>> This problem occurs with the XenServer version of 4.1.0 as well as on
>> xen-unstable at the moment.
>> Is there any hardware state which is not taken down by the disable
>> function, any subtle interactions which I have not taken account of?  I
>> have looked through the source and nothing pops out, but I am out of ideas.
>> Thanks in advance,
Attached are the two relevant patches, and two which I don't think are
relevant but might be if I am wrong.  crash_shutdown was an attempt to
make an iommu_ops which shut down all iommu functionality without saving
state.  debug-wip shows where I have put in debug statements.

kdump-fix-x2apic and apic-record-boot-mode are also in the source, but I
believe them to be unrelated to the current problem.

I have done some further debugging on the assumption that the order of
shutting down interupt remapping matters with shutting down the lapics
and ioapics, but disable_qinval causes a panic (qinval.c:222 - "queue
invalidate wait descriptor was not executed\n") if it is run before both
the lapics and ioapics are shut down.

Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Attachment: crash_shutdown_iommu_ops.patch
Description: Text Data

Attachment: debug-wip
Description: Text document

Attachment: apic-record-boot-mode.patch
Description: Text Data

Attachment: kexec-fix-x2apic.patch
Description: Text Data

Xen-devel mailing list