RE: [Xen-devel] VT-d issue on Xen 3.3.0 [WAS: Xen-3.2.1 VT-dSupp

Ed,

Can you try using the latest BIOS and the tip of xen-unstable?

Joe

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Nadolski, Ed
Sent: Wednesday, September 10, 2008 3:49 PM
To: Nadolski, Ed; Han, Weidong; Kumar, Venkat; Cui, Dexuan; Bryan York
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-devel] VT-d issue on Xen 3.3.0 [WAS: Xen-3.2.1 VT-dSupport (NOT SURE WHETHER IT'S ABUGOR...)]

I have more info: when I flash to an older BIOS, version 0559 (this required the recovery procedure) the VT-d seems to come up OK. The xm dmesg is below.

I’m surprised the old BIOS works better. So be careful about upgrading your BIOS. Any thoughts on what the bug might be? Is there anywhere I can report this to Intel?

Thanks,

(XEN) Xen EXPERIMENTAL TEST BUILD

(XEN) Xen version 3.3.0 (root@) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) Wed Sep 10 14:43:01 MDT 2008

(XEN) Latest ChangeSet: unavailable

(XEN) Xen EXPERIMENTAL TEST BUILD!

(XEN) Command line: vtd=1 iommu=1 acpi=force apic=on

(XEN) Video information:

(XEN) VGA is text mode 80x25, font 8x16

…

(XEN) Processor #0 6:15 APIC version 20

(XEN) Processor #1 6:15 APIC version 20

(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23

(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs

(XEN) [VT-D]dmar.c:446: Host address width 36

(XEN) [VT-D]dmar.c:450: ED_DEBUG: dmar:0xffff828bfff7b000

(XEN) [VT-D]dmar.c:451: ED_DEBUG: entry_header:0xffff828bfff7b030

(XEN) [VT-D]dmar.c:452: ED_DEBUG: table->length:296

(XEN) [VT-D]dmar.c:458: ED_DEBUG: entry_header:0xffff828bfff7b030 type:0x0 length:24

(XEN) [VT-D]dmar.c:463: found ACPI_DMAR_DRHD

(XEN) [VT-D]dmar.c:323: dmaru->address = feb00000

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1b.0

(XEN) [VT-D]dmar.c:458: ED_DEBUG: entry_header:0xffff828bfff7b048 type:0x0 length:32

(XEN) [VT-D]dmar.c:463: found ACPI_DMAR_DRHD

(XEN) [VT-D]dmar.c:323: dmaru->address = feb01000

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:2.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:2.1

(XEN) [VT-D]dmar.c:458: ED_DEBUG: entry_header:0xffff828bfff7b068 type:0x0 length:48

(XEN) [VT-D]dmar.c:463: found ACPI_DMAR_DRHD

(XEN) [VT-D]dmar.c:323: dmaru->address = feb02000

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:3.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:3.1

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:3.2

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:3.3

(XEN) [VT-D]dmar.c:458: ED_DEBUG: entry_header:0xffff828bfff7b098 type:0x0 length:16

(XEN) [VT-D]dmar.c:463: found ACPI_DMAR_DRHD

(XEN) [VT-D]dmar.c:323: dmaru->address = feb03000

(XEN) [VT-D]dmar.c:332: found INCLUDE_ALL

(XEN) [VT-D]dmar.c:458: ED_DEBUG: entry_header:0xffff828bfff7b0a8 type:0x1 length:88

(XEN) [VT-D]dmar.c:467: found ACPI_DMAR_RMRR

(XEN) [VT-D]dmar.c:359: ED_DEBUG: RMRR base:0x00000000000e0000 RMRR end:0x00000000000effff

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.1

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.2

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.7

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.1

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.2

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.7

(XEN) [VT-D]dmar.c:458: ED_DEBUG: entry_header:0xffff828bfff7b100 type:0x1 length:40

(XEN) [VT-D]dmar.c:467: found ACPI_DMAR_RMRR

(XEN) [VT-D]dmar.c:359: ED_DEBUG: RMRR base:0x00000000cf600000 RMRR end:0x00000000cfffffff

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:2.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:2.1

(XEN) Intel VT-d has been enabled

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Nadolski, Ed
Sent: Wednesday, September 10, 2008 1:56 PM
To: Han, Weidong; Kumar, Venkat; Cui, Dexuan; Bryan York
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] VT-d issue on Xen 3.3.0 [WAS: Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S ABUGOR...)]

I am seeing the “RMRR is incorrect” issue with Xen 3.3.0 and CentOS 5.2 on a DQ35 with the latest BIOS (JOQ3510J.86A.0942.2008.0807.1958). It happens with either IGD or PEG. CPU is E6750, main memory is 4Gb. Debug trace is below.

I’m not sure what the RMRR entries should look like. What BIOS revision worked for you? Should I change any other BIOS settings? Current video settings are:

DVMT Mode = DVMT

IGD DVMT Memory = 128Mb

IGD Aperture Size = 256Mb

Are there any other things that would be good to try?

Thanks,

Ed Nadolski

ed.nadolski@xxxxxxx

(XEN) Xen version 3.3.0 (root@) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) Tue Sep 9 17:36:19 MDT 2008

(XEN) Latest ChangeSet: unavailable

(XEN) Xen EXPERIMENTAL TEST BUILD!

(XEN) Command line: vtd=1 iommu=1 acpi=force apic=on

(XEN) Video information:

(XEN) VGA is text mode 80x25, font 8x16

(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds

……

(XEN) Processor #0 6:15 APIC version 20

(XEN) Processor #1 6:15 APIC version 20

(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23

(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs

(XEN) [VT-D]dmar.c:447: Host address width 36

(XEN) [VT-D]dmar.c:456: found ACPI_DMAR_DRHD

(XEN) [VT-D]dmar.c:323: dmaru->address = feb00000

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1b.0

(XEN) [VT-D]dmar.c:456: found ACPI_DMAR_DRHD

(XEN) [VT-D]dmar.c:323: dmaru->address = feb01000

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:2.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:2.1

(XEN) [VT-D]dmar.c:456: found ACPI_DMAR_DRHD

(XEN) [VT-D]dmar.c:323: dmaru->address = feb03000

(XEN) [VT-D]dmar.c:332: found INCLUDE_ALL

(XEN) [VT-D]dmar.c:460: found ACPI_DMAR_RMRR

(XEN) [VT-D]dmar.c:359: ED_DEBUG: RMRR base:0x00000000000e0000 RMRR end:0x00000000000effff

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.1

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.2

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1d.7

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.0

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.1

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.2

(XEN) [VT-D]dmar.c:272: found endpoint: bdf = 0:1a.7

(XEN) [VT-D]dmar.c:460: found ACPI_DMAR_RMRR

(XEN) [VT-D]dmar.c:359: ED_DEBUG: RMRR base:0x0000000000000000 RMRR end:0x0000000000000000

(XEN) [VT-D]dmar.c:363: RMRR is incorrect.

(XEN) Failed to parse ACPI DMAR. Disabling VT-d.

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Han, Weidong
Sent: Wednesday, August 06, 2008 2:59 AM
To: Kumar, Venkat; Cui, Dexuan; Bryan York
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S ABUGOR...)

Venkat,

I think you are using PEG (which means IGD is disabled), right? This is a BIOS. I suggest you use IGD at this moment.

Randy (Weidong)

From: Kumar, Venkat [mailto:Venkat.Kumar@xxxxxxx]
Sent: 2008年8月6日 16:22
To: Cui, Dexuan; Bryan York; Han, Weidong
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S ABUGOR...)

As Dexuan suggested I used xen3.2.2-rc2.

With this version of xen with vtd=1 boot parameter I could boot successfully (I was not able to boot with vtd=1 in xen-3.2.1).

But I see the following in the boot messages when I do a xm dmesg after a boot.

===================================================

(XEN) Xen heap: 14MB (14956kB)

(XEN) Domain heap initialised: DMA width 32 bits

(XEN) Processor #0 6:15 APIC version 20

(XEN) Processor #1 6:15 APIC version 20

(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23

(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs

(XEN) [VT-D]dmar.c:441: RMRR is incorrect.

(XEN) Failed to parse ACPI DMAR. Disabling VT-d.

(XEN) [VT-D]ACPI DMAR:No DMAR devices found

===================================================

If you observe the messages highlighted in RED.

VT-d is getting disabled due to some RMRR incorrect problem and in the dmar.c code this is the condition which is causing this.

if ( rmrr->base_address >= rmrr->end_address )

{

dprintk(XENLOG_ERR VTDPREFIX, "RMRR is incorrect.\n");

return -EFAULT;

}

This code is in acpi_parse_one_rmrr function.

Thx,

Venkat

================================

Venkata Kumar Duvvuru,

LSI Engenio,

Adv. Development,

Bangalore.

Mob: +91-9880318542

Off : +91-80-41978700 ( Extn : 3544 )

================================

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Cui, Dexuan
Sent: Wednesday, August 06, 2008 10:58 AM
To: Bryan York; Han, Weidong
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S A BUGOR...)

Hi Bryan,

According to our tests using the latest xen-unstable, the assignment of SATA controller (like in AHCI mode or Enhanced mode) should be OK on the DQ35 platform.

For xen-unstable, I guess you didn't specify "iommu=1" in the Xen grub entry? By default, VT-d is not enabled in the latest xen-unstable.

Thanks,
-- Dexuan

From: Bryan York [mailto:bryan.york@xxxxxxxxx]
Sent: 2008年8月6日 12:10
To: Han, Weidong
Cc: Cui, Dexuan; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S A BUGOR...)

Hello,

Thanks for the information. I'm trying to forward a PCI express SATA/PATA controller with a JMicron chipset. (Info here.) I know there have been problems with graphics cards in general, but I thought it was possible with disk controllers? Is there a list of well supported hardware (disk controllers) that can be used for VT-d?

Also, your RMRR explanation makes sense for the mercurial 3.2-testing, but what about xen-unstable? I didn't see any errors there besides "I/O Virtualisation disabled" with no other reason listed.

Thanks so much for your help on this. I will also try removing the disk controller and see if the mercurial Xen-3.2-testing boots without RMRR errors.

Regards,

-Bryan

On Tue, Aug 5, 2008 at 6:27 PM, Han, Weidong <weidong.han@xxxxxxxxx> wrote:

Hi Bryan,

I think you are using an add-on graphic card (iGfx disabled), right? We also met this issue before. Obviously it's a BIOS issue (RMRR is incorrect). When parsing apci dmar failed, VT-d will be disabled.

Randy (Weidong)

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Bryan York
Sent: 2008年8月6日 6:17
To: Cui, Dexuan
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx

Subject: RE: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S A BUGOR...)

Hello,

I'm also seeing this exact same problem. I've posted on xen-users, but did not get an answer. Venkat seems to be experiencing the same problem as me. I have the latest BIOS for my motherboard DQ35JO. (BIOS ver 933) I'm running a Q6600 as well. It hangs at "Brought up 4 CPUs." I've tried all sorts of combinations of Xen versions and kernels to get this working. And, yes, VT and VT-d are enabled in my BIOS.

1) CentOS 5.2 SRPMS compiled with Xen 3.2.1. Adding pci=nommconf does not get it to boot.

2) Mercurial Xen-3.2-testing. Here, after adding pci=nommconf, I get:

(XEN) [VT-D]dmar.c:441: RMRR is incorrect.
(XEN) Failed to parse ACPI DMAR. Disabling VT-d.
(XEN) [VT-D]ACPI DMAR:No DMAR devices found

3) Mercurial Xen-unstable, I get "I/O Virtualisation disabled" I can't find any reason why this is the case. VT-D is not enabled.

I would really like to find the cause for this. I don't know if this is a BIOS bug or what. This board is supposedly supported on Xen per the VT-D wiki. Other people seem to have the same problem as me:

http://forums12.itrc.hp.com/service/forums/bizsupport/questionanswer.do?admit=109447627+1217898320366+28353475&threadId=1234588

Thanks for any and all help,

-Bryan

The BIOS version I use may be not the exact one as yours. I mean I expect you're
not using a very old BIOS. :-)

How about trying the xen 3.2.2-rc2 and the latest xen-unstable first? There are some VT-d bug fixes between 3.2.1 (you're using it) and 3.2.2-rc2.

Thanks,

-- Dexuan

________________________________

From: Kumar, Venkat [mailto:Venkat.Kumar@xxxxxxx] Sent: 2008年8月5日 14:37 To: Cui, Dexuan; xen-devel@xxxxxxxxxxxxxxxxxxx Subject: RE: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S A BUG
OR...)

If you don't mind can you send me the BIOS version you are using?

Thx,

Venkat

================================

Venkata Kumar Duvvuru,

LSI Engenio,

Adv. Development,

Bangalore.

Mob: +91-9880318542

Off : +91-80-41978700 ( Extn : 3544 )

================================

________________________________

From: Cui, Dexuan [mailto:dexuan.cui@xxxxxxxxx] Sent: Tuesday, August 05, 2008 12:03 PM To: Kumar, Venkat; xen-devel@xxxxxxxxxxxxxxxxxxx Subject: RE: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S A BUG
OR...)

Hi Venkat,

Can you confirm you're using the latest BIOS? Can you try the xen 3.2.2-rc2 and
the latest xen-unstable on the same host?

I have the similar host. Xen 3.2.2-rc2 and the latest xen-unstable both work
well on it.

Thanks,

-- Dexuan

________________________________

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
[mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Kumar, Venkat Sent: 2008年8月5日 12:21 To: xen-devel@xxxxxxxxxxxxxxxxxxx Subject: [Xen-devel] Xen-3.2.1 VT-d Support (NOT SURE WHETHER IT'S A BUG OR...)

After building Xen-3.2.1 I could successfully boot into it if I avoid giving
vtd=1 as the boot parameter.

If I pass vtd=1 as the boot parameter to xen, the system hangs while booting.

The system hangs after the following messages

============================

Boot messages

[XEN]…..

[Xen] Brought up 2 CPUs...

=========================

My chipset DQ35 series is having VT-d support and is enabled in the BIOS as
well.

My domain0 is CentOS-5.2(2.6.18.8-xen).

Is this a possible bug or some thing else??

Any Idea??

Thx,

Venkat

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

WARNING - OLD ARCHIVES

xen-devel

RE: [Xen-devel] VT-d issue on Xen 3.3.0 [WAS: Xen-3.2.1 VT-dSupport (NOT