Re: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, wi

Dell E6400, BIOS version: A12

I hope this issue will be solved by the BIOS people, since i think i saw the same issue with my Lenovo T400, and it looks like you *must* do a cold boot, on both Lenovo & Dell.
Please update me on this issue, if you'll have any news.

Thanks for your help!

2009/6/30 Cui, Dexuan <dexuan.cui@xxxxxxxxx>

Tim,

Can you tell me the BIOS version of your Dell E6400 host?

BTW, please try to use the latest BIOS if possible, and please check your BIOS document to see if there is any known IOMMU-related issue.

Thanks,
-- Dexuan

From: Cui, Dexuan
Sent: 2009年6月30日 10:55
To: 'Tom Rotenberg'

Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

> CAP_REG=0xffffffffffffffff

HW obviously doesn't report the value correctly.

So, looks the issue is: when we change the iommu option in BIOS (e.g., from disabled to enabled), to make it really take effect, we have to make a "cold shutdown (shutdown the power supply throughly)" to the host ; or else, though Xen can see the ACPI DMAR tables，the HW doesn't respond to the MMIO accesses to the iommu registers at all -- such read access would return -1 (all-'1's) and write access would be ignored.

I remember you said after you commented the clear_fault_bits(), the host could boot fine with iommu=1 --- I guess in this case actually the DMA remapping is not enabled at all though Xen thinks HW tells it DMA remapping is enabled (Xen writes DMA_GCMD_TE into GCMD (this write is ignored)  and polls the GSTS; since reading GSTS returns -1, Xen thinks TE is on, however the underlying HW doesn't enable DMA remapping!).

Please note: this is only my deduction, anyway. :-)    You can try to create some hvm guests with devices assigned, I think the host would die weirdly since the DMAs of the assigned devices of the hvm guests are corrupting the host memory! This can be a way to verify my deduction.

And I remember you said after you commented the clear_fault_bits() and booted the host fine, you tried to uncomment the line and the host still boots fine! -- this is strange to me, I guess you might have done a cold shutdown before your uncommenting the line?

One more thing: I think your HW doesn't support SnoopControl/QueuedInvalidation/InterruptRemapping, but after you commented the clear_fault_bits(), LOOKS the host could boot fine even after Xen tries to enable them! Maybe the specific HW implementation of your host could account for this...

I think changing the iommu option in BIOS should not require a cold shutdown to the host. I'll try to raise this issue to the BIOS people.

For now, you could work around the issue by remember doing a cold shutdown every time you change the iommu BIOS option. Hope this is acceptable to you for now. :-)

Thanks,
-- Dexuan

From: Tom Rotenberg [mailto:tom.rotenberg@xxxxxxxxx]
Sent: 2009年6月29日 22:15

To: Cui, Dexuan
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

in steps 1 & 4, i mean disabling and enabling IOMMU in the BIOS (and not in Xen boot parameter). And, this is what probably caused all of this.

I am using my own version (above Xen 3.4.0), but i haven't made any changes in the hyervisor itself (but rather in the hvmloader and rombios), so it's not related to this. I prefer not to try the latest Xen, unless there is a very good reason for this, as it take long time to compile, and i can't do it right now.

I have reproduced the panic with the print, and here is the relevant part of the log:

(XEN) clear_fault_bits: CAP_REG=0xffffffffffffffff
(XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff828c8012cac6>] clear_fault_bits+0x46/0xd0
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
(XEN) rax: ffff828bfff5aff8   rbx: ffff830078f82f90   rcx: 0000000000000000
(XEN) rdx: 00000000ffffffff   rsi: ffff828bfff57000   rdi: ffff828c801f0564
(XEN) rbp: 0000000000000090   rsp: ffff828c8026fd98   r8: 0000000000004000
(XEN) r9: 0000000000003fff   r10: ffff828c80288360   r11: 0000000080287ea0
(XEN) r12: 0000000000000090   r13: 0000000000000090   r14: ffff830078f82fcc
(XEN) r15: 0000000000004800   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000007be74000   cr2: ffff828bfff5aff8
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff828c8026fd98:
(XEN)    ffff830078f82f90 ffff828c8012d35d ffff830078f824b0 ffff828c801e4257
(XEN)    ffff828c80204490 ffff828c801e4257 ffff828c802044c0 ffff828c8026fdf8
(XEN)    0000000000000000 ffff828c8012deae ffff830078f82d40 ffff828c80228100
(XEN)    000000000000001f 00000000ffffffed ffff828c80227a78 0000000000000001
(XEN)    ffff828c80205f60 ffff828c801f3400 ffff83000008bf30 ffff828c8012a9d4
(XEN)    ffff828c80227978 ffff828c8021a589 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 00000000002b23c0 ffffff8c80274000 0008bf3000000000
(XEN)    000000000008bf30 0000000000000000 ffff83000008bfc0 0000000001ce08a0
(XEN)    0000000000000000 ffffffff00000000 0000000800000000 000000010000006e
(XEN)    0000000000000003 00000000000002f8 0000000000000000 0000000000000000
(XEN)    0000000000067e9c 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffff828c801000b5 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffff830079f18000
(XEN) Xen call trace:
(XEN)    [<ffff828c8012cac6>] clear_fault_bits+0x46/0xd0
(XEN)    [<ffff828c8012d35d>] init_vtd_hw+0x15d/0x3e0
(XEN)    [<ffff828c8012deae>] intel_vtd_setup+0x5ae/0x610
(XEN)    [<ffff828c8012a9d4>] iommu_setup+0x24/0x110
(XEN)    [<ffff828c8021a589>] __start_xen+0x1859/0x27c0
(XEN)    [<ffff828c801000b5>] __high_start+0xa1/0xa3
(XEN)
(XEN) Pagetable walk from ffff828bfff5aff8:
(XEN) L4[0x105] = 000000007be7a027 5555555555555555
(XEN) L3[0x02f] = 000000007c04b063 5555555555555555
(XEN) L2[0x1ff] = 000000007c04a063 5555555555555555
(XEN) L1[0x15a] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: ffff828bfff5aff8
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

2009/6/29 Cui, Dexuan <dexuan.cui@xxxxxxxxx>

When you meet with the panic, can you apply the printk log to print CAP_REG?

In your step 1 and 4, you just mean using the xen parameter iommu=0 and iommu=1, correct?

During theses steps, you don't change any BIOS setting and there is not any HW change, correct?

I see you're using "3.4.0 19629:b2128a88e3dd", but 19629's checksum should be 69293fc2096f (http://xenbits.xensource.com/xen-3.4-testing.hg?rev/69293fc2096f).

So I assume you made your own changes to the code?

Can you try the latest changeset 19677: b2ddeb734ca7(http://xenbits.xensource.com/xen-3.4-testing.hg)?

Sorry, I don't have the same type of host, so I can't try it myself. :-(

Thanks,
-- Dexuan

From: Tom Rotenberg [mailto:tom.rotenberg@xxxxxxxxx]
Sent: 2009年6月29日 21:09

To: Cui, Dexuan
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

Well, it looks like my analysis was right.
This is how i reproduced it:
1. Disable the IOMMU
2. Turn off the laptop, and leave it turned off for a few seconds
3. Turn on the laptop
4. Enable the IOMMU (*DON'T* turn off the machine)
5. Continue
6. Xen crashes

and, now the log lines are:
(XEN) Intel VT-d Queued Invalidation supported.
(XEN) Intel VT-d Interrupt Remapping supported.

It looks like there is a problem when enabling IOMMU, without cold boot of the machine.

What do u say?

Here is the full log:
__ __            _____ _ _    ___
\ \/ /___ _ __   |___ /| || | / _ \
\ // _ \ '_ \    |_ \| || |_| | | |
/ \ __/ | | | ___) |__   _| |_| |
/_/\_\___|_| |_| |____(_) |_|(_)___/

(XEN) Xen version 3.4.0 (root@[unknown]) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Mon Jun 29 07:40:11 EDT 2009
(XEN) Latest ChangeSet: Wed Jun 24 11:15:04 2009 +0300 19629:b2128a88e3dd
(XEN) Command line: console=com1 com1=115200,8n1 iommu=1 iommu_inclusive_mapping loglvl=all guest_loglvl=all
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009f000 (usable)
(XEN) 000000000009f000 - 00000000000a0000 (reserved)
(XEN) 0000000000100000 - 000000007c04d400 (usable)
(XEN) 000000007c04d400 - 000000007c04f400 (ACPI NVS)
(XEN) 000000007c04f400 - 000000007f000000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec10000 (reserved)
(XEN) 00000000fed18000 - 00000000fed1c000 (reserved)
(XEN) 00000000fed20000 - 00000000fed90000 (reserved)
(XEN) 00000000feda0000 - 00000000feda6000 (reserved)
(XEN) 00000000fee00000 - 00000000fee10000 (reserved)
(XEN) 00000000ffe60000 - 0000000100000000 (reserved)
(XEN) System RAM: 1983MB (2031536kB)
(XEN) ACPI: RSDP 000FB9C0, 0024 (r2 DELL )
(XEN) ACPI: XSDT 7C051E00, 006C (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: FACP 7C051C9C, 00F4 (r4 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: DSDT 7C052400, 6A2B (r2 INT430 SYSFexxx     1001 INTL 20050624)
(XEN) ACPI: FACS 7C060C00, 0040
(XEN) ACPI: HPET 7C051F00, 0038 (r1 DELL    M09            1 ASL        61)
(XEN) ACPI: DMAR 7C060400, 00B0 (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: APIC 7C052000, 0068 (r1 DELL    M09     27D9020D ASL        47)
(XEN) ACPI: ASF! 7C051C00, 006A (r32 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: MCFG 7C051FC0, 003E (r16 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: TCPA 7C052300, 0032 (r1                        0 ASL         0)
(XEN) ACPI: SLIC 7C05209C, 0176 (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: SSDT 7C05032D, 066C (r1 PmRef    CpuPm     3000 INTL 20050624)
(XEN) NUMA turned off
(XEN) Faking a node at 0000000000000000-000000007c04d000
(XEN) Domain heap initialised
(XEN) DMI 2.4 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x1008
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[1004,0], pm1x_evt[1000,0]
(XEN) ACPI:                  wakeup_vec[7c060c0c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 7:7 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) Processor #1 7:7 APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [VT-D]dmar.c:485: Host address width 36
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed10000
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1b.0
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed13000
(XEN) [VT-D]dmar.c:358: found INCLUDE_ALL
(XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.7
(XEN) Intel VT-d DMAR tables have been parsed.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2527.037 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) HVM: VMX enabled
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) mce_init: init bank1
(XEN) mce_init: init bank2
(XEN) mce_init: init bank3
(XEN) mce_init: init bank4
(XEN) mce_init: init bank5
(XEN) CPU0: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU0 has no CMCI support
(XEN) CPU0: Intel(R) Core(TM)2 Duo CPU     T9400 @ 2.53GHz stepping 06
(XEN) Booting processor 1/1 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) mce_init: init bank1
(XEN) mce_init: init bank2
(XEN) mce_init: init bank4
(XEN) mce_init: init bank5
(XEN) CPU1: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU1 has no CMCI support
(XEN) CPU1: Intel(R) Core(TM)2 Duo CPU     T9400 @ 2.53GHz stepping 06
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 2 CPUs: passed.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 2 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) [VT-D]iommu.c:1761: DMAR: Forcing write-buffer flush
(XEN) Intel VT-d Snoop Control supported.
(XEN) Intel VT-d DMA Passthrough not supported.
(XEN) Intel VT-d Queued Invalidation supported.
(XEN) Intel VT-d Interrupt Remapping supported.
(XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff828c8012c951>] clear_fault_bits+0x21/0xa0
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
(XEN) rax: ffff828bfff5aff8   rbx: ffff830078f82f90   rcx: 0000000000000020
(XEN) rdx: 00000000ffffffff   rsi: ffff828bfff57000   rdi: ffff830078f82f90
(XEN) rbp: 0000000000000090   rsp: ffff828c8026fda0   r8: ffff83007be2c110
(XEN) r9: 000000000000001a   r10: 0000000000000005   r11: 0000000000000005
(XEN) r12: 0000000000000090   r13: 0000000000000090   r14: ffff830078f82fcc
(XEN) r15: 0000000000004800   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000007be74000   cr2: ffff828bfff5aff8
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff828c8026fda0:
(XEN)    ffff828c8012d32d ffff830078f824b0 ffff828c801e422f ffff828c80204490
(XEN)    ffff828c801e422f ffff828c802044c0 ffff828c8026fdf8 0000000000000000
(XEN)    ffff828c8012de7e ffff830078f82d40 ffff828c80228100 000000000000001f
(XEN)    00000000ffffffed ffff828c80227a78 0000000000000001 ffff828c80205f60
(XEN)    ffff828c801f3400 ffff83000008bf30 ffff828c8012a9d4 ffff828c80227978
(XEN)    ffff828c8021a589 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    00000000002b23c0 ffffff8c80274000 0008bf3000000000 000000000008bf30
(XEN)    0000000000000000 ffff83000008bfc0 0000000001ce08a0 0000000000000000
(XEN)    ffffffff00000000 0000000800000000 000000010000006e 0000000000000003
(XEN)    00000000000002f8 0000000000000000 0000000000000000 0000000000067e2c
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    ffff828c801000b5 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff830079f18000
(XEN) Xen call trace:
(XEN)    [<ffff828c8012c951>] clear_fault_bits+0x21/0xa0
(XEN)    [<ffff828c8012d32d>] init_vtd_hw+0x15d/0x3e0
(XEN)    [<ffff828c8012de7e>] intel_vtd_setup+0x5ae/0x610
(XEN)    [<ffff828c8012a9d4>] iommu_setup+0x24/0x110
(XEN)    [<ffff828c8021a589>] __start_xen+0x1859/0x27c0
(XEN)    [<ffff828c801000b5>] __high_start+0xa1/0xa3
(XEN)
(XEN) Pagetable walk from ffff828bfff5aff8:
(XEN) L4[0x105] = 000000007be7a027 5555555555555555
(XEN) L3[0x02f] = 000000007c04b063 5555555555555555
(XEN) L2[0x1ff] = 000000007c04a063 5555555555555555
(XEN) L1[0x15a] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: ffff828bfff5aff8
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

2009/6/29 Tom Rotenberg <tom.rotenberg@xxxxxxxxx>

I'm almost 100% sure, that this was from the same machine (since i have only 1 such Dell currently on my desk :).
However, this log was generated just after turning on the IOMMU (without cold boot). So, let me check if it happened because of this.

I'll check it now...

2009/6/29 Cui, Dexuan <dexuan.cui@xxxxxxxxx>

I have no idea about it.

> (XEN) Intel VT-d Snoop Control supported.
> (XEN) Intel VT-d DMA Passthrough not supported.
> (XEN) Intel VT-d Queued Invalidation supported.
> (XEN) Intel VT-d Interrupt Remapping supported.

This is from the log of your original mail. We can see SC/QI/IR are supported, but in your later log, we can see they are not supported.

I'm curious where the first log comes from. :-)

-- Dexuan

From: Tom Rotenberg [mailto:tom.rotenberg@xxxxxxxxx]
Sent: 2009年6月29日 20:38

To: Cui, Dexuan
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

1) Yep, i can't reproduce it anymore. Maybe i'll try it with another Dell R6400 i have here.
2) On my Lenovo T400, i don't remember the exact crash, and i don't have the log of that crash, i just remember something about "DMAR ... failure", which happened, after i turned on the IOMMU, and didn't perform a cold reboot on that machine.

Do u have any idea why these problems happen?
Does anyone else working with Dell E6400 had this problem?

Tom

2009/6/29 Cui, Dexuan <dexuan.cui@xxxxxxxxx>

It's really weird... :-)

If I understand correctly, with iommu=1 in Xen:

1) on your "Dell E6400 machine", now you can't reproduce the panic any longer;

2) on your "Lenovo T400", you can reproduce the panic sometimes -- if so, can you post the log (with the printk patch applied) when you meet with the panic? Can you find out under what condition it's easy to reproduce the panic?

Thanks,
-- Dexuan

From: Tom Rotenberg [mailto:tom.rotenberg@xxxxxxxxx]
Sent: 2009年6月29日 20:15

To: Cui, Dexuan
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

Hi,

Tried again, this time with cold reboot, it worked again. I also checked the lines you were talking about, and now they are:
(XEN) Intel VT-d Snoop Control not supported.
(XEN) Intel VT-d DMA Passthrough not supported.
(XEN) Intel VT-d Queued Invalidation not supported.
(XEN) Intel VT-d Interrupt Remapping not supported.

Could it be that there is some kind of problem with the hardware, which makes it sometimes report invalid values? (with the Lenovo T400 i'm testing, if you enable the IOMMU, without a cold-reboot, it may cause Xen to crash on boot, so you must do a cold reboot).

Here is the log:
__ __            _____ _ _    ___
\ \/ /___ _ __   |___ /| || | / _ \
\ // _ \ '_ \    |_ \| || |_| | | |
/ \ __/ | | | ___) |__   _| |_| |
/_/\_\___|_| |_| |____(_) |_|(_)___/

(XEN) Xen version 3.4.0 (root@[unknown]) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Mon Jun 29 11:04:57 EDT 2009
(XEN) Latest ChangeSet: Wed Jun 24 11:15:04 2009 +0300 19629:b2128a88e3dd
(XEN) Command line: console=com1 com1=115200,8n1 iommu=1 iommu_inclusive_mapping loglvl=all guest_loglvl=all
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009f000 (usable)
(XEN) 000000000009f000 - 00000000000a0000 (reserved)
(XEN) 0000000000100000 - 000000007c04d400 (usable)
(XEN) 000000007c04d400 - 000000007c04f400 (ACPI NVS)
(XEN) 000000007c04f400 - 000000007f000000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec10000 (reserved)
(XEN) 00000000fed18000 - 00000000fed1c000 (reserved)
(XEN) 00000000fed20000 - 00000000fed90000 (reserved)
(XEN) 00000000feda0000 - 00000000feda6000 (reserved)
(XEN) 00000000fee00000 - 00000000fee10000 (reserved)
(XEN) 00000000ffe60000 - 0000000100000000 (reserved)
(XEN) System RAM: 1983MB (2031536kB)
(XEN) ACPI: RSDP 000FB9C0, 0024 (r2 DELL )
(XEN) ACPI: XSDT 7C051E00, 006C (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: FACP 7C051C9C, 00F4 (r4 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: DSDT 7C052400, 6A2B (r2 INT430 SYSFexxx     1001 INTL 20050624)
(XEN) ACPI: FACS 7C060C00, 0040
(XEN) ACPI: HPET 7C051F00, 0038 (r1 DELL    M09            1 ASL        61)
(XEN) ACPI: DMAR 7C060400, 0120 (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: APIC 7C052000, 0068 (r1 DELL    M09     27D9020D ASL        47)
(XEN) ACPI: ASF! 7C051C00, 006A (r32 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: MCFG 7C051FC0, 003E (r16 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: TCPA 7C052300, 0032 (r1                        0 ASL         0)
(XEN) ACPI: SLIC 7C05209C, 0176 (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: SSDT 7C05032D, 066C (r1 PmRef    CpuPm     3000 INTL 20050624)
(XEN) NUMA turned off
(XEN) Faking a node at 0000000000000000-000000007c04d000
(XEN) Domain heap initialised
(XEN) DMI 2.4 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x1008
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[1004,0], pm1x_evt[1000,0]
(XEN) ACPI:                  wakeup_vec[7c060c0c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 7:7 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) Processor #1 7:7 APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [VT-D]dmar.c:485: Host address width 36
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed10000
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1b.0
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed11000
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.1
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed12000
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:3.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:3.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:3.3
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed13000
(XEN) [VT-D]dmar.c:358: found INCLUDE_ALL
(XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.7
(XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.1
(XEN) Intel VT-d DMAR tables have been parsed.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2527.058 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) HVM: VMX enabled
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) mce_init: init bank1
(XEN) mce_init: init bank2
(XEN) mce_init: init bank3
(XEN) mce_init: init bank4
(XEN) mce_init: init bank5
(XEN) CPU0: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU0 has no CMCI support
(XEN) CPU0: Intel(R) Core(TM)2 Duo CPU     T9400 @ 2.53GHz stepping 06
(XEN) Booting processor 1/1 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) mce_init: init bank1
(XEN) mce_init: init bank2
(XEN) mce_init: init bank4
(XEN) mce_init: init bank5
(XEN) CPU1: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU1 has no CMCI support
(XEN) CPU1: Intel(R) Core(TM)2 Duo CPU     T9400 @ 2.53GHz stepping 06
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 2 CPUs: passed.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 2 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) [VT-D]iommu.c:1763: DMAR: Forcing write-buffer flush
(XEN) Intel VT-d Snoop Control not supported.
(XEN) Intel VT-d DMA Passthrough not supported.
(XEN) Intel VT-d Queued Invalidation not supported.
(XEN) Intel VT-d Interrupt Remapping not supported.
(XEN) clear_fault_bits: CAP_REG=0xc0000020630270
(XEN) clear_fault_bits: CAP_REG=0xc0000020630260
(XEN) clear_fault_bits: CAP_REG=0xc9008020e30260
(XEN) clear_fault_bits: CAP_REG=0xc9008020630260
(XEN) I/O virtualisation enabled
(XEN) I/O virtualisation for PV guests disabled
(XEN) HPET: 4 timers in total, 0 timers will be used for broadcast
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:0.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:3.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:3.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:3.3
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:19.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.7
(XEN) [VT-D]iommu.c:1245:d32767 domain_context_mapping:PCIe: bdf = 0:1b.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.7
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1f.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1f.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1f.3
(XEN) [VT-D]iommu.c:1245:d32767 domain_context_mapping:PCIe: bdf = c:0.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.7
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.7
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff57000
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff56000
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff55000
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff54000
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff8070ab48
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.:   0000000076000000->0000000078000000 (460259 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff80200000->ffffffff8070ab48
(XEN) Init. ramdisk: ffffffff8070b000->ffffffff81b94400
(XEN) Phys-Mach map: ffffffff81b95000->ffffffff81f27f18
(XEN) Start info:    ffffffff81f28000->ffffffff81f284b4
(XEN) Page tables:   ffffffff81f29000->ffffffff81f3c000
(XEN) Boot stack:    ffffffff81f3c000->ffffffff81f3d000
(XEN) TOTAL:         ffffffff80000000->ffffffff82000000
(XEN) ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 132kB init memory.

Tom

2009/6/29 Cui, Dexuan <dexuan.cui@xxxxxxxxx>

So, if I understand correctly, it's really strange you can't reproduce the panic now.

With only the first printk patch applied (and don't comment the clear_fault_bits()), can you try a cold rebooting the host (plug the power cord off from the host; wait several minutes; plug the cord in and tunn on the host)? And the log?

BTW, in your original mail, the log shows:

(XEN) Intel VT-d Queued Invalidation supported.
(XEN) Intel VT-d Interrupt Remapping supported.

But in your this new log, it shows:

(XEN) Intel VT-d Queued Invalidation not supported.
(XEN) Intel VT-d Interrupt Remapping not supported.

Are you sure you have been operating on the same host? :-)

-- Dexuan

From: Tom Rotenberg [mailto:tom.rotenberg@xxxxxxxxx]
Sent: 2009年6月29日 17:01
To: Cui, Dexuan
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

Hi,

Well, just tried what you suggested, and here are the results:
* If i comment the 'clear_fault_bits()' call - then Xen boots ok, with iommu=1
The strange thing, is that after it booted one time ok (with the second patch), then it continued to boot ok, even if i reverted the second patch.

The relevant lines in the log are:
(XEN) clear_fault_bits: CAP_REG=0xc0000020630270
(XEN) clear_fault_bits: CAP_REG=0xc0000020630260
(XEN) clear_fault_bits: CAP_REG=0xc9008020e30260
(XEN) clear_fault_bits: CAP_REG=0xc9008020630260

Anyway, here is the full Xen log, with the first patch (print cap's) applied:

__ __            _____ _ _    ___
\ \/ /___ _ __   |___ /| || | / _ \
\ // _ \ '_ \    |_ \| || |_| | | |
/ \ __/ | | | ___) |__   _| |_| |
/_/\_\___|_| |_| |____(_) |_|(_)___/

(XEN) Xen version 3.4.0 (root@[unknown]) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Mon Jun 29 07:53:38 EDT 2009
(XEN) Latest ChangeSet: Wed Jun 24 11:15:04 2009 +0300 19629:b2128a88e3dd
(XEN) Command line: console=com1 com1=115200,8n1 iommu=1 iommu_inclusive_mapping loglvl=all guest_loglvl=all
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009f000 (usable)
(XEN) 000000000009f000 - 00000000000a0000 (reserved)
(XEN) 0000000000100000 - 000000007c04d400 (usable)
(XEN) 000000007c04d400 - 000000007c04f400 (ACPI NVS)
(XEN) 000000007c04f400 - 000000007f000000 (reserved)
(XEN) 00000000f8000000 - 00000000fc000000 (reserved)
(XEN) 00000000fec00000 - 00000000fec10000 (reserved)
(XEN) 00000000fed18000 - 00000000fed1c000 (reserved)
(XEN) 00000000fed20000 - 00000000fed90000 (reserved)
(XEN) 00000000feda0000 - 00000000feda6000 (reserved)
(XEN) 00000000fee00000 - 00000000fee10000 (reserved)
(XEN) 00000000ffe60000 - 0000000100000000 (reserved)
(XEN) System RAM: 1983MB (2031536kB)
(XEN) ACPI: RSDP 000FB9C0, 0024 (r2 DELL )
(XEN) ACPI: XSDT 7C051E00, 006C (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: FACP 7C051C9C, 00F4 (r4 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: DSDT 7C052400, 6A2B (r2 INT430 SYSFexxx     1001 INTL 20050624)
(XEN) ACPI: FACS 7C060C00, 0040
(XEN) ACPI: HPET 7C051F00, 0038 (r1 DELL    M09            1 ASL        61)
(XEN) ACPI: DMAR 7C060400, 0120 (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: APIC 7C052000, 0068 (r1 DELL    M09     27D9020D ASL        47)
(XEN) ACPI: ASF! 7C051C00, 006A (r32 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: MCFG 7C051FC0, 003E (r16 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: TCPA 7C052300, 0032 (r1                        0 ASL         0)
(XEN) ACPI: SLIC 7C05209C, 0176 (r1 DELL    M09     27D9020D ASL        61)
(XEN) ACPI: SSDT 7C05032D, 066C (r1 PmRef    CpuPm     3000 INTL 20050624)
(XEN) NUMA turned off
(XEN) Faking a node at 0000000000000000-000000007c04d000
(XEN) Domain heap initialised
(XEN) DMI 2.4 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x1008
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[1004,0], pm1x_evt[1000,0]
(XEN) ACPI:                  wakeup_vec[7c060c0c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 7:7 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) Processor #1 7:7 APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [VT-D]dmar.c:485: Host address width 36
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed10000
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1b.0
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed11000
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.1
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed12000
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:3.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:3.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:3.3
(XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:349: dmaru->address = fed13000
(XEN) [VT-D]dmar.c:358: found INCLUDE_ALL
(XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1d.7
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.1
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.2
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:1a.7
(XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.0
(XEN) [VT-D]dmar.c:300: found endpoint: bdf = 0:2.1
(XEN) Intel VT-d DMAR tables have been parsed.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 2527.068 MHz processor.
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) HVM: VMX enabled
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) mce_init: init bank1
(XEN) mce_init: init bank2
(XEN) mce_init: init bank3
(XEN) mce_init: init bank4
(XEN) mce_init: init bank5
(XEN) CPU0: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU0 has no CMCI support
(XEN) CPU0: Intel(R) Core(TM)2 Duo CPU     T9400 @ 2.53GHz stepping 06
(XEN) Booting processor 1/1 eip 8c000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 6144K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) mce_init: init bank1
(XEN) mce_init: init bank2
(XEN) mce_init: init bank4
(XEN) mce_init: init bank5
(XEN) CPU1: Thermal monitoring enabled (TM2)
(XEN) CMCI: CPU1 has no CMCI support
(XEN) CPU1: Intel(R) Core(TM)2 Duo CPU     T9400 @ 2.53GHz stepping 06
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 2 CPUs: passed.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 2 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) [VT-D]iommu.c:1763: DMAR: Forcing write-buffer flush
(XEN) Intel VT-d Snoop Control not supported.
(XEN) Intel VT-d DMA Passthrough not supported.
(XEN) Intel VT-d Queued Invalidation not supported.
(XEN) Intel VT-d Interrupt Remapping not supported.
(XEN) clear_fault_bits: CAP_REG=0xc0000020630270
(XEN) clear_fault_bits: CAP_REG=0xc0000020630260
(XEN) clear_fault_bits: CAP_REG=0xc9008020e30260
(XEN) clear_fault_bits: CAP_REG=0xc9008020630260
(XEN) I/O virtualisation enabled
(XEN) I/O virtualisation for PV guests disabled
(XEN) HPET: 4 timers in total, 0 timers will be used for broadcast
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:0.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:3.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:3.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:3.3
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:19.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.7
(XEN) [VT-D]iommu.c:1245:d32767 domain_context_mapping:PCIe: bdf = 0:1b.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.7
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1f.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1f.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1f.3
(XEN) [VT-D]iommu.c:1245:d32767 domain_context_mapping:PCIe: bdf = c:0.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:2.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1d.7
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.0
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.1
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.2
(XEN) [VT-D]iommu.c:1252:d32767 domain_context_mapping:PCI: bdf = 0:1a.7
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff57000
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff56000
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff55000
(XEN) [VT-D]iommu.c:620: iommu_enable_translation: iommu->reg = ffff828bfff54000
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff8070ab48
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.:   0000000076000000->0000000078000000 (460259 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff80200000->ffffffff8070ab48
(XEN) Init. ramdisk: ffffffff8070b000->ffffffff81b94400
(XEN) Phys-Mach map: ffffffff81b95000->ffffffff81f27f18
(XEN) Start info:    ffffffff81f28000->ffffffff81f284b4
(XEN) Page tables:   ffffffff81f29000->ffffffff81f3c000
(XEN) Boot stack:    ffffffff81f3c000->ffffffff81f3d000
(XEN) TOTAL:         ffffffff80000000->ffffffff82000000
(XEN) ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 132kB init memory.

Tom

2009/6/29 Cui, Dexuan <dexuan.cui@xxxxxxxxx>

diff -r 90391e2247fc xen/drivers/passthrough/vtd/iommu.c
--- a/xen/drivers/passthrough/vtd/iommu.c Tue Jun 23 11:40:27 2009 +0100
+++ b/xen/drivers/passthrough/vtd/iommu.c Mon Jun 29 10:24:31 2009 +0800
@@ -1693,7 +1693,7 @@ static int init_vtd_hw(void)
}
dma_msi_data_init(iommu, iommu->vector);
dma_msi_addr_init(iommu, cpu_physical_id(first_cpu(cpu_online_map)));
- clear_fault_bits(iommu);
+ //clear_fault_bits(iommu);
dmar_writel(iommu->reg, DMAR_FECTL_REG, 0);

/* initialize flush functions */

And, can you comment the invocation of clear_fault_bits() in init_vtd_hw() like above and see if the host can boot fine?

Thanks,
-- Dexuan

-----Original Message-----
From: Cui, Dexuan
Sent: 2009?6?29? 14:30
To: 'Tom Rotenberg'; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

diff -r 90391e2247fc xen/drivers/passthrough/vtd/iommu.c
--- a/xen/drivers/passthrough/vtd/iommu.c Tue Jun 23 11:40:27 2009 +0100
+++ b/xen/drivers/passthrough/vtd/iommu.c Mon Jun 29 10:18:23 2009 +0800
@@ -1653,6 +1653,8 @@ void clear_fault_bits(struct iommu *iomm
{
u64 val;

+ printk("clear_fault_bits: CAP_REG=0x%llx\n",
+ (unsigned long long)dmar_readq(iommu->reg,DMAR_CAP_REG));
val = dmar_readq(
iommu->reg,
cap_fault_reg_offset(dmar_readq(iommu->reg,DMAR_CAP_REG))+0x8);

Hi Tom,
Can you please apply the above patch and send us the entire log?
(I don't have the same host, so I can't try it myself.)

Thanks,
-- Dexuan

________________________________

From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Tom Rotenberg
Sent: 2009?6?28? 22:56
To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iommu=1

Hi,

I am trying to use xen 3.4 on a Dell E6400 machine. But, when i try to boot xen with a 'iommu=1' parameter in the boot line, Xen fails to boot. I have also tried to boot with and without the 'iommu_inclusive_mapping' boot option, but it didn't seem to help at all.

Does anyone knows how to solve this issue?

These are the last messages in the UART:

(XEN) Brought up 2 CPUs
(XEN) microcode.c:73:d32767 microcode: CPU1 resumed
(XEN) [VT-D]iommu.c:1761: DMAR: Forcing write-buffer flush
(XEN) Intel VT-d Snoop Control supported.
(XEN) Intel VT-d DMA Passthrough not supported.
(XEN) Intel VT-d Queued Invalidation supported.
(XEN) Intel VT-d Interrupt Remapping supported.
(XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff828c8012c951>] clear_fault_bits+0x21/0xa0
(XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor
(XEN) rax: ffff828bfff5aff8 rbx: ffff830078f82f90 rcx: 0000000000000020
(XEN) rdx: 00000000ffffffff rsi: ffff828bfff57000 rdi: ffff830078f82f90
(XEN) rbp: 0000000000000090 rsp: ffff828c8026fda0 r8: ffff83007be2c110
(XEN) r9: 000000000000001a r10: 0000000000000005 r11: 0000000000000005
(XEN) r12: 0000000000000090 r13: 0000000000000090 r14: ffff830078f82fcc
(XEN) r15: 0000000000004800 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 000000007be74000 cr2: ffff828bfff5aff8
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff828c8026fda0:
(XEN) ffff828c8012d32d ffff830078f824b0 ffff828c801e422f ffff828c80204490
(XEN) ffff828c801e422f ffff828c802044c0 ffff828c8026fdf8 0000000000000000
(XEN) ffff828c8012de7e ffff830078f82d40 ffff828c80228100 000000000000001f
(XEN) 00000000ffffffed ffff828c80227a78 0000000000000001 ffff828c80205f60
(XEN) ffff828c801f3400 ffff83000008bf40 ffff828c8012a9d4 ffff828c80227978
(XEN) ffff828c8021a589 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 00000000002b23c0 ffffff8c80274000 0008bf4000000000 000000000008bf40
(XEN) 0000000000000000 ffff83000008bfc0 0000000001ce08a0 0000000000000000
(XEN) ffffffff00000000 0000000800000000 000000010000006e 0000000000000003
(XEN) 00000000000002f8 0000000000000000 0000000000000000 0000000000067e2c
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) ffff828c801000b5 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 ffff830079f18000
(XEN) Xen call trace:
(XEN) [<ffff828c8012c951>] clear_fault_bits+0x21/0xa0
(XEN) [<ffff828c8012d32d>] init_vtd_hw+0x15d/0x3e0
(XEN) [<ffff828c8012de7e>] intel_vtd_setup+0x5ae/0x610
(XEN) [<ffff828c8012a9d4>] iommu_setup+0x24/0x110
(XEN) [<ffff828c8021a589>] __start_xen+0x1859/0x27c0
(XEN) [<ffff828c801000b5>] __high_start+0xa1/0xa3
(XEN)
(XEN) Pagetable walk from ffff828bfff5aff8:
(XEN) L4[0x105] = 000000007be7a027 5555555555555555
(XEN) L3[0x02f] = 000000007c04b063 5555555555555555
(XEN) L2[0x1ff] = 000000007c04a063 5555555555555555
(XEN) L1[0x15a] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: ffff828bfff5aff8
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

WARNING - OLD ARCHIVES

xen-devel

Re: [Xen-devel] Problem: Xen 3.4 fails to boot on Dell e6400, with iomm