WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

[Xen-devel] PCI Passthrough to HVM on xen-unstable

To: xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>
Subject: [Xen-devel] PCI Passthrough to HVM on xen-unstable
From: Jambunathan K <jambunathan@xxxxxxxxxx>
Date: Thu, 29 Nov 2007 22:28:00 +0530
Cc: Sanjeev Jorapur <sanjeev@xxxxxxxxxx>
Delivery-date: Thu, 29 Nov 2007 09:00:25 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.6 (Windows/20070728)
(Resubmitting my original post with 'xm dmesg' output)

I am working on S5000VSA Intel Server Board with the following cpu spec.

XEN-PEER-RHEL5 $ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz
stepping        : 7
cpu MHz         : 2327.512
cache size      : 4096 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat pse36
clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc up pni
monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips        : 4661.90
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Is my box vt-d capable? If the answer is yes proceed ahead to the details. If
the answer is no, how can a novice user like me identify whether or not his box
is vt-d capable.

I am attaching below snapshots of dump that I think are relevant. You could see
the following:
        a) XEN specific dom0 kernel config.
        b) menu.lst entry
        c) Guest VM (which is a Sles10 HVM) config file.
        d) dmesg | grep pciback which says that hidden device indeed exists.
        e) output of 'xm dmesg'

When I start the HVM I get the following error:

XEN-PEER-RHEL5 $ xm create sles10disk.cfg
Using config file "./sles10disk.cfg".
Error: Fail to assign device(4:0.0): maybe VT-d is not enabled, or the device is
not exist, or it has already been assigned to other domain


The following thing surprises me:

        a) If my device - (4:0.0) is indeed hidden in Dom0 but *not assigned* 
to the
HVM, realtek card shows up at (4:0.0) in HVM's lspci as below. Note that my dom0
kernel config has CONFIG_XEN_PCIDEV_BACKEND_PASS=y which I believe is 
*non-default*.

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Class ff80: Unknown device 5853:0001 (rev 01)
00:04.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8139/8139C/8139C+ (rev 20)







XEN-PEER-RHEL5 $ cat /boot/xen320-unstable/config-2.6.18.8-xen-1127 | grep XEN
CONFIG_X86_64_XEN=y
CONFIG_X86_XEN_GENAPIC=y
CONFIG_XEN_PCIDEV_FRONTEND=y
# CONFIG_XEN_PCIDEV_FE_DEBUG is not set
CONFIG_TCG_XEN=m
CONFIG_XEN=y
CONFIG_XEN_INTERFACE_VERSION=0x00030207
# XEN
CONFIG_XEN_PRIVILEGED_GUEST=y
# CONFIG_XEN_UNPRIVILEGED_GUEST is not set
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_XENBUS_DEV=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_BLKDEV_TAP=y
CONFIG_XEN_NETDEV_BACKEND=y
# CONFIG_XEN_NETDEV_PIPELINED_TRANSMITTER is not set
CONFIG_XEN_NETDEV_LOOPBACK=y
CONFIG_XEN_PCIDEV_BACKEND=y
# CONFIG_XEN_PCIDEV_BACKEND_VPCI is not set
CONFIG_XEN_PCIDEV_BACKEND_PASS=y
# CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
# CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set
# CONFIG_XEN_PCIDEV_BE_DEBUG is not set
CONFIG_XEN_TPMDEV_BACKEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_FRAMEBUFFER=y
CONFIG_XEN_KEYBOARD=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DISABLE_SERIAL=y
CONFIG_XEN_SYSFS=y
CONFIG_XEN_COMPAT_030002_AND_LATER=y
# CONFIG_XEN_COMPAT_030004_AND_LATER is not set
# CONFIG_XEN_COMPAT_030100_AND_LATER is not set
# CONFIG_XEN_COMPAT_LATEST_ONLY is not set
CONFIG_XEN_COMPAT=0x030002
CONFIG_XEN_SMPBOOT=y
CONFIG_XEN_BALLOON=y
CONFIG_XEN_DEVMEM=y

XEN-PEER-RHEL5 $ cat /boot/grub/menu.lst
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol01
#          initrd /initrd-version.img
#boot=/dev/sda

default=3
timeout=10

#0
title Red Hat Enterprise Linux Server (2.6.18-8.el5-smp)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-8.el5 ro root=/dev/VolGroup00/LogVol01
crashkernel=128M@16M
        initrd /initrd-2.6.18-8.el5.img

#1
title Red Hat Enterprise Linux Server (2.6.18-8.el5-up)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-8.el5 ro root=/dev/VolGroup00/LogVol01
crashkernel=128M@16M maxcpus=1
        initrd /initrd-2.6.18-8.el5.img

#2
title RHEL5-XEN311-RC2
    root (hd0,0)
    kernel /xen311/xen-3.1.1-rc2.gz dom0_mem=1300M loopback.nloopbacks=16
    module /xen311/vmlinuz-2.6.18-xen-311 root=/dev/VolGroup00/LogVol01  ro
showopts console=tty0
    module /xen311/initrd-2.6.18-xen-311.img


#3
title RHEL5-XEN320-UNSTABLE
    root (hd0,0)
    kernel /xen320-unstable/xen-3.2-unstable.gz dom0_mem=1300M
loopback.nloopbacks=16 vtd=1
    module /xen320-unstable/vmlinuz-2.6.18.8-xen-1127
root=/dev/VolGroup00/LogVol01  ro showopts console=tty0  pciback.hide=(04:00.0)
pciback.verbose_request=1 apic=debug maxcpus=1
    module /xen320-unstable/initrd-2.6.18.8-xen-1127.img

XEN-PEER-RHEL5 $ cat /mnt/archive/xendisks/Sles10/sles10disk.cfg
#  -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using 'xm 
create'.
# You use a separate script for each domain you want to create, or
# you can set the parameters for the domain on the xm command line.
#============================================================================

import os, re
arch = os.uname()[4]
if re.search('64', arch):
    arch_libdir = 'lib64'
else:
    arch_libdir = 'lib'

#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/usr/lib/xen/boot/hvmloader"

# The domain build function. HVM domain uses 'hvm'.
builder='hvm'

# Initial memory allocation (in megabytes) for the new domain.
#
# WARNING: Creating a domain with insufficient memory may cause out of
#          memory errors. The domain needs enough memory to boot kernel
#          and modules. Allocating less than 32MBs is not recommended.
memory = 1200

# Shadow pagetable memory for the domain, in MB.
# Should be at least 2KB per MB of domain memory, plus a few MB per vcpu.
# shadow_memory = 8

# A name for your domain. All domains must have different names.
name = "HvmSles10"

# 128-bit UUID for the domain.  The default behavior is to generate a new UUID
# on each call to 'xm create'.
#uuid = "06ed00fe-1162-4fc4-b5d8-11993ee4a8b9"

#-----------------------------------------------------------------------------
# The number of cpus guest platform has, default=1
#vcpus=1

# Enable/disable HVM guest PAE, default=1 (enabled)
#pae=1

# Enable/disable HVM guest ACPI, default=1 (enabled)
#acpi=1

# Enable/disable HVM APIC mode, default=1 (enabled)
# Note that this option is ignored if vcpus > 1
#apic=1

# List of which CPUS this domain is allowed to use, default Xen picks
#cpus = ""         # leave to Xen to pick
#cpus = "0"        # all vcpus run on CPU0
#cpus = "0-3,5,^1" # run on cpus 0,2,3,5

# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ 'type=ioemu, mac=00:16:3e:00:00:11, bridge=xenbr0, model=ne2k_pci' ]
# type=ioemu specify the NIC is an ioemu device not netfront
vif = [ 'type=ioemu, mac=00:16:3e:65:63:5d, bridge=xenbr0' ]

#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.

#disk = [ 'phy:hda1,hda1,r' ]
#disk = [ 'file:/mnt/guest02/sles10disk.img,ioemu:hda,w',
'file:/mnt/guest02/SLES-10-AMD64-EM64T-DVD1.iso,hdc:cdrom,r' ]
disk = [ 'file:/mnt/archive/xendisks/Sles10/sles10disk.img,ioemu:hda,w' ]

#----------------------------------------------------------------------------
# Configure the behaviour when a domain exits.  There are three 'reasons'
# for a domain to stop: poweroff, reboot, and crash.  For each of these you
# may specify:
#
#   "destroy",        meaning that the domain is cleaned up as normal;
#   "restart",        meaning that a new domain is started in place of the old
#                     one;
#   "preserve",       meaning that no clean-up is done until the domain is
#                     manually destroyed (using xm destroy, for example); or
#   "rename-restart", meaning that the old domain is not cleaned up, but is
#                     renamed and a new domain started in its place.
#
# The default is
#
#   on_poweroff = 'destroy'
#   on_reboot   = 'restart'
#   on_crash    = 'restart'
#
# For backwards compatibility we also support the deprecated option restart
#
# restart = 'onreboot' means on_poweroff = 'destroy'
#                            on_reboot   = 'restart'
#                            on_crash    = 'destroy'
#
# restart = 'always'   means on_poweroff = 'restart'
#                            on_reboot   = 'restart'
#                            on_crash    = 'restart'
#
# restart = 'never'    means on_poweroff = 'destroy'
#                            on_reboot   = 'destroy'
#                            on_crash    = 'destroy'

on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

#============================================================================

# New stuff
device_model = '/usr/' + arch_libdir + '/xen/bin/qemu-dm'

#-----------------------------------------------------------------------------
# boot on floppy (a), hard disk (c), Network (n) or CD-ROM (d)
# default: hard disk, cd-rom, floppy
boot="cda"

#-----------------------------------------------------------------------------
#  write to temporary files instead of disk image files
#snapshot=1

#----------------------------------------------------------------------------
# enable SDL library for graphics, default = 0
sdl=0

#----------------------------------------------------------------------------
# enable VNC library for graphics, default = 1
vnc=1

#----------------------------------------------------------------------------
# address that should be listened on for the VNC server if vnc is set.
# default is to use 'vnc-listen' setting from /etc/xen/xend-config.sxp
#vnclisten="127.0.0.1"

#----------------------------------------------------------------------------
# set VNC display number, default = domid
#vncdisplay=1

#----------------------------------------------------------------------------
# try to find an unused port for the VNC server, default = 1
#vncunused=1

#----------------------------------------------------------------------------
# enable spawning vncviewer for domain's console
# (only valid when vnc=1), default = 0
vncconsole=1

#----------------------------------------------------------------------------
# set password for domain's VNC console
# default is depents on vncpasswd in xend-config.sxp
vncpasswd=''

#----------------------------------------------------------------------------
# no graphics, use serial port
#nographic=0

#----------------------------------------------------------------------------
# enable stdvga, default = 0 (use cirrus logic device model)
stdvga=0

#-----------------------------------------------------------------------------
#   serial port re-direct to pty deivce, /dev/pts/n
#   then xm console or minicom can connect
serial='pty'


#-----------------------------------------------------------------------------
#   Qemu Monitor, default is disable
#   Use ctrl-alt-2 to connect
#monitor=1


#-----------------------------------------------------------------------------
#   enable sound card support, [sb16|es1370|all|..,..], default none
#soundhw='sb16'


#-----------------------------------------------------------------------------
#    set the real time clock to local time [default=0 i.e. set to utc]
#localtime=1


#-----------------------------------------------------------------------------
#    set the real time clock offset in seconds [default=0 i.e. same as dom0]
#rtc_timeoffset=3600

#-----------------------------------------------------------------------------
#    start in full screen
#full-screen=1


#-----------------------------------------------------------------------------
#   Enable USB support (specific devices specified at runtime through the
#                       monitor window)
#usb=1

#   Enable USB mouse support (only enable one of the following, `mouse' for
#                             PS/2 protocol relative mouse, `tablet' for
#                             absolute mouse)
#usbdevice='mouse'
#usbdevice='tablet'

#-----------------------------------------------------------------------------
#   Set keyboard layout, default is en-us keyboard.
#keymap='ja'

pci = [ '04:00.0' ]
XEN-PEER-RHEL5 $

XEN-PEER-RHEL5 $ dmesg | grep pciback
Bootdata ok (command line is root=/dev/VolGroup00/LogVol01  ro showopts
console=tty0  pciback.hide=(04:00.0) pciback.verbose_request=1 apic=debug 
maxcpus=1)
Kernel command line: root=/dev/VolGroup00/LogVol01  ro showopts console=tty0
pciback.hide=(04:00.0) pciback.verbose_request=1 apic=debug maxcpus=1
pciback 0000:04:00.0: seizing device

XEN-PEER-RHEL5 $ xm dmesg | grep VMX
(XEN) HVM: VMX enabled
XEN-PEER-RHEL5 $ xm dmesg
 __  __            _____  ____                     _        _     _
 \ \/ /___ _ __   |___ / |___ \    _   _ _ __  ___| |_ __ _| |__ | | ___
  \  // _ \ '_ \    |_ \   __) |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | / __/|__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)_____|   \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

(XEN) Xen version 3.2-unstable (root@) (gcc version 4.1.1 20070105 (Red Hat
4.1.1-52)) Thu Nov 29 12:16:30 IST 2007
(XEN) Latest ChangeSet: Sun Nov 25 12:43:13 2007 +0000 16461:d40788f07a4f
(XEN) Command line: dom0_mem=1300M loopback.nloopbacks=16 vtd=1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000009e48c000 (usable)
(XEN)  000000009e48c000 - 000000009e545000 (ACPI NVS)
(XEN)  000000009e545000 - 000000009fa42000 (usable)
(XEN)  000000009fa42000 - 000000009fa9a000 (reserved)
(XEN)  000000009fa9a000 - 000000009fab0000 (usable)
(XEN)  000000009fab0000 - 000000009fb1a000 (ACPI NVS)
(XEN)  000000009fb1a000 - 000000009fb2b000 (usable)
(XEN)  000000009fb2b000 - 000000009fb3a000 (ACPI data)
(XEN)  000000009fb3a000 - 000000009fc00000 (usable)
(XEN)  000000009fc00000 - 00000000b0000000 (reserved)
(XEN)  00000000ffc00000 - 00000000ffc0c000 (reserved)
(XEN)  0000000100000000 - 0000000160000000 (usable)
(XEN) System RAM: 4090MB (4188244kB)
(XEN) Xen heap: 14MB (14928kB)
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) Processor #0 6:15 APIC version 20
(XEN) Processor #2 6:15 APIC version 20
(XEN) Processor #4 6:15 APIC version 20
(XEN) Processor #6 6:15 APIC version 20
(XEN) Processor #1 6:15 APIC version 20
(XEN) Processor #3 6:15 APIC version 20
(XEN) Processor #5 6:15 APIC version 20
(XEN) Processor #7 6:15 APIC version 20
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
(XEN) Enabling APIC mode:  Flat.  Using 2 I/O APICs
(XEN) ACPI DMAR:No DMAR devices found
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2327.563 MHz processor.
(XEN) HVM: VMX enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Booting processor 1/2 eip 8c000
(XEN) CPU1: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Booting processor 2/4 eip 8c000
(XEN) CPU2: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Booting processor 3/6 eip 8c000
(XEN) CPU3: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Booting processor 4/1 eip 8c000
(XEN) CPU4: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Booting processor 5/3 eip 8c000
(XEN) CPU5: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Booting processor 6/5 eip 8c000
(XEN) CPU6: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Booting processor 7/7 eip 8c000
(XEN) CPU7: Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz stepping 07
(XEN) Total of 8 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer overflows in 14998 jiffies.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 8 CPUs
(XEN) AMD IOMMU: Disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff8059f33c
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000015a000000->000000015b000000 (328704 pages to be
allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff8059f33c
(XEN)  Init. ramdisk: ffffffff805a0000->ffffffff80b65800
(XEN)  Phys-Mach map: ffffffff80b66000->ffffffff80df0000
(XEN)  Start info:    ffffffff80df0000->ffffffff80df04a4
(XEN)  Page tables:   ffffffff80df1000->ffffffff80dfc000
(XEN)  Boot stack:    ffffffff80dfc000->ffffffff80dfd000
(XEN)  TOTAL:         ffffffff80000000->ffffffff81000000
(XEN)  ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Initrd len 0x5c5800, start at 0xffffffff805a0000
(XEN) Scrubbing Free RAM: ...........................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to 
Xen)
(XEN) Freed 100kB init memory.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel