[Xen-users] XCP 1.0, Ubuntu 10.04, PCI Passthrough

Hey guys,

I've been having some adventures attempting to connect a couple USB devices through XCP 1.0 dom0 into a Ubuntu 10.04 Lucid LTS domU guest.

I ended up compiling my own 2.6.37 Linux kernel on Ubuntu to get the necessary xen_pcifront support.

I read just about every forum post, etc, I could find.. and after a lot of work, I've almost got it going.. except, it doesn't work!

It's been fun anyways.

I've finally gotten the damn thing to the point where it passes through the USB controllers using PCI passthrough, and my domU recognizes them.. but basically the USB devices don't work.

These are what I'm seeing in my domU dmesg:

[ 0.030635] usbcore: registered new interface driver usbfs

[ 0.030635] usbcore: registered new interface driver hub

[ 0.032070] usbcore: registered new device driver usb

[ 0.506727] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[ 0.506765] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[ 0.506793] uhci_hcd: USB Universal Host Controller Interface driver

[ 0.605083] uhci_hcd 0000:00:00.0: new USB bus registered, assigned bus number 1

[ 0.605575] hub 1-0:1.0: USB hub found

[ 0.606111] uhci_hcd 0000:00:00.1: new USB bus registered, assigned bus number 2

[ 0.606570] hub 2-0:1.0: USB hub found

[ 0.607075] uhci_hcd 0000:00:00.2: new USB bus registered, assigned bus number 3

[ 0.607531] hub 3-0:1.0: USB hub found

[ 0.608092] uhci_hcd 0000:00:00.3: new USB bus registered, assigned bus number 4

[ 0.608568] hub 4-0:1.0: USB hub found

[ 0.609133] ehci_hcd 0000:00:00.7: new USB bus registered, assigned bus number 5

[ 0.628092] ehci_hcd 0000:00:00.7: USB 2.0 started, EHCI 1.00

[ 0.628438] hub 5-0:1.0: USB hub found

[ 0.996220] usb 5-6: new high speed USB device using ehci_hcd and address 3

[ 5.908044] ehci_hcd 0000:00:00.7: Unlink after no-IRQ? Controller is probably using the wrong IRQ.

[ 16.001977] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 16.002005] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 16.205965] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 16.205991] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 16.409975] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 16.410002] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 16.613978] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 16.614005] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 16.817991] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 16.818017] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 16.818031] hub 5-0:1.0: Cannot enable port 6. Maybe the USB cable is bad?

[ 16.873980] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 16.874743] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 17.077981] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 17.078010] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 17.281985] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 17.282014] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 17.485980] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 17.486008] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 17.689994] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 17.690021] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 17.690035] hub 5-0:1.0: Cannot enable port 6. Maybe the USB cable is bad?

[ 17.746000] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 17.746030] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 17.949973] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 17.950004] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 18.154025] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 18.154915] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 18.358022] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 18.358892] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 18.562012] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 18.563012] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 18.563032] hub 5-0:1.0: Cannot enable port 6. Maybe the USB cable is bad?

[ 18.617984] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 18.618012] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 18.822006] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 18.822036] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 19.025970] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 19.026000] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 19.229986] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 19.230017] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 19.433986] ehci_hcd 0000:00:00.7: port 6 reset error -110

[ 19.434016] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 19.434031] hub 5-0:1.0: Cannot enable port 6. Maybe the USB cable is bad?

[ 19.434063] hub 5-0:1.0: unable to enumerate USB device on port 6

[ 19.501976] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 19.502005] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 19.705985] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 19.706244] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 19.910428] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 19.911353] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 20.113978] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 20.114122] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 20.317972] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 20.317999] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 20.318014] hub 5-0:1.0: Cannot enable port 7. Maybe the USB cable is bad?

[ 20.373970] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 20.373995] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 20.577986] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 20.578016] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 20.781976] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 20.782001] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 20.986015] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 20.986043] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 21.189972] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 21.189997] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 21.190012] hub 5-0:1.0: Cannot enable port 7. Maybe the USB cable is bad?

[ 21.245971] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 21.246076] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 21.449975] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 21.449999] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 21.653973] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 21.653998] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 21.857983] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 21.858007] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 22.061970] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 22.061996] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 22.062011] hub 5-0:1.0: Cannot enable port 7. Maybe the USB cable is bad?

[ 22.117987] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 22.118017] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 22.321992] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 22.322022] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 22.525983] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 22.526014] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 22.729976] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 22.730004] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 22.934003] ehci_hcd 0000:00:00.7: port 7 reset error -110

[ 22.934034] hub 5-0:1.0: hub_port_status failed (err = -32)

[ 22.934049] hub 5-0:1.0: Cannot enable port 7. Maybe the USB cable is bad?

[ 22.934080] hub 5-0:1.0: unable to enumerate USB device on port 7

[ 23.208066] usb 1-2: new low speed USB device using uhci_hcd and address 2

[ 25.827681] ehci_hcd 0000:00:00.7: force halt; handshake e0822824 0000c000 00000000 -> -110

[ 28.208049] uhci_hcd 0000:00:00.0: Unlink after no-IRQ? Controller is probably using the wrong IRQ.

I passed through all of my USB devices from dom0,

[root@vh01 ~]# lspci

00:00.0 Host bridge: Intel Corporation E7230/3000/3010 Memory Controller Hub

00:01.0 PCI bridge: Intel Corporation E7230/3000/3010 PCI Express Root Port

00:1c.0 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 1 (rev 01)

00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5 (rev 01)

00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 6 (rev 01)

00:1d.0 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #1 (rev 01)

00:1d.1 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #2 (rev 01)

00:1d.2 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #3 (rev 01)

00:1d.3 USB Controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #4 (rev 01)

00:1d.7 USB Controller: Intel Corporation N10/ICH 7 Family USB2 EHCI Controller (rev 01)

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)

00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)

00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01)

00:1f.2 IDE interface: Intel Corporation N10/ICH7 Family SATA IDE Controller (rev 01)

00:1f.3 SMBus: Intel Corporation N10/ICH 7 Family SMBus Controller (rev 01)

04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express (rev 01)

05:07.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics Innovation) Z7/Z9 (XG20 core)

I passed through all of the 00:1d entries separately.

I can't for the life of me figure out what might be the cause of the above errors, or essentially why in the end even though it kinda detects the devices:

"[ 0.996220] usb 5-6: new high speed USB device using ehci_hcd and address 3"

"[ 23.296067] usb 1-2: new low speed USB device using uhci_hcd and address 2"

e.g. on my dom0:

xe vm-param-set other-config:pci=0/0000:00:1d.0,0/0000:00:1d.1,0/0000:00:1d.2,0/0000:00:1d.3,0/0000:00:1d.7 uuid=UUID

Essentially the devices don't work, the actual DVD drive doesn't show up, the USB UPS isn't accessible from nut (the 2 devices I wanted).. those messages above "new USB device" are the closest the devices come to being accessible.

Any ideas at all??

I've written up a full run sheet of instructions and what I did, originally for my own knowledge, but thought they might be helpful to someone... attached below

#Near as I can tell you can't do USB-only pass through with XCP 1.0

#However, you can do PCI pass-through and pass-through the entire USB controller device

#I did the below with Ubuntu 10.04 LTS Lucid

#read this guide...I read it and then kept re-reading it and realizing half the answers I wanted were in fact there just buried a bit

#http://wiki.xensource.com/xenwiki/XenPCIpassthrough

# Note my VMs are all PV... with HVM it's a different story from what I understand, read the above

#confirm that your dom0 (XCP 1.0 in my case) supports pciback

#look for pciback here

ls /sys/bus/pci/drivers/

#now check your Linux domU (guest OS) for pci-passthrough support (needs xen-pcifront module)

# (on domU)

#Your Ubuntu system must have xen-pcifront/xen_pcifront installed in the kernel for pci pass-through to work

#Basically although it's not said anywhere, your kernel version must be v2.6.37+

#e.g. uname -a should return a kernel version greater than that

#No Ubuntu 10.04 packages exist for a kernel of this version

#I also tried the kernels located here: kernel.ubuntu.com/~kernel-ppa/mainline/

#But apparently they're not compiled with Xen support or something because my XCP refused to boot the kernel complaining

#about missing modules... so I finally gave up and compiled my own kernel

# check the version

uname -a

# v2.6.37+ linux has pcifront built in:

# if you have a lower version, you need a newer kernel

# here's compile instructions I used on Lucid 10.04:

# reference: http://linuxtweaking.blogspot.com/2011/02/how-to-compile-kernel-from-kernelorg-in.html

# If you feel like living dangerously/lazily, you can try to use the 32-bit kernel I compiled..

# I've put up a link here, but naturally this comes with NO warranty at all, and Ubuntu specifically states

# compiling your own kernel voids any potential warranty/support.. so there's that..

# anyways:

# http://dev.locatrix.com/linux-headers-2.6.37_2.6.37-10.00.Custom_i386.deb

# http://dev.locatrix.com/linux-image-2.6.37_2.6.37-10.00.Custom_i386.deb

# so skip most of the below and follow along from where it starts with dpkg -i for the above files

cd ~ # or somewhere

apt-get install fakeroot kernel-wedge build-essential makedumpfile kernel-package libncurses5 libncurses5-dev

apt-get build-dep --no-install-recommends linux-image-$(uname -r)

mkdir src

cd src

wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.37.tar.gz

tar xvf linux-2.6.37.tar.gz

cd linux-2.6.37

# basically I left all the defaults alone.. seemed to already include xen and everything else

#for some god unknown reason I did have to uncheck the option labeled "one laptop per child" (OLC) support... no clue..

make menuconfig

export CONCURRENCY_LEVEL=3

make-kpkg clean

time fakeroot make-kpkg --initrd kernel-image kernel-headers

cd ..

ls -l # note the package file names and install them

dpkg -i linux-headers-2.6.37_2.6.37-10.00.Custom_i386.deb

dpkg -i linux-image-2.6.37_2.6.37-10.00.Custom_i386.deb

update-initramfs -c -k 2.6.37

update-grub

# note down the new kernel/init versions

ls -la /

# e.g. /vmlinuz-2.6.37 /initrd.img-2.6.37

#shutdown

halt

# BACK on dom0 (my XCP host)

#find VM's uuid

xe vm-list

#make sure your VM is now set to boot from the new kernel

#this may or may not apply to you.. but I configured my VM to have the kernel/ramdisk set this way:

xe vm-param-set uuid=YOURUID PV-bootloader-args="--kernel=/vmlinuz-2.6.37 --ramdisk=/initrd.img-2.6.37"

#now search for the PCI device you want to passthrough

#I wanted to passthrough a USB device, so that's what I'm doing here:

#search for the device/pci information.. couple different ways:

#easy way to identify pci devices:

lspci -tv

#best way to get the exact pci number, the device number should be from here:

ls /sys/bus/pci/devices

#some more general info for USB for confirmation

#for usb 2.0

dmesg | grep ehci_hcd

#e.g. ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 21 (level, low) -> IRQ 21

#for usb 1.1

dmesg | grep uhci_hcd

#e.g. uhci_hcd 0000:00:1d.3: irq 23, io base 0x0000ff20

#note down the pci address 0000:00:1d.7

# how I found the usb 1.1 UPS device:

dmesg | tail

#usb 2-1: USB disconnect, address 2

#usb 2-1: new low speed USB device using uhci_hcd and address 3

# regardless really use the device number from /sys/bus/pci/devices above

#now set the PCI devices to pass through

#this seems to be correct

#the 0/ seems to be the virtual bus number.. so from what I understand it should always be 0

#the below is a comma separated list of each PCI device to pass-through.. I passed through all of my USB controllers that I found

#the list could be just one device, like "other-config:pci=0/0000:00:1d.0".. I passed through all of them:

xe vm-param-set other-config:pci=0/0000:00:1d.0,0/0000:00:1d.1,0/0000:00:1d.2,0/0000:00:1d.3,0/0000:00:1d.7 uuid=YOURUID

#start up the VM again

xe vm-start uuid=YOURUID

#watch the xcp system logs as the machine boots to catch the pciback passthrough messages

tail -f /var/log/kern.log

#I received messages like "FLR functionality not supported"

#but apparently that shouldn't stop it from working

#bottom of this page talks about FLR http://wiki.xensource.com/xenwiki/XenPCIpassthrough

#FYI I have seen some information that I'm uncertain about that says you need to forward all the devices of a multifunction controller

#to get it to work in some cases...heads-up..

#this allows live attaching a pci device to the VM

#this also worked once I fixed up the linux kernel

xl list

# note down the "Name" for your VM.. use below...

# again, the PCI device number here is from "ls /sys/bus/pci/devices" (with "lspci -tv" helping you find the right one)

xl pci-attach [NAME] 0000:00:1d.7

# there's other handy xl commands..

# xl pci-detach

# xl pci-list [NAME]

Cheers,

Andrew

Andrew Eross
CTO
Locatrix Communications

eross@xxxxxxxxxxxx

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

WARNING - OLD ARCHIVES

xen-users

[Xen-users] XCP 1.0, Ubuntu 10.04, PCI Passthrough