[Xen-devel] [PATCH 0/5] pcie io space multiplexing for bootable pass through HVM domain

[Isaku Yamahata <yamahata@xxxxxxxxxxxxx>]

This patch series is for PCIe IO space multiplexing patch for commit.

It is not uncommon that a big iron for server consolidation has
many (e.g. > 16) PCIe slots. It will hold many domains, and
the administrator wants them to boot from pass through devices.
But currently up to 16 hvm domains can boot from pass through device.
This patch series address it by multiplexing PCI IO space access.

Usage:
Add the following options to dom0 kernel command line

  guestdev=<device path>+iomul
  (append "+iomul")
or
  guestiomuldev=[<segment>:]<bus>:<dev>[,[<segment:><bus>:dev]][,...]
  (In this case, don't forget to add related options. pciback.hide,
   reassign_resources or guestdev.)

Then dom0 Linux will allocate IO ports which are shared by specified devices.
And ioemu will automatically recognize IO-port-shared devices.
The unspecified devices will be treated same as before.
Note: Specifying unit to share IO port is PCI slot (device), on the other
      hand the unit of guestdev is function.
      If you specify a function to guestdev with "+iomul", all the
      functions of the given slot will share IO port even if you specify
      some of functions.


This patch series addresses the issue by multiplexing IO space access.
The patches are composed of
        Linux part: backport: preliminary patch
        Linux part: IO space ressignment code and multiplexing driver
        Linux part: guestdev kernel parameter support.
        Linux part: add kernel command line to reserve io/memory space
        xen part:   udev script for the driver
        ioemu part: make use of the PCIe io space multiplexing driver


Details:
PCI expansion ROM BIOS often uses IO port access to boot from its device
and Linux as dom0 exclusively assigns IO space to downstream PCI bridges
and the assignment unit of PCI bridge IO space is 4K. So the only up to
16 PCIe device can be accessed via IO space within 64K IO ports.
So on virtualized environment, it means only up to 16 guest domains
can boot from such pass-through devices.

The solution is to assign the same IO port region to pci devices
under same PCIe switch and disable IO bit in command register.
When accessing to one of IO port shared devices, the IO bit
of the device is enabled, and then issues IOIO.


Limitation:
- PCI devices or root complex integrated endpoints aren't supported.
- IO port of IO shared devices can't be accessed from dom0 Linux device
  driver.
  But those wouldn't be big issues because PCIe specification discourages
  the use of IO space and recommends that IO space should be used only
  for bootable device with ROM code. OS device driver should work without
  IO space access.

Test:
I don't have a machine with complicated PCIe topology nor many PCIe cards.
- PCI hotplug was tested with Linux fakephp.
- Only pci device (not bridge) hot plug/remove was tested.
- I have tested with only single multifunction PCIe

Changes from take2:
- rebased
- add kernel command line to reserve memory/io space for unused pci slot

change take 2:
- support PCI hotplug
- guestdev kernel paremeter.

thanks  

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel