[Xen-devel] Non-page aligned MMIO BAR error on dom-0 on Xen HVM setup

[Purna Chandar <purnachandar.midi@xxxxxxxxxxxxxxxxxx>]

Hi,
I am using Neterion x3100  multi-function NIC adapter on stable Xen(2.6.18) in HVM mode. I think the problem I have (explained below) is a general one and has got nothing to do with neterion card in specific.
I am trying to assign one of the functions on the adapter to a guest OS(FC11 64 bit) in pass-through mode. When I try to start the guest OS (using xm create), the dom-0 throws  the error saying the pass-through device has a "non-page aligned MMIO BAR" and cannot boot. The function infact has a BAR (bar 4) whose size is 256 bytes and the resources are mapped on a page-aligned boundary. THe guest cannot be booted for some reason because of this.

To get around with this, I used the following kernel command line option:

reassign_resources  guestdev=<b:d.f>,<b:d,f>... 

The BDFs of all the devices whose resources we want to re-assign to a page-boundary are given in guestdev parameter above. As these neterion functions are under a bridge, the bridge's BDF is also given in the guestdev option (without the bridge's BDF, I had issues with loading the neterion driver on guest OS).

With this kernel option, I am able to boot the guest OS but I see a different problem in the config space of the device in dom-0. The BAR regions are shown up as "disabled" in lspci output shown below. Without this option, the BARs are enabled.

01:00.2 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet PCIe (rev 01)
        Subsystem: Neterion Inc. X3110 Single Port 10GBase-CR                            


        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- 




        Interrupt: pin A routed to IRQ 16                                                                    
        Region 0: Memory at f9000000 (64-bit, prefetchable) [disabled] [size=8M]                             




        Region 2: Memory at fc404000 (64-bit, prefetchable) [disabled] [size=4K]                             
        Region 4: Memory at fc405000 (64-bit, prefetchable) [disabled] [size=256]                            




        [virtual] Expansion ROM at fc100000 [disabled] [size=512K]                       

What this "disabled" means is the resources are not treated as "memory mapped resources" at all (as bit-2 in COMMAND register is not set in config space). 

Surprisingly, on the guest OS, the "disabled" flag is not seen on the BAR regions. I can successfully load the driver and do some data flow on pass-through device on the guest. 

Things seem fine so far despite the problem above on dom-0.

But, when I trigger an FLR (function level reset) on the pass-through device/function on dom-0 (not on dom-u), I have problems on guest OS. The "disabled" bit appears in the lspci output on the device in the guest. The driver is not able to do any memory-mapped reads/writes and hence I am not able to load the driver. I triggered the FLR using following command:

# setpci -s 01:00.2 CAP_EXP+8.w=0xa814
(Before triggering FLR, value at CAP_EXP+8.w is 0x2814. As we have to set last bit in PCI Express capability for FLR, value used in above command is 0xa814).

Does anybody have similar issues? Is "reassign_resources" the correct kernel option to use? If not, how do I get around the "non-page alignment problem"

Any bit of help will be really helpful.

Thanks,
PurnaChandar M
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel