Re: [Xen-devel] Broadcom BCM5709 (bnx2) on Dell PowerEdge R610, Issues

[Guru Anbalagane <guru.anbalagane@xxxxxxxxxx>]

On 03/18/2011 12:46 PM, Dan Magenheimer wrote:
> cc'ing Guru...
>
>    
> > -----Original Message-----
> > From: Joshua West [mailto:jwest@xxxxxxxxxxxx]
> > Sent: Friday, March 18, 2011 1:25 PM
> > To: xen-devel@xxxxxxxxxxxxxxxxxxx
> > Subject: Re: [Xen-devel] Broadcom BCM5709 (bnx2) on Dell PowerEdge
> > R610, Issues
> >
> > Hi Guru,
> >
> > Awesome, thanks for the tip.
> >
> > I'll test out disabling cstates in the BIOS as I don't believe Xen
> > 3.4.x
> > lets you set max_cstate as an argument to xen.gz in grub.conf.
> >      
I believe max_cstate=1 works at xen command line.
Another way to set cstate is:
xenpm set-max-cstate 1
> > The patch in the changeset you mention applies to Xen 3.4.3 code.  Do
> > you have an experience with that patch functioning/helping/working with
> > Xen 3.4.x?  And if so, do you think it will end up as part of Xen 3.4.4
> > (if that ever gets tagged/released)?  Assuming disabling cstates in the
> > BIOS alleviates my problem, I'll probably give that patch a whirl with
> > cstates enabled and see if the issue comes back.  Just wondering if
> > anybody else has used that patch with Xen 3.4.3 and found success.
> >      
For Oracle VM, it applies but need to pull in few more patches to make 
it work and its not verified yet.
So for now, We recommend disabling cstates or limit to 1  or 3.


Thanks
Guru
> > Thanks.
> >
> > On 03/18/11 11:53, Guru Anbalagane wrote:
> >      
> > > This is likely related xen losing interrupts while certain cpus goes
> > > to c6 state.
> > > The below patch addresses an issue around this.
> > > http://xenbits.xen.org/hg/xen-unstable.hg/rev/1087f9a03ab6
> > > Easy workaround would be to turn off cstates in BIOS or limit cstate
> > > in xen.
> > >
> > > Hope this helps.
> > > Thanks
> > > Guru
> > >        
> > > > Message: 5
> > > > Date: Fri, 18 Mar 2011 11:39:07 -0400
> > > > From: Joshua West<jwest@xxxxxxxxxxxx>
> > > > Subject: [Xen-devel] Broadcom BCM5709 (bnx2) on Dell PowerEdge R610
> > > >      Issues
> > > > To: xen-devel@xxxxxxxxxxxxxxxxxxx
> > > > Message-ID:<4D837C9B.6030107@xxxxxxxxxxxx>
> > > > Content-Type: text/plain; charset="iso-8859-1"
> > > >
> > > > Hey folks,
> > > >
> > > > Unfortunately, ever since we went live with Xen on Dell PowerEdge
> > > > R610's, we've been having some odd and aggravating issues.  The
> > > >          
> > NIC's
> >      
> > > > tend to drop out when under heavy traffic after 1-7 days of uptime
> > > > (random, difficult to reproduce).  But before I get into the issue's
> > > > specifics, here's some information about our setup:
> > > >
> > > >      * Dell PowerEdge R610's w/ 4 Onboard Broadcom BCM5709 1-GbE
> > > >          
> > NIC's.
> >      
> > > >      * RHEL 5.6.
> > > >      * Xen 3.4.3 (from xen.org; our own compile)
> > > >      * Kernel 2.6.18.18
> > > > (http://xenbits.xensource.com/linux-2.6.18-xen.hg)
> > > > checkout 1073.
> > > >      * bnx2 driver 2.0.18c from Broadcom's netxtreme2-6.0.53 package.
> > > >        * bnx2 that ships with 2.6.18.8 doesn't support BCM5709's.
> > > >        * Had to use driver package from broadcom.com in order to get
> > > > networking.
> > > >      * NIC bonding in pairs (eth0 + eth1, etc), with options "mode=4
> > > > lacp_rate=fast miimon=100 use_carrier=1".
> > > >
> > > > What occurs is suddenly one of the NIC's in the bond stops
> > > >          
> > responding.
> >      
> > > > Gets stuck on transmitting from what I understand.  Kernel logs show
> > > >          
> > the
> >      
> > > > following, which includes extra debug information as the developers
> > > >          
> > from
> >      
> > > > Broadcom (Michael Chan and Benjamin Li) were assisting in
> > > > troubleshooting and gave me a version of bnx2 2.0.18c to run, that
> > > > prints out extra debug information upon NIC crash:
> > > >
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: NETDEV WATCHDOG: eth0: transmit
> > > > timed out
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2:<--- start FTQ dump on eth0
> > > > --->
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_RV2P_PFTQ_CTL
> > > >          
> > 10000
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_RV2P_TFTQ_CTL
> > > >          
> > 20000
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_RV2P_MFTQ_CTL
> > > >          
> > 4000
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_TBDR_FTQ_CTL
> > > >          
> > 4002
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_TDMA_FTQ_CTL
> > > >          
> > 10002
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_TXP_FTQ_CTL
> > > >          
> > 10002
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_TPAT_FTQ_CTL
> > > >          
> > 10000
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_RXP_CFTQ_CTL
> > > >          
> > 8000
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_RXP_FTQ_CTL
> > > >          
> > 100000
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0:
> > > >          
> > BNX2_COM_COMXQ_FTQ_CTL
> >      
> > > > 10000
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0:
> > > >          
> > BNX2_COM_COMTQ_FTQ_CTL
> >      
> > > > 20000
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0:
> > > >          
> > BNX2_COM_COMQ_FTQ_CTL
> >      
> > > > 10000
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: BNX2_CP_CPQ_FTQ_CTL
> > > >          
> > 4000
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: TXP mode b84c state
> > > > 80001000 evt_mask 500 pc 8001284 pc 8001284 instr 1440fffc
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: TPAT mode b84c state
> > > > 80001000 evt_mask 500 pc 8000a50 pc 8000a4c instr 38420001
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: RXP mode b84c state
> > > > 80001000 evt_mask 500 pc 8004ad0 pc 8004adc instr 14e0005d
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: COM mode b8cc state
> > > > 80008000 evt_mask 500 pc 8000a98 pc 8000a8c instr 8821
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0: CP mode b8cc state
> > > > 80000000 evt_mask 500 pc 8000c7c pc 8000928 instr 8ce800e8
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2:<--- end FTQ dump on eth0 -
> > > >          
> > -->
> >      
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0 DEBUG: intr_sem[0]
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0 DEBUG: intr_sem[0]
> > > > PCI_CMD[00100406]
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0 DEBUG:
> > > >          
> > PCI_PM[19002008]
> >      
> > > > PCI_MISC_CFG[92000088]
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0 DEBUG:
> > > > EMAC_TX_STATUS[00000008] EMAC_RX_STATUS[00000000]
> > > > Mar 18 01:40:26 xen-san-gb1 kernel: bnx2: eth0
> > > > RPM_MGMT_PKT_CTRL[40000088]
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: bnx2: eth0 DEBUG:
> > > > MCP_STATE_P0[0003610e] MCP_STATE_P1[0003610e]
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: bnx2: eth0 DEBUG:
> > > > HC_STATS_INTERRUPT_STATUS[01fe0001]
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: Ring state for ring 0 napi state
> > > >          
> > 12
> >      
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: netdev state 7
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: hw status idx 3267 last status
> > > >          
> > idx
> >      
> > > > 307c irq jiffies 100759890
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: hw tx cons a669 hw rx cons 103c
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: sw tx cons a57c a57c prod a669
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: sw rx cons f3c prod 103c
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: Current jiffies 1008f4741 HZ fa
> > > >          
> > tx
> >      
> > > > 1008f41e2 poll 100759890
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: tx stop jiffies 1008f41e2 tx
> > > >          
> > start
> >      
> > > > jiffies 0
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: irq_event c68c36 napi_event
> > > >          
> > c68c37
> >      
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: Ring state for ring 0 napi state
> > > >          
> > 12
> >      
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: netdev state 77
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: hw status idx 3267 last status
> > > >          
> > idx
> >      
> > > > 307c irq jiffies 100759890
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: hw tx cons a669 hw rx cons 103c
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: sw tx cons a57c a57c prod a669
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: sw rx cons f3c prod 103c
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: Current jiffies 1008f4741 HZ fa
> > > >          
> > tx
> >      
> > > > 1008f41e2 poll 100759890
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: tx stop jiffies 1008f41e2 tx
> > > >          
> > start
> >      
> > > > jiffies 0
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: irq_event c68c36 napi_event
> > > >          
> > c68c37
> >      
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: bnx2: eth0 NIC Copper Link is
> > > >          
> > Down
> >      
> > > > Mar 18 01:40:27 xen-san-gb1 kernel: bonding: bond0: link status
> > > > definitely down for interface eth0, disabling it
> > > >
> > > > This was then followed rather quickly by a failure with the second
> > > >          
> > NIC
> >      
> > > > (eth1) in the bond:
> > > >
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: NETDEV WATCHDOG: eth1: transmit
> > > > timed out
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2:<--- start FTQ dump on eth1
> > > > --->
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_RV2P_PFTQ_CTL
> > > >          
> > 10000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_RV2P_TFTQ_CTL
> > > >          
> > 20000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_RV2P_MFTQ_CTL
> > > >          
> > 4000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_TBDR_FTQ_CTL
> > > >          
> > 4002
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_TDMA_FTQ_CTL
> > > >          
> > 10000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_TXP_FTQ_CTL
> > > >          
> > 10002
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_TPAT_FTQ_CTL
> > > >          
> > 10000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_RXP_CFTQ_CTL
> > > >          
> > 8000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_RXP_FTQ_CTL
> > > >          
> > 100000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1:
> > > >          
> > BNX2_COM_COMXQ_FTQ_CTL
> >      
> > > > 10000
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1:
> > > >          
> > BNX2_COM_COMTQ_FTQ_CTL
> >      
> > > > 20000
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1:
> > > >          
> > BNX2_COM_COMQ_FTQ_CTL
> >      
> > > > 10000
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: BNX2_CP_CPQ_FTQ_CTL
> > > >          
> > 4000
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: TXP mode b84c state
> > > > 80005000 evt_mask 500 pc 8001294 pc 8001284 instr 38640001
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: TPAT mode b84c state
> > > > 80001000 evt_mask 500 pc 8000a58 pc 8000a5c instr 8f820014
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: RXP mode b84c state
> > > > 80001000 evt_mask 500 pc 8004ad0 pc 8004adc instr 14e0005d
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: COM mode b8cc state
> > > > 80000000 evt_mask 500 pc 8000a9c pc 8000a94 instr 3c028000
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1: CP mode b8cc state
> > > > 80008000 evt_mask 500 pc 8000c58 pc 8000c6c instr 27bdffe8
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2:<--- end FTQ dump on eth1 -
> > > >          
> > -->
> >      
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1 DEBUG: intr_sem[0]
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1 DEBUG: intr_sem[0]
> > > > PCI_CMD[00100406]
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1 DEBUG:
> > > >          
> > PCI_PM[19002008]
> >      
> > > > PCI_MISC_CFG[92000088]
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1 DEBUG:
> > > > EMAC_TX_STATUS[00000008] EMAC_RX_STATUS[00000000]
> > > > Mar 18 01:42:26 xen-san-gb1 kernel: bnx2: eth1
> > > > RPM_MGMT_PKT_CTRL[40000088]
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: bnx2: eth1 DEBUG:
> > > > MCP_STATE_P0[0003610e] MCP_STATE_P1[0003610e]
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: bnx2: eth1 DEBUG:
> > > > HC_STATS_INTERRUPT_STATUS[01fe0001]
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: Ring state for ring 0 napi state
> > > >          
> > 12
> >      
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: netdev state 7
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: hw status idx 2bb0 last status
> > > >          
> > idx
> >      
> > > > 29c4 irq jiffies 100759898
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: hw tx cons e421 hw rx cons a8ce
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: sw tx cons e334 e334 prod e421
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: sw rx cons a7ce prod a8ce
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: Current jiffies 1008fbc71 HZ fa
> > > >          
> > tx
> >      
> > > > 1008fb744 poll 100759898
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: tx stop jiffies 1008fb744 tx
> > > >          
> > start
> >      
> > > > jiffies 100239dfd
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: irq_event ab2e13 napi_event
> > > >          
> > ab2e14
> >      
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: Ring state for ring 0 napi state
> > > >          
> > 12
> >      
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: netdev state 77
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: hw status idx 2bb0 last status
> > > >          
> > idx
> >      
> > > > 29c4 irq jiffies 100759898
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: hw tx cons e421 hw rx cons a8ce
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: sw tx cons e334 e334 prod e421
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: sw rx cons a7ce prod a8ce
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: Current jiffies 1008fbc72 HZ fa
> > > >          
> > tx
> >      
> > > > 1008fb744 poll 100759898
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: tx stop jiffies 1008fb744 tx
> > > >          
> > start
> >      
> > > > jiffies 100239dfd
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: irq_event ab2e13 napi_event
> > > >          
> > ab2e14
> >      
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: bnx2: eth1 NIC Copper Link is
> > > >          
> > Down
> >      
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: bonding: bond0: link status
> > > > definitely down for interface eth1, disabling it
> > > > Mar 18 01:42:27 xen-san-gb1 kernel: bonding: bond0: Warning: No
> > > >          
> > 802.3ad
> >      
> > > > response from the link partner for any adapters in the bond
> > > >
> > > > Onto more technical details...
> > > >
> > > > The kernel we were running (2.6.18.8 from xenbits) was compiled
> > > >          
> > without
> >      
> > > > support for MSI/MSI-X originally.  So, we were experiencing these
> > > > problems with plain standard IRQ's.  Michael Chan @ Broadcom, the
> > > >          
> > author
> >      
> > > > of bnx2 if you modinfo, has told me via email:
> > > >
> > > >      * "The logs show that we haven't had an interrupt for a very
> > > >          
> > long
> >      
> > > > time. It's not clear how that interrupt was lost."
> > > >      * "So far the logs don't show any inconsistent state in the
> > > >          
> > hardware
> >      
> > > > or software. It is possible that the Xen kernel is missing an
> > > >          
> > interrupt
> >      
> > > > and not delivering to the driver. Normally, in INTA mode, the IRQ is
> > > > level triggered and should remain asserted until it is seen by the
> > > > driver and de-asserted by the driver."
> > > >
> > > > But, just in case, I compiled 2.6.18.8 with support for MSI/MSI-X
> > > >          
> > and
> >      
> > > > was able to confirm (via dmesg and lspci -vv) that the NIC's began
> > > >          
> > to
> >      
> > > > use MSI for interrupts.  Unfortunately, the NIC crash happened
> > > >          
> > anyways
> >      
> > > > (the above kernel logs is actually from when running with MSI).
> > > >
> > > > Here's whats really bugging me.  We have a Dell PowerEdge R610,
> > > >          
> > running
> >      
> > > > Xen along with the bnx2 drivers from Broadcom, thats been online for
> > > > ~220 days.  Without a failure.  The only difference is the system is
> > > >          
> > not
> >      
> > > > making use of bonding.  It has just one NIC connected to the network
> > > > with no VLAN's trunked down etc.
> > > >
> > > > It looks like I'm not alone out there, as there's a Red Hat bugzilla
> > > > report for this issue:
> > > >
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=520888
> > > >
> > > > ^^ The above has an indication of *Status
> > > > <https://bugzilla.redhat.com/page.cgi?id=fields.html#status>*:
> > > >          
> > CLOSED
> >      
> > > > DUPLICATE of bug 511368
> > > > <https://bugzilla.redhat.com/show_bug.cgi?id=511368>   , but looks
> > > >          
> > like I
> >      
> > > > don't have access to view 511368. Grrr.
> > > >
> > > > Anyways...
> > > >
> > > > 1) Has anybody else experienced this issue?
> > > > 2) Any developers care to comment on possible causes of this
> > > >          
> > problem?
> >      
> > > > 3) Anybody know of a solution?
> > > > 4) What can I do to troubleshoot further, and get developers
> > > >          
> > necessary
> >      
> > > > information?
> > > >
> > > > Lastly...
> > > >
> > > > 5) Is anybody running Intel NIC's within Dell PowerEdge R610's,
> > > >          
> > using
> >      
> > > > bonding + Xen 3.4.3 + 2.6.18.8, and can safely report success?  I
> > > >          
> > may
> >      
> > > > switch to Intel...
> > > >
> > > > Thanks!
> > > >
> > > >          
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > > http://lists.xensource.com/xen-devel
> > >        
> > --
> > Joshua West
> > Senior Systems Engineer
> > Brandeis University
> > http://www.brandeis.edu
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
> >      
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel