WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Fedora-xen] [Xen-devel] Re: [Xen-users] Continuing BUG:-iness booti

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Subject: Re: [Fedora-xen] [Xen-devel] Re: [Xen-users] Continuing BUG:-iness booting Fedora Rawhide 3.1.0-rc's (was Summary: Experiences setting up a debug serial port)
From: jim burns <jim_burn@xxxxxxxxxxxxx>
Date: Fri, 16 Sep 2011 10:55:38 -0400
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx, Fedora Xen List <fedora-xen@xxxxxxxxxx>, pasik@xxxxxx, Ian Campbell <Ian.Campbell@xxxxxxxxxxxxx>, M A Young <m.a.young@xxxxxxxxxxxx>, xen-users@xxxxxxxxxxxxxxxxxxx
Delivery-date: Fri, 16 Sep 2011 07:57:29 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bellsouth.net; s=s1024; t=1316184951; bh=QgfeQD/O4INuAHPPudPlWxSgA8N13kk7dfx55REuP0w=; h=X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:From:To:Cc:Subject:Date:Message-ID:User-Agent:In-Reply-To:References:MIME-Version:Content-Transfer-Encoding:Content-Type; b=O+9T1vLM+SLSfrDhNtnaIwOmNHc7Tbjw1G6SPOnisO5fbuESpfWs5iEHF5VHMTdncT61wyc/9fad6s3oP4MX3pgGW5ckbp1bMN3q+VBRvsLBkWwnjvepSymEseLCyACpottoyw0J5ulHF5F8aWKGRGTe6ljjWL9Cl4q9wsRXEaU=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20110915075358.GA4396@xxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <201107231426.56811.jim_burn@xxxxxxxxxxxxx> <alpine.LFD.2.02.1109150027020.22927@xxxxxxxxxxxxxxx> <20110915075358.GA4396@xxxxxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: KMail/4.7.1 (Linux/2.6.37.6-0.7-default; KDE/4.7.1; i686; ; )
On Thu September 15 2011, 3:53:58 AM, Konrad Rzeszutek Wilk wrote:
> On Wed, Sep 14, 2011 at 09:18:25PM -0400, jim burns wrote:
> > On Thu September 15 2011, 12:38:26 AM, M A Young wrote:
> > > I have a (temporary) F17 kernel with the patch from 
> > > 
http://oss.oracle.com/git/kwilk/xen.git/?p=kwilk/xen.git;a=commit;h=a7079a64
> > > 04ed2106315327fff6be3464d10814e7  (which I assume is the same patch)
> > > applied building at
> > > http://koji.fedoraproject.org/koji/taskinfo?taskID=3352367
> > > if you want to test it.
> > 
> > Oh - the patch is to the kernel, not to xen. I misread it.
> > 
> > Ok - I downloaded 
> > 
http://koji.fedoraproject.org/koji/getfile?taskID=3352368&name=kernel-3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64.rpm
> > 
> > I hope that's right. I'll try it tomorrow. Thanx.
> 
> ping?

Echo reply - router congestion.

Testing out myoung's 3.1.0 was not as straight forward as I had hoped. It did 
boot up without any BUG:s, but I did get the occasional Lock Order message. 
Log snippet at the end of the post. It doesn't seem to be directly related to 
starting guests.

The real problem comes in starting up guests. Performance is very bad. I knew 
from working with rawhide 3.0.0 (long since replaced) that performance would 
suffer - rawhide kernels are debug kernels:

jimb@insp6400 09/16/11 10:16AM:~
[511] > grep DEBUG /boot/config-3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64|grep -v 
'is not set'|wc -l
91
jimb@insp6400 09/16/11 10:16AM:~
[512] > grep DEBUG /boot/config-2.6.40.4-5.fc15.x86_64|grep -v 'is not set'|wc 
-l           
54
jimb@insp6400 09/16/11 10:18AM:~
[513] > grep DEBUG /boot/config-3.0.1-3.fc16.x86_64|grep -v 'is not set'|wc -l 
90

Starting guests is much slower under myoung's 3.1.0 than under rawhide's 3.1.0 
or 3.0.{0,1}. A cifs backed pv domu took 6 min. for 'xm create' to exit, 
during most of which, dom0 was totally unresponsive, and another 2 min. to get 
to the login screen. The 2nd attempt took 4 min. to get to the login screen - 
presumably the cifs file backed domu was still partially cached. (When I say 
unresponsive, I mean userland. I have a client computer with an iscsi target, 
and cifs/nfs connections to the dom0, and I was still getting slow traffic 
(presumably interrupt based, keep alive signals), and the dom0 hard disk light 
was constantly blinking.)

Starting a winxp domu was much worse. 'xm create' totally locked up my dom0, 
so that I had to reboot. One time, I took a nap after starting the domu, and 
got up 4 hrs. later, and dom0 was still locked up.

'xm create' did not  provide any diagnostic information. 'xl create' was a 
little more chatty:

root@insp6400 09/16/11 12:09AM:~
[544] > xl create Documents/winxp; brctl show;  ps -A|grep qemu; netstat -tlp|
grep 59; renice -11 `pidof qemu-dm`;  ps -A|grep vncv; ifconfig vif1.0 mtu 
9000
Parsing config file Documents/winxp
xc: info: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->000000000017b270
  TOTAL:         0000000000000000->000000003fc00000
  ENTRY ADDRESS: 00000000001015a0
xc: error: Could not allocate memory for HVM guest. (16 = Device or resource 
busy): Internal error
libxl: error: libxl_dom.c:284:libxl__build_hvm hvm building failed

and my serial debug log had several:

(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (1 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=9 extent: id=1 memflags=0 (0 of 
4)
(XEN) memory.c:133:d0 Could not allocate order=0 extent: id=1 memflags=0 (439 
of 2048)

Then I remembered that I recently upped the memory allocation for my winxp 
domu, from 512 to 768. This works fine under 2.6.40, the f15 non-debug 
production kernel. None the less, I knocked the allocation back down to 512, 
and my winxp domu did start up, getting to the qemu splash screen in about 2 - 
3 min., during part of which dom0 was unresponsive. However, I'm still getting 
the '(XEN) memory.c' errors, and some frequent GPF errors (a few a min.) in my 
serial debug log:

(XEN) traps.c:2956: GPF (0060): ffff82c48015354a -> ffff82c480200131

Then, rawhide and gplpv don't get along. Specifically, the xennet receive side 
driver stops working, and I have to fall back to qemu emulation. It takes 
about an hour for the winxp desktop to finish initializing, with dom0 cpu load 
on one cpu core at 72% - yum! But I'll just have to live with it - it's not 
your problem. I'll leave it up for at least a day to see if any other messages 
pop up.

Hope this was of some help.

Lock Order /var/log/messages snippet:

Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
======================================================
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] [ INFO: HARDIRQ-safe -> 
HARDIRQ-unsafe lock order detected ]
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
------------------------------------------------------
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] swapper/0 
[HC0[0]:SC1[3]:HE0:SE0] is trying to acquire:
Sep 16 01:10:19 Insp6400 kernel: [  473.537933]  (nf_conntrack_lock){+.-...}, 
at: [<ffffffffa036a7b7>] destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] and this task is already 
holding:
Sep 16 01:10:19 Insp6400 kernel: [  473.537933]  (&(&bp->lock)->rlock)
{-.-.-.}, at: [<ffffffffa0287a6a>] b44_poll+0x28/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] which would create a new lock 
dependency:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (&(&bp->lock)->rlock){-.-.-.} 
-> (nf_conntrack_lock){+.-...}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] but this new dependency 
connects a HARDIRQ-irq-safe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (&(&bp->lock)->rlock){-.-.-.}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ... which became HARDIRQ-irq-
safe at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108e176>] 
__lock_acquire+0x2ce/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff815044fd>] 
_raw_spin_lock+0x45/0x79
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa0285725>] 
b44_interrupt+0x25/0xd9 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c2958>] 
handle_irq_event_percpu+0xb4/0x271
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c2b5c>] 
handle_irq_event+0x47/0x67
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c4ef0>] 
handle_fasteoi_irq+0x8a/0xb0
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812d986f>] 
__xen_evtchn_do_upcall+0x15e/0x203
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812daf65>] 
xen_evtchn_do_upcall+0x2c/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] to a HARDIRQ-irq-unsafe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (nf_conntrack_lock){+.-...}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ... which became HARDIRQ-irq-
unsafe at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ...  [<ffffffff8108e1ed>] 
__lock_acquire+0x345/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa036bf48>] 
nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa03985ff>] 
ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8143c97d>] 
nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8143ca39>] 
nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8144621f>] 
NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff814468d9>] 
ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81415df5>] 
__netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81415ed8>] 
process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] other info that might help us 
debug this:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  Possible interrupt unsafe 
locking scenario:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]        CPU0                    
CPU1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]        ----                    
----
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   lock(nf_conntrack_lock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
local_irq_disable();
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
lock(&(&bp->lock)->rlock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
lock(nf_conntrack_lock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   <Interrupt>
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]     lock(&(&bp->lock)->rlock);
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
======================================================
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] [ INFO: HARDIRQ-safe -> 
HARDIRQ-unsafe lock order detected ]
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
------------------------------------------------------
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] swapper/0 
[HC0[0]:SC1[3]:HE0:SE0] is trying to acquire:
Sep 16 01:10:19 Insp6400 kernel: [  473.537933]  (nf_conntrack_lock){+.-...}, 
at: [<ffffffffa036a7b7>] destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] and this task is already 
holding:
Sep 16 01:10:19 Insp6400 kernel: [  473.537933]  (&(&bp->lock)->rlock)
{-.-.-.}, at: [<ffffffffa0287a6a>] b44_poll+0x28/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] which would create a new lock 
dependency:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (&(&bp->lock)->rlock){-.-.-.} 
-> (nf_conntrack_lock){+.-...}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] but this new dependency 
connects a HARDIRQ-irq-safe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (&(&bp->lock)->rlock){-.-.-.}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ... which became HARDIRQ-irq-
safe at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108e176>] 
__lock_acquire+0x2ce/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff815044fd>] 
_raw_spin_lock+0x45/0x79
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa0285725>] 
b44_interrupt+0x25/0xd9 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c2958>] 
handle_irq_event_percpu+0xb4/0x271
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c2b5c>] 
handle_irq_event+0x47/0x67
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c4ef0>] 
handle_fasteoi_irq+0x8a/0xb0
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812d986f>] 
__xen_evtchn_do_upcall+0x15e/0x203
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812daf65>] 
xen_evtchn_do_upcall+0x2c/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] to a HARDIRQ-irq-unsafe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (nf_conntrack_lock){+.-...}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ... which became HARDIRQ-irq-
unsafe at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ...  [<ffffffff8108e1ed>] 
__lock_acquire+0x345/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa036bf48>] 
nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa03985ff>] 
ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8143c97d>] 
nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8143ca39>] 
nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8144621f>] 
NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff814468d9>] 
ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81415df5>] 
__netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81415ed8>] 
process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] other info that might help us 
debug this:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  Possible interrupt unsafe 
locking scenario:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]        CPU0                    
CPU1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]        ----                    
----
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   lock(nf_conntrack_lock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
local_irq_disable();
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
lock(&(&bp->lock)->rlock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
lock(nf_conntrack_lock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   <Interrupt>
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]     lock(&(&bp->lock)->rlock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  *** DEADLOCK ***
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 2 locks held by swapper/0:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  #0:  (&(&bp->lock)->rlock)
{-.-.-.}, at: [<ffffffffa0287a6a>] b44_poll+0x28/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  #1:  (rcu_read_lock){.+.+..}, 
at: [<ffffffff8143c558>] rcu_read_lock+0x0/0x44
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] the dependencies between 
HARDIRQ-irq-safe lock and the holding lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] -> (&(&bp->lock)->rlock)
{-.-.-.} ops: 27161 {
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-HARDIRQ-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e176>] __lock_acquire+0x2ce/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff815044fd>] _raw_spin_lock+0x45/0x79
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa0285725>] b44_interrupt+0x25/0xd9 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff810c2958>] handle_irq_event_percpu+0xb4/0x271
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff810c2b5c>] handle_irq_event+0x47/0x67
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff810c4ef0>] handle_fasteoi_irq+0x8a/0xb0
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812d986f>] __xen_evtchn_do_upcall+0x15e/0x203
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf65>] xen_evtchn_do_upcall+0x2c/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-SOFTIRQ-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e19d>] __lock_acquire+0x2f5/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81504612>] _raw_spin_lock_irq+0x4f/0x82
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa02868a2>] b44_timer+0x13/0x51 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8106a83c>] run_timer_softirq+0x218/0x372
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-RECLAIM_FS-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff8108e24a>] __lock_acquire+0x3a2/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff815044fd>] _raw_spin_lock+0x45/0x79
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffffa0285725>] b44_interrupt+0x25/0xd9 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff810c2958>] handle_irq_event_percpu+0xb4/0x271
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff810c2b5c>] handle_irq_event+0x47/0x67
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff810c4ef0>] handle_fasteoi_irq+0x8a/0xb0
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff812d986f>] __xen_evtchn_do_upcall+0x15e/0x203
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff812daf65>] xen_evtchn_do_upcall+0x2c/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    INITIAL USE at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108e262>] __lock_acquire+0x3ba/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81504612>] _raw_spin_lock_irq+0x4f/0x82
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa0286637>] b44_set_mac_addr+0x5a/0x82 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8141488d>] dev_set_mac_address+0x3e/0x58
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa04a6985>] bond_enslave+0x3e6/0xa48 [bonding]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa04ad458>] bonding_store_slaves+0x106/0x174 [bonding]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff813138c4>] dev_attr_store+0x20/0x22
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff811a0d5d>] sysfs_write_file+0x108/0x144
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81142bda>] vfs_write+0xaf/0xf6
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81142dd5>] sys_write+0x4d/0x74
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150bc82>] system_call_fastpath+0x16/0x1b
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  }
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... key      at: 
[<ffffffffa028b448>] __key.37034+0x0/0xffffffffffffdbb8 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... acquired at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa036a7b7>] 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] the dependencies between the 
lock to be acquired and HARDIRQ-irq-unsafe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] -> (nf_conntrack_lock){+.-...} 
ops: 255 {
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    HARDIRQ-ON-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e1ed>] __lock_acquire+0x345/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81504846>] _raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa036bf48>] nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa03985ff>] ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143c97d>] nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143ca39>] nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8144621f>] NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff814468d9>] ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415df5>] __netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415ed8>] process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81417c0a>] net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-SOFTIRQ-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e19d>] __lock_acquire+0x2f5/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81504846>] _raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa036bf48>] nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa03985ff>] ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143c97d>] nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143ca39>] nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8144621f>] NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff814468d9>] ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415df5>] __netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415ed8>] process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81417c0a>] net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    INITIAL USE at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108e262>] __lock_acquire+0x3ba/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81504846>] _raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa036bf48>] nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa03985ff>] ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8143c97d>] nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8143ca39>] nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8144621f>] NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff814468d9>] ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81415df5>] __netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81415ed8>] process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81417c0a>] net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  }
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... key      at: 
[<ffffffffa0376018>] nf_conntrack_lock+0x18/0xffffffffffffd000 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... acquired at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa036a7b7>] 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] stack backtrace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Pid: 0, comm: swapper Not 
tainted 3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Call Trace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <IRQ>  [<ffffffff8108db25>] 
check_usage+0x37f/0x394
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d72>] ? 
check_events+0x12/0x20
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a45d>] ? 
rcu_read_lock+0x44/0x44 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007746>] ? 
xen_force_evtchn_callback+0xd/0xf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d72>] ? 
check_events+0x12/0x20
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007746>] ? 
xen_force_evtchn_callback+0xd/0xf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d5f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    INITIAL USE at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108e262>] __lock_acquire+0x3ba/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81504846>] _raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa036bf48>] nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa03985ff>] ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8143c97d>] nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8143ca39>] nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8144621f>] NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff814468d9>] ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81415df5>] __netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81415ed8>] process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81417c0a>] net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  }
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... key      at: 
[<ffffffffa0376018>] nf_conntrack_lock+0x18/0xffffffffffffd000 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... acquired at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa036a7b7>] 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] stack backtrace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Pid: 0, comm: swapper Not 
tainted 3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Call Trace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <IRQ>  [<ffffffff8108db25>] 
check_usage+0x37f/0x394
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d72>] ? 
check_events+0x12/0x20
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a45d>] ? 
rcu_read_lock+0x44/0x44 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007746>] ? 
xen_force_evtchn_callback+0xd/0xf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d72>] ? 
check_events+0x12/0x20
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007746>] ? 
xen_force_evtchn_callback+0xd/0xf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d5f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a747>] ? 
nf_conntrack_free+0x58/0x58 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062a9a>] ? 
__do_softirq+0x7e/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <EOI>  [<ffffffff810013aa>] ? 
hypercall_page+0x3aa/0x1000
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff810013aa>] ? 
hypercall_page+0x3aa/0x1000
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007701>] ? 
xen_safe_halt+0x10/0x18
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8101601e>] ? 
default_idle+0x53/0x90
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8100e2f9>] ? 
cpu_idle+0xb5/0x101
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff814e0318>] ? 
rest_init+0xdc/0xe3
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff814e023c>] ? 
csum_partial_copy_generic+0x16c/0x16c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d53b9f>] ? 
start_kernel+0x3dd/0x3ea
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d532c4>] ? 
x86_64_start_reservations+0xaf/0xb3
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d55f18>] ? 
xen_start_kernel+0x588/0x58f
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ------------[ cut here 
]------------
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] WARNING: at 
kernel/softirq.c:159 _local_bh_enable_ip+0x49/0xce()
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Hardware name: MM061            
               
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Modules linked in: fuse 
ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM 
iptable_mangle xen_pciback nfsd lockd nfs_acl auth_rpcgss sunrpc des_generic 
md4 nls_utf8 cifs fscache bridge stp llc bonding be2iscsi iscsi_boot_sysfs 
bnx2i cnic uio cxgb3i libcxgbi iw_cxgb3 cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm 
xt_comment ib_sa ib_mad ib_core ip6t_REJECT ib_addr nf_conntrack_ipv4 
nf_defrag_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 iscsi_tcp xt_state 
libiscsi_tcp nf_conntrack libiscsi ip6table_filter ipt_LOG 
scsi_transport_iscsi ip6_tables xt_physdev dell_wmi sparse_keymap 
snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device 
arc4 r852 sm_common nand snd_pcm dell_laptop microcode dcdbas b44 nand_ids 
r592 nand_ecc iwl3945 ssb memstick mtd mii iwl_legacy iTCO_wdt mac80211 
iTCO_vendor_support i2c_i801 joydev cfg80211 snd_timer rfkill snd soundcore 
snd_page_alloc tun xen_gntalloc xen_netback xen_blkback xen_gntdev xen_evtchn 
xenfs binfmt_misc
Sep 16 01:10:19 Insp6400 kernel: sdhci_pci sdhci mmc_core firewire_ohci 
firewire_core crc_itu_t wmi i915 drm_kms_helper drm i2c_algo_bit i2c_core 
video [last unloaded: scsi_wait_scan]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Pid: 0, comm: swapper Not 
tainted 3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Call Trace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <IRQ>  [<ffffffff8105c4a0>] 
warn_slowpath_common+0x83/0x9b
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7f4>] ? 
destroy_conntrack+0xad/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8105c4d2>] 
warn_slowpath_null+0x1a/0x1c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062598>] 
_local_bh_enable_ip+0x49/0xce
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8106262b>] 
local_bh_enable_ip+0xe/0x10
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81504df9>] 
_raw_spin_unlock_bh+0x40/0x44
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7f4>] 
destroy_conntrack+0xad/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a747>] ? 
nf_conntrack_free+0x58/0x58 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062a9a>] ? 
__do_softirq+0x7e/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
======================================================
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] [ INFO: HARDIRQ-safe -> 
HARDIRQ-unsafe lock order detected ]
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
------------------------------------------------------
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] swapper/0 
[HC0[0]:SC1[3]:HE0:SE0] is trying to acquire:
Sep 16 01:10:19 Insp6400 kernel: [  473.537933]  (nf_conntrack_lock){+.-...}, 
at: [<ffffffffa036a7b7>] destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] 
Sep 16 01:10:19 Insp6400 kernel: [  473.537933] and this task is already 
holding:
Sep 16 01:10:19 Insp6400 kernel: [  473.537933]  (&(&bp->lock)->rlock)
{-.-.-.}, at: [<ffffffffa0287a6a>] b44_poll+0x28/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] which would create a new lock 
dependency:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (&(&bp->lock)->rlock){-.-.-.} 
-> (nf_conntrack_lock){+.-...}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] but this new dependency 
connects a HARDIRQ-irq-safe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (&(&bp->lock)->rlock){-.-.-.}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ... which became HARDIRQ-irq-
safe at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108e176>] 
__lock_acquire+0x2ce/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff815044fd>] 
_raw_spin_lock+0x45/0x79
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa0285725>] 
b44_interrupt+0x25/0xd9 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c2958>] 
handle_irq_event_percpu+0xb4/0x271
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c2b5c>] 
handle_irq_event+0x47/0x67
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff810c4ef0>] 
handle_fasteoi_irq+0x8a/0xb0
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812d986f>] 
__xen_evtchn_do_upcall+0x15e/0x203
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812daf65>] 
xen_evtchn_do_upcall+0x2c/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] to a HARDIRQ-irq-unsafe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  (nf_conntrack_lock){+.-...}
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ... which became HARDIRQ-irq-
unsafe at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ...  [<ffffffff8108e1ed>] 
__lock_acquire+0x345/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa036bf48>] 
nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffffa03985ff>] 
ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8143c97d>] 
nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8143ca39>] 
nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8144621f>] 
NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff814468d9>] 
ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81415df5>] 
__netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81415ed8>] 
process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] other info that might help us 
debug this:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  Possible interrupt unsafe 
locking scenario:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]        CPU0                    
CPU1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]        ----                    
----
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   lock(nf_conntrack_lock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
local_irq_disable();
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
lock(&(&bp->lock)->rlock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                
lock(nf_conntrack_lock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]   <Interrupt>
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]     lock(&(&bp->lock)->rlock);
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  *** DEADLOCK ***
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 2 locks held by swapper/0:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  #0:  (&(&bp->lock)->rlock)
{-.-.-.}, at: [<ffffffffa0287a6a>] b44_poll+0x28/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  #1:  (rcu_read_lock){.+.+..}, 
at: [<ffffffff8143c558>] rcu_read_lock+0x0/0x44
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] the dependencies between 
HARDIRQ-irq-safe lock and the holding lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] -> (&(&bp->lock)->rlock)
{-.-.-.} ops: 27161 {
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-HARDIRQ-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e176>] __lock_acquire+0x2ce/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff815044fd>] _raw_spin_lock+0x45/0x79
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa0285725>] b44_interrupt+0x25/0xd9 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff810c2958>] handle_irq_event_percpu+0xb4/0x271
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff810c2b5c>] handle_irq_event+0x47/0x67
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff810c4ef0>] handle_fasteoi_irq+0x8a/0xb0
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812d986f>] __xen_evtchn_do_upcall+0x15e/0x203
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf65>] xen_evtchn_do_upcall+0x2c/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-SOFTIRQ-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e19d>] __lock_acquire+0x2f5/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81504612>] _raw_spin_lock_irq+0x4f/0x82
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa02868a2>] b44_timer+0x13/0x51 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8106a83c>] run_timer_softirq+0x218/0x372
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-RECLAIM_FS-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff8108e24a>] __lock_acquire+0x3a2/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff815044fd>] _raw_spin_lock+0x45/0x79
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffffa0285725>] b44_interrupt+0x25/0xd9 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff810c2958>] handle_irq_event_percpu+0xb4/0x271
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff810c2b5c>] handle_irq_event+0x47/0x67
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff810c4ef0>] handle_fasteoi_irq+0x8a/0xb0
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff812d986f>] __xen_evtchn_do_upcall+0x15e/0x203
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff812daf65>] xen_evtchn_do_upcall+0x2c/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
          
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    INITIAL USE at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108e262>] __lock_acquire+0x3ba/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81504612>] _raw_spin_lock_irq+0x4f/0x82
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa0286637>] b44_set_mac_addr+0x5a/0x82 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8141488d>] dev_set_mac_address+0x3e/0x58
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa04a6985>] bond_enslave+0x3e6/0xa48 [bonding]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa04ad458>] bonding_store_slaves+0x106/0x174 [bonding]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff813138c4>] dev_attr_store+0x20/0x22
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff811a0d5d>] sysfs_write_file+0x108/0x144
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81142bda>] vfs_write+0xaf/0xf6
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81142dd5>] sys_write+0x4d/0x74
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150bc82>] system_call_fastpath+0x16/0x1b
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  }
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... key      at: 
[<ffffffffa028b448>] __key.37034+0x0/0xffffffffffffdbb8 [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... acquired at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa036a7b7>] 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] the dependencies between the 
lock to be acquired and HARDIRQ-irq-unsafe lock:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] -> (nf_conntrack_lock){+.-...} 
ops: 255 {
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    HARDIRQ-ON-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e1ed>] __lock_acquire+0x345/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81504846>] _raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa036bf48>] nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa03985ff>] ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143c97d>] nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143ca39>] nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8144621f>] NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff814468d9>] ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415df5>] __netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415ed8>] process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81417c0a>] net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    IN-SOFTIRQ-W at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108e19d>] __lock_acquire+0x2f5/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81504846>] _raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa036bf48>] nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffffa03985ff>] ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143c97d>] nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8143ca39>] nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8144621f>] NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff814468d9>] ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415df5>] __netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81415ed8>] process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81417c0a>] net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
       
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    INITIAL USE at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108e262>] __lock_acquire+0x3ba/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8108f0b7>] lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81504846>] _raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa036bf48>] nf_conntrack_in+0x461/0x7dc [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffffa03985ff>] ipv4_conntrack_in+0x21/0x23 [nf_conntrack_ipv4]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8143c97d>] nf_iterate+0x4c/0x8c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8143ca39>] nf_hook_slow+0x7c/0x123
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8144621f>] NF_HOOK.constprop.4+0x46/0x5a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff814468d9>] ip_rcv+0x239/0x264
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81415df5>] __netif_receive_skb+0x4af/0x508
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81415ed8>] process_backlog+0x8a/0x151
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81417c0a>] net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81062b2e>] __do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150df7c>] call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81010bfd>] do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff81062edd>] irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff812daf6a>] xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]                                 
      
[<ffffffff8150dfce>] xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  }
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... key      at: 
[<ffffffffa0376018>] nf_conntrack_lock+0x18/0xffffffffffffd000 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  ... acquired at:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa036a7b7>] 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]    [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] 
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] stack backtrace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Pid: 0, comm: swapper Not 
tainted 3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Call Trace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <IRQ>  [<ffffffff8108db25>] 
check_usage+0x37f/0x394
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d72>] ? 
check_events+0x12/0x20
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108db7c>] 
check_irq_usage+0x42/0x88
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108e8fa>] 
__lock_acquire+0xa52/0xd0c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a45d>] ? 
rcu_read_lock+0x44/0x44 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007746>] ? 
xen_force_evtchn_callback+0xd/0xf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d72>] ? 
check_events+0x12/0x20
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007746>] ? 
xen_force_evtchn_callback+0xd/0xf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8108f0b7>] 
lock_acquire+0xf3/0x13e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007d5f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81504846>] 
_raw_spin_lock_bh+0x4a/0x7e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] ? 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7b7>] 
destroy_conntrack+0x70/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a747>] ? 
nf_conntrack_free+0x58/0x58 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062a9a>] ? 
__do_softirq+0x7e/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <EOI>  [<ffffffff810013aa>] ? 
hypercall_page+0x3aa/0x1000
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff810013aa>] ? 
hypercall_page+0x3aa/0x1000
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007701>] ? 
xen_safe_halt+0x10/0x18
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8101601e>] ? 
default_idle+0x53/0x90
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8100e2f9>] ? 
cpu_idle+0xb5/0x101
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff814e0318>] ? 
rest_init+0xdc/0xe3
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff814e023c>] ? 
csum_partial_copy_generic+0x16c/0x16c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d53b9f>] ? 
start_kernel+0x3dd/0x3ea
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d532c4>] ? 
x86_64_start_reservations+0xaf/0xb3
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d55f18>] ? 
xen_start_kernel+0x588/0x58f
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ------------[ cut here 
]------------
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] WARNING: at 
kernel/softirq.c:159 _local_bh_enable_ip+0x49/0xce()
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Hardware name: MM061            
               
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Modules linked in: fuse 
ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM 
iptable_mangle xen_pciback nfsd lockd nfs_acl auth_rpcgss sunrpc des_generic 
md4 nls_utf8 cifs fscache bridge stp llc bonding be2iscsi iscsi_boot_sysfs 
bnx2i cnic uio cxgb3i libcxgbi iw_cxgb3 cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm 
xt_comment ib_sa ib_mad ib_core ip6t_REJECT ib_addr nf_conntrack_ipv4 
nf_defrag_ipv4 nf_conntrack_ipv6 nf_defrag_ipv6 iscsi_tcp xt_state 
libiscsi_tcp nf_conntrack libiscsi ip6table_filter ipt_LOG 
scsi_transport_iscsi ip6_tables xt_physdev dell_wmi sparse_keymap 
snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device 
arc4 r852 sm_common nand snd_pcm dell_laptop microcode dcdbas b44 nand_ids 
r592 nand_ecc iwl3945 ssb memstick mtd mii iwl_legacy iTCO_wdt mac80211 
iTCO_vendor_support i2c_i801 joydev cfg80211 snd_timer rfkill snd soundcore 
snd_page_alloc tun xen_gntalloc xen_netback xen_blkback xen_gntdev xen_evtchn 
xenfs binfmt_misc
Sep 16 01:10:19 Insp6400 kernel: sdhci_pci sdhci mmc_core firewire_ohci 
firewire_core crc_itu_t wmi i915 drm_kms_helper drm i2c_algo_bit i2c_core 
video [last unloaded: scsi_wait_scan]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Pid: 0, comm: swapper Not 
tainted 3.1.0-0.rc6.git0.0.xendom0.fc17.x86_64 #1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] Call Trace:
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <IRQ>  [<ffffffff8105c4a0>] 
warn_slowpath_common+0x83/0x9b
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7f4>] ? 
destroy_conntrack+0xad/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8105c4d2>] 
warn_slowpath_null+0x1a/0x1c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062598>] 
_local_bh_enable_ip+0x49/0xce
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8106262b>] 
local_bh_enable_ip+0xe/0x10
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81504df9>] 
_raw_spin_unlock_bh+0x40/0x44
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a7f4>] 
destroy_conntrack+0xad/0xec [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa036a747>] ? 
nf_conntrack_free+0x58/0x58 [nf_conntrack]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8143c85c>] 
nf_conntrack_destroy+0x5a/0x64
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d88d>] 
skb_release_head_state+0xa7/0xef
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d530>] 
__kfree_skb+0x13/0x83
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8140d645>] 
consume_skb+0xa5/0xd1
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffffa0287af1>] 
b44_poll+0xaf/0x3ec [b44]
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81417c0a>] 
net_rx_action+0xae/0x21c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062a9a>] ? 
__do_softirq+0x7e/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062b2e>] 
__do_softirq+0x112/0x25a
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8150df7c>] 
call_softirq+0x1c/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81010bfd>] 
do_softirq+0x4b/0xa2
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81062edd>] 
irq_exit+0x5d/0xcf
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff812daf6a>] 
xen_evtchn_do_upcall+0x31/0x3e
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8150dfce>] 
xen_do_hypervisor_callback+0x1e/0x30
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  <EOI>  [<ffffffff810013aa>] ? 
hypercall_page+0x3aa/0x1000
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff810013aa>] ? 
hypercall_page+0x3aa/0x1000
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81007701>] ? 
xen_safe_halt+0x10/0x18
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8101601e>] ? 
default_idle+0x53/0x90
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff8100e2f9>] ? 
cpu_idle+0xb5/0x101
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff814e0318>] ? 
rest_init+0xdc/0xe3
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff814e023c>] ? 
csum_partial_copy_generic+0x16c/0x16c
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d53b9f>] ? 
start_kernel+0x3dd/0x3ea
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d532c4>] ? 
x86_64_start_reservations+0xaf/0xb3
Sep 16 01:10:19 Insp6400 kernel: [  473.541175]  [<ffffffff81d55f18>] ? 
xen_start_kernel+0x588/0x58f
Sep 16 01:10:19 Insp6400 kernel: [  473.541175] ---[ end trace 
dfe23b483fd11a0c ]---

 

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>