WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

[Xen-devel] Re: Bug#599089: linux-image-2.6.32-5-xen-686: Kernel Panics

To: Jason Kendall <jakendall@xxxxxxxxx>, 599089@xxxxxxxxxxxxxxx
Subject: [Xen-devel] Re: Bug#599089: linux-image-2.6.32-5-xen-686: Kernel Panics when using NFS from DomU to Dom0
From: Ian Campbell <ijc@xxxxxxxxxxxxxx>
Date: Tue, 05 Oct 2010 10:47:39 +0100
Cc: Jeremy Fitzhardinge <jeremy@xxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>, Arnd Hannemann <hannemann@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Tue, 05 Oct 2010 02:49:05 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4CA9EF03.20407@xxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4CA9EF03.20407@xxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Hi Jason,

Thanks for the report.

You stack trace looks pretty similar to one reported upstream over the
summer:
http://lists.xensource.com/archives/html/xen-devel/2010-06/msg00831.html

Unfortunately I don't think anyone ever got to the bottom of the issue.

Luckily I've been able to reproduce reasonably easily (it seems transfer
from domU to dom0 is the key rather than vice versa).

The stack trace shows that the kernel has hit a pretty rare code path.
It is trying to copy domU skb data into dom0 memory because the skb has
been in the dom0 network stack. It does this so that the guest memory
can be returned to the guest so that it doesn't all end up sitting in
the dom0 network stack. Obviously something is going wrong with this.

In addition to the kernel logging of the error I get this from the
hypervisor:
(XEN) mm.c:2062:d0 Error pfn 16d99: rd=ffff83011fefa000, od=0000000000000000, 
caf=180000000000000, taf=0000000000000000
(XEN) mm.c:658:d0 Could not get page ref for pfn 16d99
(XEN) mm.c:3621:d0 Could not get page for mach->phys update

which is consistent with the kernel's BUG_ON.

Ian.

On Mon, 2010-10-04 at 11:13 -0400, Jason Kendall wrote:
> Package: linux-2.6
> Version: 2.6.32-23
> Severity: grave
> Tags: upstream
> Justification: renders package unusable
> 
> 
> -- Running NFS inside of a DomU when connected to Dom0 causes kernel 
> panic within a few minutes. Does not matter which DomU the NFS is 
> connected from. Does not happen when connecting to NFS outside of the 
> Xen enviroment.
> 
> Notes I've made while diagnosing this issue:
> 
> 1. Able to run multiple Xen DomUs (8 in total)
> 2. MX1 has a NFS connection to Dom0 (/home) for mail delivery.
> 3. I can run MX1 without NFS with no issues, No oops or any issues
> 4. within a few mins of mounting NFS on MX1 Dom0 crashes
> 5. Clock-sources are a mixture of XEN and TSC (Xen is default, moved 2 
> DomUs to TSC as a troubleshooting step (aaa and MX1)
> 6. Mounted /home on a different DomU as a test..
> 7. Leaving the NFS mount on a different DomU did not cause Panic, 
> however, no traffic happened left it for a few hours and no panic.
> 8. Started moving files between NFS and local DomU drive and Panic happened.
> 9. Coping large amounts of files within the DomU (not over NFS) did not 
> seem to cause Oops.
> 10. Created NFS share on desktop which is not part of the Xen server to 
> see if it causes a panic (Identifies if this is a Dom0 or DomU issue)
> NOTE: Dom0 has /home which the desktop can access perfectly fine.
> 11. Moving files from Desktop NFS is ok.
> 
> Recap:
> 
> DomU <-> DomU: Ok
> DomU <-> Desktop NFS: OK
> DomU <-> Dom0: Oops
> Desktop <-> Dom0: OK
> 
> 
> -- Package-specific info:
> ** Version:
> Linux version 2.6.32-5-xen-686 (Debian 2.6.32-23) (dannf@xxxxxxxxxx) 
> (gcc version 4.3.5 (Debian 4.3.5-3) ) #1 SMP Sat Sep 18 06:00:14 UTC 2010
> 
> ** Command line:
> placeholder root=UUID=6b304f06-c37f-480f-af92-478443531682 ro quiet
> 
> ** Not tainted
> 
> ** Kernel log:
> [  157.841231] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.841242] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.841253] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.841265] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872702] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872718] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872728] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872738] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872748] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872759] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872769] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872778] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872788] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872798] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872808] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872818] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.872828] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884349] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884372] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884388] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884405] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884420] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884437] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884455] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884471] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884488] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884506] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884521] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884537] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884555] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  157.884570] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  161.642902] blkback: ring-ref 8, event-channel 8, protocol 1 (x86_32-abi)
> [  161.935808] blkback: ring-ref 9, event-channel 9, protocol 1 (x86_32-abi)
> [  168.717532] vif7.0: no IPv6 routers present
> [  172.787784] device vif8.0 entered promiscuous mode
> [  172.797787] eth0: port 9(vif8.0) entering forwarding state
> [  172.848319] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848343] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848359] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848374] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848388] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848402] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848420] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848437] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848453] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848468] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848482] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848496] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848511] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.848526] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860830] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860843] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860854] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860864] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860874] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860884] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860894] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860904] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860914] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860924] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860934] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860944] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860954] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860963] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.860974] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874077] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874091] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874101] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874111] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874121] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874131] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874141] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874151] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874161] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874170] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874180] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874190] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874200] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874210] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.874220] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.887977] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.887995] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888088] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888119] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888134] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888152] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888171] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888186] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888202] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888222] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888236] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888252] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888270] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888286] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888301] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  172.888321] physdev match: using --physdev-out in the OUTPUT, FORWARD 
> and POSTROUTING chains for non-bridged traffic is not supported anymore.
> [  177.711666] blkback: ring-ref 8, event-channel 8, protocol 1 (x86_32-abi)
> [  178.054806] blkback: ring-ref 9, event-channel 9, protocol 1 (x86_32-abi)
> [  181.495854] Switching to clocksource tsc
> [  182.880089] vif8.0: no IPv6 routers present
> 
> ** Model information
> not available
> 
> ** Loaded modules:
> Module                  Size  Used by
> xt_tcpudp               1743  8
> nf_conntrack_ipv4       7597  16
> nf_defrag_ipv4           779  1 nf_conntrack_ipv4
> xt_state                 927  16
> nf_conntrack           38079  2 nf_conntrack_ipv4,xt_state
> xt_physdev              1228  32
> iptable_filter          1790  1
> ip_tables               7690  1 iptable_filter
> x_tables                8327  4 xt_tcpudp,xt_state,xt_physdev,ip_tables
> bridge                 32999  0
> stp                      996  1 bridge
> xen_evtchn              3471  9
> xenfs                   7248  1
> nfsd                  193247  13
> exportfs                2618  1 nfsd
> nfs                   206894  0
> lockd                  49433  2 nfsd,nfs
> fscache                24654  1 nfs
> nfs_acl                 1695  2 nfsd,nfs
> auth_rpcgss            25240  2 nfsd,nfs
> sunrpc                134572  14 nfsd,nfs,lockd,nfs_acl,auth_rpcgss
> loop                    9761  32
> snd_pcm                47402  0
> snd_timer              12258  1 snd_pcm
> pl2303                 11624  0
> dcdbas                  3940  0
> snd                    34363  2 snd_pcm,snd_timer
> usbserial              21843  1 pl2303
> soundcore               3450  1 snd
> snd_page_alloc          5109  1 snd_pcm
> psmouse                44653  0
> pcspkr                  1207  0
> serio_raw               2916  0
> shpchp                 21264  0
> evdev                   5609  2
> i2c_piix4               7124  0
> pci_hotplug            18065  1 shpchp
> i2c_core               12696  1 i2c_piix4
> button                  3598  0
> processor              31041  0
> acpi_processor          3623  1 processor,[permanent]
> ext3                   94196  2
> jbd                    32161  1 ext3
> mbcache                 3762  1 ext3
> sr_mod                 10770  0
> cdrom                  26487  1 sr_mod
> ata_generic             2019  0
> sg                     15972  0
> sd_mod                 25865  5
> crc_t10dif              1012  1 sd_mod
> megaraid_mbox          21022  0
> pata_serverworks        3969  0
> ohci_hcd               17068  0
> megaraid_mm             5451  1 megaraid_mbox
> libata                116021  2 ata_generic,pata_serverworks
> qla1280                16957  0
> tg3                    88633  0
> floppy                 40931  0
> aacraid                51565  3
> libphy                 11638  1 tg3
> ehci_hcd               28255  0
> scsi_mod              101397  7 
> sr_mod,sg,sd_mod,megaraid_mbox,libata,qla1280,aacraid
> usbcore                98725  5 pl2303,usbserial,ohci_hcd,ehci_hcd
> nls_base                4541  1 usbcore
> thermal                 9206  0
> thermal_sys             9378  2 processor,thermal
> 
> ** Network interface configuration:
> 
> auto lo
> iface lo inet loopback
> 
> allow-hotplug eth0
> iface eth0 inet static
>   address 10.0.0.20
>   netmask 255.255.255.0
>   broadcast 10.0.0.255
>   gateway 10.0.0.1
>   dns-domain lab.ostlabs.com
>   dns-nameservers 10.0.0.1
> 
> ** Network status:
> *** IP interfaces and addresses:
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>      inet 127.0.0.1/8 scope host lo
>      inet6 ::1/128 scope host
>         valid_lft forever preferred_lft forever
> 2: peth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq 
> state UP qlen 1000
>      link/ether 00:06:5b:f7:eb:c3 brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::206:5bff:fef7:ebc3/64 scope link
>         valid_lft forever preferred_lft forever
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
>      link/ether 00:06:5b:f7:eb:c4 brd ff:ff:ff:ff:ff:ff
> 4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state 
> UNKNOWN
>      link/ether 00:06:5b:f7:eb:c3 brd ff:ff:ff:ff:ff:ff
>      inet 10.0.0.20/24 brd 10.0.0.255 scope global eth0
>      inet6 fe80::206:5bff:fef7:ebc3/64 scope link
>         valid_lft forever preferred_lft forever
> 5: vif1.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 6: vif2.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 7: vif3.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 8: vif4.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 9: vif5.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 10: vif6.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 11: vif7.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 12: vif8.0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc 
> pfifo_fast state UNKNOWN qlen 32
>      link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::fcff:ffff:feff:ffff/64 scope link
>         valid_lft forever preferred_lft forever
> 
> *** Device statistics:
> Inter-|   Receive                                                |  Transmit
>   face |bytes    packets errs drop fifo frame compressed 
> multicast|bytes    packets errs drop fifo colls carrier compressed
>      lo:   12990     151    0    0    0     0          0         0    
> 12990     151    0    0    0     0       0          0
>   peth0:33668865   52529    0    0    0     0          0      1400 
> 40069263   51090    0    0    0     0       0          0
>    eth1:       0       0    0    0    0     0          0         
> 0        0       0    0    0    0     0       0          0
>    eth0:32154816   49419    0    0    0     0          0      2082 
> 38628135   34224    0    0    0     0       0          0
> vif1.0:  320314    3264    0    0    0     0          0         0   
> 742272    6508    0   13    0     0       0          0
> vif2.0:  547294    6543    0    0    0     0          0         0  
> 1059726    7321    0    9    0     0       0          0
> vif3.0:  606762    1869    0    0    0     0          0         0   
> 633826    4139    0   13    0     0       0          0
> vif4.0:   67861     381    0    0    0     0          0         0   
> 336287    2440    0   13    0     0       0          0
> vif5.0:  118042    1447    0    0    0     0          0         0   
> 466361    3261    0   15    0     0       0          0
> vif6.0:  346370    2546    0    0    0     0          0         0   
> 956549    4325    0   16    0     0       0          0
> vif7.0:   18773     316    0    0    0     0          0         0   
> 303830    2198    0   10    0     0       0          0
> vif8.0:     804      13    0    0    0     0          0         0   
> 271472    1921    0   16    0     0       0          0
> 
> *** Protocol statistics:
> Ip:
>      48197 total packets received
>      38 with invalid addresses
>      0 forwarded
>      0 incoming packets discarded
>      48009 incoming packets delivered
>      34936 requests sent out
>      13 dropped because of missing route
> Icmp:
>      6 ICMP messages received
>      0 input ICMP message failed.
>      ICMP input histogram:
>          destination unreachable: 6
>      73 ICMP messages sent
>      0 ICMP messages failed
>      ICMP output histogram:
>          destination unreachable: 73
> IcmpMsg:
>          InType3: 6
>          OutType3: 73
> Tcp:
>      11 active connections openings
>      38 passive connection openings
>      6 failed connection attempts
>      0 connection resets received
>      3 connections established
>      47193 segments received
>      34001 segments send out
>      25 segments retransmited
>      0 bad segments received.
>      8 resets sent
> Udp:
>      416 packets received
>      67 packets to unknown port received.
>      0 packet receive errors
>      841 packets sent
> UdpLite:
> TcpExt:
>      3 TCP sockets finished time wait in fast timer
>      109 delayed acks sent
>      5 delayed acks further delayed because of locked socket
>      1 packets directly queued to recvmsg prequeue.
>      26117 packet headers predicted
>      940 acknowledgments not containing data payload received
>      24570 predicted acknowledgments
>      2 times recovered from packet loss by selective acknowledgements
>      76 TCP data loss events
>      TCPLostRetransmit: 4
>      25 fast retransmits
>      TCPSackShifted: 74
>      TCPSackMerged: 135
>      TCPSackShiftFallback: 20
> IpExt:
>      InBcastPkts: 602
>      OutBcastPkts: 51
>      InOctets: 32108796
>      OutOctets: 38224368
>      InBcastOctets: 159844
>      OutBcastOctets: 7380
> 
> *** Device features:
> eth0: 0x11823
> eth1: 0x9a3
> lo: 0x13865
> peth0: 0x9a3
> vif1.0: 0x10803
> vif2.0: 0x10803
> vif3.0: 0x10803
> vif4.0: 0x10803
> vif5.0: 0x10803
> vif6.0: 0x10803
> vif7.0: 0x10803
> vif8.0: 0x10803
> 
> ** PCI devices:
> 00:00.0 Host bridge [0600]: Broadcom CMIC-WS Host Bridge (GC-LE chipset) 
> [1166:0012] (rev 13)
>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> 
> 00:00.1 Host bridge [0600]: Broadcom CMIC-WS Host Bridge (GC-LE chipset) 
> [1166:0012]
>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> 
> 00:00.2 Host bridge [0600]: Broadcom CMIC-LE [1166:0000]
>      Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> 
> 00:04.0 Unassigned class [ff00]: Dell Embedded Remote Access or ERA/O 
> [1028:000c]
>      Subsystem: Dell Embedded Remote Access or ERA/O [1028:000c]
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 32, Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 11
>      Region 0: Memory at feb80000 (32-bit, prefetchable) [size=4K]
>      Region 1: I/O ports at ecf8 [size=8]
>      Region 2: I/O ports at ece8 [size=8]
>      Expansion ROM at fe000000 [disabled] [size=64K]
>      Capabilities: <access denied>
> 
> 00:04.1 Unassigned class [ff00]: Dell Remote Access Card III [1028:0008]
>      Subsystem: Dell Remote Access Card III [1028:0008]
>      Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Interrupt: pin B routed to IRQ 23
>      Region 0: Memory at fe102000 (32-bit, non-prefetchable) [size=4K]
>      Region 1: I/O ports at ec80 [size=64]
>      Region 2: Memory at feb00000 (32-bit, prefetchable) [size=512K]
>      Capabilities: <access denied>
>      Kernel driver in use: serial
> 
> 00:04.2 Unassigned class [ff00]: Dell Embedded Remote Access: BMC/SMIC 
> device [1028:000d]
>      Subsystem: Dell Embedded Remote Access: BMC/SMIC device [1028:000d]
>      Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Interrupt: pin C routed to IRQ 7
>      Region 0: I/O ports at ecf4 [size=4]
>      Capabilities: <access denied>
> 
> 00:0e.0 VGA compatible controller [0300]: ATI Technologies Inc Rage XL 
> [1002:4752] (rev 27) (prog-if 00 [VGA controller])
>      Subsystem: Dell Device [1028:0121]
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- 
> Stepping+ SERR- FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 32 (2000ns min), Cache Line Size: 64 bytes
>      Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
>      Region 1: I/O ports at e800 [size=256]
>      Region 2: Memory at fe101000 (32-bit, non-prefetchable) [size=4K]
>      [virtual] Expansion ROM at e8000000 [disabled] [size=128K]
>      Capabilities: <access denied>
> 
> 00:0f.0 Host bridge [0600]: Broadcom CSB5 South Bridge [1166:0201] (rev 93)
>      Subsystem: Broadcom CSB5 South Bridge [1166:0201]
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>      Latency: 32
>      Kernel driver in use: piix4_smbus
> 
> 00:0f.1 IDE interface [0101]: Broadcom CSB5 IDE Controller [1166:0212] 
> (rev 93) (prog-if 82 [Master PriP])
>      Subsystem: Broadcom CSB5 IDE Controller [1166:0212]
>      Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 64
>      Region 0: I/O ports at 01f0 [size=8]
>      Region 1: I/O ports at 03f4 [size=1]
>      Region 2: I/O ports at 0170 [size=8]
>      Region 3: I/O ports at 0374 [size=1]
>      Region 4: I/O ports at 08b0 [size=16]
>      Kernel driver in use: pata_serverworks
> 
> 00:0f.2 USB Controller [0c03]: Broadcom OSB4/CSB5 OHCI USB Controller 
> [1166:0220] (rev 05) (prog-if 10 [OHCI])
>      Subsystem: Broadcom OSB4/CSB5 OHCI USB Controller [1166:0220]
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 32 (20000ns max)
>      Interrupt: pin A routed to IRQ 11
>      Region 0: Memory at fe100000 (32-bit, non-prefetchable) [size=4K]
>      Kernel driver in use: ohci_hcd
> 
> 00:0f.3 ISA bridge [0601]: Broadcom CSB5 LPC bridge [1166:0225]
>      Subsystem: Broadcom Dell PowerEdge 1650/1750 [1166:0230]
>      Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>      Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 0
> 
> 00:10.0 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge 
> [1166:0101] (rev 03)
>      Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>      Capabilities: <access denied>
> 
> 00:10.2 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge 
> [1166:0101] (rev 03)
>      Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>      Capabilities: <access denied>
> 
> 00:11.0 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge 
> [1166:0101] (rev 05)
>      Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>      Capabilities: <access denied>
> 
> 00:11.2 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge 
> [1166:0101] (rev 05)
>      Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>      Capabilities: <access denied>
> 
> 02:06.0 PCI bridge [0604]: Intel Corporation 21154 PCI-to-PCI Bridge 
> [8086:b154] (prog-if 00 [Normal decode])
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 32
>      Bus: primary=02, secondary=03, subordinate=04, sec-latency=32
>      I/O behind bridge: 0000d000-0000dfff
>      Memory behind bridge: fcd00000-fcffffff
>      Prefetchable memory behind bridge: 00000000f0000000-00000000f7ffffff
>      Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- <SERR- <PERR-
>      BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
>          PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>      Capabilities: <access denied>
> 
> 03:00.0 PCI bridge [0604]: Intel Corporation 21154 PCI-to-PCI Bridge 
> [8086:b154] (prog-if 00 [Normal decode])
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 32
>      Bus: primary=03, secondary=04, subordinate=04, sec-latency=32
>      Memory behind bridge: fcf00000-fcffffff
>      Prefetchable memory behind bridge: 00000000f0000000-00000000f7ffffff
>      Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ <SERR- <PERR-
>      BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
>          PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>      Capabilities: <access denied>
> 
> 03:01.0 SCSI storage controller [0100]: QLogic Corp. ISP12160 Dual 
> Channel Ultra3 SCSI Processor [1077:1216] (rev 06)
>      Subsystem: American Megatrends Inc. QLA12160 on AMI MegaRAID 
> [101e:8493]
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 32 (16000ns min), Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 11
>      Region 0: I/O ports at dc00 [size=256]
>      Region 1: Memory at fcdff000 (32-bit, non-prefetchable) [size=4K]
>      Expansion ROM at fce00000 [disabled] [size=128K]
>      Capabilities: <access denied>
> 
> 04:00.0 RAID bus controller [0104]: American Megatrends Inc. MegaRAID 
> [101e:1960] (rev 20)
>      Subsystem: Dell PowerEdge RAID Controller 3/DC [1028:0493]
>      Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>      Latency: 32, Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 24
>      Region 0: Memory at f0000000 (32-bit, prefetchable) [size=128M]
>      Expansion ROM at fcf00000 [disabled] [size=128K]
>      Capabilities: <access denied>
>      Kernel driver in use: megaraid
> 
> 05:06.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme 
> BCM5701 Gigabit Ethernet [14e4:1645] (rev 15)
>      Subsystem: Dell Broadcom BCM5701 1000Base-T [1028:0121]
>      Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 64 (16000ns min), Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 28
>      Region 0: Memory at eff10000 (64-bit, non-prefetchable) [size=64K]
>      Capabilities: <access denied>
>      Kernel driver in use: tg3
> 
> 05:08.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme 
> BCM5701 Gigabit Ethernet [14e4:1645] (rev 15)
>      Subsystem: Dell Broadcom BCM5701 1000Base-T [1028:0121]
>      Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 64 (16000ns min), Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 29
>      Region 0: Memory at eff00000 (64-bit, non-prefetchable) [size=64K]
>      Capabilities: <access denied>
>      Kernel driver in use: tg3
> 
> 06:08.0 PCI bridge [0604]: Intel Corporation 80303 I/O Processor 
> PCI-to-PCI Bridge [8086:0309] (rev 01) (prog-if 00 [Normal decode])
>      Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>      Latency: 32, Cache Line Size: 64 bytes
>      Bus: primary=06, secondary=07, subordinate=07, sec-latency=32
>      Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort+ <SERR- <PERR-
>      BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
>          PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>      Capabilities: <access denied>
> 
> 06:08.1 RAID bus controller [0104]: Dell PowerEdge Expandable RAID 
> Controller 3/Di [1028:000a] (rev 01)
>      Subsystem: Dell PERC 3/DiB [Boxster] [1028:0121]
>      Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- 
> Stepping- SERR+ FastB2B- DisINTx-
>      Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=slow >TAbort- 
> <TAbort- <MAbort+ >SERR- <PERR- INTx-
>      Latency: 32, Cache Line Size: 64 bytes
>      Interrupt: pin A routed to IRQ 30
>      Region 0: Memory at e0000000 (32-bit, prefetchable) [size=128M]
>      Expansion ROM at efd00000 [disabled] [size=64K]
>      Capabilities: <access denied>
>      Kernel driver in use: aacraid
> 
> 
> ** USB devices:
> Bus 001 Device 002: ID 067b:aaa0 Prolific Technology, Inc. Prolific Pharos
> Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
> 
> 
> -- System Information:
> Debian Release: squeeze/sid
>    APT prefers testing
>    APT policy: (500, 'testing')
> Architecture: i386 (i686)
> 
> Kernel: Linux 2.6.32-5-xen-686 (SMP w/4 CPU cores)
> Locale: LANG=en_CA.UTF-8, LC_CTYPE=en_CA.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> 
> Versions of packages linux-image-2.6.32-5-xen-686 depends on:
> ii  debconf [debconf-2.0]         1.5.35     Debian configuration 
> management sy
> ii  initramfs-tools               0.98.4     tools for generating an 
> initramfs
> ii  linux-base                    2.6.32-23  Linux image base package
> ii  module-init-tools             3.12-1     tools for managing Linux 
> kernel mo
> 
> Versions of packages linux-image-2.6.32-5-xen-686 recommends:
> ii  firmware-linux-free           2.6.32-23  Binary firmware for various 
> driver
> ii  libc6-xen                     2.11.2-6   Embedded GNU C Library: 
> Shared lib
> 
> Versions of packages linux-image-2.6.32-5-xen-686 suggests:
> pn  grub <none>     (no description available)
> pn  linux-doc-2.6.32 <none>     (no description available)
> 
> Versions of packages linux-image-2.6.32-5-xen-686 is related to:
> pn  firmware-bnx2 <none>     (no description available)
> pn  firmware-bnx2x <none>     (no description available)
> pn  firmware-ipw2x00 <none>     (no description available)
> pn  firmware-ivtv <none>     (no description available)
> pn  firmware-iwlwifi <none>     (no description available)
> pn  firmware-linux <none>     (no description available)
> pn  firmware-linux-nonfree <none>     (no description available)
> pn  firmware-qlogic <none>     (no description available)
> pn  firmware-ralink <none>     (no description available)
> ii  xen-hypervisor-4.0-i386 [xen- 4.0.1-1    The Xen Hypervisor on i386
> 
> -- debconf information:
>    
> linux-image-2.6.32-5-xen-686/postinst/depmod-error-initrd-2.6.32-5-xen-686: 
> false
>    
> linux-image-2.6.32-5-xen-686/postinst/ignoring-do-bootloader-2.6.32-5-xen-686:
>    
> linux-image-2.6.32-5-xen-686/prerm/removing-running-kernel-2.6.32-5-xen-686: 
> true
>    linux-image-2.6.32-5-xen-686/postinst/missing-firmware-2.6.32-5-xen-686:
> 
> ------ netconsole Trace:
> 
> Oct  4 09:45:31 10.0.0.20 kernel: [173927.281586] netconsole: local port 
> 6666
> Oct  4 09:45:31 10.0.0.20 kernel: [173927.281599] netconsole: local IP 
> 10.0.0.20
> Oct  4 09:45:31 10.0.0.20 kernel: [173927.281605] netconsole: interface eth0
> Oct  4 09:45:31 10.0.0.20 kernel: [173927.281612] netconsole: remote 
> port 514
> Oct  4 09:45:31 10.0.0.20 kernel: [173927.281619] netconsole: remote IP 
> 10.0.0.100
> Oct  4 09:45:31 10.0.0.20 kernel: [173927.281627] netconsole: remote 
> ethernet address 00:1a:70:11:c5:ff
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.595807] netconsole: local port 
> 6666
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.595820] netconsole: local IP 
> 10.0.0.20
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.595830] netconsole: interface 
> peth0
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.595839] netconsole: remote 
> port 514
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.595848] netconsole: remote IP 
> 10.0.0.100
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.595859] netconsole: remote 
> ethernet address 00:1a:70:11:c5:ff
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.605629] console [netcon0] enabled
> Oct  4 09:45:47 10.0.0.20 kernel: [173943.605636] netconsole: network 
> logging started
> Oct  4 09:52:29 10.0.0.20 [174345.332640] ------------[ cut here 
> ]------------
> Oct  4 09:52:29 10.0.0.20 [174345.332716] kernel BUG at 
> /build/buildd-linux-2.6_2.6.32-23-i386-x1D1UQ/linux-2.6-2.6.32/debian/build/source_i386_xen/drivers/xen/grant-table.c:584!
>  
> 
> Oct  4 09:52:29 10.0.0.20 [174345.332805] invalid opcode: 0000 [#1]
> Oct  4 09:52:29 10.0.0.20 SMP
> Oct  4 09:52:29 10.0.0.20
> Oct  4 09:52:29 10.0.0.20 [174345.332903] last sysfs file: 
> /sys/module/configfs/initstate
> Oct  4 09:52:29 10.0.0.20 [174345.332952] Modules linked in:
> Oct  4 09:52:29 10.0.0.20  netconsole
> Oct  4 09:52:29 10.0.0.20  configfs
> Oct  4 09:52:29 10.0.0.20  xt_tcpudp
> Oct  4 09:52:29 10.0.0.20  nf_conntrack_ipv4
> Oct  4 09:52:29 10.0.0.20  nf_defrag_ipv4
> Oct  4 09:52:29 10.0.0.20  xt_state
> Oct  4 09:52:29 10.0.0.20  nf_conntrack
> Oct  4 09:52:29 10.0.0.20  xt_physdev
> Oct  4 09:52:29 10.0.0.20  iptable_filter
> Oct  4 09:52:29 10.0.0.20  ip_tables
> Oct  4 09:52:29 10.0.0.20  x_tables
> Oct  4 09:52:29 10.0.0.20  bridge
> Oct  4 09:52:29 10.0.0.20  stp
> Oct  4 09:52:29 10.0.0.20  xen_evtchn
> Oct  4 09:52:29 10.0.0.20  xenfs
> Oct  4 09:52:29 10.0.0.20  nfsd
> Oct  4 09:52:29 10.0.0.20  exportfs
> Oct  4 09:52:29 10.0.0.20  nfs
> Oct  4 09:52:29 10.0.0.20  lockd
> Oct  4 09:52:29 10.0.0.20  fscache
> Oct  4 09:52:29 10.0.0.20  nfs_acl
> Oct  4 09:52:29 10.0.0.20  auth_rpcgss
> Oct  4 09:52:29 10.0.0.20  sunrpc
> Oct  4 09:52:29 10.0.0.20  loop
> Oct  4 09:52:29 10.0.0.20  snd_pcm
> Oct  4 09:52:29 10.0.0.20  snd_timer
> Oct  4 09:52:29 10.0.0.20  snd
> Oct  4 09:52:29 10.0.0.20  soundcore
> Oct  4 09:52:29 10.0.0.20  psmouse
> Oct  4 09:52:29 10.0.0.20  pl2303
> Oct  4 09:52:29 10.0.0.20  snd_page_alloc
> Oct  4 09:52:29 10.0.0.20  usbserial
> Oct  4 09:52:29 10.0.0.20  serio_raw
> Oct  4 09:52:29 10.0.0.20  pcspkr
> Oct  4 09:52:29 10.0.0.20  shpchp
> Oct  4 09:52:29 10.0.0.20  evdev
> Oct  4 09:52:29 10.0.0.20  dcdbas
> Oct  4 09:52:29 10.0.0.20  pci_hotplug
> Oct  4 09:52:29 10.0.0.20  i2c_piix4
> Oct  4 09:52:29 10.0.0.20  i2c_core
> Oct  4 09:52:29 10.0.0.20  processor
> Oct  4 09:52:29 10.0.0.20  acpi_processor
> Oct  4 09:52:29 10.0.0.20  button
> Oct  4 09:52:29 10.0.0.20  ext3
> Oct  4 09:52:29 10.0.0.20  jbd
> Oct  4 09:52:29 10.0.0.20  mbcache
> Oct  4 09:52:29 10.0.0.20  sg
> Oct  4 09:52:29 10.0.0.20  sr_mod
> Oct  4 09:52:29 10.0.0.20  cdrom
> Oct  4 09:52:29 10.0.0.20  sd_mod
> Oct  4 09:52:29 10.0.0.20  crc_t10dif
> Oct  4 09:52:29 10.0.0.20  ata_generic
> Oct  4 09:52:29 10.0.0.20  ohci_hcd
> Oct  4 09:52:29 10.0.0.20  pata_serverworks
> Oct  4 09:52:29 10.0.0.20  megaraid_mbox
> Oct  4 09:52:29 10.0.0.20  megaraid_mm
> Oct  4 09:52:29 10.0.0.20  tg3
> Oct  4 09:52:29 10.0.0.20  floppy
> Oct  4 09:52:29 10.0.0.20  aacraid
> Oct  4 09:52:29 10.0.0.20  libphy
> Oct  4 09:52:29 10.0.0.20  qla1280
> Oct  4 09:52:29 10.0.0.20  ehci_hcd
> Oct  4 09:52:29 10.0.0.20  libata
> Oct  4 09:52:29 10.0.0.20  usbcore
> Oct  4 09:52:29 10.0.0.20  nls_base
> Oct  4 09:52:29 10.0.0.20  thermal
> Oct  4 09:52:29 10.0.0.20  scsi_mod
> Oct  4 09:52:29 10.0.0.20  thermal_sys
> Oct  4 09:52:29 10.0.0.20  [last unloaded: scsi_wait_scan]
> Oct  4 09:52:29 10.0.0.20
> Oct  4 09:52:29 10.0.0.20 [174345.334427]
> Oct  4 09:52:29 10.0.0.20 [174345.334472] Pid: 0, comm: swapper Not 
> tainted (2.6.32-5-xen-686 #1) PowerEdge 2650
> Oct  4 09:52:29 10.0.0.20 [174345.334549] EIP: 0061:[<c119803b>] EFLAGS: 
> 00010282 CPU: 1
> Oct  4 09:52:29 10.0.0.20 [174345.334622] EIP is at 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:52:29 10.0.0.20 [174345.334674] EAX: ffffffea EBX: ec847d40 
> ECX: 00000001 EDX: 00000000
> Oct  4 09:52:29 10.0.0.20 [174345.334725] ESI: 00007ff0 EDI: e9976000 
> EBP: 18375001 ESP: ec847d10
> Oct  4 09:52:29 10.0.0.20 [174345.334777]  DS: 007b ES: 007b FS: 00d8 
> GS: 00e0 SS: 0069
> Oct  4 09:52:29 10.0.0.20 [174345.334828] Process swapper (pid: 0, 
> ti=ec846000 task=ec838880 task.ti=ec846000)
> Oct  4 09:52:29 10.0.0.20 [174345.334900] Stack:
> Oct  4 09:52:29 10.0.0.20 [174345.334953]  000020d1
> Oct  4 09:52:29 10.0.0.20  c2424798
> Oct  4 09:52:29 10.0.0.20  000000ba
> Oct  4 09:52:29 10.0.0.20  c2eaea20
> Oct  4 09:52:29 10.0.0.20  c339fea0
> Oct  4 09:52:29 10.0.0.20  c20d1000
> Oct  4 09:52:29 10.0.0.20  c20d1000
> Oct  4 09:52:29 10.0.0.20  00000000
> Oct  4 09:52:29 10.0.0.20
> Oct  4 09:52:29 10.0.0.20 [174345.335118] <0>
> Oct  4 09:52:29 10.0.0.20  e9975000
> Oct  4 09:52:29 10.0.0.20  00000000
> Oct  4 09:52:29 10.0.0.20  000000ba
> Oct  4 09:52:29 10.0.0.20  117f0000
> Oct  4 09:52:29 10.0.0.20  18375001
> Oct  4 09:52:29 10.0.0.20  00000000
> Oct  4 09:52:29 10.0.0.20  000020d1
> Oct  4 09:52:29 10.0.0.20  00000000
> Oct  4 09:52:29 10.0.0.20
> Oct  4 09:52:29 10.0.0.20 [174345.335327] <0>
> Oct  4 09:52:29 10.0.0.20  ed811c70
> Oct  4 09:52:29 10.0.0.20  ed811da8
> Oct  4 09:52:29 10.0.0.20  ed81161c
> Oct  4 09:52:29 10.0.0.20  0006a875
> Oct  4 09:52:29 10.0.0.20  c11a49ca
> Oct  4 09:52:29 10.0.0.20  117fe518
> Oct  4 09:52:29 10.0.0.20  00009e8f
> Oct  4 09:52:29 10.0.0.20  117fe518
> Oct  4 09:52:29 10.0.0.20
> Oct  4 09:52:29 10.0.0.20 [174345.335573] Call Trace:
> Oct  4 09:52:29 10.0.0.20 [174345.335625]  [<c11a49ca>] ? 
> net_tx_action+0x1c8/0xeb1
> Oct  4 09:52:29 10.0.0.20 [174345.335682]  [<c10211e3>] ? 
> pvclock_clocksource_read+0xf9/0x10f
> Oct  4 09:52:29 10.0.0.20 [174345.335740]  [<c113d59e>] ? 
> cpumask_next_and+0x23/0x33
> Oct  4 09:52:29 10.0.0.20 [174345.335795]  [<c102f8c8>] ? 
> find_busiest_group+0x2e9/0x6e6
> Oct  4 09:52:29 10.0.0.20 [174345.335852]  [<c1034131>] ? 
> rebalance_domains+0x149/0x3e1
> Oct  4 09:52:29 10.0.0.20 [174345.335908]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:52:29 10.0.0.20 [174345.335963]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:52:29 10.0.0.20 [174345.336031]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:52:29 10.0.0.20 [174345.336110]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:52:29 10.0.0.20 [174345.336176]  [<c10066d7>] ? 
> xen_restore_fl_direct_end+0x0/0x1
> Oct  4 09:52:29 10.0.0.20 [174345.336247]  [<c128dca1>] ? 
> _spin_unlock_irqrestore+0xd/0xf
> Oct  4 09:52:29 10.0.0.20 [174345.336318]  [<c103b17e>] ? 
> tasklet_action+0x67/0xad
> Oct  4 09:52:29 10.0.0.20 [174345.336384]  [<c103c2e0>] ? 
> __do_softirq+0xaa/0x151
> Oct  4 09:52:29 10.0.0.20 [174345.336437]  [<c103c3b8>] ? 
> do_softirq+0x31/0x3c
> Oct  4 09:52:29 10.0.0.20 [174345.336490]  [<c103c48e>] ? 
> irq_exit+0x26/0x58
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1198c15>] ? 
> xen_evtchn_do_upcall+0x22/0x2c
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1009abf>] ? 
> xen_do_upcall+0x7/0xc
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10023a7>] ? 
> hypercall_page+0x3a7/0x1001
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1006039>] ? 
> xen_safe_halt+0xf/0x1b
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1004137>] ? 
> xen_idle+0x23/0x30
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10080a8>] ? 
> cpu_idle+0x89/0xa5
> Oct  4 09:52:29 10.0.0.20 [174345.336513] Code:
> Oct  4 09:52:29 10.0.0.20 00
> Oct  4 09:52:29 10.0.0.20 31
> Oct  4 09:52:29 10.0.0.20 d2
> Oct  4 09:52:29 10.0.0.20 8b
> Oct  4 09:52:29 10.0.0.20 04
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 89
> Oct  4 09:52:29 10.0.0.20 6c
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 30
> Oct  4 09:52:29 10.0.0.20 8d
> Oct  4 09:52:29 10.0.0.20 5c
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 30
> Oct  4 09:52:29 10.0.0.20 c7
> Oct  4 09:52:29 10.0.0.20 44
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 34
> Oct  4 09:52:29 10.0.0.20 00
> Oct  4 09:52:29 10.0.0.20 last message repeated 3 times
> Oct  4 09:52:29 10.0.0.20 89
> Oct  4 09:52:29 10.0.0.20 44
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 38
> Oct  4 09:52:29 10.0.0.20 c7
> Oct  4 09:52:29 10.0.0.20 44
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 3c
> Oct  4 09:52:29 10.0.0.20 00
> Oct  4 09:52:29 10.0.0.20 last message repeated 3 times
> Oct  4 09:52:29 10.0.0.20 e8
> Oct  4 09:52:29 10.0.0.20 e9
> Oct  4 09:52:29 10.0.0.20 9f
> Oct  4 09:52:29 10.0.0.20 e6
> Oct  4 09:52:29 10.0.0.20 ff
> Oct  4 09:52:29 10.0.0.20 85
> Oct  4 09:52:29 10.0.0.20 c0
> Oct  4 09:52:29 10.0.0.20 74
> Oct  4 09:52:29 10.0.0.20 04
> Oct  4 09:52:29 10.0.0.20 0b
> Oct  4 09:52:29 10.0.0.20 eb
> Oct  4 09:52:29 10.0.0.20 fe
> Oct  4 09:52:29 10.0.0.20 8b
> Oct  4 09:52:29 10.0.0.20 54
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 0c
> Oct  4 09:52:29 10.0.0.20 8b
> Oct  4 09:52:29 10.0.0.20 42
> Oct  4 09:52:29 10.0.0.20 10
> Oct  4 09:52:29 10.0.0.20 8b
> Oct  4 09:52:29 10.0.0.20 54
> Oct  4 09:52:29 10.0.0.20 24
> Oct  4 09:52:29 10.0.0.20 10
> Oct  4 09:52:29 10.0.0.20 89
> Oct  4 09:52:29 10.0.0.20 42
> Oct  4 09:52:29 10.0.0.20 10
> Oct  4 09:52:29 10.0.0.20 0f
> Oct  4 09:52:29 10.0.0.20 ba
> Oct  4 09:52:29 10.0.0.20 2a
> Oct  4 09:52:29 10.0.0.20
> Oct  4 09:52:29 10.0.0.20 [174345.336513] EIP: [<c119803b>]
> Oct  4 09:52:29 10.0.0.20 gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:52:29 10.0.0.20  SS:ESP 0069:ec847d10
> Oct  4 09:52:29 10.0.0.20 [174345.336513] ---[ end trace 
> 12e6e134d9f4fe6d ]---
> Oct  4 09:52:29 10.0.0.20 [174345.336513] Kernel panic - not syncing: 
> Fatal exception in interrupt
> Oct  4 09:52:29 10.0.0.20 [174345.336513] Pid: 0, comm: swapper Tainted: 
> G      D    2.6.32-5-xen-686 #1
> Oct  4 09:52:29 10.0.0.20 [174345.336513] Call Trace:
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c128c191>] ? panic+0x38/0xe4
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c128e799>] ? 
> oops_end+0x91/0x9d
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1009f21>] ? 
> do_invalid_op+0x0/0x75
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1009f8d>] ? 
> do_invalid_op+0x6c/0x75
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1092fc1>] ? 
> __alloc_pages_nodemask+0xf3/0x4e0
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c128de8b>] ? 
> error_code+0x73/0x78
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c11a49ca>] ? 
> net_tx_action+0x1c8/0xeb1
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10211e3>] ? 
> pvclock_clocksource_read+0xf9/0x10f
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c113d59e>] ? 
> cpumask_next_and+0x23/0x33
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c102f8c8>] ? 
> find_busiest_group+0x2e9/0x6e6
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1034131>] ? 
> rebalance_domains+0x149/0x3e1
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10066d7>] ? 
> xen_restore_fl_direct_end+0x0/0x1
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c128dca1>] ? 
> _spin_unlock_irqrestore+0xd/0xf
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c103b17e>] ? 
> tasklet_action+0x67/0xad
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c103c2e0>] ? 
> __do_softirq+0xaa/0x151
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c103c3b8>] ? 
> do_softirq+0x31/0x3c
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c103c48e>] ? 
> irq_exit+0x26/0x58
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1198c15>] ? 
> xen_evtchn_do_upcall+0x22/0x2c
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1009abf>] ? 
> xen_do_upcall+0x7/0xc
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10023a7>] ? 
> hypercall_page+0x3a7/0x1001
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1006039>] ? 
> xen_safe_halt+0xf/0x1b
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c1004137>] ? 
> xen_idle+0x23/0x30
> Oct  4 09:52:29 10.0.0.20 [174345.336513]  [<c10080a8>] ? 
> cpu_idle+0x89/0xa5
> Oct  4 09:53:33 10.0.0.20 [174410.032507] BUG: soft lockup - CPU#1 stuck 
> for 61s! [swapper:0]
> Oct  4 09:53:33 10.0.0.20 [174410.032507] Modules linked in:
> Oct  4 09:53:33 10.0.0.20  netconsole
> Oct  4 09:53:33 10.0.0.20  configfs
> Oct  4 09:53:33 10.0.0.20  xt_tcpudp
> Oct  4 09:53:33 10.0.0.20  nf_conntrack_ipv4
> Oct  4 09:53:33 10.0.0.20  nf_defrag_ipv4
> Oct  4 09:53:33 10.0.0.20  xt_state
> Oct  4 09:53:33 10.0.0.20  nf_conntrack
> Oct  4 09:53:33 10.0.0.20  xt_physdev
> Oct  4 09:53:33 10.0.0.20  iptable_filter
> Oct  4 09:53:33 10.0.0.20  ip_tables
> Oct  4 09:53:33 10.0.0.20  x_tables
> Oct  4 09:53:33 10.0.0.20  bridge
> Oct  4 09:53:33 10.0.0.20  stp
> Oct  4 09:53:33 10.0.0.20  xen_evtchn
> Oct  4 09:53:33 10.0.0.20  xenfs
> Oct  4 09:53:33 10.0.0.20  nfsd
> Oct  4 09:53:33 10.0.0.20  exportfs
> Oct  4 09:53:33 10.0.0.20  nfs
> Oct  4 09:53:33 10.0.0.20  lockd
> Oct  4 09:53:33 10.0.0.20  fscache
> Oct  4 09:53:33 10.0.0.20  nfs_acl
> Oct  4 09:53:33 10.0.0.20  auth_rpcgss
> Oct  4 09:53:33 10.0.0.20  sunrpc
> Oct  4 09:53:33 10.0.0.20  loop
> Oct  4 09:53:33 10.0.0.20  snd_pcm
> Oct  4 09:53:33 10.0.0.20  snd_timer
> Oct  4 09:53:33 10.0.0.20  snd
> Oct  4 09:53:33 10.0.0.20  soundcore
> Oct  4 09:53:33 10.0.0.20  psmouse
> Oct  4 09:53:33 10.0.0.20  pl2303
> Oct  4 09:53:33 10.0.0.20  snd_page_alloc
> Oct  4 09:53:33 10.0.0.20  usbserial
> Oct  4 09:53:33 10.0.0.20  serio_raw
> Oct  4 09:53:33 10.0.0.20  pcspkr
> Oct  4 09:53:33 10.0.0.20  shpchp
> Oct  4 09:53:33 10.0.0.20  evdev
> Oct  4 09:53:33 10.0.0.20  dcdbas
> Oct  4 09:53:33 10.0.0.20  pci_hotplug
> Oct  4 09:53:33 10.0.0.20  i2c_piix4
> Oct  4 09:53:33 10.0.0.20  i2c_core
> Oct  4 09:53:33 10.0.0.20  processor
> Oct  4 09:53:33 10.0.0.20  acpi_processor
> Oct  4 09:53:33 10.0.0.20  button
> Oct  4 09:53:33 10.0.0.20  ext3
> Oct  4 09:53:33 10.0.0.20  jbd
> Oct  4 09:53:33 10.0.0.20  mbcache
> Oct  4 09:53:33 10.0.0.20  sg
> Oct  4 09:53:33 10.0.0.20  sr_mod
> Oct  4 09:53:33 10.0.0.20  cdrom
> Oct  4 09:53:33 10.0.0.20  sd_mod
> Oct  4 09:53:33 10.0.0.20  crc_t10dif
> Oct  4 09:53:33 10.0.0.20  ata_generic
> Oct  4 09:53:33 10.0.0.20  ohci_hcd
> Oct  4 09:53:33 10.0.0.20  pata_serverworks
> Oct  4 09:53:33 10.0.0.20  megaraid_mbox
> Oct  4 09:53:33 10.0.0.20  megaraid_mm
> Oct  4 09:53:33 10.0.0.20  tg3
> Oct  4 09:53:33 10.0.0.20  floppy
> Oct  4 09:53:33 10.0.0.20  aacraid
> Oct  4 09:53:33 10.0.0.20  libphy
> Oct  4 09:53:33 10.0.0.20  qla1280
> Oct  4 09:53:33 10.0.0.20  ehci_hcd
> Oct  4 09:53:33 10.0.0.20  libata
> Oct  4 09:53:33 10.0.0.20  usbcore
> Oct  4 09:53:33 10.0.0.20  nls_base
> Oct  4 09:53:33 10.0.0.20  thermal
> Oct  4 09:53:33 10.0.0.20  scsi_mod
> Oct  4 09:53:33 10.0.0.20  thermal_sys
> Oct  4 09:53:33 10.0.0.20  [last unloaded: scsi_wait_scan]
> Oct  4 09:53:33 10.0.0.20
> Oct  4 09:53:33 10.0.0.20 [174410.032507]
> Oct  4 09:53:33 10.0.0.20 [174410.032507] Pid: 0, comm: swapper Tainted: 
> G      D    (2.6.32-5-xen-686 #1) PowerEdge 2650
> Oct  4 09:53:33 10.0.0.20 [174410.032507] EIP: 0061:[<c1059e72>] EFLAGS: 
> 00000202 CPU: 1
> Oct  4 09:53:33 10.0.0.20 [174410.032507] EIP is at 
> smp_call_function_many+0xed/0x17c
> Oct  4 09:53:33 10.0.0.20 [174410.032507] EAX: c145a6dc EBX: 00000002 
> ECX: 00000003 EDX: 03cd9000
> Oct  4 09:53:33 10.0.0.20 [174410.032507] ESI: c51336dc EDI: 00000001 
> EBP: c13fa2b4 ESP: ec847ba0
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  DS: 007b ES: 007b FS: 00d8 
> GS: 00e0 SS: 0069
> Oct  4 09:53:33 10.0.0.20 [174410.032507] CR0: 8005003b CR2: 094c7808 
> CR3: 0a1c5000 CR4: 00000660
> Oct  4 09:53:33 10.0.0.20 [174410.032507] DR0: 00000000 DR1: 00000000 
> DR2: 00000000 DR3: 00000000
> Oct  4 09:53:33 10.0.0.20 [174410.032507] DR6: ffff0ff0 DR7: 00000400
> Oct  4 09:53:33 10.0.0.20 [174410.032507] Call Trace:
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1006d5c>] ? 
> stop_self+0x0/0x30
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1059f1a>] ? 
> smp_call_function+0x19/0x1f
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10036be>] ? 
> xen_reboot+0x14/0x28
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1003705>] ? 
> xen_panic_event+0xa/0xd
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c128f990>] ? 
> notifier_call_chain+0x2a/0x47
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c128f9c0>] ? 
> atomic_notifier_call_chain+0xc/0xf
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c128c1af>] ? panic+0x56/0xe4
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c128e799>] ? 
> oops_end+0x91/0x9d
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1009f21>] ? 
> do_invalid_op+0x0/0x75
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1009f8d>] ? 
> do_invalid_op+0x6c/0x75
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1092fc1>] ? 
> __alloc_pages_nodemask+0xf3/0x4e0
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c128de8b>] ? 
> error_code+0x73/0x78
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c11a49ca>] ? 
> net_tx_action+0x1c8/0xeb1
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10211e3>] ? 
> pvclock_clocksource_read+0xf9/0x10f
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c113d59e>] ? 
> cpumask_next_and+0x23/0x33
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c102f8c8>] ? 
> find_busiest_group+0x2e9/0x6e6
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1034131>] ? 
> rebalance_domains+0x149/0x3e1
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10066d7>] ? 
> xen_restore_fl_direct_end+0x0/0x1
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c128dca1>] ? 
> _spin_unlock_irqrestore+0xd/0xf
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c103b17e>] ? 
> tasklet_action+0x67/0xad
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c103c2e0>] ? 
> __do_softirq+0xaa/0x151
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c103c3b8>] ? 
> do_softirq+0x31/0x3c
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c103c48e>] ? 
> irq_exit+0x26/0x58
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1198c15>] ? 
> xen_evtchn_do_upcall+0x22/0x2c
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1009abf>] ? 
> xen_do_upcall+0x7/0xc
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10023a7>] ? 
> hypercall_page+0x3a7/0x1001
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1006039>] ? 
> xen_safe_halt+0xf/0x1b
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c1004137>] ? 
> xen_idle+0x23/0x30
> Oct  4 09:53:33 10.0.0.20 [174410.032507]  [<c10080a8>] ? 
> cpu_idle+0x89/0xa5
> Oct  4 09:54:39 10.0.0.20 [174475.532508] BUG: soft lockup - CPU#1 stuck 
> for 61s! [swapper:0]
> Oct  4 09:54:39 10.0.0.20 [174475.532508] Modules linked in:
> Oct  4 09:54:39 10.0.0.20  netconsole
> Oct  4 09:54:39 10.0.0.20  configfs
> Oct  4 09:54:39 10.0.0.20  xt_tcpudp
> Oct  4 09:54:39 10.0.0.20  nf_conntrack_ipv4
> Oct  4 09:54:39 10.0.0.20  nf_defrag_ipv4
> Oct  4 09:54:39 10.0.0.20  xt_state
> Oct  4 09:54:39 10.0.0.20  nf_conntrack
> Oct  4 09:54:39 10.0.0.20  xt_physdev
> Oct  4 09:54:39 10.0.0.20  iptable_filter
> Oct  4 09:54:39 10.0.0.20  ip_tables
> Oct  4 09:54:39 10.0.0.20  x_tables
> Oct  4 09:54:39 10.0.0.20  bridge
> Oct  4 09:54:39 10.0.0.20  stp
> Oct  4 09:54:39 10.0.0.20  xen_evtchn
> Oct  4 09:54:39 10.0.0.20  xenfs
> Oct  4 09:54:39 10.0.0.20  nfsd
> Oct  4 09:54:39 10.0.0.20  exportfs
> Oct  4 09:54:39 10.0.0.20  nfs
> Oct  4 09:54:39 10.0.0.20  lockd
> Oct  4 09:54:39 10.0.0.20  fscache
> Oct  4 09:54:39 10.0.0.20  nfs_acl
> Oct  4 09:54:39 10.0.0.20  auth_rpcgss
> Oct  4 09:54:39 10.0.0.20  sunrpc
> Oct  4 09:54:39 10.0.0.20  loop
> Oct  4 09:54:39 10.0.0.20  snd_pcm
> Oct  4 09:54:39 10.0.0.20  snd_timer
> Oct  4 09:54:39 10.0.0.20  snd
> Oct  4 09:54:39 10.0.0.20  soundcore
> Oct  4 09:54:39 10.0.0.20  psmouse
> Oct  4 09:54:39 10.0.0.20  pl2303
> Oct  4 09:54:39 10.0.0.20  snd_page_alloc
> Oct  4 09:54:39 10.0.0.20  usbserial
> Oct  4 09:54:39 10.0.0.20  serio_raw
> Oct  4 09:54:39 10.0.0.20  pcspkr
> Oct  4 09:54:39 10.0.0.20  shpchp
> Oct  4 09:54:39 10.0.0.20  evdev
> Oct  4 09:54:39 10.0.0.20  dcdbas
> Oct  4 09:54:39 10.0.0.20  pci_hotplug
> Oct  4 09:54:39 10.0.0.20  i2c_piix4
> Oct  4 09:54:39 10.0.0.20  i2c_core
> Oct  4 09:54:39 10.0.0.20  processor
> Oct  4 09:54:39 10.0.0.20  acpi_processor
> Oct  4 09:54:39 10.0.0.20  button
> Oct  4 09:54:39 10.0.0.20  ext3
> Oct  4 09:54:39 10.0.0.20  jbd
> Oct  4 09:54:39 10.0.0.20  mbcache
> Oct  4 09:54:39 10.0.0.20  sg
> Oct  4 09:54:39 10.0.0.20  sr_mod
> Oct  4 09:54:39 10.0.0.20  cdrom
> Oct  4 09:54:39 10.0.0.20  sd_mod
> Oct  4 09:54:39 10.0.0.20  crc_t10dif
> Oct  4 09:54:39 10.0.0.20  ata_generic
> Oct  4 09:54:39 10.0.0.20  ohci_hcd
> Oct  4 09:54:39 10.0.0.20  pata_serverworks
> Oct  4 09:54:39 10.0.0.20  megaraid_mbox
> Oct  4 09:54:39 10.0.0.20  megaraid_mm
> Oct  4 09:54:39 10.0.0.20  tg3
> Oct  4 09:54:39 10.0.0.20  floppy
> Oct  4 09:54:39 10.0.0.20  aacraid
> Oct  4 09:54:39 10.0.0.20  libphy
> Oct  4 09:54:39 10.0.0.20  qla1280
> Oct  4 09:54:39 10.0.0.20  ehci_hcd
> Oct  4 09:54:39 10.0.0.20  libata
> Oct  4 09:54:39 10.0.0.20  usbcore
> Oct  4 09:54:39 10.0.0.20  nls_base
> Oct  4 09:54:39 10.0.0.20  thermal
> Oct  4 09:54:39 10.0.0.20  scsi_mod
> Oct  4 09:54:39 10.0.0.20  thermal_sys
> Oct  4 09:54:39 10.0.0.20  [last unloaded: scsi_wait_scan]
> Oct  4 09:54:39 10.0.0.20
> Oct  4 09:54:39 10.0.0.20 [174475.532508]
> Oct  4 09:54:39 10.0.0.20 [174475.532508] Pid: 0, comm: swapper Tainted: 
> G      D    (2.6.32-5-xen-686 #1) PowerEdge 2650
> Oct  4 09:54:39 10.0.0.20 [174475.532508] EIP: 0061:[<c1059e6e>] EFLAGS: 
> 00000202 CPU: 1
> Oct  4 09:54:39 10.0.0.20 [174475.532508] EIP is at 
> smp_call_function_many+0xe9/0x17c
> Oct  4 09:54:39 10.0.0.20 [174475.532508] EAX: c145a6dc EBX: 00000002 
> ECX: 00000003 EDX: 03cd9000
> Oct  4 09:54:39 10.0.0.20 [174475.532508] ESI: c51336dc EDI: 00000001 
> EBP: c13fa2b4 ESP: ec847ba0
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  DS: 007b ES: 007b FS: 00d8 
> GS: 00e0 SS: 0069
> Oct  4 09:54:39 10.0.0.20 [174475.532508] CR0: 8005003b CR2: 094c7808 
> CR3: 0a1c5000 CR4: 00000660
> Oct  4 09:54:39 10.0.0.20 [174475.532508] DR0: 00000000 DR1: 00000000 
> DR2: 00000000 DR3: 00000000
> Oct  4 09:54:39 10.0.0.20 [174475.532508] DR6: ffff0ff0 DR7: 00000400
> Oct  4 09:54:39 10.0.0.20 [174475.532508] Call Trace:
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1006d5c>] ? 
> stop_self+0x0/0x30
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1059f1a>] ? 
> smp_call_function+0x19/0x1f
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10036be>] ? 
> xen_reboot+0x14/0x28
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1003705>] ? 
> xen_panic_event+0xa/0xd
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c128f990>] ? 
> notifier_call_chain+0x2a/0x47
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c128f9c0>] ? 
> atomic_notifier_call_chain+0xc/0xf
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c128c1af>] ? panic+0x56/0xe4
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c128e799>] ? 
> oops_end+0x91/0x9d
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1009f21>] ? 
> do_invalid_op+0x0/0x75
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1009f8d>] ? 
> do_invalid_op+0x6c/0x75
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1092fc1>] ? 
> __alloc_pages_nodemask+0xf3/0x4e0
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c128de8b>] ? 
> error_code+0x73/0x78
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c11a49ca>] ? 
> net_tx_action+0x1c8/0xeb1
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10211e3>] ? 
> pvclock_clocksource_read+0xf9/0x10f
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c113d59e>] ? 
> cpumask_next_and+0x23/0x33
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c102f8c8>] ? 
> find_busiest_group+0x2e9/0x6e6
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1034131>] ? 
> rebalance_domains+0x149/0x3e1
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10066d7>] ? 
> xen_restore_fl_direct_end+0x0/0x1
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c128dca1>] ? 
> _spin_unlock_irqrestore+0xd/0xf
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c103b17e>] ? 
> tasklet_action+0x67/0xad
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c103c2e0>] ? 
> __do_softirq+0xaa/0x151
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c103c3b8>] ? 
> do_softirq+0x31/0x3c
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c103c48e>] ? 
> irq_exit+0x26/0x58
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1198c15>] ? 
> xen_evtchn_do_upcall+0x22/0x2c
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1009abf>] ? 
> xen_do_upcall+0x7/0xc
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10023a7>] ? 
> hypercall_page+0x3a7/0x1001
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1006039>] ? 
> xen_safe_halt+0xf/0x1b
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c1004137>] ? 
> xen_idle+0x23/0x30
> Oct  4 09:54:39 10.0.0.20 [174475.532508]  [<c10080a8>] ? 
> cpu_idle+0x89/0xa5
> Oct  4 09:55:44 10.0.0.20 [174541.028507] BUG: soft lockup - CPU#1 stuck 
> for 61s! [swapper:0]
> Oct  4 09:55:44 10.0.0.20 [174541.028507] Modules linked in:
> Oct  4 09:55:44 10.0.0.20  netconsole
> Oct  4 09:55:44 10.0.0.20  configfs
> Oct  4 09:55:44 10.0.0.20  xt_tcpudp
> Oct  4 09:55:44 10.0.0.20  nf_conntrack_ipv4
> Oct  4 09:55:44 10.0.0.20  nf_defrag_ipv4
> Oct  4 09:55:44 10.0.0.20  xt_state
> Oct  4 09:55:44 10.0.0.20  nf_conntrack
> Oct  4 09:55:44 10.0.0.20  xt_physdev
> Oct  4 09:55:44 10.0.0.20  iptable_filter
> Oct  4 09:55:44 10.0.0.20  ip_tables
> Oct  4 09:55:44 10.0.0.20  x_tables
> Oct  4 09:55:44 10.0.0.20  bridge
> Oct  4 09:55:44 10.0.0.20  stp
> Oct  4 09:55:44 10.0.0.20  xen_evtchn
> Oct  4 09:55:44 10.0.0.20  xenfs
> Oct  4 09:55:44 10.0.0.20  nfsd
> Oct  4 09:55:44 10.0.0.20  exportfs
> Oct  4 09:55:44 10.0.0.20  nfs
> Oct  4 09:55:44 10.0.0.20  lockd
> Oct  4 09:55:44 10.0.0.20  fscache
> Oct  4 09:55:44 10.0.0.20  nfs_acl
> Oct  4 09:55:44 10.0.0.20  auth_rpcgss
> Oct  4 09:55:44 10.0.0.20  sunrpc
> Oct  4 09:55:44 10.0.0.20  loop
> Oct  4 09:55:44 10.0.0.20  snd_pcm
> Oct  4 09:55:44 10.0.0.20  snd_timer
> Oct  4 09:55:44 10.0.0.20  snd
> Oct  4 09:55:44 10.0.0.20  soundcore
> Oct  4 09:55:44 10.0.0.20  psmouse
> Oct  4 09:55:44 10.0.0.20  pl2303
> Oct  4 09:55:44 10.0.0.20  snd_page_alloc
> Oct  4 09:55:44 10.0.0.20  usbserial
> Oct  4 09:55:44 10.0.0.20  serio_raw
> Oct  4 09:55:44 10.0.0.20  pcspkr
> Oct  4 09:55:44 10.0.0.20  shpchp
> Oct  4 09:55:44 10.0.0.20  evdev
> Oct  4 09:55:44 10.0.0.20  dcdbas
> Oct  4 09:55:44 10.0.0.20  pci_hotplug
> Oct  4 09:55:44 10.0.0.20  i2c_piix4
> Oct  4 09:55:44 10.0.0.20  i2c_core
> Oct  4 09:55:44 10.0.0.20  processor
> Oct  4 09:55:44 10.0.0.20  acpi_processor
> Oct  4 09:55:44 10.0.0.20  button
> Oct  4 09:55:44 10.0.0.20  ext3
> Oct  4 09:55:44 10.0.0.20  jbd
> Oct  4 09:55:44 10.0.0.20  mbcache
> Oct  4 09:55:44 10.0.0.20  sg
> Oct  4 09:55:44 10.0.0.20  sr_mod
> Oct  4 09:55:44 10.0.0.20  cdrom
> Oct  4 09:55:44 10.0.0.20  sd_mod
> Oct  4 09:55:44 10.0.0.20  crc_t10dif
> Oct  4 09:55:44 10.0.0.20  ata_generic
> Oct  4 09:55:44 10.0.0.20  ohci_hcd
> Oct  4 09:55:44 10.0.0.20  pata_serverworks
> Oct  4 09:55:44 10.0.0.20  megaraid_mbox
> Oct  4 09:55:44 10.0.0.20  megaraid_mm
> Oct  4 09:55:44 10.0.0.20  tg3
> Oct  4 09:55:44 10.0.0.20  floppy
> Oct  4 09:55:44 10.0.0.20  aacraid
> Oct  4 09:55:44 10.0.0.20  libphy
> Oct  4 09:55:44 10.0.0.20  qla1280
> Oct  4 09:55:44 10.0.0.20  ehci_hcd
> Oct  4 09:55:44 10.0.0.20  libata
> Oct  4 09:55:44 10.0.0.20  usbcore
> Oct  4 09:55:44 10.0.0.20  nls_base
> Oct  4 09:55:44 10.0.0.20  thermal
> Oct  4 09:55:44 10.0.0.20  scsi_mod
> Oct  4 09:55:44 10.0.0.20  thermal_sys
> Oct  4 09:55:44 10.0.0.20  [last unloaded: scsi_wait_scan]
> Oct  4 09:55:44 10.0.0.20
> Oct  4 09:55:44 10.0.0.20 [174541.028507]
> Oct  4 09:55:44 10.0.0.20 [174541.028507] Pid: 0, comm: swapper Tainted: 
> G      D    (2.6.32-5-xen-686 #1) PowerEdge 2650
> Oct  4 09:55:44 10.0.0.20 [174541.028507] EIP: 0061:[<c1059e6e>] EFLAGS: 
> 00000202 CPU: 1
> Oct  4 09:55:44 10.0.0.20 [174541.028507] EIP is at 
> smp_call_function_many+0xe9/0x17c
> Oct  4 09:55:44 10.0.0.20 [174541.028507] EAX: c145a6dc EBX: 00000002 
> ECX: 00000003 EDX: 03cd9000
> Oct  4 09:55:44 10.0.0.20 [174541.028507] ESI: c51336dc EDI: 00000001 
> EBP: c13fa2b4 ESP: ec847ba0
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  DS: 007b ES: 007b FS: 00d8 
> GS: 00e0 SS: 0069
> Oct  4 09:55:44 10.0.0.20 [174541.028507] CR0: 8005003b CR2: 094c7808 
> CR3: 0a1c5000 CR4: 00000660
> Oct  4 09:55:44 10.0.0.20 [174541.028507] DR0: 00000000 DR1: 00000000 
> DR2: 00000000 DR3: 00000000
> Oct  4 09:55:44 10.0.0.20 [174541.028507] DR6: ffff0ff0 DR7: 00000400
> Oct  4 09:55:44 10.0.0.20 [174541.028507] Call Trace:
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1006d5c>] ? 
> stop_self+0x0/0x30
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1059f1a>] ? 
> smp_call_function+0x19/0x1f
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10036be>] ? 
> xen_reboot+0x14/0x28
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1003705>] ? 
> xen_panic_event+0xa/0xd
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c128f990>] ? 
> notifier_call_chain+0x2a/0x47
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c128f9c0>] ? 
> atomic_notifier_call_chain+0xc/0xf
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c128c1af>] ? panic+0x56/0xe4
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c128e799>] ? 
> oops_end+0x91/0x9d
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1009f21>] ? 
> do_invalid_op+0x0/0x75
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1009f8d>] ? 
> do_invalid_op+0x6c/0x75
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1092fc1>] ? 
> __alloc_pages_nodemask+0xf3/0x4e0
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c128de8b>] ? 
> error_code+0x73/0x78
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c11a49ca>] ? 
> net_tx_action+0x1c8/0xeb1
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10211e3>] ? 
> pvclock_clocksource_read+0xf9/0x10f
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c113d59e>] ? 
> cpumask_next_and+0x23/0x33
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c102f8c8>] ? 
> find_busiest_group+0x2e9/0x6e6
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1034131>] ? 
> rebalance_domains+0x149/0x3e1
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10066d7>] ? 
> xen_restore_fl_direct_end+0x0/0x1
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c128dca1>] ? 
> _spin_unlock_irqrestore+0xd/0xf
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c103b17e>] ? 
> tasklet_action+0x67/0xad
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c103c2e0>] ? 
> __do_softirq+0xaa/0x151
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c103c3b8>] ? 
> do_softirq+0x31/0x3c
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c103c48e>] ? 
> irq_exit+0x26/0x58
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1198c15>] ? 
> xen_evtchn_do_upcall+0x22/0x2c
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1009abf>] ? 
> xen_do_upcall+0x7/0xc
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10023a7>] ? 
> hypercall_page+0x3a7/0x1001
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1006039>] ? 
> xen_safe_halt+0xf/0x1b
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c1004137>] ? 
> xen_idle+0x23/0x30
> Oct  4 09:55:44 10.0.0.20 [174541.028507]  [<c10080a8>] ? 
> cpu_idle+0x89/0xa5
> Oct  4 09:56:50 10.0.0.20 [174606.528507] BUG: soft lockup - CPU#1 stuck 
> for 61s! [swapper:0]
> Oct  4 09:56:50 10.0.0.20 [174606.528507] Modules linked in:
> Oct  4 09:56:50 10.0.0.20  netconsole
> Oct  4 09:56:50 10.0.0.20  configfs
> Oct  4 09:56:50 10.0.0.20  xt_tcpudp
> Oct  4 09:56:50 10.0.0.20  nf_conntrack_ipv4
> Oct  4 09:56:50 10.0.0.20  nf_defrag_ipv4
> Oct  4 09:56:50 10.0.0.20  xt_state
> Oct  4 09:56:50 10.0.0.20  nf_conntrack
> Oct  4 09:56:50 10.0.0.20  xt_physdev
> Oct  4 09:56:50 10.0.0.20  iptable_filter
> Oct  4 09:56:50 10.0.0.20  ip_tables
> Oct  4 09:56:50 10.0.0.20  x_tables
> Oct  4 09:56:50 10.0.0.20  bridge
> Oct  4 09:56:50 10.0.0.20  stp
> Oct  4 09:56:50 10.0.0.20  xen_evtchn
> Oct  4 09:56:50 10.0.0.20  xenfs
> Oct  4 09:56:50 10.0.0.20  nfsd
> Oct  4 09:56:50 10.0.0.20  exportfs
> Oct  4 09:56:50 10.0.0.20  nfs
> Oct  4 09:56:50 10.0.0.20  lockd
> Oct  4 09:56:50 10.0.0.20  fscache
> Oct  4 09:56:50 10.0.0.20  nfs_acl
> Oct  4 09:56:50 10.0.0.20  auth_rpcgss
> Oct  4 09:56:50 10.0.0.20  sunrpc
> Oct  4 09:56:50 10.0.0.20  loop
> Oct  4 09:56:50 10.0.0.20  snd_pcm
> Oct  4 09:56:50 10.0.0.20  snd_timer
> Oct  4 09:56:50 10.0.0.20  snd
> Oct  4 09:56:50 10.0.0.20  soundcore
> Oct  4 09:56:50 10.0.0.20  psmouse
> Oct  4 09:56:50 10.0.0.20  pl2303
> Oct  4 09:56:50 10.0.0.20  snd_page_alloc
> Oct  4 09:56:50 10.0.0.20  usbserial
> Oct  4 09:56:50 10.0.0.20  serio_raw
> Oct  4 09:56:50 10.0.0.20  pcspkr
> Oct  4 09:56:50 10.0.0.20  shpchp
> Oct  4 09:56:50 10.0.0.20  evdev
> Oct  4 09:56:50 10.0.0.20  dcdbas
> Oct  4 09:56:50 10.0.0.20  pci_hotplug
> Oct  4 09:56:50 10.0.0.20  i2c_piix4
> Oct  4 09:56:50 10.0.0.20  i2c_core
> Oct  4 09:56:50 10.0.0.20  processor
> Oct  4 09:56:50 10.0.0.20  acpi_processor
> Oct  4 09:56:50 10.0.0.20  button
> Oct  4 09:56:50 10.0.0.20  ext3
> Oct  4 09:56:50 10.0.0.20  jbd
> Oct  4 09:56:50 10.0.0.20  mbcache
> Oct  4 09:56:50 10.0.0.20  sg
> Oct  4 09:56:50 10.0.0.20  sr_mod
> Oct  4 09:56:50 10.0.0.20  cdrom
> Oct  4 09:56:50 10.0.0.20  sd_mod
> Oct  4 09:56:50 10.0.0.20  crc_t10dif
> Oct  4 09:56:50 10.0.0.20  ata_generic
> Oct  4 09:56:50 10.0.0.20  ohci_hcd
> Oct  4 09:56:50 10.0.0.20  pata_serverworks
> Oct  4 09:56:50 10.0.0.20  megaraid_mbox
> Oct  4 09:56:50 10.0.0.20  megaraid_mm
> Oct  4 09:56:50 10.0.0.20  tg3
> Oct  4 09:56:50 10.0.0.20  floppy
> Oct  4 09:56:50 10.0.0.20  aacraid
> Oct  4 09:56:50 10.0.0.20  libphy
> Oct  4 09:56:50 10.0.0.20  qla1280
> Oct  4 09:56:50 10.0.0.20  ehci_hcd
> Oct  4 09:56:50 10.0.0.20  libata
> Oct  4 09:56:50 10.0.0.20  usbcore
> Oct  4 09:56:50 10.0.0.20  nls_base
> Oct  4 09:56:50 10.0.0.20  thermal
> Oct  4 09:56:50 10.0.0.20  scsi_mod
> Oct  4 09:56:50 10.0.0.20  thermal_sys
> Oct  4 09:56:50 10.0.0.20  [last unloaded: scsi_wait_scan]
> Oct  4 09:56:50 10.0.0.20
> Oct  4 09:56:50 10.0.0.20 [174606.528507]
> Oct  4 09:56:50 10.0.0.20 [174606.528507] Pid: 0, comm: swapper Tainted: 
> G      D    (2.6.32-5-xen-686 #1) PowerEdge 2650
> Oct  4 09:56:50 10.0.0.20 [174606.528507] EIP: 0061:[<c1059e6e>] EFLAGS: 
> 00000202 CPU: 1
> Oct  4 09:56:50 10.0.0.20 [174606.528507] EIP is at 
> smp_call_function_many+0xe9/0x17c
> Oct  4 09:56:50 10.0.0.20 [174606.528507] EAX: c145a6dc EBX: 00000002 
> ECX: 00000003 EDX: 03cd9000
> Oct  4 09:56:50 10.0.0.20 [174606.528507] ESI: c51336dc EDI: 00000001 
> EBP: c13fa2b4 ESP: ec847ba0
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  DS: 007b ES: 007b FS: 00d8 
> GS: 00e0 SS: 0069
> Oct  4 09:56:50 10.0.0.20 [174606.528507] CR0: 8005003b CR2: 094c7808 
> CR3: 0a1c5000 CR4: 00000660
> Oct  4 09:56:50 10.0.0.20 [174606.528507] DR0: 00000000 DR1: 00000000 
> DR2: 00000000 DR3: 00000000
> Oct  4 09:56:50 10.0.0.20 [174606.528507] DR6: ffff0ff0 DR7: 00000400
> Oct  4 09:56:50 10.0.0.20 [174606.528507] Call Trace:
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1006d5c>] ? 
> stop_self+0x0/0x30
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1059f1a>] ? 
> smp_call_function+0x19/0x1f
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10036be>] ? 
> xen_reboot+0x14/0x28
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1003705>] ? 
> xen_panic_event+0xa/0xd
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c128f990>] ? 
> notifier_call_chain+0x2a/0x47
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c128f9c0>] ? 
> atomic_notifier_call_chain+0xc/0xf
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c128c1af>] ? panic+0x56/0xe4
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c128e799>] ? 
> oops_end+0x91/0x9d
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1009f21>] ? 
> do_invalid_op+0x0/0x75
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1009f8d>] ? 
> do_invalid_op+0x6c/0x75
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1092fc1>] ? 
> __alloc_pages_nodemask+0xf3/0x4e0
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c128de8b>] ? 
> error_code+0x73/0x78
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c119803b>] ? 
> gnttab_copy_grant_page+0x17d/0x22c
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c11a49ca>] ? 
> net_tx_action+0x1c8/0xeb1
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10211e3>] ? 
> pvclock_clocksource_read+0xf9/0x10f
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c113d59e>] ? 
> cpumask_next_and+0x23/0x33
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c102f8c8>] ? 
> find_busiest_group+0x2e9/0x6e6
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1034131>] ? 
> rebalance_domains+0x149/0x3e1
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1005fb8>] ? 
> xen_force_evtchn_callback+0xc/0x10
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10066e0>] ? 
> check_events+0x8/0xc
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10066d7>] ? 
> xen_restore_fl_direct_end+0x0/0x1
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c128dca1>] ? 
> _spin_unlock_irqrestore+0xd/0xf
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c103b17e>] ? 
> tasklet_action+0x67/0xad
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c103c2e0>] ? 
> __do_softirq+0xaa/0x151
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c103c3b8>] ? 
> do_softirq+0x31/0x3c
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c103c48e>] ? 
> irq_exit+0x26/0x58
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1198c15>] ? 
> xen_evtchn_do_upcall+0x22/0x2c
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1009abf>] ? 
> xen_do_upcall+0x7/0xc
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10023a7>] ? 
> hypercall_page+0x3a7/0x1001
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1006039>] ? 
> xen_safe_halt+0xf/0x1b
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c1004137>] ? 
> xen_idle+0x23/0x30
> Oct  4 09:56:50 10.0.0.20 [174606.528507]  [<c10080a8>] ? 
> cpu_idle+0x89/0xa5
> 
> 
> 
> 

-- 
Ian Campbell
Current Noise: Slayer - Born Of Fire

If at first you don't succeed, quit; don't be a nut about success.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel