[Xen-bugs] [Bug 1746] New: Dom0 Locked up for 4 hours "BUG: soft lockup - CPU#3 stuck for 61s!"

[bugzilla-daemon@xxxxxxxxxxxxxxxxxxx]

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1746

           Summary: Dom0 Locked up for 4 hours "BUG: soft lockup - CPU#3
                    stuck for 61s!"
           Product: Xen
           Version: 1.0
          Platform: x86-64
        OS/Version: All
            Status: NEW
          Severity: critical
          Priority: P2
         Component: Cloud Xen
        AssignedTo: xen-bugs@xxxxxxxxxxxxxxxxxxx
        ReportedBy: jfrias@xxxxxxxxx


We have a deployment of about 10 xcp 1.0 beta xen servers, and just had one
server had a very odd issue. Dom0 became unresponsive ( although xenapi
somewhat worked for querying ) for approximately 4 hours. It then recovered
itself.

We had done no changes on Dom0 and only had changed one of the Domu's to turn
off irqbalance per discussion here
http://forums.citrix.com/thread.jspa?threadID=272708&tstart=0

we are running XCP release 1.0.0-38754c (xcp)


Here's some output from dmesg + sar 

======= DMESG ==========

BUG: soft lockup - CPU#3 stuck for 61s! [swapper:0]
Modules linked in: nls_utf8 hfsplus bonding tun lockd sunrpc bridge stp llc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp
x_tables binfmt_misc dm_mirror video output sbs sbshc fan container battery ac
parport_pc lp parport nvram joydev sr_mod cdrom evdev usb_storage usb_libusual
usbhid sg thermal button processor thermal_sys bnx2 serio_raw 8250_pnp rtc_cmos
8250 serial_core rtc_core tpm_tis rtc_lib tpm tpm_bios pcspkr dm_region_hash
dm_log dm_mod ide_gd_mod megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd usbcore fbcon font tileblit bitblit softcursor [last
unloaded: ip_tables]

Pid: 0, comm: swapper Not tainted (2.6.32.12-0.7.1.xs1.0.0.298.170582xen #1)
PowerEdge R710
EIP: 0061:[<c01013a7>] EFLAGS: 00000246 CPU: 3
EIP is at 0xc01013a7
EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: ee853f78
ESI: 00117f39 EDI: 00000003 EBP: ee853f90 ESP: ee853f74
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
CR0: 8005003b CR2: b7736000 CR3: 0e713000 CR4: 00002660
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
Call Trace:
 [<c0107035>] ? xen_safe_halt+0xb5/0x150
 [<c010ac7e>] xen_idle+0x1e/0x50
 [<c0102a7b>] cpu_idle+0x3b/0x60
 [<c037b00d>] cpu_bringup_and_idle+0xd/0x10
BUG: soft lockup - CPU#3 stuck for 61s! [swapper:0]
Modules linked in: nls_utf8 hfsplus bonding tun lockd sunrpc bridge stp llc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp
x_tables binfmt_misc dm_mirror video output sbs sbshc fan container battery ac
parport_pc lp parport nvram joydev sr_mod cdrom evdev usb_storage usb_libusual
usbhid sg thermal button processor thermal_sys bnx2 serio_raw 8250_pnp rtc_cmos
8250 serial_core rtc_core tpm_tis rtc_lib tpm tpm_bios pcspkr dm_region_hash
dm_log dm_mod ide_gd_mod megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd usbcore fbcon font tileblit bitblit softcursor [last
unloaded: ip_tables]


======= SAR -B ==========
06:30:01 AM  pgpgin/s pgpgout/s   fault/s  majflt/s
07:20:01 AM    106.28   8604.10     19.54      0.00
07:30:01 AM    161.79   8180.32     34.22      0.00
11:33:56 AM  11315.18   1588.36     63.54      0.00
11:40:01 AM     28.68    798.89    694.91      0.03
11:50:01 AM      0.29    804.20     72.90      0.00

06:30:01 AM       CPU     %user     %nice   %system   %iowait    %steal    
%idle
07:20:01 AM       all      0.18      0.00      2.30      0.15      1.14    
96.23
07:30:01 AM       all      0.15      0.00      2.53      0.08      1.08    
96.16
11:33:56 AM       all      0.08      0.00      0.93      0.01      0.78    
98.20
11:40:01 AM       all      0.96      0.00      0.42      0.05      0.33    
98.24

06:30:01 AM       tps      rtps      wtps   bread/s   bwrtn/s
07:20:01 AM   2828.30    114.11   2714.19    451.26  34424.36
07:30:01 AM   3015.90     98.68   2917.23    670.15  32729.35
11:33:56 AM    497.94    299.28    198.66  45260.70   6356.23
11:40:01 AM 11165955.42 11757009.62 11187596.79 11681878.85 5099265.21
11:50:01 AM    174.25      0.08    174.17      1.15   3218.26
12:00:01 PM    192.56      0.05    192.51      0.19   3386.10

06:30:01 AM    proc/s
07:20:01 AM      0.14
07:30:01 AM      0.34
11:33:56 AM      0.70
11:40:01 AM      3.49
11:50:01 AM      0.27


-- 
Configure bugmail: 
http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-bugs