WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 		  
Home Products Support Community News
 
   
 

xen-devel

[Top] [All Lists]
<prev [Date] next>
[Advanced]
 <prev [Thread] next>

[Xen-devel] [PATCH] xen: reduce # of IPI sent for machine_restart

from [Ryan Harper] [Permanent Link][Original]
To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] [PATCH] xen: reduce # of IPI sent for machine_restart
From: Ryan Harper <ryanh@xxxxxxxxxx>
Date: Wed, 26 Apr 2006 17:26:15 -0500
Delivery-date: Wed, 26 Apr 2006 15:26:36 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.6+20040907i 
On our x460 4-node 32-way, I occasionally fail to restart and hang in
xen's reboot code.  Examining machine_restart[1], the code ends up
sending IPI's to every other cpu, which in turn send more IPIs.
Eventually the boot cpu runs machine_restart, but sometimes this breaks.
We only need to send one IPI to the boot cpu to run machine_restart code.
AFAICT[2], logical CPU0 is always going to be the boot processor.

1. xen/arch/x86/shutdown.c:203
2. xen/arch/x86/smpboot.c:1025 says:
    x86_cpu_to_apicid[0] = boot_cpu_physical_apicid;

Here is the failure I see on the serial console:

.
(XEN) Domain 0 shutdown: rebooting machine.
(XEN) **(XiElNe=)e ((fxXitaElbNel)=e. ecx,t (alifibnel=e7=7l)e .Pcr,e 
-leeinxet=x7ca7epbl)tieo. nP:r efc-ff,f 8l3i0ne0x0c0e1p2t3eie=o1b7n :- >f 
7f00)0 0P0f0f08r030e0-0e0xc000e0p0t0i0on0
:( 12X3EeN1)b Af -s>s ef0rtf00i0of0n 800'0l0e30.c00,000 00li00n10e23=
7(e67XE6) N -P) >r( e-f00eil0xce00e=e0ptx00ita0onb00:le0 f.00fc,0ff 08l
3i0n(X0e=E007N)17) 2( 8afPrcilea e-e-=ex> xce0tap00bti0leo00.n:0c, 00 
ff0lif00nf80e=300700070) 
0(P12Xre8EN-ac)exa  oce-n:p>  ti0ffo00fn:0f8 003ff000f000f80013020000800ac0010a 
2-8
(>X E0aN00c)0a *0*0 *00-*0>**0*0 
*000**0*0*00***0*0*0*0**0**00***0**0**00***0**00**********
***(nXtEeNd) * ]--------[n 
s(tXXaeEnN-)3 . 0]b--l-e- - * 
u***************************************************************
*
n(0X0E1N4)e (fCXdPUE2>:N]  
 ) (X E2N4C) 
4PUef: d2  >] (2afibllee=.cex, taliblnee.=7c,7 ) liPrnee-=7ex7)ce Pprtie-onex: 
ceffptfif8on30: 00f0ff12f81d309f00 -01> 2100d900f 0-00> 0000000000000000000
(0X00EN00) 0f
83(0XE00N)01 -21-d--9f[  -Xe> n-003.000-00un00st00ab00le00  00  CPffUf: f8  30 
105001
(29XE66N6) >R]I
P(: X EN e)0 C10P:U:[ < f ff 4f83
(0X0E0N01)4 ReIfPc6: > ]- -t-ab-l[ eX  en - N3o.t0- tuanistnatebdle  ]- -  
-N-o--t -t-[a iXnteend-3 .]-0--u-n-st
(aXbEleN )   - -N--o[t  tXaeni-n3te.d0- u]n--s-ta-bl
(eX:[E<N)f fCffP8U:30 0  00 6121
d(X9fEN>]) 
R(IIXEP:N)    CP eU0: 1 0:  [<25fff
(f8XE3N00) 0R01IP4e:f  d2 > e]0-e--  -[   NXeotn- t3.a0in-utendst ]ab--l-e -  a 
 ckNo_AtP tIaC_inirteq+d 0]x0--/0--x15

(XEN) RFLAGS: 0000000000010082   CONTEXT: hypervisor
(XEN) rax: ffff8300001e1018   rbx: ffff830000218300   rcx: 000000000000000f
(XEN) rdx: 000000000000001f   rsi: 0000000000000020   rdi: ffff830000fd7028
(XEN) rbp: ffff830000fd7018   rsp: ffff830000fd7000   r8:  0000000000000000
(XEN)  strnlen+0x0/0x45 bitmap_weight+0x0/0x1dAssertion 'spin_is_locked(lock)' 
failed, line 37, file 
/home/rharper/work/openhype/xen/unstable/hg/e/xen/include/asm/spinlock.h
(XEN) BUG at 
/home/rharper/work/openhype/xen/unstable/hg/e/xen/include/asm/spinlock.h:37
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    5
(XEN) RIP:    e010:[<ffff830000129666>]************************************
(XEN)  ack_APIC_irq+0x0/0x15(file=extable.c, line=77) Pre-exception: 
ffff8300001281ef -> 0000000000000000
(XEN) 
(XEN) RFLAGS: 0000000000010096   CONTEXT: hypervisor
(XEN) rax: 00000000ffffffff   rbx: ffff830000218300   rcx: ffff830000fef138
(XEN) 
(XEN) RFLAGS: 0000000000010082   CONTEXT: hypervisor
(XEN) rax: ffff8300001e0434   rbx: ffff830000218300   rcx: 000000000000000f
(XEN) rdx: 0000000000000000   rsi: ffff83000030b218   rdi: ffff830000313028
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    19
(XEN) RIP:    e010:[<ffff830000123e1b>]rdx: ffff830000fef170   rsi: 
ffffffffffffffff   rdi: ffff8300001cbfb1
(XEN) rbp: ffff830000fef108   rsp: ffff830000fef000   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff830000feff28   r11: 0000000000000246
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000d9411000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff830000fef000:
(XEN)    ffff8300001245f5Assertion 'diff < STACK_SIZE' failed, line 31, file 
traps.c
(XEN) BUG at traps.c:31
(XEN) rbp: ffff830000313018   rsp: ffff830000313000   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff830000313f28   r11: 0000000000000246
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 0000000010901000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff830000313000:
(XEN)   (file=extable.c, line=77) Pre-exception: ffff8300001b614b -> 
0000000000000000
(XEN) Assertion 'spin_is_locked(lock)' failed, line 37, file 
/home/rharper/work/openhype/xen/unstable/hg/e/xen/include/asm/spinlock.h
(XEN) BUG at 
/home/rharper/work/openhype/xen/unstable/hg/e/xen/include/asm/spinlock.h:37
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    8
(XEN) RIP:    e010:[<ffff830000128aca>](file=extable.c, line=77) Pre-exception: 
ffff8300001281ef -> 0000000000000000
(XEN)  0000000000000000 0000000000000000CPU:    2
(XEN) RIP:    e010:[<ffff830000105e08>]CPU:    0
(XEN) RIP:    e010:[<ffff83000014d241>] __bitmap_weight+0x99/0xe0
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
(XEN)  atomic_inc+0x0/0x19
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) rax: ffff83000014e3ae   rbx: ffff830000218300   rcx: 000000000000000f
(XEN) rdx: ffff8300003233bc   rsi: ffff8300003233c8   rdi: ffff8300001e0e6c
(XEN) rbp: ffff8300001ff008   rsp: ffff8300001ff000   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff8300001fff28   r11: 0000000000000246
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000cabeb000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN)  serial_start_sync+0x33/0xd1
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) rax: ffff8300001e03c0   rbx: ffff830000218300   rcx: 000000000000000f
(XEN) rdx: 0000000000000000   rsi: ffff83000030b218   rdi: 0000000000000008
(XEN) rbp: ffff830000fe3008   rsp: ffff830000fe2fe8   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff830000fe3f28   r11: 0000000000000246
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000dc055000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff830000fe2fe8:
(XEN)   ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e010:[<ffff8300001b614b>](file=extable.c, line=77) Pre-exception: 
ffff830000152134 -> 0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff830000fd7f28   r11: 0000000000000246
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) 
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff830000218300   rcx: 0000000000000001
(XEN) rdx: 0000000000000000   rsi: 0000000000000020   rdi: ffff83000031b080
(XEN) rbp: ffff83000031b018   rsp: ffff83000031b000   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff83000031bf28   r11: 0000000000000246
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000cc143000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff83000031b000:
(XEN)    ffff83000014efc4Assertion 'diff < STACK_SIZE' failed, line 38, file 
traps.c
(XEN) BUG at traps.c:38
(XEN)  0000002000000000 bitmap_weight+0xc/0x1drax: 0000000000000020   rbx: 
ffff8300003e8080   rcx: 0000000000000001
(XEN) Xen stack trace from rsp=ffff8300001ff000:
(XEN)   rdx: 00000000fffffffb   rsi: 0000000000000020   rdi: ffff830000ff70d0
(XEN) rbp: ffff830000ff7028   rsp: ffff830000ff6ff8   r8:  0000000000000000
(XEN) r9:  00000000deadbeef   r10: ffff830000ff7f28   r11: 0000000000000202
(XEN) r12: 00000000fee1dead   r13: 0000000000000008   r14: 0000000000000001
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000c9671000
(XEN) (file=extable.c, line=77) Pre-exception: ffff8300001b62e9 -> 
0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000c5505000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e010:[<ffff8300001b62e9>]
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff830000218300   rcx: 0000000000000001
(XEN) rdx: 0000000000000000   rsi: 0000000000000020   rdi: ffff830000357090
(XEN)  show_registers+0x249/0x718
(XEN) RFLAGS: 0000000000010082   CONTEXT: hypervisor
(XEN) rax: ffff8300001e0304   rbx: ffff830000218300   rcx: 0000000000001e8d
(XEN) rdx: 0000000000000000   rsi: 0000000000000001   rdi: ffff8300001e0304
(XEN) rbp: ffff830000fe0058   rsp: ffff830000fdfee8   r8:  00000000ffffffff
(XEN) r9:  00000000ffffffff   r10: ffff83000021a9bf   r11: ffff83000021a5cf
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000dc055000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff830000fdfee8:
(XEN)    0a00000000000400 ffff8300001e0434 ffff830000129df3 0000000000000000 
show_registers+0xab/0x718 ffff830000fdff38 0000000000000082
(XEN)   ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN)  ffff83000014d23fXen stack trace from rsp=ffff830000ff6ff8:
(XEN)    ffff8300001ff028Assertion 'diff < STACK_SIZE' failed, line 31, file 
traps.c
(XEN) BUG at traps.c:31
(XEN)  ffff83000014e3de ffff8300003233c8
(XEN)    0000000000000000 ffff8300001ff058
(XEN) RFLAGS: 0000000000010082   CONTEXT: hypervisor
(XEN) rax: ffff8300001e0304   rbx: ffff830000fea080   rcx: 00000000000010c0
(XEN) rdx: 0000000000000000   rsi: 0000000000000001   rdi: ffff8300001e0304
(XEN) rbp: ffff830000fe3f28   rsp: ffff830000fe3db8   r8:  00000000ffffffff
(XEN) r9:  00000000ffffffff   r10: ffff83000021a9bf   r11: ffff83000021a5cf
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000cc143000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff830000fe3db8:
(XEN)   
(XEN)    ffff83000031b080 ffff83000031b088
(XEN)    0aff830000fe3de8 0000000000000000 0000000000000000 ffff8300001e0434 
ffff83000014ee9e 0000000000000000 ffff830000fe3e08 ffff830000313028 
ffff83000014f0e1 fffffffffffffe80 00ff83000021a9bf 0000000000000082 
ffff8300001e03c0
(XEN)    ffff8300001cb9a2 0000000a00124738 0000000a00fdff58 0000000000000001
(XEN)    0000000a00000400 ffff8300001e0304 ffff830000fe0068 0000000000000082
(XEN)    ffff8300001268a7 0000000000000082 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 ffff830000fe0188 
ffff830000218300Assertion 'diff < STACK_SIZE' failed, line 31, file traps.c
(XEN)  number+0x47/0x38f
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff8300001cb4c0   rcx: 000000000000000a
(XEN) rdx: 0000000000000025   rsi: ffff83000021a9bf   rdi: ffff83000021a5ee
(XEN) rbp: ffff83000037f0a8   rsp: ffff83000037efe8   r8:  00000000ffffffff
(XEN) r9:  00000000ffffffff   r10: ffff83000021a9bf   r11: ffff83000021a5ee
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000c965d000
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN) Xen stack trace from rsp=ffff83000037efe8:
(XEN)   ----[ Xen-3.0-unstable    Not tainted ]----
(XEN)  0000000000000000 0000000000000082 00007cffffcecfb7 0000000000000000
(XEN)   
(XEN)   (file=extable.c, line=77) Pre-exception: ffff830000152134 -> 
0000000000000000
(XEN)  0000000000000000CPU:    1016
(XEN) RIP:    e010:[<ffff8300001b614b>]
(XEN)    ffffffffffffffb0
(XEN)    00ff83000021a9bf ffff8300001394e0 show_registers+0xab/0x718 
0000000000000000 ffff83000014e3ae
(XEN)    ffff83000021a5fb ffff83000021a9bf 00000000ffffffff 0000000000000010
(XEN)    ffff830000fe0178 ffff8300001ff068 00007cffffe00f77
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: ffff8300001e0304   rbx: ffff8300003e8080   rcx: 00000000000026e3
(XEN) rdx: 0000000000000000   rsi: 0000000000000001   rdi: ffff8300001e0304
(XEN) rbp: ffff830000ff4068   rsp: ffff830000ff3ef8   r8:  00000000ffffffff
(XEN) r9:  00000000ffffffff   r10: ffff83000021a9bf   r11: ffff83000021a5cf
(XEN) r12: 00000000fee1dead   r13: 0000000000000008   r14: 0000000000000001
(XEN)  0000000000001a5c 0000000000000000 0000000000000000 0000000000000000 
0000000000000082 0000000000000000 0000000000000000 0000000000000000 
ffff8300001e03c0 ffff83000031b098
(XEN)    0000000000000000 0000000000000000
(XEN)    0000004e00000000 0000000000000000 0000000000000000
(XEN)    ffff830000fe2fe8 00000000000003f8 0000000000000000 ffff8300001394a0 
0000000000000000 0000000000000000 0000000000000000
(XEN)   r15: 0000000000000000   cr0: 000000008005003b   cr3: 00000000c9671000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e018   cs: e010
(XEN)  0000000000000000 ffff8300001ff118Xen stack trace from 
rsp=ffff830000ff3ef8:
(XEN)   Xen stack trace from rsp=ffff830000fd7000:
(XEN)    ffff830000129df3
(XEN)    ffff8300001e0304 0000000e00000000 ffff830000152134
(XEN) Xen call trace:
(XEN)    [<ffff8300001b62e9>] 0000000000000000 ffff830000fd7028 00007cffff028fb7
(XEN)    show_registers+0x249/0x718
(XEN)    
(XEN) ************************************
(XEN) CPU0 FATAL TRAP 6 (invalid opcode), ERROR_CODE 0000, IN INTERRUPT CONTEXT.
(XEN) System shutting down -- need manual reset.
(XEN) ************************************

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@xxxxxxxxxx


diffstat output:
 shutdown.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletion(-)

Signed-off-by: Ryan Harper <ryanh@xxxxxxxxxx>
---
# HG changeset patch
# User rharper@xxxxxxxxxxxxxxxxxxxxxxxxx
# Node ID afe3fd9d527c05c93b6ea1d9499d9add51db8383
# Parent  428babd7c1e0a51aadb03a4af2e5607a364aebd8
Occasionally large smp machines fail to reboot properly and die under an IPI
storm of smp_call_function() to machine_reboot.  Only the boot processor needs
to run machine_restart, so send an IPI to CPU0.

diff -r 428babd7c1e0 -r afe3fd9d527c xen/arch/x86/shutdown.c
--- a/xen/arch/x86/shutdown.c   Tue Apr 25 10:12:16 2006
+++ b/xen/arch/x86/shutdown.c   Thu Apr 27 06:27:43 2006
@@ -20,6 +20,7 @@
 #include <xen/irq.h>
 #include <xen/console.h>
 #include <asm/msr.h>
+#include <asm/smpboot.h>
 
 /* opt_noreboot: If true, machine will need manual reset on error. */
 static int opt_noreboot = 0;
@@ -218,7 +219,12 @@
     /* Ensure we are the boot CPU. */
     if ( GET_APIC_ID(apic_read(APIC_ID)) != boot_cpu_physical_apicid )
     {
-        smp_call_function((void *)machine_restart, NULL, 1, 0);
+        cpumask_t mask = CPU_MASK_NONE;
+
+        cpu_set(0, mask); /* boot CPU is logical cpu 0 */
+
+        /* only send IPI to boot CPU. */
+        on_selected_cpus(mask, (void *)machine_restart, NULL, 1, 0);
         for ( ; ; )
             safe_halt();
     }

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
  • [Xen-devel] [PATCH] xen: reduce # of IPI sent for machine_restart, Ryan Harper <=
Previous by Date:  Re: [Xen-devel] Save/restore, migration of HVM-guest's, Mark Williamson
Next by Date:  Re: [Xen-devel] xm-test results, Ewan Mellor
Previous by Thread:  [Xen-devel] Save/restore, migration of HVM-guest's, Thorolf Godawa
Next by Thread:  [Xen-devel] [patch] fix the cdrom is not bootable, but boot device is cdrom., chenjunb
Indexes:  [Date] [Thread] [Top] [All Lists]
 
   

Copyright , Citrix Systems Inc. All rights reserved. Legal and Privacy