WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 		  
Home Products Support Community News
 
   
 

xen-users

[Top] [All Lists]
<prev [Date] next>
[Advanced]
 <prev [Thread] next>

[Xen-users] Re; System Crash ...

from [Gareth Bult] [Permanent Link][Original]
To: Xen-Users <xen-users@xxxxxxxxxxxxxxxxxxx>
Subject: [Xen-users] Re; System Crash ...
From: Gareth Bult <gareth@xxxxxxxxxxxxx>
Date: Fri, 18 Jan 2008 16:03:26 +0000 (GMT)
Delivery-date: Fri, 18 Jan 2008 08:04:30 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Hi,

I've just has a rather nasty system crash but it's generated lots of nice logging ..

Anyone know what may have caused it and whether there's anything I can do to stop it happening again ?

(Xen3.1/Stock Ubuntu)

Jan 18 14:00:31 nodec kernel: [  362.689029] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.689044]
Jan 18 14:00:31 nodec kernel: [  362.689045] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.689048]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.689092]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.689108]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.689126]  [__kmalloc+207/224] __kmalloc+0xcf/0xe0
Jan 18 14:00:31 nodec kernel: [  362.689139]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.689171]  [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110
Jan 18 14:00:31 nodec kernel: [  362.689190]  [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689211]  [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240
Jan 18 14:00:31 nodec kernel: [  362.689235]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689259]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689271]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689285]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.689300]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689327]  [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60
Jan 18 14:00:31 nodec kernel: [  362.689352]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.689375]  [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0
Jan 18 14:00:31 nodec kernel: [  362.689391]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.689405]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689427]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.689440]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689456]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.689488]  [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90
Jan 18 14:00:31 nodec kernel: [  362.689504]  [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90
Jan 18 14:00:31 nodec kernel: [  362.689516]  [skb_clone+50/784] skb_clone+0x32/0x310
Jan 18 14:00:31 nodec kernel: [  362.689534]  [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100
Jan 18 14:00:31 nodec kernel: [  362.689562]  [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130
Jan 18 14:00:31 nodec kernel: [  362.689587]  [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689607]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689622]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689638]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.689672]  [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800
Jan 18 14:00:31 nodec kernel: [  362.689688]  [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380
Jan 18 14:00:31 nodec kernel: [  362.689706]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.689727]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.689741]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.689755]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.689787]  [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0
Jan 18 14:00:31 nodec kernel: [  362.689802]  [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470
Jan 18 14:00:31 nodec kernel: [  362.689832]  [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580
Jan 18 14:00:31 nodec kernel: [  362.689864]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.689874]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.689881]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.689891]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.689900]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.689906]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.689913]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.689922]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.689934]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.689938]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.689958]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.689971]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.689979]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.689991]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.690004]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.690012]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.690020]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.690033]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.690073]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.690089]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.690125]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.690139]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.690150]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.690163]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.690170]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.690178]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.690185]  [system_call+0/139] system_call+0x0/0x8b
                                                                                                                                                                                                69,1          Top
Jan 18 14:00:31 nodec kernel: [  362.690192]
Jan 18 14:00:31 nodec kernel: [  362.690194] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.690196] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.690199] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.690203] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.690208] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.690212] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.690218] Active:58006 inactive:40628 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.690220]  free:272 slab:7704 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.690230] DMA free:1088kB min:2896kB low:3620kB high:4344kB active:232024kB inactive:162512kB present:525200kB pages_scanned:0 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.690237] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.690242] DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1108kB
Jan 18 14:00:31 nodec kernel: [  362.690265] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.690269] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.690272] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.690277] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.693502] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.693507] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.693509] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.693511] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.693565] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.693570]
Jan 18 14:00:31 nodec kernel: [  362.693571] Call Trace:

Jan 18 14:00:31 nodec kernel: [  362.693576]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.693630]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.693641]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.693659]  [__kmalloc+207/224] __kmalloc+0xcf/0xe0
Jan 18 14:00:31 nodec kernel: [  362.693677]  [pskb_expand_head+66/352] pskb_expand_head+0x42/0x160
Jan 18 14:00:31 nodec kernel: [  362.693705]  [_end+129765542/2130180472] :bridge:nf_bridge_copy_header+0x5e/0x110
Jan 18 14:00:31 nodec kernel: [  362.693723]  [_end+129743461/2130180472] :bridge:br_dev_queue_push_xmit+0x4d/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693744]  [_end+129763180/2130180472] :bridge:br_nf_post_routing+0x144/0x240
Jan 18 14:00:31 nodec kernel: [  362.693766]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.693787]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693801]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693816]  [_end+129743384/2130180472] :bridge:br_dev_queue_push_xmit+0x0/0xb0
Jan 18 14:00:31 nodec kernel: [  362.693831]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693858]  [_end+129743635/2130180472] :bridge:br_forward_finish+0x4b/0x60
Jan 18 14:00:31 nodec kernel: [  362.693879]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.693898]  [_end+129766003/2130180472] :bridge:br_nf_forward_ip+0x11b/0x1d0
Jan 18 14:00:31 nodec kernel: [  362.693913]  [_end+129763618/2130180472] :bridge:br_nf_forward_finish+0xba/0x190
Jan 18 14:00:31 nodec kernel: [  362.693923]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.693942]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.693953]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.693966]  [_end+129743560/2130180472] :bridge:br_forward_finish+0x0/0x60
Jan 18 14:00:31 nodec kernel: [  362.693988]  [_end+129743656/2130180472] :bridge:__br_forward+0x0/0x90
Jan 18 14:00:31 nodec kernel: [  362.694000]  [_end+129743763/2130180472] :bridge:__br_forward+0x6b/0x90
Jan 18 14:00:31 nodec kernel: [  362.694011]  [skb_clone+50/784] skb_clone+0x32/0x310
Jan 18 14:00:31 nodec kernel: [  362.694026]  [_end+129743237/2130180472] :bridge:br_flood+0x8d/0x100
Jan 18 14:00:31 nodec kernel: [  362.694054]  [_end+129746911/2130180472] :bridge:br_handle_frame_finish+0x127/0x130
Jan 18 14:00:31 nodec kernel: [  362.694080]  [_end+129764853/2130180472] :bridge:br_nf_pre_routing_finish+0x1bd/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694101]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694115]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.694131]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.694165]  [_end+129768044/2130180472] :bridge:br_nf_pre_routing+0x454/0x800
Jan 18 14:00:31 nodec kernel: [  362.694182]  [_end+129882151/2130180472] :ip_tables:ipt_do_table+0x23f/0x380
Jan 18 14:00:31 nodec kernel: [  362.694201]  [nf_iterate+112/160] nf_iterate+0x70/0xa0
Jan 18 14:00:31 nodec kernel: [  362.694224]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.694234]  [nf_hook_slow+113/240] nf_hook_slow+0x71/0xf0
Jan 18 14:00:31 nodec kernel: [  362.694242]  [_end+129746616/2130180472] :bridge:br_handle_frame_finish+0x0/0x130
Jan 18 14:00:31 nodec kernel: [  362.694257]  [_end+129747254/2130180472] :bridge:br_handle_frame+0x14e/0x1e0
Jan 18 14:00:31 nodec kernel: [  362.694264]  [netif_receive_skb+371/1136] netif_receive_skb+0x173/0x470
Jan 18 14:00:31 nodec kernel: [  362.694280]  [_end+128027301/2130180472] :e1000:e1000_clean_rx_irq+0x19d/0x580
Jan 18 14:00:31 nodec kernel: [  362.694300]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.694309]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.694314]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.694323]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.694331]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.694337]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.694343]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.694351]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.694363]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.694367]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.694388]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.694401]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.694408]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.694420]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.694434]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.694442]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.694451]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.694462]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.694477]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.694484]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.694500]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.694522]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.694545]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.694573]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.694586]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.694607]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.694623]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.694644]
Jan 18 14:00:31 nodec kernel: [  362.694648] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.694652] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.694659] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.694665] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.694674] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.694680] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.694688] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.694691]  free:297 slab:7704 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.694700] DMA free:1188kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.694708] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.694713] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1288kB
Jan 18 14:00:31 nodec kernel: [  362.694737] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.694742] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.694747] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.694752] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.697874] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.697881] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.697883] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.697887] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.698059] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.698064]
Jan 18 14:00:31 nodec kernel: [  362.698065] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.698068]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.698119]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.698144]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.698158]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698177]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698191]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.698209]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.698227]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.698251]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.698294]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.698332]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.698348]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.698355]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.698369]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.698383]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.698392]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.698403]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.698414]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.698432]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.698437]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.698486]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.698512]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.698526]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.698551]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.698582]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.698603]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.698621]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.698652]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.698686]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.698705]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.698745]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.698777]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.698797]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.698820]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.698838]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.698857]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.698874]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.698893]
Jan 18 14:00:31 nodec kernel: [  362.698896] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.698900] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.698905] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.698911] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.698920] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.698927] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.698935] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.698937]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.698948] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.698954] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.698959] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.698983] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.698989] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.698992] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.698995] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.701567] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.701573] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.701575] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.701577] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.701614] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.701617]
Jan 18 14:00:31 nodec kernel: [  362.701618] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.701620]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.701656]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.701668]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.701676]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701684]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701692]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.701701]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.701709]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.701724]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.701743]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.701754]  [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0
Jan 18 14:00:31 nodec kernel: [  362.701770]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.701781]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.701786]  [__wake_up+67/112] __wake_up+0x43/0x70
Jan 18 14:00:31 nodec kernel: [  362.701796]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.701804]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.701810]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.701818]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.701826]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.701839]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.701843]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.701863]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.701877]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.701884]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.701896]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.701910]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.701919]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.701927]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.701938]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.701954]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.701962]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.701978]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.701992]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.702002]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.702014]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.702023]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.702030]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.702038]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.702047]
Jan 18 14:00:31 nodec kernel: [  362.702049] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.702050] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.702055] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.702059] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.702064] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.702067] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.702073] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.702075]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.702080] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.702084] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.702088] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.702106] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.702109] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.702112] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.702114] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.704303] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.704308] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.704310] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.704312] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.704565] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.704569]
Jan 18 14:00:31 nodec kernel: [  362.704570] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.704571]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.704607]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.704619]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.704628]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704636]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704642]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.704652]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.704660]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.704674]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.704692]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.704712]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.704723]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.704734]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.704744]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.704750]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.704758]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.704767]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.704779]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.704784]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.704805]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.704820]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.704828]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.704840]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.704854]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.704863]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.704871]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.704882]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.704898]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.704906]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.704924]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.704938]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.704949]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.704963]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.704971]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.704980]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.704987]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.704996]
Jan 18 14:00:31 nodec kernel: [  362.704998] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.705001] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.705004] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.705010] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.705013] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.705018] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.705023] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.705024]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.705030] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.705035] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.705039] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.705053] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.705057] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.705059] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.705061] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707207] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.707211] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.707212] 31576 pages shared
Jan 18 14:00:31 nodec kernel: [  362.707214] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.707262] glusterfs: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.707265]
Jan 18 14:00:31 nodec kernel: [  362.707266] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.707268]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.707305]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.707313]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707321]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707327]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.707335]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.707343]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.707359]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.707377]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.707390]  [ip_rcv+894/1696] ip_rcv+0x37e/0x6a0
Jan 18 14:00:31 nodec kernel: [  362.707408]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.707417]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.707427]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.707436]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.707442]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.707450]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.707459]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.707471]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.707476]  <EOI>  [force_evtchn_callback+10/16] force_evtchn_callback+0xa/0x10
Jan 18 14:00:31 nodec kernel: [  362.707497]  [do_page_fault+2867/4816] do_page_fault+0xb33/0x12d0
Jan 18 14:00:31 nodec kernel: [  362.707510]  [copy_user_generic_unrolled+131/320] copy_user_generic_unrolled+0x83/0x140
Jan 18 14:00:31 nodec kernel: [  362.707519]  [ep_poll_callback+128/224] ep_poll_callback+0x80/0xe0
Jan 18 14:00:31 nodec kernel: [  362.707531]  [error_exit+0/121] error_exit+0x0/0x79
Jan 18 14:00:31 nodec kernel: [  362.707544]  [copy_user_generic_unrolled+78/320] copy_user_generic_unrolled+0x4e/0x140
Jan 18 14:00:31 nodec kernel: [  362.707552]  [memcpy_toiovec+77/112] memcpy_toiovec+0x4d/0x70
Jan 18 14:00:31 nodec kernel: [  362.707561]  [skb_copy_datagram_iovec+453/640] skb_copy_datagram_iovec+0x1c5/0x280
Jan 18 14:00:31 nodec kernel: [  362.707572]  [tcp_recvmsg+2755/3232] tcp_recvmsg+0xac3/0xca0
Jan 18 14:00:31 nodec kernel: [  362.707586]  [sock_common_recvmsg+48/80] sock_common_recvmsg+0x30/0x50
Jan 18 14:00:31 nodec kernel: [  362.707595]  [sock_aio_read+372/384] sock_aio_read+0x174/0x180
Jan 18 14:00:31 nodec kernel: [  362.707612]  [do_sync_read+217/288] do_sync_read+0xd9/0x120
Jan 18 14:00:31 nodec kernel: [  362.707627]  [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30
Jan 18 14:00:31 nodec kernel: [  362.707638]  [do_mmap_pgoff+1560/2128] do_mmap_pgoff+0x618/0x850
Jan 18 14:00:31 nodec kernel: [  362.707650]  [vfs_read+382/400] vfs_read+0x17e/0x190
Jan 18 14:00:31 nodec kernel: [  362.707658]  [sys_read+83/144] sys_read+0x53/0x90
Jan 18 14:00:31 nodec kernel: [  362.707667]  [system_call+134/139] system_call+0x86/0x8b
Jan 18 14:00:31 nodec kernel: [  362.707675]  [system_call+0/139] system_call+0x0/0x8b
Jan 18 14:00:31 nodec kernel: [  362.707682]
Jan 18 14:00:31 nodec kernel: [  362.707682]
Jan 18 14:00:31 nodec kernel: [  362.707685] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.707687] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.707690] CPU    0: Hot: hi:  186, btch:  31 usd:  13   Cold: hi:   62, btch:  15 usd:  52
Jan 18 14:00:31 nodec kernel: [  362.707696] CPU    1: Hot: hi:  186, btch:  31 usd: 163   Cold: hi:   62, btch:  15 usd:  57
Jan 18 14:00:31 nodec kernel: [  362.707699] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  49
Jan 18 14:00:31 nodec kernel: [  362.707704] CPU    3: Hot: hi:  186, btch:  31 usd:  97   Cold: hi:   62, btch:  15 usd:   9
Jan 18 14:00:31 nodec kernel: [  362.707709] Active:57809 inactive:40743 dirty:1 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.707710]  free:265 slab:7736 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.707716] DMA free:1060kB min:2896kB low:3620kB high:4344kB active:231236kB inactive:162972kB present:525200kB pages_scanned:18 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.707721] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.707725] DMA: 44*4kB 1*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1160kB
Jan 18 14:00:31 nodec kernel: [  362.707741] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.707744] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707747] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.707749] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.709971] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.709977] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.709978] 31578 pages shared
Jan 18 14:00:31 nodec kernel: [  362.709982] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.881155] swapper: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.881164]
Jan 18 14:00:31 nodec kernel: [  362.881165] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.881168]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.881228]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.881251]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.881266]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881280]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881295]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.881312]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.881328]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.881355]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.881392]  [_end+128027760/2130180472] :e1000:e1000_clean_rx_irq+0x368/0x580
Jan 18 14:00:31 nodec kernel: [  362.881441]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.881463]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.881488]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.881504]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.881519]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.881532]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.881549]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.881578]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.881585]  <EOI>  [xen_safe_halt+143/224] xen_safe_halt+0x8f/0xe0
Jan 18 14:00:31 nodec kernel: [  362.881639]  [xen_idle+73/144] xen_idle+0x49/0x90
Jan 18 14:00:31 nodec kernel: [  362.881647]  [cpu_idle+120/224] cpu_idle+0x78/0xe0
Jan 18 14:00:31 nodec kernel: [  362.881656]  [start_kernel+693/864] start_kernel+0x2b5/0x360
Jan 18 14:00:31 nodec kernel: [  362.881663]  [x86_64_start_kernel+318/336] _sinittext+0x13e/0x150
Jan 18 14:00:31 nodec kernel: [  362.881673] Mem-info:
Jan 18 14:00:31 nodec kernel: [  362.881677] DMA per-cpu:
Jan 18 14:00:31 nodec kernel: [  362.881684] CPU    0: Hot: hi:  186, btch:  31 usd:  22   Cold: hi:   62, btch:  15 usd:  60
Jan 18 14:00:31 nodec kernel: [  362.881692] CPU    1: Hot: hi:  186, btch:  31 usd: 161   Cold: hi:   62, btch:  15 usd:  48
Jan 18 14:00:31 nodec kernel: [  362.881699] CPU    2: Hot: hi:  186, btch:  31 usd:  36   Cold: hi:   62, btch:  15 usd:  58
Jan 18 14:00:31 nodec kernel: [  362.881705] CPU    3: Hot: hi:  186, btch:  31 usd:  74   Cold: hi:   62, btch:  15 usd:   0
Jan 18 14:00:31 nodec kernel: [  362.881715] Active:54759 inactive:43432 dirty:0 writeback:0 unstable:0
Jan 18 14:00:31 nodec kernel: [  362.881718]  free:267 slab:8169 mapped:2557 pagetables:675 bounce:0
Jan 18 14:00:31 nodec kernel: [  362.881726] DMA free:1068kB min:2896kB low:3620kB high:4344kB active:219036kB inactive:173728kB present:525200kB pages_scanned:0 all_unreclaimable? no
Jan 18 14:00:31 nodec kernel: [  362.881733] lowmem_reserve[]: 0 0 0
Jan 18 14:00:31 nodec kernel: [  362.881741] DMA: 19*4kB 6*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1068kB
Jan 18 14:00:31 nodec kernel: [  362.881760] Swap cache: add 0, delete 0, find 0/0, race 0+0
Jan 18 14:00:31 nodec kernel: [  362.881764] Free swap  = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.881766] Total swap = 2097144kB
Jan 18 14:00:31 nodec kernel: [  362.881768] Free swap:       2097144kB
Jan 18 14:00:31 nodec kernel: [  362.885123] 133120 pages of RAM
Jan 18 14:00:31 nodec kernel: [  362.885130] 20456 reserved pages
Jan 18 14:00:31 nodec kernel: [  362.885133] 31592 pages shared
Jan 18 14:00:31 nodec kernel: [  362.885135] 0 pages swap cached
Jan 18 14:00:31 nodec kernel: [  362.885149] swapper: page allocation failure. order:3, mode:0x4020
Jan 18 14:00:31 nodec kernel: [  362.885155]
Jan 18 14:00:31 nodec kernel: [  362.885156] Call Trace:
Jan 18 14:00:31 nodec kernel: [  362.885159]  <IRQ>  [__alloc_pages+806/848] __alloc_pages+0x326/0x350
Jan 18 14:00:31 nodec kernel: [  362.885208]  [_end+129764408/2130180472] :bridge:br_nf_pre_routing_finish+0x0/0x3d0
Jan 18 14:00:31 nodec kernel: [  362.885231]  [__slab_alloc+329/1456] __slab_alloc+0x149/0x5b0
Jan 18 14:00:31 nodec kernel: [  362.885244]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.885260]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.885274]  [__kmalloc_track_caller+226/240] __kmalloc_track_caller+0xe2/0xf0
Jan 18 14:00:31 nodec kernel: [  362.885292]  [__alloc_skb+111/352] __alloc_skb+0x6f/0x160
Jan 18 14:00:31 nodec kernel: [  362.885311]  [__netdev_alloc_skb+23/64] __netdev_alloc_skb+0x17/0x40
Jan 18 14:00:31 nodec kernel: [  362.885337]  [_end+128026333/2130180472] :e1000:e1000_alloc_rx_buffers+0x175/0x3a0
Jan 18 14:00:31 nodec kernel: [  362.885376]  [_end+128027648/2130180472] :e1000:e1000_clean_rx_irq+0x2f8/0x580
Jan 18 14:00:31 nodec kernel: [  362.885423]  [_end+128023320/2130180472] :e1000:e1000_clean+0x270/0x320
Jan 18 14:00:31 nodec kernel: [  362.885442]  [net_rx_action+250/688] net_rx_action+0xfa/0x2b0
Jan 18 14:00:31 nodec kernel: [  362.885467]  [__do_softirq+138/288] __do_softirq+0x8a/0x120
Jan 18 14:00:31 nodec kernel: [  362.885482]  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 18 14:00:31 nodec kernel: [  362.885496]  [do_softirq+165/256] do_softirq+0xa5/0x100
Jan 18 14:00:31 nodec kernel: [  362.885510]  [do_IRQ+98/192] do_IRQ+0x62/0xc0
Jan 18 14:00:31 nodec kernel: [  362.885524]  [evtchn_do_upcall+258/336] evtchn_do_upcall+0x102/0x150
Jan 18 14:00:31 nodec kernel: [  362.885554]  [do_hypervisor_callback+30/48] do_hypervisor_callback+0x1e/0x30
Jan 18 14:00:31 nodec kernel: [  362.885561]  <EOI>  [xen_safe_halt+143/224] xen_safe_halt+0x8f/0xe0
Jan 18 14:00:31 nodec kernel: [  362.885615]  [xen_idle+73/144] xen_idle+0x49/0x90
Jan 18 14:00:31 nodec kernel: [  362.885624]  [cpu_idle+120/224] cpu_idle+0x78/0xe0
Jan 18 14:00:31 nodec kernel: [  362.885637]  [start_kernel+693/864] start_kernel+0x2b5/0x360
Jan 18 14:00:31 nodec kernel: [  362.885652]  [x86_64_start_kernel+318/336] _sinittext+0x13e/0x150
etc ...



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
Previous by Date:  Re: [Xen-users] NVidia 169.07 on XEN, Ryan Roth
Next by Date:  Re: [Xen-users] wine in a domU, Ryan E Roth
Previous by Thread:  [Xen-users] Retrieving SDL display, Jeremie Le Hen
Next by Thread:  Re: [Xen-users] Re; System Crash ..., Gareth Bult
Indexes:  [Date] [Thread] [Top] [All Lists]
 
   

Copyright , Citrix Systems Inc. All rights reserved. Legal and Privacy