[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] OOM killer observed during heavy I/O from VMs (XEN 3.0.4 and XEN 3.1)


  • To: xen-users@xxxxxxxxxxxxxxxxxxx
  • From: Eric Tessler <maiden1134@xxxxxxxxx>
  • Date: Sat, 4 Aug 2007 23:59:53 -0700 (PDT)
  • Delivery-date: Sat, 04 Aug 2007 23:57:45 -0700
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=NNjwM905JY1RWKg6OnQItbyiABrcAeG22Ixo0xijgx0Nc4T+BnNMXWv1yhX7yFP+GC74POhXM5EK4+r13LJZ8qQ5PQS+kuLN7Coo8HbyKoCbV8/gaJuwuGb5FELZN73l414YgIlnQOIP8dz/HU7zxutl+Z4uYydk8346mnDtdTw=;
  • List-id: Xen user discussion <xen-users.lists.xensource.com>

Under both XEN 3.0.4 (2.6.16.33) and XEN 3.1 (2.6.18), I can make the OOM killer appear in dom0 of my server by doing heavy I/O from within a VM. If I start 5 VMs on the same server, each VM doing constant I/O over its boot disk (read/write a 2GB file), after about 30 minutes the OOM killer appears in dom0 and starts killing processes. This was observed using 256MB in dom0. If I bump the memory in dom0 up to 512MB, this seems to fix the problem (I have not seen the OOM killer in this configuration). This used to work in XEN 2.0.7 with 256MB in dom0.
 
Below I have included the OOM killer output and memory/slab info for the server. I want to get it working using 256MB in dom0 - this smells like a bug in the kernel/XEN as it appears there is plenty of available swap and about 40MB of memory (using 512MB swap).
 
Here is OOM killer info from the message log when it hits:
Aug  4 18:36:08  DMA: 1078*4kB 102*8kB 34*16kB 15*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 7*4096kB = 36168kB
Aug  4 18:36:08  DMA32: empty
Aug  4 18:36:08  Normal: empty
Aug  4 18:36:08  HighMem: empty
Aug  4 18:36:08  Swap cache: add 29953, delete 29286, find 9785/14072, race 0+50
Aug  4 18:36:08  Free swap  = 498904kB
Aug  4 18:36:08  Total swap = 524280kB
Aug  4 18:36:08  Free swap:       498904kB
Aug  4 18:36:08  67584 pages of RAM
Aug  4 18:36:08  0 pages of HIGHMEM
Aug  4 18:36:08  18261 reserved pages
Aug  4 18:36:13  31927 pages shared
Aug  4 18:36:19  667 pages swap cached
Aug  4 18:36:24  1 pages dirty
Aug  4 18:36:30  27115 pages writeback
Aug  4 18:36:32  1090 pages mapped
Aug  4 18:36:36  6800 pages slab
Aug  4 18:36:44  695 pages pagetables
Aug  4 18:36:51  oom-killer: gfp_mask=0x200d2, order=0
Aug  4 18:36:51   [<c0105801>] show_trace+0x21/0x30
Aug  4 18:36:51   [<c010593e>] dump_stack+0x1e/0x20
Aug  4 18:36:51   [<c0140de0>] out_of_memory+0x90/0xc0
Aug  4 18:36:52   [<c014219d>] __alloc_pages+0x2ed/0x320
Aug  4 18:36:52   [<c014d0c0>] do_wp_page+0xa0/0x4c0
Aug  4 18:36:52   [<c014ddd8>] do_swap_page+0x2f8/0x480
Aug  4 18:36:52   [<c014e972>] __handle_mm_fault+0x302/0x430
Aug  4 18:36:53   [<c011581f>] do_page_fault+0x1df/0x906
Aug  4 18:36:53   [<c01054cb>] error_code+0x2b/0x30
Aug  4 18:36:53   [<c0104f91>] handle_signal+0x81/0x170
Aug  4 18:36:53   [<c0105136>] do_signal+0xb6/0x170
Aug  4 18:36:54   [<c010522a>] do_notify_resume+0x3a/0x3c
Aug  4 18:36:54   [<c01053ef>] work_notifysig+0x13/0x18
Aug  4 18:36:54  Mem-info:
Aug  4 18:36:54  DMA per-cpu:
Aug  4 18:36:54  cpu 0 hot: high 90, batch 15 used:11
Aug  4 18:36:54  cpu 0 cold: high 30, batch 7 used:28
Aug  4 18:36:54  DMA32 per-cpu: empty
Aug  4 18:36:54  Normal per-cpu: empty
Aug  4 18:36:54  HighMem per-cpu: empty
Aug  4 18:36:54  Free pages:       36492kB (0kB HighMem)
Aug  4 18:36:54  Active:988 inactive:27911 dirty:1 writeback:26988 unstable:0 free:9123 slab:6800 mapped:1089 pagetables:695
Aug  4 18:36:54  DMA free:36492kB min:32768kB low:40960kB high:49152kB active:3952kB inactive:111644kB present:270336kB pages_scanned:29094 all_unreclaimable? no
Aug  4 18:36:54  lowmem_reserve[]: 0 0 0 0
Aug  4 18:36:54  DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Aug  4 18:36:54  lowmem_reserve[]: 0 0 0 0
Aug  4 18:36:54  Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Aug  4 18:36:54  lowmem_reserve[]: 0 0 0 0
Aug  4 18:36:54  HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Aug  4 18:36:54  lowmem_reserve[]: 0 0 0 0

Memory/slab info just before OOM hits:
MemTotal:       262344 kB
MemFree:         43616 kB
Buffers:         99044 kB
Cached:           7940 kB
SwapCached:       5420 kB
Active:           9300 kB
Inactive:       103380 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       262344 kB
LowFree:         43616 kB
SwapTotal:      524280 kB
SwapFree:       501272 kB
Dirty:            4836 kB
Writeback:       92056 kB
Mapped:           7936 kB
Slab:            25428 kB
CommitLimit:    655452 kB
Committed_AS:   765172 kB
PageTables:       2072 kB
VmallocTotal:   593912 kB
VmallocUsed:      2812 kB
VmallocChunk:   590208 kB
Cache                       Num  Total   Size  Pages
fib6_nodes                    8    113     32    113
ip6_dst_cache                28     30    256     15
ndisc_cache                   1     20    192     20
RAWv6                         4      6    640      6
UDPv6                         1      7    576      7
tw_sock_TCPv6                 0      0    128     30
request_sock_TCPv6            0      0    128     30
TCPv6                         0      0   1152      7
nbd-wi                       52     78     48     78
bridge_fdb_cache             35     59     64     59
ip_fib_alias                 15    113     32    113
ip_fib_hash                  15    113     32    113
ext3_inode_cache            309    576    460      8
ext3_xattr                    0      0     44     84
journal_handle               64    169     20    169
journal_head                196    504     52     72
revoke_table                  6    254     12    254
revoke_record                 0      0     16    203
dm_tio                    11142  11165     16    203
dm_io                     11105  11154     20    169
scsi_cmd_cache               10     10    384     10
Cache                       Num  Total   Size  Pages
sgpool-128                   34     34   3072      2
sgpool-64                    35     35   1536      5
sgpool-32                    35     35    768      5
sgpool-16                    36     40    384     10
sgpool-8                     40     40    192     20
scsi_io_context               0      0    104     37
rpc_buffers                   8      8   2048      2
rpc_tasks                     8     20    192     20
rpc_inode_cache               0      0    448      9
UNIX                         42     50    384     10
ip_mrt_cache                  0      0    128     30
tcp_bind_bucket              49    203     16    203
inet_peer_cache               1     59     64     59
secpath_cache                 0      0    128     30
xfrm_dst_cache                0      0    320     12
ip_dst_cache                 15     30    256     15
arp_cache                     7     30    128     30
RAW                           2      9    448      9
UDP                           5      9    448      9
tw_sock_TCP                   3     30    128     30
request_sock_TCP              0      0     64     59
Cache                       Num  Total   Size  Pages
TCP                          63     72   1024      4
flow_cache                    0      0    128     30
uhci_urb_priv                 0      0     40     92
blktapif_cache                0      0    108     36
blkif_cache                  15     35    112     35
cfq_ioc_pool                  0      0     48     78
cfq_pool                      0      0     96     40
crq_pool                      0      0     44     84
deadline_drq                  0      0     48     78
as_arq                     1197   1260     60     63
nfs_write_data               36     36    448      9
nfs_read_data                32     36    448      9
nfs_inode_cache               0      0    560      7
nfs_page                      0      0     64     59
isofs_inode_cache             0      0    340     11
ext2_inode_cache              0      0    420      9
dnotify_cache                 0      0     20    169
eventpoll_pwq                 0      0     36    101
eventpoll_epi                 0      0    128     30
inotify_event_cache           0      0     28    127
inotify_watch_cache           0      0     36    101
Cache                       Num  Total   Size  Pages
kioctx                        0      0    192     20
kiocb                         0      0    128     30
fasync_cache                  0      0     16    203
shmem_inode_cache          1515   1521    404      9
posix_timers_cache            1     40     96     40
uid_cache                     2     59     64     59
blkdev_ioc                  178    254     28    127
blkdev_queue                548    548    900      4
blkdev_requests            1205   1265    168     23
biovec-(256)                260    260   3072      2
biovec-128                  264    265   1536      5
biovec-64                   290    290    768      5
biovec-16                   440    560    192     20
biovec-4                    284    295     64     59
biovec-1                  32822  41006     16    203
bio                       32855  33512     64     59
sock_inode_cache            122    130    384     10
skbuff_fclone_cache          46    100    384     10
skbuff_head_cache           620    920    192     20
xen-skb-65536                 0      0  65536      1
xen-skb-32768                 0      0  32768      1
Cache                       Num  Total   Size  Pages
xen-skb-16384                 0      0  16384      1
xen-skb-8192                  0      0   8192      1
xen-skb-4096                544    557   4096      1
xen-skb-2048                 24     54   2048      2
xen-skb-512                  48     48    512      8
file_lock_cache               4     44     88     44
acpi_operand                637    736     40     92
acpi_parse_ext                0      0     44     84
acpi_parse                    0      0     28    127
acpi_state                    0      0     48     78
proc_inode_cache             17     48    328     12
sigqueue                     54     54    144     27
radix_tree_node             810   1456    276     14
bdev_cache                   82     90    448      9
sysfs_dir_cache           13325  13340     40     92
mnt_cache                    21     30    128     30
inode_cache                2416   2976    312     12
dentry_cache               4407  12989    124     31
filp                        710   1060    192     20
names_cache                   2      2   4096      1
idr_layer_cache             209    232    136     29
Cache                       Num  Total   Size  Pages
buffer_head               25038  28938     48     78
mm_struct                    87     90    448      9
vm_area_struct             1782   2478     92     42
fs_cache                     84    565     32    113
files_cache                  85    135    448      9
signal_cache                193    200    384     10
sighand_cache               191    204   1344      3
task_struct                 208    303   1280      3
anon_vma                    751   2034      8    339
pgd                          89     89   4096      1
pmd                         351    351   4096      1
size-131072(DMA)              0      0 131072      1
size-131072                   1      1 131072      1
size-65536(DMA)               0      0  65536      1
size-65536                   12     12  65536      1
size-32768(DMA)               0      0  32768      1
size-32768                    0      0  32768      1
size-16384(DMA)               0      0  16384      1
size-16384                    3      3  16384      1
size-8192(DMA)                0      0   8192      1
size-8192                   211    218   8192      1
Cache                       Num  Total   Size  Pages
size-4096(DMA)                0      0   4096      1
size-4096                    81     86   4096      1
size-2048(DMA)                0      0   2048      2
size-2048                    91     92   2048      2
size-1024(DMA)                0      0   1024      4
size-1024                   242    248   1024      4
size-512(DMA)                 0      0    512      8
size-512                    492    496    512      8
size-256(DMA)                 0      0    256     15
size-256                    892    900    256     15
size-192(DMA)                 0      0    192     20
size-192                    689    700    192     20
size-128(DMA)                 0      0    128     30
size-128                    356    360    128     30
size-96(DMA)                  0      0    128     30
size-96                    3300   3300    128     30
size-64(DMA)                  0      0     64     59
size-32(DMA)                  0      0     32    113
size-64                    5362   5782     64     59
size-32                    6732   9492     32    113
kmem_cache                  150    150    128     30
 
 
 


Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.