WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] [Fwd: high load average on hvm gentoo linux domU]

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] [Fwd: high load average on hvm gentoo linux domU]
From: Simon Bettison <simon@xxxxxxxxxxxx>
Date: Thu, 29 May 2008 16:49:25 +0100
Delivery-date: Thu, 29 May 2008 08:50:19 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
In-reply-to: <483D46BA.100@xxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <483D46BA.100@xxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.9 (X11/20071119)
Simon Bettison wrote:
> Hi All,
>
> This is my first time on this (or in fact any as I recall!) list so be
> gentle :)
>
> I am seeing some peculiar load behaviour with a gentoo linux domu which
> is acting as mailserver. The load average starts out fine but then grows
> over the course of several hours topping out at around 120 (one hundred
> and twenty!)
>
> The strange thing is the machine seems responsive, imap connections
> working ok, mail being delivered, snappy shell access etc
>
> This makes me suspicious that the load average is more a reflection of
> the vm load on the cpu slice its being given (ie its only getting say
> 0.01 of the cpu time, and would be running at say 0.11 if it was not
> virtualised) maybe I'm way off though!
>
> The setup is not overly complex:
>
> # xm list
> Name                                        ID   Mem VCPUs      State  
> Time(s)
> Domain-0                                     0  1708     1     r-----  
> 1682.4
> gentoo-spud                                  5  1024     1     -b----  
> 2792.4
> solaris                                      2  1024     1     -b----  
> 1757.4
>
> its a gentoo-linux dom0, its running on amd-vt hardware, do both domUs
> are unmodified guests...
>
> xen # cat /etc/xen/gentoo-spud
> name    = "gentoo-spud";
> memory  = 1024;
> disk = [ "file:/var/xen/gentoo/spud/domU,xvda,w" ];
> kernel  = "/var/xen/gentoo/spud/vmlinuz";
> root = "/dev/xvda ro";
> vif = [ "mac=00:16:3E:00:00:12" ];
>
> xen # cat /etc/xen/solaris.py
> name = 'solaris'
> memory = '1024'
> disk = [ 'file:/var/xen/solaris/domU,0,w' , 'phy:/dev/sdb,1,w' ,
> 'phy:/dev/sdc,2,w' ,'phy:/dev/sdd,3,w' ]
> vif = [ 'mac=00:16:3E:00:00:11' ]
>
> The solaris vm is providing zfs on the 3 physical devices ie
>
>
> solaris # zpool list
> NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
> tank   832G   200G   632G    24%  ONLINE  -
>
> solaris # zpool status
>   pool: tank
>  state: ONLINE
>  scrub: none requested
> config:
>
>         NAME        STATE     READ WRITE CKSUM
>         tank        ONLINE       0     0     0
>           raidz1    ONLINE       0     0     0
>             c0d1    ONLINE       0     0     0
>             c0d2    ONLINE       0     0     0
>             c0d3    ONLINE       0     0     0
>
> solaris # zfs list
> NAME         USED  AVAIL  REFER  MOUNTPOINT
> tank         133G   412G  24.0K  /tank
> tank/mail   1.32G   412G  1.32G  /export/zfs/mail
>
> the tank/mail share is exported over nfs:
>
> solaris # zfs get sharenfs tank/mail
> NAME       PROPERTY  VALUE      SOURCE
> tank/mail  sharenfs  rw,anon=0  local
>
> the linux box is running:
> postfix
> courier-imapd-ssl
> courier-authlib
> amavisd
> spamassassin
> clamav
>
> mail is routed through postfix on port 25 to amavis on port 10024 which
> takes care of spam and av then back into postfix on 10025 for local delivery
>
> all domains/users are virtual on postfix with delivery to
> /home/vmail/<user@domain>/
>
> /home/vmail is a symbolic link to /mnt/zfs/mail
> /mnt/zfs/mail is nfs mounted share from the solaris vm
>
> through a process of elimination I found that restarting
> courier-imapd-ssl would cause the loadavg to head back towards zero,
> where it would stay for short while but then eventually start to
> escalate again.
>
> I can't find any explanation....
>
> # vmstat 1
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> id wa
>  0  0      0 495912  14488 185348    0    0    34    64  399  296  2  1
> 96  0
>  0  0      0 494296  14488 185352    0    0     0     0  269   97  5  1
> 93  0
>  0  0      0 494296  14488 185352    0    0     0     0  207   82  0  0
> 99  0
>  0  0      0 494288  14488 185400    0    0     0     0  131   64  0  0
> 99  0
>  0  0      0 494288  14488 185452    0    0     0     0  128   60  0  0
> 100  0
>  0  0      0 494304  14488 185472    0    0     0     8  130   74  0  0
> 99  0
>  0  0      0 494304  14488 185472    0    0     0     0  144   67  0  0
> 100  0
>  0  0      0 494368  14488 185512    0    0     0     0  127   62  0  0
> 99  0
>  0  0      0 494260  14488 185512    0    0     0     0  141   73  0  0
> 100  0
>  0  0      0 494252  14488 185556    0    0     0     0  143   65  0  0
> 99  0
>  0  0      0 494252  14520 185576    0    0     0    32  116   68  0  0
> 99  0
>  0  0      0 494300  14520 185596    0    0     0     0  127   69  0  0
> 100  0
>  0  0      0 494192  14520 185596    0    0     0     0  141   68  0  0
> 99  0
>  0  0      0 494184  14520 185640    0    0     0     0  153   60  0  0
> 99  0
>  0  0      0 494184  14520 185640    0    0     0     0  122   59  0  0
> 100  0
>  0  0      0 494216  14520 185680    0    0     0     0  124   72  0  0
> 99  0
>  0  0      0 494108  14520 185680    0    0     0     0  123   67  0  0
> 100  0
>  0  0      0 494092  14520 185720    0    0     0     0  117   63  0  0
> 99  0
>  0  0      0 494092  14520 185720    0    0     0     0  119   64  0  0
> 100  0
>  0  0      0 494124  14520 185760    0    0     0     0  123   66  0  0
> 99  0
>  0  0      0 494124  14520 185760    0    0     0     0  125   70  0  0
> 99  0
>  0  0      0 494016  14520 185800    0    0     0     0  118   60  0  0
> 100  0
>  0  0      0 494016  14520 185800    0    0     0     0   93   42  0  0
> 97  0
>  0  0      0 494032  14520 185820    0    0     0     0   25   18  0  0
> 99  0
>  0  0      0 494032  14520 185820    0    0     0     0   11   12  0  0
> 100  0
>  0  0      0 494032  14520 185828    0    0     0     0   89   55  0  0
> 100  0
>  0  0      0 494032  14520 185828    0    0     0     0  138   68  0  1
> 98  0
>  0  0      0 494040  14520 185872    0    0     0     0  118   66  0  0
> 99  0
>  0  0      0 493932  14520 185872    0    0     0     0  122   63  0  0
> 99  0
>  0  0      0 493916  14520 185912    0    0     0     0  129   62  0  0
> 100  0
>  0  0      0 493916  14552 185932    0    0     0    44  126   76  0  0
> 99  0
>  0  0      0 493916  14552 185956    0    0     0     0  123   65  0  0
> 100  0
>  0  0      0 493808  14552 185956    0    0     0     0  119   58  0  0
> 99  0
>  0  0      0 493800  14552 185996    0    0     0     0  132   71  0  0
> 100  0
>  0  0      0 493800  14552 185996    0    0     0     0  120   62  0  0
> 99  0
>  0  0      0 493800  14552 186040    0    0     0    36  131   73  0  0
> 99  0
>  0  0      0 493692  14552 186040    0    0     0     0  133   68  0  0
> 100  0
>  0  0      0 493676  14552 186080    0    0     0     0  130   66  0  0
> 99  0
>  0  0      0 493676  14552 186080    0    0     0     0  125   63  0  0
> 99  0
>  0  0      0 493684  14552 186124    0    0     0     0  139   79  0  0
> 100  0
>  0  0      0 493684  14552 186124    0    0     0     0  126   68  0  0
> 99  0
>  0  0      0 493584  14552 186164    0    0     0     0  125   62  0  0
> 99  0
>  0  0      0 493584  14552 186164    0    0     0     0  127   60  0  0
> 100  0
>  0  0      0 493584  14552 186204    0    0     0     0  130   80  0  0
> 99  0
>  0  0      0 493584  14552 186204    0    0     0     0  132   69  0  0
> 99  0
>  0  0      0 493460  14564 186248    0    0     0    12  119   74  0  0
> 100  0
>  0  0      0 493460  14564 186248    0    0     0     0  129   68  0  0
> 99  0
>  0  0      0 493452  14564 186288    0    0     0     0  145   75  0  0
> 100  0
>
> thought maybe it was nfs...?
>
> # iostat -n 1
> Linux 2.6.21-xen (spud)         05/28/08
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail     10363.60        13.49         0.00        
> 0.00        11.30        13.49
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail        24.18        24.19         0.00        
> 0.00         0.00        18.67
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail         0.00         0.05         0.00        
> 0.00         0.00         5.56
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail         1.98         0.06         0.00        
> 0.00         0.00         0.05
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail      4137.37         0.06         0.00        
> 0.00         0.00         0.07
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail     12051.17         4.16         0.00        
> 0.00         0.00         4.16
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail      6151.21         7.25         0.00        
> 0.00         0.00         7.26
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail        15.48        15.49         0.00        
> 0.00         0.00        15.49
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail        26.00        26.00         0.00        
> 0.00         0.00        26.00
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail        26.98        27.00         0.00        
> 0.00         0.00        21.00
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail        26.79        26.79         0.00        
> 0.00         0.00        29.09
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail        20.99        21.00         0.00        
> 0.00         0.00        18.77
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail        21.20        21.24         0.00        
> 0.00         0.00        27.23
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail         0.00         0.05         0.00        
> 0.00         0.00         0.05
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail         1.98         0.05         0.00        
> 0.00         0.00         0.05
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail         0.00         0.05         0.00        
> 0.00         0.00         0.05
>
> Filesystem:              rBlk_nor/s   wBlk_nor/s   rBlk_dir/s  
> wBlk_dir/s   rBlk_svr/s   wBlk_svr/s
> swanney:/export/zfs/mail         2.75         2.79         0.00        
> 0.00         0.00         2.79
>
> the machine is hardly begin touched - only two mailboxes on it, very
> litle mail passing through.
>
> any pointers would be really appreciated
>
> Simon Bettison
>   
I'm not sure what exactly the issue was, but i think it might have
something to do with courier-imap + thousands of e-mails (~20k total
with some maildirs having 5k in) + NFS

installed dovecot insted and this seems to be working fine.


Simon


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>