WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] Performance questions.

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] Performance questions.
From: Kai Meyer <kai@xxxxxxxxxxxxx>
Date: Thu, 31 Jan 2008 16:31:37 -0700
Delivery-date: Thu, 31 Jan 2008 15:32:50 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.9 (X11/20071115)
I have a problem I'd like to get some input on. Here's my setup:
2x4 core Intel(R) Xeon(R) CPU E5320 @ 1.86GHz
16GB Memory
4x Barracuda 7200.10 SATA 3.0Gb/s 160-GB Hard Drive
4port 3ware 9550SXU-4LP, with the 4 drives in 2 Raid1 mirrors.
Dom-0 Centos 5.1 x86_64
kernel 2.6.18-53.1.6.el5xen

I installed 32 identical "bare metals" Centos 5.1 x86_64 instances. I put a php script that simply calculates pi to a certain degree of accuracy, and adjusted it so it took about .4 seconds to run on one of the Dom-Us. Then from another server, I am using the 'ab' command from apache to benchmark performance on the 32 servers simultaniously. Here's an example of the command I am running:

ab -n 20 -c 20 http://192.168.1.201/pi.php &

I loop this 32 times, once for each server. It's right around a concurrency of 20, and a count of anything higher, when I start getting time outs from 'ab'.
apr_poll: The timeout specified has expired (70007)

Out of the batch of 32, I will invariably get between 3 and 16 that return this error. If I lower the concurrency to 19, I'll get between 0 and 5 time outs. 18 and below, I don't get any failures. Over 20, I the number of servers that time out do increase, but very slowly.

I'm simply trying to find the threshold or cap that I am hitting so I can give a suggestion on how to monitor and manage the highest number of virtual machines possible.

What I've tried looking for is a correlation between when the time out occurs, and the following list of indicators on both the Dom-0 as well as all the Dom-Us.
Interrupts from vmstat
Context Switches from vmstat
CPU usage from vmstat/top/xentop
apache connections from the server-status module
load average

I would prefer to be able to track this sort of behavior from Dom-0. I have test results from running various different combinations of virtual machines, and concurrency levels from ab, but havn't found the underlying cause yet. Any help or pointers on what else to look for would be appreciated.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>
  • [Xen-users] Performance questions., Kai Meyer <=