WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

RE: [Xen-users] VCPU and CPU Manipulation

To: "'Omer Khalid'" <Omer.Khalid@xxxxxxx>
Subject: RE: [Xen-users] VCPU and CPU Manipulation
From: "Dustin Henning" <Dustin.Henning@xxxxxxxxxxx>
Date: Tue, 7 Oct 2008 07:50:24 -0400
Cc: 'xen-users list' <xen-users@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Tue, 07 Oct 2008 04:51:07 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <77e5896b0810070319g1fe48df4k7b902e05ef5a9da@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Organization: PRD, Inc.
References: <77e5896b0810060151n3a93ff7ei2c40fb8c4cc2bfda@xxxxxxxxxxxxxx> <-6385963530672771950@unknownmsgid> <77e5896b0810070319g1fe48df4k7b902e05ef5a9da@xxxxxxxxxxxxxx>
Reply-to: Dustin.Henning@xxxxxxxxxxx
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AckoZkEOE1ITAoJwQ5iK4URJFn8cjgACq2Iw
Omer,
        CPU affinity is which CPUs/cores a VCPU is allowed to use, while the 
CPU/core in use can vary if the CPU affinity allows for more than one 
possibility.  That said, if you pin dom0 VCPU0  to a CPU/core of your choice, 
then it won't change like you have observed, and after thinking about it, I 
believe I do pin my dom0 VCPUs manually with xm vcpu-pin commands in 
/etc/rc.local or something.  Additionally, if you only need to run one domU, 
you could still let dom0 use 3 CPUs/cores without it interfering with the 
CPU/core assigned to the domU, my example was assuming you would run 3 domUs 
and the dom0.  I am not exceptionally familiar with scheduling, but I wouldn't 
think it would come in to play when a CPU/core isn't being shared among 
multiple dom(0/U)s.  You might want to try to verify that, though, or perhaps 
someone else on the list can confirm.  Depending on what your research regards, 
it is important to note that even with PV domUs with isolated CPUs/cores, there 
will probably be some performance loss, though I should hope it would be 
negligible in such a configuration.  Good luck with your project,
        Dustin
        

--------Original Message--------
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx 
[mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Omer Khalid
Sent: Tuesday, October 07, 2008 06:19
To: Dustin.Henning@xxxxxxxxxxx
Cc: xen-users list
Subject: Re: [Xen-users] VCPU and CPU Manipulation

Hello Dustin,
Thanks a lot for the detailed explaination. It indeed clarified my 
understanding about VCPU/CPUs. 
What I have understood in a nutshell is, please correct me if its wrong, that 
"what matters is the CPU core and CPU affinity a VCPU is using rather than just 
the VCPU number; thus all the domains could have the same VCPU #, let's say 0, 
but as long as they are pinned and restricted to a particular core then they 
are restricted to only that explicity core".
Its a valid argument that to enhance resource/cpu utilization, one should not 
bother too much about which core is being used by which domU. This is 
particulary important in situations where perfomance is the key criteria. But 
in my project, and for LHC grid computing; its a policy decision that each grid 
job will be allowed to use only one core per CPU (not because of performance 
reasons but rather resource accounting reasions). In a non-virtualized 
environment, this is handled by the Batch system configuration but if the job 
is executed on a virutal machine, which I am is researching on, then comes the 
question of core utilization for a VM. Thus I stumbled upon this issue of 
tinkering with vcpu-set/vcpu-pin.
To achieve this, i first modified that /etc/xen/xend-config and restricted the 
dom0 to use only one CPU. Now my dom0 is only using one CPU while all other 
dom0 instances for each core are in --p "paused state" with no CPU allocated to 
them. Then I launched my VM with the modified config file which had vcpus=1, 
cpus="0". xm list shows the following:
[root@~]# xm vcpu-list
Name        ID     VCPUs    CPU    State     Time(s)    CPU Affinity
CernVM     1       0            0         -b-          11.3         0
Domain-0   0       0            3         r--           111.5       any cpu
Domain-0   0       1            -          --p          12.1         any cpu
Domain-0   0       2            -          --p          5.3           any cpu
Domain-0   0       3            -          --p          2.7           any cpu
I had arrived the same state earlier by using vcpu-pin/vcpu-set but following 
the above process (as you advised too) its much simpler and cleaner. 
Interestingly, I observed that before the launching the domU, dom0 was using 
CPU 2 and later on it switched to CPU 3. But I guess that's OK as its not using 
CPU 0.
So the VCPU/CPU is sorted out, what about the scheduling of these CPUs either 
using sedf or credit-scheduler? Once a domU is restricted to one core, then I 
wanted to further optimize its performance by modifying its weight using 
credit-scheduler as the application to be run in the domU is memory/cpu 
intensive.
Thanks,
Omer

On Mon, Oct 6, 2008 at 2:51 PM, Dustin Henning <Dustin.Henning@xxxxxxxxxxx> 
wrote:
Omer,
       See my response following your initial post.

--------Original Message--------
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx 
[mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Omer Khalid
Sent: Monday, October 06, 2008 04:51
To: xen-users list
Subject: [Xen-users] VCPU and CPU Manipulation
Hi,
I have a dual core SMP machine ( in total 4 cores). I have been trying to set 
restrict vcpu/cpus for my domU to one core/one vpcu but it have not fully 
worked. As there are two commands "xm vcpu-set" and "xm vcpu-pin". By using 
these commands, i have observed that the sequence in which they are used plays 
a role. e.g. I have the following state in the beginning:
[root@lxb ~]# xm vcpu-list
Name         ID     VCPUs CPU   State Time(s)      CPU Affinity
====         ==     ===== ====  ==== ======     === =====
Domain-0    0       0          3        r--      5593.4        any cpu
Domain-0    0       1          1        -b-     15361.9      any cpu
Domain-0    0       2          0        -b-     10137.5      any cpu
Domain-0    0       3          -         --p     78.9           any cpu
test_lxb      20     0          2         -b-     21169.0     any cpu
What I want to achieve is that my domU (test_lxb) uses one VCPU pinned to one 
CPU. In the above state, both my domU and dom0 are using VCPU 0 (which is 
pinned to use either CPU 3 or 2.) After few "vcpu-set" and "vcpu-pin", I reach 
the following stage where dom0 is pinned to CPU 3 and domU (test_lxb) is pinned 
to CPU 2:
[root@lxb ~]# xm vcpu-list
Name ID VCPUs CPU State Time(s) CPU Affinity
Name         ID     VCPUs CPU   State Time(s)      CPU Affinity
====         ==     ===== ====  ==== ======     ========
Domain-0    0       0         3        r--      5600.4            3
Domain-0    0       1         3        -b-     15372.5           3
Domain-0    0       2         3        -b-     10140.0           3
Domain-0    0       3         -         --p     78.9                3
test_lxb      20     0         2         -b-     21169.5           2
But domU is still using VCPU 0 which is also used by my domU; now i would like 
to restrict VCPU 0 to CPU 2 only for domU only...I am wondering how to achieve 
this last mile?
Any ideas? Thanks for you help in advance!
Regards
--
Omer

-------------------------------------------------------
CERN -- European Organization for Nuclear
Research, IT Department, CH-1211,
Geneva 23, Switzerland
       You have misinterpreted the meaning VCPU numbers.  VCPU 0 is the first 
virtual CPU for any domain, VCPU 1 is the second virtual CPU for any domain, 
etcetera.  Additional single VCPU domUs will have a VCPU 0 as well.  Each VCPU 
0 is actually a separate VCPU; they are all identified as CPU 0 to a different 
domain, and the VCPU identification just tells you what the domU sees them as 
(minus the V).  CPU indicates which CPU/core a VCPU is currently using, and CPU 
Affinity indicates which ones it is allowed to use.  Furthermore, for 
performance reasons, if you want Dom0 to only use one CPU/core, you should 
assign it only one VCPU (which will be 0, so for what you are trying to do, you 
probably ultimately want output more like this):

Name            ID      VCPU    CPU     State   Time(s) CPU Affinity
Domain-0        0       0       0       r--     5600.4  0
test_lxb        1       0       1       -b-     21169.5 1
test_abc        2       0       2       -b-     21169.5 2
test_def        3       0       3       -b-     21169.5 3

       Obviously state and time will be variable.  Additionally, which core/cpu 
is used for which domain shouldn't matter much.  Regarding getting to this 
state, the number of VCPUs dom0 has initially (and which CPUs/cores they use) 
is configurable (probably /etc/xen/xend-config).  The same is true for domUs.  
That said, see the example configs in /etc/xen for more info on how to do this, 
but you should be able to cause each domU to start up with the CPU/core you 
want it to use, and then you won't really need to use vcpu-set or vcpu-pin.  
Finally, if I don't bring it up, someone else probably will, the idea behind 
virtualization is to better use available processing power.  With that in mind, 
your domUs may not each need their own full CPU/core.  (For instance, I have a 
quad-core with four HVMs that have one vcpu each, where each uses a separate 
core, and then my dom0 has four VCPUs, where each uses a separate core; even 
this isn't by any means fully utilizing the hardware, but I am more concerned 
with maintaining optimal performance of my HVMs).  Good luck with your project,
       Dustin


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users



-- 
Omer

-------------------------------------------------------
CERN -- European Organization for Nuclear
Research, IT Department, CH-1211, 
Geneva 23, Switzerland

Phone: +41 (0) 22 767 2224
Fax:     +41 (0) 22 766 8683
E-mail : Omer.Khalid@xxxxxxx
Homepage: http://cern.ch/Omer.Khalid



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>