WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] Re: [Xen-users] Max. PV and HVM Guests

To: "Pasi Kärkkäinen" <pasik@xxxxxx>, Nick Couchman <Nick.Couchman@xxxxxxxxx>
Subject: RE: [Xen-devel] Re: [Xen-users] Max. PV and HVM Guests
From: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Date: Mon, 9 Nov 2009 07:39:48 -0800 (PST)
Cc: Robert, xen-devel@xxxxxxxxxxxxxxxxxxx, "Mr. Teo En Ming \(Zhang Enming\)" <space.time.universe@xxxxxxxxx>, xen-users@xxxxxxxxxxxxxxxxxxx, Dunkley <Robert@xxxxxxxxx>
Delivery-date: Mon, 09 Nov 2009 07:42:04 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20091109151750.GN16033@xxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Hmmm... I'll bet the problem with dom0 crashing is that
dom0 is pv_ops (2.6.31-based) and this patch, which
has been in 2.6.18.8-based dom0 for some time, has never
been put into pv_ops:

http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00143.html

That would explain why some people are seeing this problem
and others are not, and why setting dom0_mem seems to
solve the problem (as dom0_mem effectively shuts off
ballooning in dom0).

Nick and others, if you are interested in detail on
ballooning and improving memory utilization within
and between guests, see:

http://wiki.xensource.com/xenwiki/Open_Topics_For_Discussion?action=AttachFile&do=get&target=Memory+Overcommit.pdf

and

http://oss.oracle.com/projects/tmem

(Tmem is in xen-unstable and in Oracle VM 2.2... it takes
a few steps to set it up.)

> -----Original Message-----
> From: Pasi Kärkkäinen [mailto:pasik@xxxxxx]
> Sent: Monday, November 09, 2009 8:18 AM
> To: Nick Couchman
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; Mr. Teo En Ming (Zhang Enming);
> xen-users@xxxxxxxxxxxxxxxxxxx; Robert Dunkley
> Subject: Re: [Xen-devel] Re: [Xen-users] Max. PV and HVM Guests
> 
> 
> On Mon, Nov 09, 2009 at 08:06:54AM -0700, Nick Couchman wrote:
> > 
> > 
> > >>> On 2009/11/09 at 05:05, Pasi Kärkkäinen<pasik@xxxxxx> wrote: 
> > > On Mon, Nov 09, 2009 at 08:01:00PM +0800, Mr. Teo En Ming (Zhang
> > Enming) 
> > > wrote:
> > >>    No, I didn't limit dom0 memory in grub.conf.
> > >> 
> > > 
> > > You should. 
> > 
> > Really?  I thought current conventional wisdom was to allow Xen to
> > self-manage memory in both dom0 and domUs, and not to 
> manually adjust
> > this?  I run several Xen systems with anywhere from 8 to 24 
> GB of RAM
> > and 20 to 30 domUs on some of these systems and have 
> *never* specified
> > the dom0 memory at boot time - the Xen ballooning has 
> always functioned
> > perfectly fine, and never crashed my dom0.  
> >
> 
> Yes, Xen is totally OK with this, but dom0 Linux has more problems..
> 
> > Furthermore, while I'm not
> > Linux developer and so not familiar with how Linux 
> calculates buffering
> > and caching, I do know that my Linux systems dynamically 
> manage buffers
> > and caches, and when memory is reduced or some application 
> requires a
> > larger amount of physical memory, Linux reduces the amount 
> of data in
> > buffers and caches.
> > 
> 
> Yeah, it has to do with sizing the network buffers, caches etc.. 
> 
> It shouldn't _crash_, so Teo is seeing some bug I believe. But it has
> always been "best practice" to limit dom0 memory - and prevent weird
> things happening later (like "memory squeeze in netback driver").
> 
> > Of course, a lot of this depends on what you're doing in 
> dom0 - on my
> > Xen servers, my dom0 is strictly for Xen management - I'm 
> not running
> > anything else in dom0 that would require large amounts of 
> memory, memory
> > buffers and caches, etc.
> > 
> 
> Teo is running graphical stuff, X etc so it's a bit different..
> 
> -- Pasi
> 
> > 
> > > 
> > > If dom0 has all the memory at boot time, you need to balloon down
> > dom0 
> > > memory every time you create a new guest - this can (and 
> will) cause
> > 
> > > problems with the dom0 linux kernel.
> > > 
> > > Linux calculates some internal parameters/buffers/values based on
> > the
> > > _boot time_ amount of memory. And when the amount of 
> memory goes down
> > to
> > > only a small fraction of that while creating new guests bad things
> > can
> > > happen..
> > > 
> > > It still shouldn't crash though.. I bet your problem will 
> get fixed
> > when
> > > you limit the dom0 memory to say dom0_mem=512M and reboot.
> > > 
> > > -- Pasi
> > > 
> > >>    Here's my xm info output after I have shutdown all the virtual
> > machines.
> > >> 
> > >>    [root@fedora11-x86-64-host ~]# xm list
> > >>    Name                                        ID   Mem 
> VCPUs     
> > State
> > >>    Time(s)
> > >>    Domain-0                                     0  2812     2    
> > r-----
> > >>    3242.5
> > >>    [root@fedora11-x86-64-host ~]# xm info
> > >>    host                   : fedora11-x86-64-host
> > >>    release                : 2.6.30-rc3-enming.teo-tip
> > >>    version                : #1 SMP Wed Aug 19 23:14:15 SGT 2009
> > >>    machine                : x86_64
> > >>    nr_cpus                : 2
> > >>    nr_nodes               : 1
> > >>    cores_per_socket       : 2
> > >>    threads_per_core       : 1
> > >>    cpu_mhz                : 2800
> > >>    hw_caps                :
> > >>   
> > 
> bfebfbff:20100800:00000000:00000140:0400e3bd:00000000:00000001
> :00000000
> > >>    virt_caps              : hvm hvm_directio
> > >>    total_memory           : 6039
> > >>    free_memory            : 3124
> > >>    node_to_cpu            : node0:0-1
> > >>    node_to_memory         : node0:3124
> > >>    xen_major              : 3
> > >>    xen_minor              : 5
> > >>    xen_extra              : -unstable
> > >>    xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
> > hvm-3.0-x86_32
> > >>    hvm-3.0-x86_32p hvm-3.0-x86_64
> > >>    xen_scheduler          : credit
> > >>    xen_pagesize           : 4096
> > >>    platform_params        : virt_start=0xffff800000000000
> > >>    xen_changeset          : Tue Sep 01 11:34:31 2009 +0100 
> > > 20143:a7de5bd776ca
> > >>    xen_commandline        : iommu=1
> > >>    cc_compiler            : gcc version 4.4.1 20090725 (Red Hat
> > 4.4.1-2)
> > >>    (GCC)
> > >>    cc_compile_by          : root
> > >>    cc_compile_domain      : (none)
> > >>    cc_compile_date        : Thu Sep 10 07:01:13 SGT 2009
> > >>    xend_config_format     : 4
> > >> 
> > >>    --
> > >>    Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics)
> > BEng(Hons)(Mechanical
> > >>    Engineering)
> > >>    Alma Maters:
> > >>    (1) Singapore Polytechnic
> > >>    (2) National University of Singapore
> > >>    My Primary Blog:
> > [1]http://teo-en-ming-aka-zhang-enming.blogspot.com
> > >>    My Secondary Blog: [2]http://enmingteo.wordpress.com
> > >>    My Youtube videos: [3]http://www.youtube.com/user/enmingteo
> > >>    Email: [4]space.time.universe@xxxxxxxxx
> > >>    Mobile Phone (Starhub Prepaid): +65-8369-2618
> > >>    Street: Bedok Reservoir Road
> > >>    Country: Singapore
> > >> 
> > >>    On Mon, Nov 9, 2009 at 7:54 PM, Pasi Kärkkäinen 
> <[5]pasik@xxxxxx>
> > wrote:
> > >> 
> > >>      On Mon, Nov 09, 2009 at 06:52:37PM +0800, Mr. Teo En Ming
> > (Zhang 
> > > Enming)
> > >>      wrote:
> > >>      >    Hi,
> > >>      >
> > >>      >    Please watch this 4-minute video at
> > >>      >    [1][6]http://www.youtube.com/watch?v=LbLaPpwNAx4
> > >>      >
> > >>      >    I have only started 3 HVM Linux guests with 1 
> GB ram each.
> > I can't
> > >>      start
> > >>      >    the 4th HVM guest. If I attempt to start the 
> 4th instance,
> > it will
> > >>      crash
> > >>      >    dom0.
> > >>      >
> > >>      >    Are there anything in the xm dmesg output that could
> > explain the
> > >>      low limit
> > >>      >    to the number of VMs that I could start before dom0
> > becomes
> > >>      unresponsive?
> > >>      >
> > >> 
> > >>      Have you limited dom0 memory (by specifying 
> dom0_mem=XMB option
> > in
> > >>      grub.conf for xen.gz) ?
> > >> 
> > >>      What does "xm info" say about free memory before 
> starting any
> > guests?
> > >>      -- Pasi
> > >> 
> > >> References
> > >> 
> > >>    Visible links
> > >>    1. http://teo-en-ming-aka-zhang-enming.blogspot.com/
> > >>    2. http://enmingteo.wordpress.com/
> > >>    3. http://www.youtube.com/user/enmingteo
> > >>    4. mailto:space.time.universe@xxxxxxxxx
> > >>    5. mailto:pasik@xxxxxx
> > >>    6. http://www.youtube.com/watch?v=LbLaPpwNAx4
> > 
> > 
> > 
> > --------
> > This e-mail may contain confidential and privileged 
> material for the sole use of the intended recipient.  If this 
> email is not intended for you, or you are not responsible for 
> the delivery of this message to the intended recipient, 
> please note that this message may contain SEAKR Engineering 
> (SEAKR) Privileged/Proprietary Information.  In such a case, 
> you are strictly prohibited from downloading, photocopying, 
> distributing or otherwise using this message, its contents or 
> attachments in any way.  If you have received this message in 
> error, please notify us immediately by replying to this 
> e-mail and delete the message from your mailbox.  Information 
> contained in this message that does not relate to the 
> business of SEAKR is neither endorsed by nor attributable to SEAKR.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel