Hmmm... I'll bet the problem with dom0 crashing is that
dom0 is pv_ops (2.6.31-based) and this patch, which
has been in 2.6.18.8-based dom0 for some time, has never
been put into pv_ops:
http://lists.xensource.com/archives/html/xen-devel/2008-04/msg00143.html
That would explain why some people are seeing this problem
and others are not, and why setting dom0_mem seems to
solve the problem (as dom0_mem effectively shuts off
ballooning in dom0).
Nick and others, if you are interested in detail on
ballooning and improving memory utilization within
and between guests, see:
http://wiki.xensource.com/xenwiki/Open_Topics_For_Discussion?action=AttachFile&do=get&target=Memory+Overcommit.pdf
and
http://oss.oracle.com/projects/tmem
(Tmem is in xen-unstable and in Oracle VM 2.2... it takes
a few steps to set it up.)
> -----Original Message-----
> From: Pasi Kärkkäinen [mailto:pasik@xxxxxx]
> Sent: Monday, November 09, 2009 8:18 AM
> To: Nick Couchman
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; Mr. Teo En Ming (Zhang Enming);
> xen-users@xxxxxxxxxxxxxxxxxxx; Robert Dunkley
> Subject: Re: [Xen-devel] Re: [Xen-users] Max. PV and HVM Guests
>
>
> On Mon, Nov 09, 2009 at 08:06:54AM -0700, Nick Couchman wrote:
> >
> >
> > >>> On 2009/11/09 at 05:05, Pasi Kärkkäinen<pasik@xxxxxx> wrote:
> > > On Mon, Nov 09, 2009 at 08:01:00PM +0800, Mr. Teo En Ming (Zhang
> > Enming)
> > > wrote:
> > >> No, I didn't limit dom0 memory in grub.conf.
> > >>
> > >
> > > You should.
> >
> > Really? I thought current conventional wisdom was to allow Xen to
> > self-manage memory in both dom0 and domUs, and not to
> manually adjust
> > this? I run several Xen systems with anywhere from 8 to 24
> GB of RAM
> > and 20 to 30 domUs on some of these systems and have
> *never* specified
> > the dom0 memory at boot time - the Xen ballooning has
> always functioned
> > perfectly fine, and never crashed my dom0.
> >
>
> Yes, Xen is totally OK with this, but dom0 Linux has more problems..
>
> > Furthermore, while I'm not
> > Linux developer and so not familiar with how Linux
> calculates buffering
> > and caching, I do know that my Linux systems dynamically
> manage buffers
> > and caches, and when memory is reduced or some application
> requires a
> > larger amount of physical memory, Linux reduces the amount
> of data in
> > buffers and caches.
> >
>
> Yeah, it has to do with sizing the network buffers, caches etc..
>
> It shouldn't _crash_, so Teo is seeing some bug I believe. But it has
> always been "best practice" to limit dom0 memory - and prevent weird
> things happening later (like "memory squeeze in netback driver").
>
> > Of course, a lot of this depends on what you're doing in
> dom0 - on my
> > Xen servers, my dom0 is strictly for Xen management - I'm
> not running
> > anything else in dom0 that would require large amounts of
> memory, memory
> > buffers and caches, etc.
> >
>
> Teo is running graphical stuff, X etc so it's a bit different..
>
> -- Pasi
>
> >
> > >
> > > If dom0 has all the memory at boot time, you need to balloon down
> > dom0
> > > memory every time you create a new guest - this can (and
> will) cause
> >
> > > problems with the dom0 linux kernel.
> > >
> > > Linux calculates some internal parameters/buffers/values based on
> > the
> > > _boot time_ amount of memory. And when the amount of
> memory goes down
> > to
> > > only a small fraction of that while creating new guests bad things
> > can
> > > happen..
> > >
> > > It still shouldn't crash though.. I bet your problem will
> get fixed
> > when
> > > you limit the dom0 memory to say dom0_mem=512M and reboot.
> > >
> > > -- Pasi
> > >
> > >> Here's my xm info output after I have shutdown all the virtual
> > machines.
> > >>
> > >> [root@fedora11-x86-64-host ~]# xm list
> > >> Name ID Mem
> VCPUs
> > State
> > >> Time(s)
> > >> Domain-0 0 2812 2
> > r-----
> > >> 3242.5
> > >> [root@fedora11-x86-64-host ~]# xm info
> > >> host : fedora11-x86-64-host
> > >> release : 2.6.30-rc3-enming.teo-tip
> > >> version : #1 SMP Wed Aug 19 23:14:15 SGT 2009
> > >> machine : x86_64
> > >> nr_cpus : 2
> > >> nr_nodes : 1
> > >> cores_per_socket : 2
> > >> threads_per_core : 1
> > >> cpu_mhz : 2800
> > >> hw_caps :
> > >>
> >
> bfebfbff:20100800:00000000:00000140:0400e3bd:00000000:00000001
> :00000000
> > >> virt_caps : hvm hvm_directio
> > >> total_memory : 6039
> > >> free_memory : 3124
> > >> node_to_cpu : node0:0-1
> > >> node_to_memory : node0:3124
> > >> xen_major : 3
> > >> xen_minor : 5
> > >> xen_extra : -unstable
> > >> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p
> > hvm-3.0-x86_32
> > >> hvm-3.0-x86_32p hvm-3.0-x86_64
> > >> xen_scheduler : credit
> > >> xen_pagesize : 4096
> > >> platform_params : virt_start=0xffff800000000000
> > >> xen_changeset : Tue Sep 01 11:34:31 2009 +0100
> > > 20143:a7de5bd776ca
> > >> xen_commandline : iommu=1
> > >> cc_compiler : gcc version 4.4.1 20090725 (Red Hat
> > 4.4.1-2)
> > >> (GCC)
> > >> cc_compile_by : root
> > >> cc_compile_domain : (none)
> > >> cc_compile_date : Thu Sep 10 07:01:13 SGT 2009
> > >> xend_config_format : 4
> > >>
> > >> --
> > >> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics)
> > BEng(Hons)(Mechanical
> > >> Engineering)
> > >> Alma Maters:
> > >> (1) Singapore Polytechnic
> > >> (2) National University of Singapore
> > >> My Primary Blog:
> > [1]http://teo-en-ming-aka-zhang-enming.blogspot.com
> > >> My Secondary Blog: [2]http://enmingteo.wordpress.com
> > >> My Youtube videos: [3]http://www.youtube.com/user/enmingteo
> > >> Email: [4]space.time.universe@xxxxxxxxx
> > >> Mobile Phone (Starhub Prepaid): +65-8369-2618
> > >> Street: Bedok Reservoir Road
> > >> Country: Singapore
> > >>
> > >> On Mon, Nov 9, 2009 at 7:54 PM, Pasi Kärkkäinen
> <[5]pasik@xxxxxx>
> > wrote:
> > >>
> > >> On Mon, Nov 09, 2009 at 06:52:37PM +0800, Mr. Teo En Ming
> > (Zhang
> > > Enming)
> > >> wrote:
> > >> > Hi,
> > >> >
> > >> > Please watch this 4-minute video at
> > >> > [1][6]http://www.youtube.com/watch?v=LbLaPpwNAx4
> > >> >
> > >> > I have only started 3 HVM Linux guests with 1
> GB ram each.
> > I can't
> > >> start
> > >> > the 4th HVM guest. If I attempt to start the
> 4th instance,
> > it will
> > >> crash
> > >> > dom0.
> > >> >
> > >> > Are there anything in the xm dmesg output that could
> > explain the
> > >> low limit
> > >> > to the number of VMs that I could start before dom0
> > becomes
> > >> unresponsive?
> > >> >
> > >>
> > >> Have you limited dom0 memory (by specifying
> dom0_mem=XMB option
> > in
> > >> grub.conf for xen.gz) ?
> > >>
> > >> What does "xm info" say about free memory before
> starting any
> > guests?
> > >> -- Pasi
> > >>
> > >> References
> > >>
> > >> Visible links
> > >> 1. http://teo-en-ming-aka-zhang-enming.blogspot.com/
> > >> 2. http://enmingteo.wordpress.com/
> > >> 3. http://www.youtube.com/user/enmingteo
> > >> 4. mailto:space.time.universe@xxxxxxxxx
> > >> 5. mailto:pasik@xxxxxx
> > >> 6. http://www.youtube.com/watch?v=LbLaPpwNAx4
> >
> >
> >
> > --------
> > This e-mail may contain confidential and privileged
> material for the sole use of the intended recipient. If this
> email is not intended for you, or you are not responsible for
> the delivery of this message to the intended recipient,
> please note that this message may contain SEAKR Engineering
> (SEAKR) Privileged/Proprietary Information. In such a case,
> you are strictly prohibited from downloading, photocopying,
> distributing or otherwise using this message, its contents or
> attachments in any way. If you have received this message in
> error, please notify us immediately by replying to this
> e-mail and delete the message from your mailbox. Information
> contained in this message that does not relate to the
> business of SEAKR is neither endorsed by nor attributable to SEAKR.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home