WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] bring up Hypervisor on large (512GB) memory

To: <mukesh.rathor@xxxxxxxxxx>
Subject: Re: [Xen-devel] bring up Hypervisor on large (512GB) memory
From: "Jan Beulich" <jbeulich@xxxxxxxxxx>
Date: Tue, 10 Feb 2009 08:11:20 +0000
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Tue, 10 Feb 2009 00:11:08 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4990EFC7.9000907@xxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4990EFC7.9000907@xxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Minus the regression Keir talks about, we have indications that Xen (even
3.3 iirc) itself boots fine even with 1Tb. The issues come with Dom0
booting: I recently submitted a patch to add infrastructure in Xen to allow
Dom0 to get a valid initial memory map, but the kernel side changes we
have could not easily be applied to the 2.6.18 tree, so that kernel will
continue to not be suitable for this large an amount of memory. Nor can I
at this point really tell whether those changes were all that's needed to get
things going, as the testing get stalled by the testers losing access to the
single machine that had this much memory.

For SLES10 (3.2 based), where we backported the necessary hypervisor
changes, we have indications that there might be other issues (as the
testers reported even booting with mem=400G failed), but we haven't
got hard data (read: back traces), and I haven't been able to spot
further issues by static code inspection.

Jan

>>> Mukesh Rathor <mukesh.rathor@xxxxxxxxxx> 10.02.09 04:08 >>>
Hi all,

Trying to bring up the hypervisor on 512GB :

1. Started with xen 3.1.4 (last oracle release), and 512GB, the system
panic'd right away:
    (XEN) Early fatal page fault at e008:ffff828c801ce0ad
    (cr2=ffff8300cfc00000, ec=0002)

I noticed an anomaly here in the RAM map:
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f400 (usable)
(XEN)  000000000009f400 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000802ffff000 (usable)  <-------

Seems that should be 0000008000000000 ???


2. Reduce some RAM, and booting with 440GB, map makes more sense:

(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f400 (usable)
(XEN)  000000000009f400 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000006e00000000 (usable)

Panic again:
(XEN) Early fatal page fault at e008:ffff828c80210460
(cr2=ffff8300cfc00000, ec=0002)


The panic is trying to allocate bitmap in init_boot_allocator(). The bitmap
starts at cfac0000 and given the size dc1000, won't fit.


3. Moving to unstable 19164, looks like things are even more tighter!! I
    couldn't even boot with 64GB. bitmap_start:cfac0000, bitmap_size:201000,
    alloc_bitmap:ffff8300cfac0000


The only solution I can think of is moving the bitmap elsewhere, above 4GB in
this case:
    figure the size of bitmap, DIRECT map space, allocate the map,
    mark it reserved in the RAM map, and should work!

    I'd have add a loop around init_boot_allocator() in __start_xen()
    iterating thru the RAM map again, and finding space above 16M.


Am I on the right track?


Thanks,
Mukesh


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx 
http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel