This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] [timer/ticks related] dom0 hang during boot on large 1TB

To: Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Subject: Re: [Xen-devel] [timer/ticks related] dom0 hang during boot on large 1TB system
From: Mukesh Rathor <mukesh.rathor@xxxxxxxxxx>
Date: Fri, 18 Dec 2009 20:43:18 -0800
Cc: Hackel <kurt.hackel@xxxxxxxxxx>, Kurt, "Xen-devel@xxxxxxxxxxxxxxxxxxx" <Xen-devel@xxxxxxxxxxxxxxxxxxx>, Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>, jeremy@xxxxxxxx
Delivery-date: Fri, 18 Dec 2009 20:43:39 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <C750D99F.4B25%keir.fraser@xxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20091217203636.76a10aea@xxxxxxxxxxxxxxxxxxxx> <C750D99F.4B25%keir.fraser@xxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
On Fri, 18 Dec 2009 07:02:55 +0000
Keir Fraser <keir.fraser@xxxxxxxxxxxxx> wrote:

> On 18/12/2009 04:36, "Mukesh Rathor" <mukesh.rathor@xxxxxxxxxx> wrote:
> > The other fix I thought of was to change INITIAL_JIFFIES to
> > something sooner.
> > 
> > Would appreciate any help, I don't understand xen time management
> > well.
> This isn't really Xen time code, but unchanged Linux time code. I
> don't know which tree you quoted the code from -- 2.6.18 has similar
> but not identical. Anyway, I suggest try using the jiffy-comparison
> macros from <linux/jiffies.h>: time_before(), time_after(), etc.
> These are designed to work even when jiffies wraps. Feel free to send
> patch(es) for that, if you test that out and it works okay.
>  -- Keir

Ok, I came up with the following patch. Jeremy, can you please take a
look also, and comment on my fix since I noticed you've got the same 
issue in your tree. Here's a summary for your benefit:

init/calibrate.c :  calibrate_delay_direct():

                start_jiffies = get_jiffies_64();
                while (get_jiffies_64() <= (start_jiffies + tick_divider)) {
                        pre_start = start;

if first ever timer interrupt comes after start_jiffies is set, dom0 boot 
may hang if delta in timer_interrupt() is so huge that it causes jiffies 
to wrap. It appears delta is very large when memory is more than 512GB on
certain boxes causing wrap around.

why is delta in dom0->timer_interrupt() related to memory on system? 
Because hyp creates dom0, then page scrubs, then unpauses vcpu. so it
appears lot of page scurbbing results in huge delta on first tick.


Signed-off-by: Mukesh Rathor <mukesh.rathor@xxxxxxxxxx>

diff --git a/init/calibrate.c b/init/calibrate.c
index 06066a6..14f62c8 100644
--- a/init/calibrate.c
+++ b/init/calibrate.c
@@ -32,7 +32,7 @@ static unsigned long __devinit calibrate_delay_direct(void)
        unsigned long pre_start, start, post_start;
        unsigned long pre_end, end, post_end;
-       unsigned long start_jiffies;
+       u64 start_jiffies;
        unsigned long tsc_rate_min, tsc_rate_max;
        unsigned long good_tsc_sum = 0;
        unsigned long good_tsc_count = 0;
@@ -64,8 +64,8 @@ static unsigned long __devinit calibrate_delay_direct(void)
        for (i = 0; i < MAX_DIRECT_CALIBRATION_RETRIES; i++) {
                pre_start = 0;
-               start_jiffies = jiffies;
-               while (jiffies <= (start_jiffies + tick_divider)) {
+               start_jiffies = get_jiffies_64();
+               while (get_jiffies_64() <= (start_jiffies + tick_divider)) {
                        pre_start = start;
@@ -73,7 +73,7 @@ static unsigned long __devinit calibrate_delay_direct(void)
                pre_end = 0;
                end = post_start;
-               while (jiffies <=
+               while (get_jiffies_64() <=
                       (start_jiffies + tick_divider * (1 + 
delay_calibration_ticks))) {
                        pre_end = end;

Attachment: diff.out
Description: Binary data

Xen-devel mailing list
<Prev in Thread] Current Thread [Next in Thread>