Re: [Xen-devel] [timer/ticks related] dom0 hang during boot on large 1TB system

[Mukesh Rathor <mukesh.rathor@xxxxxxxxxx>]

On Mon, 21 Dec 2009 19:07:39 +0000
Keir Fraser <keir.fraser@xxxxxxxxxxxxx> wrote:

> On 21/12/2009 18:20, "Dan Magenheimer" <dan.magenheimer@xxxxxxxxxx>
> wrote:
> 
> > Not to say the problem can't or shouldn't be fixed in Xen.
> > Keir, would bad things happen if construct_dom0 is done after
> > scrub_heap_pages()?  Other than some time wastage because
> > dom0's memory would get scrubbed just before it gets
> > overwritten (which is admittedly a much bigger problem
> > when dom0_mem is not specified in the Xen boot line
> > on a machine with ginormous memory).
> 
> The problem is more likely that Xen system time started ticking some
> time earlier during boot process. I doubt it is to do with ordering of
> construct_dom0 versus boot-time scrubbing.
> 
>  -- Keir
> 

The problem is exactly how Dan described it. 'delta' for first interrupt
in dom0->timer_interrupt() goes up proportionately with amount of memory
on system. On this box, it appears more than 600GB causes delta to be 
large enough to wrap jiffies.

 1TB delta: 940b7d68a4
32GB delta: 02ae56eadb

xen->send_guest_vcpu_virq() ----> dom0->handle_IRQ() -> timer_interrupt()

timer_interrupt will call do_timer delta/NS_PER_TICK number of times.

Linux initializes jiffies to -5 minutes to catch problems from jiffies 
wrap early on. But like Dan said, dom0->calibrate_delay_direct() on 
baremetal starts running right away and is guaranteed to run in less 
than 5 minutes. We could let that assumption be true by moving page
scrub before xen->construct_dom0(), in which case the first timer
interrupt in dom0 will come in lot sooner, or just fix the loop to
account for wrap. Since jiffies just represents lower 32bits of 
jiffies_64, and get_jiffies_64() is provided for the purpose of reading
64bit version, I just avail of that. 


Thanks,
Mukesh


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel