Re: [Xen-devel] [timer/ticks related] dom0 hang during boot on large 1TB system

[Mukesh Rathor <mukesh.rathor@xxxxxxxxxx>]

On Mon, 21 Dec 2009 11:55:09 -0800
Jeremy Fitzhardinge <jeremy@xxxxxxxx> wrote:

> On 12/21/2009 11:52 AM, Mukesh Rathor wrote:
> > On Mon, 21 Dec 2009 19:07:39 +0000
> > Keir Fraser<keir.fraser@xxxxxxxxxxxxx>  wrote:
> >
> >    
> >> On 21/12/2009 18:20, "Dan Magenheimer"<dan.magenheimer@xxxxxxxxxx>
> >> wrote:
> >>
> >>      
> >>> Not to say the problem can't or shouldn't be fixed in Xen.
> >>> Keir, would bad things happen if construct_dom0 is done after
> >>> scrub_heap_pages()?  Other than some time wastage because
> >>> dom0's memory would get scrubbed just before it gets
> >>> overwritten (which is admittedly a much bigger problem
> >>> when dom0_mem is not specified in the Xen boot line
> >>> on a machine with ginormous memory).
> >>>        
> >> The problem is more likely that Xen system time started ticking
> >> some time earlier during boot process. I doubt it is to do with
> >> ordering of construct_dom0 versus boot-time scrubbing.
> >>
> >>   -- Keir
> >>
> >>      
> > The problem is exactly how Dan described it. 'delta' for first
> > interrupt in dom0->timer_interrupt() goes up proportionately with
> > amount of memory on system. On this box, it appears more than 600GB
> > causes delta to be large enough to wrap jiffies.
> >
> >   1TB delta: 940b7d68a4
> > 32GB delta: 02ae56eadb
> >
> > xen->send_guest_vcpu_virq() ---->  dom0->handle_IRQ() ->
> > timer_interrupt()
> >
> > timer_interrupt will call do_timer delta/NS_PER_TICK number of
> > times. 
> 
> How is it computing that delta?
> 
> Anyway, I'm not at all sure this will apply to a pvops dom0 kernel as
> it does timekeeping quite differently from 2.6.18-xen.
> 
>      J

delta comes from:
 
timer_inetrrupt() in time-xen.c :
...
        do {
                get_time_values_from_xen(cpu);

                /* Obtain a consistent snapshot of elapsed wallclock cycles. */
      --->      delta = delta_cpu =
                        shadow->system_timestamp + get_nsec_offset(shadow);
      --->      delta     -= processed_system_time;
                delta_cpu -= per_cpu(processed_system_time, cpu);

                /*
                 * Obtain a consistent snapshot of stolen/blocked cycles. We
                 * can use state_entry_time to detect if we get preempted here.
                 */
                do {
                        sched_time = runstate->state_entry_time;
                        barrier();
                        stolen = runstate->time[RUNSTATE_runnable] +
                                runstate->time[RUNSTATE_offline] -
                                per_cpu(processed_stolen_time, cpu);
                        blocked = runstate->time[RUNSTATE_blocked] -
                                per_cpu(processed_blocked_time, cpu);
                        barrier();
                } while (sched_time != runstate->state_entry_time);
        } while (!time_values_up_to_date(cpu));
...


At first glance, i don't understand the above algorithm. Since you've
the same code, I assumed you could also compute delta to be a large
value when dom0 starts, in which case you may observe dom0 hang. 


thanks,
Mukesh



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel