Setup: RedHat/Centos/Sci. Linux 5 update 3,
Dom0: kernel-xen-2.6.18-164.10.1.el5xen 64-bit
(on Dell Poweredge 2950 dual quad-core).
DomU: kernel-xen-2.6.18-164.10.1.el5xen 32-bit, 1 vcpu
6 domU's per dom0.
We have also seen the same problem with 2.6.18-164.9.1 and 2.6.18-164.6.1
kernels on this branch.
Symptom: On 32-bit domU only (we have never seen 64-bit domU be affected),
we observe that the clock gets set forward by a random amount of time,
between 40-80 minutes. We originally thought this was due to
rogue ntp broadcasts because the ntp broadcastclient was disabled,
but we have now disabled ntpd on all domU and the problem persists.
During the 40-80 minutes it is possible to log into the domU via
the Xen console, it appears to be otherwise functional although
the clock is stuck at the forward time. The daemons
(which are Axis-based web services) continue to function and service
requests, the node remains pingable, there is no loss of network.
Once the clock on the dom0 catches up to the time on the domU,
then the clock advances normally again on the domU as though nothing
had happened.
It is happening on any given domU on the average of every 2-3 weeks,
although they do appear to happen in time clusters. i.e. 3-4 domU's
report the clock skew within the span of a day or two, and then
nothing for a couple of weeks, and then we get another round and so forth.
Only one of the several domU's on the machine is affected at any
given time. We see this on two different pieces of hardware that
are identically configured.
There is nothing in dmesg, nothing in /var/log/messages or related
files,nothing in the xend logs. On the domU /var/log/messages
and related files have a total gap during this time. The logs of the
daemons continue to get written at a constant time stamp, although
I have seen cases where the system clock is 19:53, a daemon log
gets rotated at 19:40 and the time stamp on the rotated log shows
19:40.
Any help? Has anyone else seen anything like this?
Thanks
Steve Timm
--
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@xxxxxxxx http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
Home