RE: [Xen-devel] Xend crashes, how to debug?

[Dave Scott <Dave.Scott@xxxxxxxxxxxxx>]

The observation that speeding up xenstore reduces the frequency of crashes is 
interesting. Perhaps the failure happens when a concurrent transaction causes 
an abort? Maybe you could provoke it by running 'xm create' in a loop while 
also writing somewhere in xenstore? IIRC (although I could be mistaken) the 
standard C xenstore considers all concurrent transactions to be conflicting 
even if they operate on disjoint parts of the tree so provoking an abort would 
be easy.

> And XenServer is opensource now.. so you could always try the ocaml
> xend.
> 
> Althought I'm not sure if it's easy/direct replacement..

Sorry, couldn't resist:
<begin advert>
Feel free to give it a go. Although it's still in development (it's in a bit of 
a stabilization phase atm) mysterious toolstack crashes / segfaults are rare 
(famous last words?). The kind of bugs it's currently suffering from are mostly 
to do with the new functionality we've been integrating recently e.g. RBAC, 
ballooning etc. For more normal stuff it ought to be pretty good.

Caveats:
1. We don't have an 'xm'... instead there's a CLI called 'xe' which can do 
almost everything the API can do but the syntax is different to 'xm'. You'd 
either have to port your scripts ('xe vm-start' rather than 'xm create'?) or 
write some kind of wrapper.

2. It's much easier to install and use the whole integrated patched xen + 
patched qemu + dom0 + toolstack rather than transplant the toolstack onto 
another dom0. I'm sure it's possible but we've been focusing dev + test on the 
single environment.

http://www.xen.org/products/cloud_source.html
<end advert>

FWIW we also use an ocaml xenstore which handles concurrent transactions 
efficiently. There are some performance graphs here:

http://thomas.gazagnaire.com/pub/GH09.pdf

The reason we rewrote xenstored was because we used xenstore to report periodic 
guest performance stats to dom0. By doing this we accidentally created a 
horrible scalability bottleneck where, somewhere around 30 or 40 guests, every 
transaction aborted and the system livelocked. The new xenstored is smart 
enough to realize that these separate transactions are not conflicting and can 
be committed together.

Cheers,
Dave


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel