[Xen-devel] USB virt 2.6 status

[Harry Butterworth <harry@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Summary:

USB 2.6 back and front-end driver modules can now be loaded and unloaded
and the new xend usb code sequences the control messages to establish
the connection and tear it down as necessary.

I'm going back to forward porting the remaining USB specific 2.4 code to
complete the 2.6 back-end before resuming testing on the still
incomplete 2.6 front-end.

Detailed status:

I spent some time trying to finish the new (usb driver domain support)
xend usb code and got to the point where I thought I had most of the
code in place apart from the core state machine which sequences the
control messages between the front and back-end.  The state machine
support code was there but I found it impossible to write the actual
state machine because I didn't have a complete understanding of the
control message protocol and the behaviour of the xend messaging support
code.

I use a technique called 'sequence enumeration' for writing state
machine code which consists of: defining a mapping between the inputs to
a system and stimuli to the state machine; defining a set of state
machine responses which may be atomic or may complete asynchronously by
generating a stimulus; enumerating all possible states starting at an
initial state by considering all of the possible stimuli that may occur
in that state, what the desired state machine responses are and whether
the resulting state is a new state or is equivalent to a state
previously considered.

Sequence enumeration is a fairly painful process but it forces you to
think about all the corner cases and, provided you are rigorous enough
in defining the input->stimuli mapping and the responses, allows you to
write code which is likely to be correct.

In the case of the xend usb code I couldn't get the state machine to
close because of confusion about a few aspects of the control message
protocol concerning unloading of modules: it's unclear whether a driver
status down message is a request for xend to sequence stuff to allow the
driver to go down or an indication to xend that the driver has already
sequenced stuff and is going down. Also, if the front end is supposed to
issue disconnects for all interfaces before going down then there is the
possibility for an interface status disconnected resulting from the
back-end going down crossing in the post with an interface disconnect
request from the front end.

It's not particularly hard to construct a protocol to solve this kind of
problem from scratch (I've done it before for multi-pathing device
drivers and cluster inter-node communication) but I found it very
difficult trying to reverse engineer the intent of the existing
protocol. Also, I didn't understand the system behaviour that would
result if messages went unacknowledged as a result of a driver unloading
so I tried very hard for some time to avoid any unacknowledged messages.

I decided to defer writing the xend usb state machine code until it was
actually required for some testing when I could determine the system
behaviour experimentally.

I started testing with the new xend usb code and found that I couldn't
reliably start and stop domains without getting zombie domains that
would force me to reboot.  The problem turned out to be in
controller.py: I had to change the BackendController to be a child of
CtrlMsgRcvr rather than Controller and change lostChannel to call
self.factory.backendControllerClosed in place of
self.backend.backendClosed.

Once I could start and stop my skeletal USB driver domains reliably, I
ported a chunk of the 2.4 USB back-end into the 2.6 skeleton back-end
and coded the state machine to handle receipt of the control interface
messages in the back-end.  I made the assumption that, when unloading
the back-end module, the back-end would stop using all front-end memory
before sending the back-end driver status down message and the xend usb
code would stop sending the back-end control messages before
acknowledging the driver status down message.
On the first attempt, I had the back-end make sure to respond to all
received control messages.

The back-end state machine seemed to come out OK so I went on to the
front-end state machine (which lacked shutdown before) and tried to
enumerate a complete solution that included shutdown.

To start with, I assumed that the front-end would disconnect its
interfaces before sending a driver status down message and I again tried
to ensure that the front-end would respond to all received control
messages.  With these assumptions, I couldn't get the state machine to
close because of problems with disconnects crossing in the post.

I tried again, this time assuming that the front-end would send a driver
status down first as a request to xend to get the back-end to free up
resources and disconnect the interfaces so the front-end could exit
gracefully.

The state machine for the second attempt also got too complicated.

In the end, I came to the conclusion that the interpretation of the
existing protocol resulting in the simplest solution is that the
back-end driver status down is an indication that the back-end has
finished using all front-end resources and won't necessarily respond to
outstanding or subsequent control messages and the front-end driver
status down is an indication that the front-end won't send any more
control messages and won't necessarily respond to any outstanding or
subsequent control messages and is also a request that xend force the
back-end(s) to free up all front-end resources before acknowledging.

Given this interpretation, the front end state machine came out
relatively easily.

After this third attempt at the front and back end state machines, I
went back to the xend usb state machine in the middle and wrote the
straight through path for loading the back-end, loading the front-end,
unloading the front-end and then unloading the back-end.

I spent a very painful day trying to debug a problem in this sequence
where messages to the back end were hanging xend.  This turned out to be
a whitespace issue in the xend usb code I'd written. Being a python
novice, I had formatted a call to packMsg as follows...

msg = packMsg
(
    ...lots of parameters on multiple lines
)

writeRequest( msg )

...which is how I'd do it in C.  Unfortunately this fails silently in
python (since---I'm guessing---msg is set to the function packMsg and
the parameters are interpreted as a statement with no effect).

After fixing this issue the messaging sprang back into life and now,
after another day of debugging and a bit more hacking on the xend usb
state machine, the module unloading and loading is working well enough
to allow me to get back to porting the USB specific bits of the 2.4
back-end code into the 2.6 framework.

The xend usb state machine isn't yet a complete enumeration (and the
other state machines are a bit ragged after being reworked three times)
but it's good enough for the intended purpose: to improve the
turn-around-time for debugging the USB driver code by eliminating
reboots.

If I had known in advance it was going to be this painful to make it
work I would have put up with the reboots but now it is working it ought
to be very convenient.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel