On Sun, Jun 06, 2010 at 11:54:11AM -0700, Boris Derzhavets wrote:
> Virt-install hangs attempting to retrieve updates.img from Apache Mirror
> setup at Dom0
> with kernel 2.6.32.15
> It doesn't happen with 2.6.32.10 (12,14 final).
> Environment Xen 4.0 Dom0 on top Ubuntu 10.04 Server. Libvirt is 0.8.0 .
> Attempting to virt-install F13 PV DomU in vnc mode.
>
> I believe that builds are consistent.
> Runtime behavior is the same for .10, .14, .15. Seems to be
communicating
> problem between DomU and Dom0 during HTTP download.
>
I'm seeing PV domU network issues aswell.. when running Xen 4.0.0 on Fedora 13.
I'll have to dig more into it..
-- Pasi
> Boris.
>
> --- On Sun, 6/6/10, Jeremy Fitzhardinge <
jeremy@xxxxxxxx> wrote:
>
> From: Jeremy Fitzhardinge <
jeremy@xxxxxxxx>
> Subject: Re: [Xen-users] Re: [Xen-devel] ARP problems with xen 4.0 with
> pvops kernel 2.6.32.15
> To: "Boris Derzhavets" <
bderzhavets@xxxxxxxxx>
> Cc:
luis.silva@xxxxxxxxxxxxx,
xen-devel@xxxxxxxxxxxxxxxxxxx,
>
xen-users@xxxxxxxxxxxxxxxxxxx> Date: Sunday, June 6, 2010, 12:43 PM
>
> On 06/06/2010 03:19 AM, Boris Derzhavets wrote:
> > Network issues when working with DomUs in 2.6.32.14 and finally been
> > fixed,
> > seem to appear again in 2.6.32.15. Reverting to back to xen/stable -
> > 2.6.32.10
> > works as a fix again.
> >
>
>
There are no substantial differences between 2.6.32.14 and .15. If
> there are any differences in behaviour between them, then I'd suspect
> some inconsistency from boot to boot, or in your kernel build process.
>
> J
>
> >
> > Boris
> >
> > --- On *Thu, 6/3/10, Luís Silva /<[1]
luis.silva@xxxxxxxxxxxxx>/*
> wrote:
> >
> >
> > From: Luís Silva <[2]
luis.silva@xxxxxxxxxxxxx>
> > Subject: Re: [Xen-users] Re: [Xen-devel] ARP problems with xen 4.0
> > with pvops kernel
> > To: "Boris Derzhavets" <[3]
bderzhavets@xxxxxxxxx>
> > Cc: "Jeremy Fitzhardinge" <[4]
jeremy@xxxxxxxx>,
> > [5]
xen-devel@xxxxxxxxxxxxxxxxxxx, [6]
xen-users@xxxxxxxxxxxxxxxxxxx> > Date: Thursday, June 3, 2010, 6:20 AM
> >
> > Hello,
> >
> > Thanks for the suggestion, xen/stable works ok for me. Only
> > problem is that I have to disable offload do get dhcp to work on
> > domU, but the problem I described before doesn't exist in this
> > kernel. Later today I'm going to try a previous build I have based
> > on stable-2.6.32.x (2.6.32.13) to check if it already had this
> >
problem or not and I'll post the results.
> >
> > Luís
> >
> > On Wed, 2010-06-02 at 12:26 -0700, Boris Derzhavets wrote:
> >> Could you,please, build and try 2.6.32.10 ( xen/stable) ?
> >>
> >> Boris.
> >>
> >> --- On *Wed, 6/2/10, Luís Silva
> **/<[7]
luis.silva@xxxxxxxxxxxxx>/*
> >> wrote:
> >>
>
>>
> >> From: Luís Silva <[8]
luis.silva@xxxxxxxxxxxxx>
> >> Subject: [Xen-users] Re: [Xen-devel] ARP problems with xen
> >> 4.0 with pvops kernel
> >> To: "Jeremy Fitzhardinge" <[9]
jeremy@xxxxxxxx>
> >> Cc: [10]
xen-devel@xxxxxxxxxxxxxxxxxxx,
> [11]
xen-users@xxxxxxxxxxxxxxxxxxx> >> Date: Wednesday, June 2, 2010, 2:53 PM
> >>
> >> Hello,
> >>
> >> On Wed, 2010-06-02 at 09:06 -0700, Jeremy Fitzhardinge wrote:
> >>> On 06/02/2010 01:47 AM, Luís Silva wrote:
> >>> > Hello,
> >>> >
> >>> > I'm using the latest stable-2.6.32.x. I
already tried
> "ethtool -K
> >>> > <bridge> tx off", but that didn't make any difference.
> Also, this only
> >>> > happen with pv, in hvm mode all works ok and the domU sees
> the arp
> >>> > messages...
> >>>
> >>> Yes, ARP is a new twist on network problems. I'm guessing
> you're using
> >>> hvm without stubdoms, which means that its networking
> originates from
> >>>
qemu within dom0, whereas PV and HVM+stubdom comes via
> netback.
> >>>
> >>>
> >> Yes, when I mentioned hvm I was talking about hvm without
> >> stubdoms. I haven't tried those yet.
> >>> But aside from that, I'm stumped. Are you running any
> firewalls on
> >>> either side? Can you try disabling all the offloads (tx,
> rx, gso, tso)
> >>> on all the relevent interfaces (bridge, netback, within
the
> guest) and
> >>> see if that changes anything?
> >>>
> >>> J
> >>>
> >>>
> >>
> >> Ok, this is the bridge interface:
> >>
> >> brctl show
> >> bridge name bridge id STP enabled interfaces
> >> virbr0 8000.feffffffffff no
vif1.0
> >>
> >> ifconfig virbr0
> >> virbr0 Link encap:Ethernet HWaddr c2:ef:67:2b:a4:23
> >> inet addr:192.168.120.254 Bcast:192.168.120.255
> Mask:255.255.255.0
> >> inet6 addr: fe80::c0ef:67ff:fe2b:a423/64 Scope:Link
> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> >> RX packets:0
errors:0 dropped:0 overruns:0 frame:0
> >> TX packets:25 errors:0 dropped:0 overruns:0
> carrier:0
> >> collisions:0 txqueuelen:0
> >> RX bytes:0 (0.0
> >> B)
> >> TX bytes:4662 (4.6 KB)
> >>
> >>
> >>
> >> I'm not using firewall other than the rules defined by
> >>
libvirt. DomU has no firewall and the rules in dom0 are only
> >> these (virbr0 is natted to the outside, virbr1 is routed. The
> >> result is the same in either one of them):
> >>
> >> sudo iptables -L -n -v
> >> Chain INPUT (policy ACCEPT 241K packets, 53M bytes)
> >> pkts bytes target prot opt in out source
> destination
> >> 0
0 ACCEPT udp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 udp dpt:53
> >> 0 0 ACCEPT tcp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:53
> >>
> >>
> >> 0 0 ACCEPT udp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 udp
dpt:67
> >> 0 0 ACCEPT tcp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:67
> >> 8 515 ACCEPT udp -- virbr0 * 0.0.0.0/0
> 0.0.0.0/0 udp dpt:53
> >> 0 0
> >>
> >> ACCEPT tcp -- virbr0 *
0.0.0.0/0
> 0.0.0.0/0 tcp dpt:53
> >> 0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0
> 0.0.0.0/0 udp dpt:67
> >> 0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:67
> >>
> >> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>
>> pkts bytes target
> >> prot
> >> opt in out source destination
> >> 0 0 ACCEPT all -- * virbr1 0.0.0.0/0
> 192.168.121.0/24
> >> 0 0 ACCEPT all -- virbr1 *
> 192.168.121.0/24 0.0.0.0/0
> >>
0 0 ACCEPT all -- virbr1 virbr1 0.0.0.0/0
>
> >>
> >> 0.0.0.0/0
> >> 0 0 REJECT all -- * virbr1 0.0.0.0/0
> 0.0.0.0/0 reject-with icmp-port-unreachable
> >> 0 0 REJECT all -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 reject-with
icmp-port-unreachable
> >> 13 3448 ACCEPT all -- * virbr0 0.0.0.0/0
> 192.168.120.0/24
> >> state
> >> RELATED,ESTABLISHED
> >> 16 1374 ACCEPT all -- virbr0 *
> 192.168.120.0/24 0.0.0.0/0
> >> 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0
> 0.0.0.0/0
>
>> 0 0 REJECT all -- * virbr0 0.0.0.0/0
> 0.0.0.0/0 reject-with icmp-port-unreachable
> >> 0 0 REJECT all --
> >> virbr0
> >> * 0.0.0.0/0 0.0.0.0/0 reject-with
> icmp-port-unreachable
> >>
> >> Chain OUTPUT
(policy ACCEPT 233K packets, 27M bytes)
> >> pkts bytes target prot opt in out source
> destination
> >>
> >>
> >>
> >>
> >> And these are the various offload parameters as set at boot:
> >>
> >> Offload parameters for virbr0:
> >> rx-checksumming: on
> >> tx-checksumming: on
> >>
scatter-gather: on
> >> tcp-segmentation-offload: on
> >> udp-fragmentation-offload: on
> >> generic-segmentation-offload: on
> >> generic-receive-offload: off
> >> large-receive-offload: off
> >>
> >> Offload parameters for vif1.0:
> >> rx-checksumming: on
> >> tx-checksumming: on
> >>
scatter-gather: on
> >> tcp-segmentation-offload: on
> >> udp-fragmentation-offload: off
> >> generic-segmentation-offload: on
> >> generic-receive-offload: off
> >> large-receive-offload: off
> >>
> >> Offload parameters for eth0:
> >> rx-checksumming: on
> >> tx-checksumming: on
> >>
scatter-gather: on
> >> tcp-segmentation-offload: on
> >> udp-fragmentation-offload: off
> >> generic-segmentation-offload: off
> >> generic-receive-offload: off
> >> large-receive-offload: off
> >>
> >>
> >>
> >> To disable all checksuming I run the following commands:
> >> dom0:
> >>
> >>
sudo ethtool -K virbr0 tx off sg off tso off gso off gro off
> >> sudo ethtool -K vif1.0 tx off sg off tso off gso off gro off
> >>
> >>
> >> domU
> >>
> >> sudo ethtool -K eth0 tx off sg off tso off gso off gro off
> >>
> >>
> >>
> >> This managed to get all parameter to off in the mentioned
> >> interfaces, but unfortunately the result is the same. The arp
>
>> requests get to vif1.0, but not to eth0 on the domU.
> >>
> >> sudo tcpdump -i vif1.0 -n -vv arp
> >> tcpdump: WARNING: vif1.0: no IPv4 address assigned
> >> tcpdump: listening on vif1.0, link-type EN10MB (Ethernet),
> capture size 96 bytes
> >> 19:43:51.233378 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:52.233164 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell
192.168.120.254, length 28
> >> 19:43:53.233166 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:54.684214 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:55.684218 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:56.684232 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>
>
>>
> >>
> >> I hope this information is enough. If I can provide anything
> >> else to help debug or test, please just ask! ;)
> >>
> >> Thanks in advance,
> >> Luís
> >>
> >>> >
> >>> > Thanks,
> >>> > Luís
> >>> >
> >>>
> On Tue, 2010-06-01 at 18:20 -0700, Jeremy Fitzhardinge
> wrote:
> >>> >> On 06/01/2010 05:38 PM, Luís Silva wrote:
> >>> >> > Hello,
> >>> >> >
> >>> >> > Finally I managed to get a xen 4.0 working on ubuntu
> 10.04 with pvops
> >>> >> > kernel and libvirt. However I am having some problems
> with
> >>> >> > networking... after initial installation with
>
netinstall image in hvm
> >>> >> > mode, when I transform the vm in xen pv (via pygrub
> with the current
> >>> >> > ubuntu kernel), networking startEd to act weird...
> >>> >> >
> >>> >> > Basically I'm not using a network script from xen. I
> define a bridge
> >>> >> > (manually or via libvirt, the result is the same) and I
> use vif-bridge
> >>> >> > to connect the vif to it. But now
the weird part comes:
> I can
> >>> >> > communicate from domU to dom0, but not the other way
> >>> around,
> >>> unless I
> >>> >> > keep a ping running from domU to dom0... That's right,
> weird... while
> >>> >> > trying the ping from dom0 to domU, I used tcpdump both
> on the bridge,
> >>> >> > on the vif and on the eth0 in the domU. The arp packets
> never get to
>
>>> >> > domU, but they appear both in the bridge and the vif
> sniff's...
> >>> >>
> >>> >> What version of kernel are you using in dom0 and domU?
> There was a
> >>> >> netback bug which caused problems with dom0<->domU
> communication, but it
> >>> >> has been fixed for a while in 2.6.32 (but only recently
> in .31). The
> >>> >> workaround is to disable tx checksum offload on
your
> bridge (ethtool -K
> >>> >> <bridge> tx off).
> >>> >>
> >>> >> J
> >>> >>
> >>> >> >
> >>> >> > Here is the bridge:
> >>> >> > ifconfig virbr0
> >>> >> > virbr0 Link encap:Ethernet HWaddr
>
fe:ff:ff:ff:ff:ff
> >>> >> >
> >>>
> >>> inet addr:192.168.120.254 Bcast:192.168.120.255
> Mask:255.255.255.0
> >>> >> > inet6 addr: fe80::7cee:4bff:fe82:e63f/64
> Scope:Link
> >>> >> > UP BROADCAST RUNNING MULTICAST MTU:1500
> Metric:1
> >>> >> > RX packets:16 errors:0 dropped:0 overruns:0
>
frame:0
> >>> >> > TX packets:226 errors:0 dropped:0 overruns:0
> carrier:0
> >>> >> > collisions:0 txqueuelen:0
> >>> >> > RX bytes:952 (952.0 B) TX bytes:13953 (13.9
> KB)
> >>> >> >
> >>> >> >
> >>> >> > brctl show
> >>>
>> > bridge name bridge id STP enabled
> interfaces
> >>> >> > virbr0 8000.feffffffffff no vif5.0
> >>> >> >
> >>> >> >
> >>> >> > tcpdump -i virbr0 -vv -n
> >>> >> > tcpdump: listening on virbr0, link-type EN10MB
> (Ethernet), capture size 96 bytes
> >>> >> >
01:31:25.945151 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF],
> >>> proto ICMP (1),
> >>> length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 1, length 64
> >>> >> > 01:31:26.945361 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 2, length 64
>
>>> >> > 01:31:27.945420 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 3, length 64
> >>> >> > 01:31:28.945362 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 4, length 64
> >>> >> > 01:31:29.945364 IP
(tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317,
> >>> seq 5, length
> >>> 64
> >>> >> > 01:31:30.944300 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:30.945359 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
>
>>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 6, length 64
> >>> >> > 01:31:31.944297 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:31.945444 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 7, length 64
> >>> >> >
01:31:32.944294 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:32.945401 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> >
> >>>
> >>> 192.168.120.254 > 192.168.120.1: ICMP echo request, id
> 10317, seq 8, length 64
> >>> >> > 01:31:33.947293 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>>
>> > 01:31:34.947373 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:35.947353 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:37.948352 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:38.948399 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>>
>> > 01:31:39.948376 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:40.949356 ARP, Ethernet (len 6), IPv4 (len 4),
> Request
> >>> who-has
> >>> 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> >
> >>> >> >
> >>> >> > tcpdump -i vif5.0 -vv -n
> >>> >> > tcpdump:
WARNING: vif5.0: no IPv4 address assigned
> >>> >> > tcpdump: listening on vif5.0, link-type EN10MB
> (Ethernet), capture size 96 bytes
> >>> >> > 01:32:19.956358 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:20.956358 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:21.956359 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
>
>>> >> > 01:32:23.957311 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:24.957312 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length
> >>> 28
> >>> >> >
> >>> 01:32:25.957359 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:27.958360 ARP, Ethernet
(len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:28.958310 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:29.958362 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > Forwarding
and iptables don't seem to be the problem,
> because if I
> >>> >> > initiate a ping from domU (at the same time as the
> failing one from
> >>> >> > dom0), the ping in dom0 starts to work. As soon as I
> stop the ping in
> >>> >> > domU, the one in dom0 starts failing again...
> >>> >> >
> >>> >> > Is anyone having the same
> >>> problem? Is this a bug
> >>>
in the kernel? In
> >>> >> > dom0 or domU?
> >>> >> >
> >>> >> > Thanks in advance,
> >>> >> > Luís
> >>> >> >
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > Xen-devel mailing list
> >>>
>> > [12]
Xen-devel@xxxxxxxxxxxxxxxxxxx> <mailto:[13]
Xen-devel@xxxxxxxxxxxxxxxxxxx>
> <mailto:[14]
Xen-devel@xxxxxxxxxxxxxxxxxxx>
> >>> >> > [15]
http://lists.xensource.com/xen-devel> >>> >> >
> >>> >>
> >>>
>>
> >>> >> _______________________________________________
> >>> >> Xen-devel mailing list
> >>> >> [16]
Xen-devel@xxxxxxxxxxxxxxxxxxx> <mailto:[17]
Xen-devel@xxxxxxxxxxxxxxxxxxx>
> <mailto:[18]
Xen-devel@xxxxxxxxxxxxxxxxxxx>
> >>>
>> [19]
http://lists.xensource.com/xen-devel> >>> >>
> >>> >>
> >>> >
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> -----Inline Attachment Follows-----
> >>
> >> _______________________________________________
> >>
Xen-users mailing list
> >> [20]
Xen-users@xxxxxxxxxxxxxxxxxxx> >> [21]
http://lists.xensource.com/xen-users> >>
> >>
> >
> >
> > -----Inline Attachment Follows-----
> >
> > _______________________________________________
> > Xen-users mailing list
> > [22]
Xen-users@xxxxxxxxxxxxxxxxxxx> > </mc/compose?to=[23]
Xen-users@xxxxxxxxxxxxxxxxxxx>
> > [24]
http://lists.xensource.com/xen-users> >
> >
>
> References
>
> Visible links
> 1. file:///mc/compose?to=
luis.silva@xxxxxxxxxxxxx> 2. file:///mc/compose?to=
luis.silva@xxxxxxxxxxxxx> 3. file:///mc/compose?to=
bderzhavets@xxxxxxxxx> 4. file:///mc/compose?to=
jeremy@xxxxxxxx> 5. file:///mc/compose?to=
xen-devel@xxxxxxxxxxxxxxxxxxx> 6. file:///mc/compose?to=
xen-users@xxxxxxxxxxxxxxxxxxx> 7. file:///mc/compose?to=
luis.silva@xxxxxxxxxxxxx> 8. file:///mc/compose?to=
luis.silva@xxxxxxxxxxxxx> 9. file:///mc/compose?to=
jeremy@xxxxxxxx> 10. file:///mc/compose?to=
xen-devel@xxxxxxxxxxxxxxxxxxx> 11. file:///mc/compose?to=
xen-users@xxxxxxxxxxxxxxxxxxx> 12. file:///mc/compose?to=
Xen-devel@xxxxxxxxxxxxxxxxxxx> 13. file:///mc/compose?to=
Xen-devel@xxxxxxxxxxxxxxxxxxx> 14. file:///mc/compose?to=
Xen-devel@xxxxxxxxxxxxxxxxxxx> 15.
http://lists.xensource.com/xen-devel> 16. file:///mc/compose?to=
Xen-devel@xxxxxxxxxxxxxxxxxxx> 17. file:///mc/compose?to=
Xen-devel@xxxxxxxxxxxxxxxxxxx> 18. file:///mc/compose?to=
Xen-devel@xxxxxxxxxxxxxxxxxxx> 19.
http://lists.xensource.com/xen-devel> 20. file:///mc/compose?to=
Xen-users@xxxxxxxxxxxxxxxxxxx> 21.
http://lists.xensource.com/xen-users> 22. file:///mc/compose?to=
Xen-users@xxxxxxxxxxxxxxxxxxx> 23. file:///mc/compose?to=
Xen-users@xxxxxxxxxxxxxxxxxxx> 24.
http://lists.xensource.com/xen-users> _______________________________________________
> Xen-devel mailing list
>
Xen-devel@xxxxxxxxxxxxxxxxxxx>
http://lists.xensource.com/xen-devel_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxhttp://lists.xensource.com/xen-devel