On Sun, Jun 06, 2010 at 11:54:11AM -0700, Boris Derzhavets wrote:
> Virt-install hangs attempting to retrieve updates.img from Apache Mirror
> setup at Dom0
> with kernel 2.6.32.15
> It doesn't happen with 2.6.32.10 (12,14 final).
> Environment Xen 4.0 Dom0 on top Ubuntu 10.04 Server. Libvirt is 0.8.0 .
> Attempting to virt-install F13 PV DomU in vnc mode.
>
> I believe that builds are consistent.
> Runtime behavior is the same for .10, .14, .15. Seems to be communicating
> problem between DomU and Dom0 during HTTP download.
>
I'm seeing PV domU network issues aswell.. when running Xen 4.0.0 on Fedora 13.
I'll have to dig more into it..
-- Pasi
> Boris.
>
> --- On Sun, 6/6/10, Jeremy Fitzhardinge <jeremy@xxxxxxxx> wrote:
>
> From: Jeremy Fitzhardinge <jeremy@xxxxxxxx>
> Subject: Re: [Xen-users] Re: [Xen-devel] ARP problems with xen 4.0 with
> pvops kernel 2.6.32.15
> To: "Boris Derzhavets" <bderzhavets@xxxxxxxxx>
> Cc: luis.silva@xxxxxxxxxxxxx, xen-devel@xxxxxxxxxxxxxxxxxxx,
> xen-users@xxxxxxxxxxxxxxxxxxx
> Date: Sunday, June 6, 2010, 12:43 PM
>
> On 06/06/2010 03:19 AM, Boris Derzhavets wrote:
> > Network issues when working with DomUs in 2.6.32.14 and finally been
> > fixed,
> > seem to appear again in 2.6.32.15. Reverting to back to xen/stable -
> > 2.6.32.10
> > works as a fix again.
> >
>
> There are no substantial differences between 2.6.32.14 and .15. If
> there are any differences in behaviour between them, then I'd suspect
> some inconsistency from boot to boot, or in your kernel build process.
>
> J
>
> >
> > Boris
> >
> > --- On *Thu, 6/3/10, Luís Silva /<[1]luis.silva@xxxxxxxxxxxxx>/*
> wrote:
> >
> >
> > From: Luís Silva <[2]luis.silva@xxxxxxxxxxxxx>
> > Subject: Re: [Xen-users] Re: [Xen-devel] ARP problems with xen 4.0
> > with pvops kernel
> > To: "Boris Derzhavets" <[3]bderzhavets@xxxxxxxxx>
> > Cc: "Jeremy Fitzhardinge" <[4]jeremy@xxxxxxxx>,
> > [5]xen-devel@xxxxxxxxxxxxxxxxxxx, [6]xen-users@xxxxxxxxxxxxxxxxxxx
> > Date: Thursday, June 3, 2010, 6:20 AM
> >
> > Hello,
> >
> > Thanks for the suggestion, xen/stable works ok for me. Only
> > problem is that I have to disable offload do get dhcp to work on
> > domU, but the problem I described before doesn't exist in this
> > kernel. Later today I'm going to try a previous build I have based
> > on stable-2.6.32.x (2.6.32.13) to check if it already had this
> > problem or not and I'll post the results.
> >
> > Luís
> >
> > On Wed, 2010-06-02 at 12:26 -0700, Boris Derzhavets wrote:
> >> Could you,please, build and try 2.6.32.10 ( xen/stable) ?
> >>
> >> Boris.
> >>
> >> --- On *Wed, 6/2/10, Luís Silva
> **/<[7]luis.silva@xxxxxxxxxxxxx>/*
> >> wrote:
> >>
> >>
> >> From: Luís Silva <[8]luis.silva@xxxxxxxxxxxxx>
> >> Subject: [Xen-users] Re: [Xen-devel] ARP problems with xen
> >> 4.0 with pvops kernel
> >> To: "Jeremy Fitzhardinge" <[9]jeremy@xxxxxxxx>
> >> Cc: [10]xen-devel@xxxxxxxxxxxxxxxxxxx,
> [11]xen-users@xxxxxxxxxxxxxxxxxxx
> >> Date: Wednesday, June 2, 2010, 2:53 PM
> >>
> >> Hello,
> >>
> >> On Wed, 2010-06-02 at 09:06 -0700, Jeremy Fitzhardinge wrote:
> >>> On 06/02/2010 01:47 AM, Luís Silva wrote:
> >>> > Hello,
> >>> >
> >>> > I'm using the latest stable-2.6.32.x. I already tried
> "ethtool -K
> >>> > <bridge> tx off", but that didn't make any difference.
> Also, this only
> >>> > happen with pv, in hvm mode all works ok and the domU sees
> the arp
> >>> > messages...
> >>>
> >>> Yes, ARP is a new twist on network problems. I'm guessing
> you're using
> >>> hvm without stubdoms, which means that its networking
> originates from
> >>> qemu within dom0, whereas PV and HVM+stubdom comes via
> netback.
> >>>
> >>>
> >> Yes, when I mentioned hvm I was talking about hvm without
> >> stubdoms. I haven't tried those yet.
> >>> But aside from that, I'm stumped. Are you running any
> firewalls on
> >>> either side? Can you try disabling all the offloads (tx,
> rx, gso, tso)
> >>> on all the relevent interfaces (bridge, netback, within the
> guest) and
> >>> see if that changes anything?
> >>>
> >>> J
> >>>
> >>>
> >>
> >> Ok, this is the bridge interface:
> >>
> >> brctl show
> >> bridge name bridge id STP enabled interfaces
> >> virbr0 8000.feffffffffff no vif1.0
> >>
> >> ifconfig virbr0
> >> virbr0 Link encap:Ethernet HWaddr c2:ef:67:2b:a4:23
> >> inet addr:192.168.120.254 Bcast:192.168.120.255
> Mask:255.255.255.0
> >> inet6 addr: fe80::c0ef:67ff:fe2b:a423/64 Scope:Link
> >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >> TX packets:25 errors:0 dropped:0 overruns:0
> carrier:0
> >> collisions:0 txqueuelen:0
> >> RX bytes:0 (0.0
> >> B)
> >> TX bytes:4662 (4.6 KB)
> >>
> >>
> >>
> >> I'm not using firewall other than the rules defined by
> >> libvirt. DomU has no firewall and the rules in dom0 are only
> >> these (virbr0 is natted to the outside, virbr1 is routed. The
> >> result is the same in either one of them):
> >>
> >> sudo iptables -L -n -v
> >> Chain INPUT (policy ACCEPT 241K packets, 53M bytes)
> >> pkts bytes target prot opt in out source
> destination
> >> 0 0 ACCEPT udp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 udp dpt:53
> >> 0 0 ACCEPT tcp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:53
> >>
> >>
> >> 0 0 ACCEPT udp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 udp dpt:67
> >> 0 0 ACCEPT tcp -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:67
> >> 8 515 ACCEPT udp -- virbr0 * 0.0.0.0/0
> 0.0.0.0/0 udp dpt:53
> >> 0 0
> >>
> >> ACCEPT tcp -- virbr0 * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:53
> >> 0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0
> 0.0.0.0/0 udp dpt:67
> >> 0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0
> 0.0.0.0/0 tcp dpt:67
> >>
> >> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> >> pkts bytes target
> >> prot
> >> opt in out source destination
> >> 0 0 ACCEPT all -- * virbr1 0.0.0.0/0
> 192.168.121.0/24
> >> 0 0 ACCEPT all -- virbr1 *
> 192.168.121.0/24 0.0.0.0/0
> >> 0 0 ACCEPT all -- virbr1 virbr1 0.0.0.0/0
>
> >>
> >> 0.0.0.0/0
> >> 0 0 REJECT all -- * virbr1 0.0.0.0/0
> 0.0.0.0/0 reject-with icmp-port-unreachable
> >> 0 0 REJECT all -- virbr1 * 0.0.0.0/0
> 0.0.0.0/0 reject-with icmp-port-unreachable
> >> 13 3448 ACCEPT all -- * virbr0 0.0.0.0/0
> 192.168.120.0/24
> >> state
> >> RELATED,ESTABLISHED
> >> 16 1374 ACCEPT all -- virbr0 *
> 192.168.120.0/24 0.0.0.0/0
> >> 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0
> 0.0.0.0/0
> >> 0 0 REJECT all -- * virbr0 0.0.0.0/0
> 0.0.0.0/0 reject-with icmp-port-unreachable
> >> 0 0 REJECT all --
> >> virbr0
> >> * 0.0.0.0/0 0.0.0.0/0 reject-with
> icmp-port-unreachable
> >>
> >> Chain OUTPUT (policy ACCEPT 233K packets, 27M bytes)
> >> pkts bytes target prot opt in out source
> destination
> >>
> >>
> >>
> >>
> >> And these are the various offload parameters as set at boot:
> >>
> >> Offload parameters for virbr0:
> >> rx-checksumming: on
> >> tx-checksumming: on
> >> scatter-gather: on
> >> tcp-segmentation-offload: on
> >> udp-fragmentation-offload: on
> >> generic-segmentation-offload: on
> >> generic-receive-offload: off
> >> large-receive-offload: off
> >>
> >> Offload parameters for vif1.0:
> >> rx-checksumming: on
> >> tx-checksumming: on
> >> scatter-gather: on
> >> tcp-segmentation-offload: on
> >> udp-fragmentation-offload: off
> >> generic-segmentation-offload: on
> >> generic-receive-offload: off
> >> large-receive-offload: off
> >>
> >> Offload parameters for eth0:
> >> rx-checksumming: on
> >> tx-checksumming: on
> >> scatter-gather: on
> >> tcp-segmentation-offload: on
> >> udp-fragmentation-offload: off
> >> generic-segmentation-offload: off
> >> generic-receive-offload: off
> >> large-receive-offload: off
> >>
> >>
> >>
> >> To disable all checksuming I run the following commands:
> >> dom0:
> >>
> >> sudo ethtool -K virbr0 tx off sg off tso off gso off gro off
> >> sudo ethtool -K vif1.0 tx off sg off tso off gso off gro off
> >>
> >>
> >> domU
> >>
> >> sudo ethtool -K eth0 tx off sg off tso off gso off gro off
> >>
> >>
> >>
> >> This managed to get all parameter to off in the mentioned
> >> interfaces, but unfortunately the result is the same. The arp
> >> requests get to vif1.0, but not to eth0 on the domU.
> >>
> >> sudo tcpdump -i vif1.0 -n -vv arp
> >> tcpdump: WARNING: vif1.0: no IPv4 address assigned
> >> tcpdump: listening on vif1.0, link-type EN10MB (Ethernet),
> capture size 96 bytes
> >> 19:43:51.233378 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:52.233164 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:53.233166 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:54.684214 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:55.684218 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >> 19:43:56.684232 ARP, Ethernet (len 6), IPv4 (len 4), Request
> who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>
> >>
> >>
> >> I hope this information is enough. If I can provide anything
> >> else to help debug or test, please just ask! ;)
> >>
> >> Thanks in advance,
> >> Luís
> >>
> >>> >
> >>> > Thanks,
> >>> > Luís
> >>> >
> >>> > On Tue, 2010-06-01 at 18:20 -0700, Jeremy Fitzhardinge
> wrote:
> >>> >> On 06/01/2010 05:38 PM, Luís Silva wrote:
> >>> >> > Hello,
> >>> >> >
> >>> >> > Finally I managed to get a xen 4.0 working on ubuntu
> 10.04 with pvops
> >>> >> > kernel and libvirt. However I am having some problems
> with
> >>> >> > networking... after initial installation with
> netinstall image in hvm
> >>> >> > mode, when I transform the vm in xen pv (via pygrub
> with the current
> >>> >> > ubuntu kernel), networking startEd to act weird...
> >>> >> >
> >>> >> > Basically I'm not using a network script from xen. I
> define a bridge
> >>> >> > (manually or via libvirt, the result is the same) and I
> use vif-bridge
> >>> >> > to connect the vif to it. But now the weird part comes:
> I can
> >>> >> > communicate from domU to dom0, but not the other way
> >>> around,
> >>> unless I
> >>> >> > keep a ping running from domU to dom0... That's right,
> weird... while
> >>> >> > trying the ping from dom0 to domU, I used tcpdump both
> on the bridge,
> >>> >> > on the vif and on the eth0 in the domU. The arp packets
> never get to
> >>> >> > domU, but they appear both in the bridge and the vif
> sniff's...
> >>> >>
> >>> >> What version of kernel are you using in dom0 and domU?
> There was a
> >>> >> netback bug which caused problems with dom0<->domU
> communication, but it
> >>> >> has been fixed for a while in 2.6.32 (but only recently
> in .31). The
> >>> >> workaround is to disable tx checksum offload on your
> bridge (ethtool -K
> >>> >> <bridge> tx off).
> >>> >>
> >>> >> J
> >>> >>
> >>> >> >
> >>> >> > Here is the bridge:
> >>> >> > ifconfig virbr0
> >>> >> > virbr0 Link encap:Ethernet HWaddr
> fe:ff:ff:ff:ff:ff
> >>> >> >
> >>>
> >>> inet addr:192.168.120.254 Bcast:192.168.120.255
> Mask:255.255.255.0
> >>> >> > inet6 addr: fe80::7cee:4bff:fe82:e63f/64
> Scope:Link
> >>> >> > UP BROADCAST RUNNING MULTICAST MTU:1500
> Metric:1
> >>> >> > RX packets:16 errors:0 dropped:0 overruns:0
> frame:0
> >>> >> > TX packets:226 errors:0 dropped:0 overruns:0
> carrier:0
> >>> >> > collisions:0 txqueuelen:0
> >>> >> > RX bytes:952 (952.0 B) TX bytes:13953 (13.9
> KB)
> >>> >> >
> >>> >> >
> >>> >> > brctl show
> >>> >> > bridge name bridge id STP enabled
> interfaces
> >>> >> > virbr0 8000.feffffffffff no vif5.0
> >>> >> >
> >>> >> >
> >>> >> > tcpdump -i virbr0 -vv -n
> >>> >> > tcpdump: listening on virbr0, link-type EN10MB
> (Ethernet), capture size 96 bytes
> >>> >> > 01:31:25.945151 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF],
> >>> proto ICMP (1),
> >>> length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 1, length 64
> >>> >> > 01:31:26.945361 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 2, length 64
> >>> >> > 01:31:27.945420 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 3, length 64
> >>> >> > 01:31:28.945362 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 4, length 64
> >>> >> > 01:31:29.945364 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317,
> >>> seq 5, length
> >>> 64
> >>> >> > 01:31:30.944300 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:30.945359 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 6, length 64
> >>> >> > 01:31:31.944297 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:31.945444 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> > 192.168.120.254 > 192.168.120.1: ICMP echo request,
> id 10317, seq 7, length 64
> >>> >> > 01:31:32.944294 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:32.945401 IP (tos 0x0, ttl 64, id 0, offset 0,
> flags [DF], proto ICMP (1), length 84)
> >>> >> >
> >>>
> >>> 192.168.120.254 > 192.168.120.1: ICMP echo request, id
> 10317, seq 8, length 64
> >>> >> > 01:31:33.947293 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:34.947373 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:35.947353 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:37.948352 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:38.948399 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:39.948376 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:31:40.949356 ARP, Ethernet (len 6), IPv4 (len 4),
> Request
> >>> who-has
> >>> 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> >
> >>> >> >
> >>> >> > tcpdump -i vif5.0 -vv -n
> >>> >> > tcpdump: WARNING: vif5.0: no IPv4 address assigned
> >>> >> > tcpdump: listening on vif5.0, link-type EN10MB
> (Ethernet), capture size 96 bytes
> >>> >> > 01:32:19.956358 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:20.956358 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:21.956359 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:23.957311 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:24.957312 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length
> >>> 28
> >>> >> >
> >>> 01:32:25.957359 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:27.958360 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:28.958310 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> > 01:32:29.958362 ARP, Ethernet (len 6), IPv4 (len 4),
> Request who-has 192.168.120.1 tell 192.168.120.254, length 28
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > Forwarding and iptables don't seem to be the problem,
> because if I
> >>> >> > initiate a ping from domU (at the same time as the
> failing one from
> >>> >> > dom0), the ping in dom0 starts to work. As soon as I
> stop the ping in
> >>> >> > domU, the one in dom0 starts failing again...
> >>> >> >
> >>> >> > Is anyone having the same
> >>> problem? Is this a bug
> >>> in the kernel? In
> >>> >> > dom0 or domU?
> >>> >> >
> >>> >> > Thanks in advance,
> >>> >> > Luís
> >>> >> >
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > Xen-devel mailing list
> >>> >> > [12]Xen-devel@xxxxxxxxxxxxxxxxxxx
> <mailto:[13]Xen-devel@xxxxxxxxxxxxxxxxxxx>
> <mailto:[14]Xen-devel@xxxxxxxxxxxxxxxxxxx>
> >>> >> > [15]http://lists.xensource.com/xen-devel
> >>> >> >
> >>> >>
> >>> >>
> >>> >> _______________________________________________
> >>> >> Xen-devel mailing list
> >>> >> [16]Xen-devel@xxxxxxxxxxxxxxxxxxx
> <mailto:[17]Xen-devel@xxxxxxxxxxxxxxxxxxx>
> <mailto:[18]Xen-devel@xxxxxxxxxxxxxxxxxxx>
> >>> >> [19]http://lists.xensource.com/xen-devel
> >>> >>
> >>> >>
> >>> >
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> -----Inline Attachment Follows-----
> >>
> >> _______________________________________________
> >> Xen-users mailing list
> >> [20]Xen-users@xxxxxxxxxxxxxxxxxxx
> >> [21]http://lists.xensource.com/xen-users
> >>
> >>
> >
> >
> > -----Inline Attachment Follows-----
> >
> > _______________________________________________
> > Xen-users mailing list
> > [22]Xen-users@xxxxxxxxxxxxxxxxxxx
> > </mc/compose?to=[23]Xen-users@xxxxxxxxxxxxxxxxxxx>
> > [24]http://lists.xensource.com/xen-users
> >
> >
>
> References
>
> Visible links
> 1. file:///mc/compose?to=luis.silva@xxxxxxxxxxxxx
> 2. file:///mc/compose?to=luis.silva@xxxxxxxxxxxxx
> 3. file:///mc/compose?to=bderzhavets@xxxxxxxxx
> 4. file:///mc/compose?to=jeremy@xxxxxxxx
> 5. file:///mc/compose?to=xen-devel@xxxxxxxxxxxxxxxxxxx
> 6. file:///mc/compose?to=xen-users@xxxxxxxxxxxxxxxxxxx
> 7. file:///mc/compose?to=luis.silva@xxxxxxxxxxxxx
> 8. file:///mc/compose?to=luis.silva@xxxxxxxxxxxxx
> 9. file:///mc/compose?to=jeremy@xxxxxxxx
> 10. file:///mc/compose?to=xen-devel@xxxxxxxxxxxxxxxxxxx
> 11. file:///mc/compose?to=xen-users@xxxxxxxxxxxxxxxxxxx
> 12. file:///mc/compose?to=Xen-devel@xxxxxxxxxxxxxxxxxxx
> 13. file:///mc/compose?to=Xen-devel@xxxxxxxxxxxxxxxxxxx
> 14. file:///mc/compose?to=Xen-devel@xxxxxxxxxxxxxxxxxxx
> 15. http://lists.xensource.com/xen-devel
> 16. file:///mc/compose?to=Xen-devel@xxxxxxxxxxxxxxxxxxx
> 17. file:///mc/compose?to=Xen-devel@xxxxxxxxxxxxxxxxxxx
> 18. file:///mc/compose?to=Xen-devel@xxxxxxxxxxxxxxxxxxx
> 19. http://lists.xensource.com/xen-devel
> 20. file:///mc/compose?to=Xen-users@xxxxxxxxxxxxxxxxxxx
> 21. http://lists.xensource.com/xen-users
> 22. file:///mc/compose?to=Xen-users@xxxxxxxxxxxxxxxxxxx
> 23. file:///mc/compose?to=Xen-users@xxxxxxxxxxxxxxxxxxx
> 24. http://lists.xensource.com/xen-users
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home