Hello,
my apologies for not being sure whom to tell this problem, but it is very
strange.
Let me tell the story:
I'm using XEN (3.0.2) with SLES10 (x86_64, SunFire X4100). On one machine I
have
three virtual machines ("DomU") that are very identically configured (SLES10
x86_64 also). There is also a SLES9 (i386) acting as a multi-homed NFS server.
I can mount and access a read-only NFS filesystem on the server from Dom0
(hypervisor), and from two of the three DomUs without any problem, but from the
third DomU mount hangs and is unkillable (except kill -9). This is how I
started
to debug the problem.
To make things short: I haven't found the solution, but a some problems:
Running
tcpdump/etheral on the client (inside DomU), and on the NFS server, I found out
that a significant number (almost all) of RPC reply packets have an invalid TCP
cjhecksum on the NFS server, but not on the NFS client. When actually comparing
the packets, I found that only the checksum is different. Example:
--- nfs-client9.txt 2006-11-29 12:56:59.176133729 +0100
+++ nfs-server8.txt 2006-11-29 12:56:25.812623453 +0100
@@ -1,10 +1,10 @@
No. Time Source Destination Protocol
Info
- 9 15:10:15.488963 132.199.176.153 132.199.177.13 Portmap
V2
DUMP Reply (Call In 7)[Unreassembled Packet]
+ 8 15:10:15.497059 132.199.176.153 132.199.177.13 Portmap
V2
DUMP Reply (Call In 6)[Unreassembled Packet [incorrect TCP checksum]]
0000 00 16 3e f3 45 0d 00 c0 9f 27 44 a6 08 00 45 00 ..>.E....'D...E.
0010 01 c4 d0 3f 40 00 40 06 fd be 84 c7 b0 99 84 c7 ...?@.@.........
0020 b1 0d 00 6f 94 48 89 33 9b af 3f e4 5a 65 80 18 ...o.H.3..?.Ze..
-0030 16 a0 27 8a 00 00 01 01 08 0a 5a a1 4b 92 01 2f ..'.......Z.K../
+0030 16 a0 6c ec 00 00 01 01 08 0a 5a a1 4b 92 01 2f ..l.......Z.K../
0040 a9 da 00 00 01 8c 65 e9 c4 df 00 00 00 01 00 00 ......e.........
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 01 00 01 86 a0 00 00 00 02 00 00 00 06 00 00 ................
I' NOT saysing that _all_ TCP checksums are bad, but significantly those RPC
reply
packets seem to be affected. OK so far.
I don't know why the NFS mount is actually hanging, but the last packet
exchange
seems to be:
Server sends ACK to RPC reply with bad checksum:
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: 1023 (1023),
Seq:
2306928188, Ack: 1069064470, Len: 0
Client receives:
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: 1023 (1023),
Seq:
2306928188, Ack: 1069064470, Len: 0
Some time later I see the client sending an NFS GETATTR packet, but that's
probably when the kill came in (31 seconds later).
The other odd thing is that the multihomes NFS server has a route to
132.199.0.0
for both ethernet interfaces, but only one of those, eth0, has IP
132.199.176.153.
However when an ARP request is sent for 132.199.176.153, there are two ansers:
ARP 132.199.176.153 is at 00:c0:9f:27:44:a6
ARP 132.199.176.153 is at 00:02:b3:d9:91:a7
Only the first one is correct. However that problem may be unrelated.
Back to the issue, I doubt that XEN will just overwrite the TCP checksum of
some
specific RPC packets. Personally I could image is much more that there is some
problem in the RPC processing that might cause this. Sorry for the poor problem
description.
Just in case, the Novell/SUSE kernel versions are:
Client: 2.6.16.21-0.25-xen
Server: 2.6.5-7.282-default
Upon request I could provide the packet files as well as two PDFs that show the
packet flow.
Regards,
Ulrich
P.S: I'm subscribed to xen-users, but not to the kernel list, so maybe CC: me
for
kernel-list replies. Thanks!
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
Home