[Xen-users] unable to open xenstore issue (SLES 10 SP2)

To:	xen-users@xxxxxxxxxxxxxxxxxxx
Subject:	[Xen-users] unable to open xenstore issue (SLES 10 SP2)
From:	AP Xen <apxeng@xxxxxxxxx>
Date:	Fri, 4 Jun 2010 10:09:08 -0700
Delivery-date:	Fri, 04 Jun 2010 10:10:39 -0700
Dkim-signature:	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=NrpJI5/ESGmg/koB79Q9GVk/Al80gfonlkgvchMuVRA=; b=eEsaSHfSsNNm9Caf7Hrl3vaSa9bl414sgL+po2PRTjO7TSyF2hyaktnuIOhB54oTpz cCIkzQwng896IIVHRjBeyrPowu/QgKGL0EuITAuJG6ss7O73nG9naK6RHgu0bmelKsjd 1AG2c+kscCtaqLTiYKYAcJ7T6AWA5lbbwzaqA=
Domainkey-signature:	a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=laGtKirMDqOYLt4+ALnUxVNbgWoKCPlgtmTCEYdj/IciNLyZFYkUOHcRUOXB/O0XAJ qbin+kuMHO509AwE/5iD9ootk+iuqL4Qpr3P4hco7TDMtDSTWyaVEoMTSao+OhiWZwUJ tzyjjLfzRnfbt/1tIY7RQDPJ3RQPxExCGiYbs=
Envelope-to:	www-data@xxxxxxxxxxxxxxxxxxx
List-help:	<mailto:xen-users-request@lists.xensource.com?subject=help>
List-id:	Xen user discussion <xen-users.lists.xensource.com>
List-post:	<mailto:xen-users@lists.xensource.com>
List-subscribe:	<http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe:	<http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Sender:	xen-users-bounces@xxxxxxxxxxxxxxxxxxx

Hi,

I am running SLES 10 SP2 with the following Xen packaged with it:
xen_major              : 3
xen_minor              : 2
xen_extra              : .0_16718_16-0.3

After a YEAR of uptime, my server is acting up.

xm commands are not working. xm list stays there for ever.
xm top returns
unable to open xenstore
:Connection refuse

xm dmesg shows the following output:
(XEN) Freed 100kB init memory.
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) instrlen.c:252:d303 Cannot read from address 77b36c4a (eip
77b36c4a, mode 8)
(XEN) sh error: sh_remove_write_access(): can't remove write access to
mfn 4d60a8: guest has 1 special-use mappings of it
(XEN) domain_crash called from common.c:1850
(XEN) Domain 1129 (vcpu#0) crashed on cpu#2:
(XEN) ----[ Xen-3.2.0_16718_16-0.3  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    2
(XEN) RIP:    0008:[<000000008fb5918f>]
(XEN) RFLAGS: 0000000000010286   CONTEXT: hvm
(XEN) rax: 00000000ffacc008   rbx: 0000000000000000   rcx: 0000000083a08200
(XEN) rdx: 0000000000000000   rsi: 000000008187eca0   rdi: 0000000000000001
(XEN) rbp: 000000008a0c0cf8   rsp: 000000008a0c0cdc   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008001003b   cr4: 00000000000006b9
(XEN) cr3: 000000001f3450a0   cr2: 000000009746300f
(XEN) ds: 0023   es: 0023   fs: 0030   gs: 0000   ss: 0010   cs: 0008
(XEN) printk: 164 messages suppressed.
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1202)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1311)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1311)
(XEN) grant_table.c:340:d0 Iomem mapping not permitted
ffffffffffffffff (domain 1311)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)
(XEN) instrlen.c:252:d1670 Cannot read from address 7fefd8215f4 (eip
7fefd8215f4, mode 8)

All my domains have crashed but I am not able to find info about the
nature of the crash other than for the above info.

The xend.log does not show errors. However, I see some errors in
xend-debug.log. For some of the domains I see the following:

Traceback (most recent call last):
 File "/usr/lib64/python2.4/site-packages/xen/web/httpserver.py", line
140, in process
   resource = self.getResource()
 File "/usr/lib64/python2.4/site-packages/xen/web/httpserver.py",
line 172, in getResource
   return self.getServer().getResource(self)
 File "/usr/lib64/python2.4/site-packages/xen/web/httpserver.py",
line 351, in getResource
   return self.root.getRequestResource(req)
 File "/usr/lib64/python2.4/site-packages/xen/web/resource.py", line
39, in getRequestResource
   return findResource(self, req)
 File "/usr/lib64/python2.4/site-packages/xen/web/resource.py", line
26, in findResource
   next = resource.getPathResource(pathElement, request)
 File "/usr/lib64/python2.4/site-packages/xen/web/resource.py", line
49, in getPathResource
   val = self.getChild(path, request)
 File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line
71, in getChild
   val = self.get(x)
 File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 52, in get
   return self.domain(x)
 File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 44, in domain
   dom = self.xd.domain_lookup(x)
 File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py",
line 525, in domain_lookup
   raise XendInvalidDomain(str(domid))
XendInvalidDomain: <Fault 3: '549'>

Not sure why the domain was flagged as invalid. Doing a create on it
again went through.

There also a ton of these messages:
Traceback (most recent call last):
 File "/usr/lib64/python2.4/SocketServer.py", line 463, in
process_request_thread
   self.finish_request(request, client_address)
 File "/usr/lib64/python2.4/SocketServer.py", line 254, in finish_request
   self.RequestHandlerClass(request, client_address, self)
 File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py",
line 105, in <lambda>
   (lambda x, y, z:
 File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py",
line 65, in __init__
   server)
 File "/usr/lib64/python2.4/SocketServer.py", line 521, in __init__
   self.handle()
 File "/usr/lib64/python2.4/BaseHTTPServer.py", line 316, in handle
   self.handle_one_request()
 File "/usr/lib64/python2.4/BaseHTTPServer.py", line 310, in handle_one_request
   method()
 File "/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py",
line 82, in do_POST
   self.send_response(200)
 File "/usr/lib64/python2.4/BaseHTTPServer.py", line 367, in send_response
   self.wfile.write("%s %d %s\r\n" %
 File "/usr/lib64/python2.4/socket.py", line 256, in write
   self.flush()
 File "/usr/lib64/python2.4/socket.py", line 243, in flush
   self._sock.sendall(buffer)
error: (32, 'Broken pipe')

At this point I was forced to reboot the server to get it back and
running. I am not seeing any of the errors as of yet but I am afraid
it will show up after heavy activity of creating and destroying VMs.
Is this is a known bug that has been fixed in upstream Xen?

Thanks,
Aravindh

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

WARNING - OLD ARCHIVES

xen-users

[Xen-users] unable to open xenstore issue (SLES 10 SP2)