Pardon me for a long detailed post and for pestering the developer list
but this looks pretty serious. The bottom line is that, when I attempt
to shut down or reboot my simple xen-2.0.7 server running FC3 with two
domUs, shutting down xendomains appears to send python into an infinite
loop. This may be due to a serious misconfiguration on my part but it
is certainly a big problem.
If we shutdown the domUs manually first, the system reboots perfectly
fine.
I'll describe some history and include lots of log information. The
configuration is slightly unusual. Both domUs are VPN gateways running
the 2.4.30 kernel with openswan and openvpn. Because of the openvpn,
but are privileged domains but the only device driver configured in the
kernel is the openvpn tun/tap adapter.
Since we have three interfaces, we slightly modified the network script.
It is included below.
We first noticed problems during shutdown in that the last domU to
shutdown crashed very nastily when closing openvpn. We could see this
even when manually stopping the domUs. We surmised that the problem was
we had tun0 and tun1 defined in both domains and killing them in one
must be killing them in another. We renamed those tun adapters in the
second domain to tun10 and tun11 and that problem went away.
We still noticed very long reboot times. The hang seemed to be at
killall but the "logjam" seemed to break after xendomains errored out.
We assumed this was because killall killed something that xendomains
needed to shutdown gracefully so we added explicit K scripts in rc0.d
and rc6.d for xend and xendomains.
At first this seemed to work but it must have been only coincidental
because the problem returned. We then realized we had brain cramped and
shutdown xend before xendomains. We thought for sure we had found the
problem and put them in the proper order but, alas, the problem did not
go away.
To gain a little more insight into exactly what was happening, we
manually ran each kill script in rc6.d. When we ran K78xendomains stop,
we saw the problem, or at least many aspect of it.
>From a xen console, we saw that the first domU had cleanly shutdown but
the second one was still listed howbeit with zero memory.
The hard disks were thrashing madly and top showed python consuming 99.9
percent CPU utilization. Most of the time, during reboot, we do not see
the madly thrashing disks. They do so for a moment as one might expect
when shutting down the domains but then just blip occasionally
thereafter.
Here is what we see in the logs:
>From messages (ntpd was the last service stopped before xendomains):
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Here is the complete xend-debug.log:
network start bridge=xen-br0 netdev=eth0 antispoof=no
VIRTUAL MEMORY ARRANGEMENT:
Loaded kernel: c0100000->c0319394
Init. ramdisk: c031a000->c031a000
Phys-Mach map: c031a000->c0332000
Page tables: c0332000->c0334000
Start info: c0334000->c0335000
Boot stack: c0335000->c0336000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif1.1 domain=niagras1 mac=aa:00:00:58:4a:a7
bridge=xen-br0
VIRTUAL MEMORY ARRANGEMENT:
Loaded kernel: c0100000->c0319394
Init. ramdisk: c031a000->c031a000
Phys-Mach map: c031a000->c0332000
Page tables: c0332000->c0334000
Start info: c0334000->c0335000
Boot stack: c0335000->c0336000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 44763, 'rx_shmem_frame':
44764, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 44767, 'rx_shmem_frame':
44768, 'handle': 1}
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 69339, 'rx_shmem_frame':
69340, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 69343, 'rx_shmem_frame':
69344, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif2.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif2.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
VirqClient.virqReceived> 4
vif-bridge down vif=vif1.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif1.1 domain=niagras1 mac=aa:00:00:26:24:c4
bridge=xen-br0
Traceback (most recent call last):
File "/usr/lib/python2.3/site-packages/twisted/protocols/basic.py",
line 229, in dataReceived
return self.rawDataReceived(data)
File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1015, in rawDataReceived
self.allContentReceived()
File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 1006, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python2.3/site-packages/twisted/protocols/http.py",
line 557, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
165, in process
self.render(resrc)
File "/usr/lib/python2.3/site-packages/twisted/web/server.py", line
172, in render
body = resrc.render(self)
File "/usr/lib/python2.3/site-packages/twisted/web/resource.py", line
201, in render
return m(request)
File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 180, in
render_POST
return self.perform(req)
File "/usr/lib/python/xen/xend/server/SrvBase.py", line 103, in
perform
return self._perform(op, op_method, req)
File "/usr/lib/python/xen/xend/server/SrvBase.py", line 109, in
_perform
return self._perform_err(err, op, req)
File "/usr/lib/python/xen/xend/server/SrvBase.py", line 107, in
_perform
val = op_method(op, req)
File "/usr/lib/python/xen/xend/server/SrvDomain.py", line 47, in
op_shutdown
val = fn(req.args, {'dom': self.dom.id})
File "/usr/lib/python/xen/xend/Args.py", line 145, in __call__
return self.call_with_form_args(self.fn, fargs, xargs=xargs)
File "/usr/lib/python/xen/xend/Args.py", line 121, in
call_with_form_args
return fn(*params, **keys)
File "/usr/lib/python/xen/xend/XendDomain.py", line 477, in
domain_shutdown
val = xend.domain_shutdown(dominfo.id, reason, key)
File "/usr/lib/python/xen/xend/server/SrvDaemon.py", line 723, in
domain_shutdown
ctrl = self.domainCF.getController(dom)
File "/usr/lib/python/xen/xend/server/controller.py", line 323, in
getController
ctrl = self.createController(dom)
File "/usr/lib/python/xen/xend/server/domain.py", line 20, in
createController
return DomainController(self, dom)
File "/usr/lib/python/xen/xend/server/domain.py", line 38, in __init__
self.registerChannel()
File "/usr/lib/python/xen/xend/server/controller.py", line 220, in
registerChannel
self.channel = self.channelFactory.domChannel(self.dom)
File "/usr/lib/python/xen/xend/server/channel.py", line 83, in
domChannel
remote_port=remote_port)
File "/usr/lib/python/xen/xend/server/channel.py", line 253, in
__init__
remote_port=remote_port)
File "/usr/lib/python/xen/xend/server/channel.py", line 137, in
createPort
remote_port=int(remote_port))
xen.lowlevel.xu.PortError: Failed to map domain control interface
/usr/lib/python2.3/site-packages/twisted/web/util.py:290: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
return '''<div class="instance"><span class="instanceName">%s instance
@ %s</span>
/usr/lib/python2.3/site-packages/twisted/web/util.py:266: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
w('<div class="dict"><span class="heading">Dictionary instance @ %
s</span>' % hex(id(d)))
/usr/lib/python2.3/site-packages/twisted/web/util.py:279: FutureWarning:
hex()/oct() of negative int will return a signed string in Python 2.4
and up
w('<div class="list"><span class="heading">List instance @ %s</span>'
% hex(id(l)))
Init. ramdisk: c031a000->c031a000
Phys-Mach map: c031a000->c0332000
Page tables: c0332000->c0334000
Start info: c0334000->c0335000
Boot stack: c0335000->c0336000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge up vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 88257, 'rx_shmem_frame':
88258, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 88261, 'rx_shmem_frame':
88262, 'handle': 1}
VIRTUAL MEMORY ARRANGEMENT:
Loaded kernel: c0100000->c0319394
Init. ramdisk: c031a000->c031a000
Phys-Mach map: c031a000->c0332000
Page tables: c0332000->c0334000
Start info: c0334000->c0335000
Boot stack: c0335000->c0336000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge up vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
Loaded kernel: c0100000->c02c9df4
Init. ramdisk: c02ca000->c02ca000
Phys-Mach map: c02ca000->c02e2000
Page tables: c02e2000->c02e4000
Start info: c02e4000->c02e5000
Boot stack: c02e5000->c02e6000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
Here is the information from xend.log:
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains: failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains: failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid boot.log.1 cron.2 mail messages prelink.log
samba spooler wtmp xfrd.log
anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs
secure spooler.1 xend-debug.log Xorg.0.log
anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1
secure.1 spooler.2 xend.log Xorg.0.log.old
boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2
secure.2 vbox xend.log.1 yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done
I xend-debug.log
Row 171 Col 1 11:02 Ctrl-K H for help
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
Loaded kernel: c0100000->c02c9df4
Init. ramdisk: c02ca000->c02ca000
Phys-Mach map: c02ca000->c02e2000
Page tables: c02e2000->c02e4000
Start info: c02e4000->c02e5000
Boot stack: c02e5000->c02e6000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid boot.log.1 cron.2 mail messages prelink.log
samba spooler wtmp xfrd.log
anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs
secure spooler.1 xend-debug.log Xorg.0.log
anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1
secure.1 spooler.2 xend.log Xorg.0.log.old
boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2
secure.2 vbox xend.log.1 yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done
I xend.log
Row 6079 Col 1 11:04 Ctrl-K H for help
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown ['vpn1dc1', '2', 'halt']
[2006-01-28 22:06:09 xend] INFO (XendRoot:113) EVENT>
xend.domain.shutdown ['niagras1', '1', 'halt']
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=niagras1 id=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=1 reason=poweroff
[2006-01-28 22:06:22 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 1 poweroff 0
[2006-01-28 22:06:22 xend] INFO (XendDomain:568) Destroying domain:
name=niagras1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=0
[2006-01-28 22:06:22 xend] DEBUG (netif:305) Destroying vif domain=1
vif=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 1
[2006-01-28 22:06:22 xend] DEBUG (blkif:552) Destroying blkif domain=1
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=0
[2006-01-28 22:06:22 xend] DEBUG (blkif:408) Destroying vbd domain=1
idx=1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 1
[2006-01-28 22:06:22 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 1
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1root
[2006-01-28 22:06:22 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/niagras1swap
[2006-01-28 22:06:22 xend] INFO (console:58) Console disconnected 15
127.0.0.1 3484
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['niagras1', '1', 'poweroff']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['niagras1', '1']
[2006-01-28 22:06:22 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [15, '127.0.0.1', 3484]
[2006-01-28 22:06:23 xend] INFO (XendRoot:113) EVENT> xend.domain.died
['niagras1', '1']
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:24 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:24 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:665) Destroying vifs
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=0
[2006-01-28 22:06:24 xend] DEBUG (netif:305) Destroying vif domain=2
vif=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:674) Destroying vbds
for domain 2
[2006-01-28 22:06:24 xend] DEBUG (blkif:552) Destroying blkif domain=2
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=0
[2006-01-28 22:06:24 xend] DEBUG (blkif:408) Destroying vbd domain=2
idx=1
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:24 xend] DEBUG (XendDomainInfo:622) Closing channel to
domain 2
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.virq 4
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1root
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains: failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains: failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid boot.log.1 cron.2 mail messages prelink.log
samba spooler wtmp xfrd.log
anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs
secure spooler.1 xend-debug.log Xorg.0.log
anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1
secure.1 spooler.2 xend.log Xorg.0.log.old
boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2
secure.2 vbox xend.log.1 yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done
I xend-debug.log
Row 171 Col 1 11:02 Ctrl-K H for help
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
Loaded kernel: c0100000->c02c9df4
Init. ramdisk: c02ca000->c02ca000
Phys-Mach map: c02ca000->c02e2000
Page tables: c02e2000->c02e4000
Start info: c02e4000->c02e5000
Boot stack: c02e5000->c02e6000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid boot.log.1 cron.2 mail messages prelink.log
samba spooler wtmp xfrd.log
anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs
secure spooler.1 xend-debug.log Xorg.0.log
anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1
secure.1 spooler.2 xend.log Xorg.0.log.old
boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2
secure.2 vbox xend.log.1 yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done
I xend.log
Row 6077 Col 91 11:06 Ctrl-K H for help
[2006-01-28 22:06:24 xend] DEBUG (blkif:363) Unbinding vbd (type phy)
from VG1/vpn1dc1swap
[2006-01-28 22:06:24 xend] INFO (console:58) Console disconnected 16
127.0.0.1 3986
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:24 xend] INFO (XendRoot:113) EVENT>
xend.console.disconnect [16, '127.0.0.1', 3986]
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:26 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:26 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:26 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:26 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:28 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:28 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:28 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:28 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:30 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:30 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:30 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:30 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:32 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:32 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:32 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:32 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:34 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:34 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:34 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:34 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
Jan 28 22:05:49 xenpub1dc1 ntpd[3652]: ntpd exiting on signal 15
Jan 28 22:05:49 xenpub1dc1 ntpd: ntpd shutdown succeeded
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.0 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 2(vif1.0) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: device vif1.1 left promiscuous mode
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:22 xenpub1dc1 kernel: xen-br0: port 3(vif1.1) entering
disabled state
Jan 28 22:06:23 xenpub1dc1 net.agent[4714]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4723]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4737]: remove event not handled
Jan 28 22:06:23 xenpub1dc1 net.agent[4746]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.0 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br0: port 4(vif2.0) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: device vif2.1 left promiscuous mode
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 kernel: xen-br1: port 2(vif2.1) entering
disabled state
Jan 28 22:06:24 xenpub1dc1 net.agent[4774]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4766]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4789]: remove event not handled
Jan 28 22:06:24 xenpub1dc1 net.agent[4798]: remove event not handled
Jan 28 22:18:03 xenpub1dc1 shutdown: shutting down for system reboot
Jan 28 22:18:03 xenpub1dc1 init: Switching to runlevel: 6
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3781]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3779]: session closed for
user root
Jan 28 22:18:03 xenpub1dc1 login(pam_unix)[3780]: session closed for
user root
Jan 28 22:18:04 xenpub1dc1 xendomains: Shutting down all Xen domains:
Jan 28 22:18:05 xenpub1dc1 xendomains: Error: Internal Server Error
Jan 28 22:18:05 xenpub1dc1 xendomains: failed
Jan 28 22:18:05 xenpub1dc1 xendomains:
Jan 28 22:18:05 xenpub1dc1 rc: Stopping xendomains: failed
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel logging (proc) stopped.
Jan 28 22:18:05 xenpub1dc1 kernel: Kernel log daemon terminating.
Jan 28 22:18:06 xenpub1dc1 syslog: klogd shutdown succeeded
Jan 28 22:18:06 xenpub1dc1 exiting on signal 15
Jan 28 22:20:06 xenpub1dc1 syslogd 1.4.1: restart.
Jan 28 22:20:06 xenpub1dc1 syslog: syslogd startup succeeded
Jan 28 22:20:06 xenpub1dc1 kernel: klogd 1.4.1, log source = /proc/kmsg
started.
Jan 28 22:20:06 xenpub1dc1 kernel: Linux version 2.6.11.12-xen0
(root@xxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #4 Thu Jan
File messages not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid boot.log.1 cron.2 mail messages prelink.log
samba spooler wtmp xfrd.log
anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs
secure spooler.1 xend-debug.log Xorg.0.log
anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1
secure.1 spooler.2 xend.log Xorg.0.log.old
boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2
secure.2 vbox xend.log.1 yum.log
[root@xenpub1dc1 log]# joe xend-debug.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done
I xend-debug.log
Row 171 Col 1 11:02 Ctrl-K H for help
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 185025, 'rx_shmem_frame':
185026, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 185029, 'rx_shmem_frame':
185030, 'handle': 1}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=niagras1 mac=02:00:00:01:02:00
bridge=xen-br0
vif-bridge down vif=vif8.1 domain=niagras1 mac=aa:00:00:53:7a:5b
bridge=xen-br0
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
vif-bridge down vif=vif7.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
-bridge down vif=vif4.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif4.1 does not exist!
VIRTUAL MEMORY ARRANGEMENT:
Loaded kernel: c0100000->c02c9df4
Init. ramdisk: c02ca000->c02ca000
Phys-Mach map: c02ca000->c02e2000
Page tables: c02e2000->c02e4000
Start info: c02e4000->c02e5000
Boot stack: c02e5000->c02e6000
TOTAL: c0000000->c0400000
ENTRY ADDRESS: c0100000
vif-bridge up vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge up vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
recv_fe_driver_status> {'status': 1}
recv_fe_driver_status>
recv_fe_interface_connect {'tx_shmem_frame': 202459, 'rx_shmem_frame':
202460, 'handle': 0}
recv_fe_interface_connect {'tx_shmem_frame': 202463, 'rx_shmem_frame':
202464, 'handle': 1}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=vpn1dc1 mac=02:00:00:01:01:00
bridge=xen-br0
interface vif5.0 does not exist!
vif-bridge down vif=vif5.1 domain=vpn1dc1 mac=02:00:00:01:01:01
bridge=xen-br1
interface vif5.1 does not exist!
File xend-debug.log not changed so no update needed.
[root@xenpub1dc1 log]# ls
acpid boot.log.1 cron.2 mail messages prelink.log
samba spooler wtmp xfrd.log
anaconda.log boot.log.2 cups maillog messages.1 rpmpkgs
secure spooler.1 xend-debug.log Xorg.0.log
anaconda.syslog cron dmesg maillog.1 messages.2 rpmpkgs.1
secure.1 spooler.2 xend.log Xorg.0.log.old
boot.log cron.1 lastlog maillog.2 ppp rpmpkgs.2
secure.2 vbox xend.log.1 yum.log
[root@xenpub1dc1 log]# joe xend.log
Processing '/etc/joe/joerc'...done
Processing '/etc/joe/joerc'...done
I xend.log
Row 6157 Col 100 11:07 Ctrl-K H for help
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:36 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:36 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:36 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:36 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:38 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:38 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:38 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:38 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:40 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:40 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:40 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:40 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:42 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:42 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:42 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:42 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:44 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:44 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:44 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:44 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:46 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:46 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:46 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:46 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:48 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:48 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:48 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:48 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:50 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:50 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:50 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:50 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:52 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:52 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:52 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:52 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:54 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:54 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:54 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:54 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:56 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:56 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
[2006-01-28 22:06:56 xend] DEBUG (XendDomainInfo:634) Closing console,
domain 2
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT> xend.domain.exit
['vpn1dc1', '2', 'poweroff']
[2006-01-28 22:06:56 xend] INFO (XendRoot:113) EVENT>
xend.domain.destroy ['vpn1dc1', '2']
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:244) XendDomain>reap>
domain died name=vpn1dc1 id=2
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:247) XendDomain>reap>
shutdown id=2 reason=poweroff
[2006-01-28 22:06:57 xend] DEBUG (XendDomain:487)
domain_restart_schedule> 2 poweroff 0
[2006-01-28 22:06:57 xend] INFO (XendDomain:568) Destroying domain:
name=vpn1dc1
etc., etc., etc.
Here are the configuration files for the two domUs:
# -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using
'xm create'.
# You use a separate script for each domain you want to create, or
# you can set the parameters for the domain on the xm command line.
#============================================================================
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"
# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"
# The domain build function. Default is 'linux'.
#builder='linux'
# Initial memory allocation (in megabytes) for the new domain.
memory = 96
# A name for your domain. All domains must have different names.
name = "niagras1"
# Which CPU to start domain on?
#cpu = -1 # leave to Xen to pick
#----------------------------------------------------------------------------
# Define network interfaces.
# Number of network interfaces. Default is 1.
nics=2
# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ 'mac=aa:00:00:00:00:11, bridge=xen-br0' ]
vif = ['mac=02:00:00:01:02:00, bridge=xen-br0' ]
#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.
#disk = [ 'phy:hda1,hda1,w' ]
disk = [ 'phy:VG1/niagras1root,sda1,w', 'phy:VG1/niagras1swap,sda2,w' ]
#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the domain's
# IP config doesn't, e.g. in ifcfg-eth0 or via DHCP.
# You can use 'extra' to set the runlevel and custom environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).
# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask=
# Set default gateway.
#gateway=
# Set the hostname.
#hostname= "vm%d" % vmid
# Set root device.
root = "/dev/sda1 ro"
# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = '169.254.1.0'
# Root directory on the nfs server.
#nfs_root = '/full/path/to/root/directory'
# Sets runlevel 4.
#extra = "4"
#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is 'onreboot', which restarts the domain when it shuts
down
# with exit code reboot.
# Other values are 'always', and 'never'.
#restart = 'onreboot'
#============================================================================
# -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using
'xm create'.
# You use a separate script for each domain you want to create, or
# you can set the parameters for the domain on the xm command line.
#============================================================================
#----------------------------------------------------------------------------
# Kernel image file.
kernel = "/boot/vmlinuz-2.4.30-xenUVPN"
# Optional ramdisk.
#ramdisk = "/boot/initrd.gz"
# The domain build function. Default is 'linux'.
#builder='linux'
# Initial memory allocation (in megabytes) for the new domain.
memory = 96
# A name for your domain. All domains must have different names.
name = "vpn1dc1"
# Which CPU to start domain on?
#cpu = -1 # leave to Xen to pick
#----------------------------------------------------------------------------
# Define network interfaces.
# Number of network interfaces. Default is 1.
nics=2
# Optionally define mac and/or bridge for the network interfaces.
# Random MACs are assigned if not given.
#vif = [ 'mac=aa:00:00:00:00:11, bridge=xen-br0' ]
vif = ['mac=02:00:00:01:01:00, bridge=xen-br0', 'mac=02:00:00:01:01:01,
bridge=xen-br1' ]
#----------------------------------------------------------------------------
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
# Each disk entry is of the form phy:UNAME,DEV,MODE
# where UNAME is the device, DEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write.
#disk = [ 'phy:hda1,hda1,w' ]
disk = [ 'phy:VG1/vpn1dc1root,sda1,w', 'phy:VG1/vpn1dc1swap,sda2,w' ]
#----------------------------------------------------------------------------
# Set the kernel command line for the new domain.
# You only need to define the IP parameters and hostname if the domain's
# IP config doesn't, e.g. in ifcfg-eth0 or via DHCP.
# You can use 'extra' to set the runlevel and custom environment
# variables used by custom rc scripts (e.g. VMID=, usr= ).
# Set if you want dhcp to allocate the IP address.
#dhcp="dhcp"
# Set netmask.
#netmask=
# Set default gateway.
#gateway=
# Set the hostname.
#hostname= "vm%d" % vmid
# Set root device.
root = "/dev/sda1 ro"
# Root device for nfs.
#root = "/dev/nfs"
# The nfs server.
#nfs_server = '169.254.1.0'
# Root directory on the nfs server.
#nfs_root = '/full/path/to/root/directory'
# Sets runlevel 4.
#extra = "4"
#----------------------------------------------------------------------------
# Set according to whether you want the domain restarted when it exits.
# The default is 'onreboot', which restarts the domain when it shuts
down
# with exit code reboot.
# Other values are 'always', and 'never'.
#restart = 'onreboot'
#============================================================================
Here is the network script:
# Exit if anything goes wrong.
set -e
# First arg is the operation.
OP=$1
shift
# Pull variables in args in to environment.
for arg ; do export "${arg}" ; done
bridge=${bridge:-xen-br0}
netdev=${netdev:-eth0}
antispoof=${antispoof:-yes}
SCRIPTDIR=/etc/xen/scripts
echo "network $OP bridge=$bridge netdev=$netdev antispoof=$antispoof"
# Usage: transfer_addrs src dst
# Copy all IP addresses (including aliases) from device $src to device $dst.
transfer_addrs () {
local src=$1
local dst=$2
# Don't bother if $dst already has IP addresses.
if ip addr show dev ${dst} | egrep -q '^ *inet ' ; then
return
fi
# Address lines start with 'inet' and have the device in them.
# Replace 'inet' with 'ip addr add' and change the device name $src
# to 'dev $src'. Remove netmask as we'll add routes later.
ip addr show dev ${src} | egrep '^ *inet ' | sed -e "
s/inet/ip addr add/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
s/${src}/dev ${dst}/
" | sh -e
}
# Usage: del_addrs src
del_addrs () {
local src=$1
ip addr show dev ${src} | egrep '^ *inet ' | sed -e "
s/inet/ip addr del/
s@\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\)/[0-9]\+@\1@
s/${src}/dev ${src}/
" | sh -e
}
# Usage: transfer_routes src dst
# Get all IP routes to device $src, delete them, and
# add the same routes to device $dst.
# The original routes have to be deleted, otherwise adding them
# for $dst fails (duplicate routes).
transfer_routes () {
local src=$1
local dst=$2
# List all routes and grep the ones with $src in.
# Stick 'ip route del' on the front to delete.
# Change $src to $dst and use 'ip route add' to add.
ip route list | grep ${src} | sed -e "
h
s/^/ip route del /
P
g
s/${src}/${dst}/
s/^/ip route add /
P
d
" | sh -e
}
# Usage: create_bridge bridge
create_bridge () {
local bridge=$1
# Don't create the bridge if it already exists.
if ! brctl show | grep -q ${bridge} ; then
brctl addbr ${bridge}
brctl stp ${bridge} off
brctl setfd ${bridge} 0
fi
ifconfig ${bridge} up
}
# Usage: add_to_bridge bridge dev
add_to_bridge () {
local bridge=$1
local dev=$2
# Don't add $dev to $bridge if it's already on a bridge.
if ! brctl show | grep -q ${dev} ; then
brctl addif ${bridge} ${dev}
fi
}
# Usage: antispoofing dev bridge
# Set the default forwarding policy for $dev to drop.
# Allow forwarding to the bridge.
antispoofing () {
local dev=$1
local bridge=$2
iptables -P FORWARD DROP
iptables -A FORWARD -m physdev --physdev-in ${dev} -j ACCEPT
}
# Usage: show_status dev bridge
# Print ifconfig and routes.
show_status () {
local dev=$1
local bridge=$2
echo '============================================================'
ifconfig ${dev}
ifconfig ${bridge}
echo ' '
ip route list
echo ' '
route -n
echo '============================================================'
}
op_start () {
if [ "${bridge}" == "null" ] ; then
return
fi
# Create the bridge and give it the interface IP addresses.
# Move the interface routes onto the bridge.
while read bridge netdev
do
create_bridge ${bridge}
if [ -n "$(ip add ls dev $netdev | grep 'inet ')" ]; then
transfer_addrs ${netdev} ${bridge}
transfer_routes ${netdev} ${bridge}
fi
# Don't add $dev to $bridge if it's already on a bridge.
if ! brctl show | grep -q ${netdev} ; then
brctl addif ${bridge} ${netdev}
fi
if [ ${antispoof} == 'yes' ] ; then
antispoofing ${netdev} ${bridge}
fi
done < ${SCRIPTDIR}/bridges.txt
}
op_stop () {
if [ "${bridge}" == "null" ] ; then
return
fi
# Remove the interface from the bridge.
# Move the routes back to the interface.
while read bridge netdev
do
brctl delif ${bridge} ${netdev}
if [ -n "$(ip add ls dev $netdev | grep 'inet ')" ]; then
transfer_routes ${bridge} ${netdev}
fi
done < ${SCRIPTDIR}/bridges.txt
# It's not our place to be enabling forwarding...
}
case ${OP} in
start)
op_start
;;
stop)
op_stop
;;
status)
show_status ${netdev} ${bridge}
;;
*)
echo 'Unknown command: ' ${OP}
echo 'Valid commands are: start, stop, status'
exit 1
esac
Here is the bridges.txt file:
xen-br0 eth0
xen-br1 eth1
xen-br2 eth2
I'm stumped but it sure looks serious. Have I produced some dreadful
misconfiguration? This host is being shipped to a remote data center on
Monday and goes into production in a week. Any help would be greatly
appreciated. Thanks - John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@xxxxxxxxxxxxxxxxxxx
Financially sustainable open source development
http://www.opensourcedevel.com
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home