WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 		  
Home Products Support Community News
 
   
 

xen-users

[Top] [All Lists]
<prev [Date] next>
[Advanced]
 <prev [Thread] next>

[Xen-users] i can't run a xen - virtual machine on a glusterfs mount poi

from [Jordi Moles Blanco] [Permanent Link][Original]
To: Xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] i can't run a xen - virtual machine on a glusterfs mount point
From: Jordi Moles Blanco <jordi@xxxxxxxxx>
Date: Mon, 07 Jul 2008 18:54:21 +0200
Cc:
Delivery-date: Mon, 07 Jul 2008 09:55:05 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Reply-to: jordi@xxxxxxxxx
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.14 (X11/20080502) 
hi,
i've been using glusterfs for a while now in different scenarios and now i'm facing a really strange problem, or at least it seems so to me....
the thing is that i've got a glusterfs, with six nodes and several clients (xen-servers) mounting the shared system. The idea is to have a big shared storage for all the vm's i've got.
when i try to run a vm created from xen 3.2 which is stored on gluster, it won't run. 

it says:

Error: Device 2049 (vbd) could not be connected. Backend device not found.

and if you look at the xen log's file:

*****************
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) XendDomainInfo.create(['vm', ['name', 'xen_testdedicat1'], ['memory', '256'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.18-6-xen-amd64'], ['ramdisk', '/boot/initrd.img-2.6.18-6-xen-amd64'], ['root', '/dev/sda1 ro']]], ['device', ['vbd', ['uname', 'file:/mnt/gluster/domains/xen_testdedicat1/disk.img'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.1.164']]]]) [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) XendDomainInfo.constructDomain [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) Balloon: 262956 KiB free; need 2048; done. 
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) Adding Domain: 2
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) XendDomainInfo.initDomain: 2 256 [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) _initDomain:shadow_memory=0x0, memory_static_max=0x10000000, memory_static_min=0x0. [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) Balloon: 262948 KiB free; need 262144; done. [2008-07-07 18:51:31 4020] INFO (__init__:1072) buildDomain os=linux dom=2 vcpus=1 
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) domid          = 2
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) memsize        = 256
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) image = /boot/vmlinuz-2.6.18-6-xen-amd64 
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) store_evtchn   = 1
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) console_evtchn = 2
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) cmdline = root=/dev/sda1 ro [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) ramdisk = /boot/initrd.img-2.6.18-6-xen-amd64 
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) vcpus          = 1
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) features       =
[2008-07-07 18:51:31 4020] INFO (__init__:1072) createDevice: vbd : {'uuid': 'a4d3fb99-4d54-687e-1e5e-b3c82264547d', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'sda1', 'uname': 'file:/mnt/gluster/domains/xen_testdedicat1/disk.img', 'mode': 'w'} [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'virtual-device': '2049', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/2049'} to /local/domain/2/device/vbd/2049. [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'domain': 'xen_testdedicat1', 'frontend': '/local/domain/2/device/vbd/2049', 'uuid': 'a4d3fb99-4d54-687e-1e5e-b3c82264547d', 'dev': 'sda1', 'state': '1', 'params': '/mnt/gluster/domains/xen_testdedicat1/disk.img', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'file'} to /local/domain/0/backend/vbd/2/2049. [2008-07-07 18:51:31 4020] INFO (__init__:1072) createDevice: vif : {'ip': '192.168.1.164', 'mac': '00:16:3e:29:f1:ad', 'uuid': '2746c385-a8e6-e5e7-00bc-32fc7f3d6272'} [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'mac': '00:16:3e:29:f1:ad', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0. [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'domain': 'xen_testdedicat1', 'handle': '0', 'uuid': '2746c385-a8e6-e5e7-00bc-32fc7f3d6272', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '192.168.1.164', 'state': '1', 'frontend': '/local/domain/2/device/vif/0', 'mac': '00:16:3e:29:f1:ad', 'online': '1', 'frontend-id': '2'} to /local/domain/0/backend/vif/2/0. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': '34b70208-d172-6d75-344e-888579b73c2c', 'on_reboot': 'restart', 'start_time': '1215449492.96', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'image': "(linux (kernel /boot/vmlinuz-2.6.18-6-xen-amd64) (ramdisk /boot/initrd.img-2.6.18-6-xen-amd64) (args 'root=/dev/sda1 ro ') (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 18446744071562067968) (GUEST_VERSION 2.6) (PADDR_OFFSET 18446744071562067968) (GUEST_OS linux) (HYPERCALL_PAGE 18446744071564189696) (LOADER generic) (ENTRY 18446744071564165120) (XEN_VERSION xen-3.0)))", 'name': 'xen_testdedicat1'} [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Storing domain details: {'console/ring-ref': '2214313', 'image/entry': '18446744071564165120', 'console/port': '2', 'store/ring-ref': '2214314', 'image/loader': 'generic', 'vm': '/vm/34b70208-d172-6d75-344e-888579b73c2c', 'control/platform-feature-multiprocessor-suspend': '1', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '18446744071562067968', 'memory/target': '262144', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'console/limit': '1048576', 'image/paddr-offset': '18446744071562067968', 'image/hypercall-page': '18446744071564189696', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'xenconsoled', 'image/features/auto-translated-physmap': '1', 'name': 'xen_testdedicat1', 'domid': '2', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) DevController: writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/2/0'} to /local/domain/2/device/console/0. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) DevController: writing {'domain': 'xen_testdedicat1', 'protocol': 'vt100', 'uuid': '254941ec-2571-00da-95a9-a8d6fcce4414', 'frontend': '/local/domain/2/device/console/0', 'state': '1', 'location': '2', 'online': '1', 'frontend-id': '2'} to /local/domain/0/backend/console/2/0. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.handleShutdownWatch 
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for devices vif.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for 0.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status. 
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback 1.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for devices vbd.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for 2049.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vbd/2/2049/hotplug-status. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vbd/2/2049/hotplug-status. 
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback 2.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroy: domid=2 [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDomain(2) 
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Destroying device model
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Releasing devices
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Removing vif/0
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Removing vbd/2049
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2049 
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Removing console/0
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0 


*****************
nodes don't log any error or warning at all, and the xen-server which is trying to run the vm says this: 

************
2008-07-07 17:43:07 E [afr.c:2391:afr_writev_cbk] grup1: (path=/domains/xen_testdedicat1/disk.img child=espai2) op_ret=-1 op_errno=22 2008-07-07 17:43:07 E [fuse-bridge.c:1645:fuse_writev_cbk] glusterfs-fuse: 656: WRITE => -1 (22) 2008-07-07 17:43:07 E [afr.c:2699:afr_flush] grup1: afrfdp->fdstate[] is 0, returning ENOTCONN 2008-07-07 17:43:07 E [fuse-bridge.c:945:fuse_err_cbk] glusterfs-fuse: 657: (16) ERR => -1 (107) 
**************

i'm using this software versions

linux debian etch, kernel 2.6.18-6-xen-amd64
xen-hypervisor-3.2-1-amd64
fuse-2.7.3glfs10
glusterfs--mainline--2.5--patch-788
the thing is, however, that if i move the virtual machine create from xen 3.2 from the shared gluster mount point to a local path, it will run without any problem at all... is it that gluster is not fast enough to provide xen with the data needed to launch the vm? could i do anything to fix that? 

these are the specs files:

node side:

***********

volume espa
       type storage/posix
       option directory /mnt/compartit
end-volume

volume spai
       type performance/io-threads
       option thread-count 4
       option cache-size 32MB
       subvolumes espa
end-volume

volume espai
       type performance/write-behind
       option aggregate-size 1MB
       option flush-behind on
       subvolumes spai
end-volume

volume ultim
       type protocol/server
       subvolumes espai
       option transport-type tcp/server
       option auth.ip.espai.allow *
end-volume

***********

client/xen side:

***********

volume espai1
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.204
   option remote-subvolume espai
end-volume

volume namespace1
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.204
   option remote-subvolume nm
end-volume

volume espai2
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.206
   option remote-subvolume espai
end-volume

volume espai3
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.213
   option remote-subvolume espai
end-volume

volume espai4
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.161
   option remote-subvolume espai
end-volume

volume namespace2
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.161
   option remote-subvolume nm
end-volume

volume espai5
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.162
   option remote-subvolume espai
end-volume

volume espai6
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.163
   option remote-subvolume espai
end-volume

volume grup1
   type cluster/afr
   subvolumes espai1 espai2
end-volume

volume grup2
   type cluster/afr
   subvolumes espai3 espai4
end-volume

volume grup3
   type cluster/afr
   subvolumes espai5 espai6
end-volume

volume nm1
   type cluster/afr
   subvolumes namespace1 namespace2
end-volume

volume ultim
   type cluster/unify
   subvolumes grup1 grup2 grup3
   option scheduler rr
   option namespace nm1
end-volume

volume iot
 type performance/io-threads
 option thread-count 4
 option cache-size 32MB
 subvolumes ultim
end-volume

volume ioc
type performance/io-cache
option cache-size 64MB
option page-size 1MB
option force-revalidate-timeout 2
subvolumes iot
end-volume

***********
i've already tried for a while to play with the spec file... adding and removing write and read buffers... but the final result is the same... the vm won't run. 


any idea?

thank you.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
[More with this subject...]
<Prev in Thread] Current Thread [Next in Thread>
  • [Xen-users] i can't run a xen - virtual machine on a glusterfs mount point, Jordi Moles Blanco <=
Previous by Date:  Re: [Xen-users] PCI passthrough partial success, Christopher Isip
Next by Date:  [Xen-users] Xen 3.3 unstable on OpenSUSE 10.3 - initrd problem, Bill Morris
Previous by Thread:  [Xen-users] CPU frequency scaling allowed in xen?, piyush masrani
Next by Thread:  [Xen-users] Xen 3.3 unstable on OpenSUSE 10.3 - initrd problem, Bill Morris
Indexes:  [Date] [Thread] [Top] [All Lists]
 
   

Copyright , Citrix Systems Inc. All rights reserved. Legal and Privacy