Hello,
I have been able to reproduce the bug and found out what goes wrong.
After creating 79 VM's with all 2 vbd (root disk and swap) and one vif i
ran into the problem again.
One error caught my attention (and is only mentioned once!):
[ 628.040958] No available IRQ to bind to: increase NR_DYNIRQS.
After some searching i found:
* https://bugzilla.redhat.com/show_bug.cgi?id=442736
*
http://www.nabble.com/Unable-to-start-more-than-103-VMs-td11420853.html
I downloaded the source of "linux-image-2.6.24-23-xen" and modified
"debian/binary-custom.d/xen/patchset/001-xen-base.patch" on line 87148
Here you should change NR_DYNIRQS from 256 to 1024.
When you have a VM with 3 "devices" (in my case 2x vbd, 1x vif) you
reserve 3 Dynamic IRQ's per domU.
79 * 3 = 237 IRQ's
The default limit of 256 is reached since the dom0 also reserves some
IRQ's.
You can find out how many IRQ's you are using with:
grep Dynamic-irq /proc/interrupts |wc -l
I have now been able to address over 256 IRQ's and start 90 VM's without
any troubles.
I think Ubuntu should raise the limit from 256 to 512 or 1024 OR make it
a option wich can be passed to the kernel on boot (like suggested in the
RedHat bugreport)
I hope this helps more sysadmins in the future!
-
Met vriendelijke groet,
Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@xxxxxxxxxxxx
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl
On Tue, 2009-03-17 at 14:59 +0100, PCextreme B.V. - Wido den Hollander
wrote:
> Hello,
>
> The last few days i did some further research.
>
> It seems i have run into some limit for the Xen grant tables (although,
> that was what people told me).
>
> I tried to replicate the error so i set up a new server. Since the
> machine wich is having the troubles is a pretty heavy machine (64GB ram)
> i could not exactly replicate it.
>
> So i set up a new server with 6GB ram, but with exactly the same
> versions of Xen and i was NOT able to reproduce the error.
>
> I created 115 domU's (with 48MB of memory) and they all had network
> connectivity.
>
> So i started reading more about the Grant tables and found out that also
> vbd's are mapped in these tables.
>
> For the test i used this config for the domU:
>
> *****************************************************************
> kernel = '/boot/vmlinuz-2.6.24-22-xen'
> ramdisk = '/boot/initrd.img-2.6.24-22-xen'
> memory = 512
> vcpus = 1
>
> root = '/dev/xvda1 ro'
>
> disk = [
> 'phy:/dev/xen-domains-root/vps78-root,xvda1,w'
> 'phy:/dev/xen-domains-swap/vps78-swap,xvda2,w'
> ]
>
> name = 'vps78'
>
> vif = [ 'mac=00:16:3e:12:de:e4,vifname=vps78,bridge=vlanbr710' ]
>
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
>
> extra='xencons=tty1 rootflags=quota'
> *****************************************************************
>
> This did NOT work, it resulted in the following message:
>
> vif vif-279-0: 28 mapping shared-frames 768/769 port 8
>
> After searching in drivers/xen/interface.c i found out that it was bad,
> it is a error with code 28.
>
> As i ran out of options i remove the xvda2 line from my domU's config
> and i'll be damned, the network connectivity was back!
>
> I created a new domU (#80) and this had the same troubles, no network.
> But when i removed the xvda2 line (for the swap) the network was OK.
>
> So for the last test i added the swap after it was created:
>
> root@vps-pool-01:~# xm block-attach
> vps78 /dev/xen-domains-swap/vps78-swap xvda2 w
> root@vps-pool-01:~# dmesg
> [2540234.810635] blkback: ring-ref 299, event-channel 8, protocol 1
> (x86_64-abi)
> [2540234.810813] vbd vbd-280-51714: 28 mapping ring-ref 299 port 8
> root@vps-pool-01:~#
>
> As you can see i have the same kind of error, mapping the vbd fails.
>
> Today i ordered a new server wich is exactly the same as the machine
> wich is running into this limit. So i will be able to reproduce a exact
> setup and hopefully reproduce the errors as well.
>
> I hope to receive this server within one week.
>
> But in the meantime i hope their is somebody who has any clues of what
> this could be.
>
> root@vps-pool-01:~# xm info
> host : vps-pool-01.xen.pcextreme.nl
> release : 2.6.24-22-xen
> version : #1 SMP Mon Nov 24 21:35:54 UTC 2008
> machine : x86_64
> nr_cpus : 8
> nr_nodes : 1
> cores_per_socket : 4
> threads_per_core : 1
> cpu_mhz : 2500
> hw_caps :
> bfebfbff:20100800:00000000:00000140:040ce3bd:00000000:00000001
> total_memory : 64766
> free_memory : 26797
> max_free_memory : 26797
> max_para_memory : 26793
> max_hvm_memory : 26722
> node_to_cpu : node0:0-7
> xen_major : 3
> xen_minor : 2
> xen_extra : .1-rc1-pre
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xffff800000000000
> xen_changeset : unavailable
> cc_compiler : gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)
> cc_compile_by : buildd
> cc_compile_domain : buildd
> cc_compile_date : Fri Apr 11 01:13:51 UTC 2008
> xend_config_format : 4
> root@vps-pool-01:~#
>
> Thank you in advance!
> -
> Met vriendelijke groet,
>
> Wido den Hollander
> Hoofd Systeembeheer / CSO
> Telefoon Support Nederland: 0900 9633 (45 cpm)
> Telefoon Support België: 0900 70312 (45 cpm)
> Telefoon Direct: (+31) (0)20 50 60 104
> Fax: +31 (0)20 50 60 111
> E-mail: support@xxxxxxxxxxxx
> Website: http://www.pcextreme.nl
> Kennisbank: http://support.pcextreme.nl/
> Netwerkstatus: http://nmc.pcextreme.nl
>
>
> On Thu, 2009-03-12 at 18:47 +0100, PCextreme B.V. - Wido den Hollander
> wrote:
> > Hello,
> >
> > Today i ran into a problem with new domU's on my Xen 3.2 system.
> >
> > On this server i have 78 domU's running, but when creating domU #79 the
> > network for this domU fails.
> >
> > I allready submitted a bugreport with Ubuntu and please forgive me for
> > just posting a link:
> > https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/341846
> >
> > Since this machine is already in production i can't simply reboot or
> > build a new kernel.
> >
> > Could somebody give me a clue where to find the solution?
> >
> > -
> > Met vriendelijke groet,
> >
> > Wido den Hollander
> > Hoofd Systeembeheer / CSO
> > Telefoon Support Nederland: 0900 9633 (45 cpm)
> > Telefoon Support België: 0900 70312 (45 cpm)
> > Telefoon Direct: (+31) (0)20 50 60 104
> > Fax: +31 (0)20 50 60 111
> > E-mail: support@xxxxxxxxxxxx
> > Website: http://www.pcextreme.nl
> > Kennisbank: http://support.pcextreme.nl/
> > Netwerkstatus: http://nmc.pcextreme.nl
> >
signature.asc
Description: This is a digitally signed message part
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
Home