WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

[Xen-users] Re: Problem with network for new domU's (Grant table problem

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] Re: Problem with network for new domU's (Grant table problem/limit?) (vif vif-247-0: 28 mapping shared-frames)
From: "PCextreme B.V. - Wido den Hollander" <wido@xxxxxxxxxxxx>
Date: Wed, 18 Mar 2009 22:19:33 +0100
Delivery-date: Wed, 18 Mar 2009 14:20:37 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <1237298399.8071.59.camel@xxxxxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
Organization: PCextreme B.V.
References: <1236880073.14881.3.camel@xxxxxxxxxxxxxxxxxxxxxxxx> <1237298399.8071.59.camel@xxxxxxxxxxxxxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
Hello,

I have been able to reproduce the bug and found out what goes wrong.

After creating 79 VM's with all 2 vbd (root disk and swap) and one vif i
ran into the problem again.

One error caught my attention (and is only mentioned once!):

[ 628.040958] No available IRQ to bind to: increase NR_DYNIRQS.

After some searching i found:
* https://bugzilla.redhat.com/show_bug.cgi?id=442736
*
http://www.nabble.com/Unable-to-start-more-than-103-VMs-td11420853.html

I downloaded the source of "linux-image-2.6.24-23-xen" and modified
"debian/binary-custom.d/xen/patchset/001-xen-base.patch" on line 87148

Here you should change NR_DYNIRQS from 256 to 1024.

When you have a VM with 3 "devices" (in my case 2x vbd, 1x vif) you
reserve 3 Dynamic IRQ's per domU.

79 * 3 = 237 IRQ's

The default limit of 256 is reached since the dom0 also reserves some
IRQ's.

You can find out how many IRQ's you are using with:

grep Dynamic-irq /proc/interrupts |wc -l

I have now been able to address over 256 IRQ's and start 90 VM's without
any troubles.

I think Ubuntu should raise the limit from 256 to 512 or 1024 OR make it
a option wich can be passed to the kernel on boot (like suggested in the
RedHat bugreport)

I hope this helps more sysadmins in the future!

-  
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@xxxxxxxxxxxx
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl


On Tue, 2009-03-17 at 14:59 +0100, PCextreme B.V. - Wido den Hollander
wrote:
> Hello,
> 
> The last few days i did some further research.
> 
> It seems i have run into some limit for the Xen grant tables (although,
> that was what people told me).
> 
> I tried to replicate the error so i set up a new server. Since the
> machine wich is having the troubles is a pretty heavy machine (64GB ram)
> i could not exactly replicate it.
> 
> So i set up a new server with 6GB ram, but with exactly the same
> versions of Xen and i was NOT able to reproduce the error.
> 
> I created 115 domU's (with 48MB of memory) and they all had network
> connectivity.
> 
> So i started reading more about the Grant tables and found out that also
> vbd's are mapped in these tables.
> 
> For the test i used this config for the domU:
> 
> *****************************************************************
> kernel = '/boot/vmlinuz-2.6.24-22-xen'
> ramdisk = '/boot/initrd.img-2.6.24-22-xen'
> memory = 512
> vcpus = 1
> 
> root = '/dev/xvda1 ro'
> 
> disk = [
>                   'phy:/dev/xen-domains-root/vps78-root,xvda1,w'
>                   'phy:/dev/xen-domains-swap/vps78-swap,xvda2,w'
>               ]
> 
> name = 'vps78'
> 
> vif = [ 'mac=00:16:3e:12:de:e4,vifname=vps78,bridge=vlanbr710' ]
> 
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
> 
> extra='xencons=tty1 rootflags=quota'
> *****************************************************************
> 
> This did NOT work, it resulted in the following message:
> 
> vif vif-279-0: 28 mapping shared-frames 768/769 port 8
> 
> After searching in drivers/xen/interface.c i found out that it was bad,
> it is a error with code 28.
> 
> As i ran out of options i remove the xvda2 line from my domU's config
> and i'll be damned, the network connectivity was back!
> 
> I created a new domU (#80) and this had the same troubles, no network.
> But when i removed the xvda2 line (for the swap) the network was OK.
> 
> So for the last test i added the swap after it was created:
> 
> root@vps-pool-01:~# xm block-attach
> vps78 /dev/xen-domains-swap/vps78-swap xvda2 w
> root@vps-pool-01:~# dmesg
> [2540234.810635] blkback: ring-ref 299, event-channel 8, protocol 1
> (x86_64-abi)
> [2540234.810813] vbd vbd-280-51714: 28 mapping ring-ref 299 port 8
> root@vps-pool-01:~#
> 
> As you can see i have the same kind of error, mapping the vbd fails.
> 
> Today i ordered a new server wich is exactly the same as the machine
> wich is running into this limit. So i will be able to reproduce a exact
> setup and hopefully reproduce the errors as well.
> 
> I hope to receive this server within one week.
> 
> But in the meantime i hope their is somebody who has any clues of what
> this could be.
> 
> root@vps-pool-01:~# xm info
> host                   : vps-pool-01.xen.pcextreme.nl
> release                : 2.6.24-22-xen
> version                : #1 SMP Mon Nov 24 21:35:54 UTC 2008
> machine                : x86_64
> nr_cpus                : 8
> nr_nodes               : 1
> cores_per_socket       : 4
> threads_per_core       : 1
> cpu_mhz                : 2500
> hw_caps                :
> bfebfbff:20100800:00000000:00000140:040ce3bd:00000000:00000001
> total_memory           : 64766
> free_memory            : 26797
> max_free_memory        : 26797
> max_para_memory        : 26793
> max_hvm_memory         : 26722
> node_to_cpu            : node0:0-7
> xen_major              : 3
> xen_minor              : 2
> xen_extra              : .1-rc1-pre
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64 
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> cc_compiler            : gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)
> cc_compile_by          : buildd
> cc_compile_domain      : buildd
> cc_compile_date        : Fri Apr 11 01:13:51 UTC 2008
> xend_config_format     : 4
> root@vps-pool-01:~#
> 
> Thank you in advance!
> -  
> Met vriendelijke groet,
> 
> Wido den Hollander
> Hoofd Systeembeheer / CSO
> Telefoon Support Nederland: 0900 9633 (45 cpm)
> Telefoon Support België: 0900 70312 (45 cpm)
> Telefoon Direct: (+31) (0)20 50 60 104
> Fax: +31 (0)20 50 60 111
> E-mail: support@xxxxxxxxxxxx
> Website: http://www.pcextreme.nl
> Kennisbank: http://support.pcextreme.nl/
> Netwerkstatus: http://nmc.pcextreme.nl
> 
> 
> On Thu, 2009-03-12 at 18:47 +0100, PCextreme B.V. - Wido den Hollander
> wrote:
> > Hello,
> > 
> > Today i ran into a problem with new domU's on my Xen 3.2 system.
> > 
> > On this server i have 78 domU's running, but when creating domU #79 the
> > network for this domU fails.
> > 
> > I allready submitted a bugreport with Ubuntu and please forgive me for
> > just posting a link:
> > https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/341846
> > 
> > Since this machine is already in production i can't simply reboot or
> > build a new kernel.
> > 
> > Could somebody give me a clue where to find the solution?
> > 
> > -  
> > Met vriendelijke groet,
> > 
> > Wido den Hollander
> > Hoofd Systeembeheer / CSO
> > Telefoon Support Nederland: 0900 9633 (45 cpm)
> > Telefoon Support België: 0900 70312 (45 cpm)
> > Telefoon Direct: (+31) (0)20 50 60 104
> > Fax: +31 (0)20 50 60 111
> > E-mail: support@xxxxxxxxxxxx
> > Website: http://www.pcextreme.nl
> > Kennisbank: http://support.pcextreme.nl/
> > Netwerkstatus: http://nmc.pcextreme.nl
> > 

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
<Prev in Thread] Current Thread [Next in Thread>