so I am tenitively moving towards the xen 4/pv_ops kernel (2.6.32.18,
xen/stable-2.6.32.x as of yesterday).
I need a lot of guests, as our smallest hardware has 32GiB ram, and our median
guest has 256MiB ram.
we followed
http://lists.xensource.com/archives/html/xen-devel/2010-04/msg01537.html
and changed
NR_DYNAMIC_IRQS in arch/x86/include/asm/irq_vectors.h to 2048.
we changed the .config like so:
CONFIG_SPARSE_IRQ=n
then we applied the following patch to arch/x86/include/asm/irq_vectors.h:
--- ./linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h.old
2010-08-11 15:40:53.000000000 +0000
+++ ./linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h 2010-08-11
15:41:39.000000000 +0000
@@ -167,7 +167,7 @@
# if defined(CONFIG_X86_IO_APIC) && defined(CONFIG_SPARSE_IRQ)
extern int nr_dynamic_irqs;
# else
-# define NR_DYNAMIC_IRQS 256
+# define NR_DYNAMIC_IRQS 2048
# endif
#endif
and we now can get to 338 guests (with three disks each) on our
64GiB 16 core test server. the 339th guest appears to start, but
instead sits there on a blank console.
338 guests is enough for me, as my 64GiB boxes are reserved for 1GiB
guests and up, and even on the 512MiB and smaller guests, the 32GiB
servers rarely exceed 100 guests.
Now, if you start more guests, (we started 479) while xl works, xm hangs
forever. the domains we start after 338 are unpingable. Xend can not
restart. xenconsoled keeps dying. We kill guests down to 300, restart
xend and xenconsoled, and everything appears to be happy once again.
Anyhow, uh, like I said, it appears to work just fine for 300 guests
and below, which is enough for me. I'm guessing that much above 300
guests (338, I believe, is the tipping point when you use my setup)
we just run out of interupts.
[root@chessboard ~]# xl list |wc -l
300
[root@chessboard ~]# wc -l /proc/interrupts
1830 /proc/interrupts
so that all looks right to me. Now, it'd be a nice safety feature if
xendomains didn't hoark when it went over the limit, as I need that
to keep the existing domains running, but it looks like I've got enough
headroom here that it's not a big deal.
--
Luke S. Crawford
http://prgmr.com/xen/ - Hosting for the technically adept
http://nostarch.com/xen.htm - We don't assume you are stupid.
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home