On Tue, Feb 08, 2011 at 01:39:06PM +0100, Pim van Riezen wrote:
> Addendum:
>
> The Dells are actually R715.
> The dom0 kernel is actually vmlinuz-2.6.18-194.32.1.el5xen
>
Have you gived dom0 fixed amount of memory, and also increase dom0 vcpu weights
so that dom0 will always get enough cpu time to take care of things?
http://wiki.xensource.com/xenwiki/XenBestPractices
-- Pasi
> Cheers,
> Pim
>
> On Feb 8, 2011, at 13:22 , Pim van Riezen wrote:
>
> > Good day,
> >
> > In a scenario where we saw several dom0 nodes fall down due to a sustained
> > SYN flood to a network range, we have been investigating issues with Xen
> > under high network load. The results so far seem to be not so pretty. We
> > recreated a lab setup that can reproduce the scenario with some
> > reliability, although it takes a bit of trial-and-error to get crashes out
> > of it.
> >
> > SETUP:
> > 2x Dell R710
> > - 4x 6core AMD Opteron 6174
> > - 128GB memory
> > - Broadcom BCM5709
> > - LSI SAS2008 rev.02
> > - Emulex Saturn-X FC adapter
> > - CentOS 5.5 w/ gitco Xen 4.0.1
> >
> > 1x NexSan SATABeast FC raid
> > 1x Brocade FC switch
> > 5x Flood sources (Dell R210)
> >
> > The dom0 machines are loaded with 50 PV images, coupled to a LVM partition
> > on FC, half of which are set to start compiling a kernel in rc.local. There
> > are also 2 HVM images on both machines doing the same.
> >
> > Networking for all guests is configured in the bridging setup, attached to
> > a specific vlan that arrives tagged at the Dom0. So vifs end up in xenbr86,
> > née xenbr0.86.
> >
> > Grub conf for the dom0s:
> >
> > kernel /xen.gz-4.0.1 dom0_mem=2048M max_cstate=0 cpuidle=off
> > module /vmlinuz-2.6.18-194.11.4.el5xen ro root=LABEL=/ elevator=deadline
> > xencons=tty
> >
> > The flooding is always done to either the entire IP range the guests live
> > in (in case of SYN floods) or a sub-range of about 50 IPs (in case of UDP
> > floods), with random source addresses.
> >
> > ISSUE:
> > When the pps rate gets into the insane territory (gigabit link saturated or
> > near-saturated), the machine seems to start losing track of interrupts.
> > Depending on the severity, this leads to CPU soft lockups on random cores.
> > Under more dire circumstances, other hardware attached to the PCI bus
> > starts timing out making the kernel lose track of storage. Usually the
> > SAS-controller is the first to go, but I've also seen timeouts on the FC
> > controller.
> >
> > THINGS TRIED:
> > 1. Raising the broadcom RX ring from 255 to 3000. No noticable effects.
> > 2. Downgrading to Xen 3.4.3. No effect.
> > 3. Different Dell BIOS versions. No effect.
> > 4. Lowering number of guests -> effects get less serious. Not a serious
> > option.
> > 5. Using ipt_LIMIT in the FORWARD table set to 10000/s -> effects get less
> > serious when dealing with tcp SYN attacks. No effect when dealing with
> > 28byte UDP attacks.
> > 6. Disabling HPET as per
> > http://lists.xensource.com/archives/html/xen-devel/2010-09/msg00556.html
> > with cpuidle=0 and disabling irqbalance -> effects get less serious.
> >
> > The changes in 6 stop the machine from completely crapping itself, but I
> > still get soft lockups, although they seem to be limited to one of these
> > two paths:
> >
> > [<ffffffff8023f830>] invalidate_bh_lru+0x0/0x42
> > [<ffffffff8023f830>] invalidate_bh_lru+0x0/0x42
> > [<ffffffff8027458e>] smp_call_function_many+0x38/0x4c
> > [<ffffffff8023f830>] invalidate_bh_lru+0x0/0x42
> > [<ffffffff80274688>] smp_call_function+0x4e/0x5e
> > [<ffffffff8023f830>] invalidate_bh_lru+0x0/0x42
> > [<ffffffff8028fdd7>] on_each_cpu+0x10/0x2a
> > [<ffffffff802d7428>] kill_bdev+0x1b/0x30
> > [<ffffffff802d7a47>] __blkdev_put+0x4f/0x169
> > [<ffffffff80213492>] __fput+0xd3/0x1bd
> > [<ffffffff802243cb>] filp_close+0x5c/0x64
> > [<ffffffff8021e5d0>] sys_close+0x88/0xbd
> > [<ffffffff802602f9>] tracesys+0xab/0xb6
> >
> > and
> >
> > [<ffffffff8026f4f3>] raw_safe_halt+0x84/0xa8
> > [<ffffffff8026ca88>] xen_idle+0x38/0x4a
> > [<ffffffff8024af6c>] cpu_idle+0x97/0xba
> > [<ffffffff8064eb0f>] start_kernel+0x21f/0x224
> > [<ffffffff8064e1e5>] _sinittext+0x1e5/0x1eb
> >
> > In some scenarios, an application running on the dom0 that relies on
> > pthread_cond_timedwait seems to be hanging in all its thread on that
> > specific call. This may be related to some timing going wonky during the
> > attack, not sure.
> >
> > Is there anything more we can try?
> >
> > Cheers,
> > Pim van Riezen
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home