WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-users

Re: [Xen-users] Solving the DRBD resync issue

To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Xen-users] Solving the DRBD resync issue
From: Daniel Meszaros <meszi@xxxxxxxxxxx>
Date: Fri, 03 Jun 2011 21:06:06 +0200
Delivery-date: Fri, 03 Jun 2011 12:07:53 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <BANLkTinRDdJC7aifehKHJchXLFbgmAtWyw@xxxxxxxxxxxxxx>
List-help: <mailto:xen-users-request@lists.xensource.com?subject=help>
List-id: Xen user discussion <xen-users.lists.xensource.com>
List-post: <mailto:xen-users@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-users>, <mailto:xen-users-request@lists.xensource.com?subject=unsubscribe>
References: <4DD6B842.1040701@xxxxxxxx> <AEC6C66638C05B468B556EA548C1A77D01D57281@trantor> <4DD7AB2D.8070001@xxxxxxxx> <4DDA10EE.20501@xxxxxxxx> <BANLkTinRDdJC7aifehKHJchXLFbgmAtWyw@xxxxxxxxxxxxxx>
Sender: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.17) Gecko/20110414 Lightning/1.0b2 Thunderbird/3.1.10
Hi!

I am sorry for not answering. My SPAM filter wasn't your friend and I missed to check it in time... :-(

Am 05/23/2011 10:28 AM, schrieb Fajar A. Nugraha:
On Mon, May 23, 2011 at 2:46 PM, Daniel Brockmann wrote:

the more I think about my DRBD issue and the more I research in the net the
more I tend to explain the issue with limited CPU time for dom0.

First thing first.
By "sync problems" in your previous post, did you mean both nodes
experience split brain for the drbd resource?

One machine -luckily that one that didn't have any important machines running- lost sync so the other machine was the "newer" DRBD partner. But when starting the sync again all virtual guests of the "newer" DRBD partner weren't available anymore until I stopped the sync forcefully. Besides this the sync speed was far below that what I had when I set up the DRBD pair on both Xenserver machines.


When setup properly, you should NOT experience it, regardless of how
much CPU resource dom0 has. You should only experience SLOW disk I/O.
split brain usually occur if you don't setup fencing properly.

The funny thing: It worked fine in the beginning. I was playing around with Xenmotion with a productive mail server machine without any problems. It continuously synced with rates between 250 and 350 MByte/s which is more than okay for our needs.


It will be
better resolving _this_ instead of possibly reaching the same stage later on
again but using another replication technique, wouldn't it?

Reasons why I think it is an I/O and/or CPU time issue:

1. It worked properly when I still did not have 8 virtual guest systems
installed.
2. As soon as I start a DRBD resync my virtual guests bring kernel error
messages like "INFO: task exim4:2336 blocked for more than 120 seconds. ".
3. When starting both Xenserver machines and syncing before starting the
virtual guests a startup that's usually done in<5 minutes takes up to 60
minutes.

... which is exactly the SLOW I/O I mentioned above.

What I cannot understand is why this was occuring weeks after the setup was tested successfully. :-/

Okay, one thing changed: A few machines were added to the virtual pool. But even when these machines have been halted I could not sync the DRBD pair anymore.


I checked the XenWiki accordingly and found two promising entries that I'd
like to follow, if it's possible to apply them under a Citrix Xenserver 5.6
system:

http://wiki.xensource.com/xenwiki/XenCommonProblems#head-413e1d74442772fd5a0a94f0655a009744096627

1. How can I limit the number of vcpus my dom0 has?
2. Can I dedicate a cpu core (or cores) only for dom0?

Especially the 2nd one appears to meet what I expect. So I would be going to
check if I can configure that. How do _you_ think about it?

This thread might be able to help you:
http://xen.1045712.n5.nabble.com/XenServer-adding-additional-CPUs-to-the-control-domain-td3325379.html

I tried it differently then by installing the DRBD "loser" with Xenserver 5.6 SP2 that reserves 4 VCPUs to dom0. I additionally cut 4 GB RAM away for dom0. And I compiled DRBD from the latest sources this time (on the old systems I took precompiled binaries). But this time I couldn't even create a Xenserver storage on the DRBD device.


Personally, I suggest you step back and evaluate several things:
- do you REALLY need active-active setup?
Active-active drbd mandates protocol C (sync replication), which can
GREATLY slow down your throughput. If you can afford a small downtime
better stick with async replication.

I could afford it in the worst case. And I will keep your suggestion in mind for the case my alternative tests appears to fail again.

Actually I run Debian Squeeze on the former DRBD "loser" with a still-not-syncing DRBD device. However, now I try KVM ... and it appears to run at least as fine as my Xenserver setup before ... let's see what's gonna happen in the moment when I kill the Xenserver machine in order to be installed with Squeeze, too ... and start syncing both.

(In this moment I wished to have more than just two hosts.)


- do you KNOW how much IOPS you need?
[...]

Hmm, It's not gonna be a lot more than the machines I have. Most performance-consuming would be a virtualized Samba machine, I guess. I'll check that next week when I am at the machines again. Thanks for the suggestion.


All things considered, it might be that your best option would be
something like:
- get a separate server with lots of disks, setup raid10, install a
storage appliance OS on top (e.g.
http://www.napp-it.org/index_en.html) then export it to your xenserver
either as nfs or scsi. While nfs/scsci induce some overhead, it should
be lower compared to using drbd, OR

Hmm, my experience with NFS is that it's a lot slower than DRBD - however, ignoring the syncer rates I had lately. ;-)

iSCSI would be an option if I had the budget to have that separate storage server. But I unfortunately don't have it. :-(


- drop active-active requirement, OR

Dual-primary worked for a while. I want to check at first if I can get that running again the way it was (but using a different platform). But if it was the only way I could live with a "usual" DRBD setup, too. :-)


- beef-up your xenserver (e.g. use fast storage like SSD), upgrade
XenServer/XCP version to get dom0 to use multiple CPU core on dom0,
upgrade DRBD version to the latest, and setup proper fencing.

I am not that convenient anymore if Xenserver and my MagnyCours CPUs work well together at all. That's why I tried Squeeze and KVM now ... shipping around the limitations that Citrix included in their product.

Thank you very much for your suggestions. Regardless if it's KVM or Xen ... it will help me a lot.

CU,
Mészi.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users

<Prev in Thread] Current Thread [Next in Thread>
  • Re: [Xen-users] Solving the DRBD resync issue, Daniel Meszaros <=