This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] Xen 4.0.0x allows for data corruption in Dom0

To: Jeremy Fitzhardinge <jeremy@xxxxxxxx>
Subject: Re: [Xen-devel] Xen 4.0.0x allows for data corruption in Dom0
From: Joanna Rutkowska <joanna@xxxxxxxxxxxxxxxxxxxxxx>
Date: Mon, 08 Mar 2010 23:34:59 +0100
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Mon, 08 Mar 2010 14:35:37 -0800
Dkim-signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=messagingengine.com; h=message-id:date:from:mime-version:to:cc:subject:references:in-reply-to:content-type; s=smtpout; bh=E56+35uVBqEYyVqwfuY0Z+u/SHw=; b=lTpOWzjAGEuY04KtnrWs3bYhl3hw2bC7vX9+W62n3DFtzqJDYD6iXHwOximLh2kYpLCCRVOeS0g9+q81+naF3u79IbxvuCIfFxPv3gxGpL8bqfKI2wIAQzGEJsK1R8JpLGG1Q7p4iuuS7fT0m2ttLEUEGSyt9eGT9ONsdjJvinI=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <4B957914.4050408@xxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4B922A89.2060105@xxxxxxxxxxxxxxxxxxxxxx> <4B957914.4050408@xxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20100301 Fedora/3.0.3-1.fc12 Lightning/1.0b2pre Thunderbird/3.0.3
On 03/08/2010 11:24 PM, Jeremy Fitzhardinge wrote:
> On 03/06/2010 02:12 AM, Joanna Rutkowska wrote:
>> There is a nasty data corruption problem most likely allowed by a bug in
>> the Xen 4.0.0-x hypervisors.
>> The problem occurs with a frequency of "a few chunks per 10 GB of data
>> copied", and only when running a VM (PV domU) with a specific kernel.
>> The problem, however, affects not only the VM but also the Dom0, which
>> is of significant importance.
>> How to reproduce:
>> 1) Start at least one Xen PV VM with a pvops0 kernel. One kernel known
>> to demonstrate the problem is the one built by Michael Young, based on
>> xen/master git from Dec 23. It has recently been replaced by a newer
>> kernel, which doesn't always show the problem, but I uploaded the
>> previous one at the URL below, so people can use it for testing:
>> http://invisiblethingslab.com/pub/kernel-
>> Now you can start a dummy VM with this kernel, e.g.:
>> # xm create -c /dev/null memory=400 kernel=<path/to/kernel>
>> extra="rootdelay=1000"
>> 2) Now, in Dom0, after having started this dummy VM, create a big test
>> file, filled all with zeros. Make sure to choose a size bigger than your
>> DRAM size, to avoid fs caching effect, e.g.:
>> $ dd if=/dev/zero of=test bs=1M count=10000
>> That should create a 10GB file. Make sure to use /dev/zero and not
>> /dev/null!
>> 3) Once the test file got created, check if it really consists of zeros
>> only:
>> $ xxd test.bin | grep -v "0000 0000 0000 0000 0000 0000 0000 0000"
>> Normally you should not get any output. However, I consistently get
>> something like this:
>> 4593a000:940d 0000 0000 0000 2d40 d6fc c803 0000  ........-@......
>> 4593a010:00f6 1f52 b301 0000 b620 dcd5 ff00 0000  ...R..... ......
>> a5df0000:e542 712c 77da c9f9 a429 4b85 ecc4 9395  .Bq,w....)K.....
>> a5df0010:d9d6 971f 0d58 5c70 aba6 387d 805f 09e2  .....X\p..8}._..
>> ceecb000:f80d 0000 0000 0000 096e 1cdc e403 0000  .........n......
>> ceecb010:2460 7ef6 be01 0000 b620 dcd5 ff00 0000  $`~...... ......
>> 148432000580e 0000 0000 0000 5665 ed9d ff03 0000  X.......Ve......
>> 1484320107bcc a023 ca01 0000 b620 dcd5 ff00 0000  {..#..... ......
>> 1c548b000bc0e 0000 0000 0000 6942 387d 1b04 0000  ........iB8}....
>> 1c548b010872b 01c8 d501 0000 b620 dcd5 ff00 0000  .+....... ......
>> 225d450004448 27cd b966 b37e 1f0c e9e3 c2db b6ee  DH'..f.~........
>> 225d45010d2b2 55b8 9ef1 e818 a7e3 364d 2322 dc75  ..U.......6M#".u
>> 242056000140f 0000 0000 0000 0bb0 3704 3404 0000  ..........7.4...
>> 2420560109601 b606 e001 0000 b620 dcd5 ff00 0000  ......... ......
>> The actual data vary between tests, however, the "dcd5 ff00 0000"
>> pattern seems to be repeatable on a given system with a given hypervisor
>> binary (the above numbers are for Xen-4.0.0-rc5 built from Michael
>> Young's SRPM). The errors always occur in chunks of 32-bytes.
>> We have tested this in our lab on three different machines, with various
>> Dom0 kernels -- based on xen/master (AKA xen/stable-2.6.31) and
>> xen/stable (AKA xen/stable-2.6.32) -- and with a few Xen 4 hypervisors
>> (rc2, rc4, rc5). Not every kernel allows for reproducing the error with
>> such a simple "dummy" VM as the one given above -- e.g. the 2.6.32-based
>> kernels required some more regular VMs to be started for the problem to
>> be noticeable. However, with the previously mentioned kernel (M. Young
>> Dec23), the problem has been 100% reproducible us.
>> When downgraded to Xen 3.4.2 the problem went away.
>> Of course this problem cannot be attributed to a buggy VM kernel, as the
>> hypervisor should be resistant to any kind of "wrong" software (buggy or
>> malicious) that executes in a VM.
> Why "of course"?  You report looks to me like a bug in dom0 which is
> causing data corruption when there's another domain running.

Please note that the "of course" sentence refers to *VM* kernel not Dom0.

> I don't see anything that specifically implicates Xen.  The fact that
> the symptoms change with a different Xen version could mean kernel
> bug is effected by the Xen version (different memory layout, for
> example, or different paths in the kernel caused by different feature
> availability).

Sure, it can theoretically be anything, perhaps even a generic bug in
IA32 just accidentally triggered by some magic value in a register ;) As
I said in the first sentence it seems (to me) "most likely" to be a bug
in the hypervisor, but there is only one way to find out where it is for
sure... (to nail it down (and I'm very sorry that I cannot help with the
quest right now))


Attachment: signature.asc
Description: OpenPGP digital signature

Xen-devel mailing list
<Prev in Thread] Current Thread [Next in Thread>