[Xen-users] I/O errors in domU with LVM on DRBD

[jai lamerton <jai@xxxxxxxxxxxxxx>]

Hi,

Sorry for the need of the long winded email. Looking for some answers to the following.

I am setting up a xen PV domU on top of a LVM partitioned DRBD device. Everything was going just fine until I tried to test the filesystems in the domU.

Here is my setup;

Dom0 OS: CentOS release 5 (Final)

Kernel: 2.6.18-8.1.4.el5.centos.plusxen

Xen: xen-3.0.3-25.0.3.el5

DRBD: drbd-8.0.3-1.el5.centos

PV DomU OS: CentOS release 4.5 (Final)

Kernel: 2.6.9-55.ELxenU

my drbd.conf

global {

        usage-count yes;

}

common {

        startup {

                wfc-timeout 0;

                degr-wfc-timeout 60;

        }

        disk {

                on-io-error detach;

        }

        syncer {

                rate 100M;

                al-extents 257;

        }

        net {

                max-buffers   2048;

                unplug-watermark   2048;

                max-epoch-size  2048;

                ko-count 4;

                cram-hmac-alg "sha1";

                shared-secret "asecretstring";

                after-sb-0pri disconnect;

                after-sb-1pri consensus;

                after-sb-2pri disconnect;

                rr-conflict disconnect;

        }

}

resource r0 {

        protocol C;

        handlers {

                pri-on-incon-degr "echo 0 > /proc/sysrq-trigger ; halt -f";

                pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";

                local-io-error "echo 0 > /proc/sysrq-trigger ; halt -f";

                outdate-peer "/usr/lib64/heartbeat/drbd-peer-outdater";

        }

        on xenhostA {

                device /dev/drbd0;

                disk /dev/sda3;

                address 172.16.0.30:7788;

                meta-disk internal;

        }

        on xenhostB {

                device /dev/drbd0;

                disk /dev/sda3;

                address 172.16.0.31:7788;

                meta-disk internal;

        }

}

LVM

# pvscan

  PV /dev/drbd0   VG VGvm1      lvm2 [15.54 GB / 3.05 GB free]

# lvscan

  ACTIVE            '/dev/VGvm1/LVroot' [10.00 GB] inherit

  ACTIVE            '/dev/VGvm1/LVtmp' [500.00 MB] inherit

  ACTIVE            '/dev/VGvm1/LVswap' [2.00 GB] inherit

lvm.conf has;

filter = [ "r|/dev/sda3|", "a|/dev/drbd0|" ]

My xen domU conf

import os, re

arch = os.uname()[4]

if re.search('64', arch):

    arch_libdir = 'lib64'

else:

    arch_libdir = 'lib'

name = "centos45domU"

memory = "512"

disk = [ 'phy:/dev/mapper/VGvm1-LVroot,xvda1,w',

         'phy:/dev/mapper/VGvm1-LVtmp,xvda2,w',

         'phy:/dev/mapper/VGvm1-LVswap,xvda3,w', ]

vif = [ 'mac=00:16:3e:d2:14:70, bridge=xenbr0', ]

vfb = ["type=vnc,vncunused=1"]

bootloader="/usr/bin/pygrub"

vcpus=2

on_reboot   = 'restart'

on_crash    = 'restart'

DomU /etc/fstab

/dev/xvda1              /                       ext3    defaults,usrquota        1 1

none                    /dev/pts                devpts  gid=5,mode=620  0 0

none                    /dev/shm                tmpfs   defaults        0 0

none                    /proc                   proc    defaults        0 0

none                    /sys                    sysfs   defaults        0 0

/dev/xvda2              /tmp                    ext3    defaults        0 0

/dev/xvda3              swap                    swap    defaults        0 0

** Problem **

While doing some tests writing files in domU I got;

# dd if=/dev/zero of=/deleteme bs=1M count=100

100+0 records in

100+0 records out

# dd if=/dev/zero of=/tmp/deleteme bs=1M count=100

end_request: I/O error, dev xvda2, sector 366778

end_request: I/O error, dev xvda2, sector 366844

end_request: I/O error, dev xvda2, sector 381116

printk: 8071 messages suppressed.

Buffer I/O error on device xvda2, logical block 190558

lost page write due to I/O error on xvda2

end_request: I/O error, dev xvda2, sector 381182

end_request: I/O error, dev xvda2, sector 415678

end_request: I/O error, dev xvda2, sector 450164

end_request: I/O error, dev xvda2, sector 1204

end_request: I/O error, dev xvda2, sector 1340

end_request: I/O error, dev xvda2, sector 1454

end_request: I/O error, dev xvda2, sector 1522

end_request: I/O error, dev xvda2, sector 1652

end_request: I/O error, dev xvda2, sector 1788

end_request: I/O error, dev xvda2, sector 1902

Aborting journal on device xvda2.

ext3_abort called.

EXT3-fs error (device xvda2): ext3_journal_start_sb: Detected aborted journal

Remounting filesystem read-only

DomU /var/log/messages contains

Jun 25 10:34:29 xenpv1 kernel: SELinux: initialized (dev xvda2, type ext3),uses xattr

Jun 25 10:34:34 xenpv1 kernel: end_request: I/O error, dev xvda2, sector 264182

Jun 25 10:34:34 xenpv1 kernel: printk: 32 messages suppressed.

Jun 25 10:34:34 xenpv1 kernel: Buffer I/O error on device xvda2, logical block 132091

Jun 25 10:34:34 xenpv1 kernel: lost page write due to I/O error on xvda2

Jun 25 10:34:34 xenpv1 kernel: Buffer I/O error on device xvda2, logical block 132092

Dom0 /var/log/messages contains

Jun 25 10:34:34 xenhosta kernel: drbd0: bio would need to, but cannot, be split: (vcnt=4,idx=0,size=4096,sector=10750334)

Jun 25 10:34:34 xenhosta kernel: drbd0: bio would need to, but cannot, be split: (vcnt=4,idx=0,size=4096,sector=10750458)

Jun 25 10:34:34 xenhosta kernel: drbd0: bio would need to, but cannot, be split: (vcnt=4,idx=0,size=4096,sector=10750524)

Jun 25 10:34:34 xenhosta kernel: drbd0: bio would need to, but cannot, be split: (vcnt=4,idx=0,size=4096,sector=10750590)

However running the same test on the device that is assigned to xvda2 mounted under dom0, there is no errors.

I will also ask the DRBD lists what "bio would need to, but cannot, be split:" means.

If anyone can shed some light on this, would be great. 

Thank you kindly.

Jai.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users