Uh oh....
On Sat, Aug 05, 2006 at 08:38:03AM +0100, Ian Pratt wrote:
> > So I built -unstable changeset 10868, and ran an even heavier workload
> > (the above, plus 'bonnie' in the guests) on dom0 and two guests
> > overnight, and they experienced no soft lockups; running -unstable,
> > changeset 10868, credit scheduler. This same workload would have
> > caused soft lockups within seconds in -testing changeset 9732 using
> > the sedf scheduler; I may not have been able to get it started at all.
> > Response time remained subsecond under -unstable; -testing would have
> > been on its knees.
>
> That's good to hear. 3.0.3 is going to be a big leap forward in many
> ways.
Same environment as we've discussed earlier in this thread, now on
unstable changeset 10868, Xen on top of DRBD, this one seems to be a
reliable crash when under combined disk and network load, still
isolating but would welcome suggestions...
Steve
root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /boot/xen-3.0.gz dom0_mem=65536 com1=9600,8n1
[Multiboot-elf, <0x100000:0x7a008:0x55ff8>, shtab=0x1d0078, entry=0x100000]
module /boot/vmlinuz-2.6.16.13-xen root=801 ro console=ttyS0 ramdisk_size=32768
[Multiboot-module @ 0x1d1000, 0x494ed0 bytes]
module /boot/initrd.img-2.6.16.13-xen
[Multiboot-module @ 0x666000, 0x9a6400 bytes]
__ __ _____ ___ _ _ _
\ \/ /___ _ __ |___ / / _ \ _ _ _ __ ___| |_ __ _| |__ | | ___
\ // _ \ '_ \ |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
/ \ __/ | | | ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | | __/
/_/\_\___|_| |_| |____(_)___/ \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory
Xen version 3.0-unstable (root@xxxxxxxxxxxxxxxxx) (gcc version 3.3.5 (Debian
1:3.3.5-13)) Fri Aug 4 10:54:50 PDT 2006
Latest ChangeSet: Sat Jul 29 06:05:59 2006 +0100 10868:d2bf1a7cc131
(XEN) Command line: /boot/xen-3.0.gz dom0_mem=65536 com1=9600,8n1
(XEN) Physical RAM map:
(XEN) 0000000000000000 - 000000000009c000 (usable)
(XEN) 000000000009c000 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 000000003ffec340 (usable)
(XEN) 000000003ffec340 - 000000003fff0000 (ACPI data)
(XEN) 000000003fff0000 - 0000000040000000 (reserved)
(XEN) 00000000fec00000 - 0000000100000000 (reserved)
(XEN) System RAM: 1023MB (1048096kB)
(XEN) Xen heap: 10MB (10396kB)
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) PAE disabled.
(XEN) found SMP MP-table at 0009c1d0
(XEN) DMI 2.3 present.
(XEN) Using APIC driver default
(XEN) ACPI: RSDP (v000 IBM ) @ 0x000fdfd0
(XEN) ACPI: RSDT (v001 IBM SEREMRLD 0x00001000 IBM 0x45444f43) @ 0x3ffeff80
(XEN) ACPI: FADT (v001 IBM SEREMRLD 0x00001000 IBM 0x45444f43) @ 0x3ffeff00
(XEN) ACPI: MADT (v001 IBM SEREMRLD 0x00001000 IBM 0x45444f43) @ 0x3ffefe80
(XEN) ACPI: DSDT (v001 IBM SEREMRLD 0x00001000 MSFT 0x0100000b) @ 0x00000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x03] enabled)
(XEN) Processor #3 6:11 APIC version 17
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 6:11 APIC version 17
(XEN) ACPI: IOAPIC (id[0x0e] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 14, version 17, address 0xfec00000, GSI 0-15
(XEN) ACPI: IOAPIC (id[0x0d] address[0xfec01000] gsi_base[16])
(XEN) IOAPIC[1]: apic_id 13, version 17, address 0xfec01000, GSI 16-31
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 3 global_irq 30 dfl dfl)
(XEN) ACPI: IRQ3 used by override.
(XEN) Enabling APIC mode: Flat. Using 2 I/O APICs
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Initializing CPU#0
(XEN) Detected 1130.180 MHz processor.
(XEN) CPU: L1 I cache: 16K, L1 D cache: 16K
(XEN) CPU: L2 cache: 512K
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Intel(R) Pentium(R) III CPU family 1133MHz stepping 01
(XEN) Booting processor 1/0 eip 90000
(XEN) Initializing CPU#1
(XEN) CPU: L1 I cache: 16K, L1 D cache: 16K
(XEN) CPU: L2 cache: 512K
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Intel(R) Pentium(R) III CPU family 1133MHz stepping 01
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=0 apic2=-1 pin2=-1
(XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC
(XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed.
(XEN) ...trying to set up timer as Virtual Wire IRQ... works.
(XEN) checking TSC synchronization across 2 CPUs: passed.
(XEN) Platform timer is 1.193MHz PIT
(XEN) Brought up 2 CPUs
(XEN) Machine check exception polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Domain 0 kernel supports features = { 0000001f }.
(XEN) Domain 0 kernel requires features = { 00000000 }.
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 3c000000->3e000000 (8192 pages to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: c0100000->c05d5d74
(XEN) Init. ramdisk: c05d6000->c0f7c400
(XEN) Phys-Mach map: c0f7d000->c0f8d000
(XEN) Start info: c0f8d000->c0f8e000
(XEN) Page tables: c0f8e000->c0f94000
(XEN) Boot stack: c0f94000->c0f95000
(XEN) TOTAL: c0000000->c1400000
(XEN) ENTRY ADDRESS: c0100000
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Initrd len 0x9a6400, start at 0xc05d6000
(XEN) Scrubbing Free RAM: ...........done.
(XEN) Xen trace buffers: disabled
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
Xen).
Linux version 2.6.16.13-xen (root@n4h8) (gcc version 3.3.5 (Debian 1:3.3.5-13))
#3 SMP Fri Aug 4 10:57:44 PDT 2006
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 0000000004800000 (usable)
0MB HIGHMEM available.
72MB LOWMEM available.
DMI 2.3 present.
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x03] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: IOAPIC (id[0x0e] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 14, version 17, address 0xfec00000, GSI 0-15
ACPI: IOAPIC (id[0x0d] address[0xfec01000] gsi_base[16])
IOAPIC[1]: apic_id 13, version 17, address 0xfec01000, GSI 16-31
ACPI: INT_SRC_OVR (bus 0 bus_irq 3 global_irq 30 dfl dfl)
Enabling APIC mode: Flat. Using 2 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 50000000 (gap: 40000000:bec00000)
Built 1 zonelists
Kernel command line: root=801 ro console=ttyS0 ramdisk_size=32768
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 512 (order: 9, 8192 bytes)
Xen reported: 1130.113 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Software IO TLB enabled:
Aperture: 2 megabytes
Kernel range: 0x00000000c10f0000 - 0x00000000c12f0000
vmalloc area: c5000000-fb7fe000, maxmem 33ffe000
Memory: 47400k/73728k available (3389k kernel code, 18068k reserved, 1044k
data, 208k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 2261.60 BogoMIPS (lpj=11308023)
Security Framework v1.0.0 initialized
Capability LSM initialized
Mount-cache hash table entries: 512
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 512K
Checking 'hlt' instruction... OK.
ENABLING IO-APIC IRQs
Enabling SMP...
Brought up 2 CPUs
Initializing CPU#1
migration_cost=1895
checking if image is initramfs... it is
Freeing initrd memory: 9881k freed
Grant table initialized
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20060127
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
ACPI: PCI Interrupt Link [LPE1] (IRQs *10)
ACPI: PCI Interrupt Link [LPE2] (IRQs *10)
ACPI: PCI Interrupt Link [LPVI] (IRQs) *0, disabled.
ACPI: PCI Interrupt Link [LPUS] (IRQs *7)
ACPI: PCI Root Bridge [PCI1] (0000:01)
ACPI: PCI Interrupt Link [LPSA] (IRQs *9)
ACPI: PCI Interrupt Link [LP1A] (IRQs) *0, disabled.
ACPI: PCI Interrupt Link [LP1B] (IRQs) *0, disabled.
ACPI: PCI Interrupt Link [LP2A] (IRQs *9)
ACPI: PCI Interrupt Link [LP2B] (IRQs) *0, disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
xen_mem: Initialising balloon driver.
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
IA-32 Microcode Update Driver: v1.14-xen <tigran@xxxxxxxxxxx>
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
JFS: nTxBlock = 512, nTxLock = 4096
SGI XFS with ACLs, security attributes, realtime, large block numbers, no debug
enabled
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
RAMDISK driver initialized: 16 RAM disks of 32768K size 1024 blocksize
Xen virtual console successfully installed as ttyS0
Event-channel device installed.
blkif_init: reqs=64, pages=704, mmap_vstart=0xc0800000
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
SvrWks OSB4: IDE controller at PCI slot 0000:00:0f.1
SvrWks OSB4: chipset revision 0
SvrWks OSB4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0x0700-0x0707, BIOS settings: hda:DMA, hdb:DMA
ide1: BM-DMA at 0x0708-0x070f, BIOS settings: hdc:DMA, hdd:DMA
hda: LG CD-ROM CRN-8245B, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: ATAPI 24X CD-ROM drive, 128kB Cache, (U)DMA
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
ACPI: PCI Interrupt 0000:01:03.0[A] -> GSI 28 (level, low) -> IRQ 16
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
<Adaptec aic7892 Ultra160 SCSI adapter>
aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ACPI: PCI Interrupt 0000:01:05.0[A] -> GSI 20 (level, low) -> IRQ 17
scsi1 : IBM PCI ServeRAID 7.12.05 Build 761 <ServeRAID 4L>
Vendor: IBM Model: SERVERAID Rev: 1.00
Type: Direct-Access ANSI SCSI revision: 02
Vendor: IBM Model: SERVERAID Rev: 1.00
Type: Direct-Access ANSI SCSI revision: 02
Vendor: IBM Model: SERVERAID Rev: 1.00
Type: Processor ANSI SCSI revision: 02
Vendor: IBM Model: FTlV1 S2 Rev: 0
Type: Processor ANSI SCSI revision: 02
Fusion MPT base driver 3.03.07
Copyright (c) 1999-2005 LSI Logic Corporation
Fusion MPT SPI Host driver 3.03.07
Fusion MPT misc device (ioctl) driver 3.03.07
mptctl: Registered with Fusion MPT base driver
mptctl: /dev/mptctl @ (major,minor=10,220)
usbmon: debugfs is not available
usbcore: registered new driver libusual
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 2
input: AT Translated Set 2 keyboard as /class/input/input0
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 8
NET: Registered protocol family 20
Using IPI No-Shortcut mode
Freeing unused kernel memory: 208k freed
Loading, please wait...
Begin: Loading essential drivers... ...
logips2pp: Detected unknown logitech mouse model 0
eepro100.c:v1.09j-t 9/29/99 Donald Becker
http://www.scyld.com/network/eepro100.html
eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin
<saw@xxxxxxxxxxxxx> and others
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 27 (level, low) -> IRQ 18
eth0: 0000:00:02.0, 00:02:55:C7:CA:D8, IRQ 18.
Board assembly 754338-001, Physical connectors present: RJ45
Primary interface chip i82555 PHY #1.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x04f4518b).
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 25 (level, low) -> IRQ 19
eth1: 0000:00:0a.0, 00:02:55:C7:CA:D9, IRQ 19.
Board assembly 754338-001, Physical connectors present: RJ45
Primary interface chip i82555 PHY #1.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x04f4518b).
Intel(R) PRO/1000 Network Driver - version 6.3.9-k4
Copyright (c) 1999-2005 Intel Corporation.
Done.
Begin: Running /scripts/init-premount ...
input: PS/2 Logitech Mouse as /class/input/input1
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
device-mapper: 4.5.0-ioctl (2005-10-04) initialised: dm-devel@xxxxxxxxxx
/scripts/local-top/lvm: 36: vgchange: not found
Done.
SCSI device sda: 71096320 512-byte hdwr sectors (36401 MB)
sda: assuming Write Enabled
sda: assuming drive cache: write through
SCSI device sda: 71096320 512-byte hdwr sectors (36401 MB)
sda: assuming Write Enabled
sda: assuming drive cache: write through
sda: sda1 sda2 < sda5 >
sd 1:0:0:0: Attached scsi disk sda
SCSI device sdb: 71096320 512-byte hdwr sectors (36401 MB)
sdb: assuming Write Enabled
sdb: assuming drive cache: write through
SCSI device sdb: 71096320 512-byte hdwr sectors (36401 MB)
sdb: assuming Write Enabled
sdb: assuming drive cache: write through
sdb: unknown partition table
sd 1:0:1:0: Attached scsi disk sdb
Begin: Running /scripts/local-premount ...
Done.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
Begin: Running /scripts/log-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
INIT: version 2.86 booting
Starting the hotplug events dispatcher: udevd.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Loading /etc/console/boottime.kmap.gz
Activating swap.
Adding 1019880k swap on /dev/sda5. Priority:-1 extents:1 across:1019880k
Checking root file system...
fsck 1.37 (21-Mar-2005)
/dev/sda1: clean, 58437/512000 files, 445341/1023996 blocks
EXT3 FS on sda1, internal journal
System time was Sat Aug 12 00:22:00 UTC 2006.
Setting the System Clock using the Hardware Clock as reference...
System Clock set. System local time is now Sat Aug 12 00:22:01 UTC 2006.
Calculating module dependencies...done.
Loading modules...
auto
FATAL: Module auto not found.
3c59x
softdog
Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=60 sec
(nowayout= 0)
tulip
Linux Tulip driver version 1.1.13 (May 11, 2002)
e100
e100: Intel(R) PRO/100 Network Driver, 3.5.10-k2-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
All modules loaded.
Loading device-mapper support.
Starting Enterprise Volume Management System: device-mapper: dm-linear: Device
lookup failed
device-mapper: error adding target to table
device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table
device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table
device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table
device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table
device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table
device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table
device-mapper: dm-linear: Device lookup failed
device-mapper: error adding target to table
evms.
Checking all file systems...
fsck 1.37 (21-Mar-2005)
Setting kernel variables ...
... done.
Loading the saved-state of the serial devices...
Cannot get serial info: Invalid argument
Cannot get serial info: Invalid argument
/dev/ttyS1: No such file or directory
/dev/ttyS1: No such file or directory
Mounting local filesystems...
Cleaning /tmp /var/run /var/lock.
Cleaning: /etc/network/ifstate.
Setting up IP spoofing protection: rp_filter.
Configuring network interfaces: done.
Starting portmap daemon: portmap.
Setting the System Clock using the Hardware Clock as reference...
System Clock set. Local time: Fri Aug 11 17:22:07 PDT 2006
Recovering jove files ... Done.
Running ntpdate to synchronize clock.
Initializing random number generator...done.
Recovering nvi editor sessions... done.
Setting up X server socket directory /tmp/.X11-unix...done.
Setting up ICE socket directory /tmp/.ICE-unix...done.
INIT: Entering runlevel: 2
Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting virtual private network daemon:.
Starting portmap daemon: portmap.
Starting NFS common utilities: statd lockd.
Starting internet superserver: inetd.
* The mptctl module is missing. Please have a look at the README.Debian.gz.
.
Exporting directories for NFS kernel daemon...Installing knfsd (copyright (C)
1996 okir@xxxxxxxxxxxx).
10.27.3.21:/export/xen/fs/stevegt/crashme1/root: No such file or directory
10.27.3.18:/export/xen/fs/stevegt/xentest2/root: No such file or directory
10.27.3.17:/export/xen/fs/stevegt/xentest1/root: No such file or directory
10.27.3.8:/export/xen/fs/cclarke/umltest1/root: No such file or directory
hotaru.chaosring.Org:/export/xen/fs/stevegt/crashme1/root: No such file or
directory
rn-2.refactored-networks.com:/export/xen/fs/stevegt/xentest2/root: No such file
or directory
xentest1.stevegt.TerraLuna.Org:/export/xen/fs/stevegt/xentest1/root: No such
file or directory
umltest1.cclarke.TerraLuna.Org:/export/xen/fs/cclarke/umltest1/root: No such
file or directory
10.27.3.26:/export/xen/fs/stevegt/gforge/root: No such file or directory
10.27.3.23:/export/xen/fs/baseline/sarge/root: No such file or directory
10.27.3.22:/export/xen/fs/stevegt/build1/root: No such file or directory
10.27.3.16:/export/xen/fs/baseline/woody/root: No such file or directory
bugs.t7a.Org:/export/xen/fs/stevegt/gforge/root: No such file or directory
sarge.baseline.TerraLuna.Org:/export/xen/fs/baseline/sarge/root: No such file
or directory
build1.stevegt.TerraLuna.Org:/export/xen/fs/stevegt/build1/root: No such file
or directory
woody.baseline.TerraLuna.Org:/export/xen/fs/baseline/woody/root: No such file
or directory
10.27.3.19:/export/xen/fs/cclarke/ebond/root: No such file or directory
ebond.cclarke.TerraLuna.Org:/export/xen/fs/cclarke/ebond/root: No such file or
directory
10.27.3.29:/export/xen/fs/dmasten/mss0/root: No such file or directory
10.27.3.27:/export/xen/fs/cclarke/ccms/root: No such file or directory
10.27.3.25:/export/xen/fs/baseline/sid/root: No such file or directory
10.27.3.24:/export/xen/fs/stevegt/cvs1/root: No such file or directory
sid.stevegt.TerraLuna.Org:/export/xen/fs/baseline/sid/root: No such file or
directory
cvs1.stevegt.TerraLuna.Org:/export/xen/fs/stevegt/cvs1/root: No such file or
directory
10.27.3.28:/export/xen/fs/stevegt/t7a/root: No such file or directory
10.27.3.20:/export/xen/fs/stevegt/tcx/root: No such file or directory
tcx.TerraLuna.Org:/export/xen/fs/stevegt/tcx/root: No such file or directory
n2h54.prd.TerraLuna.Org:/export/xen: No such file or directory
n2h53.prd.TerraLuna.Org:/export/xen: No such file or directory
n2h51.prd.TerraLuna.Org:/export/xen: No such file or directory
n2h41.prd.TerraLuna.Org:/export/xen: No such file or directory
done.
Starting NFS kernel daemon: nfsdNFSD: Using /var/lib/nfs/v4recovery as the
NFSv4 state recovery directory
NFSD: unable to find recovery directory /var/lib/nfs/v4recovery
NFSD: starting 90-second grace period
mountd.
Starting OpenBSD Secure Shell server: sshdNET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
.
Starting NTP server: ntpd.
AFS module /lib/modules/2.6.16.13-xen/fs/openafs.mp.o does not exist. Not
starting AFS.
Please consider building kernel modules using instructions in
/usr/share/doc/openafs-client/README.modules
Starting DRBD resources: drbd: initialised. Version: 0.7.20 (api:79/proto:74)
drbd: SVN Revision: 2260 build by root@n4h8, 2006-08-04 11:00:44
drbd: registered as block device major 147
[ d0 drbd7: resync bitmap: bits=1048576 words=32768
drbd7: size = 4095 MB (4194303 KB)
drbd7: 0 KB marked out-of-sync by on disk bit-map.
drbd7: No usable activity log found.
drbd7: drbdsetup [3825]: cstate Unconfigured --> StandAlone
d1 drbd8: resync bitmap: bits=1048576 words=32768
drbd8: size = 4095 MB (4194303 KB)
drbd8: 0 KB marked out-of-sync by on disk bit-map.
drbd8: Found 4 transactions (46 active extents) in activity log.
drbd8: Marked additional 156 MB as out-of-sync based on AL.
drbd8: drbdsetup [3829]: cstate Unconfigured --> StandAlone
d2 drbd9: resync bitmap: bits=2621440 words=81920
drbd9: size = 9 GB (10485759 KB)
drbd9: 0 KB marked out-of-sync by on disk bit-map.
drbd9: Found 4 transactions (192 active extents) in activity log.
drbd9: drbdsetup [3833]: cstate Unconfigured --> StandAlone
d3 drbd10: resync bitmap: bits=1835008 words=57344
drbd10: size = 7167 MB (7340031 KB)
drbd10: 0 KB marked out-of-sync by on disk bit-map.
drbd10: Found 4 transactions (192 active extents) in activity log.
drbd10: Marked additional 508 MB as out-of-sync based on AL.
drbd10: drbdsetup [3837]: cstate Unconfigured --> StandAlone
d4 drbd11: resync bitmap: bits=786432 words=24576
drbd11: size = 3071 MB (3145727 KB)
drbd11: 0 KB marked out-of-sync by on disk bit-map.
drbd11: Found 4 transactions (192 active extents) in activity log.
drbd11: drbdsetup [3841]: cstate Unconfigured --> StandAlone
d5 drbd12: resync bitmap: bits=786432 words=24576
drbd12: size = 3071 MB (3145727 KB)
drbd12: 0 KB marked out-of-sync by on disk bit-map.
drbd12: Found 4 transactions (136 active extents) in activity log.
drbd12: Marked additional 508 MB as out-of-sync based on AL.
drbd12: drbdsetup [3845]: cstate Unconfigured --> StandAlone
s0 s1 s2 s3 s4 s5 n0 drbd7: drbdsetup [3883]: cstate StandAlone --> Unconnected
drbd7: drbd7_receiver [3884]: cstate Unconnected --> WFConnection
n1 drbd8: drbdsetup [3891]: cstate StandAlone --> Unconnected
drbd8: drbd8_receiver [3892]: cstate Unconnected --> WFConnection
n2 drbd9: drbdsetup [3899]: cstate StandAlone --> Unconnected
drbd9: drbd9_receiver [3900]: cstate Unconnected --> WFConnection
n3 drbd10: drbdsetup [3907]: cstate StandAlone --> Unconnected
drbd10: drbd10_receiver [3908]: cstate Unconnected --> WFConnection
n4 drbd11: drbdsetup [3915]: cstate StandAlone --> Unconnected
drbd11: drbd11_receiver [3916]: cstate Unconnected --> WFConnection
n5 drbd12: drbdsetup [3923]: cstate StandAlone --> Unconnected
drbd12: drbd12_receiver [3924]: cstate Unconnected --> WFConnection
].
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
reboot the timeout is 120 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
expire after 0 seconds. [wfc-timeout]
(These values are for resource 'ds1.t7a.org'; 0 sec -> wait forever)
To abort waiting enter 'yes' [ 18]:drbd7: drbd7_receiver [3884]: cstate
WFConnection --> WFReportParams
drbd7: Handshake successful: DRBD Network Protocol version 74
drbd7: Connection established.
drbd7: I am(S): 1:00000002:00000001:00000018:00000002:01
drbd7: Peer(S): 1:00000002:00000001:00000019:00000002:10
drbd7: drbd7_receiver [3884]: cstate WFReportParams --> WFBitMapT
drbd7: Secondary/Unknown --> Secondary/Secondary
drbd7: drbd7_receiver [3884]: cstate WFBitMapT --> SyncTarget
drbd7: Resync started as SyncTarget (need to sync 520192 KB [130048 bits set]).
drbd9: drbd9_receiver [3900]: cstate WFConnection --> WFReportParams
drbd9: Handshake successful: DRBD Network Protocol version 74
drbd9: Connection established.
drbd9: I am(S): 1:00000002:00000001:0000000d:00000001:01
drbd9: Peer(S): 1:00000002:00000001:0000000e:00000001:10
drbd9: drbd9_receiver [3900]: cstate WFReportParams --> WFBitMapT
drbd9: Secondary/Unknown --> Secondary/Secondary
drbd9: drbd9_receiver [3900]: cstate WFBitMapT --> SyncTarget
drbd9: Resync started as SyncTarget (need to sync 290816 KB [72704 bits set]).
drbd9: drbd9_receiver [3900]: cstate SyncTarget --> PausedSyncT
drbd9: Syncer waits for sync group.
drbd11: drbd11_receiver [3916]: cstate WFConnection --> WFReportParams
drbd11: Handshake successful: DRBD Network Protocol version 74
drbd11: Connection established.
drbd11: I am(S): 1:00000002:00000001:00000008:00000001:01
drbd11: Peer(S): 1:00000002:00000001:00000009:00000001:10
drbd11: drbd11_receiver [3916]: cstate WFReportParams --> WFBitMapT
drbd11: Secondary/Unknown --> Secondary/Secondary
drbd11: drbd11_receiver [3916]: cstate WFBitMapT --> SyncTarget
drbd11: Resync started as SyncTarget (need to sync 274432 KB [68608 bits set]).
drbd11: drbd11_receiver [3916]: cstate SyncTarget --> PausedSyncT
drbd11: Syncer waits for sync group.
drbd8: drbd8_receiver [3892]: cstate WFConnection --> WFReportParams
drbd8: Handshake successful: DRBD Network Protocol version 74
drbd8: Connection established.
drbd8: I am(S): 1:00000002:00000001:00000017:00000001:11
drbd8: Peer(S): 1:00000002:00000001:00000017:00000001:01
drbd8: drbd8_receiver [3892]: cstate WFReportParams --> WFBitMapS
drbd8: Secondary/Unknown --> Secondary/Secondary
drbd8: drbd8_receiver [3892]: cstate WFBitMapS --> SyncSource
drbd8: Resync started as SyncSource (need to sync 159744 KB [39936 bits set]).
drbd8: drbd8_receiver [3892]: cstate SyncSource --> PausedSyncS
drbd8: Syncer waits for sync group.
drbd12: drbd12_receiver [3924]: cstate WFConnection --> WFReportParams
drbd12: Handshake successful: DRBD Network Protocol version 74
drbd12: Connection established.
drbd12: I am(S): 1:00000002:00000001:00000001:00000001:11
drbd12: Peer(S): 1:00000002:00000001:00000001:00000001:01
drbd12: drbd12_receiver [3924]: cstate WFReportParams --> WFBitMapS
drbd12: Secondary/Unknown --> Secondary/Secondary
drbd12: drbd12_receiver [3924]: cstate WFBitMapS --> SyncSource
drbd12: Resync started as SyncSource (need to sync 520192 KB [130048 bits set]).
drbd12: drbd12_receiver [3924]: cstate SyncSource --> PausedSyncS
drbd12: Syncer waits for sync group.
drbd10: drbd10_receiver [3908]: cstate WFConnection --> WFReportParams
drbd10: Handshake successful: DRBD Network Protocol version 74
drbd10: Connection established.
drbd10: I am(S): 1:00000002:00000001:0000000b:00000001:11
drbd10: Peer(S): 1:00000002:00000001:0000000b:00000001:01
drbd10: drbd10_receiver [3908]: cstate WFReportParams --> WFBitMapS
drbd10: Secondary/Unknown --> Secondary/Secondary
drbd10: drbd10_receiver [3908]: cstate WFBitMapS --> SyncSource
drbd10: Resync started as SyncSource (need to sync 520192 KB [130048 bits set]).
drbd10: drbd10_receiver [3908]: cstate SyncSource --> PausedSyncS
drbd10: Syncer waits for sync group.
Heartbeat not configured: /etc/ha.d/ha.cf not found.
Heartbeat failure [rc=1]. Failed.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting watchdog daemon: watchdog.
openMosix: ERROR: Cannot find the /proc/hpc directory.
openMosix: ERROR: Looks like this is not an openMosix enabled kernel.
openMosix: ERROR: Configuration ABORTED.
openMosix: HINT: need to recompile your kernel with openMosix support enabled?
openMosix: HINT: need to update your boot manager (lilo, grub, etc)?
openMosix: HINT: have you read the docs at http://openMosix.sf.net/ ?
Bridge firewalling registered
device vif0.0 entered promiscuous mode
xenbr0: port 1(vif0.0) entering learning state
xenbr0: topology change detected, propagating
xenbr0: port 1(vif0.0) entering forwarding state
device peth0 entered promiscuous mode
xenbr0: port 2(peth0) entering learning state
xenbr0: topology change detected, propagating
xenbr0: port 2(peth0) entering forwarding state
Starting auto Xen domains: afs1.t7a.orgdrbd9: Secondary/Secondary -->
Secondary/Primary
device vif1.0 entered promiscuous mode
ADDRCONF(NETDEV_UP): vif1.0: link is not ready
ip_tables: (C) 2000-2006 Netfilter Core Team
Error: Device 2049 (vbd) could not be connected. Backend device not found.
xenbr0: port 3(vif1.0) entering disabled state
! build2.t7a.orgdevice vif1.0 left promiscuous mode
xenbr0: port 3(vif1.0) entering disabled state
device vif2.0 entered promiscuous mode
ADDRCONF(NETDEV_UP): vif2.0: link is not ready
drbd10: Secondary/Secondary --> Primary/Secondary
ADDRCONF(NETDEV_CHANGE): vif2.0: link becomes ready
xenbr0: port 3(vif2.0) entering learning state
xenbr0: topology change detected, propagating
xenbr0: port 3(vif2.0) entering forwarding state
ds1.t7a.orgdrbd7: Secondary/Secondary --> Secondary/Primary
device vif3.0 entered promiscuous mode
ADDRCONF(NETDEV_UP): vif3.0: link is not ready
Error: Device 2049 (vbd) could not be connected. Backend device not found.
! kdc1.t7a.orgxenbr0: port 4(vif3.0) entering disabled state
device vif3.0 left promiscuous mode
xenbr0: port 4(vif3.0) entering disabled state
drbd8: Secondary/Secondary --> Primary/Secondary
device vif4.0 entered promiscuous mode
ADDRCONF(NETDEV_UP): vif4.0: link is not ready
ADDRCONF(NETDEV_CHANGE): vif4.0: link becomes ready
xenbr0: port 4(vif4.0) entering learning state
xenbr0: topology change detected, propagating
xenbr0: port 4(vif4.0) entering forwarding state
w1.cdint.comdrbd11: Secondary/Secondary --> Secondary/Primary
device vif5.0 entered promiscuous mode
ADDRCONF(NETDEV_UP): vif5.0: link is not ready
Error: Device 2049 (vbd) could not be connected. Backend device not found.
xenbr0: port 5(vif5.0) entering disabled state
! webkdc.t7a.orgdevice vif5.0 left promiscuous mode
xenbr0: port 5(vif5.0) entering disabled state
device vif6.0 entered promiscuous mode
ADDRCONF(NETDEV_UP): vif6.0: link is not ready
drbd12: Secondary/Secondary --> Primary/Secondary
ADDRCONF(NETDEV_CHANGE): vif6.0: link becomes ready
xenbr0: port 5(vif6.0) entering learning state
xenbr0: topology change detected, propagating
xenbr0: port 5(vif6.0) entering forwarding state
[done]
_sudZUZ#Z#XZo=_ DDDD EEEEEE BBBB IIIIII AAAA NN NN
_jmZZ2!!~---~!!X##wa DD DD EE BB BB II AA AA NNN NN
.<wdP~~ -!YZL, DD DD EEEEE BBBBB II AAAAAA NNNN NN
.mX2' _%aaa__ XZ[. DD DD EE BB BB II AA AA NN NNNN
oZ[ _jdXY!~?S#wa ]Xb; DDDD EEEEEE BBBBB IIIIII AA AA NN NN
_#e' .]X2( ~Xw| )XXc
.2Z` ]X[. xY| ]oZ( Linux Version 2.6.16.13-xen
.2#; )3k; _s!~ jXf` Compiled #3 SMP Fri Aug 4 10:57:44 PDT 2006
1Z> -]Xb/ ~ __#2( Two 1.13GHz Intel Intel(R) Pentium(R) III CPU
family 1133MHz Processors
-Zo; +!4ZwaaaauZZXY' 72M RAM
*#[, ~-?!!!!!!-~ 4523.20 Bogomips Total
XUb;. n4h34
)YXL,,
+3#bc,
-)SSL,,
~~~~~
Updating the Linuxlogo... done.
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login: drbd7: Resync done (total 137 sec; paused 0 sec; 3796 K/sec)
drbd7: drbd7_worker [3826]: cstate SyncTarget --> Connected
drbd8: Syncer continues.
drbd8: drbd7_worker [3826]: cstate PausedSyncS --> SyncSource
drbd9: Syncer continues.
drbd9: drbd7_worker [3826]: cstate PausedSyncT --> SyncTarget
drbd10: Syncer continues.
drbd10: drbd7_worker [3826]: cstate PausedSyncS --> SyncSource
drbd11: Syncer continues.
drbd11: drbd7_worker [3826]: cstate PausedSyncT --> SyncTarget
drbd12: Syncer continues.
drbd12: drbd7_worker [3826]: cstate PausedSyncS --> SyncSource
drbd8: Resync done (total 211 sec; paused 137 sec; 2156 K/sec)
drbd8: drbd8_worker [3830]: cstate SyncSource --> Connected
drbd11: Resync done (total 239 sec; paused 137 sec; 2688 K/sec)
drbd11: drbd11_worker [3842]: cstate SyncTarget --> Connected
drbd9: Resync done (total 243 sec; paused 137 sec; 2740 K/sec)
drbd9: drbd9_worker [3834]: cstate SyncTarget --> Connected
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login:
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login:
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login:
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login:
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login:
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login: drbd12: Resync done (total 306 sec; paused 136 sec; 3056 K/sec)
drbd12: drbd12_worker [3846]: cstate SyncSource --> Connected
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login:
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login: drbd10: Resync done (total 310 sec; paused 136 sec; 2988 K/sec)
drbd10: drbd10_worker [3838]: cstate SyncSource --> Connected
Debian GNU/Linux 3.0 n4h34 ttyS0
n4h34 login: Unable to handle kernel paging request at virtual address c0976590
printing eip:
c03d6ed7
*pde = ma 3cf91067 pa 00f91067
*pte = ma 00000000 pa fffff000
Oops: 0000 [#1]
SMP
Modules linked in: iptable_filter ip_tables x_tables bridge drbd ipv6 nfsd
lockd sunrpc e100 tulip softdog 3c59x evdev sd_mod dm_mod thermal processor fan
e1000 eepro100 mii tg3
CPU: 0
EIP: 0061:[<c03d6ed7>] Not tainted VLI
EFLAGS: 00010206 (2.6.16.13-xen #3)
EIP is at skb_copy_bits+0x127/0x280
eax: c0976000 ebx: 000005a8 ecx: 0000016a edx: c3711720
esi: c0976590 edi: c37110e0 ebp: 000005a8 esp: c00818c8
ds: 007b es: 007b ss: 0069
Process drbd12_receiver (pid: 3924, threadinfo=c0080000 task=c3efda90)
Stack: <0>c1012ec0 00000002 c03d65ef c1d51a00 000005ea 00000042 00000000
00000000
00000020 c3edc800 c376fd64 c03d6b0f c376fd64 00000042 c37110e0 000005a8
c1a70000 c54690c0 00000000 c1d51ac0 c3edc800 c376fd64 c03dc478 c376fd64
Call Trace:
[<c03d65ef>] pskb_expand_head+0xdf/0x140
[<c03d6b0f>] __pskb_pull_tail+0x7f/0x320
[<c54690c0>] br_nf_dev_queue_xmit+0x0/0x50 [bridge]
[<c03dc478>] dev_queue_xmit+0x328/0x370
[<c5462f7e>] br_dev_queue_push_xmit+0xbe/0x140 [bridge]
[<c5469212>] br_nf_post_routing+0x102/0x1c0 [bridge]
[<c54690c0>] br_nf_dev_queue_xmit+0x0/0x50 [bridge]
[<c5462ec0>] br_dev_queue_push_xmit+0x0/0x140 [bridge]
[<c03f43f8>] nf_iterate+0x78/0x90
[<c5462ec0>] br_dev_queue_push_xmit+0x0/0x140 [bridge]
[<c5462ec0>] br_dev_queue_push_xmit+0x0/0x140 [bridge]
[<c03f447e>] nf_hook_slow+0x6e/0x110
[<c5462ec0>] br_dev_queue_push_xmit+0x0/0x140 [bridge]
[<c5463061>] br_forward_finish+0x61/0x70 [bridge]
[<c5462ec0>] br_dev_queue_push_xmit+0x0/0x140 [bridge]
[<c5468995>] br_nf_forward_finish+0x75/0x130 [bridge]
[<c5463000>] br_forward_finish+0x0/0x70 [bridge]
[<c5468b38>] br_nf_forward_ip+0xe8/0x190 [bridge]
[<c5468920>] br_nf_forward_finish+0x0/0x130 [bridge]
[<c5463000>] br_forward_finish+0x0/0x70 [bridge]
[<c03f43f8>] nf_iterate+0x78/0x90
[<c5463000>] br_forward_finish+0x0/0x70 [bridge]
[<c5463000>] br_forward_finish+0x0/0x70 [bridge]
[<c03f447e>] nf_hook_slow+0x6e/0x110
[<c5463000>] br_forward_finish+0x0/0x70 [bridge]
[<c5463167>] __br_forward+0x77/0x80 [bridge]
[<c5463000>] br_forward_finish+0x0/0x70 [bridge]
[<c5463fbf>] br_handle_frame_finish+0xdf/0x160 [bridge]
[<c5463ee0>] br_handle_frame_finish+0x0/0x160 [bridge]
[<c5467d89>] br_nf_pre_routing_finish+0xf9/0x370 [bridge]
[<c5463ee0>] br_handle_frame_finish+0x0/0x160 [bridge]
[<c0322e3a>] loopback_start_xmit+0xba/0x110
[<c0400d70>] ip_finish_output+0x0/0x220
[<c03dc07e>] dev_hard_start_xmit+0x5e/0x130
[<c03dc3b5>] dev_queue_xmit+0x265/0x370
[<c03f43f8>] nf_iterate+0x78/0x90
[<c5467c90>] br_nf_pre_routing_finish+0x0/0x370 [bridge]
[<c5467c90>] br_nf_pre_routing_finish+0x0/0x370 [bridge]
[<c03f447e>] nf_hook_slow+0x6e/0x110
[<c5467c90>] br_nf_pre_routing_finish+0x0/0x370 [bridge]
[<c5463ee0>] br_handle_frame_finish+0x0/0x160 [bridge]
[<c54685fc>] br_nf_pre_routing+0x26c/0x520 [bridge]
[<c5467c90>] br_nf_pre_routing_finish+0x0/0x370 [bridge]
[<c03f43f8>] nf_iterate+0x78/0x90
[<c5463ee0>] br_handle_frame_finish+0x0/0x160 [bridge]
[<c5463ee0>] br_handle_frame_finish+0x0/0x160 [bridge]
[<c03f447e>] nf_hook_slow+0x6e/0x110
[<c5463ee0>] br_handle_frame_finish+0x0/0x160 [bridge]
[<c546422d>] br_handle_frame+0x1ed/0x230 [bridge]
[<c5463ee0>] br_handle_frame_finish+0x0/0x160 [bridge]
[<c03dcb21>] netif_receive_skb+0x1a1/0x330
[<c03dcd87>] process_backlog+0xd7/0x190
[<c03dcf2a>] net_rx_action+0xea/0x230
[<c0125915>] __do_softirq+0xf5/0x120
[<c01259d5>] do_softirq+0x95/0xa0
[<c0125a42>] local_bh_enable+0x62/0xa0
[<c0406bd1>] tcp_prequeue_process+0x71/0x80
[<c04070e9>] tcp_recvmsg+0x349/0x750
[<c50eebfc>] dm_request+0xbc/0x100 [dm_mod]
[<c03d5085>] sock_common_recvmsg+0x55/0x70
[<c03d11cf>] sock_recvmsg+0xef/0x110
[<c03143fa>] force_evtchn_callback+0xa/0x10
[<c0147163>] mempool_alloc+0x33/0xe0
[<c01367d0>] autoremove_wake_function+0x0/0x60
[<c50eebfc>] dm_request+0xbc/0x100 [dm_mod]
[<c02b2ee0>] generic_make_request+0xf0/0x160
[<c529c680>] drbd_recv+0x90/0x190 [drbd]
[<c529cdec>] drbd_recv_header+0x2c/0xf0 [drbd]
[<c529e580>] receive_DataRequest+0x0/0x7d0 [drbd]
[<c52a045c>] drbdd+0x1c/0x150 [drbd]
[<c52a105a>] drbdd_init+0x7a/0x1a0 [drbd]
[<c52a7136>] drbd_thread_setup+0x86/0xf0 [drbd]
[<c52a70b0>] drbd_thread_setup+0x0/0xf0 [drbd]
[<c0102f75>] kernel_thread_helper+0x5/0x10
Code: 8b 4c 24 30 8b 7c 24 34 8b 91 a0 00 00 00 8b 4c 24 18 0f b7 74 ca 18 8b
4c 24 14 8d 34 06 01 fe 29 ce 8b 7c 24 38 89 d9 c1 e9 02 <f3> a5 89 d9 83 e1 03
74 02 f3 a4 89 04 24 ba 02 00 00 00 89 54
<0>Kernel panic - not syncing: Fatal exception in interrupt
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
--
Stephen G. Traugott (KG6HDQ)
Managing Partner, TerraLuna LLC
stevegt@xxxxxxxxxxxxx -- http://www.t7a.org
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home