|
xen-users
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk
I'll top post this since the thread is top heavy now...
Yeah, file: io uses VFS I believe so it leverages page cache for heavy
caching.
The I am unsure if the vbd uses the Linux block io or a modified
version of VFS that does sync writes, but if it uses VFS with sync io
that'd explain the performance loss, not so much the synchronicity,
but the chopping of io into 4k blocks (still page cache/memory
addressed) and going sync.
Run a bonnie with 4k io size and see if it is such a big difference.
I'd either run iSCSI within the domU, which could be problematic down
the line with Xen domain networking, or mount in dom0 and present to
domU over file io and leave a chunk of memory in dom0 for page caching.
-Ross
On Jan 13, 2009, at 9:47 PM, "Christopher Chen" <muffaleta@xxxxxxxxx>
wrote:
And here's more!
I decided to try something more. I mounted the iSCSI LUN on the Dom0,
created a ext3 file system, and created a image file, 10G big. The
Dom0 has 4G of RAM. The DomU has 2G allocated to it.
So I thought, what the hell...
I block-attached the image file as file://mnt/image xvdc w to the
guest, and mkfs'd it, and mounted, it and ran bonnie to use a 8G test
size (twice the size of the Dom0's RAM, and 4x the DomU's RAM). Here's
the results:
Version 1.03 ------Sequential Output------ --Sequential
Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
aenu 8G 130195 22 27364 1 46949 0 313.6 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec %CP
16 6228 99 +++++ +++ +++++ +++ 6182 97 +++++ +++ 19075 100
aenu,8G,,,130195,22,27364,1,,,46949,0,313.6,0,16,6228,99,+++++,+++,++
+++,+++,6182,97,+++++,+++,19075,100
Yeah, um, so, sustained IO is pretty good via file: driver, but still,
performance via the xvd driver perplexes me:
Version 1.03 ------Sequential Output------ --Sequential
Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
aenu 4G 47100 3 15069 0 39072 0 3971 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
+++++ +++
aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+
++++,+++,+++++,+++,+++++,+++,+++++,+++
So yeah, 36%. Pretty awesome.
I am stumped.
Cheers
cc
On Tue, Jan 13, 2009 at 4:03 PM, Christopher Chen
<muffaleta@xxxxxxxxx> wrote:
On Tue, Jan 13, 2009 at 3:53 PM, Ross Walker <rswwalker@xxxxxxxxx>
wrote:
On Jan 13, 2009, at 6:37 PM, Ross Walker <rswwalker@xxxxxxxxx>
wrote:
On Jan 13, 2009, at 5:48 PM, "Christopher Chen" <muffaleta@xxxxxxxxx
>
wrote:
Hi there!
I've been wrestling with an issue for a little bit now--
In my test environment, I have tgtd running on a Centos 5.2 box,
with
a raid 10 array backing it.
The initiators are also Centos 5.2 boxes running Xen 3.0.3
userland
with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos).
Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec
reads.
That's kind of lopsided I'd expect it the other way around.
Is this hardware RAID on the backend with write-back cache?
I've attached the iSCSI LUN to the DomU as a virtual block
device, and
I'm seeing 47MB/sec writes, and 39MB/sec reads.
How did you attach it, what Xen driver did you use phy: or file:?
Sorry, missed the virtual block device bit...
I've tried a few things, like running against a local disk, and
suprisingly, writes on the DomU are faster than the Dom0--can I
assume
the writes are buffered by the Dom0.
I'm confused.
I thought you said above you got 110MB/s on dom0 and 45MB/s on
the domU?
Never mind my comment, writes are only buffered using file: io,
but they are
buffered in the domU's page cache which is where you might be
seeing the
performance difference.
I'm going to give a shot doing the initialization from the DomU
(just
for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.
You've completely lost me now, what do you mean initialization?
Do you
mean boot domU off of iSCSI directly?
After re-reading I guessed you meant you attached to the iSCSI lun
after
booting into the VM not as the OS disk.
Again you are most likely seeing all cache affect and not the real
io.
This is all with bonnie++ -d /mnt -f -u root:root
Anyone seen this, or have any ideas?
Is any additional latency provided by the xen virtual block device
causing a degradation in TCP performance (i.e. a window size or
delayed ACK problem) or is the buffering also causing pain? I'm
going
to keep looking, but I thought I'd ask all of you.
Any layer you add is going to create latency.
If you can be a little more clearer I'm sure an accurate
explanation can
be made.
Try increasing the size of the bonnie test file to defeat the
cache, say 2x
the memory of the dom0 or domU or target which ever is largest.
The nice thing about bonnie++ -f is it sizes the file for 2x memory.
These are the numbers. In any case, the ~110MB/sec writes to the
iSCSI
target is our baseline number writing across the network. The Dom0
has
4G allocated to it--bonnie++'s test file is 8G. Any reading lower
than
that (in my mind) is degradation. I, of course, expect some effect
from the layering, but 50%?
cc
--
Chris Chen <muffaleta@xxxxxxxxx>
"I want the kind of six pack you can't drink."
-- Micah
--
Chris Chen <muffaleta@xxxxxxxxx>
"I want the kind of six pack you can't drink."
-- Micah
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
|
|
|
Home