This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] [PATCH] blkback: Fix block I/O latency issue

To: "Vincent, Pradeep" <pradeepv@xxxxxxxxxx>
Subject: Re: [Xen-devel] [PATCH] blkback: Fix block I/O latency issue
From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Date: Thu, 12 May 2011 22:51:32 -0400
Cc: Jeremy Fitzhardinge <jeremy@xxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Jan Beulich <JBeulich@xxxxxxxxxx>, Daniel Stodden <daniel.stodden@xxxxxxxxxx>
Delivery-date: Thu, 12 May 2011 19:52:25 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <C9F1B153.1500C%pradeepv@xxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20110509202403.GA27755@xxxxxxxxxxxx> <C9F1B153.1500C%pradeepv@xxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mutt/1.5.21 (2010-09-15)
> >>what were the numbers when it came to high bandwidth numbers
> Under high I/O workload, where the blkfront would fill up the queue as
> blkback works the queue, the I/O latency problem in question doesn't
> manifest itself and as a result this patch doesn't make much of a
> difference in terms of interrupt rate. My benchmarks didn't show any
> significant effect.

I have to rerun my benchmarks. Under high load (so 64Kb, four threads
writting as much as they can to a iSCSI disk), the IRQ rate for each
blkif went from 2-3/sec to ~5K/sec. But I did not do a good
job on capturing the submission latency to see if the I/Os get the
response back as fast (or the same) as without your patch.

And the iSCSI disk on the target side was an RAMdisk, so latency
was quite small which is not fair to your problem.

Do you have a program to measure the latency for the workload you
had encountered? I would like to run those numbers myself.

> The above rationale combined with relatively high disk I/O latencies
> (compared to IRQ latency) generally minimizes excessive interrupt rate.
> Also, blkfront interrupt generation mechanism works exactly the same way
> as the patched blkback. Netfront and netback generate interrupts the same
> way as well. 
> Under 'moderate' I/O workload, the rate of interrupt does go up but the
> I/O latency benefit clearly outweighs the cost extra interrupt rate (which
> isn't much for moderate I/O load anyways)
> Overall, advantages of this patch (I/O latency improvement) outweighs any
> potential fringe negative effects by a large margin and the fact that
> netfront, netback and blkfront already have the same interrupt generation
> design point should give us a lot of confidence.

If we can come up with a patch that does decrease the I/O latency _and_
does not cause a huge spike in interrupt generation that would be super.

I am particularly worried about the high interrupt generation when there is
a high I/O load. But that seems to be tied directly to the "disk" I have.
I am curious to how this works with drivers that are SSDs for example.

> That said, I do think a comprehensive interrupt throttling mechanism for
> netback, blkback and other backend I/O drivers would be useful and should
> be pursued as a separate initiative. Such a mechanism would be

Yeah, my fear is that you will disappear once this patch is in and I will
have to go forth to do this (and I don't know when I would get to it).
Would prefer to have your help here or (better) you write the patch for
both problems right now :-)

> particularly useful for netfront-netback stack which is more susceptible
> to interrupt storms than blkfront-blkback. 'IRQ coalescing' type mechanism
> that could induce delays in the order of  10s of microsecs (certainly not
> in millisecs though) to minimize interrupt generation rate would be useful
> (similar to what NICs do).

Good point. Awaiting your patch for the netfront-netback code :-)

Xen-devel mailing list

<Prev in Thread] Current Thread [Next in Thread>