Re: [Xen-devel] xen-unstable: TX/RX ring buffer exhaustion and NR_GRANT_FRAMES

[Christopher Clark <christopher.clark@xxxxxxxxxxxx>]

Hi

Here's the quick and dirty explanation of why you can't just boost that #define:

In the data structure that tracks a domain's list of foreign frames currently mapped by using grant references ('maptrack'), 11 bits are used to record the foreign grant reference index. This is to keep the data structure packed into 32 bits because you end up needing a lot of them if you're mapping a lot of foreign frames.

So the foreign domain can't hand you a grant reference index >= 2^11, so that's 2048 maximum grant entries (ie. frames you can allow others to map) per domain.
The sizeof the shared data structure stored in the shared grant table is 64 bits == 8 bytes. So 2048 of 8 bytes gives you 4 frames, which is why NR_GRANT_FRAMES is maxed out at 4 right now.

If you do just boost it above 4 then your index ends up colliding with the flags packed into the other bits and I'm not surprised it gets nasty.

Mea culpa: I hadn't anticipated each NIC needing as many entries as they actually do.
Fixing this is surgery.

For right now, if you can live with giving up some performance to get your extra NICs, you could try halving the ring sizes NET_TX_RING_SIZE and NET_RX_RING_SIZE defined in sparse/drivers/xen/netback/common.h

Christopher

On 1/10/06, Birger Tödtmann <btoedtmann@xxxxxxxxxxxxxx> wrote:

Hi,

domUs are yet not allowed to run more than 3 NICs as far as I can see
when running tests with 3.0, and the 'old' way to provide more
interfaces by enhancing the grant table structures do not work anymore
(see below).  I'd really appreciate any suggestions where I could modify
the source to get more NICs.


Am Freitag, den 11.11.2005, 12:11 +0000 schrieb harry:
> The xenidc code has the concept of a buffer resource provider which
> implements an anti-deadlock resource reservation.  Each client has uses
> a pool with it's own buffer resource provider and the anti-deadlock
> resource reservations decouple the clients from one another.  The
> underlying implementation of the buffer resource provider is currently
> trivial (individual resource pools) but the intention is that the buffer
> resource providers will share resources from a common underlying pool.
>
> This resource management strategy allows you to make better use of
> constrained resources like grant tables whilst avoiding deadlock.

Interesting thing, but I could not find documentation on xenidc - where
do I get it?  Where is the code, is it already inside Xen?


>
> On Fri, 2005-11-11 at 10:48 +0000, Keir Fraser wrote:
> > On 11 Nov 2005, at 08:59, Birger Toedtmann wrote:
> >
> > > message - see bug #183.  It was pointed out to me that it might be
> > > possible to adjust this manually in xen/include/xen/grant_table.h,
> > > however, setting ORDER_GRANT_FRAMES to 3 to get more pages of grant
> > > tables doesn't change anything.  Sooo, question #1: how can I alter the
> > > code to get more grant tables such that more nics per domU will become
> > > available?  Question #2: is it planned to make this configurable, e.g.
> > > by boot option for dom0?
> >
> > 1: you need to edit NR_GRANT_FRAMES in linux/include/asm-xen/gnttab.h
> >
> > 2: yes, it will eventually be manually configurable with a higher
> > default 'maximum'

Unfortunately, this method does not work anymore since November/
December.  After modifying NR_GRANT_FRAMES etc., the dom0 would say
"g.e. still in use!" on boot and halt after this message.  Any hints how
I can adjust the grant tables to get more NICs?



Regards,
--
Birger Tödtmann                   email:btoedtmann@xxxxxxxxxxxxxx
Technik der Rechnernetze, Institut für Experimentelle Mathematik
Universität Duisburg-Essen, Campus Essen, Germany.
skype:birger.toedtmann  pgp:0x6FB166C9  phone: +49-(0)201-1837662


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel