Hi, Espen & Kay,
> Are you assuming guest will ping the guest to a physical CPU?
Yes.
On Xen, memory is automatically assigned from the same NUMA node as a
physical CPU. Memory isn't moved from initial assignment. But if we
doesn't ping a guest to physical CPUs, the guest can run on every
physical CPUs. So, when the user uses NUMA machine, it is better to
ping the guest to physical CPUs. Because the latency becomes lower.
> How does the user figure out which devices are closer to which
> physical CPU in the platform in a QPI system without using proximity
> domain info?
The users can read machine spec to assign I/O device from the same
NUMA node as CPU to a guest.
> By at least keeping the IOMMU page tables local to the node you'll get
> lower latencies for the page table walker.
We can get lower latency by using proximity domain info of DMAR. But it
needs more modifications than my patch. I will not work for this.
If anyone develops this function, I will not be against it.
Thanks,
--
Yuji Shimada
On Thu, 20 Nov 2008 20:00:08 +0000
Espen Skoglund <espen.skoglund@xxxxxxxxxxxxx> wrote:
> You only require more memory if you duplicate the structures per
> IOMMU. While this is indeed possible (and may even be the desired
> solution) it is not what I suggested.
>
> And you're making the assumption here that the guest is assigned to
> the node of the IOMMU. As Allen points out, how does a user make this
> decision? And in many cases I would expect that you would not want to
> assign many guests to the same node anyway. By at least keeping the
> IOMMU page tables local to the node you'll get lower latencies for the
> page table walker.
>
> eSk
>
On Wed, 19 Nov 2008 10:57:10 -0800
"Kay, Allen M" <allen.m.kay@xxxxxxxxx> wrote:
>
> >Xen's user will assign a device to a closer guest. So, node of the
> >guest and node connected to IOMMU will be the same.
> >As a result, the memory performance will be improved with my patch.
>
> Are you assuming guest will ping the guest to a physical CPU? How
> does the user figure out which devices are closer to which physical
> CPU in the platform in a QPI system without using proximity domain
> info?
>
> Allen
>
> >> -----Original Message-----
> >> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
> >> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of
> >> Yuji Shimada
> >> Sent: Wednesday, November 19, 2008 12:26 AM
> >> To: Espen Skoglund
> >> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; 'Keir Fraser'
> >> Subject: Re: [Xen-devel] [PATCH] fix memory allocation from
> >> NUMA node for VT-d.
> >>
> >> Hi Espen,
> >>
> >> Your suggestion allocating memory from one of the nodes where the
> >> IOMMU is attached improves performance more. But more memory is
> >> needed, because structures are needed per IOMMU.
> >>
> >> My patch keeps the current implementation, one Device Assignment
> >> Structure and Address Translation Structure per guest.
> >>
> >> Xen's user will assign a device to a closer guest. So, node of the
> >> guest and node connected to IOMMU will be the same.
> >> As a result, the memory performance will be improved with my patch.
> >>
> >> Thanks,
> >> --
> >> Yuji Shimada
> >>
> >> On Tue, 18 Nov 2008 12:00:37 +0000
> >> Espen Skoglund <espen.skoglund@xxxxxxxxxxxxx> wrote:
> >>
> >>> Given an FSB based system the IOMMUs sit in the north-bridge. How
> >>> does this work qith QPI? Where in the system do the different IOMMUs
> >>> sit? Wouldn't it make more sense to allocate memory from one of the
> >>> nodes where the IOMMU is attached? Having the memory allocated from
> >>> the node of the guest only helps when the guest needs to update its
> >>> page tables. I'd rather optimize for page table walks in the IOMMU.
> >>>
> >>> eSk
> > >
> > >
> > >_______________________________________________
> > >Xen-devel mailing list
> > >Xen-devel@xxxxxxxxxxxxxxxxxxx
> > >http://lists.xensource.com/xen-devel
> > >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home