[Xen-devel] RE: Tmem vs order>0 allocation, workaround RFC

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

Hi Jan --

Fair enough.  You've convinced me that I shouldn't push for
tmem to be turned back on by default for the official
Xen 4.0 release.  But the patch as just checked-in by Keir
limits allocations only if tmem is enabled so I will just
document that tmem may cause problems if 32-bit-limited
devices are in the system. (I'd expect that to be rare in
the cloud environment where tmem would be most used.)

I do think it's unfortunate (turning off tmem by default)
as I suspect that "thar be (more) dragons" in Xen, when
trying to do any kind of memory utilization optimization,
that will come back and bite us.  Tmem is just the first
to aggressively pursue this and disabling it only delays
the inevitable.  For example, I'll bet improvements to
NUMA support will have many similar problems.

Anyway, thanks as usual for thinking deeply through the
issue and for trying out tmem... any new technology is
going to have some growing pains.

Thanks again,
Dan

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@xxxxxxxxxx]
> Sent: Tuesday, February 16, 2010 8:46 AM
> To: Dan Magenheimer
> Cc: Grzegorz Milos; Patrick Colp; AndrewPeace; George Dunlap; Ian
> Pratt; KeirFraser; TimDeegan; xen-devel@xxxxxxxxxxxxxxxxxxx; Kurt
> Hackel
> Subject: RE: Tmem vs order>0 allocation, workaround RFC
> 
> >>> Dan Magenheimer <dan.magenheimer@xxxxxxxxxx> 16.02.10 16:31 >>>
> >> From: Jan Beulich [mailto:JBeulich@xxxxxxxxxx]
> >> Subject: RE: Tmem vs order>0 allocation, workaround RFC
> >>
> >> >>> Dan Magenheimer <dan.magenheimer@xxxxxxxxxx> 16.02.10 16:05 >>>
> >> >Under what circumstances does dom0 require single-page-below-4G
> >> >allocations?  Is it only for bounce buffers for PCI passthrough
> >> >of old devices with 32-bit addressing limitations?  Or am I
> >>> >missing a much more common case?
> >>
> >> Not just for pass-through; all devices only supporting 32-bit
> >> addressing would have such requirements, and particularly common
> >> ones are display adapters which have DRM/AGP drivers loaded for
> >> them.
> >
> >Right, but those are statically allocated when dom0 is
> >launched, not dynamically allocated later after tmem
> >(or other memory allocation technologies) begin working,
> >right?  Whereas pass-through devices would need below-4G
> >pages later?
> 
> No, consistent/coherent allocations can happen at run time.
> Typically the largest share of the allocations would happen when
> the respective driver loads, but especially for the DRM/AGP case
> I think allocations get triggered by user mode (X initializing a
> display), which may happen at any time.
> 
> >(And 32-bit devices in a 1TB machine seems a bit of a
> >stretch, but I suppose it is good to enumerate all the
> >cases.)
> 
> Yes, but the 1Tb was just taken as an extreme example. Issues may
> arise earlier. And the display adapter part would likely remain valid
> even there - just see the use of vmalloc_32() in
> drivers/gpu/drm/drm_scatter.c for an example.
> 
> Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel