Re: Pinned, non-revocable mappings of VRAM: will bad things happen?

[Demi Marie Obenour <demiobenour@xxxxxxxxx>]

On 4/20/26 04:49, Christian König wrote:
> On 4/17/26 21:35, Demi Marie Obenour wrote:
>> On 4/17/26 03:53, Christian König wrote:
>>> On 4/16/26 18:13, Demi Marie Obenour wrote:
>>>> On 4/16/26 05:57, Christian König wrote:
> ...
>>>> Unless I am
>>>> mistaken, client workloads are highly non-uniform: a single game or
>>>> compute job might well use more VRAM than every other program on the
>>>> system combined.
>>>
>>> Yeah, perfectly correct.
>>>
>>>> Are these workloads impossible to make work well with pinning?
>>>
>>> No, as long as you don't know the workload beforehand, e.g. when you define 
>>> the limit.
>>>
>>> I mean that's why basically everybody avoids pinning and assigning fixed 
>>> amounts of resources.
>>>
>>> Even if you can make it work technically pinning usually results in a 
>>> rather bad end user experience.
>>>
>>> Regards,
>>> Christian.
>>
>> Do drivers and programs assume that they can access VRAM from the CPU?
> 
> Yes, and that is actually really important for performance.
> 
> That's why Alex and I came up with the idea of using the resize able BAR 
> feature to access all of VRAM on modern GPUs.
> 
> There are a couple of hacks which have been implemented over the years for 
> exotic platforms were MMIO/VRAM access was problematic. For example on a page 
> fault you use a GPU DMA engine to copy the VRAM buffer into system memory, 
> make the CPU memory access and then copy it back again on demand at the next 
> command submission.
> 
> But all of those hacks are basically just prove of concepts and result in 
> completely unusable performance.

I'm reminded of Asahi considering emulating unaligned accesses to
PCI BARs in a fault handler.

>> Are any of the following reasonable options?
>>
>> 1. Change the guest kernel to only map (and thus pin) a small subset
>>    of VRAM at any given time.  If unmapped VRAM is accessed the guest
>>    traps the page fault, evicts an old VRAM mapping, and creates a
>>    new one.
> 
> Yeah, that could potentially work.
> 
> This is basically what we do on the host kernel driver when we can't resize 
> the BAR for some reason. In that use case VRAM buffers are shuffled in and 
> out of the CPU accessible window of VRAM on demand.

How much is this going to hurt performance?
 
>> 2. Pretend that resizable BAR is not enabled, so the guest doesn't
>>    think it can map much of VRAM at once.  If resizable BAR is enabled
>>    on the host, it might be possible to split the large BAR mapping
>>    in a lot of ways.
> 
> That won't work. The userspace parts of the driver stack don't care how large 
> the BAR to access VRAM with the CPU is.
> 
> The expectation is that the kernel driver makes thing CPU accessible as 
> needed in the page fault handler.
> 
> It is still a good idea for your solution #1 to give the amount of "pin-able" 
> VRAM to the userspace stack as CPU visible VRAM limit so that test cases and 
> applications try to lower their usage of VRAM, e.g. use system memory bounce 
> buffers when possible.

That makes sense.

>> Or does Xen really need to allow the host to handle guest page faults?
>> That adds a huge amount of complexity to trusted and security-critical
>> parts of the system, so it really is a last resort.  Putting the
>> complexity in to the guest virtio-GPU driver is vastly preferable if
>> it can be made to work well.
> 
> Well the nested page fault handling KVM offers has proven to be extremely 
> useful. So when XEN can't do this it is clearly lacking an important feature.

I agree. However, it is a lot of work to implement, which is why I'm
looking for alternatives if possible.

KVM is part of the Linux kernel, so it can just call the Linux kernel
functions used to handle userspace page faults.  Xen is separate from
Linux, so it can't do that.  Instead, it will need to:

1. Determine that the fault needs to be handled by another VM, and
   the ID of the VM that needs to handle the fault.
2. Send a message to the VM asking it to handle the fault.
3. Block the vCPU until it gets a response.

Then the VM owning the memory will need to call the page fault handler
and provide the memory to Xen.  Xen then needs to:

4. Map the memory into the nested page tables of the VM that faulted.
5. Resume the vCPU.

> But I have one question: When XEN has a problem handling faults from the 
> guest on the host then how does that work for system memory mappings?
> 
> There is really no difference between VRAM and system memory in the handling 
> for the GPU driver stack.
> 
> Regards,
> Christian.

Generally, Xen makes the frontend (usually an unprivileged VM)
responsible for providing mappings to the backend (usually the host).
That is possible with system RAM but not with VRAM, because Xen has
no awareness of VRAM.  To Xen, VRAM is just a PCI BAR.

KVM runs in the same kernel as the GPU driver.  Xen doesn't, and that
is the source of the extra complexity.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

Attachment: OpenPGP_0xB288B55FFF9C22C1.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature