This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] VT-d scalability issue

To: "Zhai, Edwin" <edwin.zhai@xxxxxxxxx>
Subject: Re: [Xen-devel] VT-d scalability issue
From: Espen Skoglund <espen.skoglund@xxxxxxxxxxxxx>
Date: Wed, 10 Sep 2008 10:03:53 +0100
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Wed, 10 Sep 2008 02:05:33 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20080909090427.GA6704@xxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20080909090427.GA6704@xxxxxxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Not regarding the other questions/objections in this thread for a
moment --- what kind of performance improvments are we talking of here
if the vcpus are pinned?  Is it close to 1 VM or is there still some
performance degradation due to IOTLB pressure?

[And talking of IOTLB pressure, why can't Intel document the IOTLB
sizes in the chipset docs?  Or even better, why can't these values be
queried from the chipset?]


[Edwin Zhai]
> Keir,
> I have found a VT-d scalability issue and want to some feed backs.

> When I assign a pass-through NIC to a linux VM and increase the num
> of VMs, the iperf throughput for each VM drops greatly. Say, start 8
> VM running on a machine with 8 physical cpus, start 8 iperf client
> to connect each of them, the final result is only 60% of 1 VM.

> Further investigation shows vcpu migration cause "cold" cache for
> pass-through domain.  following code in vmx_do_resume try to
> invalidate orig processor's cache when 14 migration if this domain
> has pass-through device and no support for wbinvd vmexit.

> 16 if ( has_arch_pdevs(v->domain) && !cpu_has_wbinvd_exiting )
> {
>     int cpu = v->arch.hvm_vmx.active_cpu;
>     if ( cpu != -1 )
>         on_selected_cpus(cpumask_of_cpu(cpu), wbinvd_ipi, NULL, 1,

> }

> So we want to pin vcpu to free processor for domains with
> pass-through device in creation process, just like what we did for
> NUMA system.

> What do you think of it? Or have other ideas?

> Thanks,

> -- 
> best rgds,
> edwin

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

Xen-devel mailing list