WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

RE: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU

To: "Lin, Ray" <Ray.Lin@xxxxxxx>, Dante Cinco <dantecinco@xxxxxxxxx>, Konrad Wilk <konrad.wilk@xxxxxxxxxx>
Subject: RE: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough
From: Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>
Date: Thu, 18 Nov 2010 16:20:47 -0800 (PST)
Cc: Jeremy Fitzhardinge <jeremy@xxxxxxxx>, Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>, mathieu.desnoyers@xxxxxxxxxx, Andrew Thomas <andrew.thomas@xxxxxxxxxx>, keir.fraser@xxxxxxxxxxxxx, Chris Mason <chris.mason@xxxxxxxxxx>
Delivery-date: Thu, 18 Nov 2010 16:21:52 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <EB4C61A1A2501842A04B573FE42B14D60137593FF5@xxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <c044d15d-9b58-4e73-83eb-b3260a7e0fb5@default EB4C61A1A2501842A04B573FE42B14D60137593FF5@xxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
> We did suspect it, since our old setting was HZ=1000 and we assigned
> more than 10 VCPUs to domU. But we don't see the performance difference
> with HZ=100.

FWIW, it didn't appear that the problems were proportional to HZ.
Seemed more that somehow the pvclock became incorrect and spent
a lot of time rereading the pvclock value.

> -----Original Message-----
> From: Lin, Ray [mailto:Ray.Lin@xxxxxxx]
> Sent: Thursday, November 18, 2010 2:40 PM
> To: Dan Magenheimer; Dante Cinco; Konrad Wilk
> Cc: Jeremy Fitzhardinge; Xen-devel; mathieu.desnoyers@xxxxxxxxxx;
> Andrew Thomas; keir.fraser@xxxxxxxxxxxxx; Chris Mason
> Subject: RE: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2
> pvops domU kernel with PCI passthrough
> 
> 
> 
> -----Original Message-----
> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-
> bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Dan Magenheimer
> Sent: Thursday, November 18, 2010 1:21 PM
> To: Dante Cinco; Konrad Wilk
> Cc: Jeremy Fitzhardinge; Xen-devel; mathieu.desnoyers@xxxxxxxxxx;
> Andrew Thomas; keir.fraser@xxxxxxxxxxxxx; Chris Mason
> Subject: RE: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2
> pvops domU kernel with PCI passthrough
> 
> In case it is related:
> http://lists.xensource.com/archives/html/xen-devel/2010-
> 07/msg01247.html
> 
> Although I never went further on this investigation, it appeared to me
> that pvclock_clocksource_read was getting called at least an order-of-
> magnitude more frequently than expected in some circumstances for some
> kernels.  And IIRC it was scaled by the number of vcpus.
> 
> We did suspect it, since our old setting was HZ=1000 and we assigned
> more than 10 VCPUs to domU. But we don't see the performance difference
> with HZ=100.
> 
> > -----Original Message-----
> > From: Dante Cinco [mailto:dantecinco@xxxxxxxxx]
> > Sent: Thursday, November 18, 2010 12:36 PM
> > To: Konrad Rzeszutek Wilk
> > Cc: Jeremy Fitzhardinge; Xen-devel; mathieu.desnoyers@xxxxxxxxxx;
> > Andrew Thomas; keir.fraser@xxxxxxxxxxxxx; Chris Mason
> > Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2
> > pvops domU kernel with PCI passthrough
> >
> > I mentioned earlier in an previous post to this thread that I'm able
> > to apply Dulloor's xenoprofile patch to the dom0 kernel but not the
> > domU kernel. So I can't do active-domain profiling but I'm able to do
> > passive-domain profiling but I don't know how reliable the results
> are
> > since it shows pvclock_clocksource_read as the top consumer of CPU
> > cycles at 28%.
> >
> > CPU: Intel Architectural Perfmon, speed 2665.98 MHz (estimated)
> > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
> > unit mask of 0x00 (No unit mask) count 100000
> > samples  %        image name               app name
> > symbol name
> > 918089   27.9310
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           pvclock_clocksource_read
> > 217811    6.6265  domain1-modules          domain1-modules
> > /domain1-modules
> > 188327    5.7295  vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-
> debug
> > vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug
> > mutex_spin_on_owner
> > 186684    5.6795
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           __xen_spin_lock
> > 149514    4.5487
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           __write_lock_failed
> > 123278    3.7505
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           __kernel_text_address
> > 122906    3.7392
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           xen_spin_unlock
> > 90903     2.7655
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           __spin_time_accum
> > 85880     2.6127
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           __module_address
> > 75223     2.2885
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           print_context_stack
> > 66778     2.0316
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           __module_text_address
> > 57389     1.7459
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           is_module_text_address
> > 47282     1.4385  xen-syms-4.1-unstable    domain1-xen
> > syscall_enter
> > 47219     1.4365
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           prio_tree_insert
> > 46495     1.4145  vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-
> debug
> > vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug
> > pvclock_clocksource_read
> > 44501     1.3539
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           prio_tree_left
> > 32482     0.9882
> > vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-5.11.dcinco-debug
> > domain1-kernel           native_read_tsc
> >
> > I ran oprofile (0.9.5 with xenoprofile patch) for 20 seconds while
> the
> > I/Os were running. Here's the command I used:
> >
> > opcontrol --start --xen=/boot/xen-syms-4.1-unstable
> > --vmlinux=/boot/vmlinux-2.6.32.25-pvops-stable-dom0-5.7.dcinco-debug
> > --passive-domains=1
> > --passive-images=/boot/vmlinux-2.6.36-rc7-pvops-kpcif-08-2-domu-
> > 5.11.dcinco-debug
> >
> > I had to remove dom0_max_vcpus=1 (but kept dom0_vcpus_pin=true) in
> the
> > Xen command line. Otherwise, oprofile only gives the samples from
> > CPU0.
> >
> > I'm going to try perf next.
> >
> > - Dante
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>