Re: [Xen-devel] Poor HVM performance with 8 vcpus

[Tim Deegan <Tim.Deegan@xxxxxxxxxx>]

At 10:40 +0100 on 07 Oct (1254912041), Juergen Gross wrote:
> calls to shadow_alloc              438     427     424     480     436     422
> number of shadow pages in use     2765    2151    2386    2509    4885    1391
> calls to shadow_free               168     132     185     144     181     105
> calls to shadow_fault            65271   69132   60495   53756   73363   52449
> shadow_fault fast path n/p        7347    8081    6713    6134    8521    6112
> shadow_fault fast path error        14      12      15       3      13      11
> shadow_fault really guest fault  24004   25723   22815   19709   27049   19190
> shadow_fault emulates a write     1045     949    1018     995    1015     901
> shadow_fault fast emulate          424     361     449     348     387     314
> shadow_fault fixed fault         32503   34264   29624   26689   36641   26096
> calls to shadow_validate_gl2e      875     748     917     731     795     667
> calls to shadow_validate_gl3e      481     456     443     491     489     446
> calls to shadow_validate_gl4e      104      97      95     112     105      95
> calls to shadow_hash_lookup    2109654 2203254 2228896 2245849 2164727 2309059
> shadow hash hit in bucket head 2012828 2111164 2161113 2177591 2104099 2242458
> shadow hash misses                 851     840     841     910     852     838
> calls to get_shadow_status     2110031 2202828 2228769 2246689 2164213 2309241
> calls to shadow_hash_insert        438     436     428     481     437     430
> calls to shadow_hash_delete        168     150     185     154     202     128
> shadow removes write access        335     324     329     385     330     336
> shadow writeable: linux high       130     139     152     155     138     149
> shadow writeable: sl1p           14508   15402   12961   11823   16474   11472
> shadow writeable brute-force       205     185     177     230     192     187
> shadow unshadows for fork/exit       9      12      12      12      18      12
> shadow unshadows a page             10      13      13      13      19      13
> shadow walks guest tables       647527  727336  649397  646601  659655  621289
> shadow checks gwalk                526     544     535     550     614     554
> shadow flush tlb by rem wr perm    235     233     229     268     238     237
> shadow emulates invlpg           14688   15499   14604   12630   16627   11370
> shadow OOS fixup adds            14467   15335   13059   11840   16624   11339
> shadow OOS unsyncs               14467   15335   13058   11840   16624   11339
> shadow OOS evictions               566     449     565     369     589     336
> shadow OOS resyncs               14510   15407   12964   11828   16478   11481
> 
> I don't think the "shadow writable brute-force" is the problem.
> get_shadow_status seems to be a more critical candidate.

get_shadow_status is a simple hash lookup to find the shadow of a frame;
it's expected to happen multiple times per pagefault.  Even so those
numbers look high.  ~10k guest PT walks per CPU per second, each causing
3-4 shadow hash lookups.  That's much higher than the number of
pagefaults.  

I take it you reset the performance counters at the start of that run?

Are there any other numbers (outside the shadow stats) that are up
around 600k/cpu?  

I wonder whether this is caused by pagetables changing under out feet in
the shadow fault handler -- in order to avoid taking the shadow lock too
often, the fault handler walks the pagetables first, then takes the lock
and double-checks its work.   If the other cpus are aggressively writing
to the pagetables that this CPU is running on that could cause the
pagefault handler to spin, locking and unlocking the shadow lock as it
goes.   I would expect to see the OOS unsyncs number much higher if that
was the case though.  

Can you try the attached patch and see if it makes any difference?

Tim.

-- 
Tim Deegan <Tim.Deegan@xxxxxxxxxx>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

rewalk-with-lock
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel