Re: [Question] Unexpected growth of nr_cpu in `credit` Xen scheduler?

[Juergen Gross <jgross@xxxxxxxx>]

On 26.09.25 09:17, Igor Korkin wrote:
> Hi all,
>
> I'm observing a steady and abnormal increase in the `nr_cpu` value
> reported by the `credit` Xen scheduler
> (visible via `sudo xl dmesg; sudo xl debug-keys r`).
>
>
> This behavior occurs consistently when the system is subjected to
> heavy synthetic load (e.g., multiple VMs running stress workloads that
> fully saturate vCPUs).
> Over time, `nr_cpu` grows far beyond the actual number of physical or
> logical CPUs (48 in our case), and this correlates with noticeable
> performance degradation, especially under high VM density.
Can you please share the related output? Especially one instance of
the "xl debug-keys r; xl dmesg" before the test is starting, and one
with the wrong number of cpus in the data.
> We’re running on a dual-socket x86_64 server (2 × 12-core Intel Xeon
> Silver 4310 CPUs with Hyper-Threading, total 48 logical CPUs) under
> Xen 4.19.
>
> Is this growth of `nr_cpu` expected in the credit scheduler?
> If not, it may indicate a bug in CPU accounting or runqueue management
> that warrants further investigation.
>
>
> Environment details:
> - xen_version            : 4.19.0-5.25.0.38431
> - xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_641
> - xen_scheduler          : credit
> - Hardware : Dual-socket Intel Xeon Silver 4310 @ 2.10GHz (12
> cores/socket, HT enabled, 48 logical CPUs)
> - NUMA nodes : 2
> - Dom0 kernel : Debian 6.1.147-1 (6.1.0-38-amd64, SMP PREEMPT_DYNAMIC)
> - nr_cpus                : 48
> - nr_nodes               : 2
> - release                : 6.1.0-38-amd64
> - version                : #1 SMP PREEMPT_DYNAMIC Debian 6.1.147-1 (2025-08-02)
> - machine                : x86_64
> - nr_nodes               : 2
> - cores_per_socket       : 12
> - threads_per_core       : 2
> - cpu_mhz                : 2100.000
> - virt_caps              : pv hvm hvm_directio pv_directio hap shadow
> iommu_hap_pt_share vmtrace gnttab-v1 gnttab-v2
> - total_memory           : 130724
> - free_memory            : 54064
Please share the complete "xl info" output, including the xen command line.

Are there any cpupools involved? Any cpu hotplug operations?


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature