Re: [Question] Unexpected growth of nr_cpu in `credit` Xen scheduler?

[Igor Korkin <igor.korkin@xxxxxxxxx>]

Thank you for the answer

The information about xl debug-keys r I will provide soon.

Here is the output of `xl info`:

developer@host:~$ sudo xl info
host                   : host
release                : 6.1.0-38-amd64
version                : #1 SMP PREEMPT_DYNAMIC Debian 6.1.147-1 (2025-08-02)
machine                : x86_64
nr_cpus                : 48
max_cpu_id             : 47
nr_nodes               : 2
cores_per_socket       : 12
threads_per_core       : 2
cpu_mhz                : 2100.002
hw_caps                :
bfebfbff:77fef3ff:2c100800:00000121:0000000f:f3bfbfff:00405f4e:00000100
virt_caps              : pv hvm hvm_directio pv_directio hap shadow
iommu_hap_pt_share vmtrace gnttab-v1 gnttab-v2
total_memory           : 130724
free_memory            : 54067
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 19
xen_extra              : .3-pre
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p
hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Tue Jan 30 21:22:03 2024 +0100 git:199f7986eb-dirty
xen_commandline        : placeholder hap_1gb=0 hap_2mb=0 altp2m=1
hpet=legacy-replacement sched=credit dom0_mem=-54068M,max:-54068M
dom0_max_vcpus=48 dom0_vcpus_pin=1 force-ept=1 ept=ad=0 max_cstate=0
dom0-iommu=passthrough=1 watchdog=force watchdog_timeout=20
spec-ctrl=no no-real-mode edd=off
cc_compiler            : x86_64-linux-gnu-gcc (Debian 12.2.0-14+deb12u1) 12.2.0
cc_compile_by          : pkg-xen-devel
cc_compile_domain      : lists.alioth.debian.org
cc_compile_date        : Tue Sep 23 08:15:41 UTC 2025
build_id               : c3666203ea8a9657a1bcf35f963e9371e5cb51dd
xend_config_format     : 4


developer@host:~$ sudo xl cpupool-list
Name               CPUs   Sched     Active   Domain count
Pool-0              48   credit       y


developer@host:~$ sudo dmesg | grep -i hotplug
[    3.167091] smpboot: Allowing 48 CPUs, 0 hotplug CPUs


Thank you!


On Fri, Sep 26, 2025 at 11:04 AM Juergen Gross <jgross@xxxxxxxx> wrote:
>
> On 26.09.25 09:17, Igor Korkin wrote:
> > Hi all,
> >
> > I'm observing a steady and abnormal increase in the `nr_cpu` value
> > reported by the `credit` Xen scheduler
> > (visible via `sudo xl dmesg; sudo xl debug-keys r`).
> >
> >
> > This behavior occurs consistently when the system is subjected to
> > heavy synthetic load (e.g., multiple VMs running stress workloads that
> > fully saturate vCPUs).
> > Over time, `nr_cpu` grows far beyond the actual number of physical or
> > logical CPUs (48 in our case), and this correlates with noticeable
> > performance degradation, especially under high VM density.
>
> Can you please share the related output? Especially one instance of
> the "xl debug-keys r; xl dmesg" before the test is starting, and one
> with the wrong number of cpus in the data.
> > We’re running on a dual-socket x86_64 server (2 × 12-core Intel Xeon
> > Silver 4310 CPUs with Hyper-Threading, total 48 logical CPUs) under
> > Xen 4.19.
> >
> > Is this growth of `nr_cpu` expected in the credit scheduler?
> > If not, it may indicate a bug in CPU accounting or runqueue management
> > that warrants further investigation.
> >
> >
> > Environment details:
> > - xen_version            : 4.19.0-5.25.0.38431
> > - xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32
> > hvm-3.0-x86_32p hvm-3.0-x86_641
> > - xen_scheduler          : credit
> > - Hardware : Dual-socket Intel Xeon Silver 4310 @ 2.10GHz (12
> > cores/socket, HT enabled, 48 logical CPUs)
> > - NUMA nodes : 2
> > - Dom0 kernel : Debian 6.1.147-1 (6.1.0-38-amd64, SMP PREEMPT_DYNAMIC)
> > - nr_cpus                : 48
> > - nr_nodes               : 2
> > - release                : 6.1.0-38-amd64
> > - version                : #1 SMP PREEMPT_DYNAMIC Debian 6.1.147-1 
> > (2025-08-02)
> > - machine                : x86_64
> > - nr_nodes               : 2
> > - cores_per_socket       : 12
> > - threads_per_core       : 2
> > - cpu_mhz                : 2100.000
> > - virt_caps              : pv hvm hvm_directio pv_directio hap shadow
> > iommu_hap_pt_share vmtrace gnttab-v1 gnttab-v2
> > - total_memory           : 130724
> > - free_memory            : 54064
> Please share the complete "xl info" output, including the xen command line.
>
> Are there any cpupools involved? Any cpu hotplug operations?
>
>
> Juergen