[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend


  • To: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
  • From: Julian Vetter <julian@xxxxxxxxxxxxxxxx>
  • Date: Mon, 9 Nov 2026 11:16:06 +0100
  • Arc-authentication-results: i=1; strato.com; arc=none; dkim=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1794219368; s=strato-dkim-0002; d=strato.com; h=In-Reply-To:From:References:Cc:To:Subject:Date:Message-ID:Cc:Date: From:Subject:Sender; bh=z1tFFIkKDjY4xYW1PtNGBh62gyp6wCd07sTLJtfhRp4=; b=iBV54vo/NoN5wM2xVSGV7GLBiw6NcCypt1BkXWz6mix97Uk+SPHYnqvSTND229WaDB zsNAkvxcwnNXir7xfPyu3Z5c11udVYMb4lGVynjkpQeAyLC6p4pCEh9YSatNRBqVTgOg o1JAKJ9vJhq1ld7fN8ufKePFHZP8ttZn/E5zQ70H+FJmIBTYiE6cWlrMXTifZoUFENtV LkWCnP/+emJ8F9hpqAn5lDjPYAT/Iu8SvVD89k9H6LxK/Sj5sIGVghw6X9vDyQW8aWnA HI6HmezT7fdCv9LtEYWXd2zAUYev/r/HOb6boG97XgTSMSuSL6lg9Jfe/uzwYyIboNpy X2Jw==
  • Arc-seal: i=1; a=rsa-sha256; t=1794219368; cv=none; d=strato.com; s=strato-dkim-0002; b=Ly5CoXLr6I+bRApne+yZ5VfYEOur9JERXKloz51nArrfXuODkvcUiBvwjOb0RJqqAr y+6zx/tmpuX7jHH2pt1+vR4i94Ff/5k0FgtLYVirrIFegnTjkMsvu7bZ56w5UQHmyeDq 9sRGuKrIVOVSrAhNjANUgeRDFOkCPNKxiV3NBc06yhroqUm7C7gJ+x0zkMrBYCxkvg4R KDzGvZ6NNzkbjmJL6WWXjy53zb9WGUlf73B9oTJ6ZpmtIThVGK0Y04dU9xlrvEDWJUWM 82QNdxK0b/YdFohRzHG8LEidG7z8TmicoOGLi+ZUiZwJwFLGXwgLr1NNSLc9LzJTY1yt PObQ==
  • Authentication-results: eu.smtp.expurgate.cloud; dkim=pass header.s=strato-dkim-0002 header.d=outer-limits.org header.i="@outer-limits.org" header.h="In-Reply-To:From:References:Cc:To:Subject:Date:Message-ID"; dkim=permerror header.s=strato-dkim-0003 header.d=outer-limits.org header.i="@outer-limits.org"
  • Cc: xen-devel@xxxxxxxxxxxxxxxxxxxx, linux-kernel@xxxxxxxxxxxxxxx, Juergen Gross <jgross@xxxxxxxx>, Thomas Gleixner <tglx@xxxxxxxxxx>, Peter Zijlstra <peterz@xxxxxxxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>, Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>, Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>, Jiayuan Chen <jiayuan.chen@xxxxxxxxx>, "Masami Hiramatsu (Google)" <mhiramat@xxxxxxxxxx>, "Vineeth Pillai (Google)" <vineeth@xxxxxxxxxxxxxxx>, Adrian Hunter <adrian.hunter@xxxxxxxxx>
  • Delivery-date: Mon, 09 Nov 2026 10:18:48 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 11/6/26 4:39 PM, Sebastian Andrzej Siewior wrote:
On 2026-11-06 15:57:40 [+0100], Julian Vetter wrote:
We hit this issue with an Ubuntu 26.04 cloudimage guests (7.0 based
kernel with the 8250 as an nbcon console) on XCP-ng 8.3 / Xen 4.17. With
a kernel commandline adding two consoles, i.e., 'console=ttyS1
console=ttyS0'. During a live migration on that setup the printk()s from
gnttab_resume() and xen_irq_resume() queue nbcon_irq_work while the APIC
is disabled. Thus the list transitions from empty to non-empty, but
doesn't receive the IPI to empty the list again. Afterwards the list
stays non-empty, so the IPI never fires and all irq_work piles up.

The Commit 560f4deda327 ("printk/nbcon: Change nbcon_irq_work to
IRQ_WORK_LAZY") removed this particular issue because declaring the IRQ
work to be lazy, causes the timer interrupt to trigger the irq work
instead. But any other non-lazy irq_work queued in that window is still
lost the same way and will cause the list to fill up without being
handled.

Adrian has this:
    https://lore.kernel.org/all/20261102074039.26983-2-adrian.hunter@xxxxxxxxx/

I sort of asked for flush on suspend but Adrian pointed out the NMI
problem.

Thanks, I missed that. It is the same issue, just a different trigger. Adrian's example even ends in the same hung task we see (fsnotify_mark_destroy_workfn waiting on SRCU). Also, Adrian's syscore_resume() placement would cover the Xen case as well, as xen_suspend() calls syscore_resume() after those printk()s.


Looking at #1, you could probably limit it to a nop on
!arch_irq_work_has_interrupt(). The tick should not be disabled if
irq_work_needs_cpu() returns true (you don't add anything).


Yes, right, the lazy part is pointless.

And lazy items get handled via irq_work_tick() anyway. This make me
think if it gets any easier if we handle both in irq_work_tick(). On x86
raised_list is evaluated anyway so by dropping
arch_irq_work_has_interrupt() we would avoid the APIC check and handle
list in this rare case. If the IRQ-work is working then the list remains
empty.


This is maybe a cleaner solution, it needs no APIC access and no call site in the suspend paths. But irq_work_needs_cpu() would need the same change, otherwise in a CONFIG_NOHZ=y setting an idle CPU would still sit on the work until it leaves idle and re-enables the timer tick, no? I will test this.

Signed-off-by: Julian Vetter <julian.vetter@xxxxxxxxxx>

Sebastian


(Sorry, sending from my personal address, my work mail server currently mangles mails to the list.)

Julian



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.