Re: [PATCH RFC v2 00/15] rcu-tasks: let preemption outside trampolines be a quiescent state

[Yafang Shao <laoar.shao@xxxxxxxxx>]

On Sat, Sep 12, 2026 at 2:54 AM Josef Bacik <josef@xxxxxxxxxxxxxx> wrote:
>
> v1: 
> https://lore.kernel.org/all/20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@xxxxxxxxxxxxxx/
>
> v1->v2:
> - Only walk the kprobe hash while the optimizer is actually waiting (Sashiko).
> - Re-check the kprobe jump window at every QS decision instead of once at
>   preemption time (AI review).
> - Updated Documentation/RCU for the new rule (AI review).
> - Added 14/15 and 15/15 to address Paul's comments.
> - Added a comment in trace_recursion.h per Steve.
> - No change for the arm64 ftrace_static_tramp_end report, the Kconfig
>   dependency already covers it (Sashiko).
> - Re-ran the x86-64 QEMU tests, still 0.2-0.3s and clean.
>
> --- Original email ---
>
> Tasks RCU only treats a voluntary context switch, usermode or idle as a
> quiescent state, because a preempted task may be sitting in a trampoline
> that is about to be freed. That was a fine trade when PREEMPT_NONE
> servers compiled Tasks RCU away and PREEMPT desktops rarely ran
> long-lived in-kernel loops. PREEMPT_LAZY changes both halves at once:
> Tasks RCU is now real on server configs, and cond_resched() is a no-op,
> so a CPU-bound kthread or kworker only ever loses the CPU by being
> preempted, which is exactly the event Tasks RCU refuses to count.
>
> The way this showed up for us was a cgroup writeback worker draining a
> very large cgwb for around eleven minutes on an arm64 box. Nothing wrong
> with that on its own, but a BPF program detach on another CPU went
> bpf_trampoline_update() -> ftrace_shutdown() -> synchronize_rcu_tasks()
> while holding trampoline_mutex, forty-odd tasks piled up behind the
> mutex, and the hung task detector panicked the machine. The kprobe jump
> optimizer is worse in principle: it does synchronize_rcu_tasks() under
> kprobe_mutex, text_mutex and cpus_read_lock(), so one long-running
> kthread can stall static key updates and CPU hotplug for its whole run.
> The current answer is to find each such loop and add
> cond_resched_tasks_rcu_qs() to it, which is the kind of annotation
> PREEMPT_LAZY was supposed to let us stop writing.

Hello Josef,

Thanks for the great work!

We've run into exactly the same issue on our production servers
running the 6.18.y stable kernel:

[   84.393333] INFO: task bpf-thp:10274 blocked for more than 28 seconds.
[   84.393880]       Not tainted 6.18.44-3 #3.infra
[   84.394214] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[   84.394759] task:bpf-thp      state:D stack:0     pid:10274
tgid:10274 ppid:10266  task_flags:0x400100 flags:0x00080001
[   84.394764] Call Trace:
[   84.394765]  <TASK>
[   84.394767]  __schedule+0x2c1/0x6f0
[   84.394772]  schedule+0x2a/0xb0
[   84.394775]  schedule_preempt_disabled+0x15/0x30
[   84.394777]  __mutex_lock.constprop.0+0x3a2/0x9c0
[   84.394781]  __mutex_lock_slowpath+0x13/0x20
[   84.394784]  mutex_lock+0x37/0x50
[   84.394786]  bpf_trampoline_get+0x2a/0x80
[   84.394791]  check_attach_btf_id+0x32c/0x3d0
[   84.394796]  ? __pfx_mem_cgroup_css_offline+0x10/0x10
[   84.394800]  bpf_check+0x4d6/0xf20
[   84.394804]  bpf_prog_load+0x4d7/0xb00
[   84.394808]  ? avc_has_perm+0x42/0xd0
[   84.394811]  __sys_bpf+0x729/0xd50
[   84.394815]  __x64_sys_bpf+0x1a/0x30
[   84.394816]  x64_sys_call+0x19e1/0x2190
[   84.394820]  do_syscall_64+0x67/0xe10
[   84.394823]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   84.394825] RIP: 0033:0x7f2aa7e3ee5d
[   84.394831] RSP: 002b:00007ffd2b3e2a88 EFLAGS: 00000246 ORIG_RAX:
0000000000000141
[   84.394834] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f2aa7e3ee5d
[   84.394835] RDX: 0000000000000094 RSI: 00007ffd2b3e2b10 RDI: 0000000000000005
[   84.394837] RBP: 0000000000000094 R08: 00007ffd2b3e2c30 R09: 0000000000000018
[   84.394838] R10: 0000000000000070 R11: 0000000000000246 R12: 00007ffd2b3e2b10
[   84.394840] R13: 00007ffd2b3e2b10 R14: 0000000000000005 R15: 00000000215804c0
[   84.394843]  </TASK>
[   84.394848] INFO: task bpf-thp:10274 is blocked on a mutex likely
owned by task kworker/1:1:80.
[   84.395463] task:kworker/1:1     state:I stack:0     pid:80
tgid:80    ppid:2      task_flags:0x4208060 flags:0x00080000
[   84.395468] Workqueue: events bpf_link_put_deferred
[   84.395474] Call Trace:
[   84.395475]  <TASK>
[   84.395477]  __schedule+0x2c1/0x6f0
[   84.395482]  schedule+0x2a/0xb0
[   84.395485]  schedule_timeout+0xea/0x100
[   84.395488]  ? trace_hardirqs_off+0x30/0x90
[   84.395492]  ? trace_hardirqs_on+0x2b/0xa0
[   84.395494]  __wait_for_common+0x94/0x1b0
[   84.395498]  ? __pfx_schedule_timeout+0x10/0x10
[   84.395502]  wait_for_completion_state+0x21/0x40
[   84.395505]  __wait_rcu_gp+0x135/0x140
[   84.395510]  ? 0xffffffffc03fed00
[   84.395514]  synchronize_rcu_tasks+0x56/0xb0
[   84.395517]  ? __pfx_call_rcu_tasks+0x10/0x10
[   84.395521]  ? __pfx_wakeme_after_rcu+0x10/0x10
[   84.395525]  ftrace_shutdown.part.0+0xd8/0x1f0
[   84.395530]  ? 0xffffffffc03fed00
[   84.395533]  unregister_ftrace_function+0x47/0x150
[   84.395537]  ? 0xffffffffc03fed00
[   84.395539]  unregister_ftrace_direct+0x49/0xc0
[   84.395542]  bpf_trampoline_update+0x448/0x4e0
[   84.395545]  ? __radix_tree_delete+0x84/0x100
[   84.395547]  bpf_trampoline_unlink_prog+0x8f/0x150
[   84.395550]  bpf_tracing_link_release+0x1a/0x50
[   84.395553]  bpf_link_free+0x57/0xd0
[   84.395555]  bpf_link_put_deferred+0x12/0x20
[   84.395558]  process_one_work+0x1b3/0x400
[   84.395561]  worker_thread+0x1a3/0x310
[   84.395564]  kthread+0x107/0x240
[   84.395567]  ? trace_hardirqs_on+0x2b/0xa0
[   84.395569]  ? __pfx_worker_thread+0x10/0x10
[   84.395571]  ? _raw_spin_unlock_irq+0x11/0x30
[   84.395574]  ? __pfx_kthread+0x10/0x10
[   84.395577]  ret_from_fork+0x10d/0x150
[   84.395580]  ? __pfx_kthread+0x10/0x10
[   84.395583]  ret_from_fork_asm+0x1a/0x30
[   84.395588]  </TASK>
[   93.199020] rcu_tasks_wait_gp: rcu_tasks grace period number 1
(since boot) is 40249 jiffies old.

We eventually traced the root cause to a problematic BPF program
looping inside do_check() in the BPF verifier, and we've worked around
it with the following change:

--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -20411,8 +20411,7 @@ static int do_check(struct bpf_verifier_env *env)
                if (signal_pending(current))
                        return -EAGAIN;

-               if (need_resched())
-                       cond_resched();
+               cond_resched_tasks_rcu_qs();

                if (env->log.level & BPF_LOG_LEVEL2 && do_print_state) {
                        verbose(env, "\nfrom %d to %d%s:",

We've also observed random Tasks RCU stalls caused by kcompactd, but
since they don't result in hung tasks, we haven't applied a workaround
for those yet.

Hopefully we can come up with a generic solution for these Tasks RCU stalls.

[...]

-- 
Regards
Yafang