Fix possible race condition with reentrant callback groups in EventsCBGExecutor scheduler (backport #3234)

[mergify]

Description

Since #3178, although the setting of handle->in_queue = true was protected under ready_callback_groups_mutex, the read is not. This makes the following races possible with more than one worker thread:

if handle->in_queue is loaded stale false while another thread enqueues it and marks handle->in_queue = true, a duplicate handle can be pushed. Removing the callback group or node from a running executor later on can lead to a possible use-after-free if checking for work on the remaining duplicate (dangling) callback handle.

In a more rare case, if the check for handle->in_queue is loaded stale true by the last worker thread to touch it, while another thread sets handle->in_queue = false, a reentrant CBG could have ready work but not be enqueued and stay permanently starved during the lifetime of the executor.

Fixed by ensuring that handle->in_queue is checked under ready_callback_groups_mutex.

Also adds a check in remove_callback_group so that it is only removed from ready_callback_groups if it was found in the queue. previously if it wasn't ready when being removed, std::find could return ready_callback_groups.end() and unconditionally erase that from ready_callback_groups, which is undefined behavior.

Is this user-facing behavior change?

fixes the possibility of duplicate reentrant groups being pushed to the queue, or a push being skipped indefinitely if the last pending worker thread loads a stale in_queue = true

Did you use Generative AI?

This came up as a possible race while running a large benchmark sweep, partially instrumented by Claude Opus 5

Additional Information

---

This is an automatic backport of pull request #3234 done by [Mergify](https://mergify.com).

[skyegalaxy]

Pulls: #3237
Gist: https://gist.githubusercontent.com/skyegalaxy/81742caa1e7b86621f4ab0fb59e523bc/raw/ca14e451cc9dff23250986e01222fec16092d566/ros2.repos
BUILD args: --packages-above-and-dependencies rclcpp
TEST args: --packages-above rclcpp
ROS Distro: kilted
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/20176

• Linux
• Linux-aarch64
• Linux-rhel
• Windows