Skip to content

Avoid heap-allocating background processor futures#4733

Open
Abeeujah wants to merge 1 commit into
lightningdevkit:mainfrom
Abeeujah:drop-box
Open

Avoid heap-allocating background processor futures#4733
Abeeujah wants to merge 1 commit into
lightningdevkit:mainfrom
Abeeujah:drop-box

Conversation

@Abeeujah

Copy link
Copy Markdown
Contributor
The persistence loop in process_events_async boxed each task future
(ChannelManager, network graph, scorer, sweeper, LiquidityManager)
before handing it to the Joiner, incurring a heap allocation per task
on every iteration. Box::pin was only needed to satisfy the Joiner's
Unpin bound when our MSRV predated core::pin::pin!.

Now that the MSRV is 1.75, replace Box::pin with core::pin::pin! so the
futures live on the stack. Pin<&mut F> is Unpin, so the Joiner bounds
still hold. To pin every future before the Joiner borrows it, build all
five uniformly at the end of the iteration: run the synchronous timer
checks and side effects first, capture their outcomes in flags, and
gate each future's work behind those flags. Futures whose work is
skipped resolve to Ok(()) immediately, preserving the previous behavior
where an unset Joiner slot was treated as Ready(Ok(())).

The eager single poll of the ChannelManager future and the early
loop-exit (break) semantics on the scorer/sweeper timers are unchanged.

@ldk-reviews-bot

ldk-reviews-bot commented Jun 20, 2026

Copy link
Copy Markdown

I've assigned @valentinewallace as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-claude-review-bot

ldk-claude-review-bot commented Jun 20, 2026

Copy link
Copy Markdown
Collaborator

I've re-examined the full refactored persistence loop against the original control flow. Verified:

  • needs_cm_persist flag-gating + eager CM poll (lines 1123-1162): pending_monitor_writes is captured inside the async body during the eager poll, which runs after get_and_clear_needs_persistence() — race window preserved.
  • Synchronous side effects (network graph prune at 1212, have_pruned/time_passed/scorer stepping) all run before futures are constructed, matching original ordering.
  • break semantics on scorer/sweeper timers (lines 1255, 1264) and check order preserved.
  • Skipped-work futures resolve Ok(()), matching the old unset-slot semantics.
  • Pin-moves into Joiner after eager poll are sound.

No new issues found beyond what was raised in the prior review pass (the scorer_fut formatting note at lib.rs:1284-1285, already noted, remains non-blocking).

No issues found.

Comment thread lightning-background-processor/src/lib.rs Outdated
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @valentinewallace! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

// Captured by the ChannelManager persistence future below. Only meaningful when
// `needs_cm_persist` is set, declared before the futures so it outlives them
// (drop order).
let pending_monitor_writes =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can stay in the if/future and avoid the if.

let mut waker = dummy_waker();
let mut ctx = task::Context::from_waker(&mut waker);
match core::pin::Pin::new(&mut fut).poll(&mut ctx) {
task::Poll::Ready(res) => futures.set_a_res(res),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe there's any reason to change this, it would probably be simpler to keep the code structured the way it was.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've simplified it, although, I couldn't fully restore the original now that the futures are stack-pinned, Joiner holds Pin<&mut> references into them and so must be declared after all of them.

Meaning it doesn't exist yet at the eager-poll site, so the outcome has to be stashed and fed in once the Joiner is constructed. Open to a cleaner approach if you have any recommendations.

Replace Box::pin with core::pin::pin! in process_events_async now that
MSRV is 1.75. This eliminates a heap allocation per task on every
loop iteration by pinning the futures directly to the stack.

To satisfy lifetime and Joiner bounds, the loop logic was refactored
to run synchronous timer checks first, using flags to conditionally
execute the stack-pinned futures. Existing eager polling and early-break
semantics are preserved.
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.76471% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.25%. Comparing base (b6f3ad7) to head (583404e).
⚠️ Report is 3215 commits behind head on main.

Files with missing lines Patch % Lines
lightning-background-processor/src/lib.rs 86.76% 7 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4733      +/-   ##
==========================================
+ Coverage   84.54%   86.25%   +1.70%     
==========================================
  Files         137      160      +23     
  Lines       77617   111579   +33962     
  Branches    77617   111579   +33962     
==========================================
+ Hits        65625    96241   +30616     
- Misses       9949    12694    +2745     
- Partials     2043     2644     +601     
Flag Coverage Δ
tests 86.25% <86.76%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants