Async channel manager persistence in background processor#4419
Draft
joostjager wants to merge 4 commits intolightningdevkit:mainfrom
Draft
Async channel manager persistence in background processor#4419joostjager wants to merge 4 commits intolightningdevkit:mainfrom
joostjager wants to merge 4 commits intolightningdevkit:mainfrom
Conversation
Pure refactor: move the bodies of Watch::watch_channel and Watch::update_channel into methods on ChainMonitor, and have the Watch trait methods delegate to them. This prepares for adding deferred mode where the Watch methods will conditionally queue operations instead of executing them immediately. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a `deferred` parameter to `ChainMonitor::new` and `ChainMonitor::new_async_beta`. When set to true, the Watch trait methods (watch_channel and update_channel) will unimplemented!() for now. All existing callers pass false to preserve current behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the unimplemented!() stubs with a full deferred write implementation. When ChainMonitor has deferred=true, Watch trait operations queue PendingMonitorOp entries instead of executing immediately. A new flush() method drains the queue and forwards operations to the internal watch/update methods, calling channel_monitor_updated on Completed status. The BackgroundProcessor is updated to capture pending_operation_count before persisting the ChannelManager, then flush that many writes afterward - ensuring monitor writes happen in the correct order relative to manager persistence. Key changes: - Add PendingMonitorOp enum and pending_ops queue to ChainMonitor - Implement flush() and pending_operation_count() public methods - Integrate flush calls in BackgroundProcessor (both sync and async) - Add TestChainMonitor::new_deferred, flush helpers, and auto-flush in release_pending_monitor_events for test compatibility - Add create_node_cfgs_deferred for deferred-mode test networks - Add unit tests for queue/flush mechanics and full payment flow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of joining the ChannelManager persistence future with scorer/graph/sweeper writes, track it as a CmPersistState and poll it via a new Selector variant (G). This lets CM persistence run across loop iterations concurrently with event processing and network I/O. Key changes: - Add generic G future to Selector and SelectorOutput::G variant - Generalize OptionalSelector to propagate the inner future's Output - Remove the A slot from Joiner (now only used for scorer/graph/sweeper/ liquidity persistence) - Introduce CmPersistState to track in-flight CM persistence alongside the number of deferred monitor writes captured at persist start - On async CM persist completion, flush deferred monitor writes and continue back to event processing - On shutdown, await any in-flight CM persist before the final persist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
👋 Hi! I see this is a draft PR. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4419 +/- ##
==========================================
+ Coverage 86.06% 86.08% +0.01%
==========================================
Files 156 156
Lines 103188 103755 +567
Branches 103188 103755 +567
==========================================
+ Hits 88808 89313 +505
- Misses 11868 11920 +52
- Partials 2512 2522 +10
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #4351, this moves ChannelManager persistence into the
Selectorloop so it runs concurrently with event processing and network I/O, rather than being joined with scorer/graph/sweeper writes. CM persistence is tracked as aCmPersistStatepolled via a newSelectorvariant, letting it span across loop iterations. On completion, deferred monitor writes are flushed before resuming event processing.Benchmark results (simple A->B payments, 150 ms write latency):
The deferred result is expected because channel manager writing still blocks monitor writing, which holds off event processing. The speed up for immediate mode comes from CM persistence no longer blocking event processing.
Open question: the 25% improvement only applies to immediate (non-deferred) monitor writing. For the time being, we use deferred writing (for persistence ordering guarantees), and that mode shows no improvement yet. Should we pursue this PR?