maintainer,scheduler: add dispatcher drain runtime

[hongyunyan]

What problem does this PR solve?

The dispatcher-drain work split out of #4190 still mixes maintainer runtime behavior, coordinator scheduling, and public API orchestration in one review unit. This PR extracts the maintainer-side dispatcher drain runtime so reviewers can focus on target consumption, progress reporting, and drain-aware scheduling inside one changefeed after the protocol layer from #4759.

Issue Number: ref #4190

What is changed and how it works?

Background:

• This is PR 2/4 in the dispatcher-drain split stack and is based on maintainer,heartbeatpb: add drain target plumbing #4759.
• After maintainer,heartbeatpb: add drain target plumbing #4759, the drain target can already be propagated and acknowledged at maintainer-manager scope.
• The remaining work in this layer is to make each maintainer consume that target and evacuate dispatchers without refilling the target node.

Motivation:

• Let reviewers focus on maintainer-local scheduling semantics without also reading coordinator scheduling or API orchestration.
• Keep maintainer progress reporting in the same review unit as the schedulers that drive it.
• Stabilize the integration-style manager tests against future scheduler default changes.

Summary:

• add shared maintainer DrainState and wire it into controller state
• make basic and balance scheduling avoid assigning work back to the active drain target
• add a dedicated maintainer drain scheduler and drain-aware move accounting
• add DrainProgress reporting to maintainer status
• add balance-move-batch-size to scheduler config
• keep manager tests on config.NewDefaultSchedulerConfig() so new scheduler defaults do not silently break them
• add focused scheduler and operator tests for drain behavior

How it works:

• maintainers consume the latest drain target snapshot from the manager layer
• the dedicated drain scheduler moves dispatchers away from the target node while basic and balance scheduling avoid sending new work back there
• maintainer status reports the remaining dispatcher count and in-flight drain moves for the current target epoch

Check List

Tests

• Unit test
  • go test ./maintainer/...
  • go test ./pkg/scheduler ./pkg/config

Questions

Will it cause performance regression or break compatibility?

This PR does not expose a new public API by itself. It only adds maintainer-local dispatcher drain runtime and bounded balancing behavior on top of the internal drain target plumbing from #4759.

Do you need to update user documentation, design documentation or monitoring documentation?

No additional user-facing documentation is needed for this split. It is an internal decomposition of the drain-capture implementation.

Release note

None

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign flowbehappy for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/config/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cdfbd8e6-ee69-4696-ad65-1ddaa4a16bb7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch split/pr-4190-2b-maintainer-drain-runtime

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces a drainScheduler to handle node evacuation by moving dispatchers away from a target node. It integrates this into the existing scheduling framework, ensuring that basic and balance schedulers avoid the drain target to prevent churn. It also adds configuration for BalanceMoveBatchSize and enhances status reporting to include drain progress. Feedback suggests using the operator controller's CountInflightDrainMovesFromNode method in the drainScheduler to eliminate redundant logic.

[gemini-code-assist]

The drainScheduler implements its own countInflightDrainMoves method which is identical to the newly added oc.CountInflightDrainMovesFromNode(target). Using the controller's method is more efficient as it avoids creating a temporary slice of all operators and iterates over the internal map directly under a read lock.

Suggested change

	drainSlots := s.fixedDrainMoveLimit - s.countInflightDrainMoves(target)
	drainSlots := s.fixedDrainMoveLimit - s.operatorController.CountInflightDrainMovesFromNode(target)

[gemini-code-assist]

This method is redundant with oc.CountInflightDrainMovesFromNode(origin node.ID) added in maintainer/operator/operator_controller.go. Consider removing this local implementation and using the one from the operator controller.