resource_group: add client-side observability metrics for Controller

[JmPotato]

What problem does this PR solve?

Issue Number: ref #10488

What is changed and how does it work?

This is Phase 1 of the Resource Control observability improvements. It adds client-side metrics to the Resource Group Controller (client/resource_group/controller/) for better visibility into token consumption, limiter state, and throttle behavior.

Changes:

A1 — Token consumption histogram always-on + RU type breakdown

• Remove the enableControllerTraceLog guard on TokenConsumedHistogram so token consumption is always observable in production
• Add new consume_by_type counter with rru/wru labels for per-type breakdown
• Original histogram label signature unchanged (backward compatible with existing Grafana panels)

A2 — Limiter state gauges

• resource_manager_client_resource_group_token_balance — current available token balance
• resource_manager_client_resource_group_fill_rate — current effective fill rate (RU/s)
• resource_manager_client_resource_group_burst_limit — current burst limit
• Add GetFillRate() exported method to Limiter

A3 — Average RU consumption rate gauge

• resource_manager_client_resource_group_avg_ru_per_sec — EMA estimated consumption rate
• Combined with fill_rate, operators can detect consumption > refill before throttling occurs

A5 — Throttled state gauge

• resource_manager_client_resource_group_throttled — 1 if in trickle mode, 0 if normal

All new gauge metrics are observed in the 1Hz updateRunState() tick (not on the request hot path). Only the histogram observation (~200ns) runs on the request path.

Check List

Tests

• Unit test
• Existing tests verified (pre-existing failures confirmed on master)

Summary by CodeRabbit

• New Features
  • Extended monitoring with per-type resource consumption counters for read/write RU.
  • Added real-time limiter observability: token balance, fill rate, burst capacity, and throttling status.
  • Introduced average RU-per-second tracking for consumption visibility.
  • Consumption is now recorded consistently for both acquisition and response paths.
  • Automatically removes related metrics when a resource group is deleted.

[ti-chi-bot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign qiuyesuifeng for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Added Prometheus metrics and instrumentation for resource-group token consumption and limiter state: per-RU-type counters, consumption histogram, limiter gauges (token balance, fill rate, burst), avg RU/sec, throttled gauge, registration/cleanup, always-on consumption observation, and Limiter.GetFillRate.

Changes

Cohort / File(s)	Summary
Metrics Definition client/resource_group/controller/metrics/metrics.go	Added resource_group subsystem, renamed label constant to typeLabel, introduced exported metrics: TokenConsumedByTypeCounter, TokenBalanceGauge, FillRateGauge, BurstLimitGauge, AvgRUPerSecGauge, ThrottledGauge; registered them in InitAndRegisterMetrics.
Controller Integration client/resource_group/controller/group_controller.go	Initialized new metrics with labels; added observeConsumption(delta) helper; always observe token consumption (histogram + per-type counters) during acquisition and response handling; update limiter gauges (tokenBalance, fillRate, burstLimit, avgRUPerSec, throttled) on run-state refresh.
Limiter API & Locking client/resource_group/controller/limiter.go	Changed limiter mutex to sync.RWMutex for read accessors; converted IsLowTokens, GetBurst, AvailableTokens to use read lock; added exported GetFillRate() float64.
Global Cleanup client/resource_group/controller/global_controller.go	Extend resource-group cleanup to delete newly added per-group metric label values (histogram, per-type counters for rru/wru, and all new gauges).

Sequence Diagram(s)

sequenceDiagram
  participant Controller as Controller\n(group_controller.go)
  participant Limiter as Limiter\n(limiter.go)
  participant Metrics as Prometheus\n(metrics/metrics.go)

  Controller->>Limiter: request tokens / compute delta
  Limiter-->>Controller: acquisition result
  Controller->>Metrics: observe consumeTokenHistogram(amount)
  Controller->>Metrics: increment TokenConsumedByTypeCounter{resource_group,type}
  Controller->>Limiter: read state (isThrottled, GetFillRate)
  Limiter-->>Controller: fillRate / state
  Controller->>Metrics: set TokenBalanceGauge, FillRateGauge, BurstLimitGauge, AvgRUPerSecGauge, ThrottledGauge

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

• resource control: improve observability for Controller and Server modules #10488: Implements the same client-side observability improvements (per-type consumption, limiter gauges, avg RU/sec, throttled, and cleanup).

Possibly related PRs

• resourcegroup: replace blind sleep with signal-aware wait in acquireTokens #10252: Touches the same acquireTokens path and Limiter interactions; likely related at the code level.

Suggested labels

release-note-none, type/development

Suggested reviewers

• rleungx
• disksing
• nolouch

Poem

🐇 I hopped through tokens, gauges in my paw,
Counters and fill-rates now hum like a law,
RRU, WRU — each counted in rhyme,
Throttles and averages ticking in time,
Hop — metrics bloom, observability for all.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly describes the main change: adding client-side observability metrics to the Resource Group Controller.
Description check	✅ Passed	The description includes issue reference, clear explanation of changes (A1-A5), implementation details, and confirms unit tests are included.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}