[release-4.20] OCPBUGS-78193: Exclude disruption during NoExecuteTaintManager serial tests#30858
Conversation
The upstream Kubernetes NoExecuteTaintManager serial test applies NoExecute taints to worker nodes where its test pods are scheduled. This evicts all pods on those nodes that lack a matching toleration, including metrics-server replicas. When both metrics-server pods are evicted simultaneously, the metrics API (an aggregated API proxied through kube-apiserver) returns 503 for ~25-30s until replacement pods become ready. With 3 worker nodes and metrics-server running 2 replicas on 2 of them, there is a ~22% chance (2/9) that both test pods land on the metrics-server nodes, causing both replicas to be evicted at once. The existing P99-based disruption threshold does not account for this because serial jobs are a small fraction of total runs in the historical data, so the P99 is dominated by non-serial jobs that never encounter this test. The result is a very low baseline (~0-1s) with 5s of grace, which cannot absorb a 25-30s deterministic outage. This is not a product bug — the NoExecuteTaintManager test is intentionally designed to evict pods without tolerations. Filter out disruption intervals that overlap with the NoExecuteTaintManager test window so this expected disruption is not counted against the threshold. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@openshift-cherrypick-robot: Detected clone of Jira Issue OCPBUGS-78191 with correct target version. Will retitle the PR to link to the clone. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-78193, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: openshift-cherrypick-robot The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@openshift-cherrypick-robot: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
This is an automated cherry-pick of #30855
/assign openshift-ci-robot