bug: CLI fix for --load-pattern + --target-qps by viraatc · Pull Request #237 · mlcommons/endpoints

viraatc · 2026-04-01T21:56:51Z

What does this PR do?

Fixes CLI crash when --load-pattern + --target-qps are used together (IndexError: tuple index out of range), and adds test coverage to prevent regressions.

Bug fix

LoadPattern.type used alias= instead of name= on cyclopts.Parameter, and class was missing @cyclopts.Parameter(name="*") — caused cyclopts to fail resolving --load-pattern into a config key path.

Test coverage

test_cli.py: Hypothesis fuzz tests auto-discover all CLI flags from assemble_argument_collection() and test 4000 random combinations (up to 10 flags each) across offline + online/poisson + online/concurrency. Validated: catches this bug in 1.62s.
test_benchmark_command.py: Added test_concurrency_benchmark with streaming on/off — all 3 execution modes now covered.
hypothesis==6.151.10 added to test deps, schema_fuzz pytest marker.

CI & tooling

schema-updated CI job: triggers on PRs touching schema.py/config.py/cli.py — runs fuzz tests + validates YAML templates.
regenerate_templates.py: auto-generates YAML templates from schema defaults + overrides. Pre-commit hook regenerates locally on schema.py changes (skipped in CI).
Templates excluded from prettier to avoid formatting conflicts.

Type of change

Bug fix
Tests added/updated

github-actions · 2026-04-01T21:57:02Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Copilot

Pull request overview

Fixes a cyclopts CLI parsing crash triggered when --load-pattern is combined with load-pattern subfields like --target-qps / --concurrency in the online benchmark command.

Changes:

Annotates LoadPattern to adjust how cyclopts maps nested parameters (@cyclopts.Parameter(name="*")).
Updates the CLI parameter definition for LoadPattern.type to avoid the prior name collision.

Comments suppressed due to low confidence (1)

src/inference_endpoint/config/schema.py:360

This change is a regression fix for a CLI crash when combining --load-pattern with nested load-pattern fields (e.g. --target-qps). There’s existing automated test coverage for config validation in tests/unit/commands/test_benchmark.py, but no test currently exercises cyclopts parsing for this flag combination.

Add a regression test that parses benchmark online ... --load-pattern poisson --target-qps 100 (or directly parses OnlineBenchmarkConfig via cyclopts) and asserts it no longer raises and that config.settings.load_pattern.type/target_qps are set as expected.

@cyclopts.Parameter(name="*")
class LoadPattern(BaseModel):
    """Load pattern configuration.

    Different patterns use target_qps differently:
    - max_throughput: target_qps used for calculating total queries (offline, optional with default)
    - poisson: target_qps sets scheduler rate (online, required - validated)
    - concurrency: issue at fixed target_concurrency (online, required - validated)
    """

    model_config = ConfigDict(extra="forbid", frozen=True)

    type: Annotated[
        LoadPatternType,
        cyclopts.Parameter(name="--load-pattern", help="Load pattern type"),
    ] = LoadPatternType.MAX_THROUGHPUT
    target_qps: Annotated[
        float | None, cyclopts.Parameter(alias="--target-qps", help="Target QPS")
    ] = Field(None, gt=0)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/schema.py

gemini-code-assist

Code Review

This pull request modifies the LoadPattern class in the configuration schema by applying a class-level cyclopts.Parameter decorator and updating the type field's parameter definition to use the name argument instead of alias. I have no feedback to provide.

arekay-nv

Can we also add a test for this - seems like a change that shouldn't have gone in.

tests/integration/commands/test_cli.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/test.yml

tests/integration/commands/test_cli.py

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/test.yml

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.pre-commit-config.yaml

viraatc

duplicate

tests/integration/commands/test_cli.py

src/inference_endpoint/config/templates/offline_template.yaml

.github/workflows/test.yml

.pre-commit-config.yaml

viraatc · 2026-04-03T11:51:28Z

Review Council — Multi-AI Code Review

Reviewed by: Claude (Codex ran but produced investigation output, not structured findings) | Depth: standard

Found 3 issues across 3 files:

1 high (fixed)
1 medium (already fixed)
1 low (deferred)

#	File	Line	Severity	Category	Summary
1	`scripts/regenerate_templates.py`	95	high	error-handling	Pre-commit hook exited 0 on template generation failure — stale files could slip through. Fixed: now tracks failures and `sys.exit(1)`.
2	`.github/workflows/test.yml`	61	medium	security	Unpinned action SHAs in `schema-updated` job. Already fixed in latest push.
3	`tests/integration/commands/test_cli.py`	76	low	testing	`Optional` union types (`float

Also addressed all Copilot review comments (pinned SHAs, quoted pip install, heredoc for inline Python, expanded pre-commit files: regex, added except comment).

viraatc

added new schema-updated CI:

fuzz tests on CLI in CI
template validated against schema default in CI

NOTE: template now includes all supported fields

was pending items from past.
++ @rashid for thoughts?

viraatc · 2026-04-03T12:16:44Z

tests/integration/commands/test_cli.py

+@pytest.mark.schema_fuzz
+@pytest.mark.slow
+@hyp_settings(max_examples=2000, deadline=5000)
+@given(tokens=online_tokens())


Fuzz test catches 53f08fc

The bug caused --load-pattern poisson --target-qps 100 to crash:

$ inference-endpoint benchmark online \ --endpoints http://localhost:8000 --model m --dataset d.pkl \ --load-pattern poisson --target-qps 100 IndexError: tuple index out of range

Reverted the fix and ran this test — Hypothesis finds it in 1.62s:

E IndexError: tuple index out of range E Falsifying example: test_online_cli_no_crash( E tokens=['benchmark', 'online', '--endpoints', 'http://h:80', E '--model', 'm', '--dataset', 'd.pkl', E '--load-pattern', 'poisson', '--target-qps', '100', E '--name', 'test-val'], E ) ============================== 1 failed in 1.62s ===============================

viraatc · 2026-04-03T12:16:54Z

src/inference_endpoint/config/templates/offline_template.yaml

+type: offline
 model_params:
-  name: "meta-llama/Llama-3.1-8B-Instruct"
+  name: '<MODEL_NAME eg: meta-llama/Llama-3.1-8B-Instruct>'


Templates auto-generated from schema defaults by scripts/regenerate_templates.py.
Full YAML spec with placeholder overrides (model name, dataset)

Pre-commit validates templates are valid locally.
CI checks if they're up to date — if stale it will suggest to, run python scripts/regenerate_templates.py.

Is this overkill? Should we drop?

This creates more burden to the end user to understand all the flags, and we should keep the template simple. cc: @arekay-nv to review as well

I think this makes reproduction easier. Only the flags with values in <> need to be specified, all others have defaults.
We can even make this part of a pre-commit to have an integration test that takes in these templates, substitutes for an echo server just to make sure that the minimal version runs.

For beginners, they can always use the minimal command line which already has the defaults baked in.

viraatc · 2026-04-03T12:17:02Z

.github/workflows/test.yml

          pip install -e ".[dev,test,performance]"
          pip-audit
+
+  schema-updated:


new schema-updated CI job:
triggers on PRs touching schema.py/config.py/cli.py.

viraatc · 2026-04-03T12:17:11Z

.pre-commit-config.yaml

-      - id: validate-templates
-        name: Validate YAML templates against schema
-        entry: python -c "from pathlib import Path; from inference_endpoint.config.schema import BenchmarkConfig; [BenchmarkConfig.from_yaml_file(f) for f in sorted(Path('src/inference_endpoint/config/templates').glob('*.yaml'))]"
+      - id: check-templates


reuse --check mode

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/inference_endpoint/config/templates/concurrency_template.yaml

tests/integration/commands/test_cli.py

.github/workflows/test.yml

docs/DEVELOPMENT.md

Bug: LoadPattern.type had alias= instead of name= on cyclopts.Parameter, and class was missing @cyclopts.Parameter(name="*"). This caused any CLI invocation with --load-pattern to crash with IndexError. Tests: - Hypothesis fuzz tests auto-discover all CLI flags from cyclopts assemble_argument_collection() and test 4000 random combinations (offline + online/poisson + online/concurrency) - Added test_concurrency_benchmark with streaming on/off - hypothesis==6.151.10 added to test deps, schema_fuzz pytest marker CI & tooling: - schema-updated CI job: fuzz tests + template validation on schema changes - regenerate_templates.py: auto-generates YAML templates from schema defaults - Pre-commit checks templates are up to date (--check mode) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 1, 2026 21:56

viraatc requested a review from a team as a code owner April 1, 2026 21:56

github-actions bot requested review from arekay-nv and nvzhihanj April 1, 2026 21:57

Copilot started reviewing on behalf of viraatc April 1, 2026 21:57 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

src/inference_endpoint/config/schema.py Show resolved Hide resolved

gemini-code-assist bot reviewed Apr 1, 2026

View reviewed changes

arekay-nv approved these changes Apr 3, 2026

View reviewed changes

github-code-quality bot found potential problems Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Fixed Show fixed Hide fixed

viraatc force-pushed the feat/viraatc-fix1 branch from 90fe9c8 to 80a79ef Compare April 3, 2026 10:56

Copilot AI review requested due to automatic review settings April 3, 2026 10:56

Copilot started reviewing on behalf of viraatc April 3, 2026 10:56 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

github-code-quality bot found potential problems Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Dismissed Show dismissed Hide dismissed

Copilot AI review requested due to automatic review settings April 3, 2026 11:05

Copilot started reviewing on behalf of viraatc April 3, 2026 11:06 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.github/workflows/test.yml Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 3, 2026 11:32

Copilot started reviewing on behalf of viraatc April 3, 2026 11:32 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

This comment was marked as duplicate.

Sign in to view

viraatc commented Apr 3, 2026

View reviewed changes

tests/integration/commands/test_cli.py Show resolved Hide resolved

src/inference_endpoint/config/templates/offline_template.yaml Outdated Show resolved Hide resolved

.github/workflows/test.yml Show resolved Hide resolved

.pre-commit-config.yaml Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings April 3, 2026 12:14

Copilot started reviewing on behalf of viraatc April 3, 2026 12:15 View session

viraatc force-pushed the feat/viraatc-fix1 branch from 8915750 to ffb87d9 Compare April 3, 2026 12:16

viraatc commented Apr 3, 2026

View reviewed changes

Copilot AI reviewed Apr 3, 2026

View reviewed changes

viraatc force-pushed the feat/viraatc-fix1 branch from ffb87d9 to b781ff7 Compare April 3, 2026 12:21

Merge branch 'main' into feat/viraatc-fix1

637f449

Conversation

viraatc commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Bug fix

Test coverage

CI & tooling

Type of change

Uh oh!

github-actions bot commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

arekay-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

viraatc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

viraatc commented Apr 3, 2026

Review Council — Multi-AI Code Review

Uh oh!

viraatc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viraatc Apr 3, 2026

Choose a reason for hiding this comment

Fuzz test catches 53f08fc

Uh oh!

viraatc Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvzhihanj Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

arekay-nv Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

viraatc Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viraatc commented Apr 1, 2026 •

edited

Loading

github-actions bot commented Apr 1, 2026 •

edited

Loading

viraatc left a comment •

edited

Loading

viraatc left a comment •

edited

Loading

Fuzz test catches `53f08fc`

viraatc Apr 3, 2026 •

edited

Loading

viraatc Apr 3, 2026 •

edited

Loading

viraatc Apr 3, 2026 •

edited

Loading