Skip to content

docs: add benchmark segmentation guidance to prevent agent timeouts #163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LeeCampbell merged 1 commit into from
Mar 23, 2026
Merged

docs: add benchmark segmentation guidance to prevent agent timeouts #163

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion spec/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ This file provides guidance on where to find standards and specifications for th
| encoding, LEB128, DEFLATE, compression | [Histogram Encoding](./tech-standards/histogram-encoding.md) |
| log format, V2, persistence | [Histogram Encoding](./tech-standards/histogram-encoding.md) |
| xUnit, test, FluentAssertions | [Testing Standards](./tech-standards/testing-standards.md) |
| benchmark, performance, allocation, BenchmarkDotNet | [Testing Standards](./tech-standards/testing-standards.md) |
| benchmark, performance, allocation, BenchmarkDotNet, timeout | [Testing Standards](./tech-standards/testing-standards.md) |
| naming convention, XML docs, style | [Coding Standards](./tech-standards/coding-standards.md) |
| build, NuGet, AppVeyor, CI/CD | [Build System](./tech-standards/build-system.md) |
| milestone, issue, PR, GitHub | [GitHub CLI Reference](./tech-standards/github.md) |
Expand Down
22 changes: 20 additions & 2 deletions spec/tech-standards/testing-standards.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,12 +224,30 @@ Both levels of benchmark are required because:

### Running Benchmarks

> **Timeout warning:** A full benchmark run (all classes, all runtimes) can take **over 30 minutes**.
> Automated agents have a 30-minute iteration timeout and **must not** attempt a full suite in a single run.
> Always segment benchmark runs as described below.

**Segmentation strategies** (pick one or combine):

- **By benchmark class** — run one category at a time (e.g. encoding, recording, leading-zero-count)
- **By runtime** — restrict to a single target framework per run (e.g. `net8.0` only)
- **By filter** — use `--filter` to select specific benchmark methods

```bash
# Build in Release mode (required)
dotnet build HdrHistogram.Benchmarking/ -c Release

# Run specific benchmarks
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*'
# Run a SINGLE benchmark class (recommended for agents)
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*Recording32BitBenchmark*'
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*LeadingZeroCount64BitBenchmark*'

# Run benchmarks for a SINGLE runtime only
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*' --runtimes net8.0
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*' --runtimes net9.0

# Combine both: one class, one runtime (fastest, safest for agents)
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*Recording32BitBenchmark*' --runtimes net8.0

# Export results as JSON for comparison
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*' --exporters json
Expand Down
Loading