feat: add schema-compare command to test harness

[markphelps]

Summary

Adds a schema-compare command to the test harness (tools/test-harness/) that compares static (Go tree-sitter) vs runtime (Python) schema generation for regression detection.

What's new

--cog-ref flag

Build and test cog from any git ref (branch, tag, or commit SHA) without manual builds. Clones the repo, builds the CLI binary via go build and the SDK wheel via uv build, then uses both automatically:

# Test schema gen from main
uv run cog-test schema-compare --no-gpu --cog-ref main

# Test from a feature branch
uv run cog-test schema-compare --no-gpu --cog-ref feat/new-schema

# Test from a specific commit
uv run cog-test run --no-gpu --cog-ref abc1234

Resolution priority:

• CLI binary: --cog-binary > --cog-ref > --cog-version > manifest default > latest stable
• SDK: --sdk-wheel > --cog-ref (auto-built) > --sdk-version > manifest default > latest stable

Parallel schema builds

The static and runtime builds now run concurrently using distinct Docker image tags (:static and :runtime), roughly halving wall-clock time per model.

How it works

For each model, schema-compare:

1. Builds with COG_STATIC_SCHEMA=1 and without — in parallel
2. Extracts OpenAPI schemas from the Docker image labels
3. Compares the two JSON schemas for exact equality
4. Reports a structured diff on mismatch showing the exact JSON paths that diverge

cd tools/test-harness

# Compare all non-GPU models (latest release)
uv run cog-test schema-compare --no-gpu

# Compare against main branch
uv run cog-test schema-compare --no-gpu --cog-ref main

# Compare a specific fixture with a local binary
uv run cog-test schema-compare --model fixture-scalar-types --cog-binary /path/to/cog

Local fixture models

7 local fixture models (fixtures/models/) covering the full input type matrix:

Fixture	Coverage
scalar-types	str, int, float, bool, Secret
optional-types	PEP 604 X | None and Optional[X] for all types
list-types	list[X] and List[X] for str, int, Path, File
optional-list-types	list[X] | None and Optional[List[X]]
constraints-and-choices	ge/le constraints, string/int choices
file-path-types	Path, File, optional Path/File
complex-output	BaseModel structured output

These use repo: local in the manifest and are loaded from fixtures/models/ without cloning.

Example output

Schema Comparison Report (CLI ref:main)
=======================================

  + fixture-scalar-types      schemas match (static 12.3s, runtime 11.8s)
  + fixture-optional-types    schemas match (static 11.5s, runtime 11.2s)
  x fixture-complex-output    schemas differ
      $.components.schemas.Output.properties.score.title: value mismatch
        static:  "Score"
        runtime: "score"

[ask-bonk]

After reviewing the PR thoroughly, I found the code to be well-structured with proper error handling, good type annotations, and comprehensive documentation. The schema comparison logic is sound, resource cleanup is handled correctly with finally blocks, and the fixture models provide good coverage of input types.

LGTM

github run

[ask-bonk]

After reviewing the PR thoroughly, I found the code to be well-structured with proper error handling, good type annotations, and comprehensive documentation. The schema comparison logic is sound, resource cleanup is handled correctly with finally blocks, and the fixture models provide good coverage of input types.

LGTM

github run

[ask-bonk]

After reviewing the PR thoroughly, I found the code to be well-structured with proper error handling, good type annotations, and comprehensive documentation. The schema comparison logic is sound, resource cleanup is handled correctly with finally blocks, and the fixture models provide good coverage of input types.

LGTM

github run

[ask-bonk]

After thoroughly reviewing all the changes in PR #2891, I found the code to be well-structured and properly implemented:

Summary: This PR adds a schema-compare command to the test harness that compares static (Go tree-sitter) vs runtime (Python) schema generation, with parallel builds, git ref support, and comprehensive fixture models.

Key findings:

• Clean implementation with proper error handling and resource cleanup
• Good type annotations throughout Python code
• Parallel execution implemented correctly using ThreadPoolExecutor
• 7 fixture models provide comprehensive input type coverage
• Documentation is clear and complete
• Minor bug fix in pkg/schema/openapi.go (OpenAPI spec compliance)
• No breaking changes or security concerns

LGTM

github run