Skip to content

chore: replace Python test harness with Go implementation#2915

Draft
markphelps wants to merge 2 commits intomainfrom
test-harness-go
Draft

chore: replace Python test harness with Go implementation#2915
markphelps wants to merge 2 commits intomainfrom
test-harness-go

Conversation

@markphelps
Copy link
Copy Markdown
Contributor

Summary

  • replace tools/test-harness Python implementation with a full Go implementation (cmd/ + internal/ packages) while preserving manifest/fixture-driven workflows
  • add run, build, list, and schema-compare commands, including resolver/reporting/validator logic and new Go unit tests
  • convert Go tools under tools/ to standalone Go modules and update task invocations in mise.toml where needed

Validation

  • go test ./... (in tools/test-harness)
  • go run . list (in tools/test-harness)
  • go test ./... in each standalone tool module (tools/compatgen, tools/gendocs, tools/test-registry-util, tools/version, tools/weights-gen)

Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
if err != nil {
return "", fmt.Errorf("creating binary file: %w", err)
}
defer f.Close()
Copy link
Copy Markdown

@ask-bonk ask-bonk bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm Bonk, and I've done a quick review of your PR.

This PR replaces the Python test harness with a Go implementation, converting tools under tools/ to standalone Go modules. The implementation is solid overall with good package structure and proper use of errgroup for concurrent builds.

I found one actionable issue:

Unchecked error from g.Wait() - The error return from g.Wait() at line 270 is discarded. While the goroutines handle their own errors via captured variables, the WaitGroup itself can return context cancellation errors that would be lost.

Comment on lines +268 to +270
})

g.Wait()
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unchecked error from g.Wait(). Context cancellation errors would be lost here.

Suggested change
})
g.Wait()
if err := g.Wait(); err != nil {
result.Passed = false
result.Error = fmt.Sprintf("context cancelled: %v", err)
return result
}

@ask-bonk
Copy link
Copy Markdown

ask-bonk bot commented Apr 7, 2026

remote: Permission to replicate/cog.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/replicate/cog.git/': The requested URL returned error: 403

github run

@ask-bonk
Copy link
Copy Markdown

ask-bonk bot commented Apr 7, 2026

@markphelps Bonk workflow failed. Check the logs for details.

View workflow run · To retry, trigger Bonk again.

Copy link
Copy Markdown

@ask-bonk ask-bonk bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm Bonk, and I've done a quick review of your PR.

This PR replaces the Python test harness with a Go implementation, converting tools under tools/ to standalone Go modules. The implementation follows Go best practices with good package separation and proper use of errgroup for concurrency.

I found two actionable issues:

  1. Unchecked error from g.Wait() - The error return from g.Wait() is discarded. While goroutines handle their own errors via captured variables, the WaitGroup itself can return context cancellation errors that would be lost.

  2. File close without error handling - The file handle opened with os.OpenFile is closed with defer f.Close() which ignores errors. For a file opened with write permissions, close errors could indicate data loss. Although the checksum verification would likely catch this, explicit error handling is preferred.

return nil
})

g.Wait()
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unchecked error from g.Wait(). Context cancellation errors would be lost here.

Suggested change
g.Wait()
if err := g.Wait(); err != nil {
result.Passed = false
result.Error = fmt.Sprintf("context cancelled: %v", err)
return result
}

if err != nil {
return "", fmt.Errorf("creating binary file: %w", err)
}
defer f.Close()
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File handle opened for writing should have explicit close error handling. Consider using named returns and a deferred function to capture close errors:

func downloadCogBinary(tag string) (dest string, err error) {
    // ... existing code ...
    
    f, err := os.OpenFile(dest, os.O_CREATE|os.O_WRONLY, 0755)
    if err != nil {
        return "", fmt.Errorf("creating binary file: %w", err)
    }
    defer func() {
        if closeErr := f.Close(); closeErr != nil && err == nil {
            err = fmt.Errorf("closing binary file: %w", closeErr)
        }
    }()
    
    // ... rest of function ...
}

@ask-bonk
Copy link
Copy Markdown

ask-bonk bot commented Apr 8, 2026

Review submitted successfully to PR #2915.

The review includes suggestions for:

  1. runner.go:270 - Handle the error from g.Wait() to catch context cancellation
  2. resolver.go:270 - Handle file close errors explicitly using named returns and deferred closure

github run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant