[wip] Add docs-driven E2E coverage for the playground#1260
Draft
whoiskatrin wants to merge 4 commits intomainfrom
Draft
[wip] Add docs-driven E2E coverage for the playground#1260whoiskatrin wants to merge 4 commits intomainfrom
whoiskatrin wants to merge 4 commits intomainfrom
Conversation
|
whoiskatrin
force-pushed
the
playground-testing-automation
branch
2 times, most recently
from
ac87386 to
afe7b5c
Compare
whoiskatrin
force-pushed
the
playground-testing-automation
branch
from
afe7b5c to
5a3a26c
Compare
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
whoiskatrin
changed the title
…isolation, gate CI on Playwright - Only generate uncovered stubs in testing.generated.spec.ts - Gitignore generated artifacts (manifest, coverage, generated spec) - Extract shared Scenario type to e2e/types.ts - Add state reset to core, supervisor, and readonly tests - Fix PORT → DEV_PORT naming in playwright.config.ts - Comment setWorkflowStepCount .evaluate() workaround - Use Playwright-idiomatic expect.poll for auto-scroll assertion - O(n) duplicate detection in coverage report - Handle empty generated spec gracefully - Gate main CI on Playwright E2E tests passing - Add Playwright report artifact upload on failure - Bump CI timeout to 30min - Move playground-e2e.yml to nightly/manual only (main CI covers PRs) - Simplify check:testing-sync (no git diff needed with gitignored files) - Update README to reflect gitignored generated files
Replace ~1,400 lines of hand-written Playwright specs with an AI-driven test runner that interprets testing.md scenarios directly at runtime. - Parse testing.md into typed Scenario objects - Use Workers AI (Llama 4 Scout) to translate natural-language actions/expectations into Playwright commands via accessibility snapshots - Single ai-runner.spec.ts creates one test() per scenario (113 total) - Auto-skip deployed-only scenarios - Delete 6 manual spec files, coverage tracker, generation scripts - Update CI to use existing CLOUDFLARE_API_TOKEN/CLOUDFLARE_ACCOUNT_ID - Remove sync-check job and check:playground-testing-sync step Adding a new test = editing testing.md. No Playwright code needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a docs-driven browser testing workflow for the playground, using
examples/playground/testing.mdas the source of truth for what should be covered.The main goal is to make it easy to keep the playground test suite in sync with the testing guide:
testing.mdregenerates a machine-readable manifest and coverage artifactstest.fixme()entriestesting.mdand generated artifacts driftWhat changed
Docs-driven test generation
Added generation/reporting scripts under
examples/playground/scripts/:generate-e2e-from-testing.tsgenerate-testing-coverage-report.tsprepare-e2e-deps.tsThese produce:
examples/playground/e2e/testing.manifest.jsonexamples/playground/e2e/generated/testing.generated.spec.tsexamples/playground/e2e/testing.coverage.jsonexamples/playground/e2e/testing.coverage.mdPlaywright setup and manual coverage
Added a dedicated playground Playwright setup under
examples/playground/e2e/:playwright.config.tshelpers.tsmanual/coverage.tsto map implemented specs back to scenario ids fromtesting.mdUI/testability improvements
Added stable test hooks (
data-testid) and a few UX/testability improvements across playground demos and shared components, including:This keeps the browser tests resilient and lets the coverage report map cleanly to real scenarios.
CI + package scripts
Added a new workflow:
.github/workflows/playground-e2e.ymlAdded/updated scripts so the common flows are:
npm run sync:playground-testingnpm run check:playground-testing-syncnpm run test:playground:e2eHow this works in practice
examples/playground/testing.mdnpm run sync:playground-testingCurrent coverage
Because email automation was intentionally removed from this branch, the current automated coverage is:
The remaining uncovered scenarios are the email flows still documented in
examples/playground/testing.md.Testing
I ran:
npm run test:e2e -w @cloudflare/agents-playgroundnpm run check:playground-testing-syncThe playground-specific sync check and local playground browser suite are passing.