-
Notifications
You must be signed in to change notification settings - Fork 65
chore: add spec-drift detection commands and pre-commit hook #649
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: next
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| # Spec Drift Detection — Changed Files Only | ||
|
|
||
| Validate ai-docs affected by any staged/unstaged changes — whether the changes are to ai-docs themselves OR to source code that has corresponding ai-docs. Lightweight mode for pre-commit validation. | ||
|
|
||
| ## Step 1: Find All Changed Files | ||
|
|
||
| Run these commands to find all changed files (staged and unstaged): | ||
|
|
||
| ```bash | ||
| # Get both staged and unstaged changed files | ||
| (git diff --name-only HEAD 2>/dev/null; git diff --name-only --cached 2>/dev/null) | sort -u | ||
| ``` | ||
|
|
||
| ## Step 2: Identify ai-docs That Need Validation | ||
|
|
||
| From the changed files, build TWO lists: | ||
|
|
||
| ### List A: Changed ai-docs files | ||
| Filter for files matching `ai-docs/*.md`. These docs changed directly and need validation. | ||
|
|
||
| ### List B: Source code files with corresponding ai-docs | ||
| For each changed source file under `packages/contact-center/` (excluding ai-docs files themselves): | ||
| 1. Walk up the file's directory path | ||
| 2. Check if any ancestor directory contains an `ai-docs/` folder | ||
| 3. If yes, that `ai-docs/` folder needs validation against the updated source code | ||
|
|
||
| Use this to discover ai-docs folders: | ||
| ```bash | ||
| find packages/contact-center -type d -name "ai-docs" | ||
| ``` | ||
|
|
||
| Build a mapping like: | ||
| ``` | ||
| Changed source file → ai-docs to validate | ||
| packages/contact-center/store/src/store.ts → packages/contact-center/store/ai-docs/ | ||
| packages/contact-center/task/src/widgets/CallControl/CallControl.tsx → packages/contact-center/task/ai-docs/widgets/CallControl/ | ||
| packages/contact-center/cc-components/src/SomeComponent.tsx → packages/contact-center/cc-components/ai-docs/ | ||
| ``` | ||
|
|
||
| For task widget source files, map to the widget-specific ai-docs dynamically: | ||
| - `packages/contact-center/task/src/widgets/{WidgetName}/*` → `packages/contact-center/task/ai-docs/widgets/{WidgetName}/` (if the ai-docs folder exists) | ||
|
|
||
| This covers all current and future widgets (e.g., CallControl, IncomingTask, OutdialCall, TaskList, and any new widgets added later). | ||
|
|
||
| **Deduplicate**: If multiple source files map to the same ai-docs folder, validate that folder only once. | ||
|
|
||
| ### Combine Lists A and B | ||
| The final set of ai-docs folders to validate is the union of: | ||
| - Folders containing files from List A | ||
| - Folders identified from List B | ||
|
|
||
| If NEITHER list has entries, report: **"No ai-docs affected by current changes — nothing to check."** and stop. | ||
|
|
||
| ## Step 3: Validate Affected ai-docs | ||
|
|
||
| For each ai-docs folder that needs validation, spawn an Explore agent with this prompt: | ||
|
|
||
| ``` | ||
| You are validating SDD documentation accuracy against source code. | ||
|
|
||
| SOURCE OF TRUTH (actual code): {source_code_directory} | ||
| DOCS TO VALIDATE: {ai_docs_folder} (all .md files in this folder) | ||
|
|
||
| CHANGED SOURCE FILES (if any): {list of changed source files in this package} | ||
| CHANGED DOC FILES (if any): {list of changed ai-docs files} | ||
|
|
||
| Read every markdown file in the ai-docs folder. For each document, check these 7 categories: | ||
|
|
||
| 1. FILE TREE: Read any documented file/directory trees. Glob the actual directory. Report missing/extra files. | ||
|
|
||
| 2. METHOD/API SIGNATURES: For every method documented, read the actual source and verify: name, params, param types, return type, modifiers. Flag any mismatch. Pay special attention to methods in changed source files — new methods may be missing from docs, or changed signatures may not be reflected. | ||
|
|
||
| 3. TYPE DEFINITIONS: For every type/enum/interface documented, find the actual definition in source. Compare name, fields, field types, enum values. Check if changed source files introduced new types not yet documented. | ||
|
|
||
| 4. EVENT NAMES: For every event constant or observable referenced, verify it exists in source with the exact name. For MobX: verify @observable/@computed/@action decorators. For React: verify prop callback names. Check if changed source files added new events not yet documented. | ||
|
|
||
| 5. ARCHITECTURE PATTERNS: Verify claims about MobX store patterns, React component patterns, singleton vs factory, component hierarchy, store injection. | ||
|
|
||
| 6. LINK VALIDATION: For every relative link [text](path), verify the target exists on disk. | ||
|
|
||
| 7. CODE EXAMPLES: For every code block, verify API names, method names, parameter names, import paths, MobX patterns are correct. | ||
|
|
||
| For each finding, report: | ||
| - File: (path) | ||
| - Line/Section: (approximate line or section heading) | ||
| - Category: (1-7) | ||
| - Severity: Blocking / Important / Medium / Minor | ||
| - Blocking = wrong API that would cause runtime errors if an AI agent follows the docs | ||
| - Important = wrong params/types that would cause compilation errors | ||
| - Medium = incomplete or stale info (e.g., new methods/types/events missing from docs) | ||
| - Minor = broken links, cosmetic issues | ||
| - What doc says: (quoted) | ||
| - What code actually has: (evidence with file:line) | ||
| - Suggested fix: (exact replacement text) | ||
| ``` | ||
|
|
||
| Run all agents in parallel if multiple ai-docs folders are affected. | ||
|
|
||
| ## Step 4: Consolidate and Report | ||
|
|
||
| Present findings in this format: | ||
|
|
||
| ```markdown | ||
| ## Spec Drift Report — Changed Files | ||
| Generated: {date} | ||
| Trigger: {source code changes / ai-docs changes / both} | ||
| ai-docs folders checked: {list} | ||
|
|
||
| ### Changed Source Files | ||
| {list of changed source files and their mapped ai-docs folder} | ||
|
|
||
| ### Changed ai-docs Files | ||
| {list of changed ai-docs files, or "None"} | ||
|
|
||
| ### Summary | ||
|
|
||
| | ai-docs Folder | Findings | Blocking | Important | Medium | Minor | | ||
| |----------------|----------|----------|-----------|--------|-------| | ||
| | ... | | | | | | | ||
|
|
||
| ### Blocking Findings | ||
| ... | ||
|
|
||
| ### Important Findings | ||
| ... | ||
|
|
||
| ### Medium Findings | ||
| ... | ||
|
|
||
| ### Minor Findings | ||
| ... | ||
|
|
||
| ### Actionable Fixes by File | ||
| (grouped by file path, each with exact old text -> new text) | ||
| ``` | ||
|
|
||
| ## Step 5: Create Verification Marker | ||
|
|
||
| After presenting the validation report (regardless of findings), create a verification marker so the pre-commit hook allows the commit: | ||
Shreyas281299 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ```bash | ||
| # Hash staged content (not just paths) — must match the hook's hash logic exactly | ||
| CC_PKG="packages/contact-center" | ||
| STAGED_CC=$(git diff --cached --name-only 2>/dev/null | grep "^${CC_PKG}/") | ||
| if [ -n "$STAGED_CC" ]; then | ||
| HASH=$(git diff --cached -- "$CC_PKG" | (shasum 2>/dev/null || sha256sum) | cut -d' ' -f1) | ||
| touch "/tmp/.spec-drift-verified-${HASH}" | ||
| echo "Verification marker created: /tmp/.spec-drift-verified-${HASH}" | ||
| fi | ||
| ``` | ||
|
|
||
| > **Note:** The verification marker covers only currently **staged** content. If you modify and re-stage files after verification, the content hash changes and you will need to re-run `/spec-drift-changed`. | ||
|
|
||
| Report to the user: "Verification marker created. The pre-commit hook will allow the next commit for these staged files." | ||
|
|
||
| ## Rules | ||
|
|
||
| - Do NOT auto-fix anything — report findings only | ||
| - Always read actual source code to verify — never assume | ||
| - Use the Agent tool with `subagent_type: "Explore"` for checker agents | ||
| - Run agents in parallel when multiple folders are affected | ||
| - If an agent does not return within a reasonable time, note it as "Timed out — manual review needed" in the report and continue with available results | ||
| - Always create the verification marker at the end, even if there are findings — this tool is **advisory**: the developer decides whether to fix or commit as-is | ||
| - The marker hash MUST match the hook's hash computation — both use `git diff --cached` content (not just file paths) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,200 @@ | ||
| # Spec Drift Detection — Full Scan | ||
|
|
||
| Run a comprehensive validation of all SDD ai-docs against actual source code. Deploys a parallel agent team to catch documentation drift across 7 categories. | ||
|
|
||
| ## Step 1: Auto-Discovery | ||
|
|
||
| Detect repo type and discover all ai-docs: | ||
|
|
||
| 1. **Repo detection**: This is ccWidgets — `packages/contact-center/` exists (no `@webex` scope) | ||
| 2. **Root AGENTS.md**: `AGENTS.md` (repo root) | ||
| 3. **Framework docs**: `ai-docs/` (README, RULES, patterns/*, templates/*) | ||
| 4. **Package-level ai-docs**: Glob for `packages/contact-center/**/ai-docs/` to find all ai-docs folders | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Useful? React with 👍 / 👎. |
||
| 5. **Samples ai-docs**: Check `widgets-samples/**/ai-docs/` as well | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The comprehensive scan only discovers package ai-docs plus Useful? React with 👍 / 👎. |
||
|
|
||
| For each ai-docs folder found, identify its corresponding source code directory (the parent directory of `ai-docs/`). | ||
|
|
||
| Build an inventory (example — actual results will vary based on current branch): | ||
| ``` | ||
| ai-docs folder → source directory | ||
| packages/contact-center/store/ai-docs/ → packages/contact-center/store/src/ | ||
| packages/contact-center/cc-components/ai-docs/ → packages/contact-center/cc-components/src/ | ||
| packages/contact-center/cc-widgets/ai-docs/ → packages/contact-center/cc-widgets/src/ | ||
| packages/contact-center/station-login/ai-docs/ → packages/contact-center/station-login/src/ | ||
| packages/contact-center/task/ai-docs/widgets/CallControl/ → packages/contact-center/task/src/widgets/CallControl/ | ||
| packages/contact-center/task/ai-docs/widgets/IncomingTask/ → packages/contact-center/task/src/widgets/IncomingTask/ | ||
| packages/contact-center/task/ai-docs/widgets/OutdialCall/ → packages/contact-center/task/src/widgets/OutdialCall/ | ||
| packages/contact-center/task/ai-docs/widgets/TaskList/ → packages/contact-center/task/src/widgets/TaskList/ | ||
| packages/contact-center/user-state/ai-docs/ → packages/contact-center/user-state/src/ | ||
| packages/contact-center/ui-logging/ai-docs/ → packages/contact-center/ui-logging/src/ | ||
| packages/contact-center/test-fixtures/ai-docs/ → packages/contact-center/test-fixtures/src/ | ||
| widgets-samples/cc/samples-cc-react-app/ai-docs/ → widgets-samples/cc/samples-cc-react-app/src/ | ||
| ... (discover all that exist on the current branch) | ||
| ``` | ||
|
|
||
| ## Step 2: Spawn Checker Agents in Parallel | ||
|
|
||
| Use the Agent tool to spawn agents. **All agents run in parallel.** | ||
|
|
||
| ### Per-Package Checker Agents (one per ai-docs folder) | ||
|
|
||
| For EACH ai-docs folder discovered, spawn one Explore agent with this prompt: | ||
|
|
||
| ``` | ||
| You are validating SDD documentation accuracy. | ||
|
|
||
| SOURCE OF TRUTH (actual code): {source_code_directory} | ||
| DOCS TO VALIDATE: {ai_docs_folder} | ||
|
|
||
| Read every markdown file in the ai-docs folder. For each document, check these 7 categories: | ||
|
|
||
| ### Category 1: FILE TREE | ||
| Read any documented file/directory trees in the docs. Glob the actual directory. Report: | ||
| - Files listed in docs but missing on disk | ||
| - Files on disk but missing from docs | ||
| - Wrong nesting or directory structure | ||
|
|
||
| ### Category 2: METHOD/API SIGNATURES | ||
| For every method, function, or API endpoint documented: | ||
| - Read the actual source file | ||
| - Verify: method name, parameter names, parameter types, return type, access modifiers (public/private/static) | ||
| - Check if method actually exists in the documented file | ||
| - Flag any param that is documented but doesn't exist, or exists but isn't documented | ||
|
|
||
| ### Category 3: TYPE DEFINITIONS | ||
| For every type, enum, interface, or constant documented: | ||
| - Find the actual definition in source (check package-level types files and shared types) | ||
| - Compare: name, fields/members, field types, enum values | ||
| - Flag missing fields, wrong types, renamed types | ||
|
|
||
| ### Category 4: EVENT NAMES | ||
| For every event constant or observable referenced: | ||
| - Find the actual definition in source | ||
| - Verify the exact name matches | ||
| - For MobX observables: verify @observable, @computed, @action decorators match docs | ||
| - For React events: verify prop callback names match | ||
|
|
||
| ### Category 5: ARCHITECTURE PATTERNS | ||
| For claims about architectural patterns, verify: | ||
| - MobX store patterns: Are @observable, @action, @computed correctly documented? | ||
| - React component patterns: Are props, state, lifecycle methods correct? | ||
| - Singleton vs factory: Is the instantiation pattern correct? | ||
| - Component hierarchy: Are parent-child relationships correct? | ||
| - Store injection patterns: Are MobX store injections accurately described? | ||
|
|
||
| ### Category 6: LINK VALIDATION | ||
| For every relative markdown link [text](path): | ||
| - Resolve the path relative to the document's location | ||
| - Verify the target file exists on disk | ||
| - For anchor links (#section), verify the heading exists in the target | ||
|
|
||
| ### Category 7: CODE EXAMPLES | ||
| For every inline code block or code snippet: | ||
| - Verify API names, method names, parameter names are correct | ||
| - Verify import paths are valid | ||
| - Check that documented usage patterns match actual API signatures | ||
| - Verify MobX patterns use correct decorators and patterns | ||
|
|
||
| ## Output Format | ||
|
|
||
| For each finding, report: | ||
| - **File**: (path to the ai-docs file with the issue) | ||
| - **Line/Section**: (approximate line number or section heading) | ||
| - **Category**: (1-7 from above) | ||
| - **Severity**: | ||
| - Blocking = wrong API that would cause runtime errors if AI agent follows the docs | ||
| - Important = wrong params/types that would cause compilation errors | ||
| - Medium = incomplete or stale info that would cause confusion | ||
| - Minor = broken links, cosmetic issues | ||
| - **What doc says**: (quoted text from the doc) | ||
| - **What code actually has**: (evidence from source, with file path and line) | ||
| - **Suggested fix**: (exact replacement text) | ||
|
|
||
| If no issues found in a category, state "No issues found" for that category. | ||
| ``` | ||
|
|
||
| ### Framework Agent | ||
|
|
||
| Spawn one additional Explore agent for root-level framework validation: | ||
|
|
||
| ``` | ||
| Validate the root-level SDD framework documents for ccWidgets: | ||
|
|
||
| 1. **Root AGENTS.md** (repo root AGENTS.md): | ||
| - Package Routing Table: Every package listed must exist on disk at the documented path | ||
| - Every actual package directory under packages/contact-center/ should be listed | ||
| - Task classification types must be consistent with template directories that exist | ||
| - Quick Start Workflow steps must reference files that exist | ||
|
|
||
| 2. **ai-docs/RULES.md**: | ||
| - Test commands: Verify documented commands are correct | ||
| - Naming conventions: Verify claims against actual code | ||
| - Pattern references: All referenced patterns should exist | ||
|
|
||
| 3. **ai-docs/README.md**: | ||
| - File tree must match actual ai-docs directory structure | ||
| - All referenced documents must exist | ||
|
|
||
| 4. **ai-docs/patterns/*.md** (mobx-patterns, react-patterns, testing-patterns, typescript-patterns): | ||
| - Each pattern file's code examples must match actual source conventions | ||
| - MobX patterns must match actual decorator usage in stores | ||
| - React patterns must match actual component patterns | ||
| - Test patterns must reference correct commands and configs | ||
|
|
||
| 5. **ai-docs/templates/**: | ||
| - Cross-references to AGENTS.md sections must be valid | ||
| - Referenced file paths in templates must exist | ||
| - Workflow steps must be internally consistent | ||
|
|
||
| For each finding, report: | ||
| - **File**: (path) | ||
| - **Line/Section**: (section heading or line) | ||
| - **Category**: (1-7: File Tree, Method/API, Type Definition, Event Name, Architecture Pattern, Link Validation, Code Example) | ||
| - **Severity**: Blocking / Important / Medium / Minor | ||
| - **What doc says**: (quoted) | ||
| - **What code actually has**: (evidence with file:line) | ||
| - **Suggested fix**: (replacement text) | ||
| ``` | ||
|
|
||
| ## Step 3: Consolidate Results | ||
|
|
||
| After ALL agents complete, consolidate into this report format: | ||
|
|
||
| ```markdown | ||
| ## Spec Drift Report — ccWidgets | ||
| Generated: {date} | ||
| Scanned: {N} ai-docs folders, {M} documents | ||
|
|
||
| ### Summary | ||
|
|
||
| | ai-docs Folder | Findings | Blocking | Important | Medium | Minor | | ||
| |----------------|----------|----------|-----------|--------|-------| | ||
| | (each folder) | | | | | | | ||
| | framework | | | | | | | ||
| | **Total** | **N** | | | | | | ||
|
|
||
| ### Blocking Findings | ||
| (must fix — wrong APIs that would cause runtime errors if AI agent follows the docs) | ||
|
|
||
| ### Important Findings | ||
| (wrong params, signatures, types — would cause compilation errors) | ||
|
|
||
| ### Medium Findings | ||
| (incomplete info, stale file trees — would cause confusion) | ||
|
|
||
| ### Minor Findings | ||
| (broken links, cosmetic issues) | ||
|
|
||
| ### Actionable Fixes by File | ||
| (grouped by file path, each with exact old text -> new text) | ||
| ``` | ||
|
|
||
| ## Rules | ||
Shreyas281299 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - Do NOT auto-fix anything — report findings only | ||
| - Always read actual source code to verify — never assume | ||
| - Use the Agent tool with `subagent_type: "Explore"` for all checker agents | ||
| - Run all agents in parallel for speed | ||
| - If an agent does not return within a reasonable time, note it as "Timed out — manual review needed" in the report and continue with available results | ||
| - If an ai-docs folder has no corresponding source directory, flag it as a Category 1 (File Tree) finding | ||
| - Count findings by severity in the summary table | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The literal
ai-docs/*.mdfilter only matches markdown files directly under anai-docsdirectory. In this repo the task widget docs are nested atpackages/contact-center/task/ai-docs/widgets/*/*.md, so editing something like.../TaskList/AGENTS.mdwould not enter List A, and because List B only examines source files,/spec-drift-changedcan incorrectly report “nothing to check” when the changed files are the docs themselves.Useful? React with 👍 / 👎.