fix(penpal): eliminate unnecessary git check-ignore processes and full project walks#549
fix(penpal): eliminate unnecessary git check-ignore processes and full project walks#549
Conversation
Add semaphore (cap 4) to RefreshAllProjects and CheckAllProjectsHasFiles to prevent unbounded git check-ignore subprocess spawning at startup and during workspace rescans. Tighten the watcher Route 6 filter to only pass .md file events. Previously, all Create events (including scanner temp files and backup artifacts) would pass through and trigger a full project walk with a git check-ignore process. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ResolveFileInfo for single-file source resolution without walking the filesystem or spawning git check-ignore processes. Applies the same source-priority, SkipDirs, RequireSibling, and ClassifyFile rules as scanProjectSources. Add UpsertFile and RemoveFile methods on Cache for incremental mutations. UpsertFile updates ModTime/Title for existing entries or resolves source membership for new files. RemoveFile removes entries by project-relative path. Both update project metadata (HasFiles, LastModified). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the per-project debounceRefresh (which triggered full RefreshProject walks) with an accumulating debounce that collects per-file paths and ops during the 100ms window, then applies incremental UpsertFile/RemoveFile mutations. This eliminates filesystem walks and git check-ignore subprocess spawning for the common case of editing .md files. The watcher now handles Write → re-stat + re-title, Create → resolve source + insert, and Remove/Rename → remove from cache, all without touching the filesystem beyond the specific file that changed. Add IsProjectScanned method to support the guard that skips incremental updates for projects that haven't been lazily scanned yet. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RescanWith now preserves cached file data for unchanged projects instead of rescanning everything. It diffs old vs new project lists, compares source configurations via SourcesChanged, and only walks projects that are genuinely new or whose sources changed. Removed projects are cleaned up from the cache. handleDeleteFile uses RemoveFile for incremental cache removal instead of a full RefreshProject walk. refreshAfterConfigChange no longer calls populateProjects (which redundantly ran CheckAllProjectsHasFiles). Instead it does targeted git enrichment only for projects missing git info. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a file event fires for a project that hasn't been lazily scanned yet, fall back to a full RefreshProject instead of silently dropping the event. The previous guard skipped incremental updates entirely, which meant file changes were invisible until the user navigated to the project page to trigger the lazy scan. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 798066e873
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9489020eb0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
loganj
force-pushed
the
penpal-git-check-ignore-timing
branch
from
9489020 to
137a93e
Compare
outer-loop statusFeedback addressed — 1 thread(s) resolved.
Updated 2026-03-30T18:55:10.296Z |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 137a93e287
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
loganj
force-pushed
the
penpal-git-check-ignore-timing
branch
from
137a93e to
0760f2b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0760f2b404
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
loganj
force-pushed
the
penpal-git-check-ignore-timing
branch
from
0760f2b to
07acc3d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 07acc3d18d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
loganj
force-pushed
the
penpal-git-check-ignore-timing
branch
from
07acc3d to
cd0ad63
Compare
Remove gitIgnoreChecker from projectHasAnyMarkdown — this lightweight startup check no longer spawns a subprocess per project. A false positive from .md files in gitignored dirs is harmless since the full scan on first access applies proper filtering. Remove IsProjectScanned guard from flushFileEvents — incremental updates now apply regardless of scan state, avoiding fallback to full RefreshProject which spawned git check-ignore. The lazy scan on first access reconciles. Fix flaky TestSSE_BroadcastDelivery by draining background events before asserting the expected broadcast. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
loganj
force-pushed
the
penpal-git-check-ignore-timing
branch
from
cd0ad63 to
bd0a2d5
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bd0a2d593e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if hasSkippedDir(relToSource, st) { | ||
| continue |
There was a problem hiding this comment.
ResolveFileInfo does not apply the same unconditional .git directory exclusion that scanProjectSources uses, so incremental updates can admit markdown files from .git when they receive fsnotify events. In this path, filtering relies on SkipDirs, but __all_markdown__ does not include .git, so a file like .git/notes.md is accepted and cached even though a full scan would skip it. This creates inconsistent results between incremental and full refreshes and can surface phantom files until a later rescan.
Useful? React with 👍 / 👎.
1 participant
Eliminates `git check-ignore` subprocess spawns from the hot path by replacing full-project filesystem walks with targeted incremental cache mutations.
Key changes:
Covered by 15 new unit tests for `UpsertFile`, `RemoveFile`, `ResolveFileInfo`, `RescanWith`, `SourcesChanged`, and `projectHasAnyMarkdown`.
Addresses PENPAL-2225.