diff --git a/apps/penpal/ERD.md b/apps/penpal/ERD.md
index 1089932b..f713d1a5 100644
--- a/apps/penpal/ERD.md
+++ b/apps/penpal/ERD.md
@@ -98,10 +98,10 @@ see-also:
## Cache
-- **E-PENPAL-CACHE**: An in-memory cache (`sync.RWMutex`-protected) holds the full project list and per-project file lists. `RefreshProject()` walks the filesystem; `RefreshAllProjects()` runs in parallel with no concurrency limit. `RescanWith()` replaces the project list while preserving git enrichment.
+- **E-PENPAL-CACHE**: An in-memory cache (`sync.RWMutex`-protected) holds the full project list and per-project file lists. `RefreshProject()` walks the filesystem for full rescans; `RefreshAllProjects()` runs in parallel with a concurrency limit of 4. `RescanWith()` replaces the project list while preserving git enrichment and cached file data for unchanged projects — only new or source-changed projects are rescanned. Incremental mutations (`UpsertFile`, `RemoveFile`) update individual cache entries without walking the filesystem.
← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE)
-- **E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending. `EnsureProjectScanned()` is the lazy-scan entry point — it uses write-lock gating (`projectScanned` set under `mu.Lock` before scanning) to prevent concurrent requests from triggering duplicate filesystem walks. `projectHasAnyMarkdown()` performs a cheap startup check that aligns with the full scan: it uses the same gitignore checking, skips `.git`, `node_modules`, `.hg`, `.svn`, and nested worktree directories, and stops at the first `.md` file found.
+- **E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending. `EnsureProjectScanned()` is the lazy-scan entry point — it uses write-lock gating (`projectScanned` set under `mu.Lock` before scanning) to prevent concurrent requests from triggering duplicate filesystem walks. `projectHasAnyMarkdown()` performs a cheap startup check that aligns with the full scan: it uses the same gitignore checking, skips `.git`, `node_modules`, `.hg`, `.svn`, and nested worktree directories, and stops at the first `.md` file found. `CheckAllProjectsHasFiles()` runs with a concurrency limit of 4 to cap subprocess spawning. `ResolveFileInfo()` resolves source membership for a single absolute path without spawning a git check-ignore process — it applies the same source-priority, SkipDirs, RequireSibling, and ClassifyFile rules as the full walk.
← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE), [P-PENPAL-FILE-TYPES](PRODUCT.md#P-PENPAL-FILE-TYPES), [P-PENPAL-SRC-DEDUP](PRODUCT.md#P-PENPAL-SRC-DEDUP), [P-PENPAL-SRC-GITIGNORE](PRODUCT.md#P-PENPAL-SRC-GITIGNORE)
- **E-PENPAL-TITLE-EXTRACT**: `EnrichTitles()` reads the first 20 lines of each file to extract H1 headings. Titles are cached and shown as the primary display name when present.
@@ -239,7 +239,7 @@ see-also:
- **E-PENPAL-SSE**: `GET /events` is a long-lived SSE stream using `event: change` messages. Event types: `projects`, `files`, `comments`, `agents`, `navigate`. Each event carries optional `project`, `path`, `worktree` fields.
← [P-PENPAL-REALTIME](PRODUCT.md#P-PENPAL-REALTIME)
-- **E-PENPAL-WATCHER**: The file watcher bridges `fsnotify` to SSE. Two-tier watch strategy: base (shallow workspace + project root directories) and dynamic (deep, per-focus). Debounce at 100ms per event key.
+- **E-PENPAL-WATCHER**: The file watcher bridges `fsnotify` to SSE. Two-tier watch strategy: base (shallow workspace + project root directories) and dynamic (deep, per-focus). Debounce at 100ms per event key. File events use an accumulating debounce that collects per-file paths and ops during the debounce window, then applies incremental cache mutations (`UpsertFile`/`RemoveFile`) instead of full project walks. Only `.md` file events trigger cache updates — non-`.md` Create events (e.g., scanner temp files) are filtered out. Structural events (workspace changes, source auto-detect) still use the original debounce with full discovery.
← [P-PENPAL-REALTIME](PRODUCT.md#P-PENPAL-REALTIME), [P-PENPAL-LIVE-UPDATE](PRODUCT.md#P-PENPAL-LIVE-UPDATE)
- **E-PENPAL-FOCUS**: `windowFocuses map[string]focusTarget` (one entry per browser window) drives dynamic watches. Union of all window focuses determines the watched set. File focus watches only the file's parent directory. Project focus watches all source directories + `.penpal/comments/`.
diff --git a/apps/penpal/TESTING.md b/apps/penpal/TESTING.md
index a184ebaf..59173f3b 100644
--- a/apps/penpal/TESTING.md
+++ b/apps/penpal/TESTING.md
@@ -64,7 +64,7 @@ see-also:
| Source Types — anchors (P-PENPAL-SRC-ANCHORS, SRC-ANCHORS-GROUP, SRC-ANCHORS-NESTED) | discovery_test.go (TestClassifyAnchorsFile, TestGroupAnchorsPaths, TestGroupAnchorsPaths_MarkerOnlyModule, TestAnchorsFileOrder, TestAnchorsRequireSibling) | — | — | — |
| Source Types — claude-plans (P-PENPAL-SRC-CLAUDE-PLANS) | — | — | — | — |
| Source Types — manual (P-PENPAL-SRC-MANUAL) | — | — | grouping_test.go (TestBuildFileGroups_ManualSourceDirHeadings) | — |
-| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_SkipsGitignored, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans) | — | — | — |
+| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_IgnoresGitignore, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans, TestResolveFileInfo, TestUpsertFile, TestRemoveFile, TestRescanWith_PreservesUnchangedProjects, TestSourcesChanged) | — | — | — |
| Worktree Support (P-PENPAL-WORKTREE) | discovery/worktree_test.go, cache/worktree_test.go | Layout.test.tsx | worktree_test.go (API + MCP) | — |
| Worktree Dropdown (P-PENPAL-PROJECT-WORKTREE-DROPDOWN) | — | Layout.test.tsx | — | — |
| Git Integration (P-PENPAL-GIT-INFO) | — | — | — | — |
diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go
index a784dbea..8c5c5726 100644
--- a/apps/penpal/internal/cache/cache.go
+++ b/apps/penpal/internal/cache/cache.go
@@ -342,6 +342,13 @@ func (c *Cache) FindFile(projectName, filePath string) *FileInfo {
return nil
}
+// IsProjectScanned returns whether a project has had a full file scan.
+func (c *Cache) IsProjectScanned(projectName string) bool {
+ c.mu.RLock()
+ defer c.mu.RUnlock()
+ return c.projectScanned[projectName]
+}
+
// EnsureProjectScanned triggers a full file scan for a project if it hasn't
// been scanned yet. Returns true if a scan was actually performed (first call
// for this project). This is the lazy-scan entry point — called when a user
@@ -388,15 +395,18 @@ func (c *Cache) RefreshProject(projectName string) {
}
}
-// RefreshAllProjects rescans all projects' files and updates metadata
-// E-PENPAL-CACHE: parallel refresh with no concurrency limit.
+// RefreshAllProjects rescans all projects' files and updates metadata.
+// E-PENPAL-CACHE: parallel refresh with concurrency limit of 4.
func (c *Cache) RefreshAllProjects() {
projects := c.Projects()
var wg sync.WaitGroup
+ sem := make(chan struct{}, 4)
for _, p := range projects {
wg.Add(1)
go func(qn string) {
defer wg.Done()
+ sem <- struct{}{}
+ defer func() { <-sem }()
c.RefreshProject(qn)
}(p.QualifiedName())
}
@@ -410,14 +420,17 @@ var errFoundMarkdown = errors.New("found markdown")
// CheckAllProjectsHasFiles does a cheap per-project check to set HasFiles
// without doing a full file scan. For each project, it walks the project
// root and stops as soon as it finds any .md file.
-// E-PENPAL-SCAN: lightweight startup check — no file list is built.
+// E-PENPAL-SCAN: lightweight startup check with concurrency limit of 4.
func (c *Cache) CheckAllProjectsHasFiles() {
projects := c.Projects()
var wg sync.WaitGroup
+ sem := make(chan struct{}, 4)
for _, p := range projects {
wg.Add(1)
go func(p discovery.Project) {
defer wg.Done()
+ sem <- struct{}{}
+ defer func() { <-sem }()
found := projectHasAnyMarkdown(p.Path)
c.mu.Lock()
for i := range c.projects {
@@ -433,13 +446,12 @@ func (c *Cache) CheckAllProjectsHasFiles() {
}
// projectHasAnyMarkdown walks the directory tree and returns true as soon as
-// it finds any .md file. Aligns skip behavior with scanProjectSources: skips
-// .git, node_modules, .hg, .svn, nested worktrees/submodules, and gitignored dirs.
-// E-PENPAL-SCAN: lightweight startup check consistent with full scan filtering.
+// it finds any .md file. Skips .git, node_modules, .hg, .svn, and nested
+// worktrees/submodules. Does NOT use git check-ignore — a false positive
+// from a .md file in a gitignored directory is harmless since the full scan
+// on first access applies proper filtering.
+// E-PENPAL-SCAN: lightweight startup check — no subprocess spawned.
func projectHasAnyMarkdown(projectPath string) bool {
- gitChecker := newGitIgnoreChecker(projectPath)
- defer gitChecker.Close()
-
err := filepath.WalkDir(projectPath, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return nil
@@ -456,10 +468,6 @@ func projectHasAnyMarkdown(projectPath string) bool {
return filepath.SkipDir
}
}
- // E-PENPAL-SRC-GITIGNORE: skip gitignored directories.
- if path != projectPath && gitChecker.IsIgnored(path) {
- return filepath.SkipDir
- }
return nil
}
if strings.HasSuffix(d.Name(), ".md") {
@@ -520,22 +528,107 @@ func (c *Cache) RefreshProjectGitInfo(name string) {
// preserving existing git info for known projects.
// E-PENPAL-CACHE: replaces the project list while preserving git enrichment.
func (c *Cache) RescanWith(projects []discovery.Project) {
- // Preserve enrichment data (git info) for projects we already know about
+ // Snapshot current state before replacing
c.mu.RLock()
existing := make(map[string]discovery.Project)
+ existingScanned := make(map[string]bool)
+ existingFiles := make(map[string][]FileInfo)
for _, p := range c.projects {
- existing[p.QualifiedName()] = p
+ qn := p.QualifiedName()
+ existing[qn] = p
+ existingScanned[qn] = c.projectScanned[qn]
+ if files, ok := c.projectFiles[qn]; ok {
+ existingFiles[qn] = files
+ }
}
c.mu.RUnlock()
+ // Preserve git enrichment only when the project path hasn't changed.
+ // If the path changed, Git metadata would be stale and enrichGitInfo
+ // needs to re-discover it (it skips projects where Git != nil).
for i := range projects {
- if prev, ok := existing[projects[i].QualifiedName()]; ok {
+ if prev, ok := existing[projects[i].QualifiedName()]; ok && prev.Path == projects[i].Path {
projects[i].Git = prev.Git
}
}
c.SetProjects(projects)
- c.RefreshAllProjects()
+
+ // Determine which projects need scanning
+ var toScan []string
+ newNames := make(map[string]bool, len(projects))
+ for _, p := range projects {
+ qn := p.QualifiedName()
+ newNames[qn] = true
+
+ prev, existed := existing[qn]
+ if !existed {
+ // New project: needs scan
+ toScan = append(toScan, qn)
+ } else if !existingScanned[qn] {
+ // Existed but never scanned: needs scan
+ toScan = append(toScan, qn)
+ } else if SourcesChanged(prev.Sources, p.Sources) {
+ // Sources changed: needs rescan
+ toScan = append(toScan, qn)
+ } else {
+ // Unchanged: preserve cached files
+ c.mu.Lock()
+ c.projectFiles[qn] = existingFiles[qn]
+ c.projectScanned[qn] = true
+ c.updateProjectMetadataLocked(qn, existingFiles[qn])
+ c.mu.Unlock()
+ }
+ }
+
+ // Clean up removed projects
+ c.mu.Lock()
+ for name := range existingFiles {
+ if !newNames[name] {
+ delete(c.projectFiles, name)
+ delete(c.projectScanned, name)
+ }
+ }
+ c.mu.Unlock()
+
+ // Scan only the projects that need it, with concurrency limit
+ if len(toScan) > 0 {
+ var wg sync.WaitGroup
+ sem := make(chan struct{}, 4)
+ for _, qn := range toScan {
+ wg.Add(1)
+ go func(qn string) {
+ defer wg.Done()
+ sem <- struct{}{}
+ defer func() { <-sem }()
+ c.RefreshProject(qn)
+ }(qn)
+ }
+ wg.Wait()
+ }
+}
+
+// SourcesChanged returns true if two source lists differ materially.
+// E-PENPAL-CACHE: used by RescanWith to detect which projects need rescanning.
+func SourcesChanged(a, b []discovery.FileSource) bool {
+ if len(a) != len(b) {
+ return true
+ }
+ for i := range a {
+ if a[i].Name != b[i].Name || a[i].Type != b[i].Type ||
+ a[i].RootPath != b[i].RootPath || a[i].SourceTypeName != b[i].SourceTypeName {
+ return true
+ }
+ if len(a[i].Files) != len(b[i].Files) {
+ return true
+ }
+ for j := range a[i].Files {
+ if a[i].Files[j] != b[i].Files[j] {
+ return true
+ }
+ }
+ }
+ return false
}
// EnrichTitles fills in missing Title fields for files in the given project.
@@ -586,6 +679,325 @@ func extractTitle(path string) string {
return ""
}
+// ResolveFileInfo resolves source membership for a single absolute .md file path
+// within a project. It applies the same source-priority, SkipDirs, RequireSibling,
+// and ClassifyFile rules as scanProjectSources but without walking the filesystem
+// or spawning a git check-ignore process. Returns FileInfo entries for each source
+// that claims the file (typically one typed source + __all_markdown__). Returns nil
+// if no source claims the file.
+// E-PENPAL-SCAN: single-file source resolution for incremental cache updates.
+// ResolveFileInfo resolves source membership for a single absolute .md file path
+// without a filesystem walk. It applies the same exclusion rules as
+// scanProjectSources: nested git worktree/submodule detection, gitignore
+// ancestor-directory checks (P-PENPAL-SRC-GITIGNORE), SkipDirs filtering, and
+// RequireSibling validation.
+func ResolveFileInfo(project *discovery.Project, absPath string) []FileInfo {
+ if !strings.HasSuffix(absPath, ".md") {
+ return nil
+ }
+
+ info, err := os.Stat(absPath)
+ if err != nil {
+ return nil
+ }
+ if info.IsDir() {
+ return nil
+ }
+
+ relToProject, err := filepath.Rel(project.Path, absPath)
+ if err != nil || strings.HasPrefix(relToProject, "..") {
+ return nil
+ }
+
+ title := extractTitle(absPath)
+ var results []FileInfo
+ typedClaimed := false
+
+ for _, source := range project.Sources {
+ isAllMarkdown := source.Name == "__all_markdown__"
+
+ if source.Type == "thoughts" || source.Type == "tree" {
+ rootPath := source.RootPath
+ if rootPath == "" {
+ continue
+ }
+
+ // Check containment
+ if !strings.HasPrefix(absPath, rootPath+"/") && absPath != rootPath {
+ continue
+ }
+
+ // Skip files under nested git worktrees/submodules.
+ if isUnderNestedGitRepo(absPath, rootPath) {
+ continue
+ }
+
+ // P-PENPAL-SRC-GITIGNORE: skip files whose ancestor directory
+ // is gitignored (source root itself is exempt).
+ if isAncestorDirGitIgnored(absPath, rootPath, project.Path) {
+ continue
+ }
+
+ relToSource, err := filepath.Rel(rootPath, absPath)
+ if err != nil {
+ continue
+ }
+
+ // Check SkipDirs against each path component
+ stName := source.SourceTypeName
+ if stName == "" {
+ stName = source.Name
+ }
+ st := discovery.GetSourceType(stName)
+
+ if !isAllMarkdown && typedClaimed {
+ continue // already claimed by an earlier typed source
+ }
+
+ if hasSkippedDir(relToSource, st) {
+ continue
+ }
+
+ // RequireSibling check
+ if st != nil && st.RequireSibling != "" {
+ siblingPath := filepath.Join(filepath.Dir(absPath), st.RequireSibling)
+ if _, err := os.Stat(siblingPath); err != nil {
+ continue
+ }
+ }
+
+ // ClassifyFile
+ fileType := "other"
+ if st != nil && st.ClassifyFile != nil {
+ fileType = st.ClassifyFile(relToSource)
+ if fileType == "" {
+ continue // skip this file for this source
+ }
+ } else {
+ if strings.Contains(relToSource, "research") {
+ fileType = "research"
+ } else if strings.Contains(relToSource, "plan") {
+ fileType = "plan"
+ }
+ }
+
+ if !isAllMarkdown {
+ typedClaimed = true
+ }
+
+ results = append(results, FileInfo{
+ Project: project.QualifiedName(),
+ Workspace: project.WorkspaceName,
+ ProjectPath: project.Path,
+ Source: source.Name,
+ SourceType: source.Type,
+ SourceAuto: source.Auto,
+ Path: relToSource,
+ FullPath: relToProject,
+ Name: filepath.Base(absPath),
+ Title: title,
+ ModTime: info.ModTime(),
+ FileType: fileType,
+ })
+
+ } else if source.Type == "files" {
+ if !isAllMarkdown && typedClaimed {
+ continue
+ }
+ found := false
+ for _, f := range source.Files {
+ if f == absPath {
+ found = true
+ break
+ }
+ }
+ if !found {
+ continue
+ }
+
+ fileType := "other"
+ lower := strings.ToLower(filepath.Base(absPath))
+ if strings.Contains(lower, "research") {
+ fileType = "research"
+ } else if strings.Contains(lower, "plan") {
+ fileType = "plan"
+ }
+
+ if !isAllMarkdown {
+ typedClaimed = true
+ }
+
+ results = append(results, FileInfo{
+ Project: project.QualifiedName(),
+ Workspace: project.WorkspaceName,
+ ProjectPath: project.Path,
+ Source: source.Name,
+ SourceType: source.Type,
+ SourceAuto: source.Auto,
+ Path: filepath.Base(absPath),
+ FullPath: relToProject,
+ Name: filepath.Base(absPath),
+ Title: title,
+ ModTime: info.ModTime(),
+ FileType: fileType,
+ })
+ }
+ }
+
+ return results
+}
+
+// hasSkippedDir checks whether any directory component between the source root
+// and the file matches the source type's SkipDirs.
+func hasSkippedDir(relToSource string, st *discovery.SourceType) bool {
+ if st == nil || len(st.SkipDirs) == 0 {
+ return false
+ }
+ dir := filepath.Dir(relToSource)
+ if dir == "." {
+ return false
+ }
+ for _, component := range strings.Split(dir, string(filepath.Separator)) {
+ if st.SkipDirs[component] {
+ return true
+ }
+ }
+ return false
+}
+
+// isUnderNestedGitRepo walks parent directories from absPath up to (but not
+// including) rootPath, returning true if any intermediate directory contains a
+// .git file (not directory), indicating a nested git worktree or submodule.
+// This mirrors the nested-repo check in scanProjectSources without spawning
+// a subprocess.
+func isUnderNestedGitRepo(absPath, rootPath string) bool {
+ dir := filepath.Dir(absPath)
+ for dir != rootPath && strings.HasPrefix(dir, rootPath+"/") {
+ gitEntry := filepath.Join(dir, ".git")
+ if fi, err := os.Lstat(gitEntry); err == nil && !fi.IsDir() {
+ return true
+ }
+ dir = filepath.Dir(dir)
+ }
+ return false
+}
+
+// isAncestorDirGitIgnored walks parent directories from absPath up to (but not
+// including) rootPath, running a one-shot `git check-ignore -q` on each.
+// Returns true if any ancestor directory is gitignored.
+// P-PENPAL-SRC-GITIGNORE: the source root itself is exempt (always scanned).
+func isAncestorDirGitIgnored(absPath, rootPath, projectPath string) bool {
+ dir := filepath.Dir(absPath)
+ for dir != rootPath && strings.HasPrefix(dir, rootPath+"/") {
+ cmd := exec.Command("git", "-C", projectPath, "check-ignore", "-q", dir)
+ if cmd.Run() == nil {
+ return true // exit code 0 means ignored
+ }
+ dir = filepath.Dir(dir)
+ }
+ return false
+}
+
+// UpsertFile adds or updates file entries in the cache for the given absolute path.
+// For existing entries (matched by FullPath), re-stats for ModTime and re-extracts
+// Title. For new files, resolves source membership and inserts.
+// E-PENPAL-CACHE: incremental cache mutation without filesystem walk.
+func (c *Cache) UpsertFile(projectName string, project *discovery.Project, absPath string) bool {
+ // Perform all filesystem and git I/O outside the lock.
+ relToProject, err := filepath.Rel(project.Path, absPath)
+ if err != nil {
+ return false
+ }
+
+ info, err := os.Stat(absPath)
+ if err != nil {
+ return false
+ }
+
+ title := extractTitle(absPath)
+ resolved := ResolveFileInfo(project, absPath)
+
+ // Acquire lock only for the short critical section that mutates the cache.
+ c.mu.Lock()
+ defer c.mu.Unlock()
+
+ files := c.projectFiles[projectName]
+
+ // Check if any entries already exist for this path
+ updated := false
+ for i := range files {
+ if files[i].FullPath == relToProject {
+ files[i].ModTime = info.ModTime()
+ files[i].Title = title
+ updated = true
+ }
+ }
+
+ if updated {
+ sort.Slice(files, func(i, j int) bool {
+ return files[i].ModTime.After(files[j].ModTime)
+ })
+ c.projectFiles[projectName] = files
+ c.updateProjectMetadataLocked(projectName, files)
+ return true
+ }
+
+ // New file — use pre-resolved source membership
+ if len(resolved) == 0 {
+ return false
+ }
+
+ files = append(files, resolved...)
+ sort.Slice(files, func(i, j int) bool {
+ return files[i].ModTime.After(files[j].ModTime)
+ })
+ c.projectFiles[projectName] = files
+ c.updateProjectMetadataLocked(projectName, files)
+ return true
+}
+
+// RemoveFile removes all cache entries with the given project-relative path.
+// E-PENPAL-CACHE: incremental cache mutation without filesystem walk.
+func (c *Cache) RemoveFile(projectName, fullPath string) bool {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+
+ files := c.projectFiles[projectName]
+ if files == nil {
+ return false
+ }
+
+ n := 0
+ for _, f := range files {
+ if f.FullPath != fullPath {
+ files[n] = f
+ n++
+ }
+ }
+ if n == len(files) {
+ return false // nothing removed
+ }
+
+ files = files[:n]
+ c.projectFiles[projectName] = files
+ c.updateProjectMetadataLocked(projectName, files)
+ return true
+}
+
+// updateProjectMetadataLocked updates HasFiles and LastModified for a project.
+// Must be called with c.mu held for writing.
+func (c *Cache) updateProjectMetadataLocked(projectName string, files []FileInfo) {
+ for i := range c.projects {
+ if c.projects[i].QualifiedName() == projectName {
+ c.projects[i].HasFiles = len(files) > 0
+ if len(files) > 0 {
+ c.projects[i].LastModified = files[0].ModTime
+ }
+ break
+ }
+ }
+}
+
// ScanProjectSourcesForWorktree scans a project's sources remapped to a worktree path.
// Each source's RootPath under the project is remapped to the equivalent path under
// the worktree. Sources whose directory doesn't exist in the worktree are skipped.
diff --git a/apps/penpal/internal/cache/cache_test.go b/apps/penpal/internal/cache/cache_test.go
index 19e4e6cf..b0feafc8 100644
--- a/apps/penpal/internal/cache/cache_test.go
+++ b/apps/penpal/internal/cache/cache_test.go
@@ -521,27 +521,23 @@ func TestCheckAllProjectsHasFiles(t *testing.T) {
}
}
-// E-PENPAL-SCAN: verifies projectHasAnyMarkdown skips gitignored dirs.
-func TestProjectHasAnyMarkdown_SkipsGitignored(t *testing.T) {
+// E-PENPAL-SCAN: projectHasAnyMarkdown does NOT check gitignore — it's a
+// lightweight startup check where false positives are harmless. Verifying that
+// .md files in gitignored dirs still count as "has markdown".
+func TestProjectHasAnyMarkdown_IgnoresGitignore(t *testing.T) {
tmpDir := t.TempDir()
runGit(t, tmpDir, "init")
runGit(t, tmpDir, "config", "user.email", "test@test.com")
runGit(t, tmpDir, "config", "user.name", "test")
- // All .md files are in a gitignored directory
+ // .md files only in a gitignored directory — still returns true
os.WriteFile(filepath.Join(tmpDir, ".gitignore"), []byte("build/\n"), 0644)
os.MkdirAll(filepath.Join(tmpDir, "build"), 0755)
os.WriteFile(filepath.Join(tmpDir, "build", "output.md"), []byte("# Gen"), 0644)
- if projectHasAnyMarkdown(tmpDir) {
- t.Error("expected false: only .md files are in gitignored dir")
- }
-
- // Add a non-gitignored .md file and re-check
- os.WriteFile(filepath.Join(tmpDir, "README.md"), []byte("# README"), 0644)
if !projectHasAnyMarkdown(tmpDir) {
- t.Error("expected true: README.md is not gitignored")
+ t.Error("expected true: .md exists even though gitignored (gitignore not checked)")
}
}
@@ -701,3 +697,483 @@ func TestCache_FindFile(t *testing.T) {
})
}
}
+
+// E-PENPAL-SCAN: verifies single-file source resolution matches scan behavior.
+func TestResolveFileInfo_ThoughtsSource(t *testing.T) {
+ tmpDir := t.TempDir()
+ thoughtsDir := filepath.Join(tmpDir, "thoughts")
+ os.MkdirAll(filepath.Join(thoughtsDir, "research"), 0755)
+
+ filePath := filepath.Join(thoughtsDir, "research", "topic.md")
+ os.WriteFile(filePath, []byte("# My Research"), 0644)
+
+ project := &discovery.Project{
+ Name: "test",
+ Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: thoughtsDir, Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+ },
+ }
+
+ results := ResolveFileInfo(project, filePath)
+ if len(results) != 2 {
+ t.Fatalf("expected 2 results (thoughts + __all_markdown__), got %d", len(results))
+ }
+
+ // First result should be the typed source
+ if results[0].Source != "thoughts" {
+ t.Errorf("expected first source 'thoughts', got %q", results[0].Source)
+ }
+ if results[0].FileType != "research" {
+ t.Errorf("expected fileType 'research', got %q", results[0].FileType)
+ }
+ if results[0].Title != "My Research" {
+ t.Errorf("expected title 'My Research', got %q", results[0].Title)
+ }
+ if results[0].FullPath != "thoughts/research/topic.md" {
+ t.Errorf("expected fullPath 'thoughts/research/topic.md', got %q", results[0].FullPath)
+ }
+
+ // Second result should be __all_markdown__
+ if results[1].Source != "__all_markdown__" {
+ t.Errorf("expected second source '__all_markdown__', got %q", results[1].Source)
+ }
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo respects SkipDirs.
+func TestResolveFileInfo_SkipDirs(t *testing.T) {
+ tmpDir := t.TempDir()
+ os.MkdirAll(filepath.Join(tmpDir, "node_modules", "pkg"), 0755)
+ filePath := filepath.Join(tmpDir, "node_modules", "pkg", "readme.md")
+ os.WriteFile(filePath, []byte("# Dep"), 0644)
+
+ project := &discovery.Project{
+ Name: "test",
+ Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+ },
+ }
+
+ results := ResolveFileInfo(project, filePath)
+ // __all_markdown__ has SkipDirs for node_modules
+ if len(results) != 0 {
+ t.Fatalf("expected 0 results (node_modules is in SkipDirs), got %d", len(results))
+ }
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo respects RequireSibling.
+func TestResolveFileInfo_RequireSibling(t *testing.T) {
+ tmpDir := t.TempDir()
+
+ // Directory WITH ANCHORS.md sibling
+ withSibling := filepath.Join(tmpDir, "module-a")
+ os.MkdirAll(withSibling, 0755)
+ os.WriteFile(filepath.Join(withSibling, "ANCHORS.md"), []byte("---\nprefix: A\n---\n"), 0644)
+ os.WriteFile(filepath.Join(withSibling, "PRODUCT.md"), []byte("# Product"), 0644)
+
+ // Directory WITHOUT ANCHORS.md sibling
+ withoutSibling := filepath.Join(tmpDir, "module-b")
+ os.MkdirAll(withoutSibling, 0755)
+ os.WriteFile(filepath.Join(withoutSibling, "PRODUCT.md"), []byte("# Orphan"), 0644)
+
+ project := &discovery.Project{
+ Name: "test",
+ Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "anchors", Type: "tree", SourceTypeName: "anchors", RootPath: tmpDir, Auto: true},
+ },
+ }
+
+ // File with sibling should be included
+ results := ResolveFileInfo(project, filepath.Join(withSibling, "PRODUCT.md"))
+ if len(results) != 1 {
+ t.Fatalf("expected 1 result for file with ANCHORS.md sibling, got %d", len(results))
+ }
+
+ // File without sibling should be excluded
+ results = ResolveFileInfo(project, filepath.Join(withoutSibling, "PRODUCT.md"))
+ if len(results) != 0 {
+ t.Fatalf("expected 0 results for file without ANCHORS.md sibling, got %d", len(results))
+ }
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo returns nil for non-.md files.
+func TestResolveFileInfo_NonMdFile(t *testing.T) {
+ tmpDir := t.TempDir()
+ filePath := filepath.Join(tmpDir, "readme.txt")
+ os.WriteFile(filePath, []byte("hello"), 0644)
+
+ project := &discovery.Project{
+ Name: "test",
+ Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+ },
+ }
+
+ results := ResolveFileInfo(project, filePath)
+ if len(results) != 0 {
+ t.Fatalf("expected 0 results for non-.md file, got %d", len(results))
+ }
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo dedup — first typed source wins.
+func TestResolveFileInfo_SourcePriority(t *testing.T) {
+ tmpDir := t.TempDir()
+ thoughtsDir := filepath.Join(tmpDir, "thoughts")
+ os.MkdirAll(thoughtsDir, 0755)
+ filePath := filepath.Join(thoughtsDir, "plan.md")
+ os.WriteFile(filePath, []byte("# Plan"), 0644)
+
+ project := &discovery.Project{
+ Name: "test",
+ Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: thoughtsDir, Auto: true},
+ // A second typed source covering the same path
+ {Name: "manual", Type: "tree", SourceTypeName: "manual", RootPath: thoughtsDir},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+ },
+ }
+
+ results := ResolveFileInfo(project, filePath)
+ // Should get 2: first typed source (thoughts) + __all_markdown__
+ // The second typed source (manual) should be skipped
+ if len(results) != 2 {
+ t.Fatalf("expected 2 results, got %d", len(results))
+ }
+ if results[0].Source != "thoughts" {
+ t.Errorf("expected first source 'thoughts', got %q", results[0].Source)
+ }
+ if results[1].Source != "__all_markdown__" {
+ t.Errorf("expected second source '__all_markdown__', got %q", results[1].Source)
+ }
+}
+
+// E-PENPAL-CACHE: verifies UpsertFile updates existing entries.
+func TestUpsertFile_ExistingFile(t *testing.T) {
+ tmpDir := t.TempDir()
+ filePath := filepath.Join(tmpDir, "thoughts", "plan.md")
+ os.MkdirAll(filepath.Join(tmpDir, "thoughts"), 0755)
+ os.WriteFile(filePath, []byte("# Old Title"), 0644)
+
+ c := New()
+ projectName := "test"
+ c.SetProjects([]discovery.Project{
+ {Name: "test", Path: tmpDir, Sources: []discovery.FileSource{
+ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: filepath.Join(tmpDir, "thoughts"), Auto: true},
+ }},
+ })
+
+ // Pre-populate cache with an entry
+ c.SetProjectFiles(projectName, []FileInfo{
+ {Project: "test", Source: "thoughts", FullPath: "thoughts/plan.md", Name: "plan.md", Title: "Old Title", ModTime: time.Now().Add(-1 * time.Hour)},
+ })
+
+ // Update the file on disk
+ os.WriteFile(filePath, []byte("# New Title"), 0644)
+
+ project := c.FindProject(projectName)
+ ok := c.UpsertFile(projectName, project, filePath)
+ if !ok {
+ t.Fatal("UpsertFile returned false, expected true")
+ }
+
+ files := c.ProjectFiles(projectName)
+ if len(files) != 1 {
+ t.Fatalf("expected 1 file, got %d", len(files))
+ }
+ if files[0].Title != "New Title" {
+ t.Errorf("expected title 'New Title', got %q", files[0].Title)
+ }
+}
+
+// E-PENPAL-CACHE: verifies UpsertFile adds new files via source resolution.
+func TestUpsertFile_NewFile(t *testing.T) {
+ tmpDir := t.TempDir()
+ thoughtsDir := filepath.Join(tmpDir, "thoughts")
+ os.MkdirAll(thoughtsDir, 0755)
+ filePath := filepath.Join(thoughtsDir, "new-note.md")
+ os.WriteFile(filePath, []byte("# Fresh Note"), 0644)
+
+ c := New()
+ projectName := "test"
+ project := discovery.Project{
+ Name: "test", Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: thoughtsDir, Auto: true},
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+ },
+ }
+ c.SetProjects([]discovery.Project{project})
+ c.SetProjectFiles(projectName, nil) // empty cache
+
+ ok := c.UpsertFile(projectName, &project, filePath)
+ if !ok {
+ t.Fatal("UpsertFile returned false, expected true")
+ }
+
+ files := c.ProjectFiles(projectName)
+ if len(files) != 2 {
+ t.Fatalf("expected 2 files (thoughts + __all_markdown__), got %d", len(files))
+ }
+
+ // Check the typed source entry
+ found := false
+ for _, f := range files {
+ if f.Source == "thoughts" && f.FullPath == "thoughts/new-note.md" {
+ found = true
+ if f.Title != "Fresh Note" {
+ t.Errorf("expected title 'Fresh Note', got %q", f.Title)
+ }
+ }
+ }
+ if !found {
+ t.Error("expected thoughts source entry for new file")
+ }
+}
+
+// E-PENPAL-CACHE: verifies UpsertFile returns false for excluded files.
+func TestUpsertFile_ExcludedFile(t *testing.T) {
+ tmpDir := t.TempDir()
+ os.MkdirAll(filepath.Join(tmpDir, "node_modules"), 0755)
+ filePath := filepath.Join(tmpDir, "node_modules", "readme.md")
+ os.WriteFile(filePath, []byte("# Dep"), 0644)
+
+ c := New()
+ projectName := "test"
+ project := discovery.Project{
+ Name: "test", Path: tmpDir,
+ Sources: []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+ },
+ }
+ c.SetProjects([]discovery.Project{project})
+ c.SetProjectFiles(projectName, nil)
+
+ ok := c.UpsertFile(projectName, &project, filePath)
+ if ok {
+ t.Error("UpsertFile should return false for file in SkipDirs")
+ }
+
+ files := c.ProjectFiles(projectName)
+ if len(files) != 0 {
+ t.Fatalf("expected 0 files, got %d", len(files))
+ }
+}
+
+// E-PENPAL-CACHE: verifies RemoveFile removes entries and updates metadata.
+func TestRemoveFile(t *testing.T) {
+ now := time.Now()
+ older := now.Add(-1 * time.Hour)
+
+ c := New()
+ projectName := "test"
+ c.SetProjects([]discovery.Project{
+ {Name: "test", Path: "/tmp/test"},
+ })
+ c.SetProjectFiles(projectName, []FileInfo{
+ {Project: "test", Source: "thoughts", FullPath: "thoughts/plan.md", Name: "plan.md", ModTime: now},
+ {Project: "test", Source: "thoughts", FullPath: "thoughts/old.md", Name: "old.md", ModTime: older},
+ })
+
+ ok := c.RemoveFile(projectName, "thoughts/plan.md")
+ if !ok {
+ t.Fatal("RemoveFile returned false, expected true")
+ }
+
+ files := c.ProjectFiles(projectName)
+ if len(files) != 1 {
+ t.Fatalf("expected 1 file, got %d", len(files))
+ }
+ if files[0].FullPath != "thoughts/old.md" {
+ t.Errorf("expected remaining file 'thoughts/old.md', got %q", files[0].FullPath)
+ }
+
+ // Verify metadata updated
+ project := c.FindProject(projectName)
+ if !project.HasFiles {
+ t.Error("project should still have files")
+ }
+ if !project.LastModified.Equal(older) {
+ t.Errorf("expected LastModified to be older time, got %v", project.LastModified)
+ }
+}
+
+// E-PENPAL-CACHE: verifies RemoveFile returns false for non-existent entries.
+func TestRemoveFile_NotFound(t *testing.T) {
+ c := New()
+ c.SetProjectFiles("test", []FileInfo{
+ {Project: "test", FullPath: "readme.md"},
+ })
+
+ ok := c.RemoveFile("test", "nonexistent.md")
+ if ok {
+ t.Error("RemoveFile should return false for non-existent file")
+ }
+}
+
+// E-PENPAL-CACHE: verifies RemoveFile clears HasFiles when last file removed.
+func TestRemoveFile_ClearsHasFiles(t *testing.T) {
+ c := New()
+ c.SetProjects([]discovery.Project{
+ {Name: "test", Path: "/tmp/test", HasFiles: true},
+ })
+ c.SetProjectFiles("test", []FileInfo{
+ {Project: "test", FullPath: "only.md", ModTime: time.Now()},
+ })
+
+ c.RemoveFile("test", "only.md")
+
+ project := c.FindProject("test")
+ if project.HasFiles {
+ t.Error("project should have HasFiles=false after removing last file")
+ }
+}
+
+// E-PENPAL-CACHE: verifies SourcesChanged detects material differences.
+func TestSourcesChanged(t *testing.T) {
+ base := []discovery.FileSource{
+ {Name: "thoughts", Type: "tree", RootPath: "/a/thoughts", SourceTypeName: "thoughts"},
+ {Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+ }
+
+ // Identical
+ same := []discovery.FileSource{
+ {Name: "thoughts", Type: "tree", RootPath: "/a/thoughts", SourceTypeName: "thoughts"},
+ {Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+ }
+ if SourcesChanged(base, same) {
+ t.Error("identical sources should not be reported as changed")
+ }
+
+ // Different count
+ fewer := base[:1]
+ if !SourcesChanged(base, fewer) {
+ t.Error("different count should be reported as changed")
+ }
+
+ // Different root path
+ moved := []discovery.FileSource{
+ {Name: "thoughts", Type: "tree", RootPath: "/b/thoughts", SourceTypeName: "thoughts"},
+ {Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+ }
+ if !SourcesChanged(base, moved) {
+ t.Error("different RootPath should be reported as changed")
+ }
+
+ // Different name
+ renamed := []discovery.FileSource{
+ {Name: "rp1", Type: "tree", RootPath: "/a/thoughts", SourceTypeName: "rp1"},
+ {Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+ }
+ if !SourcesChanged(base, renamed) {
+ t.Error("different Name should be reported as changed")
+ }
+
+ // Different files list
+ withFiles := []discovery.FileSource{
+ {Name: "manual", Type: "files", Files: []string{"/a/foo.md"}},
+ }
+ withDiffFiles := []discovery.FileSource{
+ {Name: "manual", Type: "files", Files: []string{"/a/bar.md"}},
+ }
+ if !SourcesChanged(withFiles, withDiffFiles) {
+ t.Error("different Files should be reported as changed")
+ }
+}
+
+// E-PENPAL-CACHE: verifies RescanWith preserves cache for unchanged projects.
+func TestRescanWith_PreservesUnchangedProjects(t *testing.T) {
+ tmpDir := t.TempDir()
+
+ // Set up two projects
+ projADir := filepath.Join(tmpDir, "proj-a")
+ projBDir := filepath.Join(tmpDir, "proj-b")
+ os.MkdirAll(projADir, 0755)
+ os.MkdirAll(projBDir, 0755)
+ os.WriteFile(filepath.Join(projADir, "readme.md"), []byte("# A"), 0644)
+ os.WriteFile(filepath.Join(projBDir, "readme.md"), []byte("# B"), 0644)
+
+ sourcesA := []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: projADir, Auto: true},
+ }
+ sourcesB := []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: projBDir, Auto: true},
+ }
+
+ c := New()
+ c.SetProjects([]discovery.Project{
+ {Name: "proj-a", Path: projADir, Sources: sourcesA},
+ {Name: "proj-b", Path: projBDir, Sources: sourcesB},
+ })
+
+ // Simulate initial scan for proj-a
+ c.SetProjectFiles("proj-a", []FileInfo{
+ {Project: "proj-a", Source: "__all_markdown__", FullPath: "readme.md", Name: "readme.md", Title: "A", ModTime: time.Now()},
+ })
+
+ // proj-b is not scanned yet
+
+ // RescanWith the same projects (unchanged sources)
+ c.RescanWith([]discovery.Project{
+ {Name: "proj-a", Path: projADir, Sources: sourcesA},
+ {Name: "proj-b", Path: projBDir, Sources: sourcesB},
+ })
+
+ // proj-a should still have its cached files (not re-walked)
+ filesA := c.ProjectFiles("proj-a")
+ if len(filesA) != 1 {
+ t.Fatalf("expected proj-a to preserve 1 cached file, got %d", len(filesA))
+ }
+ if filesA[0].Title != "A" {
+ t.Errorf("expected preserved title 'A', got %q", filesA[0].Title)
+ }
+
+ // proj-b was never scanned, so RescanWith should scan it now
+ filesB := c.ProjectFiles("proj-b")
+ if len(filesB) != 1 {
+ t.Fatalf("expected proj-b to have 1 file after rescan, got %d", len(filesB))
+ }
+}
+
+// E-PENPAL-CACHE: verifies RescanWith cleans up removed projects.
+func TestRescanWith_RemovesOldProjects(t *testing.T) {
+ keepSources := []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/keep", Auto: true},
+ }
+ removeSources := []discovery.FileSource{
+ {Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/remove", Auto: true},
+ }
+
+ c := New()
+ c.SetProjects([]discovery.Project{
+ {Name: "keep", Path: "/tmp/keep", Sources: keepSources},
+ {Name: "remove", Path: "/tmp/remove", Sources: removeSources},
+ })
+ c.SetProjectFiles("keep", []FileInfo{
+ {Project: "keep", FullPath: "readme.md"},
+ })
+ c.SetProjectFiles("remove", []FileInfo{
+ {Project: "remove", FullPath: "readme.md"},
+ })
+
+ // RescanWith only the "keep" project (same sources → preserved)
+ c.RescanWith([]discovery.Project{
+ {Name: "keep", Path: "/tmp/keep", Sources: keepSources},
+ })
+
+ // "remove" should be gone from cache
+ filesRemoved := c.ProjectFiles("remove")
+ if len(filesRemoved) != 0 {
+ t.Errorf("expected removed project to have 0 cached files, got %d", len(filesRemoved))
+ }
+
+ // "keep" should still have its files (unchanged sources → preserved)
+ filesKeep := c.ProjectFiles("keep")
+ if len(filesKeep) != 1 {
+ t.Errorf("expected kept project to preserve 1 cached file, got %d", len(filesKeep))
+ }
+}
diff --git a/apps/penpal/internal/server/manage.go b/apps/penpal/internal/server/manage.go
index 42dd7e09..f9e23e97 100644
--- a/apps/penpal/internal/server/manage.go
+++ b/apps/penpal/internal/server/manage.go
@@ -8,6 +8,7 @@ import (
"os"
"path/filepath"
"strings"
+ "sync"
"time"
"github.com/loganj/penpal/internal/config"
@@ -26,6 +27,9 @@ func expandTilde(path string) string {
}
// refreshAfterConfigChange saves the config and re-discovers all projects.
+// RescanWith preserves cached data for unchanged projects and only scans
+// new or source-changed ones, so we skip the redundant populateProjects
+// call and just enrich git info in the background.
func (s *Server) refreshAfterConfigChange() {
if err := config.Save(s.cfgPath, s.cfg); err != nil {
log.Printf("Warning: could not save config: %v", err)
@@ -34,7 +38,33 @@ func (s *Server) refreshAfterConfigChange() {
s.cache.RescanWith(projects)
s.watcher.Refresh(s.workspacePaths(), projects)
s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged})
- go s.populateProjects()
+ go s.enrichGitInfo()
+}
+
+// enrichGitInfo updates git info for all projects that don't have it yet.
+func (s *Server) enrichGitInfo() {
+ projects := s.cache.Projects()
+ var wg sync.WaitGroup
+ sem := make(chan struct{}, 8)
+
+ for _, p := range projects {
+ if p.Name == "(root)" || p.Git != nil {
+ continue
+ }
+ wg.Add(1)
+ go func(p discovery.Project) {
+ defer wg.Done()
+ sem <- struct{}{}
+ defer func() { <-sem }()
+ git := discovery.GetGitInfo(p.Path)
+ s.cache.EnrichProject(p.QualifiedName(), git)
+ }(p)
+ }
+
+ wg.Wait()
+ if len(projects) > 0 {
+ s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged})
+ }
}
// handleAPIWorkspaces dispatches workspace management requests.
diff --git a/apps/penpal/internal/server/server.go b/apps/penpal/internal/server/server.go
index 7dbb8ca7..3deef897 100644
--- a/apps/penpal/internal/server/server.go
+++ b/apps/penpal/internal/server/server.go
@@ -1054,8 +1054,8 @@ func (s *Server) handleDeleteFile(w http.ResponseWriter, r *http.Request) {
}
s.cfgMu.Unlock()
- // Refresh cache so the file disappears from listings
- s.cache.RefreshProject(qualifiedName)
+ // E-PENPAL-CACHE: remove file from cache incrementally (no walk).
+ s.cache.RemoveFile(qualifiedName, filePath)
s.watcher.Broadcast(watcher.Event{Type: watcher.EventFilesChanged, Project: qualifiedName})
w.WriteHeader(http.StatusNoContent)
}
diff --git a/apps/penpal/internal/server/sse_test.go b/apps/penpal/internal/server/sse_test.go
index be8d4390..0a2fd6b5 100644
--- a/apps/penpal/internal/server/sse_test.go
+++ b/apps/penpal/internal/server/sse_test.go
@@ -101,37 +101,41 @@ func TestSSE_BroadcastDelivery(t *testing.T) {
// Broadcast a change event
s.watcher.Broadcast(watcher.Event{Type: watcher.EventFilesChanged, Project: "test-proj"})
- line1, err := reader.ReadString('\n')
- if err != nil {
- t.Fatalf("reading event line 1: %v", err)
- }
- if line1 != "event: change\n" {
- t.Errorf("expected %q, got %q", "event: change\n", line1)
- }
+ // Read SSE events until we find our specific broadcast. Background
+ // goroutines (populateProjects) may emit events that race with ours.
+ var found bool
+ for attempts := 0; attempts < 10; attempts++ {
+ line1, err := reader.ReadString('\n')
+ if err != nil {
+ t.Fatalf("reading event line: %v", err)
+ }
+ if line1 != "event: change\n" {
+ t.Fatalf("expected event line, got %q", line1)
+ }
- line2, err := reader.ReadString('\n')
- if err != nil {
- t.Fatalf("reading event line 2: %v", err)
- }
- dataStr := strings.TrimPrefix(line2, "data: ")
- dataStr = strings.TrimSuffix(dataStr, "\n")
- var evt watcher.Event
- if err := json.Unmarshal([]byte(dataStr), &evt); err != nil {
- t.Fatalf("parsing event JSON %q: %v", dataStr, err)
- }
- if evt.Type != "files" {
- t.Errorf("expected event type %q, got %q", "files", evt.Type)
- }
- if evt.Project != "test-proj" {
- t.Errorf("expected project %q, got %q", "test-proj", evt.Project)
- }
+ line2, err := reader.ReadString('\n')
+ if err != nil {
+ t.Fatalf("reading data line: %v", err)
+ }
+ dataStr := strings.TrimPrefix(line2, "data: ")
+ dataStr = strings.TrimSuffix(dataStr, "\n")
- line3, err := reader.ReadString('\n')
- if err != nil {
- t.Fatalf("reading event line 3: %v", err)
+ // Read blank separator
+ if _, err := reader.ReadString('\n'); err != nil {
+ t.Fatalf("reading separator: %v", err)
+ }
+
+ var evt watcher.Event
+ if err := json.Unmarshal([]byte(dataStr), &evt); err != nil {
+ t.Fatalf("parsing event JSON %q: %v", dataStr, err)
+ }
+ if evt.Type == "files" && evt.Project == "test-proj" {
+ found = true
+ break
+ }
}
- if line3 != "\n" {
- t.Errorf("expected blank separator, got %q", line3)
+ if !found {
+ t.Error("did not receive expected files event for test-proj")
}
}
diff --git a/apps/penpal/internal/watcher/watcher.go b/apps/penpal/internal/watcher/watcher.go
index c5e96f32..1a85700f 100644
--- a/apps/penpal/internal/watcher/watcher.go
+++ b/apps/penpal/internal/watcher/watcher.go
@@ -70,6 +70,13 @@ type Watcher struct {
windowFocuses map[string]focusTarget
baseWatched map[string]struct{}
dynamicWatched map[string]struct{}
+
+ // E-PENPAL-WATCHER: accumulating debounce for incremental file updates.
+ // Collects per-file paths and ops during the debounce window, then applies
+ // incremental cache mutations instead of full project walks.
+ filePending map[string]map[string]fsnotify.Op // projectName → absPath → accumulated op
+ filePendingMu sync.Mutex
+ fileTimers map[string]*time.Timer // projectName → debounce timer
}
// New creates a new watcher
@@ -90,6 +97,8 @@ func New(c *cache.Cache, act *activity.Tracker) (*Watcher, error) {
windowFocuses: make(map[string]focusTarget),
baseWatched: make(map[string]struct{}),
dynamicWatched: make(map[string]struct{}),
+ filePending: make(map[string]map[string]fsnotify.Op),
+ fileTimers: make(map[string]*time.Timer),
}
return w, nil
@@ -578,13 +587,29 @@ func (w *Watcher) handleEvent(event fsnotify.Event) {
return
}
- // Only care about .md files for file list updates
- if !strings.HasSuffix(path, ".md") && event.Op&fsnotify.Create == 0 {
+ // E-PENPAL-WATCHER: only .md file events trigger cache updates. Non-.md
+ // Create events (e.g., scanner temp files, backup artifacts) are ignored.
+ // However, directory Create/Rename events may contain .md files that need
+ // to be discovered, so we walk those directories before returning.
+ if !strings.HasSuffix(path, ".md") {
+ if event.Op&(fsnotify.Create|fsnotify.Rename) != 0 {
+ if info, err := os.Stat(path); err == nil && info.IsDir() {
+ filepath.Walk(path, func(p string, fi os.FileInfo, err error) error {
+ if err != nil {
+ return nil
+ }
+ if !fi.IsDir() && strings.HasSuffix(p, ".md") {
+ w.debounceFileEvent(projectName, p, fsnotify.Create)
+ }
+ return nil
+ })
+ }
+ }
return
}
// Record activity for .md file changes before debouncing
- if strings.HasSuffix(path, ".md") && w.activity != nil {
+ if w.activity != nil {
if project := w.cache.FindProject(projectName); project != nil {
if relPath, err := filepath.Rel(project.Path, path); err == nil {
evtType := activity.FileModified
@@ -596,11 +621,8 @@ func (w *Watcher) handleEvent(event fsnotify.Event) {
}
}
- w.debounceRefresh(projectName, func() {
- w.cache.RefreshProject(projectName)
- w.cache.RefreshProjectGitInfo(projectName)
- w.Broadcast(Event{Type: EventFilesChanged, Project: projectName})
- })
+ // E-PENPAL-WATCHER: accumulate file event for incremental cache update.
+ w.debounceFileEvent(projectName, path, event.Op)
}
// findProjectForPath finds which project a path belongs to by checking
@@ -654,3 +676,63 @@ func (w *Watcher) debounceRefresh(key string, fn func()) {
fn()
})
}
+
+// debounceFileEvent accumulates a file event for a project and schedules
+// an incremental cache update after the debounce window (100ms).
+// E-PENPAL-WATCHER: accumulating debounce for incremental file updates.
+func (w *Watcher) debounceFileEvent(projectName, absPath string, op fsnotify.Op) {
+ w.filePendingMu.Lock()
+ defer w.filePendingMu.Unlock()
+
+ if w.filePending[projectName] == nil {
+ w.filePending[projectName] = make(map[string]fsnotify.Op)
+ }
+ w.filePending[projectName][absPath] |= op
+
+ if timer, ok := w.fileTimers[projectName]; ok {
+ timer.Stop()
+ }
+
+ w.fileTimers[projectName] = time.AfterFunc(100*time.Millisecond, func() {
+ w.flushFileEvents(projectName)
+ })
+}
+
+// flushFileEvents drains accumulated file events for a project and applies
+// incremental cache mutations. Falls back to a full RefreshProject if the
+// project hasn't been scanned yet.
+// E-PENPAL-WATCHER: incremental cache update from accumulated file events.
+func (w *Watcher) flushFileEvents(projectName string) {
+ w.filePendingMu.Lock()
+ events := w.filePending[projectName]
+ delete(w.filePending, projectName)
+ delete(w.fileTimers, projectName)
+ w.filePendingMu.Unlock()
+
+ if len(events) == 0 {
+ return
+ }
+
+ project := w.cache.FindProject(projectName)
+ if project == nil {
+ return
+ }
+
+ // Apply incremental updates regardless of scan state. For unscanned
+ // projects this adds files to a partial cache; the lazy scan on first
+ // access will reconcile with a full walk.
+ for absPath, op := range events {
+ if op&(fsnotify.Remove|fsnotify.Rename) != 0 {
+ relPath, err := filepath.Rel(project.Path, absPath)
+ if err == nil {
+ w.cache.RemoveFile(projectName, relPath)
+ }
+ }
+ if op&(fsnotify.Create|fsnotify.Write) != 0 {
+ w.cache.UpsertFile(projectName, project, absPath)
+ }
+ }
+
+ w.cache.RefreshProjectGitInfo(projectName)
+ w.Broadcast(Event{Type: EventFilesChanged, Project: projectName})
+}
diff --git a/apps/penpal/junk.md b/apps/penpal/junk.md
new file mode 100644
index 00000000..1e272e5c
--- /dev/null
+++ b/apps/penpal/junk.md
@@ -0,0 +1,25 @@
+# The History of Some Mechanical Keyboards
+
+Mechanical keyboards have experienced a remarkable resurgence in popularity over the past decade, but their history stretches back to the earliest days of personal computing. Understanding how we got here requires a look at the evolution of keyboard technology and the communities that kept mechanical switches alive during the membrane era.
+
+## The Early Days
+
+The first computer keyboards were directly descended from typewriters. IBM's Model F, introduced in 1981 alongside the original IBM PC, used buckling spring switches that produced a distinctive tactile feel and audible click. This was followed by the legendary Model M in 1985, which refined the buckling spring mechanism and became one of the most beloved keyboards ever produced. Many Model M keyboards from the 1980s are still in daily use today, a testament to their build quality.
+
+During this period, all keyboards were effectively mechanical. The concept of a "mechanical keyboard" as a distinct category didn't exist because there was no alternative. Every key had its own discrete switch mechanism, and keyboards were built like tanks. They were also expensive, often costing the equivalent of several hundred dollars in today's money.
+
+## The Membrane Revolution
+
+The 1990s brought a dramatic shift. As personal computers moved from business tools to consumer products, cost pressure drove manufacturers toward membrane keyboards. Instead of individual switches, membrane keyboards use two flexible plastic sheets with conductive traces. When a key is pressed, the sheets make contact and complete a circuit. This design is dramatically cheaper to manufacture but sacrifices the tactile feedback and durability that mechanical switches provide.
+
+By the early 2000s, membrane keyboards had almost completely taken over the consumer market. Mechanical keyboards survived primarily in niche applications like point-of-sale terminals and some industrial equipment. The few enthusiasts who preferred mechanical switches had to hunt for remaining stock of older keyboards or pay premium prices for the limited models still in production.
+
+## The Renaissance
+
+The mechanical keyboard renaissance began around 2008-2010, driven by gaming communities and programmer forums. Cherry, a German company that had been manufacturing MX switches since 1983, suddenly found renewed demand for their products. The Cherry MX Blue, with its tactile bump and audible click, became the gateway switch for many newcomers. The Cherry MX Red, a smooth linear switch, found favor with gamers who wanted rapid key presses without tactile resistance.
+
+What truly accelerated the movement was the emergence of custom keyboard communities online. Forums and subreddits dedicated to mechanical keyboards created spaces where enthusiasts could share their builds, discuss switch preferences, and organize group buys for custom keycap sets. This community-driven approach transformed mechanical keyboards from a niche preference into a full-blown hobby with its own culture, vocabulary, and aesthetic sensibility.
+
+## The Modern Landscape
+
+Today the mechanical keyboard market is thriving. Dozens of switch manufacturers compete alongside Cherry, offering hundreds of switch variants with different actuation forces, travel distances, and sound profiles. Hot-swappable PCBs allow users to change switches without soldering. Custom keycap sets are produced in every color combination imaginable. The hobby continues to grow, blending engineering, aesthetics, and community in ways that the original IBM engineers could never have anticipated.