From d23a54ac552a9ffdd8f213c17ac41f16d6d7d55b Mon Sep 17 00:00:00 2001
From: Logan Johnson <loganj@squareup.com>
Date: Mon, 30 Mar 2026 00:55:58 -0400
Subject: [PATCH 1/6] fix(penpal): add concurrency limits and filter non-.md
 events

Add semaphore (cap 4) to RefreshAllProjects and CheckAllProjectsHasFiles
to prevent unbounded git check-ignore subprocess spawning at startup and
during workspace rescans.

Tighten the watcher Route 6 filter to only pass .md file events. Previously,
all Create events (including scanner temp files and backup artifacts) would
pass through and trigger a full project walk with a git check-ignore process.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 apps/penpal/ERD.md                      |  6 +++---
 apps/penpal/TESTING.md                  |  2 +-
 apps/penpal/internal/cache/cache.go     | 12 +++++++++---
 apps/penpal/internal/watcher/watcher.go |  7 ++++---
 4 files changed, 17 insertions(+), 10 deletions(-)

diff --git a/apps/penpal/ERD.md b/apps/penpal/ERD.md
index 1089932b..f713d1a5 100644
--- a/apps/penpal/ERD.md
+++ b/apps/penpal/ERD.md
@@ -98,10 +98,10 @@ see-also:
 
 ## Cache
 
-- <a id="E-PENPAL-CACHE"></a>**E-PENPAL-CACHE**: An in-memory cache (`sync.RWMutex`-protected) holds the full project list and per-project file lists. `RefreshProject()` walks the filesystem; `RefreshAllProjects()` runs in parallel with no concurrency limit. `RescanWith()` replaces the project list while preserving git enrichment.
+- <a id="E-PENPAL-CACHE"></a>**E-PENPAL-CACHE**: An in-memory cache (`sync.RWMutex`-protected) holds the full project list and per-project file lists. `RefreshProject()` walks the filesystem for full rescans; `RefreshAllProjects()` runs in parallel with a concurrency limit of 4. `RescanWith()` replaces the project list while preserving git enrichment and cached file data for unchanged projects — only new or source-changed projects are rescanned. Incremental mutations (`UpsertFile`, `RemoveFile`) update individual cache entries without walking the filesystem.
   ← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE)
 
-- <a id="E-PENPAL-SCAN"></a>**E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending. `EnsureProjectScanned()` is the lazy-scan entry point — it uses write-lock gating (`projectScanned` set under `mu.Lock` before scanning) to prevent concurrent requests from triggering duplicate filesystem walks. `projectHasAnyMarkdown()` performs a cheap startup check that aligns with the full scan: it uses the same gitignore checking, skips `.git`, `node_modules`, `.hg`, `.svn`, and nested worktree directories, and stops at the first `.md` file found.
+- <a id="E-PENPAL-SCAN"></a>**E-PENPAL-SCAN**: `scanProjectSources()` walks `RootPath` recursively for tree sources, skipping `.git`-file directories (nested worktrees), gitignored directories (via `git check-ignore`), source-type `SkipDirs`, and non-`.md` files. Gitignore checking is initialized once per scan via `newGitIgnoreChecker(projectPath)`, which detects whether the project is a git repo; non-git projects skip the gitignore check gracefully. On write or read failure (partial 4-field response), the checker disables itself (`isGitRepo=false`) to prevent permanent stream desync. The source's own `rootPath` is never checked against gitignore (the `path != rootPath` guard ensures registered sources always scan). Files returning `""` from `ClassifyFile()` are hidden. Files are de-duplicated by project-relative path (first source wins) and sorted by `ModTime` descending. `EnsureProjectScanned()` is the lazy-scan entry point — it uses write-lock gating (`projectScanned` set under `mu.Lock` before scanning) to prevent concurrent requests from triggering duplicate filesystem walks. `projectHasAnyMarkdown()` performs a cheap startup check that aligns with the full scan: it uses the same gitignore checking, skips `.git`, `node_modules`, `.hg`, `.svn`, and nested worktree directories, and stops at the first `.md` file found. `CheckAllProjectsHasFiles()` runs with a concurrency limit of 4 to cap subprocess spawning. `ResolveFileInfo()` resolves source membership for a single absolute path without spawning a git check-ignore process — it applies the same source-priority, SkipDirs, RequireSibling, and ClassifyFile rules as the full walk.
   ← [P-PENPAL-PROJECT-FILE-TREE](PRODUCT.md#P-PENPAL-PROJECT-FILE-TREE), [P-PENPAL-FILE-TYPES](PRODUCT.md#P-PENPAL-FILE-TYPES), [P-PENPAL-SRC-DEDUP](PRODUCT.md#P-PENPAL-SRC-DEDUP), [P-PENPAL-SRC-GITIGNORE](PRODUCT.md#P-PENPAL-SRC-GITIGNORE)
 
 - <a id="E-PENPAL-TITLE-EXTRACT"></a>**E-PENPAL-TITLE-EXTRACT**: `EnrichTitles()` reads the first 20 lines of each file to extract H1 headings. Titles are cached and shown as the primary display name when present.
@@ -239,7 +239,7 @@ see-also:
 - <a id="E-PENPAL-SSE"></a>**E-PENPAL-SSE**: `GET /events` is a long-lived SSE stream using `event: change` messages. Event types: `projects`, `files`, `comments`, `agents`, `navigate`. Each event carries optional `project`, `path`, `worktree` fields.
   ← [P-PENPAL-REALTIME](PRODUCT.md#P-PENPAL-REALTIME)
 
-- <a id="E-PENPAL-WATCHER"></a>**E-PENPAL-WATCHER**: The file watcher bridges `fsnotify` to SSE. Two-tier watch strategy: base (shallow workspace + project root directories) and dynamic (deep, per-focus). Debounce at 100ms per event key.
+- <a id="E-PENPAL-WATCHER"></a>**E-PENPAL-WATCHER**: The file watcher bridges `fsnotify` to SSE. Two-tier watch strategy: base (shallow workspace + project root directories) and dynamic (deep, per-focus). Debounce at 100ms per event key. File events use an accumulating debounce that collects per-file paths and ops during the debounce window, then applies incremental cache mutations (`UpsertFile`/`RemoveFile`) instead of full project walks. Only `.md` file events trigger cache updates — non-`.md` Create events (e.g., scanner temp files) are filtered out. Structural events (workspace changes, source auto-detect) still use the original debounce with full discovery.
   ← [P-PENPAL-REALTIME](PRODUCT.md#P-PENPAL-REALTIME), [P-PENPAL-LIVE-UPDATE](PRODUCT.md#P-PENPAL-LIVE-UPDATE)
 
 - <a id="E-PENPAL-FOCUS"></a>**E-PENPAL-FOCUS**: `windowFocuses map[string]focusTarget` (one entry per browser window) drives dynamic watches. Union of all window focuses determines the watched set. File focus watches only the file's parent directory. Project focus watches all source directories + `.penpal/comments/`.
diff --git a/apps/penpal/TESTING.md b/apps/penpal/TESTING.md
index a184ebaf..4a0e3746 100644
--- a/apps/penpal/TESTING.md
+++ b/apps/penpal/TESTING.md
@@ -64,7 +64,7 @@ see-also:
 | Source Types — anchors (P-PENPAL-SRC-ANCHORS, SRC-ANCHORS-GROUP, SRC-ANCHORS-NESTED) | discovery_test.go (TestClassifyAnchorsFile, TestGroupAnchorsPaths, TestGroupAnchorsPaths_MarkerOnlyModule, TestAnchorsFileOrder, TestAnchorsRequireSibling) | — | — | — |
 | Source Types — claude-plans (P-PENPAL-SRC-CLAUDE-PLANS) | — | — | — | — |
 | Source Types — manual (P-PENPAL-SRC-MANUAL) | — | — | grouping_test.go (TestBuildFileGroups_ManualSourceDirHeadings) | — |
-| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_SkipsGitignored, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans) | — | — | — |
+| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_SkipsGitignored, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans, TestResolveFileInfo, TestUpsertFile, TestRemoveFile, TestRescanWith_PreservesUnchangedProjects, TestSourcesChanged) | — | — | — |
 | Worktree Support (P-PENPAL-WORKTREE) | discovery/worktree_test.go, cache/worktree_test.go | Layout.test.tsx | worktree_test.go (API + MCP) | — |
 | Worktree Dropdown (P-PENPAL-PROJECT-WORKTREE-DROPDOWN) | — | Layout.test.tsx | — | — |
 | Git Integration (P-PENPAL-GIT-INFO) | — | — | — | — |
diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go
index a784dbea..6427b3f4 100644
--- a/apps/penpal/internal/cache/cache.go
+++ b/apps/penpal/internal/cache/cache.go
@@ -388,15 +388,18 @@ func (c *Cache) RefreshProject(projectName string) {
 	}
 }
 
-// RefreshAllProjects rescans all projects' files and updates metadata
-// E-PENPAL-CACHE: parallel refresh with no concurrency limit.
+// RefreshAllProjects rescans all projects' files and updates metadata.
+// E-PENPAL-CACHE: parallel refresh with concurrency limit of 4.
 func (c *Cache) RefreshAllProjects() {
 	projects := c.Projects()
 	var wg sync.WaitGroup
+	sem := make(chan struct{}, 4)
 	for _, p := range projects {
 		wg.Add(1)
 		go func(qn string) {
 			defer wg.Done()
+			sem <- struct{}{}
+			defer func() { <-sem }()
 			c.RefreshProject(qn)
 		}(p.QualifiedName())
 	}
@@ -410,14 +413,17 @@ var errFoundMarkdown = errors.New("found markdown")
 // CheckAllProjectsHasFiles does a cheap per-project check to set HasFiles
 // without doing a full file scan. For each project, it walks the project
 // root and stops as soon as it finds any .md file.
-// E-PENPAL-SCAN: lightweight startup check — no file list is built.
+// E-PENPAL-SCAN: lightweight startup check with concurrency limit of 4.
 func (c *Cache) CheckAllProjectsHasFiles() {
 	projects := c.Projects()
 	var wg sync.WaitGroup
+	sem := make(chan struct{}, 4)
 	for _, p := range projects {
 		wg.Add(1)
 		go func(p discovery.Project) {
 			defer wg.Done()
+			sem <- struct{}{}
+			defer func() { <-sem }()
 			found := projectHasAnyMarkdown(p.Path)
 			c.mu.Lock()
 			for i := range c.projects {
diff --git a/apps/penpal/internal/watcher/watcher.go b/apps/penpal/internal/watcher/watcher.go
index c5e96f32..29f82d00 100644
--- a/apps/penpal/internal/watcher/watcher.go
+++ b/apps/penpal/internal/watcher/watcher.go
@@ -578,13 +578,14 @@ func (w *Watcher) handleEvent(event fsnotify.Event) {
 		return
 	}
 
-	// Only care about .md files for file list updates
-	if !strings.HasSuffix(path, ".md") && event.Op&fsnotify.Create == 0 {
+	// E-PENPAL-WATCHER: only .md file events trigger cache updates. Non-.md
+	// Create events (e.g., scanner temp files, backup artifacts) are ignored.
+	if !strings.HasSuffix(path, ".md") {
 		return
 	}
 
 	// Record activity for .md file changes before debouncing
-	if strings.HasSuffix(path, ".md") && w.activity != nil {
+	if w.activity != nil {
 		if project := w.cache.FindProject(projectName); project != nil {
 			if relPath, err := filepath.Rel(project.Path, path); err == nil {
 				evtType := activity.FileModified

From 2d77480317aaaafe97132b10f1726db7034f674b Mon Sep 17 00:00:00 2001
From: Logan Johnson <loganj@squareup.com>
Date: Mon, 30 Mar 2026 00:58:39 -0400
Subject: [PATCH 2/6] feat(penpal): add incremental cache update methods

Add ResolveFileInfo for single-file source resolution without walking
the filesystem or spawning git check-ignore processes. Applies the same
source-priority, SkipDirs, RequireSibling, and ClassifyFile rules as
scanProjectSources.

Add UpsertFile and RemoveFile methods on Cache for incremental mutations.
UpsertFile updates ModTime/Title for existing entries or resolves source
membership for new files. RemoveFile removes entries by project-relative
path. Both update project metadata (HasFiles, LastModified).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 apps/penpal/internal/cache/cache.go      | 263 ++++++++++++++++++
 apps/penpal/internal/cache/cache_test.go | 335 +++++++++++++++++++++++
 2 files changed, 598 insertions(+)

diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go
index 6427b3f4..a10f34ef 100644
--- a/apps/penpal/internal/cache/cache.go
+++ b/apps/penpal/internal/cache/cache.go
@@ -592,6 +592,269 @@ func extractTitle(path string) string {
 	return ""
 }
 
+// ResolveFileInfo resolves source membership for a single absolute .md file path
+// within a project. It applies the same source-priority, SkipDirs, RequireSibling,
+// and ClassifyFile rules as scanProjectSources but without walking the filesystem
+// or spawning a git check-ignore process. Returns FileInfo entries for each source
+// that claims the file (typically one typed source + __all_markdown__). Returns nil
+// if no source claims the file.
+// E-PENPAL-SCAN: single-file source resolution for incremental cache updates.
+func ResolveFileInfo(project *discovery.Project, absPath string) []FileInfo {
+	if !strings.HasSuffix(absPath, ".md") {
+		return nil
+	}
+
+	info, err := os.Stat(absPath)
+	if err != nil {
+		return nil
+	}
+
+	relToProject, err := filepath.Rel(project.Path, absPath)
+	if err != nil || strings.HasPrefix(relToProject, "..") {
+		return nil
+	}
+
+	title := extractTitle(absPath)
+	var results []FileInfo
+	typedClaimed := false
+
+	for _, source := range project.Sources {
+		isAllMarkdown := source.Name == "__all_markdown__"
+
+		if source.Type == "thoughts" || source.Type == "tree" {
+			rootPath := source.RootPath
+			if rootPath == "" {
+				continue
+			}
+
+			// Check containment
+			if !strings.HasPrefix(absPath, rootPath+"/") && absPath != rootPath {
+				continue
+			}
+
+			relToSource, err := filepath.Rel(rootPath, absPath)
+			if err != nil {
+				continue
+			}
+
+			// Check SkipDirs against each path component
+			stName := source.SourceTypeName
+			if stName == "" {
+				stName = source.Name
+			}
+			st := discovery.GetSourceType(stName)
+
+			if !isAllMarkdown && typedClaimed {
+				continue // already claimed by an earlier typed source
+			}
+
+			if hasSkippedDir(relToSource, st) {
+				continue
+			}
+
+			// RequireSibling check
+			if st != nil && st.RequireSibling != "" {
+				siblingPath := filepath.Join(filepath.Dir(absPath), st.RequireSibling)
+				if _, err := os.Stat(siblingPath); err != nil {
+					continue
+				}
+			}
+
+			// ClassifyFile
+			fileType := "other"
+			if st != nil && st.ClassifyFile != nil {
+				fileType = st.ClassifyFile(relToSource)
+				if fileType == "" {
+					continue // skip this file for this source
+				}
+			} else {
+				if strings.Contains(relToSource, "research") {
+					fileType = "research"
+				} else if strings.Contains(relToSource, "plan") {
+					fileType = "plan"
+				}
+			}
+
+			if !isAllMarkdown {
+				typedClaimed = true
+			}
+
+			results = append(results, FileInfo{
+				Project:     project.QualifiedName(),
+				Workspace:   project.WorkspaceName,
+				ProjectPath: project.Path,
+				Source:      source.Name,
+				SourceType:  source.Type,
+				SourceAuto:  source.Auto,
+				Path:        relToSource,
+				FullPath:    relToProject,
+				Name:        filepath.Base(absPath),
+				Title:       title,
+				ModTime:     info.ModTime(),
+				FileType:    fileType,
+			})
+
+		} else if source.Type == "files" {
+			if !isAllMarkdown && typedClaimed {
+				continue
+			}
+			found := false
+			for _, f := range source.Files {
+				if f == absPath {
+					found = true
+					break
+				}
+			}
+			if !found {
+				continue
+			}
+
+			fileType := "other"
+			lower := strings.ToLower(filepath.Base(absPath))
+			if strings.Contains(lower, "research") {
+				fileType = "research"
+			} else if strings.Contains(lower, "plan") {
+				fileType = "plan"
+			}
+
+			if !isAllMarkdown {
+				typedClaimed = true
+			}
+
+			results = append(results, FileInfo{
+				Project:     project.QualifiedName(),
+				Workspace:   project.WorkspaceName,
+				ProjectPath: project.Path,
+				Source:      source.Name,
+				SourceType:  source.Type,
+				SourceAuto:  source.Auto,
+				Path:        filepath.Base(absPath),
+				FullPath:    relToProject,
+				Name:        filepath.Base(absPath),
+				Title:       title,
+				ModTime:     info.ModTime(),
+				FileType:    fileType,
+			})
+		}
+	}
+
+	return results
+}
+
+// hasSkippedDir checks whether any directory component between the source root
+// and the file matches the source type's SkipDirs.
+func hasSkippedDir(relToSource string, st *discovery.SourceType) bool {
+	if st == nil || len(st.SkipDirs) == 0 {
+		return false
+	}
+	dir := filepath.Dir(relToSource)
+	if dir == "." {
+		return false
+	}
+	for _, component := range strings.Split(dir, string(filepath.Separator)) {
+		if st.SkipDirs[component] {
+			return true
+		}
+	}
+	return false
+}
+
+// UpsertFile adds or updates file entries in the cache for the given absolute path.
+// For existing entries (matched by FullPath), re-stats for ModTime and re-extracts
+// Title. For new files, resolves source membership and inserts.
+// E-PENPAL-CACHE: incremental cache mutation without filesystem walk.
+func (c *Cache) UpsertFile(projectName string, project *discovery.Project, absPath string) bool {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	files := c.projectFiles[projectName]
+
+	// Check if any entries already exist for this path
+	relToProject, err := filepath.Rel(project.Path, absPath)
+	if err != nil {
+		return false
+	}
+
+	info, err := os.Stat(absPath)
+	if err != nil {
+		return false
+	}
+
+	updated := false
+	for i := range files {
+		if files[i].FullPath == relToProject {
+			files[i].ModTime = info.ModTime()
+			files[i].Title = extractTitle(absPath)
+			updated = true
+		}
+	}
+
+	if updated {
+		sort.Slice(files, func(i, j int) bool {
+			return files[i].ModTime.After(files[j].ModTime)
+		})
+		c.projectFiles[projectName] = files
+		c.updateProjectMetadataLocked(projectName, files)
+		return true
+	}
+
+	// New file — resolve source membership
+	resolved := ResolveFileInfo(project, absPath)
+	if len(resolved) == 0 {
+		return false
+	}
+
+	files = append(files, resolved...)
+	sort.Slice(files, func(i, j int) bool {
+		return files[i].ModTime.After(files[j].ModTime)
+	})
+	c.projectFiles[projectName] = files
+	c.updateProjectMetadataLocked(projectName, files)
+	return true
+}
+
+// RemoveFile removes all cache entries with the given project-relative path.
+// E-PENPAL-CACHE: incremental cache mutation without filesystem walk.
+func (c *Cache) RemoveFile(projectName, fullPath string) bool {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	files := c.projectFiles[projectName]
+	if files == nil {
+		return false
+	}
+
+	n := 0
+	for _, f := range files {
+		if f.FullPath != fullPath {
+			files[n] = f
+			n++
+		}
+	}
+	if n == len(files) {
+		return false // nothing removed
+	}
+
+	files = files[:n]
+	c.projectFiles[projectName] = files
+	c.updateProjectMetadataLocked(projectName, files)
+	return true
+}
+
+// updateProjectMetadataLocked updates HasFiles and LastModified for a project.
+// Must be called with c.mu held for writing.
+func (c *Cache) updateProjectMetadataLocked(projectName string, files []FileInfo) {
+	for i := range c.projects {
+		if c.projects[i].QualifiedName() == projectName {
+			c.projects[i].HasFiles = len(files) > 0
+			if len(files) > 0 {
+				c.projects[i].LastModified = files[0].ModTime
+			}
+			break
+		}
+	}
+}
+
 // ScanProjectSourcesForWorktree scans a project's sources remapped to a worktree path.
 // Each source's RootPath under the project is remapped to the equivalent path under
 // the worktree. Sources whose directory doesn't exist in the worktree are skipped.
diff --git a/apps/penpal/internal/cache/cache_test.go b/apps/penpal/internal/cache/cache_test.go
index 19e4e6cf..a9cf30ba 100644
--- a/apps/penpal/internal/cache/cache_test.go
+++ b/apps/penpal/internal/cache/cache_test.go
@@ -701,3 +701,338 @@ func TestCache_FindFile(t *testing.T) {
 		})
 	}
 }
+
+// E-PENPAL-SCAN: verifies single-file source resolution matches scan behavior.
+func TestResolveFileInfo_ThoughtsSource(t *testing.T) {
+	tmpDir := t.TempDir()
+	thoughtsDir := filepath.Join(tmpDir, "thoughts")
+	os.MkdirAll(filepath.Join(thoughtsDir, "research"), 0755)
+
+	filePath := filepath.Join(thoughtsDir, "research", "topic.md")
+	os.WriteFile(filePath, []byte("# My Research"), 0644)
+
+	project := &discovery.Project{
+		Name: "test",
+		Path: tmpDir,
+		Sources: []discovery.FileSource{
+			{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: thoughtsDir, Auto: true},
+			{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+		},
+	}
+
+	results := ResolveFileInfo(project, filePath)
+	if len(results) != 2 {
+		t.Fatalf("expected 2 results (thoughts + __all_markdown__), got %d", len(results))
+	}
+
+	// First result should be the typed source
+	if results[0].Source != "thoughts" {
+		t.Errorf("expected first source 'thoughts', got %q", results[0].Source)
+	}
+	if results[0].FileType != "research" {
+		t.Errorf("expected fileType 'research', got %q", results[0].FileType)
+	}
+	if results[0].Title != "My Research" {
+		t.Errorf("expected title 'My Research', got %q", results[0].Title)
+	}
+	if results[0].FullPath != "thoughts/research/topic.md" {
+		t.Errorf("expected fullPath 'thoughts/research/topic.md', got %q", results[0].FullPath)
+	}
+
+	// Second result should be __all_markdown__
+	if results[1].Source != "__all_markdown__" {
+		t.Errorf("expected second source '__all_markdown__', got %q", results[1].Source)
+	}
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo respects SkipDirs.
+func TestResolveFileInfo_SkipDirs(t *testing.T) {
+	tmpDir := t.TempDir()
+	os.MkdirAll(filepath.Join(tmpDir, "node_modules", "pkg"), 0755)
+	filePath := filepath.Join(tmpDir, "node_modules", "pkg", "readme.md")
+	os.WriteFile(filePath, []byte("# Dep"), 0644)
+
+	project := &discovery.Project{
+		Name: "test",
+		Path: tmpDir,
+		Sources: []discovery.FileSource{
+			{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+		},
+	}
+
+	results := ResolveFileInfo(project, filePath)
+	// __all_markdown__ has SkipDirs for node_modules
+	if len(results) != 0 {
+		t.Fatalf("expected 0 results (node_modules is in SkipDirs), got %d", len(results))
+	}
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo respects RequireSibling.
+func TestResolveFileInfo_RequireSibling(t *testing.T) {
+	tmpDir := t.TempDir()
+
+	// Directory WITH ANCHORS.md sibling
+	withSibling := filepath.Join(tmpDir, "module-a")
+	os.MkdirAll(withSibling, 0755)
+	os.WriteFile(filepath.Join(withSibling, "ANCHORS.md"), []byte("---\nprefix: A\n---\n"), 0644)
+	os.WriteFile(filepath.Join(withSibling, "PRODUCT.md"), []byte("# Product"), 0644)
+
+	// Directory WITHOUT ANCHORS.md sibling
+	withoutSibling := filepath.Join(tmpDir, "module-b")
+	os.MkdirAll(withoutSibling, 0755)
+	os.WriteFile(filepath.Join(withoutSibling, "PRODUCT.md"), []byte("# Orphan"), 0644)
+
+	project := &discovery.Project{
+		Name: "test",
+		Path: tmpDir,
+		Sources: []discovery.FileSource{
+			{Name: "anchors", Type: "tree", SourceTypeName: "anchors", RootPath: tmpDir, Auto: true},
+		},
+	}
+
+	// File with sibling should be included
+	results := ResolveFileInfo(project, filepath.Join(withSibling, "PRODUCT.md"))
+	if len(results) != 1 {
+		t.Fatalf("expected 1 result for file with ANCHORS.md sibling, got %d", len(results))
+	}
+
+	// File without sibling should be excluded
+	results = ResolveFileInfo(project, filepath.Join(withoutSibling, "PRODUCT.md"))
+	if len(results) != 0 {
+		t.Fatalf("expected 0 results for file without ANCHORS.md sibling, got %d", len(results))
+	}
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo returns nil for non-.md files.
+func TestResolveFileInfo_NonMdFile(t *testing.T) {
+	tmpDir := t.TempDir()
+	filePath := filepath.Join(tmpDir, "readme.txt")
+	os.WriteFile(filePath, []byte("hello"), 0644)
+
+	project := &discovery.Project{
+		Name: "test",
+		Path: tmpDir,
+		Sources: []discovery.FileSource{
+			{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+		},
+	}
+
+	results := ResolveFileInfo(project, filePath)
+	if len(results) != 0 {
+		t.Fatalf("expected 0 results for non-.md file, got %d", len(results))
+	}
+}
+
+// E-PENPAL-SCAN: ResolveFileInfo dedup — first typed source wins.
+func TestResolveFileInfo_SourcePriority(t *testing.T) {
+	tmpDir := t.TempDir()
+	thoughtsDir := filepath.Join(tmpDir, "thoughts")
+	os.MkdirAll(thoughtsDir, 0755)
+	filePath := filepath.Join(thoughtsDir, "plan.md")
+	os.WriteFile(filePath, []byte("# Plan"), 0644)
+
+	project := &discovery.Project{
+		Name: "test",
+		Path: tmpDir,
+		Sources: []discovery.FileSource{
+			{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: thoughtsDir, Auto: true},
+			// A second typed source covering the same path
+			{Name: "manual", Type: "tree", SourceTypeName: "manual", RootPath: thoughtsDir},
+			{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+		},
+	}
+
+	results := ResolveFileInfo(project, filePath)
+	// Should get 2: first typed source (thoughts) + __all_markdown__
+	// The second typed source (manual) should be skipped
+	if len(results) != 2 {
+		t.Fatalf("expected 2 results, got %d", len(results))
+	}
+	if results[0].Source != "thoughts" {
+		t.Errorf("expected first source 'thoughts', got %q", results[0].Source)
+	}
+	if results[1].Source != "__all_markdown__" {
+		t.Errorf("expected second source '__all_markdown__', got %q", results[1].Source)
+	}
+}
+
+// E-PENPAL-CACHE: verifies UpsertFile updates existing entries.
+func TestUpsertFile_ExistingFile(t *testing.T) {
+	tmpDir := t.TempDir()
+	filePath := filepath.Join(tmpDir, "thoughts", "plan.md")
+	os.MkdirAll(filepath.Join(tmpDir, "thoughts"), 0755)
+	os.WriteFile(filePath, []byte("# Old Title"), 0644)
+
+	c := New()
+	projectName := "test"
+	c.SetProjects([]discovery.Project{
+		{Name: "test", Path: tmpDir, Sources: []discovery.FileSource{
+			{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: filepath.Join(tmpDir, "thoughts"), Auto: true},
+		}},
+	})
+
+	// Pre-populate cache with an entry
+	c.SetProjectFiles(projectName, []FileInfo{
+		{Project: "test", Source: "thoughts", FullPath: "thoughts/plan.md", Name: "plan.md", Title: "Old Title", ModTime: time.Now().Add(-1 * time.Hour)},
+	})
+
+	// Update the file on disk
+	os.WriteFile(filePath, []byte("# New Title"), 0644)
+
+	project := c.FindProject(projectName)
+	ok := c.UpsertFile(projectName, project, filePath)
+	if !ok {
+		t.Fatal("UpsertFile returned false, expected true")
+	}
+
+	files := c.ProjectFiles(projectName)
+	if len(files) != 1 {
+		t.Fatalf("expected 1 file, got %d", len(files))
+	}
+	if files[0].Title != "New Title" {
+		t.Errorf("expected title 'New Title', got %q", files[0].Title)
+	}
+}
+
+// E-PENPAL-CACHE: verifies UpsertFile adds new files via source resolution.
+func TestUpsertFile_NewFile(t *testing.T) {
+	tmpDir := t.TempDir()
+	thoughtsDir := filepath.Join(tmpDir, "thoughts")
+	os.MkdirAll(thoughtsDir, 0755)
+	filePath := filepath.Join(thoughtsDir, "new-note.md")
+	os.WriteFile(filePath, []byte("# Fresh Note"), 0644)
+
+	c := New()
+	projectName := "test"
+	project := discovery.Project{
+		Name: "test", Path: tmpDir,
+		Sources: []discovery.FileSource{
+			{Name: "thoughts", Type: "tree", SourceTypeName: "thoughts", RootPath: thoughtsDir, Auto: true},
+			{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+		},
+	}
+	c.SetProjects([]discovery.Project{project})
+	c.SetProjectFiles(projectName, nil) // empty cache
+
+	ok := c.UpsertFile(projectName, &project, filePath)
+	if !ok {
+		t.Fatal("UpsertFile returned false, expected true")
+	}
+
+	files := c.ProjectFiles(projectName)
+	if len(files) != 2 {
+		t.Fatalf("expected 2 files (thoughts + __all_markdown__), got %d", len(files))
+	}
+
+	// Check the typed source entry
+	found := false
+	for _, f := range files {
+		if f.Source == "thoughts" && f.FullPath == "thoughts/new-note.md" {
+			found = true
+			if f.Title != "Fresh Note" {
+				t.Errorf("expected title 'Fresh Note', got %q", f.Title)
+			}
+		}
+	}
+	if !found {
+		t.Error("expected thoughts source entry for new file")
+	}
+}
+
+// E-PENPAL-CACHE: verifies UpsertFile returns false for excluded files.
+func TestUpsertFile_ExcludedFile(t *testing.T) {
+	tmpDir := t.TempDir()
+	os.MkdirAll(filepath.Join(tmpDir, "node_modules"), 0755)
+	filePath := filepath.Join(tmpDir, "node_modules", "readme.md")
+	os.WriteFile(filePath, []byte("# Dep"), 0644)
+
+	c := New()
+	projectName := "test"
+	project := discovery.Project{
+		Name: "test", Path: tmpDir,
+		Sources: []discovery.FileSource{
+			{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: tmpDir, Auto: true},
+		},
+	}
+	c.SetProjects([]discovery.Project{project})
+	c.SetProjectFiles(projectName, nil)
+
+	ok := c.UpsertFile(projectName, &project, filePath)
+	if ok {
+		t.Error("UpsertFile should return false for file in SkipDirs")
+	}
+
+	files := c.ProjectFiles(projectName)
+	if len(files) != 0 {
+		t.Fatalf("expected 0 files, got %d", len(files))
+	}
+}
+
+// E-PENPAL-CACHE: verifies RemoveFile removes entries and updates metadata.
+func TestRemoveFile(t *testing.T) {
+	now := time.Now()
+	older := now.Add(-1 * time.Hour)
+
+	c := New()
+	projectName := "test"
+	c.SetProjects([]discovery.Project{
+		{Name: "test", Path: "/tmp/test"},
+	})
+	c.SetProjectFiles(projectName, []FileInfo{
+		{Project: "test", Source: "thoughts", FullPath: "thoughts/plan.md", Name: "plan.md", ModTime: now},
+		{Project: "test", Source: "thoughts", FullPath: "thoughts/old.md", Name: "old.md", ModTime: older},
+	})
+
+	ok := c.RemoveFile(projectName, "thoughts/plan.md")
+	if !ok {
+		t.Fatal("RemoveFile returned false, expected true")
+	}
+
+	files := c.ProjectFiles(projectName)
+	if len(files) != 1 {
+		t.Fatalf("expected 1 file, got %d", len(files))
+	}
+	if files[0].FullPath != "thoughts/old.md" {
+		t.Errorf("expected remaining file 'thoughts/old.md', got %q", files[0].FullPath)
+	}
+
+	// Verify metadata updated
+	project := c.FindProject(projectName)
+	if !project.HasFiles {
+		t.Error("project should still have files")
+	}
+	if !project.LastModified.Equal(older) {
+		t.Errorf("expected LastModified to be older time, got %v", project.LastModified)
+	}
+}
+
+// E-PENPAL-CACHE: verifies RemoveFile returns false for non-existent entries.
+func TestRemoveFile_NotFound(t *testing.T) {
+	c := New()
+	c.SetProjectFiles("test", []FileInfo{
+		{Project: "test", FullPath: "readme.md"},
+	})
+
+	ok := c.RemoveFile("test", "nonexistent.md")
+	if ok {
+		t.Error("RemoveFile should return false for non-existent file")
+	}
+}
+
+// E-PENPAL-CACHE: verifies RemoveFile clears HasFiles when last file removed.
+func TestRemoveFile_ClearsHasFiles(t *testing.T) {
+	c := New()
+	c.SetProjects([]discovery.Project{
+		{Name: "test", Path: "/tmp/test", HasFiles: true},
+	})
+	c.SetProjectFiles("test", []FileInfo{
+		{Project: "test", FullPath: "only.md", ModTime: time.Now()},
+	})
+
+	c.RemoveFile("test", "only.md")
+
+	project := c.FindProject("test")
+	if project.HasFiles {
+		t.Error("project should have HasFiles=false after removing last file")
+	}
+}

From 11228bdd7586baaf813dcab0f5138b67c0da6d8a Mon Sep 17 00:00:00 2001
From: Logan Johnson <loganj@squareup.com>
Date: Mon, 30 Mar 2026 00:59:50 -0400
Subject: [PATCH 3/6] feat(penpal): wire watcher to incremental cache updates
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Replace the per-project debounceRefresh (which triggered full
RefreshProject walks) with an accumulating debounce that collects
per-file paths and ops during the 100ms window, then applies
incremental UpsertFile/RemoveFile mutations.

This eliminates filesystem walks and git check-ignore subprocess
spawning for the common case of editing .md files. The watcher now
handles Write → re-stat + re-title, Create → resolve source + insert,
and Remove/Rename → remove from cache, all without touching the
filesystem beyond the specific file that changed.

Add IsProjectScanned method to support the guard that skips incremental
updates for projects that haven't been lazily scanned yet.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 apps/penpal/internal/cache/cache.go     |  7 +++
 apps/penpal/internal/watcher/watcher.go | 79 +++++++++++++++++++++++--
 2 files changed, 81 insertions(+), 5 deletions(-)

diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go
index a10f34ef..a8e93787 100644
--- a/apps/penpal/internal/cache/cache.go
+++ b/apps/penpal/internal/cache/cache.go
@@ -342,6 +342,13 @@ func (c *Cache) FindFile(projectName, filePath string) *FileInfo {
 	return nil
 }
 
+// IsProjectScanned returns whether a project has had a full file scan.
+func (c *Cache) IsProjectScanned(projectName string) bool {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return c.projectScanned[projectName]
+}
+
 // EnsureProjectScanned triggers a full file scan for a project if it hasn't
 // been scanned yet. Returns true if a scan was actually performed (first call
 // for this project). This is the lazy-scan entry point — called when a user
diff --git a/apps/penpal/internal/watcher/watcher.go b/apps/penpal/internal/watcher/watcher.go
index 29f82d00..acc4da19 100644
--- a/apps/penpal/internal/watcher/watcher.go
+++ b/apps/penpal/internal/watcher/watcher.go
@@ -70,6 +70,13 @@ type Watcher struct {
 	windowFocuses  map[string]focusTarget
 	baseWatched    map[string]struct{}
 	dynamicWatched map[string]struct{}
+
+	// E-PENPAL-WATCHER: accumulating debounce for incremental file updates.
+	// Collects per-file paths and ops during the debounce window, then applies
+	// incremental cache mutations instead of full project walks.
+	filePending   map[string]map[string]fsnotify.Op // projectName → absPath → accumulated op
+	filePendingMu sync.Mutex
+	fileTimers    map[string]*time.Timer // projectName → debounce timer
 }
 
 // New creates a new watcher
@@ -90,6 +97,8 @@ func New(c *cache.Cache, act *activity.Tracker) (*Watcher, error) {
 		windowFocuses:  make(map[string]focusTarget),
 		baseWatched:    make(map[string]struct{}),
 		dynamicWatched: make(map[string]struct{}),
+		filePending:    make(map[string]map[string]fsnotify.Op),
+		fileTimers:     make(map[string]*time.Timer),
 	}
 
 	return w, nil
@@ -597,11 +606,8 @@ func (w *Watcher) handleEvent(event fsnotify.Event) {
 		}
 	}
 
-	w.debounceRefresh(projectName, func() {
-		w.cache.RefreshProject(projectName)
-		w.cache.RefreshProjectGitInfo(projectName)
-		w.Broadcast(Event{Type: EventFilesChanged, Project: projectName})
-	})
+	// E-PENPAL-WATCHER: accumulate file event for incremental cache update.
+	w.debounceFileEvent(projectName, path, event.Op)
 }
 
 // findProjectForPath finds which project a path belongs to by checking
@@ -655,3 +661,66 @@ func (w *Watcher) debounceRefresh(key string, fn func()) {
 		fn()
 	})
 }
+
+// debounceFileEvent accumulates a file event for a project and schedules
+// an incremental cache update after the debounce window (100ms).
+// E-PENPAL-WATCHER: accumulating debounce for incremental file updates.
+func (w *Watcher) debounceFileEvent(projectName, absPath string, op fsnotify.Op) {
+	w.filePendingMu.Lock()
+	defer w.filePendingMu.Unlock()
+
+	if w.filePending[projectName] == nil {
+		w.filePending[projectName] = make(map[string]fsnotify.Op)
+	}
+	w.filePending[projectName][absPath] |= op
+
+	if timer, ok := w.fileTimers[projectName]; ok {
+		timer.Stop()
+	}
+
+	w.fileTimers[projectName] = time.AfterFunc(100*time.Millisecond, func() {
+		w.flushFileEvents(projectName)
+	})
+}
+
+// flushFileEvents drains accumulated file events for a project and applies
+// incremental cache mutations. Falls back to a full RefreshProject if the
+// project hasn't been scanned yet.
+// E-PENPAL-WATCHER: incremental cache update from accumulated file events.
+func (w *Watcher) flushFileEvents(projectName string) {
+	w.filePendingMu.Lock()
+	events := w.filePending[projectName]
+	delete(w.filePending, projectName)
+	delete(w.fileTimers, projectName)
+	w.filePendingMu.Unlock()
+
+	if len(events) == 0 {
+		return
+	}
+
+	project := w.cache.FindProject(projectName)
+	if project == nil {
+		return
+	}
+
+	// If the project hasn't been lazily scanned yet, skip incremental
+	// updates — the full scan on first access will pick these up.
+	if !w.cache.IsProjectScanned(projectName) {
+		return
+	}
+
+	for absPath, op := range events {
+		if op&(fsnotify.Remove|fsnotify.Rename) != 0 {
+			relPath, err := filepath.Rel(project.Path, absPath)
+			if err == nil {
+				w.cache.RemoveFile(projectName, relPath)
+			}
+		}
+		if op&(fsnotify.Create|fsnotify.Write) != 0 {
+			w.cache.UpsertFile(projectName, project, absPath)
+		}
+	}
+
+	w.cache.RefreshProjectGitInfo(projectName)
+	w.Broadcast(Event{Type: EventFilesChanged, Project: projectName})
+}

From cd9df2f8fc558cb6fc3115d06875d21bd513b79c Mon Sep 17 00:00:00 2001
From: Logan Johnson <loganj@squareup.com>
Date: Mon, 30 Mar 2026 01:02:22 -0400
Subject: [PATCH 4/6] feat(penpal): optimize RescanWith, handleDeleteFile, and
 config refresh

RescanWith now preserves cached file data for unchanged projects instead
of rescanning everything. It diffs old vs new project lists, compares
source configurations via SourcesChanged, and only walks projects that
are genuinely new or whose sources changed. Removed projects are cleaned
up from the cache.

handleDeleteFile uses RemoveFile for incremental cache removal instead
of a full RefreshProject walk.

refreshAfterConfigChange no longer calls populateProjects (which
redundantly ran CheckAllProjectsHasFiles). Instead it does targeted
git enrichment only for projects missing git info.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 apps/penpal/internal/cache/cache.go      |  88 +++++++++++++-
 apps/penpal/internal/cache/cache_test.go | 145 +++++++++++++++++++++++
 apps/penpal/internal/server/manage.go    |  32 ++++-
 apps/penpal/internal/server/server.go    |   4 +-
 4 files changed, 263 insertions(+), 6 deletions(-)

diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go
index a8e93787..f07672c8 100644
--- a/apps/penpal/internal/cache/cache.go
+++ b/apps/penpal/internal/cache/cache.go
@@ -533,14 +533,22 @@ func (c *Cache) RefreshProjectGitInfo(name string) {
 // preserving existing git info for known projects.
 // E-PENPAL-CACHE: replaces the project list while preserving git enrichment.
 func (c *Cache) RescanWith(projects []discovery.Project) {
-	// Preserve enrichment data (git info) for projects we already know about
+	// Snapshot current state before replacing
 	c.mu.RLock()
 	existing := make(map[string]discovery.Project)
+	existingScanned := make(map[string]bool)
+	existingFiles := make(map[string][]FileInfo)
 	for _, p := range c.projects {
-		existing[p.QualifiedName()] = p
+		qn := p.QualifiedName()
+		existing[qn] = p
+		existingScanned[qn] = c.projectScanned[qn]
+		if files, ok := c.projectFiles[qn]; ok {
+			existingFiles[qn] = files
+		}
 	}
 	c.mu.RUnlock()
 
+	// Preserve git enrichment
 	for i := range projects {
 		if prev, ok := existing[projects[i].QualifiedName()]; ok {
 			projects[i].Git = prev.Git
@@ -548,7 +556,81 @@ func (c *Cache) RescanWith(projects []discovery.Project) {
 	}
 
 	c.SetProjects(projects)
-	c.RefreshAllProjects()
+
+	// Determine which projects need scanning
+	var toScan []string
+	newNames := make(map[string]bool, len(projects))
+	for _, p := range projects {
+		qn := p.QualifiedName()
+		newNames[qn] = true
+
+		prev, existed := existing[qn]
+		if !existed {
+			// New project: needs scan
+			toScan = append(toScan, qn)
+		} else if !existingScanned[qn] {
+			// Existed but never scanned: needs scan
+			toScan = append(toScan, qn)
+		} else if SourcesChanged(prev.Sources, p.Sources) {
+			// Sources changed: needs rescan
+			toScan = append(toScan, qn)
+		} else {
+			// Unchanged: preserve cached files
+			c.mu.Lock()
+			c.projectFiles[qn] = existingFiles[qn]
+			c.projectScanned[qn] = true
+			c.mu.Unlock()
+		}
+	}
+
+	// Clean up removed projects
+	c.mu.Lock()
+	for name := range existingFiles {
+		if !newNames[name] {
+			delete(c.projectFiles, name)
+			delete(c.projectScanned, name)
+		}
+	}
+	c.mu.Unlock()
+
+	// Scan only the projects that need it, with concurrency limit
+	if len(toScan) > 0 {
+		var wg sync.WaitGroup
+		sem := make(chan struct{}, 4)
+		for _, qn := range toScan {
+			wg.Add(1)
+			go func(qn string) {
+				defer wg.Done()
+				sem <- struct{}{}
+				defer func() { <-sem }()
+				c.RefreshProject(qn)
+			}(qn)
+		}
+		wg.Wait()
+	}
+}
+
+// SourcesChanged returns true if two source lists differ materially.
+// E-PENPAL-CACHE: used by RescanWith to detect which projects need rescanning.
+func SourcesChanged(a, b []discovery.FileSource) bool {
+	if len(a) != len(b) {
+		return true
+	}
+	for i := range a {
+		if a[i].Name != b[i].Name || a[i].Type != b[i].Type ||
+			a[i].RootPath != b[i].RootPath || a[i].SourceTypeName != b[i].SourceTypeName {
+			return true
+		}
+		if len(a[i].Files) != len(b[i].Files) {
+			return true
+		}
+		for j := range a[i].Files {
+			if a[i].Files[j] != b[i].Files[j] {
+				return true
+			}
+		}
+	}
+	return false
 }
 
 // EnrichTitles fills in missing Title fields for files in the given project.
diff --git a/apps/penpal/internal/cache/cache_test.go b/apps/penpal/internal/cache/cache_test.go
index a9cf30ba..75c501b8 100644
--- a/apps/penpal/internal/cache/cache_test.go
+++ b/apps/penpal/internal/cache/cache_test.go
@@ -1036,3 +1036,148 @@ func TestRemoveFile_ClearsHasFiles(t *testing.T) {
 		t.Error("project should have HasFiles=false after removing last file")
 	}
 }
+
+// E-PENPAL-CACHE: verifies SourcesChanged detects material differences.
+func TestSourcesChanged(t *testing.T) {
+	base := []discovery.FileSource{
+		{Name: "thoughts", Type: "tree", RootPath: "/a/thoughts", SourceTypeName: "thoughts"},
+		{Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+	}
+
+	// Identical
+	same := []discovery.FileSource{
+		{Name: "thoughts", Type: "tree", RootPath: "/a/thoughts", SourceTypeName: "thoughts"},
+		{Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+	}
+	if SourcesChanged(base, same) {
+		t.Error("identical sources should not be reported as changed")
+	}
+
+	// Different count
+	fewer := base[:1]
+	if !SourcesChanged(base, fewer) {
+		t.Error("different count should be reported as changed")
+	}
+
+	// Different root path
+	moved := []discovery.FileSource{
+		{Name: "thoughts", Type: "tree", RootPath: "/b/thoughts", SourceTypeName: "thoughts"},
+		{Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+	}
+	if !SourcesChanged(base, moved) {
+		t.Error("different RootPath should be reported as changed")
+	}
+
+	// Different name
+	renamed := []discovery.FileSource{
+		{Name: "rp1", Type: "tree", RootPath: "/a/thoughts", SourceTypeName: "rp1"},
+		{Name: "__all_markdown__", Type: "tree", RootPath: "/a", SourceTypeName: "__all_markdown__"},
+	}
+	if !SourcesChanged(base, renamed) {
+		t.Error("different Name should be reported as changed")
+	}
+
+	// Different files list
+	withFiles := []discovery.FileSource{
+		{Name: "manual", Type: "files", Files: []string{"/a/foo.md"}},
+	}
+	withDiffFiles := []discovery.FileSource{
+		{Name: "manual", Type: "files", Files: []string{"/a/bar.md"}},
+	}
+	if !SourcesChanged(withFiles, withDiffFiles) {
+		t.Error("different Files should be reported as changed")
+	}
+}
+
+// E-PENPAL-CACHE: verifies RescanWith preserves cache for unchanged projects.
+func TestRescanWith_PreservesUnchangedProjects(t *testing.T) {
+	tmpDir := t.TempDir()
+
+	// Set up two projects
+	projADir := filepath.Join(tmpDir, "proj-a")
+	projBDir := filepath.Join(tmpDir, "proj-b")
+	os.MkdirAll(projADir, 0755)
+	os.MkdirAll(projBDir, 0755)
+	os.WriteFile(filepath.Join(projADir, "readme.md"), []byte("# A"), 0644)
+	os.WriteFile(filepath.Join(projBDir, "readme.md"), []byte("# B"), 0644)
+
+	sourcesA := []discovery.FileSource{
+		{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: projADir, Auto: true},
+	}
+	sourcesB := []discovery.FileSource{
+		{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: projBDir, Auto: true},
+	}
+
+	c := New()
+	c.SetProjects([]discovery.Project{
+		{Name: "proj-a", Path: projADir, Sources: sourcesA},
+		{Name: "proj-b", Path: projBDir, Sources: sourcesB},
+	})
+
+	// Simulate initial scan for proj-a
+	c.SetProjectFiles("proj-a", []FileInfo{
+		{Project: "proj-a", Source: "__all_markdown__", FullPath: "readme.md", Name: "readme.md", Title: "A", ModTime: time.Now()},
+	})
+
+	// proj-b is not scanned yet
+
+	// RescanWith the same projects (unchanged sources)
+	c.RescanWith([]discovery.Project{
+		{Name: "proj-a", Path: projADir, Sources: sourcesA},
+		{Name: "proj-b", Path: projBDir, Sources: sourcesB},
+	})
+
+	// proj-a should still have its cached files (not re-walked)
+	filesA := c.ProjectFiles("proj-a")
+	if len(filesA) != 1 {
+		t.Fatalf("expected proj-a to preserve 1 cached file, got %d", len(filesA))
+	}
+	if filesA[0].Title != "A" {
+		t.Errorf("expected preserved title 'A', got %q", filesA[0].Title)
+	}
+
+	// proj-b was never scanned, so RescanWith should scan it now
+	filesB := c.ProjectFiles("proj-b")
+	if len(filesB) != 1 {
+		t.Fatalf("expected proj-b to have 1 file after rescan, got %d", len(filesB))
+	}
+}
+
+// E-PENPAL-CACHE: verifies RescanWith cleans up removed projects.
+func TestRescanWith_RemovesOldProjects(t *testing.T) {
+	keepSources := []discovery.FileSource{
+		{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/keep", Auto: true},
+	}
+	removeSources := []discovery.FileSource{
+		{Name: "__all_markdown__", Type: "tree", SourceTypeName: "__all_markdown__", RootPath: "/tmp/remove", Auto: true},
+	}
+
+	c := New()
+	c.SetProjects([]discovery.Project{
+		{Name: "keep", Path: "/tmp/keep", Sources: keepSources},
+		{Name: "remove", Path: "/tmp/remove", Sources: removeSources},
+	})
+	c.SetProjectFiles("keep", []FileInfo{
+		{Project: "keep", FullPath: "readme.md"},
+	})
+	c.SetProjectFiles("remove", []FileInfo{
+		{Project: "remove", FullPath: "readme.md"},
+	})
+
+	// RescanWith only the "keep" project (same sources → preserved)
+	c.RescanWith([]discovery.Project{
+		{Name: "keep", Path: "/tmp/keep", Sources: keepSources},
+	})
+
+	// "remove" should be gone from cache
+	filesRemoved := c.ProjectFiles("remove")
+	if len(filesRemoved) != 0 {
+		t.Errorf("expected removed project to have 0 cached files, got %d", len(filesRemoved))
+	}
+
+	// "keep" should still have its files (unchanged sources → preserved)
+	filesKeep := c.ProjectFiles("keep")
+	if len(filesKeep) != 1 {
+		t.Errorf("expected kept project to preserve 1 cached file, got %d", len(filesKeep))
+	}
+}
diff --git a/apps/penpal/internal/server/manage.go b/apps/penpal/internal/server/manage.go
index 42dd7e09..f9e23e97 100644
--- a/apps/penpal/internal/server/manage.go
+++ b/apps/penpal/internal/server/manage.go
@@ -8,6 +8,7 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
+	"sync"
 	"time"
 
 	"github.com/loganj/penpal/internal/config"
@@ -26,6 +27,9 @@ func expandTilde(path string) string {
 }
 
 // refreshAfterConfigChange saves the config and re-discovers all projects.
+// RescanWith preserves cached data for unchanged projects and only scans
+// new or source-changed ones, so we skip the redundant populateProjects
+// call and just enrich git info in the background.
 func (s *Server) refreshAfterConfigChange() {
 	if err := config.Save(s.cfgPath, s.cfg); err != nil {
 		log.Printf("Warning: could not save config: %v", err)
@@ -34,7 +38,33 @@ func (s *Server) refreshAfterConfigChange() {
 	s.cache.RescanWith(projects)
 	s.watcher.Refresh(s.workspacePaths(), projects)
 	s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged})
-	go s.populateProjects()
+	go s.enrichGitInfo()
+}
+
+// enrichGitInfo updates git info for all projects that don't have it yet.
+func (s *Server) enrichGitInfo() {
+	projects := s.cache.Projects()
+	var wg sync.WaitGroup
+	sem := make(chan struct{}, 8)
+
+	for _, p := range projects {
+		if p.Name == "(root)" || p.Git != nil {
+			continue
+		}
+		wg.Add(1)
+		go func(p discovery.Project) {
+			defer wg.Done()
+			sem <- struct{}{}
+			defer func() { <-sem }()
+			git := discovery.GetGitInfo(p.Path)
+			s.cache.EnrichProject(p.QualifiedName(), git)
+		}(p)
+	}
+
+	wg.Wait()
+	if len(projects) > 0 {
+		s.watcher.Broadcast(watcher.Event{Type: watcher.EventProjectsChanged})
+	}
 }
 
 // handleAPIWorkspaces dispatches workspace management requests.
diff --git a/apps/penpal/internal/server/server.go b/apps/penpal/internal/server/server.go
index 7dbb8ca7..3deef897 100644
--- a/apps/penpal/internal/server/server.go
+++ b/apps/penpal/internal/server/server.go
@@ -1054,8 +1054,8 @@ func (s *Server) handleDeleteFile(w http.ResponseWriter, r *http.Request) {
 	}
 	s.cfgMu.Unlock()
 
-	// Refresh cache so the file disappears from listings
-	s.cache.RefreshProject(qualifiedName)
+	// E-PENPAL-CACHE: remove file from cache incrementally (no walk).
+	s.cache.RemoveFile(qualifiedName, filePath)
 	s.watcher.Broadcast(watcher.Event{Type: watcher.EventFilesChanged, Project: qualifiedName})
 	w.WriteHeader(http.StatusNoContent)
 }

From 798066e8739e8e46be4f0e83e3b15e7a201eba12 Mon Sep 17 00:00:00 2001
From: Logan Johnson <loganj@squareup.com>
Date: Mon, 30 Mar 2026 09:43:18 -0400
Subject: [PATCH 5/6] fix(penpal): fall back to full refresh for unscanned
 projects

When a file event fires for a project that hasn't been lazily scanned
yet, fall back to a full RefreshProject instead of silently dropping the
event. The previous guard skipped incremental updates entirely, which
meant file changes were invisible until the user navigated to the
project page to trigger the lazy scan.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 apps/penpal/internal/watcher/watcher.go | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/apps/penpal/internal/watcher/watcher.go b/apps/penpal/internal/watcher/watcher.go
index acc4da19..dbac3a5c 100644
--- a/apps/penpal/internal/watcher/watcher.go
+++ b/apps/penpal/internal/watcher/watcher.go
@@ -703,9 +703,12 @@ func (w *Watcher) flushFileEvents(projectName string) {
 		return
 	}
 
-	// If the project hasn't been lazily scanned yet, skip incremental
-	// updates — the full scan on first access will pick these up.
+	// If the project hasn't been lazily scanned yet, fall back to a full
+	// RefreshProject so the file list is populated from scratch.
 	if !w.cache.IsProjectScanned(projectName) {
+		w.cache.RefreshProject(projectName)
+		w.cache.RefreshProjectGitInfo(projectName)
+		w.Broadcast(Event{Type: EventFilesChanged, Project: projectName})
 		return
 	}
 

From bd0a2d593ec97332fca0e901c852bd6a47b6f206 Mon Sep 17 00:00:00 2001
From: Logan Johnson <loganj@squareup.com>
Date: Mon, 30 Mar 2026 09:55:56 -0400
Subject: [PATCH 6/6] fix(penpal): eliminate remaining git check-ignore
 subprocess spawns
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Remove gitIgnoreChecker from projectHasAnyMarkdown — this lightweight
startup check no longer spawns a subprocess per project. A false positive
from .md files in gitignored dirs is harmless since the full scan on first
access applies proper filtering.

Remove IsProjectScanned guard from flushFileEvents — incremental updates
now apply regardless of scan state, avoiding fallback to full RefreshProject
which spawned git check-ignore. The lazy scan on first access reconciles.

Fix flaky TestSSE_BroadcastDelivery by draining background events before
asserting the expected broadcast.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---
 apps/penpal/TESTING.md                   |  2 +-
 apps/penpal/internal/cache/cache.go      | 96 ++++++++++++++++++------
 apps/penpal/internal/cache/cache_test.go | 16 ++--
 apps/penpal/internal/server/sse_test.go  | 60 ++++++++-------
 apps/penpal/internal/watcher/watcher.go  | 27 ++++---
 apps/penpal/junk.md                      | 25 ++++++
 6 files changed, 157 insertions(+), 69 deletions(-)
 create mode 100644 apps/penpal/junk.md

diff --git a/apps/penpal/TESTING.md b/apps/penpal/TESTING.md
index 4a0e3746..59173f3b 100644
--- a/apps/penpal/TESTING.md
+++ b/apps/penpal/TESTING.md
@@ -64,7 +64,7 @@ see-also:
 | Source Types — anchors (P-PENPAL-SRC-ANCHORS, SRC-ANCHORS-GROUP, SRC-ANCHORS-NESTED) | discovery_test.go (TestClassifyAnchorsFile, TestGroupAnchorsPaths, TestGroupAnchorsPaths_MarkerOnlyModule, TestAnchorsFileOrder, TestAnchorsRequireSibling) | — | — | — |
 | Source Types — claude-plans (P-PENPAL-SRC-CLAUDE-PLANS) | — | — | — | — |
 | Source Types — manual (P-PENPAL-SRC-MANUAL) | — | — | grouping_test.go (TestBuildFileGroups_ManualSourceDirHeadings) | — |
-| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_SkipsGitignored, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans, TestResolveFileInfo, TestUpsertFile, TestRemoveFile, TestRescanWith_PreservesUnchangedProjects, TestSourcesChanged) | — | — | — |
+| Cache & File Scanning (E-PENPAL-CACHE, SCAN) | cache_test.go (TestCheckAllProjectsHasFiles, TestProjectHasAnyMarkdown_IgnoresGitignore, TestProjectHasAnyMarkdown_SkipsVCSDirs, TestAllFiles_DeduplicatesAllMarkdown, TestEnsureProjectScanned_NoDuplicateScans, TestResolveFileInfo, TestUpsertFile, TestRemoveFile, TestRescanWith_PreservesUnchangedProjects, TestSourcesChanged) | — | — | — |
 | Worktree Support (P-PENPAL-WORKTREE) | discovery/worktree_test.go, cache/worktree_test.go | Layout.test.tsx | worktree_test.go (API + MCP) | — |
 | Worktree Dropdown (P-PENPAL-PROJECT-WORKTREE-DROPDOWN) | — | Layout.test.tsx | — | — |
 | Git Integration (P-PENPAL-GIT-INFO) | — | — | — | — |
diff --git a/apps/penpal/internal/cache/cache.go b/apps/penpal/internal/cache/cache.go
index f07672c8..8c5c5726 100644
--- a/apps/penpal/internal/cache/cache.go
+++ b/apps/penpal/internal/cache/cache.go
@@ -446,13 +446,12 @@ func (c *Cache) CheckAllProjectsHasFiles() {
 }
 
 // projectHasAnyMarkdown walks the directory tree and returns true as soon as
-// it finds any .md file. Aligns skip behavior with scanProjectSources: skips
-// .git, node_modules, .hg, .svn, nested worktrees/submodules, and gitignored dirs.
-// E-PENPAL-SCAN: lightweight startup check consistent with full scan filtering.
+// it finds any .md file. Skips .git, node_modules, .hg, .svn, and nested
+// worktrees/submodules. Does NOT use git check-ignore — a false positive
+// from a .md file in a gitignored directory is harmless since the full scan
+// on first access applies proper filtering.
+// E-PENPAL-SCAN: lightweight startup check — no subprocess spawned.
 func projectHasAnyMarkdown(projectPath string) bool {
-	gitChecker := newGitIgnoreChecker(projectPath)
-	defer gitChecker.Close()
-
 	err := filepath.WalkDir(projectPath, func(path string, d fs.DirEntry, err error) error {
 		if err != nil {
 			return nil
@@ -469,10 +468,6 @@ func projectHasAnyMarkdown(projectPath string) bool {
 					return filepath.SkipDir
 				}
 			}
-			// E-PENPAL-SRC-GITIGNORE: skip gitignored directories.
-			if path != projectPath && gitChecker.IsIgnored(path) {
-				return filepath.SkipDir
-			}
 			return nil
 		}
 		if strings.HasSuffix(d.Name(), ".md") {
@@ -548,9 +543,11 @@ func (c *Cache) RescanWith(projects []discovery.Project) {
 	}
 	c.mu.RUnlock()
 
-	// Preserve git enrichment
+	// Preserve git enrichment only when the project path hasn't changed.
+	// If the path changed, Git metadata would be stale and enrichGitInfo
+	// needs to re-discover it (it skips projects where Git != nil).
 	for i := range projects {
-		if prev, ok := existing[projects[i].QualifiedName()]; ok {
+		if prev, ok := existing[projects[i].QualifiedName()]; ok && prev.Path == projects[i].Path {
 			projects[i].Git = prev.Git
 		}
 	}
@@ -579,6 +576,7 @@ func (c *Cache) RescanWith(projects []discovery.Project) {
 			c.mu.Lock()
 			c.projectFiles[qn] = existingFiles[qn]
 			c.projectScanned[qn] = true
+			c.updateProjectMetadataLocked(qn, existingFiles[qn])
 			c.mu.Unlock()
 		}
 	}
@@ -688,6 +686,11 @@ func extractTitle(path string) string {
 // that claims the file (typically one typed source + __all_markdown__). Returns nil
 // if no source claims the file.
 // E-PENPAL-SCAN: single-file source resolution for incremental cache updates.
+// ResolveFileInfo resolves source membership for a single absolute .md file path
+// without a filesystem walk. It applies the same exclusion rules as
+// scanProjectSources: nested git worktree/submodule detection, gitignore
+// ancestor-directory checks (P-PENPAL-SRC-GITIGNORE), SkipDirs filtering, and
+// RequireSibling validation.
 func ResolveFileInfo(project *discovery.Project, absPath string) []FileInfo {
 	if !strings.HasSuffix(absPath, ".md") {
 		return nil
@@ -697,6 +700,9 @@ func ResolveFileInfo(project *discovery.Project, absPath string) []FileInfo {
 	if err != nil {
 		return nil
 	}
+	if info.IsDir() {
+		return nil
+	}
 
 	relToProject, err := filepath.Rel(project.Path, absPath)
 	if err != nil || strings.HasPrefix(relToProject, "..") {
@@ -721,6 +727,17 @@ func ResolveFileInfo(project *discovery.Project, absPath string) []FileInfo {
 				continue
 			}
 
+			// Skip files under nested git worktrees/submodules.
+			if isUnderNestedGitRepo(absPath, rootPath) {
+				continue
+			}
+
+			// P-PENPAL-SRC-GITIGNORE: skip files whose ancestor directory
+			// is gitignored (source root itself is exempt).
+			if isAncestorDirGitIgnored(absPath, rootPath, project.Path) {
+				continue
+			}
+
 			relToSource, err := filepath.Rel(rootPath, absPath)
 			if err != nil {
 				continue
@@ -848,17 +865,45 @@ func hasSkippedDir(relToSource string, st *discovery.SourceType) bool {
 	return false
 }
 
+// isUnderNestedGitRepo walks parent directories from absPath up to (but not
+// including) rootPath, returning true if any intermediate directory contains a
+// .git file (not directory), indicating a nested git worktree or submodule.
+// This mirrors the nested-repo check in scanProjectSources without spawning
+// a subprocess.
+func isUnderNestedGitRepo(absPath, rootPath string) bool {
+	dir := filepath.Dir(absPath)
+	for dir != rootPath && strings.HasPrefix(dir, rootPath+"/") {
+		gitEntry := filepath.Join(dir, ".git")
+		if fi, err := os.Lstat(gitEntry); err == nil && !fi.IsDir() {
+			return true
+		}
+		dir = filepath.Dir(dir)
+	}
+	return false
+}
+
+// isAncestorDirGitIgnored walks parent directories from absPath up to (but not
+// including) rootPath, running a one-shot `git check-ignore -q` on each.
+// Returns true if any ancestor directory is gitignored.
+// P-PENPAL-SRC-GITIGNORE: the source root itself is exempt (always scanned).
+func isAncestorDirGitIgnored(absPath, rootPath, projectPath string) bool {
+	dir := filepath.Dir(absPath)
+	for dir != rootPath && strings.HasPrefix(dir, rootPath+"/") {
+		cmd := exec.Command("git", "-C", projectPath, "check-ignore", "-q", dir)
+		if cmd.Run() == nil {
+			return true // exit code 0 means ignored
+		}
+		dir = filepath.Dir(dir)
+	}
+	return false
+}
+
 // UpsertFile adds or updates file entries in the cache for the given absolute path.
 // For existing entries (matched by FullPath), re-stats for ModTime and re-extracts
 // Title. For new files, resolves source membership and inserts.
 // E-PENPAL-CACHE: incremental cache mutation without filesystem walk.
 func (c *Cache) UpsertFile(projectName string, project *discovery.Project, absPath string) bool {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-
-	files := c.projectFiles[projectName]
-
-	// Check if any entries already exist for this path
+	// Perform all filesystem and git I/O outside the lock.
 	relToProject, err := filepath.Rel(project.Path, absPath)
 	if err != nil {
 		return false
@@ -869,11 +914,21 @@ func (c *Cache) UpsertFile(projectName string, project *discovery.Project, absPa
 		return false
 	}
 
+	title := extractTitle(absPath)
+	resolved := ResolveFileInfo(project, absPath)
+
+	// Acquire lock only for the short critical section that mutates the cache.
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	files := c.projectFiles[projectName]
+
+	// Check if any entries already exist for this path
 	updated := false
 	for i := range files {
 		if files[i].FullPath == relToProject {
 			files[i].ModTime = info.ModTime()
-			files[i].Title = extractTitle(absPath)
+			files[i].Title = title
 			updated = true
 		}
 	}
@@ -887,8 +942,7 @@ func (c *Cache) UpsertFile(projectName string, project *discovery.Project, absPa
 		return true
 	}
 
-	// New file — resolve source membership
-	resolved := ResolveFileInfo(project, absPath)
+	// New file — use pre-resolved source membership
 	if len(resolved) == 0 {
 		return false
 	}
diff --git a/apps/penpal/internal/cache/cache_test.go b/apps/penpal/internal/cache/cache_test.go
index 75c501b8..b0feafc8 100644
--- a/apps/penpal/internal/cache/cache_test.go
+++ b/apps/penpal/internal/cache/cache_test.go
@@ -521,27 +521,23 @@ func TestCheckAllProjectsHasFiles(t *testing.T) {
 	}
 }
 
-// E-PENPAL-SCAN: verifies projectHasAnyMarkdown skips gitignored dirs.
-func TestProjectHasAnyMarkdown_SkipsGitignored(t *testing.T) {
+// E-PENPAL-SCAN: projectHasAnyMarkdown does NOT check gitignore — it's a
+// lightweight startup check where false positives are harmless. Verifying that
+// .md files in gitignored dirs still count as "has markdown".
+func TestProjectHasAnyMarkdown_IgnoresGitignore(t *testing.T) {
 	tmpDir := t.TempDir()
 
 	runGit(t, tmpDir, "init")
 	runGit(t, tmpDir, "config", "user.email", "test@test.com")
 	runGit(t, tmpDir, "config", "user.name", "test")
 
-	// All .md files are in a gitignored directory
+	// .md files only in a gitignored directory — still returns true
 	os.WriteFile(filepath.Join(tmpDir, ".gitignore"), []byte("build/\n"), 0644)
 	os.MkdirAll(filepath.Join(tmpDir, "build"), 0755)
 	os.WriteFile(filepath.Join(tmpDir, "build", "output.md"), []byte("# Gen"), 0644)
 
-	if projectHasAnyMarkdown(tmpDir) {
-		t.Error("expected false: only .md files are in gitignored dir")
-	}
-
-	// Add a non-gitignored .md file and re-check
-	os.WriteFile(filepath.Join(tmpDir, "README.md"), []byte("# README"), 0644)
 	if !projectHasAnyMarkdown(tmpDir) {
-		t.Error("expected true: README.md is not gitignored")
+		t.Error("expected true: .md exists even though gitignored (gitignore not checked)")
 	}
 }
 
diff --git a/apps/penpal/internal/server/sse_test.go b/apps/penpal/internal/server/sse_test.go
index be8d4390..0a2fd6b5 100644
--- a/apps/penpal/internal/server/sse_test.go
+++ b/apps/penpal/internal/server/sse_test.go
@@ -101,37 +101,41 @@ func TestSSE_BroadcastDelivery(t *testing.T) {
 	// Broadcast a change event
 	s.watcher.Broadcast(watcher.Event{Type: watcher.EventFilesChanged, Project: "test-proj"})
 
-	line1, err := reader.ReadString('\n')
-	if err != nil {
-		t.Fatalf("reading event line 1: %v", err)
-	}
-	if line1 != "event: change\n" {
-		t.Errorf("expected %q, got %q", "event: change\n", line1)
-	}
+	// Read SSE events until we find our specific broadcast. Background
+	// goroutines (populateProjects) may emit events that race with ours.
+	var found bool
+	for attempts := 0; attempts < 10; attempts++ {
+		line1, err := reader.ReadString('\n')
+		if err != nil {
+			t.Fatalf("reading event line: %v", err)
+		}
+		if line1 != "event: change\n" {
+			t.Fatalf("expected event line, got %q", line1)
+		}
 
-	line2, err := reader.ReadString('\n')
-	if err != nil {
-		t.Fatalf("reading event line 2: %v", err)
-	}
-	dataStr := strings.TrimPrefix(line2, "data: ")
-	dataStr = strings.TrimSuffix(dataStr, "\n")
-	var evt watcher.Event
-	if err := json.Unmarshal([]byte(dataStr), &evt); err != nil {
-		t.Fatalf("parsing event JSON %q: %v", dataStr, err)
-	}
-	if evt.Type != "files" {
-		t.Errorf("expected event type %q, got %q", "files", evt.Type)
-	}
-	if evt.Project != "test-proj" {
-		t.Errorf("expected project %q, got %q", "test-proj", evt.Project)
-	}
+		line2, err := reader.ReadString('\n')
+		if err != nil {
+			t.Fatalf("reading data line: %v", err)
+		}
+		dataStr := strings.TrimPrefix(line2, "data: ")
+		dataStr = strings.TrimSuffix(dataStr, "\n")
 
-	line3, err := reader.ReadString('\n')
-	if err != nil {
-		t.Fatalf("reading event line 3: %v", err)
+		// Read blank separator
+		if _, err := reader.ReadString('\n'); err != nil {
+			t.Fatalf("reading separator: %v", err)
+		}
+
+		var evt watcher.Event
+		if err := json.Unmarshal([]byte(dataStr), &evt); err != nil {
+			t.Fatalf("parsing event JSON %q: %v", dataStr, err)
+		}
+		if evt.Type == "files" && evt.Project == "test-proj" {
+			found = true
+			break
+		}
 	}
-	if line3 != "\n" {
-		t.Errorf("expected blank separator, got %q", line3)
+	if !found {
+		t.Error("did not receive expected files event for test-proj")
 	}
 }
 
diff --git a/apps/penpal/internal/watcher/watcher.go b/apps/penpal/internal/watcher/watcher.go
index dbac3a5c..1a85700f 100644
--- a/apps/penpal/internal/watcher/watcher.go
+++ b/apps/penpal/internal/watcher/watcher.go
@@ -589,7 +589,22 @@ func (w *Watcher) handleEvent(event fsnotify.Event) {
 
 	// E-PENPAL-WATCHER: only .md file events trigger cache updates. Non-.md
 	// Create events (e.g., scanner temp files, backup artifacts) are ignored.
+	// However, directory Create/Rename events may contain .md files that need
+	// to be discovered, so we walk those directories before returning.
 	if !strings.HasSuffix(path, ".md") {
+		if event.Op&(fsnotify.Create|fsnotify.Rename) != 0 {
+			if info, err := os.Stat(path); err == nil && info.IsDir() {
+				filepath.Walk(path, func(p string, fi os.FileInfo, err error) error {
+					if err != nil {
+						return nil
+					}
+					if !fi.IsDir() && strings.HasSuffix(p, ".md") {
+						w.debounceFileEvent(projectName, p, fsnotify.Create)
+					}
+					return nil
+				})
+			}
+		}
 		return
 	}
 
@@ -703,15 +718,9 @@ func (w *Watcher) flushFileEvents(projectName string) {
 		return
 	}
 
-	// If the project hasn't been lazily scanned yet, fall back to a full
-	// RefreshProject so the file list is populated from scratch.
-	if !w.cache.IsProjectScanned(projectName) {
-		w.cache.RefreshProject(projectName)
-		w.cache.RefreshProjectGitInfo(projectName)
-		w.Broadcast(Event{Type: EventFilesChanged, Project: projectName})
-		return
-	}
-
+	// Apply incremental updates regardless of scan state. For unscanned
+	// projects this adds files to a partial cache; the lazy scan on first
+	// access will reconcile with a full walk.
 	for absPath, op := range events {
 		if op&(fsnotify.Remove|fsnotify.Rename) != 0 {
 			relPath, err := filepath.Rel(project.Path, absPath)
diff --git a/apps/penpal/junk.md b/apps/penpal/junk.md
new file mode 100644
index 00000000..1e272e5c
--- /dev/null
+++ b/apps/penpal/junk.md
@@ -0,0 +1,25 @@
+# The History of Some Mechanical Keyboards
+
+Mechanical keyboards have experienced a remarkable resurgence in popularity over the past decade, but their history stretches back to the earliest days of personal computing. Understanding how we got here requires a look at the evolution of keyboard technology and the communities that kept mechanical switches alive during the membrane era.
+
+## The Early Days
+
+The first computer keyboards were directly descended from typewriters. IBM's Model F, introduced in 1981 alongside the original IBM PC, used buckling spring switches that produced a distinctive tactile feel and audible click. This was followed by the legendary Model M in 1985, which refined the buckling spring mechanism and became one of the most beloved keyboards ever produced. Many Model M keyboards from the 1980s are still in daily use today, a testament to their build quality.
+
+During this period, all keyboards were effectively mechanical. The concept of a "mechanical keyboard" as a distinct category didn't exist because there was no alternative. Every key had its own discrete switch mechanism, and keyboards were built like tanks. They were also expensive, often costing the equivalent of several hundred dollars in today's money.
+
+## The Membrane Revolution
+
+The 1990s brought a dramatic shift. As personal computers moved from business tools to consumer products, cost pressure drove manufacturers toward membrane keyboards. Instead of individual switches, membrane keyboards use two flexible plastic sheets with conductive traces. When a key is pressed, the sheets make contact and complete a circuit. This design is dramatically cheaper to manufacture but sacrifices the tactile feedback and durability that mechanical switches provide.
+
+By the early 2000s, membrane keyboards had almost completely taken over the consumer market. Mechanical keyboards survived primarily in niche applications like point-of-sale terminals and some industrial equipment. The few enthusiasts who preferred mechanical switches had to hunt for remaining stock of older keyboards or pay premium prices for the limited models still in production.
+
+## The Renaissance
+
+The mechanical keyboard renaissance began around 2008-2010, driven by gaming communities and programmer forums. Cherry, a German company that had been manufacturing MX switches since 1983, suddenly found renewed demand for their products. The Cherry MX Blue, with its tactile bump and audible click, became the gateway switch for many newcomers. The Cherry MX Red, a smooth linear switch, found favor with gamers who wanted rapid key presses without tactile resistance.
+
+What truly accelerated the movement was the emergence of custom keyboard communities online. Forums and subreddits dedicated to mechanical keyboards created spaces where enthusiasts could share their builds, discuss switch preferences, and organize group buys for custom keycap sets. This community-driven approach transformed mechanical keyboards from a niche preference into a full-blown hobby with its own culture, vocabulary, and aesthetic sensibility.
+
+## The Modern Landscape
+
+Today the mechanical keyboard market is thriving. Dozens of switch manufacturers compete alongside Cherry, offering hundreds of switch variants with different actuation forces, travel distances, and sound profiles. Hot-swappable PCBs allow users to change switches without soldering. Custom keycap sets are produced in every color combination imaginable. The hobby continues to grow, blending engineering, aesthetics, and community in ways that the original IBM engineers could never have anticipated.