feat: spec preview support + local spec reading#1043
feat: spec preview support + local spec reading#1043dslovinsky wants to merge 9 commits intomainfrom
Conversation
Read API specs from local filesystem instead of fetching from dev-docs.alchemy.com at index time. Upload spec content to Redis under branch-scoped keys so preview mode can serve edited specs. - Add fetchLocalApiSpec() for reading specs from fern/api-specs/ - Add specsDir option to ContentSource for local spec reads - Add spec uploader (preview-specs.ts) for Redis upload - Wire up spec generation + upload in preview and production indexers - Run targeted spec generation on slow-path reindex (rest/rpc only) Co-Authored-By: Claude <noreply@anthropic.com>
🔗 Preview Mode
|
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 358ee2a509
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
- Throw early if specsDir is set but metadata.json is missing
(catches forgotten `pnpm generate`)
- Replace `execSync("pnpm generate")` with Promise.all on both
generate:rest and generate:rpc to avoid race from shell backgrounding
Co-Authored-By: Claude <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1b7a45e5d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
- Rewrite fetchLocalApiSpec to walk specsDir directly instead of reading metadata.json, eliminating the generate:metadata dependency - Simplify fail-fast check to just verify specsDir exists - Remove all generate:metadata calls from preview.ts - Add pnpm generate step to index-main-content CI workflow Co-Authored-By: Claude <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c532d641bb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| uses: ./.github/actions/setup-pnpm | ||
|
|
||
| - name: Generate API specs | ||
| run: pnpm generate |
There was a problem hiding this comment.
This new step runs pnpm generate, but that script is defined as pnpm run generate:rest & pnpm run generate:rpc in package.json, which runs one generator in the background and returns the foreground command's status; a failure in the background branch does not fail the step. In that case the workflow still runs pnpm index:main, so Redis/Algolia can be updated with a partial spec set from only one generator. Please run the two generators in separate steps (or otherwise propagate both exit codes) so indexing cannot continue after a partial generation failure.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed: split into separate pnpm generate:rest and pnpm generate:rpc CI steps so either failing correctly fails the workflow.
| await Promise.all([ | ||
| execAsync("pnpm generate:rest"), | ||
| execAsync("pnpm generate:rpc"), | ||
| ]); |
There was a problem hiding this comment.
The preview startup now uses exec() (via promisify(exec)) to run both generators, which buffers child stdout/stderr with Node's default maxBuffer; if generator output is verbose (for example, many lint diagnostics), this throws ERR_CHILD_PROCESS_STDIO_MAXBUFFER and aborts preview boot before indexing completes. Switching to a streaming API (spawn/execFile with inherited stdio) or setting an explicit larger buffer avoids this failure mode.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed: replaced promisify(exec) with a spawn-based helper using inherited stdio — no buffering limits.
- Split pnpm generate into separate generate:rest and generate:rpc CI steps so either failing correctly fails the workflow - Replace promisify(exec) with spawn-based helper to avoid maxBuffer limits on verbose generator output Co-Authored-By: Claude <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03ba8f6200
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - name: Generate REST API specs | ||
| run: pnpm generate:rest |
There was a problem hiding this comment.
Now that this job explicitly generates specs before pnpm index:main, main indexing has become the path that publishes spec content to Redis, but the workflow still triggers only on fern/docs.yml pushes. I checked the workflows in .github/workflows and this is the only one that runs pnpm index:main, so merges that modify src/openapi/**, src/openrpc/**, or fern/remote-specs.json can skip this job and leave production main:*spec* keys stale.
Useful? React with 👍 / 👎.
Extract spec upload from the content indexer into a dedicated CLI (scripts/upload-specs.ts) that computes SHA-256 hashes and only uploads changed specs to Redis. Add sync-specs.yml workflow to trigger on spec source changes to main. Simplify gh-pages-deploy to chain off sync-specs instead of Publish Docs. Co-Authored-By: Claude <noreply@anthropic.com>
Merge sync-specs.yml into index-main-content.yml so a single workflow handles generate → upload-specs → index → revalidate for all content changes (docs.yml, spec sources, remote-spec-updated). Switch gh-pages-deploy to direct path triggers instead of workflow_run. Co-Authored-By: Claude <noreply@anthropic.com>
Output changed URLs to fern/api-specs/changed-specs.json instead of stdout so logging can use console.info normally. Only walk alchemy/ and chains/ subdirectories instead of filtering by filename. Co-Authored-By: Claude <noreply@anthropic.com>
Replaced by upload-specs.ts which handles change detection via Redis. Co-Authored-By: Claude <noreply@anthropic.com>
Remove fetchApiSpec, getSpecInfo, getMetadata, and fetchWithRetries — all dead code since specs are always read locally. Rename fetchLocalApiSpec to readApiSpec for clarity. Co-Authored-By: Claude <noreply@anthropic.com>
1 participant
Summary
fern/api-specs/) instead of fetching fromdev-docs.alchemy.comat index time{branch}:{specType}-spec:{specUrl}) so preview mode serves edited specsChanges
fetchLocalApiSpec()— reads specs from local filesystem viametadata.jsonspecsDiroption onContentSourcefor local spec readsuploadSpecs()uploader — writes spec JSON to Redis with TTL for preview branchespreview.ts— runspnpm generateon startup, targeted generation on reindexpreview-watchers.ts— passes changed file path to--reindex={{changed}}Companion PR: OMGWINNING/docs-site (ds/spec-preview)
Test plan
pnpm preview, verify spec edit appears on preview URLpnpm generate && pnpm index:main→ verify specs uploaded undermain:keyspnpm test:runpasses (209/209)