fix: improve sigv4 stream upload performance#993
Conversation
There was a problem hiding this comment.
Pull request overview
This PR optimizes AWS SigV4 streaming (chunked) upload handling by reducing buffer-copy overhead in the chunk parser and caching derived signing keys to avoid repeated HMAC derivations during per-chunk verification.
Changes:
- Replaces per-write
Buffer.concatbehavior in the SigV4 chunk parser with a segmented buffer queue + in-place consumption and offset-based delimiter scanning. - Hardens chunk header parsing/error messages and adds coverage for fragmented parsing cases (header/trailer boundaries).
- Caches derived SigV4 signing keys inside
SignatureV4to speed up repeated chunk signature validations within the same scope.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/storage/protocols/s3/signature-v4-stream.ts | Refactors the streaming chunk parser to use a segmented queue, adds delimiter search offsets, and reduces hashing work. |
| src/storage/protocols/s3/signature-v4.ts | Adds an in-instance derived signing key cache and switches call sites to use it. |
| src/test/signature-v4-stream.test.ts | Updates expectations for new error strings and adds tests for fragmented parsing + signing-key cache reuse. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Coverage Report for CI Build 24139634727Coverage increased (+0.1%) to 80.916%Details
Uncovered Changes
Coverage RegressionsNo coverage regressions found. Coverage Stats💛 - Coveralls |
62ff991 to
eb1bf6b
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
eb1bf6b to
7c27fca
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
7c27fca to
121b6f0
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
121b6f0 to
233c41f
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
What kind of change does this PR introduce?
Performance
What is the current behavior?
Sigv4 chunked parser uses
Buffer.concatand rebuilds the unread buffer per chunk O(n^2)What is the new behavior?
Use segmented queue and consume slice in place.
Add header/trailer offsets to scan delimiter from previous boundary instead of full payload.
Harden header parsing and drop writing raw line.
Create the digest only when needed (skip unsigned trailer, zero length terminal chunk).
Cache derived signing key.