Skip to content

fix: improve sigv4 stream upload performance#993

Open
ferhatelmas wants to merge 1 commit intomasterfrom
ferhat/sigv4-stream
Open

fix: improve sigv4 stream upload performance#993
ferhatelmas wants to merge 1 commit intomasterfrom
ferhat/sigv4-stream

Conversation

@ferhatelmas
Copy link
Copy Markdown
Member

What kind of change does this PR introduce?

Performance

What is the current behavior?

Sigv4 chunked parser uses Buffer.concat and rebuilds the unread buffer per chunk O(n^2)

What is the new behavior?

Use segmented queue and consume slice in place.
Add header/trailer offsets to scan delimiter from previous boundary instead of full payload.
Harden header parsing and drop writing raw line.
Create the digest only when needed (skip unsigned trailer, zero length terminal chunk).
Cache derived signing key.

@ferhatelmas ferhatelmas requested a review from a team as a code owner April 8, 2026 11:29
Copilot AI review requested due to automatic review settings April 8, 2026 11:29
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes AWS SigV4 streaming (chunked) upload handling by reducing buffer-copy overhead in the chunk parser and caching derived signing keys to avoid repeated HMAC derivations during per-chunk verification.

Changes:

  • Replaces per-write Buffer.concat behavior in the SigV4 chunk parser with a segmented buffer queue + in-place consumption and offset-based delimiter scanning.
  • Hardens chunk header parsing/error messages and adds coverage for fragmented parsing cases (header/trailer boundaries).
  • Caches derived SigV4 signing keys inside SignatureV4 to speed up repeated chunk signature validations within the same scope.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
src/storage/protocols/s3/signature-v4-stream.ts Refactors the streaming chunk parser to use a segmented queue, adds delimiter search offsets, and reduces hashing work.
src/storage/protocols/s3/signature-v4.ts Adds an in-instance derived signing key cache and switches call sites to use it.
src/test/signature-v4-stream.test.ts Updates expectations for new error strings and adds tests for fragmented parsing + signing-key cache reuse.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@coveralls
Copy link
Copy Markdown

coveralls commented Apr 8, 2026

Coverage Report for CI Build 24139634727

Coverage increased (+0.1%) to 80.916%

Details

  • Coverage increased (+0.1%) from the base build.
  • Patch coverage: 23 uncovered changes across 2 files (300 of 323 lines covered, 92.88%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
src/storage/protocols/s3/signature-v4-stream.ts 298 276 92.62%
src/storage/protocols/s3/signature-v4.ts 25 24 96.0%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 37494
Covered Lines: 30516
Line Coverage: 81.39%
Relevant Branches: 4222
Covered Branches: 3239
Branch Coverage: 76.72%
Branches in Coverage %: Yes
Coverage Strength: 315.6 hits per line

💛 - Coveralls

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: ferhat elmas <elmas.ferhat@gmail.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants