Add streaming APIs to build cache provider interface and all cache plugins by Copilot · Pull Request #5746 · microsoft/rushstack

Copilot · 2026-04-05T04:11:54Z

…P plugin Add optional tryGetCacheEntryStreamByIdAsync and trySetCacheEntryStreamAsync methods to ICloudBuildCacheProvider. Implement streaming in HttpBuildCacheProvider and update OperationBuildCache to prefer streaming when available. Add fetchStreamAsync to WebClient and stream write support to FileSystemBuildCacheProvider. Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/050e10a7-3cad-4da4-93e5-9941453283b9 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/050e10a7-3cad-4da4-93e5-9941453283b9 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

…file Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/798065bf-040f-462e-8ea9-b9ca5659196d Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

iclanton · 2026-04-05T04:16:19Z

@copilot - add APIs to FileSystem in node-core-library for the fs APIs you're making use of in this PR. Ideally, none of our projects (except for FileSystem itself) should use fs. Everything should use FileSystem. The change to node-core-library will also need a changefile. Take a look at the conventions in FileSystem and follow that pattern.

…rary and update consumers to use FileSystem instead of fs directly Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/5ad20d4d-c9a4-4855-bb13-8dd9e2c1350b Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot · 2026-04-05T04:31:41Z

@copilot - add APIs to FileSystem in node-core-library for the fs APIs you're making use of in this PR. Ideally, none of our projects (except for FileSystem itself) should use fs. Everything...

Added createReadStream() and createWriteStream() static methods to FileSystem in node-core-library, along with FileSystemReadStream and FileSystemWriteStream type aliases (following the same pattern as FileSystemStats and FolderItem). Updated FileSystemBuildCacheProvider and OperationBuildCache to use the new FileSystem APIs instead of importing from node:fs directly. Change file for @rushstack/node-core-library included as a minor bump. Commit: 468caf7

iclanton · 2026-04-05T04:33:42Z

@copilot - are there async variants of those fs APIs? Should we expose and use those too?

Copilot · 2026-04-05T04:34:33Z

@copilot - are there async variants of those fs APIs? Should we expose and use those too?

No, fs.createReadStream() and fs.createWriteStream() don't have async variants — they're already inherently asynchronous. They return stream objects immediately and all the actual I/O happens asynchronously through the stream events/pipeline. The fs/promises module doesn't include them for this reason.

iclanton · 2026-04-05T05:15:42Z

@copilot - take a look at the other FileSystem APIs. Many have an optional options parameter with useful utilities. For example, the write operations have an option to create the parent folder if it doesn't exist. Consider adding useful options to these new APIs

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/199d4b3e-1f3f-44e1-9fc6-7b4a0e027c7e Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot · 2026-04-05T05:26:42Z

@copilot - take a look at the other FileSystem APIs. Many have an optional options parameter with useful utilities. For example, the write operations have an option to create the parent folder if it...

Added IFileSystemCreateWriteStreamOptions with ensureFolderExists to createWriteStream(), following the same pattern as IFileSystemWriteBinaryFileOptions and other write APIs. Updated FileSystemBuildCacheProvider to use the new option instead of manual ensureFolderAsync(). For createReadStream, there aren't useful options from the existing FileSystem conventions (read methods use encoding/convertLineEndings which don't apply to raw streams). Commit: 91161d8

iclanton · 2026-04-05T06:29:57Z

@copilot - can we add stream support to the other cache plugins?

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/6212baeb-266c-4823-94df-251c69a8f74c Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

…y streaming upload Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/6212baeb-266c-4823-94df-251c69a8f74c Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot · 2026-04-05T06:48:14Z

@copilot - can we add stream support to the other cache plugins?

Added streaming support to both remaining cache plugins:

Amazon S3 plugin (733ef5c):

Refactored AmazonS3Client._makeRequestAsync to extract signing into _buildSignedRequest
Added _makeStreamRequestAsync using WebClient.fetchStreamAsync() with UNSIGNED-PAYLOAD for AWS Sig V4 signing of streaming uploads
Added getObjectStreamAsync() (with retry support) and uploadObjectStreamAsync() (single attempt, since consumed streams can't be replayed)
Added tryGetCacheEntryStreamByIdAsync and trySetCacheEntryStreamAsync to AmazonS3BuildCacheProvider

Azure Storage plugin (733ef5c):

Added tryGetCacheEntryStreamByIdAsync using blobClient.download() with readableStreamBody
Added trySetCacheEntryStreamAsync using blockBlobClient.uploadStream()
Added _logBlobError helper for consistent error handling across buffer and stream methods

All three cache plugins (HTTP, S3, Azure) now implement both streaming methods. All existing tests pass (45 S3, 10 Azure).

iclanton · 2026-04-05T19:45:15Z

@copilot - Can you put usage of this functionality behind a Rush experiment?

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/63fb5abe-e500-4a9c-bd82-3ed613989ef4 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot · 2026-04-05T20:00:23Z

@copilot - Can you put usage of this functionality behind a Rush experiment?

Added a useStreamingBuildCache experiment flag in experiments.json. The streaming code paths in OperationBuildCache now only activate when both the experiment is enabled AND the cloud provider implements the streaming methods. Without the experiment enabled, all cache operations use the existing buffer-based approach. Commit: 2493ed8

...rushstack/node-core-library/copilot-stream-cache-entry-for-http-plugin_2026-04-05-04-27.json

common/changes/@microsoft/rush/copilot-stream-cache-entry-for-http-plugin_2026-04-05-03-56.json

Co-authored-by: Ian Clanton-Thuon <iclanton@users.noreply.github.com>

iclanton

@copilot - This change introduces a lot of duplicated code. I've called out some instances of it. Can you review with that in mind and clean that up?

libraries/rush-lib/src/logic/buildCache/FileSystemBuildCacheProvider.ts

iclanton · 2026-04-05T20:48:05Z

libraries/rush-lib/src/utilities/WebClient.ts

+ * Makes an HTTP request that resolves as soon as headers are received, providing the
+ * response body as a readable stream. This avoids buffering the entire response in memory.
+ */
+const makeStreamRequestAsync: StreamFetchFn = async (


@copilot - This function should be able to be DRY'd up with makeRequestAsync.

Extracted shared _makeRawRequestAsync function that handles URL parsing, protocol selection, redirect following, body sending, and error handling. makeRequestAsync and makeStreamRequestAsync now both delegate to this shared core, each providing only their response-handling callback. Commit: a5dfa47

rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3Client.ts

iclanton · 2026-04-05T20:49:54Z

rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3Client.ts

+   * does not use retry logic because the stream is consumed after the first attempt and cannot be
+   * replayed. The caller should handle failures accordingly.
+   */
+  public async uploadObjectStreamAsync(objectName: string, objectStream: Readable): Promise<void> {


@copilot - This should be able to be DRY'd up with uploadObjectStreamAsync.

uploadObjectAsync and uploadObjectStreamAsync now both use the shared _makeSignedRequestAsync method. The upload methods still differ in retry behavior (buffer retries, stream doesn't), so they remain as separate public methods. Commit: a5dfa47

rush-plugins/rush-amazon-s3-build-cache-plugin/src/AmazonS3Client.ts

…lient private member type mismatch Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/1b32730c-9ce0-4e75-a2e1-45f06a628960 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

libraries/rush-lib/src/utilities/WebClient.ts

+    case WebClientProxy.Fiddler:
+      // For debugging, disable cert validation
+      // eslint-disable-next-line
+      process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';


…dCacheProvider Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/c103fb17-b8b1-4d02-88ba-9db236e9f48f Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot · 2026-04-05T21:42:18Z

@copilot - This change introduces a lot of duplicated code. I've called out some instances of it. Can you review with that in mind and clean that up?

Addressed all duplication instances across the three files:

WebClient.ts: Extracted _makeRawRequestAsync shared by both makeRequestAsync and makeStreamRequestAsync
AmazonS3Client.ts: Merged request methods into _makeSignedRequestAsync with overloads; extracted _handleGetResponseAsync for shared GET response handling
HttpBuildCacheProvider.ts: Extracted _makeHttpCoreRequestAsync shared by _makeHttpRequestAsync and _makeHttpStreamRequestAsync

Net result: -4 lines (337 added, 341 removed). Commit: a5dfa47

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/c103fb17-b8b1-4d02-88ba-9db236e9f48f Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot AI and others added 4 commits April 5, 2026 03:35

fix lint warnings, add change files and API report update

2462253

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/050e10a7-3cad-4da4-93e5-9941453283b9 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

address code review feedback: improve type annotations and JSDoc

5c783a9

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/050e10a7-3cad-4da4-93e5-9941453283b9 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Replace manual change files with rush change --bulk generated change …

1da21c9

…file Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/798065bf-040f-462e-8ea9-b9ca5659196d Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot AI assigned Copilot and iclanton Apr 5, 2026

Copilot created this pull request from a session on behalf of iclanton April 5, 2026 04:11 View session

github-project-automation bot added this to Bug Triage Apr 5, 2026

github-project-automation bot moved this to Needs triage in Bug Triage Apr 5, 2026

Copilot started work on behalf of iclanton April 5, 2026 04:16 View session

Copilot AI requested a review from iclanton April 5, 2026 04:32

Copilot finished work on behalf of iclanton April 5, 2026 04:32

Copilot started work on behalf of iclanton April 5, 2026 04:33 View session

Copilot finished work on behalf of iclanton April 5, 2026 04:35

Copilot started work on behalf of iclanton April 5, 2026 05:15 View session

Add ensureFolderExists option to FileSystem.createWriteStream

91161d8

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/199d4b3e-1f3f-44e1-9fc6-7b4a0e027c7e Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot finished work on behalf of iclanton April 5, 2026 05:27

Copilot started work on behalf of iclanton April 5, 2026 06:30 View session

Copilot AI and others added 2 commits April 5, 2026 06:43

Add streaming support to Amazon S3 and Azure Storage build cache plugins

733ef5c

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/6212baeb-266c-4823-94df-251c69a8f74c Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Address code review: add documentation for type assertion and no-retr…

84a44b7

…y streaming upload Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/6212baeb-266c-4823-94df-251c69a8f74c Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot AI changed the title ~~Add streaming APIs to build cache provider interface and HTTP plugin~~ Add streaming APIs to build cache provider interface and all cache plugins Apr 5, 2026

Copilot finished work on behalf of iclanton April 5, 2026 06:49

Copilot started work on behalf of iclanton April 5, 2026 19:45 View session

Gate streaming build cache behind useStreamingBuildCache experiment flag

2493ed8

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/63fb5abe-e500-4a9c-bd82-3ed613989ef4 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot finished work on behalf of iclanton April 5, 2026 20:01

iclanton reviewed Apr 5, 2026

View reviewed changes

...rushstack/node-core-library/copilot-stream-cache-entry-for-http-plugin_2026-04-05-04-27.json Outdated Show resolved Hide resolved

iclanton reviewed Apr 5, 2026

View reviewed changes

common/changes/@microsoft/rush/copilot-stream-cache-entry-for-http-plugin_2026-04-05-03-56.json Outdated Show resolved Hide resolved

iclanton added 4 commits April 5, 2026 13:42

Update changelogs.

e654a36

Co-authored-by: Ian Clanton-Thuon <iclanton@users.noreply.github.com>

Expand FileSystem.createWriteStream.

7902281

Use destructuring on rushConfiguration in PhasedScriptAction.

f58fc21

Clean up some duplicated types.

dd2acb4

iclanton reviewed Apr 5, 2026

View reviewed changes

Copilot started work on behalf of iclanton April 5, 2026 21:12 View session

Fix CI: add missing useStreamingBuildCache to bridge plugin, fix WebC…

704485d

…lient private member type mismatch Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/1b32730c-9ce0-4e75-a2e1-45f06a628960 Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

github-advanced-security bot found potential problems Apr 5, 2026

View reviewed changes

Copilot finished work on behalf of iclanton April 5, 2026 21:26

Copilot AI requested a review from iclanton April 5, 2026 21:26

Copilot started work on behalf of iclanton April 5, 2026 21:26 View session

DRY up duplicated code across WebClient, AmazonS3Client, and HttpBuil…

a5dfa47

…dCacheProvider Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/c103fb17-b8b1-4d02-88ba-9db236e9f48f Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Address code review: use Buffer.isBuffer for clearer body length check

66fcffc

Agent-Logs-Url: https://github.com/microsoft/rushstack/sessions/c103fb17-b8b1-4d02-88ba-9db236e9f48f Co-authored-by: iclanton <5010588+iclanton@users.noreply.github.com>

Copilot finished work on behalf of iclanton April 5, 2026 21:47

Conversation

Copilot AI commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

iclanton commented Apr 5, 2026

Uh oh!

Copilot AI commented Apr 5, 2026

Uh oh!

Uh oh!

Uh oh!

iclanton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iclanton Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iclanton Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Check failure

Copilot AI commented Apr 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Apr 5, 2026 •

edited

Loading