chore(ci): shard release build into per-target matrix#5206
Closed
chore(ci): shard release build into per-target matrix#5206
Conversation
Replace the serial build job in release-shared.yml with an 8-way matrix
on large-linux-x86, one shard per published platform package. Each shard
uploads only its own slice; smoke-test, publish, and brew/scoop consumers
pattern-download every shard via `merge-multiple: true` and regenerate
dist/checksums.txt locally. No merge job, no re-upload of bulk bytes.
Refactor apps/cli/scripts/build.ts to accept --target <pkg-name>: when
set, only that shard's bun binary, Go binary (legacy), archive, and
Linux package(s) are built (deb/rpm for glibc, apk for musl). Each musl
shard now cross-compiles its own Go binary directly because
CGO_ENABLED=0 makes the output identical to glibc, removing the
cross-shard copyFile. Without --target, end-to-end behaviour is preserved
for local dev.
Add apps/cli/scripts/checksums.ts that hashes every
supabase_${version}_*.{tar.gz,zip,deb,rpm,apk} found in dist/ into
dist/checksums.txt. Called by build.ts in all-targets mode and by the
three CI jobs that need the file (publish, publish-homebrew,
publish-scoop) after their pattern-download.
Add a `build_only` input on release-shared.yml that gates smoke-test
(publish/brew/scoop auto-skip via their needs: chain). Wire up a new
build-pr.yml that runs the build matrix on PRs touching build-relevant
files, across both legacy and next shells, so build breakages surface
before merge without waiting for the full smoke-test suite.
Drop .github/workflows/build-pr.yml and instead expand smoke-test-pr.yml's
path filter to cover everything that affects the build phase or runtime
behaviour: apps/cli/scripts/{build,checksums,sync-versions}.ts,
apps/cli/src/**, apps/cli/package.json, apps/cli-go/**, packages/cli-*/**,
root workspace files (package.json, pnpm-lock.yaml, pnpm-workspace.yaml),
and .github/actions/setup/**. Previously the filter only watched
apps/cli/** and release-shared.yml, so changes to those other paths could
land on develop without any PR-time build + smoke validation.
Keep the build_only input on release-shared.yml as a generic capability
for any future caller (or ad-hoc workflow_dispatch) that wants to skip
the smoke-test matrix and only exercise the build shards.
Introduce a new step in the release-shared.yml workflow to generate checksums for the built artifacts using a dedicated script. This ensures that the integrity of the published packages can be verified post-build.
jgoux
approved these changes
May 7, 2026
Add retry mechanism for transient Docker failures during smoke tests. Introduce a set of transient exit codes and refactor the Docker test execution to handle retries, improving reliability on shared CI runners. The changes ensure that flaky image pulls do not cause test failures, providing clearer output on success or failure.
Change the runner for the build job in release-shared.yml from large-linux-x86 to blacksmith-8vcpu-ubuntu-2404, optimizing the CI environment for better performance and resource allocation.
Update the runner for the build job in release-shared.yml from blacksmith-8vcpu-ubuntu-2404 to larger-runner-4cpu, aiming to optimize resource allocation and improve CI performance.
Change the runner for the build job in release-shared.yml from larger-runner-4cpu to ubuntu-latest, ensuring compatibility with the latest CI environment and potentially improving build performance.
Refactor the build job in release-shared.yml to utilize a matrix strategy for runners, allowing parallel execution across multiple target platforms. Each target is now assigned to a specific runner architecture, improving build efficiency and reducing queue times.
…ct handling Update the release-shared.yml workflow to consolidate the build job into a single high-vCPU runner, improving build efficiency by reducing setup overhead. Modify artifact upload and download steps to handle all targets collectively, streamlining the process. Adjust descriptions for clarity and ensure consistency across the workflow.
Change the runner for the build job in release-shared.yml from blacksmith-32vcpu-ubuntu-2404 to blacksmith-32vcpu-ubuntu-2404-arm, ensuring compatibility with ARM architecture and potentially enhancing build performance for target platforms.
…ld jobs Update the release-shared.yml workflow to introduce separate build jobs for x86 and ARM architectures, utilizing a matrix strategy for parallel execution. Modify artifact handling to ensure correct uploads and downloads per target, and enhance clarity in job descriptions. Add checksum generation steps to verify artifact integrity post-build.
…ase workflow Update the release-shared.yml workflow to allow building for multiple targets in both x86 and ARM architectures. Modify the build script to accept multiple target arguments, enhancing flexibility and clarity. Adjust artifact handling to ensure correct uploads for each target, improving the overall build process.
… release workflow Refactor the release-shared.yml workflow to merge x86 and ARM build jobs into a single job that builds for all targets. Update artifact upload and download steps to handle all targets collectively, improving clarity and efficiency in the build process. Adjust verification steps to include all target architectures.
Member
Author
|
Dropping the PR as none of it really improve the overall CI time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Splits the single serial
buildjob inrelease-shared.ymlinto an 8-way matrix onlarge-linux-x86(one shard per published platform package), and broadens the path filter onsmoke-test-pr.ymlso PRs targetingdeveloprun the full build + smoke validation whenever they touch anything that affects the build phase or runtime behaviour.Why
buildjob runs all 8bun --compiletargets, the 6 Go cross-compiles (legacy), and the 6 nfpm packages serially on a single runner. Wall-clock time is bounded by the sum of every target.smoke-test-pr.ymlwas scoped toapps/cli/**+release-shared.yml. A regression inapps/cli-go/**,packages/cli-*/**, root workspace files, or the shared setup action would not trigger a PR build, so build breakages could land ondevelopundetected.What changes
Sharded build matrix in
release-shared.ymllarge-linux-x86, one per platform package:cli-darwin-arm64,cli-darwin-x64,cli-linux-{arm64,x64},cli-linux-{arm64,x64}-musl,cli-windows-{arm64,x64}.cli-build-shard-${shell}-${version}-${target}.smoke-test/publish/publish-homebrew/publish-scoopswitch fromname:topattern: cli-build-shard-${shell}-${version}-* + merge-multiple: true. Artifact bytes hit storage exactly once.Setup GoandPre-download Go modulesare gated oninputs.shell == 'legacy';Install nfpmis gated onstartsWith(matrix.target, 'cli-linux-'), so darwin/windows shards skip irrelevant setup.build_onlyinput (defaultfalse, so existing callers are unaffected). Whentrue,smoke-testis skipped viaif:andpublish/publish-homebrew/publish-scoopauto-skip via theirneeds:chain. Useful for ad-hocworkflow_dispatchruns that just want to validate the build matrix.apps/cli/scripts/build.tsrefactor--target <pkg-name>flag builds only one shard's outputs (bun binary + Go binary for legacy + archive for standard targets + deb/rpm for glibc Linux + apk for musl).--target, the script's end-to-end behaviour is preserved for local dev.CGO_ENABLED=0produces output identical to the glibc build), so the cross-shardcopyFilefrom glibc is gone.apps/cli/scripts/checksums.tsNew self-contained script that walks
dist/supabase_${version}_*.{tar.gz,zip,deb,rpm,apk}and writesdist/checksums.txt. Called bybuild.tsin all-targets mode and by the three consumer CI jobs (publish,publish-homebrew,publish-scoop) right after their pattern-download.Broader path filter on
smoke-test-pr.ymlReplaces the old
apps/cli/**+ workflow paths with an explicit list of every path that can affect the build phase or how the built artifacts behave at runtime:apps/cli/scripts/{build,checksums,sync-versions}.tsapps/cli/src/**,apps/cli/package.jsonapps/cli-go/**packages/cli-*/**package.json,pnpm-lock.yaml,pnpm-workspace.yaml.github/actions/setup/**.github/workflows/{release-shared,smoke-test-pr}.ymlNow any PR to
developthat could break the release pipeline runs the full build + smoke matrix before merge.Compatibility
release-shared.yml's public input contract stays backward-compatible: the newbuild_onlyinput hasdefault: false.pnpm exec bun apps/cli/scripts/build.ts --version <v>still produces every binary, archive, package, andchecksums.txtindist/.Test plan
smoke-test-pr.ymlruns against the new sharded build matrix and all 8 shards are green.develop, confirm the matrix-fanoutbuild, the pattern-download insmoke-test/publish/brew/scoop, and the inlineGenerate checksumsstep all behave as expected.dist/artifact set on GitHub Releases matches the current release's file list (12 archives + packages +checksums.txt).