chore(ci): unify install + GitHub counts into one refresh-counts workflow#604
Merged
Conversation
…counts.yml
The two scheduled workflows that refresh installs-cache.json — registry
counts (npm/PyPI/crates) and GitHub-side counts (clones/releases/ghPackages)
— were running on staggered crons (06:00 and 06:30 UTC) but BOTH wrote
the same file and BOTH wrote the same two fields (`total` and
`fetchedAt`) from different inputs. Whenever the first workflow's PR
hadn't merged before the second ran (typical: PR merge cycle is 1-2h),
the second workflow opened a PR that conflicted on those shared fields.
Evidence in PR history:
2026-05-08:
PR 582 (registry) merged 10:49
PR 588 (GH-side, MANUAL backfill) merged 12:49
PR 590 (GH-side, automated) CLOSED 12:57 — superseded by PR 588
2026-05-09:
PR 601 (registry) opened 07:56, merged 10:09
PR 602 (GH-side) opened 08:14, merged 10:14 (took 2h)
PR 603 (GH-side, again) opened 10:23, merged 10:24 — same-day duplicate
Two days running, the second-of-two PR ended either closed-and-replaced
or as a duplicate re-run.
Fix: combine both into a single workflow (.github/workflows/refresh-counts.yml)
that runs once daily, shares one checkout, and produces one PR.
Beyond merging the YAML, this also fixes the dual-writer problem at
the data-design level. Step 1 (bash, registry counters) writes ONLY
the per-source HWMs (npm/pypi/crates) and intentionally does NOT touch
`total` or `fetchedAt`. Step 2 (Node, GH-side counters) reads the cache
that step 1 just wrote — seeing fresh registry HWMs alongside its own
fresh GH-side data — and writes the canonical `total` + `fetchedAt`
based on the installs.data.ts formula. Single canonical writer for the
displayed total.
Removed:
.github/workflows/update-installs.yml
.github/workflows/update-github-counts.yml
Added:
.github/workflows/refresh-counts.yml
The schema-regression guard from update-installs.yml is preserved in
step 1. The Node script (scripts/update-github-counts.mjs) is unchanged
— it already reads the cache before computing total, so it picks up
step 1's writes naturally.
YAML parsed cleanly via `yaml` package; no other repo references to
the two removed workflow filenames.
The original update-installs.yml ran without `set -e` on purpose. Single- package curl failures fall through (curl exits non-zero, jq emits nothing, the `$(())` capture treats empty as 0, NPM_TOTAL just doesn't grow that iteration) and the per-source HWM logic ensures we never regress the cached value. With `pipefail` enabled, one transient registry blip would abort the whole daily refresh. Step 3 (PR-open) keeps `set -euo pipefail` — that's a sequence of discrete commands where any failure should bubble up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Combines the two scheduled count-refresh workflows into one. The two were racing on
installs-cache.jsonand producing duplicate or mutually-conflicting PRs almost daily.The conflict
update-installs.ymlandupdate-github-counts.ymlran 30 min apart on overlapping fields:npm,pypi,cratesclones*,releases*,ghPackagestotalfetchedAtBoth computed
totalfrom different inputs (each saw its own fresh data + the cached version of the other half's fields), so the second-of-the-day PR always conflicted with the first ontotalwhenever the first hadn't merged yet — which was almost always, since PR merge cycles take 1–2 hours.Evidence in PR history
Two days running, the second PR ended either closed-and-replaced or as a duplicate re-run.
What changed
New:
.github/workflows/refresh-counts.yml(single workflow, single cron at 06:00 UTC, single PR per day).Removed:
update-installs.ymlandupdate-github-counts.yml.Beyond merging the YAML — also fixes the dual-writer problem at the data-design level:
npm,pypi,crates). Intentionally does NOT touchtotalorfetchedAt. The schema-regression guard fromupdate-installs.ymlis preserved.clones/releases/ghPackages— and writes the canonicaltotal+fetchedAtusing the same formula asinstalls.data.ts. Single canonical writer for the displayed total.The Node script (
scripts/update-github-counts.mjs) is unchanged — it already reads the cache before computing total, so it picks up step 1's writes naturally.Failure modes
The new design trades partial-update resilience for atomic-update consistency. Given that the GH API failure mode is rare and recoverable on the next day, while the conflict mode was happening daily and required manual intervention, this is the right trade.
Test plan
yamlpackage — 5 steps, single cron0 6 * * *, job namerefreshupdate-installs.ymlscript with only thetotal/fetchedAtwrites removed (verified line-by-line)node scripts/update-github-counts.mjsinvocation as the old workflowworkflow_dispatchsmoke test post-merge (recommended before relying on the cron)Tokens
Same as before:
ORG_TRAFFIC_TOKEN(PAT) forgh pr createand/traffic/clonesaccess; falls back toGITHUB_TOKENfor the GH API if absent.git pushfalls back to the persistedGITHUB_TOKEN(contents: writefrom the job's permissions block) — same pattern as both removed workflows.Out of scope
The
adoption-snapshot.ymlandinstalls-cache-schema.ymlworkflows touch related concerns but are not in scope here. They run on different cadences and don't conflict with the two consolidated workflows.