Commit 49a7238
Rewatch: replace wave scheduler with DAG + critical-path priority (#8374)
* Rewatch: replace wave scheduler with DAG + critical-path priority
Instead of compiling modules in topologically-sorted waves (which stall
on the slowest file per wave), schedule them from a priority queue on a
work-stealing dispatcher. Priority is the length of the module's longest
downstream dependency chain, so bottlenecks start first.
Workers run inside a `rayon::in_place_scope` bounded to
`rayon::current_num_threads()` so the heap ordering actually decides
dispatch. Completions flow back over `std::sync::mpsc`; the dispatcher
updates pending-dep counters, propagates the dirty flag, and pushes
newly-ready modules back onto the heap. `BuildState` mutations happen
after the scope exits so worker reads and main-thread writes stay
disjoint. Cycle detection now triggers when the heap drains without
completing the universe.
Signed-Off-By: Jaap Frolich <jaap@tella.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Rewatch: tighten scheduler invariants and sort results by module name
- Restore the `.unwrap()` on `package.namespace.to_suffix()` for mlmap
dispatch so a missing namespace suffix panics loudly instead of being
silently papered over.
- Drop the defensive `.unwrap_or_default()` on the post-completion
dependents lookup and the `if *count == 0 { continue }` guard on the
pending-deps decrement; both can only fire when the graph itself is
inconsistent, and hiding that would mask bugs. Align with the
unwrap-style assertions the old scheduler used.
- Sort `results_buffer` by module name before applying results so the
returned `compile_errors` / `compile_warnings` strings and the per-
package `.compiler.log` writes are deterministic across runs despite
non-deterministic completion order.
Signed-Off-By: Jaap Frolich <jaap@tella.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Rewatch: persist propagated dirtiness across failed builds
When a predecessor's cmi changes the dispatcher only marks the local
dirty_set; it used to not write compile_dirty = true back onto
build_state. If the first compile error aborted dispatch before those
dependents were scheduled, their compile_dirty stayed false in
build_state, and the next incremental build would skip recompiling them
even though an upstream .cmi had changed — producing stale output.
Persist dirty_set onto build_state.modules[].compile_dirty right before
applying compile results. The success path in the result loop still
overrides compile_dirty = false for modules that actually recompiled
cleanly in this run, so only the propagated-but-not-scheduled modules
retain the flag.
Signed-Off-By: Jaap Frolich <jaap@tella.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* CHANGELOG: DAG scheduler for rewatch
Signed-Off-By: Jaap Frolich <jaap@tella.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent bcc333c commit 49a7238
2 files changed
Lines changed: 437 additions & 304 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
| |||
0 commit comments