Skip to content

Commit c5e0e02

Browse files
authored
fix: prevent cron from permanently dying after handler errors (#16219)
When any error occurs during a cron job, the cron stops firing permanently and never recovers until the process is restarted. This happens because of a bug in croner v9 combined with missing error handling in the cron setup. Croner's `protect: true` option sets an internal `blocking` flag before running the handler and only clears it after the handler completes. In v9, if the handler throws, the flag is never cleared because there's no `try/finally` around it. Every future cron sees `blocking = true` and skips, so the cron is dead forever. Even without the croner bug, the error from the handler was propagating as an unhandled promise, which can crash the process. One example where this affected us was if there is a db error while running the job. Even if that error is temporary and would be resolved during the next run, the cron job would no longer run until the process is restarted. ## Two changes fix this: - Bumped croner from v9 to v10, which wraps the handler in `try/finally` so the `blocking` flag is always cleared regardless of whether the handler throws. This was a confirmed bug in v9 - Added the `catch` option to the Cron constructor, which tells croner to catch handler errors and pass them to a callback instead of letting them escape as unhandled rejections. The callback logs the error via `payload.logger.error` so it's visible in application logs. --- - To see the specific tasks where the Asana app for GitHub is being used, see below: - https://app.asana.com/0/0/1213983836266032
1 parent 54189e1 commit c5e0e02

6 files changed

Lines changed: 12248 additions & 4263 deletions

File tree

packages/payload/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@
107107
"busboy": "^1.6.0",
108108
"ci-info": "^4.1.0",
109109
"console-table-printer": "2.12.1",
110-
"croner": "9.1.0",
110+
"croner": "10.0.1",
111111
"dataloader": "2.2.3",
112112
"deepmerge": "4.3.1",
113113
"file-type": "21.3.4",

packages/payload/src/bin/index.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,8 @@ export const bin = async () => {
4343
{
4444
// Do not run consecutive crons if previous crons still ongoing
4545
protect: true,
46+
// TODO: Remove this compatibility option in 4.0. This only exists to ensure backwards-compatibility between Croner v9 and Croner v10 cron syntax
47+
sloppyRanges: true,
4648
},
4749
)
4850

packages/payload/src/index.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -754,8 +754,13 @@ export class BasePayload {
754754
})
755755
},
756756
{
757+
catch: (err) => {
758+
this.logger.error({ err, msg: 'Error in job queue cron job handler' })
759+
},
757760
// Do not run consecutive crons if previous crons still ongoing
758761
protect: true,
762+
// TODO: Remove this compatibility option in 4.0. This only exists to ensure backwards-compatibility between Croner v9 and Croner v10 cron syntax
763+
sloppyRanges: true,
759764
},
760765
)
761766

packages/payload/src/queues/operations/handleSchedules/index.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,10 @@ export function checkQueueableTimeConstraints({
141141
? queueScheduleStats?.tasks?.[taskConfig.slug]?.lastScheduledRun
142142
: queueScheduleStats?.workflows?.[workflowConfig?.slug ?? '']?.lastScheduledRun
143143

144-
const nextRun = new Cron(scheduleConfig.cron).nextRun(lastScheduledRun ?? undefined)
144+
const nextRun = new Cron(scheduleConfig.cron, {
145+
// TODO: Remove this compatibility option in 4.0. This only exists to ensure backwards-compatibility between Croner v9 and Croner v10 cron syntax
146+
sloppyRanges: true,
147+
}).nextRun(lastScheduledRun ?? undefined)
145148

146149
if (!nextRun) {
147150
return false

0 commit comments

Comments
 (0)