Skip to content

DAG task lifecycle hooks silently skipped when task failure triggers Omit cascade on downstream tasks #16057

Open
Bug
#16058
Open
DAG task lifecycle hooks silently skipped when task failure triggers Omit cascade on downstream tasks#16057
Bug
#16058
@fcolombo7

Description

@fcolombo7
fcolombo7
opened on Apr 30, 2026

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When a DAG task fails and has downstream dependent tasks, those downstream tasks are marked Omitted. Any lifecycle hook declared on the failing task is never scheduled.
This affects all DAG topologies where the failing task has at least one downstream dependent.

What happens when you execute the sample workflow attached below:

Image

What I expected to happen:

Image

The notify hook on step-a should fire when step-a fails, regardless of whether step-a has downstream dependents. Downstream tasks should be marked Omitted only after all hooks on the failing task have completed.

Version(s)

v4.0.2, v4.0.5, 0eb6267

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hook-bug-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: step-a
            template: fail-inner
            hooks:
              notify:
                template: echo
                expression: tasks["step-a"].status == "Failed"
          - name: step-b
            template: succeed-inner
            depends: step-a

    - name: fail-inner
      dag:
        tasks:
          - name: inner
            template: fail-pod

    - name: succeed-inner
      dag:
        tasks:
          - name: inner
            template: succeed-pod

    - name: fail-pod
      container:
        image: alpine:3.23
        command: [sh, -c]
        args: ["exit 1"]

    - name: succeed-pod
      container:
        image: alpine:3.23
        command: [sh, -c]
        args: ["echo done"]

    - name: echo
      container:
        image: alpine:3.23
        command: [echo]
        args: ["hook fired"]

Logs from the workflow controller

{"time":"2026-04-30T08:38:02.541709186Z","level":"INFO","msg":"Processing workflow","lastSeenVersion":"","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","phase":"","resourceVersion":"701913157"}
{"time":"2026-04-30T08:38:02.553802829Z","level":"INFO","msg":"resolved artifact repository","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","artifactRepositoryRef":{"default":true}}
{"time":"2026-04-30T08:38:02.555229614Z","level":"INFO","msg":"Task-result reconciliation","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","component":"workflow_worker","numObjs":0}
{"time":"2026-04-30T08:38:02.555291112Z","level":"INFO","msg":"updated phase","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","fromPhase":"","toPhase":"Running","component":"workflow_worker"}
{"time":"2026-04-30T08:38:02.555369943Z","level":"WARN","msg":"Node was nil, will be initialized as type Skipped","namespace":"hxdr-processing","component":"workflow_worker","workflow":"hook-bug-rxph8"}
{"time":"2026-04-30T08:38:02.556007549Z","level":"INFO","msg":"was unable to obtain node, letting display name to be nodeName","component":"workflow_worker","workflow":"hook-bug-rxph8","boundaryID":"","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.556038162Z","level":"INFO","msg":"node initialized","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","node":"hook-bug-rxph8","phase":"Running","message":""}
{"time":"2026-04-30T08:38:02.557104092Z","level":"INFO","msg":"Event occurred","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"},"fieldPath":"","kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","component":"workflow_worker","type":"Normal","reason":"WorkflowRunning","message":"Workflow Running"}
{"time":"2026-04-30T08:38:02.557373467Z","level":"WARN","msg":"was unable to obtain the node","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeID":"hook-bug-rxph8-231077340","taskName":"step-a"}
{"time":"2026-04-30T08:38:02.557419344Z","level":"WARN","msg":"was unable to obtain the node","taskName":"step-b","nodeID":"hook-bug-rxph8-281410197","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.557430755Z","level":"WARN","msg":"was unable to obtain the node","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeID":"hook-bug-rxph8-281410197","taskName":"step-b","component":"workflow_worker"}
{"time":"2026-04-30T08:38:02.557507729Z","level":"WARN","msg":"was unable to obtain the node","namespace":"hxdr-processing","component":"workflow_worker","workflow":"hook-bug-rxph8","nodeID":"hook-bug-rxph8-231077340","taskName":"step-a"}
{"time":"2026-04-30T08:38:02.558240925Z","level":"WARN","msg":"was unable to obtain the node","nodeID":"hook-bug-rxph8-231077340","taskName":"step-a","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.558258175Z","level":"INFO","msg":"All of node dependencies completed","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeName":"hook-bug-rxph8.step-a","dependencies":null}
{"time":"2026-04-30T08:38:02.558337572Z","level":"WARN","msg":"Node was nil, will be initialized as type Skipped","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.558654851Z","level":"INFO","msg":"node initialized","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","node":"hook-bug-rxph8-231077340","phase":"Running","message":""}
{"time":"2026-04-30T08:38:02.559189186Z","level":"WARN","msg":"was unable to obtain the node","nodeID":"hook-bug-rxph8-158441414","taskName":"inner","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.559204461Z","level":"WARN","msg":"was unable to obtain the node","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeID":"hook-bug-rxph8-158441414","taskName":"inner"}
{"time":"2026-04-30T08:38:02.559420877Z","level":"WARN","msg":"was unable to obtain the node","nodeID":"hook-bug-rxph8-158441414","taskName":"inner","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.559439528Z","level":"INFO","msg":"All of node dependencies completed","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeName":"hook-bug-rxph8.step-a.inner","dependencies":null}
{"time":"2026-04-30T08:38:02.559466871Z","level":"WARN","msg":"Node was nil, will be initialized as type Skipped","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","component":"workflow_worker"}
{"time":"2026-04-30T08:38:02.561275107Z","level":"INFO","msg":"node initialized","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","message":"","node":"hook-bug-rxph8-158441414","phase":"Pending"}
{"time":"2026-04-30T08:38:02.561642363Z","level":"INFO","msg":"getExecutorLogOpts","workflow":"hook-bug-rxph8","loglevel":"info","namespace":"hxdr-processing","nodeName":"hook-bug-rxph8.step-a.inner","nodeID":"hook-bug-rxph8-158441414","component":"workflow_worker"}
{"time":"2026-04-30T08:38:02.561909522Z","level":"INFO","msg":"getExecutorLogOpts","nodeID":"hook-bug-rxph8-158441414","loglevel":"info","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeName":"hook-bug-rxph8.step-a.inner"}
{"time":"2026-04-30T08:38:02.563554166Z","level":"INFO","msg":"getExecutorLogOpts","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeName":"hook-bug-rxph8.step-a.inner","nodeID":"hook-bug-rxph8-158441414","loglevel":"info"}
{"time":"2026-04-30T08:38:02.619104625Z","level":"INFO","msg":"add pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:02.620025865Z","level":"INFO","msg":"Created pod","nodeName":"hook-bug-rxph8.step-a.inner","nodeID":"hook-bug-rxph8-158441414","podName":"hook-bug-rxph8-fail-pod-158441414","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.620106533Z","level":"WARN","msg":"was unable to obtain the node","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeID":"hook-bug-rxph8-281410197","taskName":"step-b","component":"workflow_worker"}
{"time":"2026-04-30T08:38:02.620291132Z","level":"WARN","msg":"was unable to obtain the node","taskName":"step-b","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeID":"hook-bug-rxph8-281410197"}
{"time":"2026-04-30T08:38:02.620517468Z","level":"INFO","msg":"TaskSet Reconciliation","namespace":"hxdr-processing","component":"workflow_worker","workflow":"hook-bug-rxph8"}
{"time":"2026-04-30T08:38:02.621013565Z","level":"INFO","msg":"reconcileAgentPod","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:02.627086138Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:02.627939571Z","level":"INFO","msg":"Workflow to be dehydrated","namespace":"hxdr-processing","component":"workflow_worker","workflow":"hook-bug-rxph8","Workflow Size":1796}
{"time":"2026-04-30T08:38:02.644870561Z","level":"INFO","msg":"Workflow update successful","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","phase":"Running","lastSeenVersion":"701913157","resourceVersion":"701913165"}
{"time":"2026-04-30T08:38:02.644962837Z","level":"INFO","msg":"Event occurred","reason":"WorkflowNodeRunning","message":"Running node hook-bug-rxph8","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"},"fieldPath":"","kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","type":"Normal","component":"workflow_worker"}
{"time":"2026-04-30T08:38:02.64498548Z","level":"INFO","msg":"Event occurred","message":"Running node hook-bug-rxph8.step-a","component":"workflow_worker","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"},"fieldPath":"","kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","type":"Normal","reason":"WorkflowNodeRunning"}
{"time":"2026-04-30T08:38:02.692696293Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:03.638424389Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:04.680255958Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:06.664619838Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:07.670408178Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:08.678119865Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:09.777745233Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:12.620124913Z","level":"INFO","msg":"Processing workflow","resourceVersion":"701913165","lastSeenVersion":"701913157","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","phase":"Running"}
{"time":"2026-04-30T08:38:12.621090875Z","level":"INFO","msg":"Task-result reconciliation","numObjs":1,"namespace":"hxdr-processing","component":"workflow_worker","workflow":"hook-bug-rxph8"}
{"time":"2026-04-30T08:38:12.62133436Z","level":"INFO","msg":"Pod failed","displayName":"inner","templateName":"fail-pod","pod":"hook-bug-rxph8-fail-pod-158441414","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","message":"main: Error (exit code 1)"}
{"time":"2026-04-30T08:38:12.621360031Z","level":"INFO","msg":"Pod restart check: analyzing pod status","podName":"hook-bug-rxph8-fail-pod-158441414","workflow":"hook-bug-rxph8","containerStatuses":2,"podPhase":"Failed","podReason":"","podMessage":"","namespace":"hxdr-processing","component":"workflow_worker","initContainerStatuses":1}
{"time":"2026-04-30T08:38:12.621404615Z","level":"INFO","msg":"Pod restart check: pod does not qualify for restart","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","podName":"hook-bug-rxph8-fail-pod-158441414","podReason":""}
{"time":"2026-04-30T08:38:12.621745553Z","level":"WARN","msg":"was unable to obtain the node","taskName":"step-b","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","component":"workflow_worker","nodeID":"hook-bug-rxph8-281410197"}
{"time":"2026-04-30T08:38:12.621764419Z","level":"WARN","msg":"was unable to obtain the node","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","nodeID":"hook-bug-rxph8-281410197","taskName":"step-b"}
{"time":"2026-04-30T08:38:12.622193414Z","level":"INFO","msg":"Outbound nodes set","nodeID":"hook-bug-rxph8-231077340","outbound":["hook-bug-rxph8-158441414"],"component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:12.622216891Z","level":"INFO","msg":"node phase changed","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","node":"hook-bug-rxph8-231077340","fromPhase":"Running","toPhase":"Failed"}
{"time":"2026-04-30T08:38:12.622250044Z","level":"INFO","msg":"node finished","namespace":"hxdr-processing","node":"hook-bug-rxph8-231077340","finishedAt":"2026-04-30T08:38:12Z","component":"workflow_worker","workflow":"hook-bug-rxph8"}
{"time":"2026-04-30T08:38:12.622287129Z","level":"WARN","msg":"was unable to obtain the node","nodeID":"hook-bug-rxph8-281410197","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","taskName":"step-b"}
{"time":"2026-04-30T08:38:12.622451296Z","level":"INFO","msg":"node initialized","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","node":"hook-bug-rxph8-281410197","phase":"Omitted","message":" (message: omitted: depends condition not met)"}
{"time":"2026-04-30T08:38:12.622554063Z","level":"INFO","msg":"Outbound nodes set","nodeID":"hook-bug-rxph8","outbound":["hook-bug-rxph8-281410197"],"workflow":"hook-bug-rxph8","namespace":"hxdr-processing","component":"workflow_worker"}
{"time":"2026-04-30T08:38:12.62277553Z","level":"INFO","msg":"node phase changed","fromPhase":"Running","toPhase":"Failed","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","node":"hook-bug-rxph8"}
{"time":"2026-04-30T08:38:12.622797356Z","level":"INFO","msg":"node finished","node":"hook-bug-rxph8","finishedAt":"2026-04-30T08:38:12Z","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","component":"workflow_worker"}
{"time":"2026-04-30T08:38:12.622854104Z","level":"INFO","msg":"TaskSet Reconciliation","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:12.622862647Z","level":"INFO","msg":"reconcileAgentPod","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:12.622876202Z","level":"INFO","msg":"updated phase","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","fromPhase":"Running","toPhase":"Failed"}
{"time":"2026-04-30T08:38:12.622915108Z","level":"INFO","msg":"Marking workflow completed","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:12.622931875Z","level":"INFO","msg":"Marking workflow as pending archiving","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:12.623422585Z","level":"INFO","msg":"Event occurred","fieldPath":"","kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","type":"Warning","reason":"WorkflowFailed","message":"","component":"workflow_worker","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"}}
{"time":"2026-04-30T08:38:12.623717208Z","level":"INFO","msg":"Workflow to be dehydrated","component":"workflow_worker","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","Workflow Size":4004}
{"time":"2026-04-30T08:38:12.638297554Z","level":"INFO","msg":"Workflow update successful","workflow":"hook-bug-rxph8","namespace":"hxdr-processing","component":"workflow_worker","resourceVersion":"701913333","phase":"Failed","lastSeenVersion":"701913165"}
{"time":"2026-04-30T08:38:12.638476075Z","level":"INFO","msg":"Event occurred","fieldPath":"","kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","type":"Normal","reason":"WorkflowNodeRunning","message":"Running node hook-bug-rxph8.step-a.inner: main: Error (exit code 1)","component":"workflow_worker","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"}}
{"time":"2026-04-30T08:38:12.63893103Z","level":"INFO","msg":"Event occurred","component":"workflow_worker","apiVersion":"argoproj.io/v1alpha1","type":"Warning","reason":"WorkflowNodeFailed","message":"Failed node hook-bug-rxph8.step-a.inner: main: Error (exit code 1)","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"},"fieldPath":"","kind":"Workflow"}
{"time":"2026-04-30T08:38:12.638997684Z","level":"INFO","msg":"Event occurred","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"},"component":"workflow_worker","fieldPath":"","kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","type":"Warning","reason":"WorkflowNodeFailed","message":"Failed node hook-bug-rxph8.step-a"}
{"time":"2026-04-30T08:38:12.639036565Z","level":"INFO","msg":"Event occurred","fieldPath":"","component":"workflow_worker","kind":"Workflow","apiVersion":"argoproj.io/v1alpha1","type":"Warning","reason":"WorkflowNodeFailed","message":"Failed node hook-bug-rxph8","object":{"name":"hook-bug-rxph8","namespace":"hxdr-processing"}}
{"time":"2026-04-30T08:38:13.647792376Z","level":"INFO","msg":"queueing pod for cleanup after","podName":"hook-bug-rxph8-fail-pod-158441414","action":"deletePod","after":5000000000,"component":"pod_controller","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:13.648444786Z","level":"INFO","msg":"archiving workflow","component":"archive_worker","uid":"50cad03e-ed67-40f5-b4ca-6a173f3db159","namespace":"hxdr-processing","workflow":"hook-bug-rxph8"}
{"time":"2026-04-30T08:38:14.739053229Z","level":"INFO","msg":"Queueing workflow for delete due to TTL","phase":"Failed","workflow":"hxdr-processing/hook-bug-rxph8","addAfter":118000000000,"component":"gc_controller"}
{"time":"2026-04-30T08:38:18.648218723Z","level":"INFO","msg":"cleaning up pod","component":"pod_controller","podName":"hook-bug-rxph8-fail-pod-158441414","key":"hxdr-processing/hook-bug-rxph8-fail-pod-158441414/deletePod","action":"deletePod","namespace":"hxdr-processing"}
{"time":"2026-04-30T08:38:18.658154979Z","level":"INFO","msg":"update pod event","component":"pod_controller","pod":"hook-bug-rxph8-fail-pod-158441414"}
{"time":"2026-04-30T08:38:18.665262697Z","level":"INFO","msg":"delete pod event","pod":"hook-bug-rxph8-fail-pod-158441414","component":"pod_controller"}
{"time":"2026-04-30T08:40:13.002419201Z","level":"INFO","msg":"Deleting garbage collected workflow","workflow":"hxdr-processing/hook-bug-rxph8","component":"gc_controller"}
{"time":"2026-04-30T08:40:13.016836493Z","level":"INFO","msg":"Successful request to be deleted","component":"gc_controller","workflow":"hxdr-processing/hook-bug-rxph8"}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions