Ignore 406 Not Acceptable during docker manifest list

[itechdima]

No description provided.

[flx42]

Hello @itechdima, in which case is this a problem?

Thanks!

[itechdima]

Hello @flx42
We're finally trying to use your approach with enroot importer in pyxis
I've patched a bit caching importer example to adapt to our filesystem, but it appears, that our container registry responses with 406 Not Allowed if I have enroot import with tag+digest for the image (this one)

Original error:

root@login-0:~#  srun --container-image=cr.eu-north1.nebius.cloud#ml-containers/training_diag:12.9.0-ubuntu24.04-20260223082816 hostname
cpu-bind=MASK - worker-1, task  0  0 [7044]: mask 0x3 set
pyxis: importing docker image: cr.eu-north1.nebius.cloud#ml-containers/training_diag:12.9.0-ubuntu24.04-20260223082816
[2026-02-25T17:15:51.021] error: pyxis: importer get failed with exit code 1
[2026-02-25T17:15:51.021] error: pyxis: printing importer log file:
[2026-02-25T17:15:51.021] error: pyxis:     [INFO] Querying registry for permission grant
[2026-02-25T17:15:51.021] error: pyxis:     [INFO] Authenticating with user: <anonymous>
[2026-02-25T17:15:51.021] error: pyxis:     [INFO] Authentication succeeded
[2026-02-25T17:15:51.021] error: pyxis:     [INFO] Querying registry for permission grant
[2026-02-25T17:15:51.021] error: pyxis:     [INFO] Authenticating with user: <anonymous>
[2026-02-25T17:15:51.021] error: pyxis:     [INFO] Authentication succeeded
[2026-02-25T17:15:51.021] error: pyxis:     [INFO] Fetching image manifest list
[2026-02-25T17:15:51.021] error: pyxis:     [ERROR] URL https://cr.eu-north1.nebius.cloud/v2/ml-containers/training_diag/manifests/sha256:e12d6abdf91980bedaf4b3d14660394be1dc21ff749dcdb28eedc3dfaeea2232 returned error code: 406 Not Acceptable
[2026-02-25T17:15:51.021] error: pyxis: failed to import docker image with importer: /opt/slurm_scripts/pyxis_caching_importer.sh
[2026-02-25T17:15:51.193] error: pyxis: couldn't start container
[2026-02-25T17:15:51.193] error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
[2026-02-25T17:15:51.193] error: Failed to invoke spank plugin stack
[2026-02-25T17:15:52.130] error: pyxis: child 7148 failed with error code: 1
srun: error: worker-1: task 0: Exited with exit code 1
srun: Terminating StepId=82.0

I suppose, it's also could be an issue with container registry and not sure how hard it is to fix. Also, not quite sure here whether it's expected from any container registry to respond in such a way.
Manually changing docker.sh in the cluster to add 406 as one more ignored code for curl, it starts working.

Another fix, that could work - add additional patch the caching importer (in our specific implementation) to not use digest at all and rely only on image tag, but that's less robust than your exampe

But if I'm wrong here and a registry shouldn't respond with 406 - I'll bring it to them and feel free to close this PR