[Download] Harden resume validation by anujbharambe · Pull Request #4143 · huggingface/huggingface_hub

anujbharambe · 2026-04-24T19:59:46Z

Summary

Persist resume metadata alongside .incomplete files to detect mismatches and restart safely.
Validate resume responses with If-Range and Content-Range before appending data.
Add regression coverage for resume metadata mismatch and updated http_get behavior.

Details

This change hardens resume behavior for large downloads by:

Writing a small resume state sidecar that stores ETag, expected size, and a normalized URL.
Rejecting resumes when metadata mismatches or the server returns an unexpected Content-Range.
Falling back to a clean restart when resume safety checks fail.

This addresses unreliable resume cases and prevents silent corruption due to appending the wrong byte ranges.

Reviewers

@kripper

Closes #4060

Note

Medium Risk
Changes core download/resume behavior by introducing new validation paths (ETag/Content-Range checks) and restart fallbacks, which could affect large-file downloads and retry semantics if edge cases are mishandled.

Overview
Hardens resumable downloads to prevent silent corruption. http_get now supports an optional etag, sends it via If-Range on resume, handles 416 Range Not Satisfiable by restarting cleanly, and validates 206 responses by checking Content-Range matches the requested resume position (otherwise restart from scratch).

Persists and validates resume state for .incomplete files. _download_to_tmp_and_move now writes a sidecar .metadata file (etag/expected size/normalized URL) and deletes/restarts when metadata or size is inconsistent, cleaning up the metadata file on completion or when force_download=True. Tests add coverage for metadata mismatch resets and update http_get retry mocks to include status_code/Content-Range behavior.

^{Reviewed by Cursor Bugbot for commit 299fda4. Bugbot is set up for automated code reviews on this repo. Configure here.}

This commit hardens the download resumption logic to prevent data corruption and improve reliability: - Implements symmetric validation for ETag and expected_size in resume metadata. - Fixes unreachable 416 (Range Not Satisfiable) handler by checking status before raising. - Ensures etag/Range parameters are correctly forwarded during connection-error retries. - Fixes Content-Range validation to account for initial Range offsets and enforces strict validation. - Always persists resume metadata sidecars from the first download attempt. - Ensures metadata sidecars are cleared when force_download=True, even if incomplete file is missing. - Removes redundant ETag formatting logic as normalization is handled upstream. - Removes error-prone sentinel values ('none') in favor of unambiguous empty strings. - Adds comprehensive tests for Range-based resumes and metadata integrity.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 299fda4. Configure here.}

anujbharambe · 2026-04-25T08:11:56Z

Hey @kripper @Wauplin - this PR is ready for review.

All Cursor Bugbot comments have been addressed and resolved. The remaining open Bugbot findings are low-severity style observations (dead code branch, duplicate regex), not correctness issues.

Summary of what this does:
--> Persists a '.metadata' sidecar alongside '.incomplete' files to detect stale resumes (changed etag/size/url) and restart safely
--> Adds 'If-Range' header and 'Content-Range' validation in 'http-get'
--> Handles 416 (Range Not Satisfiable) gracefully instead of crashing
--> Adds test coverage for metadata mismatch and http_get retry behavior

Closes #4060