-
-
Notifications
You must be signed in to change notification settings - Fork 396
Add append method to VaryingDimension for explicit chunk extension #3870
Copy link
Copy link
Open
Description
Context:
From review discussion in #3802 (comment), @dcherian suggested adding an append(self, sizes: Iterable[int]) method to VaryingDimension to allow explicitly extending chunk edges.
Current behavior:
VaryingDimension.resize(new_extent) handles growth by appending a single chunk covering the gap:
# [10, 10, 10] resized from 30 to 45 produces [10, 10, 10, 15]This works but gives the caller no control over how the new region is partitioned.
Proposed behavior:
# Explicit chunk extension
dim = VaryingDimension([10, 10, 10], extent=30)
dim = dim.append([10, 5]) # -> edges=(10, 10, 10, 10, 5), extent=45
# Would raise on FixedDimension (regular grids don't have explicit edges)Open questions:
- Should this live on
VaryingDimension,ChunkGrid, orArray? - Should
FixedDimension.appendraise, or convert toVaryingDimension? - How does this interact with metadata serialization? Appending chunks to a regular grid would change the serialization format from
"regular"to"rectilinear". - Should
Array.resize()accept an optionalchunksparameter that delegates to this? (Also noted in the design doc's resize section)
Related:
- feat: unified chunk grid with rectilinear chunk/shard support #3802 (unified chunk grid PR)
shall we also add
append(self, sizes: Iterable[int])to allow a different mode of extension (would fail for the FixedDimension).
Originally posted by @dcherian in #3802 (comment)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels