fix: avoid ListView take_reduce rebuild for dense selections

[dimitarvdimitrov]

The take_reduce path rebuilds the ListView, which is expensive.

For dense selections (density above the rebuild threshold) there's little garbage to reclaim, so skip the rebuild and let the regular take path handle it. The rebuild is still worthwhile for sparse takes where we'd otherwise drag around a lot of unused elements when exporting (e.g. to DuckDB).

This PR does a very crude estimation of the number of elements that stay in the ListViewArray.

Potentially we should do this estimation at export time in the ListViewExporter, but that's less trivial.

[codspeed-hq]

Merging this PR will not alter performance

✅ 1122 untouched benchmarks
⏩ 1455 skipped benchmarks¹

---

_{Comparing mitko/avoid-list-view-take-reduce-rebuild (df8bb7d) with develop (d8c156f)}

Footnotes

1. 1455 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[gatesn]

Am I missing something? Why would we ever rebuild? The whole point of ListView is to have cheap take/filter.

It should be explicit from the user if they want to garbage collect

[dimitarvdimitrov]

Am I missing something? Why would we ever rebuild? The whole point of ListView is to have cheap take/filter.

It should be explicit from the user if they want to garbage collect

The rebuild is still worthwhile for sparse takes where we'd otherwise drag around a lot of unused elements when exporting e.g. duckdb's ListViewExporter

vortex/vortex-duckdb/src/exporter/list_view.rs

Lines 80 to 85 in 11d607e

	let elements_exporter =
	new_array_exporter_with_flatten(elements.clone(), cache, ctx, true)?;
	if !elements.is_empty() {
	elements_exporter.export(0, elements.len(), &mut duckdb_elements, ctx)?;
	}

ideally we do this as heuristics in the exporter as opposed to in each operation like take. I was thinking of doing that next, but it might take a bit longer

[gatesn]

ideally we do this as heuristics in the exporter as opposed to in each operation like take

Yeah, 100%. Agree priority might not be to do it now, but eager GC impacts all Vortex users

[dimitarvdimitrov]

looking into why the benchmarks regressed...

[joseph-isaacs]

i think this should move? and also be pub crate?

[dimitarvdimitrov]

any suggestion for where to move it? it's now used in arrays/listview/compute and arrays/filter/execute