Skip to content

feat: BMI for bit filtering#10136

Draft
devanbenz wants to merge 3 commits into
apache:mainfrom
devanbenz:db/10098/bmi-null-bool-filter
Draft

feat: BMI for bit filtering#10136
devanbenz wants to merge 3 commits into
apache:mainfrom
devanbenz:db/10098/bmi-null-bool-filter

Conversation

@devanbenz

@devanbenz devanbenz commented Jun 12, 2026

Copy link
Copy Markdown

DRAFT

This commit adds the ability for bit filtering to be done using the _pext_u64 BMI.

Right now this PR is directly copying the bit_util.rs code from: https://github.com/apache/arrow-rs/pull/9848/changes#diff-f2fe5b601534125d0ba033762b62db93ae2f81c71a543b6694df33773ebfa308R939-R976 -- I've just moved the compress function in to arrow-buffer. This draft PR is mostly meant to allow for me to benchmark on my ARM machine and x86_64 machine.

I noticed while bench-marking using the BMI path (filter_bits_compress) leads to a regression when we have a very dense or a very sparse filter mask. Besides those two outliers we gain a large performance gain from BMI.

Case (kept) strategy no-BMI2 BMI2 Delta
1023/1024 (dense) lazy 1.88 us 2.16 us +15%
optimized 1.21 us 2.23 us +84%
sliced 1.84 us 2.41 us +31%
9/10 lazy ~103 us 2.98 us -97%
optimized ~85 us 3.02 us -96%
sliced ~104 us 2.92 us -97%
1/2 lazy ~39 us 1.94 us -95%
optimized ~39 us 1.97 us -95%
sliced ~40 us 2.16 us -95%
1/10 lazy 7.37 us 1.55 us -79%
optimized 5.26 us 1.53 us -71%
sliced 7.41 us 1.74 us -77%
1/1024 (sparse) lazy 0.89 us 0.75 us -16%
optimized 99 ns 103 ns +3.5%
sliced 0.89 us 0.74 us -16%

/// Replace with `value.compress(mask)` when `uint_gather_scatter_bits`
/// is stabilised: <https://github.com/rust-lang/rust/issues/149069>
#[inline]
pub fn compress(value: u64, mask: u64) -> u64 {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant