GH-48277: [C++][Parquet] unpack with shuffle algorithm

[AntoinePrv]

Rationale for this change

The current bit-unpacking algorithm (which is implemented as a C++ code generator script in Python) does not fully leverage SIMD operations: all loads and some bitshifts use scalar operations, leaving performance on the table.

What changes are included in this PR?

1. Devise new bit-unpacking algorithms that fully leverage SIMD operations, for various parameter values of (packed bit width, destination integer width, SIMD register size). Different algorithms are necessary for different parameter values, because of straddling issues with some bit offsets.

2. Implement these new algorithms entirely in C++ using metaprogramming: the tables necessary for efficient SIMD swizzling and shifting are computed at compile-time using constexpr code (the exception is AVX-512 which is still using the legacy Python code generation script).

3. Implement low-level generic fallbacks for SIMD operations that are not available in all SIMD instruction sets, such as some flavor of bit-shifting. These fallbacks are also being contributed to xsimd, but have not been merged there yet.

Benchmark results on a AVX2 CPU (AMD Zen 2) on Linux (Ubuntu 24.04):

• Parquet decoding:

  • between 30% and 60% faster on DELTA_BINARY_PACKED integer decoding
  • up to 25% faster on RLE boolean decoding

• Parquet reading:

  • around 60% faster on BIT_PACKED levels reading
  • up to 70% faster on RLE levels reading
  • up to 20% faster on reading or skipping OPTIONAL and REPEATED records

Are these changes tested?

Yes, by the current extensive bit-unpacking and Parquet decoding tests.

Are there any user-facing changes?

No

• GitHub Issue: [C++][Parquet] Better simd unpack algorithm #48277

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #48277 has been automatically assigned in GitHub to PR creator.

[AntoinePrv]

@pitrou apart from R-lint, this is looking pretty good.

[pitrou]

@ursabot please benchmark lang=C++

[voltrondatabot]

Benchmark runs are scheduled for commit a4bfe8a. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

[conbench-apache-arrow]

Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit a4bfe8a.

There were 37 benchmark results indicating a performance regression:

• Pull Request Run on amd64-c6a-4xlarge-linux at 2025-11-27 19:32:37Z
  • BM_UnpackUint64 (C++) with params=DynamicAligned/47/64, source=cpp-micro, suite=arrow-bpacking-benchmark
  • IsInInt64SmallSet (C++) with params=64, source=cpp-micro, suite=arrow-compute-scalar-set-lookup-benchmark
• and 35 more (see the report linked below)

The full Conbench report has more details.

[AntoinePrv]

@pitrou I'm running this locally, and I made an error when fixing ASAN over-reading problem.
These latest benchmarks are not doing well.

[pitrou]

@ursabot please benchmark lang=C++

[voltrondatabot]

Benchmark runs are scheduled for commit dd3ec0d. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

[conbench-apache-arrow]

Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit dd3ec0d.

There were 19 benchmark results indicating a performance regression:

• Pull Request Run on amd64-c6a-4xlarge-linux at 2025-11-28 15:12:47Z
  • BM_UnpackUint32 (C++) with params=DynamicUnaligned/20/64, source=cpp-micro, suite=arrow-bpacking-benchmark
  • BM_DeltaLengthDecodingByteArray (C++) with params=max-string-length:8/batch-size:2048, source=cpp-micro, suite=parquet-encoding-benchmark
• and 17 more (see the report linked below)

The full Conbench report has more details.

[pitrou]

@ursabot please benchmark lang=C++

[voltrondatabot]

Benchmark runs are scheduled for commit 408ef04. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

[pitrou]

@ursabot please benchmark lang=C++

[conbench-apache-arrow]

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 408ef04.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[conbench-apache-arrow]

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit 408ef04.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[pitrou]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: b638570

Submitted crossbow builds: ursacomputing/crossbow @ actions-158ce97c66

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

[pitrou]

Unfortunately the "AMD64 Windows R release" failure looks related to this PR: it's deterministic (I've restarted it twice), it doesn't occur on git main, and the test where it fails/crashes uses Parquet.

arrow/r/tests/testthat/test-dplyr-summarize.R

Lines 260 to 271 in ebaaf07

	test_that("n_distinct() with many batches", {
	skip_if_not_available("parquet")
	tf <- tempfile()
	write_parquet(dplyr::starwars, tf, chunk_size = 20)
	ds <- open_dataset(tf)
	expect_equal(
	ds |> summarise(n_distinct(sex, na.rm = FALSE)) |> collect(),
	ds |> collect() |> summarise(n_distinct(sex, na.rm = FALSE))
	)
	})

[pitrou]

@jonkeane How would you go about diagnosing a R test that crashes/stops with no error message on Windows (see above)?

[pitrou]

@ursabot please benchmark lang=C++

[rok]

Benchmark runs are scheduled for commit b638570. Watch https://buildkite.com/apache-arrow and https://conbench.arrow-dev.org for updates. A comment will be posted here when the runs are complete.