Repository navigation
Conversation
…d runs at full speed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
CtrProcess(src/lib.rs) keeps the counter in a localu128while it fills a batch and writes it back toivonce at the end, instead of storing it toivafter every block and loading it again for the next one. Output,ivandstateare unchanged byte for byte.Why
Cargo.tomlasks foraes = "0.9.2"with no upper bound, so anyone on Rust 1.89+ gets aes 0.9.4. aes 0.9.4 turns its VAES backends on by default and picks them at runtime. In 0.9.2 they already existed, but only behind--cfg aes_backend="avx256"or"avx512", so master was also slow for anyone building with that flag. On x86 CPUs with AVX-512 and VAES, the backend encrypts 64 blocks per batch (0.9.2's default AES-NI backend does 8). With it,ctr256andCtr256ran about 2.6x slower: 5.9 → 2.3 GB/s at 512 KiB on an Intel Xeon Platinum 8370C, and the same on AMD Zen 4.Cause: in the batch loop (
src/lib.rslines 247–250 on master,*k = block_from_slice(ctr); ctr_next(ctr);),ctr_nextstores the counter as two 8-byte halves andblock_from_slicereads it back as one 16-byte load. That load can't be forwarded from two smaller stores, so every block waited for the previous block's store to reach cache, one after another. With 8-block batches LLVM fully unrolls the loop and keeps the counter in registers, so this never showed up on 0.9.2's default backend. With 64 blocks it keeps the loop, and the counter goes through memory (checked in the disassembly of the 0.9.4 build). After this change the AVX-512 backend is 40–56% faster than on 0.9.2, where before it was 60% slower.Machines with only AES-NI, or with 256-bit VAES and no AVX-512 (0.9.4 uses 30-block batches there), aren't slowed down (the laptop numbers below are noisy; if anything they're faster).
Benchmarks
MB/s, median of 3 passes, release build with
lto = "fat",codegen-units = 1. Each column is a separate crate depending on WarpCrypto with aes pinned to=0.9.2or=0.9.4. "master" is d6adf6f. "this PR" is the first commit (9419218), which cleared the batch buffer once per call. The second commit clears it once per batch again, as master does, so short calls skip the 1 KiB clear. In its 0.9.4 build the counter still stays in registers, and the per-batch clear is 16 512-bit stores per 64 blocks. A rerun on the Zen 3 laptop showed no difference between the two commits beyond its noise. The Zen 4 machine wasn't rerun.AMD EPYC, Zen 4 (family 25 model 17; AVX-512 + VAES → 64-block backend on 0.9.4), Linux, rustc 1.99.1
AMD Ryzen 5 PRO 5675U, Zen 3 (VAES, no AVX-512 → 30-block backend on 0.9.4), Windows, rustc 1.97.1. This is a laptop that was busy with other work, so expect about ±15% noise.
IGE doesn't use the batch path, and its numbers don't change.
Tests
ctr_matches_reference_across_batch_widthstest: checksctr256andCtr256against a one-block-at-a-time CTR built straight onaes. It covers lengths around the 8-, 30- and 64-block batch sizes, three startingstates, data starting off 16-byte alignment, and counters that carry into the high 64 bits or wrap past 2^128 in the middle of a batch. The last case also wraps inside the 4-block batches of the software backend. Under Miri it uses a smaller set of lengths.cargo test(debug and--release) andcargo clippy --all-targets -- -D warningspass with aes 0.9.2 (the locked version) and 0.9.4 on Windows/Zen 3, which exercises the 30-block backend, on both commits. The first commit also passed on Linux/Zen 4, which exercises the 64-block one.cargo +1.85 test --releasepasses (MSRV), checked on the first commit.RUSTFLAGS='--cfg aes_backend="soft"' cargo +nightly miri test --test rustpassed all 14 tests on the first commit, on Linux. On the second commit, the 7 CTR tests (ctr_filter) also pass under Miri, on Windows.Cargo.lockstays on aes 0.9.2 so the 1.85 CI job keeps working, and the wheels built from it don't change. The speedup reaches crates that resolve aes 0.9.4 themselves. No public API changes.If you want this out as a patch release, it would fit as 2.2.1. I've left the version alone.