Skip to content

feat: add ZipFileBuilder for parallelizing compression - #911

Open
alex wants to merge 3 commits into
zip-rs:masterfrom
alex:zip-file-builder
Open

feat: add ZipFileBuilder for parallelizing compression#911
alex wants to merge 3 commits into
zip-rs:masterfrom
alex:zip-file-builder

Conversation

@alex

@alex alex commented Aug 8, 2026

Copy link
Copy Markdown

Compresses a file's contents independently of any ZipWriter, so that multiple files can be compressed on separate threads and then appended serially with ZipWriter::add_prepared_file.

Copilot AI lite review requested due to automatic review settings August 8, 2026 13:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new “prepare then append” workflow to support parallelizing per-file compression work outside of ZipWriter, while still producing correct ZIP metadata when appending the pre-compressed payloads.

Changes:

  • Introduces ZipFileBuilder to compress a single file entry into memory while tracking CRC-32 and sizes, producing a PreparedZipFile.
  • Adds ZipWriter::add_prepared_file to append a PreparedZipFile by copying its already-compressed bytes verbatim (including handling ZIP64 header patching and stream/data-descriptor mode).
  • Adds tests covering roundtrip reading, duplicate-name rejection (and writer recovery), stream mode behavior, and encryption rejection.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@Its-Just-Nans Its-Just-Nans left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,

Thanks for the MR, a little question in comment.

Also, what do you think about adding a test with a thread (like the documentation)?

Comment thread src/write.rs Outdated
crc32: u32,
uncompressed_size: u64,
compressed_size: u64,
data: Vec<u8>,

@Its-Just-Nans Its-Just-Nans Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not using a Box<[u8]> ?

and change line 2526

-let data = cursor.into_inner();
+let data = cursor.into_inner().into_boxed_slice();

@alex

alex commented Aug 11, 2026

Copy link
Copy Markdown
Author

I originally had a test case with some threads (similar to the doc comment), but I removed it on the theory that the borrow checker meant it had to work :-) Happy to add it back.

alex and others added 2 commits August 11, 2026 18:08
Compresses a file's contents independently of any ZipWriter, so that
multiple files can be compressed on separate threads and then appended
serially with ZipWriter::add_prepared_file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 11, 2026 22:10
@alex
alex force-pushed the zip-file-builder branch from b657a1a to 888f3bc Compare August 11, 2026 22:10
@alex

alex commented Aug 11, 2026

Copy link
Copy Markdown
Author

Addressed, thanks!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/write.rs:1801

  • add_prepared_file() writes the already-compressed payload via ZipWriter::write_all(), which unconditionally updates ZipWriterStats (CRC32 + byte count) on the compressed bytes (see ZipWriter::write at write.rs:761+). For prepared files the CRC/sizes are already known, so this does extra CPU work proportional to the compressed size and can partially defeat the purpose of off-thread compression. Writing directly to the inner writer avoids the redundant hashing and bookkeeping.
        // start_entry leaves the inner writer as a bare Storer (a compression encoder is only
        // installed by start_file*), so the already-compressed bytes pass through unchanged.
        let result = self.write_all(&data);
        self.ok_or_abort_file(result)?;

Skips ZipWriter::write's per-byte stats bookkeeping, which is redundant
for pre-compressed data whose CRC and sizes are already known.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 11, 2026 22:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/write.rs:2332

  • Doc comment reads as if encryption-free options would fail. It likely meant that creating a builder will fail when encryption is requested; adding a comma clarifies the intended meaning.
/// Encryption is not supported; pass options without encryption or creation will fail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants