feat: prototype async API, with demonstrable perf improvements via benchmark - #73

Closed

Pr0methean wants to merge 162 commits into

Commits on Sep 7, 2023

use cfg-if in a few places
cosmicexplorer
committed
d9075e8
continue smoothing with feature guards
cosmicexplorer
committed
d0355d2
add feature info for docs.rs
cosmicexplorer
committed
426a621
use num_enum to clean up the System type
cosmicexplorer
committed
dc52677
use displaydoc and thiserror to remove some boilerplate
cosmicexplorer
committed
c1e1272
make DateTime parsing and error messaging much cleaner
cosmicexplorer
committed
ff76ca2
avoid making this a breaking change and add a doctest
cosmicexplorer
committed
00c6721

Commits on Sep 22, 2023

Commits on Sep 23, 2023

Limiter works now!!!
cosmicexplorer
committed
179ea98
get reader works!
cosmicexplorer
committed
0e93517
use TryStreamExt
cosmicexplorer
committed
8de2093
add criterion benchmarks that show async is slower lol
cosmicexplorer
committed
fee0cb3
ok, but at least async reading is much much faster than sync!!
cosmicexplorer
committed
24371b3
do lots of i/o copying and get very interesting results
cosmicexplorer
committed
8fc3138
mess around a little
cosmicexplorer
committed
d60985c
add tmp testproject
cosmicexplorer
committed
21a233c

Commits on Sep 25, 2023

make lots of changes at once oops
cosmicexplorer
committed
c9c4658
move write head stuff into ops::Drop impls
cosmicexplorer
committed
5f0cbbf
add PermitState enum for invariant checking
cosmicexplorer
committed
7faf9bd
add num_enum for easy transposition with atomics
cosmicexplorer
committed
18db885
expand Lease::AlreadyTaken
cosmicexplorer
committed
8bd8063
remove wrapper mod
cosmicexplorer
committed
eb02433

Commits on Sep 26, 2023

try to make the ring buffer atomic????
cosmicexplorer
committed
315f32e
allow truncating
cosmicexplorer
committed
1885103
initial impl of RingFuturized
cosmicexplorer
committed
a6d7a32
modify Read/WritePermit to remove the lifetime
cosmicexplorer
committed
239d246
use Ring for tokio!
cosmicexplorer
committed
e75422e
generalize the ReadAdapter
cosmicexplorer
committed
6f6f901
refactor RingFuturized via Pusher
cosmicexplorer
committed
729f325
make everything compile and pass tests again
cosmicexplorer
committed
11d606e
fix more stuff
cosmicexplorer
committed
e61b7d4
changes to tmp
cosmicexplorer
committed
6e76427

Commits on Sep 27, 2023

try to improve throughput in AsyncIoAdapter
cosmicexplorer
committed
fb4d8fc
make extraction like a second faster!!
cosmicexplorer
committed
436c688
set permissions on extracted files
cosmicexplorer
committed
42ea15b
split out work to create directories
cosmicexplorer
committed
012d6a8
split extraction into several pipelined phases
cosmicexplorer
committed
40023b0
almost on par with sync now!!!!!!
cosmicexplorer
committed
02890a0
another speedup
cosmicexplorer
committed
66f0652
parallelize dir creation
cosmicexplorer
committed
031b08e
remove unused imports
cosmicexplorer
committed
e53ca77
tiny refactor
cosmicexplorer
committed
0a9859a
get EXTREMELY close to sync performance!!!
cosmicexplorer
committed
609416b
fix matching task and we BEAT sync now!!!
cosmicexplorer
committed
dfde875
cleanup
cosmicexplorer
committed
da4135e
comments
cosmicexplorer
committed
0a2660a
tiny optimizations to the dir creation phase
cosmicexplorer
committed
784086c
use read_exact() and preallocate because we know exactly how much to read
cosmicexplorer
committed
013797f
reduce PathBuf allocations
cosmicexplorer
committed
0c32e9b
optimize buffering
cosmicexplorer
committed
391a739
write permission to file!
cosmicexplorer
committed
f3c65c4
be more explicit about how we open extraction output files
cosmicexplorer
committed
c3f01b1

Commits on Sep 30, 2023

refactor several things
cosmicexplorer
committed
d38e07f
improve benchmarks
cosmicexplorer
committed
33a034d
improve tmp harness
cosmicexplorer
committed
3223047
tiny improvement
cosmicexplorer
committed
bc601d7
remove extraneous trait bounds
cosmicexplorer
committed
ca7dba5
remove unnecessary macro
cosmicexplorer
committed
cc03c68
remove a lot of stuff we don't need
cosmicexplorer
committed
dcb71e6
remove FixedLengthFile
cosmicexplorer
committed
c78e598
remove changes to zipcrypto
cosmicexplorer
committed
6201f72
remove some changes to Cargo.toml
cosmicexplorer
committed
4ded289
move async stuff to tokio subdir
cosmicexplorer
committed
6d65c26
clean up Cargo.toml
cosmicexplorer
committed
4315346
expand async deflate support
cosmicexplorer
committed
355cf72

Commits on Oct 1, 2023

add BufWriter
cosmicexplorer
committed
9c5f0e3
we have a good BufWriter now!
cosmicexplorer
committed
ef76c6a
only one test failure!!!
cosmicexplorer
committed
c9dccc5
absurd pinning procedures fail to fix test
cosmicexplorer
committed
10653c2

Commits on Oct 2, 2023

reader tests work!
cosmicexplorer
committed
672c117
refactor some tests
cosmicexplorer
committed
1422278
propose alternative testing?
cosmicexplorer
committed
ba8868b
remove unused imports
cosmicexplorer
committed
a1cc633
nobody knows how to fix this for now lmao
cosmicexplorer
committed
7598b73
DOWN TO ONE TEST FAILURE
cosmicexplorer
committed
ed65552
all tests pass now
cosmicexplorer
committed
6d03139
code suddenly started working
cosmicexplorer
committed
4d53091
fix up some error messages
cosmicexplorer
committed
d8bf219
switch to normal pin-project crate and remove Unpin bounds!
cosmicexplorer
committed
9c50a08

Commits on Oct 5, 2023

move away from pin-project in favor of Pin<Box<...>>
cosmicexplorer
committed
1ff110a
add extract_simple() method
cosmicexplorer
committed
0095dec
writing works?????
cosmicexplorer
committed
c6a58e2
clean up some stuff
cosmicexplorer
committed
ba342bd
Merge branch 'use-displaydoc-thiserror' into async-executor
cosmicexplorer
committed
ba94c94
Merge branch 'num-enum-system' into async-executor
cosmicexplorer
committed
694eb56
Merge branch 'use-cfg-if' into async-executor
cosmicexplorer
committed
dc9103f
Merge branch 'fix-datetime-api' into async-executor
cosmicexplorer
committed
b421ac2
add finish_into_readable()
cosmicexplorer
committed
2b5c126
add merge_archive
cosmicexplorer
committed
64692bb
add broken merge_archive benchmark
cosmicexplorer
committed
39f10cc
make several things less public
cosmicexplorer
committed
1a27805
add new_append()
cosmicexplorer
committed
d93919f
expose Shared and SharedData
cosmicexplorer
committed
917de42
add is_dir() and unix_mode()
cosmicexplorer
committed
aaf4d2f
fix errors
cosmicexplorer
committed
8e0e1d3
add os.rs
cosmicexplorer
committed
5e1dae4
make os submodule
cosmicexplorer
committed
193d575

Commits on Oct 6, 2023

add file descriptor validation from tokio-pipe
cosmicexplorer
committed
b4a9ea9
copy_file_range() works????/
cosmicexplorer
committed
3cb9dd8
ownership comes into play
cosmicexplorer
committed
f45f8f1
refactor fd wrappers
cosmicexplorer
committed
c92484e
flesh out pipe interactions
cosmicexplorer
committed
7bf627c
add benchmark which shows copy_file_range EXACTLY the same as sync?
cosmicexplorer
committed
be22e05
add copy-file-range
cosmicexplorer
committed
5921729
clean a lot of stuff up
cosmicexplorer
committed
c46d124
make central directory parsing faster (?)
cosmicexplorer
committed
8ebd379
cleanup
cosmicexplorer
committed
d00e547
make tmp work by fixing memchr dep
cosmicexplorer
committed
e25c68f
write_async changes
cosmicexplorer
committed
e6a3f78
check for vectorized write
cosmicexplorer
committed
6047440
more cleanup
cosmicexplorer
committed
c9c5811
streamline local header writing
cosmicexplorer
committed
b8c21ce
move LocalHeaderBuffer to spec
cosmicexplorer
committed
05d7854
introduce another wrapper
cosmicexplorer
committed
270b6cd
buffer more
cosmicexplorer
committed
3d7d49a
streamline more
cosmicexplorer
committed
d7c7026
more streamlining
cosmicexplorer
committed
fe3577b
more streamlining
cosmicexplorer
committed
18be9da
more streamlining
cosmicexplorer
committed
25aa63c
more streamlining
cosmicexplorer
committed
3899460
vectorize write for more places
cosmicexplorer
committed
d52920a
cleanup
cosmicexplorer
committed
3728d4d
more streamlining
cosmicexplorer
committed
5b657e1
fix endianness
cosmicexplorer
committed
e85498b

Commits on Oct 7, 2023

add static assertion
cosmicexplorer
committed
512a833
fix compilation errors
cosmicexplorer
committed
77e78fe

Commits on Oct 8, 2023

delete some code
cosmicexplorer
committed
f4957ec
clean up RawArgs a bit
cosmicexplorer
committed
cfe9f77
remove tokio-pipe dep
cosmicexplorer
committed
0009b2d
fix memchr dep
cosmicexplorer
committed
a273b84
cleanup
cosmicexplorer
committed
a4d640b
fix a TERRIBLE bug lmfao
cosmicexplorer
committed
4110282
improve perf of locating the initial zip central directory
cosmicexplorer
committed
52a5c9b
avoid dynamic memory allocation when searching for *end* of central dir record
cosmicexplorer
committed
9a994c2
fix a bug and simplify our header parsing a lot
cosmicexplorer
committed
c886931
remove debug statements
cosmicexplorer
committed
9392044

Commits on Oct 10, 2023

enable splicing pipes to and from files
cosmicexplorer
committed
1cf3fb9
impl IntoRawFd for MutateInnerOffset
cosmicexplorer
committed
5c82074
slightly improve file splice test
cosmicexplorer
committed
ba6d8d6
test requesting more splice than available
cosmicexplorer
committed
542cda0
add test for vectored write impls
cosmicexplorer
committed
455a942
clean up buf reader/writer
cosmicexplorer
committed
35c04c7
tiny refactor
cosmicexplorer
committed
6af327b
fix bug in merge_archive() -- support nonzero archive_offset
cosmicexplorer
committed
6d64862
add rayon dep
cosmicexplorer
committed
aea5359
tiny fix to doctest
cosmicexplorer
committed
3c2bf8d
add another rayon usage
cosmicexplorer
committed
d07a48e
fix note
cosmicexplorer
committed
87f21bb
add impl for splitting into files, but tests are broken
cosmicexplorer
committed
6843d2d

Commits on Oct 11, 2023

stop making Limiter impl Seek
cosmicexplorer
committed
1da7060
Revert "stop making Limiter impl Seek"
cosmicexplorer
committed
d48617c

Commits on Oct 16, 2023

Commits on Oct 17, 2023

impl for MutateInnerOffset
cosmicexplorer
committed
f755697

Commits on May 1, 2024

Merge remote-tracking branch 'cosmicex/async-executor' into oldpr409
Pr0methean
committed
952bfc9

Commits on May 2, 2024

Merge branch 'master' into oldpr409
Pr0methean
authored
049ce91