Add support for the compressed-blobs REAPI proposal - #386

Merged

mostynb merged 35 commits into

buchgr:masterfrom

mostynb:compressed_blobs_support

Mar 2, 2021

Commits on Mar 1, 2021

use the proposed compressed-blobs REAPI
mostynb
committed
eb612d7
[disk] store CAS blobs in compressed form
mostynb
committed
712e06e
[disk] try to preserve atimes when migrating CAS blobs
mostynb
committed
e9fbf6e
[bytestream] add support for compressed-blobs reads and writes
mostynb
committed
f522e31
[http] allow zstd-encoded CAS GETs
mostynb
committed
e1d3c4d
[http] accept zstd-encoded PUTs
mostynb
committed
ba069f8
[http] document how to upload/download zstd-compressed data
mostynb
committed
5eac267
[zstd] limit concurrency to 1 for single-use {en,de}coders
mostynb
committed
997695a
[zstd] use decoder and encoder pools
mostynb
committed
fe70a24
[bytestream] be more careful about calling Close on the result of Get calls
mostynb
committed
0accc93
[disk] make the CAS storage mode (zstd/uncompressed) configurable
mostynb
committed
6cc58ac
[disk] fan out when migrating old data files
mostynb
committed
2176041
[disk] use raw uncompressed CAS blob files
mostynb
committed
038e067
[disk] validate cache file hash values when loading
mostynb
committed
f8eb86c
remove --destructive_compression_prototype flag
mostynb
committed
512ad74
doc: mention compressed-blobs in the gRPC section of the readme file
mostynb
committed
d5ad6e7
[disk] fix space accounting
mostynb
committed
e873612
[httpproxy] don't read remote file headers for CAS.v2 Contains size value
mostynb
committed
eaffa82
[disk] make compressed cas blob files valid .zst
mostynb
committed
67a2ad0
[disk] encode logical CAS blob sizes in the filename
mostynb
committed
9154319
[disk] fix --storage_mode uncompressed uploads
mostynb
committed
3832c9c
[proxy] add cmdline flags for http and gcs backend proxies
mostynb
committed
f907889
[proxy] switch between v1 and v2 CAS blobs depending on the storage mode
mostynb
committed
0cdfdeb
[s3proxy] don't read remote file headers for CAS.v2 Contains size value
mostynb
committed
163eb40
[http] don't interact with files in tests, use disk.Cache public API
mostynb
committed
9af561e
[disk] rework the disk storage to avoid fs operations while holding the LRU lock
mostynb
committed
0bdc1b4
[casblob] sanity check file size
mostynb
committed
63ba6fc
[casblob] add note that we never store empty files
mostynb
committed
b12351e
[casblob] add missing Close on error conditions
mostynb
committed
605d444
[casblob] make the GetUncompressedReadCloser and GetZstdReadCloser comments more explicit
mostynb
committed
c24f861
[casblob] GetLegacyZstdReadCloser should pass errors to readers
mostynb
committed
ed9901a
[casblob] read and write chunk offset table with one binary.Read/Write call
mostynb
committed
311fb8e
[casblob] proxy backends should return the logical size for Get
mostynb
committed
ff4c853
[casblob] test a basic assumption about the sizeof int
mostynb
committed
564d20e
[disk] accept ascii characters in random string of blob filenames
mostynb
committed
48e8649

Navigation Menu

Add support for the compressed-blobs REAPI proposal - #386

Add support for the compressed-blobs REAPI proposal#386
mostynb merged 35 commits into
buchgr:masterfrom
mostynb:compressed_blobs_support

Commits on Mar 1, 2021

use the proposed compressed-blobs REAPI

[disk] store CAS blobs in compressed form

[disk] try to preserve atimes when migrating CAS blobs

[bytestream] add support for compressed-blobs reads and writes

[http] allow zstd-encoded CAS GETs

[http] accept zstd-encoded PUTs

[http] document how to upload/download zstd-compressed data

[zstd] limit concurrency to 1 for single-use {en,de}coders

[zstd] use decoder and encoder pools

[bytestream] be more careful about calling Close on the result of Get calls

[disk] make the CAS storage mode (zstd/uncompressed) configurable

[disk] fan out when migrating old data files

[disk] use raw uncompressed CAS blob files

[disk] validate cache file hash values when loading

remove --destructive_compression_prototype flag

doc: mention compressed-blobs in the gRPC section of the readme file

[disk] fix space accounting

[httpproxy] don't read remote file headers for CAS.v2 Contains size value

[disk] make compressed cas blob files valid .zst

[disk] encode logical CAS blob sizes in the filename

[disk] fix --storage_mode uncompressed uploads

[proxy] add cmdline flags for http and gcs backend proxies

[proxy] switch between v1 and v2 CAS blobs depending on the storage mode

[s3proxy] don't read remote file headers for CAS.v2 Contains size value

[http] don't interact with files in tests, use disk.Cache public API

[disk] rework the disk storage to avoid fs operations while holding the LRU lock

[casblob] sanity check file size

[casblob] add note that we never store empty files

[casblob] add missing Close on error conditions

[casblob] make the GetUncompressedReadCloser and GetZstdReadCloser comments more explicit

[casblob] GetLegacyZstdReadCloser should pass errors to readers

[casblob] read and write chunk offset table with one binary.Read/Write call

[casblob] proxy backends should return the logical size for Get

[casblob] test a basic assumption about the sizeof int

[disk] accept ascii characters in random string of blob filenames