s3s

S3 Service Adapter

crate	version	docs	coverage
s3s
s3s-aws			n/a
s3s-multipart
s3s-chunked
s3s-time
s3s-sigv2
s3s-sigv4
s3s-rfc2047
s3s-fs			n/a
s3s-http3			n/a
s3s-model			n/a
s3s-policy			n/a
s3s-proxy		n/a (binary crate)	n/a
s3s-test
s3s-e2e		n/a (binary crate)	n/a
s3s-wasm	not published (internal)	n/a	n/a

Coverage badges come from the CI coverage run and link to Codecov; crates outside that run show n/a.

📚 Development documentation for the main branch is available on GitHub Pages and is rebuilt once a day.

This experimental project intends to offer an ergonomic adapter for building S3-compatible services.

s3s implements Amazon S3 REST API in the form of a generic hyper service. S3-compatible services can focus on the S3 API itself and don't have to care about the HTTP layer.

s3s-aws provides useful types and integration with aws-sdk-s3.

s3s-multipart is a general-purpose asynchronous streaming parser for multipart/form-data. It is not tied to S3, and s3s builds on it for POST Object form uploads.

• Parsing: it consumes a stream of bytes::Bytes chunks and yields parts with their headers and data.
• S3 integration: crates/s3s/src/http/multipart.rs keeps the S3 contract on top of the parser.
• Protocol tests: crates/s3s-multipart/tests/protocol/.
• Benchmarks: crates/s3s-multipart/benches/ — parse_throughput, take_data_stream, and vs_multer, which compares it with multer.
• Fuzzing: fuzz/fuzz_targets/multipart_parser.rs.
• RFC differences: deliberate differences from RFC 2046 section 5.1 and RFC 7578 are documented in the Compatibility notes.

s3s-rfc2047 provides RFC 2047 MIME encoded-word encoding and decoding for non-ASCII header values.

s3s-fs implements the S3 API based on file system, as a sample implementation. It is designed for integration testing, which can be used to mock an S3 client. It also provides a binary for debugging. Play it!

The same file system can also be served over HTTP/3: build the binary with the optional http3 feature and pass --http3.

cargo install s3s-fs --features binary,http3
s3s-fs --http3 --port 8014 /data

• http://host:port keeps serving HTTP/1.1 and HTTP/2 over TCP, while https://host:port serves HTTP/3 on the same port.
• The server does not send Alt-Svc, so an HTTP/3 client connects explicitly.
• Without --cert and --key a self-signed certificate is generated; --cert-out writes the certificate a client has to trust. All three options require --http3.
• A default build has no HTTP/3 dependencies, and s3s-fs is published after s3s-http3.

s3s-http3 is an experimental, opt-in HTTP/3 transport: it serves an S3Service, or any tower::Service, over QUIC, so the same S3 API is reachable over UDP with TLS 1.3 and the h3 ALPN protocol. The adapter is server-side only, and its API may change while the HTTP/3 ecosystem evolves. Runnable servers are in crates/s3s-fs/examples/http3-server.rs and crates/s3s-http3/examples/serve-with.rs.

The other workspace members are supporting crates:

• s3s-time — the date and time wire formats of the S3 API (RFC 3339 date-time, IMF-fixdate, epoch seconds), and the value type behind s3s::dto::Timestamp.
• s3s-chunked — the aws-chunked streaming request-body decoder, initialized as a placeholder while the implementation is under development.
• s3s-sigv2, s3s-sigv4 — AWS Signature Version 2 and Version 4 parsing, canonicalization and signing.
• s3s-model — the S3 protocol model used by the code generator: the S3 error codes.
• s3s-policy — the S3 policy language model with wildcard pattern matching.
• s3s-proxy — a proxy implementation used by the end-to-end tests.
• s3s-test — a reusable test harness for S3-compatible services.
• s3s-e2e — the end-to-end test runner built on it.
• s3s-wasm — an internal crate (publish = false) that runs s3s under WebAssembly in its test suite.

Compatibility

100% S3 compatibility is not a realistic goal, because there is no single authoritative S3 specification to measure against: the S3 API is effectively defined by Amazon S3, and Amazon S3's own behavior does not always match the S3 API documentation exactly.

This project therefore aims for practical compatibility with real S3 clients and workflows on a best-effort basis. Compatibility is pursued as far as it is practical, not promised, and each known difference from Amazon S3 is tracked and handled as a separate issue rather than assumed away.

How it works

The diagram above shows how s3s works.

s3s converts HTTP requests to operation inputs before calling the user-defined service.

s3s converts operation outputs or errors to HTTP responses after calling the user-defined service.

The data types, serialization and deserialization are generated from the smithy model in aws-sdk-rust repository. We apply manual hacks to fix some problems in smithy server codegen and make s3s ready to use now.

Security

S3Service and other adapters in this project are not a complete security boundary. If they are exposed to the Internet directly, they may be attacked.

It is up to the user to implement security enhancements such as HTTP body length limits, object-size limits, rate limits and back pressure.

Authentication is required for production deployments. Without calling set_auth, the service accepts anonymous (unsigned) requests and skips authorization entirely: every S3 operation is open to any client that can reach the service, and signed requests fail with NotImplemented because no authentication provider is configured. A forgotten set_auth turns the service into a publicly readable and writable endpoint.

For streaming uploads (PUT Object, UploadPart), s3s applies a default 5 GiB object-size limit matching the AWS single-PUT limit; set S3Config::put_object_max_size to None to disable it and enforce deployment-specific caps in the S3 implementation. For production, set it explicitly even though the default is already 5 GiB. POST Object keeps using S3Config::post_object_max_file_size.

List-type responses (ListObjects, ListBuckets, ...) are serialized in full by s3s: their memory usage grows with the number of entries the S3 implementation returns. Implementations should paginate (max-keys / continuation tokens) and deployments should bound response sizes.

Docker

Docker images are available at GitHub Container Registry (GHCR).

See Docker documentation for usage details.

Contributing

• Development Guide

Sponsor

We have a reward funds pool for contributors: #174

If my open-source work has been helpful to you, please sponsor me.

Every little bit helps. Thank you!