Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,40 @@ This design provides the same durability guarantees as calling `fsync` after eve
go get github.com/alialaee/logfile
```

## Benchmarks

See [benchmark/README.md](benchmark/README.md).

Here's the latest result from my laptop (M4 MacBook Air 2025, on-battery):

```
=== Logfile Write (This library)
Total bytes written: 477 MB
Time taken: 576.585416ms
Throughput: 828.85 MB/s

=== Direct File Write+Flush ===
Total bytes written: 119 MB
Time taken: 1.022756167s
Throughput: 116.82 MB/s

=== Tidwall WAL Write ===
Total bytes written: 119 MB
Time taken: 1.117973042s
Throughput: 106.87 MB/s

=== Hashicorp Raft Write ===
Total bytes written: 119 MB
Time taken: 1.197531459s
Throughput: 99.77 MB/s

=== Pebble Record Write ===
Total bytes written: 477 MB
Time taken: 678.86125ms
Throughput: 703.98 MB/s
```


## Usage

```go
Expand Down
44 changes: 44 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Benchmarking

The benchmarking is a work in progress, but the initial results are promising. The benchmark writes 50 records of random sizes between 512 bytes and 1MB, with a concurrency of 20 goroutines (each writing the same set of records), and measures the total time taken for all writes to complete. If the library doesn't support concurrent writes, it will just write the records sequentially in a single goroutine.


# Libraries

- **Logfile**: the library implemented in this repo.
- **Direct File Write+Flush**: a simple implementation that writes each record directly to the file and calls `Sync` after each write, without any batching or optimization.
- **Tidwall WAL**: a popular write-ahead log library that supports concurrent writes and fsync, but doesn't use group commit.
- **Hashicorp Raft**: the write-ahead log implementation used in Hashicorp's Raft library, which also supports concurrent writes but doesn't use group commit.
- **Pebble Record**: the record writer used in the Pebble key-value store.


## Results

Here's the initial benchmark results on my machine:

```
=== Logfile Write ===
Total bytes written: 477 MB
Time taken: 576.585416ms
Throughput: 828.85 MB/s

=== Direct File Write+Flush ===
Total bytes written: 119 MB
Time taken: 1.022756167s
Throughput: 116.82 MB/s

=== Tidwall WAL Write ===
Total bytes written: 119 MB
Time taken: 1.117973042s
Throughput: 106.87 MB/s

=== Hashicorp Raft Write ===
Total bytes written: 119 MB
Time taken: 1.197531459s
Throughput: 99.77 MB/s

=== Pebble Record Write ===
Total bytes written: 477 MB
Time taken: 678.86125ms
Throughput: 703.98 MB/s
```
55 changes: 55 additions & 0 deletions benchmark/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
module github.com/alialaee/logfile/benchmark

go 1.26.1

require (
github.com/alialaee/logfile v0.0.0-20260311130226-838864baad48
github.com/cockroachdb/pebble v1.1.5
github.com/hashicorp/raft v1.4.0
github.com/hashicorp/raft-wal v0.4.2
github.com/tidwall/wal v1.2.1
)

replace github.com/alialaee/logfile => ../

require (
github.com/armon/go-metrics v0.4.1 // indirect
github.com/benbjohnson/immutable v0.4.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cockroachdb/errors v1.11.3 // indirect
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b // indirect
github.com/cockroachdb/redact v1.1.5 // indirect
github.com/coreos/etcd v3.3.27+incompatible // indirect
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf // indirect
github.com/coreos/pkg v0.0.0-20220810130054-c7d1c02cb6cf // indirect
github.com/fatih/color v1.9.0 // indirect
github.com/getsentry/sentry-go v0.27.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/hashicorp/go-hclog v0.14.1 // indirect
github.com/hashicorp/go-immutable-radix v1.3.0 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-msgpack v1.1.5 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.17 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.15.0 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.42.0 // indirect
github.com/prometheus/procfs v0.9.0 // indirect
github.com/rogpeppe/go-internal v1.9.0 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.2.0 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/tinylru v1.2.1 // indirect
go.etcd.io/bbolt v1.3.6 // indirect
golang.org/x/exp v0.0.0-20230626212559-97b1e661b5df // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
google.golang.org/protobuf v1.33.0 // indirect
)
Loading
Loading