Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ jobs:
images: ghcr.io/txpipe/dolos
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=raw,value=stable,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=semver,pattern=v{{major}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=semver,pattern=v{{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=raw,value=stable,enable=${{ startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-') }}
type=semver,pattern=v{{major}},enable=${{ startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-') }}
type=semver,pattern=v{{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-') }}
type=semver,pattern=v{{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=sha

Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ Dolos uses three distinct storage backends, each serving a specific purpose:
- **Database**: `<storage.path>/wal`

### Where the indexes live
There is no standalone index store — it was removed in v1.7. Every index is a
projection, and lives in the store that holds what it projects:
There is no standalone index store. Every index is a projection and lives in
the store that holds what it projects:
- the live-UTxO tags (by address, payment, stake, policy, asset, script ref) project the UTxO set and live in the `StateStore` (`StateStore::utxos_by_tag`, written through `StateWriter::apply_utxo_tags` in the same batch as the set)
- the archive tags and the exact lookups (by block hash, block number, tx hash) project the block history and live in the `ArchiveStore` (`ArchiveStore::slots_by_tag` / `slot_by_*`, written through `ArchiveWriter::apply_index` in the same batch as the blocks)

Expand Down
9 changes: 7 additions & 2 deletions crates/fjall/src/archive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ This module implements the `ArchiveStore` trait using [Fjall](https://github.com
| 3 | `archive-tags` | Block tags, append-only | Prefix scans by dimension and key |
| 4 | `index-exact` | Block hash / block number / tx hash → slot | Point lookups |

The last two are projections of the blocks. They were a separate database until v1.7 (`<storage.path>/index`), and moving them here is what lets them be written in the same batch as the block locations they point at, so the history and its lookups commit together. They keep the compaction settings the standalone store gave them — `l0_threshold = 8`, `memtable_size_mb = 128` — so their behavior did not change with the move.
The last two are projections of the blocks. Keeping them here lets the history
and its lookups commit in the same batch. Both keyspaces use
`l0_threshold = 8` and `memtable_size_mb = 128`.

## Key Schemas

Expand Down Expand Up @@ -75,7 +77,10 @@ Internal prefix constants:

## Stelae

The `indexes` stele layer is the sorted output of `ArchiveStore::iter_archive_tags` followed by `iter_exact_records`, and the input of `ArchiveWriter::append_prehashed`. It is byte-identical to what the standalone index store produced, so the move required no media-type bump and no backfill. See `scan.rs` for the prefix walk both traversals share.
The `indexes` stele layer is the sorted output of
`ArchiveStore::iter_archive_tags` followed by `iter_exact_records`, and the
input of `ArchiveWriter::append_prehashed`. See `scan.rs` for the prefix walk
both traversals share.

## Pruning

Expand Down
10 changes: 3 additions & 7 deletions crates/fjall/src/state/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,9 @@ Value: (empty)
- `lookup_key`: the logical key, stored verbatim so a prefix scan can find it
- `txo_ref`: `[tx_hash:32][index:4]`, the UTxO the tag points at

They are a projection of the UTxO set, so they are written in the same batch as the set (`StateWriter::apply_utxo_tags` / `undo_utxo_tags`) and read through `StateStore::utxos_by_tag`. Before v1.7 they lived in a separate index database; nothing about the key encoding changed with the move, only which journal, cache and write batch they live under.
They are a projection of the UTxO set, so they are written in the same batch as
the set (`StateWriter::apply_utxo_tags` / `undo_utxo_tags`) and read through
`StateStore::utxos_by_tag`.

Note the asymmetry with the archive's tags, which hash their key: here the lookup key is stored whole, because a live-UTxO query knows the key it is asking about and wants the exact refs back.

Expand Down Expand Up @@ -194,9 +196,3 @@ writer.commit()?;
| Schema parameter | Not required | Required |
| Entity keyspaces | Unified with hash prefix | Separate per namespace |
| Multimap support | Not supported | Supported |

## Migration Notes

This 4-keyspace design is **not backward compatible** with previous versions that used separate keyspaces per entity type. Users must recreate their state databases when upgrading.

The removal of the schema parameter from `StateStore::open()` is also a breaking API change.
4 changes: 1 addition & 3 deletions docs/content/architecture/data-layer.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,13 @@ Under the configured `storage.path`, a running node maintains three on-disk stor
| **Archive** | Immutable history: full block bodies, time-indexed logs of entity changes, and the reverse lookups over them — slots by address, payment credential, stake credential, policy, asset, datum and more, plus slot by block hash, block number and tx hash. | History is append-only and read differently from current state, so it uses its own layout (see below). The lookups are a projection of the blocks, so they live beside them and commit with them. |
| **Mempool** | Submitted-but-unconfirmed transactions and their lifecycle state. | Pending transactions are transient and overlaid on top of committed state during validation. |

Before v1.7 the reverse lookups lived in a fourth store of their own, configured under `[storage.index]`. That store is gone, and so is the table.

## Storage traits and pluggable backends

Each store is defined as a trait in `dolos-core` — `WalStore`, `StateStore`, `ArchiveStore`, and `MempoolStore` — and the rest of Dolos only ever talks to those traits. Concrete implementations are selected at runtime in `src/adapters/storage.rs`, which wraps each backend in an enum so a node can mix engines per store.

Available backends:

- **redb** (`dolos-redb3`) — an embedded ACID B+tree store. It backs the WAL (the only WAL implementation) and the mempool; its state and archive config variants were removed in v1.7, so a configuration naming them fails to load.
- **redb** (`dolos-redb3`) — an embedded ACID B+tree store. It backs the WAL (the only WAL implementation) and the mempool. It is not a state or archive backend.
- **fjall** (`dolos-fjall`) — an LSM-tree engine tuned for write-heavy workloads with many hot keys. The only persistent backend for the state and archive stores, and the default for both.
- **no-op** — a store that silently discards writes, used to *disable* the archive.
- **in-memory** — non-persistent stores for testing and ephemeral nodes. State and archive have builtin implementations in `dolos-core` backed by ordered maps, which serve their traits in full; the WAL and mempool use redb's memory backend instead.
Expand Down
31 changes: 13 additions & 18 deletions docs/content/configuration/schema.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,14 +109,12 @@ The `storage` section controls how Dolos stores data in the local file system. E
| version | string | "v4" |

- `path`: root directory where all data will be stored.
- `version`: storage schema version (`v0` - `v4`). This release only reads
`v4`, which stores independently compressed zstd-3 block frames using the
bundled dictionary and folds the former index store into state and archive;
there are no compression settings. Run `dolos init` to upgrade a readable
older configuration, or follow the manual steps in the
[migration guide](https://docs.txpipe.io/dolos/migration/dolos-v1-7) when it
cannot be read. Relabelling an older data directory as `v4` is unsupported
and is not detected.
- `version`: storage schema version (`v0` - `v4`). Dolos reads `v4`, which
stores independently compressed zstd-3 block frames using the bundled
dictionary and keeps indexes in the state and archive stores; there are no
compression settings. Run `dolos init` and
[bootstrap](../bootstrap) to replace data in another format. Relabelling a
data directory as `v4` is unsupported and is not detected.

Besides the stores, this directory holds `scratch/`: where `dolos snapshot
publish`, `dolos snapshot backfill`, `dolos bootstrap stelae` and `dolos
Expand Down Expand Up @@ -155,7 +153,7 @@ deletes the files it has consumed as it goes. `--download-dir` moves it.
| worker_threads | integer | 4 |
| memtable_size_mb | integer | 64 |

- `backend`: `fjall` or `in_memory` (defaults to `fjall`). `in_memory` keeps the whole state in process memory: it is ephemeral — everything is lost on restart — and it holds the UTxO set and all entities in RAM, so it suits devnets, tooling and tests rather than a node following a public network. The `redb` value was removed in v1.7: a configuration still naming it fails to load, and an existing redb state directory has to be re-bootstrapped (a stelae restore is the shortest path).
- `backend`: `fjall` or `in_memory` (defaults to `fjall`). `in_memory` keeps the whole state in process memory: it is ephemeral — everything is lost on restart — and it holds the UTxO set and all entities in RAM, so it suits devnets, tooling and tests rather than a node following a public network.
- `path`: optional override for the state path (defaults to `<storage.path>/state`).
- `cache`: size (MB) of the state cache.
- `max_history`: maximum number of slots to keep before pruning.
Expand All @@ -175,23 +173,20 @@ deletes the files it has consumed as it goes. `--download-dir` moves it.
| worker_threads | integer | 4 |
| memtable_size_mb | integer | 64 |

- `backend`: `fjall`, `in_memory`, or `no_op` (defaults to `fjall`). `in_memory` keeps the whole archive in process memory, block bodies included: ephemeral — everything is lost on restart — so it suits devnets, tooling and tests rather than a node following a public network. `no_op` disables the archive entirely. The `redb` value was removed in v1.7: a configuration still naming it fails to load, and an existing redb archive has to be re-bootstrapped (a stelae restore is the shortest path).
- `backend`: `fjall`, `in_memory`, or `no_op` (defaults to `fjall`). `in_memory` keeps the whole archive in process memory, block bodies included: ephemeral — everything is lost on restart — so it suits devnets, tooling and tests rather than a node following a public network. `no_op` disables the archive entirely.
- `path`: optional override for the archive path (defaults to `<storage.path>/archive`).
- `blocks_path`: optional override for block segment files.
- `cache`: size (MB) of the archive index cache.
- `max_journal_size`, `flush_on_commit`, `l0_threshold`, `worker_threads`, `memtable_size_mb`: Fjall tuning options.

Block segment files are always compressed: every block body is written as one zstd frame (level 3, checksummed) with the dictionary bundled in the binary and addressed by its physical location, from the first block of a fresh instance on. Nothing about this is configurable — there is no profile, dictionary path, cache bound, training, sealing or maintenance command — and the layout is part of storage `v4`. A key the archive table does not know, such as the `block_compression` table of an earlier build, fails the configuration load rather than being ignored.
Block segment files are always compressed: every block body is written as one zstd frame (level 3, checksummed) with the dictionary bundled in the binary and addressed by its physical location, from the first block of a fresh instance on. Nothing about this is configurable — there is no profile, dictionary path, cache bound, training, sealing or maintenance command — and the layout is part of storage `v4`. Unknown archive settings fail configuration loading.

The dictionary is part of the storage format, not a setting: every build carries the same 112,640-byte asset (SHA-256 `c47b2eb1f69a997bf01a720f26bcc1b668e0fddb95431e91c90cebf4f9b87139`, zstd dictionary id `1075630411`), every frame names it in its header, and a frame written for any other dictionary is refused on read. Its provenance and the evaluation behind it are in the repository under `crates/flatfiles/dictionary/`.

A data directory written by an earlier layout is not readable and is not converted: bring a `v4` instance up by a fresh bootstrap (`dolos bootstrap mithril` or `relay`) or by importing a logical snapshot (`dolos bootstrap stelae`, the shortest path), both of which write the compressed layout from the first block. The version check is the only enforcement and it is at the configuration level: a `v3` or older `dolos.toml` is refused with the remedy named, and `dolos init` rewrites it. There is no check of the data itself, so a data directory from an older layout whose configuration was relabelled `v4` by hand is unsupported and is not detected — its blocks are not frames and reads of them fail.

### `storage.index` section — removed in v1.7

The standalone index store is gone. Its lookups moved into the stores that hold what they project: the live-UTxO tags into `storage.state`, and the historical tags and exact lookups into `storage.archive`. Drop the table from the configuration; v1.7 bumps the storage version, so an existing data directory has to be re-bootstrapped (a stelae restore is the shortest path) and its configuration revisited anyway.

Any tuning the table carried (`cache`, `worker_threads`, `max_journal_size`, and the rest) should be re-applied to `storage.state` and `storage.archive`, which now carry the keyspaces. `index.backend = "no_op"` had no replacement and needs none: `archive.backend = "no_op"` is the ledger-only switch and drops the historical lookups with the archive, while the live-UTxO tags stay with the UTxO set they project.
Dolos opens only storage declared as `v4`. Run `dolos init` and bootstrap with
Mithril, a relay, or a compatible Stelae snapshot to create the required
layout. The version check applies to the configuration; relabelling data in
another format as `v4` is unsupported and is not detected.

### `storage.mempool` section

Expand Down
2 changes: 1 addition & 1 deletion docs/content/installation/binaries.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@ npm install @txpipe/dolos
| [dolos-aarch64-apple-darwin.tar.gz](https://github.com/txpipe/dolos/releases/latest/download/dolos-aarch64-apple-darwin.tar.gz) | Apple Silicon macOS |
| [dolos-x86_64-pc-windows-msvc.tar.gz](https://github.com/txpipe/dolos/releases/latest/download/dolos-x86_64-pc-windows-msvc.tar.gz) | x64 Windows |
| [dolos-x86_64-unknown-linux-gnu.tar.gz](https://github.com/txpipe/dolos/releases/latest/download/dolos-x86_64-unknown-linux-gnu.tar.gz) | x64 Linux |
| [dolos-aarch64-unknown-linux-gnu.tar.gz](https://github.com/txpipe/dolos/releases/latest/download/dolos-aarch64-unknown-linux-gnu.tar.gz) | ARM64 Linux |
| [dolos-aarch64-unknown-linux-gnu.tar.gz](https://github.com/txpipe/dolos/releases/latest/download/dolos-aarch64-unknown-linux-gnu.tar.gz) | ARM64 Linux |
20 changes: 19 additions & 1 deletion docs/content/installation/docker.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ docker run ghcr.io/txpipe/dolos:latest

The result of the above command should show _Dolos'_ command-line help message.

`latest` deliberately tracks the `main` branch. It is useful for testing the
current development build, but it is mutable and is not the stable-release
channel. Use `stable` for the most recent stable release or an exact version tag
for reproducible deployments.

## Entry Point

The entry-point of the image points to _Dolos_ executable. You can pass the same command-line arguments that you would pass to the binary release running bare-metal. For example:
Expand Down Expand Up @@ -39,7 +44,20 @@ To use a versioned image, replace the `latest` tag by the desired version with t
ghcr.io/txpipe/dolos:v1
```

The `v1` tag is a floating alias that tracks the latest `1.x` release. For production deployments, consider pinning to a specific version (e.g. `v1.0.0`).
The `v1` tag is a floating alias that tracks the latest stable `1.x` release. For production deployments, consider pinning to a specific version (e.g. `v1.0.0`).

## Prerelease Images

Prerelease images are published under their exact version tag and an immutable
commit SHA tag. For example:

```sh
docker pull ghcr.io/txpipe/dolos:v2.0.0-alpha.0
docker run ghcr.io/txpipe/dolos:v2.0.0-alpha.0 --help
```

Prereleases do not update `stable`, major aliases such as `v2`, or minor
aliases such as `v2.0`. The `latest` tag follows `main` independently.

## Multiple Architectures

Expand Down
19 changes: 9 additions & 10 deletions src/adapters/storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -136,13 +136,12 @@ pub fn inspect_existing_data(
))
}

/// The storage version this binary reads. A store built by an older dolos is
/// not migrated in place: the supported path off it is a fresh `dolos init`
/// followed by a restore or a re-sync.
/// The only storage version this binary reads. Data in another format must be
/// cleared and bootstrapped again.
pub const CURRENT_STORAGE_VERSION: StorageVersion = StorageVersion::V4;

/// The migration guide the refusal points an operator at.
pub const MIGRATION_GUIDE_URL: &str = "https://docs.txpipe.io/dolos/migration/dolos-v1-7";
/// The bootstrap guide the refusal points an operator at.
pub const BOOTSTRAP_GUIDE_URL: &str = "https://docs.txpipe.io/dolos/bootstrap";

/// Refuse a configuration at any storage version but the current one.
///
Expand All @@ -156,7 +155,7 @@ fn check_storage_version(version: &StorageVersion) -> Result<(), Error> {
return Err(Error::StorageError(format!(
"unsupported storage version `{version}`, this dolos only supports \
`{CURRENT_STORAGE_VERSION}`; run `dolos init` to upgrade the configuration and \
re-bootstrap the data — see the migration guide at {MIGRATION_GUIDE_URL}"
re-bootstrap the data — see the bootstrap guide at {BOOTSTRAP_GUIDE_URL}"
)));
}
Ok(())
Expand Down Expand Up @@ -1400,8 +1399,8 @@ mod tests {
toml::from_str(&toml).unwrap()
}

/// A v1.6-era configuration is refused, and the refusal names both the
/// tool that performs the migration and the guide that describes it.
/// An unsupported configuration is refused, and the refusal names both
/// the tool that prepares it and the bootstrap guide.
#[test]
fn older_storage_versions_are_refused_with_the_remedy() {
for stale in [
Expand All @@ -1425,8 +1424,8 @@ mod tests {
"refusal must name the remedy: {message}"
);
assert!(
message.contains(MIGRATION_GUIDE_URL),
"refusal must point at the migration guide: {message}"
message.contains(BOOTSTRAP_GUIDE_URL),
"refusal must point at the bootstrap guide: {message}"
);
}

Expand Down
Loading
Loading