Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,6 @@ tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }
stats_alloc = "0.1"
serial_test = "3"
dolos-redb3 = { path = "crates/redb3" }
stelae = { path = "crates/stelae" }

[target.'cfg(unix)'.dev-dependencies]
nix = { version = "0.30.1", features = ["signal", "resource"] }
Expand Down
52 changes: 36 additions & 16 deletions adrs/004_stelae_snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,40 +309,60 @@ The arithmetic is counted in layers, because layers are what the ceiling counts:

### Code layout

Two crates, both workspace members. The split is the protocol/profile boundary made mechanical: **`cargo tree -p stelae` must contain no `dolos-*` package**, so extracting the protocol later is a directory move rather than a refactor.
Four crates, all workspace members until the extraction. The Stelae half is two of them — the protocol a third party implements from and the profile-generic lifecycle machinery — and the boundary is checkable: **`cargo tree -e normal --all-features` for `stelae` and `stelae-driver` must contain no `dolos-*` package**, so extracting the pair is a directory move rather than a refactor.

```
crates/stelae/ # package `stelae` — protocol, zero dolos deps
```text
crates/stelae/ # package `stelae` — the wire protocol, zero dolos deps
lib.rs # errors, protocol constants, envelope media types
frame.rs # deterministic CBOR-seq record read/write
inscription.rs # schema, JCS encode/verify, digest, history invariant
frame.rs # deterministic CBOR-seq record read/write, Limits
codec.rs # fixed-arity decode helpers for layer content records
inscription.rs # schema, JCS encode/verify, digest, history invariant (history_for)
profile.rs # Profile trait, layer-kind registry, media-type & tag naming rules
digest.rs # streaming sha256 + zstd (diffId + blob digest in one pass)
sign.rs # Ed25519 detached signatures, trusted keys, k-of-n
plan.rs # restore planning: layer selection, progress file, resume, preflight
layer.rs # reading a layer without holding it
plan.rs # progress file, resume, remaining-bytes accounting
progress.rs # Observer: what a transfer says about itself while running
transport.rs # the SteleReader/SteleWriter seam and the blob index
dir.rs # a stele on a local filesystem
oci.rs # feature `oci`: push with blob-skip, pull missing-only, tags, referrers
tests/toy_profile.rs # a second, trivial profile — proves the core carries no Dolos assumption

crates/stelae-driver/ # package `stelae-driver` — profile-generic lifecycle, zero dolos deps
lib.rs # the driver's Error
profile.rs # DriverProfile: the dataset policy stelae::Profile deliberately refuses
predecessor.rs # Predecessor/First: what a publish follows and may carry forward
publish.rs # the chained-publish lifecycle: open, Tuning, Publishing, Chained, standing
restore.rs # Budget/Checkpoint/Outlook: restore bounds and the resume checkpoint
preflight.rs # one free-space policy, in both directions
reporting.rs # counting layers and records for the two drivers to report
retry.rs # bounded patience for an external that fails in bursts
digests.rs # the digests-layer codec (Cardano immutable-DB file hashes)

crates/snapshot/ # package `dolos-snapshot` — the io.txpipe.dolos.cardano profile
lib.rs # DolosProfile: name/version, media types, tag rendering, position/parameters
lib.rs # DolosProfile, driver re-exports, profile constants, error mapping
namespaces.rs # the closed set of state namespaces a Dolos stele carries
layers/{blocks,indexes,logs,state,digests}.rs
export.rs # stores -> layers, generic over dolos_core::Domain
restore.rs # layers -> store writes, per-epoch checkpointing
restore.rs # layer selection and restore into store writes
planning.rs # one epoch selection, one reading of the plan it produces
registry.rs # store-typed publish/preview/restore over OCI; Point, verify, inspect
publisher.rs # publishing as a sequence of steps a command drives
node.rs # registry auth and scratch-dir policy from node configuration
backfill.rs # feature `backfill`: the epoch-at-a-time publisher daemon

crates/mithril/ # package `dolos-mithril` — the aggregator fetch; no stelae dependency
```

New deps: `zstd`, `serde_jcs`, `ed25519-dalek`, `oci-client`. The Dolos-side crate keeps the name `snapshot` because that is this project's word for the artifact (`dolos snapshot`, `[snapshot]`, `tests/e2e/snapshot.rs`); `stele` is the protocol's word for the same thing.

Everything is built against the engine-agnostic core traits. Existing APIs used: `ArchiveStore::get_range` / `iter_logs`, `StateStore::iter_entities` / `read_cursor`, `ArchiveWriter::apply` / `write_log`, `StateWriter::write_entity` / `apply_utxoset` / `set_cursor`, `IndexStore::initialize_schema`, `index_delta_from_utxo_delta`, `seed_wal_from_state`, `CardanoConfig.stop_epoch`. Missing APIs to add (thin wrappers over existing backend internals in both redb and fjall):
The Dolos-side crate keeps the name `snapshot` because that is this project's word for the artifact (`dolos snapshot`, `[snapshot]`, `tests/e2e/snapshot.rs`); `stele` is the protocol's word for the same thing. The planned `sign.rs` (Ed25519 detached signatures, trusted keys, k-of-n) has not been built: signing stays specified above and unimplemented, and nothing else in this section is aspirational.

1. `StateStore::iter_utxos()` — full UTxO-set iteration (export + live-UTxO index rebuild).
2. `IndexStore` iteration of archive tag/exact records by epoch range (export).
3. `IndexWriter::append_prehashed(records)` — direct insertion of pre-hashed records (restore).
Everything is built against the engine-agnostic core traits (`ArchiveStore`, `StateStore`, `IndexStore` and their writers, `seed_wal_from_state`, `CardanoConfig.stop_epoch`). The store APIs the initial design listed as missing — `StateStore::iter_utxos()`, epoch-ranged iteration of archive tag/exact records, `IndexWriter::append_prehashed` — have all since landed in `dolos-core` and its backends.

### CLI and configuration

- `dolos snapshot publish [--repo oci://…] [--output-dir DIR] [--epochs N..M] [--dry-run]` — export layers; `--output-dir` writes blobs + inscription to disk, `--repo` pushes with blob-skip and moves tags.
- `dolos snapshot digest` — compute and print the canonical inscription and its sha256 from local stores (what independent verifiers run and sign).
- `dolos snapshot verify | sign --key FILE | inspect`.
- `dolos snapshot verify | inspect`; `sign --key FILE` belongs to the signature
phase, which is specified and not yet built ("Code layout" above).
- `dolos bootstrap snapshot` gains source-scheme dispatch: `oci://` → new path; https template / `--file` → legacy tarball, unchanged. `--point epoch-E|latest`; existing `--continue` drives resume; `sync.max_history` bounds how much history is fetched (subsumes the old `full`/`ledger` variants).

```toml
Expand Down
7 changes: 7 additions & 0 deletions crates/snapshot/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,13 @@ pub(crate) use stelae_driver::scope_key;
/// [`export::publish`] and [`restore::restore_dir`] hold for the transports.
pub use stelae::progress;

/// The protocol surface a host reaches through this crate rather than by
/// naming `stelae` itself: only this crate depends on the stelae crates, so
/// there is one pin point and one place to absorb upstream churn. These cover
/// what a test or a binary opens a stele with; anything else a consumer needs
/// joins the list rather than justifying a second dependency edge.
pub use stelae::{dir, inscription, transport, SteleReader};

use dolos_cardano::model::{
AccountEpochLog, AccountState, AssetState, DRepState, DatumState, EpochState, EraSummary,
FixedNamespace, GovState, PendingMirState, PendingRewardState, PoolState, ProposalState,
Expand Down
Loading
Loading