diff --git a/docs/plans/2026-08-10-graceful-content-lock-deletion.md b/docs/plans/2026-08-10-graceful-content-lock-deletion.md index 0e3519f..0b57a08 100644 --- a/docs/plans/2026-08-10-graceful-content-lock-deletion.md +++ b/docs/plans/2026-08-10-graceful-content-lock-deletion.md @@ -14,7 +14,7 @@ ## Status and provenance -- Plan status: **accepted product design; Tasks 1–5 committed and Task 6 implemented pending commit; Tasks 7–10 remain**. +- Plan status: **accepted product design; Tasks 1, 2, 4, 5, and 6 committed; the omitted Task 3 prerequisite is implemented pending commit; Tasks 7–10 remain**. - Repository inspected: `/home/u/Projects/Synonym/Pubky/locks-public`. - Planning base when written: clean `master` at `ba49a77`. - There has been no production deployment. New persistence may require a clean pre-production database; no historical backfill is required. @@ -153,11 +153,12 @@ Both drain endpoints return `200` with the same closed aggregate body: "status": "active", "accepted_count": 0, "terminal_count": 0, - "cancellation_enqueued_count": 0 + "cancellation_enqueued_count": 0, + "cleanup_token": "<43-character-unpadded-base64url>" } ``` -`status` is exactly `active` or `completed`. The response contains no drain ID, replay flag, Bundle ID, reader, Payment Request ID, address, payment reference, or raw error. Exact replay returns the same aggregate body. +`status` is exactly `active` or `completed`. The response contains no drain ID, replay flag, Bundle ID, reader, Payment Request ID, address, payment reference, or raw error. Replay preserves the frozen drain identity, cancellation count, and cleanup token while returning its latest monotonic aggregate progress. ### Per-Bundle status @@ -200,6 +201,8 @@ The canonical persisted `request_state` is one of these exact closed snake-case `expired` is returned when the invoice has a durable `payment_expired_at`; otherwise the persisted observation state maps one-to-one to `undetected`, `detected`, or `confirmed`. `confirmations` and `amount_matched` remain orthogonal factual fields. +Locks maps `rejected`, `canceled`, and `proposal_expired` requests to `VerificationTaskStatus::Expired`. An `accepted` request whose `payment_state` is `expired` also maps to `Expired`. These terminal outcomes carry no failure message and never map to `Failed`. + Drain classification uses the persisted state without inference from invoice delivery or Bitcoin observation: - `accepted` is accepted and blocking; @@ -420,7 +423,12 @@ cargo test --workspace --no-run - Modify: `locks-server/src/paykit_http_client.rs` - Modify: `locks-server/src/app_state/mod.rs` - Create: `locks-service/src/application/ports/payment_drain.rs` +- Create: `locks-service/src/application/ports/payment_drain_repository.rs` - Create: `locks-service/src/application/use_cases/drain_lock_payments.rs` +- Create: `locks-service/src/infrastructure/postgres/payment_drains.rs` +- Create: `locks-service/migrations/0015_content_lock_payment_drains.sql` +- Modify: `locks-service/src/infrastructure/postgres/content_lock_deletions.rs` +- Modify: `locks-service/src/infrastructure/postgres/verification_task_claims.rs` - Test: `locks-server/src/paykit_http_client.rs` - Test: deletion use-case tests and HTTP integration fixtures @@ -428,6 +436,8 @@ cargo test --workspace --no-run **RED:** Test exact signed JSON, no `minimum_confirmations` leak, aggregate redaction, local application of confirmations, canceled/rejected/expired transitions, timely matched confirmation continuation, and retryable transport errors. +**GREEN:** Freeze Paykit obligation identity, criterion, cutoff status, and authoritative invoice window in the deletion snapshot; persist the opaque drain token and aggregate under the deletion claim fence. Paykit aggregate progress is monotonic: `accepted_count` may only decrease to zero, `terminal_count` may only increase by the same amount, and `cancellation_enqueued_count` plus the opaque cleanup token remain immutable. `completed` requires `accepted_count == 0` and cannot regress to `active`. Locks persists each newer aggregate under the live deletion lease, but still requires every frozen local obligation to become terminal before phase advancement; aggregate completion never bypasses Locks-local confirmation or reorg reconciliation. Exclude ordinary verification workers for every surviving deletion snapshot. Immediately before external entitlement storage, an ordinary claimed worker durably marks entitlement publication under its exact lease. Deletion admission locks and owns every matching task row before snapshot/reset: a committed publication marker blocks deletion, while committed deletion ownership blocks publication and every ordinary claim/retry/terminal write. Ambiguous entitlement publication retains the marker until an ordinary retry reconciles an equivalent entitlement and terminalizes the task. Publish entitlements before terminalizing their task and reconcile only an equivalent existing entitlement. Compose the client and repository in `AppState`. Task 9 supplies the queue polling/supervision that invokes this phase use case. + **Suggested commit:** `feat(paykit): drain deleting lock payments` ### Task 8: Implement final credential/read draining diff --git a/locks-e2e/tests/creator_publishing_http.rs b/locks-e2e/tests/creator_publishing_http.rs index 7a30a35..95a0130 100644 --- a/locks-e2e/tests/creator_publishing_http.rs +++ b/locks-e2e/tests/creator_publishing_http.rs @@ -827,6 +827,7 @@ impl FakePaykitServer { Some(json!({ "bundle_id": BUNDLE_ID, "lock_resource": lock_resource, + "payment_in": 24, "reader": creator().to_string(), })) ); @@ -872,14 +873,20 @@ async fn fake_invoice_handler( State(state): State>>, headers: HeaderMap, body: Bytes, -) -> StatusCode { +) -> (StatusCode, Json) { let mut state = state.lock().await; state.invoice_count += 1; state.invoice_body = Some(serde_json::from_slice(&body).unwrap()); state.invoice_signature = headers .get("X-Paykit-Signature") .map(|value| value.to_str().unwrap().to_owned()); - StatusCode::CREATED + ( + StatusCode::CREATED, + Json(json!({ + "invoice_created_at": "2026-08-12T10:00:00Z", + "payment_deadline": "2026-08-13T10:00:00Z", + })), + ) } async fn fake_status_handler( diff --git a/locks-e2e/tests/postgres_runtime.rs b/locks-e2e/tests/postgres_runtime.rs index 26fcd04..af93f1e 100644 --- a/locks-e2e/tests/postgres_runtime.rs +++ b/locks-e2e/tests/postgres_runtime.rs @@ -10,7 +10,7 @@ use axum::extract::ConnectInfo; use axum::http::{Request, StatusCode, header}; use axum::routing::post; use locks_core::ids::{ - BundleId, CreatorPubky, GuardedResourceHash, LockServerPubky, PubkyLockResource, + BundleId, CreatorPubky, GuardedResourceHash, LockServerPubky, PubkyLockResource, TaskId, }; use locks_core::lock_policy::{ AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, Criterion, GuardedResource, LockLogic, @@ -29,16 +29,19 @@ use locks_server::worker::{VerificationWorker, WorkerTick}; use locks_service::application::models::{ AccessCredential, AccessCredentialLookupKey, ContentLockDeletionJob, ContentLockOwnershipStatus, CreatorAuthorityAuthKind, CreatorAuthorityRecord, - CreatorAuthoritySecret, VerificationTaskStatus, + CreatorAuthoritySecret, VerificationTaskRecord, VerificationTaskStatus, +}; +use locks_service::application::ports::{ + ContentLockDeletionRepository, VerificationTaskRepository, }; -use locks_service::application::ports::ContentLockDeletionRepository; use locks_service::infrastructure::memory::{ content_locks::InMemoryContentLockRepository, entitlements::InMemoryEntitlementRepository, guarded_resources::InMemoryGuardedResourceRepository, lock_service_pointers::InMemoryLockServicePointerRepository, }; use locks_service::infrastructure::postgres::{ - CreatorAuthoritySecretCipher, PostgresContentLockDeletionRepository, run_migrations, + CreatorAuthoritySecretCipher, PostgresContentLockDeletionRepository, + PostgresVerificationTaskRepository, run_migrations, }; use serde_json::{Value, json}; use sqlx::postgres::PgPoolOptions; @@ -111,6 +114,61 @@ async fn postgres_runtime_state_survives_app_state_recreation() { database.cleanup().await; } +#[tokio::test] +async fn manual_completion_hides_legacy_paykit_admission_without_authoritative_window() { + let Some(database) = TestDatabase::create().await else { + return; + }; + let submitted = paykit_submission_for(&paykit_content_lock()); + let task = VerificationTaskRecord { + task_id: TaskId::from_str(&uuid::Uuid::new_v4().to_string()).unwrap(), + creator: submitted.pubky_lock_resource.creator().clone(), + submitted_proof_bundle: submitted.clone(), + status: VerificationTaskStatus::Pending, + submitted_at: datetime!(2026-08-12 06:00:00 UTC), + started_at: None, + completed_at: None, + failure_message: None, + }; + PostgresVerificationTaskRepository::new(database.pool().clone()) + .insert_verification_task(task.clone()) + .await + .unwrap(); + sqlx::query( + "INSERT INTO paykit_task_admissions + (verification_task_id, ready, ready_at) + VALUES ($1::uuid, TRUE, now())", + ) + .bind(task.task_id.to_string()) + .execute(database.pool()) + .await + .unwrap(); + + let response = router(app_state(database.pool().clone())) + .oneshot(json_request( + "POST", + "/verification-task-completions", + json!({ + "creator": submitted.pubky_lock_resource.creator(), + "bundle_id": submitted.bundle_id, + }), + )) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::NOT_FOUND); + assert_eq!( + response_json(response).await, + json!({ + "error": { + "code": "verification_task_not_found", + "message": "verification task not found" + } + }) + ); + + database.cleanup().await; +} + #[tokio::test] async fn postgres_runtime_readyz_returns_ready_without_leaking_runtime_details() { let Some(database) = TestDatabase::create().await else { @@ -275,9 +333,18 @@ async fn snapshotted_unready_paykit_replay_ignores_tombstoned_lock_and_reader_re let call = paykit_state.fetch_add(1, Ordering::SeqCst); async move { if call == 0 { - StatusCode::INTERNAL_SERVER_ERROR + ( + StatusCode::INTERNAL_SERVER_ERROR, + axum::Json(json!({ "error": "injected" })), + ) } else { - StatusCode::OK + ( + StatusCode::OK, + axum::Json(json!({ + "invoice_created_at": "2026-08-12T10:00:00Z", + "payment_deadline": "2026-08-13T10:00:00Z", + })), + ) } } }), diff --git a/locks-server/src/api/verification.rs b/locks-server/src/api/verification.rs index a701b31..d1c7d1e 100644 --- a/locks-server/src/api/verification.rs +++ b/locks-server/src/api/verification.rs @@ -20,7 +20,9 @@ use locks_service::application::use_cases::submit_proof_bundle::{ use locks_service::application::use_cases::validate_paykit_payment_submission::{ ValidatePaykitPaymentSubmissionRequest, ValidatePaykitPaymentSubmissionUseCase, }; -use locks_service::infrastructure::postgres::PostgresPaykitTaskAdmissionRepository; +use locks_service::infrastructure::postgres::{ + PaykitInvoiceWindow, PostgresPaykitTaskAdmissionRepository, +}; use locks_service::infrastructure::verifiers::registry::StaticCriterionVerifierRegistry; use crate::api::dtos::{ @@ -116,13 +118,20 @@ async fn maybe_prepare_paykit_submission( let admissions = PostgresPaykitTaskAdmissionRepository::new(pool.clone()); if let Some(admission) = admissions.find_existing(submitted).await? { if admission.requires_paykit { - create_paykit_invoice(state, &admission.task.submitted_proof_bundle).await?; - admissions.mark_ready(&admission.task).await?; + let invoice_window = create_paykit_invoice( + state, + &admission.task.submitted_proof_bundle, + admission.payment_in, + ) + .await?; + admissions + .mark_ready(&admission.task, invoice_window) + .await?; } return Ok(Some(admission.task.into())); } } - ValidatePaykitPaymentSubmissionUseCase::new(state.content_locks().as_ref()) + let validated = ValidatePaykitPaymentSubmissionUseCase::new(state.content_locks().as_ref()) .execute(ValidatePaykitPaymentSubmissionRequest { submitted_proof_bundle: submitted.clone(), }) @@ -140,10 +149,17 @@ async fn maybe_prepare_paykit_submission( if let Some(pool) = state.postgres_pool() { let task = submit_use_case.prepare_task(submitted.clone()).await?; let admissions = PostgresPaykitTaskAdmissionRepository::new(pool.clone()); - let admission = admissions.reserve(task).await?; + let admission = admissions.reserve(task, validated.payment_in).await?; if admission.requires_paykit { - create_paykit_invoice(state, &admission.task.submitted_proof_bundle).await?; - admissions.mark_ready(&admission.task).await?; + let invoice_window = create_paykit_invoice( + state, + &admission.task.submitted_proof_bundle, + admission.payment_in, + ) + .await?; + admissions + .mark_ready(&admission.task, invoice_window) + .await?; } return Ok(Some(admission.task.into())); } @@ -160,6 +176,7 @@ async fn maybe_prepare_paykit_submission( .create_invoice(&PaykitInvoiceRequest { bundle_id: submitted.bundle_id.to_string(), lock_resource: submitted.pubky_lock_resource.to_string(), + payment_in: validated.payment_in, reader: reader.to_string(), }) .await @@ -170,14 +187,15 @@ async fn maybe_prepare_paykit_submission( async fn create_paykit_invoice( state: &AppState, submitted: &SubmittedProofBundle, -) -> Result<(), ApiError> { + payment_in: u64, +) -> Result { let reader = submitted.reader_public_key.as_ref().ok_or_else(|| { ApiError::new( ApiErrorCode::InvalidRequest, "paykit-payment requires reader_public_key", ) })?; - state + let response = state .paykit_http_client() .ok_or_else(|| { ApiError::new( @@ -188,10 +206,15 @@ async fn create_paykit_invoice( .create_invoice(&PaykitInvoiceRequest { bundle_id: submitted.bundle_id.to_string(), lock_resource: submitted.pubky_lock_resource.to_string(), + payment_in, reader: reader.to_string(), }) .await - .map_err(map_paykit_invoice_error) + .map_err(map_paykit_invoice_error)?; + Ok(PaykitInvoiceWindow { + invoice_created_at: response.invoice_created_at, + payment_deadline: response.payment_deadline, + }) } fn map_paykit_invoice_error(error: PaykitClientError) -> ApiError { diff --git a/locks-server/src/app_state/mod.rs b/locks-server/src/app_state/mod.rs index dd87d52..906bf4a 100644 --- a/locks-server/src/app_state/mod.rs +++ b/locks-server/src/app_state/mod.rs @@ -21,8 +21,8 @@ use locks_service::{ ContentLockOwnershipRepository, ContentLockRepository, CreatorAuthorityManager, CreatorAuthorityStore, CreatorConnectFlowStore, EntitlementRepository, FrontendSessionCodeStore, FrontendSessionStore, GuardedResourceRepository, - LegacyCreatorConnectFlowClient, LockServicePointerRepository, VerificationTaskClaimer, - VerificationTaskRepository, + LegacyCreatorConnectFlowClient, LockServicePointerRepository, PaymentDrainClient, + PaymentDrainRepository, VerificationTaskClaimer, VerificationTaskRepository, }, }, infrastructure::{ @@ -31,6 +31,7 @@ use locks_service::{ content_lock_deletions::InMemoryContentLockDeletionRepository, content_lock_ownership::InMemoryContentLockOwnershipRepository, verification_task_claims::InMemoryVerificationTaskClaimer, + verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence, verification_tasks::InMemoryVerificationTaskRepository, }, postgres::{ @@ -38,7 +39,8 @@ use locks_service::{ PostgresContentLockDeletionRepository, PostgresContentLockOwnershipRepository, PostgresCreatorAuthorityStore, PostgresCreatorConnectFlowStore, PostgresFrontendSessionCodeStore, PostgresFrontendSessionStore, - PostgresVerificationTaskClaimer, PostgresVerificationTaskRepository, + PostgresPaymentDrainRepository, PostgresVerificationTaskClaimer, + PostgresVerificationTaskRepository, }, pubky::{ AuthorizingPubkyHomeserverStorageClient, LegacyCookieCreatorAuthorityManager, @@ -153,6 +155,7 @@ pub struct AppState { lock_service_pointers: Arc, content_lock_ownership: Arc, content_lock_deletions: Arc, + payment_drains: Option>, verification_tasks: Arc, verification_task_claimer: Arc, entitlements: Arc, @@ -175,6 +178,7 @@ pub struct AppState { verification_submission_rate_limiter: Arc, reader_pubky_resolver: Arc, paykit_http_client: Option>, + payment_drain_client: Option>, } impl std::fmt::Debug for AppState { @@ -208,12 +212,18 @@ impl std::fmt::Debug for AppState { impl AppState { pub fn new_empty_in_memory(config: LockServerRuntimeConfig) -> Self { - let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::new()); - let verification_task_claimer = - Arc::new(InMemoryVerificationTaskClaimer::with_task_repository( + let verification_task_deletion_fence = + Arc::new(InMemoryVerificationTaskDeletionFence::new()); + let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::with_deletion_fence( + Arc::clone(&verification_task_deletion_fence), + )); + let verification_task_claimer = Arc::new( + InMemoryVerificationTaskClaimer::with_task_repository_and_deletion_fence( vec![], verification_tasks.clone(), - )); + Arc::clone(&verification_task_deletion_fence), + ), + ); let access_credentials = Arc::new(InMemoryAccessCredentialStore::new()); let creator_authority_store = InMemoryCreatorAuthorityStore::new(); let creator_authorities = Arc::new(creator_authority_store.clone()); @@ -231,6 +241,11 @@ impl AppState { let private_runtime = PrivateRuntimeAdapters { content_lock_ownership: Arc::new(InMemoryContentLockOwnershipRepository::new()), + content_lock_deletions: Arc::new( + InMemoryContentLockDeletionRepository::with_verification_task_fence( + verification_task_deletion_fence, + ), + ), verification_tasks, verification_task_claimer, access_credentials, @@ -258,12 +273,18 @@ impl AppState { lock_service_pointers: Arc, entitlements: Arc, ) -> Self { - let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::new()); - let verification_task_claimer = - Arc::new(InMemoryVerificationTaskClaimer::with_task_repository( + let verification_task_deletion_fence = + Arc::new(InMemoryVerificationTaskDeletionFence::new()); + let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::with_deletion_fence( + Arc::clone(&verification_task_deletion_fence), + )); + let verification_task_claimer = Arc::new( + InMemoryVerificationTaskClaimer::with_task_repository_and_deletion_fence( vec![], verification_tasks.clone(), - )); + Arc::clone(&verification_task_deletion_fence), + ), + ); let access_credentials = Arc::new(InMemoryAccessCredentialStore::new()); let creator_authority_store = InMemoryCreatorAuthorityStore::new(); let creator_authorities = Arc::new(creator_authority_store.clone()); @@ -280,6 +301,11 @@ impl AppState { let private_runtime = PrivateRuntimeAdapters { content_lock_ownership: Arc::new(InMemoryContentLockOwnershipRepository::new()), + content_lock_deletions: Arc::new( + InMemoryContentLockDeletionRepository::with_verification_task_fence( + verification_task_deletion_fence, + ), + ), verification_tasks, verification_task_claimer, access_credentials, @@ -307,12 +333,18 @@ impl AppState { where S: PubkyHomeserverStorageClient + Clone + 'static, { - let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::new()); - let verification_task_claimer = - Arc::new(InMemoryVerificationTaskClaimer::with_task_repository( + let verification_task_deletion_fence = + Arc::new(InMemoryVerificationTaskDeletionFence::new()); + let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::with_deletion_fence( + Arc::clone(&verification_task_deletion_fence), + )); + let verification_task_claimer = Arc::new( + InMemoryVerificationTaskClaimer::with_task_repository_and_deletion_fence( vec![], verification_tasks.clone(), - )); + Arc::clone(&verification_task_deletion_fence), + ), + ); let access_credentials = Arc::new(InMemoryAccessCredentialStore::new()); let creator_authority_store = InMemoryCreatorAuthorityStore::new(); let creator_authorities = Arc::new(creator_authority_store.clone()); @@ -346,6 +378,11 @@ impl AppState { let private_runtime = PrivateRuntimeAdapters { content_lock_ownership: Arc::new(InMemoryContentLockOwnershipRepository::new()), + content_lock_deletions: Arc::new( + InMemoryContentLockDeletionRepository::with_verification_task_fence( + verification_task_deletion_fence, + ), + ), verification_tasks, verification_task_claimer, access_credentials, @@ -403,6 +440,9 @@ impl AppState { content_lock_ownership: Arc::new(PostgresContentLockOwnershipRepository::new( pool.clone(), )), + content_lock_deletions: Arc::new(PostgresContentLockDeletionRepository::new( + pool.clone(), + )), verification_tasks, verification_task_claimer, access_credentials, @@ -467,6 +507,9 @@ impl AppState { content_lock_ownership: Arc::new(PostgresContentLockOwnershipRepository::new( pool.clone(), )), + content_lock_deletions: Arc::new(PostgresContentLockDeletionRepository::new( + pool.clone(), + )), verification_tasks, verification_task_claimer, access_credentials, @@ -517,11 +560,12 @@ impl AppState { )) }) }); - let content_lock_deletions: Arc = - match postgres_pool.as_ref() { - Some(pool) => Arc::new(PostgresContentLockDeletionRepository::new(pool.clone())), - None => Arc::new(InMemoryContentLockDeletionRepository::new()), - }; + let payment_drains: Option> = postgres_pool + .as_ref() + .map(|pool| Arc::new(PostgresPaymentDrainRepository::new(pool.clone())) as Arc<_>); + let payment_drain_client = paykit_http_client + .as_ref() + .map(|client| Arc::clone(client) as Arc); Self { config, @@ -531,7 +575,8 @@ impl AppState { guarded_resources: creator_repositories.guarded_resources, lock_service_pointers: creator_repositories.lock_service_pointers, content_lock_ownership: private_runtime.content_lock_ownership, - content_lock_deletions, + content_lock_deletions: private_runtime.content_lock_deletions, + payment_drains, verification_tasks: private_runtime.verification_tasks, verification_task_claimer: private_runtime.verification_task_claimer, entitlements: creator_repositories.entitlements, @@ -554,6 +599,7 @@ impl AppState { verification_submission_rate_limiter, reader_pubky_resolver, paykit_http_client, + payment_drain_client, } } @@ -585,6 +631,14 @@ impl AppState { &self.content_lock_deletions } + pub fn payment_drains(&self) -> Option<&Arc> { + self.payment_drains.as_ref() + } + + pub fn payment_drain_client(&self) -> Option<&Arc> { + self.payment_drain_client.as_ref() + } + pub fn lock_service_pointers(&self) -> &Arc { &self.lock_service_pointers } diff --git a/locks-server/src/app_state/private_runtime.rs b/locks-server/src/app_state/private_runtime.rs index af87ed1..b0ce136 100644 --- a/locks-server/src/app_state/private_runtime.rs +++ b/locks-server/src/app_state/private_runtime.rs @@ -11,10 +11,10 @@ use locks_service::application::{ PendingCreatorConnectFlowRecord, }, ports::{ - AccessCredentialStore, ContentLockOwnershipRepository, CreatorAuthorityManager, - CreatorAuthorityStore, CreatorConnectFlowStore, FrontendSessionCodeStore, - FrontendSessionStore, LegacyCreatorConnectFlowClient, VerificationTaskClaimer, - VerificationTaskRepository, + AccessCredentialStore, ContentLockDeletionRepository, ContentLockOwnershipRepository, + CreatorAuthorityManager, CreatorAuthorityStore, CreatorConnectFlowStore, + FrontendSessionCodeStore, FrontendSessionStore, LegacyCreatorConnectFlowClient, + VerificationTaskClaimer, VerificationTaskRepository, }, }; use time::OffsetDateTime; @@ -23,6 +23,7 @@ use tokio::sync::RwLock; #[derive(Clone)] pub(super) struct PrivateRuntimeAdapters { pub(super) content_lock_ownership: Arc, + pub(super) content_lock_deletions: Arc, pub(super) verification_tasks: Arc, pub(super) verification_task_claimer: Arc, pub(super) access_credentials: Arc, diff --git a/locks-server/src/paykit_http_client.rs b/locks-server/src/paykit_http_client.rs index a5c6be8..0744804 100644 --- a/locks-server/src/paykit_http_client.rs +++ b/locks-server/src/paykit_http_client.rs @@ -4,6 +4,10 @@ use async_trait::async_trait; use base64::Engine; use base64::engine::general_purpose::URL_SAFE_NO_PAD; use locks_core::ids::{BundleId, CreatorPubky}; +use locks_service::application::ports::{ + PaymentDrainCleanupToken, PaymentDrainClient, PaymentDrainClientError, PaymentDrainStatus, + PaymentDrainSummary, PaymentRequestState, PaymentRequestStatus, PaymentState, +}; use locks_service::infrastructure::verifiers::paykit_payment::{ PaykitPaymentStatus, PaykitPaymentStatusClient, PaykitPaymentStatusError, PaykitPaymentStatusKind, @@ -11,6 +15,8 @@ use locks_service::infrastructure::verifiers::paykit_payment::{ use pubky_common::crypto::Keypair; use reqwest::StatusCode; use serde::{Deserialize, Serialize}; +use time::OffsetDateTime; +use time::format_description::well_known::Rfc3339; use url::Url; use crate::config::{ @@ -41,6 +47,8 @@ pub enum PaykitClientError { }, #[error("Paykit status response was invalid: {0}")] InvalidStatusResponse(reqwest::Error), + #[error("Paykit invoice response was invalid: {0}")] + InvalidInvoiceResponse(String), } #[derive(Debug, Clone)] @@ -54,9 +62,49 @@ pub struct PaykitHttpClient { pub struct PaykitInvoiceRequest { pub bundle_id: String, pub lock_resource: String, + pub payment_in: u64, pub reader: String, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PaykitInvoiceResponse { + pub invoice_created_at: OffsetDateTime, + pub payment_deadline: OffsetDateTime, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct PaykitInvoiceResponseBody { + invoice_created_at: String, + payment_deadline: String, +} + +#[derive(Debug, Serialize)] +struct PaymentDrainRequest<'a> { + lock_resource: &'a str, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct PaymentDrainResponseBody { + status: String, + accepted_count: u64, + terminal_count: u64, + cancellation_enqueued_count: u64, + cleanup_token: String, +} + +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct PaymentRequestStatusResponseBody { + request_state: String, + payment_state: String, + invoice_created_at: String, + payment_deadline: String, + confirmations: u32, + amount_matched: bool, +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize)] pub struct PaykitStatusRequest { pub creator: String, @@ -109,17 +157,33 @@ impl PaykitHttpClient { pub async fn create_invoice( &self, request: &PaykitInvoiceRequest, - ) -> Result<(), PaykitClientError> { + ) -> Result { let response = self.signed_post("invoices", request).await?; - if response.status().is_success() { - Ok(()) - } else { - Err(PaykitClientError::NonSuccess { + if !response.status().is_success() { + return Err(PaykitClientError::NonSuccess { operation: "invoice creation", status: response.status(), - }) + }); } + + let body = response + .json::() + .await + .map_err(|error| PaykitClientError::InvalidInvoiceResponse(error.to_string()))?; + let invoice_created_at = OffsetDateTime::parse(&body.invoice_created_at, &Rfc3339) + .map_err(|error| PaykitClientError::InvalidInvoiceResponse(error.to_string()))?; + let payment_deadline = OffsetDateTime::parse(&body.payment_deadline, &Rfc3339) + .map_err(|error| PaykitClientError::InvalidInvoiceResponse(error.to_string()))?; + if payment_deadline < invoice_created_at { + return Err(PaykitClientError::InvalidInvoiceResponse( + "payment_deadline precedes invoice_created_at".to_owned(), + )); + } + Ok(PaykitInvoiceResponse { + invoice_created_at, + payment_deadline, + }) } pub async fn transaction_status( @@ -141,6 +205,97 @@ impl PaykitHttpClient { .map_err(PaykitClientError::InvalidStatusResponse) } + pub async fn start_payment_drain( + &self, + lock_resource: &str, + ) -> Result { + self.payment_drain("payment-request-drains", lock_resource) + .await? + .ok_or(PaymentDrainClientError::NotFound) + } + + pub async fn lookup_payment_drain( + &self, + lock_resource: &str, + ) -> Result, PaymentDrainClientError> { + self.payment_drain("payment-request-drain-lookups", lock_resource) + .await + } + + pub async fn payment_request_status( + &self, + creator: &str, + bundle_id: &str, + ) -> Result, PaymentDrainClientError> { + let response = self + .signed_post( + "payment-requests/status", + &PaykitStatusRequest { + creator: creator.to_owned(), + bundle_id: bundle_id.to_owned(), + }, + ) + .await + .map_err(|_| PaymentDrainClientError::Transport)?; + if response.status() == StatusCode::NOT_FOUND { + return Ok(None); + } + map_drain_error(response.status())?; + let body = response + .json::() + .await + .map_err(|_| PaymentDrainClientError::MalformedSuccess)?; + let invoice_created_at = OffsetDateTime::parse(&body.invoice_created_at, &Rfc3339) + .map_err(|_| PaymentDrainClientError::MalformedSuccess)?; + let payment_deadline = OffsetDateTime::parse(&body.payment_deadline, &Rfc3339) + .map_err(|_| PaymentDrainClientError::MalformedSuccess)?; + if payment_deadline < invoice_created_at { + return Err(PaymentDrainClientError::MalformedSuccess); + } + Ok(Some(PaymentRequestStatus { + request_state: PaymentRequestState::parse(&body.request_state) + .ok_or(PaymentDrainClientError::MalformedSuccess)?, + payment_state: PaymentState::parse(&body.payment_state) + .ok_or(PaymentDrainClientError::MalformedSuccess)?, + invoice_created_at, + payment_deadline, + confirmations: body.confirmations, + amount_matched: body.amount_matched, + })) + } + + async fn payment_drain( + &self, + path: &str, + lock_resource: &str, + ) -> Result, PaymentDrainClientError> { + let response = self + .signed_post(path, &PaymentDrainRequest { lock_resource }) + .await + .map_err(|_| PaymentDrainClientError::Transport)?; + if response.status() == StatusCode::NOT_FOUND { + return Ok(None); + } + map_drain_error(response.status())?; + let body = response + .json::() + .await + .map_err(|_| PaymentDrainClientError::MalformedSuccess)?; + let status = PaymentDrainStatus::parse(&body.status) + .ok_or(PaymentDrainClientError::MalformedSuccess)?; + if (status == PaymentDrainStatus::Completed) != (body.accepted_count == 0) { + return Err(PaymentDrainClientError::MalformedSuccess); + } + Ok(Some(PaymentDrainSummary { + status, + accepted_count: body.accepted_count, + terminal_count: body.terminal_count, + cancellation_enqueued_count: body.cancellation_enqueued_count, + cleanup_token: PaymentDrainCleanupToken::parse(&body.cleanup_token) + .ok_or(PaymentDrainClientError::MalformedSuccess)?, + })) + } + fn endpoint(&self, path: &str) -> Url { let mut endpoint = self.server_url.clone(); endpoint.set_query(None); @@ -174,6 +329,15 @@ impl PaykitHttpClient { } } +fn map_drain_error(status: StatusCode) -> Result<(), PaymentDrainClientError> { + match status { + status if status.is_success() => Ok(()), + StatusCode::CONFLICT => Err(PaymentDrainClientError::Conflict), + status if status.is_server_error() => Err(PaymentDrainClientError::Server), + _ => Err(PaymentDrainClientError::Server), + } +} + fn bounded_http_client( connect_timeout: Duration, request_timeout: Duration, @@ -185,6 +349,32 @@ fn bounded_http_client( .map_err(PaykitClientError::Http) } +#[async_trait] +impl PaymentDrainClient for PaykitHttpClient { + async fn start_payment_drain( + &self, + lock_resource: &locks_core::ids::PubkyLockResource, + ) -> Result { + PaykitHttpClient::start_payment_drain(self, &lock_resource.to_string()).await + } + + async fn lookup_payment_drain( + &self, + lock_resource: &locks_core::ids::PubkyLockResource, + ) -> Result, PaymentDrainClientError> { + PaykitHttpClient::lookup_payment_drain(self, &lock_resource.to_string()).await + } + + async fn payment_request_status( + &self, + creator: &CreatorPubky, + bundle_id: &BundleId, + ) -> Result, PaymentDrainClientError> { + PaykitHttpClient::payment_request_status(self, &creator.to_string(), &bundle_id.to_string()) + .await + } +} + #[async_trait] impl PaykitPaymentStatusClient for PaykitHttpClient { async fn transaction_status( @@ -272,7 +462,7 @@ mod tests { assert_eq!( String::from_utf8(body).unwrap(), format!( - "{{\"bundle_id\":\"{BUNDLE_ID}\",\"lock_resource\":\"{LOCK_RESOURCE}\",\"reader\":\"{READER}\"}}" + "{{\"bundle_id\":\"{BUNDLE_ID}\",\"lock_resource\":\"{LOCK_RESOURCE}\",\"payment_in\":24,\"reader\":\"{READER}\"}}" ) ); } @@ -365,7 +555,16 @@ mod tests { let expected_body = canonical_body_bytes(&invoice_request()).unwrap(); let expected_signature = sign_body(&keypair, &expected_body); - client.create_invoice(&invoice_request()).await.unwrap(); + let response = client.create_invoice(&invoice_request()).await.unwrap(); + + assert_eq!( + response.invoice_created_at, + time::macros::datetime!(2026-08-12 10:00:00 UTC) + ); + assert_eq!( + response.payment_deadline, + time::macros::datetime!(2026-08-13 10:00:00 UTC) + ); let request = captured.single(); assert_eq!(request.path, "/invoices"); @@ -407,6 +606,172 @@ mod tests { assert_eq!(request.signature, Some(expected_signature)); } + #[tokio::test] + async fn payment_drain_calls_use_exact_signed_bodies_and_closed_responses() { + let captured = CapturedRequests::default(); + let server_url = spawn_test_server(captured.clone()).await; + let client = PaykitHttpClient::from_parts( + &server_url, + reqwest::Client::new(), + Keypair::from_secret(&[9_u8; 32]), + ) + .unwrap(); + + let started = client.start_payment_drain(LOCK_RESOURCE).await.unwrap(); + assert_eq!(started.status, PaymentDrainStatus::Active); + assert_eq!(started.accepted_count, 1); + assert_eq!( + started.cleanup_token.as_str(), + "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + ); + let looked_up = client + .lookup_payment_drain(LOCK_RESOURCE) + .await + .unwrap() + .unwrap(); + assert_eq!(looked_up, started); + let status = client + .payment_request_status(CREATOR, BUNDLE_ID) + .await + .unwrap() + .unwrap(); + assert_eq!(status.request_state, PaymentRequestState::Accepted); + assert_eq!(status.payment_state, PaymentState::Detected); + assert!(status.amount_matched); + + let requests = captured.all(); + assert_eq!(requests.len(), 3); + assert_eq!(requests[0].path, "/payment-request-drains"); + assert_eq!(requests[1].path, "/payment-request-drain-lookups"); + assert_eq!(requests[2].path, "/payment-requests/status"); + assert_eq!( + String::from_utf8(requests[0].body.clone()).unwrap(), + format!("{{\"lock_resource\":\"{LOCK_RESOURCE}\"}}") + ); + assert_eq!(requests[0].body, requests[1].body); + assert_eq!( + String::from_utf8(requests[2].body.clone()).unwrap(), + format!("{{\"bundle_id\":\"{BUNDLE_ID}\",\"creator\":\"{CREATOR}\"}}") + ); + assert!(requests.iter().all(|request| request.signature.is_some())); + assert!(requests.iter().all(|request| { + !String::from_utf8_lossy(&request.body).contains("minimum_confirmations") + })); + } + + #[tokio::test] + async fn payment_drain_http_errors_preserve_not_found_conflict_and_server_failure() { + for (status, expected) in [ + ( + axum::http::StatusCode::NOT_FOUND, + PaymentDrainClientError::NotFound, + ), + ( + axum::http::StatusCode::CONFLICT, + PaymentDrainClientError::Conflict, + ), + ( + axum::http::StatusCode::SERVICE_UNAVAILABLE, + PaymentDrainClientError::Server, + ), + ] { + let server_url = spawn_configured_drain_server(status, "{}").await; + let client = PaykitHttpClient::from_parts( + &server_url, + reqwest::Client::new(), + Keypair::from_secret(&[9_u8; 32]), + ) + .unwrap(); + assert_eq!( + client.start_payment_drain(LOCK_RESOURCE).await, + Err(expected) + ); + } + } + + #[tokio::test] + async fn payment_drain_rejects_malformed_unknown_or_inconsistent_success_bodies() { + for body in [ + "not-json", + r#"{"status":"active","accepted_count":0,"terminal_count":0,"cancellation_enqueued_count":0}"#, + r#"{"status":"complete","accepted_count":0,"terminal_count":0,"cancellation_enqueued_count":0,"cleanup_token":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}"#, + r#"{"status":"active","accepted_count":0,"terminal_count":0,"cancellation_enqueued_count":0,"cleanup_token":"short"}"#, + r#"{"status":"active","accepted_count":0,"terminal_count":0,"cancellation_enqueued_count":0,"cleanup_token":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA","extra":true}"#, + r#"{"status":"completed","accepted_count":1,"terminal_count":0,"cancellation_enqueued_count":0,"cleanup_token":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}"#, + ] { + let server_url = spawn_configured_drain_server(axum::http::StatusCode::OK, body).await; + let client = PaykitHttpClient::from_parts( + &server_url, + reqwest::Client::new(), + Keypair::from_secret(&[9_u8; 32]), + ) + .unwrap(); + assert_eq!( + client.start_payment_drain(LOCK_RESOURCE).await, + Err(PaymentDrainClientError::MalformedSuccess) + ); + } + } + + #[tokio::test] + async fn payment_drain_accepts_completed_cancellation_only_aggregate() { + let server_url = spawn_configured_drain_server( + axum::http::StatusCode::OK, + r#"{"status":"completed","accepted_count":0,"terminal_count":0,"cancellation_enqueued_count":1,"cleanup_token":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}"#, + ) + .await; + let client = PaykitHttpClient::from_parts( + &server_url, + reqwest::Client::new(), + Keypair::from_secret(&[9_u8; 32]), + ) + .unwrap(); + + let summary = client.start_payment_drain(LOCK_RESOURCE).await.unwrap(); + assert_eq!(summary.status, PaymentDrainStatus::Completed); + assert_eq!(summary.accepted_count, 0); + assert_eq!(summary.terminal_count, 0); + assert_eq!(summary.cancellation_enqueued_count, 1); + } + + #[tokio::test] + async fn payment_drain_timeout_is_transport_failure() { + let server_url = spawn_hanging_test_server().await; + let client = PaykitHttpClient::from_parts( + &server_url, + bounded_http_client(Duration::from_millis(10), Duration::from_millis(25)).unwrap(), + Keypair::from_secret(&[9_u8; 32]), + ) + .unwrap(); + assert_eq!( + client.start_payment_drain(LOCK_RESOURCE).await, + Err(PaymentDrainClientError::Transport) + ); + } + + #[tokio::test] + async fn payment_request_status_rejects_unknown_or_reversed_success_body() { + for body in [ + r#"{"request_state":"unknown","payment_state":"detected","invoice_created_at":"2026-08-12T10:00:00Z","payment_deadline":"2026-08-13T10:00:00Z","confirmations":0,"amount_matched":true}"#, + r#"{"request_state":"accepted","payment_state":"unknown","invoice_created_at":"2026-08-12T10:00:00Z","payment_deadline":"2026-08-13T10:00:00Z","confirmations":0,"amount_matched":true}"#, + r#"{"request_state":"accepted","payment_state":"detected","invoice_created_at":"2026-08-13T10:00:00Z","payment_deadline":"2026-08-12T10:00:00Z","confirmations":0,"amount_matched":true}"#, + ] { + let server_url = + spawn_configured_payment_request_status_server(axum::http::StatusCode::OK, body) + .await; + let client = PaykitHttpClient::from_parts( + &server_url, + reqwest::Client::new(), + Keypair::from_secret(&[9_u8; 32]), + ) + .unwrap(); + assert_eq!( + client.payment_request_status(CREATOR, BUNDLE_ID).await, + Err(PaymentDrainClientError::MalformedSuccess) + ); + } + } + #[tokio::test] async fn create_invoice_times_out_when_paykit_does_not_respond() { let server_url = spawn_hanging_test_server().await; @@ -443,6 +808,30 @@ mod tests { assert!(matches!(error, PaykitClientError::Http(error) if error.is_timeout())); } + #[tokio::test] + async fn create_invoice_rejects_malformed_or_inconsistent_success_body() { + for body in [ + r#"{"invoice_created_at":"2026-08-12T10:00:00Z"}"#, + r#"{"invoice_created_at":"not-a-time","payment_deadline":"2026-08-13T10:00:00Z"}"#, + r#"{"invoice_created_at":"2026-08-13T10:00:00Z","payment_deadline":"2026-08-12T10:00:00Z"}"#, + r#"{"invoice_created_at":"2026-08-12T10:00:00Z","payment_deadline":"2026-08-13T10:00:00Z","extra":true}"#, + ] { + let server_url = + spawn_configured_invoice_server(axum::http::StatusCode::OK, body).await; + let client = PaykitHttpClient::from_parts( + &server_url, + reqwest::Client::new(), + Keypair::from_secret(&[9_u8; 32]), + ) + .unwrap(); + + assert!(matches!( + client.create_invoice(&invoice_request()).await, + Err(PaykitClientError::InvalidInvoiceResponse(_)) + )); + } + } + #[tokio::test] async fn status_not_found_and_invalid_success_body_use_retryable_client_error() { for (status, body) in [ @@ -474,6 +863,7 @@ mod tests { bundle_id: BUNDLE_ID.to_owned(), lock_resource: LOCK_RESOURCE.to_owned(), reader: READER.to_owned(), + payment_in: 24, } } @@ -490,6 +880,10 @@ mod tests { assert_eq!(requests.len(), 1); requests[0].clone() } + + fn all(&self) -> Vec { + self.0.lock().unwrap().clone() + } } #[derive(Clone, Debug, PartialEq, Eq)] @@ -503,6 +897,15 @@ mod tests { let app = Router::new() .route("/invoices", post(capture_invoice)) .route("/transactions/status", post(capture_status)) + .route("/payment-request-drains", post(capture_payment_drain)) + .route( + "/payment-request-drain-lookups", + post(capture_payment_drain_lookup), + ) + .route( + "/payment-requests/status", + post(capture_payment_request_status), + ) .with_state(captured); let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); @@ -515,7 +918,23 @@ mod tests { async fn spawn_hanging_test_server() -> String { let app = Router::new() .route("/invoices", post(hang)) - .route("/transactions/status", post(hang)); + .route("/transactions/status", post(hang)) + .route("/payment-request-drains", post(hang)); + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + format!("http://{addr}") + } + + async fn spawn_configured_invoice_server( + status: axum::http::StatusCode, + body: &'static str, + ) -> String { + let app = Router::new() + .route("/invoices", post(configured_status)) + .with_state(ConfiguredStatusResponse { status, body }); let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); let addr = listener.local_addr().unwrap(); tokio::spawn(async move { @@ -545,6 +964,36 @@ mod tests { format!("http://{addr}") } + async fn spawn_configured_drain_server( + status: axum::http::StatusCode, + body: &'static str, + ) -> String { + let app = Router::new() + .route("/payment-request-drains", post(configured_status)) + .with_state(ConfiguredStatusResponse { status, body }); + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + format!("http://{addr}") + } + + async fn spawn_configured_payment_request_status_server( + status: axum::http::StatusCode, + body: &'static str, + ) -> String { + let app = Router::new() + .route("/payment-requests/status", post(configured_status)) + .with_state(ConfiguredStatusResponse { status, body }); + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { + axum::serve(listener, app).await.unwrap(); + }); + format!("http://{addr}") + } + async fn configured_status( State(response): State, ) -> impl IntoResponse { @@ -568,7 +1017,10 @@ mod tests { .map(|value| value.to_str().unwrap().to_owned()), body: body.to_vec(), }); - (axum::http::StatusCode::CREATED, "accepted") + Json(json!({ + "invoice_created_at": "2026-08-12T10:00:00Z", + "payment_deadline": "2026-08-13T10:00:00Z", + })) } async fn capture_status( @@ -589,4 +1041,58 @@ mod tests { "amount_matched": true, })) } + + async fn capture_payment_drain( + State(captured): State, + headers: HeaderMap, + body: Bytes, + ) -> impl IntoResponse { + capture_request(&captured, &headers, body, "/payment-request-drains"); + drain_response() + } + + async fn capture_payment_drain_lookup( + State(captured): State, + headers: HeaderMap, + body: Bytes, + ) -> impl IntoResponse { + capture_request(&captured, &headers, body, "/payment-request-drain-lookups"); + drain_response() + } + + async fn capture_payment_request_status( + State(captured): State, + headers: HeaderMap, + body: Bytes, + ) -> impl IntoResponse { + capture_request(&captured, &headers, body, "/payment-requests/status"); + Json(json!({ + "request_state": "accepted", + "payment_state": "detected", + "invoice_created_at": "2026-08-12T10:00:00Z", + "payment_deadline": "2026-08-13T10:00:00Z", + "confirmations": 0, + "amount_matched": true, + })) + } + + fn capture_request(captured: &CapturedRequests, headers: &HeaderMap, body: Bytes, path: &str) { + captured.push(CapturedRequest { + path: path.to_owned(), + signature: headers + .get(SIGNATURE_HEADER) + .map(|value| value.to_str().unwrap().to_owned()), + body: body.to_vec(), + }); + } + + fn drain_response() -> Json { + Json(json!({ + "status": "active", + "accepted_count": 1, + "terminal_count": 0, + "cancellation_enqueued_count": 0, + "cleanup_token": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + })) + } } diff --git a/locks-service/migrations/0014_paykit_invoice_windows.sql b/locks-service/migrations/0014_paykit_invoice_windows.sql new file mode 100644 index 0000000..87b1b9b --- /dev/null +++ b/locks-service/migrations/0014_paykit_invoice_windows.sql @@ -0,0 +1,28 @@ +ALTER TABLE paykit_task_admissions +ADD COLUMN payment_in_hours BIGINT, +ADD COLUMN invoice_created_at TIMESTAMPTZ, +ADD COLUMN payment_deadline TIMESTAMPTZ; + +-- Rows created before this migration have no authoritative payment window to +-- backfill. Preserve that all-NULL legacy state so the application can fail +-- closed instead of fabricating invoice facts. Every post-migration admission +-- writes a positive payment_in_hours and becomes ready only with a complete +-- immutable timestamp window. +ALTER TABLE paykit_task_admissions +ADD CONSTRAINT paykit_task_admissions_invoice_window_valid + CHECK ( + (payment_in_hours IS NULL + AND invoice_created_at IS NULL + AND payment_deadline IS NULL) + OR + (payment_in_hours > 0 + AND NOT ready + AND invoice_created_at IS NULL + AND payment_deadline IS NULL) + OR + (payment_in_hours > 0 + AND ready + AND invoice_created_at IS NOT NULL + AND payment_deadline IS NOT NULL + AND invoice_created_at <= payment_deadline) + ); diff --git a/locks-service/migrations/0015_content_lock_payment_drains.sql b/locks-service/migrations/0015_content_lock_payment_drains.sql new file mode 100644 index 0000000..b9913a2 --- /dev/null +++ b/locks-service/migrations/0015_content_lock_payment_drains.sql @@ -0,0 +1,92 @@ +ALTER TABLE content_lock_deletion_task_snapshot + ADD COLUMN creator TEXT, + ADD COLUMN bundle_id TEXT, + ADD COLUMN pubky_lock_resource TEXT, + ADD COLUMN criterion_id TEXT, + ADD COLUMN status_at_cutoff TEXT, + ADD COLUMN paykit_admission_required BOOLEAN, + ADD COLUMN payment_in_hours BIGINT, + ADD COLUMN invoice_created_at TIMESTAMPTZ, + ADD COLUMN payment_deadline TIMESTAMPTZ, + ADD COLUMN resolved_status TEXT, + ADD COLUMN resolved_at TIMESTAMPTZ; + +ALTER TABLE verification_tasks + ADD COLUMN entitlement_publication_claim_token UUID, + ADD COLUMN deletion_job_id UUID REFERENCES content_lock_deletion_jobs(job_id); + +COMMENT ON COLUMN verification_tasks.entitlement_publication_claim_token IS + 'Claim token that crossed the external entitlement-publication boundary.'; +COMMENT ON COLUMN verification_tasks.deletion_job_id IS + 'Task-row ownership fence set by graceful deletion admission.'; + +ALTER TABLE content_lock_deletion_task_snapshot + ADD CONSTRAINT content_lock_deletion_task_snapshot_cutoff_status_valid + CHECK (status_at_cutoff IN ('pending', 'in_progress', 'completed', 'failed', 'expired')), + ADD CONSTRAINT content_lock_deletion_task_snapshot_identity_all_or_none + CHECK ( + (creator IS NULL AND bundle_id IS NULL AND pubky_lock_resource IS NULL + AND criterion_id IS NULL AND status_at_cutoff IS NULL) + OR + (creator IS NOT NULL AND bundle_id IS NOT NULL + AND pubky_lock_resource IS NOT NULL AND status_at_cutoff IS NOT NULL) + ), + ADD CONSTRAINT content_lock_deletion_task_snapshot_admission_shape_valid + CHECK ( + (paykit_admission_required IS NULL + AND payment_in_hours IS NULL + AND invoice_created_at IS NULL + AND payment_deadline IS NULL) + OR + (paykit_admission_required = FALSE + AND payment_in_hours IS NULL + AND invoice_created_at IS NULL + AND payment_deadline IS NULL) + OR + (paykit_admission_required = TRUE + AND payment_in_hours IS NULL + AND invoice_created_at IS NULL + AND payment_deadline IS NULL) + OR + (paykit_admission_required = TRUE + AND payment_in_hours > 0 + AND invoice_created_at IS NOT NULL + AND payment_deadline IS NOT NULL + AND invoice_created_at <= payment_deadline) + ), + ADD CONSTRAINT content_lock_deletion_task_snapshot_resolution_valid + CHECK ( + (resolved_status IS NULL AND resolved_at IS NULL) + OR + (resolved_status IN ('completed', 'failed', 'expired') AND resolved_at IS NOT NULL) + ), + ADD CONSTRAINT content_lock_deletion_task_snapshot_bundle_unique + UNIQUE (deletion_job_id, creator, bundle_id); + +CREATE INDEX content_lock_deletion_task_snapshot_resolution_idx + ON content_lock_deletion_task_snapshot (deletion_job_id, resolved_status); + +CREATE TABLE content_lock_payment_drains ( + deletion_job_id UUID PRIMARY KEY + REFERENCES content_lock_deletion_jobs(job_id) ON DELETE CASCADE, + status TEXT NOT NULL, + accepted_count BIGINT NOT NULL, + terminal_count BIGINT NOT NULL, + cancellation_enqueued_count BIGINT NOT NULL, + cleanup_token TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL, + updated_at TIMESTAMPTZ NOT NULL, + CONSTRAINT content_lock_payment_drains_status_valid + CHECK (status IN ('active', 'completed')), + CONSTRAINT content_lock_payment_drains_counts_valid + CHECK ( + accepted_count >= 0 + AND terminal_count >= 0 + AND cancellation_enqueued_count >= 0 + ), + CONSTRAINT content_lock_payment_drains_cleanup_token_shape + CHECK ( + length(cleanup_token) = 43 + AND cleanup_token ~ '^[A-Za-z0-9_-]{43}$' + ) +); diff --git a/locks-service/src/application/ports/entitlement.rs b/locks-service/src/application/ports/entitlement.rs index 9158df8..e635e7a 100644 --- a/locks-service/src/application/ports/entitlement.rs +++ b/locks-service/src/application/ports/entitlement.rs @@ -35,3 +35,22 @@ pub trait EntitlementRepository: Send + Sync { bundle_id: &BundleId, ) -> Result<(), ApplicationError>; } + +pub(crate) fn same_entitlement_decision( + existing: &VerifiedProofBundle, + candidate: &VerifiedProofBundle, +) -> bool { + if existing.verification_result.criteria.len() != candidate.verification_result.criteria.len() { + return false; + } + let mut normalized_candidate = candidate.clone(); + for (candidate_result, existing_result) in normalized_candidate + .verification_result + .criteria + .iter_mut() + .zip(&existing.verification_result.criteria) + { + candidate_result.verified_at = existing_result.verified_at; + } + existing == &normalized_candidate +} diff --git a/locks-service/src/application/ports/mod.rs b/locks-service/src/application/ports/mod.rs index 97ecdbf..a40450e 100644 --- a/locks-service/src/application/ports/mod.rs +++ b/locks-service/src/application/ports/mod.rs @@ -7,6 +7,8 @@ mod creator_authority; mod entitlement; mod guarded_resources; mod lock_policy; +mod payment_drain; +mod payment_drain_repository; mod runtime; mod verification; @@ -17,5 +19,47 @@ pub use creator_authority::*; pub use entitlement::*; pub use guarded_resources::*; pub use lock_policy::*; +pub use payment_drain::*; +pub use payment_drain_repository::*; pub use runtime::*; pub use verification::*; + +#[cfg(test)] +mod payment_drain_contract_tests { + use super::{ + PaymentDrainCleanupToken, PaymentDrainClient, PaymentDrainClientError, PaymentDrainStatus, + PaymentRequestState, PaymentState, + }; + + fn assert_object_safe(_: &dyn PaymentDrainClient) {} + + #[test] + fn payment_drain_port_is_object_safe_and_closed_values_parse_strictly() { + let _ = assert_object_safe; + assert_eq!( + PaymentDrainStatus::parse("active"), + Some(PaymentDrainStatus::Active) + ); + assert_eq!(PaymentDrainStatus::parse("complete"), None); + assert_eq!( + PaymentRequestState::parse("proposal_expired"), + Some(PaymentRequestState::ProposalExpired) + ); + assert_eq!(PaymentRequestState::parse("unknown"), None); + assert_eq!(PaymentState::parse("expired"), Some(PaymentState::Expired)); + assert_eq!(PaymentState::parse("late"), None); + let token = + PaymentDrainCleanupToken::parse("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA").unwrap(); + assert_eq!( + token.as_str(), + "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + ); + assert!(PaymentDrainCleanupToken::parse("short").is_none()); + assert_eq!(format!("{token:?}"), "PaymentDrainCleanupToken()"); + let _ = PaymentDrainClientError::NotFound; + let _ = PaymentDrainClientError::Conflict; + let _ = PaymentDrainClientError::MalformedSuccess; + let _ = PaymentDrainClientError::Transport; + let _ = PaymentDrainClientError::Server; + } +} diff --git a/locks-service/src/application/ports/payment_drain.rs b/locks-service/src/application/ports/payment_drain.rs new file mode 100644 index 0000000..4b5e55c --- /dev/null +++ b/locks-service/src/application/ports/payment_drain.rs @@ -0,0 +1,143 @@ +use std::fmt; + +use async_trait::async_trait; +use base64::Engine; +use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use locks_core::ids::{BundleId, CreatorPubky, PubkyLockResource}; +use time::OffsetDateTime; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PaymentDrainStatus { + Active, + Completed, +} + +impl PaymentDrainStatus { + pub fn parse(value: &str) -> Option { + match value { + "active" => Some(Self::Active), + "completed" => Some(Self::Completed), + _ => None, + } + } +} + +#[derive(Clone, PartialEq, Eq)] +pub struct PaymentDrainCleanupToken(String); + +impl PaymentDrainCleanupToken { + pub fn parse(value: &str) -> Option { + if value.len() != 43 || value.contains('=') { + return None; + } + let decoded: [u8; 32] = URL_SAFE_NO_PAD.decode(value).ok()?.try_into().ok()?; + (URL_SAFE_NO_PAD.encode(decoded) == value).then(|| Self(value.to_owned())) + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl fmt::Debug for PaymentDrainCleanupToken { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str("PaymentDrainCleanupToken()") + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PaymentDrainSummary { + pub status: PaymentDrainStatus, + pub accepted_count: u64, + pub terminal_count: u64, + pub cancellation_enqueued_count: u64, + pub cleanup_token: PaymentDrainCleanupToken, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PaymentRequestState { + Proposed, + ProposalExpired, + Accepted, + Rejected, + Canceled, + ProofSubmitted, + ActiveRecurring, + RecoveryRequired, + InvalidConflict, +} + +impl PaymentRequestState { + pub fn parse(value: &str) -> Option { + match value { + "proposed" => Some(Self::Proposed), + "proposal_expired" => Some(Self::ProposalExpired), + "accepted" => Some(Self::Accepted), + "rejected" => Some(Self::Rejected), + "canceled" => Some(Self::Canceled), + "proof_submitted" => Some(Self::ProofSubmitted), + "active_recurring" => Some(Self::ActiveRecurring), + "recovery_required" => Some(Self::RecoveryRequired), + "invalid_conflict" => Some(Self::InvalidConflict), + _ => None, + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PaymentState { + Undetected, + Detected, + Confirmed, + Expired, +} + +impl PaymentState { + pub fn parse(value: &str) -> Option { + match value { + "undetected" => Some(Self::Undetected), + "detected" => Some(Self::Detected), + "confirmed" => Some(Self::Confirmed), + "expired" => Some(Self::Expired), + _ => None, + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PaymentRequestStatus { + pub request_state: PaymentRequestState, + pub payment_state: PaymentState, + pub invoice_created_at: OffsetDateTime, + pub payment_deadline: OffsetDateTime, + pub confirmations: u32, + pub amount_matched: bool, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum PaymentDrainClientError { + NotFound, + Conflict, + MalformedSuccess, + Transport, + Server, +} + +#[async_trait] +pub trait PaymentDrainClient: Send + Sync { + async fn start_payment_drain( + &self, + lock_resource: &PubkyLockResource, + ) -> Result; + + async fn lookup_payment_drain( + &self, + lock_resource: &PubkyLockResource, + ) -> Result, PaymentDrainClientError>; + + async fn payment_request_status( + &self, + creator: &CreatorPubky, + bundle_id: &BundleId, + ) -> Result, PaymentDrainClientError>; +} diff --git a/locks-service/src/application/ports/payment_drain_repository.rs b/locks-service/src/application/ports/payment_drain_repository.rs new file mode 100644 index 0000000..a1239da --- /dev/null +++ b/locks-service/src/application/ports/payment_drain_repository.rs @@ -0,0 +1,82 @@ +use async_trait::async_trait; +use locks_core::ids::{BundleId, CreatorPubky, PubkyLockResource, TaskId}; +use time::OffsetDateTime; +use uuid::Uuid; + +use crate::application::errors::ApplicationError; +use crate::application::models::VerificationTaskStatus; + +use super::PaymentDrainSummary; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PaymentDrainObligation { + pub task_id: TaskId, + pub creator: CreatorPubky, + pub bundle_id: BundleId, + pub lock_resource: PubkyLockResource, + pub criterion_id: String, + pub invoice_created_at: OffsetDateTime, + pub payment_deadline: OffsetDateTime, + pub status: VerificationTaskStatus, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PaymentDrainTerminalTransition { + pub status: VerificationTaskStatus, + pub entitlement_publication_token: Option, +} + +#[async_trait] +pub trait PaymentDrainRepository: Send + Sync { + async fn store_payment_drain( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + summary: &PaymentDrainSummary, + ) -> Result; + + async fn get_payment_drain( + &self, + deletion_job_id: Uuid, + ) -> Result, ApplicationError>; + + async fn reconcile_payment_drain( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + summary: &PaymentDrainSummary, + ) -> Result; + + async fn list_obligations( + &self, + deletion_job_id: Uuid, + ) -> Result, ApplicationError>; + + async fn begin_entitlement_publication( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + task_id: &TaskId, + ) -> Result, ApplicationError>; + + async fn persist_terminal_obligation( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + task_id: &TaskId, + transition: PaymentDrainTerminalTransition, + ) -> Result; + + async fn all_obligations_terminal( + &self, + deletion_job_id: Uuid, + ) -> Result; +} diff --git a/locks-service/src/application/ports/verification.rs b/locks-service/src/application/ports/verification.rs index eec1528..255faec 100644 --- a/locks-service/src/application/ports/verification.rs +++ b/locks-service/src/application/ports/verification.rs @@ -62,6 +62,17 @@ pub trait VerificationTaskRepository: Send + Sync { /// Worker-facing port for claiming verification task leases. #[async_trait] pub trait VerificationTaskClaimer: Send + Sync { + /// Fences entitlement publication before external storage I/O. + /// + /// Returns false when the exact lease is lost or deletion already owns the task. + async fn begin_claimed_entitlement_publication( + &self, + task_id: &TaskId, + worker_id: &str, + claim_token: &uuid::Uuid, + now: time::OffsetDateTime, + ) -> Result; + /// Claims one pending or expired in-progress verification task for a worker. /// /// Returns `Ok(None)` when no task is claimable. Every successful claim includes a fresh diff --git a/locks-service/src/application/use_cases/complete_verification_task.rs b/locks-service/src/application/use_cases/complete_verification_task.rs index 7d2c4a7..bf4d7d9 100644 --- a/locks-service/src/application/use_cases/complete_verification_task.rs +++ b/locks-service/src/application/use_cases/complete_verification_task.rs @@ -1,7 +1,7 @@ use async_trait::async_trait; use time::OffsetDateTime; -use locks_core::ids::{LockServerPubky, TaskId}; +use locks_core::ids::{BundleId, CreatorPubky, LockServerPubky, TaskId}; use locks_core::lock_policy::ContentLock; use locks_core::verification::{ EntitlementLifetime, VERIFIED_PROOF_BUNDLE_VERSION, VerificationResult, VerifiedProofBundle, @@ -15,7 +15,7 @@ use crate::application::models::{ }; use crate::application::ports::{ Clock, ContentLockRepository, CriterionVerifierRegistry, EntitlementRepository, - VerificationTaskClaimer, VerificationTaskRepository, + VerificationTaskClaimer, VerificationTaskRepository, same_entitlement_decision, }; use crate::application::use_cases::entitlement_check::verify_content_lock_identity; @@ -54,6 +54,56 @@ struct ClaimFencedTaskRepository<'a> { clock: &'a dyn Clock, } +struct ClaimFencedEntitlementRepository<'a> { + inner: &'a dyn EntitlementRepository, + claim: &'a ClaimedVerificationTask, + claimer: &'a dyn VerificationTaskClaimer, + worker_id: &'a str, + clock: &'a dyn Clock, +} + +#[async_trait] +impl EntitlementRepository for ClaimFencedEntitlementRepository<'_> { + async fn insert_verified_proof_bundle( + &self, + entitlement: VerifiedProofBundle, + ) -> Result<(), ApplicationError> { + if !self + .claimer + .begin_claimed_entitlement_publication( + &self.claim.task.task_id, + self.worker_id, + &self.claim.claim_token, + self.clock.now(), + ) + .await? + { + return Err(ApplicationError::VerificationTaskClaimLost); + } + self.inner.insert_verified_proof_bundle(entitlement).await + } + + async fn get_verified_proof_bundle( + &self, + creator: &CreatorPubky, + bundle_id: &BundleId, + ) -> Result, ApplicationError> { + self.inner + .get_verified_proof_bundle(creator, bundle_id) + .await + } + + async fn delete_verified_proof_bundle( + &self, + creator: &CreatorPubky, + bundle_id: &BundleId, + ) -> Result<(), ApplicationError> { + self.inner + .delete_verified_proof_bundle(creator, bundle_id) + .await + } +} + #[async_trait] impl VerificationTaskRepository for ClaimFencedTaskRepository<'_> { async fn insert_verification_task( @@ -213,19 +263,8 @@ impl<'a> CompleteVerificationTaskUseCase<'a> { .await { Ok(Some(existing)) if same_entitlement_decision(&existing, &entitlement) => {} - Ok(_) => { - self.persist_failed_task(task, viewer_safe_failure_message(&error).to_owned()) - .await?; - return Err(error); - } - Err(lookup_error) => { - self.persist_failed_task( - task, - viewer_safe_failure_message(&lookup_error).to_owned(), - ) - .await?; - return Err(lookup_error); - } + Ok(Some(_)) => return Err(error), + Ok(None) | Err(_) => return Err(ApplicationError::VerificationPending), } } @@ -257,7 +296,14 @@ impl<'a> CompleteVerificationTaskUseCase<'a> { }); } let fenced_tasks = ClaimFencedTaskRepository { - claim, + claim: claim.clone(), + claimer, + worker_id, + clock: self.clock, + }; + let fenced_entitlements = ClaimFencedEntitlementRepository { + inner: self.entitlements, + claim: &claim, claimer, worker_id, clock: self.clock, @@ -265,7 +311,7 @@ impl<'a> CompleteVerificationTaskUseCase<'a> { CompleteVerificationTaskUseCase::new( &fenced_tasks, self.content_locks, - self.entitlements, + &fenced_entitlements, self.verifiers, self.clock, self.verified_by.clone(), @@ -389,26 +435,6 @@ fn viewer_safe_failure_message(error: &ApplicationError) -> &'static str { } } -fn same_entitlement_decision( - existing: &VerifiedProofBundle, - candidate: &VerifiedProofBundle, -) -> bool { - if existing.verification_result.criteria.len() != candidate.verification_result.criteria.len() { - return false; - } - - let mut normalized_candidate = candidate.clone(); - for (candidate_result, existing_result) in normalized_candidate - .verification_result - .criteria - .iter_mut() - .zip(&existing.verification_result.criteria) - { - candidate_result.verified_at = existing_result.verified_at; - } - existing == &normalized_candidate -} - #[cfg(test)] mod tests { use std::str::FromStr; @@ -476,6 +502,7 @@ mod tests { datetime!(2026-05-29 12:01:00 UTC), datetime!(2026-05-29 12:02:00 UTC), datetime!(2026-05-29 12:03:00 UTC), + datetime!(2026-05-29 12:04:00 UTC), ]); let use_case = CompleteVerificationTaskUseCase::new( &tasks, @@ -583,6 +610,82 @@ mod tests { ); } + #[tokio::test] + async fn ambiguous_entitlement_write_without_read_back_stays_pending() { + let content_lock = content_lock_fixture(true); + let tasks = FakeTasks::new(Some(pending_task_for(&content_lock))); + let content_locks = FakeContentLocks::new(Some(content_lock)); + let entitlements = FakeEntitlements::with_ambiguous_write_and_absent_read_back(); + let verifier = FakeVerifier { + satisfied: true, + error: None, + }; + let registry = FakeRegistry { + verifier: Some(&verifier), + }; + let clock = SequenceClock::new(vec![ + datetime!(2026-05-29 12:01:00 UTC), + datetime!(2026-05-29 12:02:00 UTC), + ]); + + let result = CompleteVerificationTaskUseCase::new( + &tasks, + &content_locks, + &entitlements, + ®istry, + &clock, + lock_server(), + ) + .execute(CompleteVerificationTaskRequest { task_id: task_id() }) + .await; + + assert_eq!(result, Err(ApplicationError::VerificationPending)); + assert!( + !tasks + .updates() + .iter() + .any(|task| task.status == VerificationTaskStatus::Failed) + ); + } + + #[tokio::test] + async fn ambiguous_entitlement_write_with_failed_read_back_stays_pending() { + let content_lock = content_lock_fixture(true); + let tasks = FakeTasks::new(Some(pending_task_for(&content_lock))); + let content_locks = FakeContentLocks::new(Some(content_lock)); + let entitlements = FakeEntitlements::with_ambiguous_write_and_failed_read_back(); + let verifier = FakeVerifier { + satisfied: true, + error: None, + }; + let registry = FakeRegistry { + verifier: Some(&verifier), + }; + let clock = SequenceClock::new(vec![ + datetime!(2026-05-29 12:01:00 UTC), + datetime!(2026-05-29 12:02:00 UTC), + ]); + + let result = CompleteVerificationTaskUseCase::new( + &tasks, + &content_locks, + &entitlements, + ®istry, + &clock, + lock_server(), + ) + .execute(CompleteVerificationTaskRequest { task_id: task_id() }) + .await; + + assert_eq!(result, Err(ApplicationError::VerificationPending)); + assert!( + !tasks + .updates() + .iter() + .any(|task| task.status == VerificationTaskStatus::Failed) + ); + } + #[tokio::test] async fn current_claim_owner_completes_after_equivalent_entitlement_was_published() { let content_lock = content_lock_fixture(true); @@ -628,6 +731,7 @@ mod tests { datetime!(2026-05-29 12:05:00 UTC), datetime!(2026-05-29 12:06:00 UTC), datetime!(2026-05-29 12:07:00 UTC), + datetime!(2026-05-29 12:08:00 UTC), ]); let completed = CompleteVerificationTaskUseCase::new( &tasks, @@ -707,10 +811,10 @@ mod tests { record: "verified_proof_bundle" }) ); - assert_eq!( - retry_tasks.updates().last().unwrap().status, - VerificationTaskStatus::Failed - ); + let updates = retry_tasks.updates(); + let stored = updates.last().unwrap(); + assert_eq!(stored.status, VerificationTaskStatus::InProgress); + assert_eq!(stored.failure_message, None); } #[tokio::test] @@ -1127,6 +1231,16 @@ mod tests { #[async_trait] impl VerificationTaskClaimer for LostClaimClaimer { + async fn begin_claimed_entitlement_publication( + &self, + _task_id: &TaskId, + _worker_id: &str, + _claim_token: &uuid::Uuid, + _now: time::OffsetDateTime, + ) -> Result { + Ok(true) + } + async fn claim_next_verification_task( &self, _worker_id: &str, @@ -1250,6 +1364,8 @@ mod tests { struct FakeEntitlements { stored: Mutex>, fail_after_store: bool, + hide_stored: bool, + fail_read: bool, } impl FakeEntitlements { @@ -1257,6 +1373,26 @@ mod tests { Self { stored: Mutex::new(Vec::new()), fail_after_store: true, + hide_stored: false, + fail_read: false, + } + } + + fn with_ambiguous_write_and_absent_read_back() -> Self { + Self { + stored: Mutex::new(Vec::new()), + fail_after_store: true, + hide_stored: true, + fail_read: false, + } + } + + fn with_ambiguous_write_and_failed_read_back() -> Self { + Self { + stored: Mutex::new(Vec::new()), + fail_after_store: true, + hide_stored: false, + fail_read: true, } } @@ -1295,6 +1431,14 @@ mod tests { creator: &CreatorPubky, bundle_id: &BundleId, ) -> Result, ApplicationError> { + if self.fail_read { + return Err(ApplicationError::Storage { + message: "entitlement read unavailable".to_owned(), + }); + } + if self.hide_stored { + return Ok(None); + } Ok(self .stored .lock() diff --git a/locks-service/src/application/use_cases/drain_lock_payments.rs b/locks-service/src/application/use_cases/drain_lock_payments.rs new file mode 100644 index 0000000..5b46ba8 --- /dev/null +++ b/locks-service/src/application/use_cases/drain_lock_payments.rs @@ -0,0 +1,554 @@ +use locks_core::ids::{ContentLockPath, LockServerPubky, PubkyLockResource}; +use locks_core::lock_policy::VerifierType; +use locks_core::verification::{ + CriterionVerificationResult, EntitlementLifetime, VERIFIED_PROOF_BUNDLE_VERSION, + VerificationResult, VerifiedProofBundle, +}; + +use crate::application::errors::ApplicationError; +use crate::application::models::{ + ClaimedContentLockDeletionJob, ContentLockDeletionPhase, VerificationTaskStatus, +}; +use crate::application::ports::{ + Clock, ContentLockDeletionRepository, EntitlementRepository, PaymentDrainClient, + PaymentDrainClientError, PaymentDrainRepository, PaymentDrainStatus, PaymentDrainSummary, + PaymentDrainTerminalTransition, PaymentRequestState, PaymentRequestStatus, PaymentState, + same_entitlement_decision, +}; + +pub struct DrainLockPaymentsUseCase<'a> { + deletions: &'a dyn ContentLockDeletionRepository, + drains: &'a dyn PaymentDrainRepository, + paykit: &'a dyn PaymentDrainClient, + entitlements: &'a dyn EntitlementRepository, + clock: &'a dyn Clock, + verified_by: LockServerPubky, + minimum_confirmations: u32, +} + +impl<'a> DrainLockPaymentsUseCase<'a> { + pub fn new( + deletions: &'a dyn ContentLockDeletionRepository, + drains: &'a dyn PaymentDrainRepository, + paykit: &'a dyn PaymentDrainClient, + entitlements: &'a dyn EntitlementRepository, + clock: &'a dyn Clock, + verified_by: LockServerPubky, + minimum_confirmations: u32, + ) -> Self { + Self { + deletions, + drains, + paykit, + entitlements, + clock, + verified_by, + minimum_confirmations, + } + } + + pub async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> Result { + match claim.job.phase { + ContentLockDeletionPhase::StartPaymentDrain => self.start(claim, worker_id).await, + ContentLockDeletionPhase::DrainPayments => self.drain(claim, worker_id).await, + _ => Err(ApplicationError::InvalidContentLockDeletionState { + message: "payment drain use case requires a payment drain phase".to_owned(), + }), + } + } + + async fn start( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> Result { + let lock_resource = lock_resource(&claim); + let summary = match self.paykit.start_payment_drain(&lock_resource).await { + Ok(summary) => summary, + Err(PaymentDrainClientError::Conflict) => self + .paykit + .lookup_payment_drain(&lock_resource) + .await + .map_err(map_client_error)? + .ok_or_else(|| ApplicationError::Verifier { + message: "Paykit payment drain conflict could not be reconciled".to_owned(), + })?, + Err(error) => return Err(map_client_error(error)), + }; + let now = self.clock.now(); + if !self + .drains + .store_payment_drain( + claim.job.job_id, + worker_id, + claim.claim_token, + now, + &summary, + ) + .await? + { + return Ok(false); + } + Ok(self + .deletions + .advance_phase( + claim.job.job_id, + worker_id, + claim.claim_token, + now, + ContentLockDeletionPhase::DrainPayments, + ) + .await? + .is_some()) + } + + async fn drain( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> Result { + let lock_resource = lock_resource(&claim); + let summary = self + .paykit + .lookup_payment_drain(&lock_resource) + .await + .map_err(map_client_error)? + .ok_or_else(|| ApplicationError::Verifier { + message: "Paykit payment drain not found".to_owned(), + })?; + let persisted = self + .drains + .get_payment_drain(claim.job.job_id) + .await? + .ok_or_else(|| ApplicationError::InvalidContentLockDeletionState { + message: "payment drain cleanup token is missing".to_owned(), + })?; + validate_aggregate_progress(&persisted, &summary)?; + let now = self.clock.now(); + if !self + .drains + .reconcile_payment_drain( + claim.job.job_id, + worker_id, + claim.claim_token, + now, + &summary, + ) + .await? + { + return Ok(false); + } + + for obligation in self.drains.list_obligations(claim.job.job_id).await? { + if matches!( + obligation.status, + VerificationTaskStatus::Completed + | VerificationTaskStatus::Expired + | VerificationTaskStatus::Failed + ) { + continue; + } + let status = self + .paykit + .payment_request_status(&obligation.creator, &obligation.bundle_id) + .await + .map_err(map_client_error)? + .ok_or_else(|| ApplicationError::Verifier { + message: "Paykit payment request not found".to_owned(), + })?; + if status.invoice_created_at != obligation.invoice_created_at + || status.payment_deadline != obligation.payment_deadline + { + return Err(ApplicationError::InvalidVerificationTaskState { + message: "Paykit payment window changed during deletion".to_owned(), + }); + } + let Some(next_status) = classify_payment_task(status, self.minimum_confirmations)? + else { + continue; + }; + let now = self.clock.now(); + let mut entitlement_publication_token = None; + if next_status == VerificationTaskStatus::Completed { + let Some(publication_token) = self + .drains + .begin_entitlement_publication( + claim.job.job_id, + worker_id, + claim.claim_token, + now, + &obligation.task_id, + ) + .await? + else { + return Ok(false); + }; + let entitlement = VerifiedProofBundle { + version: VERIFIED_PROOF_BUNDLE_VERSION, + bundle_id: obligation.bundle_id.clone(), + pubky_lock_resource: obligation.lock_resource.clone(), + verification_result: VerificationResult { + criteria: vec![CriterionVerificationResult { + criterion_id: obligation.criterion_id.clone(), + satisfied: true, + verified_at: now, + verified_by: self.verified_by.clone(), + verifier_type: VerifierType::PaykitPayment, + }], + }, + entitlement_lifetime: EntitlementLifetime::Unbounded, + }; + persist_entitlement(self.entitlements, entitlement).await?; + entitlement_publication_token = Some(publication_token); + } + if !self + .drains + .persist_terminal_obligation( + claim.job.job_id, + worker_id, + claim.claim_token, + now, + &obligation.task_id, + PaymentDrainTerminalTransition { + status: next_status, + entitlement_publication_token, + }, + ) + .await? + { + return Ok(false); + } + } + + if summary.status != PaymentDrainStatus::Completed + || !self + .drains + .all_obligations_terminal(claim.job.job_id) + .await? + { + return Ok(false); + } + Ok(self + .deletions + .advance_phase( + claim.job.job_id, + worker_id, + claim.claim_token, + self.clock.now(), + ContentLockDeletionPhase::DrainExistingCredentials, + ) + .await? + .is_some()) + } +} + +fn lock_resource(claim: &ClaimedContentLockDeletionJob) -> PubkyLockResource { + PubkyLockResource::new( + claim.job.creator.clone(), + ContentLockPath::from_lock_id(claim.job.lock_id.clone()), + ) +} + +async fn persist_entitlement( + entitlements: &dyn EntitlementRepository, + entitlement: VerifiedProofBundle, +) -> Result<(), ApplicationError> { + if let Err(error) = entitlements + .insert_verified_proof_bundle(entitlement.clone()) + .await + { + let existing = entitlements + .get_verified_proof_bundle( + entitlement.pubky_lock_resource.creator(), + &entitlement.bundle_id, + ) + .await?; + if !existing + .as_ref() + .is_some_and(|existing| same_entitlement_decision(existing, &entitlement)) + { + return Err(error); + } + } + Ok(()) +} + +fn map_client_error(error: PaymentDrainClientError) -> ApplicationError { + ApplicationError::Verifier { + message: match error { + PaymentDrainClientError::NotFound => "Paykit payment drain not found", + PaymentDrainClientError::Conflict => "Paykit payment drain conflict", + PaymentDrainClientError::MalformedSuccess => { + "Paykit payment drain returned malformed success" + } + PaymentDrainClientError::Transport => "Paykit payment drain transport failure", + PaymentDrainClientError::Server => "Paykit payment drain server failure", + } + .to_owned(), + } +} + +fn validate_aggregate_progress( + persisted: &PaymentDrainSummary, + current: &PaymentDrainSummary, +) -> Result<(), ApplicationError> { + let accepted_delta = persisted.accepted_count.checked_sub(current.accepted_count); + let terminal_delta = current.terminal_count.checked_sub(persisted.terminal_count); + if persisted.cleanup_token != current.cleanup_token + || persisted.cancellation_enqueued_count != current.cancellation_enqueued_count + || accepted_delta.is_none() + || terminal_delta.is_none() + || accepted_delta != terminal_delta + || (current.status == PaymentDrainStatus::Completed && current.accepted_count != 0) + || (current.status == PaymentDrainStatus::Active && current.accepted_count == 0) + || (persisted.status == PaymentDrainStatus::Completed + && current.status != PaymentDrainStatus::Completed) + { + return Err(ApplicationError::InvalidContentLockDeletionState { + message: "Paykit payment drain aggregate changed incompatibly for deletion job" + .to_owned(), + }); + } + Ok(()) +} + +pub(crate) fn classify_payment_task( + status: PaymentRequestStatus, + minimum_confirmations: u32, +) -> Result, ApplicationError> { + if matches!( + status.request_state, + PaymentRequestState::RecoveryRequired + | PaymentRequestState::InvalidConflict + | PaymentRequestState::ProofSubmitted + | PaymentRequestState::ActiveRecurring + ) { + return Err(ApplicationError::InvalidVerificationTaskState { + message: "Paykit payment request entered an unsupported failure state".to_owned(), + }); + } + if matches!( + status.request_state, + PaymentRequestState::Rejected + | PaymentRequestState::Canceled + | PaymentRequestState::ProposalExpired + ) || (status.request_state == PaymentRequestState::Accepted + && status.payment_state == PaymentState::Expired) + { + return Ok(Some(VerificationTaskStatus::Expired)); + } + + if status.request_state != PaymentRequestState::Accepted || !status.amount_matched { + return Ok(None); + } + + if minimum_confirmations == 0 + && matches!( + status.payment_state, + PaymentState::Detected | PaymentState::Confirmed + ) + { + return Ok(Some(VerificationTaskStatus::Completed)); + } + + Ok((status.payment_state == PaymentState::Confirmed + && status.confirmations >= minimum_confirmations) + .then_some(VerificationTaskStatus::Completed)) +} + +#[cfg(test)] +mod tests { + use base64::Engine; + use base64::engine::general_purpose::URL_SAFE_NO_PAD; + use time::macros::datetime; + + use crate::application::models::VerificationTaskStatus; + use crate::application::ports::{ + PaymentDrainCleanupToken, PaymentDrainStatus, PaymentDrainSummary, PaymentRequestState, + PaymentRequestStatus, PaymentState, + }; + + use super::{classify_payment_task, validate_aggregate_progress}; + + #[test] + fn aggregate_progress_accepts_only_equal_monotonic_transfer() { + let initial = summary(PaymentDrainStatus::Active, 2, 3, 1, 1); + assert!( + validate_aggregate_progress(&initial, &summary(PaymentDrainStatus::Active, 1, 4, 1, 1)) + .is_ok() + ); + assert!( + validate_aggregate_progress( + &initial, + &summary(PaymentDrainStatus::Completed, 0, 5, 1, 1) + ) + .is_ok() + ); + + for invalid in [ + summary(PaymentDrainStatus::Active, 3, 2, 1, 1), + summary(PaymentDrainStatus::Active, 1, 3, 1, 1), + summary(PaymentDrainStatus::Active, 1, 5, 1, 1), + summary(PaymentDrainStatus::Active, 1, 4, 2, 1), + summary(PaymentDrainStatus::Active, 1, 4, 1, 2), + summary(PaymentDrainStatus::Completed, 1, 4, 1, 1), + ] { + assert!(validate_aggregate_progress(&initial, &invalid).is_err()); + } + let completed = summary(PaymentDrainStatus::Completed, 0, 5, 1, 1); + assert!( + validate_aggregate_progress( + &completed, + &summary(PaymentDrainStatus::Active, 0, 5, 1, 1) + ) + .is_err() + ); + } + + fn summary( + status: PaymentDrainStatus, + accepted_count: u64, + terminal_count: u64, + cancellation_enqueued_count: u64, + token_byte: u8, + ) -> PaymentDrainSummary { + PaymentDrainSummary { + status, + accepted_count, + terminal_count, + cancellation_enqueued_count, + cleanup_token: PaymentDrainCleanupToken::parse( + &URL_SAFE_NO_PAD.encode([token_byte; 32]), + ) + .unwrap(), + } + } + + #[test] + fn terminal_unpaid_states_expire_without_failure() { + for request_state in [ + PaymentRequestState::Rejected, + PaymentRequestState::Canceled, + PaymentRequestState::ProposalExpired, + ] { + assert_eq!( + classify_payment_task(status(request_state, PaymentState::Undetected, 0, false), 6), + Ok(Some(VerificationTaskStatus::Expired)) + ); + } + assert_eq!( + classify_payment_task( + status( + PaymentRequestState::Accepted, + PaymentState::Expired, + 0, + false + ), + 6, + ), + Ok(Some(VerificationTaskStatus::Expired)) + ); + } + + #[test] + fn confirmations_are_applied_only_by_locks() { + assert_eq!( + classify_payment_task( + status( + PaymentRequestState::Accepted, + PaymentState::Detected, + 0, + true + ), + 0, + ), + Ok(Some(VerificationTaskStatus::Completed)) + ); + assert_eq!( + classify_payment_task( + status( + PaymentRequestState::Accepted, + PaymentState::Detected, + 0, + true + ), + 1, + ), + Ok(None) + ); + assert_eq!( + classify_payment_task( + status( + PaymentRequestState::Accepted, + PaymentState::Confirmed, + 5, + true + ), + 6, + ), + Ok(None) + ); + assert_eq!( + classify_payment_task( + status( + PaymentRequestState::Accepted, + PaymentState::Confirmed, + 6, + true + ), + 6, + ), + Ok(Some(VerificationTaskStatus::Completed)) + ); + } + + #[test] + fn timely_matched_payment_stays_pending_after_deadline_until_confirmed() { + let mut value = status( + PaymentRequestState::Accepted, + PaymentState::Detected, + 0, + true, + ); + value.payment_deadline = datetime!(2026-08-11 10:00:00 UTC); + assert_eq!(classify_payment_task(value, 6), Ok(None)); + } + + #[test] + fn classification_failure_states_fail_closed() { + for request_state in [ + PaymentRequestState::RecoveryRequired, + PaymentRequestState::InvalidConflict, + PaymentRequestState::ProofSubmitted, + PaymentRequestState::ActiveRecurring, + ] { + assert!( + classify_payment_task( + status(request_state, PaymentState::Undetected, 0, false), + 6, + ) + .is_err() + ); + } + } + + fn status( + request_state: PaymentRequestState, + payment_state: PaymentState, + confirmations: u32, + amount_matched: bool, + ) -> PaymentRequestStatus { + PaymentRequestStatus { + request_state, + payment_state, + invoice_created_at: datetime!(2026-08-12 10:00:00 UTC), + payment_deadline: datetime!(2026-08-13 10:00:00 UTC), + confirmations, + amount_matched, + } + } +} diff --git a/locks-service/src/application/use_cases/mod.rs b/locks-service/src/application/use_cases/mod.rs index 1218395..ae472c7 100644 --- a/locks-service/src/application/use_cases/mod.rs +++ b/locks-service/src/application/use_cases/mod.rs @@ -4,6 +4,7 @@ pub mod create_content_lock; #[cfg(test)] mod credential_flow_tests; pub mod delete_guarded_resource; +pub mod drain_lock_payments; mod entitlement_check; pub mod exchange_frontend_session_code; pub mod get_creator_authority_status; diff --git a/locks-service/src/application/use_cases/validate_paykit_payment_submission.rs b/locks-service/src/application/use_cases/validate_paykit_payment_submission.rs index a2cfbc1..7498097 100644 --- a/locks-service/src/application/use_cases/validate_paykit_payment_submission.rs +++ b/locks-service/src/application/use_cases/validate_paykit_payment_submission.rs @@ -12,6 +12,12 @@ pub struct ValidatePaykitPaymentSubmissionRequest { pub submitted_proof_bundle: SubmittedProofBundle, } +/// Immutable canonical payment facts required before Paykit invoice side effects. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ValidatedPaykitPaymentSubmission { + pub payment_in: u64, +} + /// Validates payment proof identity before invoice side effects are allowed. pub struct ValidatePaykitPaymentSubmissionUseCase<'a> { content_locks: &'a dyn ContentLockRepository, @@ -27,7 +33,7 @@ impl<'a> ValidatePaykitPaymentSubmissionUseCase<'a> { pub async fn execute( &self, request: ValidatePaykitPaymentSubmissionRequest, - ) -> Result<(), ApplicationError> { + ) -> Result { let submitted = request.submitted_proof_bundle; let content_lock = self .content_locks @@ -53,15 +59,22 @@ impl<'a> ValidatePaykitPaymentSubmissionUseCase<'a> { if proof.verifier_type != VerifierType::PaykitPayment { return Err(ApplicationError::InvalidPaykitPaymentSubmission); } - let criterion_matches = content_lock.criteria.iter().any(|criterion| { - criterion.criterion_id == proof.criterion_id - && criterion.verifier_type == VerifierType::PaykitPayment - }); - if !criterion_matches { - return Err(ApplicationError::InvalidPaykitPaymentSubmission); - } - - Ok(()) + let criterion = content_lock + .criteria + .iter() + .find(|criterion| { + criterion.criterion_id == proof.criterion_id + && criterion.verifier_type == VerifierType::PaykitPayment + }) + .ok_or(ApplicationError::InvalidPaykitPaymentSubmission)?; + let params = criterion + .paykit_payment_params() + .map_err(|_| ApplicationError::InvalidPaykitPaymentSubmission)? + .ok_or(ApplicationError::InvalidPaykitPaymentSubmission)?; + + Ok(ValidatedPaykitPaymentSubmission { + payment_in: params.payment_in(), + }) } } @@ -84,7 +97,10 @@ mod tests { }; use locks_core::verification::{Proof, SUBMITTED_PROOF_BUNDLE_VERSION, SubmittedProofBundle}; - use super::{ValidatePaykitPaymentSubmissionRequest, ValidatePaykitPaymentSubmissionUseCase}; + use super::{ + ValidatePaykitPaymentSubmissionRequest, ValidatePaykitPaymentSubmissionUseCase, + ValidatedPaykitPaymentSubmission, + }; use crate::application::errors::ApplicationError; use crate::application::ports::ContentLockRepository; @@ -177,7 +193,10 @@ mod tests { }) .await; - assert_eq!(result, Ok(())); + assert_eq!( + result, + Ok(ValidatedPaykitPaymentSubmission { payment_in: 24 }) + ); } #[tokio::test] diff --git a/locks-service/src/infrastructure/memory/content_lock_deletions.rs b/locks-service/src/infrastructure/memory/content_lock_deletions.rs index 8efcd24..4494d62 100644 --- a/locks-service/src/infrastructure/memory/content_lock_deletions.rs +++ b/locks-service/src/infrastructure/memory/content_lock_deletions.rs @@ -1,4 +1,7 @@ -use std::collections::{HashMap, HashSet}; +use std::{ + collections::{HashMap, HashSet}, + sync::Arc, +}; use async_trait::async_trait; use locks_core::ids::{CreatorPubky, LockId}; @@ -14,6 +17,7 @@ use crate::application::{ }, ports::ContentLockDeletionRepository, }; +use crate::infrastructure::memory::verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence; type JobKey = (CreatorPubky, LockId); @@ -26,17 +30,35 @@ struct StoredJob { } /// In-memory deletion repository with the same lease-fencing semantics as PostgreSQL. -#[derive(Debug, Default)] +#[derive(Debug)] pub struct InMemoryContentLockDeletionRepository { jobs: RwLock>, force_receipts: RwLock>, publication_intents: RwLock>, + verification_task_fence: Arc, +} + +impl Default for InMemoryContentLockDeletionRepository { + fn default() -> Self { + Self::with_verification_task_fence(Arc::new(InMemoryVerificationTaskDeletionFence::new())) + } } impl InMemoryContentLockDeletionRepository { pub fn new() -> Self { Self::default() } + + pub fn with_verification_task_fence( + verification_task_fence: Arc, + ) -> Self { + Self { + jobs: RwLock::new(HashMap::new()), + force_receipts: RwLock::new(HashSet::new()), + publication_intents: RwLock::new(HashMap::new()), + verification_task_fence, + } + } } #[async_trait] @@ -109,6 +131,7 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { job.validate_frozen_identity()?; job.validate_state(false)?; let key = (job.creator.clone(), job.lock_id.clone()); + let mut verification_tasks = self.verification_task_fence.records.write().await; let intents = self.publication_intents.read().await; let mut jobs = self.jobs.write().await; let receipts = self.force_receipts.read().await; @@ -120,6 +143,24 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { record: "content_lock_deletion_job", }); } + let matching_task_ids = verification_tasks + .iter() + .filter_map(|(task_id, task)| { + (task.creator == job.creator && task.lock_id == job.lock_id).then_some(*task_id) + }) + .collect::>(); + if matching_task_ids.iter().any(|task_id| { + verification_tasks + .get(task_id) + .is_some_and(|task| task.entitlement_publication_claim_token.is_some()) + }) { + return Err(ApplicationError::ContentLockDeletionInProgress); + } + for task_id in matching_task_ids { + if let Some(task) = verification_tasks.get_mut(&task_id) { + task.deletion_job_id = Some(job.job_id); + } + } jobs.insert( key, StoredJob { @@ -293,6 +334,16 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { if intents.contains_key(&key) { return Ok(PrepareForceDeletionResult::PublicationInProgress); } + let verification_tasks = self.verification_task_fence.records.read().await; + if verification_tasks.values().any(|task| { + task.creator == *creator + && task.lock_id == *lock_id + && task.deletion_job_id.is_some() + && task.entitlement_publication_claim_token.is_some() + }) { + return Ok(PrepareForceDeletionResult::PublicationInProgress); + } + drop(verification_tasks); let mut jobs = self.jobs.write().await; let mut receipts = self.force_receipts.write().await; if receipts.contains(&key) { diff --git a/locks-service/src/infrastructure/memory/mod.rs b/locks-service/src/infrastructure/memory/mod.rs index d1a1e28..1854ea9 100644 --- a/locks-service/src/infrastructure/memory/mod.rs +++ b/locks-service/src/infrastructure/memory/mod.rs @@ -6,4 +6,5 @@ pub mod entitlements; pub mod guarded_resources; pub mod lock_service_pointers; pub mod verification_task_claims; +pub mod verification_task_deletion_fence; pub mod verification_tasks; diff --git a/locks-service/src/infrastructure/memory/verification_task_claims.rs b/locks-service/src/infrastructure/memory/verification_task_claims.rs index 377a826..db07493 100644 --- a/locks-service/src/infrastructure/memory/verification_task_claims.rs +++ b/locks-service/src/infrastructure/memory/verification_task_claims.rs @@ -9,12 +9,14 @@ use crate::application::models::{ ClaimedVerificationTask, VerificationTaskRecord, VerificationTaskStatus, }; use crate::application::ports::{VerificationTaskClaimer, VerificationTaskRepository}; +use crate::infrastructure::memory::verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence; /// In-memory verification task claimer used to model worker lease semantics. #[derive(Default)] pub struct InMemoryVerificationTaskClaimer { records: RwLock>, task_repository: Option>, + deletion_fence: Arc, } #[derive(Debug, Clone)] @@ -29,6 +31,14 @@ struct ClaimableVerificationTask { impl InMemoryVerificationTaskClaimer { /// Creates a claimer seeded with unclaimed task records. pub fn new(records: Vec) -> Self { + let deletion_fence = Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks(&records)); + Self::with_deletion_fence(records, deletion_fence) + } + + pub fn with_deletion_fence( + records: Vec, + deletion_fence: Arc, + ) -> Self { Self { records: RwLock::new( records @@ -43,6 +53,7 @@ impl InMemoryVerificationTaskClaimer { .collect(), ), task_repository: None, + deletion_fence, } } @@ -56,10 +67,24 @@ impl InMemoryVerificationTaskClaimer { claimer } + pub fn with_task_repository_and_deletion_fence( + records: Vec, + task_repository: Arc, + deletion_fence: Arc, + ) -> Self { + let mut claimer = Self::with_deletion_fence(records, deletion_fence); + claimer.task_repository = Some(task_repository); + claimer + } + /// Creates a claimer seeded with already-claimed task records. pub fn with_claimed_tasks( records: Vec<(VerificationTaskRecord, String, time::OffsetDateTime)>, ) -> Self { + let tasks = records + .iter() + .map(|(task, _, _)| task.clone()) + .collect::>(); Self { records: RwLock::new( records @@ -76,23 +101,55 @@ impl InMemoryVerificationTaskClaimer { .collect(), ), task_repository: None, + deletion_fence: Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks(&tasks)), } } } #[async_trait] impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { + async fn begin_claimed_entitlement_publication( + &self, + task_id: &TaskId, + worker_id: &str, + claim_token: &uuid::Uuid, + now: time::OffsetDateTime, + ) -> Result { + let mut fence_records = self.deletion_fence.records.write().await; + let records = self.records.read().await; + let owned = records.iter().any(|record| { + record.task.task_id == *task_id + && record.task.status == VerificationTaskStatus::InProgress + && record.claimed_by.as_deref() == Some(worker_id) + && record.claim_token.as_ref() == Some(claim_token) + && record + .claim_expires_at + .is_some_and(|expires| expires >= now) + }); + let Some(fence) = fence_records.get_mut(task_id) else { + return Ok(false); + }; + if !owned || fence.deletion_job_id.is_some() { + return Ok(false); + } + fence.entitlement_publication_claim_token = Some(*claim_token); + Ok(true) + } + async fn claim_next_verification_task( &self, worker_id: &str, now: time::OffsetDateTime, claim_expires_at: time::OffsetDateTime, ) -> Result, ApplicationError> { + let fence_records = self.deletion_fence.records.read().await; let mut records = self.records.write().await; - let Some(index) = records - .iter() - .position(|record| record.is_claimable_at(now)) - else { + let Some(index) = records.iter().position(|record| { + record.is_claimable_at(now) + && fence_records + .get(&record.task.task_id) + .is_some_and(|fence| fence.deletion_job_id.is_none()) + }) else { return Ok(None); }; @@ -129,6 +186,13 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { now: time::OffsetDateTime, next_attempt_at: time::OffsetDateTime, ) -> Result, ApplicationError> { + let fence_records = self.deletion_fence.records.read().await; + if fence_records + .get(task_id) + .is_none_or(|fence| fence.deletion_job_id.is_some()) + { + return Ok(None); + } let mut records = self.records.write().await; let Some(record) = records.iter_mut().find(|record| { record.task.task_id == *task_id @@ -175,6 +239,13 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { message: "claimed task transition must be terminal".to_owned(), }); } + let mut fence_records = self.deletion_fence.records.write().await; + if fence_records + .get(&task.task_id) + .is_none_or(|fence| fence.deletion_job_id.is_some()) + { + return Ok(None); + } let mut records = self.records.write().await; let Some(record) = records.iter_mut().find(|record| { record.task.task_id == task.task_id @@ -197,6 +268,9 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { record.claim_token = None; record.claim_expires_at = None; record.next_attempt_at = None; + if let Some(fence) = fence_records.get_mut(&record.task.task_id) { + fence.entitlement_publication_claim_token = None; + } Ok(Some(record.task.clone())) } } @@ -219,18 +293,31 @@ impl ClaimableVerificationTask { #[cfg(test)] mod tests { - use std::{str::FromStr, sync::Arc}; + use std::{collections::BTreeMap, str::FromStr, sync::Arc}; use serde_json::json; use time::macros::datetime; - use locks_core::ids::{BundleId, CreatorPubky, PubkyLockResource, TaskId}; - use locks_core::lock_policy::VerifierType; + use locks_core::ids::{ + BundleId, CreatorPubky, GuardedResourceHash, LockId, PubkyLockResource, TaskId, + }; + use locks_core::lock_policy::{ + AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, + LockServerConfig, VerifierType, + }; use locks_core::verification::{Proof, SUBMITTED_PROOF_BUNDLE_VERSION, SubmittedProofBundle}; use super::InMemoryVerificationTaskClaimer; - use crate::application::models::{VerificationTaskRecord, VerificationTaskStatus}; - use crate::application::ports::{VerificationTaskClaimer, VerificationTaskRepository}; + use crate::application::errors::ApplicationError; + use crate::application::models::{ + ContentLockDeletionJob, PrepareForceDeletionResult, VerificationTaskRecord, + VerificationTaskStatus, + }; + use crate::application::ports::{ + ContentLockDeletionRepository, VerificationTaskClaimer, VerificationTaskRepository, + }; + use crate::infrastructure::memory::content_lock_deletions::InMemoryContentLockDeletionRepository; + use crate::infrastructure::memory::verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence; use crate::infrastructure::memory::verification_tasks::InMemoryVerificationTaskRepository; const LOCK_ID: &str = "000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG"; @@ -250,6 +337,219 @@ mod tests { ); } + #[tokio::test] + async fn publication_first_blocks_in_memory_deletion_admission() { + let job = deletion_job(); + let task = task_for_lock( + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d20", + VerificationTaskStatus::Pending, + &job.lock_id, + ); + let fence = Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks( + std::slice::from_ref(&task), + )); + let claimer = + InMemoryVerificationTaskClaimer::with_deletion_fence(vec![task], Arc::clone(&fence)); + let deletions = InMemoryContentLockDeletionRepository::with_verification_task_fence(fence); + let claimed = claimer + .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .await + .unwrap() + .unwrap(); + assert!( + claimer + .begin_claimed_entitlement_publication( + &claimed.task.task_id, + "worker-a", + &claimed.claim_token, + NOW, + ) + .await + .unwrap() + ); + + assert_eq!( + deletions.insert_job(job).await, + Err(ApplicationError::ContentLockDeletionInProgress) + ); + } + + #[tokio::test] + async fn deletion_first_blocks_in_memory_publication_and_terminal_transition() { + let job = deletion_job(); + let task = task_for_lock( + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d21", + VerificationTaskStatus::Pending, + &job.lock_id, + ); + let fence = Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks( + std::slice::from_ref(&task), + )); + let claimer = + InMemoryVerificationTaskClaimer::with_deletion_fence(vec![task], Arc::clone(&fence)); + let deletions = InMemoryContentLockDeletionRepository::with_verification_task_fence(fence); + let claimed = claimer + .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .await + .unwrap() + .unwrap(); + deletions.insert_job(job).await.unwrap(); + + assert!( + !claimer + .begin_claimed_entitlement_publication( + &claimed.task.task_id, + "worker-a", + &claimed.claim_token, + NOW, + ) + .await + .unwrap() + ); + let completed = claimed + .task + .transition_to(VerificationTaskStatus::Completed, NOW, None) + .unwrap(); + assert_eq!( + claimer + .persist_claimed_verification_task_transition( + completed, + "worker-a", + &claimed.claim_token, + NOW, + ) + .await + .unwrap(), + None + ); + } + + #[tokio::test] + async fn deletion_owned_publication_marker_blocks_in_memory_force_escalation() { + let job = deletion_job(); + let task = task_for_lock( + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d24", + VerificationTaskStatus::Pending, + &job.lock_id, + ); + let task_id = task.task_id; + let fence = Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks( + std::slice::from_ref(&task), + )); + let deletions = + InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::clone(&fence)); + deletions.insert_job(job.clone()).await.unwrap(); + { + let mut records = fence.records.write().await; + let record = records.get_mut(&task_id).unwrap(); + assert_eq!(record.deletion_job_id, Some(job.job_id)); + record.entitlement_publication_claim_token = Some(uuid::Uuid::new_v4()); + } + + assert_eq!( + deletions + .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .await + .unwrap(), + PrepareForceDeletionResult::PublicationInProgress + ); + assert!( + deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap() + .force_requested_at + .is_none() + ); + } + + #[tokio::test] + async fn retry_retains_publication_fence_until_reconciled_terminal_transition() { + let job = deletion_job(); + let task = task_for_lock( + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d22", + VerificationTaskStatus::Pending, + &job.lock_id, + ); + let fence = Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks( + std::slice::from_ref(&task), + )); + let claimer = + InMemoryVerificationTaskClaimer::with_deletion_fence(vec![task], Arc::clone(&fence)); + let deletions = InMemoryContentLockDeletionRepository::with_verification_task_fence(fence); + let first = claimer + .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .await + .unwrap() + .unwrap(); + assert!( + claimer + .begin_claimed_entitlement_publication( + &first.task.task_id, + "worker-a", + &first.claim_token, + NOW, + ) + .await + .unwrap() + ); + let retry_at = NOW + time::Duration::seconds(10); + claimer + .schedule_verification_task_retry( + &first.task.task_id, + "worker-a", + &first.claim_token, + NOW, + retry_at, + ) + .await + .unwrap() + .unwrap(); + + assert_eq!( + deletions.insert_job(job.clone()).await, + Err(ApplicationError::ContentLockDeletionInProgress) + ); + + let second = claimer + .claim_next_verification_task( + "worker-b", + retry_at, + retry_at + time::Duration::minutes(5), + ) + .await + .unwrap() + .unwrap(); + assert!( + claimer + .begin_claimed_entitlement_publication( + &second.task.task_id, + "worker-b", + &second.claim_token, + retry_at, + ) + .await + .unwrap() + ); + let completed = second + .task + .transition_to(VerificationTaskStatus::Completed, retry_at, None) + .unwrap(); + claimer + .persist_claimed_verification_task_transition( + completed, + "worker-b", + &second.claim_token, + retry_at, + ) + .await + .unwrap() + .unwrap(); + + deletions.insert_job(job).await.unwrap(); + } + #[tokio::test] async fn pending_task_can_be_claimed_and_transitions_to_in_progress() { let pending = task( @@ -643,6 +943,14 @@ mod tests { } fn task(task_id: &str, status: VerificationTaskStatus) -> VerificationTaskRecord { + task_for_lock(task_id, status, &LockId::from_str(LOCK_ID).unwrap()) + } + + fn task_for_lock( + task_id: &str, + status: VerificationTaskStatus, + lock_id: &LockId, + ) -> VerificationTaskRecord { VerificationTaskRecord { task_id: TaskId::from_str(task_id).unwrap(), creator: CreatorPubky::from_str("pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy").unwrap(), @@ -650,7 +958,7 @@ mod tests { version: SUBMITTED_PROOF_BUNDLE_VERSION, bundle_id: BundleId::from_str("000G40R40M30E209185GR38E1W").unwrap(), pubky_lock_resource: PubkyLockResource::from_str(&format!( - "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy/pub/locks.app/{LOCK_ID}.json" + "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy/pub/locks.app/{lock_id}.json" )) .unwrap(), reader_public_key: None, @@ -667,4 +975,36 @@ mod tests { failure_message: None, } } + + fn deletion_job() -> ContentLockDeletionJob { + ContentLockDeletionJob::new( + uuid::Uuid::new_v4(), + ContentLock { + version: CONTENT_LOCK_VERSION, + creator: CreatorPubky::from_str( + "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy", + ) + .unwrap(), + primary_resource: Some( + GuardedResource::new( + "/priv/locks.app/content/post.json".to_owned(), + GuardedResourceHash::from_bytes([7; 32]), + "application/json".to_owned(), + 42, + ) + .unwrap(), + ), + secondary_resources: BTreeMap::new(), + criteria: vec![], + lock_logic: LockLogic::All { criteria: vec![] }, + access_policy: AccessPolicy { + requested_credential_ttl_seconds: 900, + }, + lock_server: LockServerConfig { override_: None }, + created_at: datetime!(2026-08-12 04:00:00 UTC), + }, + datetime!(2026-08-12 05:00:00 UTC), + ) + .unwrap() + } } diff --git a/locks-service/src/infrastructure/memory/verification_task_deletion_fence.rs b/locks-service/src/infrastructure/memory/verification_task_deletion_fence.rs new file mode 100644 index 0000000..993cbc9 --- /dev/null +++ b/locks-service/src/infrastructure/memory/verification_task_deletion_fence.rs @@ -0,0 +1,52 @@ +use std::collections::HashMap; + +use locks_core::ids::{CreatorPubky, LockId, TaskId}; +use tokio::sync::RwLock; +use uuid::Uuid; + +use crate::application::models::VerificationTaskRecord; + +#[derive(Debug, Clone)] +pub(crate) struct InMemoryVerificationTaskFenceRecord { + pub(crate) creator: CreatorPubky, + pub(crate) lock_id: LockId, + pub(crate) entitlement_publication_claim_token: Option, + pub(crate) deletion_job_id: Option, +} + +/// Shared in-memory serialization state for verification publication and deletion admission. +#[derive(Debug, Default)] +pub struct InMemoryVerificationTaskDeletionFence { + pub(crate) records: RwLock>, +} + +impl InMemoryVerificationTaskDeletionFence { + pub fn new() -> Self { + Self::default() + } + + pub(crate) fn from_tasks(tasks: &[VerificationTaskRecord]) -> Self { + Self { + records: RwLock::new( + tasks + .iter() + .map(|task| { + ( + task.task_id, + InMemoryVerificationTaskFenceRecord { + creator: task.creator.clone(), + lock_id: task + .submitted_proof_bundle + .pubky_lock_resource + .lock_id() + .clone(), + entitlement_publication_claim_token: None, + deletion_job_id: None, + }, + ) + }) + .collect(), + ), + } + } +} diff --git a/locks-service/src/infrastructure/memory/verification_tasks.rs b/locks-service/src/infrastructure/memory/verification_tasks.rs index 45aa255..4413e2f 100644 --- a/locks-service/src/infrastructure/memory/verification_tasks.rs +++ b/locks-service/src/infrastructure/memory/verification_tasks.rs @@ -1,4 +1,4 @@ -use std::collections::HashMap; +use std::{collections::HashMap, sync::Arc}; use async_trait::async_trait; use tokio::sync::RwLock; @@ -8,11 +8,21 @@ use locks_core::ids::{BundleId, CreatorPubky, TaskId}; use crate::application::errors::ApplicationError; use crate::application::models::VerificationTaskRecord; use crate::application::ports::VerificationTaskRepository; +use crate::infrastructure::memory::verification_task_deletion_fence::{ + InMemoryVerificationTaskDeletionFence, InMemoryVerificationTaskFenceRecord, +}; /// In-memory verification task repository. -#[derive(Debug, Default)] +#[derive(Debug)] pub struct InMemoryVerificationTaskRepository { records: RwLock>, + deletion_fence: Arc, +} + +impl Default for InMemoryVerificationTaskRepository { + fn default() -> Self { + Self::with_deletion_fence(Arc::new(InMemoryVerificationTaskDeletionFence::new())) + } } impl InMemoryVerificationTaskRepository { @@ -20,6 +30,13 @@ impl InMemoryVerificationTaskRepository { pub fn new() -> Self { Self::default() } + + pub fn with_deletion_fence(deletion_fence: Arc) -> Self { + Self { + records: RwLock::new(HashMap::new()), + deletion_fence, + } + } } #[async_trait] @@ -28,6 +45,7 @@ impl VerificationTaskRepository for InMemoryVerificationTaskRepository { &self, task: VerificationTaskRecord, ) -> Result<(), ApplicationError> { + let mut fence_records = self.deletion_fence.records.write().await; let mut records = self.records.write().await; if records.contains_key(&task.task_id) || records.values().any(|existing| { @@ -40,6 +58,19 @@ impl VerificationTaskRepository for InMemoryVerificationTaskRepository { record: "verification_task", }); } + fence_records.insert( + task.task_id, + InMemoryVerificationTaskFenceRecord { + creator: task.creator.clone(), + lock_id: task + .submitted_proof_bundle + .pubky_lock_resource + .lock_id() + .clone(), + entitlement_publication_claim_token: None, + deletion_job_id: None, + }, + ); records.insert(task.task_id, task); Ok(()) } @@ -82,7 +113,9 @@ impl VerificationTaskRepository for InMemoryVerificationTaskRepository { } async fn delete_verification_task(&self, task_id: &TaskId) -> Result<(), ApplicationError> { + let mut fence_records = self.deletion_fence.records.write().await; self.records.write().await.remove(task_id); + fence_records.remove(task_id); Ok(()) } } diff --git a/locks-service/src/infrastructure/postgres/content_lock_deletions.rs b/locks-service/src/infrastructure/postgres/content_lock_deletions.rs index 1d39261..0a364a5 100644 --- a/locks-service/src/infrastructure/postgres/content_lock_deletions.rs +++ b/locks-service/src/infrastructure/postgres/content_lock_deletions.rs @@ -138,6 +138,33 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { if deletion_cutoff_exists { return Err(ApplicationError::ContentLockDeletionInProgress); } + sqlx::query( + "SELECT task_id FROM verification_tasks + WHERE creator = $1 + AND submitted_proof_bundle->>'pubky_lock_resource' = $2 + FOR UPDATE", + ) + .bind(job.creator.to_string()) + .bind(&lock_resource) + .fetch_all(&mut *transaction) + .await + .map_err(storage_error)?; + let publication_in_progress: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 FROM verification_tasks + WHERE creator = $1 + AND submitted_proof_bundle->>'pubky_lock_resource' = $2 + AND entitlement_publication_claim_token IS NOT NULL + )", + ) + .bind(job.creator.to_string()) + .bind(&lock_resource) + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + if publication_in_progress { + return Err(ApplicationError::ContentLockDeletionInProgress); + } sqlx::query( "INSERT INTO content_lock_deletion_jobs (job_id, creator, lock_id, frozen_content_lock, deletion_started_at, state, phase, @@ -160,8 +187,29 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { .map_err(map_insert_error)?; sqlx::query( "INSERT INTO content_lock_deletion_task_snapshot - (deletion_job_id, verification_task_id) - SELECT $1, task_id + (deletion_job_id, verification_task_id, creator, bundle_id, + pubky_lock_resource, criterion_id, status_at_cutoff, + paykit_admission_required) + SELECT $1, task_id, creator, bundle_id, + submitted_proof_bundle->>'pubky_lock_resource', + ( + SELECT proof->>'criterion_id' + FROM jsonb_array_elements(submitted_proof_bundle->'proofs') AS proof + WHERE proof->>'verifier_type' = 'paykit-payment' + LIMIT 1 + ), + status, + ( + EXISTS ( + SELECT 1 + FROM jsonb_array_elements(submitted_proof_bundle->'proofs') AS proof + WHERE proof->>'verifier_type' = 'paykit-payment' + ) + OR EXISTS ( + SELECT 1 FROM paykit_task_admissions AS admission + WHERE admission.verification_task_id = verification_tasks.task_id + ) + ) FROM verification_tasks WHERE creator = $2 AND submitted_proof_bundle->>'pubky_lock_resource' = $3", @@ -172,6 +220,23 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { .execute(&mut *transaction) .await .map_err(storage_error)?; + sqlx::query( + "UPDATE verification_tasks AS task + SET status = 'pending', started_at = NULL, claimed_by = NULL, + claim_token = NULL, claim_expires_at = NULL, + entitlement_publication_claim_token = NULL, deletion_job_id = $1, + next_attempt_at = NULL, + last_attempt_error = NULL, updated_at = $2 + FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.deletion_job_id = $1 + AND snapshot.verification_task_id = task.task_id + AND task.status IN ('pending', 'in_progress')", + ) + .bind(job.job_id) + .bind(job.deletion_started_at) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; transaction.commit().await.map_err(storage_error) } @@ -281,9 +346,25 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { "SELECT EXISTS ( SELECT 1 FROM content_lock_deletion_task_snapshot AS snapshot - JOIN paykit_task_admissions AS admission + LEFT JOIN paykit_task_admissions AS admission ON admission.verification_task_id = snapshot.verification_task_id - WHERE snapshot.deletion_job_id = $1 AND admission.ready = FALSE + WHERE snapshot.deletion_job_id = $1 + AND ( + snapshot.paykit_admission_required IS NULL + OR ( + snapshot.paykit_admission_required = TRUE + AND ( + snapshot.criterion_id IS NULL + OR admission.verification_task_id IS NULL + OR admission.ready = FALSE + OR admission.payment_in_hours IS NULL + OR admission.payment_in_hours <= 0 + OR admission.invoice_created_at IS NULL + OR admission.payment_deadline IS NULL + OR admission.invoice_created_at > admission.payment_deadline + ) + ) + ) )", ) .bind(job_id) @@ -297,6 +378,31 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { .to_owned(), }); } + sqlx::query( + "UPDATE content_lock_deletion_task_snapshot AS snapshot + SET payment_in_hours = admission.payment_in_hours, + invoice_created_at = admission.invoice_created_at, + payment_deadline = admission.payment_deadline, + resolved_status = CASE + WHEN snapshot.status_at_cutoff IN ('completed', 'failed', 'expired') + THEN snapshot.status_at_cutoff + ELSE NULL + END, + resolved_at = CASE + WHEN snapshot.status_at_cutoff IN ('completed', 'failed', 'expired') + THEN $2 + ELSE NULL + END + FROM paykit_task_admissions AS admission + WHERE snapshot.deletion_job_id = $1 + AND snapshot.paykit_admission_required = TRUE + AND admission.verification_task_id = snapshot.verification_task_id", + ) + .bind(job_id) + .bind(now) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; } let sql = format!( "UPDATE content_lock_deletion_jobs @@ -438,6 +544,21 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { job.state, ContentLockDeletionState::Queued | ContentLockDeletionState::Running ) { + let entitlement_publication_in_progress: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 FROM verification_tasks + WHERE deletion_job_id = $1 + AND entitlement_publication_claim_token IS NOT NULL + )", + ) + .bind(job.job_id) + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + if entitlement_publication_in_progress { + transaction.commit().await.map_err(storage_error)?; + return Ok(PrepareForceDeletionResult::PublicationInProgress); + } let sql = format!( "UPDATE content_lock_deletion_jobs SET force_requested_at = COALESCE(force_requested_at, $3), @@ -672,10 +793,13 @@ fn storage_display(error: impl std::fmt::Display) -> ApplicationError { #[cfg(test)] mod tests { - use std::{collections::BTreeMap, str::FromStr}; + use std::{collections::BTreeMap, str::FromStr, sync::Mutex}; + use async_trait::async_trait; use locks_core::{ - ids::{BundleId, CreatorPubky, GuardedResourceHash, PubkyLockResource, TaskId}, + ids::{ + BundleId, CreatorPubky, GuardedResourceHash, LockServerPubky, PubkyLockResource, TaskId, + }, lock_policy::{ AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, LockServerConfig, VerifierType, @@ -694,9 +818,20 @@ mod tests { ContentLockDeletionState, PrepareForceDeletionResult, VerificationTaskRecord, VerificationTaskStatus, }, - ports::{ContentLockDeletionRepository, VerificationTaskRepository}, + ports::{ + Clock, ContentLockDeletionRepository, EntitlementRepository, + PaymentDrainCleanupToken, PaymentDrainClient, PaymentDrainClientError, + PaymentDrainRepository, PaymentDrainStatus, PaymentDrainSummary, + PaymentDrainTerminalTransition, PaymentRequestState, PaymentRequestStatus, + PaymentState, VerificationTaskClaimer, VerificationTaskRepository, + }, + use_cases::drain_lock_payments::DrainLockPaymentsUseCase, + }, + infrastructure::memory::entitlements::InMemoryEntitlementRepository, + infrastructure::postgres::{ + PostgresPaymentDrainRepository, PostgresVerificationTaskClaimer, + PostgresVerificationTaskRepository, testing::TestDatabase, }, - infrastructure::postgres::{PostgresVerificationTaskRepository, testing::TestDatabase}, }; const CREATOR: &str = "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy"; @@ -879,8 +1014,10 @@ mod tests { let lock = content_lock(); let task = verification_task(&lock, BundleId::from_bytes([3; 16])); let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); - let admission = admissions.reserve(task.clone()).await.unwrap(); + let admission = admissions.reserve(task.clone(), 24).await.unwrap(); assert!(admission.requires_paykit); + assert_eq!(admission.payment_in, 24); + assert_eq!(admission.invoice_window, None); let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); repository @@ -918,7 +1055,7 @@ mod tests { let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); let mut external_calls = 0; let result = admissions - .reserve(verification_task(&lock, BundleId::from_bytes([4; 16]))) + .reserve(verification_task(&lock, BundleId::from_bytes([4; 16])), 24) .await; if result.is_ok() { external_calls += 1; @@ -938,9 +1075,10 @@ mod tests { let database = TestDatabase::create().await; let lock = content_lock(); - let task = verification_task(&lock, BundleId::from_bytes([6; 16])); + let mut task = verification_task(&lock, BundleId::from_bytes([6; 16])); + task.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::PaykitPayment; let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); - let admission = admissions.reserve(task).await.unwrap(); + let admission = admissions.reserve(task.clone(), 24).await.unwrap(); let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); repository .insert_job(ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap()) @@ -974,7 +1112,22 @@ mod tests { assert_eq!(still_running.state, ContentLockDeletionState::Running); assert_eq!(still_running.phase, ContentLockDeletionPhase::Withdraw); - admissions.mark_ready(&admission.task).await.unwrap(); + let invoice_window = crate::infrastructure::postgres::PaykitInvoiceWindow { + invoice_created_at: datetime!(2026-08-12 05:01:00 UTC), + payment_deadline: datetime!(2026-08-13 05:01:00 UTC), + }; + admissions + .mark_ready(&admission.task, invoice_window) + .await + .unwrap(); + let replay = admissions + .find_existing(&task.submitted_proof_bundle) + .await + .unwrap() + .unwrap(); + assert!(!replay.requires_paykit); + assert_eq!(replay.payment_in, 24); + assert_eq!(replay.invoice_window, Some(invoice_window)); let advanced = repository .advance_phase( claim.job.job_id, @@ -991,6 +1144,56 @@ mod tests { database.cleanup().await; } + #[tokio::test] + async fn payment_drain_phase_rejects_snapshotted_legacy_admission_without_invoice_window() { + use crate::infrastructure::postgres::verification_tasks::PostgresVerificationTaskRepository; + + let database = TestDatabase::create().await; + let lock = content_lock(); + let task = verification_task(&lock, BundleId::from_bytes([7; 16])); + PostgresVerificationTaskRepository::new(database.pool().clone()) + .insert_verification_task(task.clone()) + .await + .unwrap(); + sqlx::query( + "INSERT INTO paykit_task_admissions + (verification_task_id, ready, ready_at) + VALUES ($1::uuid, TRUE, now())", + ) + .bind(task.task_id.to_string()) + .execute(database.pool()) + .await + .unwrap(); + + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + repository + .insert_job(ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap()) + .await + .unwrap(); + let claim = repository + .claim_next("worker", NOW, NOW + time::Duration::seconds(60)) + .await + .unwrap() + .unwrap(); + + let result = repository + .advance_phase( + claim.job.job_id, + "worker", + claim.claim_token, + NOW, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await; + assert!(matches!( + result, + Err(ApplicationError::InvalidContentLockDeletionState { message }) + if message == "payment drain cannot start before reserved Paykit admissions are ready" + )); + + database.cleanup().await; + } + #[tokio::test] async fn paykit_admission_insert_failure_rolls_back_verification_task() { use crate::infrastructure::postgres::PostgresPaykitTaskAdmissionRepository; @@ -1017,7 +1220,7 @@ mod tests { let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); let task = verification_task(&content_lock(), BundleId::from_bytes([5; 16])); - assert!(admissions.reserve(task).await.is_err()); + assert!(admissions.reserve(task, 24).await.is_err()); let task_count = sqlx::query_scalar::<_, i64>("SELECT COUNT(*) FROM verification_tasks") .fetch_one(database.pool()) .await @@ -1279,6 +1482,466 @@ mod tests { database.cleanup().await; } + #[tokio::test] + async fn deletion_lease_takes_exclusive_ownership_of_snapshotted_paykit_obligation() { + use crate::infrastructure::postgres::{ + PaykitInvoiceWindow, PostgresPaykitTaskAdmissionRepository, + }; + + let database = TestDatabase::create().await; + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let drains = PostgresPaymentDrainRepository::new(database.pool().clone()); + let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); + let ordinary = PostgresVerificationTaskClaimer::new(database.pool().clone()); + let lock = content_lock(); + let mut task = verification_task(&lock, BundleId::from_bytes([9; 16])); + task.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::PaykitPayment; + admissions.reserve(task.clone(), 24).await.unwrap(); + let window = PaykitInvoiceWindow { + invoice_created_at: NOW, + payment_deadline: NOW + time::Duration::hours(24), + }; + admissions.mark_ready(&task, window).await.unwrap(); + + let ordinary_claim = ordinary + .claim_next_verification_task("ordinary", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + + assert!( + !ordinary + .begin_claimed_entitlement_publication( + &ordinary_claim.task.task_id, + "ordinary", + &ordinary_claim.claim_token, + NOW, + ) + .await + .unwrap() + ); + + let ordinary_completed = ordinary_claim + .task + .clone() + .transition_to(VerificationTaskStatus::Completed, NOW, None) + .unwrap(); + assert!( + ordinary + .persist_claimed_verification_task_transition( + ordinary_completed, + "ordinary", + &ordinary_claim.claim_token, + NOW, + ) + .await + .unwrap() + .is_none() + ); + + let withdraw_claim = deletions + .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + deletions + .advance_phase( + job.job_id, + "deletion", + withdraw_claim.claim_token, + NOW, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await + .unwrap() + .unwrap(); + let start_claim = deletions + .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + let token = + PaymentDrainCleanupToken::parse("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA").unwrap(); + let summary = PaymentDrainSummary { + status: PaymentDrainStatus::Active, + accepted_count: 1, + terminal_count: 0, + cancellation_enqueued_count: 0, + cleanup_token: token, + }; + assert!( + drains + .store_payment_drain( + job.job_id, + "deletion", + start_claim.claim_token, + NOW, + &summary, + ) + .await + .unwrap() + ); + assert!( + drains + .store_payment_drain( + job.job_id, + "deletion", + start_claim.claim_token, + NOW, + &summary, + ) + .await + .unwrap() + ); + let divergent = PaymentDrainSummary { + accepted_count: 2, + ..summary.clone() + }; + assert!( + drains + .store_payment_drain( + job.job_id, + "deletion", + start_claim.claim_token, + NOW, + &divergent, + ) + .await + .is_err() + ); + assert_eq!( + drains.get_payment_drain(job.job_id).await.unwrap(), + Some(summary) + ); + let obligations = drains.list_obligations(job.job_id).await.unwrap(); + assert_eq!(obligations.len(), 1); + assert_eq!(obligations[0].task_id, task.task_id); + assert_eq!(obligations[0].invoice_created_at, window.invoice_created_at); + assert_eq!(obligations[0].payment_deadline, window.payment_deadline); + assert!(!drains.all_obligations_terminal(job.job_id).await.unwrap()); + + deletions + .advance_phase( + job.job_id, + "deletion", + start_claim.claim_token, + NOW, + ContentLockDeletionPhase::DrainPayments, + ) + .await + .unwrap() + .unwrap(); + let drain_claim = deletions + .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + let publication_token = drains + .begin_entitlement_publication( + job.job_id, + "deletion", + drain_claim.claim_token, + NOW, + &task.task_id, + ) + .await + .unwrap() + .unwrap(); + assert_eq!( + drains + .begin_entitlement_publication( + job.job_id, + "deletion", + drain_claim.claim_token, + NOW, + &task.task_id, + ) + .await + .unwrap(), + Some(publication_token) + ); + assert_eq!( + deletions + .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .await + .unwrap(), + PrepareForceDeletionResult::PublicationInProgress + ); + assert!( + !drains + .persist_terminal_obligation( + job.job_id, + "deletion", + drain_claim.claim_token, + NOW, + &task.task_id, + PaymentDrainTerminalTransition { + status: VerificationTaskStatus::Completed, + entitlement_publication_token: Some(Uuid::new_v4()), + }, + ) + .await + .unwrap() + ); + assert!( + drains + .persist_terminal_obligation( + job.job_id, + "deletion", + drain_claim.claim_token, + NOW, + &task.task_id, + PaymentDrainTerminalTransition { + status: VerificationTaskStatus::Completed, + entitlement_publication_token: Some(publication_token), + }, + ) + .await + .unwrap() + ); + assert!(drains.all_obligations_terminal(job.job_id).await.unwrap()); + let retained_marker: Option = sqlx::query_scalar( + "SELECT entitlement_publication_claim_token FROM verification_tasks WHERE task_id = $1", + ) + .bind(task.task_id.as_uuid()) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(retained_marker, None); + + database.cleanup().await; + } + + #[tokio::test] + async fn entitlement_publication_fence_committed_first_blocks_deletion_cutoff() { + let database = TestDatabase::create().await; + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); + let ordinary = PostgresVerificationTaskClaimer::new(database.pool().clone()); + let lock = content_lock(); + let task = verification_task(&lock, BundleId::from_bytes([11; 16])); + tasks.insert_verification_task(task).await.unwrap(); + let claim = ordinary + .claim_next_verification_task("ordinary", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + assert!( + ordinary + .begin_claimed_entitlement_publication( + &claim.task.task_id, + "ordinary", + &claim.claim_token, + NOW, + ) + .await + .unwrap() + ); + + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + assert_eq!( + deletions.insert_job(job.clone()).await, + Err(ApplicationError::ContentLockDeletionInProgress) + ); + assert!( + deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .is_none() + ); + + database.cleanup().await; + } + + #[tokio::test] + async fn completed_paykit_aggregate_waits_for_locks_confirmation_and_local_terminal_state() { + use crate::infrastructure::postgres::{ + PaykitInvoiceWindow, PostgresPaykitTaskAdmissionRepository, + }; + + let database = TestDatabase::create().await; + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let drains = PostgresPaymentDrainRepository::new(database.pool().clone()); + let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); + let entitlements = InMemoryEntitlementRepository::new(); + let lock = content_lock(); + let mut task = verification_task(&lock, BundleId::from_bytes([10; 16])); + task.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::PaykitPayment; + admissions.reserve(task.clone(), 24).await.unwrap(); + let window = PaykitInvoiceWindow { + invoice_created_at: NOW, + payment_deadline: NOW + time::Duration::hours(24), + }; + admissions.mark_ready(&task, window).await.unwrap(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let withdraw = deletions + .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + deletions + .advance_phase( + job.job_id, + "deletion", + withdraw.claim_token, + NOW, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await + .unwrap() + .unwrap(); + let start = deletions + .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + let token = + PaymentDrainCleanupToken::parse("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA").unwrap(); + let initial_summary = PaymentDrainSummary { + status: PaymentDrainStatus::Active, + accepted_count: 1, + terminal_count: 0, + cancellation_enqueued_count: 0, + cleanup_token: token.clone(), + }; + drains + .store_payment_drain( + job.job_id, + "deletion", + start.claim_token, + NOW, + &initial_summary, + ) + .await + .unwrap(); + deletions + .advance_phase( + job.job_id, + "deletion", + start.claim_token, + NOW, + ContentLockDeletionPhase::DrainPayments, + ) + .await + .unwrap() + .unwrap(); + let claim = deletions + .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + let paykit = MutablePaymentDrainClient { + summary: PaymentDrainSummary { + status: PaymentDrainStatus::Completed, + accepted_count: 0, + terminal_count: 1, + cancellation_enqueued_count: 0, + cleanup_token: token, + }, + status: Mutex::new(PaymentRequestStatus { + request_state: PaymentRequestState::Accepted, + payment_state: PaymentState::Detected, + invoice_created_at: window.invoice_created_at, + payment_deadline: window.payment_deadline, + confirmations: 0, + amount_matched: true, + }), + }; + let clock = FixedClock(NOW); + let use_case = DrainLockPaymentsUseCase::new( + &deletions, + &drains, + &paykit, + &entitlements, + &clock, + LockServerPubky::from_str("pubky7ir1ttte48bcp4zjychjyscicrwi1j34mtt91ptsafdbjmr8g9eo") + .unwrap(), + 6, + ); + assert!( + !use_case + .execute_claimed(claim.clone(), "deletion") + .await + .unwrap() + ); + assert!(!drains.all_obligations_terminal(job.job_id).await.unwrap()); + assert!( + entitlements + .get_verified_proof_bundle(&task.creator, &task.submitted_proof_bundle.bundle_id,) + .await + .unwrap() + .is_none() + ); + + { + let mut status = paykit.status.lock().unwrap(); + status.payment_state = PaymentState::Confirmed; + status.confirmations = 6; + } + assert!(use_case.execute_claimed(claim, "deletion").await.unwrap()); + assert!(drains.all_obligations_terminal(job.job_id).await.unwrap()); + assert!( + entitlements + .get_verified_proof_bundle(&task.creator, &task.submitted_proof_bundle.bundle_id,) + .await + .unwrap() + .is_some() + ); + assert_eq!( + deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap() + .phase, + ContentLockDeletionPhase::DrainExistingCredentials + ); + + database.cleanup().await; + } + + struct FixedClock(time::OffsetDateTime); + + impl Clock for FixedClock { + fn now(&self) -> time::OffsetDateTime { + self.0 + } + } + + struct MutablePaymentDrainClient { + summary: PaymentDrainSummary, + status: Mutex, + } + + #[async_trait] + impl PaymentDrainClient for MutablePaymentDrainClient { + async fn start_payment_drain( + &self, + _lock_resource: &PubkyLockResource, + ) -> Result { + Ok(self.summary.clone()) + } + + async fn lookup_payment_drain( + &self, + _lock_resource: &PubkyLockResource, + ) -> Result, PaymentDrainClientError> { + Ok(Some(self.summary.clone())) + } + + async fn payment_request_status( + &self, + _creator: &CreatorPubky, + _bundle_id: &BundleId, + ) -> Result, PaymentDrainClientError> { + Ok(Some(*self.status.lock().unwrap())) + } + } + #[tokio::test] async fn read_rejects_corrupt_frozen_manifest_identity() { let database = TestDatabase::create().await; diff --git a/locks-service/src/infrastructure/postgres/migrations.rs b/locks-service/src/infrastructure/postgres/migrations.rs index db6b1b7..5c7e599 100644 --- a/locks-service/src/infrastructure/postgres/migrations.rs +++ b/locks-service/src/infrastructure/postgres/migrations.rs @@ -55,6 +55,24 @@ mod tests { assert_table_exists(&mut connection, "content_lock_publication_intents").await; assert_table_exists(&mut connection, "content_lock_deletion_task_snapshot").await; assert_table_exists(&mut connection, "paykit_task_admissions").await; + assert_column_exists( + &mut connection, + "paykit_task_admissions", + "payment_in_hours", + ) + .await; + assert_column_exists( + &mut connection, + "paykit_task_admissions", + "invoice_created_at", + ) + .await; + assert_column_exists( + &mut connection, + "paykit_task_admissions", + "payment_deadline", + ) + .await; assert_column_exists(&mut connection, "verification_tasks", "creator").await; assert_column_exists(&mut connection, "verification_tasks", "bundle_id").await; assert_column_exists(&mut connection, "verification_tasks", "next_attempt_at").await; diff --git a/locks-service/src/infrastructure/postgres/mod.rs b/locks-service/src/infrastructure/postgres/mod.rs index 43c6845..a347b3f 100644 --- a/locks-service/src/infrastructure/postgres/mod.rs +++ b/locks-service/src/infrastructure/postgres/mod.rs @@ -14,8 +14,9 @@ pub mod creator_connect_flows; pub mod errors; pub mod frontend_sessions; pub mod migrations; +pub mod payment_drains; mod proof_admission; -pub use proof_admission::PostgresPaykitTaskAdmissionRepository; +pub use proof_admission::{PaykitInvoiceWindow, PostgresPaykitTaskAdmissionRepository}; #[cfg(test)] pub(crate) mod testing; pub mod verification_task_claims; @@ -29,5 +30,6 @@ pub use creator_connect_flows::PostgresCreatorConnectFlowStore; pub use errors::PostgresError; pub use frontend_sessions::{PostgresFrontendSessionCodeStore, PostgresFrontendSessionStore}; pub use migrations::run_migrations; +pub use payment_drains::PostgresPaymentDrainRepository; pub use verification_task_claims::PostgresVerificationTaskClaimer; pub use verification_tasks::PostgresVerificationTaskRepository; diff --git a/locks-service/src/infrastructure/postgres/payment_drains.rs b/locks-service/src/infrastructure/postgres/payment_drains.rs new file mode 100644 index 0000000..a290b7d --- /dev/null +++ b/locks-service/src/infrastructure/postgres/payment_drains.rs @@ -0,0 +1,635 @@ +use std::str::FromStr; + +use async_trait::async_trait; +use locks_core::ids::{BundleId, CreatorPubky, PubkyLockResource, TaskId}; +use sqlx::{FromRow, PgPool}; +use time::OffsetDateTime; +use uuid::Uuid; + +use crate::application::errors::ApplicationError; +use crate::application::models::VerificationTaskStatus; +use crate::application::ports::{ + PaymentDrainCleanupToken, PaymentDrainObligation, PaymentDrainRepository, PaymentDrainStatus, + PaymentDrainSummary, PaymentDrainTerminalTransition, +}; + +#[derive(Debug, Clone)] +pub struct PostgresPaymentDrainRepository { + pool: PgPool, +} + +impl PostgresPaymentDrainRepository { + pub fn new(pool: PgPool) -> Self { + Self { pool } + } +} + +#[derive(FromRow)] +struct ObligationRow { + task_id: Uuid, + creator: String, + bundle_id: String, + pubky_lock_resource: String, + criterion_id: String, + invoice_created_at: OffsetDateTime, + payment_deadline: OffsetDateTime, + status: String, +} + +#[derive(FromRow)] +struct DrainRow { + status: String, + accepted_count: i64, + terminal_count: i64, + cancellation_enqueued_count: i64, + cleanup_token: String, +} + +#[async_trait] +impl PaymentDrainRepository for PostgresPaymentDrainRepository { + async fn store_payment_drain( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + summary: &PaymentDrainSummary, + ) -> Result { + let counts = summary_counts(summary)?; + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let owns_claim: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 FROM content_lock_deletion_jobs + WHERE job_id = $1 AND state = 'running' AND claimed_by = $2 + AND claim_token = $3 AND claim_expires_at >= $4 + AND phase = 'start_payment_drain' AND force_requested_at IS NULL + )", + ) + .bind(deletion_job_id) + .bind(worker_id) + .bind(claim_token) + .bind(now) + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + if !owns_claim { + transaction.rollback().await.map_err(storage_error)?; + return Ok(false); + } + let existing = sqlx::query_as::<_, DrainRow>( + "SELECT status, accepted_count, terminal_count, + cancellation_enqueued_count, cleanup_token + FROM content_lock_payment_drains + WHERE deletion_job_id = $1 FOR UPDATE", + ) + .bind(deletion_job_id) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)?; + if let Some(existing) = existing { + let existing = row_to_summary(existing)?; + if !valid_aggregate_progress(&existing, summary) { + return Err(invalid_deletion( + "Paykit payment drain aggregate changed for deletion job", + )); + } + sqlx::query( + "UPDATE content_lock_payment_drains + SET status = $2, accepted_count = $3, terminal_count = $4, updated_at = $5 + WHERE deletion_job_id = $1", + ) + .bind(deletion_job_id) + .bind(drain_status_to_database(summary.status)) + .bind(counts.0) + .bind(counts.1) + .bind(now) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + } else { + sqlx::query( + "INSERT INTO content_lock_payment_drains + (deletion_job_id, status, accepted_count, terminal_count, + cancellation_enqueued_count, cleanup_token, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $7)", + ) + .bind(deletion_job_id) + .bind(drain_status_to_database(summary.status)) + .bind(counts.0) + .bind(counts.1) + .bind(counts.2) + .bind(summary.cleanup_token.as_str()) + .bind(now) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + } + transaction.commit().await.map_err(storage_error)?; + Ok(true) + } + + async fn get_payment_drain( + &self, + deletion_job_id: Uuid, + ) -> Result, ApplicationError> { + sqlx::query_as::<_, DrainRow>( + "SELECT status, accepted_count, terminal_count, + cancellation_enqueued_count, cleanup_token + FROM content_lock_payment_drains WHERE deletion_job_id = $1", + ) + .bind(deletion_job_id) + .fetch_optional(&self.pool) + .await + .map_err(storage_error)? + .map(row_to_summary) + .transpose() + } + + async fn reconcile_payment_drain( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + summary: &PaymentDrainSummary, + ) -> Result { + let counts = summary_counts(summary)?; + let updated = sqlx::query( + "UPDATE content_lock_payment_drains AS drain + SET status = $5, accepted_count = $6, terminal_count = $7, updated_at = $4 + FROM content_lock_deletion_jobs AS deletion + WHERE deletion.job_id = $1 AND deletion.state = 'running' + AND deletion.claimed_by = $2 AND deletion.claim_token = $3 + AND deletion.claim_expires_at >= $4 AND deletion.phase = 'drain_payments' + AND deletion.force_requested_at IS NULL + AND drain.deletion_job_id = deletion.job_id + AND drain.cleanup_token = $8 + AND drain.cancellation_enqueued_count = $9 + AND drain.accepted_count >= $6 + AND drain.terminal_count <= $7 + AND drain.accepted_count - $6 = $7 - drain.terminal_count + AND NOT (drain.status = 'completed' AND $5 <> 'completed') + AND (($5 = 'completed' AND $6 = 0) OR ($5 = 'active' AND $6 > 0))", + ) + .bind(deletion_job_id) + .bind(worker_id) + .bind(claim_token) + .bind(now) + .bind(drain_status_to_database(summary.status)) + .bind(counts.0) + .bind(counts.1) + .bind(summary.cleanup_token.as_str()) + .bind(counts.2) + .execute(&self.pool) + .await + .map_err(storage_error)?; + Ok(updated.rows_affected() == 1) + } + + async fn list_obligations( + &self, + deletion_job_id: Uuid, + ) -> Result, ApplicationError> { + let rows = sqlx::query_as::<_, ObligationRow>( + "SELECT snapshot.verification_task_id AS task_id, snapshot.creator, + snapshot.bundle_id, snapshot.pubky_lock_resource, + snapshot.criterion_id, + snapshot.invoice_created_at, snapshot.payment_deadline, + COALESCE(snapshot.resolved_status, snapshot.status_at_cutoff) AS status + FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.deletion_job_id = $1 + AND snapshot.paykit_admission_required = TRUE + ORDER BY snapshot.verification_task_id", + ) + .bind(deletion_job_id) + .fetch_all(&self.pool) + .await + .map_err(storage_error)?; + rows.into_iter().map(row_to_obligation).collect() + } + + async fn begin_entitlement_publication( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + task_id: &TaskId, + ) -> Result, ApplicationError> { + let publication_token = Uuid::new_v4(); + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let owns_claim: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 FROM content_lock_deletion_jobs + WHERE job_id = $1 AND state = 'running' AND claimed_by = $2 + AND claim_token = $3 AND claim_expires_at >= $4 + AND phase = 'drain_payments' AND force_requested_at IS NULL + FOR UPDATE + )", + ) + .bind(deletion_job_id) + .bind(worker_id) + .bind(claim_token) + .bind(now) + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + if !owns_claim { + transaction.rollback().await.map_err(storage_error)?; + return Ok(None); + } + let admitted = sqlx::query_scalar( + "UPDATE verification_tasks AS task + SET entitlement_publication_claim_token = + COALESCE(task.entitlement_publication_claim_token, $6), + updated_at = $4 + FROM content_lock_deletion_jobs AS deletion, + content_lock_deletion_task_snapshot AS snapshot + WHERE deletion.job_id = $1 AND deletion.state = 'running' + AND deletion.claimed_by = $2 AND deletion.claim_token = $3 + AND deletion.claim_expires_at >= $4 AND deletion.phase = 'drain_payments' + AND deletion.force_requested_at IS NULL + AND snapshot.deletion_job_id = deletion.job_id + AND snapshot.verification_task_id = task.task_id + AND snapshot.paykit_admission_required = TRUE + AND snapshot.resolved_status IS NULL + AND task.task_id = $5::uuid + AND task.deletion_job_id = deletion.job_id + AND task.status IN ('pending', 'in_progress') + RETURNING task.entitlement_publication_claim_token", + ) + .bind(deletion_job_id) + .bind(worker_id) + .bind(claim_token) + .bind(now) + .bind(task_id.to_string()) + .bind(publication_token) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)?; + transaction.commit().await.map_err(storage_error)?; + Ok(admitted) + } + + async fn persist_terminal_obligation( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + task_id: &TaskId, + transition: PaymentDrainTerminalTransition, + ) -> Result { + let PaymentDrainTerminalTransition { + status, + entitlement_publication_token, + } = transition; + if !matches!( + status, + VerificationTaskStatus::Completed | VerificationTaskStatus::Expired + ) { + return Err(ApplicationError::InvalidVerificationTaskState { + message: "payment drain transition must be completed or expired".to_owned(), + }); + } + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let updated = sqlx::query( + "UPDATE verification_tasks AS task + SET status = $6, started_at = COALESCE(started_at, $5), completed_at = $5, + failure_message = NULL, claimed_by = NULL, claim_token = NULL, + claim_expires_at = NULL, next_attempt_at = NULL, + last_attempt_error = NULL, entitlement_publication_claim_token = NULL, + updated_at = $5 + FROM content_lock_deletion_jobs AS deletion, + content_lock_deletion_task_snapshot AS snapshot + WHERE deletion.job_id = $1 AND deletion.state = 'running' + AND deletion.claimed_by = $2 AND deletion.claim_token = $3 + AND deletion.claim_expires_at >= $5 AND deletion.phase = 'drain_payments' + AND deletion.force_requested_at IS NULL + AND snapshot.deletion_job_id = deletion.job_id + AND snapshot.verification_task_id = task.task_id + AND snapshot.paykit_admission_required = TRUE + AND snapshot.resolved_status IS NULL + AND task.task_id = $4::uuid + AND task.status IN ('pending', 'in_progress') + AND task.entitlement_publication_claim_token IS NOT DISTINCT FROM $7", + ) + .bind(deletion_job_id) + .bind(worker_id) + .bind(claim_token) + .bind(task_id.to_string()) + .bind(now) + .bind(status_to_database(status)) + .bind(entitlement_publication_token) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + if updated.rows_affected() != 1 { + transaction.rollback().await.map_err(storage_error)?; + return Ok(false); + } + let resolved = sqlx::query( + "UPDATE content_lock_deletion_task_snapshot + SET resolved_status = $3, resolved_at = $4 + WHERE deletion_job_id = $1 AND verification_task_id = $2::uuid + AND resolved_status IS NULL", + ) + .bind(deletion_job_id) + .bind(task_id.to_string()) + .bind(status_to_database(status)) + .bind(now) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + if resolved.rows_affected() != 1 { + return Err(invalid_deletion( + "payment drain snapshot resolution lost its fence", + )); + } + transaction.commit().await.map_err(storage_error)?; + Ok(true) + } + + async fn all_obligations_terminal( + &self, + deletion_job_id: Uuid, + ) -> Result { + sqlx::query_scalar( + "SELECT NOT EXISTS ( + SELECT 1 FROM content_lock_deletion_task_snapshot + WHERE deletion_job_id = $1 + AND paykit_admission_required = TRUE + AND COALESCE(resolved_status, status_at_cutoff) + NOT IN ('completed', 'failed', 'expired') + )", + ) + .bind(deletion_job_id) + .fetch_one(&self.pool) + .await + .map_err(storage_error) + } +} + +fn row_to_obligation(row: ObligationRow) -> Result { + Ok(PaymentDrainObligation { + task_id: TaskId::from_str(&row.task_id.to_string()).map_err(storage_display)?, + creator: CreatorPubky::from_str(&row.creator).map_err(storage_display)?, + bundle_id: BundleId::from_str(&row.bundle_id).map_err(storage_display)?, + lock_resource: PubkyLockResource::from_str(&row.pubky_lock_resource) + .map_err(storage_display)?, + criterion_id: row.criterion_id, + invoice_created_at: row.invoice_created_at, + payment_deadline: row.payment_deadline, + status: status_from_database(&row.status)?, + }) +} + +fn row_to_summary(row: DrainRow) -> Result { + Ok(PaymentDrainSummary { + status: match row.status.as_str() { + "active" => PaymentDrainStatus::Active, + "completed" => PaymentDrainStatus::Completed, + _ => { + return Err(invalid_deletion( + "persisted Paykit payment drain status is invalid", + )); + } + }, + accepted_count: u64::try_from(row.accepted_count).map_err(storage_display)?, + terminal_count: u64::try_from(row.terminal_count).map_err(storage_display)?, + cancellation_enqueued_count: u64::try_from(row.cancellation_enqueued_count) + .map_err(storage_display)?, + cleanup_token: PaymentDrainCleanupToken::parse(&row.cleanup_token) + .ok_or_else(|| invalid_deletion("persisted Paykit cleanup token is invalid"))?, + }) +} + +fn summary_counts(summary: &PaymentDrainSummary) -> Result<(i64, i64, i64), ApplicationError> { + Ok(( + i64::try_from(summary.accepted_count).map_err(storage_display)?, + i64::try_from(summary.terminal_count).map_err(storage_display)?, + i64::try_from(summary.cancellation_enqueued_count).map_err(storage_display)?, + )) +} + +fn valid_aggregate_progress(previous: &PaymentDrainSummary, current: &PaymentDrainSummary) -> bool { + let accepted_delta = previous.accepted_count.checked_sub(current.accepted_count); + let terminal_delta = current.terminal_count.checked_sub(previous.terminal_count); + previous.cleanup_token == current.cleanup_token + && previous.cancellation_enqueued_count == current.cancellation_enqueued_count + && accepted_delta.is_some() + && accepted_delta == terminal_delta + && !(previous.status == PaymentDrainStatus::Completed + && current.status != PaymentDrainStatus::Completed) + && ((current.status == PaymentDrainStatus::Completed && current.accepted_count == 0) + || (current.status == PaymentDrainStatus::Active && current.accepted_count > 0)) +} + +fn drain_status_to_database(status: PaymentDrainStatus) -> &'static str { + match status { + PaymentDrainStatus::Active => "active", + PaymentDrainStatus::Completed => "completed", + } +} + +fn status_from_database(value: &str) -> Result { + match value { + "pending" => Ok(VerificationTaskStatus::Pending), + "in_progress" => Ok(VerificationTaskStatus::InProgress), + "completed" => Ok(VerificationTaskStatus::Completed), + "failed" => Ok(VerificationTaskStatus::Failed), + "expired" => Ok(VerificationTaskStatus::Expired), + _ => Err(ApplicationError::InvalidVerificationTaskState { + message: "unknown snapshotted verification task status".to_owned(), + }), + } +} + +fn status_to_database(status: VerificationTaskStatus) -> &'static str { + match status { + VerificationTaskStatus::Completed => "completed", + VerificationTaskStatus::Expired => "expired", + _ => unreachable!("validated terminal payment drain status"), + } +} + +fn invalid_deletion(message: &str) -> ApplicationError { + ApplicationError::InvalidContentLockDeletionState { + message: message.to_owned(), + } +} + +fn storage_error(error: sqlx::Error) -> ApplicationError { + storage_display(error) +} + +fn storage_display(error: impl std::fmt::Display) -> ApplicationError { + ApplicationError::Storage { + message: error.to_string(), + } +} + +#[cfg(test)] +mod tests { + use base64::Engine; + use base64::engine::general_purpose::URL_SAFE_NO_PAD; + use time::OffsetDateTime; + use uuid::Uuid; + + use crate::application::ports::{ + PaymentDrainCleanupToken, PaymentDrainRepository, PaymentDrainStatus, PaymentDrainSummary, + }; + use crate::infrastructure::postgres::testing::TestDatabase; + + use super::PostgresPaymentDrainRepository; + + #[tokio::test] + async fn payment_drain_migration_creates_durable_token_table() { + let database = TestDatabase::create().await; + let exists: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 FROM information_schema.tables + WHERE table_schema = current_schema() + AND table_name = 'content_lock_payment_drains' + )", + ) + .fetch_one(database.pool()) + .await + .unwrap(); + assert!(exists); + database.cleanup().await; + } + + #[tokio::test] + async fn reconciliation_persists_only_monotonic_aggregate_progress_under_live_claim() { + let database = TestDatabase::create().await; + let repository = PostgresPaymentDrainRepository::new(database.pool().clone()); + let job_id = Uuid::new_v4(); + let claim_token = Uuid::new_v4(); + let now = OffsetDateTime::now_utc(); + sqlx::query( + "INSERT INTO content_lock_deletion_jobs + (job_id, creator, lock_id, frozen_content_lock, deletion_started_at, + state, phase, claimed_by, claim_token, claim_expires_at) + VALUES ($1, 'creator', 'lock', '{}'::jsonb, $2, + 'running', 'drain_payments', 'worker', $3, $4)", + ) + .bind(job_id) + .bind(now) + .bind(claim_token) + .bind(now + time::Duration::minutes(5)) + .execute(database.pool()) + .await + .unwrap(); + let token = PaymentDrainCleanupToken::parse(&URL_SAFE_NO_PAD.encode([7_u8; 32])).unwrap(); + sqlx::query( + "INSERT INTO content_lock_payment_drains + (deletion_job_id, status, accepted_count, terminal_count, + cancellation_enqueued_count, cleanup_token, created_at, updated_at) + VALUES ($1, 'active', 2, 3, 1, $2, $3, $3)", + ) + .bind(job_id) + .bind(token.as_str()) + .bind(now) + .execute(database.pool()) + .await + .unwrap(); + + let completed = PaymentDrainSummary { + status: PaymentDrainStatus::Completed, + accepted_count: 0, + terminal_count: 5, + cancellation_enqueued_count: 1, + cleanup_token: token.clone(), + }; + assert!( + repository + .reconcile_payment_drain(job_id, "worker", claim_token, now, &completed) + .await + .unwrap() + ); + assert_eq!( + repository.get_payment_drain(job_id).await.unwrap(), + Some(completed) + ); + + let divergent = PaymentDrainSummary { + status: PaymentDrainStatus::Completed, + accepted_count: 0, + terminal_count: 6, + cancellation_enqueued_count: 1, + cleanup_token: token, + }; + assert!( + !repository + .reconcile_payment_drain(job_id, "worker", claim_token, now, &divergent) + .await + .unwrap() + ); + assert!( + !repository + .reconcile_payment_drain(job_id, "worker", Uuid::new_v4(), now, &divergent) + .await + .unwrap() + ); + + database.cleanup().await; + } + + #[tokio::test] + async fn start_phase_replay_persists_monotonic_progress_after_crash_before_phase_advance() { + let database = TestDatabase::create().await; + let repository = PostgresPaymentDrainRepository::new(database.pool().clone()); + let job_id = Uuid::new_v4(); + let claim_token = Uuid::new_v4(); + let now = OffsetDateTime::now_utc(); + sqlx::query( + "INSERT INTO content_lock_deletion_jobs + (job_id, creator, lock_id, frozen_content_lock, deletion_started_at, + state, phase, claimed_by, claim_token, claim_expires_at) + VALUES ($1, 'creator', 'lock', '{}'::jsonb, $2, + 'running', 'start_payment_drain', 'worker', $3, $4)", + ) + .bind(job_id) + .bind(now) + .bind(claim_token) + .bind(now + time::Duration::minutes(5)) + .execute(database.pool()) + .await + .unwrap(); + let token = PaymentDrainCleanupToken::parse(&URL_SAFE_NO_PAD.encode([8_u8; 32])).unwrap(); + let active = PaymentDrainSummary { + status: PaymentDrainStatus::Active, + accepted_count: 1, + terminal_count: 0, + cancellation_enqueued_count: 0, + cleanup_token: token.clone(), + }; + assert!( + repository + .store_payment_drain(job_id, "worker", claim_token, now, &active) + .await + .unwrap() + ); + + let completed = PaymentDrainSummary { + status: PaymentDrainStatus::Completed, + accepted_count: 0, + terminal_count: 1, + cancellation_enqueued_count: 0, + cleanup_token: token, + }; + assert!( + repository + .store_payment_drain(job_id, "worker", claim_token, now, &completed) + .await + .unwrap() + ); + assert_eq!( + repository.get_payment_drain(job_id).await.unwrap(), + Some(completed) + ); + + database.cleanup().await; + } +} diff --git a/locks-service/src/infrastructure/postgres/proof_admission.rs b/locks-service/src/infrastructure/postgres/proof_admission.rs index 3e68ca8..79a7483 100644 --- a/locks-service/src/infrastructure/postgres/proof_admission.rs +++ b/locks-service/src/infrastructure/postgres/proof_admission.rs @@ -16,10 +16,21 @@ const PROOF_ADMISSION_LOCK_NAMESPACE: &str = "locks:proof-admission:v1"; pub struct PaykitTaskAdmission { /// The durable task associated with the public Bundle handle. pub task: VerificationTaskRecord, + /// Immutable whole-hour payment window sent to Paykit for exact replay. + pub payment_in: u64, + /// Immutable timestamps returned by Paykit once the reservation is ready. + pub invoice_window: Option, /// Whether the caller must create/reconcile the Paykit invoice before making the task claimable. pub requires_paykit: bool, } +/// Immutable Paykit invoice timestamps bound to one admitted verification task. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PaykitInvoiceWindow { + pub invoice_created_at: time::OffsetDateTime, + pub payment_deadline: time::OffsetDateTime, +} + /// PostgreSQL coordinator for durable persist-before-Paykit admission. #[derive(Debug, Clone)] pub struct PostgresPaykitTaskAdmissionRepository { @@ -39,7 +50,10 @@ impl PostgresPaykitTaskAdmissionRepository { ) -> Result, ApplicationError> { let sql = format!( "SELECT {VERIFICATION_TASK_ROW_COLUMNS}, - COALESCE(admission.ready, TRUE) AS paykit_ready + COALESCE(admission.ready, TRUE) AS paykit_ready, + admission.payment_in_hours, + admission.invoice_created_at, + admission.payment_deadline FROM verification_tasks AS task LEFT JOIN paykit_task_admissions AS admission ON admission.verification_task_id = task.task_id @@ -55,12 +69,16 @@ impl PostgresPaykitTaskAdmissionRepository { return Ok(None); }; let ready = existing.paykit_ready; + let payment_in = payment_in_from_database(existing.payment_in_hours)?; + let invoice_window = invoice_window_from_row(&existing, ready)?; let existing = row_to_task(existing.task)?; if existing.submitted_proof_bundle != *submitted { return Err(ApplicationError::VerificationTaskConflict); } Ok(Some(PaykitTaskAdmission { task: existing, + payment_in, + invoice_window, requires_paykit: !ready, })) } @@ -69,7 +87,9 @@ impl PostgresPaykitTaskAdmissionRepository { pub async fn reserve( &self, task: VerificationTaskRecord, + payment_in: u64, ) -> Result { + let payment_in_hours = payment_in_to_database(payment_in)?; let row = VerificationTaskWriteRow::try_from(&task)?; let lock_id = task.submitted_proof_bundle.pubky_lock_resource.lock_id(); let mut transaction = self.pool.begin().await.map_err(storage_error)?; @@ -77,7 +97,10 @@ impl PostgresPaykitTaskAdmissionRepository { let existing_sql = format!( "SELECT {VERIFICATION_TASK_ROW_COLUMNS}, - COALESCE(admission.ready, TRUE) AS paykit_ready + COALESCE(admission.ready, TRUE) AS paykit_ready, + admission.payment_in_hours, + admission.invoice_created_at, + admission.payment_deadline FROM verification_tasks AS task LEFT JOIN paykit_task_admissions AS admission ON admission.verification_task_id = task.task_id @@ -91,13 +114,19 @@ impl PostgresPaykitTaskAdmissionRepository { .map_err(storage_error)? { let ready = existing.paykit_ready; + let stored_payment_in = payment_in_from_database(existing.payment_in_hours)?; + let invoice_window = invoice_window_from_row(&existing, ready)?; let existing = row_to_task(existing.task)?; - if existing.submitted_proof_bundle != task.submitted_proof_bundle { + if existing.submitted_proof_bundle != task.submitted_proof_bundle + || stored_payment_in != payment_in + { return Err(ApplicationError::VerificationTaskConflict); } transaction.commit().await.map_err(storage_error)?; return Ok(PaykitTaskAdmission { task: existing, + payment_in: stored_payment_in, + invoice_window, requires_paykit: !ready, }); } @@ -118,10 +147,12 @@ impl PostgresPaykitTaskAdmissionRepository { insert_task(&mut transaction, row).await?; sqlx::query( - "INSERT INTO paykit_task_admissions (verification_task_id, ready) - VALUES ($1::uuid, FALSE)", + "INSERT INTO paykit_task_admissions + (verification_task_id, ready, payment_in_hours) + VALUES ($1::uuid, FALSE, $2)", ) .bind(task.task_id.to_string()) + .bind(payment_in_hours) .execute(&mut *transaction) .await .map_err(storage_error)?; @@ -129,25 +160,42 @@ impl PostgresPaykitTaskAdmissionRepository { Ok(PaykitTaskAdmission { task, + payment_in, + invoice_window: None, requires_paykit: true, }) } /// Makes a reserved task claimable after Paykit confirms invoice creation/replay. - pub async fn mark_ready(&self, task: &VerificationTaskRecord) -> Result<(), ApplicationError> { + pub async fn mark_ready( + &self, + task: &VerificationTaskRecord, + invoice_window: PaykitInvoiceWindow, + ) -> Result<(), ApplicationError> { + if invoice_window.payment_deadline < invoice_window.invoice_created_at { + return Err(ApplicationError::VerificationTaskConflict); + } let result = sqlx::query( "UPDATE paykit_task_admissions - SET ready = TRUE, ready_at = COALESCE(ready_at, now()) - WHERE verification_task_id = $1::uuid", + SET ready = TRUE, + ready_at = COALESCE(ready_at, now()), + invoice_created_at = COALESCE(invoice_created_at, $2), + payment_deadline = COALESCE(payment_deadline, $3) + WHERE verification_task_id = $1::uuid + AND ( + (ready = FALSE AND invoice_created_at IS NULL AND payment_deadline IS NULL) + OR + (ready = TRUE AND invoice_created_at = $2 AND payment_deadline = $3) + )", ) .bind(task.task_id.to_string()) + .bind(invoice_window.invoice_created_at) + .bind(invoice_window.payment_deadline) .execute(&self.pool) .await .map_err(storage_error)?; if result.rows_affected() == 0 { - return Err(ApplicationError::MissingRecord { - record: "paykit_task_admission", - }); + return Err(ApplicationError::VerificationTaskConflict); } Ok(()) } @@ -158,6 +206,45 @@ struct PaykitAdmissionRow { #[sqlx(flatten)] task: VerificationTaskRow, paykit_ready: bool, + payment_in_hours: Option, + invoice_created_at: Option, + payment_deadline: Option, +} + +fn payment_in_to_database(payment_in: u64) -> Result { + i64::try_from(payment_in) + .ok() + .filter(|value| *value > 0) + .ok_or(ApplicationError::VerificationTaskConflict) +} + +fn payment_in_from_database(payment_in: Option) -> Result { + payment_in + .and_then(|value| u64::try_from(value).ok()) + .filter(|value| *value > 0) + .ok_or(ApplicationError::Storage { + message: "invalid Paykit payment window stored in Postgres".to_owned(), + }) +} + +fn invoice_window_from_row( + row: &PaykitAdmissionRow, + ready: bool, +) -> Result, ApplicationError> { + match (ready, row.invoice_created_at, row.payment_deadline) { + (false, None, None) => Ok(None), + (true, Some(invoice_created_at), Some(payment_deadline)) + if invoice_created_at <= payment_deadline => + { + Ok(Some(PaykitInvoiceWindow { + invoice_created_at, + payment_deadline, + })) + } + _ => Err(ApplicationError::Storage { + message: "invalid Paykit invoice window stored in Postgres".to_owned(), + }), + } } async fn insert_task( diff --git a/locks-service/src/infrastructure/postgres/verification_task_claims.rs b/locks-service/src/infrastructure/postgres/verification_task_claims.rs index 53b9b7d..5ee74d3 100644 --- a/locks-service/src/infrastructure/postgres/verification_task_claims.rs +++ b/locks-service/src/infrastructure/postgres/verification_task_claims.rs @@ -26,6 +26,34 @@ impl PostgresVerificationTaskClaimer { #[async_trait] impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { + async fn begin_claimed_entitlement_publication( + &self, + task_id: &TaskId, + worker_id: &str, + claim_token: &uuid::Uuid, + now: time::OffsetDateTime, + ) -> Result { + let updated = sqlx::query( + "UPDATE verification_tasks + SET entitlement_publication_claim_token = $3, updated_at = $4 + WHERE task_id = $1::uuid AND status = 'in_progress' + AND claimed_by = $2 AND claim_token = $3 AND claim_expires_at >= $4 + AND deletion_job_id IS NULL + AND NOT EXISTS ( + SELECT 1 FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.verification_task_id = verification_tasks.task_id + )", + ) + .bind(task_id.to_string()) + .bind(worker_id) + .bind(claim_token) + .bind(now) + .execute(&self.pool) + .await + .map_err(storage_error)?; + Ok(updated.rows_affected() == 1) + } + async fn claim_next_verification_task( &self, worker_id: &str, @@ -50,10 +78,23 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { WHERE ((status = 'pending' AND (next_attempt_at IS NULL OR next_attempt_at <= $3)) OR (status = 'in_progress' AND claim_expires_at < $3)) + AND deletion_job_id IS NULL AND NOT EXISTS ( SELECT 1 FROM paykit_task_admissions WHERE verification_task_id = verification_tasks.task_id - AND ready = FALSE + AND ( + ready = FALSE + OR payment_in_hours IS NULL + OR payment_in_hours <= 0 + OR invoice_created_at IS NULL + OR payment_deadline IS NULL + OR invoice_created_at > payment_deadline + ) + ) + AND NOT EXISTS ( + SELECT 1 + FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.verification_task_id = verification_tasks.task_id ) AND creator = split_part(submitted_proof_bundle->>'pubky_lock_resource', '/', 1) AND bundle_id = submitted_proof_bundle->>'bundle_id' @@ -103,6 +144,12 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { AND claimed_by = $2 AND claim_token = $3 AND claim_expires_at >= $4 + AND deletion_job_id IS NULL + AND NOT EXISTS ( + SELECT 1 + FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.verification_task_id = verification_tasks.task_id + ) RETURNING {VERIFICATION_TASK_ROW_COLUMNS}" ); let row = sqlx::query_as::<_, VerificationTaskRow>(&sql) @@ -144,6 +191,7 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { claimed_by = NULL, claim_token = NULL, claim_expires_at = NULL, + entitlement_publication_claim_token = NULL, next_attempt_at = NULL, last_attempt_error = NULL, updated_at = $4 @@ -152,6 +200,12 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { AND claimed_by = $2 AND claim_token = $3 AND claim_expires_at >= $4 + AND deletion_job_id IS NULL + AND NOT EXISTS ( + SELECT 1 + FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.verification_task_id = verification_tasks.task_id + ) RETURNING {VERIFICATION_TASK_ROW_COLUMNS}" ); let row = sqlx::query_as::<_, VerificationTaskRow>(&sql) @@ -201,6 +255,172 @@ mod tests { const NOW: time::OffsetDateTime = datetime!(2026-05-29 12:10:00 UTC); const CLAIM_EXPIRES_AT: time::OffsetDateTime = datetime!(2026-05-29 12:15:00 UTC); + #[tokio::test] + async fn waiting_publication_update_rechecks_task_row_deletion_fence() { + let database = TestDatabase::create().await; + let repository = PostgresVerificationTaskRepository::new(database.pool().clone()); + let claimer = PostgresVerificationTaskClaimer::new(database.pool().clone()); + let pending = task( + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d13", + VerificationTaskStatus::Pending, + datetime!(2026-05-29 12:00:00 UTC), + ); + repository + .insert_verification_task(pending.clone()) + .await + .unwrap(); + let claim = claimer + .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .await + .unwrap() + .unwrap(); + + let mut transaction = database.pool().begin().await.unwrap(); + sqlx::query("SELECT task_id FROM verification_tasks WHERE task_id = $1 FOR UPDATE") + .bind(pending.task_id.as_uuid()) + .fetch_one(&mut *transaction) + .await + .unwrap(); + + let waiting_claimer = claimer.clone(); + let task_id = pending.task_id; + let claim_token = claim.claim_token; + let publication = tokio::spawn(async move { + waiting_claimer + .begin_claimed_entitlement_publication(&task_id, "worker-a", &claim_token, NOW) + .await + }); + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + assert!(!publication.is_finished()); + + let deletion_job_id = uuid::Uuid::new_v4(); + sqlx::query( + "INSERT INTO content_lock_deletion_jobs + (job_id, creator, lock_id, frozen_content_lock, deletion_started_at, state, phase) + VALUES ($1, $2, $3, '{}'::jsonb, $4, 'queued', 'withdraw')", + ) + .bind(deletion_job_id) + .bind(pending.creator.to_string()) + .bind(LOCK_ID) + .bind(NOW) + .execute(&mut *transaction) + .await + .unwrap(); + sqlx::query("UPDATE verification_tasks SET deletion_job_id = $1 WHERE task_id = $2") + .bind(deletion_job_id) + .bind(pending.task_id.as_uuid()) + .execute(&mut *transaction) + .await + .unwrap(); + transaction.commit().await.unwrap(); + + assert!(!publication.await.unwrap().unwrap()); + database.cleanup().await; + } + + #[tokio::test] + async fn retry_retains_publication_marker_until_reconciled_terminal_transition() { + let database = TestDatabase::create().await; + let repository = PostgresVerificationTaskRepository::new(database.pool().clone()); + let claimer = PostgresVerificationTaskClaimer::new(database.pool().clone()); + let pending = task( + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d17", + VerificationTaskStatus::Pending, + datetime!(2026-05-29 12:00:00 UTC), + ); + repository + .insert_verification_task(pending.clone()) + .await + .unwrap(); + let first = claimer + .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .await + .unwrap() + .unwrap(); + assert!( + claimer + .begin_claimed_entitlement_publication( + &pending.task_id, + "worker-a", + &first.claim_token, + NOW, + ) + .await + .unwrap() + ); + let retry_at = NOW + time::Duration::seconds(10); + claimer + .schedule_verification_task_retry( + &pending.task_id, + "worker-a", + &first.claim_token, + NOW, + retry_at, + ) + .await + .unwrap() + .unwrap(); + + let retained: Option = sqlx::query_scalar( + "SELECT entitlement_publication_claim_token + FROM verification_tasks + WHERE task_id = $1", + ) + .bind(pending.task_id.as_uuid()) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(retained, Some(first.claim_token)); + + let second = claimer + .claim_next_verification_task( + "worker-b", + retry_at, + retry_at + time::Duration::minutes(5), + ) + .await + .unwrap() + .unwrap(); + assert!( + claimer + .begin_claimed_entitlement_publication( + &pending.task_id, + "worker-b", + &second.claim_token, + retry_at, + ) + .await + .unwrap() + ); + let completed = second + .task + .transition_to(VerificationTaskStatus::Completed, retry_at, None) + .unwrap(); + claimer + .persist_claimed_verification_task_transition( + completed, + "worker-b", + &second.claim_token, + retry_at, + ) + .await + .unwrap() + .unwrap(); + + let cleared: Option = sqlx::query_scalar( + "SELECT entitlement_publication_claim_token + FROM verification_tasks + WHERE task_id = $1", + ) + .bind(pending.task_id.as_uuid()) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(cleared, None); + + database.cleanup().await; + } + #[tokio::test] async fn claims_oldest_pending_task_first() { let database = TestDatabase::create().await; @@ -248,7 +468,7 @@ mod tests { datetime!(2026-05-29 12:00:00 UTC), ); - let first = admissions.reserve(pending.clone()).await.unwrap(); + let first = admissions.reserve(pending.clone(), 24).await.unwrap(); assert!(first.requires_paykit); assert!( claimer @@ -258,14 +478,33 @@ mod tests { .is_none() ); - let replay = admissions.reserve(pending.clone()).await.unwrap(); + let replay = admissions.reserve(pending.clone(), 24).await.unwrap(); assert!(replay.requires_paykit); assert_eq!(replay.task, pending); - admissions.mark_ready(&pending).await.unwrap(); - let ready_replay = admissions.reserve(pending.clone()).await.unwrap(); + let invoice_window = crate::infrastructure::postgres::PaykitInvoiceWindow { + invoice_created_at: datetime!(2026-05-29 12:00:00 UTC), + payment_deadline: datetime!(2026-05-30 12:00:00 UTC), + }; + admissions + .mark_ready(&pending, invoice_window) + .await + .unwrap(); + let divergent_window = crate::infrastructure::postgres::PaykitInvoiceWindow { + invoice_created_at: datetime!(2026-05-29 12:00:01 UTC), + payment_deadline: datetime!(2026-05-30 12:00:01 UTC), + }; + assert!( + admissions + .mark_ready(&pending, divergent_window) + .await + .is_err() + ); + let ready_replay = admissions.reserve(pending.clone(), 24).await.unwrap(); assert!(!ready_replay.requires_paykit); assert_eq!(ready_replay.task, pending); + assert_eq!(ready_replay.payment_in, 24); + assert_eq!(ready_replay.invoice_window, Some(invoice_window)); assert_eq!( claimer .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) @@ -280,6 +519,50 @@ mod tests { database.cleanup().await; } + #[tokio::test] + async fn legacy_paykit_admission_without_authoritative_window_fails_closed() { + use crate::infrastructure::postgres::PostgresPaykitTaskAdmissionRepository; + + let database = TestDatabase::create().await; + let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); + let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); + let pending = task( + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d16", + VerificationTaskStatus::Pending, + datetime!(2026-05-29 12:00:00 UTC), + ); + tasks + .insert_verification_task(pending.clone()) + .await + .unwrap(); + sqlx::query( + "INSERT INTO paykit_task_admissions + (verification_task_id, ready, ready_at) + VALUES ($1::uuid, TRUE, now())", + ) + .bind(pending.task_id.to_string()) + .execute(database.pool()) + .await + .unwrap(); + + assert!( + admissions + .find_existing(&pending.submitted_proof_bundle) + .await + .is_err() + ); + let claimer = PostgresVerificationTaskClaimer::new(database.pool().clone()); + assert!( + claimer + .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .await + .unwrap() + .is_none() + ); + + database.cleanup().await; + } + #[tokio::test] async fn does_not_claim_terminal_tasks() { let database = TestDatabase::create().await; diff --git a/locks-service/src/infrastructure/postgres/verification_tasks.rs b/locks-service/src/infrastructure/postgres/verification_tasks.rs index 77a4513..cf0a68c 100644 --- a/locks-service/src/infrastructure/postgres/verification_tasks.rs +++ b/locks-service/src/infrastructure/postgres/verification_tasks.rs @@ -154,7 +154,11 @@ impl VerificationTaskRepository for PostgresVerificationTaskRepository { completed_at = $8, failure_message = $9, updated_at = now() - WHERE task_id = $1::uuid", + WHERE task_id = $1::uuid + AND NOT EXISTS ( + SELECT 1 FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.verification_task_id = verification_tasks.task_id + )", ) .bind(row.task_id) .bind(row.creator) @@ -189,7 +193,14 @@ impl VerificationTaskRepository for PostgresVerificationTaskRepository { AND NOT EXISTS ( SELECT 1 FROM paykit_task_admissions WHERE verification_task_id = verification_tasks.task_id - AND ready = FALSE + AND ( + ready = FALSE + OR payment_in_hours IS NULL + OR payment_in_hours <= 0 + OR invoice_created_at IS NULL + OR payment_deadline IS NULL + OR invoice_created_at > payment_deadline + ) )" ); let row = sqlx::query_as::<_, VerificationTaskRow>(&sql) @@ -213,7 +224,14 @@ impl VerificationTaskRepository for PostgresVerificationTaskRepository { AND NOT EXISTS ( SELECT 1 FROM paykit_task_admissions WHERE verification_task_id = verification_tasks.task_id - AND ready = FALSE + AND ( + ready = FALSE + OR payment_in_hours IS NULL + OR payment_in_hours <= 0 + OR invoice_created_at IS NULL + OR payment_deadline IS NULL + OR invoice_created_at > payment_deadline + ) )" ); let row = sqlx::query_as::<_, VerificationTaskRow>(&sql) @@ -495,6 +513,36 @@ mod tests { database.cleanup().await; } + #[tokio::test] + async fn legacy_paykit_admission_without_authoritative_window_is_hidden_from_all_lookups() { + let database = TestDatabase::create().await; + let repo = PostgresVerificationTaskRepository::new(database.pool().clone()); + let pending = task(VerificationTaskStatus::Pending); + let task_id = pending.task_id; + let creator = pending.creator.clone(); + let bundle_id = pending.submitted_proof_bundle.bundle_id.clone(); + repo.insert_verification_task(pending).await.unwrap(); + sqlx::query( + "INSERT INTO paykit_task_admissions + (verification_task_id, ready, ready_at) + VALUES ($1::uuid, TRUE, now())", + ) + .bind(task_id.to_string()) + .execute(database.pool()) + .await + .unwrap(); + + assert_eq!(repo.get_verification_task(&task_id).await.unwrap(), None); + assert_eq!( + repo.get_verification_task_by_handle(&creator, &bundle_id) + .await + .unwrap(), + None + ); + + database.cleanup().await; + } + #[tokio::test] async fn insert_rejects_task_when_record_creator_diverges_from_submitted_bundle() { let database = TestDatabase::create().await;