Skip to content

fix: identify RequestError::Transport as infrastructure error - #870

Draft
aintnostressin wants to merge 133 commits into
mainfrom
fix/pubky-errors-as-infra
Draft

fix: identify RequestError::Transport as infrastructure error#870
aintnostressin wants to merge 133 commits into
mainfrom
fix/pubky-errors-as-infra

Conversation

@aintnostressin

@aintnostressin aintnostressin commented May 20, 2026

Copy link
Copy Markdown
Contributor

It should abort homeserver run in case of infrastructure error.

2026-05-20T07:11:32.112505Z ERROR event_processor.run:dx.user_events.process: nexus_watcher::service::indexer::key_based:51: Failed to subscribe to event stream: Request(Transport(reqwest::Error { kind: Request, url: "https://8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo/events-stream?user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo%3A0&limit=50&path=%2Fpub%2F", source: hyper_util::client::legacy::Error(Connect, ConnectError("dns error", CouldNotResolveHost)) })) service=KeyBasedEventProcessor with HS ID: 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo homeserver=8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo
2026-05-20T07:11:32.112574Z ERROR event_processor.run: nexus_watcher::service::indexer::key_based:136: Non-infrastructure user error; continuing with next user hs_id=8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo action="skip_user" error=PubkyClientError(RequestFailed { message: "error sending request for url (https://8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo/events-stream?user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo%3A0&limit=50&path=%2Fpub%2F)" }) service=KeyBasedEventProcessor with HS ID: 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo

After fix:

2026-05-21T06:39:42.129542Z  INFO event_processor.run: nexus_watcher::service::indexer::key_based:113: Found 1 users on HS 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo service=KeyBasedEventProcessor with HS ID: 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo
2026-05-21T06:39:42.129600Z  INFO event_processor.run:dx.user_events.process: pubky::actors::event_stream:362: Subscribing to event stream for 1 user(s) on homeserver pubky8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo service=KeyBasedEventProcessor with HS ID: 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo homeserver=8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo
2026-05-21T06:39:42.400418Z ERROR event_processor.run:dx.user_events.process: nexus_watcher::service::indexer::key_based:51: Failed to subscribe to event stream: Request(Transport(reqwest::Error { kind: Request, url: "https://8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo/events-stream?user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo%3A0&limit=50&path=%2Fpub%2F", source: hyper_util::client::legacy::Error(Connect, ConnectError("dns error", CouldNotResolveHost)) })) service=KeyBasedEventProcessor with HS ID: 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo homeserver=8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo
2026-05-21T06:39:42.400497Z ERROR event_processor.run: nexus_watcher::service::indexer::key_based:126: Infrastructure error while processing user; aborting homeserver run hs_id=8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo action="abort_hs" error=PubkyClientError(RequestFailed { is_transport: true, message: "error sending request for url (https://8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo/events-stream?user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo%3A0&limit=50&path=%2Fpub%2F)" }) service=KeyBasedEventProcessor with HS ID: 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo
2026-05-21T06:39:42.400538Z ERROR nexus_watcher::service::indexer:105: Event processor failed for KeyBasedEventProcessor with HS ID: 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo: PubkyClientError(RequestFailed { is_transport: true, message: "error sending request for url (https://8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo/events-stream?user=au1h984texykfj8sz5ggac5ffft3rzaqecgjii3tfec54fk4fbgo%3A0&limit=50&path=%2Fpub%2F)" })
2026-05-21T06:39:42.400561Z  INFO nexus_watcher::service::backoff:61: Homeserver 8pinxxgqs41n4aididenw5apqp1urfmzdztr8jt4abrkdn435ewo backed off for 60s

ok300 and others added 30 commits February 15, 2026 12:58
…HS from run_all

- NexusWatcher::start now spawns 3 parallel tokio tasks:
  1. Default homeserver processing (run_default_homeserver)
  2. Other homeservers processing (run_all, excluding default HS)
  3. Reserved for future use (TODO placeholder)
- Added run_default_homeserver to TEventProcessorRunner trait
- Modified homeservers_by_priority to exclude the default homeserver
- Added Send + Sync bounds to TEventProcessorRunner trait
- Updated all tests and rustdocs

Co-authored-by: ok300 <106775972+ok300@users.noreply.github.com>
…match

Co-authored-by: ok300 <106775972+ok300@users.noreply.github.com>
Fix pkarr version mismatch in PubkyId conversion
…s` (#761)

* refactor: extract periodic task loop into reusable run_periodic_tasks

* Fix periodic task ticks to skip missed intervals

Set MissedTickBehavior::Skip on the interval timer in run_periodic_tasks
so that if a task takes longer than its tick interval, missed ticks are
dropped instead of queuing up a burst of back-to-back invocations.

Added tests verifying that slow tasks don't accumulate queued ticks and
that fast tasks still tick at the expected cadence.

* chore: rename variables for clarity

* chore: make task_runner module private

* log: NexusWatcher::start logs task outcomes on exit

* chore: cargo fmt

* chore: limit visibility of Task structs to pub(crate)

* fix: correctly set task name for panicking tasks

* fix: ensure NexusWatcher::start returns Err if any task panics

---------

Co-authored-by: Claude <noreply@anthropic.com>
* nexus-api tests: use unseeded testnet

* deps: update pubky dependency to latest 0.7.x
* ref: runners and indexers

* chore: use runner pub function

* extract_retry_event_info -> handle_error (#785)

* remove get_homeserver_id fn (#784)

* Extract common TEventProcessorRunner::run (#789)

* Add HsEventProcessor rustdoc

* Add KeyBasedEventProcessor rustdoc

* Add HsEventProcessorRunner rustdoc

* Add KeyBasedEventProcessorRunner rustdoc

* Add TODO for KeyBasedEventProcessor::run_internal

* Extract no-op default TEventProcessorRunner::post_run fn

---------

Co-authored-by: aintnostressin <aintnostressin@proton.me>
Co-authored-by: ok300 <106775972+ok300@users.noreply.github.com>
* ref: runners and indexers

* chore: use runner pub function

* extract_retry_event_info -> handle_error (#785)

* remove get_homeserver_id fn (#784)

* TEventProcessor: expand logic of handle_error

* Clarify comments

* fix: remove InternalError from is_infrastructure

* Expand logic for EventProcessorError::is_infrastructure_err

* chore: remove unused imports

---------

Co-authored-by: tipogi <tipogi@protonmail.com>
Co-authored-by: aintnostressin <aintnostressin@proton.me>
@aintnostressin
aintnostressin requested a review from ok300 May 20, 2026 07:28
@greptile-apps

greptile-apps Bot commented May 20, 2026

Copy link
Copy Markdown

Greptile Summary

This PR fixes a misclassification where RequestError::Transport (e.g. DNS failures, connection refused) was being treated as a per-user error and silently skipped rather than aborting the homeserver run. The fix threads an is_transport: bool flag through PubkyClientError::RequestFailed and uses it in EventProcessorError::is_infrastructure() to correctly trigger the abort-HS path.

  • PubkyClientError::RequestFailed gains an is_transport field; only the RequestError::Transport variant sets it true, while Server (non-404/5xx), Validation, and DecodeJson variants set it false.
  • is_infrastructure() now has an explicit PubkyClientError arm, returning true only for transport errors, so the caller aborts and backs off the homeserver instead of skipping individual users.

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped to transport-layer error classification and does not touch any data path, state mutation, or persistence logic.

The two changed files touch only error classification logic. The is_transport flag is plumbed correctly through all RequestError variants, and the new is_infrastructure() arm correctly gates on that flag. All non-transport client errors retain their existing non-infrastructure classification.

No files require special attention.

Important Files Changed

Filename Overview
nexus-common/src/db/connectors/pubky.rs Adds is_transport: bool field to RequestFailed, sets it true only for RequestError::Transport, and false for Server (non-4xx/5xx), Validation, and DecodeJson variants.
nexus-common/src/models/event/errors.rs Extends is_infrastructure() to handle the PubkyClientError arm, returning true only when RequestFailed { is_transport: true } and false for all other client-error variants; also updates client_error() constructor for the new RequestFailed shape.

Reviews (4): Last reviewed commit: "treat only transport errors as infra err..." | Re-trigger Greptile

Comment thread nexus-common/src/models/event/errors.rs Outdated
@aintnostressin
aintnostressin removed the request for review from ok300 May 20, 2026 09:20
@aintnostressin
aintnostressin marked this pull request as draft May 20, 2026 09:20
@greptile-apps

greptile-apps Bot commented May 20, 2026

Copy link
Copy Markdown

Want your agent to iterate on Greptile's feedback? Try greploops.

@aintnostressin

Copy link
Copy Markdown
Contributor Author

@greptile review again

@aintnostressin aintnostressin changed the title fix: treat some PubkyClientError variants as infra errors fix: identify RequestError::Transport as infrastructure error May 21, 2026
@aintnostressin
aintnostressin marked this pull request as ready for review May 21, 2026 07:21
@aintnostressin aintnostressin added 🐞 bug Something isn't working 👀 watcher Nexus indexer related operations and removed 🐞 bug Something isn't working labels May 21, 2026
@aintnostressin

Copy link
Copy Markdown
Contributor Author

I was testing this locally and it may cause a stall in homeserver indexing process if we process an event line and user doesn't have a PKARR record published

2026-05-21T11:02:07.708222Z  INFO event_processor.run:event_batch.process: nexus_watcher::service::indexer:125: ProcessEventLine: PUT pubky://ujikizdojcuwn3m7nfypzt1xwiz8t68rqr9qbdbagmgotwd1duho/pub/pubky.app/files/0032VXNACT47G service=HsEventProcessor with HS ID: 8um71us3fyw6h8wbcxb5ar3rwusy1a6u49956ikzojg3gcwd1dty batch.size=501
2026-05-21T11:02:07.710154Z  WARN event_processor.run:event_batch.process:event.process: nexus_watcher::service::indexer:157: Infrastructure error, stopping batch: PubkyClientError: Request failed (is_transport: true): error sending request for url (https://_pubky.ujikizdojcuwn3m7nfypzt1xwiz8t68rqr9qbdbagmgotwd1duho/pub/pubky.app/files/0032VXNACT47G): client error (Connect): dns error: pkarr could not resolve host service=HsEventProcessor with HS ID: 8um71us3fyw6h8wbcxb5ar3rwusy1a6u49956ikzojg3gcwd1dty batch.size=501 event.resource=files event.uri=pubky://ujikizdojcuwn3m7nfypzt1xwiz8t68rqr9qbdbagmgotwd1duho/pub/pubky.app/files/0032VXNACT47G event.type=PUT event.user_id=ujikizdojcuwn3m7nfypzt1xwiz8t68rqr9qbdbagmgotwd1duho event.resource_id="0032VXNACT47G" instance=HsEventProcessor with HS ID: 8um71us3fyw6h8wbcxb5ar3rwusy1a6u49956ikzojg3gcwd1dty otel.status_code="ERROR" otel.status_message=PubkyClientError: Request failed (is_transport: true): error sending request for url (https://_pubky.ujikizdojcuwn3m7nfypzt1xwiz8t68rqr9qbdbagmgotwd1duho/pub/pubky.app/files/0032VXNACT47G): client error (Connect): dns error: pkarr could not resolve host

@aintnostressin

Copy link
Copy Markdown
Contributor Author

This may indicate a broader problem:

If we are acting in the context of the homeserver (processing event lines) we could fetch data directly from that homeserver instead of performing dns resolution

@aintnostressin
aintnostressin marked this pull request as draft May 21, 2026 12:28
@aintnostressin

Copy link
Copy Markdown
Contributor Author

Possible fix for problems described above: #877

Base automatically changed from feat/user-key-indexing to feat/dx-events-by-user June 1, 2026 12:27
@tipogi tipogi added this to the 2026-July milestone Jun 8, 2026
Base automatically changed from feat/dx-events-by-user to main June 30, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

👀 watcher Nexus indexer related operations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants