Skip to content

fix(rpcsmartrouter): say why a write's outcome is unclear (MAG-4380) - #546

Merged
avitenzer merged 4 commits into
mainfrom
fix/MAG-4380-unclear-write-reason
Nov 9, 2026
Merged

avitenzer merged 4 commits into
mainfrom
fix/MAG-4380-unclear-write-reason

Conversation

@AnnaR-prog

@AnnaR-prog AnnaR-prog commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Found in the real-node test of the GraphQL interface (epic MAG-4368), but not specific to GraphQL: this is the write path on every interface. The ticket now sits under MAG-2769.

Revised on 2026-11-08 and rebased on main 2d99d4b7.

  • The cause of a timeout now always gets a timeout reason (details below). In the previous version, a dial or response timeout and a gateway's 504 were reported as "a node's answer was lost after the transaction was sent". That is false for a dial timeout, which never sends anything.
  • The previous body claimed "full package passes with -race, 0 races". That was wrong: TestHealthCommand_RejectsUnbootableCrossValidation failed under -race, as it also did on main at the old fork point 0d38f68b. The Verification section now lists exactly what was run.

A write whose outcome the router cannot know ends with "transaction status unclear", a 500, and the advice to verify on-chain before resubmitting. That verdict was right in every case Tom tested. The reason in the message was not: it was one fixed string, "timeout reached before any node responded", whatever had happened.

  • U-lost (Fuel Sepolia, 2026-10-15): both nodes executed the transfer, and both answers became a proxy's 502 within 0.3 s. The client was told, 0.3 s in, that the request timed out before any node responded.
  • U-01: one node rejected the write, and the other took it and went silent. The client got the same text, although a node had answered.

The ticket's other half, the log line that said status=200 while the client got a 500, is already fixed on main. MAG-4400's relay_end event replaced that line and logs an unclear write with status: 500. This PR carries only the reason.

The reason now follows the cause

The verdict does not change. unknownWriteOutcome is byte-identical to main, so every write that was "unclear" before still is, and no other write becomes unclear. Once the verdict is "unknown", unclearWriteReason names why. The first rule that applies wins:

# What happened Reason
1 No endpoint answered before a deadline timeout reached before any node responded (unchanged text)
2 Some endpoints answered (a rejection, a 502, a refusal), but not every one did before a deadline timeout reached before every node responded
3 Every endpoint came back before a deadline, none with an answer, and a connection broke after it was established: a reset, an EOF, a gateway's 502 a node's answer was lost after the transaction was sent
4 Anything else: a cancellation with budget left, or an error nothing classifies a request to a node ended without an answer (new)
  • What counts as a timeout. An endpoint that came back with a timeout did not answer either, so it counts with the endpoints still silent. That covers the budget, the attempt's own deadline, a dial or a response that timed out, and a gateway's 504. The timeouts are exactly the attempts relay_end logs as outcome: timeout, shared through protocolErrorOutcome, so the client's reason and the logged attempts cannot disagree. The 504 is the one addition, because it is a timeout one hop further out. Cloudflare's 522 and 524 classify as the same code, so they count too.
  • Why a timeout must not read as "answer lost". The router cannot tell a dial timeout from a response timeout: both are PROTOCOL_CONNECTION_TIMEOUT (error_codes.go). A dial that timed out never sent the transaction, so "lost after the transaction was sent" would be false there.
  • Precedence. A timeout or a silence beats a lost answer. The request spent its time waiting on the node that never answered, which is what the client lived through, and the advice is the same either way. A lost answer beats rule 4.
  • The frame is unchanged. The message still opens with transaction status unclear: and still ends with the transaction may already have been submitted — verify on-chain before resubmitting. A client keying on either sees no difference. No reason says "failed", and a test now pins that for every reason.

Flagged: rule 4 is a new client-visible string

a request to a node ended without an answer covers the failures that keep the outcome open but are neither a timeout nor a broken connection: a cancellation while budget remained, and an error the classifier does not recognise (UNKNOWN_ERROR). Calling those "answer lost after the transaction was sent" or "timeout reached" would claim something the router does not know, so this reason claims neither a timeout nor a delivery.

Known limitation (not changed here)

A streaming client (WebSocket or gRPC) that hangs up mid-write stops the relay with CallerGone. If an endpoint is still silent at that point, rule 2 gives "timeout reached before every node responded" although no deadline fired. This only reaches the relay_end log, because the client is already gone. The HTTP listeners never produce CallerGone.

Why the ticket's suggested strings are not used

  • "A node did not answer before the budget." Rules 1 and 2 keep the old sentence's shape ("timeout reached before … responded"). The all-silent case keeps its exact text, and the not-every-node case differs by one word. Also, a timeout can come from a deadline other than the budget.
  • "A node answered with a transport-level failure after receiving the transaction." This claims the node received the transaction, which the router cannot know. A 502 is the proxy speaking, and a reset only says that the connection broke. Rule 3 states only what the router saw: it sent the transaction, and the answer did not come back.

Code

  • rpcsmartrouter_server.go:
    • unclearWriteError carries the reason.
    • errUnknownWriteOutcome keeps its name and text for rule 1, and three siblings carry rules 2-4.
    • unclearWriteOutcomeFor counts each transport failure into writeCutOffs (timed out, or lost), asks main's unchanged unknownWriteOutcome for the verdict, and only then asks unclearWriteReason.
    • writeOutcomeIsUnknown stays the yes/no wrapper its existing tests call.
  • relay_end.go:
    • The attempt-outcome switch is extracted unchanged into protocolErrorOutcome, so writeCutOffs reads the same timeouts.
    • An unclear write is matched by type (unclearWriteError), so every reason still ends as outcome: write_unknown with status: 500.
    • Its error field carries the message the client read, reason included.
  • testdata/relay_end/write_unknown.json: in this golden, node-a answered and node-b stayed silent. Its error now says "timeout reached before every node responded", and that line is the only change.
  • docs/RELAY-END-EVENT.md: the write_unknown row lists the four reasons and what counts as a timeout.
  • common/error_codes.go: one comment that named errUnknownWriteOutcome as the only unclear-write error now names the type.

Where the text is quoted

  • smart-router-automation keys on the prefix only: UNCLEAR_WRITE_SENTINEL = "transaction status unclear" and UNCLEAR_WRITE_STATUS = 500 (tests/simulator/_router_sentinels.py). The full old sentence appears there only in comments.
  • This repo: the tests key on transaction status unclear and the advice. The relay_end doc quotes only the opening words.

So this is not marked breaking. The text in the middle of the message changes for every unclear write that is not all-silent.

Verification (local, 2026-11-08)

On unmodified main 2d99d4b7: TestUnclearWriteSaysWhy fails 5 of its 7 cases. The client's message and relay_end's error both say "timeout reached before any node responded" for:

  • every answer lost (two 502s);
  • a single node's answer lost (502);
  • one node silent and the other's answer lost;
  • one node rejected the write and the other stayed silent;
  • one gateway timed out (504) and the other's answer was lost (502).

The all-silent and two-504 cases pass on main, because the fixed text happens to be right there.

On the previous head of this PR (397a644f, rebased as 82990a30), the new tests fail where that version said "answer lost" for something else:

  • TestUnclearWriteSaysWhy: two 504s, and a 504 beside a 502.
  • TestUnclearWriteOutcomeFor_ReasonFollowsTheFailure: a dial timeout, an attempt deadline, a cancellation when the budget ran out, a reset beside a timeout, a cancellation with budget left, and an unclassified error. Each row also asserts the code the real classifier gave its failure, so it tests the case it names.

On this head:

  • go test -race -count=1 -run 'TestUnclearWrite|TestWriteCutOffs|TestUnknownWrite|TestWriteOutcome|TestMayHaveReachedNode|TestRelayEnd|TestRESTListener_WriteOutcome|TestWithUnclearWriteStatus' ./protocol/rpcsmartrouter/ passes, with no race reported. That covers every relay_end exit-path golden and the existing write-outcome, REST write-outcome and wrapper tests, unchanged.
  • go test -count=1 ./protocol/rpcsmartrouter/... ./protocol/common/... without -race passes (rpcsmartrouter 293 s, common 8 s).
  • The full package was not run with -race locally. CI's unit-tests-race covers it.
  • gofmt, goimports and go vet are clean, and golangci-lint v2 (latest) --new-from-rev origin/main reports 0 issues.

New tests (in protocol/rpcsmartrouter/write_outcome_reason_test.go):

  • TestUnclearWriteSaysWhy: seven writes end to end through the REST listener. It checks the client's message and the relay_end event (outcome, status, error).
  • TestUnclearWriteOutcomeFor_ReasonFollowsTheFailure: one real processor per row, with failures sent through the real classifier.
  • TestUnclearWriteReason: the rule table, including precedence and the count clamps.
  • TestWriteCutOffs_Classification: which failures count as a timeout or as a broken connection, next to what relay_end logs for each attempt, so the two lists cannot drift.
  • TestUnclearWriteErrorsReachTheClientVerbatim: each of the four messages reaches the client exactly as written through the error envelope, redaction included, and none says "failed".

Jira ticket

Jira ticket: MAG-4380


Author Checklist

  • read the contribution guide
  • included the correct type prefix in the PR title (fix)
  • confirmed ! in the type prefix if API or client breaking change: not breaking (the message keeps its opening words and its advice; only the reason in the middle changes)
  • targeted the main branch
  • provided a link to the relevant issue or specification: MAG-4380
  • reviewed "Files changed" and left comments if necessary
  • included the necessary unit and integration tests
  • updated the relevant documentation or specification: docs/RELAY-END-EVENT.md (write_unknown row) and the Go doc comments on the unclear-write errors and reasons
  • confirmed all CI checks have passed (pending)

🤖 Generated with Claude Code

@AnnaR-prog
AnnaR-prog requested a review from a team as a code owner October 18, 2026 12:46
@AnnaR-prog
AnnaR-prog force-pushed the fix/MAG-4380-unclear-write-reason branch from ca94075 to 397a644 Compare October 20, 2026 12:25
@AnnaR-prog AnnaR-prog changed the title fix(rpcsmartrouter): say why a write's outcome is unclear, and log the status the client gets (MAG-4380) fix(rpcsmartrouter): say why a write's outcome is unclear (MAG-4380) Oct 20, 2026
A write whose outcome the router cannot know ends with "transaction
status unclear". The verdict was right in every case, but the reason
was one fixed string, "timeout reached before any node responded",
whatever happened. On Fuel Sepolia (2026-10-15) both nodes executed a
transfer and both answers became a proxy's 502 within 0.3 s. The
client was told, 0.3 s in, that no node had responded before a
timeout. A node that rejected the write beside a silent sibling got
the same text, though a node had answered.

The verdict does not change: unknownWriteOutcome is untouched. Once it
says the outcome is unknown, unclearWriteReason names why, and the
first rule that applies wins:

- no endpoint answered before a deadline: "timeout reached before any
  node responded" (the old text);
- some did, but not every one: "timeout reached before every node
  responded";
- every endpoint came back before a deadline, none with an answer, and
  a connection broke after it was established (a reset, an EOF, a
  gateway's 502): "a node's answer was lost after the transaction was
  sent";
- anything else, such as a cancellation or an error nothing classifies:
  "a request to a node ended without an answer".

An endpoint that came back with a timeout did not answer either, so it
counts with the silent ones, whatever the deadline was. That covers the
budget, the attempt's own deadline, and a dial or a response that timed
out, which is relay_end's timeout, now shared through
protocolErrorOutcome. It also covers a gateway's 504, which is the same
thing one hop further out. A dial that timed out never sent the
transaction, so it must not be reported as an answer lost after
sending. A timeout or a silence wins over a lost answer: the request
spent its time waiting on the node that never answered, and the advice
is the same either way.

The message still opens with "transaction status unclear:" and still
ends with the advice to verify on-chain before resubmitting, so a client
keying on either is unaffected.

relay_end (MAG-4400) already logs an unclear write with the status the
client gets, 500, which was the ticket's second half. It now matches
any unclearWriteError, so every reason is logged as write_unknown, and
its error field carries the message the client read. The write_unknown
golden (one node answered, one silent) now says "before every node
responded", and docs/RELAY-END-EVENT.md lists the reasons.

Tests:
- TestUnclearWriteSaysWhy drives seven writes through the REST listener
  (silences, 502s, 504s and a rejection). It checks the client's
  message and the relay_end event.
- TestUnclearWriteOutcomeFor_ReasonFollowsTheFailure sends one failure
  per row through the real classifier: a dial timeout, a deadline, a
  cancellation with and without budget left, a reset, an EOF, an
  unclassified error, and mixes.
- TestUnclearWriteReason, TestWriteCutOffs_Classification and
  TestUnclearWriteErrorsReachTheClientVerbatim pin the rules, the shared
  timeout classification, and each message through the error envelope.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@AnnaR-prog
AnnaR-prog force-pushed the fix/MAG-4380-unclear-write-reason branch from 397a644 to 668aee6 Compare November 8, 2026 15:28
…e reason, not only errUnknownWriteOutcome (MAG-4380)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 668aee6d. I found no correctness problems. I pushed one comment-only commit (b7a45bb7).

1. Stale comment (fixed in b7a45bb7). withUnclearWriteStatus still said the listener "renders errUnknownWriteOutcome". It now renders whichever unclearWriteError the write ended with, so the comment says that. No code change.

Checked and fine:

  • The verdict is unchanged. unknownWriteOutcome is untouched. The only change to the loop that feeds it is dropping the break, so cutOffMidDelivery gets the same value, and writeOutcomeIsUnknown is now unclearWriteOutcomeFor(...) != nil.
  • Silence only comes from a deadline. For Stateful, WaitForResults returns early only when every session of the batch answered (checkEndProcessing) or on a success. Otherwise it waits for the context. So an endpoint with no record at all means the budget ran out, or the caller went away (the CallerGone limitation you documented). Rules 1 and 2 saying "timeout reached" is true on the HTTP listeners.
  • The counts and clamps. A timed-out attempt is already in answered (it is a protocol error), so dispatched - answered + timedOut can't exceed dispatched unless answered > dispatched. The min/max clamp covers that. The rate-limited-retry path (cumulative SessionsDispatched) gives rule 2: the 429s count as answers, which matches the table's "a refusal" row.
  • writeCutOffs.add and relay_end agree. Both call protocolErrorOutcome with requestRanOutOfRoad on the same GetStopReason(). The only extras are the 504/522/524 code (LavaErrorNodeGatewayTimeout, which is MayHaveReachedNode) and the reset/EOF/502 codes. The extraction of protocolErrorOutcome is behaviour-preserving, and IsRateLimited is nil-safe, so dropping the old LavaError != nil guard on that path is fine.
  • relay_end matches by type. errors.As(in.err, &unclearWrite) is the only consumer outside SendParsedRelay. No listener matches errUnknownWriteOutcome by value, so every reason still gets write_unknown and 500. A repo-wide grep finds no other errors.Is on it.
  • Client strings. smart-router-automation keys only on UNCLEAR_WRITE_SENTINEL = "transaction status unclear" and status 500 (tests/simulator/_router_sentinels.py). The full old sentence appears only in comments and skill docs, so no automation test breaks.
  • Memoisation. unknownWriteOnce still computes once, and both return sites hand back the same *unclearWriteError.

Non-blocking observation (not changed): relay_end's error_code for write_unknown still comes from relayEndWriteUnknownCode (the first MayHaveReachedNode attempt), so a 504-plus-502 write can log NODE_BAD_GATEWAY beside the reason "timeout reached before every node responded". Both are true, and that field predates this PR.

Verification on b7a45bb7 (local, 2026-11-09):

  • go build ./..., go vet and gofmt are clean on rpcsmartrouter and common. golangci-lint --new-from-rev=origin/main reports 0 issues.
  • go test -race -count=1 -run 'TestUnclearWrite|TestWriteCutOffs|TestUnknownWrite|TestWriteOutcome|TestMayHaveReachedNode|TestRelayEnd|TestRESTListener_WriteOutcome|TestWithUnclearWriteStatus' ./protocol/rpcsmartrouter/ passes (46 s).
  • go test -count=1 ./protocol/rpcsmartrouter/... ./protocol/common/... passes (292 s and 8 s).
  • The branch merges into current main (36030802) with no conflicts. It stays two linear commits, so rebase-merge works.

avitenzer and others added 2 commits November 9, 2026 13:26
Resolves the conflict with #671 (MAG-3980): the budget-ran-out unclear-write return keeps
#671's headed result and returns the write's reason error instead of errUnknownWriteOutcome.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving after the review above. The only fix was a stale comment (b7a45bb7). Main's #671 conflict is resolved in 67cc4ce6 and keeps both behaviours. Main has since been merged in again. Locally on 67cc4ce6: go test -count=1 passes on rpcsmartrouter, common and chainlib, and the targeted write-outcome and relay_end tests pass under -race. Merging as a squash, since the branch has merge commits.

@avitenzer
avitenzer merged commit e90f7e3 into main Nov 9, 2026
@avitenzer
avitenzer deleted the fix/MAG-4380-unclear-write-reason branch November 9, 2026 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants