Repository navigation
fix(rpcsmartrouter): say why a write's outcome is unclear (MAG-4380) - #546
Conversation
ca94075 to
397a644
Compare
A write whose outcome the router cannot know ends with "transaction status unclear". The verdict was right in every case, but the reason was one fixed string, "timeout reached before any node responded", whatever happened. On Fuel Sepolia (2026-10-15) both nodes executed a transfer and both answers became a proxy's 502 within 0.3 s. The client was told, 0.3 s in, that no node had responded before a timeout. A node that rejected the write beside a silent sibling got the same text, though a node had answered. The verdict does not change: unknownWriteOutcome is untouched. Once it says the outcome is unknown, unclearWriteReason names why, and the first rule that applies wins: - no endpoint answered before a deadline: "timeout reached before any node responded" (the old text); - some did, but not every one: "timeout reached before every node responded"; - every endpoint came back before a deadline, none with an answer, and a connection broke after it was established (a reset, an EOF, a gateway's 502): "a node's answer was lost after the transaction was sent"; - anything else, such as a cancellation or an error nothing classifies: "a request to a node ended without an answer". An endpoint that came back with a timeout did not answer either, so it counts with the silent ones, whatever the deadline was. That covers the budget, the attempt's own deadline, and a dial or a response that timed out, which is relay_end's timeout, now shared through protocolErrorOutcome. It also covers a gateway's 504, which is the same thing one hop further out. A dial that timed out never sent the transaction, so it must not be reported as an answer lost after sending. A timeout or a silence wins over a lost answer: the request spent its time waiting on the node that never answered, and the advice is the same either way. The message still opens with "transaction status unclear:" and still ends with the advice to verify on-chain before resubmitting, so a client keying on either is unaffected. relay_end (MAG-4400) already logs an unclear write with the status the client gets, 500, which was the ticket's second half. It now matches any unclearWriteError, so every reason is logged as write_unknown, and its error field carries the message the client read. The write_unknown golden (one node answered, one silent) now says "before every node responded", and docs/RELAY-END-EVENT.md lists the reasons. Tests: - TestUnclearWriteSaysWhy drives seven writes through the REST listener (silences, 502s, 504s and a rejection). It checks the client's message and the relay_end event. - TestUnclearWriteOutcomeFor_ReasonFollowsTheFailure sends one failure per row through the real classifier: a dial timeout, a deadline, a cancellation with and without budget left, a reset, an EOF, an unclassified error, and mixes. - TestUnclearWriteReason, TestWriteCutOffs_Classification and TestUnclearWriteErrorsReachTheClientVerbatim pin the rules, the shared timeout classification, and each message through the error envelope. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
397a644 to
668aee6
Compare
…e reason, not only errUnknownWriteOutcome (MAG-4380) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
avitenzer
left a comment
There was a problem hiding this comment.
Reviewed at 668aee6d. I found no correctness problems. I pushed one comment-only commit (b7a45bb7).
1. Stale comment (fixed in b7a45bb7). withUnclearWriteStatus still said the listener "renders errUnknownWriteOutcome". It now renders whichever unclearWriteError the write ended with, so the comment says that. No code change.
Checked and fine:
- The verdict is unchanged.
unknownWriteOutcomeis untouched. The only change to the loop that feeds it is dropping thebreak, socutOffMidDeliverygets the same value, andwriteOutcomeIsUnknownis nowunclearWriteOutcomeFor(...) != nil. - Silence only comes from a deadline. For
Stateful,WaitForResultsreturns early only when every session of the batch answered (checkEndProcessing) or on a success. Otherwise it waits for the context. So an endpoint with no record at all means the budget ran out, or the caller went away (theCallerGonelimitation you documented). Rules 1 and 2 saying "timeout reached" is true on the HTTP listeners. - The counts and clamps. A timed-out attempt is already in
answered(it is a protocol error), sodispatched - answered + timedOutcan't exceeddispatchedunlessanswered > dispatched. Themin/maxclamp covers that. The rate-limited-retry path (cumulativeSessionsDispatched) gives rule 2: the 429s count as answers, which matches the table's "a refusal" row. writeCutOffs.addand relay_end agree. Both callprotocolErrorOutcomewithrequestRanOutOfRoadon the sameGetStopReason(). The only extras are the 504/522/524 code (LavaErrorNodeGatewayTimeout, which isMayHaveReachedNode) and the reset/EOF/502 codes. The extraction ofprotocolErrorOutcomeis behaviour-preserving, andIsRateLimitedis nil-safe, so dropping the oldLavaError != nilguard on that path is fine.- relay_end matches by type.
errors.As(in.err, &unclearWrite)is the only consumer outsideSendParsedRelay. No listener matcheserrUnknownWriteOutcomeby value, so every reason still getswrite_unknownand 500. A repo-wide grep finds no othererrors.Ison it. - Client strings. smart-router-automation keys only on
UNCLEAR_WRITE_SENTINEL = "transaction status unclear"and status 500 (tests/simulator/_router_sentinels.py). The full old sentence appears only in comments and skill docs, so no automation test breaks. - Memoisation.
unknownWriteOncestill computes once, and both return sites hand back the same*unclearWriteError.
Non-blocking observation (not changed): relay_end's error_code for write_unknown still comes from relayEndWriteUnknownCode (the first MayHaveReachedNode attempt), so a 504-plus-502 write can log NODE_BAD_GATEWAY beside the reason "timeout reached before every node responded". Both are true, and that field predates this PR.
Verification on b7a45bb7 (local, 2026-11-09):
go build ./...,go vetand gofmt are clean onrpcsmartrouterandcommon.golangci-lint --new-from-rev=origin/mainreports 0 issues.go test -race -count=1 -run 'TestUnclearWrite|TestWriteCutOffs|TestUnknownWrite|TestWriteOutcome|TestMayHaveReachedNode|TestRelayEnd|TestRESTListener_WriteOutcome|TestWithUnclearWriteStatus' ./protocol/rpcsmartrouter/passes (46 s).go test -count=1 ./protocol/rpcsmartrouter/... ./protocol/common/...passes (292 s and 8 s).- The branch merges into current main (
36030802) with no conflicts. It stays two linear commits, so rebase-merge works.
avitenzer
left a comment
There was a problem hiding this comment.
Approving after the review above. The only fix was a stale comment (b7a45bb7). Main's #671 conflict is resolved in 67cc4ce6 and keeps both behaviours. Main has since been merged in again. Locally on 67cc4ce6: go test -count=1 passes on rpcsmartrouter, common and chainlib, and the targeted write-outcome and relay_end tests pass under -race. Merging as a squash, since the branch has merge commits.
Description
Found in the real-node test of the GraphQL interface (epic MAG-4368), but not specific to GraphQL: this is the write path on every interface. The ticket now sits under MAG-2769.
A write whose outcome the router cannot know ends with "transaction status unclear", a 500, and the advice to verify on-chain before resubmitting. That verdict was right in every case Tom tested. The reason in the message was not: it was one fixed string, "timeout reached before any node responded", whatever had happened.
The ticket's other half, the log line that said
status=200while the client got a 500, is already fixed on main. MAG-4400'srelay_endevent replaced that line and logs an unclear write withstatus: 500. This PR carries only the reason.The reason now follows the cause
The verdict does not change.
unknownWriteOutcomeis byte-identical to main, so every write that was "unclear" before still is, and no other write becomes unclear. Once the verdict is "unknown",unclearWriteReasonnames why. The first rule that applies wins:relay_endlogs asoutcome: timeout, shared throughprotocolErrorOutcome, so the client's reason and the logged attempts cannot disagree. The 504 is the one addition, because it is a timeout one hop further out. Cloudflare's 522 and 524 classify as the same code, so they count too.PROTOCOL_CONNECTION_TIMEOUT(error_codes.go). A dial that timed out never sent the transaction, so "lost after the transaction was sent" would be false there.transaction status unclear:and still ends withthe transaction may already have been submitted — verify on-chain before resubmitting. A client keying on either sees no difference. No reason says "failed", and a test now pins that for every reason.Flagged: rule 4 is a new client-visible string
a request to a node ended without an answercovers the failures that keep the outcome open but are neither a timeout nor a broken connection: a cancellation while budget remained, and an error the classifier does not recognise (UNKNOWN_ERROR). Calling those "answer lost after the transaction was sent" or "timeout reached" would claim something the router does not know, so this reason claims neither a timeout nor a delivery.Known limitation (not changed here)
A streaming client (WebSocket or gRPC) that hangs up mid-write stops the relay with
CallerGone. If an endpoint is still silent at that point, rule 2 gives "timeout reached before every node responded" although no deadline fired. This only reaches therelay_endlog, because the client is already gone. The HTTP listeners never produceCallerGone.Why the ticket's suggested strings are not used
Code
rpcsmartrouter_server.go:unclearWriteErrorcarries the reason.errUnknownWriteOutcomekeeps its name and text for rule 1, and three siblings carry rules 2-4.unclearWriteOutcomeForcounts each transport failure intowriteCutOffs(timed out, or lost), asks main's unchangedunknownWriteOutcomefor the verdict, and only then asksunclearWriteReason.writeOutcomeIsUnknownstays the yes/no wrapper its existing tests call.relay_end.go:protocolErrorOutcome, sowriteCutOffsreads the same timeouts.unclearWriteError), so every reason still ends asoutcome: write_unknownwithstatus: 500.errorfield carries the message the client read, reason included.testdata/relay_end/write_unknown.json: in this golden, node-a answered and node-b stayed silent. Itserrornow says "timeout reached before every node responded", and that line is the only change.docs/RELAY-END-EVENT.md: thewrite_unknownrow lists the four reasons and what counts as a timeout.common/error_codes.go: one comment that namederrUnknownWriteOutcomeas the only unclear-write error now names the type.Where the text is quoted
UNCLEAR_WRITE_SENTINEL = "transaction status unclear"andUNCLEAR_WRITE_STATUS = 500(tests/simulator/_router_sentinels.py). The full old sentence appears there only in comments.transaction status unclearand the advice. Therelay_enddoc quotes only the opening words.So this is not marked breaking. The text in the middle of the message changes for every unclear write that is not all-silent.
Verification (local, 2026-11-08)
On unmodified main
2d99d4b7:TestUnclearWriteSaysWhyfails 5 of its 7 cases. The client's message andrelay_end'serrorboth say "timeout reached before any node responded" for:The all-silent and two-504 cases pass on main, because the fixed text happens to be right there.
On the previous head of this PR (
397a644f, rebased as82990a30), the new tests fail where that version said "answer lost" for something else:TestUnclearWriteSaysWhy: two 504s, and a 504 beside a 502.TestUnclearWriteOutcomeFor_ReasonFollowsTheFailure: a dial timeout, an attempt deadline, a cancellation when the budget ran out, a reset beside a timeout, a cancellation with budget left, and an unclassified error. Each row also asserts the code the real classifier gave its failure, so it tests the case it names.On this head:
go test -race -count=1 -run 'TestUnclearWrite|TestWriteCutOffs|TestUnknownWrite|TestWriteOutcome|TestMayHaveReachedNode|TestRelayEnd|TestRESTListener_WriteOutcome|TestWithUnclearWriteStatus' ./protocol/rpcsmartrouter/passes, with no race reported. That covers everyrelay_endexit-path golden and the existing write-outcome, REST write-outcome and wrapper tests, unchanged.go test -count=1 ./protocol/rpcsmartrouter/... ./protocol/common/...without-racepasses (rpcsmartrouter293 s,common8 s).-racelocally. CI'sunit-tests-racecovers it.go vetare clean, and golangci-lint v2 (latest)--new-from-rev origin/mainreports 0 issues.New tests (in
protocol/rpcsmartrouter/write_outcome_reason_test.go):TestUnclearWriteSaysWhy: seven writes end to end through the REST listener. It checks the client's message and therelay_endevent (outcome,status,error).TestUnclearWriteOutcomeFor_ReasonFollowsTheFailure: one real processor per row, with failures sent through the real classifier.TestUnclearWriteReason: the rule table, including precedence and the count clamps.TestWriteCutOffs_Classification: which failures count as a timeout or as a broken connection, next to whatrelay_endlogs for each attempt, so the two lists cannot drift.TestUnclearWriteErrorsReachTheClientVerbatim: each of the four messages reaches the client exactly as written through the error envelope, redaction included, and none says "failed".Jira ticket
Jira ticket: MAG-4380
Author Checklist
fix)!in the type prefix if API or client breaking change: not breaking (the message keeps its opening words and its advice; only the reason in the middle changes)mainbranchwrite_unknownrow) and the Go doc comments on the unclear-write errors and reasons🤖 Generated with Claude Code