Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/RELAY-END-EVENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ On Cosmos the fields are read, as `tx_code` is, only on a write the spec marks s
| --- | --- | --- |
| `ok` | An upstream's or the cache's reply that is not a node error | `SendParsedRelay` returns a reply and no error: the first success (stateless, stateful), the quorum winner (cross-validation), or a cache hit |
| `node_error` | A node's error reply, passed through | `SendParsedRelay` returns a node error with no error: `processNonCrossValidationResult` when nothing succeeded, or a cached node error. A node's rate-limit reply is one too. |
| `write_unknown` | "transaction status unclear", HTTP 500 | `SendParsedRelay` returns `errUnknownWriteOutcome` (`writeOutcomeIsUnknown`): a stateful relay where no upstream accepted, and one was still silent when the budget ran out or its connection died after the request went out |
| `write_unknown` | "transaction status unclear: …" with the reason, HTTP 500 | `SendParsedRelay` returns an `unclearWriteError` (`unclearWriteOutcomeFor`): a stateful relay where no upstream accepted, and one was still silent when the budget ran out or its connection died after the request went out. The reason says which, first match wins: no node answered before a deadline ("timeout reached before any node responded"), some did but not every one ("… before every node responded"), a connection broke after it was established (a reset, an EOF or a 502: "a node's answer was lost after the transaction was sent"), or anything else, such as a cancellation or an unclassified error ("a request to a node ended without an answer"). An attempt that timed out counts as not answered, whatever the deadline: the budget, the attempt's own, a dial or a response, or a gateway's 504 |
| `no_answer` | An error, after at least one upstream was asked | `SendParsedRelay` returns an error with a dispatched attempt: every attempt failed in transport, timed out, was rate limited or answered an HTTP error status, or cross-validation found no quorum. When an upstream answered, the client gets its status and body as is, and `status` says which (MAG-4481) |
| `rejected_before_dispatch` | An error; no upstream was asked | `SendRelay` or the WebSocket manager: the request does not parse (unsupported shape, refused batch). `SendParsedRelay` with nothing dispatched: invalid cross-validation headers, the cross-validation capacity check, an empty provider pool, an unavailable pinned provider (the select-provider directive), every endpoint failing consistency pre-validation, or the deadline expiring before dispatch |
| `router_error` | An error, or nothing usable | The router's own machinery failed (`routerFault`): its task loop could not start, the task channel closed without a final instruction, or a relay had no request data. Also `SendParsedRelay` returning neither a reply nor an error, which includes a panic inside it |
Expand Down
4 changes: 2 additions & 2 deletions protocol/common/error_codes.go
Original file line number Diff line number Diff line change
Expand Up @@ -392,8 +392,8 @@ var (
// transaction answered with ABORTED can be on chain. It does not decide a write's verdict
// here. A gRPC status error comes back as a node error carrying the node's own message, and
// writeOutcomeIsUnknown reads this flag only from protocol errors, so the caller gets Sui's
// ABORTED answer, with its "retriable with another submission" text, rather than
// errUnknownWriteOutcome.
// ABORTED answer, with its "retriable with another submission" text, rather than a "transaction
// status unclear" error (unclearWriteError).
//
// A node error, not PROTOCOL_PROVIDER_ABORTED (1013). That code is CategoryInternal, which
// would report the endpoint as unreachable and rank the node's own message below internal
Expand Down
34 changes: 22 additions & 12 deletions protocol/rpcsmartrouter/relay_end.go
Original file line number Diff line number Diff line change
Expand Up @@ -372,8 +372,10 @@ func (rpcss *RPCSmartRouterServer) buildRelayEnd(ctx context.Context, in relayEn
end.readTransaction(in.protocolMessage, reply)
end.status = in.result.GetStatusCode()

// Every reason a write's outcome is unclear is one unclearWriteError, so match the type, not one value.
var unclearWrite *unclearWriteError
switch {
case errors.Is(in.err, errUnknownWriteOutcome):
case errors.As(in.err, &unclearWrite):
end.outcome = relayOutcomeWriteUnknown
case in.err == nil && !served:
end.outcome = relayOutcomeRouterError // neither an answer nor an error: the router lost the result
Expand All @@ -399,7 +401,8 @@ func (rpcss *RPCSmartRouterServer) buildRelayEnd(ctx context.Context, in relayEn
// What the listener sends, whether or not the router returned a result to carry it.
end.status = http.StatusInternalServerError
end.errorCode = relayEndWriteUnknownCode(in.cause, end.attempts, requestRanOutOfRoad(end.stopReason))
end.errorText = relayEndText(errUnknownWriteOutcome.Error())
// The message the client reads, with the reason it gives (MAG-4380).
end.errorText = relayEndText(unclearWrite.Error())
default:
cause := in.cause
if cause == nil {
Expand Down Expand Up @@ -505,27 +508,34 @@ func relayEndProtocolErrorAttempt(protocolError relaycore.RelayError, ranOutOfRo
if protocolError.Response != nil {
attempt.latency = protocolError.Response.RelayResult.AttemptLatency
}
attempt.outcome = protocolErrorOutcome(protocolError, ranOutOfRoad)
attempt.errorCode = relayEndCode(protocolError.LavaError)
if err := protocolError.GetError(); err != nil {
attempt.message = relayEndText(err.Error())
}
return attempt
}

// protocolErrorOutcome is how a transport failure ended its attempt. An unclear write reads its
// timeouts from here too (writeCutOffs), so the reason the client gets and relay_end's attempts
// agree on which attempts timed out.
func protocolErrorOutcome(protocolError relaycore.RelayError, ranOutOfRoad bool) string {
err := protocolError.GetError()
switch {
case protocolError.LavaError.IsRateLimited() || errors.Is(err, common.StatusCodeError429):
attempt.outcome = attemptOutcomeRateLimited
return attemptOutcomeRateLimited
case errors.Is(err, context.Canceled) || protocolError.LavaError == common.LavaErrorContextCanceled:
// Cancelled because the budget ran out is a timeout; otherwise the router stopped waiting.
attempt.outcome = attemptOutcomeCancelled
if ranOutOfRoad {
attempt.outcome = attemptOutcomeTimeout
return attemptOutcomeTimeout
}
return attemptOutcomeCancelled
case errors.Is(err, context.DeadlineExceeded) || protocolError.LavaError == common.LavaErrorContextDeadline ||
protocolError.LavaError == common.LavaErrorConnectionTimeout || protocolError.LavaError == common.LavaErrorNoResponseTimeout:
attempt.outcome = attemptOutcomeTimeout
return attemptOutcomeTimeout
default:
attempt.outcome = attemptOutcomeTransportError
return attemptOutcomeTransportError
}
attempt.errorCode = relayEndCode(protocolError.LavaError)
if err != nil {
attempt.message = relayEndText(err.Error())
}
return attempt
}

// relayEndErrorCode names, from the error registry, why a relay that did not end on an answer
Expand Down
6 changes: 5 additions & 1 deletion protocol/rpcsmartrouter/relay_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -354,14 +354,18 @@ func TestRelayEnd_ExitPaths(t *testing.T) {
},
},
{
// node-a answered and node-b never did, so the reason is a node still silent, not that
// nobody responded (MAG-4380).
name: "write_unknown",
fanOut: true,
nodes: func(t *testing.T) []relayEndNode {
return []relayEndNode{{"node-a", relayEndReply(200, relayEndNonceLow, 0)}, {"node-b", relayEndSilent}}
},
body: relayEndWrite,
timeout: 1500 * time.Millisecond,
check: func(t *testing.T, _ *common.RelayResult, err error) { require.ErrorIs(t, err, errUnknownWriteOutcome) },
check: func(t *testing.T, _ *common.RelayResult, err error) {
require.ErrorIs(t, err, errUnclearWriteNodeSilent)
},
},
{
// Nothing comes back at all: the router answers for itself, and the client still gets
Expand Down
131 changes: 112 additions & 19 deletions protocol/rpcsmartrouter/rpcsmartrouter_server.go
Original file line number Diff line number Diff line change
Expand Up @@ -1405,12 +1405,44 @@ func (rpcss *RPCSmartRouterServer) ParseRelay(
return protocolMessage, nil
}

// errUnknownWriteOutcome is returned when a write ended with no endpoint having answered. "Failed"
// would be a claim we cannot support: our deadline cannot un-send a transaction, and reporting a
// failure invites a resubmit of one that may already be on chain.
var errUnknownWriteOutcome = errors.New("transaction status unclear: timeout reached before any node responded; the transaction may already have been submitted — verify on-chain before resubmitting")
// unclearWriteError is what a write whose outcome we do not know ends with. "Failed" would be a
// claim we cannot support: our deadline cannot un-send a transaction, and reporting a failure
// invites a resubmit of one that may already be on chain.
//
// The message reads the same around its reason — "transaction status unclear: <reason>; the
// transaction may already have been submitted — verify on-chain before resubmitting" — so a client
// keying on the opening words or acting on the advice is unaffected by which reason it carries. The
// reason says why the outcome is unknown, and it has to be the true one (MAG-4380): every unclear
// write used to say "timeout reached before any node responded", including one where both nodes
// executed the transfer and both answers became a proxy's 502 within 0.3 s.
type unclearWriteError struct {
reason string
}

func (e *unclearWriteError) Error() string {
return "transaction status unclear: " + e.reason + "; the transaction may already have been submitted — verify on-chain before resubmitting"
}

// The reasons, in the order unclearWriteReason tries them. A timeout covers every deadline: the
// request's budget, the attempt's own, a dial or a response that timed out, and a gateway's 504.
var (
// errUnknownWriteOutcome: no endpoint answered before a deadline.
errUnknownWriteOutcome = &unclearWriteError{reason: "timeout reached before any node responded"}
// errUnclearWriteNodeSilent: an endpoint answered, with a failure, but another one the write was
// sent to did not answer before a deadline, and it may hold the transaction.
errUnclearWriteNodeSilent = &unclearWriteError{reason: "timeout reached before every node responded"}
// errUnclearWriteAnswerLost: every endpoint came back before a deadline, none with an answer, and
// at least one connection broke after it was established (a reset, an EOF, a gateway's 502), so
// that node may have executed the transaction and its answer was lost. Nothing timed out.
errUnclearWriteAnswerLost = &unclearWriteError{reason: "a node's answer was lost after the transaction was sent"}
// errUnclearWriteNoAnswer: every endpoint came back before a deadline, none with an answer, and
// what leaves the outcome open is neither a timeout nor a broken connection: a cancellation, or
// an error nothing classifies. It claims neither a timeout nor a delivery.
errUnclearWriteNoAnswer = &unclearWriteError{reason: "a request to a node ended without an answer"}
)

// writeOutcomeIsUnknown reports whether a failed request was a write whose outcome we do not know.
// unclearWriteOutcomeFor is the same verdict with its reason.
//
// The rule is silence: nothing succeeded, and some endpoint we asked did not answer at all — it may
// be holding the transaction. A hung endpoint records no error, so any rule inspecting only the
Expand All @@ -1427,12 +1459,18 @@ var errUnknownWriteOutcome = errors.New("transaction status unclear: timeout rea
// Evaluated here rather than in the results manager because a failed request reaches the client
// through either of two branches in SendParsedRelay; deciding once keeps that invisible.
func writeOutcomeIsUnknown(protocolMessage chainlib.ProtocolMessage, relayProcessor *relaycore.RelayProcessor) bool {
return unclearWriteOutcomeFor(protocolMessage, relayProcessor) != nil
}

// unclearWriteOutcomeFor is writeOutcomeIsUnknown with the reason: nil when the outcome is known,
// otherwise the unclearWriteError that names why it is not (see unclearWriteReason).
func unclearWriteOutcomeFor(protocolMessage chainlib.ProtocolMessage, relayProcessor *relaycore.RelayProcessor) *unclearWriteError {
if chainlib.GetStateful(protocolMessage) != common.CONSISTENCY_SELECT_ALL_PROVIDERS {
return false
return nil
}
if relayProcessor == nil {
// The request failed before a processor existed, so nothing was ever dispatched.
return false
return nil
}

successResults, nodeErrors, protocolErrors := relayProcessor.GetResultsData()
Expand All @@ -1444,21 +1482,26 @@ func writeOutcomeIsUnknown(protocolMessage chainlib.ProtocolMessage, relayProces
// endpoints than answered, and the verdict flips to "failed" while one is still silent.
dispatched := relayProcessor.GetUsedProviders().SessionsDispatched()

ranOutOfRoad := requestRanOutOfRoad(relayProcessor.GetStopReason())

// A node error is a reply, so it settles that endpoint. A transport error may not: an
// unclassified one proves nothing, and a reset, EOF or timeout can arrive after the request was
// already on the wire.
cutOffMidDelivery := false
var cutOffs writeCutOffs
for _, protocolError := range protocolErrors {
// Unmatched errors classify as UNKNOWN_ERROR, which carries the flag for the same reason,
// so a nil check here would be dead code rather than a safety net.
if protocolError.LavaError != nil && protocolError.LavaError.MayHaveReachedNode {
cutOffMidDelivery = true
break
}
cutOffs.add(protocolError, ranOutOfRoad)
}

return unknownWriteOutcome(len(successResults), answered, dispatched,
requestRanOutOfRoad(relayProcessor.GetStopReason()), cutOffMidDelivery)
if !unknownWriteOutcome(len(successResults), answered, dispatched, ranOutOfRoad, cutOffMidDelivery) {
return nil
}
return unclearWriteReason(answered, dispatched, cutOffs)
}

// withUnclearWriteStatus forces the HTTP status of an unclear-write reply to 500.
Expand All @@ -1478,7 +1521,8 @@ func withUnclearWriteStatus(result *common.RelayResult) *common.RelayResult {
stamped := *result
stamped.StatusCode = http.StatusInternalServerError
// MAG-4218: an unclear write outcome is the one case where the router must speak instead of
// the node. Drop the forwarded upstream answer so the listener renders errUnknownWriteOutcome.
// the node. Drop the forwarded upstream answer so the listener renders the unclearWriteError and
// its reason.
stamped.ClientFacing = nil
return &stamped
}
Expand Down Expand Up @@ -1535,6 +1579,54 @@ func unknownWriteOutcome(successes, answered, dispatched int, ranOutOfRoad, cutO
return cutOffMidDelivery
}

// writeCutOffs counts the transport failures of a write that its unclear reason tells apart.
type writeCutOffs struct {
// timedOut: the endpoint did not answer before a deadline, whichever one ended it. That is what
// relay_end calls a timeout (the budget, the attempt's deadline, a dial or a response that timed
// out), plus a gateway's timeout (504), which is the same thing one hop further out.
timedOut int
// lost: the connection broke after it was established — a reset, an EOF, a gateway's 502 — so
// the node may hold the transaction and its answer did not come back.
lost int
}

func (c *writeCutOffs) add(protocolError relaycore.RelayError, ranOutOfRoad bool) {
switch {
case protocolErrorOutcome(protocolError, ranOutOfRoad) == attemptOutcomeTimeout,
protocolError.LavaError == common.LavaErrorNodeGatewayTimeout:
c.timedOut++
case protocolError.LavaError == common.LavaErrorConnectionReset,
protocolError.LavaError == common.LavaErrorConnectionClosed,
protocolError.LavaError == common.LavaErrorNodeBadGateway:
c.lost++
}
}

// unclearWriteReason names why a write's outcome is unknown. Call it only once unknownWriteOutcome
// has said so; it always returns a reason.
//
// An endpoint that came back with a timeout did not answer either, so it counts with the ones still
// silent. The first rule that applies wins:
// - no endpoint answered before a deadline: errUnknownWriteOutcome;
// - some did, not every one: errUnclearWriteNodeSilent. A timeout or a silence wins over a lost
// answer: the request spent its time waiting on the node that never answered, which is what
// the client lived through, and the advice is the same either way;
// - a connection broke after it was established: errUnclearWriteAnswerLost;
// - anything else, a cancellation or an error nothing classifies: errUnclearWriteNoAnswer.
func unclearWriteReason(answered, dispatched int, cutOffs writeCutOffs) *unclearWriteError {
notAnswered := min(max(dispatched-answered+cutOffs.timedOut, 0), dispatched)
switch {
case notAnswered == dispatched:
return errUnknownWriteOutcome
case notAnswered > 0:
return errUnclearWriteNodeSilent
case cutOffs.lost > 0:
return errUnclearWriteAnswerLost
default:
return errUnclearWriteNoAnswer
}
}

// wakeWaitCap bounds how long a client relay that ends an idle period (or arrives while the wake
// burst it triggered is still in flight) waits for the first endpoint poll to land before it is
// routed. One upstream round trip is the intent; the cap keeps a slow upstream from turning idle
Expand Down Expand Up @@ -1629,11 +1721,12 @@ func (rpcss *RPCSmartRouterServer) SendParsedRelay(

// Both failure returns below ask the same question, and the answer cannot differ between them.
// Memoised so the results snapshot and its two locks are taken once.
// nil when the write's outcome is known, otherwise the error naming why it is not.
var unknownWriteOnce sync.Once
var unknownWriteAnswer bool
unknownWrite := func() bool {
var unknownWriteAnswer *unclearWriteError
unknownWrite := func() *unclearWriteError {
unknownWriteOnce.Do(func() {
unknownWriteAnswer = writeOutcomeIsUnknown(protocolMessage, relayProcessor)
unknownWriteAnswer = unclearWriteOutcomeFor(protocolMessage, relayProcessor)
})
return unknownWriteAnswer
}
Expand All @@ -1658,10 +1751,10 @@ func (rpcss *RPCSmartRouterServer) SendParsedRelay(
// failure hardest to diagnose, a silent upstream, and no Lava-Retries or
// Lava-Provider-Address to say what was tried. unknownWrite is true only for a processor
// that dispatched, so the processor is never nil there.
if unknownWrite() {
unclear := withUnclearWriteStatus(&common.RelayResult{})
rpcss.appendHeadersToRelayResult(ctx, unclear, relayProcessor.ProtocolErrors(), relayProcessor, protocolMessage, protocolMessage.GetApi().GetName(), analytics, false)
return unclear, errUnknownWriteOutcome
if unclear := unknownWrite(); unclear != nil {
unclearResult := withUnclearWriteStatus(&common.RelayResult{})
rpcss.appendHeadersToRelayResult(ctx, unclearResult, relayProcessor.ProtocolErrors(), relayProcessor, protocolMessage, protocolMessage.GetApi().GetName(), analytics, false)
return unclearResult, unclear
}

// A hung upstream: the request was dispatched and the processing budget ran out before any
Expand Down Expand Up @@ -1750,15 +1843,15 @@ func (rpcss *RPCSmartRouterServer) SendParsedRelay(
//
// Safe to hoist: writeOutcomeIsUnknown returns false unless this is a stateful relay with no
// successes, so a served write and every read are unaffected.
if unknownWrite() {
if unclear := unknownWrite(); unclear != nil {
if analytics != nil {
analytics.Success = false
}
// The reply keeps this request's headers; only the status and the message the client reads
// are replaced. The underlying error goes to relay_end's attempts rather than being
// concatenated into the reply — "failed relay, insufficient results" in front of "status
// unclear" would tell the customer both things at once.
return withUnclearWriteStatus(returnedResult), errUnknownWriteOutcome
return withUnclearWriteStatus(returnedResult), unclear
}
if err != nil {
returnedResult = rpcss.withNoAnswerTimeout(returnedResult, protocolMessage)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
}
],
"chain": "ETH1",
"error": "transaction status unclear: timeout reached before any node responded; the transaction may already have been submitted — verify on-chain before resubmitting",
"error": "transaction status unclear: timeout reached before every node responded; the transaction may already have been submitted — verify on-chain before resubmitting",
"error_code": "PROTOCOL_CONTEXT_DEADLINE",
"event": "relay_end",
"has_reply": false,
Expand Down
Loading