Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/RATE-LIMIT-HOLDOFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,9 @@ off" means on its path. This table is the catalog; add a row when wiring a new c
| Recovery probe (`recovery_probe.go`) | Skip the replay while held off (verdict stays inconclusive), instead of burning replay-attempt budget on a vendor that said stop | 429 probe responses (Retry-After floor honoured) | live |
| Chain tracker / endpoint poller (`chaintracker/chain_tracker.go`, `endpointstate/endpoint_monitor.go`) | Records its own 429s under the url alone — the key the WS and gRPC subscription paths use — never under the relay path's (provider, url) key: relay selection holds a provider off while every url the registry knows for it is held, so a tracker entry there would pull the provider's healthy legs out of selection whenever its one polled leg is limited, and a backup-tier leg with no relay traffic would never be cleared. Reads both keys: the first fetch waits out an existing hold-off before each attempt, and a rate-limited attempt ends the init burst instead of retrying into the limit — the applied hold-off rides the start error as its Retry-After, and the monitor's start retry waits it out with the tracker in `retrying_start` naming the 429, never `starting` with no error; the dedicated poll waits out the applied hold-off (Retry-After when sent, else the strike schedule) and whatever the registry holds for the url — a relay's, a probe's or a subscription's hold-off, or the provider escalation. Its timer wakes at the normal cadence and reads the registry on every wake: while the url is held off nothing is sent, not even the poll scheduled before the hold-off began, and the next wake looks again (MAG-4387). It used to fold the hold-off into one timer when the timer was armed, so a poll already scheduled went out into the limit and a hold-off lifted early was slept out in full — after a burst of 429s that froze the url's tip for up to 35 minutes while relays on the url were answered. The idle wake burst does not poll a url that is held off: `WakePoll` sends nothing and returns `chaintracker.ErrorPollNowHeldOff` (also the typed 429, remaining wait as Retry-After), and the burst counts the url as not polled. An explicit `/debug/poll-now` still polls, as an explicit `lava-select-provider` outranks a relay hold-off. Because each wake reads the registry, an answer recorded for the url by anyone — a relay included — resumes the poll within one interval. A held wake runs at the base cadence, not the failure backoff: a refused poll counts as a failed one, and the backoff it built up would otherwise delay the resume by up to a minute. Clears its url-only entry only after its own 429 was answered; a fetch that got no answer (timeout, refused or reset connection) clears nothing | The typed 429 from the head or block-hash fetch — an HTTP 429, a refused ws handshake, or a JSON-RPC error body with code 429 / "too many requests", which `chainlib.RateLimitedReplyError` types for the body form on both fetchers — `chainlib.ChainFetcher` and the per-endpoint `endpointstate.EndpointPoller` behind every per-url tracker, including its `CustomMessage` behind the SVM (Solana / KOII) latest-slot poll via `chainlib.RateLimitedErrorBody` (MAG-4165) | live |
| Probe loop (`runProbeCycleCore` in `rpcsmartrouter_server.go`) | Busy, not dead: an endpoint whose observation went stale while its url is held off under either key (`URLReadyAt`) is left out of its provider's QoS sample — neither healthy nor counted — and a provider whose every endpoint is in that state sends no sample that cycle. A stale url used to be scored availability 0 every cycle for the length of the hold-off, routing reads away from a healthy vendor that only asked us to slow down (MAG-4387). A fresh observation is judged as usual, held off or not, and the proactive re-enable pass is unaffected | Nothing — read only | live |
| WS subscriptions (`direct_ws_subscription_manager.go`) | A rate-limited connect/subscribe is held off instead of scored (no availability sample), selection skips held endpoints while something ready remains, and the pool's reconnect ladder is floored by the dial's Retry-After. Keyed per WS URL — no provider name exists on this path, so vendor-tier escalation does not apply | 429 handshakes and subscribe failures | live |
| WS subscriptions (`direct_ws_subscription_manager.go`) | A rate-limited connect/subscribe is held off instead of scored (no availability sample) and the subscribe moves on to the next endpoint while one is left (MAG-4730), selection skips held endpoints while something ready this subscribe has not tried remains, and the pool's reconnect ladder is floored by the dial's Retry-After. Keyed per WS URL — no provider name exists on this path, so vendor-tier escalation does not apply | 429 handshakes and subscribe failures | live |
| WS / gRPC transports | Recognition first: a 429 on the WS upgrade or a corroborated gRPC rate limit produces the typed sentinel these consumers key on | handshake / metadata 429s | live |
| gRPC streaming subscriptions (`direct_grpc_subscription_manager.go`) | Selection skips held-off endpoints while something ready remains; the full tier stays when nothing is. A stream the upstream ends for rate holds its endpoint off, so the client's resubscribe goes elsewhere; a stream's first message clears it. Keyed per URL, as on the WS path | Corroborated rate limits that end a stream (`common.RateLimitFromGRPC`, retry delay read from the stream's trailer) | live |
| gRPC streaming subscriptions (`direct_grpc_subscription_manager.go`) | Selection skips held-off endpoints while something ready this subscribe has not tried remains; the full tier stays when nothing is. A stream the upstream ends for rate holds its endpoint off, so the client's resubscribe goes elsewhere; a stream's first message clears it. Keyed per URL, as on the WS path | Corroborated rate limits that end a stream (`common.RateLimitFromGRPC`, retry delay read from the stream's trailer) | live |

## Metrics

Expand Down
117 changes: 67 additions & 50 deletions protocol/rpcsmartrouter/direct_grpc_subscription_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ func NewDirectGRPCSubscriptionManager(
metricsManager: metrics.SafeMetrics(metricsManager),
chainID: chainID,
apiInterface: apiInterface,
subscriptionEndpoints: newSubscriptionEndpoints("DirectGRPC", "no gRPC endpoints available",
subscriptionEndpoints: newSubscriptionEndpoints("DirectGRPC", "no gRPC endpoints available", chainID,
grpcEndpoints, grpcBackupEndpoints, optimizer),
config: config,
rateLimiter: NewGRPCClientRateLimiter(config),
Expand Down Expand Up @@ -606,64 +606,81 @@ func (dgm *DirectGRPCSubscriptionManager) createNewSubscription(
if err != nil {
return nil, nil, fmt.Errorf("failed to select endpoint: %w", err)
}
// From here on the request goes to the endpoint: its pool dials it, reflection asks it for the
// method, and the stream opens on it.
sentAt := time.Now()
defer func() {
chainlib.SubscriptionUpstreamFrom(ctx).Sent(dgm.endpointNames.name(endpoint.Url), sentAt, time.Since(sentAt), err)
}()

// Get or create pool
pool, err := dgm.getOrCreatePool(endpoint)
if err != nil {
return nil, nil, fmt.Errorf("failed to get pool: %w", err)
}
// Open the stream there, or on the next endpoint while one cannot take it
// (subscribeWithFailover). Each attempt reports its outcome against its own endpoint.
var (
pool *UpstreamGRPCPool
conn *UpstreamGRPCStreamConnection
methodDesc *desc.MethodDescriptor
stream grpc.ClientStream
subCtx context.Context
subCancel context.CancelFunc
)
endpoint, err = dgm.subscribeWithFailover(ctx, clientKey, endpoint, func(endpoint *common.NodeUrl) (err error) {
// From here on the request goes to the endpoint: its pool dials it, reflection asks it for the
// method, and the stream opens on it.
sentAt := time.Now()
defer func() {
chainlib.SubscriptionUpstreamFrom(ctx).Sent(dgm.endpointNames.name(endpoint.Url), sentAt, time.Since(sentAt), err)
}()

// Get or create pool
pool, err = dgm.getOrCreatePool(endpoint)
if err != nil {
return fmt.Errorf("failed to get pool: %w", err)
}

// Get a connection, with this stream's slot counted on it already (see
// ReserveConnectionForStream); every failure below releases the slot.
conn, err := pool.ReserveConnectionForStream(ctx)
if err != nil {
return nil, nil, fmt.Errorf("failed to get connection: %w", err)
}
// Get a connection, with this stream's slot counted on it already (see
// ReserveConnectionForStream); every failure below releases the slot.
conn, err = pool.ReserveConnectionForStream(ctx)
if err != nil {
return fmt.Errorf("failed to get connection: %w", err)
}

// Connection established — pin client to this endpoint for future
// subscriptions. Mirrors DirectWSSubscriptionManager.startUpstreamSubscription
// (Epoch is unused for direct RPC — there's no provider rotation).
dgm.stickyStore.Set(clientKey, &lavasession.StickySession{
Provider: endpoint.Url,
Epoch: 0,
})
// Connection established — pin client to this endpoint for future
// subscriptions. Mirrors DirectWSSubscriptionManager.startUpstreamSubscription
// (Epoch is unused for direct RPC — there's no provider rotation).
dgm.stickyStore.Set(clientKey, &lavasession.StickySession{
Provider: endpoint.Url,
Epoch: 0,
})

// Parse service and method
svc, methodName := rpcInterfaceMessages.ParseSymbol(methodPath)
// Parse service and method
svc, methodName := rpcInterfaceMessages.ParseSymbol(methodPath)

// Get method descriptor
methodDesc, err := conn.GetMethodDescriptor(ctx, svc, methodName)
if err != nil {
pool.NotifyStreamRemoved(conn)
return nil, nil, fmt.Errorf("failed to get method descriptor: %w", err)
}
// Get method descriptor
methodDesc, err = conn.GetMethodDescriptor(ctx, svc, methodName)
if err != nil {
pool.NotifyStreamRemoved(conn)
return fmt.Errorf("failed to get method descriptor: %w", err)
}

// Verify it's a server-streaming method
if !methodDesc.IsServerStreaming() {
pool.NotifyStreamRemoved(conn)
return nil, nil, fmt.Errorf("method %s is not a server-streaming method", methodPath)
}
// Verify it's a server-streaming method
if !methodDesc.IsServerStreaming() {
pool.NotifyStreamRemoved(conn)
return fmt.Errorf("method %s is not a server-streaming method", methodPath)
}

// The subscription context is rooted in the manager, NOT in ctx. conn.NewStream
// binds the upstream stream's whole life to the context it was created with, and
// with sharing enabled later clients join this same stream — so rooting it in the
// creating client's request context meant the first client to disconnect killed
// the stream for every joiner. Rooted here, only the subscription's own teardown
// ends it: the last client leaving (removeClientFromSubscription) or Stop.
subCtx, subCancel := context.WithCancel(dgm.ctx)
// The subscription context is rooted in the manager, NOT in ctx. conn.NewStream
// binds the upstream stream's whole life to the context it was created with, and
// with sharing enabled later clients join this same stream — so rooting it in the
// creating client's request context meant the first client to disconnect killed
// the stream for every joiner. Rooted here, only the subscription's own teardown
// ends it: the last client leaving (removeClientFromSubscription) or Stop.
subCtx, subCancel = context.WithCancel(dgm.ctx)

// Create the upstream stream
stream, err := dgm.createUpstreamStream(subCtx, conn.GetConn(), methodPath, requestData, methodDesc)
// Create the upstream stream
stream, err = dgm.createUpstreamStream(subCtx, conn.GetConn(), methodPath, requestData, methodDesc)
if err != nil {
subCancel()
pool.NotifyStreamRemoved(conn)
return fmt.Errorf("failed to create stream: %w", err)
}
return nil
})
if err != nil {
subCancel()
pool.NotifyStreamRemoved(conn)
return nil, nil, fmt.Errorf("failed to create stream: %w", err)
return nil, nil, err
}

// Generate IDs
Expand Down
76 changes: 45 additions & 31 deletions protocol/rpcsmartrouter/direct_ws_subscription_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,7 @@ func NewDirectWSSubscriptionManager(
chainID: chainID,
apiInterface: apiInterface,
subscriptionEndpoints: newSubscriptionEndpoints("DirectWS",
"no WebSocket endpoints available (primary and backup both exhausted)",
"no WebSocket endpoints available (primary and backup both exhausted)", chainID,
wsEndpoints, wsBackupEndpoints, optimizer),
config: config,
rateLimiter: NewClientRateLimiter(config),
Expand Down Expand Up @@ -615,50 +615,64 @@ func (dwsm *DirectWSSubscriptionManager) StartSubscription(

// No active subscription found, create new one
// Select best endpoint using sticky session, optimizer, or first available
startTime := time.Now()
selectedEndpoint, err := dwsm.selectEndpoint(ctx, clientKey, nil)
if err != nil {
dwsm.settlePendingSubscription(hashedParams, false)
dwsm.metricsManager.SetFailedWsSubscriptionRequestMetric(dwsm.chainID, dwsm.apiInterface)
return nil, nil, fmt.Errorf("failed to select WebSocket endpoint: %w", err)
}

utils.LavaFormatTrace("DirectWS: creating new upstream subscription",
utils.LogAttr("hashedParams", utils.ToHexString(hashedParams)),
utils.LogAttr("endpoint", sanitizeEndpointURL(selectedEndpoint.Url)),
// Subscribe upstream there, or on the next endpoint while one cannot take the
// subscription (subscribeWithFailover). Each attempt reports its outcome against its own
// endpoint, to the relay record, the optimizer and the hold-off.
var (
pool *UpstreamWSPool
conn *UpstreamWSConnection
upstreamSub *rpcclient.ClientSubscription
firstMsg *rpcclient.JsonrpcMessage
msgChan chan *rpcclient.JsonrpcMessage
)
selectedEndpoint, err = dwsm.subscribeWithFailover(ctx, clientKey, selectedEndpoint, func(endpoint *common.NodeUrl) (err error) {
utils.LavaFormatTrace("DirectWS: creating new upstream subscription",
utils.LogAttr("hashedParams", utils.ToHexString(hashedParams)),
utils.LogAttr("endpoint", sanitizeEndpointURL(endpoint.Url)),
)
startTime := time.Now()

// Get or create upstream pool for selected endpoint
pool := dwsm.GetOrCreatePool(selectedEndpoint)
// Get or create upstream pool for the endpoint
pool = dwsm.GetOrCreatePool(endpoint)

// Get a connection, with this subscription's slot counted on it already (see
// ReserveConnectionForSubscription); every failure below releases the slot.
conn, err := pool.ReserveConnectionForSubscription(ctx)
if err != nil {
chainlib.SubscriptionUpstreamFrom(ctx).Sent(dwsm.endpointNames.name(selectedEndpoint.Url), startTime, time.Since(startTime), err)
dwsm.noteSubscribeFailure(selectedEndpoint.Url, err)
dwsm.settlePendingSubscription(hashedParams, false)
dwsm.metricsManager.SetFailedWsSubscriptionRequestMetric(dwsm.chainID, dwsm.apiInterface)
return nil, nil, fmt.Errorf("failed to get WebSocket connection: %w", err)
}
// Get a connection, with this subscription's slot counted on it already (see
// ReserveConnectionForSubscription); every failure below releases the slot.
conn, err = pool.ReserveConnectionForSubscription(ctx)
if err != nil {
chainlib.SubscriptionUpstreamFrom(ctx).Sent(dwsm.endpointNames.name(endpoint.Url), startTime, time.Since(startTime), err)
dwsm.noteSubscribeFailure(endpoint.Url, err)
return fmt.Errorf("failed to get WebSocket connection: %w", err)
}

// Create upstream subscription
upstreamSub, firstMsg, msgChan, err := dwsm.createUpstreamSubscription(ctx, conn, subscriptionParams, subscribeMethod)
chainlib.SubscriptionUpstreamFrom(ctx).Sent(dwsm.endpointNames.name(selectedEndpoint.Url), startTime, time.Since(startTime), err)
// Create upstream subscription
upstreamSub, firstMsg, msgChan, err = dwsm.createUpstreamSubscription(ctx, conn, subscriptionParams, subscribeMethod)
chainlib.SubscriptionUpstreamFrom(ctx).Sent(dwsm.endpointNames.name(endpoint.Url), startTime, time.Since(startTime), err)
if err != nil {
pool.NotifySubscriptionRemoved(conn)
dwsm.noteSubscribeFailure(endpoint.Url, err)
return fmt.Errorf("failed to create upstream subscription: %w", err)
}

// Report success to optimizer. The endpoint answered, so any standing rate-limit
// hold-off for it is stale.
dwsm.holdoffs().RecordAnswer(endpoint.Url, endpoint.Url)
if dwsm.optimizer != nil {
latency := time.Since(startTime)
dwsm.optimizer.AppendRelayData(endpoint.Url, latency, 1, 0) // cu=1, syncBlock=0 (unknown)
}
return nil
})
if err != nil {
pool.NotifySubscriptionRemoved(conn)
dwsm.noteSubscribeFailure(selectedEndpoint.Url, err)
dwsm.settlePendingSubscription(hashedParams, false)
dwsm.metricsManager.SetFailedWsSubscriptionRequestMetric(dwsm.chainID, dwsm.apiInterface)
return nil, nil, fmt.Errorf("failed to create upstream subscription: %w", err)
}

// Report success to optimizer. The endpoint answered, so any standing rate-limit
// hold-off for it is stale.
dwsm.holdoffs().RecordAnswer(selectedEndpoint.Url, selectedEndpoint.Url)
if dwsm.optimizer != nil {
latency := time.Since(startTime)
dwsm.optimizer.AppendRelayData(selectedEndpoint.Url, latency, 1, 0) // cu=1, syncBlock=0 (unknown)
return nil, nil, err
}

// Store sticky session for client affinity
Expand Down
16 changes: 16 additions & 0 deletions protocol/rpcsmartrouter/direct_ws_subscription_manager_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,11 @@ type mockSubscriptionServer struct {
// rejectSubscribes makes the node answer every subscribe with a JSON-RPC error, which is
// a failed subscribe rather than a subscription that ends later (MAG-2685).
rejectSubscribes atomic.Bool

// subscribeRefusal, when set, is the JSON-RPC error object the node answers every subscribe
// with instead: which refusal it is decides whether the router may try another endpoint
// (MAG-4730).
subscribeRefusal atomic.Pointer[rpcclient.JsonError]
}

// setRefusing turns the node's outage on or off.
Expand Down Expand Up @@ -165,6 +170,17 @@ func (ms *mockSubscriptionServer) handleWS(w http.ResponseWriter, r *http.Reques
continue
}

if refusal := ms.subscribeRefusal.Load(); refusal != nil {
response := map[string]interface{}{
"jsonrpc": "2.0",
"id": req.ID,
"error": map[string]interface{}{"code": refusal.Code, "message": refusal.Message},
}
respBytes, _ := json.Marshal(response)
ms.safeWriteMessage(conn, websocket.TextMessage, respBytes)
continue
}

if ms.rejectSubscribes.Load() {
response := map[string]interface{}{
"jsonrpc": "2.0",
Expand Down
Loading