Skip to content

fix(utils): make the log level and stderr logger safe to change while others log (MAG-4740) - #617

Merged
nimrod-teich merged 2 commits into
mainfrom
mag-4740
Nov 2, 2026
Merged

nimrod-teich merged 2 commits into
mainfrom
mag-4740

Conversation

@nimrod-teich

Copy link
Copy Markdown
Contributor

#Closes MAG-4740

Jira ticket: MAG-4740

Note

Left unmerged for review. The change is to the logger's shared state, plus tests.

Why

utils.SetGlobalLoggingLevel assigned two process-wide variables without synchronization:

  • the level, which every LavaFormat* call reads before formatting;
  • zerolog's package Logger, which every enabled call reads.

Under -race, any test that sets the level failed whenever a goroutine left over from an earlier test was still logging. main fails the same way:

Test On main under -race Leftover goroutine
TestHealthCommand_RejectsUnbootableCrossValidation 2 of 6 runs, after a crafted-relay test a chainproxy connector logging its shutdown
TestGetUniqueGuidResponseDeterministic, TestSetGlobalLoggingLevel_VolumeChangesAcrossLevels 2 of 2 package runs with -count=3 TestHappyFlow's relays monitor, started on context.Background()

A test that changes the level while four goroutines log reports 11 races on main.

What changed

  • utils/lavalog.go: the level is an atomic.Int32, and the stderr sink is an atomic.Pointer[zerolog.Logger]. That is the pattern debugBufferLogger beside them already uses.
    • SetGlobalLoggingLevel stores both.
    • LavaFormatLog and IsDebugEnabled load them.
    • init still sets zerolog's global logger once, before any goroutine. Nothing else in the binary imports github.com/rs/zerolog/log.
  • utils.SetStdLogger(logger) (restore func()) swaps the sink for tests that capture output. The four tests that assigned zerolog's global logger now use it: the redisstore and performance TestMains, the metrics page test, and the cross-validation fleet capture. The in-package captureLog in utils uses it too.
  • utils/log_event.go: LogEvent writes through the same sink.
  • TestHappyFlow: its monitor runs on t.Context(), so its probes stop logging when it ends.
  • CreateChainLibMocks: the comment above it describes the code as it is.

How to verify

go test -race -count=3 ./utils/ -run TestLoggingStateChangesWhileOthersLog
go test -race -count=3 ./protocol/metrics/
go test -race -count=6 ./protocol/rpcsmartrouter/ -run 'TestSendCraftedRelays_IgnoresCrossValidationPolicy|TestHealthCommand_RejectsUnbootableCrossValidation'
Run main This branch
level and sink changed while 4 goroutines log 11 races 3 of 3 pass
protocol/metrics, -race -count=3 2 of 2 runs fail 3 of 3 runs pass
crafted relay + health command, -race -count=6 2 of 6 fail 6 of 6 pass

TestLoggingStateChangesWhileOthersLog is the regression test. It only finds a race under -race, and it passes without it on either code.

Also run here:

  • utils, ecosystem/cache/redisstore and protocol/performance pass under -race, so the capture tests still capture.
  • The Fleet and CrossValidation tests in protocol/rpcsmartrouter pass under -race.
  • go vet ./... passes.

Full go test ./protocol/rpcsmartrouter/ passes without -race (252 s).

Screenshots

No UI change.

…hile others log (MAG-4740)

SetGlobalLoggingLevel assigned the level and zerolog's package Logger
with plain writes, and every LavaFormat* call reads both. Under -race,
any test that set the level failed whenever an earlier test's goroutine
was still logging: TestHealthCommand_RejectsUnbootableCrossValidation
after a crafted-relay test's connector, and the metrics level tests
after TestHappyFlow's monitor, which ran on context.Background().

The level is an atomic.Int32 and the stderr sink an
atomic.Pointer[zerolog.Logger], as debugBufferLogger already is.
SetStdLogger swaps the sink for tests that capture output; the four
tests that assigned zerolog's global logger use it. LogEvent writes
through the same sink. TestHappyFlow's monitor runs on t.Context().
TestLoggingStateChangesWhileOthersLog changes both while goroutines
log; on main the same scenario reports 11 races.

@avitenzer avitenzer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I reviewed the diff and re-ran the evidence locally. CI here has no go test lane, so these runs are the only test evidence.

The bug and the fix reproduce. Base 0a971917 against this head af13ae8a, on the same machine at the same time:

Run base this PR
go test -race -count=3 ./protocol/metrics/ fails: SetGlobalLoggingLevel from TestGetUniqueGuidResponseDeterministic races TestHappyFlow's monitor goroutine passes
crafted relay + health command, -race -count=6 fails: the health command's SetGlobalLoggingLevel races a leftover chainproxy Connector loop 6 of 6 pass

The regression test catches both halves. TestLoggingStateChangesWhileOthersLog can't compile on main because it calls SetStdLogger. So I ran it with -overlay against two mutants of this PR's lavalog.go. Reverting only the level to a plain variable gives 3 races and a FAIL. Reverting only the sink gives 36 races and a FAIL.

Also passing:

  • go build ./... and go vet ./....
  • utils, redisstore and performance under -race, so their TestMain captures still capture.
  • The 51 Fleet|CrossValidation tests in rpcsmartrouter under -race.
  • This head merged onto current main c3aa77d0, which is conflict-free: go test ./... passes in all 38 packages, and the full go test -race ./protocol/rpcsmartrouter/ passes.

Checked and fine:

  • utils is the only importer of github.com/rs/zerolog/log in the dependency graph of ./cmd/..., so leaving zerolog's global at its init value changes no output.
  • No utils var initializer or earlier init() logs, so the nil stdLogger before init can't be reached. debugBufferLogger uses the same pattern.
  • Both commands still set JsonFormat before the level.

Optional, none blocking:

  1. newStdLogger still reads zerolog's plain package global, because zerologlog.Output(w) is Logger.Output(w). That is safe only while init stays its only writer. zerolog.New(w).Level(level).With().Timestamp().Logger() builds the same sink, since Timestamp() is a hook and Output copies hooks. With it, the zerolog/log import and the assignment in init could go.
  2. UseMockLogLevel's doc comment (protocol/chainlib/common_test_utils.go:270) still justifies itself by the race this PR removes. setMockLogLevelOnce's "(see there)" now points at a one-line comment.
  3. rollingLogLogger is now the only plain global that LavaFormatLog reads on every call. It's written at startup, where the default is off, and by TestLogEvent_ReachesEveryEnabledSink, so it's fine for this ticket.

The branch is 3 commits behind main and needs an update before it can merge. The update is conflict-free.

@avitenzer

Copy link
Copy Markdown
Contributor

@nimrod-teich #617 is approved and we'd like it rebase-merged. GitHub blocks it on the required dev-sim-prtests / preflight, which is red for a reason that has nothing to do with this PR.

Cause. The gate installs smart-router-helm-chart main on dev-prtests, and that cluster runs an Envoy Gateway older than v1.7.0. The chart now renders two fields that only exist from v1.7.0:

The install fails with field not declared in schema on both. I checked the Envoy Gateway CRDs: v1.5.0 to v1.6.1 lack both fields, and v1.7.0 onwards has them. The chart's docs/GATEWAY_API_SETUP.md still installs v1.6.1, while 6.30.0's changelog was rehearsed on v1.8.3.

Run 41451503238 (this PR, chart 6.30.0) and run 41347308204 (#624) fail the same way. The last green run, 41316433885, was on chart 6.25.0. The PR image itself built and started fine.

Fix. Upgrade Envoy Gateway on dev-prtests to v1.8.3, to match where the chart was rehearsed. No repo provisions that cluster: the gate only detects its GatewayClass, and it was set up in #126. If it was installed per GATEWAY_API_SETUP.md, the upgrade is a server-side apply of v1.8.3's install.yaml. Until then, every approval's preflight fails.

Then either re-run the gate for #617 or rebase-merge it with your bypass. The branch is 3 commits behind main. The rebase is conflict-free (canBeRebased: true), and build, vet, go test ./... and rpcsmartrouter -race all pass on it merged with main c3aa77d0.

avitenzer added a commit that referenced this pull request Nov 2, 2026
…MAG-4454)

TestGetUniqueGuidResponseForError_LogsTheGUID captured its lines by
assigning zerolog's global logger. MAG-4740 (#617) moves every
LavaFormat* call onto an atomic stderr sink, after which that assignment
captures nothing and the test fails, whichever of the two PRs lands
second.

The test now reads the debug log ring, as the listener test beside it
does. LavaFormatLog writes the same fields there whichever stderr sink is
installed, so the test holds before and after #617, and it still fails
when the error-masking line drops GUID.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 57 minutes
@avitenzer

Copy link
Copy Markdown
Contributor

Update: I merged main into this branch (0d56706, level with c3aa77d0) and re-ran the gate. Run 41478240947 failed at 16:09Z the same way as before: .spec.routingType and .spec.provider.kubernetes.envoyDaemonSet.pod.priorityClassName: field not declared in schema. So dev-prtests' Envoy Gateway is still older than v1.7.0.

#554, #555, #559 and #616 are now reviewed, approved and level with main too. With all five merged onto main together:

  • the build and go vet pass;
  • the touched packages' tests pass;
  • protocol/rpcsmartrouter passes under -race.

One cross-PR catch: #559's new GUID test captured logs by assigning zerolog's global logger, which this PR retires. It would have failed whichever of the two merged second. It's fixed on #559's side, so the two can merge in either order.

@nimrod-teich all five wait only on dev-sim-prtests / preflight. Once Envoy Gateway is upgraded we'll re-run the gate and rebase-merge them. If the upgrade will take a while, please rebase-merge them with your bypass.

@nimrod-teich
nimrod-teich merged commit aca006b into main Nov 2, 2026
@nimrod-teich
nimrod-teich deleted the mag-4740 branch November 2, 2026 21:12
nimrod-teich added a commit that referenced this pull request Nov 2, 2026
…name the GUID on the error line (MAG-4454) (#559)

* fix(utils): cut long log strings on a rune boundary (MAG-4454)

FormatLongString cut at byte offsets, so a multi-byte character at
either cut was split and the JSON log writer replaced its bytes with
U+FFFD. Each cut now moves inward to the nearest rune boundary, at most
three bytes; bytes that are not UTF-8 are cut where they were.

* fix(metrics): carry the request's GUID under GUID on the error-masking line (MAG-4454)

UniqueGuidResponseForError logged the request's GUID only under
msgSeed, so a search for the request's lines by "GUID":"<id>" missed
the line that records why it failed. It now carries GUID too, written
as a quoted decimal string like every other line; msgSeed stays. The
WebSocket marshal-failure line, which had the GUID only under seed,
gets GUID as well.

* fix(chainlib): cut the request body on every listener's log lines unless --debug-relays (MAG-4454)

The GraphQL and REST listeners logged the whole request body at info,
signed transactions included, and every listener's failure line
(LogRequestAndResponse, error level) logged it whole, JSON-RPC and
Tendermint included, although those two cut their own request line.

relayBodyForLog is now the one rule: the body's head and tail,
relayMsgLogMaxChars bytes in all, or the whole body under
--debug-relays. Each listener logs that form on its request line and
hands the same form to its failure line: JSON-RPC, Tendermint POST,
REST POST and non-POST, GraphQL, gRPC unary and stream subscribe, and
the WebSocket manager. The debug-level success lines are unchanged.

The WebSocket manager's RelayMsgLogMaxChars option only ever held the
package constant, so it goes; AnalyzeWebSocketErrorAndGetFormattedMessage
takes the logged frame as a string.

* docs(logs): name the stream label that selects relay_end writes (MAG-4454)

The doc said to select writes by the collector's stream label without
giving it. The collector labels router lines matching
"stateful":("1"|true) with stateful="1", which is also why a | json
extraction of stateful lands as stateful_extracted.

* test(metrics): read the GUID line test's output from the debug ring (MAG-4454)

TestGetUniqueGuidResponseForError_LogsTheGUID captured its lines by
assigning zerolog's global logger. MAG-4740 (#617) moves every
LavaFormat* call onto an atomic stderr sink, after which that assignment
captures nothing and the test fails, whichever of the two PRs lands
second.

The test now reads the debug log ring, as the listener test beside it
does. LavaFormatLog writes the same fields there whichever stderr sink is
installed, so the test holds before and after #617, and it still fails
when the error-masking line drops GUID.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Took 57 minutes

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: avitenzer <tenzer@clara.co.uk>
AnnaR-prog added a commit that referenced this pull request Nov 3, 2026
…l-dedup + review fixes)

Conflicts resolved so both sides survive:

- utils/lavalog.go: main's #617 (atomic globalLogLevel, atomic stdLogger sink,
  newStdLogger, SetStdLogger, currentLogLevel) kept whole. The PR's
  setStderrLogger is dropped: main's newStdLogger is the same builder, and
  setStderrLogger wrote the non-atomic zerolog global that #617 retired and
  read defaultGlobalLogLevel, which #617 removed. The PR's level table
  (logLevels, getLogLevel, RollingLoggerSetup) and its sink loop are kept,
  with the stderr event opened on stdLogger.Load() instead of
  &zerologlog.Logger. The commented-out logger block main deleted stays deleted.
- utils/lavalog_test.go: both tests kept (the PR's ToHexString pin, main's
  FormatLongString rune-boundary test), imports unioned.
- protocol/metrics/smartrouter_metrics_manager.go: the PR's register-as-built
  form, with main's help text (#554 batch/default overflow "invalid UTF-8",
  #621 customer-id, #624 csm_provider_blocked, #623 cache_failed_total).
- protocol/metrics/default_method_label_test.go (no textual conflict): main's
  #554 test called RecordRelaySuccess/RecordRelayError, which the PR folded
  into recordRelayOutcome; it now calls recordRelayOutcome with success true
  and false.
- utils/specfetcher (#606): github.go and gitlab.go are the PR's (its
  repoHost flow, a8fb8cd, already merges the listing flow #606 merged);
  listing_api.go is dropped, since nothing calls listingAPI or
  fetchFromListingAPI once that flow is in; listing_api_test.go is kept as
  main has it and now pins the PR's flow. Same resolution as the #604 x #606
  trial merge 5e0c88f3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants