Repository navigation
fix(utils): make the log level and stderr logger safe to change while others log (MAG-4740) - #617
Conversation
…hile others log (MAG-4740) SetGlobalLoggingLevel assigned the level and zerolog's package Logger with plain writes, and every LavaFormat* call reads both. Under -race, any test that set the level failed whenever an earlier test's goroutine was still logging: TestHealthCommand_RejectsUnbootableCrossValidation after a crafted-relay test's connector, and the metrics level tests after TestHappyFlow's monitor, which ran on context.Background(). The level is an atomic.Int32 and the stderr sink an atomic.Pointer[zerolog.Logger], as debugBufferLogger already is. SetStdLogger swaps the sink for tests that capture output; the four tests that assigned zerolog's global logger use it. LogEvent writes through the same sink. TestHappyFlow's monitor runs on t.Context(). TestLoggingStateChangesWhileOthersLog changes both while goroutines log; on main the same scenario reports 11 races.
avitenzer
left a comment
There was a problem hiding this comment.
Approving. I reviewed the diff and re-ran the evidence locally. CI here has no go test lane, so these runs are the only test evidence.
The bug and the fix reproduce. Base 0a971917 against this head af13ae8a, on the same machine at the same time:
| Run | base | this PR |
|---|---|---|
go test -race -count=3 ./protocol/metrics/ |
fails: SetGlobalLoggingLevel from TestGetUniqueGuidResponseDeterministic races TestHappyFlow's monitor goroutine |
passes |
crafted relay + health command, -race -count=6 |
fails: the health command's SetGlobalLoggingLevel races a leftover chainproxy Connector loop |
6 of 6 pass |
The regression test catches both halves. TestLoggingStateChangesWhileOthersLog can't compile on main because it calls SetStdLogger. So I ran it with -overlay against two mutants of this PR's lavalog.go. Reverting only the level to a plain variable gives 3 races and a FAIL. Reverting only the sink gives 36 races and a FAIL.
Also passing:
go build ./...andgo vet ./....utils,redisstoreandperformanceunder-race, so theirTestMaincaptures still capture.- The 51
Fleet|CrossValidationtests inrpcsmartrouterunder-race. - This head merged onto current main
c3aa77d0, which is conflict-free:go test ./...passes in all 38 packages, and the fullgo test -race ./protocol/rpcsmartrouter/passes.
Checked and fine:
utilsis the only importer ofgithub.com/rs/zerolog/login the dependency graph of./cmd/..., so leaving zerolog's global at itsinitvalue changes no output.- No
utilsvar initializer or earlierinit()logs, so the nilstdLoggerbeforeinitcan't be reached.debugBufferLoggeruses the same pattern. - Both commands still set
JsonFormatbefore the level.
Optional, none blocking:
newStdLoggerstill reads zerolog's plain package global, becausezerologlog.Output(w)isLogger.Output(w). That is safe only whileinitstays its only writer.zerolog.New(w).Level(level).With().Timestamp().Logger()builds the same sink, sinceTimestamp()is a hook andOutputcopies hooks. With it, thezerolog/logimport and the assignment ininitcould go.UseMockLogLevel's doc comment (protocol/chainlib/common_test_utils.go:270) still justifies itself by the race this PR removes.setMockLogLevelOnce's "(see there)" now points at a one-line comment.rollingLogLoggeris now the only plain global thatLavaFormatLogreads on every call. It's written at startup, where the default isoff, and byTestLogEvent_ReachesEveryEnabledSink, so it's fine for this ticket.
The branch is 3 commits behind main and needs an update before it can merge. The update is conflict-free.
|
@nimrod-teich #617 is approved and we'd like it rebase-merged. GitHub blocks it on the required Cause. The gate installs
The install fails with Run 41451503238 (this PR, chart 6.30.0) and run 41347308204 (#624) fail the same way. The last green run, 41316433885, was on chart 6.25.0. The PR image itself built and started fine. Fix. Upgrade Envoy Gateway on dev-prtests to v1.8.3, to match where the chart was rehearsed. No repo provisions that cluster: the gate only detects its GatewayClass, and it was set up in #126. If it was installed per Then either re-run the gate for #617 or rebase-merge it with your bypass. The branch is 3 commits behind main. The rebase is conflict-free ( |
…MAG-4454) TestGetUniqueGuidResponseForError_LogsTheGUID captured its lines by assigning zerolog's global logger. MAG-4740 (#617) moves every LavaFormat* call onto an atomic stderr sink, after which that assignment captures nothing and the test fails, whichever of the two PRs lands second. The test now reads the debug log ring, as the listener test beside it does. LavaFormatLog writes the same fields there whichever stderr sink is installed, so the test holds before and after #617, and it still fails when the error-masking line drops GUID. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 57 minutes
|
Update: I merged #554, #555, #559 and #616 are now reviewed, approved and level with
One cross-PR catch: #559's new GUID test captured logs by assigning zerolog's global logger, which this PR retires. It would have failed whichever of the two merged second. It's fixed on #559's side, so the two can merge in either order. @nimrod-teich all five wait only on |
…name the GUID on the error line (MAG-4454) (#559) * fix(utils): cut long log strings on a rune boundary (MAG-4454) FormatLongString cut at byte offsets, so a multi-byte character at either cut was split and the JSON log writer replaced its bytes with U+FFFD. Each cut now moves inward to the nearest rune boundary, at most three bytes; bytes that are not UTF-8 are cut where they were. * fix(metrics): carry the request's GUID under GUID on the error-masking line (MAG-4454) UniqueGuidResponseForError logged the request's GUID only under msgSeed, so a search for the request's lines by "GUID":"<id>" missed the line that records why it failed. It now carries GUID too, written as a quoted decimal string like every other line; msgSeed stays. The WebSocket marshal-failure line, which had the GUID only under seed, gets GUID as well. * fix(chainlib): cut the request body on every listener's log lines unless --debug-relays (MAG-4454) The GraphQL and REST listeners logged the whole request body at info, signed transactions included, and every listener's failure line (LogRequestAndResponse, error level) logged it whole, JSON-RPC and Tendermint included, although those two cut their own request line. relayBodyForLog is now the one rule: the body's head and tail, relayMsgLogMaxChars bytes in all, or the whole body under --debug-relays. Each listener logs that form on its request line and hands the same form to its failure line: JSON-RPC, Tendermint POST, REST POST and non-POST, GraphQL, gRPC unary and stream subscribe, and the WebSocket manager. The debug-level success lines are unchanged. The WebSocket manager's RelayMsgLogMaxChars option only ever held the package constant, so it goes; AnalyzeWebSocketErrorAndGetFormattedMessage takes the logged frame as a string. * docs(logs): name the stream label that selects relay_end writes (MAG-4454) The doc said to select writes by the collector's stream label without giving it. The collector labels router lines matching "stateful":("1"|true) with stateful="1", which is also why a | json extraction of stateful lands as stateful_extracted. * test(metrics): read the GUID line test's output from the debug ring (MAG-4454) TestGetUniqueGuidResponseForError_LogsTheGUID captured its lines by assigning zerolog's global logger. MAG-4740 (#617) moves every LavaFormat* call onto an atomic stderr sink, after which that assignment captures nothing and the test fails, whichever of the two PRs lands second. The test now reads the debug log ring, as the listener test beside it does. LavaFormatLog writes the same fields there whichever stderr sink is installed, so the test holds before and after #617, and it still fails when the error-masking line drops GUID. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Took 57 minutes --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: avitenzer <tenzer@clara.co.uk>
…l-dedup + review fixes) Conflicts resolved so both sides survive: - utils/lavalog.go: main's #617 (atomic globalLogLevel, atomic stdLogger sink, newStdLogger, SetStdLogger, currentLogLevel) kept whole. The PR's setStderrLogger is dropped: main's newStdLogger is the same builder, and setStderrLogger wrote the non-atomic zerolog global that #617 retired and read defaultGlobalLogLevel, which #617 removed. The PR's level table (logLevels, getLogLevel, RollingLoggerSetup) and its sink loop are kept, with the stderr event opened on stdLogger.Load() instead of &zerologlog.Logger. The commented-out logger block main deleted stays deleted. - utils/lavalog_test.go: both tests kept (the PR's ToHexString pin, main's FormatLongString rune-boundary test), imports unioned. - protocol/metrics/smartrouter_metrics_manager.go: the PR's register-as-built form, with main's help text (#554 batch/default overflow "invalid UTF-8", #621 customer-id, #624 csm_provider_blocked, #623 cache_failed_total). - protocol/metrics/default_method_label_test.go (no textual conflict): main's #554 test called RecordRelaySuccess/RecordRelayError, which the PR folded into recordRelayOutcome; it now calls recordRelayOutcome with success true and false. - utils/specfetcher (#606): github.go and gitlab.go are the PR's (its repoHost flow, a8fb8cd, already merges the listing flow #606 merged); listing_api.go is dropped, since nothing calls listingAPI or fetchFromListingAPI once that flow is in; listing_api_test.go is kept as main has it and now pins the PR's flow. Same resolution as the #604 x #606 trial merge 5e0c88f3. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#Closes MAG-4740
Jira ticket: MAG-4740
Note
Left unmerged for review. The change is to the logger's shared state, plus tests.
Why
utils.SetGlobalLoggingLevelassigned two process-wide variables without synchronization:LavaFormat*call reads before formatting;Logger, which every enabled call reads.Under
-race, any test that sets the level failed whenever a goroutine left over from an earlier test was still logging.mainfails the same way:mainunder-raceTestHealthCommand_RejectsUnbootableCrossValidationTestGetUniqueGuidResponseDeterministic,TestSetGlobalLoggingLevel_VolumeChangesAcrossLevels-count=3TestHappyFlow's relays monitor, started oncontext.Background()A test that changes the level while four goroutines log reports 11 races on
main.What changed
utils/lavalog.go: the level is anatomic.Int32, and the stderr sink is anatomic.Pointer[zerolog.Logger]. That is the patterndebugBufferLoggerbeside them already uses.SetGlobalLoggingLevelstores both.LavaFormatLogandIsDebugEnabledload them.initstill sets zerolog's global logger once, before any goroutine. Nothing else in the binary importsgithub.com/rs/zerolog/log.utils.SetStdLogger(logger) (restore func())swaps the sink for tests that capture output. The four tests that assigned zerolog's global logger now use it: the redisstore and performanceTestMains, the metrics page test, and the cross-validation fleet capture. The in-packagecaptureLoginutilsuses it too.utils/log_event.go:LogEventwrites through the same sink.TestHappyFlow: its monitor runs ont.Context(), so its probes stop logging when it ends.CreateChainLibMocks: the comment above it describes the code as it is.How to verify
mainprotocol/metrics,-race -count=3-race -count=6TestLoggingStateChangesWhileOthersLogis the regression test. It only finds a race under-race, and it passes without it on either code.Also run here:
utils,ecosystem/cache/redisstoreandprotocol/performancepass under-race, so the capture tests still capture.FleetandCrossValidationtests inprotocol/rpcsmartrouterpass under-race.go vet ./...passes.Full
go test ./protocol/rpcsmartrouter/passes without-race(252 s).Screenshots
No UI change.