vlagent: preserve CRI stdout/stderr stream as a searchable field

[patsevanton]

Problem

vlagent's Kubernetes collector parses the CRI stdout/stderr marker but discards it, so logs cannot be filtered by output stream at query time. Alerting/querying can only match on _msg with regular expressions, which forces VictoriaLogs to scan the entire log stream (mostly stdout info/debug noise) just to find the rare error/panic line.

Expected behavior

Store the container output stream (the CRI stdout/stderr marker) as a dedicated, searchable field so it can be used as a cheap pre-filter, e.g.:

_time: 2m
  | kubernetes.pod_labels.app:=my-app
  | stream:=stderr
  | _msg:~"panic:"

This lets alerting rules (and any query) narrow down to stderr before running the expensive _msg regex, instead of matching against the whole stdout+stderr flow.

Current behavior

In app/vlagent/kubernetescollector/processor.go, parseCRILine reads the stream marker into criLine.stream (streamStdout / streamStderr), but it is used only to pick the correct partial-line merge buffer (partialCRIStdout / partialCRIStderr) in TryAddLine. After that the stream value is dropped and never added to the log fields:

• parseCRILine — processor.go:496-538 (stream parsed here)
• TryAddLine — processor.go:141-149 (stream used only for joinPartialLines)
• addLineInternal / addRow — processor.go:225-266 (only _msg, parsed JSON fields, klog fields and pod metadata are written; no stream field)

As a result, the only stream-related knob is -kubernetesCollector.streamFields, which configures VictoriaLogs _stream fields (default kubernetes.container_name,kubernetes.pod_name,kubernetes.pod_namespace) and has nothing to do with stdout/stderr.

Why it matters

Sending errors/panics/500s to stderr while normal logs go to stdout is a common, useful contract, but right now that separation is lost on ingestion. Without a stream field, every error/panic alert must run a regex over the full stdout+stderr flow, which is the most expensive operation for VictoriaLogs and grows with log volume. A stream:=stderr pre-filter would reduce the scanned data by ~99% for typical applications.

Proposed solution

Add support for preserving the output stream as a field. For example:

• always emit stream with value stdout or stderr from the parsed CRI marker; and/or
• gate it behind a flag such as -kubernetesCollector.includeStream (default false) for backward compatibility.

References

• Repo: https://github.com/VictoriaMetrics/VictoriaLogs
• Component: app/vlagent/kubernetescollector/processor.go
• Related prior work: partial CRI line merging across interleaved stdout/stderr (vlagent: interleaved stdout and stderr partial CRI lines not handled correctly #1215, PR app/vlagent: support interleaved CRI log streams #1218)

[vadimalekseev]

When implementing the Kubernetes collector, we discussed adding the stream field to logs, but decided against it for the following reasons:

1. It confuses users. Logs would end up with two fields: _stream and stream (or file.stream). Most users would not know which one to choose.
2. Most applications write to a single stream. Cases where an application writes to both streams simultaneously are rare, and cases where a user wants to filter logs by stream are even rarer.

If I understand correctly, your goal is query optimization. It might be better to optimize the regular expression or filter logs by other fields instead. For example, if you know that your structured logs are written to stdout and contain a level field, you can add a level:="" filter to remove those logs and leave only stderr.

If we add this functionality, it should be under a separate flag and disabled by default, so that only users who understand OS I/O streams know what it refers to. It should be hidden behind a flag with a clear name and should not confuse users with the _stream term.

I will bring this up with the team again so we don't make this decision in isolation.

[patsevanton]

We run a large set of log-based alerts against VictoriaLogs (panic / error / config-miss / retry). Most of them cannot pre-filter by CRI stdout/stderr, so every evaluation scans the full container log flow and then matches _msg.

That is expensive. Typical apps write info/debug JSON to stdout and the rare panic/error to stderr. Without a first-class stream field from vlagent, VictoriaLogs has to walk the noisy stdout volume just to find those lines.

Below is a live snapshot from one production cluster. Service and namespace names are anonymized. “Scanned now” is the number of log lines in the alert’s time window after namespace/cluster/container filters, before the expensive _msg match. “If stream:=stderr” is the same window restricted to CRI stderr.

Alert	Window	Scanned now (stdout+stderr)	If stream:=stderr	Reduction
[PANIC RECOVER] on ns-billing (level=error)	7d	773,255	273	99.96%
config miss: item icon (app-checkout-ui)	1h*	3,696	0†	—
config miss: currency info (app-checkout-ui)	1h*	3,696	0†	—
config miss: transaction name (app-checkout-ui)	1h*	3,696	0†	—
config miss: billing info (app-checkout-ui)	1h*	3,696	0†	—
routeType == url but routeUrl == null (app-checkout-ui)	1h*	3,696	0†	—
unknown routeType (app-checkout-ui)	1h*	3,696	0†	—
failed to load/refresh data/configs (app-catalog-gw)	10m	1,088,598	857	99.92%
retry with retries>=5 (app-catalog-consumer-bg)	5m	160	0†	—
panic (app-storefront)	1m	50	0†	—
panic (app-inventory-guard)	1m	99,612	0†	—
panic (container=~app-catalog.*)	1m	132,224	14	99.99%
panic (ns-geo)	5m	40,610	70	99.83%
panic (ns-catalog-gw)	5m	604,906	446	99.93%
panic (app-bid-store.*)	1h	20,594	242	98.82%
error minus known noise (ns-bid-store)	30m	33,042	242	99.27%
error minus known noise (ns-ads-publisher)	5m	1,011,712	0†	—
error + "context" (ns-ads-publisher)	1h	2,894,957	0†	—
error minus known noise (ns-ads-builder)	1h	54,730	227	99.59%
"deadlock" (ns-ads-builder)	30m	54,624	226	99.59%
error minus kafka/tcp noise (ns-ads-auction)	30m	3,660	3	99.92%
same, shorter window (ns-ads-auction)	5m	1,224	3	99.75%
error + "context" + duration extract (ns-ads-publisher)	30m	2,892,672	0†	—
cronjob errors (app-bid-store-stats-loader)	1h	0	0	—
"panic:" on stderr (ns-ads-publisher)	5m	already stream="stderr"	0	n/a
"panic:" (ns-ads-builder)	5m	18,392	79	99.57%
empty payer number (ns-partner-credit)	3m	260	0†	—

* These rules have no _time filter in production. The 1h counts are a lower bound; the real scan is the full retained range.

† stderr is empty because this app currently writes JSON (including errors) to stdout. stream:=stderr only helps after the process actually emits panics/errors on stderr. That is the contract this issue is about.

Takeaways

1. For apps that already split CRI streams, a stream:=stderr pre-filter drops scanned volume by ~99.5–99.99% on the noisy services (app-catalog-gw ~1.09M → 857 lines / 10m; ns-ads-builder ~55k → 227 / 1h; ns-geo ~41k → 70 / 5m).
2. Several high-rate alerts (ns-ads-publisher ~1.0M / 5m, ~2.9M / 1h) currently have zero stderr lines. They still pay a full _msg scan over stdout. Preserving the CRI marker does not magically shrink those queries until logging goes to stderr — but without the field, there is no cheap way to write the alerts that can use the split.
3. Two panic alerts already try stream="stderr". That only works in our setup because another collector path stores stream. vlagent’s Kubernetes collector parses the CRI marker and then drops it, so the same alerts cannot be written against vlagent-ingested logs.

Please keep the CRI stdout/stderr marker as a searchable field (always, or behind -kubernetesCollector.includeStream). For log-based alerting this is the difference between scanning millions of info lines per evaluation and scanning a few hundred stderr lines.

[valyala]

Let's add add this field with the name output_stream. In this case it won't be confused with the _stream. There is no need in hiding this field under command-line flag, since it doesn't resolve any issues, while complicates configuring of the vlagent. So new version of vlagent must add this field unconditionally.

[angelofallars]

Hiya! I'm working on this issue and will have a PR out soon

[angelofallars]

@vadimalekseev I just opened #1791 which adds a new field to the Kubernetes Collector called _output_stream (I added a leading underscore cause that seems to be the convention for "internal" fields). Eagerly awaiting review!

[vadimalekseev]

@angelofallars thank you for creating the pull request!

which adds a new field to the Kubernetes Collector called _output_stream (I added a leading underscore cause that seems to be the convention for "internal" fields)

Could you please rename it to output_stream as @valyala mentioned? This is not an internal field of vlagent or VictoriaLogs - it is a regular field, like Pod labels, ip and others.

Currently, field names starting with an underscore represent one of key concepts and output stream is not one of them. Also, field names sorted alphabetically, so _output_stream will be at the top of fields list and can confuse users.

[angelofallars]

@angelofallars thank you for creating the pull request!

which adds a new field to the Kubernetes Collector called _output_stream (I added a leading underscore cause that seems to be the convention for "internal" fields)

Could you please rename it to output_stream as @valyala mentioned? This is not an internal field of vlagent or VictoriaLogs - it is a regular field, like Pod labels, ip and others.

Currently, field names starting with an underscore represent one of key concepts and output stream is not one of them. Also, field names sorted alphabetically, so _output_stream will be at the top of fields list and can confuse users.

Thanks for the explanation! I just learned that key concepts are represented with underscores. I just renamed it to output_stream in the PR.

[vadimalekseev]

vlagent preserves CRI output stream in the field named output_stream starting from the commit 0acff4d. This commit will be included in the next release of VictoriaLogs. In the meantime, you can verify this functionality by building vlagent from the commit according to these instructions.