[opentelemetry-instrumentation-genai-langchain] Record streaming telemetry for the LangChain instrumentation - #482
Merged
lmolkova merged 5 commits intoAug 28, 2026
Conversation
A streamed call was indistinguishable from a synchronous one. The callback-based handler never routes chunks through util-genai's stream wrappers, so gen_ai.request.stream and gen_ai.response.time_to_first_chunk were never set and the gen_ai.client.operation.time_to_first_chunk and time_per_output_chunk histograms never recorded a point. The wrappers own that bookkeeping for instrumentations that receive an SDK stream to proxy, which a callback handler never does. Add GenAIInvocation.record_stream_chunk() as the public entry point for reporting chunk arrival without a stream, and implement on_llm_new_token to call it once per output chunk. Streaming is inferred from a token arriving rather than from the stream invocation param, which LangChain derives from the model's streaming field and leaves False for a plain .stream() call. LangChain's contentless end-of-stream marker is skipped so it does not contribute a spurious near-zero inter-chunk gap.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds support for streaming timing telemetry to the callback-based LangChain instrumentation by introducing a util-level API (GenAIInvocation.record_stream_chunk()) that can be invoked from token callbacks, enabling gen_ai.request.stream, gen_ai.response.time_to_first_chunk, and the streaming timing histograms to be emitted for LangChain .stream() / .astream().
Changes:
- Add
GenAIInvocation.record_stream_chunk()inopentelemetry-util-genaito record per-chunk arrival timing without wrapping an SDK stream. - Implement
on_llm_new_tokenin the LangChain callback handler to callrecord_stream_chunk()for each real output chunk (skipping LangChain’s end-of-stream marker). - Add unit tests in both util-genai and langchain packages to validate streamed vs non-streamed behavior and marker handling.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| util/opentelemetry-util-genai/src/opentelemetry/util/genai/_invocation.py | Adds record_stream_chunk() to mark streaming and record per-chunk timing. |
| util/opentelemetry-util-genai/tests/test_handler_metrics.py | Adds direct tests for record_stream_chunk() and its metric/span effects. |
| util/opentelemetry-util-genai/.changelog/461.added | Towncrier fragment documenting the new util entry point. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/utils.py | Adds is_stream_end_marker() helper for filtering LangChain’s contentless terminal marker. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/callback_handler.py | Implements on_llm_new_token to report streaming chunk arrivals. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_streaming.py | Adds streaming/non-streaming tests and an end-of-stream marker regression test. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/.changelog/461.fixed | Towncrier fragment documenting the LangChain streaming telemetry fix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Pull request dashboard statusMerged · refreshed 2026-08-29 00:34 UTC Status above doesn't look right?
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
on_llm_new_token calls GenAIInvocation.record_stream_chunk(), which this branch adds to opentelemetry-util-genai and which the released 1.1b0 does not have. The declared floor therefore understated the requirement: the oldest tox factor installs only the instrumentation editable and resolves its declared deps from PyPI at their floors, so py310-test-instrumentation-genai-langchain-oldest would install util-genai 1.1b0 and raise AttributeError on the first streamed token. A released instrumentation would fail the same way for anyone resolving util-genai to the declared floor. Declare the current unreleased dev version and install util-genai from the workspace in the oldest requirements until it ships, matching what open-telemetry#269 did for the anthropic and openai instrumentations. scripts/check_deps.py enforces both halves of that pairing, and requires the editable install to be removed again once the floor points at a release. Also folds in the review follow-ups: rename the changelog fragments to the PR number, add a streaming conformance scenario mirroring the anthropic and openai packages, and restore the blank line ruff-format wants between record_stream_chunk and _on_stream_chunk.
sfc-gh-zeningchen
force-pushed
the
langchain-streaming-telemetry
branch
from
August 26, 2026 23:01
b434163 to
c78b4a2
Compare
lmolkova
approved these changes
Aug 26, 2026
lmolkova
left a comment
lmolkova
left a comment
Member
There was a problem hiding this comment.
Thank you! Please add conformance tests with expected streaming metrics and it should be good to go.
sfc-gh-zeningchen
force-pushed
the
langchain-streaming-telemetry
branch
2 times, most recently
from
August 27, 2026 01:24
d50e850 to
c295369
Compare
lmolkova
approved these changes
Aug 27, 2026
sfc-gh-zeningchen
force-pushed
the
langchain-streaming-telemetry
branch
from
August 27, 2026 07:35
c295369 to
10dda48
Compare
lmolkova
enabled auto-merge
August 28, 2026 23:21
github-project-automation
Bot
moved this to Done
in GenAI Semantic Conventions and Instrumentation libraries
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Streamed LangChain calls never set
gen_ai.request.streamorgen_ai.response.time_to_first_chunk, and never recorded thetime_to_first_chunk/time_per_output_chunkhistograms added in #269,because the shared stream wrappers are the only producers of those values and a
callback-based instrumentation has no stream to wrap. This adds
GenAIInvocation.record_stream_chunk()toopentelemetry-util-genaias theentry point for reporting chunk arrival without a stream, and implements
on_llm_new_tokento call it once per output chunk. The capability lives in theutil rather than the instrumentation, per the direction on #8.
Fixes #481
Type of change
How has this been tested?
New
tests/test_streaming.pyin the langchain package covers.stream(),.astream(), a non-streamed.invoke()asserting no stream attributes ormetric points appear, and a parametrized case separating LangChain's
end-of-stream marker from a genuinely content-free chunk. These patch the
model's
_stream/_astreamrather than replaying a cassette, because theassertions are about chunk timing, which a cassette does not reproduce. Two
tests in
test_handler_metrics.pycoverrecord_stream_chunk()directly.util-genaisuite: 332 passedlangchainsuite: 239 passedlangchain==0.3.21,langchain-core0.3.86, wherechunk_positiondoes not exist) and atlatest (
langchain==1.3.16,langchain-core1.6.0)tox -e precommit(ruff, ruff-format) andpyrightcleanChecklist