Project Proposal: OpenTelemetry Support Self Assessment - #3435
kaspernissen wants to merge 8 commits into
Conversation
Signed-off-by: Kasper Borg Nissen <kasper.nissen@dash0.com>
Signed-off-by: Kasper Borg Nissen <kasper.nissen@dash0.com>
Signed-off-by: Kasper Borg Nissen <kasper.nissen@dash0.com>
|
LGTM, what’s the next step? I’m happy to not only participate but also serve as co-lead if that’s desirable @kaspernissen |
Signed-off-by: Kasper Borg Nissen <kasper.nissen@dash0.com>
|
Awesome, thanks @mhausenblas - really glad to have you on this. Just pushed the update with you as co-lead. From here:
Glad to have a group forming around this. |
| - workstream | ||
| - workstreams | ||
| - yahn | ||
| - Baykara |
|
Hi @kaspernissen. The thought behind this project means well, but I don't believe that it's appropriate for OpenTelemetry as a project to be scoring other projects in the CNCF based on their support for us. I discussed this with @caniszczyk and he agrees – the CNCF actively tries to avoid CNCF projects judging each other, as it's too easy for projects and maintainers to take offense at the grade we are giving them, even if our intentions are good. But, just to be clear, I think the spirit of what you are trying to do is great! I have some suggestions. It feels like the goals of this project are to improve OpenTelemetry support in OSS and to allow projects to promote themselves based on their support. If we remove the scoring aspect, what would that look like? One thing that would be very helpful would be implementation/integration guides on the website, based on the type of OSS project.
Down the road, it's possible that we could work with the CNCF to create certification programs for some of these classes of support. Our highest priority on that front is a certification program for the OpenTelemetry Collector; we'll definitely be doing that. But I could see the value in badges or certifications that cover other forms of OpenTelemetry support, like native instrumentation. Even before we have badges to hand out, we could give space for projects to promote their OpenTelemetry support on the website and in places like the registry (and its successor the ecosystem-explorer). Again, apologies, because I can see how excited you are for this. But I hope that feedback makes sense. Let me know what you think about these proposals for how to re-implement the goals you are after. Or, if I've misunderstood the goals, let me know what other goals you are trying to achieve. |
|
I like the intent of this project, but I have concerns about the specifics:
I suggest that we make the following changes to this proposal:
We don't want to be in the business of subjective, opinionated labeling of third parties. No matter how well intended we are there is good chance it will end up creating arguments and conflict. I suggest that we start small, within Otel and move carefully. If it works well within Otel and is proven to be a good, objective measure, we can consider extending the scope. |
|
Looks like @tedsuo and I cross-posted independently, but we mostly agree about concerning areas. |
|
@tedsuo @tigrannajaryan I am in the same page with you folks, but having talked with @kaspernissen I would love to reiterate this is not about scoring, it is about evaluating support and helping CNCF projects to grow. And in no way this is about vendors. I've been working a bit on the automation for this (as @tigrannajaryan mentioned "automatic evaluations where possible"), because having a framework to define what these evaluations are so projects can run them themselves is a must for an initiative like this. These evaluations then can be used for conformance too. As pointed out, libraries, Kubernetes controllers, sidecars, databases, message brokers all require different approaches for performing the evaluations, so what I would love to see this group do is to refine what those evaluations would be. The dimensions included in this proposal allows us to categorize the evaluation results so if a project is interested in improving one of its dimensions it can drill down to find specific suggestions. I hope this makes sense.. |
C'mon, man. This proposal literally states that OpenTelemetry should assign maturity levels to other CNCF projects. That is the main thrust of this proposal. It's true that this document does start by saying that it is not an evaluation, but then it immediately proposes that we grade projects on a scale of 0-3 based on their support for OpenTelemetry, with the main deliverable being a visualization for comparison purposes. How is that not an industry analyst report?
I completely agree with your point that all of these types of projects require different approaches. But precisely to that point, I don't see how a generic, high-level analysis framework would be very helpful to engineers that want to actually improve their telemetry. These kinds of frameworks and reports can definitely be helpful to CTOs and other decision makers at end user companies that are trying to decide which technologies to adopt. But that is also the primary reason why OpenTelemetry as a project shouldn't be issuing them. Saying "please don't use our comparative analysis to make judgements" wouldn't change their utility, and other projects will see through that. If the CNCF wants to develop reports on the state of observability within the CNCF, and use a framework like this, I don't think that anyone from this project would have any objections. But as a CNCF project, we've been directly asked by CNCF leadership to not get into this business, as part of a general guideline that CNCF projects should not evaluate each other.
If the ultimate value in this project lies in making these suggestions and providing better tools to implement them, how about we start there? A high-level comparative framework is not a required step for publishing implementation guides, and there's definitely a lot of excitement for both improved documentation and better tooling to support these efforts. We would absolutely appreciate any efforts on these fronts.
What do you think about organizing an effort around those pieces? Are there any specific parts that you are particularly interested in? I know I know, "the maturity model" is the specific part you're interest in! But help us understand something more concrete. Is there a particular kind of project you are most interested in helping? Libraries? Kubernetes? Message brokers? Can you present a run through on a specific project you're trying to engage with, where making progress is difficult? |
|
@tedsuo fair pushback. Let me clarify where I actually am on this, because I think we're closer than the thread suggests. The proposal is a draft, and was put forward to start exactly this kind of discussion. The dimensions and levels were one way to talk about maturity on a range. You're right that the doc as written specifies 0-3 levels and a comparative visualization. I'm fully open to dropping both the levels and the comparative visualization. They were a starting point for the conversation, not the point of the proposal. The point is the underlying observation: "supports OpenTelemetry" is becoming a binary claim that hides enormous variance across projects, and end users absorb that cost when they migrate their tooling. As OTel becomes the foundation for observability in many organizations, that variance is going to matter more, not less. I think we agree on this. The question is what to do about it. On your concrete suggestions, I'm in. Implementation guides per project type, better tooling, support for instrumentation score, native instrumentation guidance for libraries, semconv guidance for services. Those are the kind of deliverables that would actually help projects mature their OTel support, and they're concrete in a way a comparative framework isn't. The framing I'd add: applied guides. We work with maintainers to figure out where the gaps are, then write guides grounded in those gaps rather than in the abstract. The dimensions from the original proposal become internal scaffolding for what the guides need to cover, not a public deliverable. That's effectively the pattern the five gateway evaluations followed: structured look at where the gaps are, maintainer conversations, upstream changes, and then a writeup that captures what good looks like for that project type. There's also room for a deterministic evaluation tool alongside the guides. Something maintainers can run against their project's actual telemetry output and get back a factual report of what's emitted, what semconv it follows, and where the gaps are. No score, just diagnostic information they can act on. That's the direction @salaboy's automation work is already heading, and it complements the instrumentation score effort. It also gives the applied-guides pattern a way to scale past evaluations done by hand. On your specific question around project interest, I'm interested in helping where I can. I started with gateways, since that was a place where I saw visible variance in the support provided. That's really what sparked the idea for this draft proposal. On the CNCF leadership constraint, understood. If the framework dimensions turn out to be useful as input to a CNCF-level effort someday, that's a separate conversation. The OTel-side scope should be the implementation work. If that's roughly what you had in mind, I'd be glad to redraft the proposal along those lines and start the discussion in the group on which project types to prioritize after gateways. |
|
Thanks @kaspernissen for being so flexible! Regarding the "supports OpenTelemetry" concern, I completely agree with you. Unless we start providing more guidance, people will start making up their own definitions. But even beyond the definitions, there's nuance to OpenTelemetry support that people will miss without guidance, leading to sub-optimal implementations. It feels like we're all in alignment with solving those problems, and it's a good time to start thinking about them. I can think of three general classes of support that all require different guidance. Let me know if you see others.
In the first case, the OpenTelemetry Collector, we've seen some genuine confusion, with some companies concluding that "OpenTelemetry Collector" is a generic term. We added a definition of a Collector Distribution here to make it more clear that this is a trademarked term meant to refer to a specific codebase, with some wiggle room for forking and repackaging (but not infinite room). There's been interest in taking this another step now that we have graduated, and providing a Collector Certification. The CNCF only provides that kind of support for graduated projects, which is why we've been waiting. For the other two cases – library and service instrumentation – I like your idea of "applied guides" as well as providing tooling that can help make it easier, not to mention verify correctness. In the roadmap we are working on, the SemConv Tooling SIG is owning this workstream, with a focus on library instrumentation. But even without tooling, guides that describe common pitfalls and best practices would be helpful. I can see how your framework was aiming to address that, especially for services. If we turn this into guides I think we have agreement on a way to move forwards. Writing the guides will also show us the gaps in our toolchain for supporting this. All that said... I'm not sure we need a new SIG for this? The Comms SIG would be a great place to start the work on guides. @svrnm what do you think? The SemConv Tooling SIG like I mentioned is handling the tooling. And the Collector SIG along with the GC/TC can handle the Collector certification once the CNCF is able to make those resources available to us. |
|
Hi everyone, thanks @kaspernissen for kicking off this initiative, and a special thanks to @salaboy for pointing me to it. I’ve been reading through the thread, and I strongly agree with the concerns raised by @tigrannajaryan and @tedsuo regarding the risks of OpenTelemetry acting as a "rating agency" for third-party projects. Providing a top-down score could indeed lead to friction. However, I think the core need remains: CNCF projects and end-users need guidance on what "OTel support" actually means and how to improve it. To bridge this gap, I’d love to propose an alternative approach and offer my availability to contribute to it. Instead of a centralized scoring/reporting system, we could reframe the Maturity Model as an Opt-in Self-Assessment Framework. Here is what I’m proposing:
This approach keeps OpenTelemetry in the role of an enabler rather than a judge, while still providing the clarity the ecosystem needs and that this initiative aims to provide. It could also be worth considering involving one of the CNCF TAGs in this initiative to help us collaborate and reach other projects. I'd be more than happy to contribute to this initiative. Let me know what you think! |
Would like to have other @open-telemetry/docs-maintainers speak to this as well, here are my thoughts: we had the idea of growing the instructions (aka applied guides) for building components (for collector, but also instrumentation libararies and ESPECIALLY native instrumantation for libraries and applications) into the documentation for a long long time. But like many other things in docs it never grew beyond a certain base level because nobody owned that part of the docs. Our collector docs are right now a prime example for healthy docs, since we have a well functioning co-ownership between the SIG maintainers and @tiffany76 from our end. A similar success story is the ecosystem explorer which is part of Comms that is owned by @jaydeluca. Both projects are also so successful because they grow organically, so instead of doing one big splash we started small and build things incrementally. If we have 2-3 people who are really enthusiastic about making this happen, I am optimistic about it working out, but again deferring to other Comms maintainers. |
|
I support and agree with what @svrnm has shared. |
|
➕ 💯 on the underlying problem — "supports OpenTelemetry" has become a binary claim that can be used to mean anything!, and a more formal vocabulary is better. I'd love to see things like "this library instruments via OTel APIs and uses OTel semconv" or "this app honors OTEL_EXPORTER_OTLP_ENDPOINT and can ship to any OTLP backend." One observation about greenfield bias. Most of the seven dimensions (integration surface, semconv currency, OTEL_* config, multi-signal) nautrally favor projects that had OTLP and current otel conventions as day-1 design choices. Established projects that didn't have OTel when they started will naturally score lower. Also, if we can consider "observable behavior" - like does OTLP come out? is semconv followed? are OTEL_* working?, rather than implementation choices (using OTel API internally or other one) that'd be helpful too— it stops penalizing incumbents who reach the same outcome via a different path. Hopefully this can be addressed before the proposal is accepted |
reyang
added
the
triage:tc-reviewed
|
First, apologies for the slow turnaround. A few other priorities pulled me away, and I'll be on PTO in a couple of weeks. Before then, I want to leave this in a state where we can keep moving forward, so let me try to pull the threads together and suggest a path forward. Reading through the discussion again, I think there's a lot of alignment, and the conversation has helped sharpen the idea. A few points I want to explicitly support:
No new SIG for now. I agree we don't need a new SIG at this stage. Moving the work and discussion into the Comms SIG, as @svrnm suggested, feels like the right approach. We can grow it organically there and revisit the structure later if needed. Concrete next steps:
Since I'll be out for a while, I don't want this to stall. I'm happy to keep driving it when I'm back, but it would be great to find a co-driver or a few volunteers to keep the Comms SIG discussion moving in the meantime. If anyone is interested in picking up a thread, whether it's the tooling prototype or the guides, I'd love to collaborate. Thanks everyone for the thoughtful feedback so far. It's genuinely improved the proposal. |
Yes, it would be good to know for us who is eventually doing the work. This is too much work for one individual (@kaspernissen), so there should be 2 or 3 people more that sign up to help with this. |
|
Hey @kaspernissen @svrnm, sorry for the silence on this one. I was mostly on PTO over the last month and completely missed the conversation. I'd like to help drive this, and I'm glad the opt-in self-assessment framing resonated. That's the part I'm most interested in, so I'm happy to sign up as one of the co-drivers @svrnm is asking for, either on the tooling prototype or on the maintainer-facing guides. The one caveat is that I'm still fairly new to the OpenTelemetry project, so I'd need some guidance on how things work here. Before committing to something concrete, a couple of questions. Has the initiative moved forward in the meantime? I'm mostly wondering whether it made it into the Comms SIG as a discussion item, and whether the reframed proposal already exists somewhere I should be reading. And is there any documentation on the lifecycle of an initiative like this, meaning how it gets proposed, where it lives, and how it moves from discussion to actual deliverables? Happy to read up on my own so I don't reinvent anything or step on existing process. If it's easier than writing it all out here, I'm also glad to join a Comms SIG call or do a quick sync to align on next steps and get to know each other properly. |
…-maturity-model # Conflicts: # .cspell.yaml
|
@graz-dev welcome, and no need to apologize for going quiet. I was away most of July and have spent the last few weeks catching up, so the silence here is on me. Ready to pick it back up now. Your questions, honestly: The reframed proposal does exist, as of a few minutes ago. Short version: the 0-3 levels and the comparative assessment are gone, along with anything that has OpenTelemetry assessing other projects. What's left is opt-in tooling maintainers run against their own telemetry, plus guides per project type. Working group under Comms SIG, not a new SIG. @svrnm on staffing: I've put @graz-dev down as co-lead. That's two of us with something to own. I'd still like one or two more for the guides and work in general. @mhausenblas the scope has changed a lot since you offered to co-lead, so I've marked you as pending confirmation rather than assuming it carries over. Either way is fine, just let me know. @graz-dev being new isn't a problem, I'm fairly new to this part of the project myself and still learning how it works. The next SIG Comms meeting is Tuesday and I can't make that one. Let's get in touch and discuss, and then we can bring it to the next SIG Comms meeting. If you want to join Tuesday anyway, go for it, but no pressure to drive it solo. |
graz-dev
left a comment
graz-dev
left a comment
There was a problem hiding this comment.
Thanks @kaspernissen just added a couple of comments here.
I'll join the next SIG meeting to understang how things works, then let's sync to understand how to present the initiative :)
|
|
||
| OpenTelemetry has become the de facto standard for producing telemetry in cloud native systems, and a growing number of projects across the ecosystem emit their telemetry through it. But "supports OpenTelemetry" has become a binary claim that hides enormous variance in what a project actually delivers, and adopters absorb the cost of that variance when they integrate. | ||
|
|
||
| This project aims to help maintainers close that gap themselves. It has two parts: opt-in tooling a maintainer can run against their own project's telemetry to get a factual report of what is emitted and where the gaps are, and maintainer-facing guidance on what good OpenTelemetry support looks like for different classes of projects. |
There was a problem hiding this comment.
probably here a first artifact to work on could be a classification model for the projects.
categories into the CNCF landscape are more usage oriented we probably need something more architecture oriented (?).
There was a problem hiding this comment.
Yes, that's a good idea. Will add some wording around this being the first artifact to work on.
|
ya'll might want to follow this new project that might be helpful in providing some tooling to be used to validate/score semantic convention conformance as one dimension of maturity: https://github.com/open-telemetry/semantic-conventions-conformance |
kaspernissen
changed the title
Co-authored-by: Graziano Casto <graziano.casto@outlook.com>
|
Thank you, @jaydeluca! Very interesting project, highly relevant for these efforts as well. |
graz-dev
left a comment
graz-dev
left a comment
There was a problem hiding this comment.
Added Project classification model as first deliverable for the initiative and added more details about the self-assessment tool.
Co-authored-by: Graziano Casto <graziano.casto@outlook.com>
|
|
||
| > **Revision note.** An earlier version of this proposal described a descriptive maturity model with 0–3 levels per dimension, applied by this project to other cloud native projects and published as a comparative assessment. Review feedback made clear that OpenTelemetry should not be in the business of evaluating or labeling other projects, which is also consistent with CNCF guidance that projects avoid judging one another. That framing has been removed. The proposal now centers on self-assessment tooling and maintainer guidance, with the original dimensions retained only as internal scaffolding for what the tooling checks and the guides cover. See the discussion on [PR #3435](https://github.com/open-telemetry/community/pull/3435). | ||
|
|
||
| Support is rarely all-or-nothing. Projects mature unevenly: integration surfaces, semantics, configuration, trace modeling, and multi-signal workflows tend to evolve on different timelines. In practice, adopters hit projects where traces flow via OTLP but metrics are still Prometheus-only, where semantic conventions are several versions out of date, or where standard `OTEL_*` configuration is quietly ignored in favor of project-specific flags. Each integration becomes its own learning curve, and what works for one project rarely carries cleanly to the next. |
There was a problem hiding this comment.
standard
OTEL_*configuration
Note: If referring to env variables, then they are optional in OTel and support really varies across language implementations.
| ### Current challenges | ||
|
|
||
| - There is no shared definition of what "supports OpenTelemetry" means, so projects define it for themselves and adopters cannot tell what a given claim covers. | ||
| - Maintainers who want to improve their OpenTelemetry support have no way to check their project's actual telemetry output against current conventions and expectations. Gaps are found by users, not by maintainers. |
There was a problem hiding this comment.
support have no way to check their project's actual telemetry output against current conventions and expectations.
"no way" is a bit extreme phrasing. OTel's own Weaver tooling, esp. the live-check is a solid tool (still evolving) for this purpose.
|
|
||
| Objectives: | ||
|
|
||
| 1. **Opt-in self-assessment tooling.** Build tooling a maintainer runs against their own project's emitted telemetry, producing a factual report: what signals are emitted, which semantic conventions are followed, which standard configuration is honored, and where the gaps are. The output is diagnostic information and next steps, not a score, and it belongs to whoever ran it. |
There was a problem hiding this comment.
Hopefully, we are going to enhance OTel's own existing tools like weaver for this.
|
|
||
| #### Assessment dimensions (internal scaffolding) | ||
|
|
||
| The earlier draft organized OpenTelemetry support into seven dimensions. Those dimensions are retained as internal structure for deciding what the tooling checks and what the guides must cover. They are not a public grading rubric, and the associated 0–3 maturity levels have been dropped. |
There was a problem hiding this comment.
we probably don't need to describe the history here. Its useful in the PR desc, so folks reviewing it new has some idea how this proposal evolved.
|
|
||
| 2. **Self-assessment tooling.** Tooling a maintainer can point at their own project's telemetry to get a factual report of what is emitted and where the gaps are, with concrete next steps and links into the guides. Delivered incrementally. | ||
|
|
||
| **Scope for the first iteration: analysis of telemetry the maintainer already has.** The input is a captured OTLP payload produced by the project during something the maintainer already runs. The tool does not start the project, does not generate load, and does not re-run it under varying configuration. This keeps the first iteration deterministic and replayable, makes it usable in CI, and keeps the adoption cost close to zero. |
There was a problem hiding this comment.
could we use this to self-evaluate OTel itself? OTel clients emit their own telemetry, OTel Collector, OTel Arrow's engine all emit their own telemetry, following OTel sem conv (where applicable) or their own conventions.
If OTel itself can use this model, that'd be the biggest validation and advertisement for this.
11 participants
This PR contains a project proposal for the OpenTelemetry Support Maturity Model - a descriptive framework for evaluating how cloud native projects support OpenTelemetry across multiple dimensions.
Draft model can be seen here: https://docs.google.com/document/d/1KvRtYqdSR1ii-SLV2wEv0MH-j9Mh5xA5u7f51kO6xpw/edit?usp=sharing
Note: similar to #3000, this proposal will be used to socialize the effort and identify additional collaborators. Several contributors expressed interest on #3247; specific roles, TC sponsor, and GC liaison will be confirmed during review.
Why now
Future considerations (out of scope for this proposal)
Contributors on #3247 raised the idea of a follow-up conformance program in the spirit of Prometheus conformance. This proposal explicitly does not include conformance work - the descriptive model is the necessary first step, and a separate proposal would be the right place to take it further.
Related