OpenTelemetry Is No ‘Magic Button’ for Observability

OpenTelemetry has emerged as an essential component of the observability experience, as its continued development increasingly covers DevOps needs for the developer and operations. However, as one of the major open source projects at this time, it still needs work. Its functionality — while arguably already essential for observability — remains a work in progress, the success of which hinges on the continued support and hard work of the community. Its usefulness also depends on the observability tools and platforms used in conjunction with OpenTelemetry.

“Some people treat OpenTelemetry as a magic button, but ultimately, it’s supposed to help you better understand your observability data — and that depends on what you’re using it with,” Ryan Perry, an engineering director at Grafana, told The New Stack. “The whole point of observability is to be able to ask questions about your systems and get answers without needing to add new and siloed instrumentation.”

Before and After

But before OpenTelemetry, and even before observability, there was monitoring. Telemetry data consisting of logs, metrics, and, more recently, traces, offers the required data to be scrutinized or collected with monitoring. However, once it’s collected and observed through monitoring, it doesn’t mean that much if the data has not been parsed or channeled in an appropriate way to eliminate irrelevant telemetry data.

At the same time, the observation of events or performance as an operator using different telemetry data is useful and has been useful to a certain extent. But it falls short of observability, which involves the analysis and drawing actionable insights based on inferences from the data that was collected using monitoring.

OpenTelemetry offers a standardized process for observability. It can be seen as three main components: standards, SDKs, and the collector. The standards ensure interoperability, the SDKs simplify application instrumentation and the collector acts as a vendor-neutral agent.

It is used to make sense of telemetry data consisting of metrics, logs, and traces. It is also more than just being vendor-neutral in that it is designed to allow the user to integrate the observability tools of their choice into a common approach, unifying them.

Since its inception in 2019, OpenTelemetry became workable for accommodating telemetry data consisting of traces, followed by metrics, and, more recently, logs in 2023.

What this means in practice is users will look to observability to gain an understanding of and do other things. Usually, you can ask your live collection of information about your systems, such as why a server ran out of memory, why a trace is slow, why a request is slow, or why there are error logs, and be able to get that answer without doing something new. A collector is strong in getting logs, metrics, traces, and profiles in a way that is increasingly integrated over time, pointing you in the right direction to the right tool to answer your questions. But in practice, it is a tool such as those like Grafana Cloud Profiles (powered by the OSS database Pyroscope) that can supplement your observability to “better answer whatever question you’re asking with logs, metrics, traces, profiles, or some combination of all of these,” said Grafana’s Perry, who was also the CEO and co-founder of continuous profiling tool provider Pyroscope, which Grafana Labs acquired in 2023.

Leading observability providers offer support, maintenance, and development of OpenTelemetry. These include organizations such as Grafana, Honeycomb, Datadog, Dynatrace, Splunk and others. They have a collective interest in making OpenTelemetry better.

At the same time, they also offer a lot of work to, in some cases, make OpenTelemetry even better by offering tools that take advantage of its functionalities. These tools provide different analytics often specialized for various use cases, as they compete to offer unique features. They do this by contributing to the project itself and by creating and improving its solutions that work in conjunction with OpenTelemetry, namely providing the deep analytics required for proper observability.

In the case of Grafana Labs, this is done through its Grafana visualizations of various telemetry data, of course. Grafana has played a crucial role in improving the usability of its visualizations and apps, as well as contributing significantly to OpenTelemetry as one of its top ten contributors.

Interoperability

Grafana is indeed a huge proponent of OpenTelemetry’s vision of interoperability and the lack of vendor lock-in, Grafana CTO Tom Wilkie, told The New Stack during GrafanaCon, its annual user’s conference, in Amsterdam in April. “This resonates with Grafana’s ‘Big Tent’ philosophy and our culture of open thinking. However, OpenTelemetry is a vast project with a lot of variability in quality. Some areas, like metrics, haven’t fully lived up to their promise yet,” he said.

In metrics, for example, there are issues such as no way to remove a metric once it’s declared, leading to potential memory leaks “if there’s a lot of churn,” Wilkie said. “These rough edges prevent us from wholeheartedly recommending it for production use for metrics right now — but we’re working on fixing those. We see it as our responsibility to smooth out these issues. Additionally, it’s crucial to note that none of this is proprietary,” he said. “We contribute to the project, ensuring that our improvements benefit the whole community. We have our distribution of OpenTelemetry components, which allows us to ship bug fixes and security updates quicker than waiting for upstream approvals.”

OpenTelemetry integration for a Grafana Alloy configuration to use with Grafana Application Observability.

Grafana Alloy distribution of the OpenTelemetry Collector is open source, as well as its distributions of the associated .NET and Java SDKs. “Customers seek something they can rely on with support and maintenance contracts, which we provide,” Wilkie said.

Beyla

Grafana’s open source Beyla offers tracing via eBPF, which is another way to collect traces for telemetry. Its development parallels that of the OpenTelemetry project’s profiler, the development of which Grafana Labs is heavily involved as a major contributor. The profiler should prove to be useful for users because it goes deeper for observability analysis by extending to the code level. It instrumentalizes a deeper analysis of metrics, traces and logs, by extending telemetrics data pulled together in a unified stream that extends to the code level for applications throughout the network. Code is analyzed and stored.

With Beyla, you’re working more towards profiling, which is closely linked to tracing, Ryan Perry, an engineering director at Grafana, told The New Stack. “Profiling can tell you what your code was actually doing for a specific period, linking trace spans with profiling data,” Perry said. “For example, Beyla might tell you that your code was running for seven seconds, and profiling will break down how that time was spent on different functions.”

BC Gain is founder and principal analyst for ReveCom Media. His obsession with computers began when he hacked a Space Invaders console to play all day for 25 cents at the local video arcade in the early 1980s. He then…

Read more from B. Cameron Gain