Lab 1 - Observability Primer
Lab Goal
Define common observability and monitoring terms and gain an understanding the role
of OpenTelemetry and its components.
Introduction - What is observability?
How
effectively you can understand system behavior from the outside using the data it generates.
Introduction - What is monitoring?
The continuous process of watching and tracking system health based on pre-defined set of data.
Introduction - What is telemetry?
The process of recording and sending data from remote components to a backend.
Data types include: metrics, logs, events and traces.
Introduction - What is instrumentation?
The code that records and measure behavior of an application or infrastructure component.
Manual instrumentation includes any code you add and automatic instrumentation is what is
provided for you "out of the box".
Introduction - Hello, OpenTelemetry!
A set of standardized vendor-agnostic SDKs, APIs, and tools for ingesting, transforming,
and sending telemetry to observability back-end(s).
Introduction - Part of the CNCF
OTel was the result of merging OpenCensus and OpenTracing and joined the Cloud Native Computing
Foundation (CNCF) in 2019. Benefits include a broad ecosystem, active community engagement,
support, continued growth and investment.
Introduction - What OpenTelemetry is not
- only a tracing tool
- telemetry backend and storage system
- observability UI
Introduction - OpenTelemetry API/SDK
- API defines data types and how to generate telemetry data.
- SDK defines a language-specific implementation of the API, plus configuration, data processing and exporting.
Introduction - OpenTelemetry language SDKs
Introduction - OpenTelemetry instrumentation libraries
OTel instrumentation can already be found in a broad number of libraries - check the
Registry
to see current options.
In this workshop we'll use the
opentelemetry-instrumentation-flask
library
built on the OTel WSGI middleware to observe web requests in a Flask app.
Introduction - Registry for libraries
Check the
Registry
to see if your favorite library is already instrumented.
Introduction - OpenTelemetry collector
The Collector is a proxy that receives, processes and exports telemetry data in OTLP,
Prometheus, and many proprietary tools.
Introduction - OpenTelemetry protocol
The OpenTelemetry Protocol (OTLP) specification describes the encoding, transport, and delivery mechanism of telemetry data between telemetry sources, intermediate nodes such as collectors and telemetry backends.
Read more at OpenTelemetry Protocol Specification
Introduction - OpenTelemetry resource
A Resource represents the entity producing telemetry as resource attributes.
For example, a process producing telemetry that is running in a container on Kubernetes has a
process name, pod name, namespace, and possibly a deployment name. All four of these attributes
can be included in the Resource.
You can use resource information to better investigate interesting behavior. If your telemetry
indicates latency in your system, you can narrow down to a specific container, pod, or
Kubernetes deployment.
Read more at
Resource Semantic Conventions
Introduction - Types of instrumentation
- Automatic - observability agent runs alongside application and adds instrumentation without you
needing to make code changes
- Programmatic - a mix of both, where you pull in pre-instrumented dependencies and manually add metadata (e.g. labels)
- Manual - you set up observability library and add instrumentation code in your application
Introduction - Automatic instrumentation
No code changes - nothing to see here! Agent running alongside application injects
instrumentation at runtime:
from flask import Flask, request
app = Flask(__name__)
@app.route("/server_request")
def server_request():
print(request.args.get("param"))
return "served"
if __name__ == "__main__":
app.run(port=8082)
Introduction - Programmatic instrumentation
Use a framework/technology specific instrumentation library to capture basic method calls,
requires configuration and code changes:
from flask import Flask, request
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
BatchSpanProcessor,
ConsoleSpanExporter,
)
from opentelemetry.trace import get_tracer_provider, set_tracer_provider
set_tracer_provider(TracerProvider())
get_tracer_provider().add_span_processor(
BatchSpanProcessor(ConsoleSpanExporter())
)
instrumentor = FlaskInstrumentor()
app = Flask(__name__)
instrumentor.instrument_app(app)
@app.route("/server_request")
def server_request():
print(request.args.get("param"))
return "served"
if __name__ == "__main__":
app.run(port=8082)
Introduction - Manual instrumentation
Requires configuring OpenTelemetry libraries and instrumenting every method call you care about:
from flask import Flask, request
from opentelemetry.instrumentation.wsgi import collect_request_attributes
from opentelemetry.propagate import extract
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
BatchSpanProcessor,
ConsoleSpanExporter,
)
from opentelemetry.trace import (
SpanKind,
get_tracer_provider,
set_tracer_provider,
)
app = Flask(__name__)
set_tracer_provider(TracerProvider())
tracer = get_tracer_provider().get_tracer(__name__)
get_tracer_provider().add_span_processor(
BatchSpanProcessor(ConsoleSpanExporter())
)
@app.route("/server_request")
def server_request():
with tracer.start_as_current_span(
"server_request",
context=extract(request.headers),
kind=SpanKind.SERVER,
attributes=collect_request_attributes(request.environ),
):
print(request.args.get("param"))
return "served"
if __name__ == "__main__":
app.run(port=8082)
Lab completed - Results
We defined common terminology and gained a basic understanding of the OpenTelemetry project
and its components.
Next up, installing and configuring OTel in the demo application...
Contact - are there any questions?