Lab Goal
This lab walks you through one approach to linking metrics to traces via exemplars.
Traces & metrics - What's an exemplar?
Exemplars are the way to connect traces to metrics allowing you to jump from "hey what's that weird spike?" on a metric chart
directly to a trace associated with that context with one click!
Traces & metrics - Open source metrics
-
OpenTelemetry -
Metrics SDK is marked as "mixed" and under active development. Python SDK does not yet
support exemplars.
-
Prometheus - Stable APIs,
extensive ecosystem of instrumented libraries and dashboards.
Our goal is to tie metrics with traces via exemplars, so we'll instrument metrics with Prometheus and keep
traces instrumented with OpenTelemetry!
Traces & metrics - Prometheus config file
In the metrics/prometheus
directory you can view the config file named
prometheus.yml
:
global:
scrape_interval: 5s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
Normally you monitor other targets over HTTP and scrape their endpoints, but we are
going to start with Prometheus itself. The scrape_configs section is where you tell Prometheus which targets to
scrape to collect metrics from.
Traces & metrics - Building Prometheus image
Run the following command in your terminal:
$ podman build -t workshop-prometheus:v2.54.1 -f ./metrics/prometheus/Buildfile-prom
Traces & metrics - Running Prometheus
Run the following command to start the Prometheus container:
$ podman run -p 9090:9090 workshop-prometheus:v2.54.1 --config.file=/etc/prometheus/prometheus.yml
Verify our targets are configured by opening the targets page in the Prometheus console at
http://localhost:9090/targets
,
noting that Prometheus is scraping itself and waiting for the
hello-otel
application to come online:
Traces & metrics - Stopping Prometheus container
For now we'll stop the container by using CTRL-C
, as we will later run the
application and metrics collection tooling together in a single pod.
Traces & Metrics - Using Prometheus metrics
Traces & metrics - Modifying the build
Open metrics/Buildfile-metrics and ensure the command to install the
prometheus_flask_exporter
has been added:
FROM python:3.12-bullseye
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
RUN pip install opentelemetry-api \
opentelemetry-sdk \
opentelemetry-exporter-otlp \
opentelemetry-instrumentation-flask \
opentelemetry-instrumentation-jinja2 \
opentelemetry-instrumentation-requests \
prometheus-flask-exporter
COPY . .
CMD [ "flask", "run", "--host=0.0.0.0"]
Traces & metrics - Adding imports
Open the file metrics/app.py
and ensure the imports include the line
using the prometheus-flask-exporter
to import
PrometheusMetrics
as shown:
from opentelemetry.trace import set_tracer_provider
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from prometheus_flask_exporter import PrometheusMetrics
Traces & metrics - Instrument Prometheus Flask metrics
Continue down the file metrics/app.py
and verify that the programmatic Flask
metric instrumentation has been added as shown:
provider = TracerProvider()
processor = SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
provider.add_span_processor(processor)
set_tracer_provider(provider)
app = Flask("hello-otel")
FlaskInstrumentor().instrument_app(app)
Jinja2Instrumentor().instrument()
RequestsInstrumentor().instrument()
metrics = PrometheusMetrics(app)
Traces & metrics - Building the app image
$ podman build -t hello-otel:metrics -f metrics/Buildfile-metrics
Verify you get a success message in the console like below:
Successfully tagged localhost/hello-otel:metrics
81039de9e73baf0c2ee04d75f7c4ed0361cd97cf927f46020e295a30ec34af8f
Traces & metrics - Run the pod
Run the pod with the app, Prometheus, and Jaeger:
$ podman play kube metrics/app-prom-pod.yml
Open a browser and make several requests to the pages below to generate metrics and tracing data:
Traces & metrics - Verifying Flask metrics
Prometheus uses a pull-model where a Prometheus server scrapes a
/metric
endpoint. Let's check our metrics endpoint to verify that the
prometheus-flask-exporter
is working. Open
http://localhost:8001/metrics and confirm you see
metrics prefixed with
flask_*
and
python_*
:
Traces & metrics - Verifying targets up in Prometheus
Verify our targets are up and being scraped by opening the targets page in the Prometheus
console at
http://localhost:9090/targets
,
noting that Prometheus is scraping itself and now also the
hello-otel
application (with the status UP):
Traces & metrics - Stopping the pod
Now let's stop the pod and all the containers running so that we can start adding our exemplars
to explore a journey from metrics to traces:
$ podman play kube metrics/app-prom-pod.yml
Traces & metrics - Updating Prometheus configuration
We start by adding exemplars to our Prometheus metrics which will be tying in our traces from
OpenTelemetry. Open up the metrics/prometheus/prometheus.yml
file, add or
ensure there is a scrape job defined for the hello-otel
application as
shown, and save the file:
global:
scrape_interval: 5s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "hello-otel"
static_configs:
- targets: ["localhost:5000"]
Traces & metrics - Updating Prometheus buildfile
Open up the metrics/prometheus/Buildfile-prom
and ensure the CMD is using
the exemplar storage feature by adding the flag as shown:
FROM prom/prometheus:v2.54.1
ADD prometheus.yml /etc/prometheus
ENTRYPOINT [ "prometheus" ]
CMD [ "--config.file=/etc/prometheus/prometheus.yml", "--enable-feature=exemplar-storage" ]
Traces & metrics - Rebuilding Prometheus
Run the following command in your terminal:
$ podman build -t workshop-prometheus:v2.54.1 -f metrics/prometheus/Buildfile-prom
Traces & metrics - Updating imports
Open metrics/app.py and ensure the import for the Counter
metric from Prometheus Client has been added as shown:
import random
import re
import urllib3
import requests
from flask import Flask, render_template, request
from breeds import breeds
from opentelemetry.trace import set_tracer_provider
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from prometheus_flask_exporter import PrometheusMetrics
from prometheus_client import Counter
Traces & metrics - Create a counter
In the same file metrics/app.py ensure that a counter for the number of
homepage loads is added as shown:
provider = TracerProvider()
processor = SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
provider.add_span_processor(processor)
set_tracer_provider(provider)
app = Flask("hello-otel")
FlaskInstrumentor().instrument_app(app)
Jinja2Instrumentor().instrument()
RequestsInstrumentor().instrument()
metrics = PrometheusMetrics(app)
HITS_COUNTER = Counter('hits_counter', 'count of homepage loads')
...
Traces & metrics - Increment the counter
Continuing down in the same file, ensure code is added in the index()
to
increment (increase) the hits_counter
and configure
exemplars to attach to the metric as shown. Save the file when done:
@app.route('/')
def index():
global HITS
span = trace.get_current_span()
trace_id = '{:032x}'.format(span.get_span_context().trace_id)
HITS = HITS + 1
span.set_attribute("hits", HITS)
HITS_COUNTER.inc(1, exemplar={"trace_id": trace_id, "trace_url": f"http://localhost:16686/trace/{trace_id}"})
msg = f'This webpage has been viewed {HITS} times'
return msg
...
Traces & metrics - Updating app image
Open metrics/Buildfile-metrics
and ensure the command to install
prometheus_flask_exporter
is added as shown:
FROM python:3.12-bullseye
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
RUN pip install opentelemetry-api \
opentelemetry-sdk \
opentelemetry-exporter-otlp \
opentelemetry-instrumentation-flask \
opentelemetry-instrumentation-jinja2 \
opentelemetry-instrumentation-requests \
prometheus-flask-exporter
COPY . .
CMD [ "flask", "run", "--host=0.0.0.0"]
Traces & metrics - Build image
$ podman build -t hello-otel:metrics -f metrics/Buildfile-metrics
Verify you get a success message in the console like below:
Successfully tagged
localhost/hello-otel:metrics c10270357ea602ad7af1c7456b36959e04dca4a828e4af4a22e107bb99
Traces & metrics - Running the pod
Run the pod with the app, Prometheus and Jaeger:
$ podman play kube metrics/app-prom-pod.yml
Open a browser and make several requests (over time, refresh the browser to generate more in
the resulting graph on the next slide) to the homepage to generate metrics and traces:
Traces & metrics - Verify exemplars
Open the Prometheus console in your browser
http://localhost:9090
and query
hits_counter_total
in Graph view. Select by clicking on the
Show Exemplars
button in the middle to see them as little blue diamonds (note
you need to reduce the time window down to 5 minutes or so to see a graph plots):
Traces & metrics - From metric to trace
Click on one of the exemplars (blue dot) on the chart and copy the trace_url
i
nto a new browser tab:
Traces & metrics - Verify exemplars
Ta-da! You should see the trace waterfall for one of the requests made:
Traces & metrics - Stopping the pod
Let's clean up all the running containers by stopping the pod as follows:
$ podman play kube metrics/app-prom-pod.yml
Lab completed - Results
We reviewed the concept of exemplars and one way approach to linking Prometheus native metrics to OpenTelemetry traces!
Contact - are there any questions?
Lab and workshop completed!