Lab 8 - Instrumenting Applications
Lab Goal
This lab introduces client libraries and shows you how to use them to add Prometheus
metrics to applications and services. You'll get hands-on and instrument a sample application
to start collecting metrics.
Instrumenting - Generalized or specific metrics
In this workshop you've been given pre-instrumented demo applications (such as the services demo)
that leverage generalized auto instrumentation. These leverage exporters to provide general
observability metrics, but not specific business guided data. To be able to specifically
instrument your applications and services, you'll use language specific Prometheus client
libraries to track for the insights you want.
Let's start with a review of the metrics types in Prometheus, look at using the
Prometheus client libraries, and finally instrumenting an example Java application
using the Prometheus Java client library.
Instrumenting - Requirements for lab
This lab concludes with an example exercise where you instrument a simple Java application with
the four Prometheus metric types and collect them with a running Prometheus instance. The focus
is coding instrumentation, so let's assume you have the following:
- working Prometheus instance, such as you used in previous labs in this workshop
-
a basic understanding of coding, Java skills are not needed (but nice) for this
lab as you'll be walked through all you need to do and provided a working Java project to
with start
Intermezzo - Reviewing Prometheus metrics collection
The basics of how Prometheus collects metrics from target systems and applications is to scrape
using a pull mechanism as follows:
- Targets are scraped over the HTTP protocol, standard is
/metrics
path.
- Targets provide
current states
for each metric, sending:
- single sample for each tracked time series
- metric name
- label set
- sample value
- Each scraped sample is stored with a server-side timestamp added, building a set of
time series.
Intermezzo - Exposing target metrics
For Prometheus to be able to scrape a target, that target must expose metrics in the proper
format over HTTP. An example taken from the example service used later in this lab shows the
format you can manually verify in your browser on the path
http://localhost:7777/metrics
:
# HELP java_app_g is a gauge metric
# TYPE java_app_g gauge
java_app_g -1.50935252450572
# HELP java_app_h is a histogram metric
# TYPE java_app_h histogram
java_app_h_bucket{le="0.005",} 0.0
java_app_h_bucket{le="0.01",} 0.0
java_app_h_bucket{le="0.025",} 2.0
java_app_h_bucket{le="0.05",} 3.0
java_app_h_bucket{le="+Inf",} 162.0
java_app_h_count 162.0
java_app_h_sum 388.8431679332764
# HELP java_app_s is a summary metric (request size in bytes)
# TYPE java_app_s summary
java_app_s{quantile="0.5",} 2.3390745837068434
java_app_s{quantile="0.9",} 4.10579993595773
java_app_s{quantile="0.99",} 4.10579993595773
java_app_s_count 11.0
java_app_s_sum 27.438201238052876
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 11.0
Intermezzo - Instrumenting target for metrics
Because a target provides only the current values for the previously shared metrics, Prometheus
is responsible for collecting these individual values over time and creating time series. The
important part to remember is that the individual target application or service is only
instrumented to keep track of the current state of its metrics and does not ever buffer any
historical metrics states.
The various implementation details can be found in
Prometheus exposition formats documentation.
Instead of serialize the exposition format yourself, there are various client libraries to
assist you with the protocol serialization and more. Let's look closer at how they can help.
Instrumenting - Prometheus client libraries
The provided Prometheus
client libraries
assist with instrumenting your application code. From application code you're creating a metrics
registry to track all metrics objects, creating and updating metrics objects (counters, gauges,
histograms, and summaries), and exposing the results to Prometheus over HTTP. The client library
architecture:
Instrumenting - Using a client library
Using a client library from your application code is laid out in the following overview, numbered
in the order that each step would be implemented and used. The final piece of the puzzle is
Prometheus scraping the /metrics
endpoint:
Intermezzo - Reviewing metrics types: Counters
There are four metrics types you'll be exploring in Prometheus for this lab.
The first is a Counter
. Counters track cumulative totals over time, such as
the total number of seconds spent handling requests. Counters may only decrease in value when the
process that exposes them restarts, in which case their last value is forgotten and it's reset
to zero. A counter metric is serialized like this:
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 11.0
Intermezzo - Reviewing metrics types: Gauges
Gauges
track current tallies, things that increase or decrease over time,
such as memory usage or a temperature. A gauge metric is serialized like this:
# HELP java_app_g is a gauge metric
# TYPE java_app_g gauge
java_app_g -1.50935252450572
Intermezzo - Reviewing metrics types: Histograms
Histograms
allow you to to track the distribution of a set of observed
values, such as request latencies, across a set of buckets. They also track the total number of
observed values, and the cumulative sum of the observed values. A histogram metric is serialized
as a list of counter series, with one per bucket, and an le
label indicating the
latency upper bound of each bucket counter:
# HELP java_app_h is a histogram metric
# TYPE java_app_h histogram
java_app_h_bucket{le="0.005",} 0.0
java_app_h_bucket{le="0.01",} 0.0
java_app_h_bucket{le="0.025",} 2.0
java_app_h_bucket{le="0.05",} 3.0
java_app_h_bucket{le="+Inf",} 162.0
java_app_h_count 162.0
java_app_h_sum 388.8431679332764
Intermezzo - Reviewing metrics types: Summaries
Summaries
are tracking the distribution of a set of values, such as request
latencies, as a set of quantiles. A quantile, is like a percentile, but indicated with a range
from 0 to 1 instead of 0 to 100. For example, a quantile 0.5 is the 50th percentile. Like a
histogram, summaries also track the totals and cumulative sums of the observed values. A summary
metric is serialized with the quantile label indicating the quantile:
# HELP java_app_s is a summary metric (request size in bytes)
# TYPE java_app_s summary
java_app_s{quantile="0.5",} 2.3390745837068434
java_app_s{quantile="0.9",} 4.10579993595773
java_app_s{quantile="0.99",} 4.10579993595773
java_app_s_count 11.0
java_app_s_sum 27.438201238052876
Instrumenting - Some library coding details to consider
Client libraries provide interfaces for creating and using metrics and each library can be
slightly different for each type of metric.
Depending on the type of metric, constructors
will require different options.
For example, creating a histogram will require specifying a bucket configuration and a counter
would not need any parameters.
Metric objects also expose distinct state update methods for each type of metric. For example,
counters provide methods to increment the current value but never provide a method to set the
counter to an arbitrary value. Gauges on the other hand can be set an absolute value and also
provide methods to decrease the current value.
Instrumenting - Worried about library efficiency?
Don't worry, be happy!
All official Prometheus
client libraries
are implemented with efficiency and concurrency safety in mind. State updates are highly optimized
such that incrementing a counter millions of times a second will still perform well. Also, state
updates and reads from metric states are fully concurrency-safe. This means you can update metric
values from multiple threads without locking issues. Application are able to handle multiple
scrapes safely at the same time.
Instrumenting - What metrics to track: USE
When you are just getting started and are unsure of what metrics you want to track, a good starting
point can be the
USE Method.
It's summarized as follows:
For every resource, check utilization, saturation, and errors.
These are a set of metrics useful for measuring things that behave like
resources, used
or unused (queues, CPUs, memory, etc)
- Utilization: the average time that the resource was busy servicing work
- Saturation: the degree to which the resource has extra work which it can't service, often queued
- Errors: the count of error events
Instrumenting - What metrics to track: RED
The goal of the
Red Method
is to ensure that the software application functions properly for the end-users above all
else. These are the three key metrics you want to monitor for each service in your architecture:
- Rate: request counters
- Error: error counters
- Duration: distributions of time each request takes (histograms or summaries)
See also, the
Prometheus documentation on instrumentation
for best practices for instrumenting different types of systems.
Instrumenting - Best practices metric names
Metric name of time series describes an aspect of the system being monitored. They are not
interpreted by Prometheus in any meaningful way, so here are a few best practices for metric
names:
- ensure human readability
- ensure valid, matching regular expression
[a-zA-Z_:][a-zA-Z0-9_:]*
-
ensure clarity of origin with prefix, such as
prometheus_
or
java_app_
-
ensure unit suffix adhering to
base units,
such as
prometheus_tsdb_storage_blocks_bytes
or
prometheus_engine_query_duration_seconds
Instrumenting - More best practices metric names
Naming the basic metric types of counters, gauges, histograms and summaries
have their own best
practices as follows:
- counters named with suffix
_total
, such as prometheus_http_requests_total
- gauges are exposing the current number of queries, so something like
prometheus_engine_queries
-
Histograms and summaries also produce counter time series, these receive the following
suffixes, which are auto-appended so you'll never have to manually specify:
java_app_h_sum
for total sum of observations
java_app_h_count
for total count of observations
java_app_h_bucket
for individual buckets of histogram
Instrumenting - Metric label dangers!
Carving up your metrics with labels might feel very useful in the beginning, but be aware that
each label creates a new dimension. This means that for each unique set of labels creates a
unique time series to be tracked, stored, and handled during queries by Prometheus. The number
of concurrently active time series is a bottle neck for Prometheus at scale (a few million is
a guideline for a large server).
Label dimensions for metrics are multiplicative, so if you add a status_code
and method
labels to your metric the total series number is the product of
the number of different status codes and methods (all valid combinations). Then multiply that
cardinality by the number of targets for the overall time series cost.
Instrumenting - Avoiding metric cardinality explosions
To avoid time series explosions, also known as cardinality bombs
, consider
keeping the number of possible values well bounded for labels. Several really bad examples:
- storing IP addresses in a label value
- storing email addresses in a label value
- storing full HTTP paths in a label value
- especially if they contain IDs or other unbounded cardinality information
These examples create rapidly ever-increasing numbers of series that will overload your
Prometheus server quickly.
Instrumenting - Example Java application
For the rest of this lab you'll be working on exercises that walk you through instrumenting
a simple Java application using the
Prometheus Java client library.
For this lab we assume you have installed Java 8+ and Maven 3.8+.
Instrumenting - Java metrics demo project
First, we'll give you a project with all the Java layout you need and a completed example of
the four basic metrics instrumented for an example application. To install this locally:
Instrumenting - Download and unzip project
Once you have downloaded the project using the link provided previously, just unzip into its
own directory as shown (Note: examples shown are on Mac OSX system):
$ unzip workshop-prometheus/prometheus-java-metrics-demo-v0.1.zip
Archive: /Users/erics/Downloads/prometheus-java-metrics-demo-v0.1.zip
a66e4ea8ea9d1313b257fcf3b3321d9ce47a86e8
creating: prometheus-java-metrics-demo-v0.1/
extracting: prometheus-java-metrics-demo-v0.1/.gitignore
inflating: prometheus-java-metrics-demo-v0.1/README.md
inflating: prometheus-java-metrics-demo-v0.1/basic-pom.xml
inflating: prometheus-java-metrics-demo-v0.1/pom.xml
creating: prometheus-java-metrics-demo-v0.1/src/
creating: prometheus-java-metrics-demo-v0.1/src/main/
creating: prometheus-java-metrics-demo-v0.1/src/main/java/
creating: prometheus-java-metrics-demo-v0.1/src/main/java/io/
creating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/
creating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/java_apps/
inflating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/java_apps/BasicJavaMetrics.java
inflating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/java_apps/JavaMetrics.java
Instrumenting - Opening project in your IDE
Here you see the project opened in VSCode (use any tooling you like), which allows you to explore
the project single class file with a fully instrumented example Java application:
Instrumenting - Running the Java example
You can either run the JavaMetrics.java
class file from your IDE or from the
command line in the project root directory, as for example is shown below where you see the
example as started successfully:
$ cd prometheus-java-metrics-demo-v0.1/
$ mvn clean install (watch for BUILD SUCCESS)
$ java -jar target/java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Validating Java metrics setup
Make sure the Java metrics endpoint is working by opening the endpoint to be scraped by a
Prometheus instance on http://localhost:7777/metrics
, showing something
like this:
# HELP java_app_s is a summary metric (request size in bytes)
# TYPE java_app_s summary
java_app_s{quantile="0.5",} 2.679717814859738
java_app_s{quantile="0.9",} 4.566657867333372
java_app_s{quantile="0.99",} 4.927313848318692
java_app_s_count 512.0
java_app_s_sum 1343.9017287309503
# HELP java_app_h is a histogram metric
# TYPE java_app_h histogram
java_app_h_bucket{le="0.005",} 1.0
java_app_h_bucket{le="0.01",} 1.0
...
java_app_h_bucket{le="10.0",} 512.0
java_app_h_bucket{le="+Inf",} 512.0
java_app_h_count 512.0
java_app_h_sum 1291.5300871683055
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 512.0
# HELP java_app_g is a gauge metric
# TYPE java_app_g gauge
java_app_g 5.5811320747117765
Instrumenting - Prometheus configuration new metrics
While the metrics are exposed in this example on localhost:7777
, they will
not be scraped by Prometheus until you have updated its configuration to add this new end point.
Let's update our workshop-prometheus.yml
file to add the java application job as
shown along with comments for clarity (this is the minimum needed, with a few custom labels for
fun):
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping java metrics.
- job_name: "java_app"
static_configs:
- targets: ["localhost:7777"]
labels:
job: "java_app"
env: "workshop-lab8"
Instrumenting - Start Prometheus instance
Start the prometheus instance (for container Prometheus, see next slide) and then watch the
console where you started the Java application as it will report each time a new scrape is done
by logging Handled :/metrics
:
$ ./prometheus --config.file=support/workshop-prometheus.yml
===========Java application log===============
Java example metrics setup successful...
Java example service started...
Handled :/metrics
Handled :/metrics
Handled :/metrics
Handled :/metrics
...
Intermezzo - Using container Prometheus?
Astute users of this workshop might notice we are using the binary installation of Prometheus
for the examples shown here. Nothing prevents you from using a container version of Prometheus
as demonstrated in earlier labs from this workshop.
Well, there is one problem you'll notice, that the container version of Prometheus can't find the
locally locally running Java metrics example running on your local machine on port 7777. The
Java target will be listed as DOWN, so we'll leave this fix as an an exercise for you to enjoy
sorting out if you want to run Prometheus in a container for these exercises!
Instrumenting - Starting with basic Java metrics
Now stop the running Java metrics example (from the console just hit CTRL + C).
Let's start by creating your own Java metrics application starting with the minimal setup needed
to get your Java application running and exposing it's /metrics
. Instead of
coding it all by hand, we've put a staring point in the file
BasicJavaMetrics.java
found in the same directory, so open this file in your
IDE:
src/main/java/io/chronosphere/java_apps/BasicJavaMetrics.java
Instrumenting - Exploring BasicJavaMetrics.jar
First a comment about the last three import lines at the top of this file. They are all that's
needed from the client library to setup a basic metrics registry and expose the metrics endpoint.
Look for the main method, this is where the action happens in this class:
public static void main(String[] args) throws Exception {
CollectorRegistry registry = CollectorRegistry.defaultRegistry;
// Set up and start just default Java metrics and expose endpoint.
//DefaultExports.initialize();
startHttpServer(registry);
System.out.println("");
System.out.println("Basic Java metrics setup successful...");
System.out.println("");
// Insert your code here for application or microservice.
System.out.println("My application or service started...");\
System.out.println("");
}
Instrumenting - Building and running basic metrics
To help you run this BasicJavaMetrics.java
class file you are given a
separate basic-pom.xml
file, so either point your IDE build process at this
new pom file, or follow te command line example shown below:
$ cd prometheus-java-metrics-demo-v0.1/
$ mvn -f basic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Validating basic metrics setup
Make sure the basic metrics endpoint is working by opening the endpoint to be scraped by a
Prometheus instance on localhost:9999/metrics
.
But wait, it's showing a blank screen???
Instrumenting - Uncommenting default Java metrics
If you were looking closely at the main method, you might have noticed that there was a
commented out line //DefaultExports.initialize();
that can be uncommented
to initialize all the default Java metrics:
public static void main(String[] args) throws Exception {
CollectorRegistry registry = CollectorRegistry.defaultRegistry;
// Set up and start just default Java metrics and expose endpoint.
DefaultExports.initialize(); <<<<< UNCOMMENT THIS
startHttpServer(registry);
System.out.println("");
System.out.println("Basic Java metrics setup successful...");
System.out.println("");
// Insert your code here for application or microservice.
System.out.println("My application or service started...");\
System.out.println("");
}
Instrumenting - Rebuilding and running basic metrics
After editing the default metrics initialization line, you have to rebuild the Java executable
jar file and run it again:
$ mvn -f basic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Trying validating again
This time when you check the endpoint at localhost:9999/metrics
you're going
to find all kinds of Java metrics listed as shown below:
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 1498.0
# HELP jvm_classes_loaded_total The total number of classes loaded since JVM started
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 1498.0
# HELP jvm_classes_unloaded_total The total number of classes unloaded since JVM started
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 0.0
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 8192.0
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
Instrumenting - Adding a counter metric
Now that you have a working setup, you'll start with defining your first real instrumented
metric, a counter. To do this you will adjust the existing main class to add a method to start
a counter and giving it the previously created registry,startCounter(registry)
.
In the next slide you'll see the updated main method with the default initialized metrics
method removed.
Instrumenting - Adding comment and counter method
Note the larger and bold printed lines with a comment and the
startCounter(registry)
method call. Next we need to clean up our imports
to remove the default initialization and include our new counter from the instrumentation
library.
public static void main(String[] args) throws Exception {
CollectorRegistry registry = CollectorRegistry.defaultRegistry;
// Define and start a counter metric.
startCounter(registry);
startHttpServer(registry);
System.out.println("");
System.out.println("Basic Java metrics setup successful...");
System.out.println("");
// Insert your code here for application or microservice.
System.out.println("My application or service started...");\
System.out.println("");
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can remove
the DefaultExports
line and add the Counter
import
as follows:
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.common.TextFormat;
import io.prometheus.client.hotspot.DefaultExports; <<<<< REMOVE
import io.prometheus.client.Counter; <<<<< ADD
Instrumenting - Defining your counter method
Just below the main method, you can add in the following startCounter
method that will take a registry that you passed in and build a new counter before starting a
thread to run the counter which will auto increment every 5 seconds. Feel free to cut and paste
the code below:
private static void startCounter(CollectorRegistry registry) {
Counter counter = Counter.build()
.namespace("java_app")
.name("c")
.help("is a counter metric")
.register(registry);
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.inc(1);
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
}
Instrumenting - Rebuilding for counter metric
After defining the counter metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f basic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Validating your counter metric
Check the endpoint at localhost:9999/metrics
you're going to find the
counter metric as shown below:
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 2.0
Instrumenting - Adding a gauge metric
Next up we can add a gauge metric and you'll do that by renaming the startCounter
method and expanding it to add the building of a gauge metric. In the next slide you'll see the
updated main method with the renamed startMetrics(registry)
method.
Instrumenting - Renaming start metrics method
Note the larger and bold printed lines with a comment and the
startMetrics(registry)
method call. Next you'll need to import your new
gauge metric from the instrumentation library.
public static void main(String[] args) throws Exception {
CollectorRegistry registry = CollectorRegistry.defaultRegistry;
// Defining and starting metrics.
startMetrics(registry);
startHttpServer(registry);
System.out.println("");
System.out.println("Basic Java metrics setup successful...");
System.out.println("");
// Insert your code here for application or microservice.
System.out.println("My application or service started...");\
System.out.println("");
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Gauge
import as follows:
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.common.TextFormat;
import io.prometheus.client.Counter;
import io.prometheus.client.Gauge; <<<<< ADD
Instrumenting - Defining your gauge metric
You're now going to expand the renamed startMetrics
method to build a new
gauge metric directly after the counter metric definition. Feel free to cut and paste the gauge
definition below:
private static void startCounter(CollectorRegistry registry) {
Counter counter = Counter.build()
.namespace("java_app")
.name("c")
.help("is a counter metric")
.register(registry);
Gauge gauge = Gauge.build()
.namespace("java_app")
.name("g")
.help("is a gauge metric")
.register(registry);
Instrumenting - Starting your gauge metric
You're now ready to set your gauge to a random number between -5 and 10 in the thread you've
started. Feel free to cut and paste the new line:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.inc(1);
gauge.set(-5 + (Math.random() * (10 - -5)));
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
}
Instrumenting - Rebuilding for gauge metric
After defining the gauge metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f basic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Validating your gauge metric
Check the endpoint at localhost:9999/metrics
you're going to find the
gauge metric as shown below with a random number that changes every 5 seconds:
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 2.0
# HELP java_app_g is a gauge metric
# TYPE java_app_g gauge
java_app_g 4.762373540089195
Instrumenting - Adding a histogram metric
Next up we can add a histogram metric. We no longer need to update the main method as you only
need to update your imports and add the new histogram metric to the
startMetrics
method.
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Histogram
import as follows:
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.common.TextFormat;
import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram; <<<<< ADD
Instrumenting - Defining your histogram metric
You're now going to expand the startMetrics
method to build a new
histogram metric directly after the previously added gauge metric definition. Feel free to cut
and paste the histogram definition below:
Gauge gauge = Gauge.build()
.namespace("java_app")
.name("g")
.help("is a gauge metric")
.register(registry);
Histogram histogram = Histogram.build()
.namespace("java_app")
.name("h")
.help("is a histogram metric")
.register();
Instrumenting - Starting your histogram metric
Now you'll set the histogram buckets using random numbers in the thread you've started. Feel
free to cut and paste to add the new line:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.inc(1);
gauge.set(-5 + (Math.random() * (10 - -5)));
histogram.observe(0 + (Math.random() * (5 - 0)));
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
Instrumenting - Rebuilding for histogram metric
After defining the histogram metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f basic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Validating your histogram metric
Check the endpoint at localhost:9999/metrics
for the histogram metric as
shown below with a random number that changes every 5 seconds:
# HELP java_app_h is a histogram metric
# TYPE java_app_h histogram
java_app_h_bucket{le="0.005",} 0.0
java_app_h_bucket{le="0.01",} 0.0
...
java_app_h_bucket{le="10.0",} 2.0
java_app_h_bucket{le="+Inf",} 2.0
java_app_h_count 2.0
java_app_h_sum 5.434964201169219
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 2.0
# HELP java_app_g is a gauge metric
# TYPE java_app_g gauge
java_app_g 4.762373540089195
Instrumenting - Adding a summary metric
Finally, you're adding a summary metric. We no longer need to update the main method as you only
need to update your imports and add the new histogram metric to the
startMetrics
method.
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Summary
import as follows:
import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.exporter.common.TextFormat;
import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.Summary; <<<<< ADD
Instrumenting - Defining your summary metric
You're now going to expand the startMetrics
method to build a new
summary metric directly after the previously added histogram metric definition. Feel free to cut
and paste the summary definition below:
Histogram histogram = Histogram.build()
.namespace("java_app")
.name("h")
.help("is a histogram metric")
.register();
Summary summary = Summary.build()
.quantile(0.5, 0.05) // Add 50th percentile with 5% tolerance
.quantile(0.9, 0.01) // Add 90th percentile with 1% tolerance
.quantile(0.99, 0.001) // Add 99th percentile with 0.1% tolerance
.namespace("java_app")
.name("s")
.help("is a summary metric (request size in bytes)")
.register(registry);
Instrumenting - Starting your summary metric
Now you'll set the summary quantiles using random numbers in the thread you've started. Feel
free to cut and paste to add the new line:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.inc(1);
gauge.set(-5 + (Math.random() * (10 - -5)));
histogram.observe(0 + (Math.random() * (5 - 0)));
summary.observe(0 + (Math.random() * (5 - 0)));
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
Instrumenting - Rebuilding for summary metric
After defining the summary metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f basic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Validating your summary metric
Check the endpoint at localhost:9999/metrics
for the summary metric as
shown below with a random number that changes every 5 seconds:
# HELP java_app_s is a summary metric (request size in bytes)
# TYPE java_app_s summary
java_app_s{quantile="0.5",} 2.3090768039675225
java_app_s{quantile="0.9",} 2.3090768039675225
java_app_s{quantile="0.99",} 2.3090768039675225
java_app_s_count 2.0
java_app_s_sum 6.163592371161444
# HELP java_app_h is a histogram metric
# TYPE java_app_h histogram
java_app_h_bucket{le="0.005",} 0.0
...
java_app_h_bucket{le="+Inf",} 2.0
java_app_h_count 2.0
java_app_h_sum 5.434964201169219
# HELP java_app_c is a counter metric
# TYPE java_app_c counter
java_app_c 2.0
# HELP java_app_g is a gauge metric
# TYPE java_app_g gauge
java_app_g 4.762373540089195
Instrumenting - Prometheus configuration adjustment
While the metrics are exposed in this example on localhost:9999
, they will
not be scraped by Prometheus until you have updated workshop-prometheus.yml
file
by adding the java application job as shown (I've added a few custom labels for fun):
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping java metrics.
- job_name: "java_app"
static_configs:
- targets: ["localhost:9999"]
labels:
job: "java_app"
env: "workshop-lab8"
Instrumenting - Start Prometheus instance
Start your Prometheus instance (for container Prometheus, see next slide) and then watch the
console where you started the Java application as it will report each time a new scrape is done
by logging Handled :/metrics
:
$ ./prometheus --config.file=support/workshop-prometheus.yml
===========Java application log===============
Java example metrics setup successful...
Java example service started...
Handled :/metrics
Handled :/metrics
Handled :/metrics
Handled :/metrics
...
Instrumenting - Scraping Java metrics
You can validate that the Java metrics you just instrumented in your application are available
in the Prometheus console localhost:9090
as shown. Feel free to query and
explore:
Lab completed - Results
Next up, metrics monitoring at scale...
Contact - are there any questions?