Lab Goal

This lab introduces client libraries and shows you how to use them to add Prometheus metrics to applications and services. You'll get hands-on and instrument a sample application to start collecting metrics.

Instrumenting - Generalized or specific metrics

In this workshop you've been given pre-instrumented demo applications (such as the services demo) that leverage generalized auto instrumentation. These leverage exporters to provide general observability metrics, but not specific business guided data. To be able to specifically instrument your applications and services, you'll use language specific Prometheus client libraries to track for the insights you want.

Let's start with a review of the metrics types in Prometheus, look at using the Prometheus client libraries, and finally instrumenting an example Java application using the Prometheus Java client library.

Instrumenting - Requirements for lab

This lab concludes with an example exercise where you instrument a simple Java application with the four Prometheus metric types and collect them with a running Prometheus instance. The focus is coding instrumentation, so let's assume you have the following:

1. working Prometheus instance, such as you used in previous labs in this workshop
2. a basic understanding of coding, Java skills are not needed (but nice) for this lab as you'll be walked through all you need to do and provided a working Java project to with start

Intermezzo - Reviewing Prometheus metrics collection

The basics of how Prometheus collects metrics from target systems and applications is to scrape using a pull mechanism as follows:

1. Targets are scraped over the HTTP protocol, standard is /metrics path.
2. Targets provide current states for each metric, sending:
• single sample for each tracked time series
• metric name
• label set
• sample value
3. Each scraped sample is stored with a server-side timestamp added, building a set of time series.

Intermezzo - Exposing target metrics

For Prometheus to be able to scrape a target, that target must expose metrics in the proper format over HTTP. An example taken from the example service used later in this lab shows the format you can manually verify in your browser on the path http://localhost:7777/metrics:

							
								# HELP java_app_g is a gauge metric
								# TYPE java_app_g gauge
								java_app_g -1.50935252450572
								# HELP java_app_h is a histogram metric
								# TYPE java_app_h histogram
								java_app_h_bucket{le="0.005",} 0.0
								java_app_h_bucket{le="0.01",} 0.0
								java_app_h_bucket{le="0.025",} 2.0
								java_app_h_bucket{le="0.05",} 3.0
								java_app_h_bucket{le="+Inf",} 162.0
								java_app_h_count 162.0
								java_app_h_sum 388.8431679332764
								# HELP java_app_s is a summary metric (request size in bytes)
								# TYPE java_app_s summary
								java_app_s{quantile="0.5",} 2.3390745837068434
								java_app_s{quantile="0.9",} 4.10579993595773
								java_app_s{quantile="0.99",} 4.10579993595773
								java_app_s_count 11.0
								java_app_s_sum 27.438201238052876
								# HELP java_app_c is a counter metric
								# TYPE java_app_c counter
								java_app_c 11.0
							
						

Intermezzo - Instrumenting target for metrics

Because a target provides only the current values for the previously shared metrics, Prometheus is responsible for collecting these individual values over time and creating time series. The important part to remember is that the individual target application or service is only instrumented to keep track of the current state of its metrics and does not ever buffer any historical metrics states.

The various implementation details can be found in Prometheus exposition formats documentation. Instead of serialize the exposition format yourself, there are various client libraries to assist you with the protocol serialization and more. Let's look closer at how they can help.

Instrumenting - Prometheus client libraries

The provided Prometheus client libraries assist with instrumenting your application code. From application code you're creating a metrics registry to track all metrics objects, creating and updating metrics objects (counters, gauges, histograms, and summaries), and exposing the results to Prometheus over HTTP. The client library architecture:

Instrumenting - Using a client library

Using a client library from your application code is laid out in the following overview, numbered in the order that each step would be implemented and used. The final piece of the puzzle is Prometheus scraping the /metrics endpoint:

Intermezzo - Reviewing metrics types: Counters

There are four metrics types you'll be exploring in Prometheus for this lab.

The first is a Counter. Counters track cumulative totals over time, such as the total number of seconds spent handling requests. Counters may only decrease in value when the process that exposes them restarts, in which case their last value is forgotten and it's reset to zero. A counter metric is serialized like this:

							
								# HELP java_app_c is a counter metric
								# TYPE java_app_c counter
								java_app_c 11.0
							
						

Intermezzo - Reviewing metrics types: Gauges

Gauges track current tallies, things that increase or decrease over time, such as memory usage or a temperature. A gauge metric is serialized like this:

							
								# HELP java_app_g is a gauge metric
								# TYPE java_app_g gauge
								java_app_g -1.50935252450572
							
						

Intermezzo - Reviewing metrics types: Histograms

Histograms allow you to to track the distribution of a set of observed values, such as request latencies, across a set of buckets. They also track the total number of observed values, and the cumulative sum of the observed values. A histogram metric is serialized as a list of counter series, with one per bucket, and an le label indicating the latency upper bound of each bucket counter:

							
								# HELP java_app_h is a histogram metric
								# TYPE java_app_h histogram
								java_app_h_bucket{le="0.005",} 0.0
								java_app_h_bucket{le="0.01",} 0.0
								java_app_h_bucket{le="0.025",} 2.0
								java_app_h_bucket{le="0.05",} 3.0
								java_app_h_bucket{le="+Inf",} 162.0
								java_app_h_count 162.0
								java_app_h_sum 388.8431679332764
							
						

Intermezzo - Reviewing metrics types: Summaries

Summaries are tracking the distribution of a set of values, such as request latencies, as a set of quantiles. A quantile, is like a percentile, but indicated with a range from 0 to 1 instead of 0 to 100. For example, a quantile 0.5 is the 50th percentile. Like a histogram, summaries also track the totals and cumulative sums of the observed values. A summary metric is serialized with the quantile label indicating the quantile:

							
								# HELP java_app_s is a summary metric (request size in bytes)
								# TYPE java_app_s summary
								java_app_s{quantile="0.5",} 2.3390745837068434
								java_app_s{quantile="0.9",} 4.10579993595773
								java_app_s{quantile="0.99",} 4.10579993595773
								java_app_s_count 11.0
								java_app_s_sum 27.438201238052876
							
						

Instrumenting - Some library coding details to consider

Client libraries provide interfaces for creating and using metrics and each library can be slightly different for each type of metric.

Depending on the type of metric, constructors will require different options. For example, creating a histogram will require specifying a bucket configuration and a counter would not need any parameters.

Metric objects also expose distinct state update methods for each type of metric. For example, counters provide methods to increment the current value but never provide a method to set the counter to an arbitrary value. Gauges on the other hand can be set an absolute value and also provide methods to decrease the current value.

Instrumenting - Worried about library efficiency?

Don't worry, be happy!

All official Prometheus client libraries are implemented with efficiency and concurrency safety in mind. State updates are highly optimized such that incrementing a counter millions of times a second will still perform well. Also, state updates and reads from metric states are fully concurrency-safe. This means you can update metric values from multiple threads without locking issues. Application are able to handle multiple scrapes safely at the same time.

Instrumenting - What metrics to track: USE

When you are just getting started and are unsure of what metrics you want to track, a good starting point can be the USE Method. It's summarized as follows:

For every resource, check utilization, saturation, and errors.

These are a set of metrics useful for measuring things that behave like resources, used or unused (queues, CPUs, memory, etc)

• Utilization: the average time that the resource was busy servicing work
• Saturation: the degree to which the resource has extra work which it can't service, often queued
• Errors: the count of error events

Instrumenting - What metrics to track: RED

The goal of the Red Method is to ensure that the software application functions properly for the end-users above all else. These are the three key metrics you want to monitor for each service in your architecture:

• Rate: request counters
• Error: error counters
• Duration: distributions of time each request takes (histograms or summaries)

See also, the Prometheus documentation on instrumentation for best practices for instrumenting different types of systems.

Instrumenting - Best practices metric names

Metric name of time series describes an aspect of the system being monitored. They are not interpreted by Prometheus in any meaningful way, so here are a few best practices for metric names:

• ensure human readability
• ensure valid, matching regular expression [a-zA-Z_:][a-zA-Z0-9_:]*
• ensure clarity of origin with prefix, such as prometheus_ or java_app_
• ensure unit suffix adhering to base units, such as prometheus_tsdb_storage_blocks_bytes or prometheus_engine_query_duration_seconds

Instrumenting - More best practices metric names

Naming the basic metric types of counters, gauges, histograms and summaries have their own best practices as follows:

• counters named with suffix _total, such as prometheus_http_requests_total
• gauges are exposing the current number of queries, so something like prometheus_engine_queries
• Histograms and summaries also produce counter time series, these receive the following suffixes, which are auto-appended so you'll never have to manually specify:
• java_app_h_sum for total sum of observations
• java_app_h_count for total count of observations
• java_app_h_bucket for individual buckets of histogram

Instrumenting - Metric label dangers!

Carving up your metrics with labels might feel very useful in the beginning, but be aware that each label creates a new dimension. This means that for each unique set of labels creates a unique time series to be tracked, stored, and handled during queries by Prometheus. The number of concurrently active time series is a bottle neck for Prometheus at scale (a few million is a guideline for a large server).

Label dimensions for metrics are multiplicative, so if you add a status_code and method labels to your metric the total series number is the product of the number of different status codes and methods (all valid combinations). Then multiply that cardinality by the number of targets for the overall time series cost.

Instrumenting - Avoiding metric cardinality explosions

To avoid time series explosions, also known as cardinality bombs, consider keeping the number of possible values well bounded for labels. Several really bad examples:

• storing IP addresses in a label value
• storing email addresses in a label value
• storing full HTTP paths in a label value
• especially if they contain IDs or other unbounded cardinality information

These examples create rapidly ever-increasing numbers of series that will overload your Prometheus server quickly.

Instrumenting - Example Java application

For the rest of this lab you'll be working on exercises that walk you through instrumenting a simple Java application using the Prometheus Java client library.

For this lab we assume you have installed Java 8+ and Maven 3.8+.

Instrumenting - Java metrics demo project

First, we'll give you a project with all the Java layout you need and a completed example of the four basic metrics instrumented for an example application. To install this locally:

1. Download and unzip the Prometheus Java Metrics Demo
2. Unzip the prometheus-java-metrics-demo-v0.1.zip file in your workshop directory
3. Open the project in your favorite IDE (examples shown here use VSCode)

Instrumenting - Download and unzip project

Once you have downloaded the project using the link provided previously, just unzip into its own directory as shown (Note: examples shown are on Mac OSX system):

							
								$ unzip workshop-prometheus/prometheus-java-metrics-demo-v0.1.zip

								Archive:  /Users/erics/Downloads/prometheus-java-metrics-demo-v0.1.zip
								a66e4ea8ea9d1313b257fcf3b3321d9ce47a86e8
								creating: prometheus-java-metrics-demo-v0.1/
								extracting: prometheus-java-metrics-demo-v0.1/.gitignore
								inflating: prometheus-java-metrics-demo-v0.1/README.md
								inflating: prometheus-java-metrics-demo-v0.1/basic-pom.xml
								inflating: prometheus-java-metrics-demo-v0.1/pom.xml
								creating: prometheus-java-metrics-demo-v0.1/src/
								creating: prometheus-java-metrics-demo-v0.1/src/main/
								creating: prometheus-java-metrics-demo-v0.1/src/main/java/
								creating: prometheus-java-metrics-demo-v0.1/src/main/java/io/
								creating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/
								creating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/java_apps/
								inflating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/java_apps/BasicJavaMetrics.java
								inflating: prometheus-java-metrics-demo-v0.1/src/main/java/io/chronosphere/java_apps/JavaMetrics.java
							
						

Instrumenting - Opening project in your IDE

Here you see the project opened in VSCode (use any tooling you like), which allows you to explore the project single class file with a fully instrumented example Java application:

Instrumenting - Running the Java example

You can either run the JavaMetrics.java class file from your IDE or from the command line in the project root directory, as for example is shown below where you see the example as started successfully:

							
								$ cd prometheus-java-metrics-demo-v0.1/

								$ mvn clean install  (watch for BUILD SUCCESS)

								$ java -jar target/java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar

								Java example metrics setup successful...

								Java example service started...
							
						

Instrumenting - Validating Java metrics setup

Make sure the Java metrics endpoint is working by opening the endpoint to be scraped by a Prometheus instance on http://localhost:7777/metrics, showing something like this:

							
								# HELP java_app_s is a summary metric (request size in bytes)
								# TYPE java_app_s summary
								java_app_s{quantile="0.5",} 2.679717814859738
								java_app_s{quantile="0.9",} 4.566657867333372
								java_app_s{quantile="0.99",} 4.927313848318692
								java_app_s_count 512.0
								java_app_s_sum 1343.9017287309503
								# HELP java_app_h is a histogram metric
								# TYPE java_app_h histogram
								java_app_h_bucket{le="0.005",} 1.0
								java_app_h_bucket{le="0.01",} 1.0
								...
								java_app_h_bucket{le="10.0",} 512.0
								java_app_h_bucket{le="+Inf",} 512.0
								java_app_h_count 512.0
								java_app_h_sum 1291.5300871683055
								# HELP java_app_c is a counter metric
								# TYPE java_app_c counter
								java_app_c 512.0
								# HELP java_app_g is a gauge metric
								# TYPE java_app_g gauge
								java_app_g 5.5811320747117765
							
						

Instrumenting - Prometheus configuration new metrics

While the metrics are exposed in this example on localhost:7777, they will not be scraped by Prometheus until you have updated its configuration to add this new end point. Let's update our workshop-prometheus.yml file to add the java application job as shown along with comments for clarity (this is the minimum needed, with a few custom labels for fun):

							
								scrape_configs:

									# Scraping Prometheus.
									- job_name: "prometheus"
									static_configs:
										- targets: ["localhost:9090"]

									# Scraping java metrics.
									- job_name: "java_app"
									static_configs:
										- targets: ["localhost:7777"]
								          labels:
								            job: "java_app"
								            env: "workshop-lab8"
							
						

Instrumenting - Start Prometheus instance

Start the prometheus instance (for container Prometheus, see next slide) and then watch the console where you started the Java application as it will report each time a new scrape is done by logging Handled :/metrics:

							
								$ ./prometheus --config.file=support/workshop-prometheus.yml

								===========Java application log===============
								Java example metrics setup successful...

								Java example service started...

								Handled :/metrics
								Handled :/metrics
								Handled :/metrics
								Handled :/metrics
								...
							
						

Intermezzo - Using container Prometheus?

Astute users of this workshop might notice we are using the binary installation of Prometheus for the examples shown here. Nothing prevents you from using a container version of Prometheus as demonstrated in earlier labs from this workshop.

Well, there is one problem you'll notice, that the container version of Prometheus can't find the locally locally running Java metrics example running on your local machine on port 7777. The Java target will be listed as DOWN, so we'll leave this fix as an an exercise for you to enjoy sorting out if you want to run Prometheus in a container for these exercises!

Instrumenting - Starting with basic Java metrics

Now stop the running Java metrics example (from the console just hit CTRL + C).

Let's start by creating your own Java metrics application starting with the minimal setup needed to get your Java application running and exposing it's /metrics. Instead of coding it all by hand, we've put a staring point in the file BasicJavaMetrics.java found in the same directory, so open this file in your IDE:

							
								src/main/java/io/chronosphere/java_apps/BasicJavaMetrics.java
							
						

Instrumenting - Exploring BasicJavaMetrics.jar

First a comment about the last three import lines at the top of this file. They are all that's needed from the client library to setup a basic metrics registry and expose the metrics endpoint. Look for the main method, this is where the action happens in this class:

							
								public static void main(String[] args) throws Exception {

								CollectorRegistry registry = CollectorRegistry.defaultRegistry;

								// Set up and start just default Java metrics and expose endpoint.
								//DefaultExports.initialize();
								startHttpServer(registry);
								System.out.println("");
								System.out.println("Basic Java metrics setup successful...");
								System.out.println("");

								// Insert your code here for application or microservice.
								System.out.println("My application or service started...");\
								System.out.println("");
								}
							
						

Instrumenting - Building and running basic metrics

To help you run this BasicJavaMetrics.java class file you are given a separate basic-pom.xml file, so either point your IDE build process at this new pom file, or follow te command line example shown below:

							
								$ cd prometheus-java-metrics-demo-v0.1/

								$ mvn -f basic-pom.xml clean install  (watch for BUILD SUCCESS)

								$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar

								Java example metrics setup successful...

								Java example service started...
							
						

Instrumenting - Validating basic metrics setup

Make sure the basic metrics endpoint is working by opening the endpoint to be scraped by a Prometheus instance on localhost:9999/metrics.

But wait, it's showing a blank screen???

Instrumenting - Uncommenting default Java metrics

If you were looking closely at the main method, you might have noticed that there was a commented out line //DefaultExports.initialize(); that can be uncommented to initialize all the default Java metrics:

							
								public static void main(String[] args) throws Exception {

								CollectorRegistry registry = CollectorRegistry.defaultRegistry;

								// Set up and start just default Java metrics and expose endpoint.
								DefaultExports.initialize();    <<<<< UNCOMMENT THIS
								startHttpServer(registry);
								System.out.println("");
								System.out.println("Basic Java metrics setup successful...");
								System.out.println("");

								// Insert your code here for application or microservice.
								System.out.println("My application or service started...");\
								System.out.println("");
								}
							
						

Instrumenting - Rebuilding and running basic metrics

After editing the default metrics initialization line, you have to rebuild the Java executable jar file and run it again:

							
								$ mvn -f basic-pom.xml clean install  (watch for BUILD SUCCESS)

								$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar

								Java example metrics setup successful...

								Java example service started...
							
						

Instrumenting - Trying validating again

This time when you check the endpoint at localhost:9999/metrics you're going to find all kinds of Java metrics listed as shown below:

							
								# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
								# TYPE jvm_classes_loaded gauge
								jvm_classes_loaded 1498.0
								# HELP jvm_classes_loaded_total The total number of classes loaded since JVM started
								# TYPE jvm_classes_loaded_total counter
								jvm_classes_loaded_total 1498.0
								# HELP jvm_classes_unloaded_total The total number of classes unloaded since JVM started
								# TYPE jvm_classes_unloaded_total counter
								jvm_classes_unloaded_total 0.0
								# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
								# TYPE jvm_buffer_pool_used_bytes gauge
								jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
								jvm_buffer_pool_used_bytes{pool="direct",} 8192.0
								jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
							
						

Instrumenting - Adding a counter metric

Now that you have a working setup, you'll start with defining your first real instrumented metric, a counter. To do this you will adjust the existing main class to add a method to start a counter and giving it the previously created registry,startCounter(registry). In the next slide you'll see the updated main method with the default initialized metrics method removed.

Instrumenting - Adding comment and counter method

Note the larger and bold printed lines with a comment and the startCounter(registry) method call. Next we need to clean up our imports to remove the default initialization and include our new counter from the instrumentation library.

							
								public static void main(String[] args) throws Exception {

								CollectorRegistry registry = CollectorRegistry.defaultRegistry;

								// Define and start a counter metric.
								startCounter(registry);

								startHttpServer(registry);
								System.out.println("");
								System.out.println("Basic Java metrics setup successful...");
								System.out.println("");

								// Insert your code here for application or microservice.
								System.out.println("My application or service started...");\
								System.out.println("");
								}
							
						

Instrumenting - Updating your import statements

At the top of the basic metrics file you'll find the client library imports. You can remove the DefaultExports line and add the Counter import as follows:

							
								import io.prometheus.client.CollectorRegistry;
								import io.prometheus.client.exporter.common.TextFormat;
								import io.prometheus.client.hotspot.DefaultExports;  <<<<< REMOVE
								import io.prometheus.client.Counter;                 <<<<< ADD
							
						

Instrumenting - Defining your counter method

Just below the main method, you can add in the following startCounter method that will take a registry that you passed in and build a new counter before starting a thread to run the counter which will auto increment every 5 seconds. Feel free to cut and paste the code below:

							
								private static void startCounter(CollectorRegistry registry) {
									Counter counter = Counter.build()
														.namespace("java_app")
														.name("c")
														.help("is a counter metric")
														.register(registry);

									Thread bgThread = new Thread(() -> {
										while (true) {
											try {
												counter.inc(1);
												TimeUnit.SECONDS.sleep(5);
											} catch (InterruptedException e) {
												e.printStackTrace();
											}
										}
									});
									bgThread.start();
								}
							
						

Instrumenting - Rebuilding for counter metric

After defining the counter metric, you have to rebuild the Java executable jar file and run it again:

							
								$ mvn -f basic-pom.xml clean install  (watch for BUILD SUCCESS)

								$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar

								Java example metrics setup successful...

								Java example service started...
							
						

Instrumenting - Validating your counter metric

Check the endpoint at localhost:9999/metrics you're going to find the counter metric as shown below:

							
								# HELP java_app_c is a counter metric
								# TYPE java_app_c counter
								java_app_c 2.0
							
						

Instrumenting - Adding a gauge metric

Next up we can add a gauge metric and you'll do that by renaming the startCounter method and expanding it to add the building of a gauge metric. In the next slide you'll see the updated main method with the renamed startMetrics(registry) method.

Instrumenting - Renaming start metrics method

Note the larger and bold printed lines with a comment and the startMetrics(registry) method call. Next you'll need to import your new gauge metric from the instrumentation library.

							
								public static void main(String[] args) throws Exception {

								CollectorRegistry registry = CollectorRegistry.defaultRegistry;

								// Defining and starting metrics.
								startMetrics(registry);

								startHttpServer(registry);
								System.out.println("");
								System.out.println("Basic Java metrics setup successful...");
								System.out.println("");

								// Insert your code here for application or microservice.
								System.out.println("My application or service started...");\
								System.out.println("");
								}
							
						

Instrumenting - Updating your import statements

At the top of the basic metrics file you'll find the client library imports. You can add the Gauge import as follows:

							
								import io.prometheus.client.CollectorRegistry;
								import io.prometheus.client.exporter.common.TextFormat;
								import io.prometheus.client.Counter;
								import io.prometheus.client.Gauge;                 <<<<< ADD
							
						

Instrumenting - Defining your gauge metric

You're now going to expand the renamed startMetrics method to build a new gauge metric directly after the counter metric definition. Feel free to cut and paste the gauge definition below:

							
								private static void startCounter(CollectorRegistry registry) {
									Counter counter = Counter.build()
														.namespace("java_app")
														.name("c")
														.help("is a counter metric")
														.register(registry);

									Gauge gauge = Gauge.build()
													.namespace("java_app")
													.name("g")
													.help("is a gauge metric")
													.register(registry);
							
						

Instrumenting - Starting your gauge metric

You're now ready to set your gauge to a random number between -5 and 10 in the thread you've started. Feel free to cut and paste the new line:

							
									Thread bgThread = new Thread(() -> {
										while (true) {
											try {
												counter.inc(1);

												gauge.set(-5 + (Math.random() * (10 - -5)));

												TimeUnit.SECONDS.sleep(5);
											} catch (InterruptedException e) {
												e.printStackTrace();
											}
										}
									});
									bgThread.start();
								}
							
						

Instrumenting - Rebuilding for gauge metric

After defining the gauge metric, you have to rebuild the Java executable jar file and run it again:

							
								$ mvn -f basic-pom.xml clean install  (watch for BUILD SUCCESS)

								$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar

								Java example metrics setup successful...

								Java example service started...
							
						

Instrumenting - Validating your gauge metric

Check the endpoint at localhost:9999/metrics you're going to find the gauge metric as shown below with a random number that changes every 5 seconds:

							
								# HELP java_app_c is a counter metric
								# TYPE java_app_c counter
								java_app_c 2.0
								# HELP java_app_g is a gauge metric
								# TYPE java_app_g gauge
								java_app_g 4.762373540089195
							
						

Instrumenting - Adding a histogram metric

Next up we can add a histogram metric. We no longer need to update the main method as you only need to update your imports and add the new histogram metric to the startMetrics method.

Instrumenting - Updating your import statements

At the top of the basic metrics file you'll find the client library imports. You can add the Histogram import as follows:

							
								import io.prometheus.client.CollectorRegistry;
								import io.prometheus.client.exporter.common.TextFormat;
								import io.prometheus.client.Counter;
								import io.prometheus.client.Gauge;
								import io.prometheus.client.Histogram;        <<<<< ADD
							
						

Instrumenting - Defining your histogram metric

You're now going to expand the startMetrics method to build a new histogram metric directly after the previously added gauge metric definition. Feel free to cut and paste the histogram definition below:

							
								Gauge gauge = Gauge.build()
									.namespace("java_app")
									.name("g")
									.help("is a gauge metric")
									.register(registry);

								Histogram histogram = Histogram.build()
									.namespace("java_app")
									.name("h")
									.help("is a histogram metric")
									.register();
							
						

Instrumenting - Starting your histogram metric

Now you'll set the histogram buckets using random numbers in the thread you've started. Feel free to cut and paste to add the new line:

							
									Thread bgThread = new Thread(() -> {
										while (true) {
											try {
												counter.inc(1);
												gauge.set(-5 + (Math.random() * (10 - -5)));

												histogram.observe(0 + (Math.random() * (5 - 0)));

												TimeUnit.SECONDS.sleep(5);
											} catch (InterruptedException e) {
												e.printStackTrace();
											}
										}
									});
							
						

Instrumenting - Rebuilding for histogram metric

After defining the histogram metric, you have to rebuild the Java executable jar file and run it again:

							
								$ mvn -f basic-pom.xml clean install  (watch for BUILD SUCCESS)

								$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar

								Java example metrics setup successful...

								Java example service started...
							
						

Instrumenting - Validating your histogram metric

Check the endpoint at localhost:9999/metrics for the histogram metric as shown below with a random number that changes every 5 seconds:

							
								# HELP java_app_h is a histogram metric
								# TYPE java_app_h histogram
								java_app_h_bucket{le="0.005",} 0.0
								java_app_h_bucket{le="0.01",} 0.0
								...
								java_app_h_bucket{le="10.0",} 2.0
								java_app_h_bucket{le="+Inf",} 2.0
								java_app_h_count 2.0
								java_app_h_sum 5.434964201169219
								# HELP java_app_c is a counter metric
								# TYPE java_app_c counter
								java_app_c 2.0
								# HELP java_app_g is a gauge metric
								# TYPE java_app_g gauge
								java_app_g 4.762373540089195
							
						

Instrumenting - Adding a summary metric

Finally, you're adding a summary metric. We no longer need to update the main method as you only need to update your imports and add the new histogram metric to the startMetrics method.

Instrumenting - Updating your import statements

At the top of the basic metrics file you'll find the client library imports. You can add the Summary import as follows:

							
								import io.prometheus.client.CollectorRegistry;
								import io.prometheus.client.exporter.common.TextFormat;
								import io.prometheus.client.Counter;
								import io.prometheus.client.Gauge;
								import io.prometheus.client.Histogram;
								import io.prometheus.client.Summary;        <<<<< ADD
							
						

Instrumenting - Defining your summary metric

You're now going to expand the startMetrics method to build a new summary metric directly after the previously added histogram metric definition. Feel free to cut and paste the summary definition below:

							
								Histogram histogram = Histogram.build()
									.namespace("java_app")
									.name("h")
									.help("is a histogram metric")
									.register();

								Summary summary = Summary.build()
									.quantile(0.5, 0.05) // Add 50th percentile with 5% tolerance
									.quantile(0.9, 0.01) // Add 90th percentile with 1% tolerance
									.quantile(0.99, 0.001) // Add 99th percentile with 0.1% tolerance
									.namespace("java_app")
									.name("s")
									.help("is a summary metric (request size in bytes)")
									.register(registry);
							
						

Instrumenting - Starting your summary metric

Now you'll set the summary quantiles using random numbers in the thread you've started. Feel free to cut and paste to add the new line:

							
									Thread bgThread = new Thread(() -> {
										while (true) {
											try {
												counter.inc(1);
												gauge.set(-5 + (Math.random() * (10 - -5)));
												histogram.observe(0 + (Math.random() * (5 - 0)));

												summary.observe(0 + (Math.random() * (5 - 0)));

												TimeUnit.SECONDS.sleep(5);
											} catch (InterruptedException e) {
												e.printStackTrace();
											}
										}
									});
							
						

Instrumenting - Rebuilding for summary metric

After defining the summary metric, you have to rebuild the Java executable jar file and run it again:

							
								$ mvn -f basic-pom.xml clean install  (watch for BUILD SUCCESS)

								$ java -jar target/basic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar

								Java example metrics setup successful...

								Java example service started...
							
						

Instrumenting - Validating your summary metric

Check the endpoint at localhost:9999/metrics for the summary metric as shown below with a random number that changes every 5 seconds:

							
								# HELP java_app_s is a summary metric (request size in bytes)
								# TYPE java_app_s summary
								java_app_s{quantile="0.5",} 2.3090768039675225
								java_app_s{quantile="0.9",} 2.3090768039675225
								java_app_s{quantile="0.99",} 2.3090768039675225
								java_app_s_count 2.0
								java_app_s_sum 6.163592371161444
								# HELP java_app_h is a histogram metric
								# TYPE java_app_h histogram
								java_app_h_bucket{le="0.005",} 0.0
								...
								java_app_h_bucket{le="+Inf",} 2.0
								java_app_h_count 2.0
								java_app_h_sum 5.434964201169219
								# HELP java_app_c is a counter metric
								# TYPE java_app_c counter
								java_app_c 2.0
								# HELP java_app_g is a gauge metric
								# TYPE java_app_g gauge
								java_app_g 4.762373540089195
							
						

Instrumenting - Prometheus configuration adjustment

While the metrics are exposed in this example on localhost:9999, they will not be scraped by Prometheus until you have updated workshop-prometheus.yml file by adding the java application job as shown (I've added a few custom labels for fun):

							
								scrape_configs:

									# Scraping Prometheus.
									- job_name: "prometheus"
									static_configs:
										- targets: ["localhost:9090"]

									# Scraping java metrics.
									- job_name: "java_app"
									static_configs:
										- targets: ["localhost:9999"]
								          labels:
								            job: "java_app"
								            env: "workshop-lab8"
							
						

Instrumenting - Start Prometheus instance

Start your Prometheus instance (for container Prometheus, see next slide) and then watch the console where you started the Java application as it will report each time a new scrape is done by logging Handled :/metrics:

							
								$ ./prometheus --config.file=support/workshop-prometheus.yml

								===========Java application log===============
								Java example metrics setup successful...

								Java example service started...

								Handled :/metrics
								Handled :/metrics
								Handled :/metrics
								Handled :/metrics
								...
							
						

Instrumenting - Scraping Java metrics

You can validate that the Java metrics you just instrumented in your application are available in the Prometheus console localhost:9090 as shown. Feel free to query and explore:

Lab completed - Results

Next up, metrics monitoring at scale...

• Video introduction to workshop (Open Source Edinburgh meetup)
• Getting started with cloud native o11y workshops
• This workshop project repository
• O11y workshop collection
• Prometheus project
• PromLens project
• Perses project
• Report an issue with this workshop

Contact - are there any questions?

Eric D. Schabell
Director Evangelism
Contact: @ericschabell {@fosstodon.org) or https://www.schabell.org

Up next in workshop...

Conclusion - Metrics Monitoring at Scale