Instrumenting - Example Java application
For the rest of this lab you've chosen to run your instrumenting experience from the source
project on your local machine.
For this lab we assume you have installed Java 8+ and Maven 3.8+.
Instrumenting - Java metrics demo project
First, we'll give you a project with all the Java layout you need and a completed example of
the four basic metrics instrumented for an example application. To install this locally:
Instrumenting - Download and unzip project
Once you have downloaded the project using the link provided previously, just unzip into its
own directory as shown (Note: examples shown are on Mac OSX system):
$ unzip workshop-prometheus/prometheus-java-metrics-demo-v0.3.zip
Archive: /Users/erics/Downloads/prometheus-java-metrics-demo-v0.3.zip
a66e4ea8ea9d1313b257fcf3b3321d9ce47a86e8
creating: prometheus-java-metrics-demo-v0.3/
extracting: prometheus-java-metrics-demo-v0.3/.gitignore
inflating: prometheus-java-metrics-demo-v0.3/README.md
inflating: prometheus-java-metrics-demo-v0.3/basic-pom.xml
inflating: prometheus-java-metrics-demo-v0.3/pom.xml
creating: prometheus-java-metrics-demo-v0.3/src/
creating: prometheus-java-metrics-demo-v0.3/src/main/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/io/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/java_apps/
inflating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/java_apps/BasicJavaMetrics.java
inflating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/java_apps/MyBasicJavaMetrics.java
Instrumenting - Opening project in your IDE
Here you see the project opened in VSCode (use any tooling you like), which allows you to explore
the project single class file with a fully instrumented example Basic Java Metric application:
Instrumenting - Running the Java example
You can either run the BasicJavaMetrics.java
class file from your IDE or
from the command line in the project root directory, as for example is shown below where you see
the example as started successfully:
$ cd prometheus-java-metrics-demo-v0.3/
$ mvn clean install (watch for BUILD SUCCESS)
$ java -jar target/java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating Java metrics setup
Make sure the Java metrics endpoint is working by opening the endpoint to be scraped by a
Prometheus instance on http://localhost:7777/metrics
, showing something
like this:
# HELP java_app_c_total example counter
# TYPE java_app_c_total counter
java_app_c_total{status="error"} 239.0
java_app_c_total{status="ok"} 478.0
# HELP java_app_g_seconds is a gauge metric
# TYPE java_app_g_seconds gauge
java_app_g_seconds{value="value"} 7.29573889110867
# HELP java_app_h_seconds is a histogram metric
# TYPE java_app_h_seconds histogram
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.005"} 0
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.01"} 0
...
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="10.0"} 10
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="+Inf"} 239
java_app_h_seconds_count{method="GET",path="/",status_code="200"} 239
java_app_h_seconds_sum{method="GET",path="/",status_code="200"} 28475.853574282995
# HELP java_app_s_seconds is summary metric (request latency in seconds)
# TYPE java_app_s_seconds summary
java_app_s_seconds{status="ok",quantile="0.5"} 2.870230936180606
java_app_s_seconds{status="ok",quantile="0.95"} 4.888056778494996
java_app_s_seconds{status="ok",quantile="0.99"} 4.903344773262025
java_app_s_seconds_count{status="ok"} 239
java_app_s_seconds_sum{status="ok"} 607.9779550254922
Instrumenting - Prometheus configuration new metrics
While the metrics are exposed in this example on localhost:7777
, they will
not be scraped by Prometheus until you have updated its configuration to add this new end point.
Let's update our workshop-prometheus.yml
file to add the java application job as
shown along with comments for clarity (this is the minimum needed, with a few custom labels for
fun):
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping java metrics.
- job_name: "java_app"
static_configs:
- targets: ["localhost:7777"]
labels:
job: "java_app"
env: "workshop-lab8"
Instrumenting - Start Prometheus instance
Start the prometheus instance and then validate in your Prometheus console using the query
console and up
to validate the endpoint is being scraped:
$ ./prometheus --config.file=support/workshop-prometheus.yml
===========Java application log===============
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Starting my Java metrics
Now stop the running Java metrics example (from the console just hit CTRL + C).
Let's start by creating your own Java metrics application starting with the minimal setup needed
to get your Java application running and exposing it's /metrics
. Instead of
coding it all by hand, we've put a starting point in the file
MyBasicJavaMetrics.java
found in the same directory, so open this file in your
IDE:
src/main/java/io/chronosphere/java_apps/MyBasicJavaMetrics.java
Instrumenting - Exploring MyBasicJavaMetrics
This class has a few basic imports, then a variable you can use to set the port you want your
metrics to be exposed on. The main method has two parts, first you instrument this class with
metrics, currently commented out as placeholders:
public static void main(String[] args) throws InterruptedException, IOException {
// Set up and default Java metrics.
//JvmMetrics.builder().register();
// Initialize Counter.
// TODO: counter init
// Initialize Gauge.
// TODO: gauge init
// Initialize Histogram.
// TODO: histogram init
// Initialize Summary.
// TODO: summary init
// Start thread and apply values to metrics.
// TODO: set up this thread and populate metrics.
Instrumenting - Exploring more MyBasicJavaMetrics
The second half of the main method is used to setup a simple web server to serve the metrics
up for scraping by Prometheus and then a few print statements to notify you that instrumentation
has completed and that your application or service has started:
HTTPServer server = HTTPServer.builder().port(METRICS_PORT).buildAndStart();
System.out.println("HTTPServer listening on port http://localhost:" + server.getPort() + "/metrics");
System.out.println("");
System.out.println("Basic Java metrics setup successful...");
System.out.println("");
System.out.println("My application or service started...");
System.out.println("");
}
Instrumenting - Building and running my metrics
To help you run this MyBasicJavaMetrics.java
class file you are given a
separate mybasic-pom.xml
file, so either point your IDE build process at this
new pom file, or follow the command line example shown below:
$ cd prometheus-java-metrics-demo-v0.3/
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/mybasic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Validating basic metrics setup
Make sure the basic metrics endpoint is working by opening the endpoint to be scraped by a
Prometheus instance on localhost:9999/metrics
.
But wait, it's showing a blank screen???
Instrumenting - Uncommenting default Java metrics
If you were looking closely at the main method, you might have noticed that there was a
commented out line //JvmMetrics.builder().register();
that can be uncommented
to initialize all the default Java metrics:
// Set up and default Java metrics.
//JvmMetrics.builder().register(); <<<<< UNCOMMENT THIS LINE
// Initialize Counter.
// TODO: counter init
Instrumenting - Rebuilding and running basic metrics
After editing the default metrics initialization line, you have to rebuild the Java executable
jar file and run it again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/mybasic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
Java example metrics setup successful...
Java example service started...
Instrumenting - Trying validating again
This time when you check the endpoint at localhost:9999/metrics
you're going
to find all kinds of Java metrics listed such as:
jvm_classes_loaded 1498.0
jvm_classes_loaded_total 1498.0
jvm_classes_unloaded_total 0.0
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 8192.0
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
...
Instrumenting - Adding a counter metric
Now that you have a working setup, you'll start with defining your first real instrumented
metric, a counter. To do this you will adjust the existing main class to add a method to
initialize a counter and create a private method defining your counter. In the next slide you'll
see the updated main method with the default initialized metrics method removed.
Instrumenting - Initializing the counter
Note the larger and bold printed line is what you need to initialize a counter. Next you need
to add the private method below the main method where you define how the counter is initialized:
// Set up and default Java metrics.
JvmMetrics.builder().register();
// Initialize Counter.
Counter counter = counter();
// Initialized Gauge.
// TODO: gauge init
Instrumenting - Private counter method
This is the method to initialize the counter:
private static Counter counter() {
Counter counter = Counter.builder()
.name("java_app_c")
.help("example counter")
.labelNames("status")
.register();
return counter;
}
Instrumenting - Updating your import statements
At the top of your basic metrics file you'll find the client library imports. One new line as
appeared if your IDE automatically imports for you, otherwise make sure it's there:
import io.prometheus.metrics.core.metrics.Counter;
Note that now you are developing your own instrumented metrics, you can comment out the default
JVM metrics line and remove its corresponding import statement.
Instrumenting - Populating your metrics
The last TODO marker in the main method asks you to start a thread and populate your metrics.
This is only for this workshop that we are going to populate our metrics with data, some of it
random numbers. Feel free to cut and paste the code below that refreshes the metrics values
every second:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Rebuilding for counter metric
After defining the counter metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/mybasic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your counter metric
Check the endpoint at localhost:9999/metrics
you're going to find the
counter metric as shown below:
java_app_c_total{status="error"} 8.0
java_app_c_total{status="ok"} 16.0
Instrumenting - Adding a gauge metric
Next up we can add a gauge metric and you'll do that by initializing a gauge, and then creating
a private method to create the gauge. In the next slide you'll see the
updated main method.
Instrumenting - Initializing a gauge
Note the larger and bold printed line is what you need to initialize a gauge. On the next
slide you'll add the private method below the main method defining how this gauge is initialized:
// Set up and default Java metrics.
//JvmMetrics.builder().register();
// Initialize Counter.
Counter counter = counter();
// Initialized Gauge.
Gauge gauge = gauge();
// Initialize Histogram.
// TODO: histogram init
Instrumenting - Private gauge method
This is the method to initialize the gauge:
private static Gauge gauge() {
Gauge gauge = Gauge.builder()
.name("java_app_g")
.help("is a gauge metric")
.labelNames("value")
.unit(Unit.SECONDS)
.register();
return gauge;
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Gauge
import as follows:
import io.prometheus.metrics.core.metrics.Gauge;
Instrumenting - Populating your gauge metric
You're now going to expand the thread initialization in the main method to populate the gauge
metric. Feel free to cut and paste the gauge definition below:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
gauge.labelValues("value").set(rand(-5, 10));
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Random number generating method
Notice that you are using a random rand()
method call in the population of
the gauge to grab random numbers, so make sure you add this method below to the bottom of this
class:
private static double rand(double min, double max) {
return min + (Math.random() * (max - min));
}
Instrumenting - Rebuilding for gauge metric
After defining the gauge metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/mybasic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your gauge metric
Check the endpoint at localhost:9999/metrics
you're going to find the
gauge metric as shown below with a random number that changes every couple of seconds:
java_app_c_total{status="error"} 6.0
java_app_c_total{status="ok"} 12.0
java_app_g_seconds{value="value"} -2.051171083831968
Instrumenting - Adding a histogram metric
Next up we can add a histogram metric and you'll do that by initializing it before creating
a private method to create the histogram. In the next slide you'll see the updated main method.
Instrumenting - Initializing a histogram
Note the larger and bold printed line is what you need to initialize a histogram. On the next
slide you'll add the private methode below the main method defining how this histogram is
initialized:
// Initialize Counter.
Counter counter = counter();
// Initialized Gauge.
Gauge gauge = gauge();
// Initialize Histogram.
Histogram histogram = histogram();
long start = System.nanoTime();
// Initialized Summary.
// TODO: summary init
Instrumenting - Private histogram method
This is the method to initialize the histogram:
private static Histogram histogram() {
Histogram histogram = Histogram.builder()
.name("java_app_h")
.help("is a histogram metric")
.unit(Unit.SECONDS)
.labelNames("method", "path", "status_code")
.register();
return histogram;
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Histogram
import as follows:
import io.prometheus.metrics.core.metrics.Histogram;
Instrumenting - Populating histogram metric
You're now going to expand the thread initialization in the main method to populate the histogram
metric. Feel free to cut and paste the gauge definition below:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
gauge.labelValues("value").set(rand(-5, 10));
histogram.labelValues("GET", "/", "200").observe(Unit.nanosToSeconds(System.nanoTime() - start));
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Rebuilding for histogram metric
After defining the histogram metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/mybasic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your histogram metric
Check the endpoint at localhost:9999/metrics
for the histogram metric as
shown below with a random number that changes every couple of seconds:
# HELP java_app_c_total example counter
# TYPE java_app_c_total counter
java_app_c_total{status="error"} 6.0
java_app_c_total{status="ok"} 12.0
# HELP java_app_g_seconds is a gauge metric
# TYPE java_app_g_seconds gauge
java_app_g_seconds{value="value"} 9.56286725499423
# HELP java_app_h_seconds is a histogram metric
# TYPE java_app_h_seconds histogram
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.005"} 0
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.01"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.025"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.05"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.1"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.25"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.5"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="1.0"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="2.5"} 3
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="5.0"} 5
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="10.0"} 6
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="+Inf"} 6
java_app_h_seconds_count{method="GET",path="/",status_code="200"} 6
java_app_h_seconds_sum{method="GET",path="/",status_code="200"} 15.072195015999998
Instrumenting - Adding a summary metric
Finally, you're adding a summary metric and you'll do that by initializing it before creating
a private method to create the summary. In the next slide you'll see the updated main method.
Instrumenting - Initializing a summary
Note the larger and bold printed line is what you need to initialize a summary. On the next
slide you'll add the private methode below the main method defining how this summary is
initialized:
// Initialize Histogram.
Histogram histogram = histogram();
long start = System.nanoTime();
// Initialized Summary.
Summary summary = summary();
// Start thread and apply values to metrics.
Thread bgThread = new Thread(() -> {
Instrumenting - Private summary method
This is the method to initialize the summary:
// Creating a summary.
private static Summary summary() {
Summary summary = Summary.builder()
.name("java_app_s")
.help("is summary metric (request latency in seconds)")
.unit(Unit.SECONDS)
.quantile(0.5, 0.01)
.quantile(0.95, 0.005)
.quantile(0.99, 0.005)
.labelNames("status")
.register();
return summary;
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Summary
import as follows:
import io.prometheus.metrics.core.metrics.Summary;
Instrumenting - Populating summary metric
You're now going to expand the thread initialization in the main method to populate the summary
metric. Feel free to cut and paste the gauge definition below:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
gauge.labelValues("value").set(rand(-5, 10));
histogram.labelValues("GET", "/", "200").observe(Unit.nanosToSeconds(System.nanoTime() - start));
summary.labelValues("ok").observe(rand(0, 5));
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Rebuilding for summary metric
After defining the summary metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ java -jar target/mybasic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your summary metric
Check the endpoint at localhost:9999/metrics
for the summary metric as
shown below with a random number that changes every couple of seconds:
# HELP java_app_c_total example counter
# TYPE java_app_c_total counter
java_app_c_total{status="error"} 6.0
java_app_c_total{status="ok"} 12.0
# HELP java_app_g_seconds is a gauge metric
# TYPE java_app_g_seconds gauge
java_app_g_seconds{value="value"} 9.56286725499423
# HELP java_app_h_seconds is a histogram metric
# TYPE java_app_h_seconds histogram
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.005"} 0
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.01"} 1
...
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="10.0"} 6
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="+Inf"} 6
java_app_h_seconds_count{method="GET",path="/",status_code="200"} 6
java_app_h_seconds_sum{method="GET",path="/",status_code="200"} 15.072195015999998
# HELP java_app_s_seconds is summary metric (request latency in seconds)
# TYPE java_app_s_seconds summary
java_app_s_seconds{status="ok",quantile="0.5"} 3.0098277118187493
java_app_s_seconds{status="ok",quantile="0.95"} 4.617804632747773
java_app_s_seconds{status="ok",quantile="0.99"} 4.617804632747773
java_app_s_seconds_count{status="ok"} 5
java_app_s_seconds_sum{status="ok"} 14.307305756153882
Instrumenting - Prometheus configuration adjustment
While the metrics are exposed in this example on localhost:9999
, they will
not be scraped by Prometheus until you have updated workshop-prometheus.yml
file by adding the java application job as shown (I've added a few custom labels for fun):
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping java metrics.
- job_name: "java_app"
static_configs:
- targets: ["localhost:9999"]
labels:
job: "java_app"
env: "workshop-lab8"
Instrumenting - Start Prometheus instance
Start your Prometheus instance (for container Prometheus, see next slide) and verify the
metrics are scraped in the Prometheus console (by querying UP
for example):
$ ./prometheus --config.file=support/workshop-prometheus.yml
===========Query results for UP:===============
up{env="workshop-lab8", instance="localhost:9999", job="java_app"} 1
Instrumenting - Scraping Java metrics
You can validate that the Java metrics you just instrumented in your application are available
in the Prometheus console localhost:9090
as shown. Feel free to query and
explore:
Lab completed - Results
Next up, metrics monitoring at scale...
Contact - are there any questions?
Instrumenting - Example Java application For the rest of this lab you've chosen to run your instrumenting experience from the source
project on your local machine. For this lab we assume you have installed Java 8+ and Maven 3.8+.