Instrumenting - Example Java application
For the rest of this lab you've chosen to run your instrumenting experience from the source
project as a container image on your local machine.
For this lab we assume you are using Podman, but you can also use Docker.
Instrumenting - Java metrics demo project
First, we'll give you a project with all the Java layout you need and a completed example of
the four basic metrics instrumented for an example application. To install this locally:
Instrumenting - Download and unzip project
Once you have downloaded the project using the link provided previously, just unzip into its
own directory as shown (Note: examples shown are on Mac OSX system):
$ unzip workshop-prometheus/prometheus-java-metrics-demo-v0.3.zip
Archive: /Users/erics/Downloads/prometheus-java-metrics-demo-v0.3.zip
a66e4ea8ea9d1313b257fcf3b3321d9ce47a86e8
creating: prometheus-java-metrics-demo-v0.3/
extracting: prometheus-java-metrics-demo-v0.3/.gitignore
inflating: prometheus-java-metrics-demo-v0.3/README.md
inflating: prometheus-java-metrics-demo-v0.3/basic-pom.xml
inflating: prometheus-java-metrics-demo-v0.3/pom.xml
creating: prometheus-java-metrics-demo-v0.3/src/
creating: prometheus-java-metrics-demo-v0.3/src/main/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/io/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/
creating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/java_apps/
inflating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/java_apps/BasicJavaMetrics.java
inflating: prometheus-java-metrics-demo-v0.3/src/main/java/io/chronosphere/java_apps/MyBasicJavaMetrics.java
Instrumenting - Opening project in your IDE
Here you see the project opened in VSCode (use any tooling you like), which allows you to explore
the project single class file with a fully instrumented example Basic Java Metric application:
Instrumenting - Running the Java example
You have to build the BasicJavaMetrics
application from the command line
in the project root directory, before you can build your container image. First build your
Java using Maven as shown here:
$ cd prometheus-java-metrics-demo-v0.3/
$ mvn clean install (watch for BUILD SUCCESS)
Instrumenting - Building Java container
Now you need to use the provided Buildfile
and podman tooling to build your
container image as shown:
$ podman build -t basic-java-metrics -f Buildfile
STEP 1/4: FROM openjdk:19
STEP 2/4: ADD target/java_metrics-1.0-xxxx.jar java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
--> 622fb698bbab
STEP 3/4: ENTRYPOINT ["java", "-jar","java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar"]
--> ff4ac83c2ac0
STEP 4/4: EXPOSE 7777
COMMIT basic-java-metrics
--> 21d6c8d06793
Successfully tagged localhost/basic-java-metrics:latest
21d6c8d06793f854f798f0c5ffd8ad3cd1a4960575ec8c83ef23d04702047f43
Instrumenting - Running Java container
Now run the container image as shown, noting it's going to be found on port 7777:
$ podman run -p 7777:7777 basic-java-metrics
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating Java metrics setup
Make sure the Java metrics endpoint is working by opening the endpoint to be scraped by a
Prometheus instance on http://localhost:7777/metrics
, showing something
like this:
# HELP java_app_c_total example counter
# TYPE java_app_c_total counter
java_app_c_total{status="error"} 239.0
java_app_c_total{status="ok"} 478.0
# HELP java_app_g_seconds is a gauge metric
# TYPE java_app_g_seconds gauge
java_app_g_seconds{value="value"} 7.29573889110867
# HELP java_app_h_seconds is a histogram metric
# TYPE java_app_h_seconds histogram
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.005"} 0
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.01"} 0
...
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="10.0"} 10
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="+Inf"} 239
java_app_h_seconds_count{method="GET",path="/",status_code="200"} 239
java_app_h_seconds_sum{method="GET",path="/",status_code="200"} 28475.853574282995
# HELP java_app_s_seconds is summary metric (request latency in seconds)
# TYPE java_app_s_seconds summary
java_app_s_seconds{status="ok",quantile="0.5"} 2.870230936180606
java_app_s_seconds{status="ok",quantile="0.95"} 4.888056778494996
java_app_s_seconds{status="ok",quantile="0.99"} 4.903344773262025
java_app_s_seconds_count{status="ok"} 239
java_app_s_seconds_sum{status="ok"} 607.9779550254922
Instrumenting - Prometheus configuration new metrics
While the metrics are exposed in this example on localhost:7777
, they will
not be scraped by Prometheus until you have updated its configuration to add this new end point.
Let's update our workshop-prometheus.yml
file to add the java application job as
shown along with comments for clarity (this is the minimum needed, with a few custom labels for
fun):
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping java metrics.
- job_name: "java_app"
static_configs:
- targets: ["localhost:7777"]
labels:
job: "java_app"
env: "workshop-lab8"
Instrumenting - Start Prometheus instance
Start the prometheus instance by rebuilding the container image and then running it. Validate in
your Prometheus console using the query console and up
to validate the
endpoint is being scraped:
$ podman build -t workshop-prometheus:v2.54.1 -f Buildfile
STEP 1/2: FROM prom/prometheus:v2.54.1
STEP 2/2: ADD workshop-prometheus.yml /etc/prometheus
COMMIT workshop-prometheus:v2.54.1
--> 9fe643412975
Successfully tagged localhost/workshop-prometheus:v2.54.1
9fe6434129753c0248f068af3739bd4f01dd3d3a36dafaa130f1c3b794a279df
$ podman run -p 9090:9090 workshop-prometheus:v2.54.1 --config.file=/etc/prometheus/workshop-prometheus.yml
...
ts=2024-05-30T07:36:27.291Z level=info msg="TSDB started"
ts=2024-05-30T07:36:27.291Z level=info msg="Loading configuration file" filename=/etc/prometheus/workshop-prometheus.yml
ts=2024-05-30T07:36:27.293Z level=info msg="Completed loading configuration file" filename=/etc/prometheus/workshop-prometheus.yml
ts=2024-05-30T07:36:27.293Z level=info msg="Server is ready to receive web requests."
ts=2024-05-30T07:36:27.293Z level=info component="rule manager" msg="Starting rule manager..."
Intermezzo - Prometheus container issues?
When you try to view the Java metrics from Prometheus, you'll notice that the current
configuration does not work. The target is shown DOWN
and unreachable, why?
The container version of Prometheus can't find the locally running Java metrics example when
listed as localhost
because that is local to the Prometheus container. We
need to find out what the IP address is on your local machine for these containers as follows
(if following the workshop, you've seen this earlier). Using the container variable we can
update our Prometheus configuration file as follows:
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping java metrics.
- job_name: "java_app"
static_configs:
- targets: ["host.containers.internal:7777"]
labels:
job: "java_app"
env: "workshop-lab8"
Instrumenting - Restart Prometheus instance
Restart the prometheus instance by rebuilding the container image and then running it. Validate in
your Prometheus console using the query console and up
to validate the
endpoint is being scraped:
$ podman build -t workshop-prometheus:v2.54.1 -f Buildfile
STEP 1/2: FROM prom/prometheus:v2.54.1
STEP 2/2: ADD workshop-prometheus.yml /etc/prometheus
COMMIT workshop-prometheus:v2.54.1
--> 9fe643412975
Successfully tagged localhost/workshop-prometheus:v2.54.1
9fe6434129753c0248f068af3739bd4f01dd3d3a36dafaa130f1c3b794a279df
$ podman run -p 9090:9090 workshop-prometheus:v2.54.1 --config.file=/etc/prometheus/workshop-prometheus.yml
...
ts=2024-05-30T07:36:27.291Z level=info msg="TSDB started"
ts=2024-05-30T07:36:27.291Z level=info msg="Loading configuration file" filename=/etc/prometheus/workshop-prometheus.yml
ts=2024-05-30T07:36:27.293Z level=info msg="Completed loading configuration file" filename=/etc/prometheus/workshop-prometheus.yml
ts=2024-05-30T07:36:27.293Z level=info msg="Server is ready to receive web requests."
ts=2024-05-30T07:36:27.293Z level=info component="rule manager" msg="Starting rule manager..."
Instrumenting - Starting my Java metrics
Now stop the running Java metrics example (from the console just hit CTRL + C).
Let's start by creating your own Java metrics application starting with the minimal setup needed
to get your Java application running and exposing it's /metrics
. Instead of
coding it all by hand, we've put a starting point in the file
MyBasicJavaMetrics.java
found in the same directory, so open this file in your
IDE:
src/main/java/io/chronosphere/java_apps/MyBasicJavaMetrics.java
Instrumenting - Exploring MyBasicJavaMetrics
This class has a few basic imports, then a variable you can use to set the port you want your
metrics to be exposed on. The main method has two parts, first you instrument this class with
metrics, currently commented out as placeholders:
public static void main(String[] args) throws InterruptedException, IOException {
// Set up and default Java metrics.
//JvmMetrics.builder().register();
// Initialize Counter.
// TODO: counter init
// Initialize Gauge.
// TODO: gauge init
// Initialize Histogram.
// TODO: histogram init
// Initialize Summary.
// TODO: summary init
// Start thread and apply values to metrics.
// TODO: set up this thread and populate metrics.
Instrumenting - Exploring more MyBasicJavaMetrics
The second half of the main method is used to setup a simple web server to serve the metrics
up for scraping by Prometheus and then a few print statements to notify you that instrumentation
has completed and that your application or service has started:
HTTPServer server = HTTPServer.builder().port(METRICS_PORT).buildAndStart();
System.out.println("HTTPServer listening on port http://localhost:" + server.getPort() + "/metrics");
System.out.println("");
System.out.println("Basic Java metrics setup successful...");
System.out.println("");
System.out.println("My application or service started...");
System.out.println("");
}
Instrumenting - Building and running my metrics
To help you run this MyBasicJavaMetrics.java
class file you are given a
separate mybasic-pom.xml
file, so either point your IDE build process at this
new pom file, or follow the command line example shown below:
$ cd prometheus-java-metrics-demo-v0.3/
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
Instrumenting - Editing container buildfile
Now you need to use the provided Buildfile
but you need to edit this to
pull the new jar file into your container image, so open your Buildfile, edit as shown to use
your newly built jar file:
FROM openjdk:19
ADD target/mybasic_java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
ENTRYPOINT ["java", "-jar","java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar"]
EXPOSE 9999
Instrumenting - Building new container
Build your new container image pulling the new jar file as shown:
$ podman build -t basic-java-metrics -f Buildfile
STEP 1/4: FROM openjdk:19
STEP 2/4: ADD target/mybasic_java_metrics-1.0-xxxx.jar java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar
--> 622fb698bbab
STEP 3/4: ENTRYPOINT ["java", "-jar","java_metrics-1.0-SNAPSHOT-jar-with-dependencies.jar"]
--> ff4ac83c2ac0
STEP 4/4: EXPOSE 9999
COMMIT basic-java-metrics
--> 21d6c8d06793
Successfully tagged localhost/basic-java-metrics:latest
21d6c8d06793f854f798f0c5ffd8ad3cd1a4960575ec8c83ef23d04702047f43
Instrumenting - Running new container
Now run the new container image as shown, noting it's going to be found on port 9999 for the
MyBasicJavaMetrics class
:
$ podman run -p 9999:9999 basic-java-metrics
HTTPServer listening on port http:
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating basic metrics setup
Make sure the basic metrics endpoint is working by opening the endpoint to be scraped by a
Prometheus instance on localhost:9999/metrics
.
But wait, it's showing a blank screen???
Instrumenting - Uncommenting default Java metrics
If you were looking closely at the main method, you might have noticed that there was a
commented out line //JvmMetrics.builder().register();
that can be uncommented
to initialize all the default Java metrics:
// Set up and default Java metrics.
//JvmMetrics.builder().register(); <<<<< UNCOMMENT THIS LINE
// Initialize Counter.
// TODO: counter init
Instrumenting - Rebuilding and running container
After editing the default metrics initialization line, you have to rebuild the Java executable
jar file and run the new container image:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ podman build -t basic-java-metrics -f Buildfile
$ podman run -p 9999:9999 basic-java-metrics
HTTPServer listening on port http://localhost:9999/metrics
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Trying validating again
This time when you check the endpoint at localhost:9999/metrics
you're going
to find all kinds of Java metrics listed such as:
jvm_classes_loaded 1498.0
jvm_classes_loaded_total 1498.0
jvm_classes_unloaded_total 0.0
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
jvm_buffer_pool_used_bytes{pool="direct",} 8192.0
jvm_buffer_pool_used_bytes{pool="mapped - 'non-volatile memory'",} 0.0
...
Instrumenting - Adding a counter metric
Now that you have a working setup, you'll start with defining your first real instrumented
metric, a counter. To do this you will adjust the existing main class to add a method to
initialize a counter and create a private method defining your counter. In the next slide you'll
see the updated main method with the default initialized metrics method removed.
Instrumenting - Initializing the counter
Note the larger and bold printed line is what you need to initialize a counter. Next you need
to add the private method below the main method where you define how the counter is initialized:
// Set up and default Java metrics.
JvmMetrics.builder().register();
// Initialize Counter.
Counter counter = counter();
// Initialized Gauge.
// TODO: gauge init
Instrumenting - Private counter method
This is the method to initialize the counter:
private static Counter counter() {
Counter counter = Counter.builder()
.name("java_app_c")
.help("example counter")
.labelNames("status")
.register();
return counter;
}
Instrumenting - Updating your import statements
At the top of your basic metrics file you'll find the client library imports. One new line as
appeared if your IDE automatically imports for you, otherwise make sure it's there:
import io.prometheus.metrics.core.metrics.Counter;
Note that now you are developing your own instrumented metrics, you can comment out the default
JVM metrics line and remove its corresponding import statement.
Instrumenting - Populating your metrics
The last TODO marker in the main method asks you to start a thread and populate your metrics.
This is only for this workshop that we are going to populate our metrics with data, some of it
random numbers. Feel free to cut and paste the code below that refreshes the metrics values
every second:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Rebuilding for counter metric
After defining the counter metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ podman build -t basic-java-metrics -f Buildfile
$ podman run -p 9999:9999 basic-java-metrics
HTTPServer listening on port http://localhost:9999/metrics
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your counter metric
Check the endpoint at localhost:9999/metrics
you're going to find the
counter metric as shown below:
java_app_c_total{status="error"} 8.0
java_app_c_total{status="ok"} 16.0
Instrumenting - Adding a gauge metric
Next up we can add a gauge metric and you'll do that by initializing a gauge, and then creating
a private method to create the gauge. In the next slide you'll see the
updated main method.
Instrumenting - Initializing a gauge
Note the larger and bold printed line is what you need to initialize a gauge. On the next
slide you'll add the private method below the main method defining how this gauge is initialized:
// Set up and default Java metrics.
//JvmMetrics.builder().register();
// Initialize Counter.
Counter counter = counter();
// Initialized Gauge.
Gauge gauge = gauge();
// Initialize Histogram.
// TODO: histogram init
Instrumenting - Private gauge method
This is the method to initialize the gauge:
private static Gauge gauge() {
Gauge gauge = Gauge.builder()
.name("java_app_g")
.help("is a gauge metric")
.labelNames("value")
.unit(Unit.SECONDS)
.register();
return gauge;
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Gauge
import as follows:
import io.prometheus.metrics.core.metrics.Gauge;
Instrumenting - Populating your gauge metric
You're now going to expand the thread initialization in the main method to populate the gauge
metric. Feel free to cut and paste the gauge definition below:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
gauge.labelValues("value").set(rand(-5, 10));
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Random number generating method
Notice that you are using a random rand()
method call in the population of
the gauge to grab random numbers, so make sure you add this method below to the bottom of this
class:
private static double rand(double min, double max) {
return min + (Math.random() * (max - min));
}
Instrumenting - Rebuilding for gauge metric
After defining the gauge metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ podman build -t basic-java-metrics -f Buildfile
$ podman run -p 9999:9999 basic-java-metrics
HTTPServer listening on port http://localhost:9999/metrics
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your gauge metric
Check the endpoint at localhost:9999/metrics
you're going to find the
gauge metric as shown below with a random number that changes every couple of seconds:
java_app_c_total{status="error"} 6.0
java_app_c_total{status="ok"} 12.0
java_app_g_seconds{value="value"} -2.051171083831968
Instrumenting - Adding a histogram metric
Next up we can add a histogram metric and you'll do that by initializing it before creating
a private method to create the histogram. In the next slide you'll see the updated main method.
Instrumenting - Initializing a histogram
Note the larger and bold printed line is what you need to initialize a histogram. On the next
slide you'll add the private methode below the main method defining how this histogram is
initialized:
// Initialize Counter.
Counter counter = counter();
// Initialized Gauge.
Gauge gauge = gauge();
// Initialize Histogram.
Histogram histogram = histogram();
long start = System.nanoTime();
// Initialized Summary.
// TODO: summary init
Instrumenting - Private histogram method
This is the method to initialize the histogram:
private static Histogram histogram() {
Histogram histogram = Histogram.builder()
.name("java_app_h")
.help("is a histogram metric")
.unit(Unit.SECONDS)
.labelNames("method", "path", "status_code")
.register();
return histogram;
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Histogram
import as follows:
import io.prometheus.metrics.core.metrics.Histogram;
Instrumenting - Populating histogram metric
You're now going to expand the thread initialization in the main method to populate the histogram
metric. Feel free to cut and paste the gauge definition below:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
gauge.labelValues("value").set(rand(-5, 10));
histogram.labelValues("GET", "/", "200").observe(Unit.nanosToSeconds(System.nanoTime() - start));
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Rebuilding for histogram metric
After defining the histogram metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ podman build -t basic-java-metrics -f Buildfile
$ podman run -p 9999:9999 basic-java-metrics
HTTPServer listening on port http://localhost:9999/metrics
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your histogram metric
Check the endpoint at localhost:9999/metrics
for the histogram metric as
shown below with a random number that changes every couple of seconds:
# HELP java_app_c_total example counter
# TYPE java_app_c_total counter
java_app_c_total{status="error"} 6.0
java_app_c_total{status="ok"} 12.0
# HELP java_app_g_seconds is a gauge metric
# TYPE java_app_g_seconds gauge
java_app_g_seconds{value="value"} 9.56286725499423
# HELP java_app_h_seconds is a histogram metric
# TYPE java_app_h_seconds histogram
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.005"} 0
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.01"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.025"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.05"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.1"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.25"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.5"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="1.0"} 1
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="2.5"} 3
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="5.0"} 5
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="10.0"} 6
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="+Inf"} 6
java_app_h_seconds_count{method="GET",path="/",status_code="200"} 6
java_app_h_seconds_sum{method="GET",path="/",status_code="200"} 15.072195015999998
Instrumenting - Adding a summary metric
Finally, you're adding a summary metric and you'll do that by initializing it before creating
a private method to create the summary. In the next slide you'll see the updated main method.
Instrumenting - Initializing a summary
Note the larger and bold printed line is what you need to initialize a summary. On the next
slide you'll add the private methode below the main method defining how this summary is
initialized:
// Initialize Histogram.
Histogram histogram = histogram();
long start = System.nanoTime();
// Initialized Summary.
Summary summary = summary();
// Start thread and apply values to metrics.
Thread bgThread = new Thread(() -> {
Instrumenting - Private summary method
This is the method to initialize the summary:
// Creating a summary.
private static Summary summary() {
Summary summary = Summary.builder()
.name("java_app_s")
.help("is summary metric (request latency in seconds)")
.unit(Unit.SECONDS)
.quantile(0.5, 0.01)
.quantile(0.95, 0.005)
.quantile(0.99, 0.005)
.labelNames("status")
.register();
return summary;
}
Instrumenting - Updating your import statements
At the top of the basic metrics file you'll find the client library imports. You can add the
Summary
import as follows:
import io.prometheus.metrics.core.metrics.Summary;
Instrumenting - Populating summary metric
You're now going to expand the thread initialization in the main method to populate the summary
metric. Feel free to cut and paste the gauge definition below:
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.labelValues("ok").inc();
counter.labelValues("ok").inc();
counter.labelValues("error").inc();
gauge.labelValues("value").set(rand(-5, 10));
histogram.labelValues("GET", "/", "200").observe(Unit.nanosToSeconds(System.nanoTime() - start));
summary.labelValues("ok").observe(rand(0, 5));
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
Instrumenting - Rebuilding for summary metric
After defining the summary metric, you have to rebuild the Java executable jar file and run it
again:
$ mvn -f mybasic-pom.xml clean install (watch for BUILD SUCCESS)
$ podman build -t basic-java-metrics -f Buildfile
$ podman run -p 9999:9999 basic-java-metrics
HTTPServer listening on port http://localhost:9999/metrics
Basic Java metrics setup successful...
My application or service started...
Instrumenting - Validating your summary metric
Check the endpoint at localhost:9999/metrics
for the summary metric as
shown below with a random number that changes every couple of seconds:
# HELP java_app_c_total example counter
# TYPE java_app_c_total counter
java_app_c_total{status="error"} 6.0
java_app_c_total{status="ok"} 12.0
# HELP java_app_g_seconds is a gauge metric
# TYPE java_app_g_seconds gauge
java_app_g_seconds{value="value"} 9.56286725499423
# HELP java_app_h_seconds is a histogram metric
# TYPE java_app_h_seconds histogram
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.005"} 0
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="0.01"} 1
...
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="10.0"} 6
java_app_h_seconds_bucket{method="GET",path="/",status_code="200",le="+Inf"} 6
java_app_h_seconds_count{method="GET",path="/",status_code="200"} 6
java_app_h_seconds_sum{method="GET",path="/",status_code="200"} 15.072195015999998
# HELP java_app_s_seconds is summary metric (request latency in seconds)
# TYPE java_app_s_seconds summary
java_app_s_seconds{status="ok",quantile="0.5"} 3.0098277118187493
java_app_s_seconds{status="ok",quantile="0.95"} 4.617804632747773
java_app_s_seconds{status="ok",quantile="0.99"} 4.617804632747773
java_app_s_seconds_count{status="ok"} 5
java_app_s_seconds_sum{status="ok"} 14.307305756153882
Instrumenting - Prometheus configuration adjustment
While the metrics are exposed in this example on localhost:9999
, they will
not be scraped by Prometheus until you have updated workshop-prometheus.yml
file
by adding the java application job as shown (I've added a few custom labels for fun). Note this
was already queried for the IP address to use for the container running the Java application:
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping java metrics.
- job_name: "java_app"
static_configs:
- targets: ["host.containers.internal:9999"]
labels:
job: "java_app"
env: "workshop-lab8"
Instrumenting - Start Prometheus instance
Start your Prometheus instance (for container Prometheus, see next slide) and verify the
metrics are scraped in the Prometheus console (by querying UP
for example):
$ podman build -t workshop-prometheus:v2.54.1 -f Buildfile
STEP 1/2: FROM prom/prometheus:v2.54.1
STEP 2/2: ADD workshop-prometheus.yml /etc/prometheus
COMMIT workshop-prometheus:v2.54.1
--> 9fe643412975
Successfully tagged localhost/workshop-prometheus:v2.54.1
9fe6434129753c0248f068af3739bd4f01dd3d3a36dafaa130f1c3b794a279df
$ podman run -p 9090:9090 workshop-prometheus:v2.54.1 --config.file=/etc/prometheus/workshop-prometheus.yml
...
ts=2024-05-30T07:36:27.291Z level=info msg="TSDB started"
ts=2024-05-30T07:36:27.291Z level=info msg="Loading configuration file" filename=/etc/prometheus/workshop-prometheus.yml
ts=2024-05-30T07:36:27.293Z level=info msg="Completed loading configuration file" filename=/etc/prometheus/workshop-prometheus.yml
ts=2024-05-30T07:36:27.293Z level=info msg="Server is ready to receive web requests."
ts=2024-05-30T07:36:27.293Z level=info component="rule manager" msg="Starting rule manager..."
===========Query results for UP:===============
up{env="workshop-lab8", instance="192.168.1.4:9999", job="java_app"}
Instrumenting - Scraping Java metrics
You can validate that the Java metrics you just instrumented in your application are available
in the Prometheus console localhost:9090
as shown. Feel free to query and
explore:
Lab completed - Results
Next up, metrics monitoring at scale...
Contact - are there any questions?
Instrumenting - Example Java application For the rest of this lab you've chosen to run your instrumenting experience from the source
project as a container image on your local machine. For this lab we assume you are using Podman, but you can also use Docker.