Services demo - Installing and running
To install the services demo we're providing a self contained project you can download, unzip,
and run an installation script to quickly get started. Just follow these easy steps:
- Download and unzip the Prometheus Services Demo Installer.
- Run 'init.sh' file.
- Follow the instructions in the terminal output to start the services.
Services demo - Download and unzip
Once you have downloaded the project using the link provided previously, just unzip into its
own directory as shown (Note: examples shown are on Mac OSX system):
$ unzip prometheus-service-demo-installer-v1.0.zip
creating: prometheus-service-demo-installer-v1.0/
inflating: prometheus-service-demo-installer-v1.0/.gitignore
inflating: prometheus-service-demo-installer-v1.0/README.md
creating: prometheus-service-demo-installer-v1.0/docs/
creating: prometheus-service-demo-installer-v1.0/docs/demo-images/
inflating: prometheus-service-demo-installer-v1.0/docs/demo-images/workshop.png
inflating: prometheus-service-demo-installer-v1.0/init.sh
creating: prometheus-service-demo-installer-v1.0/installs/
inflating: prometheus-service-demo-installer-v1.0/installs/README
creating: prometheus-service-demo-installer-v1.0/installs/demo/
inflating: prometheus-service-demo-installer-v1.0/installs/demo/Buildfile
inflating: prometheus-service-demo-installer-v1.0/installs/demo/Dockerfile
inflating: prometheus-service-demo-installer-v1.0/installs/demo/LICENSE
inflating: prometheus-service-demo-installer-v1.0/installs/demo/README.md
...
creating: prometheus-service-demo-installer-v1.0/support/
extracting: prometheus-service-demo-installer-v1.0/support/README
inflating: prometheus-service-demo-installer-v1.0/support/unzip.vbs
inflating: prometheus-service-demo-installer-v1.0/support/workshop-prometheus.yml
Services demo - Exploring the installer
Now change into the project directory explore the demo source code found in the
installs directory as shown. This is the code that the installer will build
automatically:
$ cd prometheus-service-demo-installer-v1.0
$ ls -1
README.md
docs/
init.sh
installs/
support/
$ ls installs/demo/
Buildfile LICENSE Dockerfile README.md api.go
batch.go client.go cpu.go crossbuild.sh disk.go
go.mod go.sum holiday.go intermittent_metric.go
main.go memory.go
Services demo - Run installation script
Time to build this project, so let's run the init script to build the services
project. Run the command below and see the next slide for the output and explanation:
$ ./init.sh
Services demo - Installation introduction
The installation starts off with a nice ascii art sign to let you know you are about to install
the services project:
##########################################################################
## ##
## Setting up the Prometheus Service Demo Installer ##
## ##
## #### #### ### # # ##### ##### # # ##### # # #### ##
## # # # # # # ## ## # # # # # # # # ##
## #### #### # # # # # ### # ##### ### # # ### ##
## # # # # # # # # # # # # # # # ##
## # # # ### # # ##### # # # ##### ### #### ##
## ##
## #### ##### ##### # # ##### #### ##### ##
## # # # # # # # # # ##
## ### ### ##### # # # # ### ##
## # # # # # # # # # ##
## #### ##### # # # ##### #### ##### ##
## ##
## #### ##### # # ### ##
## # # # ## ## # # ##
## # # ### # # # # # ##
## # # # # # # # ##
## #### ##### # # ### ##
## ##
## ##### # # #### ##### ### # # ##### ##### ##
## # ## # # # # # # # # # # ##
## # # # # ### # ##### # # ### ##### ##
## # # ## # # # # # # # # # ##
## ##### # # #### # # # ##### ##### ##### # # ##
## ##
## brought to you by Eric D. Schabell ##
## ##
## git@gitlab.com:o11y-workshops/prometheus-service-demo-installer.git ##
## ##
##########################################################################
Services demo - Testing and building
Next the installer tests if you have GO installed and a valid version, if not you are pointed
to where to install it. In this example you see that it's a valid version of Go so it continues
on to build the services demo code:
Checking for Go version needed:
Go major version is good...
Go minor version is good...
Your Go version 1.21 accepted!
Building the services...
Prometheus service project built successfully!
Services demo - Completed building
If all goes well you have now see the following successful build message with pointers to how
to run the services demo binary you built and have the link to verify the metrics are exposed
in your browser:
======================================================
= =
= Installer complete, get ready to rock! =
= =
= The Prometheus services can be started from a =
= terminal with the following command: =
= =
= $ ./installs/demo/services_demo =
= =
= You can make sure the service is running by =
= viewing the exported metrics at: =
= =
= http://localhost:8080/metrics =
= =
======================================================
Services demo - Testing the metrics
After starting the services binary as shown in the terminal output:
$ ./installs/demo/services_demo
It should have metrics scraping endpoints available so that you can query them with PromQL.
Test at
http://localhost:8080/metrics:
# HELP demo_api_http_requests_in_progress The current number of API HTTP requests in progress.
# TYPE demo_api_http_requests_in_progress gauge
demo_api_http_requests_in_progress 1
# HELP demo_api_request_duration_seconds A histogram of the API HTTP request durations in seconds.
# TYPE demo_api_request_duration_seconds histogram
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0001"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00015000000000000001"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00022500000000000002"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0003375"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.00050625"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.000759375"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0011390624999999999"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0017085937499999998"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0025628906249999996"} 0
demo_api_request_duration_seconds_bucket{method="GET",path="/api/bar",status="200",le="0.0038443359374999994"} 0
...
Services demo - Prometheus configuration
While the metrics are exposed, they will not be scraped by Prometheus until you have updated its
configuration to add this new end point. Let's update our
workshop-prometheus.yml file to add the services job as shown along with
comments for clarity:
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# Scraping services demo.
- job_name: "services"
static_configs:
- targets: ["localhost:8080"]
Services demo - Applying the configuration
Previously you learned to restart the Prometheus to apply configuration changes. If you have
a running instance you don't want to lose a period of collecting time series data, so let's send
a restart signal instead using the
kill command. First we find the
Prometheus server process id (PID) using one command, then apply it using the command as shown
below:
# Locate the Prometheus process id (PID).
#
$ ps aux | grep prometheus
erics 94110 1:31PM 0:05.28 ./prometheus --config.file=workshop-prometheus.yml
erics 97648 2:43PM 0:00.00 grep prometheus
# Send a restart signal to the PID we found.
#
$ kill -s HUP 94110
Intermezzo - Support helper configuration file
The previous Prometheus configuration changes were applied by you to your workshop file. In
case you want to return to this workshop at a later time, we provide a working adjusted
configuration file in the support directory, just start Prometheus as follows to start scraping
both Prometheus and the service demo endpoints:
$ ./prometheus --config.file=support/workshop-prometheus.yml
Services demo - Configuration applied
No matter which method you used to apply the new Prometheus configuration, validate it by
checking the log output, you should see something like this (example of using a signal to
restart):
...
ts=2024-05-29T12:57:28.154Z level=info msg="Loading configuration file" filename=workshop-prometheus.yml
ts=2024-05-29T12:57:28.227Z level=info msg="Completed loading of configuration file" filename=workshop-prometheus.yml
totalDuration=72.287041ms db_storage=3.042µs remote_storage=2.416µs web_handler=2.667µs
query_engine=3.791µs scrape=70.481917ms scrape_sd=53.75µs notify=6.834µs notify_sd=2.958µs
rules=4.5µs tracing=7.167µs
...
Services demo - Validating setup
The last check to make sure it's working, execute
demo_api_http_requests_in_progress
in the Prometheus console to generate a graph output something like the following (note, this has
been running awhile, hence the full graph):
Lab completed - Results
Next up, exploring basic queries...
Contact - are there any questions?
Eric D. Schabell
Director Evangelism
Contact: @ericschabell {@fosstodon.org) or https://www.schabell.org
Director Evangelism
Contact: @ericschabell {@fosstodon.org) or https://www.schabell.org