Service discovery - Setting up the targets
Our service discovery integration using file based discovery will need a file to watch, which
will be a targets.yml
file that needs to be located in the same Prometheus
instance directory holding your workshop-prometheus.yml
file. Edit this
target file to look like this (note for container installs we are using a podman variable, if
using source install then replace all configuration targets with
localhost:PORT_NUMBER
):
- targets: ["host.containers.internal:11111"]
- targets: ["host.containers.internal:22222"]
labels:
job: "services"
env: "production"
- targets: ["host.containers.internal:44444"]
labels:
job: "services"
env: "development"
Intermezzo - A word about Prometheus setup
The next step is configuring a Prometheus instance to make use of the file based discovery
targeting our newly created targets.yml
file. If you have been following the
previous workshop labs, you most likely have a heavily configured
workshop-prometheus.yml
file. In the next slide you'll see the minimum
setup you'll use for the rest of this lab. To achieve the same configuration, it's handy if you
just comment out any unused or unneeded parts of the configuration by commenting out lines using
#
(hash mark or pound sign).
Now let's get to work on that configuration...
Service discovery - Configuring file base discovery
Now we need to adjust the configuration of our Prometheus instance to use file based discovery
and watch the newly created targets.yml
file. To do that ensure the
workshop-prometheus.yml
file contains the following:
# workshop config
global:
scrape_interval: 5s
scrape_configs:
# Scraping Prometheus.
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
# File based discovery.
- job_name: "file-sd-workshop"
file_sd_configs:
- files:
- "targets.yml"
Service discovery - Adding targets to buildfile
Before you rebuild the container image, be sure to add the newly created
targets.yml
file to the buildfile as follows, then rebuild the container
image using this file:
FROM prom/prometheus:v2.54.1
ADD workshop-prometheus.yml /etc/prometheus
ADD targets.yml /etc/prometheus
Service discovery - Building a container image
Now you can build your own container image with our custom configuration inserted:
$ podman build -t workshop-prometheus:v2.54.1 -f Buildfile
STEP 1/3: FROM prom/prometheus:v2.54.1
Resolving "prom/prometheus" using unqualified-search
registries (/etc/containers/registries.conf.d/999-podman-machine.conf)
Trying to pull docker.io/prom/prometheus:v2.54.1...
Getting image source signatures
Copying blob sha256:54512e4fd08c47b3ca9a1a819b67275b22bf3a26be2d068369e330cc2dcaaa30
Copying blob sha256:ae2aea4a76d0bd645dd670e4146ffb8c6700e5cff84fffc00c57e0546b40e3fe
Copying blob sha256:9a4e2bd7c8b5eff5563de824d2eda1ab776820b49b8b3371404f796ff14bf5f2
Copying config sha256:dd202374baaf7882936f1edebed305ff0d1a7dfd9dfb6a09b198f550043d02d2
Writing manifest to image destination
Storing signatures
STEP 2/3: ADD workshop-prometheus.yml /etc/prometheus
STEP 3/3: ADD targets.yml /etc/prometheus
COMMIT workshop-prometheus
--> a6433a85d68f
Successfully tagged localhost/workshop-prometheus:v2.54.1
a6433a85d68faa925da7673a2bbb3bd4a9674f3dca5061d9ae34b5ab1b4d1a77
Service discovery - Starting Prometheus instance
To start Prometheus is depending on whether you've chosen to install from the source binary, or
run it in an open source container. Start using a command below that matches your choice:
$ ./prometheus --config.file=workshop-prometheus.yml
$ podman run -p 9090:9090 workshop-prometheus:v2.54.1 --config.file=/etc/prometheus/workshop-prometheus.yml
Service discovery - Validating file based discovery
After starting Prometheus, validate the scraping is working through the file based discovery by
opening the console at http://localhost:9090
and selecting from the menu
STATUS -> TARGETS
and finding the two production and one development
services environments:
Service discovery - Validating using PromQL
You can also validate that the three services environments are being scraped by using a PromQL
query UP{}
as follows:
Service discovery - Updating targets for discovery
Now we can update our targets.yml
file and have our Prometheus instance
discovering those changes dynamically. Let's promote our development services infrastructure to
STAGING by updating the env
label:
- targets:
- "host.containers.internal:11111"
- "host.containers.internal:22222"
labels:
job: "services"
env: "production"
- targets:
- "host.containers.internal:44444"
labels:
job: "services"
env: "staging"
Service discovery - Validating staging updates
After saving the targets.yml
file changes, you can query the services
targets using the same methods shown earlier (here with PromQL) to verify Prometheus picked
up the changes dynamically:
Service discovery - Individual exercise
Before you complete this lab, a final exercise to apply all you have learned.
Start a new instance of your services infrastructure and run it on an unused port on your machine
(localhost:33333 for example).
Add this new infrastructure target, give it a job label, and name the environment testing.
Service discovery - Validating testing updates
Verify the discovery is collecting testing metrics by querying the services targets using the same
methods shown earlier (here with PromQL):
Lab completed - Results
Next up, metrics monitoring at scale...
Contact - are there any questions?
Service discovery - Setting up the targets Our service discovery integration using file based discovery will need a file to watch, which
will be a targets.yml file that needs to be located in the same Prometheus
instance directory holding your workshop-prometheus.yml file. Edit this
target file to look like this (note for container installs we are using a podman variable, if
using source install then replace all configuration targets with localhost:PORT_NUMBER ): - target s: [ "host.containers.internal:11111" ]
- target s: [ "host.containers.internal:22222" ]
label s: jo b: "services" en v: "production" - target s: [ "host.containers.internal:44444" ]
label s: jo b: "services" en v: "development"