Services demo - Container installation notes

This lab guides you through installing the services demo using Podman, an open source container tooling set that works on Mac OSX, Linux, and Windows systems. This workshop assumes you have a working Podman installation from earlier labs in this workshop.

Note: if you need to install Podman, see the earlier lab for installing Prometheus in a container and return here when done.

Services demo - Downloading the project

To install the services demo in a container we'll need the source project to build our container. We're providing a self contained project you can download, unzip, and configure to build a container:

Click here to download the Prometheus Services Demo Installer.

Services demo - Unzipping the project

Once you have downloaded the project using the link provided previously, just unzip into its own directory as shown (Note: examples shown are on Mac OSX system):
unzip services

Services demo - Building a container image

Now you can build your own container image with the provided build file (output has been abbreviated to save space): 
							
								$ podman build -t prometheus_services_demo:v1 -f installs/demo/Buildfile

								[1/2] STEP 1/5: FROM golang:1.17-alpine AS builder
								[1/2] STEP 2/5: WORKDIR /source
								[1/2] STEP 3/5: COPY . /source
								[1/2] STEP 4/5: RUN go mod download
								[1/2] STEP 5/5: RUN go build -v -o prometheus_demo_service .
								[2/2] STEP 1/4: FROM alpine:3
								[2/2] STEP 2/4: COPY --from=builder /source/prometheus_demo_service  /bin/prometheus_demo_service
								[2/2] STEP 3/4: EXPOSE     8080
								[2/2] STEP 4/4: ENTRYPOINT [ "/bin/prometheus_demo_service" ]
								[2/2] COMMIT prometheus_services_demo:v1
								Successfully tagged localhost/prometheus_services_demo:v1
								f0f7afad800b643a40938e04c1dd2d6c394fb6c0197929e3db40df2e425a8127

Services demo - Verifying built image

Looking up our built images to verify everything went well and we should see something like this: 
							
								$ podman images

								REPOSITORY                          TAG          IMAGE ID      CREATED        SIZE
								localhost/prometheus_services_demo  v1           231a2e5431b4  4 seconds ago  20.3 MB
								docker.io/library/alpine            3            f6648c04cd6c  2 days ago     7.95 MB
								docker.io/library/golang            1.17-alpine  0ebb35c98346  12 months ago  321 MB

Services demo - Start your demo engines!

Now it's time to start the services demo in a detached container mapping our port 8080 to the containers exposed port 8080: 
							
								$ podman run -d -p 8080:8080 prometheus_services_demo:v1

								e924f60535bf8fa1b80a7dc2f18574223f0cb44468362fc2b4a78471af5aafbc

Services demo - Testing the metrics

After starting the services binary as shown in the terminal output, it should have metrics scraping end points available so that you can query them with PromQL. Test this at http://localhost:8080/metrics:
services metrics

Prometheus - Adjusting settings live

Using this container image we built means any changes you need to make to the configuration that were in the flags used to start the server will require a new image be built, the old container stopped, and the new container started using the new image.

Now let's find out the actual IP address of our container on our local machine so that we can update the static prometheus scrap configuration to start collecting data from our services demo.

Services demo - Finding our container IP address

Run the following command (or something like it on your machine) to determine the IP address we need to update our Prometheus configuration file: 
							
								$ ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}'

								192.168.1.4
In our case the container IP is 192.168.1.4 and we are scraping port 8080 for it's metrics, so let's add this to our Prometheus configuration file.

Services demo - Prometheus configuration updates

While the metrics are exposed, they will not be scraped by Prometheus until you have updated its configuration to add this new end point. Let's update our workshop-prometheus.yml file to add the services job as shown along with comments for clarity (noting that the IP address for the services demo is the IP we discovered in the previous slide): 
							
								scrape_configs:

									# Scraping Prometheus.
									- job_name: "prometheus"
									static_configs:
										- targets: ["localhost:9090"]

									# Scraping services demo.
									- job_name: "services"
									static_configs:
										targets: ["192.168.1.4:8080"]

Services demo - Rebuilding a new Prometheus image

Now you can rebuild your Prometheus container image with our new configuration inserted: 
							
								$ podman build -t workshop-prometheus -f Buildfile

								STEP 1/2: FROM prom/prometheus:v2.46.0
								STEP 2/2: ADD workshop-prometheus.yml /etc/prometheus
								COMMIT workshop-prometheus
								--> b63d3b6d2139
								Successfully tagged localhost/workshop-prometheus:latest
								b63d3b6d2139c3a28eeab4b8d65169a1b4d77da503c51a587340e0a1b0a52b8a

Services demo - Restart your Prometheus image

First you stop the running container, then start it again as we did before, but this time the newest rebuilt image will be used:

$ podman run -p 9090:9090 workshop-prometheus --config.file=/etc/prometheus/workshop-prometheus.yml
starting

Services demo - Validating setup

The last check to make sure it's working, execute demo_api_http_requests_in_progress in the Prometheus console to generate a graph output something like the following (note, this has been running awhile, hence the full graph):
services metrics

Services demo - Starting multiple instances

The output is more interesting if you have multiple instances of the services demo running. This is pretty easy by just starting a new container with different port mapping. Let's start a second services demo in a new container and map our local machine port 8081 (or any other open port you have) to the container port 8080 as follows: 
							
								$ podman run -d -p 8081:8080 prometheus_services_demo:v1

								e924f60535bf8fa1b80a7dc2f18574223f0cb44468362fc2b4a78471af5aazzz

Services demo - Testing the metrics

After starting the services binary as shown in the terminal output, it should have metrics scraping end points available so that you can query them with PromQL. Test this at http://localhost:8081/metrics:
services metrics

Services demo - Prometheus configuration updates

Now we need to update our workshop-prometheus.yml file to add the new services demo instance (noting that the IP address for the new services demo is the same IP we discovered earlier): 
							
								scrape_configs:

									# Scraping Prometheus.
									- job_name: "prometheus"
									static_configs:
										- targets: ["localhost:9090"]

									# Scraping services demo.
									- job_name: "services"
									static_configs:
										targets: ["192.168.1.4:8080"]
										targets: ["192.168.1.4:8081"]

Services demo - Rebuilding a new Prometheus image

Now you can rebuild your Prometheus container image with our new configuration inserted: 
							
								$ podman build -t workshop-prometheus -f Buildfile

								STEP 1/2: FROM prom/prometheus:v2.46.0
								STEP 2/2: ADD workshop-prometheus.yml /etc/prometheus
								COMMIT workshop-prometheus
								--> b63d3b6d2139
								Successfully tagged localhost/workshop-prometheus:latest
								b63d3b6d2139c3a28eeab4b8d65169a1b4d77da503c51a587340e0a1b0a52b8a

Services demo - Restart your Prometheus image

First you stop the running container, then start it again as we did before, but this time the newest rebuilt image will be used:

$ podman run -p 9090:9090 workshop-prometheus --config.file=/etc/prometheus/workshop-prometheus.yml
starting

Services demo - Validating setup

The last check makes sure it's working by executing up{job=services} in the Prometheus console validating both instances are being scraped:
services up

Services demo - Removing second services demo instance

The final exercise is for you to stop the second services demo container, remove the target for 8081 from your Prometheus configuration, stop the Prometheus container, rebuild the Prometheus container with the updated configuration, and restart a Prometheus container that is only scraping Prometheus and one services demo on port 8080.

Lab completed - Results

completed
Next up, exploring basic queries...
references

Contact - are there any questions?

Eric D. Schabell
Director Evangelism
Contact: @ericschabell {@fosstodon.org) or https://www.schabell.org

Up next in workshop...

Lab 4 - Exploring Basic Queries