Lab 7 - Pipeline integration with OpenTelemetry
Lab Goal
This lab explores how to integrate a Fluent Bit pipeline with OpenTelemetry.
Intermezzo - Jumping to the solution
If you happen to be exploring Fluent Bit as an architect and want to jump to the solution in
action, we've included the configuration files in the easy install project from the source
install support directory, see the previous installing from source lab. Instead of creating all
the configurations as shown in this lab, you'll find them ready to use as shown below from the
fluentbit-install-demo root directory:
$ ls -l support/configs-lab-7/
-rw-r--r--@ 1 erics staff 166 Aug 1 15:34 Buildfile-fb
-rw-r--r-- 1 erics staff 215 Jul 27 09:47 Buildfile-otel
-rw-r--r-- 1 erics staff 73 Jul 25 13:51 Buildfile-prom
-rw-r--r-- 1 erics staff 1589 Aug 1 15:50 workshop-fb.yaml
-rw-r--r-- 1 erics staff 245 Jul 30 15:42 workshop-otel.yaml
-rw-r--r-- 1 erics staff 209 Jul 27 09:23 workshop-prom.yml
OTel integration - The use cases
There are many reasons why an organization might want to pass their telemetry pipeline output
onwards to OpenTelemetry (OTel), specifically through an
OpenTelemetry Collector.
To facilitate this integration, Fluent Bit from the release of version 3.1 added support for
converting its telemetry data into the OTel format.
In this lab, we'll explore what Fluent Bit offers to convert pipeline data into the correct format for sending onwards to OTel.
In this lab, we'll explore what Fluent Bit offers to convert pipeline data into the correct format for sending onwards to OTel.
OTel integration - The architecture
This is an architectural overview of what we are building. Here you can see the pipeline
collecting log events, processing them into an OTel Envelope to satisfy the OTel schema
(resource, scope, attributes), pushing to an OTel Collector, and finally to both the collectors
console output and to a separate log file:
OTel integration - Initial log configuration
We begin configuration of our telemetry pipeline in the
INPUT phase with a
simple dummy plugin generating sample log events as follows:
# This file is our workshop Fluent Bit configuration.
#
service:
flush: 1
log_level: info
pipeline:
# This entry generates a success message for the workshop.
inputs:
- name: dummy
dummy: '{"service" : "backend", "log_entry" : "Generating a 200 success code."}'
...
OTel integration - Initial configuration outputs
Now ensure the output section of our configuration file
workshop-fb.yaml
following the inputs section is as follows:
# This entry directs all tags (it matches any we encounter)
# to print to standard output, which is our console.
#
outputs:
- name: stdout
match: '*'
format: json_lines
OTel integration - Running initial pipeline (source)
To see if our configuration works we can test run it with our Fluent Bit installation. Depending
on the chosen install method, here we show how to run it using the source installation followed
by the container version. Below the source install is shown from the directory we created to hold
all our configuration files:
# source install.
#
$ [PATH_TO]/fluent-bit --config=workshop-fb.yaml
OTel integration - Console output initial pipeline (source)
The console output should look something like this. This runs until exiting with CTRL_C:
...
[2024/08/01 15:41:55] [ info] [input:dummy:dummy.0] initializing
[2024/08/01 15:41:55] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/08/01 15:41:55] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/01 15:41:55] [ info] [sp] stream processor started
{"date":1722519715.919221,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519716.252186,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519716.583481,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519716.917044,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519717.250669,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519717.584412,"service":"backend","log_entry":"Generating a 200 success code."}
...
OTel integration - Testing initial pipeline (container)
Let's now try testing our configuration by running it using a container image. First thing that
is needed is to open in our favorite editor, a new file called
Buildfile-fb. This is going to be used to build a new container image
and insert our configuration. Note this file needs to be in the same directory as our
configuration file, otherwise adjust the file path names:
FROM cr.fluentbit.io/fluent/fluent-bit:3.1.4
COPY ./workshop-fb.yaml /fluent-bit/etc/workshop-fb.yaml
CMD [ "fluent-bit", "-c", "/fluent-bit/etc/workshop-fb.yaml"]
OTel integration - Building initial pipeline (container)
Now we'll build a new container image, naming it with a version tag, as follows using the
Buildfile-fb and assuming you are in the same directory:
$ podman build -t workshop-fb:v12 -f Buildfile-fb
STEP 1/3: FROM cr.fluentbit.io/fluent/fluent-bit:3.1.4
STEP 2/3: COPY ./workshop-fb.yaml /fluent-bit/etc/workshop-fb.yaml
--> b4ed3356a842
STEP 3/3: CMD [ "fluent-bit", "-c", "/fluent-bit/etc/workshop-fb.yaml"]
COMMIT workshop-fb:v4
--> bcd69f8a85a0
Successfully tagged localhost/workshop-fb:v12
bcd69f8a85a024ac39604013bdf847131ddb06b1827aae91812b57479009e79a
OTel integration - Running initial pipeline (container)
Run the initial pipeline using this container command:
$ podman run --rm workshop-fb:v12
OTel integration - Console output initial pipeline (container)
The console output should look something like this. This runs until exiting with CTRL_C:
...
[2024/08/01 15:41:55] [ info] [input:dummy:dummy.0] initializing
[2024/08/01 15:41:55] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/08/01 15:41:55] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/01 15:41:55] [ info] [sp] stream processor started
{"date":1722519715.919221,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519716.252186,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519716.583481,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519716.917044,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519717.250669,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":1722519717.584412,"service":"backend","log_entry":"Generating a 200 success code."}
...
OTel integration - OpenTelemetry envelope
Within the pipeline after ingesting log events, we need to process them into an OTel compatible
schema before we can push them onwards to an OTel Collector. Fluent Bit v3.1+ offers a processor
to put our logs into an OTel envelope.
The OTel Envelope is putting our log events into the correct format (
The OTel Envelope is putting our log events into the correct format (
resource,
scope, attributes). Let's try adding this to our
workshop-fb.yaml file and explore the changes to our log events...
OTel integration - Processing the envelope
The configuration of our
processor section needs an entry for logs. Fluent
Bit provides a simple processor called opentelemetry_envelope and is added
as follows after the inputs section:
...
pipeline:
# This entry generates an success message for the workshop.
inputs:
- name: dummy
dummy: '{"service" : "backend", "log_entry" : "Generating a 200 success code."}'
processors:
logs:
- name: opentelemetry_envelope
...
OTel integration - Testing envelope pipeline (source)
To see if our configuration works we can test run it with our Fluent Bit installation. Depending
on the chosen install method, here we show how to run it using the source installation followed
by the container version. Below the source install is shown from the directory we created to hold
all our configuration files:
# source install.
#
$ [PATH_TO]/fluent-bit --config=workshop-fb.yaml
OTel integration - Console output envelope pipeline (source)
The console output should look something like this. Note each log event has now been put into
the OTel Envelope format, providing for a
resource, scope,
and attributes. This runs until exiting with CTRL_C:
...
[2024/08/01 16:26:52] [ info] [input:dummy:dummy.0] initializing
[2024/08/01 16:26:52] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/08/01 16:26:52] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/01 16:26:52] [ info] [sp] stream processor started
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722522413.113665,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":4294967294.0}
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722522414.113558,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":4294967294.0}
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722522415.11368,"service":"backend","log_entry":"Generating a 200 success code."}
...
OTel integration - Building envelope pipeline (container)
Now we'll build a new container image, naming it with a version tag, as follows using the
Buildfile-fb and assuming you are in the same directory:
$ podman build -t workshop-fb:v13 -f Buildfile-fb
STEP 1/3: FROM cr.fluentbit.io/fluent/fluent-bit:3.1.4
STEP 2/3: COPY ./workshop-fb.yaml /fluent-bit/etc/workshop-fb.yaml
--> b4ed3356a842
STEP 3/3: CMD [ "fluent-bit", "-c", "/fluent-bit/etc/workshop-fb.yaml"]
COMMIT workshop-fb:v4
--> bcd69f8a85a0
Successfully tagged localhost/workshop-fb:v13
bcd69f8a85a024ac39604013bdf847131ddb06b1827aae91812b57479009e79a
OTel integration - Running envelope pipeline (container)
Run the initial pipeline using this container command:
$ podman run --rm workshop-fb:v13
OTel integration - Console output initial pipeline (container)
The console output should look something like this. Note each log event has now been put into
the OTel Envelope format, providing for a
resource, scope,
and attributes. This runs until exiting with CTRL_C:
...
[2024/08/01 16:26:52] [ info] [input:dummy:dummy.0] initializing
[2024/08/01 16:26:52] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/08/01 16:26:52] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/01 16:26:52] [ info] [sp] stream processor started
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722522413.113665,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":4294967294.0}
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722522414.113558,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":4294967294.0}
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722522415.11368,"service":"backend","log_entry":"Generating a 200 success code."}
...
OTel integration - Inside the architecture
The work up to now has been focused on our telemetry pipeline with Fluent Bit. We configured log
event ingestion, processing to an OTel Envelope, and it's ready to push onwards as shown here in
this architectural component:
OTel integration - Pushing to the collector
When sending telemetry data in the OTel Envelope we are using the OpenTelemetry Protocol (OTLP),
which is the standard to ingest into a collector. The next step will be for us to configure an
OTel Collector to receive telemetry data and point our Fluent Bit instances at that collector as
depicted in this diagram:
OTel integration - Options installing OTel Collector
The rest of this workshop can be done using the path of source installations or container
installations of the OTel Collector, so please click on the option you want to use for
the rest of this workshop: