Eric D. Schabell

Mastering Fluent Bit: Beginners Guide for Contributing to our CNCF Project Docs

eric@schabell.org (eric@schabell.org) — Fri, 15 May 2026 07:00:00 +0200

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

Each article in this series addresses a single topic by providing insights into what the topic is, why we are interested in exploring that topic, where to get started with the topic, and how to get hands-on with learning about the topic as it relates to the Fluent Bit project.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, contributing to the Fluent Bit docs project. This is a follow-up to our previous article on contributing to the Fluent Bit project website, and this time we go a step further by tackling documentation contributions. If we can find something undocumented, clarify something confusing, or fix a gap between what the code does and what the docs say — that is a genuine contribution that the community notices.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Contributing to the Fluent Bit docs project?

Before diving into the hands-on steps, let's understand why contributing to the Fluent Bit docs matters and why it's a great fit for anyone new to contributing to a CNCF project.

The Fluent Bit documentation is the first place developers go when they are trying to figure out how to configure an input plugin, wire up an output, or understand a pipeline behavior. Gaps in that documentation have a real cost — people waste time, open issues that aren't actually bugs, or simply give up. When you fix a doc, you are helping every developer who comes after you.

Contributing docs also gives you a natural reason to dig into the code. You often end up reading the actual source to understand what a plugin really does before you write about it. That is how you start building the intuition that eventually gets you contributing code too. But for now, let's start with documentation and get our first PR in.

Where to get started

This time we have two repositories we need to work on the docs project. The docs project does not live in the same place as the code project, and both are relevant to any fixes we are making as we want to verify everything against the existing code.

The first project is the code project found at fluent/fluent-bit, the upstream canonical source code repository, and we need to fork it to our own username/fluent-bit-fork. Easy enough to do using the GitHub UI. We reference the code project when we need to understand what a feature actually does in practice before we document it. Note, as with our previous article, the fork has been renamed with the -fork suffix to make it easy to identify at a glance. We clone this fork to our location machine.

# Fork the original code project using GitHub.

# Check out the fork locally, here using my fork as an example.

$ git clone git@github.com:eschabell/fluent-bit-fork.git

# Add the upstream website repo.

$ git remote add upstream https://github.com/fluent/fluent-bit.git

The second project is the docs project found at fluent/fluent-bit-docs, the upstream documentation repository where all the actual docs content lives, and we need to fork this to our own username/fluent-bit-docs-fork. This is where our documentation changes will land.

# Fork the original docs project using GitHub.

# Check out the fork locally, here using my fork as an example.

$ git clone git@github.com:eschabell/fluent-bit-docs-fork.git

# Add the upstream website repo.

$ git remote add upstream https://github.com/fluent/fluent-bit-docs.git

So now we've set up both locally, forking each repository on GitHub, then cloning and adding the upstream remote for each, just as we did in the previous article.

This is a habit worth building from day one — always sync from upstream before starting any new contribution, This prevents the diverged branch headache that will slow down our pull request later, see below for my example:

# Fetch any upstream work done by others.

$ git fetch upstream

# Sync local fork with the upstream changes.

$ git rebase upstream/master

# Last step, push to your fork's repository.

$ git push

Now we are ready to get started with our first change to the Fluent Bit docs.

Finding a fix that is needed

There are two natural ways to find something worth fixing. The first is the GitHub issues list on the fluent-bit-docs repository. Look for issues tagged with documentation, help wanted, or good first issue. These are explicitly flagged by maintainers as accessible starting points. I'll be honest with you though, we don't have a lot of those. We tend to keep our issues list on the docs site to a minimum.

The second, and frankly more satisfying, is to discover something yourself. Again, being honest with you, using AI to assist you in this is making this path easier than you might think. Either way, when exploring a plugin page and notice a spelling mistake, a missing configuration option, or just wanting to add an example configuration you've used yourself are all valid ways to get started.

Maybe you try to follow an example and find it no longer works with the current version. Sometimes you can discover a mismatch between what the docs describe and what the code in your fluent-bit-fork actually does (or your AI explorations reveal). That kind of gap is a real contribution waiting to happen. Cross-referencing the docs against the code is exactly the kind of work maintainers appreciate and rarely have enough time to do themselves.

Either way, a nice-to-have habit worth building: open a GitHub issue on fluent-bit-docs before starting the work. Describe what you found and what you plan to fix. It takes just a few minutes, gives maintainers visibility, allows you to ask any questions you might have, and means your PR can reference fixing a specific issue number when you submit it.

Start with a branch

Once you have found something to fix, it should be noted that the common practice is to never work directly on our main branch. Every change, no matter how small, gets its own branch. The branch name should be descriptive enough to focus on the changes you are making. I personally always use my name to preface all branches as it makes it easier to find in larger listings of branches for bigger projects.

Below is an example of a real branch I was working on recently in the Fluent Bit docs project.

# Starting in our forked master.

$ git checkout -b erics_processors_tda_fixes

Switched to a new branch 'erics_processors_tda_fixes'

Note this is also listing the path the the file I'm working on during this fix. Another standard way of working that I like to apply.

Now make the actual documentation changes. The fluent-bit-docs repository uses Markdown files organized by topic area. Find the relevant file, make your edit, and use the structure and conventions you see in the surrounding content as your guide. The project has a specific style and voice — mirror it rather than going off on your own. Keep the change focused. One file per commit (and if possible, per PR) is a better habit than bundling five things together, especially when you are starting out.

Testing documentation changes

The docs project uses Vale for prose linting, which checks style consistency and catches common documentation issues before they reach the reviewers. Run it locally before committing so you are not finding out about style violations in CI after the PR is already open.

The following shows a run that basically passes, but has a few suggestions mentioned. It's common to find them outside the. applied changes, and if you do, it's not a bad habit to try and fix them with the changes you are making to this file.

# Testing docs changes uses Vale to parse against our project rules.

$ vale pipeline/processors/tda.md

pipeline/processors/tda.md
 65:1   suggestion  Spelling check: 'x_t'?                                                  FluentBit.Spelling
 65:12  suggestion  Spelling check: 'x_t'?                                                  FluentBit.Spelling
 91:27  suggestion  Spelling check: 'x_i'?                                                  FluentBit.Spelling
 94:18  suggestion  Spelling check: 'x_i'?                                                  FluentBit.Spelling
 94:24  suggestion  Spelling check: 'x_j'?                                                  FluentBit.Spelling
 156:4  suggestion  'Interpreting Betti numbers' should use sentence-style capitalization.  FluentBit.Headings

✔ 0 errors, 0 warnings and 6 suggestions in 1 file.

# Once you have completed all fixes, the Vale test should run as follows.

$ vale pipeline/processors/tda.md

✔ 0 errors, 0 warnings and 0 suggestions in 1 file.

This is the docs equivalent of making sure the unit tests pass before you commit code — it shows respect for the reviewers' time and signals that you know the contribution process.

Committing changes

Before committing, take a moment to compose a proper commit message. This is not optional and it is not a formality. A good commit message tells the story of what changed and why, and it is the permanent record you are attaching your name to.

The format to follow: a short subject line (under 72 characters) describing the change, a blank line, then a bullet list in the body covering specifically what was modified and the reason behind each change. Note that a good habit is to indicate the file change location in the first line as shown.

# A good commit message.

docs: pipeline: processors: tda: doc validation fixes

  - Fix sentence-case heading (TDA title)
  - Merge hard-wrapped paragraph lines for GitBook rendering
  - Add LaTeX math delimiters to R^{mD} notation
  - Fix MD060 table column alignment in parameters and metrics tables
  - Sort configuration parameters table alphabetically.

  Fixes issue #2999.

The last line in the commit is used to point to the issue that this change is addressing. If you tag your commit message with the issue number, it will auto-close the issue on the merge of this docs PR into the master branch. The following is the workflow for adding, committing, and pushing your changes.

# When ready to submit our changes.

$ git add pipeline/processors/tda.md

# Commit the changes using a signed commit (assumes GPG set up).

$ git commit -S

# Push the changes to our repository

$ git push --set-upstream origin erics_processors_tda_fixes

Now we open a pull request against fluent/fluent-bit-docs from the GitHub UI. In the pull request description it should automatically fill with your commit message details. Be sure to explicitly request a review — do not assume the PR will be picked up automatically, and make sure to tag a reviewer. If for some reason it's not possible to request a review through the UI, then feel free to post a comment after submitting the PR and ask me (@eschabell) to review as I'm always happy to help.

AI good habits for contributors

This section is worth paying close attention to, because AI tooling is now part of many developers' daily workflow and if used carelessly in an open source context it can create real problems and broken trust with core project maintainers.

Here are my personal ground rules for working with AI assistance on open source projects and how I work with Fluent Bit projects.

Use your local fork filesystem

Configure your AI tool to work against your local fork checkouts — not a downloaded copy to /tmp or any other ephemeral scratch directory. This is important because your working directory is already version-controlled. Every change the AI proposes is immediately visible via git diff, which means you always know exactly what changed before you decide to commit anything. It also saves on token usage and bandwidth speeding up the AI results to your queries.

Never let AI modify without your approval

I always set a personal rule: no line of code or documentation changes without your explicit review and approval, line by line. AI tools should propose changes and you accept, reject, or modify them. This is not just good open source hygiene — it is how you learn the codebase and the project's conventions. The Fluent Bit docs and website have a specific voice and structure. Furthermore, you are putting your name (signing the commits) on any changes you are pushing, so you might want to make sure you agree with each line that is being modified in your name.

Never let AI touch git

This one is non-negotiable for me in git interactions with my upstream repositories. AI does not commit, does not push, does not fork, and does not open pull requests in my inner developer loop. I do all of that manually. Commits are attribution. When you sign a commit with your name and email, you are asserting that you wrote or have the right to submit that content. In a CNCF project operating under a DCO (Developer Certificate of Origin), this is a legal and community trust matter, not a formality. Keep your hands on the wheel for all git operations and you will also understand the processes and retain your skills.

Check for tests when adding new code

This one is more for code based repositories, but good to know as background information here. If your contribution goes beyond a documentation change and into actual code — a plugin, a configuration example, a script — check whether the Fluent Bit fork has existing test patterns for that area. Follow them. If you are adding testable behavior, add tests. Ask in the PR or issue if you are unsure what test coverage is expected. Maintainers would rather answer that question up front than request changes after review.

Always provide a proper commit message

Every commit should have a clear, structured message. At minimum: a short subject line describing what changed, followed by a list in the body describing why and what specifically was modified. It must be signed or it will fail on DCO sign-off in the CI/CD process. Check the contributing guide for the expected standard. A good commit message is also your own paper trail — if a maintainer asks why you changed something, your commit history should answer the question. Remember, you are signing this, not your AI tooling.

Nice to have: open an issue before a PR

For anything beyond a trivially obvious contribution, the best practice is to open a GitHub issue first. Describe what you want to write or fix, get a signal from the maintainers that it is welcome, then do the work and open the PR referencing that you are fixing that issue.

More in the series

In this article we explored step-by-step what it takes to make our first documentation contribution to the Fluent Bit docs project — from setting up our fork, testing Vale locally, and submitting a pull request. Finally we tried to help with establishing good habits around AI tooling along the way.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Get started with Fluent Bit today using this online free workshop.

Mastering Fluent Bit: Beginners Guide for Contributing to our CNCF Project Website

eric@schabell.org (eric@schabell.org) — Wed, 6 May 2026 07:00:00 +0200

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, contributing to the Fluent Bit project website. This article will be a hands-on exploration of how to get started contributing blog articles to the Fluent Bit project website, something that is very accessible to newcomers and a great way to become part of the community.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Contributing to the Fluent Bit website?

Before diving into the hands-on steps, let's understand why contributing to the Fluent Bit website matters and why it's a great starting point for those of us new to contributing to a CNCF project.

The Fluent Bit project website is where the community shares knowledge, tutorials, and updates with the world. Contributing a blog article means you are directly adding value to the project — helping other developers learn, discover use cases, and get hands-on experience with Fluent Bit telemetry pipelines. No deep knowledge of C is required, no need to wrestle with complex build pipelines. Just ideas, a bit of Hugo familiarity, and a willingness to follow the contribution process.

As a CNCF graduated project, Fluent Bit has an active and welcoming community. Getting your name into the contributor history by writing a blog article is a genuinely meaningful step into open source participation.

Where to get started

The Fluent Bit website lives at fluent/fluent-bit-website on GitHub. Before we touch a single line, there are two repositories we need to understand and keep straight in our heads throughout this process.

The first is fluent/fluent-bit-website, the upstream canonical repository owned by the Fluent community. We never push directly to this one.

The second is your-username/fluent-bit-website-fork, a personal fork of the repository where all our work happens before it goes upstream via a pull request. Note it's been renamed to add the -fork to the repository name, a standard practice to easily identify forked projects.

Forking the repository on GitHub, then we clone our fork locally, and finally, we add the upstream remote so we can always sync our local copy with what the community is doing directly from our command line.

# Fork the original website using GitHub.

# Check out the fork locally, here using my fork as an example.

$ git clone git@github.com:eschabell/fluent-bit-website-fork.git

# Add the upstream website repo.

$ git remote add upstream https://github.com/fluent/fluent-bit-website.git

It's a habit worth building from day one — always sync from upstream before starting any new contribution, This prevents the diverged branch headache that will slow down our pull request later, see below for my example:

# Fetch any upstream work done by others.

$ git fetch upstream

# Sync local fork with the upstream changes.

$ git rebase upstream/main

# Last step, push to your fork's repository.

$ git push

With our fork cloned, verify the site builds locally. The Fluent Bit website uses Hugo, so install it (an exercise left to the reader), then run the Hugo server and open a browser to confirm the site renders before making any changes.

# Start your local copy of the website on http://localhost:9999

$ hugo server -D -p 9999

...

Watching for config changes in .../fluent-bit-website-fork/config.toml
Start building sites …
hugo v0.161.1
                  │ EN
──────────────────┼─────
 Pages            │ 337
 Paginator pages  │   0
 Non-page files   │   0
 Static files     │ 315
 Processed images │   0
 Aliases          │   1
 Cleaned          │   0

Built in 319 ms
Environment: "development"
Serving pages from disk
Running in Fast Render Mode.
Web Server is available at //localhost:9999/ (bind address 127.0.0.1)
Press Ctrl+C to stop

...

If all goes well we should see this on http://localhost:9999 on your local machine:

Now we are ready to get started with our first change to the Fluent Bit website, maybe submitting an article?

How to contribute our first article

The Fluent Bit website is a Hugo site, which means every blog article is a Markdown file with a YAML front matter block at the top. This header is not optional — Hugo uses it to generate the page metadata, listing views, author information, dates, and tags. Getting the front matter right is the first practical task. It covers fields like title, date, author, tags, and any project-specific fields the site uses.

This is a sample from the Fluent Bit latest release announcement blog, found under content/announcements/v5.0/:

---

title: 'v5.0.6'
description: 'Next generation Telemetry Agent for Logs, Metrics and Traces.'
url: "/announcements/v5.0.6/"
release_date: 2026-05-21
publishdate: 2026-05-21
ver: v5.0.6
herobg: "/images/hero@2x.jpg"
latestVer: true
---

The fastest way to get this right is not to try and write it from memory. Instead, we open the blog content directory (content/posts/) in our local fork checkout and find a recently merged article to use as your reference. Mirror its file naming convention, the directory placement, and the front matter fields. Read a few existing articles to understand the tone and content that fits the Fluent Bit blog — tutorials, integration guides, project updates, and community spotlights are all examples of what works well there.

Writing our article

With front matter understood, we need to write our article in a Markdown file locally and use the Hugo server to preview it as we work. The audience is the Fluent Bit community — developers, operators, and platform engineers — so assume technical literacy but do not assume deep Fluent Bit expertise, especially for introductory topics.

Submitting our changes

Once the article is written and previewed correctly, here is the process to follow to submit your changes to the website project for maintainer review. Create a new branch off the synced fork — we never want to work directly on our main branch. Commit your new file with a clear, descriptive commit message. Push to your fork as shown below.

# Work on a branch.

$ git checkout -b erics_my_new_article

# When ready to submit our changes.

$ git add content/posts/my-new-fb-article.md

# Commit the changes using a signed commit (assumes GPG set up).

$ git commit -S

# Push the changes to our repository

$ git push --set-upstream origin erics_my_new_article

Now we open a pull request against fluent/fluent-bit-website from the GitHub UI. In the pull request description, explain what the article covers and why it is a good fit for the blog. Then explicitly request a review — do not assume the PR will be picked up automatically, and make sure to tag a reviewer. If for some reason it's not possible to request a review through the UI, then feel free to post a comment after submitting the PR and ask me (@eschabell) to review as I'm always happy to help.

AI good habits for contributors

Here are my personal ground rules for working with AI assistance on open source projects and how I work with Fluent Bit projects.

Use your local fork filesystem

Never let AI modify without your approval

I always set a personal rule: no line of code or documentation changes without your explicit review and approval, line by line. AI tools should propose changes and you accept, reject, or modify them. This is not just good open source hygiene — it is how you learn the codebase and the project's conventions. The Fluent Bit docs and website have a specific voice and structure. Furthermore, you are putting your name (signing the commits) on any changes you are pushing, so you might want to make real sure you agree with each line that is being modified in your name.

Never let AI touch git

Check for tests when adding new code

This one is more for code based repositories, but good to know as background information here. If your contribution goes beyond a blog article and into actual code — a plugin, a configuration example, a script — check whether the Fluent Bit fork has existing test patterns for that area. Follow them. If you are adding testable behavior, add tests. Ask in the PR or issue if you are unsure what test coverage is expected. Maintainers would rather answer that question up front than request changes after review.

Always provide a proper commit message

Nice to have: open an issue before a PR

This is more for code repositories than for the website project, but good background information. For anything beyond a trivially obvious contribution, the best practice is to open a GitHub issue first. Describe what you want to write or fix, get a signal from the maintainers that it is welcome, then do the work and open the PR referencing that you are fixing that issue.

More in the series

In this article we explored step-by-step what it takes to make our first article contribution to the Fluent Bit project website — from setting up our fork and getting Hugo running locally, to configuring a blog article and submitting a pull request. Finally we tried to help with establishing good habits around AI tooling along the way.

A New Chapter: Joining SUSE as Technical Advocacy Lead

eric@schabell.org (eric@schabell.org) — Fri, 1 May 2026 07:00:00 +0200

There are moments in your career where the path forward becomes surprisingly clear. After years of building, shipping, and advocating in the open source and cloud native ecosystem, I find myself at one of those moments.

Today I'm excited to share that I've joined SUSE as Technical Advocacy Lead — and I couldn't be more energized about what lies ahead.

Let me back up a bit, because getting here is a story worth telling.

The Chronosphere Chapter Closes

Most of you who follow this blog know that for the past few years I've been doing what I love most: building technical advocacy teams, creating hands-on content, and helping developers navigate the world of observability. My time at Chronosphere was a whirlwind of Fluent Bit, Perses, OpenTelemetry, Prometheus, and more conference stages than I can count. We built something genuinely good there — a team that cared deeply about community, about making the hard stuff accessible, and about doing the work rather than just talking about it.

Then came the acquisition by Palo Alto Networks (PANW), and with it the kind of organizational pivot that signals a company's priorities are shifting. PANW is a security company first, and that changes everything about how a technical advocacy function fits into the picture. There was nothing wrong with the outcome — acquisitions are part of the tech landscape — but for me it meant it was time to start looking at what came next.

That's a different kind of crossroads than most people talk about. It's not about being unhappy or burned out. It's about knowing what kind of work you do best, where you add real value, and having the self-awareness to admit when the environment no longer fits. I've been fortunate enough in this career to know the difference.

Taking the Break (And Actually Taking It)

Before jumping into anything new, I made a deliberate choice to step back for three months to look for something new I really wanted to do. I want to be honest about this: I did not treat that break the way a lot of people treat them. I stopped measuring my days by travel (last year I was on the road for +140 days), conference submissions, workshops, and article output. I worked on the open source project Fluent Bit. I played a lot of golf. I went back to some of the things that remind me why I got into this world in the first place while researching possible new adventures.

When you've been running at full speed for years — speaking, publishing, socializing, building hands-on workshops, evangelizing across communities — you can lose track of what fuels you versus what drains you. That break gave me the reset I didn't know I needed. I came back genuinely curious again. Hungry for the right reasons.

Three months sounds like a long time when you're a person who typically has six blog posts scheduled in advance. Trust me, it went fast, and I'd recommend it to anyone navigating a transition.

Why SUSE?

When I started looking at what was next, I had a simple filter: I wanted to be somewhere with deep open source roots, genuine community investment, and a team I already trusted. SUSE checked all three boxes in a way that made the decision feel less like a job search and more like coming home.

The open source credentials speak for themselves. SUSE has been at the center of the Linux and enterprise open source world for decades. This is not a company that added an open source strategy as an afterthought or a marketing play. It runs all the way through the culture. And for someone who has spent the better part of their career operating at the intersection of community and technology, that matters more than most people realize.

But honestly? What tipped the scales was the people.

When I started having conversations with the team, I kept running into old friends. People I'd shared stages with, collaborated on projects with, and grabbed beers with at KubeCon hallway tracks over the years. The open source world is small in the best possible way — you get to know who is real and who is performing, who does the work and who just talks about it. The people at SUSE who I already knew were always the ones doing the work. That is not a coincidence; it says something about the culture that attracts and keeps them.

It felt less like a job interview and more like a reunion of friends hanging out in a pub.

The Role: Technical Advocacy Lead

The title is Technical Advocacy Lead, and I'll tell you what that means to me in practice.

I'll be working with a team of very senior technology advocates — people who have deep expertise, strong opinions about how things should be done, and the kind of track record that means they don't need to be told what good looks like. My job, something I'm always trying to insist on, is player-coaching. I always try to be more of a guide while remaining hands-on in the open source technologies that we all know and love. There is always so much to do and learn, new technology arriving on the scene as old ones depart. What's not to love about learning and sharing with others?

I've led advocacy teams before, and what I've learned is that the most important thing you can do for a team of strong individual contributors is get out of their way — strategically. Guiding the direction, building the frameworks for their production, and removing the friction. Let the experts be experts and share that with the world.

What does the work look like in practice? The four pillars I've lived by for years still apply: speaking, publishing, socializing, and hands-on content generation. If you've been reading this blog, you know that's exactly how I operate. Conference talks that go deep on real technical problems. Blog series that walk you through the details rather than hand-waving past them. Workshops where you actually get your hands dirty. Community conversations that happen at hallway tracks and meetup tables as much as on stage.

That framework doesn't change. The technology stack and the problems we're solving alongside the community will evolve, and I'm looking forward to getting deep on them all.

What's Coming on This Blog

One thing that won't change: this blog. I've been writing here for nearly two decades — no ads, no clickbait, no ulterior motive beyond sharing what I'm learning and working on with anyone who finds it useful. That continues.

You can expect content to start reflecting the world I'm operating in at SUSE. That means Linux, open source infrastructure, cloud native ecosystems, AI, and whatever technical problems are worth working through in public. The Mastering Fluent Bit series isn't going anywhere either — that work belongs to the community and I'll keep contributing to it.

Ready to Go

Three months of rest. A role that fits. A team I'm genuinely excited to work with. A company built on values I share. I've been doing this long enough to recognize when the conditions are right. The conditions are right.

If you're heading to an event in the coming months and want to connect, find me on the usual channels. And if you're working on something interesting in the open source or cloud native space and want to collaborate, reach out — that's always been how the best work gets started.

Here's to the next chapter. Let's build something worth talking about.

Installing openSUSE Tumbleweed on MacBook Pro 13 inch (late 2011)

eric@schabell.org (eric@schabell.org) — Thu, 12 Feb 2026 07:00:00 +0100

It's been awhile since I looked at my old Macbook Pro 13 inch from late 2011, with 125GB SSD and 8GB RAM. I've been on occasion posting articles about how to install Linux distributions on it and thought I'd give openSUSE a try this time.

This is a machine I've taken on trips around the world and back in the day ran many sessions, workshops, and demos sharing all that developer goodness.

Below are the steps and adjustments needed to get openSUSE working on these laptops in no time.

The first step is to pick between the two options that are available to us for the desktop. I've chosen openSUSE Tumbleweed, which has the promise of "You install it once and enjoy it forever. No longer do you have to worry every six months about massive system upgrades that risk bricking your system."

The second step is to get an USB stick big enough to hold the openSUSE Tumbleweed iso image. Note we are not worried about version numbers, as you "...get frequent updates that not only address vulnerabilities or squash bugs, but reflect latest features and developments, such as fresh kernels, fresh drivers and recent desktop environment versions."

I am a bit command line oriented so I just downloaded the offline 64-bit version offered as shown here:

This iso image needs to be written onto the USB stick we have so I've used a modern MacBook and the terminal to run diskutil list which shows me the USB is disk4:

Now make sure it's not mounted with diskutil unmountDisk /dev/disk4 and then run the following which shows not progress bar but takes quite awhile to complete the iso write to the USB stick so be patient:

sudo sh -c "cat PATH_TO/openSUSE-Tumbleweed-VERSION-Media.iso > /dev/rdisk4"

Once finished you can remove the USB device and move over to the old MacBook.

Installing on Macbook Pro 13 inch (late 2011)

Insert the USB device you created above, there is a port on the left side for this, (re-)start your Macbook Pro and be sure to hold down the Option (or alt) key, just to the left of the CMD key. This opens a menu of options to start this machine from and we'll need to use the EFI option as that's our USB image. Here you see I have the original macbook OSX setup installed.

Now it's booting from the device and you just can follow the normal installation. It really helps if you have a network cable connection you can plug your Macbook Pro into as the broadcom wifi device does not work out of the box.

First select from the boot menu the Installation option to install openSUSE Tumbleweed:

The installer will run through a bunch of checks as it works towards the partitioning of the disk.

It asks what type of system you want to install, I chose here the Gnome Desktop, feel free to explore any of the desktops you prefer.

After this you end up at the disk phase where you are able to look at the partitioner tool. I always chose to leave the recovery HD and remove the OSX partition, then dividing that up into a /, /boot, /boot/efi, and swap partitions. If you wan to explore, then open the Expert Partitioner on the bottom of the screen (exercise left to you who knows what you are doing).

As I've done this before on this MacBook I can just jump ahead and accept whatever the defaults are here by clicking on the Next button on the bottom right.

Next up, setting your timezone.

And your user.

Finally you can review all the things that are about to happen on this machine if you approve and click on the Next button.

Now you get to enjoy all the installation unfolding before your very eyes.

Once you've completed the installer, you should see user login, something like the following.

After logging in you will see the introduction to openSUSE. You've done it!

This is just the basic starting point, we need to now update the packages, add a repository for the wireless drivers and development kernels, and install them before this is completed.

Updating the installation

This part is best done on the command line to ensure each step of the process is run without problems. Log in and open a terminal to start (note, our user password we setup is what you use for root when asked by sudo). We need to use the zypper package manager to refresh and update all the packages from the installer.

sudo zypper refresh

sudo zypper dist-upgrade

This will pull a bunch of new updated packages and install them. For the next part, fixing the wifi device, we need to install a new repository. The easiest way to do this is fine the YaST2 application in the menus and click on the Add button on the bottom right.

This opens the list of available repositories and we want to select from the Community Repositories as shown.

Next select the Packman repository as shown and click on OK to install.

Once that completes we can move on to fixing the wifi device.

Wifi setup

Now the only thing missing is a wifi driver so there are a few things to be done that require that network cable to be connected as we install the development packages for the kernel we are running and then build the broadcom driver for that kernel.

Back to the terminal command line, install the following packages from the new repository we just added.

sudo zypper install broadcom-wl broadcom-wl-kmp-default kernel-devel

You will be warned about +30 packages that are going to be installed, but that's what we want. Note that the kmp-default suffix is openSUSE's convention for kernel module packages built against the default kernel. Tumbleweed handles the kernel module build automatically as part of the package install.

The following shows the installation of packages.

Now we want to trigger a rebuild (should be automatic, but better to be safe), so we need to install dams and lsmod as follows.

# Install dkms and lsmod
sudo zypper install dkms lsmod

Now trigger the rebuild and verify.

# Trigger DKMS rebuild (usually automatic, but to force it)
sudo dkms autoinstall

# Verify
lsmod | grep wl

If you don't see the models, it might be because you need to reboot the system to load the kernel. Once you've done that you can set up your wireless connection in openSUSE Tumbleweed in the top right drop down menu.

Pretty straight forward once you have the right device drivers, so hope you enjoyed this update to the latest openSUSE Tumbleweed on your old Macbook Pro 13 inch from late 2011!

OpenSearchCon EU - Fluent Bit & OpenSearch: The Power Couple of Open Source Observability

eric@schabell.org (eric@schabell.org) — Wed, 4 Feb 2026 07:00:00 +0100

After my adventures with the OpenSearch maintainers and developers in their booth at KubeCon NA this year, I was excited to share the results of our adventures at their OpenSearchCon EU event.

So with full enthusiasm I submitted a session about the observability power couple that is Fluent Bit and OpenSearch when used together. This would provide the attendees with a good overview of what the CNCF graduated project Fluent Bit, an end-to-end observability pipeline, can do for them.

To complement and visualize your pipeline telemetry data, OpenSearch was added to the mix and we worked hard to provide a streamlined demo project providing the hands-on experience of setting up the entire integration.

Let's look at the session and supporting hands-on learning content we are providing.

First off, the session was accepted as you can see in the image below:

The session as it was submitted is described below and will be given on Friday, 17 April 2026 in Prague:

Fluent Bit & OpenSearch: The Power Couple of Open Source Observability

In the world of cloud-native observability, the "Golden Signal" is only as valuable as the pipeline that carries it. To build a truly vendor-neutral, scalable observability stack, look to Fluent Bit for high-performance telemetry routing and OpenSearch for robust search and analytics. This session dives into why these two projects form the "Power Couple" of the open-source ecosystem. One excels at being a lightweight, sub-millisecond processor at the edge, the other providing the distributed backend required to make sense of that data.

What we will cover:

The "Why" Behind the Pairing: An exploration of how Fluent Bit’s low-resource footprint perfectly complements OpenSearch’s high-performance indexing.
Streamlined Integration: Practical demonstrations of configuring the Fluent Bit OpenSearch output plugin.
Zero to Dashboard: A walkthrough of the developer workflow—from containerized deployment using Podman to visualizing live telemetry in OpenSearch Dashboards.

Attendees will walk away with an understanding of the advantages to this couple and access to a free online hands-on workshop: https://o11y-workshops.gitlab.io/workshop-fluentbit

When: 17 April 2026, 16:30-17:10 in room Bohemia 1 by Patrick Stephens

As you can see this session is supported by a free online hands-on workshop, specifically the following lab for this specific session where we can repeat what we see in this session as an attendee: https://o11y-workshops.gitlab.io/workshop-fluentbit/#/11

Also of note, I am personally unable to attend this event and I really do not like to back out of anything I have previously committed to. The only solution was to find another Fluent Bit maintainer and friend of mine to take over and crush the delivery of this session. This could only be Patrick Stephens, a fellow Fluent Bit Maintainer who's an excellent speaker in his own right.

You can find the rest of the schedule online, but don't miss Patrick and this hands-on integration session with Fluent Bit telemetry pipelines and OpenSearch visualization backend. A special thanks to Pat for helping out!

KubeCon EU 2026 - From Hallucinations to Hardware: Diagnosing LLM Failures

eric@schabell.org (eric@schabell.org) — Sat, 31 Jan 2026 07:00:00 +0100

I'm excited to share that a talk I submitted, with my good friend Ryan Peirce, was accepted to Cloud Native AI & Kubeflow Day!

In case you were still unaware, KubeCon + CloudNativeCon is the flagship conference of the Cloud Native Computing Foundation (CNCF), gathering adopters and technologists from leading open source and cloud native communities. The event brings together the entire cloud native ecosystem for education, collaboration, and networking opportunities.

Every year this event features multiple co-located events, and we're excited to be presenting at Cloud Native AI & Kubeflow Day, which focuses on the intersection of cloud native technologies and artificial intelligence workloads.

Below are all the details of our session at the time of this writing.

On Monday, 23 March in Amsterdam, Ryan will be taking the stage to demonstrate real-world troubleshooting of LLM-powered applications for the following session.

From Hallucinations to Hardware: Diagnosing LLM Failures

Generative AI apps can hallucinate—or fail—at the worst possible times. In this live demo session, we'll interact with an LLM-powered application designed to surface both entertaining hallucinations and real-world GPU performance issues. Using open source tools like Prometheus, OTel, NVIDIA DCGM, and OpenInference, we'll troubleshoot problems in real time and trace them from user experience down to infrastructure. See how observability gives engineers and SREs the visibility they need to keep AI systems reliable.

Time: 15:20-15:45

This is a hands-on, live demonstration session where he will be working with a real LLM-powered application. He has intentionally designed it to showcase both the amusing side of AI hallucinations and the serious infrastructure challenges that can impact production systems. The session will walk through the complete troubleshooting journey, from identifying user-facing issues all the way down to GPU performance bottlenecks.

Ryan will be leveraging a powerful stack of open source observability tools including Prometheus for metrics collection, OpenTelemetry for distributed tracing, NVIDIA DCGM for GPU telemetry, and OpenInference for LLM-specific observability. This combination provides the comprehensive visibility needed to diagnose and resolve issues in modern AI infrastructure.

For engineers and SREs working with AI systems, this session will demonstrate practical approaches to maintaining reliability in production environments. The live demo format means you'll see real troubleshooting workflows in action rather than theoretical examples.

Be sure to check the Cloud Native AI & Kubeflow Day schedule for the exact time and location of the session as the time can be adjusted leading up to the co-located event.

Don't miss this session as Ryan is an amazing rock star on stage and his live demo is jaw dropping!

(Note that in the session URL you will see my name listed [from-hallucinations-to-hardware-diagnosing-llm-failures-eric-d-schabell-ryan-peirce], but I will not be on stage. Ryan was kind enough to cover for me as I won't be available to present on that day. Thanks Ryan!)

Mastering Fluent Bit: Developers Guide to Forwarding Metrics with Prometheus Remote Write

eric@schabell.org (eric@schabell.org) — Tue, 27 Jan 2026 07:00:00 +0100

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, integrating Fluent Bit with Prometheus. In case you missed the previous article, check out the developer guide to monitoring health metrics with Prometheus where you explore how to monitor the health of your telemetry data pipelines.

This article will complete our hands-on exploration of Prometheus integration, helping developers leverage Fluent Bit's powerful metrics capabilities. We'll look at the final pattern for integrating Fluent Bit with Prometheus in your observability infrastructure.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Integrating with Prometheus?

Before diving into the hands-on examples, let's understand why Prometheus integration matters for Fluent Bit users. Prometheus is the de facto standard for metrics collection and monitoring in cloud native environments. It's another CNCF graduated project that provides a time-series database optimized for operational monitoring. The combination of Fluent Bit's lightweight, high-throughput telemetry pipeline with Prometheus's battle-tested metrics storage creates a powerful observability solution.

Fluent Bit provides several ways to integrate with Prometheus, the first of which we covered in the first article. In the second article we explored Fluent Bit monitoring itself and exposing internal pipeline metrics, giving you visibility into the health and performance of your telemetry infrastructure. Understanding how your telemetry pipeline is performing is critical for maintaining reliable observability.

In this third and final article, we integrate with Prometheus using Fluent Bit as a metrics proxy, scraping metrics from various sources and forwarding them to Prometheus. This is particularly useful when we need to aggregate metrics from multiple sources or transform them before they reach Prometheus.

Let's dive into this final pattern, forwarding telemetry pipeline metrics to Prometheus with remote write.

Where to get started

You should have explored the previous articles in this series to install and get started with Fluent Bit on your developer local machine, either using the source code or container images. Links at the end of this article will point you to a free hands-on workshop that lets you explore more of Fluent Bit in detail.

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.2.2 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's explore the Prometheus integration pattern for forwarding Fluent Bit metrics that will help you with your observability infrastructure.

How to integrate with Prometheus

See this article for details about the service section of the configurations used in the rest of this article, but for now we plan to focus on our Fluent Bit pipeline and specifically the Prometheus integration capabilities that can be of great help in managing metrics in your observability stack.

Below in the figure you see the phases of a telemetry pipeline. Metrics collected by input plugins flow through the pipeline and can be routed to Prometheus-compatible outputs.

Understanding how metrics flow through Fluent Bit's pipeline is essential for effective Prometheus integration. Input plugins collect metrics, which then pass through filters for transformation, before being routed to output plugins that deliver metrics to Prometheus.

Forwarding metrics using Prometheus remote write

The final integration pattern demonstrates using Fluent Bit as a metrics proxy that scrapes existing Prometheus endpoints and forwards metrics using the Prometheus remote write protocol. This pattern is useful when you need to aggregate metrics from multiple sources, transform metrics before they reach Prometheus, or push metrics to remote Prometheus backends.

The prometheus_scrape input plugin allows Fluent Bit to collect metrics from any Prometheus-compatible endpoint. The prometheus_remote_write output plugin then pushes these metrics to a Prometheus server or compatible backend using the remote write API.

Let's create a configuration that scrapes metrics from a local application (we'll simulate this with HashiCorp Vault's metrics endpoint format) and forwards them using remote write:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    # Scrape metrics from an application's Prometheus endpoint
    - name: prometheus_scrape
      host: 127.0.0.1
      port: 8200
      tag: app_metrics
      metrics_path: /v1/sys/metrics
      scrape_interval: 10s

    # Also collect host metrics
    - name: node_exporter_metrics
      tag: host_metrics
      scrape_interval: 5

  outputs:
    # Forward all metrics to Prometheus using remote write
    - name: prometheus_remote_write
      match: '*'
      host: prometheus-server.example.com
      port: 9090
      uri: /api/v1/write
      add_label:
        - datacenter us-west-2
        - cluster production

This configuration demonstrates a powerful pattern where Fluent Bit acts as a metrics aggregator. It scrapes application metrics from a Prometheus endpoint, collects host metrics using the node exporter plugin, and forwards everything to a central Prometheus server using the remote write protocol. The custom labels datacenter and cluster are added to all metrics, providing organizational context for querying.

For environments where you need secure remote write with authentication, the configuration would look like this:

service:
  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: node_exporter_metrics
      tag: host_metrics
      scrape_interval: 5

    - name: fluentbit_metrics
      tag: pipeline_metrics
      scrape_interval: 2

  outputs:
    - name: prometheus_remote_write
      match: '*'
      host: prometheus.example.com
      port: 443
      uri: /api/v1/write
      tls: on
      tls.verify: on
      http_user: ${PROMETHEUS_USERNAME}
      http_passwd: ${PROMETHEUS_PASSWORD}
      add_label:
        - service my-application
        - environment production

This configuration securely pushes metrics to a remote Prometheus server using TLS and HTTP basic authentication. The credentials are read from environment variables, following security best practices. Both host metrics and Fluent Bit pipeline metrics are collected and forwarded together.

Let's examine a more advanced scenario where we scrape metrics from multiple sources and use tagging to route them appropriately:

service:
  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    # Scrape Redis metrics
    - name: prometheus_scrape
      host: redis-server
      port: 9121
      tag: redis_metrics
      metrics_path: /metrics
      scrape_interval: 15s

    # Scrape PostgreSQL metrics
    - name: prometheus_scrape
      host: postgres-server
      port: 9187
      tag: postgres_metrics
      metrics_path: /metrics
      scrape_interval: 15s

    # Collect host metrics
    - name: node_exporter_metrics
      tag: node_metrics
      scrape_interval: 10

  outputs:
    # Send infrastructure metrics to operations Prometheus
    - name: prometheus_remote_write
      match: node_metrics
      host: ops-prometheus.internal
      port: 9090
      uri: /api/v1/write
      add_label:
        - team operations
        - tier infrastructure

    # Send application metrics to development Prometheus
    - name: prometheus_remote_write
      match: '*_metrics'
      host: dev-prometheus.internal
      port: 9090
      uri: /api/v1/write
      add_label:
        - team development
        - tier application

This configuration demonstrates Fluent Bit's routing capabilities applied to metrics before they are sent to the Prometheus backend. Infrastructure metrics (from node_exporter_metrics) are routed to an operations Prometheus instance, while application metrics (Redis and PostgreSQL) are routed to a development team's Prometheus instance. Each destination receives appropriately labeled metrics for easy filtering and organization.

The Prometheus remote write protocol also supports compression for efficient network usage. You can enable compression by adding the compress parameter:

outputs:
    - name: prometheus_remote_write
      match: '*'
      host: prometheus-server.example.com
      port: 9090
      uri: /api/v1/write
      compress: gzip

This is particularly useful when pushing metrics over high-latency or bandwidth-constrained networks.

It's left to the reader to explore their metrics telemetry data with their own Prometheus instance and to browse through the collected metrics telemetry data. A primer to do this if you need help can be found in this hands-on free online Prometheus workshop.

More in the series

In this article you explored the final of three powerful patterns for integrating Fluent Bit with Prometheus: forwarding metrics with Fluent Bit to Prometheus. This article is based on this online free workshop.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild.

Mastering Fluent Bit: Developers Guide to Monitoring Health Metrics with Prometheus

eric@schabell.org (eric@schabell.org) — Tue, 20 Jan 2026 07:00:00 +0100

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, integrating Fluent Bit with Prometheus. In case you missed the previous article, check out the developer guide to routing metrics to Prometheus where you explore how to collect and direct metrics telemetry data to your Prometheus instance.

This article will continue our hands-on exploration of Prometheus integration, helping developers leverage Fluent Bit's powerful metrics capabilities. We'll look at the second of three essential patterns for integrating Fluent Bit with Prometheus in your observability infrastructure.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Integrating with Prometheus?

Fluent Bit provides several ways to integrate with Prometheus, the first of which we covered in the previous article. In this article we'll explore Fluent Bit monitoring itself and exposing internal pipeline metrics, giving you visibility into the health and performance of your telemetry infrastructure. Understanding how your telemetry pipeline is performing is critical for maintaining reliable observability.

The third and final way to integrate with Prometheus is to use Fluent Bit can act as a metrics proxy, scraping metrics from various sources and forwarding them to Prometheus. This is particularly useful when you need to aggregate metrics from multiple sources or transform them before they reach Prometheus. This will be explored in a future article.

Let's dive into the second pattern, exposing internal telemetry pipeline metrics to Prometheus.

Where to get started

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.2.2 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's explore the Prometheus integration pattern for monitoring Fluent Bit metrics that will help you with your observability infrastructure.

How to integrate with Prometheus

Below in the figure you see the phases of a telemetry pipeline. Metrics collected by input plugins flow through the pipeline and can be routed to Prometheus-compatible outputs.

Monitoring telemetry pipeline health

The second integration pattern focuses on monitoring the health and performance of your Fluent Bit instance. Understanding how your telemetry pipeline is performing is critical for maintaining reliable observability infrastructure. Fluent Bit provides the fluentbit_metrics input plugin that exposes internal pipeline metrics.

The Fluent Bit Metrics plugin collects valuable information about your pipeline including uptime, input plugin throughput, output plugin performance, buffer usage, and error counts. These metrics help you answer questions like: Is my pipeline processing data fast enough? Are there any backpressure issues? How much memory is my pipeline consuming?

To demonstrate this pattern, let's create a configuration file called fluent-bit.yaml that exposes internal Fluent Bit metrics:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    # Simulate application logs for the pipeline to process
    - name: dummy
      tag: app.logs
      dummy: '{"message":"Application log entry","level":"info"}'

    # Collect Fluent Bit internal metrics
    - name: fluentbit_metrics
      tag: internal_metrics
      scrape_interval: 2

  outputs:
    # Send application logs to stdout
    - name: stdout
      match: app.logs
      format: json_lines

    # Expose internal metrics for Prometheus
    - name: prometheus_exporter
      match: internal_metrics
      host: 0.0.0.0
      port: 2022
      add_label:
        - instance fluent-bit-dev
        - component telemetry-pipeline

This configuration demonstrates a realistic scenario where Fluent Bit is processing application logs while simultaneously exposing its own internal metrics. The fluentbit_metrics input plugin collects metrics about the pipeline itself, and these are exposed on a separate port (2022) from any host metrics we might be collecting.

Let's run this configuration as follows:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.2.2
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

# Note: For container deployments collecting linux host metrics, you need
# to mount the host's /proc and /sys filesystems:
# $ podman run --rm -v /proc:/host/proc:ro -v /sys:/host/sys:ro -p 2021:2021 fb

$ podman run --rm fb 

...
{"date":1768901032.490993,"message":"Application log entry","level":"info"}
{"date":1768901033.490052,"message":"Application log entry","level":"info"}
{"date":1768901034.488256,"message":"Application log entry","level":"info"}
...

Our console output shows the generated log message and in the background, being staged on the endpoint for a Prometheus instance to scrap are our Fluent Bit instance metrics.

Now verify the metrics are being tagged with custom labels by opening a browser window to http://localhost:2022/metrics and we should see the following:

# HELP fluentbit_uptime Number of seconds that Fluent Bit has been running.
# TYPE fluentbit_uptime counter
fluentbit_uptime{instance="fluent-bit-dev",component="telemetry-pipeline",hostname="Erics-MacBook-Pro.local"} 126
# HELP fluentbit_logger_logs_total Total number of logs
# TYPE fluentbit_logger_logs_total counter
fluentbit_logger_logs_total{instance="fluent-bit-dev",component="telemetry-pipeline",message_type="error"} 0
fluentbit_logger_logs_total{instance="fluent-bit-dev",component="telemetry-pipeline",message_type="warn"} 0
fluentbit_logger_logs_total{instance="fluent-bit-dev",component="telemetry-pipeline",message_type="info"} 14
fluentbit_logger_logs_total{instance="fluent-bit-dev",component="telemetry-pipeline",message_type="debug"} 0
fluentbit_logger_logs_total{instance="fluent-bit-dev",component="telemetry-pipeline",message_type="trace"} 0
# HELP fluentbit_routing_logs_records_total Total log records routed from input to output
# TYPE fluentbit_routing_logs_records_total counter
fluentbit_routing_logs_records_total{instance="fluent-bit-dev",component="telemetry-pipeline",input="dummy.0",output="stdout.0"} 125
# HELP fluentbit_routing_logs_bytes_total Total bytes routed from input to output (logs)
# TYPE fluentbit_routing_logs_bytes_total counter
...

These metrics provide valuable insights into your pipeline's health. The fluentbit_uptime counter tells you how long your instance has been running. The fluentbit_input_records_total and fluentbit_input_bytes_total counters show throughput for each input plugin. The fluentbit_output_records_total, fluentbit_output_retries_total, and fluentbit_output_errors_total counters help you monitor output plugin performance and detect delivery issues.

To integrate this with Prometheus, add a scrape configuration to your Prometheus configuration file prometheus.yml as follows:

scrape_configs:
  - job_name: 'fluent-bit-health-metrics'
    static_configs:
      - targets: ['localhost:2022']
    scrape_interval: 10s

This configuration tells Prometheus to scrape the Fluent Bit metrics endpoint every 10 seconds. The metrics will then be available for querying in Prometheus and can be visualized in the Prometheus console or using the Perses project for dashboards.

With these metrics in Prometheus, you can create alerts for scenarios like high retry rates (indicating backpressure or destination issues), zero input records (indicating data collection problems), or high memory usage.

More in the series

In this article you explored the second of three powerful patterns for integrating Fluent Bit with Prometheus: monitoring Fluent Bit health metrics. In the following article we will continue onwards to look at scraping and forwarding Prometheus metrics with remote write. This article is based on this online free workshop.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, forwarding Prometheus metrics.

Mastering Fluent Bit: Developers Guide to Routing Metrics to Prometheus

eric@schabell.org (eric@schabell.org) — Thu, 8 Jan 2026 07:00:00 +0100

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, integrating Fluent Bit with Prometheus. In case you missed the previous article, check out the developer guide to telemetry pipeline routing where you explore how to direct telemetry data to different destinations based on tags, patterns, and conditions.

This article will be a hands-on exploration of Prometheus integration that helps you as a developer leverage Fluent Bit's powerful metrics capabilities. We'll look at the first of three essential patterns for integrating Fluent Bit with Prometheus in your observability infrastructure.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Integrating with Prometheus?

Fluent Bit provides several ways to integrate with Prometheus. You can expose metrics endpoints that Prometheus can scrape (pull model), push metrics directly to Prometheus using the remote write protocol, or even scrape existing Prometheus endpoints and route those metrics through your telemetry pipeline. This flexibility allows Fluent Bit to act as a metrics aggregator, forwarder, or even a replacement for dedicated metrics agents in resource-constrained environments.What is Prometheus Integration?

There are several compelling reasons to integrate Fluent Bit with Prometheus in your infrastructure. First, Fluent Bit can collect system-level metrics using its built-in Node Exporter Metrics plugin, eliminating the need to deploy a separate Prometheus Node Exporter. This reduces resource usage and simplifies your deployment.

Second, Fluent Bit can monitor itself and expose internal pipeline metrics, giving you visibility into the health and performance of your telemetry infrastructure. Understanding how your telemetry pipeline is performing is critical for maintaining reliable observability. This will be covered in a future article.

Third, Fluent Bit can act as a metrics proxy, scraping metrics from various sources and forwarding them to Prometheus. This is particularly useful when you need to aggregate metrics from multiple sources or transform them before they reach Prometheus. This will be explored in a future article.

Let's dive into the first pattern, collecting system-level metrics using its built-in Node Exporter Metrics plugin.

Where to get started

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.2.2 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's explore the three Prometheus integration patterns that will help you with your observability infrastructure.

How to integrate with Prometheus

Below in the figure you see the phases of a telemetry pipeline. Metrics collected by input plugins flow through the pipeline and can be routed to Prometheus-compatible outputs.

Now let's look at the first of three most useful Prometheus integration patterns that developers will want to know about.

Routing metrics through Fluent Bit to Prometheus

The first integration pattern involves collecting host-level metrics using Fluent Bit's built-in Node Exporter Metrics plugin and exposing them for Prometheus to scrape. This pattern is incredibly valuable because it allows you to collect system metrics without deploying a separate Prometheus Node Exporter agent.

The Node Exporter Metrics input plugin implements a subset of the collectors available in the original Prometheus Node Exporter. It collects CPU statistics, memory usage, disk I/O, network interface statistics, filesystem information, and more. The beauty of this approach is that all these metrics flow through Fluent Bit's pipeline, where you can transform, filter, and route them as needed.

To demonstrate this pattern, let's create a configuration file called fluent-bit.yaml that collects host metrics and exposes them through a Prometheus endpoint:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: node_exporter_metrics
      tag: node_metrics
      scrape_interval: 2

  outputs:
    - name: prometheus_exporter
      match: node_metrics
      host: 0.0.0.0
      port: 2021
      add_label:
        - app fluent-bit
        - environment development

    # testing output to console

    - name: stdout
      match: node_metrics
      format: json_lines

Our configuration uses the node_exporter_metrics input plugin to collect system metrics every two seconds. The prometheus_exporter output plugin then exposes these metrics on port 2021 in a format that Prometheus can scrape. We've also added the custom labels app and environment that will be attached to all metrics, making it easier to filter and query them in Prometheus.

Let's run this configuration as follows:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.2.2
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

# Note: For container deployments collecting linux host metrics, you need
# to mount the host's /proc and /sys filesystems:
# $ podman run --rm -v /proc:/host/proc:ro -v /sys:/host/sys:ro -p 2021:2021 fb

$ podman run --rm fb 

...
[2026/01/19 15:25:47.115361000] [ warn] [input:node_exporter_metrics:node_exporter_metrics.0] calling IORegistryEntryGetChildEntry is failed
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="0",mode="user"} = 25039.200000000001
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="0",mode="system"} = 9067.2999999999993
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="0",mode="nice"} = 0
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="0",mode="idle"} = 48662.790000000001
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="1",mode="user"} = 23096.860000000001
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="1",mode="system"} = 7764.7299999999996
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="1",mode="nice"} = 0
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="1",mode="idle"} = 52016.459999999999
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="2",mode="user"} = 20056.130000000001
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="2",mode="system"} = 6364.9700000000003
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="2",mode="nice"} = 0
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="2",mode="idle"} = 56597.839999999997
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="3",mode="user"} = 17696.98
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="3",mode="system"} = 5385.8999999999996
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="3",mode="nice"} = 0
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="3",mode="idle"} = 60055.519999999997
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="4",mode="user"} = 412.75
2026-01-19T14:25:47.114513143Z node_cpu_seconds_total{cpu="4",mode="system"} = 116.18000000000001
...

Note that the warning entry is an OSX specific issue with the node_exporter_metrics input plugin. The node_exporter_metrics plugin tries to collect system metrics similar to Prometheus Node Exporter. On OSX, it uses Apple's IOKit framework to access hardware information through the IORegistry (a hierarchical database of hardware devices).

Our console output for testing shows all the available metrics about this machine that are being collected every 1s. This gives us something to work with and query once it's sent to a Prometheus backend.

Now verify the metrics are being tagged with custom labels by opening a browser window to http://localhost:2021/metrics and we should see the following:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="0",mode="user"} 25095.049999999999
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="0",mode="system"} 9092.3299999999999
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="0",mode="nice"} 0
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="0",mode="idle"} 48874.290000000001
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="1",mode="user"} 23145.150000000001
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="1",mode="system"} 7784.9700000000003
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="1",mode="nice"} 0
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="1",mode="idle"} 52240.839999999997
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="2",mode="user"} 20091.23
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="2",mode="system"} 6379.5500000000002
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="2",mode="nice"} 0
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="2",mode="idle"} 56841.639999999999
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="3",mode="user"} 17723.43
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="3",mode="system"} 5396.7299999999996
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="3",mode="nice"} 0
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="3",mode="idle"} 60312.18
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="4",mode="user"} 412.89999999999998
node_cpu_seconds_total{app="fluent-bit",environment="development",cpu="4",mode="system"} 116.25
...

Notice how the metrics include our custom labels app="fluent-bit" and environment="development". These labels are automatically added to every metric by the Prometheus exporter output plugin, making it easy to identify and filter metrics in your Prometheus queries.

To integrate this with Prometheus, add a scrape configuration to your Prometheus configuration file prometheus.yml as follows:

scrape_configs:
  - job_name: 'fluent-bit-node-metrics'
    static_configs:
      - targets: ['localhost:2021']
    scrape_interval: 10s

The Node Exporter Metrics plugin supports numerous collectors including CPU, disk I/O, filesystem, load average, memory, network interface, and more. You can selectively enable or disable collectors based on your monitoring needs, and set individual scrape intervals for each collector type.

It's left to the reader to run their own Prometheus instance with this configuration and to explore the collected metrics telemetry data. A primer to do this if you need help can be found in this hands-on free online Prometheus workshop.

More in the series

In this article you explored the first of three powerful patterns for integrating Fluent Bit with Prometheus: collecting and exposing host metrics. In the following article we will continue onwards to look at monitoring Fluent Bit's internal pipeline health and using Fluent Bit as a metrics proxy with remote write capabilities. This article is based on this online free workshop.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, we'll explore monitoring Fluent Bit's internal pipeline health.

DevOpsCon Amsterdam 2026: It’s my docs, and I will cry if I want to!

eric@schabell.org (eric@schabell.org) — Mon, 5 Jan 2026 07:00:00 +0100

My last DevOpsCon event was in Munich, so when I saw they were expanding and visiting my home country in Amsterdam, Netherlands I could not resist. I submitted a few things and the one that was selected is a talk I'm going to be testing based on my experiences over the last year as a maintainer on the Fluent Bit CNCF project.

I will be on-site at the Van der Valk Hotel Amsterdam-Amstel for DevOpsCon Amsterdam from 20-24 April 2026. They will have networking, sessions, and hands-on workshops available for all who attend.

Below you'll find my talk details and the schedule.

My talk was added to the Observability & Reliability track and is scheduled for Wednesday, 22 April 2026 at the time of this writing.

It’s my docs, and I will cry if I want to!

(Surviving and reviving documentation in a decade-old open source project)

Documentation is often the unsung hero — or the forgotten relic — of open source software. After ten years of development, Fluent Bit, a high-performance logging and metrics processor, has grown into a complex ecosystem of plugins, integrations, and configuration options. Meanwhile, its documentation has struggled to keep pace. In this talk, I’ll share the journey of inheriting and maintaining a decade’s worth of documentation debt in the Fluent Bit Docs project. We’ll explore what happens when docs evolve slower than code, the challenges of balancing accuracy with accessibility, and how technical debt in docs is just as real (and painful) as in code.

I’ll share my experiences with:

Auditing and triaging outdated documentation at scale (getting started)
Building collaboration between developers, contributors, and users (trying to!)
Modernizing doc structure and tooling without burning it all down (step-by-baby-step)
Restoring your sanity — and maybe even your love for docs (the ongoing rewards!)

Whether you’re a doc maintainer, an open source contributor, or just someone who’s cried over a barebones README, this talk will commiserate and (hopefully) motivate you to never fear a docs challenge again!

Looking forward to seeing you there and getting feedback on what you think of this topic!

2025 Year in Review: Community, Contribution, and Changes

eric@schabell.org (eric@schabell.org) — Wed, 31 Dec 2025 07:00:00 +0100

As we ease into the last few days of 2025, I like to close out this one last task: reflecting back on all the fun, challenges, and travel over the past year. While it takes a bit of research to gather all the facts and figures, it’s a rewarding effort. I’ve been sharing on this site for two decades now—no ads, no ulterior motives—just an honest effort to communicate with the world.

This year, the focus for me and my teams remained on the four pillars of our role: speaking, publishing, socializing, and hands-on content generation.

While 2024 was about observability (o11y) taking center stage, 2025 was the year we put those theories into practice, accelerated our momentum, and stated to gain serious traction in the wild.

Read on to find out more!

This was my third full year working for Chronosphere, an observability vendor that has really started to blossom. I remember when I started here, nobody knew who we were. Now it's a common occurrence to run into existing customers and to hear people say, "Chronosphere, I've heard of that!"

Travel and Events

Travel and speaking at events around the world is a major focus of what I do. I'm teaching, sharing, and freely contributing to both open source projects and to the general public knowledge around observability technologies.

As you can see in the attached figure, I had 17 trips to 36 cities and 8 countries with the following destinations:

Portland, OR
Salt Lake City, UT
Spain
Las Vegas, NV
London, UK (2
Glasgow, Scotland
Boston, MA
New York City, NY
Washington DC
Raleigh, NC
Oklahoma City, OK
Manchester, UK
Dublin, Ireland
Atlanta, GA

Some of the events:

SREDay London
Devoxx UK
Cloud Native Rejekts EU
KubeCon EU
Cloud Native Los Angeles
Tech Meetup Glasgow
Cloud Native London
Red Hat Summit NA
SRECon EU
All Things Open
Cloud Native Rejects NA
KubeCon NA

If 2024 was about strategic travel, 2025 was about being where the community is. The travel schedule was relentless but incredibly rewarding. From London to Atlanta, the message remained clear: observability is the backbone of modern operations.

Publishing

I’ve continued to preach the open-source gospel as a Cloud Native Computing Foundation (CNCF) Ambassador. My team spent 2025 bridging the gap between metrics, events, traces, logs, and dashboards.

This year my blog turned 20 years in production! The reach of our content in 2025 exceeded all expectations. Sharing knowledge is only half the battle; ensuring it reaches the right people is the other half. This year, my blog surpassed 900,000 views, a testament to the community's growing hunger for cloud-native and observability insights.

I topped out in Oct - Nov at 205k views. At the time of this writing it's around 4.7 million views and I'm happy to see that it has such value for so many visitors as that's always been the goal.

This blog offers free content (see the license at the footer of this blog) and without any advertising. It's about learning and not about earning. I've published 37 articles here this year on a large range of topics, but mostly focused on cloud native observability. I've also continued to find syndication with DZone, posting 16 articles in featured posts.

Workshops and hands-on content

This was the Year of the Workshop. We focused heavily on Fluent Bit telemetry pipelines, Prometheus, OpenTelemetry, and Perses visualization.

I personally shifted my focus from Perses to Fluent Bit and have since become a maintainer. I'm working extensively on the docs project, some on the website, and have even managed to start contributing code to the main Fluent Bit project. Good to be back hands on with open source projects, where I have my roots.

This was the year I also started working with AI in my coding, using vibe coding as it's known. This has been a fun journey and it definitely has accelerated both my coding and my production. It has not gone on without a few bumps in the road, but that's the fun of learning new things!

The Fairway Journey

While the tech world moved fast, I made sure my time on the grass kept pace. 2025 was the first full year of focus for my golf game. I logged an incredible 123 rounds, taking my clubs to Spain, Florida, North Carolina, and Oregon. All that practice paid off—I started the year at a 9.4, but I’m closing out 2025 with my handicap down to a 5.6.

The goal is not for me to get to scratch, I just enjoy a nice time and want to hit the ball rather regularly where I aim it. This is the realization that life is moving along and that I'm not going upwards in any sports I choose to play, but slowly declining.

It's a natural evolution and I think golf is the one game I can play for a very, very long time and the enjoyment will not diminish no matter the handicap.

What's next?

Finally, the year wrapped up with some major news!

I do not know how this will change work and my 2026 plans, but I'm very excited for this next chapter. I am looking forward to helping scale Chronosphere's reach next year.

Thanks for the Journey

This journey would be nothing without our readers, listeners, watchers, and those providing feedback. I thank you from the bottom of my heart. I’m already looking forward to 2026 and the new challenges it will bring.

Happy holidays and end of the year to you and your families as you transition into 2026, I'm looking forward to seeing you all online and in person next year!

Mastering Fluent Bit: Developer Guide to Telemetry Pipeline Routing

eric@schabell.org (eric@schabell.org) — Wed, 3 Dec 2025 07:00:00 +0100

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, using Fluent Bit routing for developers. In case you missed the previous article, check out the top three telemetry pipeline filters where you explore the most useful filters for manipulating and controlling your telemetry data.

This article will be a hands-on exploration of routing patterns that help you as a developer building sophisticated telemetry pipelines. We'll look at how to direct telemetry data to different destinations based on tags, patterns, and conditions in your Fluent Bit configurations.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.2.0 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's explore how routing works in Fluent Bit and why it's essential for building production-ready telemetry pipelines.

Understanding routing in telemetry pipelines

See this article for details about the service section of the configurations used in the rest of this article, but for now we plan to focus on our Fluent Bit pipeline and specifically the routing capabilities that let us direct telemetry data to appropriate destinations.

Below in the figure you see the phases of a telemetry pipeline. Routing happens throughout the pipeline but is most evident in the final output phase, where we decide which processed events go to which destinations.

Routing in Fluent Bit determines which events flow to which outputs, and there are two primary routing approaches available to developers:

Tag-based routing

Every event in Fluent Bit carries a tag, and outputs use the match parameter to subscribe to events with specific tags. Tags are simple strings that act as identifiers for event streams, similar to topics in message queues. This tag-based routing system is the traditional foundation of Fluent Bit's routing capabilities.

Tags typically follow a hierarchical naming convention like app.frontend.logs or system.metrics.cpu. This structure makes it easy to create wildcard patterns for flexible matching. The match parameter in outputs supports wildcards: a single asterisk (*) matches any characters at one level, while a double asterisk (**) matches multiple levels of the hierarchy.

For example, a match pattern of app.* would catch app.logs and app.metrics, but not app.frontend.logs. Meanwhile, app.** would match all three. This wildcard system provides the flexibility needed for complex routing scenarios while keeping configurations manageable.

Conditional routing

Conditional routing is a newer approach available in Fluent Bit 4.2+ that evaluates individual records and routes them to different outputs based on their content. Unlike tag-based routing which operates on entire data chunks, conditional routing provides per-record routing decisions.

This mechanism uses a routes block defined within input configurations. Each route specifies conditions that determine which records match, and the outputs where matching records should be sent. Conditions can examine any field within the record using comparison operators like equals, greater than, regular expressions, and array membership tests.

Conditional routing is particularly powerful when you need to split logs based on severity levels, route records from different services to dedicated outputs, or implement complex multi-condition routing logic. It provides fine-grained control without requiring filters to modify tags, making your routing configuration more explicit and easier to understand.

The key advantage is that routing decisions are made per-record at the input stage, before any filtering or processing occurs. This enables efficient routing that can send subsets of data to different destinations based on content, without processing or forwarding unnecessary records.

Combining routing mechanisms

In production environments, you rarely send all your telemetry data to a single destination. Different types of logs need different handling. Error logs might go to an alerting system, audit logs to long-term storage, and debug logs might be dropped entirely to save costs.

You can combine both routing approaches in a single configuration. Use tag-based routing for broad categorization by source or input type, and conditional routing for fine-grained decisions based on record content like severity level, error codes, or specific field values. Together, these routing approaches give you complete control over your telemetry pipeline.

Understanding these routing patterns is essential for developers who want to build sophisticated telemetry pipelines that efficiently direct data where it needs to go while minimizing costs and maximizing value.

Now let's look at practical implementations of these routing patterns that developers need to master.

Basic tag-based routing with wildcards

The traditional routing mechanism in Fluent Bit is the tag-based system. Every input assigns data a human-readable identifier (tag), and every output uses a match pattern to select which events it receives. Let's create a configuration that demonstrates basic routing patterns.

First, create a configuration file called fluent-bit.yaml as follows:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy

      tag: app.frontend.requests

      dummy: '{"service":"frontend","level":"info","message":"Request processed"}'

    - name: dummy
      tag: app.backend.errors
      dummy: '{"service":"backend","level":"error","message":"Database connection failed"}'
    
    - name: dummy
      tag: system.metrics
      dummy: '{"cpu_usage":75,"memory_usage":60}'


  outputs:
    - name: stdout
      match: app.frontend.*
      format: json_lines
    
    - name: stdout
      match: app.backend.*
      format: json_lines
    
    - name: stdout
      match: system.*
      format: json_lines

Our configuration creates three different dummy inputs, each with its own tag representing different sources of telemetry data. The tag structure uses a hierarchical naming convention, similar to Java package names or DNS domains, which makes it easy to create wildcard patterns.

The outputs section shows three stdout outputs, each matching a different tag pattern. The asterisk (*) is a wildcard that matches any characters at that level. This means app.frontend.* matches any event with a tag starting with app.frontend., regardless of what follows.

Let's run this configuration to see tag-based routing in action:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.2.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-18 10:15:23.456789","service":"frontend","level":"info","message":"Request processed"}
{"date":"2025-12-18 10:15:24.567890","service":"backend","level":"error","message":"Database connection failed"}
{"date":"2025-12-18 10:15:25.678901","cpu_usage":75,"memory_usage":60}
...

Each event is routed to its corresponding output based on the tag match pattern. This basic routing pattern is the foundation for more complex scenarios. Note that we use stdout for most examples, leaving it to the reader to experiment with other output destinations in their own environment for routing to different destinations.

You can also use the double asterisk (**) wildcard to match multiple levels. Let's modify our outputs to demonstrate this:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy

      tag: app.frontend.requests

      dummy: '{"service":"frontend","level":"info","message":"Request processed"}'

    - name: dummy
      tag: app.backend.errors
      dummy: '{"service":"backend","level":"error","message":"Database connection failed"}'
    
    - name: dummy
      tag: system.metrics
      dummy: '{"cpu_usage":75,"memory_usage":60}'


  outputs:
    - name: stdout
      match: app.**
      format: json_lines

Running this configuration shows that the single output with app.** matches both app.frontend.requests and app.backend.errors, but not system.metrics. The double asterisk matches any number of tag levels, making it perfect for catching all events under a hierarchical namespace.

Conditional routing

While tag-based routing directs entire chunks of data based on their source, conditional routing evaluates individual records and makes routing decisions based on their content. This feature, available in Fluent Bit 4.2 and greater, uses a routes block within input configurations to define routing rules.

Let's create a configuration that routes logs to different destinations based on their severity level:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"service":"api","level":"ERROR","message":"Failed to connect"}'
      routes:
        logs:
          - name: critical_errors
            condition:
              op: or
              rules:
                - field: "$level"
                  op: eq
                  value: "ERROR"
                - field: "$level"
                  op: eq
                  value: "FATAL"
            to:
              outputs:
                - critical_output
    
    - name: dummy
      tag: app.logs
      dummy: '{"service":"api","level":"INFO","message":"Request completed"}'
      routes:
        logs:
          - name: normal_logs
            condition:
              op: and
              rules:
                - field: "$level"
                  op: eq
                  value: "INFO"
            to:
              outputs:
                - info_output

  outputs:
    - name: stdout
      alias: critical_output
      format: json_lines
    
    - name: stdout
      alias: info_output
      format: json_lines

The routes block is where conditional routing configuration happens. Each route has several key components:

name - A unique identifier for the route
condition - The logic block that determines which records match
condition.op - The logical operator (and or or) for combining multiple rules
condition.rules - An array of rules to evaluate against each record
to.outputs - The destination outputs (referenced by alias or name) for matching records

Each rule in the condition.rules array specifies:

field - The field to examine using record accessor syntax ($level, $service, etc.)
op - The comparison operator (eq, neq, gt, lt, gte, lte, regex, not_regex, in, not_in)
value - The value to compare against (can be a single value or array for in/not_in operators)
context - Optional parameter specifying where to look for the field (body, metadata, otel_resource_attributes, etc.)

When a record arrives, Fluent Bit evaluates the conditions for each route in order. Records are sent to outputs whose conditions they match. This allows per-record routing decisions based on the actual content of each log entry.

Let's run this configuration:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.2.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-18 10:30:45.123456","service":"api","level":"ERROR","message":"Failed to connect"}
{"date":"2025-12-18 10:30:46.234567","service":"api","level":"INFO","message":"Request completed"}
...

The ERROR logs are routed to the critical_output while INFO logs go to info_output, all based on examining the level field in each record.

You can also define a default route to catch records that don't match any other conditions:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"service":"api","level":"ERROR","message":"Failed to connect"}'
      routes:
        logs:
          - name: error_logs
            condition:
              op: and
              rules:
                - field: "$level"
                  op: eq
                  value: "ERROR"
            to:
              outputs:
                - error_output
          
          - name: default_logs
            condition:
              default: true
            to:
              outputs:
                - default_output

  outputs:
    - name: stdout
      alias: error_output
      format: json_lines
    
    - name: stdout
      alias: default_output
      format: json_lines

The route with condition.default: true acts as a catch-all for any records that don't match the earlier routes.

Let's create a more complex example that routes based on multiple conditions:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"service":"api","level":"ERROR","environment":"production","response_time":6000}'
      routes:
        logs:
          - name: high_priority_errors
            condition:
              op: and
              rules:
                - field: "$level"
                  op: eq
                  value: "ERROR"
                - field: "$environment"
                  op: eq
                  value: "production"
                - field: "$response_time"
                  op: gt
                  value: 5000
            to:
              outputs:
                - critical_output
                - audit_output

          - name: all_other_logs
            condition:
              default: true
            to:
              outputs:
                - general_output

  outputs:
    - name: stdout
      alias: critical_output
      format: json_lines
    
    - name: stdout
      alias: audit_output
      format: json_lines

    - name: stdout

      alias: general_output
      format: json_lines

This configuration uses the and operator to combine three conditions. Only records that are ERROR level, from production, AND have a response time greater than 5000ms will be sent to both the critical_output and audit_output. All other records go to general_output.

Let's run this configuration:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.2.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-18 10:45:12.345678","service":"api","level":"ERROR","environment":"production","response_time":6000}
{"date":"2025-12-18 10:45:12.345678","service":"api","level":"ERROR","environment":"production","response_time":6000}
...

Notice how the high-priority error appears twice in the output because it's being sent to two different outputs (critical_output and audit_output).

Conditional routing also supports the in and not_in operators for matching against arrays of values:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"service":"payment","level":"ERROR","message":"Transaction failed"}'
      routes:
        logs:
          - name: critical_services
            condition:
              op: and
              rules:
                - field: "$service"
                  op: in
                  value: ["payment", "authentication", "database"]
            to:
              outputs:
                - critical_output

          - name: standard_logs
            condition:
              default: true
            to:
              outputs:
                - standard_output

  outputs:
    - name: stdout
      alias: critical_output
      format: json_lines
    
    - name: stdout
      alias: standard_output
      format: json_lines

This configuration routes logs from critical services (payment, authentication, database) to a dedicated output, while all other services go to standard output. The in operator makes it easy to match against multiple values without creating separate rules for each one.

Advanced routing with regular expressions and nested tags

Let's look at advanced routing which we can implement using tag-based fan-out, where a single event is sent to multiple destinations by having multiple outputs with overlapping match patterns:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: prod.svc.auth.logs
      dummy: '{"service":"auth","environment":"production","level":"ERROR","message":"Token validation failed"}'

  outputs:
    - name: stdout
      match: 'prod.**'
      format: json_lines
      
    - name: stdout
      match_regex: '.*\.auth\..*'
      format: json_lines
    
    - name: stdout
      match: '**.logs'
      format: json_lines

Running this configuration shows that the single input event matches all three output patterns:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.2.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-18 11:15:45.567890","service":"auth","environment":"production","level":"ERROR","message":"Token validation failed"}
{"date":"2025-12-18 11:15:45.567890","service":"auth","environment":"production","level":"ERROR","message":"Token validation failed"}
{"date":"2025-12-18 11:15:45.567890","service":"auth","environment":"production","level":"ERROR","message":"Token validation failed"}
...

The event appears three times because it matches:

prod.** (production environment)
.*\.auth\..* (auth service)
**.logs (logs data type)

This fan-out pattern is essential for implementing multi-destination routing without duplicating configuration or using additional filters.

Important routing considerations

When implementing routing in your Fluent Bit pipelines, keep these important points in mind:

Design tag structure carefully - Tags are the backbone of your routing strategy. Use hierarchical naming with clear levels that represent environment, service type, service name, and data type. This makes it easy to create flexible match patterns.
Be aware of ordering - Fluent Bit processes outputs in the order they're defined in the configuration. For most routing scenarios this doesn't matter, but if you're doing advanced processing or using filters that depend on routing results, order can be significant.
Use match patterns efficiently - While you can create many specific outputs with exact tag matches, it's often more maintainable to use wildcard patterns and fewer outputs. Balance specificity with simplicity.
Consider performance implications - Every output adds overhead. If you're routing to many destinations, test the performance impact and consider whether you can consolidate outputs or use features like tag-based fan-out more efficiently.
Test your routing logic - Use the stdout output during development to verify that events are being routed as expected. It's easy to make mistakes with wildcard patterns, and seeing the actual output helps catch routing errors early.
Document your routing patterns - Complex routing configurations can be difficult to understand months later. Add comments explaining your tag structure and routing logic to help future maintainers.

This covers the essential routing patterns that developers need to master when building Fluent Bit telemetry pipelines for their applications.

More in the series

In this article you learned how to implement simple and more sophisticated routing patterns in Fluent Bit using tags, wildcards, and content-based routing. This article is based on this online free workshop, see the lab for advanced telemetry routing.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, exploring Fluent Bit routing metrics telemetry data to Prometheus.

Mastering Fluent Bit: Top 3 Telemetry Pipeline Filters for Developers

eric@schabell.org (eric@schabell.org) — Wed, 26 Nov 2025 07:00:00 +0100

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, using Fluent Bit filters for developers. In case you missed the previous article, check out three tips for using telemetry pipeline multiline parsers where you explore how to handle complex multiline log messages.

This article will be a hands-on exploration of filters that help you as a developer testing out your Fluent Bit pipelines. We'll take a look at the top three filters you'll want to know about when building your telemetry pipeline configurations in Fluent Bit.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Where to get started

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.0.8 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's look at the top three filters that will help you with your local development testing of Fluent Bit pipelines.

Filtering in a telemetry pipeline

See this article for details about the service section of the configurations used in the rest of this article, but for now we plan to focus on our Fluent Bit pipeline and specifically the filters that can be of great help in managing our telemetry data during testing in our inner developer loop.

Below in the figure you see the phases of a telemetry pipeline. The third phase is filter, which is where we can modify, enrich, or drop records based on specific criteria.

Filters in Fluent Bit are powerful tools that operate on records after they've been parsed but before they reach their destination. Unlike processors that work on raw data streams, filters work on structured records, giving you the ability to manipulate individual fields, add metadata, remove sensitive information, or exclude records entirely based on conditions.

In production environments you need full control of the data you're collecting. Filtering lets you alter the collected data before delivering it to a destination. Each available filter can be used to match, exclude, or enrich your logs with specific metadata. Fluent Bit supports many filters, and understanding the most useful ones will dramatically improve your development experience.

Now let's look at the most interesting filters that developers will want to know more about.

1. Modify filter

One of the most versatile filters for telemetry pipelines that developers will encounter is the Modify filter. The Modify filter allows you to change records using rules and conditions, giving you the power to add new fields, rename existing ones, remove unwanted data, and conditionally manipulate your telemetry based on specific criteria.

To provide an example we start by creating a test configuration file called fluent-bit.yaml that demonstrates the Modify filter's capabilities:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs

      dummy: '{"environment":"dev","level":"info","message":"Application started","memory_mb":512}'

  filters:
    - name: modify
      match: '*'
      add:
        - service_name my-application
        - version 1.2.3
        - processed true
      rename:
        - environment env
        - memory_mb mem_usage
      remove:
        - level

  outputs:
    - name: stdout
      match: '*'
      format: json_lines

Our configuration uses the modify filter with several different operations. The add operation inserts new fields into the record. This is extremely useful for adding metadata that your observability backend expects, such as service names, versions, or deployment information. The rename operation changes field names to match your preferred naming conventions or to comply with backend requirements. The remove operation strips out fields you don't want to send to your destination, which can reduce storage costs and improve query performance.

Let's run this configuration to see the Modify filter in action:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-05 14:23:45.678901","env":"dev","message":"Application started","mem_usage":512,"service_name":"my-application","version":"1.2.3","processed":"true"}
{"date":"2025-12-05 14:23:46.789012","env":"dev","message":"Application started","mem_usage":512,"service_name":"my-application","version":"1.2.3","processed":"true"}
...

Notice how the output has been transformed?

The original environment field is now env, memory_mb is now mem_usage, the level field has been removed entirely, and we've added three new fields: service_name, version, and processed. This kind of transformation is essential when you're working with multiple services that produce logs in different formats but need to be standardized before sending to your observability backend.

The Modify filter also supports conditional operations using the Condition parameter. This allows you to apply modifications only when specific criteria are met. Let's extend our example to demonstrate conditional modifications:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"environment":"production","level":"error","message":"Database connection failed","response_time":5000}'

    - name: dummy
      tag: app.logs
      dummy: '{"environment":"dev","level":"info","message":"Request processed","response_time":150}'

  filters:
    - name: modify
      match: '*'
      condition:
        - key_value_equals environment production
      add:
        - priority high
        - alert true

    - name: modify
      match: '*'
      condition:
        - key_value_equals level error
      add:
        - severity critical


  outputs:
    - name: stdout
      match: '*'
      format: json_lines

Let's run this configuration:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-05 14:30:12.345678","environment":"production","level":"error","message":"Database connection failed","response_time":5000,"priority":"high","alert":"true","severity":"critical"}
{"date":"2025-12-05 14:30:13.456789","environment":"dev","level":"info","message":"Request processed","response_time":150}
...

The first record matches both conditions (production environment AND error level), so it gets priority, alert, and severity fields added. The second record doesn't match any conditions, so it passes through unchanged.

This conditional logic is incredibly powerful for implementing routing rules, prioritizing certain types of logs, or adding context based on the content of your telemetry data.

2. Grep filter

Another essential filter that developers need in their telemetry toolkit is the Grep filter. The Grep filter allows you to match or exclude specific records based on regular expression patterns, giving you fine-grained control over which events flow through your pipeline. This is particularly useful during development when you want to focus on specific types of logs or exclude noisy events that aren't relevant to your current debugging session.

To demonstrate the power of the Grep filter, let's create a configuration that filters application logs:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"level":"DEBUG","message":"Processing request 12345","service":"api"}'

    - name: dummy
      tag: app.logs
      dummy: '{"level":"ERROR","message":"Failed to connect to database","service":"api"}'

    - name: dummy
      tag: app.logs
      dummy: '{"level":"INFO","message":"Request completed successfully","service":"api"}'

    - name: dummy
      tag: app.logs
      dummy: '{"level":"WARN","message":"High memory usage detected","service":"api"}'

  filters:
    - name: grep
      match: '*'
      regex:
        - level ERROR|WARN


  outputs:
    - name: stdout
      match: '*'
      format: json_lines

Our configuration uses the grep filter with a regex parameter to keep only records where the level field matches either ERROR or WARN. This kind of filtering is invaluable when you're troubleshooting production issues and need to focus on problematic events while ignoring routine informational logs.

Let's run this configuration:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-05 15:10:23.456789","level":"ERROR","message":"Failed to connect to database","service":"api"}
{"date":"2025-12-05 15:10:24.567890","level":"WARN","message":"High memory usage detected","service":"api"}
...

Notice that only the ERROR and WARN level logs appear in the output. The DEBUG and INFO logs have been filtered out completely. This dramatically reduces the volume of logs you need to process during development and testing.

The Grep filter also supports excluding records using the exclude parameter. Let's modify our configuration to demonstrate this:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"message":"User login successful","user":"alice@example.com"}'

    - name: dummy
      tag: app.logs
      dummy: '{"message":"Health check passed","endpoint":"/health"}'

    - name: dummy
      tag: app.logs
      dummy: '{"message":"Database query executed","query":"SELECT * FROM users"}'

    - name: dummy
      tag: app.logs
      dummy: '{"message":"Metrics endpoint called","endpoint":"/metrics"}'

  filters:
    - name: grep
      match: '*'
      exclude:
        - message /health|/metrics


  outputs:
    - name: stdout
      match: '*'
      format: json_lines

Let's run this updated configuration:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-05 15:20:34.567890","message":"User login successful","user":"alice@example.com"}
{"date":"2025-12-05 15:20:35.678901","message":"Database query executed","query":"SELECT * FROM users"}
...

The health check and metrics endpoint logs have been excluded from the output. This is extremely useful for filtering out routine monitoring traffic that generates high volumes of logs but provides little value during debugging. By combining regex to include specific patterns and exclude to filter out unwanted patterns, you can create sophisticated filtering rules that give you exactly the logs you need.

An important note about the Grep filter is that it supports matching nested fields using the record accessor format. For example, if you have JSON logs with nested structures like {"kubernetes":{"pod_name":"my-app-123"}}, you can use $kubernetes['pod_name'] as the key to match against nested values.

3. Record Modifier filter

The third essential filter for developers is the Record Modifier filter. While the Modify filter focuses on adding, renaming, and removing fields using static values, the Record Modifier filter excels at appending fields with dynamic values such as environment variables and removing or allowing specific keys using pattern matching. This makes it ideal for injecting runtime context into your logs.

Let's create a configuration that demonstrates the Record Modifier filter:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"message":"Application event","request_id":"req-12345","response_time":250,"internal_debug":"sensitive data","trace_id":"trace-abc"}'

  filters:
    - name: record_modifier
      match: '*'
      record:
        - hostname ${HOSTNAME}
        - pod_name ${POD_NAME}
        - namespace ${NAMESPACE}
      remove_key:
        - internal_debug


  outputs:
    - name: stdout
      match: '*'
      format: json_lines

Our configuration uses the record_modifier filter with several powerful features. The record parameter adds new fields with values from environment variables. This is incredibly useful in containerized environments where hostname, pod names, and namespace information are available as environment variables but need to be injected into your logs for proper correlation and filtering in your observability backend. The remove_key parameter strips out sensitive fields that shouldn't be sent to your logging destination.

Let's run this configuration with some environment variables set:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm \
  -e HOSTNAME=dev-server-01 \
  -e POD_NAME=my-app-pod-abc123 \
  -e NAMESPACE=production \
  fb

...
{"date":"2025-12-05 16:15:45.678901","message":"Application event","request_id":"req-12345","response_time":250,"trace_id":"trace-abc","hostname":"dev-server-01","pod_name":"my-app-pod-abc123","namespace":"production"}
...

Notice how the environment variables have been injected into the log record, and the internal_debug field has been removed. This pattern is essential for enriching your logs with contextual information that helps you understand where the logs originated in your distributed system.

The Record Modifier filter also supports the allowlist_key parameter (and its legacy alias whitelist_key), which works inversely to remove_key. Instead of specifying which fields to remove, you specify which fields to keep, and all others are removed:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"message":"User action","user_id":"12345","email":"user@example.com","password_hash":"abc123","session_token":"xyz789","action":"login","timestamp":"2025-12-05T16:20:00Z"}'

  filters:
    - name: record_modifier
      match: '*'
      allowlist_key:
        - message
        - user_id
        - action
        - timestamp


  outputs:
    - name: stdout
      match: '*'
      format: json_lines

Let's run this configuration:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-12-05 16:20:01.234567","message":"User action","user_id":"12345","action":"login","timestamp":"2025-12-05T16:20:00Z"}
...

The sensitive fields (email, password_hash, session_token) have been completely stripped out, leaving only the allowlisted fields. This approach is particularly useful when you're dealing with logs that might contain sensitive information and you want to take a cautious approach by explicitly defining what's safe to send to your logging backend.

Another powerful feature of the Record Modifier filter is the ability to generate UUIDs for each record. This is invaluable for tracking and correlating individual log entries across your distributed system:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"message":"Processing request","service":"api"}'

  filters:
    - name: record_modifier
      match: '*'
      uuid_key: event_id


  outputs:
    - name: stdout
      match: '*'
      format: json_lines

When you run this configuration, each record will have a unique event_id field added automatically, making it easy to reference specific log entries in your observability tools.

This covers the top three filters for developers getting started with Fluent Bit while trying to transform and filter their telemetry data effectively and speed up their inner development loop.

More in the series

In this article you learned about three powerful Fluent Bit filters that improve the inner developer loop experience. This article is based on this online free workshop.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, exploring Fluent Bit routing as there are new ways to for developers to leverage this feature.

Mastering Fluent Bit: 3 Tips for Telemetry Pipeline Multiline Parsers for Developers

eric@schabell.org (eric@schabell.org) — Thu, 20 Nov 2025 07:00:00 +0100

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, using Fluent Bit multiline parsers for developers. In case you missed the previous article, check out using telemetry pipeline processors where you explore the top three telemetry data processors for developers.

This article will be a dive into parsers that help developer testing of Fluent Bit pipelines when dealing with difficult and long multiline log messages. We'll take a look at using multiline parsers for your telemetry pipeline configuration in Fluent Bit.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Where to get started

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.0.8 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's look at the three tips for multiline parsers and how they help you manage complex log entries during your local development testing.

Multiline parsing in a telemetry pipeline

See this article for details about the service section of the configurations used in the rest of this article, but for now we plan to focus on our Fluent Bit pipeline and specifically the multiline parsers that can be of great help in managing our telemetry data during testing in our inner developer loop.

Below in the figure you see the phases of a telemetry pipeline. The second phase is parser, which is where unstructured input data is turned into structured data.

Note that in this article we explore Fluent Bit using multiline parsers that we can configure to process data in the input of our telemetry pipeline, but this is shown here as a separate phase.

The challenge developers often face is that real-world applications don't always log messages on a single line. Stack traces, error messages, and debug output frequently span multiple lines. These multiline messages need to be concatenated before they can be properly parsed and processed.

Fluent Bit provides multiline parsers to solve this exact problem. A multiline parser can recognize when multiple lines of log data belong together and concatenate them into a single event before further processing.

An example of multiline log data that developers encounter daily would be a Java stack trace:

Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!

  at com.myproject.module.MyProject.badMethod(MyProject.java:22)
  at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
  at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
  at com.myproject.module.MyProject.someMethod(MyProject.java:10)
  at com.myproject.module.MyProject.main(MyProject.java:6)

Without multiline parsing, each line would be treated as a separate log entry. With multiline parsing, all these lines are correctly concatenated into a single structured event that maintains the complete context of the error.

The Fluent Bit multiline parser engine exposes two ways to configure the feature:

Built-in multiline parsers
Configurable multiline parsers

Fluent Bit provides pre-configured built-in parsers for common use cases such as:

docker - Process log entries generated by Docker container engine
cri - Process log entries generated by CRI-O container engine
go - Process log entries from Go applications
python - Process log entries from Python applications
ruby - Process log entries from Ruby applications
java - Process log entries from Java applications

For cases where the built-in parsers don't fit your needs, you can define custom multiline parsers. These custom parsers use regular expressions and state machines to identify the start and continuation of multiline messages. Let's look at how to configure a custom multiline parser that developers will want to know more about.

Now let's look at the most interesting tips for multiline parsers that developers will want to know more about.

1. Configurable multiline parser

One of the more common use cases for telemetry pipelines that developers will encounter is dealing with stack traces and error messages that span multiple lines. These multiline messages need special handling to ensure they are concatenated properly before being sent to their destination.

To provide an example we start by creating a test log file called test.log with multiline Java stack trace data:

single line...
Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)
another line...

Next, let's create a multiline parser configuration. We create a new file called parsers_multiline.yaml in our favorite editor and add the following configuration:

parsers:

  - name: multiline-regex-test
    type: regex
    flush_timeout: 1000
    rules:
      - state: start_state
        regex: '/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/'
        next_state: cont
      - state: cont
        regex: '/^\s+at.*/'
        next_state: cont

Let's break down what this multiline parser does:

name - We give our parser a unique name multiline-regex-test
type - We specify the type as regex for regular expression based parsing
flush_timeout - After 1000ms of no new matching lines, the buffer is flushed
rules - We define the state machine rules that control multiline detection

The rules section is where the magic happens. A multiline parser uses states to determine which lines belong together:

The start_state rule matches lines that begin a new multiline message. In our case, the pattern matches a timestamp followed by any text, which identifies the first line of our Java exception.
The cont (continuation) rule matches lines that are part of the multiline message. Our pattern matches lines starting with whitespace followed by "at", which identifies the stack trace lines.
Each rule specifies a next_state which tells Fluent Bit what state to transition to after matching. This creates a state machine that can handle complex multiline patterns.

When the parser sees a line matching start_state, it begins a new multiline buffer. Any subsequent lines matching the cont pattern are appended to that buffer. When a line doesn't match either pattern, or when the flush timeout expires, the complete multiline message is emitted as a single event.

Now let's create our main Fluent Bit configuration file fluent-bit.yaml that uses this multiline parser:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on
  parsers_file: parsers_multiline.yaml

pipeline:
  inputs:
    - name: tail
      path: test.log
      read_from_head: true
      multiline.parser: multiline-regex-test

  outputs:
    - name: stdout
      match: '*'

Note several important configuration points here:

We include the parsers_file in the service section to load our multiline parser definitions
We use the tail input plugin to read from our test log file
We set read_from_head: true to read the entire file from the beginning
Most importantly, we specify multiline.parser: multiline-regex-test to apply our multiline parser

The multiline parser is applied at the input stage, which is the recommended approach. This ensures that lines are concatenated before any other processing occurs.

Let's run this configuration to see the multiline parser in action:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
[0] tail.0: [[1750332967.679671000, {}], {"log"=>"single line...
"}]
[1] tail.0: [[1750332967.679677000, {}], {"log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)
"}]
[2] tail.0: [[1750332967.679677000, {}], {"log"=>"another line...
...

Notice how the output shows three distinct events:

The single line message passes through unchanged
The entire stack trace is concatenated into one event, preserving the complete error context
The final single line message passes through unchanged

This is exactly what we want. The multiline parser successfully identified the start of the Java exception and concatenated all the stack trace lines into a single structured event.

2. Extracting structured data from multiline messages

Once you have your multiline messages properly concatenated, you'll often want to extract specific fields from them. Fluent Bit supports this through the parser filter, which can be applied after multiline parsing.

Let's extend our example to extract the date and message components from the concatenated stack trace. First, we'll add a regular expression parser to our parsers_multiline.yaml file:

parsers:

  - name: multiline-regex-test
    type: regex
    flush_timeout: 1000
    rules:
      - state: start_state
        regex: '/([a-zA-Z]+ \d+ \d+\:\d+\:\d+)(.*)/'
        next_state: cont
      - state: cont
        regex: '/^\s+at.*/'
        next_state: cont

  - name: named-capture-test
    format: regex
    regex: '/^(?<date>[a-zA-Z]+ \d+ \d+\:\d+\:\d+)\s+(?<message>(.|\n)*)$/m'

The new named-capture-test parser uses named capture groups to extract:

date - The timestamp at the start of the message
message - The remaining content including all newlines

Note the /m modifier at the end of the regex, which enables multiline mode where . (dot) can match newline characters.

Now we update our main configuration to apply this parser using the parser filter:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on
  parsers_file: parsers_multiline.yaml

pipeline:
  inputs:
    - name: tail
      path: test.log
      read_from_head: true
      multiline.parser: multiline-regex-test

  filters:
    - name: parser
      match: '*'
      key_name: log
      parser: named-capture-test


  outputs:
    - name: stdout
      match: '*'

We've added a parser filter that:

Matches all events with match: '*'
Looks at the log field with key_name: log
Applies the named-capture-test parser to extract structured fields

Running this enhanced configuration produces:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
[0] tail.0: [[1750333602.460984000, {}], {"log"=>"single line...
"}]
[1] tail.0: [[1750333602.460998000, {}], {"date"=>"Dec 14 06:41:08", "message"=>"Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)
"}]
[2] tail.0: [[1750333602.460998000, {}], {"log"=>"another line...
"}]
...

Now the multiline Java exception event contains structured fields:

date contains the timestamp
message contains the complete exception and stack trace

This structured format makes it much easier to query, analyze, and alert on these error events in your observability backend.

3. Important considerations for multiline parsers

When working with multiline parsers, keep these important points in mind:

Apply multiline parsing at the input stage - While you can apply multiline parsing using the multiline filter, the recommended approach is to configure it directly on the input plugin using multiline.parser. This ensures lines are concatenated before any other processing.

Understand flush timeout behavior - The flush_timeout parameter determines how long Fluent Bit waits for additional matching lines before emitting the multiline buffer. Set this value based on your application's logging patterns. Too short and you might break up valid multiline messages. Too long and you'll introduce unnecessary latency.

Use specific state patterns - Make your regular expressions as specific as possible to avoid false matches. The start_state pattern should uniquely identify the beginning of a multiline message, and continuation patterns should only match valid continuation lines.

Be aware of resource implications - Multiline parsers buffer lines in memory until the complete message is ready. For applications with very large multiline messages (like huge stack traces), this can consume significant memory. The multiline parser bypasses the buffer_max_size limit to ensure complete messages are captured.

Test with real data - Always test your multiline parser configurations with actual log data from your applications. Edge cases in log formatting can cause unexpected parsing behavior.

This covers the three tips for developers getting started with Fluent Bit multiline parsers while trying to handle complex multiline log messages and speed up their inner development loop.

More in the series

In this article you learned how to use Fluent Bit multiline parsers to properly handle log messages that span multiple lines. This article is based on this online free workshop.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, exploring some of the more interesting Fluent Bit filters for developers.

Mastering Fluent Bit: Top 3 Telemetry Pipeline Processors for Developers

eric@schabell.org (eric@schabell.org) — Thu, 13 Nov 2025 07:00:00 +0100

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, using Fluent Bit processors for developers. In case you missed the previous article, check out the top tips on using telemetry pipeline parsers for developers where you get tips on cleaning up your telemetry data for better developer experiences.

This article will be a hands-on tour of the things that help you as a developer testing out your Fluent Bit pipelines. We'll take a look at the top 3 processors you'll want to know about when building your telemetry pipeline configurations in Fluent Bit.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Where to get started

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.0.8 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's look at the top three processors that will help you with your local development testing of Fluent Bit pipelines.

Processing in a telemetry pipeline

See this article for details about the service section of the configurations used in the rest of this article, but for now we plan to focus on our Fluent Bit pipeline and specifically the processors that can be of great help in managing our telemetry data during testing in our inner developer loop.

Processors in Fluent Bit are powerful components that sit between the input and output phases of your telemetry pipeline. They allow you to manipulate, transform, and enrich your telemetry data before it reaches its destination. Unlike filters which operate on records, processors work on the raw data stream level, giving you fine-grained control over how your data flows through the pipeline.

The processor phase happens after data is ingested but before it's formatted for output. This makes processors ideal for operations that need to happen at scale across your entire data stream, such as content modification, metrics extraction, and data aggregation.

Keeping all of this in mind, let's look at the most interesting processors that developers will want to know more about.

1. Content Modifier processor

One of the most common use cases for telemetry pipelines that developers will encounter is the need to add, modify, or remove fields from your telemetry data. The Content Modifier processor gives you the ability to manipulate the structure and content of your events as they flow through the pipeline.

To provide an example we start with a simple Fluent Bit configuration file fluent-bit.yaml containing a configuration using the dummy plugin to generate events that we'll then modify:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"environment":"dev","message":"Application started"}'
      
  processors:
    logs:
      - name: content_modifier
        action: insert
        key: pipeline_version
        value: "1.0.0"
        
      - name: content_modifier
        action: insert
        key: processed_timestamp
        value: "${HOSTNAME}"
        
      - name: content_modifier
        action: rename
        renamed_key: env
        key: environment
        
  outputs:
    - name: stdout
      match: '*'
      format: json_lines
      json_date_format: java_sql_timestamp

Our configuration uses the content_modifier processor three times to demonstrate different actions. First, we insert a new field called pipeline_version with a static value. Second, we insert a processed_timestamp field that references an environment variable. Third, we rename the environment field to env for consistency.

Let's run this to confirm our working test environment:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-10-26 20:45:12.123456","env":"dev","message":"Application started","pipeline_version":"1.0.0","processed_timestamp":"localhost"}
{"date":"2025-10-26 20:45:13.234567","env":"dev","message":"Application started","pipeline_version":"1.0.0","processed_timestamp":"localhost"}
...

Note how each event now contains the additional fields we configured, and the original environment field has been renamed to env. This processor is invaluable for standardizing your telemetry data before it reaches your backend systems.

2. Metrics Selector processor

Another critical use case for developers working with telemetry data is the ability to extract and select specific metrics from your event streams. The Metrics Selector processor allows you to filter and route metrics based on their labels and values, giving you precise control over which metrics flow to which destinations.

To demonstrate this we'll create a configuration that generates different types of metrics and uses the metrics selector to route them appropriately:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: metrics.cpu
      dummy: '{"metric":"cpu_usage","value":75.5,"host":"server01","env":"production"}'
      
    - name: dummy
      tag: metrics.memory
      dummy: '{"metric":"memory_usage","value":82.3,"host":"server01","env":"production"}'
      
    - name: dummy
      tag: metrics.disk
      dummy: '{"metric":"disk_usage","value":45.2,"host":"server02","env":"staging"}'
      
  processors:
    logs:
      - name: metrics_selector
        metric_name: cpu_usage
        action: include
        label: env
        operation_type: prefix_match
        match: prod
        
  outputs:
    - name: stdout
      match: 'metrics.cpu'
      format: json_lines
      json_date_format: java_sql_timestamp
      
    - name: stdout
      match: 'metrics.*'
      format: json_lines
      json_date_format: java_sql_timestamp

Our configuration generates three different metric types and uses the metrics_selector processor to filter CPU metrics that match production environments. This allows you to create sophisticated routing rules based on your metric characteristics.

Let's run this configuration:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-10-26 21:10:33.456789","metric":"cpu_usage","value":75.5,"host":"server01","env":"production"}
{"date":"2025-10-26 21:10:33.567890","metric":"memory_usage","value":82.3,"host":"server01","env":"production"}
{"date":"2025-10-26 21:10:33.678901","metric":"disk_usage","value":45.2,"host":"server02","env":"staging"}
...

The metrics selector processor helps you focus on the metrics that matter most during development and testing, reducing noise and improving the signal-to-noise ratio in your telemetry data.

3. OpenTelemetry Envelope processor

The third essential processor that developers need to understand is the OpenTelemetry Envelope processor. This processor transforms your Fluent Bit telemetry data into the OpenTelemetry protocol format, enabling seamless integration with the broader OpenTelemetry ecosystem. As organizations increasingly adopt OpenTelemetry as their standard for observability data, this processor becomes critical for ensuring your Fluent Bit pipelines can communicate effectively with OpenTelemetry collectors and backends.

The OpenTelemetry Envelope processor wraps your telemetry data in the standard OpenTelemetry format, preserving all the semantic conventions and structures that make OpenTelemetry powerful. This includes proper handling of resource attributes, instrumentation scope, and the telemetry signal types that are core to OpenTelemetry.

For comprehensive coverage of integrating Fluent Bit with OpenTelemetry, I highly recommend exploring these detailed articles:

Telemetry Pipelines: Integrating Fluent Bit with OpenTelemetry, Part 1 - This article covers the fundamentals of integrating Fluent Bit with OpenTelemetry, including configuration patterns and best practices for getting started.

Integrating Fluent Bit with OpenTelemetry, Part 2 - This follow-up article dives deeper into advanced integration scenarios, troubleshooting tips, and real-world use cases for production deployments.

To demonstrate how the OpenTelemetry Envelope processor works, let's create a configuration that wraps application logs in OpenTelemetry format:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: dummy
      tag: app.logs
      dummy: '{"level":"info","service":"user-api","message":"User login successful","user_id":"12345"}'
      
    - name: dummy
      tag: app.logs
      dummy: '{"level":"error","service":"payment-api","message":"Payment processing failed","transaction_id":"tx-9876"}'
      
  processors:
    logs:
      - name: opentelemetry_envelope
        resource:
          service_name: my-application
          service_version: 1.2.3
          deployment_environment: production
        instrumentation_scope:
          name: fluent-bit
          version: 4.2.0
        
  outputs:
    - name: stdout
      match: '*'
      format: json_lines
      json_date_format: java_sql_timestamp

Our configuration uses the opentelemetry_envelope processor to wrap each log entry with OpenTelemetry metadata. The resource section adds attributes that describe the source of the telemetry data, such as the service name and deployment environment. The instrumentation_scope section identifies the tool that collected the data, which is essential for proper attribution in OpenTelemetry systems.

Let's run this configuration to see the OpenTelemetry envelope in action:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-10-26 22:15:30.123456","resource":{"service_name":"my-application","service_version":"1.2.3","deployment_environment":"production"},"instrumentation_scope":{"name":"fluent-bit","version":"4.1.0"},"level":"info","service":"user-api","message":"User login successful","user_id":"12345"}
{"date":"2025-10-26 22:15:31.234567","resource":{"service_name":"my-application","service_version":"1.2.3","deployment_environment":"production"},"instrumentation_scope":{"name":"fluent-bit","version":"4.1.0"},"level":"error","service":"payment-api","message":"Payment processing failed","transaction_id":"tx-9876"}
...

Notice how each log entry now includes the OpenTelemetry resource attributes and instrumentation scope information. This standardized format ensures that when your telemetry data reaches an OpenTelemetry collector or backend, it will be properly categorized and can be correlated with other telemetry signals like traces and metrics from your distributed system.

This covers the top 3 processors for developers getting started with Fluent Bit while trying to leverage processors to transform and enrich their telemetry data quickly and speed up their inner development loop.

More in the series

In this article you learned about three powerful Fluent Bit processors that improve the inner developer loop experience. This article is based on this online free workshop.

Cloud Native Rejekts NA 2025 - Expanding Your Toolbox: Beginners Guide to Controlling Kubernetes Logs (slides)

eric@schabell.org (eric@schabell.org) — Sat, 8 Nov 2025 18:00:00 +0100

Today I spoke at the always fun event taking place before KubeCon + CloudNativeCon NA, Cloud Native Rejekts.

Since my transfer from the application development domains to the cloud native observability domain more than three years ago, I've never missed a single CFP for this event. This year it was in another amazing location, The Academy of Medicine in Atlanta, GA.

In the past I've shared observability topics in a session on Perses, a session providing OpenTelemetry insights, and a session on integrating Fluent Bit with OpenTelemetry including a live demo. Today I told a cloud native story helping developers gain control of their logs in Kubernetes environments while within their inner development loop.

Below you will find the session slides, abstract, and location details for this Atlanta based event.

My session focused on Kubernetes logging and how to get some sort of control back as a developer working on applications and just wanting to zoom in on the application they are deploying. Here are the slides:

And the abstract for the talk:

Expanding Your Toolbox: Beginners Guide to Controlling Kubernetes Logs

Drowning in logs from your Kubernetes clusters? Struggling to scale observability without overwhelming your telemetry systems? You're not alone—and there's a better way. In this talk, you’ll learn how to efficiently manage and streamline logging data from source to destination using telemetry pipelines.

We’ll walk through the key stages of a modern telemetry pipeline—collection, parsing, filtering, routing, and forwarding—demonstrating how to build powerful, flexible pipelines that can handle logs from any source to any destination. Along the way, you’ll see a live demo in a real Kubernetes environment, where we’ll deploy your first telemetry pipeline tailored to a real-world use case.

Whether you're debugging production issues, operating multi-tenant clusters, or just trying to cut through the noise, this session will give you the tools and patterns you need to simplify and scale log collection. Plus, you’ll get access to a self-paced, hands-on workshop to continue exploring after the session: o11y-workshops.gitlab.io/workshop-fluentbit.

This talk includes a 15-minute live demo showcasing key integration phases:

Deploying a real workload: Start with a Kubernetes cluster running a real application (a CMS) that generates log data.
Installing a telemetry pipeline: Deploy a telemetry pipeline to the cluster to begin streaming logs from all containers.
Streaming to an output: Route collected logs to an external destination for analysis or storage.
Optimizing log volume: Refine the pipeline to filter out noisy or unnecessary logs—reducing telemetry costs and improving signal-to-noise ratio.
Filtering logs: Enrich logs with metadata, isolate error-level telemetry, and ensure only the necessary logs (e.g., error logs from the CMS) are exported securely from the cluster.

Date: 8 Nov 2025

Time: 14:35–15:05

Room: Crystal Dining Room

They recorded all sessions, so watch for it to publish online. Thanks to all who came to the talk and I enjoyed sharing real fun hands-on developer learning with Kubernetes and Fluent Bit.

Mastering Fluent Bit: Controlling Logs with Fluent Bit on Kubernetes (updated)

eric@schabell.org (eric@schabell.org) — Tue, 4 Nov 2025 13:00:00 +0100

This is a special update to the original Controlling Logs with Fluent Bit on Kubernetes article published previously that is no longer valid.

The issue requiring this update arose over last weekend when I discovered that Broadcom, who acquired VMWare, who were the custodians of the bitnami catalog, did something not so nice to all of us.

In this issue, they announced that the catalog would be moved in August 2025 to a legacy catalog in favor of a 'secure' catalog and no longer updated.

This meant that the current Logs Control Easy Install project, the workshop Controlling Logs with Fluent Bit on Kubernetes, and the original article had to all be updated. I spent this last weekend and most of this week bringing all three of these artifacts up to date with a new application workload to replace the use of Ghost CMS.

Let's take a look at what these changes meant to the original article and provide you with a new guide, using Fluent Bit to get control of logs on a Kubernetes cluster.

As with all articles in this series, I'm providing a short introduction to Fluent Bit before sharing how to use a Fluent Bit telemetry pipeline on a Kubernetes cluster to take control of all the logs being generated.

What is Fluent Bit?

Before diving into Fluent Bit, let's step back and look at the position of this project within the Fluent organization. If we look at the Fluent organization on Github, we find Fluentd and Fluent Bit projects hosted there. The back story is that the project started with log parsing project Fluentd joining the CNCF in 2026 and reaching Graduated status in 2019.

Once it became apparent that the world was heading into cloud native Kubernetes environments, the solution was not designed for the flexible and light weight requirements that Kubernetes solutions demanded. Fluent bit was born from the need to have a low resource using, high throughput, and highly scalable log management solution for cloud native Kubernetes environments. The project was started within the Fluent organization as a sub-project in 2017 and the rest is now 10 years of history with the release of v4 last week!

Fluent Bit has become so much more than a flexible and lightweight log pipeline solution, now able to process metrics and traces and becoming a telemetry pipeline collection tool of choice for those looking to put the control over their telemetry data right at the source where it's being collected.

Let's get started with Fluent Bit and see what we can do for ourselves!

Why control logs on a Kubernetes cluster?

When you dive into the cloud native world this means you are deploying containers on Kubernetes. The complexities increase dramatically as your applications and microservices interact in this complex and dynamic infrastructure landscape.

Deployments can auto-scale, pods spin up and are taken down as the need arises, and underlying all of this are the various Kubernetes controlling components. All of these things are generating telemetry data and Fluent Bit is a wonderfully simple way to take control of them across a Kubernetes cluster. It provides a way of collecting everything through a central telemetry pipeline as you go while providing the ability to parse, filter, and route all your telemetry data.

For developers, this article will demonstrate using Fluent Bit as a single point of log collection on a development Kubernetes cluster with a deployed workload. Finally, all examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines

Where to get started

To ensure you are ready to start controlling your Kubernetes cluster logs, the rest of this article assumes you have completed this previous article. This ensures you are running a two node Kubernetes cluster with a workload running in the form of Ghost CMS and Fluent Bit is installed to collect all container logs.

If you did not work through the previous article, I've provided a Logs Control Easy Install project repository that you can download, unzip, and run with one command to spin up the Kubernetes cluster with the above setup on your local machine.

Using either path, once set up you are able to see the logs from Fluent Bit containing everything generated on this running cluster. This would be the logs across two namespaces; kube-system and logging.

You can verify that they are up and running by browsing those namespaces, shown here on my local machine:

$ kubectl --kubeconfig target/logdemoconfig.yaml get pods --namespace kube-system

NAME                                           READY   STATUS   RESTARTS  AGE
coredns-668d6bf9bc-2skfg                       1/1     Running  0         2m43s
coredns-668d6bf9bc-79jpn                       1/1     Running  0         2m43s
etcd-logdemo-control-plane                     1/1     Running  0         2m51s
kindnet-7lm5s                                  1/1     Running  0         2m43s
kube-apiserver-logdemo-control-plane           1/1     Running  0         2m50s
kube-controller-manager-logdemo-control-plane  1/1     Running  0         2m50s
kube-proxy-2q9db                               1/1     Running  0         2m43s
kube-scheduler-logdemo-control-plane           1/1     Running  0         2m50s

$ kubectl --kubeconfig target/logdemoconfig.yaml get pods --namespace default

NAME                             READY   STATUS    RESTARTS   AGE
log-generator-7cdff9fcf8-qgw9k   1/1     Running   0          5m4s

$ kubectl --kubeconfig target/logdemoconfig.yaml get pods --namespace logging

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-vk2rf   1/1     Running   0          4m49s

The initial configuration for the Fluent Bit instance is to collect all container logs, from all namespaces, shown in in the fluent-bit-helm.yaml configuration file used in our set up, highlighted in bold below:

args:

  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml

config:
  extraFiles:
    fluent-bit.yaml: |
      service:
       flush: 1
       log_level: info
       http_server: true
       http_listen: 0.0.0.0
       http_port: 2020
      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            read_from_head: true
            path: /var/log/containers/*.log

            multiline.parser: docker, cri
        outputs:
          - name: stdout
            match: '*'

To see all the logs collected, we can dump the Fluent Bit log file as follows, using the pod name we found above:

$ kubectl --kubeconfig target/logdemoconfig.yaml logs fluent-bit-vk2rf --nanmespace logging

[OUTPUT-CUT-DUE-TO-LOG-VOLUME]

...

You will notice if you browse that you have error messages, info messages, if you look hard enough some logs from Kubernetes pods, the application workload, and even your Fluent Bit instance. As a developer working on your cluster, how can you find anything useful in this flood of logging? The good thing is you do have a single place to look for them!

Another point to mention is that by using the Fluent Bit tail input plugin and setting it to read from the beginning of each log file, we have ensured that our log telemetry data is taken from all our logs. If we didn't set this to collect from the beginning of the log file, our telemetry pipeline would miss everything that was generated before the Fluent Bit instance started. This ensures we have the workload startup messages and can test on standard log telemetry events each time we modify our pipeline configuration.

Let's start taking control of our logs and see how we as developers can make some use of the logs data we want to see during our local development testing.

Taking back control

The first thing we can do is to focus our log collection efforts on just the workload we are interested in and in this example we are looking to find problems with our Log Generator application deployment. As you are not interested in the logs from anything happening in the kube-system namespace, you can narrow the focus of your Fluent Bit input plugin to only examine Log Generator log files.

This can be done by making a new configuration file called myfluent-bit-helm.yaml file and change the default path as follows in bold:

args:

  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml

config:
  extraFiles:
    fluent-bit.yaml: |
      service:
       flush: 1
       log_level: info
       http_server: true
       http_listen: 0.0.0.0
       http_port: 2020
      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            read_from_head: true
            path: /var/log/containers/*generator*
            multiline.parser: docker, cri
        outputs:
          - name: stdout
            match: '*'

Next step is to update the Fluent Bit instance with a helm update command as follows:

$ helm upgrade --kubeconfig target/logdemoconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-mzktk   1/1     Running   0          28s

Now explore the logs being collected by Fluent Bit and notice that all the kube-system namespace logs are no longer there and we can focus on our deployed workload.

$ kubectl --kubeconfig target/logdemoconfig.yaml logs fluent-bit-mzktk --nanmespace logging

...
[0] kube.var.log.containers.log-generator-7cdff9fcf8-qgw9k_default_log-generator-f1be04153240cf686ba99566f79d64a0e2496a4b555dd833000e1ff222f73dc8.log: [[1762440627.216814972, {}], {"time"=>"2025-11-06T14:50:27.216814972Z", "stream"=>"stdout", "_p"=>"F", "log"=>"Thu Nov  6 14:50:27 UTC 2025 - This is a log message from the log generator."}]
[0] kube.var.log.containers.log-generator-7cdff9fcf8-qgw9k_default_log-generator-f1be04153240cf686ba99566f79d64a0e2496a4b555dd833000e1ff222f73dc8.log: [[1762440630.220889184, {}], {"time"=>"2025-11-06T14:50:30.220889184Z", "stream"=>"stdout", "_p"=>"F", "log"=>"Thu Nov  6 14:50:30 UTC 2025 - {message:true 200 success}"}]
[0] kube.var.log.containers.log-generator-7cdff9fcf8-qgw9k_default_log-generator-f1be04153240cf686ba99566f79d64a0e2496a4b555dd833000e1ff222f73dc8.log: [[1762440633.226102188, {}], {"time"=>"2025-11-06T14:50:33.226102188Z", "stream"=>"stderr", "_p"=>"F", "log"=>"Thu Nov  6 14:50:33 UTC 2025 - {message:false 500 error}"}]

...

This is just a selection of log lines from the total output. If you look closer you see these logs have their own sort of format, so let's standardize them so that JSON is the output format and make the various timestamps a bit more readable by changing your Fluent Bit output plugin configuration as follows:

args:

  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml

config:
  extraFiles:
    fluent-bit.yaml: |
      service:
       flush: 1
       log_level: info
       http_server: true
       http_listen: 0.0.0.0
       http_port: 2020
      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            read_from_head: true
            path: /var/log/containers/*ghost*
            multiline.parser: docker, cri
        outputs:
          - name: stdout

            match: '*'

            format: json_lines

            json_date_format: java_sql_timestamp

Update the Fluent Bit instance using a helm update command as follows:

$ helm upgrade --kubeconfig target/logdemoconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-gqsc8   1/1     Running   0          42s

Now explore the logs being collected by Fluent Bit and notice the output changes:

$ kubectl --kubeconfig target/logdemoconfig.yaml logs fluent-bit-gqsc8 --nanmespace logging

...

{"date":"2025-11-06 14:51:44.314963","time":"2025-11-06T14:51:44.314963452Z","stream":"stdout","_p":"F","log":"Thu Nov  6 14:51:44 UTC 2025 - This is a log message from the log generator."}
{"date":"2025-11-06 14:51:47.317883","time":"2025-11-06T14:51:47.317883265Z","stream":"stdout","_p":"F","log":"Thu Nov  6 14:51:47 UTC 2025 - {message:true 200 success}"}
{"date":"2025-11-06 14:51:50.321730","time":"2025-11-06T14:51:50.321730204Z","stream":"stderr","_p":"F","log":"Thu Nov  6 14:51:50 UTC 2025 - {message:false 500 error}"}

...

Now if we look closer at the array of messages and being the developer we are, we've noticed a mix of stderr and stdout log lines. Let's take control and trim out all the lines that do not contain stderr, as we are only interested in what is broken.

We need to add a filter section to our Fluent Bit configuration using the grep filter and targeting a regular expression to select the keys stream or stderr as follows:

args:

  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml

config:
  extraFiles:
    fluent-bit.yaml: |
      service:
       flush: 1
       log_level: info
       http_server: true
       http_listen: 0.0.0.0
       http_port: 2020
      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            read_from_head: true
            path: /var/log/containers/*ghost*
            multiline.parser: docker, cri

        filters:
          - name: grep
            match: '*'

            regex: stream stderr
        outputs:
          - name: stdout

            match: '*'

            format: json_lines

            json_date_format: java_sql_timestamp

Update the Fluent Bit instance using a helm update command as follows:

$ helm upgrade --kubeconfig target/logdemoconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-npn8n   1/1     Running   0          12s

Now explore the logs being collected by Fluent Bit and notice the output changes:

$ kubectl --kubeconfig target/logdemoconfig.yaml logs fluent-bit-npn8n --nanmespace logging

...

{"date":"2025-11-06 14:52:56.411711","time":"2025-11-06T14:52:56.411711621Z","stream":"stderr","_p":"F","log":"Thu Nov  6 14:52:56 UTC 2025 - {message:false 500 error}"}
{"date":"2025-11-06 14:53:07.426493","time":"2025-11-06T14:53:07.426493474Z","stream":"stderr","_p":"F","log":"Thu Nov  6 14:53:07 UTC 2025 - {message:false 500 error}"}
{"date":"2025-11-06 14:53:18.441807","time":"2025-11-06T14:53:18.441807795Z","stream":"stderr","_p":"F","log":"Thu Nov  6 14:53:18 UTC 2025 - {message:false 500 error}"}
{"date":"2025-11-06 14:53:29.453509","time":"2025-11-06T14:53:29.453509211Z","stream":"stderr","_p":"F","log":"Thu Nov  6 14:53:29 UTC 2025 - {message:false 500 error}"}
{"date":"2025-11-06 14:53:40.462905","time":"2025-11-06T14:53:40.462905134Z","stream":"stderr","_p":"F","log":"Thu Nov  6 14:53:40 UTC 2025 - {message:false 500 error}"}

...

Now we are no longer seeing standard output log events as our telemetry pipeline is filtering to only show standard error tagged logs!

This exercise has shown how to format and prune our logs using our Fluent Bit telemetry pipeline on a Kubernetes cluster. Now let's look at how to enrich our log telemetry data.

We are going to add tags to every standard error line pointing the on-call developer to the SRE they need to contact. To do this, we expand our filter section of the Fluent Bit configuration using the modify filter and targeting the keys stream or stderr to remove those keys and add two new keys STATUS and ACTION as follows:

args:

  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml

config:
  extraFiles:
    fluent-bit.yaml: |
      service:
       flush: 1
       log_level: info
       http_server: true
       http_listen: 0.0.0.0
       http_port: 2020
      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            read_from_head: true
            path: /var/log/containers/*ghost*
            multiline.parser: docker, cri

        filters:
          - name: grep
            match: '*'

            regex: stream stderr

          - name: modify
            match: '*'
            condition: Key_Value_Equals stream stderr
            remove: stream
            add:
              - STATUS REALLY_BAD
              - ACTION CALL_SRE
        outputs:
          - name: stdout

            match: '*'

            format: json_lines

            json_date_format: java_sql_timestamp

Update the Fluent Bit instance using a helm update command as follows:

$ helm upgrade --kubeconfig target/logdemoconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-ftfs4   1/1     Running   0          32s

Now explore the logs being collected by Fluent Bit and notice the output changes where the stream key is missing and two new ones have been added at the end of each error log event:

$ kubectl --kubeconfig target/logdemoconfig.yaml logs fluent-bit-ftfs4 --nanmespace logging

...

[CUT-LINE-FOR-VIEWING] {message:false 500 error}","STATUS":"REALLY_BAD_ERROR","ACTION":"CALL_SRE"}
[CUT-LINE-FOR-VIEWING] {message:false 500 error}","STATUS":"REALLY_BAD_ERROR","ACTION":"CALL_SRE"}
[CUT-LINE-FOR-VIEWING] {message:false 500 error}","STATUS":"REALLY_BAD_ERROR","ACTION":"CALL_SRE"}
[CUT-LINE-FOR-VIEWING] {message:false 500 error}","STATUS":"REALLY_BAD_ERROR","ACTION":"CALL_SRE"}
[CUT-LINE-FOR-VIEWING] {message:false 500 error}","STATUS":"REALLY_BAD_ERROR","ACTION":"CALL_SRE"}

...

Now we have a running Kubernetes cluster, an application workload generating logs, and using a Fluent Bit telemetry pipeline to gather and take control of our log telemetry data.

Initially we found that gathering all log telemetry data was flooding too much information to be able to sift out the important events for our development needs. We then started taking control back of our log telemetry data by narrowing our collection strategy, by filtering, and finally by enriching our telemetry data.

More in the series

In this article you learned how to use Fluent Bit on a Kubernetes cluster to take control of your telemetry data. This article is based on this (updated) online free workshop.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, a developers guide to Fluent Bit service section configuration.

Mastering Fluent Bit: Top Tip Using Telemetry Pipeline Parsers for Developers

eric@schabell.org (eric@schabell.org) — Tue, 21 Oct 2025 07:00:00 +0200

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit.

Each article in this series addresses a single topic by providing insights into what the topic is, why we are interested in exploring that topic, where to get started with the topic, and how to get hands-on with learning about the topic as it relates to the Fluent Bit project.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, using Fluent Bit tips for developers. In case you missed the previous article, check out the top 3 telemetry pipeline output plugins for developers where you get tips on the best of Fluent Bit for your developer experiences.

This article will be a hands-on tour of the things that help you as a developer testing out your Fluent Bit pipelines. We'll take a look at the top tip when using a parser for your telemetry pipeline configuration in Fluent Bit.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Where to get started

You can verify that you have a functioning installation by testing your Fluent Bit, either using a source installation or a container installation as shown below:

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.0.8 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's look at a few tips and tricks to help you with your local development testing of Fluent Bit input plugins.

Parsing in a telemetry pipeline

See this article for details about the service section of the configurations used in the rest of this article, but for now we plan to focus on our Fluent Bit pipeline and specifically the parsers that can be of great help in managing our telemetry data during testing in our inner developer loop.

Below in the figure you see the phases of a telemetry pipeline. The second phase is parser, which is where unstructured input data is turned into structured data. Fluent Bit does this using Parsers that we can configure to manipulate the unstructured data producing structured data for the next phases of our pipeline.

An example of this can be found in the online workshop, where we see an example of unstructured log data:

192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

When unstructured log data is parsed by Fluent Bit, the results become structured data such as the following:

  "host":    "192.168.2.20",
  "user":    "-",
  "method":  "GET",
  "path":    "/cgi-bin/try/",
  "code":    "200",
  "size":    "3395"
}

The Fluent Bit parser engine is configurable and can process log entries based in two formats:

JSON maps
Regular expressions

By default, Fluent Bit provides a set of pre-configured parsers that can be used for different use cases, such as logs from these formats:

Apache
NGINX
Docker
Syslog rfc5424
Syslog rfc3164

Parsers tend to be defined in configuration files that are loaded at start time in the main Fluent Bit configuration file. We can also load parsers from the command line, but we won't be covering this here. Keeping all of this in mind, let's look at the most interesting parser that developers will want to know more about.

1. Regular expression parser

One of the more common use cases for telemetry pipelines that developers will encounter is having multiple event streams producing data that creates the situation that keys are not unique if parsed without some cleanup. To illustrate how Fluent Bit can easily provide us with a means to both parse and filter events from multiple input sources to clean up any duplicate keys before sending onwards to a destination.

To provide an example we start with a simple Fluent Bit configuration file fluent-bit.yaml containing a configuration using the dummy plugin to generate two types of events, both using the same key to cause confusion if we try querying without cleaning them up first:

service:

  flush: 1
  log_level: info

  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    # This entry generates a successful message.
    - name:  dummy
      tag:   event.success
      dummy: '{"message":"true 200 success"}'

    # This entry generates a failure message.
    - name:  dummy
      tag:   event.error
      dummy: '{"message":"false 500 error"}'

  outputs:
    - name: stdout
      match: '*'

      format: json_lines

      json_date_format: java_sql_timestamp

Our configuration is tagging each successful event with event.success and failure events with event.error. The confusion will be caused by configuring the dummy message with the same key, message, for both event definitions. This will cause our incoming events to be confusing to deal with.

Let's run this to confirm out working test environment:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-10-26 19:59:34.508732","message":"true 200 success"}
{"date":"2025-10-26 19:59:34.508837","message":"false 500 error"}
{"date":"2025-10-26 19:59:35.509396","message":"true 200 success"}
{"date":"2025-10-26 19:59:35.509456","message":"false 500 error"}
{"date":"2025-10-26 19:59:36.508828","message":"true 200 success"}
...

Now we have dirty ingested data coming into our pipeline, showing that we have multiple messages on the same key. To be able to clean this up for usage before passing on to the backend (output), we need to make use of both the Parser and Filter phases.

First, in the Parser phase, where unstructured data is converted into structured data, we'll make use of the built in Regular expression parser plugin to structure the duplicate messages into something more usable. To set up the parser configuration we create a new file called parsers.yaml in our favorite editor. Add the following configuration, where we are defining a parser, naming the parser message_cleaning_parser, selecting the built-in regex parser, and applying the regular expression shown here to convert each message into a structured format (note this actually is applied to incoming messages in the next phase of the telemetry pipeline):

# This parser uses the built-in parser plugin and applies the
# regex to all incoming events.
#
parsers:
  - name: message_cleaning_parser
    format: regex
    regex: '^(?<valid_message>[^ ]+) (?<code>[^ ]+) (?<type>[^ ]+)$'

In the Filter phase is where the previously defined parser is put to the test. To set up the filters configuration we create a new section as shown below and add the following configuration, where we are defining filters, naming a new filter parser, matching all incoming messages to apply this filter, looking for the key message to select the value to be fed into the parser, and applying the parser message_cleaning_parser to it:

service:

  flush: 1
  log_level: info

  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

  parsers_file: parsers.yaml

pipeline:
  inputs:
    # This entry generates a successful message.
    - name:  dummy
      tag:   event.success
      dummy: '{"message":"true 200 success"}'

    # This entry generates a failure message.
    - name:  dummy
      tag:   event.error
      dummy: '{"message":"false 500 error"}'

  filters:
    - name: parser
      match: '*'
      key_name: message
      parser: message_cleaning_parser

  outputs:
    - name: stdout
      match: '*'

      format: json_lines

      json_date_format: java_sql_timestamp

Also note that we have to include the parsers_file by name to ensure our filters can find the parser we defined. Now when we run the configuration we see the following:

# For source installation.
$ fluent-bit --config fluent-bit.yaml

# For container installation after building new image with your 
# configuration using a Buildfile as follows:
#
# FROM ghcr.io/fluent/fluent-bit:4.1.0
# COPY ./fluent-bit.yaml /fluent-bit/etc/fluent-bit.yaml
# COPY ./parsers.yaml /fluent-bit/etc/parsers.yaml
# CMD [ "fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.yaml" ]
#
$ podman build -t fb -f Buildfile

$ podman run --rm fb 

...
{"date":"2025-10-26 20:15:54.233766","valid_message":"true","code":"200","type":"success"}
{"date":"2025-10-26 20:15:54.234199","valid_message":"false","code":"500","type":"error"}
{"date":"2025-10-26 20:15:55.234238","valid_message":"true","code":"200","type":"success"}
{"date":"2025-10-26 20:15:55.234323","valid_message":"false","code":"500","type":"error"}
{"date":"2025-10-26 20:15:56.233915","valid_message":"true","code":"200","type":"success"}
{"date":"2025-10-26 20:15:56.234009","valid_message":"false","code":"500","type":"error"}
...

Note the alternating generated event lines with parsed messages that now contain keys for each field to simplify later querying. The message key has been parsed to show valid_message, solving the confusing use case.

This covers the top tip for developers getting started with Fluent Bit while trying to leverage a parser to clean up their telemetry data quickly and speed up their inner development loop.

More in the series

In this article you learned a few handy tricks for using Fluent Bit output plugins and routing to improve the inner developer loop experience. This article is based on this online free workshop.

All Things Open 2025 - Cloud Metrics: The Cost-Value Equation (slides)

eric@schabell.org (eric@schabell.org) — Tue, 14 Oct 2025 14:00:00 +0200

As I previously mentioned, I missed the 2024 All Things Open, having last spoken in 2023, but I'm back this year at All Things Open in Raleigh, NC!

I'm a big fan of the All Things Open conferences and meetups, as they have been a rock star organization since they started and always put on great events in the state of North Carolina. This year is the 13th edition and I've only missed the events across the pandemic years.

The selection committee was kind enough to select one of my favorite topics that I've been working on within my current team at Chronosphere, cloud metrics and cost-value equations. Today I was on-site and gave this burst-talk session. I wanted to share the slides and my thoughts on the attendee reactions.

First, here are the slides with speaker notes if you download them.

Finally, let's look at the abstract and details of my session.

Cloud Metrics: The Cost-Value Equation

Are you collecting just about every metric under the sun and the kitchen sink too? Understanding the cost of collecting metrics and the usefulness of those metrics is the only way to scale in a cloud native world. You can’t get away with just collecting everything as you grow.

Your observability teams need to make decisions about what to collect, what to drop, what to aggregate, and still be able to alert, triage, remediate, and do their root cause analysis on a daily basis. Gain immediate insights into high cost data (DPPS), when to drop time series data, and how to determine when the value of that data is at its lowest. Session includes a recorded demo video of it in action.

Thanks to all who attended this talk and for all the questions! I look forward to seeing you all in Raleigh again soon!

SRECon Eu 2025 - The Executive Signal (slides)

eric@schabell.org (eric@schabell.org) — Wed, 8 Oct 2025 18:00:00 +0200

"SREcon25 Europe/Middle East/Africa is a gathering of engineers who care deeply about site reliability, systems engineering, and working with complex distributed systems at scale. SREcon strives to challenge both those new to the profession as well as those who have been involved in it for decades. The conference has a culture of critical thought, deep technical insights, continuous improvement, and innovation."

Today I shared some insights into the executive signal and gave some important tips that everyone can use to cut through the noise and drive change. Below I share the slides from my talk.

The following are the slides from my session, which was just 4 mins!

The title and abstract are provided below for completeness:

The Executive Signal: How to Cut Through the Noise and Drive Change

Cloud native teams often live closest to the problems—but their voices are the last to be heard. Despite offering real solutions, many engineers find their concerns ignored by executive decision-makers. The result? Ongoing firefighting, chronic on-call stress, and ultimately, burnout. With developers spending more than 10 hours a week reacting to incidents, faster and smarter executive action isn’t optional—it’s essential.

In this lightning talk, you’ll learn how to bridge the communication gap between technical teams and leadership. We’ll cover three actionable techniques you can start using today to make your message resonate—turning engineering pain into executive priority. If you want to stop spinning in operational chaos and start getting leadership buy-in, this talk will give you the tools to drive real change—without waiting for burnout to make the case for you.

One core topic: This talk will show 3 simple tricks on how to engage effectively with leadership to invoke change.

Thanks to all who joined us on Wed, 8 October for the rapid fire lightning talks!