Planet Puppet

Example 42: Announcing Pabawi, a web frontend for classic infrastructures

2026-01-16T00:00:00+00:00

Is there still space in AI and Cloud native days for a new tool to manage classic infrastructures based on physical or virtual servers? If you handle good old servers, and use Puppet, Bolt or Ansible to configure and manage them, then the answer might be yes, and we may have a nice tool for you. Pabawi is a new, modern, sleek web frontend to inventory your systems, check their configuration management status, and run actions on them. Needless to sat that it’s also Open Source. Version 0.4.0, just released after a few weeks of AI based coding, has integrations with Bolt, Hiera, PuppetDB and PuppetServer. Current version is expected to be executed by a Puppet developer or user on her/his localhost, featuring: Multi-Source Inventory: View and manage nodes from Bolt inventory and PuppetDB Command Execution: Run ad-hoc commands on remote nodes with whitelist security Task Execution: Execute Bolt tasks with automatic parameter support Puppet Integration: Trigger Puppet agent runs with full configuration control Package Management: Install and manage packages across your infrastructure Execution History: Track all operations with detailed results and re-execution capability Dynamic Inventory: Automatically discover nodes from PuppetDB Node Facts: View comprehensive system information from Puppet agents Puppet Reports: Browse detailed Puppet run reports with metrics and resource changes Catalog Inspection: Examine compiled Puppet catalogs and resource relationships Event Tracking: Monitor individual resource changes and failures over time PQL Queries: Filter nodes using PuppetDB Query Language Hiera Data Browser: Explore hierarchical configuration data and key usage analysis Puppet code analysis: Analyses Puppet code base, reporting class usage, lint issues, outdated modules in Puppetfile More robust users management and RBAC, and Ansible and other tools integrations are planned for future versions. Check the git repository for installation instructions and details. Screenshots Here are some screenshots, from a test setup with few nodes. The interface adapts well and is still fast on way larger setups (currently tested with infrastructures up to 350 nodes). Nodes Inventory Node inventory with multi-source support, blazing fast search and filtering options Task Execution Bolt task execution interface with automatic parameters discovery Executions Tracking Execution history with filtering and detailed execution results with re-run capabilities Node Details Node detail page with access to facts, reports, events, managed resources and other useful info Puppet reports Puppet reports view, with run times and number of affected resources Free installation service (time limited) Pabawi is easy to integrate in your existing Puppet / Bolt infrastructure and has contextual setup instructions for each integration, but if you want help in trying it in your infrastructure, example42 offers a launch special, time limited, free installation service, no strings attached. In a shared screen call we can setup Pabawi together, either using its container image or via npm. We want to test and validate it in different conditions, collecting users’ feedback and suggestions. Just contact me on LinkedIN for planning a call: half an hour should be enough to set it up and see how it works. Alessandro Franceschi

R.I.Pienaar - www.devco.net: Choria Hierarchical Data

2025-11-24T23:00:00+00:00

As most are aware, I created the widely used Hiera system in Puppet. I introduced it in 2011, and it has since become essentially the only way to use Puppet in any meaningful fashion. Given its widespread adoption, I donated the code to Puppet, and it became integrated with Puppet core.

Unfortunately, during this integration we lost some key values—the command line and the ability to use it in scripts and elsewhere.

Meanwhile, our world is changing, and we are ever more focussing on small, single purpose compute. I intend to create a new kind of Configuration Management system that is focussed on the small, single, purpose needs. Filling the gap that has always existed - how to manage our applications rather than systems, an area Puppet has always been weak at.

There is then still the need for hierarchical data and given that I have the flexibility to start completely fresh I am at best taking some inspiration from Hiera.

So today I’ll introduce a new tool called Choria Hierarchical Data - current code name tinyhiera but that might change.

NOTE This post was first posted on the Choria blog.

Read on for the details

The big difference here is that this new system supports just one data file, you express a data structure that is the complete outcome of the query and then configure in the same file how to override and extend that data file.

Let’s look at it, here I’ll show the different sections, in reality it’s just one file.

This is the data we wish to manage:

data:
    log_level: INFO
    packages:
        - httpd
    web:
        listen_port: 80
        tls: false

Lets define the hierarchy for overriding this data; this should be familiar to Puppet users:

---
hierarchy:
    merge: deep # or first
    order:
        - env:{{ lookup('env') }}
        - role:{{ lookup('role') }}
        - host:{{ lookup('hostname') }}

And finally, we show the overrides, here we just specify how the data will be extended:

overrides:
    env:prod:
        log_level: WARN

    role:web:
        packages:
            - nginx

        web:
            listen_port: 443
            tls: true

    host:web01:
        log_level: TRACE
        web:
            listen_port: 8080

We can now call our CLI tool to resolve this entire structure.

We query the data setting the role fact to web, the data from the role:web section is merged into the data from the data section:

$ tinyhiera parse data.yaml role=web
{
  "log_level": "INFO",
  "packages": [
    "ca-certificates",
    "nginx"
  ],
  "web": {
    "listen_port": 443,
    "tls": true
  }
}
$ tinyhiera parse data.yaml role=web --query web.listen_port
443

Features and Differences

This is the basic feature, similar to what you might have used in Puppet except differing in a few areas:

It’s all one file and one data structure, it’s not key-lookup orientated. The entire outcome is rendered in one query
Expressions used in overrides and data lookups are a full query language called expr-lang, this means we can call many functions, generate data, derive data and easily extend this over time
The data is typed, if your facts have rich data the results can also be rich data, if value is a integer, array or map the result will be the same type x="{{ lookup("value") }}"
The CLI has built in system facts, can take facts on the CLI, read them from JSON and YAML files, parse your environment variables as facts or all at the same time
The CLI can emit data as environment variables for use in scripts
There is a Golang library to use it in your own code

Environment Variable Output

We want this to be usable in scripts, the easiest might be dig into the data and just get a single value:

PORT=$(tinyhiera parse data.yaml role=web --query web.listen_port)

But this involves many calls to the command, we can instead emit all the data as variables.

$ tinyhiera parse data.yaml role=web --env
HIERA_LOG_LEVEL=INFO

You can see the data is emitted as environment variables that you can just source into your shell script, obviously in this use case it benefits to have flat data.

Built-in facts

We have many ways to get facts into it such as files, environment variables, JSON and YAML files. The CLI also includes it’s own mini fact source called system which will return facts about the system it is running on.

To view the facts fully resolved, we can run the following, I am removing much of the details here but you can see this offers enough to make most required decisions:

$ tinyhiera facts --system-facts
{
  "host": {
    "info": {
      "hostname": "grime.local",
      "uptime": 1843949,
      "bootTime": 1762258234,
      "procs": 711,
      "os": "darwin",
      "platform": "darwin",
      "platformFamily": "Standalone Workstation",
      "platformVersion": "26.1",
      "kernelVersion": "25.1.0",
      "kernelArch": "arm64"
    }
  },
  "memory": {
    "swap": { },
    "virtual": { }
  },
  "network": {
    "interfaces": [ ]
  },
  "partition": {
    "partitions": [ ],
    "usage": [ ]
  }
}

Long-term view

Autonomous Agents

The goal here is to incorporate this into various other places, first we pull it into a Choria Autonomous Agents, these agents own the full lifecycle of an application:

Deploy dependencies
Configure the application
Run the application
Monitor the application
Restart the application on failure
Orchestrate rolling upgrades
Present APIs for interacting with the management layer

Agents can fetch data from a Key-Value store, but it’s kind of all the same data. With Hiera integrated into the KV feature, we get:

  - name: watch_tag
    type: kv
    interval: 10s
    success_transition: regional_update
    state_match: [RUN, RESTART]
    properties:
      bucket: NATS
      key: config
      hiera_config: true

Here the autonomous agent will check KV every 10 seconds for data, fully resolve it using Hiera and save the resulting data into the Autonomous Agent internal data store. If the KV data chanfes or the referenced facts change to the extent that the data change, the machine will update the stored data and fire the regional_update event.

This way we can create role-orientated specific data all in the same Key-Value key.

Configuration Management

I want to create a new CM system that takes the model we are used to in Puppet but brings it to scripts and reusable APIs.

The script use case would essentially be standalone commands:

$ marionette package ensure zsh --version 1.2.3
$ marionette service ensure ssh-server --enable --running
$ marionette service info httpd

This should be familiar to Puppet users, we’re basically pulling the resources and RAL into standalone commands. The commands will be fully idempotent like Puppet and support multiple Operating Systems.

For integration with other languages, you can also deal with JSON-in-JSON-out style:

# Can be driven by JSON in and returns JSON out instead
$ echo '{"name":"zsh", "version":"1.2.3"}'|marionette package apply
{
....
}

I do not want to create something huge like Puppet, but just basically enough that can enable the package-config-service trio pattern, ultimately a manifest would look like this:

cat <<EOF>manifest.yaml
data:
    resources:
        - package:
            name: myapp
            ensure: present
        - service:
            name: myapp
            ensure: running
            enabled: true

hierarchy:
    merge: deep
    order:
        - platform:{{ lookup('host.info.platform') }}
        - hostname:{{ lookup('host.info.hostname') }}

overrides:
    platform:darwin
        resources:
            - package:
                ensure: MyApp-1.2.3.dmg
EOF

$ marionette apply manifest.json --system-facts

Pull this into Autonomous Agents and you have self-contained, self-managing applications similar to Kubernetes Operators but anywhere.

Use this in scripts, simple setups, during container creation or orchestration, or even in CI pipelines.

We should even be able to compile this manifest, or a full autonomous agent, into a single, static, binary that you can just run as ./setup or ./manage and the entire lifecycle is managed with zero dependencies.

Availability and Status

You can get the command on GitHub choria-io/tinyhiera, we are pretty far along but expect some more breaking changes including a potential name change.

Overlook InfraTech: The new gem server in town

2025-10-06T00:00:00+00:00

If you’re in the Ruby world you may have been following the RubyGems shenanigans of the last few weeks. They suffered a poorly communicated corporate led rug-pull, not all that dissimilar to the games Perforce has played over the last year or so. Amusingly, they also claimed it was for security purposes without providing any actual rationale supporting that claim.

As you can guess, this caused a lot of consternation, especially in projects like OpenVox based on Ruby and in companies like Overlook InfraTech whose livelihood depends on Ruby.

You may want to read that last bit again. This heavily impacts companies whose livelihoods depend on Ruby.

Ah. Now that’s the root of the problem, isn’t it? There aren’t enough companies financially supporting the projects they’ve built their own businesses on top of. For example, Puppetserver would not be possible without JRuby and yet Puppet/Perforce didn’t bother helping the project through its funding crisis until they were publicly named and shamed and it became a PR problem for them.

We don’t believe in doing business that way and we now have an opportunity to live up to our ideals. The original RubyGems team has announced a community led and governed gem server. This means that we now have the opportunity to help build a new ecosystem that’s not supported and dominated by a single company out of touch with its community.

Post by @getajobmike@ruby.social

View on Mastodon

They don’t currently have a way to sponsor the project, so we are using GitHub Sponsors to throw down $100USD monthly for each of the current developer team. We encourage you and/or your employers to do the same by clicking the {sponsor} icon on each of these GitHub profiles.

Here’s to a better and more resilient Ruby ecosystem going forward!

Camptocamp Blog: Platform Engineering at CNS Munich

2025-08-26T14:33:58+00:00

This year Cloud Native Summit (CNS) in Munich gathered adopters and technologists from open source and cloud native communities.

Compared to last year's Kubernetes Communities Days (KCD), the agenda dedicated many slots to Platform Engineering. This discipline can be seen as a way to fill gaps between different stakeholders of an Internal Developer Platform (IDP). Last year I wrote an article on this topic at the KCD Munich. This year I would like to share fresh ideas or principles from some talks given during the conference.

Identifying problems before solutions

Day 1 opened with a presentation of "Product thinking" approach made by Stéphane Di Cesare (Deutsche Kreditbank) and Dominik Schmidle (Giant Swarm). They reminded some core principles and techniques of "Platform Engineering" to manage the link between problem space and solution space.

Day 1 also closed with a panel discussion on "Platform Engineering" animated by Max Körbächer (Liquid Reply), focused on the adoption of IDPs by end users and within projects.

Video 1 Platform Thinking

Video 2 Panel Discussion

AI workloads on K8s

"Use platform engineering for AI workloads on K8s", that was the motto of the presentation given by Mario-Leander Reimer (QAware GmbH). He tackled emerging challenges for deploying Agentic AI workloads and introduced an example of what a platform should cover including quality plane and compliance plane. His talk is part of an open source initiative launched at QAware this year to mature the concepts of a dedicated control plane: the agentic layer.

Video 3 Architecting and Building a K8s-based AI Platform

Composable platforms on Kubernetes

Hossein Salahi (Liquid Reply) presented a reference implementation to streamline the delivery of services for developers. Some of the challenges were to link dev and ops, to manage services in configuration, to orchestrate infrastructure deployment and to manage access. For that purpose the open source tools Kratix codify a contract between dev and ops. Combined with Backstage, this abstraction allows services to be packaged and delivered to developers.

Video 4 Modular Platform Engineering with Kratix and Backstage

Reduce cognitive workloads

Interestingly the problem of cognitive workloads due to some platform complexity was a recurring theme. But Michel Murabito (Mia Platform) positioned platform engineering as a way to reduce cognitive workloads. He highlighted essential building blocks and features that can reduce mental burden.

Video 5 Creating a smooth Developer Experience

Involved user as contributor

Lian Li (lianmakesthings) told us a story of what can go wrong when building a platform based on real examples and some facts. She advocated some receipts that can improve communication among teams in an organisation and explain some benefits of having contributing users in the loop.

Video 6 Many Cooks, One Platform: Balancing Ownership and Contribution for the Perfect Broth

Convergence

Evelyn Osman (enmacc) discussed the importance of convergence when building a platform in order to avoid fragmentation in teams and solutions. She shared some important steps that foster innovations and better meet user expectations.

Video 7 Convergence on Platforms

Conclusion

This is not an exhaustive list of what was discussed, but somehow it made it clear that Platform Engineering approach can help picking the right solution in the open source landscape. Some enablers already provide a strong integration with Kubernetes to develop and publish new services on top of it. And depending on the organization, some shift in practices also might be required to follow the evolution of the platform at build and runtime. The conference was a great place to exchange with the community on technical and organization challenges.

References

betadots: 🧩 Puppet Module Update Process

2025-07-11T07:59:30+00:00

🔍 1. Identify and Analyze the Module

Extract a module name from the Puppetfile.
Search for the module on Puppet Forge.
Compare available versions and identify the latest one.

📝 2. Review Changes in the Module

Follow the Project URL on the Forge page to the GitHub repository.
Check recent changes under Releases or in the Changelog.

Example:

⚠️ 3. Watch for Breaking Changes

Possible breaking changes:
- Removal of support for EOL software
- API changes
- Renamed parameters or variables
Note: Not every breaking change will affect your setup.
- Example: Dropping EL6 support likely doesn’t concern you.

🔍 4. Evaluate Other Changes

Linked pull requests (PRs) may offer additional insights.
Check whether changes are understandable and whether breaking changes apply to your environment.

🌱 5. Integrate into Development Branches

If the module is considered ready for update:
- Integrate into a feature or development branch.

🌲 6. Control Repo Branch Structure

Typically present:

development
production
Optionally: staging before production
Additionally: 0–n feature branches

🧪 7. Testing in Development Environment

Test the updated module in a suitable environment (VM, container).
Observe how it interacts with other modules.

Possible findings:

Dependencies on specific module versions
New facts writing additional data to PuppetDB
New or modified parameters requiring Hiera data

🔄 8. Forward the Change

Once all adjustments are complete:
- Pass the change to the next branch for further testing or rollout preparation.

🤖 9. Automation with Renovate Bot

This process is quite detailed and time-consuming.
With Renovate Bot, collecting and reviewing relevant updates becomes much easier.
Renovate can be integrated into GitLab or GitHub.
It works similarly to GitHub’s built-in Dependabot, but is more flexible and configurable.

🧰 10. Automation with VoxBox

You can also try using the Vox Pupuli VoxBox to list all dependencies for the entire control repo.
- This doesn't work with every setup.
- If there are private modules in the Puppetfile, the person running the command must have access to them.
- Example command:
- podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:latest r10k:dependencies
More useful information can be found in the Vox Pupuli VoxBox documentation

betadots: 🧩 Puppet Modul Update Prozess

2025-07-11T07:58:40+00:00

🔍 1. Modul identifizieren und analysieren

Einen Modulnamen aus dem Puppetfile entnehmen.
In der Puppet Forge nach dem Modul suchen.
Verfügbare Versionen vergleichen und die neueste identifizieren.

📝 2. Änderungen im Modul prüfen

Über den Link zur Project URL auf der Forge-Seite zum GitHub-Repository wechseln.
Dort unter Releases oder im Changelog die letzten Änderungen prüfen:

Beispiel:

⚠️ 3. Auf Breaking Changes achten

Mögliche Breaking Changes:
- Entfernung der Unterstützung für EOL-Software
- Änderungen an der API
- Umbenennung von Variablen
Hinweis: Nicht jede Breaking Change ist für jedes Setup relevant.
- Beispiel: Der Entfall von EL6-Support betrifft euch vermutlich nicht.

🔍 4. Sonstige Änderungen bewerten

Auch verlinkte Pull Requests (PRs) können Aufschluss geben.
Prüfen, ob Änderungen nachvollziehbar sind und keine Breaking Changes enthalten, die euch betreffen.

🌱 5. Aufnahme in Entwicklungszweige

Wenn das Modul als updatefähig eingeschätzt wird:
- Aufnahme in einen Feature- oder Development-Branch

🌲 6. Branch-Struktur im Control-Repo

Typischerweise vorhanden:

development
staging
production
Zusätzlich: 0–n Feature-Branches

🧪 7. Tests in Entwicklungsumgebung

Test des neuen Moduls in geeigneter Umgebung (VM, Container)
Beobachtung des Zusammenspiels mit anderen Modulen

Mögliche Erkenntnisse:

Abhängigkeit zu anderen Modul-Versionen
Neue Facts, die zusätzliche Daten in die PuppetDB schreiben
Neue oder geänderte Parameter, die Hiera-Daten erfordern

🔄 8. Change weiterreichen

Wenn alle Anpassungen erledigt sind:
- Übergabe des Changes in den nächsten Branch zur weiteren Prüfung oder Vorbereitung für den Rollout.

🤖 9. Automatisierung mit Renovate Bot

Dieser Prozess ist sehr kleinteilig und kann zeitaufwendig sein.
Mit Hilfe von Renovate Bot lässt sich das Sichten und Sammeln der relevanten Informationen deutlich erleichtern.
Renovate ist ein Bot, der sich in GitLab oder GitHub integrieren lässt.
Er funktioniert ähnlich wie der GitHub-eigene Dependabot, ist aber deutlich flexibler und konfigurierbarer.

🧰 10. Automatisierung mit VoxBox

Man kann auch versuchen mit der Vox Pupuli Voxbox die Abhängigkeiten für das gesamte Control-Repo anzeigen zu lassen.
- Das klappt nicht bei jedem Setup.
- Wenn im Puppetfile private Module sind, muss der jenige der es ausführt, Zugriff auf diese haben.
- Beispielbefehl:
- podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:latest r10k:dependencies
Weitere nützliche Informationen gibt es in der Vox Pupuli VoxBox Dokumentation

Overlook InfraTech: So much for being developer first

2025-06-12T00:00:00+00:00

Well, it appears that Perforce has taken another step down the enshittification journey. For some strange reason, it appears that they don’t want you writing Puppet modules unless you’re a paying customer. That’s right, they killed the open source PDK and sequestered the source away, just like they did with Puppet. They would like you to pay for the privilege of creating content for them.

This is a very strange strategic move on their part. Their ecosystem is almost entirely supported by community authors building and maintaining Puppet modules. A full 96% of their paying customers use modules supported by Vox Pupuli, many of which are open source users. They barely even maintain their own modules in the puppetlabs namespace. Most of the contributions there have for years come from community members using the Trusted Contributors program that I built when I was still there.

On a related note, it also appears that they’ve shut down the Trusted Contributors program as it’s no longer mentioned on their website.

In reality, they don’t even have a dedicated modules team anymore! One with any sense at all would think that given this lack that they’d want to encourage people to build and maintain quality content in the ecosystem rather than putting barriers in place to make that harder to do. 🤔

My only assumption can be that the Puppet Core sales aren’t so hot and they need something, anything, to juice the numbers a bit.

But does it actually matter?

In all reality, many of us never really liked using the PDK. It’s really big and inflexible. The customization options are confusing and tedious. It generates way too many random config files for services that most people never used, unless you customize it, which again is super awkward. And maybe worst of all, it was designed primarily for the use of Puppet’s modules team (it still existed back then) rather than being designed for community developers. For example, until a few years ago, it configured PR testing and publishing workflows with GitHub actions that assumed that you were using Honeycomb for profiling and hardcoded the puppetlabs namespace.

Vox Pupuli developer tools

Luckily, Vox Pupuli has been working on replacements for a while. We believe that developer tools should be small and purpose-built rather than trying to jam pack all the things for all the people into a single massively overloaded tool.

Rather than using the exact same command used to

create a module from a template
run unit testing in CI pipeline
manage installed Puppet versions
build and publish modules
drop into the debugger REPL that you probably didn’t even know existed,

Vox Pupuli’s toolkit includes smaller tools that do one or two things well instead of being mediocre at trying to do everything.

For example, our modulesync tool makes it easy to create, manage, and update a group of modules based on a template. No more broken dependencies because you haven’t remembered to update one of your modules for the last four years. All your modules, whether that’s one or one hundred, will be in a consistent state and tested against your current infrastructure.

Or you can get zero-install access to a whole suite of module testing and management tasks with voxbox. Update your module’s metadata, validate all of its dependencies, and publish a new version all with a few podman commands.

The Vox Pupuli Devkit SDK

Now that easy access to the PDK is disappearing, we’ve put together a project to collect all these tools and more into a coherent suite that’s usable, discoverable, and maintainable. We’ll build the missing parts along the way. For example, I’m currently working on an asdf / mise plugin to easily manage OpenVox versions using an industry standard tool rather than something bespoke that’s constrained to very specific and limited workflows.

We don’t have a timeline on this yet. But luckily, there’s not much development on the PDK except to make it harder for you to use. You can very likely simply continue to use the version you already have installed until we have more news for you.

Watch this space and we’ll soon have information for you about the new Devkit SDK!

Overlook InfraTech: OpenVox InfraTales - MacOS Signing and Notarization

2025-05-12T00:00:00+00:00

Hi everyone! Nick here. I figured it was time I finally write a blog post. If you’ve seen me around on the Vox Pupuli Slack, you know that I’ve been deep in the trenches of OpenVox build code for some time now, getting the various bits to a secure and stable place where it’s easily accessible to community contributors.

Today’s story is about wrestling with MacOS’s Gatekeeper so we can release a production-ready MacOS agent build.

No unsigned code, ever

Perforce recently dropped a LinkedIn post about their latest Puppet Core release, bragging about how secure and enterprise-ready Puppet Core is. The one that stuck out to me: Fully signed binaries and agents - no unsigned code, ever

They’re right - unsigned binaries are like showing up to a formal dinner party in a crab costume. In OpenVox-land, all of our Linux binaries are fully signed. However, just like how Perforce’s internal dev builds are not always signed, our prerelease, non-final MacOS and Windows agents were unsigned (Windows remains so until we navigate the extortion racket that is paying for an Extended Validation code signing certificate from a vendor).

I applaud Perforce on their journey towards creating the safest, most secure Puppet environment possible, so I thought I’d write this little guide to help them take the next step. In order to reach the utopia of a fully signed and notarized MacOS agent that Gatekeeper is happy with, we have to not only sign the right things, but notarize it as well.

Gatekeeper: The ever vigilant security guard of MacOS

Apple’s Gatekeeper ensures that the software you download and run is trusted and safe. If your software isn’t both signed and notarized in the correct way, it will flat out refuse to open it (without extra shenanigans), warning that you potentially downloaded a radioactive piece of malware and you should be ashamed of yourself.

To make Gatekeeper happy, all you used to have to do in the halcyon days of pre-Sonoma was to sign the specific binary entrypoints into your app with an “application developer identity” (certificate + private key). Let’s dive into the GitHub mines and see how this is currently done.

Warning: There be dragons in these mines

The way that Perforce does this signing process for their current MacOS agents (or at least they did before taking everything private, but I don’t believe it has changed) is by sending the three executable files in the puppet-agent package to a signing server, which signs those files, and then sends them back to continue the build process. You can see how this is defined in the puppet-agent repo. This utilizes Vanagon’s ExtraFilesSigner to lay down a bash file to run the command, rsync the file over, run the command, then rsync the file back. This process is embedded in Vanagon’s MacOS platform build process. This code ends up with a dmg that contains a pkg installer, which contains all the files, including those three signed binaries. Finally, the pkg file must be signed with a different “installer developer identity”. Perforce does this by mounting the created dmg, signing the pkg with that identity, then recreating the dmg file with that signed pkg.

Signing and Notarization: Everyone’s favorite migraines

While this worked fine before, as of MacOS 15 Sonoma, the requirements are much more stringent. Not only must you sign the these binaries, but you must sign every single binary, dylib, and bundle file, no exceptions. And on top of that, you have to sign all of them specifying it should use the hardened runtime via the --options runtime flag to the codesign tool, as well a secure timestamp via the --timestamp flag. Doesn’t matter if the files are already signed in a different manner. MacOS 15 has a refined palate and will only consume the finest artisinal binaries. Yes, it’s probably for the best, but I digress.

Additionally, you must notarize either the dmg or pkg file. This involves sending your file to Apple, where their system scans for any hidden nasties and verifies everything that should be signed, is signed in the correct way. Once approved, you then must staple the notarization to the dmg or pkg so Gatekeeper can stop judging us.

Bringing a first pass of order to chaos

Right now in OpenVox, unlike Linux agents where we use containers to build them, we’re building the MacOS agent in a VM that is configured in a particular way with the appropriate bits and bobs installed. This is similar to how Perforce does it, except that this particular VM template also includes the bits needed for signing and notarization. This is not highly portable and accessible to people yet, but we wanted to get this agent into your hands as soon as possible. Soon, we’ll be moving the process into GitHub Actions and hopefully finding a way to build it locally without a Mac as well. We’ll take this awkward Ford Pinto and turn it into a Ferrari. Or at least a Honda Accord.

For this first pass at MacOS agent signing and notarizing, I moved the process entirely into Vanagon. While this may not be where we want to leave it since it reduces potential future flexibility, it makes sense for now since the only MacOS thing we sign is the agent, and this keeps the signing code from being spread across three different repos. This also streamlines the process, since we can do the signing and notarizing as part of the build, rather than having to unpack the build, sign the pkg, then repackage it into a dmg.

This new code hard-codes where to look for binaries that need signing within the agent package. This is not at all ideal, and is ripe for future improvements like moving this definition back into the vanagon project itself. But because it isn’t obvious which files need signing (i.e. binaries don’t all live in the same place, don’t all have a particular suffix), this is an easy way for us to utilize find -exec to sign all the pieces. Another improvement made here is that at every step, we verify the signing/notarizing worked correctly. This gives us an extra layer of confidence that the delivered agent will not shout obscenities at you.

As before, we sign the pkg file, but we do it before building the dmg. Then, we build the dmg and sign that, after which we notarize using the notarytool in Xcode. Usually, the notary process takes less than 2 minutes. On our initial submissions, it took days as they were being manually reviewed. Now that the system recognizes our submission, it is much faster. Once notarization is approved, we staple that notarization to the dmg and then check with Gatekeeper that it isn’t going to complain.

The Bureaucracy: D-U-N-S numbers and Overlook InfraTech, Inc.

Figuring out this whole process was quite a bear. Apple’s documentation mostly assumes you are developing with the Xcode GUI app, and not trying to do everything from the command line. It isn’t terribly clear exactly what needs to be signed and notarized and how without some more digging and trial and error.

Another thing that isn’t terribly ideal is that in order to have a developer account tied to an organization, that organization has to be a full fledged legal entity with a D-U-N-S number. If you ever start a business, you’ll find that you need a franky ridiculous amount of different identification numbers that tell people your business is, in fact, a real business, and this number is just one of many. Vox Pupuli doesn’t have this, so the agent is signed with the Overlook InfraTech, Inc. account. This has the side effect that our company name shows as the one that owns the puppet and pxp-agent background tasks. This should absolutely say Vox Pupuli and hopefully one of these days we can change that.

Final thoughts: Let’s work together

To my knowledge, Perforce does not have their MacOS 15 agent notarized, nor signed with MacOS 15’s newer stringent requirements (repeating my disclaimer: I do not know for sure, as none of us has visibility into their current build process or access to their current agents without signing a EULA which prevents you from inspecting any of the code anyway). I’d love to submit some PRs to their repos, but unfortunately, they have seemingly completely abandoned them in favor of their internal forks. That’s a bummer, because security is a team sport, and we all benefit when the ecosystem is safe. Nevertheless, I invite them to take a look at our work here to assist in getting their MacOS agents notarized, and I’d be more than happy to work with their engineers to assist. Thrilled, in fact. And I promise I’m not being sarcastic about that.

If you’re curious for more details about how we did this or how we do any of the OpenVox things, drop by the Vox Pupuli Slack or IRC, both of which you can find in the Connect menu of voxpupuli.org. I’m always happy to chat, even while I’m debugging why a binary isn’t signing properly at 2AM. If you have any other topics you’d be interested in hearing more about in this blog, drop me a line any time.

Overlook InfraTech: The OSU Open Source Lab needs your help today

2025-05-01T00:00:00+00:00

Have you been looking at the OpenVox project and wondered what you could do to help out? What if I told you about an opportunity to help us, and also hundreds of other OSS projects that you probably rely on? The OSU Open Source Lab is OpenVox’s primary package host and helped us get the project off the ground right from the start. They help hundreds of open source projects with critical infrastructure needs that many projects (like us) would not have been able to afford on their own.

🔔 tl;dr; the Open Source Lab needs your help

Not only that, but they have been training the next generation of FOSS DevOps and system infrastructure engineers for YEARS. Many of the first Puppet community practitioners came up through the OSL. They contributed many of the first Puppet modules. The venerable Pro Puppet second edition was written by OSL students in collaboration with PuppetLabs developers, including myself.

OSL has played a HUGE part in Puppet’s story and helped Puppet, Vox Pupuli, and OpenVox get to where they are today. I’d wager that your company uses OSL resources on a regular basis and may not even realize it. Take a look at the list and you’re certain to find something business critical.

However, with the current crisis in academic funding in the USA, they are in critical need of funding and without your help, they may cease to exist. See this blog post for more information and go talk to your boss today! Don’t hesitate to drop them some cash yourself if you can spare it.

Camptocamp Blog: Talos Linux: a new standard for on-premises Kubernetes clusters?

2025-04-22T17:44:21+00:00

A few weeks ago, I was at KubeCon Europe 2025 in London and I had the opportunity to attend a presentation that tackled the monumental challenge of migrating 35 Kubernetes clusters in an air-gapped environment from nodes deployed with a mix of kubeadm/Ansible/Puppet to Talos Linux nodes deployed using Cluster API.

While the presentation was quite interesting (you can get the slides here and watch the session recording on CNCF's YouTube Channel), I want to dive more into the Talos Linux project and its features.

What is Talos Linux?

Talos Linux is a modern Linux distribution purpose-built for running Kubernetes clusters. Some noteworthy characteristics of Talos Linux are:

Immutable: Talos Linux is designed to be immutable and always runs from a SquashFS image. This means that the operating system is read-only and cannot be modified at runtime. This immutability provides a strong security posture and means that there is no need to worry about unintended changes to the operating system.
Minimal: Talos Linux is a minimal operating system that only includes the components necessary to run Kubernetes. All the OS is built from the ground-up and no unnecessary components are included. This minimalism reduces the attack surface and improves performance.
Ephemeral and Declarative: Talos Linux nodes are ephemeral and everything written to disk is reconstructible. It is also declarative, meaning that the desired state of the system is defined in a configuration file and gRPC API, which is perfect for someone that loves automation and reproducibility, like myself.
Secure: As a consequence of its design, Talos Linux provides enhanced security features, ensuring that the system remains robust against various threats.
Agnostic: Talos Linux is cloud-agnostic, allowing it to run on various cloud providers and on-premises environments without vendor lock-in.

In summary, and I'm quoting the official documentation, "Talos is meant to do one thing: maintain a Kubernetes cluster, and it does this very, very well."

My thoughts

I have been following the Talos Linux project for a while, and I was gladly surprised to see a Swiss-bank like PostFinance being on the forefront of adopting such modern solutions like Talos Linux and Cluster API.

I think Talos Linux will be a key player in the Kubernetes ecosystem, especially for organizations looking for an on-premises solution that's secure, efficient and easy to manage.

The fact that Talos is declarative and immutable might seem like a drawback at first for someone used to the old ways of managing infrastructure with Ansible or Puppet, but I believe that this is the future of managing Kubernetes clusters.

I want my nodes to behave like pods that I can easily create, destroy, and replace. Besides, I don't want to deal with the overhead of managing the operating system. I already have enough to deal with the on-premises infrastructure for the network and storage and the Kubernetes cluster itself, so why not offload the management of the operating system to a purpose-built distribution like Talos?

With Omni, Sidero Lab's SaaS platform for managing Talos Linux clusters, I believe Sidero Labs have a good revenue model to continue developing Talos Linux. As a fan of open-source, we are all aware of the challenges of maintaining a project like Talos Linux, and I believe that having a SaaS platform to manage Talos Linux clusters is a good way to ensure the project's sustainability.

Talos Linux vs. other solutions

Red Hat OpenShift is a well-known solution in large enterprises. However, more than a Kubernetes distribution, it is a complete platform that includes a lot of features and components, including CI/CD tools, monitoring, etc. It is also expensive.

On the other hand, Talos Linux shines with its simplicity and minimalism, which brings more flexibility and allows teams to choose their solution to complete the platform as they see fit.

RKE2 is another Kubernetes distribution that focuses on simplicity and security, making it a strong contender for organizations looking for a lightweight solution. However, it still requires an underlying operating system that you need to operate.

Bonus

While at KubeCon, I had the opportunity to visit the Sidero Labs' booth and talk to the team behind Talos Linux. I thank the team for a warm welcome and great conversations about the project.

Go further

I wanted to keep this blog post short and not too technical, but if you want to learn more about Talos Linux, I recommend checking out the following resources:

Overlook InfraTech: A whirlwind conversation on trademarks

2025-03-20T00:00:00+00:00

Quite a few people over the last few months have helpfully pointed out the name collisions in the OpenVox Puppet fork. If you didn’t recognize it, there’s also a VoIP/PBX provider called OpenVox. Surely that’s got to be a violation of some kind of law, right? Even Google gets it very wrong! See how our information is interleaved in this autogenerated profile?

Hold up though. Not so fast because it’s not quite that simple. Lemme tell you a quick story about a name that I’m pretty familiar with: Bolt.

The Bolt story

In August of 2017, Puppet began work on a new project intended to be an agent-less orchestration tool for automating the manual work it takes to maintain your infrastructure. Sure, it started life as an Ansible competitor, but it grew to be pretty neat. For the first time, you could trivially and seamlessly interleave shell scripts and Puppet code and just push that configuration out to nodes in your infrastructure as needed.

Imagine my surprise when I visited the Slack booth at KubeCon the following year and saw their new application library called… none other than Bolt. It had gone through several name iterations, including Slapp which is my personal favorite, but Bolt is what they settled on. I asked if they knew about our tool and they nodded and brushed off my concerns. Obviously, I raised it to our Legal team and was surprised when they also brushed it off.

So then I started noticing all the things named Bolt.

In the Pacific Northwest where I lived, we had the Bolt Bus
There were the Bolt e-bike rentals
There’s the Bolt rideshare and rental company
There’s the silly Bolt movie about “superhero” dog.
There’s an ecommerce platform named Bolt
There’s a mobile app tool named Bolt
There’s a CMS named Boltcms
There’s an investment platform named Bolt.
There’s a gear lock brand named BOLT
There’s a biotech company named Bolt
There’s an insurance company named Bolt

Aaaaand I’m tired of typing out that list, so I’m just gonna stop at that.

What purpose does trademark serve?

Clearly there’s more to trademark enforcement than just the name itself. As it turns out, that’s only a small part of it. At this point, I’ll take a real quick aside to remind you that I’m an open source software nerd, not a lawyer. And I’m certainly not your lawyer so none of this is legal advice. In any case, the concept of trademark law isn’t really to give you an asset to own, it’s to protect consumers in the market from brand confusion. You’ll notice that none of those companies I listed are named Bolt and also do infrastructure orchestration.

A trademark is a word, phrase, or logo that identifies the source of goods or services. Trademark law protects a business’ commercial identity or brand by discouraging other businesses from adopting a name or logo that is “confusingly similar” to an existing trademark. The goal is to allow consumers to easily identify the producers of goods and services and avoid confusion.
– Wikipedia

There’s a commonly used rubric called the DuPont Factors which can help identify whether there is a probable infringement. The rubric has 13 criteria, but they boil down roughly to

Similarity of the marks.
Similarity of goods/services.
Similarity of trade channels.

So that explains why we aren’t concerned with the fact that there’s a product out there named OpenVox but is in a totally unrelated industry. In other words, someone looking for an infrastructure management system isn’t likely to accidentally purchase a PBX system, even if the name is the same.

What about the filenames?

But there’s another point that comes up frequently that’s not quite as clear. That’s the name and paths of the executables and the configuration files and such within the OpenVox packages. At this point, they’re exactly the same as the original Puppet packages. In other words, to configure OpenVox, you edit the puppet.conf file and to run the agent you invoke puppet agent -t. We negotiated this point very heavily with Perforce in order to not cause community disruption.

There are a few non-legally-defensible reasons why Perforce might agree to this usage and a couple reasons that are legally defensible. First off, let’s look at why Perforce might be motivated to allow this usage. These are not reasons that would stand up in court, if it came to that, but are ways in which Perforce benefits from it.

They get a little bit of free marketing. Every time someone types puppet agent -t they’re reminded of where OpenVox came from.
If someone using the system were to be confused by the name, it would likely be to Perforce’s benefit. In other words, if someone trialled OpenVox and typed ‘puppet’ a whole bunch, there is a small chance that when they submit an order to their purchasing department that they might accidentally request a Puppet purchase. It would never go the other direction.
If we had to change all the commands and paths today, then the ecosystem would be forced to write complex abstractions or explanations with every module, tool, tutorial, README, guide, etc. It’s more likely that they’d take the easy way out and choose one or the other, thereby fragmenting the ecosystem.

That last point deserves more explanation. OpenVox and Puppet by themselves aren’t terribly useful. Their value comes from the modules that enable the management of Apache, MySQL, SELinux, etc. So the modules and the tools that make building modules easier are of utmost importance. All these modules and tools and the documentation around them are currently written with the expectation of the existing paths.

If we changed these paths, then every single part of the ecosystem would have to account for the path change. It’s a comparatively high lift to abstract the name and paths out, whether it’s in code or docs, so it’s likely that most people will just choose a side and then search and replace ‘openvox’ for ‘puppet’. This is why Puppet modules and GitHub repos are all still in the puppetlabs namespace, for example.

There are two reasons why Perforce would be on the losing end of that schism. First is that Perforce is now attempting to sell something infinitely more expensive than its former price of free. Very few people creating and sharing free open source content would choose to support that ecosystem rather than the FOSS alternative. The second reason is that 96% of Puppet’s enterprise customers use Vox Pupuli modules and, for all practical purposes, 100% of them use Vox Pupuli’s dev tools since they’re vendored into the PDK. Vox Pupuli is obviously choosing to support OpenVox even if that has to be at the exclusion of Puppet itself.

There are also legally defensible reasons. Even if Perforce weren’t convinced by the reasons already stated, these would almost certainly lead to them losing a court case if it came to it.

By the time that someone has researched, and purchased (or not), and downloaded, and installed, and configured the system, it’s highly unlikely that they would be confused about whether they were running Puppet or OpenVox. Remember that the goal of trademark laws are to protect against confusion in the market. Several of the DuPont factors look at trade channels and the difference between downloading an OSS tool and talking to Sales is quite stark. If this were the only point we had to go on, it probably would not be enough. But there are better defenses.
Trademark is intended to protect a brand, not functionality. If something is used for functional purposes, then it cannot under the Functionality Doctrine be considered a trademark infringement. The canonical case here is Sega v. Accolade of 1992. Sega included a “Trademark Security System” which relied on the string “SEGA” to exist in a known memory address on a game cartridge. If the code were present, the game system would display a trademark notification and play the game. If it were not present, then the game wouldn’t play. Accolade added the code to their games and won the court case when Sega sued them for trademark infringement because the trademark was used for functionality.
Command names and file paths are a functional interface to a piece of software. In other words, if a tool or system wants to integrate with Puppet, then it will often do so by configuring puppet.conf or running puppet commands. If we are to build a Puppet-compatible product, then it has to use the same command names and paths. Our industry is littered with examples of this going back for many decades. Think about how many MTAs install /usr/sbin/sendmail, for example. On my Mac, that’s provided by Postfix. The actual Sendmail company, now owned by my former employer Proofpoint, has had three decades to dispute that use and has not. Another example is MySQL, owned by what might be the most litigious company in existence (Oracle.) When MariahDB forked away, it was ten years before they bothered to replace the admin commands like mysqladmin and even today they ship symlinks so you can still run that command.

The path forward

We have no concerns about trademark infringement today. The OpenVox PBX system is in an entirely different industry and won’t be confused with the OpenVox configuration management system. Perforce and OpenVox are both incentivized for us to keep using the existing command names and paths. And even if they weren’t, there is significant legal precedence to defend our use.

With that said, it would benefit our own brand to have our own command names and paths. Over the next year or so, we have a roadmap to achieving this safely and with as little community disruption as possible. We will first build an integration acceptance test framework so that we can tell quickly when we screw it up. Then we’ll gradually start replacing commands and paths with our own versions while leaving symlinks in place so that integrations continue to work. Then finally, we’ll extract those symlinks into an optional openvox-puppet-compat package that will likely never go away.

In order to make Puppet extensions universal, we aren’t planning to ever change the internal file hierarchy that makes up the documented API. And if that ever does change, we’ll do it in cooperation with Perforce to retain compatibility.

Overlook InfraTech: Paying their fair share

2025-02-28T00:00:00+00:00

The other day someone shared a Puppet Community Slack post in which a very senior Perforce sales engineer stated plainly that the reason for their sequestered source shenanigans is that they’re “asking open source users to pay their fair share.” Now, I don’t want to sound like I’m poking fun at this particular corporate flunky, he’s only repeating what he’s clearly been told over and over again, but What. The. Fuck.

Besides making it exceedingly clear that the only thing that Perforce understands about open source communities is a bunch of people who happen to own wallets or purses, this also demonstrates a devastating lack of understanding of their own product. Let’s break it down, shall we?

What you think of as a big beastly monolithic application called Puppet Enterprise is mostly a lot of open source tools jammed together with some proprietary business logic duct tape and baling wire. No surprise there, that’s how most enterprise apps are built these days.

The puppetserver application is business logic implemented in the open source Clojure runtime. It drives puppet via the open source jRuby runtime. These are packaged up into a jar with a lot of open source components like the open source Bouncy Castle which is of course built on top of the open source OpenSSL library. This jar is then run with the open source openjdk Java runtime. To build it, they use the open source Jenkins CI/CD platform. Packages are generated with the open source fpm tool.

And that’s only puppetserver. The puppet-agent package uses open source nssm (now defunct) to run on Windows because they couldn’t be arsed to write their own service management. PuppetDB is just a business logic wrapper over the open source PostgreSQL database server which was chosen because it was the only one with the primitives they needed. The Puppet Enterprise Console is built on the open source Ember framework. I could go on listing the projects that Puppet depends on, but that would easily triple the length of this article.

There is very literally not a single part of Puppet Enterprise that stands alone. More of the PE installer is open source packages than it’s Puppet Enterprise itself. And yet how much do they contribute back? Do you think they “pay their fair share?”

No. It’s zero. When jRuby put out a desperate call for funding, do you think Perforce chipped in? Have they ever contributed to OpenSSL? Has @whack ever seen a dime for the thousands of packages Puppet built with fpm? No. None. Perforce has never contributed a penny. They’ve just taken and taken.

Update, hey neat! Looks like raising the issue finally got jRuby some support. Let’s hope that they continue funding their dependencies.

Post by @headius@mastodon.social

View on Mastodon

To be clear though, that’s not the part of this that’s so offensive. Perforce’s sponsorship record is pretty standard for the corporate world. Puppet itself, prior to Perforce, didn’t contribute all that much more. What is utterly offensive is this idea that somehow the very community that built the foundation that Puppet sits on isn’t “paying their fair share”.

Let me tell you about “their fair share.” Many of you know that I spent thirteen years at Puppet and I ran Community for a large part of that. So I can give you actual real numbers. A full 96% of Puppet’s paid customers uses Vox Pupuli’s open source community modules. And the remaining 4% would have except their policies required them to write all their own content.

When it comes to paid Puppet customers, we should acknowledge how many of them are customers because of the community. I’ve attended many “vendor appreciation dinners” where the point of me being there as a community representative was to make sure that Puppet, Inc continued to remember that investing in community was required. The single biggest deal Puppet ever closed hinged on a community interaction that I and some Vox Pupuli members enabled. We talked through a hypothetical solution to the problem at hand and then I built it on my spare time as a community member.

Then if you want, we can get down to raw contributor data and I could show you how many community members contribute to Puppet codebases more even than most Puppet employees. A quick count shows that only 54 of the top 100 were Puppet employees.

Puppet’s community is the foundation for Puppet Enterprise and the whole company. When we did the massive upgrade from 3 to 4 and needed all the tooling to find all the weird edge cases, the community is who pointed out what we needed to validate and built tools like catalog-diff to give us a starting point. When we pulled core resources out to modules in Puppet 6 and broke every testing pipeline in the world, the community worked with Josh to fix it. And now that Perforce is insisting on shooting its own feet, the community is once again stepping up to ensure that an open source community actually continues to exist and help each other out. That community still includes their paid clients and somehow Perforce continues to lose sight of that fact.

The fact that the messaging that their representatives share is that open source community users need to step up and “pay their fair share” is a slap in the face to the community that has provided the foundation for the entire company to exist on. Without these people Puppet by Perforce would not exist.

Puppet by Perforce doesn’t pay their fair share in any way shape or form, and for them to self-righteously demand that from their users demonstrates how little they understand the space and how parasitic they are. I wish that they weren’t so representative of the industry.

The reality is that open source projects help your company. They accelerate your GTM (go to market) velocity a TON. And open-sourcing your own product leads to rapid adoption by open source community users who give you so much free marketing and product feedback and are the final line of QA before changes hit your paying customers. All in addition to the code and docs and expertise contributions. But this doesn’t come without cost to you. If you use the benefits of open source software then you have to pay the costs of open source software. It’s not a lot. It just means that you have to understand that the value you derive from open source communities is inherently a function of your product being freely available. When the vultures swoop in and try to alter that power balance, then you lose the benefits of the open source community.

And before you demand a “fair share” from your users, make sure that you’re doing the same. If your company uses open source software then contribute back. Either sponsor the project or contribute code, resources, or time. And if your community contributes code, or docs, or troubleshooting, or anything then freaking appreciate them for it.

Overlook InfraTech: CfgMgmtCamp wrapup

2025-02-17T00:00:00+00:00

Well, we’re back from Belgium after another successful week of conferences. First we attended FOSDEM in Brussels, then we hustled down to Ghent for the CfgMgmtCamp speaker’s dinner. This year is special to me because it’s the first year that I’ve not gone to represent someone else. It was also special because the atmosphere was so positive. There were a ton of great talks, more sponsorship dollars, attendance was up, more people showed up for the optional third day add-on than ever before, and people responded really well to the talk lineup. Chef even showed up to try and rebuild their community engagement!

In other words, it felt a bit like a revival. But I’ll come back to that. (Or you can skip directly to it, just like online recipes!)

Main stage panel at FOSDEM

It started on Jan. 21, 10 days before FOSDEM. There were some last minute schedule changes that I won’t speculate about too much ^{(they’re the same link, just two days apart)} and I was invited to participate in a panel discussion about project forks, why they happen, what they can do to the community and so on. Read more about it and see the recording right on the FOSDEM schedule page.

What a fun experience! Dawn was the original architect of the Puppet community and was my boss for all of a hot minute when I moved onto the team. But somehow this is the first time I’ve shared a stage with her. Everyone on stage was super knowledgeable and I learned a ton. In particular, Stephen Walli explains that statistically two criteria predict the success of a community fork

The community is put into existential crisis by actions of the original IP holder, and
One or more respected non-employee members of the community step up to serve as anchors.

This bodes very well for the future of OpenVox.

A busy week in Ghent

The main stage lineup at CfgMgmtCamp was fascinating. First, Hazel Weakly talked about the humanity behind software, configuration, and infrastructure management. This is an interesting perspective to me. As I see it, humanity is the unsaid reason behind the devops movement in the first place but we’ve now shifted far away from it. It’s no surprise that Puppet’s slogan used to be the human-forward “freeing you to do what the robots can’t” and now it’s bland corporate scare-words “avoid business disruption and minimize downtime.”

The thematic arc of the morning took us through these ideals we held when starting the devops movement, to the nitty gritty of OpenTofu’s escape from the corporate thumbscrews and recapturing their own personal power, to the real value of a content ecosystem and musing on a better way to build sustainable ecosystems.

Of course, the rest of the conference was just as good. We had two full days of two solid Puppet / OpenVox tracks covering all kinds of topics, from the technical to the speculative to the damage control. On the main stage Adam talked about the hypergraph of the future. James kept you on your toes by throwing fire again. There were a couple of talks about compliance. Stahnke showed us how he put his EPEL and Puppet knowledge to work with NixOS.

But there was still a vibe running through. Florian talked about toxic corporate environments and Nick explained how “source available” is such a woefully inadequate substitute for open source and ways to better appreciate your contributors. A lot of talks hinted at disillusionment with corporatism and the current state of the world. Lunch conversations debated the merits of various open source licenses and whether or not the GPL would prevent the licensing smash-and-grab ploys that so many once-great companies are pulling today. I lost track of how many people thanked me for my “yes, I’m a dirty hippie” slide.

I even overheard someone at a table nearby talking about their challenges in forming a tech union. I didn’t catch who or where it was, so if this was you, I’d love to hear more about it.

And that’s not even mentioning the overwhelming interest in OpenVox. Our community dinner (traditionally the Puppet Community and now the Vox Pupuli Community) filled the first floor of the Royal India. This was a nice healthy increase from years past and brought together several generations of Puppet / OpenVox community collaborators.

Some people were there for their first time and some had been there since the beginning; back when CfgMgmtCamp was still called a Puppet Camp and none of us had been acquired by private equity yet. But we were all united by two common interests: 1) keeping a community alive despite the best efforts of Puppet’s current owner to repeatedly shoot its own feet, and 2) excellent Indian cuisine enjoyed with friends.

For context: the conference is two days long, but they book the venue for a third day to make space for mini-events to tag along and colocate. Puppet has traditionally used this for a Puppet Community Day; Ansible and Foreman have similar events.

The optional third day is where this enthusiasm really shone through though. Not only were there more people who stayed for the third day than in years past, but every single person left the Puppet Community Day to join the OpenVox working session breakout. And as a measure of their commitment, at 5pm there were still 40-50 people in the room still hard at it, which is unheard of. Most people trickle out to catch their trains during the day.

We did not get quite as much done as we’d hoped, but everyone had the opportunity to participate and we set the groundwork for a stable and healthy project. In short, we adopted a Fedora SIG style governance and decided to remain an independent project via OpenCollective for the time being. As we mature a bit, we’ll continue investigating the Linux Foundation and other options.

Notes for the entire third day, including the Puppet Community Day and the OpenVox working session breakout can be seen in our wiki, thanks to Gene Liverman and Alex Fisher.

Insights

I think the positive vibes were mainly the result of three different factors.

First, the obvious

I don’t need to spend much time on this. With the media coverage and the furor around the clumsy way Perforce is stumbling through its community changes and the blatant money-grab, it’s no surprise to see excitement in our community. We shall see how much of that excitement is lasting.

What’s old is new again

For years, it seemed inevitable; cloud was eating our lunch. Every workload was becoming cloud-native, whether or not it really fit there, and the idea of managing configuration of long-lived instances was beginning to feel quaint. But there was a lot of foreshadowing, especially for those of us who’d been around for a few boom-bust cycles. A lot of the same problems being solved, a lot of the same challenges being (re-)discovered, opaque builds, semi-reproducible artifacts. And so much of it is tied together with not much more than the shell scripts I used to build production chroots back in the day.

The expected savings from cloud-native workflows haven’t really materialized either. If your workload is suited for, designed for, and optimized for the cloud then it can often be cheaper. But costs that were trivial at testing or prototyping stages have a tendency to unexpectedly balloon out of control at scale.

It’s not that cloud is bad. We’re actually moving a lot of workflows (such as the OpenVox build pipelines) to containers and it’s drastically simplifying what we’re doing. But now that the craze is dying down, people are realizing that it’s not the be-all end-all that it was promised to be. There are a ton of workflows well-suited to cloud & containers, especially transient and stateless tasks. But there are also a lot of workloads that benefit from being on-prem, or on long-lived stateful instances, or just don’t need the layers and layers of complexity.

And for these jobs, good old fashioned configuration management is right here waiting with its predictable pricing. The pendulum is swinging back to a hybrid infrastructure and we’ll all benefit from playing in the same sandbox. This year we saw a resurgence in config management community investment. But we also had a full Kubernetes track. And I find that very promising.

The industry, the world, and everything

There’s a lot going on right now. There’s somehow a blustery orange man-baby in the American White House again. For some unexplained reason, that gives Elon Musk the ability to go Twitter on the American government and slash essential departments protecting everything from consumer rights to the environment. “AI” is unsustainably disrupting industries and exploding the wealth gap. A lot of companies & industries aren’t fully recovered from COVID yet.

And right in the middle of that are company after company turning to corporate vampires and sucking the open source communities that built them dry. Is it any wonder that people are activating?

Back in the early 2000’s, I regularly protested with the EFF on the Santa Monica Pier in support of all sorts of digital rights and privacy protections. Not all the bystanders knew who the EFF was or what we were talking about. But most did or were curious to hear about it.

I’m getting some of the same vibes now. People are realizing that they need to take their interests into their own hands again. VC money has been cheap and easy for a very long time. And along with the foosball tables, that meant quite a bit of budget flexibility. It didn’t matter so much if you spent a few days polishing up a patch to contribute upstream when the runway was so long.

That’s been on the downswing for a while, but this last year really brought everything to a head. The investors and board members have come calling and they want their pound of flesh. And when it comes down to it, your boss cares more about the bottom line and the market cap than they do about you or your silly open source ideals. The days of founding a startup with a pair of dice are done. No more Uber but for dogwalking or AirBnb but for retirement homes or Tinder but for telling jokes.

Now it’s time for solidarity. People are realizing that the company they once trusted doesn’t actually have their best interests in mind. They’re seeing through the permissive license and contributor license agreement blinders. And they’re viscerally coming to grips with the idea that very few businesses include “provide something of value” as an actual business goal.

And they’re starting to do something about it.

betadots: Das neue Puppet Open Source Projekt heißt "OpenVox"

2025-01-28T15:00:38+00:00

Puppet by Perforce hat angekündigt,
dass die Open Source Pakete ab 2025 nur noch nach Zustimmung einer nicht weiter definierten EULA verfügbar gemacht werden sollen.

In early 2025, Puppet will begin to ship any new binaries and packages developed by our team to a private, hardened, and controlled location. Community contributors will have free access to this private repo under the terms of an End-User License Agreement (EULA) for development use.

Dabei muss berücksichtigt werden, dass die kostenfreie Variante nur für maximal 25 Nodes gilt.
Wer mehr Systeme mit Puppet Open Source verwalten will, benötigt eine Puppet Labs Support Commercial License.

Die Open Source Community hat das Gespräch mit dem Puppet Management gesucht.

Hier das Resultat:

Das neue Puppet Open Source Projekt heißt OpenVox
- Vorbereitung
- Aus Puppet wird OpenVox und Puppet Open Source
- Source Code, Pakete und Container
- Ich nutze Puppet Open Source. Was muss ich tun?
- Puppet Open Source
- OpenVox
- Die weitere Entwicklung
- Wie kann man helfen?
- Professioneller Support
- Zusammenfassung

Vorbereitung

Vor dem ersten Gespräch mit Puppet hat sich die Community im Vorfeld besprochen und analysiert, ob man als Open Source Projekt in der Lage sein kann, die weitere Entwicklung zu stemmen.
Das Feedback aus der Open Source Community und von Open Source Usern war sehr positiv.

Einige Nutzer haben sogar direkt angekündigt, dass man Entwickler für dedizierte Aufgaben zur Verfügung stellen möchte.

Aus Puppet wird OpenVox und Puppet Open Source

Die Open Source Community hat sich mit Puppet by Perforce darauf geeinigt, dass die weitere Entwicklung von Puppet in einem Fork unter der Verwaltung der Open Source Community erfolgen soll.

Nach einer öffentlichen Abstimmung Ende 2024 innerhalb der Community, hat man sich für den Namen OpenVox entschieden und beschlossen, dass die Entwicklung innerhalb der bereits existierenden Vox Pupuli Puppet Open Source Community auf GitHub stattfinden soll.

Da Perforce die Namensrechte an Puppet besitzt, hat man gemeinsam mit Perforce entschieden, dass die Pakete, die durch das OpenVox Projekt erstellt werden, nicht mehr Puppet im Namen haben sollen.
Dies ist auch in Hinsicht auf Differenzierbarkeit der Herkunft der Pakete sinnvoll.
Die Open Source Pakete von Perforce werden weiterhin puppet-agent, puppetdb und puppetserver heißen.

Gemeinsam mit Perforce wurde vereinbart, dass man sowohl die Kommandozeile, wie auch Namen der Konfigurationsdateien und die Schnittstellen zur Erweiterung von Puppet ( /lib/puppet in Modulen) zu existierenden Installation und Code kompatibel halten möchte.

Die weitere Kompatibilität soll durch ein Language Steering Committee sichergestellt werden, in dem alle Beteiligten gemeinsam über Änderungen an der Puppet DSL oder Inkompatibilitäten entscheiden.

Source Code, Pakete und Container

Der frei verfügbare Puppet Open Source Code wurde in das OpenVox GitHub Projekt geforkt.

Im ersten Schritt hat das Re-branding stattgefunden.
Für Anwender ist es wichtig zu wissen, dass Kommandozeilen Tools, Bibliotheken und Puppet Erweiterungen auch unter OpenVox vollständig kompatibel zu Puppet Open Source sind.

Aktuell werden die Pipelines zur Erstellung der RedHat und Debian Pakete auf öffentliche GitHub Actions umgestellt und sollen sehr bald zur Verfügung gestellt werden.
Erste reproduzierbare Build wurden bereits getetstet.
Pakete für Windows und MacOS sind ebenfalls geplant.

Die Puppet Container (puppetserver und puppetdb) werden nicht mehr weiter gepflegt!

Statt dessen werden OpenVox Container zur Verfügung gestellt. Aktuell wird an OpenVox Server und OpenVoxDB gearbeitet.

Ich nutze Puppet Open Source. Was muss ich tun?

Es gibt 2 Möglichkeiten:

Puppet Open Source
OpenVox

Puppet Open Source

Man muss bei Puppet die EULA akzeptieren und sich ab 25 Nodes um eine Lizenz kümmern.

OpenVox

Die Installations-Anleitung für die Migration findet man auf der
OpenVox Projekt Seite.

Da die Konfigurationspfade und -dateien, sowie die Kommandos alle identisch sind, muss keine weitere Anpassung oder Umstellung vorgenommen werden.
Auch der existierende Puppet Code funktioniert ohne Anpassungen.

Nutzer von OpenVox werden gebeten die aktuellen Pakete und Container in einer Testumgebung zu validieren.

Die weitere Entwicklung

Es gibt eine ganze Reihe von Wünschen aus der Community, die bisher leider nicht umgesetzt wurden.
Zuerst möchte man aktuelle Betriebssystemversionen unterstützen. Hierbei hatte die Community oft nach Paketen für Debian stable und testing gefragt.

Im nächsten Schritt soll der Code von alten Artefakten befreit werden.
Nicht mehr gepflegte Ruby Erweiterungen sollen durch moderne Implementierungen ersetzt werden.
Der Code Zoo (Jruby, Clojure, Java) soll aufgeräumt werden.

Wie kann man helfen?

Die Puppet Community möchte sich als Professionelles Open Source Projekt etablieren.
Dazu sucht man die finanzielle Unterstützung von Usern, Firmen und öffentlichen Einrichtungen für Open Source Arbeit.

Auch individuelle Zuarbeit ist gerne gesehen. Es muss Dokumentation geschrieben werden, die Pakete müssen getestet werden.
Die Community braucht die Unterstützung der Anwender, sei es durch Zuarbeit, Code Reviews oder Bug Reports.

Im nächsten Schritt benötigt man Spiegel Server, Bandbreite und Storage.

Professioneller Support

Open Source Support ist für viele Kunden ein wichtiger Aspekt bei der Wahl der Mittel.
Für OpenVox gibt es bereits jetzt schon 2 Firmen, die Support anbieten:

Wir von der betadots GmbH unterstützen Kunden bei Planung, Installation, Updates von OpenVox, Puppet Open Source und Puppet Enterprise und kümmern uns um die Weiterbildung der Mitarbeiter im Umfeld von Git und GitLab, Foreman/Katello, Puppet/OpenVox und Linux HA.

In Amerikanischem Raum wird die Firma overlookinfratech Ansprechpartner für OpenVox Support sein.

Weitere Firmen werden auf der Support Übersicht von OpenVox gelistet werden.

Zusammenfassung

OpenVox ist ein drop-in Ersatz für Puppet Open Source.

Die Community hat gezeigt, dass sie den Wunsch nach Weiterentwicklung hat und in der Lage ist diese Arbeiten durchzuführen.

Professioneller Support und Trainings stehen in den USA und in Europa zur Verfügung.

Open Source ist die Basis für den Erfolg vieler Unternehmen. Deshalb: Unterstützt Eure Community.

Für weitere Informationen stehen wir gerne zur Verfügung.

Datacenters need automation

betadots GmbH

Overlook InfraTech: First release, hot off the presses!

2025-01-21T00:00:00+00:00

It’s been quite a journey, y’all. But we’re excited to announce the first release of OpenVox, the community-maintained open source implementation of Puppet. OpenVox 8.11 is functionally equivalent to Puppet and should be a drop-in replacement. Be aware, of course, that even though you can type the same commands, use all the same modules and extensions, and configure the same settings, OpenVox is not yet tested to the same standard that Puppet is.

Migrating is fairly simple, just replace the packages following instructions on the handy dandy new install page. You’ll notice that they’re still using the apt|yum.overlookinfratech.com repositories. As we get our infrastructure built out, these will probably be moved to the voxpupuli.org namespace. Please don’t use these packages on critical production infrastructures yet, unless you’re comfortable with troubleshooting and reporting back on the silly errors we’ve made while rebranding and rebuilding.

If you’d like professional assistance in the migration, check out our new OpenVox migration service.

We consider OpenVox a soft-fork because we intend to maintain downstream compatibility for as long as we are able. As such, Vox Pupuli is working with Perforce create a Puppet™️ Standards Steering Committee to set the direction of features and language evolutions.

The OpenVox project goals are pretty straightforward and aim to alleviate pain points observed by the community over the last few years:

Modernizing the OpenVox codebase and ecosystem including supporting current operating systems.
Recentering and focusing on community requirements; user needs will drive development.
Democratizing platform support by allowing community members to contribute what they need instead of waiting for business requirements to align.
Maintaining an active and responsive open source community like the rest of Vox Pupuli’s namespace.

Migration Service

Overlook InfraTech is an actual company with bills to pay and everything, so you knew there was going to be a plug here eventually. But we’ll keep it short and tasteful. We have been instrumental to the OpenVox fork right from the very beginning and we know inside and out what changes to expect. We are best situated to help you, so if you’d like professional assistance, or even just someone to look over your (virtual) shoulder during the process, we have a very reasonably priced migration service. Whether you’re ready to jump in and try it out today, or you’d rather wait a bit and let the dust settle, we’ll be here to help.

betadots: The Ruby side of Puppet - Part 3 - Custom Types and Providers

2025-01-20T18:50:29+00:00

This is the final post in a three-part serie, covering the concepts and best practices for extending Puppet using custom facts, custom functions and custom types and providers.

Part 1 explores how to create custom facts, which allow nodes to send information to the Puppet Server.

Part 2 discusses building custom functions to process data or execute specific tasks.

Part 3 (this post) focuses on custom types and providers, allowing you to extend Puppet's DSL and control system resources.

Types are at the core of Puppet’s declarative DSL. Types describe the desired state of the system, targeting specific, configurable parts (sometimes OS-specific). Puppet provides a set of core types, available with any Puppet agent installation.

Some of the core types include:

file
user
group
package
service

Why Create Custom Types and Providers:

There are several reasons to develop custom types and providers:

Avoid relying on the often unreliable or hard-to-maintain exec resources
Manage an application that requires CLI commands for configuration.
Handle configurations with highly specific syntax that existing Puppet types cannot manage (like file or concat).

Content:

General concepts
Types and providers in modules
Type description
Provider implementation
Using custom types in Puppet DSL
Resource API
Summary

General concepts

A custom type consists of two main parts:

Type definiton: This defines how the type will be used in the Puppet DSL.
Provider implementation(s): One or more providers per type define how to interact with the system to manage resources.

Type definition

The type describes how Puppet resources are declared in the DSL. For example, to manage an application’s configuration:

app_config { <setting>:
  ensure   => present,
  property => <value>,
  param    => <app_args>,
  provider => <auth_method>,
}

In the type definition, you typically have a namevar (a key identifier) and mutiple parameters and properties.

Namevar: The key identifying the resource instance in the type declaration.
Properties: These represent something measurable on the target system, such as a user’s UID or GID, or an application’s config value.
Parameters: Parameters influence how Puppet manages a resource but don’t directly map to something measurable on the system. For example, manage_home in the user type is a parameter that affects Puppet’s behavior but isn’t a property of the user account.

The difference between parameters and properties is nicely described on the Puppet Type/Provider development page:

Properties:

"Properties correspond to something measurable on the target system. For example, the UID and GID of a user account are properties, because their current state can be queried or changed. In practical terms, setting a value for a property causes a method to be called on the provider."

Parameters:

"Parameters change how Puppet manages a resource, but do not necessarily map directly to something measurable. For example, the user type’s managehome attribute is a parameter — its value affects what Puppet does, but the question of whether Puppet is managing a home directory isn’t an innate property of the user account."

In our example, the property is the value of the config setting, whereas the parameter specifies application attributes.

Provider implementations

Providers define the mechanisms for managing the state of the resources described by types. They handle:

Determining whether the resource already exists.
Creating or removing resources.
Modifying resources.
Optionally, listing all existing resources of the type.

Types and providers in modules

Custom types and providers are placed within the lib/puppet directory of a module.

The type file is located in lib/puppet/type and named after the resource type (app_config.rb in this case).
Provider implementations go into the lib/puppet/provider directory, inside a subdirectory named after the resource type. Each provider is named according to its provider implementation (e.g., ruby.rb, cli.rb, cert.rb, token.rb, user.rb)

For example:

Type: app_config
Providers:
- cert
- token
- user

# <modulepath>/<modulename>
modules/application/
    \- lib/
        \- puppet/
            |- type/
            |    \- app_config.rb
            \- provider/
                \- app_config/
                    |- cert.rb
                    |- token.rb
                    \- user.rb

Type description

New custom types can be crated using two different API versions.
APIv1 is the old, classic way using getters and setters within the providers.
APIv2 is a new implementation which is integrated into PDK - this implementation is also called Resource API.

In the next section we introduce the APIv1 implementation. The Resource API implementation is handled later in this document.

APIv1

The traditional method uses Puppet::Type.newtype, which defines the type.
It is recommended to add the type documentation into the code.
This allows users to run puppet describe <type> on their system to display the documentation.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  # ... the code ...
end

Management

The most important feature of a type is to add or remove something from the system.
This is usually handled with the ensure property.
To enable ensure, a single line is needed: ensurable

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
end

Namevar

When declaring the type, one must specify a title - one could refer to this as a type instance identifier.
Usually this reflects for something the type is managing.

user { 'betadots': # <- title
}

The most simple implementation is to add a parameter with the name :name.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
end

Passing the namevar: true key to the parameter is another way to identify a namevar:

newparam(:key, namevar: true) do
  desc 'The app config setting key to manage. see app_cli conf --help'
end

Properties

Next we add the other property. Each property can be validated by its content.
In our demo case we expect a string value.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value)do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
end

Besides this one can provide specific valid values which are validated automatically:

newproperty(:enable) do
  newvalue(:true)
  newvalue(:false)
end

Please note that arrays as property values are validated in a different way:

From Puppet custom types website:

By default, if a property is assigned multiple values in an array:

It is considered in sync if any of those values matches the current value.

If none of those values match, the first one is used when syncing the property.

If all array values should be matched, the property needs the
array_matchingto be set to :all. The default value is :first

newproperty(:flags, :array_matching => :all) do
  # ...
end

Accessing values can be done in two ways:

should for properties, value for parameters
value for both, properties and parameters

We prefer to use the explizit methods as this makes it more clear, whether we deal with a property or a parameter.

Parameters

Parameters are defined in a similar way:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        cli_args => ['-p'], # persistency
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value) do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI options to use during command execution."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s is not a an array" % value
      end
    end
    defaultto ['-p']
  end
end

Boolean parameters

Parameter which get a bool value should be handled in a different way to avoid code repetition

require 'puppet/parameter/boolean'
# ...
newparam(:force, :boolean => true, :parent => Puppet::Parameter::Boolean)

Automatic relationships

Within the type we can specify soft dependencies between different types.

E.g. the app_cli should use a user which must be available on the system

autorequire(:user) do
  ['app']
end

From now on the new custom type can already be used in Puppet DSL, the compiler will create a catalog but the agent will produce an error, as there are no functional providers.

Agent side prerun evaluation

It is possible to have the agent first check some things prior the catalog will be applied.
This is what the :pre_run_check method can be used for

def pre_run_check
  File.exist?('/opt/app/bin/app.exe') && raise Puppet::Error, "App not installed"
end

Features

When using multiple providers (similar to the package resource) we want to be sure that the provider has all required implementations (features).
A type can require a feature:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        file     => '/opt/app/etc/app.cfg',
        cli_args => ['-p'], # persistency
      }
  }
  ensurable
  # global feature
  # feature :cli, "The cli feature requires some parameters", :methods => [:cli]
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value) do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
  # The property config_file must be set when using the cli 
  #   option to configure app.
  # The cli provider will check for a feature.
  newproperty(:file, :required_features => %w{cli}) do
    desc "The config file to use."
    validate do |value|
      unless (File.expand_path(value) == value)
        raise ArgumentError, "%s is not an absolute path" % value
      end
    end
    defaultto: '/opt/app/etc/app.cfg'
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI options to use during command execution."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s is not a an array" % value
      end
    end
    defaultto: ['-p']
  end
end

Provider implementation

Once the type is defined, the provider must handle how the resource is managed. Providers typically implement methods to:

Check if the resource exists (exists?).
Create a new resource (create).
Read existing resource properties (prefetch or a getter)
Modify a resource (flush or a setter).
Delete a resource (destroy).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider."
end

It is also possible to re-use and extend existing providers classes.
Common code can be placed inside a generic provider (lib/puppet/provider/app_config.rb).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli, :parent => Puppet::Provider::App_config) do
  desc "The app_config types cli provider reuses shared/common code from app_config.rb."
end

If you want to add shared functionality to multiple providers, you can place your code into the PuppetX module directory: lib/puppet_x/<company_name>/<unique class name>.

# lib/puppet/provider/app_config/cli.rb
require_relative '../../puppet_x/betadots/app_api.rb'
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider reuses shared/common code from app_config.rb."
end

A new provider can be created by inheriting from and extending an existing provider:

# lib/puppet/provider/app_config/token.rb
Puppet::Type.type(:app_config).provide(:token, :parent => :api, :source => :api) do
  desc "The app_config types token provideris an extension to the api provider."
end

Additionally one can reuse any provider from any type:

# lib/puppet/provider/app_config/file.rb
Puppet::Type.type(:app_config).provide(:file, :parent => Puppet::Type.type(:ini_setting).provider(:ruby)) do
  desc "The app_config types token provideris an extension to the api provider."
end

Selection of provider

The Puppet Agent must know which provider to use, if multiple providers are present.
You can use of the provider meta parameterr or let providers perform checks to determine if they are valid for the system. The options include:

Check	Example	Description
commands	`commands :app => "/opt/app/bin/app.exe"`	Valid if the command `/opt/app/bin/app.exe` exists. The command can be later used using `:app`.
confine - exists	`confine :exists => "/opt/app/etc/app.conf"`	Valid if the file exists.
confine - bool	`confine :true => /^10\./.match(%x{/opt/app/bin/app.exec version})`	Valid if Version is 10.x
confine - Fact	`confine 'os.family' => :debian`	Valid if the fact value matches
confing - feature	`confine :feature => :cli`	Valid if the provider has the feature
defaultfor	`defaultfor 'os.family' => :debian`	Use this provider as default for Debian-based systems.

Reading and applying configuration

The provider needs the ability to create, read, update and delete individual configuration states (CRUD API).

Each configuration property needs a getter (read) and a setter (modify) method.

# lib/puppet/provider/app_config/cli.rb
#   app_config: { 'enable_logging':
#     ensure   => present,
#     value    => true,
#   }
#
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider."

  # confine to existing command
  commands :app => "/opt/app/bin/app.exe"

  def exists?
    @result = app_cli('list').split(%r{\n}).grep(%r{^#{resource[:key]}})
    if @result.length > 1
      raise ParserError, 'found multiple config items found, please fix this'
    end
    return false if @result.empty?
    return true unless @result.empty?
  end

  def create
    app(['set', resource[:name], resource[:value]])
  end

  def destroy
    app(['rm', resource[:name]])
  end

  # getter
  def value()
    @result[0].split[1]
  end

  # setter
  def value=(value)
    app(['set', resource[:name], resource[:value]])
  end
end

It is recommended to also create the instances class method, which can collect all instances of a resource type into a hash.
The namevar is the hash key, which has a hash of paramters and properties as value.

require 'yaml'
def self.instances
  instances = []
  YAML.load(app_cli('list')).each do |key, value|
    attributes_hash = { key: key, ensure: :present, value: value, name: key}
    instances << new(attributes_hash)
  end
  instances
end

Sometimes it is not possible to collect all instances, such as with the file resource. In such cases no instance method is defined.

The instance method is used indirectly be each type when calling prefetch to get the acutal configuration and returns the@property_hash instance variable.

def self.prefetch(resources)
  resources.each_key do |name|
    provider = instances.find { |instance| instance.name == name }
    resources[name].provider = provider if provider
  end
end

This allows the type getter and setter to read an manipulate the instance variable, instead of writing getter and setter methods for each type.
This behavior is added by declaring the mk_resource_methods class method.

Once this is implemented you can run the command puppet resource app_config to retrieve all existing configurations.

Refresh events

In some cases it is required to refresh a resource, for example, to restart a service, remount a mount, or rerun an exec resource. To allow a type/provider to react to a refresh event, it needs special handling.

Within the type the refreshable feature must be activated and a refresh definition is added.

The following examples are taken from puppet service type and provider. The feature describes, which provider definition should be executed upon refresh event.

# lib/puppet/type/services.rb
# ...
  feature :refreshable, "The provider can restart the service.",
    :methods => [:restart]
# ...
  def refresh
    # Only restart if we're actually running
    if (@parameters[:ensure] || newattr(:ensure)).retrieve == :running
      provider.restart
    else
      debug "Skipping restart; service is not running"
    end
  end
# ...

Using custom types in Puppet DSL

Once the custom type and provider are implemented, you can use them in your manifests as follows:

# Type declaration
app_config { 'enable_logging':
  ensure    => present,
  value     => true,
  require   => File['/opt/app/etc/app.cfg'],
  subscribe => Service|'app'|,
}
# Type reference
service { 'app':
  ensure    => running,
  subscribe => App_config['enable_logging'],
}
# Type declaration using lambda and splat operator
$config_hash.each |String $key, Hash $hash| {
  app_config { $key:
    * => $hash,
  }
}
# Virtual or exported resource
@@app_config { 'app_mount':
  ensure => present,
  value  => "${facts['networking']['fqdn']}/srv/app_mount",
  tag    => 'blog',
}
# Resource collector
App_config <<| tag == 'blog' |>>

Resource API

The modern resource API has one limitation: one can not refresh types!
Besides this it also consists of types and providers which must be placed at the same location as the existing implementation.

Resource API Type

The resource API type uses an attributes hash to list all parameters and properties. Within the type key one can use any of the built-in Puppet data types.

# lib/puppet/type/app_config.rb
require 'puppet/resource_api'

Puppet::ResourceApi.register_type(
  name: 'app_config',
  docs: <<-EOS,
      @summary The type to manage app config settings
      @example
      app_config { 'key':
        ensure => present,
        value  => 'value',
      }

      This type provides Puppet with the capabilities to manage our application config
    EOS
  features: []
  attributes: {
    ensure: {
      type:    'Enum[present, absent]',
      desc:    'Whether the setting should be added or removed',
      default: 'present',
    },
    name: {
      type:     'String',
      desc:     'The setting to manage',
      behavior: :namevar,
    },
    value: {
      type: 'Variant[String, Integer, Array]',
      desc: 'The value to set'
    }
  }
)

Respource API Provider

The most simple solution is to make use of the SimpleProvider class.
This provider needs up to 4 defines:

get
create
update
delete

# lib/puppet/provider/app_config/cli.rb
require 'puppet/resource_api/simple_provider'
require 'open3'

class Puppet::Provider::AppConfig::Cli < Puppet::ResourceApi::SimpleProvider
  $app_command = '/opt/app/bin/app.exe'

  # Get: fetching all readable config settings into a hash
  def get(context)
    context.debug('Returning data from command')

    @result = []
    stdout, stderr, status = Open3.capture3('/opt/app/bin/app.exe list')
    raise ParseError "Error running command: \n#{stderr}" unless status.success?
    # Missing error handling. This is just for demo
    @command_result = stdout.split(%r{\n})
    @command_result.each do |line|
      next if line == '---'
      key = line.split(':')[0]
      value = line.split(':')[1].strip
      hash = {
        'ensure': 'present',
        'name': key,
        'value': value,
      }
      @result += [ hash ]
    end
    @result
  end

  # Create: add a new config setting with name and should value
  def create(context, name, should)
    context.notice("Creating '#{name}' with #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end

  # Update: correct an existing setting with name and replace with should value
  def update(context, name, should)
    context.notice("Updating '#{name}' with #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end

  # Delete: remove a config setting
  def delete(context, name)
    context.notice("Deleting '#{name}'")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe rm #{name}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end
end

Summary

Types and providers are not complex. They basically describe the CRUD behavior of Puppet.
The type defines how to use the Puppet DSL, while the provider controls how to check and handle specific settings.

Please stop using exec for almost anything that is not super simple. In most cases, a simple type and provider will allow better analysis, error handling and control over what is happening. Besides this: custom types and providers execute faster compared to exec type usage - far more faster!

The example application and the working types and providers are available on GitHub:

The app: workshop demo app
The Puppet module with types and providers for the app: workshop demo module

App usage:

# Install APP:
git clone https://github.com/betadots/workshop-demo-app /opt/app
/opt/app/bin/app.exe

# Install Puppet module:
git clone https://github.com/betadots/workshop-demo-module -b ruby_workshop modules/app
# Type API V1
puppet resource app_config --modulepath modules
# Type Resource API
puppet resource app_config2 --modulepath modules

Happy puppetizing,
Martin

betadots: Die Ruby-Seite von Puppet - Teil 3 - Benutzerdefinierte Typen und Provider

2025-01-20T18:48:32+00:00

Dies ist der letzte Beitrag in einer dreiteiligen Serie, die die Konzepte und Best Practices für die Erweiterung von Puppet mithilfe von benutzerdefinierten Fakten, benutzerdefinierten Funktionen und benutzerdefinierten Typen sowie Providern behandelt.

Teil 1 untersucht, wie man benutzerdefinierte Fakten erstellt, die es Knoten ermöglichen, Informationen an den Puppet-Server zu senden.

Teil 2 behandelt den Aufbau benutzerdefinierter Funktionen zur Verarbeitung von Daten oder zur Ausführung spezifischer Aufgaben.

Teil 3 (dieser Beitrag) konzentriert sich auf benutzerdefinierte Typen und Provider, mit denen Puppets DSL erweitert und Systemressourcen verwaltet werden können.

Typen stehen im Zentrum der deklarativen DSL von Puppet. Typen beschreiben den gewünschten Zustand des Systems und zielen auf spezifische, konfigurierbare Teile ab (manchmal betriebssystemspezifisch). Puppet bietet eine Reihe von Kern-Typen, die mit jeder Puppet-Agent-Installation verfügbar sind.

Einige der Kern-Typen umfassen:

file
user
group
package
service

Warum benutzerdefinierte Typen und Provider erstellen:

Es gibt mehrere Gründe, benutzerdefinierte Typen und Provider zu entwickeln:

Vermeiden von den oft unzuverlässigen oder schwer zu wartenden exec-Ressourcen.
Verwalten einer Anwendung, die CLI-Befehle zur Konfiguration benötigt.
Handhaben von Konfigurationen mit hochspezifischer Syntax, die bestehende Puppet-Typen nicht verwalten können (wie file oder concat).

Inhalt:

Allgemeine Konzepte
Typen und Provider in Modulen
Typbeschreibung
Provider-Implementierung
Verwendung benutzerdefinierter Typen in der Puppet DSL
Resourcen-API
Zusammenfassung

Allgemeine Konzepte

Ein Typ besteht aus zwei Hauptteilen:

Typdefinition: Diese definiert, wie der Typ in der Puppet DSL verwendet wird.
Provider-Implementierung(en): Ein oder mehrere Provider pro Typ definieren, wie mit dem System interagiert wird, um Ressourcen zu verwalten.

Typdefinition

Der Typ beschreibt, wie Puppet-Ressourcen in der DSL deklariert werden. Zum Beispiel, um die Konfiguration einer Anwendung zu verwalten:

app_config { <setting>:
  ensure   => present,
  property => <value>,
  param    => <app_args>,
  provider => <auth_method>,
}

Typdefinition haben einen Namevar (ein Schlüsselbezeichner) und mehrere Parameter und Properties (Eigenschaften).

Namevar: Der Schlüssel, der die Ressourceninstanz in der Typdeklaration identifiziert.
Properties (Eigenschaften): Diese repräsentieren etwas Messbares im Zielsystem, wie die UID oder GID eines Benutzers oder einen Konfigurationswert einer Anwendung.
Parameter: Parameter beeinflussen, wie Puppet eine Ressource verwaltet, spiegeln jedoch nicht direkt etwas Messbares im System wider. Zum Beispiel ist manage_home im user-Typ ein Parameter, der Puppets Verhalten beeinflusst, aber keine Eigenschaft des Benutzerkontos ist.

Der Unterschied zwischen Parametern und Eigenschaften wird auf der Puppet-Typ/Provider-Entwicklungsseite gut beschrieben:

Properties:

"Eigenschaften entsprechen etwas Messbarem im Zielsystem. Zum Beispiel sind die UID und GID eines Benutzerkontos Eigenschaften, da ihr aktueller Zustand abgefragt oder geändert werden kann. Praktisch bedeutet das, dass das Festlegen eines Wertes für eine Eigenschaft eine Methode im Provider aufruft."

Parameter:

"Parameter ändern, wie Puppet eine Ressource verwaltet, entsprechen jedoch nicht unbedingt direkt etwas Messbarem. Zum Beispiel ist das managehome-Attribut des Benutzertyps ein Parameter – sein Wert beeinflusst, was Puppet tut, aber die Frage, ob Puppet ein Home-Verzeichnis verwaltet, ist keine angeborene Eigenschaft des Benutzerkontos."

In unserem Beispiel ist die Eigenschaft der Wert des Konfigurationseinstellungen, während der Parameter die Anwendungsattribute angibt.

Provider-Implementierungen

Provider definieren die Mechanismen zur Verwaltung des Zustands der durch Typen beschriebenen Ressourcen. Sie behandeln:

Bestimmung, ob die Ressource bereits existiert
Erstellen oder Entfernen von Ressourcen.
Ändern von Ressourcen.
Optional das Auflisten aller vorhandenen Ressourcen des Typs.

Typen und Provider in Modulen

Benutzerdefinierte Typen und Provider werden im Verzeichnis lib/puppet eines Moduls platziert.

Die Typdatei befindet sich in lib/puppet/type und ist nach dem Ressourcentyp benannt (app_config.rb in diesem Fall).
Provider-Implementierungen gehen in das Verzeichnis lib/puppet/provider, in ein Unterverzeichnis, das nach dem Ressourcentyp benannt ist. Jeder Provider ist nach seiner Provider-Implementierung benannt (z.B. ruby.rb, cli.rb, cert.rb, token.rb, user.rb)

Beispiel:

Typ: app_config
Provider:
- cert
- token
- user

# <modulpfad>/<modulname>
modules/application/
    \- lib/
        \- puppet/
            |- type/
            |    \- app_config.rb
            \- provider/
                \- app_config/
                    |- cert.rb
                    |- token.rb
                    \- user.rb

Typbeschreibung

Neue benutzerdefinierte Typen können mit zwei verschiedenen API-Versionen erstellt werden. APIv1 ist die alte, klassische Methode, die Getter und Setter innerhalb der Provider verwendet. APIv2 ist eine neue Implementierung, die in PDK integriert ist – diese Implementierung wird auch als Resource-API bezeichnet.

Im nächsten Abschnitt stellen wir die APIv1-Implementierung vor. Die Implementierung der Ressourcen-API wird später in diesem Dokument behandelt.

APIv1

Die traditionelle Methode verwendet Puppet::Type.newtype, die den Typ definiert. Es wird empfohlen, die Typdokumentation in den Code einzufügen. Dies ermöglicht es den Benutzern, puppet describe <type> auf ihrem System auszuführen, um die Dokumentation anzuzeigen.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten
    zur Authentifizierung: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  # ... der Code ...
end

Verwaltung

Die wichtigste Funktion eines Typs besteht darin, etwas zum System hinzuzufügen oder zu entfernen. Dies wird normalerweise mit der ensure-Eigenschaft behandelt. Um ensure zu aktivieren, ist eine einzige Zeile erforderlich: ensurable

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten
    zur Authentifizierung: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
end

Namevar

Bei der Deklaration des Typs muss ein Titel angegeben werden – dies könnte man als Typinstanz-Identifikator bezeichnen. In der Regel spiegelt dies wider, wofür der Typ verantwortlich ist.

user { 'betadots': # <- Titel
}

Die einfachste Implementierung besteht darin, einen Parameter mit dem Namen :name hinzuzufügen.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
end

Das Übergeben des Schlüssels namevar: true an den Parameter ist eine weitere Möglichkeit, einen Namevar zu identifizieren:

newparam(:key, namevar: true) do
  desc 'Der Schlüssel des Anwendungsconfig-Elements, das verwaltet werden soll. Siehe app_cli conf --help'
end

Eigenschaften

Als Nächstes fügen wir die andere Eigenschaft hinzu. Jede Eigenschaft kann durch ihren Inhalt validiert werden. In unserem Demo-Fall erwarten wir einen Stringwert.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
  newproperty(:value) do
    desc "Der zu setzende Konfigurationswert."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s ist kein gültiger Wert" % value
      end
    end
  end
end

Außerdem kann man spezifische gültige Werte angeben, die automatisch validiert werden:

newproperty(:enable) do
  newvalue(:true)
  newvalue(:false)
end

Bitte beachten, dass Arrays als Eigenschaftswerte auf andere Weise validiert werden:

Von der Website Puppet benutzerdefinierte Typen:

Standardmäßig wird eine Eigenschaft, die mehrere Werte in einem Array zugewiesen bekommt:

Als synchron betrachtet, wenn einer dieser Werte dem aktuellen Wert entspricht.

Wenn keiner dieser Werte übereinstimmt, wird der erste Wert beim Synchronisieren der Eigenschaft verwendet.

Wenn alle Array-Werte übereinstimmen sollen, muss die Eigenschaft array_matching auf :all gesetzt werden. Der Standardwert ist :first.

newproperty(:flags, :array_matching => :all) do
  # ...
end

Der Zugriff auf Werte kann auf zwei Arten erfolgen:

should für Eigenschaften, value für Parameter.
value für sowohl Eigenschaften als auch Parameter.

Wir bevorzugen die expliziten Methoden, da dies klarer macht, ob wir es mit einer Eigenschaft oder einem Parameter zu tun haben.

Parameter

Parameter werden auf ähnliche Weise definiert:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        cli_args => ['-p'], # Persistenz
      }
  }
  ensurable
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
  newproperty(:value) do
    desc "Der zu setzende Konfigurationswert."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s ist kein gültiger Wert" % value
      end
    end
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI-Optionen, die während der Befehlsausführung verwendet werden sollen."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s ist kein Array" % value
      end
    end
    defaultto ['-p']
  end
end

Boolean-Parameter

Parameter, die einen booleschen Wert erhalten, sollten auf andere Weise behandelt werden, um Codewiederholungen zu vermeiden.

require 'puppet/parameter/boolean'
# ...
newparam(:force, :boolean => true, :parent => Puppet::Parameter::Boolean)

Automatische Abhängigkeiten

Innerhalb des Typs können wir weiche Abhängigkeiten zwischen verschiedenen Typen angeben.

Beispiel: Das app_cli sollte einen Benutzer verwenden, der im System verfügbar sein muss.

autorequire(:user) do
  ['app']
end

Von nun an kann der neue benutzerdefinierte Typ bereits in Puppet DSL verwendet werden, der Compiler wird ein Katalog erstellen, aber der Agent wird einen Fehler produzieren, da es keine funktionalen Provider gibt.

Vorabprüfung auf der Agentenseite

Es ist möglich, dass der Agent zunächst einige Dinge überprüft, bevor der Katalog angewendet wird. Dafür kann die Methode :pre_run_check verwendet werden.

def pre_run_check
  File.exist?('/opt/app/bin/app.exe') && raise Puppet::Error, "App nicht installiert"
end

Merkmale

Bei der Verwendung mehrerer Provider (ähnlich der Ressourcenpakete) wollen wir sicherstellen, dass der Provider alle erforderlichen Implementierungen (Merkmale - Features) hat. Ein Typ kann ein Merkmal erfordern:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        file     => '/opt/app/etc/app.cfg',
        cli_args => ['-p'], # Persistenz
      }
  }
  ensurable
  # globales Merkmal
  # feature :cli, "Das CLI-Merkmal erfordert einige Parameter", :methods => [:cli]
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
  newproperty(:value) do
    desc "Der zu setzende Konfigurationswert."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s ist kein gültiger Wert" % value
      end
    end
  end
  # Die Eigenschaft config_file muss gesetzt werden, wenn die CLI
  #   Option zur Konfiguration von App verwendet wird.
  # Der CLI-Provider wird nach einem Merkmal suchen.
  newproperty(:file, :required_features => %w{cli}) do
    desc "Die zu verwendende Konfigurationsdatei."
    validate do |value|
      unless (File.expand_path(value) == value)
        raise ArgumentError, "%s ist kein absoluter Pfad" % value
      end
    end
    defaultto '/opt/app/etc/app.cfg'
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI-Optionen, die während der Befehlsausführung verwendet werden sollen."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s ist kein Array" % value
      end
    end
    defaultto ['-p']
  end
end

Provider-Implementierung

Sobald der Typ definiert ist, muss der Anbieter steuern, wie die Ressource verwaltet wird. Anbieter implementieren typischerweise Methoden, um:

Überprüfen, ob die Ressource existiert (exists?).
Eine neue Ressource erstellen (create).
Vorhandene Ressourcenattribute lesen (prefetch oder ein Getter).
Eine Ressource ändern (flush oder ein Setter).
Eine Ressource löschen (destroy).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli) do
  desc "Der CLI-Anbieter des app_config-Typs."
end

Es ist auch möglich, bestehende Anbieterklassen wiederzuverwenden und zu erweitern. Gemeinsamer Code kann in einem generischen Anbieter (lib/puppet/provider/app_config.rb) platziert werden.

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli, :parent => Puppet::Provider::App_config) do
  desc "Der CLI-Anbieter des app_config-Typs verwendet gemeinsamen Code aus app_config.rb."
end

Wenn eine gemeinsame Funktionalität für mehrere Anbieter hinzugefügt werden soll, kann man den Code im Puppet_X-Modulverzeichnis ablegen: lib/puppet_x/<unternehmens_name>/<eindeutiger Klassenname>.

# lib/puppet/provider/app_config/cli.rb
require_relative '../../puppet_x/betadots/app_api.rb'
Puppet::Type.type(:app_config).provide(:cli) do
  desc "Der CLI-Anbieter des app_config-Typs verwendet gemeinsamen Code aus app_config.rb."
end

Ein neuer Anbieter kann erstellt werden, indem er von einem bestehenden Anbieter erbt und diesen erweitert:

# lib/puppet/provider/app_config/token.rb
Puppet::Type.type(:app_config).provide(:token, :parent => :api, :source => :api) do
  desc "Der Token-Anbieter des app_config-Typs ist eine Erweiterung des API-Anbieters."
end

Zusätzlich kann man jeden Anbieter von jedem Typ wiederverwenden:

# lib/puppet/provider/app_config/file.rb
Puppet::Type.type(:app_config).provide(:file, :parent => Puppet::Type.type(:ini_setting).provider(:ruby)) do
  desc "Der Datei-Anbieter des app_config-Typs ist eine Erweiterung des API-Anbieters."
end

Auswahl des Anbieters

Der Puppet-Agent muss wissen, welcher Anbieter zu verwenden ist, wenn mehrere Anbieter vorhanden sind. Man kann den provider-Meta-Parameter verwenden oder die Anbieter Überprüfungen durchführen lassen, um festzustellen, ob sie für das System gültig sind. Die Optionen umfassen:

Überprüfung	Beispiel	Beschreibung
Befehle	`commands :app => "/opt/app/bin/app.exe"`	Gültig, wenn der Befehl `/opt/app/bin/app.exe` existiert. Der Befehl kann später mit `:app` verwendet werden.
Einschränkung - existiert	`confine :exists => "/opt/app/etc/app.conf"`	Gültig, wenn die Datei existiert.
Einschränkung - bool	`confine :true => /^10\./.match(%x{/opt/app/bin/app.exec version})`	Gültig, wenn die Version 10.x ist.
Einschränkung - Fakt	`confine 'os.family' => :debian`	Gültig, wenn der Faktwert übereinstimmt.
Einschränkung - Funktion	`confine :feature => :cli`	Gültig, wenn der Anbieter die Funktion hat.
defaultfor	`defaultfor 'os.family' => :debian`	Anbieter als Standard für Debian-basierte Systeme.

Lesen und Anwenden von Konfigurationen

Der Anbieter benötigt die Fähigkeit, einzelne Konfigurationszustände zu erstellen, zu lesen, zu aktualisieren und zu löschen (CRUD-API).

Jede Konfigurationseigenschaft benötigt eine getter- (lesen) und eine setter- (ändern) Methode.

# lib/puppet/provider/app_config/cli.rb
#   app_config: { 'enable_logging':
#     ensure   => present,
#     value    => true,
#   }
#
Puppet::Type.type(:app_config).provide(:cli) do
  desc "Der CLI-Anbieter des app_config-Typs."

  # Einschränkung auf bestehenden Befehl
  commands :app => "/opt/app/bin/app.exe"

  def exists?
    @result = app_cli('list').split(%r{\n}).grep(%r{^#{resource[:key]}})
    if @result.length > 1
      raise ParserError, 'Mehrere Konfigurationselemente gefunden, bitte beheben Sie dies'
    end
    return false if @result.empty?
    return true unless @result.empty?
  end

  def create
    app(['set', resource[:name], resource[:value]])
  end

  def destroy
    app(['rm', resource[:name]])
  end

  # getter
  def value()
    @result[0].split[1]
  end

  # setter
  def value=(value)
    app(['set', resource[:name], resource[:value]])
  end
end

Es wird empfohlen, auch die instances-Klassenmethode zu erstellen, die alle Instanzen eines Ressourcentyps in einem Hash sammeln kann. Der Namevar ist der Hash-Schlüssel, der einen Hash von Parametern und Eigenschaften als Wert hat.

require 'yaml'
def self.instances
  instances = []
  YAML.load(app_cli('list')).each do |key, value|
    attributes_hash = { key: key, ensure: :present, value: value, name: key}
    instances << new(attributes_hash)
  end
  instances
end

Manchmal ist es nicht möglich, alle Instanzen zu sammeln, z. B. beim file-Ressourcentyp. In solchen Fällen wird keine instance-Methode definiert.

Die instance-Methode wird indirekt von jedem Typ verwendet, wenn prefetch aufgerufen wird, um die aktuelle Konfiguration zu erhalten und die @property_hash-Instanzvariable zurückzugeben.

def self.prefetch(resources)
  resources.each_key do |name|
    provider = instances.find { |instance| instance.name == name }
    resources[name].provider = provider if provider
  end
end

Dies ermöglicht es dem Typ, Getter und Setter zu verwenden, um die Instanzvariable zu lesen und zu manipulieren, anstatt für jeden Typ getter- und setter-Methoden zu schreiben. Dieses Verhalten wird durch die Deklaration der mk_resource_methods-Klassenmethode hinzugefügt.

Sobald dies implementiert ist, kann man den Befehl puppet resource app_config ausführen, um alle vorhandenen Konfigurationen abzurufen.

Refresh-Ereignisse

In einigen Fällen ist es erforderlich, eine Ressource zu aktualisieren (refresh), beispielsweise um einen Dienst neu zu starten, ein Laufwerk erneut zu mounten oder eine Exec-Ressource erneut auszuführen. Damit ein Typ/Anbieter auf ein Aktualisierungsereignis reagieren kann, ist eine spezielle Behandlung erforderlich.

Innerhalb des Typs muss die refreshable-Funktion aktiviert werden, und eine refresh-Definition wird hinzugefügt.

Die folgenden Beispiele stammen vom Puppet service Typ und Anbieter. Die Funktion beschreibt, welche Anbieterdefinition bei einem Aktualisierungsereignis ausgeführt werden soll.

# lib/puppet/type/services.rb
# ...
  feature :refreshable, "Der Anbieter kann den Dienst neu starten.",
    :methods => [:restart]
# ...
  def refresh
    # Nur neu starten, wenn wir tatsächlich laufen
    if (@parameters[:ensure] || newattr(:ensure)).retrieve == :running
      provider.restart
    else
      debug "Neustart überspringen; Dienst läuft nicht"
    end
  end
# ...

Verwendung benutzerdefinierter Typen in Puppet DSL

Sobald der benutzerdefinierte Typ und Anbieter implementiert sind, kann man diesen in Puppet Manifesten wie folgt verwenden:

# Typdeklaration
app_config { 'enable_logging':
  ensure    => present,
  value     => true,
  require   => File['/opt/app/etc/app.cfg'],
  subscribe => Service['app'],
}
# Typreferenz
service { 'app':
  ensure    => running,
  subscribe => App_config['enable_logging'],
}
# Typdeklaration mit Lambda und Splat-Operator
$config_hash.each |String $key, Hash $hash| {
  app_config { $key:
    * => $hash,
  }
}
# Virtuelle oder exportierte Ressource
@@app_config { 'app_mount':
  ensure => present,
  value  => "${facts['networking']['fqdn']}/srv/app_mount",
  tag    => 'blog',
}
# Ressourcen-Sammler
App_config <<| tag == 'blog' |>>

Resourcen-API

Die moderne Ressourcen-API hat eine Einschränkung: Man kann Typen nicht aktualisieren! Außerdem besteht sie aus Typen und Anbietern, die am selben Ort wie die bestehende Implementierung platziert werden müssen.

Ressourcen-API-Typ

Der Ressourcen-API-Typ verwendet einen Attributs-Hash, um alle Parameter und Eigenschaften aufzulisten. Innerhalb des type-Schlüssels kann man jeden der integrierten Puppet-Datentypen verwenden.

# lib/puppet/type/app_config.rb
require 'puppet/resource_api'

Puppet::ResourceApi.register_type(
  name: 'app_config',
  docs: <<-EOS,
      @summary Der Typ zur Verwaltung von App-Konfigurationseinstellungen
      @example
      app_config { 'key':
        ensure => present,
        value  => 'value',
      }

      Dieser Typ bietet Puppet die Möglichkeit, unsere Anwendungsconfig zu verwalten.
    EOS
  features: []
  attributes: {
    ensure: {
      type:    'Enum[present, absent]',
      desc:    'Ob die Einstellung hinzugefügt oder entfernt werden soll',
      default: 'present',
    },
    name: {
      type:     'String',
      desc:     'Die Einstellung, die verwaltet werden soll',
      behavior: :namevar,
    },
    value: {
      type: 'Variant[String, Integer, Array]',
      desc: 'Der Wert, der gesetzt werden soll'
    }
  }
)

Ressourcen-API-Anbieter

Die einfachste Lösung ist die Verwendung der Klasse SimpleProvider. Dieser Anbieter benötigt bis zu 4 Definitionen:

get
create
update
delete

# lib/puppet/provider/app_config/cli.rb
require 'puppet/resource_api/simple_provider'
require 'open3'

class Puppet::Provider::AppConfig::Cli < Puppet::ResourceApi::SimpleProvider
  $app_command = '/opt/app/bin/app.exe'

  # Get: Alle lesbaren Konfigurationseinstellungen in einen Hash abrufen
  def get(context)
    context.debug('Gibt Daten vom Befehl zurück')

    @result = []
    stdout, stderr, status = Open3.capture3('/opt/app/bin/app.exe list')
    raise ParseError "Fehler beim Ausführen des Befehls: \n#{stderr}" unless status.success?
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    @command_result = stdout.split(%r{\n})
    @command_result.each do |line|
      next if line == '---'
      key = line.split(':')[0]
      value = line.split(':')[1].strip
      hash = {
        'ensure': 'present',
        'name': key,
        'value': value,
      }
      @result += [ hash ]
    end
    @result
  end

  # Create: Eine neue Konfigurationseinstellung mit Name und Sollwert hinzufügen
  def create(context, name, should)
    context.notice("Erstelle '#{name}' mit #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    return true unless status.success?
  end

  # Update: Eine vorhandene Einstellung mit Namen korrigieren und durch den Sollwert ersetzen
  def update(context, name, should)
    context.notice("Aktualisiere '#{name}' mit #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    return true unless status.success?
  end

  # Delete: Eine Konfigurationseinstellung entfernen
  def delete(context, name)
    context.notice("Lösche '#{name}'")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe rm #{name}")
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    return true unless status.success?
  end
end

Zusammenfassung

Typen und Anbieter sind nicht komplex. Sie beschreiben im Grunde das CRUD-Verhalten von Puppet. Der Typ definiert, wie man die Puppet DSL verwendet, während der Anbieter steuert, wie spezifische Einstellungen überprüft und behandelt werden.

Eine exzessive Nutzung der exec Resource ist zu vermweiden. In den meisten Fällen ermöglicht ein einfacher Typ und Anbieter eine bessere Analyse, Fehlerbehandlung und Kontrolle über das, was passiert. Darüber hinaus: benutzerdefinierte Typen und Anbieter führen schneller aus im Vergleich zur Verwendung des exec-Typs - viel schneller!

Die Beispielanwendung und die funktionierenden Typen und Anbieter aus diesem Posting sind auf GitHub verfügbar:

Die App: Workshop-Demo-App
Das Puppet-Modul mit Typen und Anbietern für die App: Workshop-Demo-Modul

App-Nutzung:

# APP installieren:
git clone https://github.com/betadots/workshop-demo-app /opt/app
/opt/app/bin/app.exe

# Puppet-Modul installieren:
git clone https://github.com/betadots/workshop-demo-module -b ruby_workshop modules/app
# Type API V1
puppet resource app_config --modulepath modules
# Type Resource API
puppet resource app_config2 --modulepath modules

Viel Spaß beim Puppetisieren,

Martin

betadots: The Ruby side of Puppet - Part 2 - Custom Functions

2025-01-14T16:04:08+00:00

This is the second post in a series of three, covering the concepts and best practices for extending Puppet using custom facts, custom functions and custom types and providers.

Part 1 covers Custom Facts, which explain how a node can provide information to the Puppet Server.

Part 2 (this post) focuses on Custom Functions, demonstrating how to develop functions for data processing and execution.

Part 3 covers Custom Types and Providers, showing how to extend Puppet's DSL functionality.

Custom Functions in Puppet:

Functions in Puppet are executed on the Puppet server, specifically inside the compiler. These functions have access to all facts and Puppet variables, as long as they exist within the function’s namespace.

There are two main types of functions in Puppet:

Statement functions
Return value functions

Statement functions can interact with Puppet internals. The most common statement functions are include or contain, which add classes to the catalog, or noop and notice, which set all resources into simulation mode or print entries into the Puppet server log file.

Return value functions such as lookup or read_url, return values from Hiera or external sources like web servers.
Some return value functions can also manipulate data (each, filter, map, reduce).

Why Create Custom Functions?

Here are some reasons you might develop custom functions in Puppet:

Writing a custom Hiera lookup function
Converting data structures for specific needs

Content:

Function development
Functions in Modules
General API
Dispatcher and Define
Using functions in Puppet DSL
Summary

Function development

Since functions are executed only at compile time, it is advisable to first focus on their core functionality before integrating them with the Function API. This allows you to test the logic with local data, avoiding issues on the Puppet server that could break compilation or even crash the server.

For example, imagine you have a data structure that outlines various aspects of a system's configuration. Using a function, you can expose specific parts of this data structure as variables.

Example Data Structure:

server_hash:
  'postfix':
    'config':
      'transport': 'mta'
      'ssl': true
  'apache':
    'config':
      'vhost': 'server.domain.tld'
      'document_root': '/srv/www/html/server.domain.tld'
      'ssl': true
      'proxy_pass': 'http://localhost:8080'
  'tomcat':
    'config':
      'application': 'crm'
      'service': 'frontend'

Assume that this data structure comes from Hiera, and you want to allow other teams, who may write their own modules, to reuse this data without performing additional lookups. This would allow you to modify the data structure internally while still providing teams with the required information.

Example Function:

# test data
server_hash = {
  'postfix' => {
    'config' => {
      'transport' => 'mta',
      'ssl' => true
    }
  },
  'apache' => {
    'config' => {
      'vhost' => 'server.domain.tld',
      'document_root' => '/srv/www/html/server.domain.tld',
      'ssl' => true,
      'proxy_pass' => 'http://localhost:8080'
    }
  },
  'tomcat' => {
    'config' => {
      'application' => 'crm',
      'service' => 'frontend'
    }
  }
}
# function definition
def read_service_config(data, service)
  data[service]
end
# testing the function
puts read_service_config(server_hash, 'tomcat')

Functions in Modules

Puppet provides an API for creating custom functions. To ensure the Puppet agent can locate these functions, they must be placed in specific directories within a module.

Files inside the module's lib directory are synchronized across all agents. Note that it is not possible to limit which files are synchronized.

Custom functions can be added to one of two directory structures:

lib/puppet/parser/functions - Functions API v1
lib/puppet/functions - Functions API v2

The newer Functions API v2 supports subdirectories, allowing you to namespace your functions. The filename should match the function name.

In this post, we’ll focus on the modern API v2.

Example Directory Structure:

# <modulepath>/stdlib/
  \- lib/
    \- puppet/
      \- functions/
        \- stdlib/
          \- to_yaml.rb

# stdlib/lib/puppet/functions/stdlib/to_yaml.rb
Puppet::Functions.create_function(:'stdlib::to_yaml') do
  # ...
end

We recommend prefixing custom functions with the module name to avoid naming conflicts.

General API

To create custom functions in Puppet, use the following structure:

Example:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  # ...
end

Note that the namespace must be identical to the module name.

Dispatcher and Define

Within the do ... end of the function, you need to add a dispatcher. The dispatcher validates the input data and links it to a named definition, which executes the corresponding Ruby code.

The dispatcher can check data types using param and map values to Ruby symbols. The definition uses the dispatcher’s name and the parameters provided in the dispatcher.

The final command in the definition determines the return value.

Example Dispatcher for an IPv4 Address:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end
end

Here’s a table of common parameter methods used in dispatchers:

Method	Description
`param` or `required_param`	A mandatory argument. May occur multiple times. Position: All mandatory arguments must come first.
`optional_param`	An argument that can be omitted. You can use any number of these. When there are multiple optional arguments, users can only pass latter ones if they also provide values for the prior ones. This also applies to repeated arguments. Position: Must come after any required arguments.
`repeated_params` or `optional_repeated_params`	A repeatable argument, which can receive zero or more values. A signature can only use one repeatable argument. Position: Must come after any non-repeating arguments.
`required_repeated_param`	A repeatable argument, which must receive one or more values. A signature can only use one repeatable argument. Position: Must come after any non-repeating arguments.
`block_param` or `required_block_param`	A mandatory lambda (block of Puppet code). A signature can only use one block. Position: Must come after all other arguments.
`optional_block_param`	An optional lambda. A signature can only use one block. Position: Must come after all other arguments.

Adding a Definition:

Description	Dispatcher	Definition
Name	`dispatch :ip`	`def ip`
Parameter	`param ..., :ipaddr`	`def ip(ipaddr)`

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'resolv'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  def ip(ipaddr)
    Resolv.getname ipaddr
  end
end

Now additional dispatchers and definitions can be added:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'socket'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  dispatch :dnsname do
    param 'Regexp[/.*/]', :dnsname
  end

  dispatch :local_hostname do
    param 'Regexp[//]', :local_hostname
  end

  def ip(ipaddr)
    Addrinfo.tcp(ipaddr, '80').getnameinfo[0]
  end

  def dnsname(dnsname)
    Addrinfo.ip(dnsname).getnameinfo[0]
  end

  def local_hostname
    Socket.gethostname
  end
end

Using functions in Puppet DSL

Within Puppet the functions usually gets executed during compile time.
A special case is the deferred function which gets executed on the agent.

Let's consider two use-cases:

using resolv function on the compiler
using resolv function on the agent

Use case 1: compiler execution

The function can be used anywhere in the code:

class betadots::resolver (
  Stdlib::Fqdn $local_hostname = betadots::resolve(),
) {
  $ip_from_google = betadots::resolve('www.google.com')
}

Use case 2: agent execution

The function must be declared to run in deferred mode in Puppet. Avoid using the function within the class header, as this could potentially overwrite the function call from hiera data.

class betadots::local_resolve (
) {
  $local_api_ip = Deferred('betadots::resolve', ['api.int.domain.tld'])
}

Summary

Puppet provides a stable API for creating custom functions.
Before creating a new function, check whether a solutions already exists in Stdlib- or Extlib-Module.

Remember to prefix your functions with your module name to avoid conflicts and help identify their origin.

Happy puppetizing,
Martin

betadots: Die Ruby-Seite von Puppet - Teil 2 - Benutzerdefinierte Funktionen

2025-01-14T16:01:03+00:00

Dies ist der zweite Beitrag einer dreiteiligen Serie, in der die Konzepte und Best Practices zum Erweitern von Puppet mithilfe von benutzerdefinierten Facts, benutzerdefinierten Funktionen und benutzerdefinierten Typen und Providern behandelt werden.

Teil 1 behandelt Benutzerdefinierte Facts, die erklären, wie ein Knoten Informationen an den Puppet-Server übermitteln kann.

Teil 2 (dieser Beitrag) konzentriert sich auf Benutzerdefinierte Funktionen und zeigt, wie man Funktionen für Datenverarbeitung und Ausführung entwickelt.

Teil 3 stellt dar, wie man die DSL-Funktionalität von Puppet mit neuen Typen und Provider erweitert.

Benutzerdefinierte Funktionen in Puppet:

Funktionen in Puppet werden auf dem Puppet-Server, insbesondere im Compiler, ausgeführt. Diese Funktionen haben Zugriff auf alle Facts und Puppet-Variablen, solange sie mit dem Namensraum angegeben werden.

Es gibt zwei Haupttypen von Funktionen in Puppet:

Anweisungsfunktionen
Rückgabefunktionen

Anweisungsfunktionen können mit den internen Mechanismen von Puppet interagieren. Die gebräuchlichsten Anweisungsfunktionen sind include und contain, die Klassen zum Katalog hinzufügen, oder noop und notice, die alle Ressourcen in den Simulationsmodus versetzen oder Einträge in die Protokolldatei des Puppet-Servers schreiben.

Rückgabefunktionen wie lookup oder read_url geben Werte aus Hiera oder externen Quellen wie Webservern zurück. Einige Rückgabefunktionen können auch Daten manipulieren (each, filter, map, reduce).

Warum benutzerdefinierte Funktionen erstellen?

Hier sind einige Gründe, warum man benutzerdefinierte Funktionen in Puppet entwickeln könnte:

Erstellen einer benutzerdefinierten Hiera-Suchfunktion
Umwandeln von Datenstrukturen für spezifische Anforderungen

Inhalt:

Funktion-Entwicklung
Funktionen in Modulen
Allgemeine API
Dispatcher und Definition
Verwendung von Funktionen in der Puppet-DSL
Zusammenfassung

Funktion-Entwicklung

Da Funktionen nur zur Kompilierzeit ausgeführt werden, empfiehlt es sich, sich zunächst auf ihre Kernfunktionalität zu konzentrieren, bevor sie in die Funktions-API integriert werden. Dadurch kann die Logik mit lokalen Daten getestet werden, ohne dass Probleme auf dem Puppet-Server auftreten, die die Kompilierung unterbrechen oder den Server sogar zum Absturz bringen könnten.

Als Beispiel wird eine Datenstruktur definiert, die verschiedene Aspekte der Konfiguration eines Systems darstellt. Mit einer Funktion kann man bestimmte Teile dieser Datenstruktur als Variablen verfügbar machen.

Beispiel für eine Datenstruktur:

server_hash:
  'postfix':
    'config':
      'transport': 'mta'
      'ssl': true
  'apache':
    'config':
      'vhost': 'server.domain.tld'
      'document_root': '/srv/www/html/server.domain.tld'
      'ssl': true
      'proxy_pass': 'http://localhost:8080'
  'tomcat':
    'config':
      'application': 'crm'
      'service': 'frontend'

Angenommen, diese Datenstruktur stammt aus Hiera, und man möchte anderen Teams, die ihre eigenen Module schreiben, die Wiederverwendung dieser Daten ohne zusätzliche Abfragen ermöglichen. Dadurch kann man die interne Datenstruktur ändern und die Funktion anpassen, während die Teams weiterhin die benötigten Informationen zur Verfügung gestellt bekommen.

Beispiel für eine Funktion:

# Test Daten
server_hash = {
  'postfix' => {
    'config' => {
      'transport' => 'mta',
      'ssl' => true
    }
  },
  'apache' => {
    'config' => {
      'vhost' => 'server.domain.tld',
      'document_root' => '/srv/www/html/server.domain.tld',
      'ssl' => true,
      'proxy_pass' => 'http://localhost:8080'
    }
  },
  'tomcat' => {
    'config' => {
      'application' => 'crm',
      'service' => 'frontend'
    }
  }
}
# Funktion definition
def read_service_config(data, service)
  data[service]
end
# Testen der Funktion
puts read_service_config(server_hash, 'tomcat')

Funktionen in Modulen

Puppet stellt eine API zum Erstellen benutzerdefinierter Funktionen zur Verfügung. Damit der Puppet Compiler diese Funktionen finden kann, müssen sie in bestimmten Verzeichnissen innerhalb eines Moduls abgelegt werden.

Dateien im lib-Verzeichnis des Moduls werden auf alle Agenten synchronisiert. Es ist nicht möglich, zu steuern, welche Dateien synchronisiert werden.

Benutzerdefinierte Funktionen können zu einer der beiden folgenden Verzeichnisstrukturen hinzugefügt werden:

lib/puppet/parser/functions - Funktionen API v1
lib/puppet/functions - Funktionen API v2

Die neuere Funktionen-API v2 unterstützt Unterverzeichnisse, sodass man Funktionen in Namespaces organisieren können. Der Dateiname sollte mit dem Funktionsnamen übereinstimmen.

In diesem Beitrag konzentrieren wir uns auf die moderne API v2.

Beispiel für eine Verzeichnisstruktur:

# <modulepath>/stdlib/
  \- lib/
    \- puppet/
      \- functions/
        \- stdlib/
          \- to_yaml.rb

# stdlib/lib/puppet/functions/stdlib/to_yaml.rb
Puppet::Functions.create_function(:'stdlib::to_yaml') do
  # ...
end

Wir empfehlen, benutzerdefinierte Funktionen mit dem Modulnamen zu präfixen, um Namenskonflikte zu vermeiden.

Allgemeine API

Um benutzerdefinierte Funktionen in Puppet zu erstellen, verwendet man die folgende Struktur:

Beispiel:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  # ...
end

Bitte beachten, dass der Funktions Namespace identisch zum Modulnamen (betadots) sein muss.

Dispatcher und Definition

Innerhalb des do ... end Blocks der Funktion muss ein Dispatcher und eine Definition hinzugefügt werden. Der Dispatcher validiert die Eingabedaten und verknüpft sie mit einer benannten Definition, die den entsprechenden Ruby-Code ausführt.

Der Dispatcher kann Datentypen mithilfe von param prüfen und Werte Ruby-Symbolen zuordnen. Die Definition verwendet den Namen des Dispatchers und die im Dispatcher bereitgestellten Parameter.

Der letzte Befehl in der Definition bestimmt den Rückgabewert.

Beispiel für einen Dispatcher für eine IPv4-Adresse:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end
end

Hier ist eine Tabelle mit gängigen Methoden für Parameter in Dispatchern:

Methode	Beschreibung
`param` oder `required_param`	Ein obligatorisches Argument. Kann mehrfach verwendet werden. Position: Alle Pflichtargumente müssen zuerst erscheinen.
`optional_param`	Ein Argument, das ausgelassen werden kann.
`repeated_params` oder `optional_repeated_params`	Ein wiederholbares Argument, das null oder mehr Werte annehmen kann.
`required_repeated_param`	Ein wiederholbares Argument, das einen oder mehr Werte annehmen muss.
`block_param` oder `required_block_param`	Ein obligatorisches Lambda.
`optional_block_param`	Ein optionales Lambda.

Hinzufügen einer Definition:

Beschreibung	Dispatcher	Definition
Name	`dispatch :ip`	`def ip`
Parameter	`param ..., :ipaddr`	`def ip(ipaddr)`

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'resolv'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  def ip(ipaddr)
    Resolv.getname ipaddr
  end
end

Nun können zusätzliche Dispatcher und Definitionen hinzugefügt werden:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'socket'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  dispatch :dnsname do
    param 'Regexp[/.*/]', :dnsname
  end

  dispatch :local_hostname do
    param 'Regexp[//]', :local_hostname
  end

  def ip(ipaddr)
    Addrinfo.tcp(ipaddr, '80').getnameinfo[0]
  end

  def dnsname(dnsname)
    Addrinfo.ip(dnsname).getnameinfo[0]
  end

  def local_hostname
    Socket.gethostname
  end
end

Verwendung von Funktionen in der Puppet-DSL

Innerhalb von Puppet werden Funktionen in der Regel während der Kompilierzeit ausgeführt.
Eine besondere Ausnahme bildet die deferred function, die auf dem Agenten ausgeführt wird.

Im folgenden betrachten wir die zwei Anwendungsfälle:

Verwendung der resolve-Funktion im Compiler
Verwendung der resolve-Funktion auf dem Agenten

Anwendungsfall 1: Ausführung im Compiler

Die Funktion kann an beliebiger Stelle im Code verwendet werden:

class betadots::resolver (
  Stdlib::Fqdn $local_hostname = betadots::resolve(),
) {
  $ip_from_google = betadots::resolve('www.google.com')
}

Anwendungsfall 2: Ausführung auf dem Agenten

Die Funktion muss in deferred mode deklariert werden, um in Puppet auf dem Agenten ausgeführt zu werden. Die Funktion sollte nicht als Parameter im Header der Klasse verwendet werden, da dies möglicherweise den Funktionsaufruf mit hiera-Daten überschreiben könnte.

class betadots::local_resolve (
) {
  $local_api_ip = Deferred('betadots::resolve', ['api.int.domain.tld'])
}

Zusammenfassung

Puppet bietet eine stabile API zum Erstellen benutzerdefinierter Funktionen.
Bevor eine neue Funktion erstellt wird, bitte prüfen, ob bereits eine Lösung im Stdlib- oder Extlib-Modul vorhanden ist.

Funktionen sollten mit dem Modulnamen zu präfixen, um Konflikte zu vermeiden und deren Herkunft zu kennzeichnen.

Viel Erfolg beim Puppetisieren,
Martin

Overlook InfraTech: The strength of community

2025-01-08T00:00:00+00:00

Everything’s been super hectic with the holidays and so nobody noticed that I didn’t post the last two weeks, right? I joke, but not really. We’ve been super busy too and not just with the regular holiday stuff. In between the celebrations and glühwein, we’ve been working on community building.

As a community, we decided on the name for our new community maintained Puppet™️ fork and we decided that Vox Pupuli would adopt it. Now we’re busy creating the organization governance, porting all the repositories in, and setting up the pipelines and build tooling to streamline package publication. You’ll see a lot more information about things soon when everything’s done and we make the official announcement, but that’s a lot of work for a few weeks over the holidays when people are usually checked out!

Our biggest remaining challenge right now is determining release criteria that balances establishing our own brand presence against ecosystem safety all while respecting Perforce’s trademarks. I’m talking about agreeing on what names need to be changed before we can release packages and call them our own. I’m grateful for David, Jake, Sara, and others from Perforce who have given us loads of their own time to help us get to a resolution in everyone’s benefit.

This ability to pull together and work to find solutions, even over strong disagreements, is one of the indicators of a strong community. I’m confident that this spirit of collaboration is going to continue as we build the new Puppet™️ Language Steering Committee to guide our shared future. I’m quite excited for this steering committee because it’s got representatives from the entire community, even including Perforce itself. It’s the first time that community user needs sit at the same table as the business needs do without anyone having veto power over the other.

Being at the same table ensures that we don’t diverge too much and that we stay compatible and interoperable. These efforts, along with others like converging on acceptable compatibility names, will help ensure that the same Puppet™️ Forge works for all of us, that Vox Pupuli developer tools remain usable on Puppet Enterprise, that the PDK builds modules for whatever Puppet™️ implementation you happen to be running, and that our entire ecosystem remains collaborative and engaging regardless of where we work.

I’m kind of digging where we are. Some of it may have been born out of conflict and there were some hurt feelings all around. But being able to talk things over, share our needs, and work together to resolve them means that together we are a community and together we will build an ecosystem that’s stronger than it ever has been before.

betadots: Die Ruby-Seite von Puppet - Teil 1 - Benutzerdefinierte Fakten

2025-01-01T17:26:54+00:00

Dies ist der erste von drei Beiträgen, die die Konzepte und Best Practices zur Erweiterung von Puppet mit Hilfe von benutzerdefinierten Fakten, benutzerdefinierten Funktionen, benutzerdefinierten Typen und Providern behandeln.

Teil 1 (dieser Beitrag) erklärt benutzerdefinierte Fakten und wie ein Puppet Agent dem Puppet-Server Informationen bereitstellen kann.

Teil 2 konzentriert sich auf benutzerdefinierte Funktione und erläutert detailliert, wie Datenverarbeitungs- oder Ausführungsfunktionen entwickelt werden können.

Teil 3 deckt benutzerdefinierte Typen und Provider ab, die die Funktionalität der Puppet-DSL erweitern.

Benutzerdefinierte Fakten:

Es gibt mehrere Gründe, benutzerdefinierte Fakten zu entwickeln:

Wann immer ein VM-Namensschema verwendet wird, empfehlen wir, das Namensschema als benutzerdefinierten Fakt bereitzustellen.
Es ist manchmal erforderlich, den Zustand bestimmter Konfigurationen zu bestimmen oder ob bestimmte Software installiert ist.
Die Rechenzentrumsabteilung benötigt möglicherweise einen Überblick über die physische Hardware, Details zu den Hardwareanbietern und End-of-Life (EOL)-Supportdaten. Darüber hinaus kann die Finanzabteilung Informationen über die Anzahl der verwendeten kommerziellen Lizenzen und die Versionen der installierten Software benötigen.

Wann immer Informationen von einem Puppet Agenten benötigt werden, bietet Puppet die Möglichkeit, benutzerdefinierte Fakten zu implementieren.

Inhalt:

Faktenerstellung
Fakten in Modulen
Allgemeine API
Einschränkungen
Hilfsfunktionen
Zugriff auf andere Fakten
Rückgabewerte
Windows-Fakten
Weitere Konzepte
Codierungsstrategien und strukturierte Daten
Zusammenfassung

Faktenerstellung

Benutzerdefinierte Fakten können lokal auf einem Arbeitsplatz oder auf dem System entwickelt werden, auf dem der Fakt benötigt wird. Bevor der ungetestete Fakt in den Puppet-Code aufgenommen wird, kann der Faktladepfad auf zwei Arten angegeben und unabhängig gestestet werden.

Die folgende Verzeichnisstruktur wird verwendet:

ls ~/new_fact/
betadots_application_version.rb

Um den Fakt auszuführen, verwenden man einen der folgenden Befehle:

FACTERLIB=~/new_fact facter betadots_application_version
facter --custom-dir ~/new_fact betadots_application_version

Fakten in Modulen

Puppet stellt eine API zur Entwicklung benutzerdefinierter Fakten bereit. Um sicherzustellen, dass der Puppet-Agent sie findet, müssen benutzerdefinierte Fakten in einem bestimmten Verzeichnis innerhalb eines Moduls abgelegt werden.

Alle Dateien im lib-Verzeichnis von Modulen werden auf alle Agenten synchronisiert. Es ist nicht möglich, einzuschränken, welche Dateien synchronisiert werden.

Benutzerdefinierte Fakten sollten im Verzeichnis lib/facter eines Moduls abgelegt werden. Obwohl man einen beliebigen Dateinamen wählen kann, wird empfohlen, den Namen des Fakts als Dateinamen zu verwenden, um eine einfache Identifikation zu gewährleisten. Der Dateiname muss mit .rb enden.

Wir empfehlen außerdem, dem Namen des benutzerdefinierten Fakts einen Firmen- oder Abteilungspräfix hinzuzufügen.

Allgemeine API

Facter bietet eine API zum Erstellen neuer benutzerdefinierter Fakten.

Beispiel:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  setcode do
    # Code hier
  end
end

Der Block setcode do ... end enthält den gesamten Ruby-Code für den Fakt. Das Ergebnis des letzten Befehls oder der letzten Variablen wird als Rückgabewert des Fakts verwendet.

Einschränkungen

Da alle benutzerdefinierten Fakten an alle Puppet-Agenten verteilt werden, ist es wichtig sicherzustellen, dass betriebssystem- oder anwendungsspezifische Fakten nur dort ausgeführt werden, wo es erforderlich ist.

Dies wird durch Einschränkungen (confining) erreicht.

Einschränkungen können auf einfache oder strukturierte Fakten oder auf anderen Ruby-Code wie File.exist? angewendet werden.

Einfache Fakt-Einschränkung:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine kernel: 'Linux'

  setcode do
    # Code hier
  end
end

Einschränkung unter Verwendung eines Ruby-Blocks und Fakten:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine 'os' do |os|
    os['family'] == 'RedHat'
  end

  setcode do
    # Code hier
  end
end

Einschränkung unter Verwendung eines Ruby-Blocks und Ruby-Code:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    # Code hier
  end
end

Hilfsfunktionen

Facter bietet mehrere Hilfsmethoden, die verwendet werden können, ohne dass die Klasse facter erforderlich ist:

Hilfsfunktion	Beschreibung
`Facter::Core::Execution::execute`	Führt ein ausführbares Programm auf dem System mit einer Timeout-Option aus, um das Aufhängen von Facter-Läufen zu verhindern.
`Facter::Core::Execution::which`	Prüft, ob ein ausführbares Programm verfügbar ist. Funktioniert auf jedem unterstützten Betriebssystem und durchsucht die Standardpfade `ENV['PATH'] + ['/sbin', '/usr/sbin']`.
`Facter.value`	Bietet Zugriff auf den Wert eines anderen Fakts. Achtung! Loops vermeiden!

Weitere Informationen findet man in der API-Dokumentation.

Bitte beachten, dass Facter::Core::Execution::exec zugunsten von Facter::Core::Execution::execute veraltet ist. Dies ist wichtig, wenn man von älteren Versionen von Facter migriert.

Die timeout-Options-Hash kann auf zwei Arten festgelegt werden:

Gültig für den gesamten Fakt:

Facter.add('<name>', {timeout: 5}) do
  ...
end

Pro Ausführungsmethode:

Facter::Core::Execution::execute('<cmd>', options = {:timeout => 5})

Zum Beispiel, wenn eine Anwendung ihre Konfiguration über /opt/app/bin/app config zurückgibt, kann man ein Timeout von 5 Sekunden festlegen:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
  end
end

Das Ergebnis des letzten Befehls oder die Angabe einer Variable wird als Faktenergebnis verwendet.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    config_list = Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
    config_list
  end
end

Zugriff auf andere Fakten

Es ist möglich, dass ein benutzerdefinierter Fakt den Wert eines anderen Fakts verwendet, indem er Facter.value verwendet. Dies sollte jedoch vorsichtig geschehen, um zyklische Abhängigkeiten zwischen Fakten zu vermeiden.

In unserem Beispiel hat die Anwendung unterschiedliche Konfigurationspfade für verschiedene Betriebssysteme.

RedHat - /etc/sysconfig/application
Debian - /etc/default/application

Beispiel mit Zugriff auf einen anderen Fakt:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config')
  end
end

Das Beispiel ruft den os-Fakt ab und verwendet den family-Schlüssel, um das Betriebssystem zu vergleichen, und wendet dann eine Logik an, um zu bestimmen, welchen Anwendungskonfigurationspfad geprüft werden soll.

Rückgabewerte

Puppet erwartet, dass benutzerdefinierte Fakten einen gültigen Werttyp zurückgeben, insbesondere Strings, Ganze Zahlen oder Arrays.
Wenn der Wert eines Fakts nil oder undef ist, wird der Fakt einfach nicht definiert.

Windows-Fakten

Facter ist eine plattformübergreifende API und stellt die gleichen Funktionen auf allen Betriebssystemen bereit. Es ist jedoch wichtig zu beachten, dass Pfade und ausführbare Dateien in einer Windows-Umgebung unterschiedlich sein können.

Mit Hilfe des Facter::Core::Execution.execute werden üblicherweise Powershell Kommandos gestartet.

Das folgende Beispiel erstellt einen Windows-Fakt:

# ~/new_fact/windows_version.rb
Facter.add(:windows_version) do
  confine osfamily: 'windows'

  setcode do
    Facter::Core::Execution.execute('reg query "HKLM\Software\Microsoft\Internet Explorer" /v svcVersion', {:timeout => 5})
  end
end

Das Beispiel verwendet den reg query-Befehl, um die Version des Internet Explorers zu erhalten.

Bitte beachten, dass die Win32 API calls seit Ruby 1.9 deprecated sind.
Man kann statt dessen die Fiddle oder andere Ruby Bibliotheken nutzen.

Weitere Konzepte

Protokollierung

Zur Analyse eines Facts kann man auf die debug Methode zugreifen.

Facter.add(:betadots_application_version) do
  setcode do
    Facter.debug("Custom fact 'betadots_application_version' running")
    ...
  end
end

Es sind folgende Log Level möglich: debug, info, warn, error und fatal.

Aggregierung

Eine weitere Option ist das Zusammenstellen von Facts als Hash mit Hilfe von aggregates und chunks.
Jeder chunkbenötigt einen Namen (key) und einen Wert (value), wobei der Wert entweder ein Array oder ein Hash sein muss!

Nachdem Facter alle chunks eingelesen und validiert hat, werden diese aggregiert. Als default werden die Arrays oder Hashes gemerged.
Es besteht die Möglichkeit, eine eigene Aggregierungs Funktion zu schreiben.

Wir gehen davon aus, dass unsere Beispielanwendung es uns ermöglicht, Einstellungen auszulesen: /opt/app/bin/app config <key>.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version, :type => :aggregate) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  chunk(:config_version) do
    {:version => Facter::Core::Execution.execute('/opt/app/bin/app config version')}
  end

  chunk(:config_cache_size) do
    {:cache_size => Facter::Core::Execution.execute('/opt/app/bin/app config cache_size')}
  end

  chunk(:config_log_level) do
    {:log_level => Facter::Core::Execution.execute('/opt/app/bin/app config log_level')}
  end
end

Ergebnis:

betadots_application_version => {
  version => '1.2.4',
  cache_size => '400M',
  log_level => 'info',
}

Gewichtung

Facter ermöglicht es, den Rückgabe Wert auf unterschiedliche Arten zu ermitteln. Die einzelnen Arten bekommen eine Gewichtung, damit Facter danach entscheiden kann, welche Art (Methode) vals Rückgabewert verwendet werden soll.

Wir gehen davon aus, dass unsere Besipielanwendung entweder über systemd oder über eine andere Art gestartet wurde (PID Datei).

Hinweis: Externe Fakten haben eine Gewichtung von 1000. Man kann externe Fakten überschreiben, indem man für den Custom Fakt den gleichen Namen verwendet und eine Gewichtung über 1000 hinterlegt.

Weitere Informationen findet man in der Fact precedence Sektion der Custom facts overview Seite.

Facter.add('application_running') do
  has_weight 100

  setcode do
    Facter::Core::Execution.execute('systemctl status app')
  end
end

Facter.add('application_running') do
  has_weight 50

  setcode do
    File.exist?('/var/run/application.pid')
  end
end

Blocking und Caching

Normalerweise werden Facts bei jedem Lauf eines Puppet Agenten erneut ermittelt. Dies kann bei manchen Facts dazu führen, dass Facter mehr Zeit benötigt, bis alle Facts geladen sind. Dies gilt insbesondere, wenn Facts eine Anwendung abfragen und diese Abfrage eine hohe Last erzeugt und lange dauert oder es gibt "teure" Facts, die gar nicht benötigt werden.
Facter ermöglicht es, einen Cache zu verwenden, so dass Facts nur einmalig und danch erst nach der Lebensdauer des Cache Objektes erneut abgefragt werden und man kann Blacklisten mit unerwünschen Facts pflegen.

Die Facter Konfiguration erfolgt in /etc/puppetlabs/facter/facter.conf auf *nix Systemen oder in C:\ProgramData\PuppetLabs\facter\etc\facter.conf auf Windows.
Bitte beachten, dass die Konfiguration auf dem Puppet Agent erfolgen muss!

# /etc/puppetlabs/facter/facter.conf
facts : {
  blocklist : [ "file system", "EC2", "processors.isa" ]
  ttls : [
    { "timezone": 30 days },
    { "my-fact-group": 30 days },
  ]
}
fact-groups : {
  my-fact-group : [ "os", "ssh.version"]
}

Innerhalb des Elementes blocklist kann man entweder einen Fact, ein Subelement eines Fact Hashes oder eine Facter Gruppe angeben.
Existierende Facter Gruppen kann man aus Facter auslesen:

Facter block groups

facter --list-block-groups
EC2
  - ec2_metadata
  - ec2_userdata
file system
  - mountpoints
  - filesystems
  - partitions
hypervisors
  - hypervisors

Facter cache groups

facter --list-cache-groups
EC2
  - ec2_metadata
  - ec2_userdata
GCE
  - gce
...

Außerdem kann man eigene Facter Gruppen innerhalb der facts-group Sektion hinterlegen.

Bitte beachten, dass das Dylan Ratcliffe's facter_cache Module seit Puppet 6 nicht mehr notwendig ist.

Mehr Informationen findet man auf der Puppet Webseite im Bereich facter.conf settings.

Codierungsstrategien und strukturierte Daten

Anzahl dr Fakten redizieren
Hashes verwenden
Generischer Code und Ruby Module

Wenn ein Fakt komplexe Daten zurückgeben muss, sollten strukturierte Daten verwendet werden, um hierarchische Informationen bereitzustellen. Weitere Informationen zur Strukturierung von Daten finden Sie in der Puppet-Dokumentation zu strukturierten Daten in benutzerdefinierten Fakten.

Eine bewährte Praxis besteht darin, sich für eine Struktur zu entscheiden, wenn ein Fakt mehr als eine einzelne Zeichenfolge zurückgeben soll. Solche Daten werden in der Regel in Hashes oder Arrays organisiert, die wahlweise direkt einen Hash liefern oder über aggregates zusammengesetzt werden:

Lösung 1: Hash in Ruby:

Facter.add(:betadots_file_check) do
  setcode do
    file_list = case Facter.value(:kernel)
                when 'windows'
                  [
                    'c:/Program Data/Application/bin/app',
                    'c:/backup/state.txt',
                  ]
                when 'Linux'
                  [
                    '/opt/app/bin/app',
                    '/etc/backup/state.txt',
                  ]
                end

    result = {}

    file_list.each |name| do
      result[name] = {
        :size => File.size(name),
        :world_writable => File.world_writable?(name)
      } if File.exist?(name)
    end

    result
  end
end

Facter Ergebnis:

{
  '/opt/app/bin/app' => {
    size => 64328,
    world_writable => null,
  },
  '/etc/backup/state.txt' => {
    size => 0,
    world_writable => 438,
  }
}

Lösung 2: Hashes mit Hilfe von aggregate:

Facter.add(:betadots_file_check, :type => :aggregate) do
  file_list = case Facter.value(:kernel)
              when 'windows'
                [
                  'c:/Program Data/Application/bin/app',
                  'c:/backup/state.txt',
                ]
              when 'Linux'
                [
                  '/opt/app/bin/app',
                  '/etc/backup/state.txt',
                ]
              end

  chunk(:file_size) do
    result = {}

    file_list.each |name| do
      result[name] = { :size => File.size(name) } if File.exist?(name)
    end

    result
  end

  chunk(:file_world_writable) do
    result = {}

    file_list.each |name| do
      result[name] = { :world_writable => File.world_writable?(name) } if File.exist?(name)
    end

    result
  end
end

Facter Ergebnis:

{
  '/opt/app/bin/app' => {
    size => 64328,
    world_writable => null,
  },
  '/etc/backup/state.txt' => {
    size => 0,
    world_writable => 438,
  }
}

Ein weiteres Facter Besipiel liest das Agent Zertifikat aus und liefert die Zertifikatserweiterungen als Fact Hash:

Facter.add(:betadots_cert_extension) do
  setcode do
    require 'openssl'
    require 'puppet'
    require 'puppet/ssl/oids'

    # set variables
    extension_hash = {}
    certdir = Puppet.settings[:certdir]
    certname = Puppet.settings[:certname]
    certificate_file = "#{certdir}/#{certname}.pem"

    # get puppet ssl oids
    oids = {}
    Puppet::SSL::Oids::PUPPET_OIDS.each do |o|
      oids[o[0]] = o[1]
    end

    # read the certificate
    cert = OpenSSL::X509::Certificate.new File.read certificate_file

    # cert extensions differs if we run via agent (numeric) or via facter (names)
    cert.extensions.each do |extension|
      case extension.oid.to_s
      when %r{^1\.3\.6\.1\.4\.1\.34380\.1\.1}
        short_name = oids[extension.oid]
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      when %r{^pp_}
        short_name = extension.oid
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      end
    end

    extension_hash
  end
end

Zusammenfassung

Dieser Beitrag hat die Grundlagen zur Erstellung von benutzerdefinierten Fakten in Puppet beschrieben. Benutzerdefinierte Fakten sind ein leistungsfähiges Werkzeug, um einem Puppet-Server oder einem Management-Tool Informationen zu liefern. Facts werden in Modulen hinterlegt und sollten bei Bedarf eingeschränkt (confine) werden. Die Rückgabewerte sollten als Hash oder Array und nur in seltenen Fällen als Bool, String oder Integer ausgegeben werden.

Ein letzter Hinweis: Nur lokale Kommandos ausführen! Wenn man Remote Systeme in Kombination mit Facter nutzen möchte, müssen diese Hochverfügbar und skaliert aufgebaut sein.

Happy puppetizing,
Martin

betadots: The Ruby side of Puppet - Part 1 - Custom Facts

2025-01-01T17:26:04+00:00

This is the first in a series of three posts covering the concepts and best practices for extending Puppet using custom facts, custom functions and custom types and providers.

Part 1 (this post) explains custom facts and how a node can provide information to the Puppet Server.

Part 2 focuses on custom functions, detailing how to develop data processing or execution functions.

Part 3 covers custom types and providers, extending Puppet DSL functionality.

Custom Facts:

There are several reasons to develop custom facts:

Whenever a VM naming model is used, we recommend providing the naming schema as a custom fact.
It is sometimes necessary to determine the state of certain configurations or whether specific software is installed.
The datacenter department may need an overview of physical hardware, hardware vendor details, and end-of-life (EOL) support dates. Additionally, the finance department may require information on the number of commercial licenses in use and the versions of installed software.

Whenever information from a node is needed, Puppet offers the option to deploy custom facts.

Content:

Fact Development
Facts in Modules
General API
Confining
Helpers
Accessing Other Facts
Return Values
Windows Facts
Additional Concepts
Coding Strategies and Structured Data
Summary

Fact Development

You can develop custom facts locally on a workstation or on the system where the fact is needed. Before adding the untested fact to your Puppet code, you can specify the fact load path in two ways.

The following directory structure is used:

ls ~/new_fact/
betadots_application_version.rb

To execute the fact, use one of the following commands:

FACTERLIB=~/new_fact facter betadots_application_version
facter --custom-dir ~/new_fact betadots_application_version

Facts in Modules

Puppet provides an API to develop custom facts. To ensure the Puppet agent finds them, custom facts must be placed in a specific directory within a module.

All files in the lib directory of modules are synchronized to all agents. It’s not possible to restrict which files are synchronized.

Custom facts should be placed in the lib/facter directory of a module. Although you can choose any filename, it's recommended to use the fact name as the filename for easy identification. The filename must end with .rb.

We also recommend prefixing the custom fact name with a company or department identifier.

General API

Facter provides an API to create new custom facts.

Example:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  setcode do
    # Code here
  end
end

The setcode do ... end block contains all the Ruby code for the fact. The result of the last command or variable is used as the fact’s return value.

Confining

Since all custom facts are distributed to all Puppet agents, it’s essential to ensure that OS- or application-specific facts are only executed where necessary.

This is achieved by confining facts.

Confinement can be applied to simple or structured facts or any other Ruby code, such as File.exist?.

Simple fact confinement:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine kernel: 'Linux'

  setcode do
    # Code here
  end
end

Confinement using Ruby block and facts:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine 'os' do |os|
    os['family'] == 'RedHat'
  end

  setcode do
    # Code here
  end
end

Confinement using Ruby block and Ruby code:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    # Code here
  end
end

Helpers

Facter provides several helper methods that can be used without requiring the facter class:

Helper	Description
`Facter::Core::Execution::execute`	Runs an executable on the system with a timeout option to prevent hanging facter runs.
`Facter::Core::Execution::which`	Checks if an executable is available. Works on any supported OS, searching the default paths `ENV['PATH'] + ['/sbin', '/usr/sbin']`
`Facter.value`	Provides access to the value of any other fact. Be careful to avoid loops!

More information is available in the API documentation.

Please note that Facter::Core::Execution::exec has been deprecated in favor of Facter::Core::Execution::execute. This is important when migrating from older versions of Facter.

The timeout options hash can be set in two ways:

Valid for the whole fact:

Facter.add('<name>', {timeout: 5}) do
  ...
end

Per execute method

Facter::Core::Execution::execute('<cmd>', options = {:timeout => 5})

For example, if an application returns its configuration via /opt/app/bin/app config, you can set a timeout of 5 seconds:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
  end
end

The result of the last command or a variable with content is used as fact result.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    config_list = Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
    config_list
  end
end

Accessing Other Facts

It’s possible for a custom fact to leverage the value of another fact by using Facter.value.However, this should be done cautiously to avoid introducing cyclic dependencies between facts.

In our example the application has to different config paths for different OS.

RedHat - /etc/sysconfig/application
Debian - /etc/default/application

Example with access to another fact:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config')
  end
end

The example retreives the os fact and uses the family key to compare the OS, then applies logic to determine which application config path to check for.

Return values

Puppet expects custom facts to return a valid value type, specifically strings, integers, booleans, or structured data (such as arrays and hashes). If the fact cannot determine a valid value, it should return nil or undef to indicate that no fact value is available.

Windows Facts

Windows-based systems support custom facts just like Linux or other Unix-like systems.
The main difference is that many Windows-specific tasks (such as checking installed software or reading from the registry)may require platform-specific Ruby code.

Within the Facter::Core::Execution.execute usually powershell commands is used.

Here's an example of a Windows custom fact that retrieves the version of Internet Explorer:

# ~/new_fact/betadots_internet_explorer_version.rb
Facter.add(:betadots_internet_explorer_version) do
  confine kernel: 'windows'

  setcode do
    Facter::Core::Execution.execute('reg query "HKLM\Software\Microsoft\Internet Explorer" /v svcVersion')
  end
end

In this example, we use the reg querycommand to check the Internet Explorer version in the Windows registry.

Please note that the Win32 API calls are deprecated since ruby 1.9.
Please consider using Fiddle or other Ruby-based libraries for interacting with the system.

Additional Concepts

Logging

It's often useful to include logging within custom facts to help with troubleshooting and development. You can log messages using Puppet's built-in logging mechanism:

Facter.add(:betadots_application_version) do
  setcode do
    Facter.debug("Custom fact 'betadots_application_version' running")
    ...
  end
end

Logging levels include debug, info, warn, error, and fatal.

Aggregates

Another option to create structured facts is the usage of chunks.
Each chunk must have a name and returns a data structure which must be of type array or hash.
After reading all chunks, the chunks will be aggregated. The default is to merge arrays or hashes.

It is possible to write a different aggregate definition.

We assume that our app can directly read config keys: /opt/app/bin/app config <key> returns the value.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version, :type => :aggregate) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  chunk(:config_version) do
    {:version => Facter::Core::Execution.execute('/opt/app/bin/app config version')}
  end

  chunk(:config_cache_size) do
    {:cache_size => Facter::Core::Execution.execute('/opt/app/bin/app config cache_size')}
  end

  chunk(:config_log_level) do
    {:log_level => Facter::Core::Execution.execute('/opt/app/bin/app config log_level')}
  end
end

Result:

betadots_application_version => {
  version => '1.2.4',
  cache_size => '400M',
  log_level => 'info',
}

Weight

Facter offers the option to select a value based on priority of a fact.
Let's assume that an application is either started form systemd or in any other way (so it has a pid file).

Note: external facts have a built in weight value of 1000. Overriding external facts is possible by creating a fact with the same name and specifying a weight value over 1000.

More details can be found in the Fact precedence section of the Custom facts overview page.

Facter.add('application_running') do
  has_weight 100

  setcode do
    Facter::Core::Execution.execute('systemctl status app')
  end
end

Facter.add('application_running') do
  has_weight 50

  setcode do
    File.exist?('/var/run/application.pid')
  end
end

Blocking and Caching

By default, facts are recalculated each time they are queried. In certain scenarios, this might be inefficient, especially for computationally expensive facts or facts that rarely change. Facter offers a caching mechanism that allows you to persist fact values between Puppet runs.

Besides this some facts might be of no interest, but take long time to read.
In this case Facters blocklist can be used.

Configuration takes place in /etc/puppetlabs/facter/facter.conf on *nix systems or C:\ProgramData\PuppetLabs\facter\etc\facter.conf on Windows.
Please note that this configuratoin must be done on the Puppet agents!

# /etc/puppetlabs/facter/facter.conf
facts : {
  blocklist : [ "file system", "EC2", "processors.isa" ]
  ttls : [
    { "timezone": 30 days },
    { "my-fact-group": 30 days },
  ]
}
fact-groups : {
  my-fact-group : [ "os", "ssh.version"]
}

As you can see: within the blocklist we can mention the facts itself, a sub fact, or a fact group.
Existing facter groups can be read on the command line:

Facter block groups

facter --list-block-groups
EC2
  - ec2_metadata
  - ec2_userdata
file system
  - mountpoints
  - filesystems
  - partitions
hypervisors
  - hypervisors

Facter cache groups

facter --list-cache-groups
EC2
  - ec2_metadata
  - ec2_userdata
GCE
  - gce
...

Besides this we can add our own fact groups using the facts-group setting.

Please note that in the past people were using Dylan Ratcliffe's facter_cache Module. This is no longer needed as of Puppet 6, when the facter.conf settings were introduced.

The Puppet website provides more information on facter.conf settings.

Coding Strategies and Structured Data

Limit the total number of facts
Use data hashes
Generic code and Ruby modules

In more advanced cases, facts can return structured data, such as arrays or hashes. Structured facts allow for more complex data to be passed back to the Puppet server.

It is absolutely required to limit the total number of facts for a node. Main reason is the performance of PuppetDB.

Hashes or arrays can be built directly in Ruby code or via Facter aggregates.

e.g. We need the state of several files.

Solution 1: Hash in Ruby:

Facter.add(:betadots_file_check) do
  setcode do
    file_list = case Facter.value(:kernel)
                when 'windows'
                  [
                    'c:/Program Data/Application/bin/app',
                    'c:/backup/state.txt',
                  ]
                when 'Linux'
                  [
                    '/opt/app/bin/app',
                    '/etc/backup/state.txt',
                  ]
                end

    result = {}

    file_list.each |name| do
      result[name] = { 'size' => File.size(name), 'world_writable' => File.world_writable?(name) } if File.exist?(name)
    end

    result
  end
end

Will return:

{
  /opt/app/bin/app => {
    size => 64328,
    world_writable => null,
  },
  /etc/backup/state.txt => {
    size => 0,
    world_writable => 438,
  }
}

Solution 2: Hashes using aggregate:

Facter.add(:betadots_file_check, :type => :aggregate) do
  file_list = case Facter.value(:kernel)
              when 'windows'
                [
                  'c:/Program Data/Application/bin/app',
                  'c:/backup/state.txt',
                ]
              when 'Linux'
                [
                  '/opt/app/bin/app',
                  '/etc/backup/state.txt',
                ]
              end

  chunk(:file_size) do
    result = {}

    file_list.each |name| do
      result[name] = { :size => File.size(name) } if File.exist?(name)
    end

    result
  end

  chunk(:file_world_writable) do
    result = {}

    file_list.each |name| do
      result[name] = { :world_writable => File.world_writable?(name) } if File.exist?(name)
    end

    result
  end
end

Will return:

{
  /opt/app/bin/app => {
    size => 64328,
    world_writable => null,
  },
  /etc/backup/state.txt => {
    size => 0,
    world_writable => 438,
  }
}

Another example reads the Puppet agent certificate and provides the certificate extensions as a Facter hash:

Facter.add(:betadots_cert_extension) do
  setcode do
    require 'openssl'
    require 'puppet'
    require 'puppet/ssl/oids'

    # set variables
    extension_hash = {}
    certdir = Puppet.settings[:certdir]
    certname = Puppet.settings[:certname]
    certificate_file = "#{certdir}/#{certname}.pem"

    # get puppet ssl oids
    oids = {}
    Puppet::SSL::Oids::PUPPET_OIDS.each do |o|
      oids[o[0]] = o[1]
    end

    # read the certificate
    cert = OpenSSL::X509::Certificate.new File.read certificate_file

    # cert extensions differs if we run via agent (numeric) or via facter (names)
    cert.extensions.each do |extension|
      case extension.oid.to_s
      when %r{^1\.3\.6\.1\.4\.1\.34380\.1\.1}
        short_name = oids[extension.oid]
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      when %r{^pp_}
        short_name = extension.oid
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      end
    end

    extension_hash
  end
end

Summary

Custom facts are a powerful way to extend Puppet’s built-in facts with your organization’s specific information. Whether you’re retrieving application versions, monitoring system configuration, or providing custom details for a specific environment, custom facts help ensure that Puppet has the right data to apply the correct configuration.

In this post, we’ve covered the basics of custom fact development, including:

How to create custom facts in a local environment or as part of a module.
The API for creating and confining facts.
Accessing other facts and helper methods to enhance fact functionality.
Returning structured data for complex requirements.

One more advise: Run local commands only! If connections to remote systems are required you must ensure scalability and high availability.

Happy puppetizing,
Martin

Overlook InfraTech: Letting go of control

2024-12-19T00:00:00+00:00

Events of the last week have me thinking about something that’s true in a lot of contexts. A lot of people like to have control of a relationship; control of a situation; control of the message; or in this case, control of a community. I get it, realizing that you don’t have control can be unsettling. But in reality, not having control is the natural order of things and you’ll have a much better time of it once you realize that.

Let’s dive in. A controlling relationship is an easy start to the conversation, since most of us have either been in one or have witnessed one. We may have even been the offender; it’s very easy to accidentally fall into this role under certain circumstances. A relationship controller is quite often someone who’s terrified that the other(s) will leave or hurt them and so they tighten their grip more and more and more trying to prevent this. We all know the usual outcome; that grip becomes so tight that it forces an escape. The inability to let go of control leads to exactly the outcome that was most feared to begin with.

The same thing happens when one tries too hard to control a message. I don’t have to explain it to you. Google the Barbra Streisand effect for that.

As a seasoned community leader, I’m most interested in the context of community engagement. And in particular, technical open source communities centered around a product ecosystem — although this fundamentally applies to other community types as well. See, a lot of Product communities have a power imbalance at their very core. If the company can gatekeep merges, and packaging, and releases — then they have an outsized say in where the product goes. If they don’t want to invest in supporting Apple Silicon, it ain’t gonna happen even if community members volunteer to do the work.

Many companies mistake this imbalance as having control, even to the point of treating community members as if they were unpaid employees. I’ll give you a personal example of that. For most of our history, Puppet had an excellent community relationship because we remembered that we built ourselves off of their passion. As money got tight, some of the niceties got strained and some of the technical engagement started lagging. But we always treated them with respect and supported them as they grew and evolved the community. When they created Vox Pupuli, we celebrated and we never stopped celebrating it. Just before the pandemic did us in, a large part of what turned out to be Puppet’s final PuppetConf/Puppetize keynote centered Vox Pupuli as the independent community group that our success hinged on.

This changed rapidly after the Perforce acquisition. The community started being treated as liabilities and resources. At one point, the VP of Product at the time insisted on holding a community Zoom call the very next day or two to get feedback on some ideas he had. He said, “of course they’ll do it. They want to have input on what we sell!”

I’m sure they do. I collect feedback like that all the time. But community members who are freely giving their time for something they enjoy in a space where they feel welcomed and valued do not want to feel obligated to do so! That’s called a job and you collect a paycheck for it.

He didn’t understand when I tried to explain that he didn’t control them. They weren’t employees and he couldn’t expect them to prioritize his demands. He almost fired me when I refused.

Most companies in this position don’t understand how fragile and tenuous this power imbalance is. It only lasts so long as you own the roadmap and people still care enough to put up with your bullshit. As soon as the frustrations community feels about your behavior outweighs the work it is to just take on the roadmap themselves, then this power imbalance evaporates in an instant. And it’s generally irreversible because you’ve already tipped your hand as to how you’ll behave when you feel like you have the advantage. Your voice is no longer welcomed.

⚠️ A Product company never has control over their open source user community. They only have an illusion of it and by the time they realize this, it’s generally too late to recover.

To build a healthier community, they should be actively relinquishing control and cultivating influence instead. They will get very similar results, they just have to be more patient, honest, open, transparent,… You know, all the things that make a relationship work.

When you realize that you have zero control but as much influence as you are willing to develop, then you start behaving in ways that build a strong, loyal, and supportive community that will help see you through your missteps and always make sure that you have a seat at the table when decisions are being made.

You don’t lie to your community. You don’t always have to tell every single detail, and you can always couch things in softer terms or provide context. But you never lie.
You don’t make empty promises or excuses. You’re clear about what you can or can’t do.
You actively listen to their feedback and you care about it. In specific, you hear the pain that they have more than the words they use or the solutions they offer. You never “empty listen” and then throw it in the trash.
You take action on that feedback if warranted, but you’re always clear what should be expected.
You get really good at hearing what community wants and getting there before they have to ask for it. This gives you more influence in the growth patterns, just like cultivating a garden.
You participate in the day-to-day of the community and get to know what people value.
You are a role model, not a dictator.
You remind rather than enforce. In all the years I participated in and then managed the Puppet community, I rarely had to moderate users more than reminding them of the code of conduct.
You meet community where they are and where they want to be. You help them build a better table rather than insisting that they sit at yours.

In short, listen more than you proclaim, realize the value of your community members and never take them for granted, and help them build the space that they want — where they can feel welcome, accepted, and valued. Connect with people in a human way, rather than as a corporation using carefully crafted corporate doublespeak.

And let go of this anxiety-driven idea of control. You never had it anyways, and influence is far more valuable. Influence helps you build a lasting community and solid open source project in which you have a respected seat at the table.

Overlook InfraTech: Contributions of ‘Ghost Engineers’

2024-12-05T00:00:00+00:00

Last week was Thanksgiving for the USA and while I don’t personally celebrate the holiday or the violent colonization it represents, it does encourage me to reflect back on some of the things I’m most thankful for. Once again, you won’t be surprised to hear that it’s people. Coworkers, friends, community members, industry professionals that I learn from and am inspired by every day. People that I’m lucky enough to collaborate with, which probably includes many of you reading this. People in my communities are what I’m thankful for.

Collaboration is one of the things that I’ve been thinking about a lot over the last few months while getting this fun little venture off the ground. I think that most companies don’t prioritize collaboration as much as they should and I want to avoid that trap. And I’m definitely thinking about macro business decisions like why did Puppet, inc. invest so much time and resources into building an orchestration tool from scratch instead of integrating with one of the many existing options, even Ansible itself?

🔆 But even more, I’m thinking about micro level choices, because I think those are how you define yourself as a people and then that culture bubbles up into how your business runs, from top to bottom.

Many of you know that I’ve been involved in the roller derby world for a very long time. My partner has been a founding member of several teams and we are in the home of one of the premiere leagues in the world. One of the things that continues to draw me to the sport is their sense of community. The big dollars haven’t invaded yet and so they’re just there to have fun. They’ll beat each other up on the track and then share an afterparty. When a bout concludes, the audience will rush to circle the track because every single player will high-five every single fan who wants it. Then if you pay attention, there’s a ceremony in which the teams give gifts to each other and award several MVP awards to the other team. It’s worth pointing out that these awards don’t go to the player scoring the most points or other easily trackable stats. They almost always go to the players who are the best at holding their team together into a formidable force as observed by the people playing against them.

Recently, someone at Stanford who apparently knows very little about teamwork published a study that claimed to reveal that almost 10% of all software engineers were lazy and ineffective and didn’t contribute anything to the bottom line. Their rationale for this proclamation was they contributed few or no lines of code and the author deemed them “ghost engineers.” 🤨

A lot of people more qualified than I explained how LOC is maybe the least useful metric possible so I won’t go there. Instead, I want to look at it more holistically. This is a symptom of the bleed ‘em dry stack-ranking mentality that wants an easily measured metric so that toxic companies can reward those on top and punish those on the bottom in the misguided idea that it will improve performance and help the company hit revenue targets. Of course, trivially measured and manipulated metrics like this have nothing to do with the success of your company. If I have a choice between doing something to benefit the business or something that will get me a raise, what do you think I’m going to pick?

The other day, Mekka Okereke posted a sports metric that I’d never heard of and it’s super intriguing. We’ve all heard of the stats like points scored, number of rebounds or blocks or assists. But he talks about a new metric that makes me think about those roller derby MVP awards. It’s called the plus-minus and seems to be a simple comparison of how well your team does when you’re on the court vs. how it does when you’re not playing. 🤯

Huh. I can see parallels in tech…. He goes on to say that

Post by @mekkaokereke@hachyderm.io

View on Mastodon

This resonates so much with me. I’ve always worked towards being this person, and I’ve always really valued others with this mindset. It’s a shame that it’s so rare, but again, tech companies rarely incentivize it.

But that brings me to the actual point of this ramble. The more we micro-optimize and track things like lines of code, or commits, or tickets closed, or time on the clock, or even story points, the more we lose track of the bigger shared company success goal. And the more we incentivize people to game the system. It’s easy to write more lines of code or chop up work into a million tickets and commits. It’s much harder to enable actual value.

I don’t know the answer. Not yet. And maybe I never will; this is a challenge people have been grappling with for ages and in a backwards sort of way, it’s why we have the terrible metrics we do today. But I do know that the answer is based on building a culture of shared team success rather than on myopically focusing on individual metrics. Because these “ghost engineers” are the biggest contributors to our actual success and it’s about damn time they get some MVP awards of their own. And recognition and compensation to match.

Overlook InfraTech: Future proofing open source

2024-11-23T00:00:00+00:00

We had a slight disagreement with our attorney while drafting our articles of incorporation and related documents. He’d used some very aggressive language around the ownership of intellectual property and whatnot. It was pretty standard lingo but it just rubbed us both the wrong way and we asked him to tone it down significantly. This prompted a long discussion about our open source business model. We talked about the ownership of open source contributions, both to our repositories and to those owned by Vox Pupuli or other community members. We talked about contributor license agreements (CLAs) and the fact that our business model involves more upstream contributions than internal development. We explained that any future employees would almost certainly be established open source contributors and that we didn’t believe it was ethical to try to claim ownership of their personal OSS work whether it was done on “work time” or not.

At one point he asked, “if you don’t want to own anything, then how will you build value in your company?”

Ah, now that’s the problem, isn’t it? See, we don’t really ascribe to the capitalistic idea that valuation is the only thing that matters. We’re more interested in making the world a better place, in building value across the ecosystem, and in making a decent living along the way. We’re not aiming to be a billion dollar acquisition target. That’s why our company is registered as a public benefit corporation so that our fiduciary duty is not exclusively measured in dollars and shareholder return.

That said, those are big words for current me. In a potential wildly successful timeline, what’s future me going to do when staring down an investment offer with a lot of zeros attached? We need to protect our ideals against not only unethical venture capital & private equity vultures, but also against our own temptations.

Recently Jay Graber claimed that Bluesky was billionaire proof and I genuinely hope it’s true and wish them well. But all across the Internet, experts had lots to say about it. In particular, Cory Doctorow talked about Ulysses pacts which is recognizing that humans are fallible and protecting yourself against temptation. Things like removing booze from your house to help you get through a dry January or setting screen time limits so you go to bed instead of doomscrolling all night.

Open source licenses and eschewing greedy CLAs are Ulysses pacts. They mean that when we are tempted, we literally cannot claw our projects back from the community commons. Even more so, by staying OSS only and avoiding or limiting open core products, we reduce our valuation potential to make that billion dollar investment or buyout offer less likely in the first place.

It comes with challenges, of course. It means that we have to work harder to build a sustainable business model instead of taking the easy way out. It means that hiring might be harder because nobody’s going to become a tech bro billionaire on our stock options.

But I think it’s doable. And I don’t quite have the hubris to say that we’re “billionaire proof” but I think we’re building the right protections to ensure that our community is and always will be open. And I do hope you’ll come along with us.

For what it’s worth, the attorney did end up rewriting things to focus on open source. Small wins, but they’re important and they add up.

Overlook InfraTech: Alpha community package availability

2024-11-13T00:00:00+00:00

Well, I’ve got some great news for you. Thanks to hard work by a few volunteers, we have alpha community-built packages up and the starts of a download mirror list, which you can find on the new “downloads” page. But let’s talk a bit before you go racing off to install and make sure you know what you’re dipping into.

This project is kind of a preliminary state. It’s just a simple rebuild of the packages, with Perforce’s blessing. There have been no alterations other than build instructions, and no name or branding changes. The repositories are currently just simple mirrors in the Overlook InfraTech namespace. There will almost certainly be some organizational changes as we get going.

You should be able to simply install the package and end up with exactly the same commands as before. However, these should be considered kick-the-tires builds, not something to run in production just yet. We don’t yet have a fully robust test pipeline operational, although with community help we’ll get there soon.

🥳 You can help out by trying these out on non-critical infrastructure. Let us know of any installation failures or anything weird that you run into. And if you have anything else to contribute or would like to be involved in other ways, just drop us a message and let us know what’s on your mind.

In other news, I realized something funny the other day. I’ve been writing in this blog as if y’all knew who I was. And some of you probably connected the dots because of how I shared the posts. But just for clarity’s sake, I wrote an “about us” page with my name and face. If you go check it out, you’ll notice another familiar face.

I’d like to announce that Nick Burgan has joined this little endeavor! He left Perforce a month or two ago, intending to take a nice peaceful break from the corporate world. But the community packaging effort piqued his interest again. He contributed the build system, so he’s to thank for the quick packaging turnaround, and decided that he’d like to be part of the project long term. All the better for us!

Overlook InfraTech: It was only a matter of time

2024-11-08T00:00:00+00:00

Well, it looks like they did it. Perforce has all but closed the Puppet source. To be fair, they didn’t actually change the license itself, but they’ve gone as far as they could and still remain compliant. They’re forking projects internally where all their development will happen and pinkie-swear promise that ….. eventually ….. that work will make it to the public repositories.

But we’ve seen how they keep promises. Heck, they never did get around to paying out the survey compensation that y’all donated to Vox Pupuli a few years ago.

And as far as not changing the license goes, I have a little backstory to share there too. See, when Perforce leadership first started flirting with the idea of changing the license, I talked them out of it by explaining how the Contributor License Agreement had been managed.

First, let’s explain what a CLA is, if you haven’t run into one before. We’ll start with just a commit. When you contribute to an OSS project with a given license, your contribution is implicitly and hand-wavily licensed similarly. But here’s the sticky point–you still own the code you contributed. So that 2 lines of code that I contributed to the Linux kernel a thousand years ago is still mine and if they ever decide to do something wonky like ahem, close the source, they would need to get my permission or reimplement the patch.

The CLA is designed to get around that. They’re all a little different and generally they don’t take your ownership away, but they do grant perpetual license to use and to re-license any code you contribute. If you’ve contributed to Puppet and you had to go sign the thing to make the CLA Bot quit yelling at you, that’s what you signed away.

But here’s the problem. It was inconsistently maintained over the history of the project. It didn’t even exist for the first years of Puppet and then later on it would regularly crash or corrupt itself and months would go by without CLA enforcement. If Perforce actually tried to change the license, it would require a long and costly audit and then months or years of tracking down long-gone contributors and somehow convincing them to agree to the license change.

I honestly think that’s the real reason they didn’t change the license. Apache 2 allows them to close the source away and only requires that they tell you what they changed. A changelog might be enough to stay technically compliant. This lets them pretend to be open source without actually participating.

But we know what this sequestered source will mean. In short, from whenever they implement this on, you’ll never again have assurance that the source code you’re looking at in the repo is the same as what’s running on your servers. You won’t know if they’re running intrusive trackers or exfiltrating private information about your infrastructure. And you certainly won’t get access to any new features. The Debian packages that lavamind builds and the FreeBSD packages that smortex maintains will never again be anything remotely resembling current.

Which I imagine is how they want it.

But that honestly only touches the surface of what’s wrong with this approach. What happens when community tooling, like Vox’s test pipelines for example, break unexpectedly because they don’t know about internal EULA-only changes that haven’t made it public yet? You might think that’s hyperbole, but it’s something that regularly happened even with the source being open! Many of you remember the debacle of Puppet 6 demoting core resource types to installable modules. With sequestered source, this is only going to get more common and worse. Even paying customers are going to feel the pain. The community is really good at catching all the small things before they snowball into big problems and before they ship to customers. With this OSP-as-an-afterthought approach, that last bit of QA is going to land firmly in paying customers’ laps.

In any case, last week I told y’all that I was fiddling with a build pipeline to get packages built from the repositories I’d mirrored. Several people reached out to help and it looks like we can have the build pipeline cranking and geographically distributed hosting for community-built packages next week.

I’ve still got a lot of things to work out, but if you’re interested in helping just give me a shout and let’s get the ball rolling. And you’re always welcome to show up to the Vox Pupuli sync call next Tuesday at an ungodly time of the morning for me using the current Zoom URL. We have a lot of details to work out; whether we simply build packages or deviate from the published source and become a community uptream fork; how we intend to stay ahead of CVEs and get timely security patches out; what sort of community governance structure we’d like; and so on. But these are the early days. We’ve got some time to work out the kinks.

Cheers and stay well. This week has been a rollercoaster for a lot of us.

Overlook InfraTech: A Flock of Flutter devs

2024-10-31T00:00:00+00:00

Flutter is a neat little UI toolkit allowing you to build applications targeting most major platforms. It started out mobile only, iOS and Android. Today it supports web and desktop too. I poked at it back in the day when I was considering some fun mobile app dev side projects. It made a pretty big splash initially, with a lot of community interest. Was it to be the fabled universal toolkit?

But over the years, I heard less and less about it and at some point got the impression that the project was defunct. How sad. So when Hazel Weakly boosted a blog post about a Flutter fork the other day, it piqued my interest. So much of the story sounded oddly familiar. It’s an “open source” project supported by a corporate entity that claims to be an OSS cheerleader but doesn’t actually show up. The symptoms are really easy to see. Things like:

An understaffed core dev team
Pull requests and issues that get ignored for years
Weird infrastructure that makes it hard for external contributors to participate
Lack of interest in community features
Needlessly contentious reviews and community friction
Devs who don’t actually use the product
Few or no commits to upstream projects

Oh, wait. Are we still talking about Flutter?

Anyways, it made me think that maybe it would benefit the Puppet community to have their own fork that could advance at the pace that we need it to. This is a model that’s worked out pretty well for Fedora and other upstream community forks. Not everything makes it back downstream, but most of the things that really matter do. And it achieves this goal without fragmenting the community. This sounds like what Flock wants to do with the Flutter community and I’m really excited for them.

I don’t know that I am ready to take on such a large project, but the idea is intriguing. It makes me wonder how many other people are having similar thoughts.

Since I’ve already got the public repos mirrored and syncing any updates, I might as well try to get them building packages too. This seems like a fun little baby step without actually committing to anything. I’m mostly just dipping my toes in the water and feeling things out. But if anyone is excited by the idea and wants to help out, just let me know.

And Happy Halloween for anyone celebrating. It’s my favorite holiday of the year.

Overlook InfraTech: Community stewardship matters

2024-10-21T00:00:00+00:00

Some years ago I got caught up in a very annoying conflict with James over r/puppet. I don’t remember if he’d created the space or not, but he’d been the sole moderator for years. He’d been doing an excellent job, but recently he’d been drifting away from the Puppet community and spending more of his time on his mgmt project. (Which is totally cool and you should check it out sometime.)

Some moderation tasks had begun slipping by. Spam posts took a while to get removed. The look and feel of the forum hadn’t kept up to use the newer theming capabilities. You know, minor things that felt like increasingly irritating papercuts over time. I thought I’d take a little work off his hands and asked for mod status to which he immediately replied that no company should ever have full administrative control over its own community and denied my request. What a jerk! 🤣

We eventually came to a compromise by which he gave me limited mod status as long as I brought along two committed community members to take full admin roles and have veto power over me. And then some time later as we built trust, I also got full admin and didn’t have to ask the other mods for help updating the forum theming. For some eight years or so, this arrangement has worked out very well.

But I’m here now to tell you that James was right all along.

Community Stewardship 👍

Many people are passingly familiar with the idea of checks and balances in which branches of the government can prevent excesses in the other branches. The push & pull in opposition to each other creates a stability in the whole system similarly to how cantilever truss bridges support weight. In contrast, governments that concentrate all the power in a single branch are generally inherently unstable and usually require totalitarian methods to prevent unrest.

The parallel to community is obvious, although the consequences are rarely so brutal. When a person is ready to leave a community, they just do it. No police state stops them at the border. And then the community collapses. Sometimes quickly and sometimes in excruciating slow motion. It’s almost inevitable when a community doesn’t have power over itself.

We’ve seen examples of this for ages in the tech world. Right now we’re watching a particularly drama-filled MiniMusk Mullenweg Meltdown ravage the Wordpress community because he has nothing to keep him in check, but that’s far from the only example. I was involved (very peripherally) with Andover way back in the day (get offa my lawn you kids! ) when VA swooped in and booted out all the original community volunteer SourceForge admins. Without the corrective pushback of community volunteers, the place very quickly devolved into the first case of enshittification that I personally witnessed. SourceForge still exists, but it’s the barest shadow of itself. I can’t remember the last time I even looked at it before five minutes ago.

And closer to the Puppet ecosystem, just look at the roller coaster that SaltStack has been on since the VMware/Broadcom acquisition. First in January they posted a “no open source without direct revenues” manifesto. Then in May, Tom Hatch, creator of Salt, stepped down. And finally in September, the community manager, Jimmy Chunga, left the building.

Given that timeline, observe the difference in another VMware owned community forum between May and September. Today, there’s a single orphaned post from 25 days ago with zero replies. When a corporate entity owns a community, there’s very little to keep them prioritizing community needs, and people respond to that.

Perhaps a better way of stating James’ objection is that

📌 A community always has ultimate power over itself, even if sometimes that power is exercised by simply leaving.

“Corporate Stewardship” 🤢

This is why I don’t truly believe there’s such a thing as “corporate stewardship” with implied ownership. The very definition of the word is (emphasis mine):

stewardship, n. the careful and responsible management of something entrusted to one’s care

One cannot be a steward of something you own and have iron control over! Being a community steward means responsibly managing the community on behalf of the community instead of to “control the message” or mine it for leads or other kinds of monetization. And when a corporate entity has unfettered power over a community, without the checks and balances pushback to keep it in line, it’s an unstable system that inevitably falls into enshittification. Corporate entities simply are not capable of telling themselves no, especially when they believe that there’s money on the table.

So why did it work for `r/puppet` and other aspects of the Puppet Community for so long?

In reality it didn’t. Puppet’s community never fell under corporate stewardship. It only appeared that way because some of the folk in admin/owner roles were employed by Puppet. But the key is that we were all community members first and foremost. We participated in community, we helped people in the Slack and asked for help ourselves, we built Puppet modules and other tools to satisfy our needs, we helped connect people with needs to people with solutions. And then we also did our jobs, just like everyone else in the community.

Vox Pupuli leaders and other community members always set the ground rules for the community and the Puppet Community Team just implemented them. That’s not to say that the company didn’t have any say–we certainly did help steer the ship. But Puppet’s influence was gentle and always followed the needs of the community itself.

Our problem was that we never wrote down and codified these practices. For example, we had a very firm rule that any Puppet employee with a leadership role or other decision making influence would not run for the Vox Pupuli management committee to avoid a conflict-of-interest and having undue control over community decisions. That’s why you’ve never seen myself or David Sandilands on the PMC. But that was an informal rule that we held ourselves to, it wasn’t part of the Vox bylaws.

In short

The Puppet Community has always been under community stewardship, even though it wasn’t always obvious as such. And that’s the only way that a healthy vibrant community can prosper. Unfortunately, we were sloppy and didn’t protect against future iterations of the company that weren’t as responsible.

Now as I’ve become exclusively a community member, I’m becoming more and more convinced that we need to firmly re-establish community leadership of the Puppet ecosystem. I’m still not quite sure what that means, but one thing’s for sure: we gotta write it down this time!

Overlook InfraTech: Everything is on fire….

2024-10-16T00:00:00+00:00

I’ve been pretty quiet since leaving Puppet in July. And circumstances this week aren’t exactly how I wanted to pop my head up again and tell you what I was up to. I’ve done some private consulting with a few of y’all over the years, and had planned to expand on that. Check out the Services page for more information if you’re interested. But that’s not what I want to talk about today.

I want to talk a bit about Perforce’s actions and what I think that means for the community & ecosystem. They seized control of the Puppet Community Slack, booted all the admins from the moderator team, and banned me from the space. For context of how momentous that was, up until this week the Puppet Community has only ever banned one single person ever. To me, that implies that they’re gearing up for something that they want super strict control over messaging for. We’ve all seen this game before.

I know from personal experience with the company that they do not value the community for itself. In 2023, they flew me to Boston to explain the community to execs & VPs. Halfway through my presentation, they cut me off to demand to know how we monetize and then they pivoted into speculation on how we would monetize. There was zero interest in the idea that a healthy community supported a strong ecosystem from which the entire value of the company and product was derived. None. If the community didn’t literally hand them dollars, then they didn’t want to invest in supporting it.

But that’s to be expected, I suppose. They are a private equity company after all.

Now here’s something that many people don’t realize: the Apache 2 license does not require that the source be kept open. In fact, simply publishing a changelog might be enough to stay compliant.

I might be paranoid, but it would not surprise me one bit if they were about to close the source and make it really hard to use Open Source Puppet. That would tie right into what I know about their values. Why should they invest in supporting users who don’t hand them dollars?

I’m obviously not super happy with that idea, so I’ve spent the last couple days busily mirroring public OSS repositories from puppetlabs. I don’t quite know what happens next, but if worse comes to worst, then I’ll at least be able to republish packages.

Laurent Domb Blog: Game Day Every Day: How AWS Prepares for the Unexpected

2024-09-08T22:50:04+00:00

I had the pleasure to speak to John Gaffney from PYMNTS.com about resiliences on AWS, and how we help our customers to build a resilience posture that maps business requirements.

Full interview here: https://www.pymnts.com/news/security-and-risk/2024/chaos-engineering-aws-chief-technologist-on-preparing-for-the-unexpected/

R.I.Pienaar - www.devco.net: Lab Infra Rebuild Part 6

2024-07-31T01:00:00+00:00

This is the final in a series of posts about rebuilding my lab infrastructure, see the initial post here.

Today we’ll wrap things up with a look at some SaaS tools I use and a general look through some small utilities and things I use to bring it all together.

I’ve been enjoying my summer for the last 3 months hence the hiatus of posts.

Email

Long ago I used to run my own Zimbra but I gave up on that a few years ago, been with Fastmail ever since and really happy with their service.

Delivering email from my 20+ VMs all over the world though is tricky, getting all the various DKIM and other settings right for a big set of IP addresses and ranges quickly becomes a maintenance nightmare. But ther’s a constant trickle of stuff from them - cron job, monitoring, backup statusses and more.

After some looking around at options I found SMTP2GO who have a very generous 1000 / month free tier. This is usually fine for me but I ended up paying them anyway for a annual account. This way I have just one egress point to consider in my various email policy setups and thus far, for delivering system emails, this has been a great time saver.

Read on about DNS, Git, SSO and more.

DNS

Till this rebuild I hosted my own DNS, I’ve had a set of quite stable DNS servers on the same IP addresses for over a decade so it’s worked well.

I’ve always wanted to be rid of hosting this myself but most services are request based billing and I always felt this would not go well since you have no way of controlling how many DNS requests you get.

After some research I found ClouDNS who have a 50 zone / 2000 record plan that allows 200 million queries a month for $5/month. This is plenty and incredibly cheap. They have Geo DNS servers, support is responsive and knowledgeable and have a decent API.

For comparison DNSimple is $30/month and charges per zone and 10c per million queries a month. That’s crazy.

My usage is around 20 million queries a month so I am very comfortable in this level of service and for $5/month there is simply no way to compete with this on any kind of self hosting setup.

They have a bind zone file import feature and I use their API to do daily backups of all the zones into my local Git server using a little tool I wrote call CloudDNS Backup

Git

Speaking of Git hosting, I quite like Gitea though should probably move to Foregejo. The split from Gitea happened just as I was building things up so I am still on that.

Gitea is pretty great, it does a reasonable job of being a GitHub facsimile and uses act (also great for local action testing) to provide reasonably compatible self hosted GitHub Actions.

Gitea and Foregejo are both single binary/single process tools so really easy to get going, for my needs even their SQLite support is ideal.

SSO

Getting all things authenticated and managing users is a nightmare, I use Okta free tier to front almost all my HTTP stuff and it’s been great. With Apache mod_auth_openidc it’s easy to stick that in-front of most things even static sites.

Object Storage

I used to use Digitalocean S3 compatible storage (still do a bit tbh), but I am slowly moving all my use of that over to a private Minio instance. I do not need it to be super high available but do need the data on it secure so this runs on my Hetzner Backup server with its 6 disk redundant storage setup.

To be honest I think Minio is just unusable. It’s not that the tool is bad, it’s really great, I really want to love it, it’s that the project is just crazy with releases. It’s in everyones interest to upgrade and run the latest software but with 50 releases THIS YEAR ALONE (IT IS AUGUST!), I do not know how anyone use this in any seriousness. This is not software made to be used by real teams in the real world with real pressures on their time. Further their Release Notes are of the git log --oneline variety which does not help, commit logs are developer UX not end user UX.

So I am probably looking at an alternative soon. This is kind of why my migration to it have stalled until I can figure out what to do with this.

Security

We all have to stay on top of security alerts, in the good old days you’d just read Bugtraq and know all there is to know. These days though things are much more complex with the amount of things we run and how much goes on the internet.

I use OpenCVE to track and alert me of any CVEs on the main tools I care about.

I’ve also for the first time set my Enterprise Linux machines to auto update themselves, it’s a bit scary to be honest and I exclude kernel updates, but so far it’s been ok. Once a httpd update messed me around a bit but Puppet soon fixed that on next run so was a small inconvenience. On the whole I’d strongly recommend using this.

Monitoring

Apart from the obvious Grafana/Prometheus pair that I do self-host I use a few other things. Graphite to store my IoT data in, it just seems a bit more suitable though alas it’s in a sad state and mostly dying. I might need to revisit what I do there in time.

I have deployed Apprise everywhere and it’s really great it can notify a huge list of services and having it on every machine ready to use is great. I have it integrated in a reusable Gitea action so any failing builds result in alerts.

Apprise sends some alerts and statuses to a dedicated private Mastodon account others to Slack and others to Victorops and to Pushover. This is well worth looking into. Geting Slack/Mastadon reach outs from cron or other tools is really good.

Victorops I use for my Prometheus alerts and some other things, I like its ability to silence and to give me a clear view of the current state of things when I am not around computers. Probably better options now but it’s cheap and just works.

I use Pulsetic to do external checks on my websites, they have a very generous free tier though I did recently upgrade to a paid plan. They allow you to make great external visible or private dashboard like this one for my Choria Project Infrastructure

Letsencrypt

I use LE for TLS like more or less everyone else. ClouDNS is supported for a DNS authenticator in acme.sh so that’s a good fit.

I have a difficult problem in that I want to do a globally redundant hosting of some static files on the same name but I do not want to pay for a GSLB. So I ended up making an Action that will manage those certificates daily, renew them and commit them to my Puppet repository from where they get rolled out to the webservers. This works great.

Awesome Lists

I’ll take liberty and do a plug for my Free-for-dev list that currently has 1500+ services listed. It focus on services that provide generous free tiers generally in the devopsey world many of them suited for home labs. Most of the ones above came from this list, this is literally why I maintain this list. Currently this is the 4th most popular Awesome List with nearly 90 000 stars. Built by 1600+ people collaboratively on GitHub

If you’re not aware I also strongly recommend subscribing to This Week in Self-Hosted if home labs is your thing this will be invaluable.

In general if you are not yet on board with the whole Awesome List movement you really should, check out a giant list in sindresorhus/awesome and you’ll find many tools to track, discover, search and more.

These are the new Yahoo at grass roots level, it’s amazing and one of the most real and relevant resources out there today.

Conclusion

Well that about sums it up.

After this we’ll get back into some general blogging, there’s a fair bit I did not get into like my recent intro into 3D printing and more but this series related to my Home Lab build has more or less run its course.

betadots: Puppet best practice

2024-05-08T08:10:51+00:00

At betadots, during our Puppet code reviews, we often receive requests for a comprehensive summary of best practices and guidelines.
In response, we've compiled this article to delve deep into Puppet's best practices and implementations.

Table of content

Control Repo
- Test your control-repo!
Hiera
Node Classification
Library Module Usage
Profiles: Doing Them right
Module Development
- General Concepts and Structure
- Parameters and Variables
- Declaration and References
- Class and Resource Ordering
- Documentation
Summary
Examples
- environment.conf
- bin/config_script.sh
- manifests/site.pp
- Hiera Config
- Simple Hiera Data
- Module Classes
- Class Spec Tests

Control Repo

The control repo's layout is crucial. We scrutinize files such as environment.conf, Puppetfile, manifests/site.pp, and hiera.yaml.

In environment.conf, we focus on settings like config_version, modulepath, and environment_timeout.

When examining Puppetfile, we emphasize library modules, versions, and sources. While Puppet forge offers convenience, mirroring library module GIT repositories internally is advisable for control and security. Using git tags in Puppetfile for module versioning facilitates pre-upgrade reviews and eliminates Puppet server internet dependencies.

Test your Control Repo

Thorough testing validates code changes effectively. Our tests include:

Linting with puppet lint
Unit testing with rspec puppet
Acceptance testing using puppet beaker or puppet litmus
Catalog diffing with catalog diff tool and catalog diff viewer

Consider adopting PDK - Puppet Development Kit or leveraging onceover.

Ensure you're using the right version of RSpec-Puppet, distinguishing between maintained projects by Rodjek and Puppet Inc..

For your reference:

The official website for puppet-lint is now located at https://puppetlabs.github.io/puppet-lint/.
Please note that the older version is no longer maintained.
Similarly, for RSpec, we advise consulting the README file in the official GitHub repository. Be cautious not to mistake it for the project by Rodjek. While they may appear similar currently, this could change in the future.

Hiera

Hiera is pivotal, covering node classification, library module API parameters, and infrastructure differences. We stress correct Hiera implementation, aligning hierarchy layers with infrastructure complexity while balancing diversity and maintainability.
All Hiera layers should rely solely on Puppet or trusted facts.

Next we analyze the data and check for data keys and their module namespace usage.

If no module namespace is used, we can be sure that all hiera values are looked up using explizit lookup function.
This has a huge impact on analyzing and understanding code and data, as class parameters are mapped to different names within hiera.

Node Classification

In manifests/site.pp, we inspect resource defaults, node declarations, and data processing.

Setting resource defaults simplifies coding, while following the roles and profiles pattern aids in managing numerous identically configured systems.

For divergent infrastructures, Hiera-based node classification offers flexibility, shifting complexity from static code to dynamic data.

We usually recommend to set the following resource defaults:

file : disable backup
exec : path parameter default
package : default provider (only needed in an environment with Windows nodes)

Another topic we look for is data processing based on node data like fqdn or network information. Here we ask to please move this to a custom fact.
Nice side effect: this will drastically reduce the compilation time, as the node calculates its own data.

Library Module Usage

Leverage upstream library modules whenever possible to solve problems efficiently. However, consider community contributions or feature requests for missing functionalities rather than internal forks or rewrites.

Modern libraries adopting the data-in-modules pattern facilitate configuration solely through Hiera data, minimizing Puppet code development.

Profiles: Doing Them Right

Profiles encapsulate infrastructure components like user management, authentication, and application deployment. Combining resource and library module declarations within profiles streamlines configuration management.

Clear profile class naming aids in identifying responsibilities and maintaining a single source of truth.

The main win over just using library module data is the capability to identify which profile declares the library class.

Any subprofile which is split into multiple files - maybe due to better readability or maintainability - must be specified being a private subprofile.
This can be achieved by placing assert_private() into the beginning of the class body.

The concept of profile module development is the same as for library modules.

Module Development

Module development involves several considerations:

General Concepts and Structure: Utilize PDK for module creation and consider modulesync for managing multiple modules.
Parameters and Variables: Emphasize class parameters for flexibility and avoid unnecessary variable mapping.
Declaration and References: Ensure resource references are in the same class as the declaration and use Puppet-lint plugins for reference checks.
Class and Resource Ordering: Maintain proper class and resource ordering for predictable execution.
Documentation: Leverage Puppet Strings for comprehensive module documentation, including parameter usage, limitations, and code examples.

General Concepts and Structure

To simply get started with a new module we recommend using PDK. It will help to ramp up the module skeleton.
If you create a class with PDK, it will also create a simple test class for it.

In case that many modules must be managed, we recommend to switch to modulesync from Voxpupuli Puppet community.

Every module must have a dedicated API.

That means that there are specific classes to be used by others, which have parameter to allow adoptions.
Any subclass which is not an API endpoint, must be marked as being private.

Testing should be done on all classes, the private ones should verify for compiler error, whereas your API classes spec tests should also cover different parameters and operating systems.

Differences between operating systems must be done via hiera data in library or profile modules.

If you find self written profile class code repeated or copied within multiple modules you must check if building an additional internal library module can be used to reduce the multiple development overhead.

Testing must at least cover linting and unit tests.
Acceptance tests need VM automation or container technologies and should be added as soon as possible.

Parameters and Variables

Class Parameters allow flexible usage of profiles. Any parameter can be validated using a Puppet Data Type. You also can create custom data types if the basic ones are not sufficient for you use case. In the past validation was often done with assert_* or validate_* functions in the class body. With data types you can specify the type of data you are expecting right in the class header and don't need to care about validation later on.

With class parameters you can specify different settings for specific use cases like access to a dev and a prod machine via hiera.

Please stop doing variable mapping!

We often see that hash parameters are not used directly but that certain elements of a hash are placed into a new variable.
This only makes sense if you need the specific value multiple times.
Rule of thumb, if you need a special piece of data more then twice, you might do a mapping.
But in most cases we recommend to not do it!

Variable mapping produces code which is hard to understand, because you must always keep in mind the original variable and the mapped variables.

Declaration and References

Using resource references, you can set default values and ordering.

We sometimes see references on resources which are not local to the class.
The class config declares the file and a class service declares the service.

Whenever someone refactors on of the classes, you must also take care on the references.

We highly recommend to make use of the reference on declaration outside of class puppet-lint plugin.

Class and Resource Ordering

We mentioned that references should be local to a declaration.
But how to ensure resource ordering?

Normally resources inside a class are always processed in the same order as they appear in the class.
There are some resource types which have soft autorequire dependecies, like user and their primary group.

In general you must ensure that classes are always done in right order.
Otherwise puppet tries to make an educated guess in which order resources and classes might be.

Classes and resources ordering should be separate from declaration.

Documentation

Classes can have many parameters. So where do you look for information on how to make use of them?
This is where Puppet strings will be used.
Puppet strings allows you to render the file REFERENCE.md using a rake task.

Additionally it offers the option to run a documentation web server with access to your completely deployed control-repository.
It will check for any file updates and update the documentation automatically.
We recommend to put a web server with user authentication in front of the puppet strings web service.

Next to class responsibility, limitations, usage and parameter documentation we recommend to document any complex code.
We sometimes see multiple usages of filter, map or reduce functions which parse and restructure data.
Any new person working on the code must be able to understand why and what is done. Data examples can be useful.

Summary

Structured and documented code simplifies Puppet management. Adopting automated testing and coding principles from reputable library modules - like the ones from Puppet Inc and Voxpupuli - enhances code quality and reliability.

Examples

Below are examples illustrating various Puppet practices.

environment.conf

config_version = 'bin/config_script.sh $environmentpath $environment'
modulepath = site:modules:$basemodulepath

bin/config_script.sh

#!/bin/bash
if [ -x /usr/bin/git ]; then
  ENVGITDIR="$1/environments/$2/.git"
  /usr/bin/git --git-dir "${ENVGITDIR}" log --pretty=format:"%h - %an, %ad : %s" -1
else
  echo "no git - environment $1"
fi
exit 0

manifests/site.pp

File {
  backup => false,
}
Exec {
  path => $facts['path'],
}
if $facts['os']['family'] == 'windows' {
  Package {
    provider => 'choco',
  }
}

# Node classification - sorted Hash
# e.g.
# classes_hash:
#   '01_base': 'profile::base'
#   '02_security': 'profile::security'
#   '12_application': 'profile::application::billing'
#   '13_service':
#     - 'profile::application::billing::backend'
#     - 'profile::application::billing::admin_ui'
#
# overwriting identifier classes:
# classes_hash:
#   '13_service': 'profile::application::billing2'
#
# disabling identifier classes:
# classes_hash:
#   '13_service': ''
#
# $element[0]: the hash key identifier
# $element[1]: the class name to load
#
$classes_hash = lookup('classes_hash', { 'value_type' => Hash, 'default_value' => {} })
$classes_hash.keys.sort.each |$key| {
  if $classes_hash[$key] != '' {
    contain $classes_hash[$key]
  } else {
    echo { $key:
      message  => "Class for ${key} on ${facts['networking']['fqdn']} is disabled",
      withpath => false,
    }
  }
}
node default {}

Hiera Config

---
version: 5

defaults:
  datadir: data
  lookup_key: eyaml_lookup_key
  options:
    pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
    pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"

hierarchy:
  - name: "node yaml hierarchy"
    paths:
      - "nodes/%{trusted.certname}.yaml"
  - name: "role yaml hierarchy"
    paths:
      - "role/%{trusted.extensions.pp_role}-%{trusted.extensions.pp_env}.yaml"
      - "role/%{trusted.extensions.pp_role}.yaml"
  - name: "os yaml hierarchy"
    paths:
      - "os/%{facts.os.family}-%{facts.os.release.major}.yaml"
      - "os/%{facts.os.family}.yaml"
  - name: "zone yaml hierarchy"
    paths:
      - "zone/%{trusted.extentions.pp_zone}.yaml"
  - name: "Common"
    path: "common.yaml"

Simple Hiera Data

# data/common.yaml
---
# Node classification using classes_hash
classes_hash:
  '00_base_ssh': 'ssh'
  '01_base_nft': 'profile::nftables'
  '02_base_nft_rules': 'profile::nftables::rules::base'

# Commmon SSH data
# using saz-ssh: https://forge.puppet.com/modules/saz/ssh/readme
ssh::server_options:
    Protocol: '2'
    ListenAddress:
        - '127.0.0.0'
        - "%{::facts.networking.hostname}"
    PasswordAuthentication: 'yes'
    SyslogFacility: 'AUTHPRIV'
    UsePAM: 'yes'
    X11Forwarding: 'yes'

ssh::server::match_block:
  filetransfer:
    type: 'group'
    options:
      ChrootDirectory: '/home/sftp'
      ForceCommand: 'internal-sftp'

# Adding core resources
# requires puppetlabs-stdlib : https://forge.puppet.com/modules/puppetlabs/stdlib/readme
stdlib::manage::create_resources:
  'package':
    'nano':
      ensure: 'absent'
    'vim':
      ensure: 'present'
  'user':
    'monitoring':
      ensure: 'present'
      uid: '11225'
      gid: '11225'
      password: '!'

Module Classes

# @summary Installs and configures the application
# @author betadots GmbH
# @param parameter1 Parameter to set thing 1
# @param parameter2 Parameter to set thing 2
# @example
#     include application
#   or
#     class { 'application':
#       parameter1 => 'value,
#     }
#
class application (
  Datatype $parameter1,
  Datatype $parameter2 = 'default',
) {
  contain application::install
  contain application::config
  contain application::service

  Class['application::install']
  -> Class['application::config']
  ~> Class['application::service']
}

class application::install {
  assert_private()
  # Puppet DSL
}

...

Class Spec Tests

# spec/classes/application_spec.rb
describe 'application' do
  on_supported_os.each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }

      it { is_expected.to compile.with_all_deps }
      it { is_expected.to contain_class('application::install')}
    end
  end
end

# spec/classes/application_install_spec.rb
describe 'application::install' do
  on_supported_os.each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }

      it { is_expected.to compile.and_raise_error(%{private}) }
    end
  end
end

Happy puppetizing,

Martin

R.I.Pienaar - www.devco.net: Lab Infra Rebuild Part 5

2024-04-25T06:00:00+00:00

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today let’s look at VMs and Baremetals and Operating Systems.

Virtual Machines

As I mentioned I’ve been a Linode customer since essentially their day one and have had many 100s of machines there for personal and client needs.

I soured off them quite significantly after they botched their Kubernetes release by not having Europe side support past basic triage initially that led to multiple multi hour outages and so I have been making some changes.

My most recent incarnation had about 5 or 6 Linode machines - Puppet Server, Choria Repos * 3, 2 x DNS, General Services machine.

Today I have 2 Linode machines left. I was reluctant to move them as they were DNS servers for many domains but I changed my way of hosting domains also during this so less of a concern now.

They are now just RPM/Deb repos for Choria and I will move those elsewhere soon also. That’ll be the first time I am without Linode machines since basically 2003, such a shame. One is in the US to provide a nearby mirror there, I might keep it and just scale it down to lower spec. But with the recent changes at Linode it feels a bit like it’s time to consider alternatives.

Previously I had Digital Ocean droplets x 3 for my Kubernetes cluster, as discussed that’s all gone now too.

I used to have quite a selection of Vultr machines, I don’t recall why I left them really I think I felt Linode was just the rolls royce of this kind of Cloud provider and so consolidated to simplify my business accounting etc

Baremetal

In the previous iteration I had only 1 hosted physical machine and that was my Backups machine running Bacula on a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm. I do not need much from this machine, it wakes up, do backups then sleep again till tomorrow. So the spinning rust is fine for that, I just need lots of it. I rebuilt on a new one just to get a hardware and OS refresh.

Of course I do still use Virtual Machines just managed by Cockpit as per the 2nd part in this series.

I got a pair of Hetzner AX41-NVMe (64GB RAN Ryzen 5 3600) with 2 x 512 NVMe SSDs each. I was expecting to add a 3rd but really these 2 plus the Ryzen at my office turns out to be plenty for my needs. They have some upgrades available - more RAM, Extra Disks, SATA can be added etc. I don’t know if Hetzner supports upgrading running machines but this is nice little platform. At EUR37 per machine that puts them between a 4GB and 8GB shared CPU Linode. You really can’t complain. I might get a 3rd one just for the sake of it and spread my development machines out more. Something for after the summer.

Performance wise moving from my Droplets and other VMs to these machines have been amazing, for an investment equalling 1 Linode I can run several VMs with no additional cost to expand to another VM or to shuffle memory allocations - or just to allocate more since 64GB is way more than I need. This really is a no brainer for my needs.

I’ve been a Hetzner customer also since a very long time, it’s not clear how long but it feels like maybe 2008 or so. They’ve had their ups and downs and dodgy datacenters, dodgy connectivity and dodgy hardware, bad english support, but in the past few years I think they’re really firmed up quite nicely and my machines there have not given me trouble so I felt it’s safe to lean on them a bit more. A few months in now and I’ve not had one minute of problems.

Read on about Operating Systems and more.

Operating System

A bit of history is needed I guess. I started with Softlanding Linux System around 1993, after installing this on a 20MB HDD I removed from a Novell machine the disk promptly died. A bit of hits and misses later I eventually got it quite happy.

Then I moved to Slackware which was the successor to SLS of course. I tried a few things but once RedHat released their first preview in October 1994 on Halloween it was pretty much just RedHat all the way from there. I’ve used all the versions, even the 5.x series that was a total disaster after they moved to ELF but kept at it.

When RedHat removed their free editions I dabbled with Debian but I don’t like the inconsistencies, community, approach or really anything at all about Debian.

Luckily of course CentOS came to the rescue, I even donated hardware to them - a IBM Bladecenter full of Blades. Of course then for some reason they decided to join RedHat, I never did understand the thinking here. Needless to say that did go about as well as it was obvious to anyone it would.

This left a few choices, mainly Rocky Linux and AlmaLinux. I initially went with Rocky Linux it seemed to be on a good track and had the CentOS founder on board, they did seem to loose steam a bit and had a few fits and starts. Now seems fairly solid so I wouldn’t have qualms using them but I settled on AlmaLinux. AlmaLinux seemed at the time to have a bit more backers, a bit more polish and just worked better for my needs.

Then of course RedHat removed the source repos from CentOS Stream this left things quite difficult. Rocky Linux, then, stated they are going to get their SRPMs from a bunch of places to keep the bug-for-bug EL rebuild thing going by means which I really do not think is sustainable. From using paid-for cloud instances and then getting SRPMs using those or using the UBI container images etc, basically they go spelunking all over the place to find source RPMs and then somehow promise to be a complete like-for-like distribution of RHEL. I am not convinced.

Almalinux went another way, they decided to drop the 1:1 RHEL promise and instead moving to promising ABI/Binary compatible with RHEL. They probably also get their SRPMs from some interesting places but they are forging a path of innovation which already resulted in them returning some old hardware support for example. I am quite happy with my choice.

To complicate matters a bit more - or simplify them I guess - the Open Enterprise Linux Association was created to be a source of SRPMs for EL rebuilders like Alma and Rocky. Their mission is to deliver All sources necessary to achieve a 1:1 / bug-for-bug compatible version of EL which will be distributed via Git, encouraging community collaboration. Hopefully this will help matters along. This is supported by Oracle and SUSE. Oracle of course have their own Oracle Enterprise Linux that was dunked into the same problem - and no doubt the source of all the headaches for RedHat.

Anyway, so a lot of history, I like EL. I will keep using EL even if its a slightly different EL. I don’t think its inherently better or worse than other alternatives - and I do not care. The best tool for the job is often the one you know best. I have 30 years of EL experience and that works for me. It’s all just SystemdOS now anyway.

Conclusion

That’s about it really for hardware and OS. The consolidation, moving to Baremetals etc have ended up resulting in quite a bit of money saved. Even if I got a 3rd VM host at Hetzner I’d still be more than a $100 down on monthly expenses but got a lot more.

I have some travel coming up in the next few weeks so the next installment might be a while.

Camptocamp Blog: Unveiling the Simplicity of Cluster Mesh for Kubernetes Deployments

2024-04-16T13:30:57+00:00

Unveiling the Simplicity of Cluster Mesh for Kubernetes Deployments

During Kubecon EU 2024, among a crowd of tech enthusiasts and Kubernetes aficionados, Liz Rice the Queen bee, demo’ed multi-cluster networking. This is Cluster Mesh 101 with Cilium.

Here are a few paragraphs summarizing the experience.

Overview

Cluster Mesh extends the networking plane across multiple clusters. It enables connection among endpoints of connected clusters. Two noticeable features are:
i) Network Policy Enforcement as implemented by Cilium prevails even under this network setup
ii) Services can load balance requests among clusters just by using annotations

Networking Adventures Begin

To the surprise of many, Liz announces she will be running the demo over the venue's Wi-Fi.

Presentation gets started with connectivity tests over VPN connections, routes propagation validation, checks of the BGP peering and visualization of routing tables. The setup is a running k8s cluster in GKE and another in EKS. Node to node Network connectivity is the final objective here, and do not forget all assigned IPs should be not overlapping. Cilium cannot create a bridge between two cloud providers. No black magic.

This is foreplay preparing the ground for the demo. The steps for reproducing this can be found in official Cilium here.

Enter Cilium's Cluster Mesh

With the foundational work laid, it's time to kickstart the demo. Liz demonstrates how to enable all necessary components with cilium clustermesh enable in both clusters. These trigger the deployment of the clustermesh-apiserver into each cluster, along with the generation of all required certificates. This component goes the extra mile by attempting to auto-detect the optimal service type for LoadBalancer, ensuring the Cluster Mesh control plane is efficiently exposed to other clusters.

A simple cilium clustermesh connect builds the bridge between clusters, just as if we had a single network plane between Pods across all clusters.

Now cilium clustermesh status echos:

✅ All 2 nodes are connected to all clusters
🔌 Cluster Connections:

cilium-cli-ci-multicluster-2-168: 2/2 configured, 2/2 connected

Traffic load balancing and failover in the hive:

Till now demo showed and validated the piping between Pods. We can successfully communicate with a Pod on another cluster using its IP address for example. But this wouldn't be very practical in a real world-scenario. Moreover no DNS service can help us fetch this dynamic IP. In fact, this is the raison d'être of k8s Service object.

What Cilium proposes is extending native Service resources into a cluster-mesh-aware Service using annotations.

Cilium provides a pragmatic solution through global services with auto discovery and failover mechanisms.

By using service.cilium.io/global=true makes a service global, which meansmatching Pods across clusters. Or in other words we extend service’s backends to use Pods in remote clusters. Then the service’s traffic is balanced across clusters.

Questions? This is probably better explained by Ciilium documentation here

apiVersion: v1
kind: Service
metadata:
  name: rebel-base
  annotations:
    service.cilium.io/global: "true"
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    name: rebel-base

With service.cilium.io/affinity=local|remote we can fine tune the global service to prefer local or remote Pods. With local we could designate our local cluster as the primary destination, while the remote Pods serve as a backup.

The following represents a service, which is global and prefers using endpoints found locally:

apiVersion: v1
kind: Service
metadata:
  name: rebel-base
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/affinity: "local"
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    name: rebel-base

Should the primary cluster encounter any issues or downtime, traffic seamlessly shifts to the backup cluster, ensuring continuity of service.
In essence, Cilium offers a straightforward approach to traffic management, enhancing reliability by providing a failover mechanism that ensures service accessibility remains intact in the face of pod disruptions.

Conclusion

A few interesting use cases arise when using Cluster Mesh. The one we focused on in this article is about leveraging remote clusters Pods as service's Pod backend as failover mechanism. We can also think of using a global service for moving workloads around, for example for lowering computational costs into cheaper regions.

So there you have it, folks. A whirlwind tour of multicluster networking traffic management with Cilium, served up with a dose of honey.
Who knew cluster meshing would be that simple?

Contact us

Needs a demo, or dig into some specifics on Cilium?
Ping us here: https://camptocamp.com/consulting

R.I.Pienaar - www.devco.net: Lab Infra Rebuild Part 4

2024-04-11T09:00:00+00:00

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today I’ll talk about my physical office and office hardware.

Office Space

When my son started going to school I did not look forward to all the driving so figured a office near his school would be good, I’d spend the days there and come home after pick up. I rented a nice place in a town called Mosta, it was nice and had ample storage and would have made a really great maker space as it had about 4 car garages worth of underground storage that was well lit and ventilated.

Unfortunately this place was opposite a school and parking was absolute hell. I ended up just not using it for months at a time since I could drive home in 12 minutes or spend 45 minutes finding parking, no thanks. I gave up trying to find a garage to rent around there, it’s just crazy.

When I started looking the 2nd place in Żebbuġ I saw seemed perfect, a fantastic bright 7m x 7m office with a underground 1.5 car garage at a reasonable price. I took that and just recently extended my lease to 5 years. My sister-in-law is also moving her business to the same street and there’s a 3D printer shop around the corner, bonus. Parking is always easy even on the street and it’s like 5 minutes from my house.

Here I am able to set up 3 workstations, my main desk, one for guests with a little Desktop for my son and big work station for soldering, assembling IoT projects and such. I also have a nice 2 seater sofa with a coffee table. With the garage I have space to put things like a lazer cutter and more in future - though I am eyeing a small space across the road as a workshop space also for that kind of thing.

Location wise I could not want more, it’s a easy walk or cycle from my house through a beautiful car-free valley and the town is pleasant enough with food options and a small corner shop nearby.

Read on for more about the hardware.

Desktop / Laptop

I’ve been using using Apple desktops and laptops almost exclusively since about 2007. I do like the graphical UI but loathe the BSD based shell, so generally my mantra is: MacOS Shells are for SSH to Linux machines.

I’ve had every iMac since the very first plastic white ones, I really liked that form factor, so I just kept buying each model that came out. I especially liked the time when Apple sold visually complimentary displays for these machines and you could have a quite pleasing dual screen setup.

Alas those days are gone, now to be honest, every iMac dual screen setup just looks like rubbish, so I just can’t with that anymore and so it was time to change. I now have 2 useless Display port old Cinema displays, guess they go to the trash.

I have a MacBook 16" M2 PRO 32GB/1TB SSD, which I think is pretty much perfect - despite coffee damage knocking out all the ports on the right side, it’s still awesome.

I’ve always had a disconnect between Laptop and Desktop speeds and this time round the stars aligned reasonably well that one can almost have parity between the 2, so after some looking around I figured a Mac Mini will give me near enough performance and good enough options for displays.

To be honest I was not too convinced about moving to a Mini - but the iMacs were never really speed daemons so maybe it would be fine? So I picked up a Mac Mini M2 PRO 32GB/1TB SSD and really moving between the 2 just feels exactly the same performance wise. This has been a bigger deal than I anticipated for my general happiness when switching setup - something I do all the time.

The mini sitting on my desk is a bit annoying, I’m considering options to mount it on the wall behind the display.

For my display I went with a LG 40" Curved UltraWide 5K2K Nano IPS Monitor. I’ve used a 32 inch ultra wide for a few years, this has been a great upgrade. It’s quite pricey around EUR1200, but it’s a really nice display and as it’s a Thunderbolt monitor it just works flawlessly with the Macs. I did buy BetterDisplay to get some more control, well worth it - I’d say essential if you have a monitor like this. I do wish it was brighter though.

I used to have a Moode Audio system with some high end DACs and Amp paired with JBL monitors for audio, I got those before HomePods existed. Now tossed all that out for just 2 HomePod Minis on my desk, they’re fantastic.

I use a Logitech Brio 4K Ultra HD Webcam - works well on MacOS including it’s management software. My previous camera did not handle the sync frequency of my new office lights well and so was instantly useless.

Today I am replacing my Razer Abyssus (2014) with a new Abyssus that doesn’t look like it’s been to war.

Keyboard wise I use the MS Natural 4000 and have 4 more in boxes so I don’t need to think about keyboard for years, they last about 3 to 4 years each.

Storage

I’ve been a Qnap user since forever, like maybe 2005 or so. The first Qnap I bought was some weird ARM CPU and they ran a patched Linux Kernel to get big partition support. When the machine died I had quite a bit of trouble to access my files but did get there eventually - but did make me reconsider my backup strategy.

Their next gen kit was Intel based and ran stock kernels so I gave them another go. What I got then was a 4 bay TS-439. This machine moved with me from London to Malta and now 14 years later it still got a security update just before I retired it, unbelievable.

Later I got a TS-451+ as the old ones CPU was a bit slow, the old one moved to my office and I synced to it daily as a backup. But as the old one is now 14 years old it seems I am just pushing my luck with this box, so I retired it. Moved the TS-451+ to the position of backup machine and got a new TS-453E.

The TS-453E is interesting because it’s a kind of LTS hardware that will have full hardware support till 2029 and probably software far past that. I got it with an extended warranty also so this should set me up for 5 years at least on this platform.

I set the primary NAS up in 2 x RAID-1 configuration and basically perform backups like this:

Real time RAID-1 mirror between 2 disk for the primary volume
Daily sync my Dropbox onto the primary NAS
Daily sync primary to the backup NAS that is running a RAID-5 setup in another location in Malta
Daily sync the primary to a 5TB external USB drive that I can take to a 3rd offsite when I leave the office for months over summer
Monthly I do a full bitrot check across all the files on the primary
Monthly I do a manual sync between the RAID-1 volumes of the primary NAS
Monthly I sync the full primary NAS to Finland on a machine with a RAID-6
My primary desktop does TimeMachine backups to the primary regularly

This achieves the following redundancy:

+1 disk redundancy for any change I made now thanks to the first RAID-1
daily same-country offsite to a RAID-5 storage
monthly OS-level integrity checks of all files and the chance to restore any to the primary
monthly on-site backup for long term recovery should I notice a accidental file deletion or corruption
monthly backup to another country.

In the end every file lands on about 11 disks in duplicate, 3-4 locations, 2 countries, different raid levels and with bitrot detection at multiple RAID levels both standard Linux kernel and NAS kernels and at the OS level. I could make the final Finland archive a 3 month rolling archive since I have enough space there for it.

The machine I mention in Finland is a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm drives it runs Bacula for my server backups, but for above I just rsync to it. It’s a slow crappy machine tbh but for the purpose of mostly just keepting files on disks mostly idle, it’s perfectly fine and the price is great. I’ve had similar machines for a decade or more, I cycle them every 3 years to new hardware.

Linux Shell

I mentioned earlier that I live by: MacOS Shells are for SSH to Linux machines, so I need a Linux machine.

I used to use the NUC machines but as they were ancient and dying I needed something else. Slack suggested I look at the Asus Mini PC PN51 and what a great suggestion that was.

I have the Ryzen 7 5700U based machine with 64GB RAM and 1TB M.2 SSD in it, this machine runs CentOS and later Almalinux flawlessly. I used to use it bare but that was a bit of a waste now it runs Cockpit as per my previous post and I have a 16GB VM there dedicated as my shell box.

Along with that I have 6-10 VMs elsewhere making up a dev environment.

Miscellaneous Hardware

At my office I have a few other things:

One of my old NUC machines is a Linux Desktop for my son, mainly for minecraft/youtube when he visits
A Krups Nescafe Dolce Gusto Infinissima Touch Black Automatic Coffee Machine - it’s rubbish, don’t buy these coffee machines
A bar fridge
Xiaomi air purifier
Xiaomi robot vacuum - the nice one with auto bag empty feature so the place keeps clean when I am away for months at a time over summer
A Prusa MK4 and it’s many accessories (this is a whole blog post to be made)
A 15 year old Brother printer that is enormous and has finally now probably met it’s end, probably fixable, but I want an excuse to buy a smaller printer
Ubiquity Dream Machine and a 8 port switch

Everything is new except the Brother printer, but we’ll fix that soon.

I try to not clutter the place up so I keep things to a minimum here and as mentioned I might move my workshop across the road to another unit and get some more 3D printers, Lazer cutters and so.

Conclusion

That’s about it for the physical location and physical hardware. I’ll need to redo my home office as I gutted it and brought most things here but that’s for later, it’s mainly used now for its real purpose of being a media room with a 100 inch 4K project Dolby Atmos audio setup.

Camptocamp Blog: Beyond the Buzz: Embracing the Magic of eBPF in Kubernetes

2024-04-10T14:02:32+00:00

Beyond the Buzz: Embracing the Magic of eBPF in Kubernetes

In a time where the buzz around Artificial Intelligence (AI) seems to overshadow everything else, this year's KubeCon Europe offered a refreshing perspective. While AI continues to be a hot topic, some in the Kubernetes community are starting to feel a bit tired of it. With all the hype and uncertainty surrounding AI, another hero has emerged: eBPF (Extended Berkeley Packet Filter).

AI: A Distant Shining Horizon

AI has certainly added some excitement to discussions about cloud-native technologies, from automating cluster troubleshooting to hosting AI on Kubernetes. But not everyone is fully on board. While AI has proved to be of great assistance - e.g. suggesting, fixing and reviewing code written by humans- some folks worry that too much focus on AI might distract from more practical, ready-to-implement advancements. The feeling is clear: while AI offers lots of possibilities, the horizon is a bit uncertain and overly hyped.

eBPF: Here and Now in Kubernetes Innovation

In contrast, eBPF is all about practical innovation. It's a technology that delivers real results, right here, right now. We need solutions for network security, observability, and performance today, and that's where eBPF shines. Unlike the abstract promises of AI, eBPF offers concrete tools and methods to improve Kubernetes environments immediately. For example, when it comes to network security, eBPF-powered tools like Cilium can do the job without needing the complexity of AI.
This shift towards valuing what's immediately useful over what's exciting but distant was noticeable at KubeCon. As we dive deeper into what eBPF can do, it becomes clear why this technology has captured the attention of the Kubernetes community.

The Way Forward with eBPF

By embracing eBPF, the Kubernetes community isn't just adopting new tools; it's championing a philosophy of practical, tangible progress. As we explore the latest eBPF innovations – from better security to revolutionary observability tools – we see the benefits eBPF brings to Kubernetes. It's a journey grounded in reality, offering not just a vision of the future, but a roadmap to get there.

Redefining Efficiency: eBPF's Radical Return to Linux's Roots

eBPF represents a refreshing departure from the conventional approach of stacking layer upon layer in pursuit of functionality, often resulting in bloated, resource-intensive systems, even for simple tasks like rendering a webpage or routing network packets between nodes or clusters. With eBPF, we're venturing back into the depths of the Linux system, where innovation meets efficiency. Here, we witness a paradigm shift—a departure from the status quo. The results speak for themselves: a nearly negligible overhead and remarkable responsiveness. In fact, eBPF brings us so close to real-time processing that it's revolutionizing how we think about performance in cloud-native environments. We're not just optimizing; we're redefining what's possible, and eBPF is leading the charge.

eBPF talks from KubeCon 2024

Cilium: Connecting, Observing, and Securing Service Mesh and Beyond with eBPF
Speakers: Liz Rice, Christine Kim, Nico Meisenzahl, Vlad Ungureanu
https://youtu.be/wq1TxZw1AaY?si=JTyhE333QfsGht0T
Dealing with eBPF’s Observability Data Deluge
Speaker: Anna Kapuścińska
https://youtu.be/yWB8n_e4N14?si=OyMJEKzbxS5zxA5P
Unlock Energy Consumption in the Cloud with eBPF
Speaker: Leonard Pahlke
https://youtu.be/lW9pZoKRJVs?si=rX5CQMaFuZBm8bBT
Fast and Efficient Log Processing with Wasm and eBPF
Speaker: Michael Yuan
https://youtu.be/4u7nUpZxr3g?si=pkcpoEwOeDaH5HcE
No 'Soup' for You! Enforcing Network Policies for Host Processes via eBPF
Speaker: Vinay Kulkarni
https://youtu.be/AWAf3H4Qwq8?si=qVqQfWb3J_905BCJ
eBPF’s Abilities and Limitations: The Truth
Speakers: Liz Rice & John Fastabend
https://youtu.be/tClsqnZMN6I?si=TyMFTMk4Q45K6T2v

R.I.Pienaar - www.devco.net: Lab Infra Rebuild Part 3

2024-04-07T08:00:00+00:00

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today I’ll talk a bit about Configuration Management having previously mentioned I am ditching Kubernetes.

Server Management

The general state of server management is pretty sad, you have Ansible or Puppet and a long tail of things that just can’t work or are under terrible corporate control that you just can’t touch them.

I am, as most people are aware, a very long term Puppet user since almost day 1 and have contributed significant features like Hiera and the design of Data in Modules. I’ve not had much need/interest in being involved in that community for ages but I want to like Puppet and I want to keep using it where appropriate.

I think Puppet is more or less finished - there are bug fixes and stuff of course - but in general core Puppet is stable, mature and does what one wants and have extension points one needs. There’s not really any reason not to use if it fits your needs/tastes and should things go pear shaped its a easy fork. One can essentially stay on a current version for years at this point and it’s fine. There used to be some issues around packaging but even this is Apache-2 now. If you have the problem it solves it does so very well, but the industry have moved on so not much scope for extending it imo.

All the action is in the content - modules on the forge. Vox Pupuli are doing an amazing job, I honestly do not know how they do so much or maintain so many modules, it’s really impressive.

There’s a general state of rot though, modules I used to use are almost all abandoned with a few moved to Vox. I wonder if stats about this is available, but I get the impression that content wise things are also taking a huge dive there and Vox are holding everything afloat with Puppet hoping to make money from selling commercial modules - I doubt it will sustain a business their size but it’s a good idea.

Read on for more about Puppet today in my infra.

Given the general state of things in the server management world I decided to use Puppet again for this round of infrastructure rebuild. I can’t see this lasting alas. Most people are aware that Puppet have been bought by Perforce and have had a huge shift in people and such. It’s inevitable that revenue generation is the main push at the moment.

Unfortunately the way this play out is pretty unpleasant. Here’s an example: I have an ancient, and crappy, monitoring script that runs on the node and checks last_run_summary.yaml to infer the current status of runs - are they failing etc. It’s worked for a very long time and last_run_summary.yaml is a contract that’s to be maintained. Something in recent Puppet broke this script (the last_run_summary.yaml now behaves differently) so I thought I’d ask on their Slack if there is something newer/maintained before I fixed mine.

Immediately from Puppet people you get a message saying to use Puppet Enterprise for this. In a community Slack where people are just asking questions about a 200 line script. The suggestion is to move to a price-not-disclosed product vs a 200 line script. Without even so much as a question about needs or environment or if the suggestion would be relevant. It’s just corporate enshitification. Eventually I got some good answers from the community, despite Puppets best efforts.

This outcome is of course entirely predictable, I can only hope Vox doesn’t get burned in the inevitable slide into the sewer.

Managing Puppet

I have 2 old EL based machines that could not make the trip to Puppet 8 due some legacy there, the rest got moved to EL9 with a Puppet Server. I am though a big fan of running puppet apply based builds and will likely move to that instead of the server. Apply based workflows present a few problems though, primarily how you get the code on the nodes and how the workflow is around that.

I’d like a git based flow where I commit a change, CI package it and puts it on a repo and the fleet updates to it. Ideally the fleet updates to it asap. Further I want visibility into the runs, node-side monitoring which fits my event based world-view and allows me central control to trigger runs and do scheduled maintenance.

So I built a system to orchestrate Puppet that’ll release soon called Puppet Control.

Provides a nice cli for triggering runs, querying runs etc
Git+CI based flow handles fetching, validating and deploying code to nodes for apply, including tamper detection
Has concurrently control built in for fine-grained resource management of file servers used to deliver code bundles
Includes a run scheduler with concurrency controls that include ability to say for eg: only 1 database server out of all database server can run Puppet at a time but webservers can run 10 concurrently
Can do on-demand runs as soon as possible subject to concurrency control to ensure the shared infra performs at peak
Has various ways to find nodes in certain states like pctl nodes failing to find nodes with failing resources
Can show real-time events of Puppet runs
Can have maintenance declared that will stop all scheduled Puppet runs
Expose run statistics to Prometheus for runtimes, changes, health and more

There’s an optional video below with more details and show the code release flow etc:

I’ll release this eventually, it’s dependent on some work happening in Choria at the moment.

The concurrency control is a big deal. Scheduling Puppet runs is quite a difficult task. Usually the solution Puppet users do is just to spread the runs in cron by some random time distribution.

That leaves the problem of fast Puppet runs during maintenance windows though. In the past we had a command mco puppet runall 100 which would discover all the nodes then in a loop ask their state and schedule more as some stopped - the goal being to keep as close to 100 running at a time. The choice of 100 nodes is related to the capacity of the Puppet Server infrastructure.

This worked fine but it was very resource intensive on the Choria/MCollective network as 1000s of RPC requests have to be made to know the current state and it was not suited to using in an ongoing fashion. With Puppet Control every run is a run that happens at the desired concurrency, but without a central orchestrator trying to make all the choices. It’s significantly cheaper on the network.

More significantly by allowing the concurrency group name to be configured on a per node basis one can have different policies by type of machine. This is a big deal, let’s say we are using puppet apply but we do not wish to have our 5 database machines all do concurrent runs and potentially restarting at the same time. By creating a concurrency governor for just those machines set to 1 we prevent that.

With this in place triggering all the nodes to run at their groups configured concurrency is a simple pctl nodes trigger which takes 2 seconds to complete. From there the nodes will run without overwhelming the Servers.

Another interesting thing here is that this model maps well onto Ansible local mode as well. So in theory, unexplored theory, this same central controller and scheduler could be made for Ansible.

This is built on Choria Concurrency Governors which is an amazing distributed system building block.

Choria

No surprise that I am using Choria for a large part of this, with a bit of a twist though. Choria, as released on choria.io, is actually a distribution of a much larger system that is tailored for Puppet users. That official Choria release requires Puppet Agent and will not support unofficial builds or unsupported deviations from that.

With the writing being on the wall though for Puppet this leaves me with a problem, I have no easy to use Public distribution of Choria for non Puppet users. Puppet provides the following to Choria:

Deployment of the packages and files to nodes
Management of policies and plugins on nodes
Certificate Authority with certs on every node
Optional discovery source of truth in PuppetDB
Libraries for managing packages and services

These are actually quite significant hurdles to cross to create a fully Puppet-free Choria distribution.

That said for a long time Choria have had another life as a large scale orchestrator that is not Puppet related. This implies:

It can self-provision at a rate of 1000+ nodes / minute
Deploy its own plugins at a rate of multiple plugins delivered to millions of nodes in minutes
Manage its own security both integrated with a CA or using a new JWT+ed25519 based approach
Integrate with non Puppet data sources using external extension points
Can upgrade itself in place without any help from Puppet in an Over-The-Air type self-upgrade system
Has centralised RBAC integrated with tools like Open Policy Agent

In the last 2 years I was on a related contract and all these components have been firmed up and made much more capable, reliable and horizontally scalable and used in anger in the real world in some quite serious mission critical builds.

Thus, my Choria infrastructure is actually a highbrid between Puppet and Non Puppet. Puppet places the RPMs and plugins that require Puppet (package, service, legacy plugins), but Choria self provisions everything else and owns the life of the agent and more. I am running the new Protocol and with Open Policy Agent based RBAC. I’ve made some changes to the various Puppet models to enable this and will start looking for some early adopters.

This means my many Raspberry PI - from Xmas lights to sensors and HVAC control - are all now managed by Choria as well as the provisioner caters for them since those are without Puppet.

I wrote about the new protocol in an ADR and you can see there is scope for integration into TPMs and more. This is the future world view of Choria and already in use on some 100s of thousands of machines.

Conclusion

So that’s a bit about managing the machines without Kubernetes or a ISP managing them.

As I’ve been out of active Puppet use for a few years it’s been interesting to come back with some semi-fresh eyes and rethink some of the old things I believed was true when I used it constantly.

Choria will play a critical role in the path forward as I’ll move much into containers managed as per the previous blog post leaving the problem Puppet solves to quite a thin layer. I’ve some thoughts on doing something to at least replace the most basic package-config-service trio of CM with something in Choria long term.

R.I.Pienaar - www.devco.net: Lab Infra Rebuild Part 2

2024-03-21T08:00:00+00:00

Previously I blogged about rebuilding my personal infra, focussing on what I had before.

Today we’ll start into what I used to replace the old stuff. It’s difficult to know where to start but I think a bit about VM and Container management is as good as any.

Kubernetes

My previous build used a 3 node Kubernetes Cluster hosted at Digital Ocean. It hosted:

Public facing websites like this blog (WordPress in the past), Wiki, A few static sites etc
Monitoring: Prometheus, Grafana, Graphite
A bridge from The Things Network for my LoRaWAN devices
3 x redundant Choria Brokers and AAA
Container Registry backed by Spaces (Digital Ocean object storage)
Ingress and Okta integration via Vouch
Service discovery and automatic generation of configurations for Prom, Ingress etc

Apart from the core cluster I had about 15 volumes, 3 Spaces, an Ingress load balancer with a static IP and a managed MySQL database.

I never got around to go full GitOps on this setup, it just seemed too much to do for a one man infra to both deploy all that and maintain the discipline. Of course I am not a stranger to the discipline required being from Puppet world, but something about the whole GitOps setup just seemed like A LOT.

I quite liked all of this, when Kubernetes works it is a pleasant experience, some highlights:

Integration with cloud infra like LBs is an amazing experience
Integration with volumes to provide movable storage is really great and hard to repeat
I do not mind YAML and the diffable infrastructure is really great, no surprise there. I hold myself largely to blame for the popularity of YAML in infra tools at large thanks to Hiera etc, so I can’t complain.
Complete abstraction of node complexities is a double-edged sword but I think in the end I come to appreciate it
I do like the container workflow and it was compatible with some pre-k8s thoughts I had on this
Easy integration between CI and infrastructure with the kubectl rollout abstraction

Some things I just did not like, I will try to mention some things that not the usual gripes:

Access to managed k8s infra is great, but not knowing how its put together for the particular cloud can make debugging things hard. I had some Cilium failures that was a real pain
API deprecations are constant, production software rely on Beta APIs and will just randomly break. I expected this, but over the 3 years this happened more than I expected. You really have to be on top of all the versions of all the things
The complimentary management tooling is quite heavy like I mentioned around GitOps. Traditional CM had a quick on-ramp and was suitable at small scale, I miss that
I had to move from Linode K8s to Digital Ocean K8s. The portability promises of pure kubernetes is lost if you do not take a lot of care
Logging from the k8s infra is insane, ever-changing, unusable unless you really really are into this stuff like very deep and very on-top of every version change
Digital Ocean does forced upgrades of the k8s, this is fine. The implication is that all the nodes will be replaced so Prometheus polling source will change with big knock on effect. The way DO does it though involves 2 full upgrades for every 1 upgrade doubling the pain
It just seem like no-one wants to even match the features Hiera have in terms of customization of data
Helm

In the end it all just seemed like a lot for my needs and was ever slightly fragile. I went on a 3 month sabbatical last year and the entire infra went to hell twice, all on its own, because I neglected some upgrades during this time and when Digital Ocean landed their upgrade it all broke. It’s a big commitment.

See the full entry for detail of what I am doing instead.

Container Management

So let’s look at what I am working on for container management. My current R&D focus in Choria is around Autonomous Agents that can manage one thing forever, one thing like a Container. So I am dog-fooding some of this work where I need containers and will move things into containers as I progress down this path.

Looking at my likes list from the Kubernetes section above we can imagine where I am focussing with the tooling I am building, lets just jump right in with what I have today:

containers:
  tally:
    image: registry.choria.io/choria/tally
    image_tag: latest
    syslog: true
    kv_update: true
    restart_files:
      - /etc/tally/config/choria.conf
    volumes:
      - /etc/tally/config:/tally/config
    ports:
      - 9010:8080
    register_ports:
      - protocol: http
        service: tally
        ip: %{facts.networking.ip}
        cluster: %{facts.location}
        port: 9010
        priority: 1
        annotations:
          prometheus.io/scrape: true

This is Puppet Hiera data that defines:

A running Container for a Choria related service that passively listens to events and expose metrics to Prometheus
It will watch a file on the host file system and restart the container if the file changes, Puppet manages the file in question
It supports rolling upgrades via Key-Value store updates but defaults to latest
It exposes the port to service discovery with some port specific annotations

This creates a Choria Autonomous Agent that will forever manage the container. If health checks fail the port will not be published to Service Discovery anymore and remediation will kick in etc.

Of course Puppet is an implementation detail - anything that can press the YAML file and place it into Choria can do this. Choria can also download and deploy these automations as a plugin at runtime via securely signed artifacts. So this supports a fully CI/CD driven GitOps like flow that has no Puppet involvement.

To replace the kubectl rollout process I support KV updates like choria kv put HOIST container.tally.tag 0.0.4 (we have Go APIs for this also), the container will listen for this kv update and perform an upgrade. Upgrades support rolling strategies like say 2 at at time out of a cluster of 20 etc. The kv put is the only interaction needed to do this and no active orchestration is needed.

Service discovery is also in a Choria Key-Value bucket and I can query it:

$ sd find
[1] tally @ ve2 [http://192.168.1.10:9010]
   prometheus.io/port: 9010

Or generate Prometheus configurations:

$ sd prometheus
- targets:
    - 192.168.1.10:9010
  labels:
    __meta_choria_cluster_name: ve2
    __meta_choria_ip: 192.168.1.10
    __meta_choria_port: "9010"
    __meta_choria_priority: "1"
    __meta_choria_protocol: http
    __meta_choria_service: tally

So one can imagine how that integrates with Prometheus file based SD or HTTP based. In future I’ll add things to manage ingress configurations automatically etc, and of course Service Discovery -> file flow can ge managed using Autonomous Agents also.

Actual building of containers have not changed much from earlier thoughts about this and the above system - called Hoist - will focus on strengthening those thoughts.

Virtualization

Previously I ran some various mixes of KVM things: I had Puppet code to generate libvirt configuration files, later I got lazy and used the RedHat graphical machines manager - I just don’t change my VMs much to be honest so don’t need a lot of clever things.

As I was looking to run 3 or 4 baremetal machines with VMs on-top I wanted something quite nice with a nice UI and started looking around and found numerous Youtubers going on like crazy about Proxmox. I tried Proxmox for a week on some machines and had some thoughts about this experience.

It is nice with a broad feature set that is quite a good all round product in this space, I can see if done right this will be formidable tool to use and would consider it in future again.

Is seems like this is a company who has engineers, paid to engineer, and they will engineer things all day long. Ditto product owners etc. There’s a lot there and lots of it feels half-baked, awkward or incomplete or in-progress. Combined with there just being A LOT it seemed like a mess. I bet this is a company who just love sprint based work and fully embrace the sprints being quite isolated approach. It shows in obvious ways.

It’s Debian based. I have used EL from their first Beta. Slackware before. SLS before (1993!). In kubernetes you can get away with not caring so much about your hosts but in a virtualized environment you will need to make sure you have ways to manage updates, backups and such of those baremetals. I do not have any tooling specifically built for Debian so I really do not want that lift. I also just do not care even remotely for Debian or its community.

Ultimately I do not need integration with Ceph etc, I don’t need (oh hell no do I not need) a database-driven file system developed by Proxmox and for my needs I do not need SDN. All these things are useful but it seemed like it would tick some of the boxes I listed in Kubernetes conns.

After some looking around at options I came across Cockpit which comes integrated already into EL based distros and while it’s not as full featured as Proxmox I find that to be a feature rather than a shortcoming. It does just the right things and I can easily just not install things I do not want.

I think its firewall management needs a bunch of work still, but that’s ok I do not want to manage firewalls in this manner (more later) so that is just fine, I also do not need its package management - no problem, just uninstall the feature.

Really not much to say or complain about here - and having nothing to say is a huge feature - it makes virtual machines, allow me to edit their configurations, see their consoles etc. Just what I need and no more. One annoying thing with it is that I cannot figure out how to trigger a re-install of a machine, I did not look too deep though.

EDIT: I do have a few things to say about it after all:

It took like 2 minutes to install on an EL machine.
When you are not using it, there is nothing in the process tree. It’s fully based on socket activation. No resources wasted.
It does not take over and camp on everything. It uses the same commands and APIs Puppet/Ansible/You does, you can keep using CLI tools you know. Or progressively learn.
It uses your normal system users via PAM so not much to deploy regarding authentication etc.
It support EL/Debian/Ubuntu and more.

This is huge, it just is super lightweight and gets out of your way and does not prescribe much. It does not invent new things and does not invent new terminology. Huge.

Conclusion

So that is roughly what I am doing for Virtualization and Container management after Kubernetes. Container management is a work in progress so a lot of my components are now just stuff running on VMs but as I progress to improve Hoist a bit I’ll gradually move into it for more things, this has been something I’ve wanted to finish for a long time so I am glad to get the chance.

Further as my Lab is really that after all, a place for R&D, using the area I am focussing on in Choria in anger to solve some real problems has been invaluable and I wanted more of that, this influenced some of these decisions.

binford2k.com: Cranky community members

2024-03-21T00:00:00+00:00

You know the kind. There’s that one person who always has all the opinions. Sometimes there are many of them. But they’re critical about that thing you built, or the workflow you designed, or that issue that you haven’t responded to since they filed it last year. Especially if you hear it long enough, this criticism can cut deeply.

Technical Open Source community members always have something to complain about. Maybe they don’t feel like their work is appreciated. Maybe bugs or pull requests aren’t addressed in a timely manner. Maybe they have opinions about your product’s functionality. Maybe your product is genuinely bad and they’re trying to tell you that.

Whatever the reasons, you’ll soon come to learn that few communities all agree on everything. There is always some amount of dissent and so there will always be someone complaining about something. You’ll also find that corporate and community interests are rarely perfectly aligned. Engaging in a technical community will always require navigating these imperfect alignments.

So what are we to do? Luckily, there are some strategies you can employ to engage with unruly community members, primarily by understanding them better and learning how to make the jabs less hurtful.

Work is not your identity

The most helpful strategy is to emotionally separate yourself from your work and from your company. Chances are that someone else is responsible for your product’s architecture, for its design, for its featureset, its roadmap, its priorities, and so on. Chances are that you are building something that’s largely dictated by others, and that their decision making is driven by business needs and priorities.

These business needs can be driven by all sorts of things. Maybe there’s a market play going on. Or portfolio diversification. Maybe the company is preparing for IPO or an acquisition. Maybe it’s working for a new certification. Perhaps there just aren’t resources to do it the right way and corners had to be cut. Maybe you don’t even know what’s happening behind the scenes.

The key is that it’s unlikely that you’re responsible for any of this, so you shouldn’t own the complaint either. The most you can really do is to ensure that the community member files a ticket to get the feedback in their own words and then raise the ticket to the appropriate party. And don’t feel obligated to defend the underlying business need. Chances are that’s not your job either.

Community members raise concerns because they care

With a few exceptions, people complain because they care and want to help you improve something. They aren’t complaining just to be jerks.

The ones who are complaining to be jerks are usually readily identifiable – often because other community members will call them out on their toxic behaviors. You don’t have to give these people the time of the day. Report them to your community managers, share the code of conduct, or just flat out ignore them.

Spend your quality time with the ones who care and are trying to help. Remember that the words they say aren’t always what they mean. Listen for the problem underneath and remember that people are often reacting out of frustration, especially if it’s an ongoing problem they’re referring to.

Communication styles are hard

And that brings me to my last point. Most of us are engaging in global communities or diverse communities of people from all walks of life. The life story that shaped your communication style is likely not the same as that of the customer support engineer from the deep Bronx and that of the SRE from Berlin or Melbourne and that of the infrastructure architect from Johannesburg and that of the OSS contributor from Atlanta.

We all have different stories and life experiences. Some of us went to school and some learned from the school of hard knocks. Some of us took computer classes as school kids and some of us are busy learning a new profession and how to manage effective blameless code reviews. Some of us were trained on the American shit sandwich feedback style and some of us don’t mince words and just say what we mean. Some of us are neurospicy in the way that makes us speak bluntly and some of us are neurospicy in the way that finds that incredibly offensive.

The point is that what you find to be insulting and snarky is usually not meant that way at all. When the German SRE provides curt feedback that’s direct and to the point, they’re not doing it to be an asshole; they’re doing it out of kindness. Because to them, that’s the most effective way to communicate their needs and help you do your job better. Now it’s on you to choose how you want to hear it.

To recap

If you aren’t personally responsible for intentionally making something bad, then don’t take feedback personally.
Remember that people are providing feedback because they want to help make things better.
You don’t have to choose to hear negativity in the way people communicate. It probably wasn’t meant that way.

Happy open sourcing. Go forth and collaborate!

R.I.Pienaar - www.devco.net: Lab Infra Rebuild Part 1

2024-03-20T08:00:00+00:00

I’ve been posting on socials a bit about rebuilding my lab and some opinions I had on tools, approaches and more. Some people have asked for a way to keep up with my efforts, so I figured it might be time to post here for the first time since 2018!

In this post I’ll focus on what came before, a bit of a recap of my previous setup. Additionally, to a general software refresh I have also been in Malta now 8 years and a lot of my office hardware was purchased around the time of moving here, so we’ll also cover replacing NAS servers and more.

My previous big OS rebuilt was around CentOS 7 days, so that’s about 3 years ago now, high time to revisit some choices.

My infra falls in the following categories:

Development machines: usually 10 or so Virtual Machines that I use mainly in developing Choria.io
Office support equipment: NAS, Printers, Desktops, Laptops etc
Networking equipment: mainly home networking stuff for my locations
Hosting publicly visible items: This blog, DNS, Choria Package repos etc
Management infrastructure: Choria, Puppet, etc
Monitoring infrastructure: Prometheus and friends
Backups: for everything
General all-purpose things like source control etc
My actual office

Below I’ll do a quick run through all the equipment, machines, devices etc. I use regularly. I’ve largely replaced it all and will detail that in the following posts. It’s not huge infra or anything, all told about 20 to 30 instances in 5 or 6 locations.

Development Machines

I’ve had 3 x Intel NUC machines, each with 512GB solid state and 8GB RAM since 2016. They have served me well and really I could see them going for a while longer.

They were each a KVM host and had on them 3 to 8 CentOS VMs. One had some bigger ones as that was my general shell machine and the other were mainly there to add some servers to my node count for Choria development. Choria being inherently a distributed system having some more nodes help.

For my needs they were great but unfortunately they have a hardware problem, their BIOS battery die and it is soldered on to the board so not exactly user serviceable.

Two of these are just retiring - I will remove the SSD and see what it can be used for, more on that later - but otherwise they are done.

One of them is actually a bit newer so it’s finding a new home as a little desktop for my 6 y/o mainly for Scratch and Minecraft etc.

I have been using RedHat since version 0.9 Halloween Beta release then used CentOS for ages - even donated hardware - and after they seemed to be intent on self-destruction I started moving to Alma Linux.

Public Hosting and Management

I host my blog and a few other public bits myself. When I rebuilt all my things 3 years ago I wanted to get some more real Kubernetes experience so I got 3 x 8GB droplets from Digital Ocean with a Kubernetes cluster.

There I hosted the sites, used their managed MySQL, used their Object store and volumes.

It’s been fine generally, I don’t think I like the complexity for what I needed but it was good experience - more on this later. This cluster started with Linode managed KE but it was early days for them in Kubernetes and so I bailed - their support was terrible at the time wrt Kubernetes.

This was surprising as I’ve been a Linode customer since almost their day one and they have been consistently amazing. But when they launched managed Kubernetes it seemed their EU timezone support people were like level 1 only and I always ended up waiting till US time to get things resolved, several times many hour outages. It’s a sad outcome, at the time I moved all I felt comfortable away from them fearing it was a start of a downward slide in quality. Apart from things I am reluctant to move Linode has essentially lost me as a customer.

So I moved to Digital Ocean. It was fine, I have only really one pain with them - they do forced updates of the Kubernetes version (fine) but the way they do it always result in 2 full upgrades and node replacements. This means my outgoing IPs for things like Prometheus kept changing which was extremely annoying.

The Kubernetes infra also ran a 3 node Choria Broker cluster, Prometheus+Alert Manager and Graphite. Alerts to Victorops that I pay for.

Apart from that I had Linode machines for Puppet, Name Servers, Choria Package repos and some general use machines that I should have killed years ago.

I guess the bills for this came to like $400 a month, helps to have a company to bill this to.

Office Equipment

My main office desktop is one of the last Intel iMacs with an old, like 2009 era, Apple Thunderbolt display. I’ve had iMacs since the very first one and they were one of my favourite form factors. It’s a bit sad about this one as it was a really nice machine but Apple is moving fast to stop supporting Intel so now there are no more OS updates.

The screen being ridiculously old and Thunderbolt only - no HDMI - is now just useless.

The current iMacs do look nice but the problem is there is no screen I can put next to them that don’t look like crap and I want either a big monitor or 2, it’s just not working anymore to use iMacs.

I have an old tank Brother printer that is maybe 15 to 20 years old. It just refuses to die but I think its paper feed has now dried up so maybe time to go.

Other than things actually stuck to my office I use a range of new Macbooks - currently a 16 inch M2 Pro with 32GB RAM.

For file storage I use Dropbox of course but QNAP devices for larger storage. I have an old TS-439 that is now 14 years old and still getting security updates (!!!), this is at my office. At home I have a TS-451+. Each provide around 4TB of usable space.

At home I also have a 34 inch ultra-wide display that I put my Laptop on when I sit at the desk here. I got this 7 years ago now so getting a bit old.

Networking Equipment

I have 4 locations I work from often with bits of equipment scattered everywhere. Some time ago I had a mix of networking equipment but I’ve been standardising on Ubiquiti. I am still sad about Apple retiring their Wi-Fi range. Mikrotik was nice enough but I wanted something a bit more polished.

In Malta I have 3 Dream Machines (the round tube one) and around 16 switches, APs, etc. My house is 450 year old with 1.5 meter thick walls so every room gets an AP. I like the single vendor system as all have a single management pane and generally things like VPNs etc. just work.

In Latvia I have a Dream Machine Pro as I have a set of surveillance cameras there, I am considering updating to the Pro at my main house at least also as I’d like to add a camera outside.

In Malta I used to be on Melita for everything, now on Go who have a really great new all fibre to the house network. I have one location left to move and then I’ll have 1Gb links everywhere. In Latvia a 4G link that’s actually quite surprisingly good (190 Mbps down 11 up).

Firewalls tended to be a mix of the Ubiquiti devices and iptables.

General Infra

My mail has been hosted at Fastmail for years and they are amazing. The web ui is a bit of a mess to be honest but at least it doesn’t keep changing. Their mail hosting features are rock solid though.

I used GitHub for everything since they introduced unlimited Private repos. I do pay for GitHub though.

Backups

I have a 30TB Hetzner machine that runs Bacula. It does daily backups of everything I have and does a full 3 month rotation of Full backups with 1 month of Incrementals onto a RAID-10 disk setup.

The main QNAP syncs my Dropbox onto its disks daily.

The main QNAP is set up as 2 x RAID-1 volumes. The main volume is synchronised daily to the office QNAP and I do monthly manual disk rot checks and sync between the RAID-1 volumes. This gives me a 1 month old full backup of the entire QNAP at home and daily off-sites. Monthly I also sync the QNAP to the Hetzner machine.

This means every file hits 9 drives in 3 locations over 2 countries with access to file versions going back 3 months. Ample time to recover from user error like deleting the wrong files and also a lot of redundancy built in.

Physical Office

I’ve had an Office in a town called Mosta here for 4 years now, I have not used it much because parking around it was hell. On paper it was 5 minutes from School and I would stay there while the boy is at School - in practise it could take me 40 minutes to get parking and 12 to drive back home. And when I found parking it could be very far from the office - walking along pavements in Summer is no joke here.

It just never worked, I ended up working from home most of the time. I should have bailed out of it years ago but just never got around to it.

Conclusion

So that about rounds it up, keep reading to hear what literally everything is being replaced with and what new things are being added to the mix to boot!

Part 2 covers Kubernetes, Virtualization and Containers
Part 3 covers Server Management, Puppet and Choria
Part 4 covers My Office and Office Hardware
Part 5 covers VMs, Baremetals and Operating Systems
Part 6 wraps up the series with a look at SaaS and other tools

binford2k.com: Recovering archived Puppet blog posts

2024-02-27T00:00:00+00:00

The Puppet blog has long been a treasure trove of content. You never knew what you might find; a product announcement, an industry analysis, a user interview, a technical post. And it never deleted content, so people got into the habit of linking to blog posts to use as reference or documentation.

This was really great in a lot of ways, but it came with its downsides. Outdated content didn’t always get updated expediently and the amount of content just kept growing so there really wasn’t a good way to manage updates. Only the content that was actively noticed and complained about was updated. So links across the web often pointed to old and outdated content….

🔔 Unfortunately due to how Google indexing works, that really meant that the old, outdated, and often inaccurate content surfaced at the top of search results way too darn often!

During the acquisition, the new marketing team made the decision to declare bankruptcy and start over. They decided to refocus the blog on mostly industry news and product updates and asked the Community and Engineering teams to republish still-relevant content onto the engineering blog. They kept some of the old content that was performing well from an SEO standpoint and still relevant, but archived most of it.

Understandably, this was dismaying for those of us using these posts for documentation! But don’t fret, there is a blog archive located at https://prod-puppet-blog.netlify.app/blog/.

I’ve created a shortcut a method for retrieving pages from the archive if you have the URL.

First drag this link to your bookmarks folder and give it a reasonable name like “retrieve archived Puppet blog posts.”
Then when you see a link that leads to an archived post that you’d like to recover right click and copy it.
Click the bookmark and paste the URL into the dialog you see.

When you click OK, it will take you to directly the archived content. Save the page for your own reference or republish it as long as you respect copyright. 🚨 Please don’t link to the archive, as there’s no guarantee how long it will stay running.

Happy excavating!

(image from https://www.worldhistory.org/image/1353/archaeology/)

betadots: Meine erste Woche bei Betadots

2024-02-19T08:54:41+00:00

2024-02-13

Die erste Woche fing fulminant an mit einem Besuch beim Configuration Management Camp in Belgien in Gent. Ich bin in CCC-Kreisen unterwegs und das CfgMgmtCamp ist den CCC-Veranstaltungen ziemlich ähnlich - haufenweise Leute in ungezwungener Kleidung, an jeder Ecke ein Gesprächspartner, der mehr zu deinem Thema weiß als du selbst. Die Vortragsauswahl war deutlich schwankender, zwei Vorträge mag ich hervorheben:

Der erste Vortrag ist Adam Jacob. Das ist der Erfinder von Chef. Chef wurde vor zwei Jahren (+/-) an Progress verkauft und seitdem ist es still um die Software geworden. Eigentlich lohnen sich die 45min Einführung aus diesem Podcast

https://corecursive.com/configuring-identity-adam-jacob/

ich mag den Podcast mit eigenen Worten zusammenfassen: Auch als Chef weißt du manchmal nicht, wo der Weg hinführt. Du musst mit deiner eigenen Unsicherheit klar kommen und an deine Angestellten kommunizieren.
Wie dem auch sei, Adam hat die Firma verkauft. Danach endet der Podcast. Man hört ja gelegentlich gerüchte von Startupgründern, die sich eine Insel kaufen und sich zur Ruhe setzen, aber Adam hat einen Vortrag gehalten über ein neues Stück Software, an dem er arbeitet.
Der Vortrag hatte es in sich -- und zwar nicht vom Inhalt, sondern vom Präsentationsstil. Der längste Verkaufsmonolog, den ich je gehört habe. Das fing an mit 20 Minuten Smalltalk, wie geil ChatGPT sei, dass wir (das Publikum) das geil finden, dass Adam das geil findet. Der obligatorische SAP-Witz, dass ChatGPT kein SAP installieren kann und dann die Pointe ins Publikum, ob jemand schon mal manuell durch diese Hölle gegangen sei. Und dann der lange Anlauf, er habe eine GUI geschaffen, die nach unten Terraform für Amazon-Web-Services (AWS) und Google-Cloud-Plattform (GCP) schaffen kann.
Das war ein emotivierender Vortrag. Danach hast du das Gefühl "cooler Typ. Los, packen wir es an!"; Inhaltlich bin ich nicht zufrieden, aber auf die Präsentation bin ich neidisch. Wenn ich ein Unternehmen gründen würde, würde ich Adam als Chief Evangelist einstellen. 🥲

Workshop mit Rafael zum eBPF. Bei eBPF muss ich erstmal ausholen wofür man das braucht -- ich hatte mal ein Problem, dass mir ein Prozess eine Logdatei weggelöscht hat. Immer und immer wieder. Ich fand aber den Prozess nicht in Cron und nicht in DBUS und nicht in ...; Wie löst man das Problem? Nun ja, du schreibst ein Kernelmodul "Wenn jemand auf diese Pfadangabe ein Dateihandle erzeugt oder einen Syscall-unlink auslöst, dass logge mir den Prozessnamen und den Prozessowner!"; Weil Kernelmodule kein Zuckerschlecken sind, gibt es dafür Helferchen - zum Beispiel SystemTap. Das ist eine Hilfestellung um Kernelmodule zu schreiben.
Manchmal braucht es aber nicht das ganze Kernelmodul. Es geht einfacher - mit eBPF. Die Abkürzung stand mal für extended-Berkeley-Packet-Filter und will seitdem nicht mehr ausgesprochen werden, weil das Projekt über seine Idee hinausgewachsen ist. Du schreibst ein kleines C-Programm und lädst das in den Kernel. Anschließend rufst du auf: Wann immer jemand den Syscall-Open ausführen will, führe mein Programm aus! Es gibt Syscalls, Tracepoints, Kernel-Probes und Netzwerkpakete. Es ist sehr einfach verglichen mit Kernelmodulen, aber unterschätze den Lernaufwand dennoch nicht!
Der Vortrag begann mit Kubernetes-Architekturen und wie sich Kunden mit Netzwerkdebuggen und Failover und Netzwerk-vs-Georedundanz ins Knie geschossen haben. Schon allein die Probemhinführung hat sich gelohnt um den Vortrag zu besuchen! Dann ging es um die Packet-Filter und die Abstraktionsebene darüber, die Rafaels Firma verkauft um den Netzwerkstack durchdebuggen zu können. Anschließend gab es das Tutorial zum selber durcharbeiten mit Mentor im Raum. Das Tutorial kann ich übrigens empfehlen!

https://isovalent.com/labs/getting-started-with-ebpf/

Abendprogramm: Der erste Tag in einer Bar nahe des Campus. Die Bar war deutlich überfüllt, dafür gab es wieder diese Unmenge schlauer Gesprächspartner. Also haben wir über 5G-Standard gesprochen, über Google-freie Androiden, über Bier brauen.
Der zweite Tag war ein Abendessen mit bekannten Persönlichkeiten in der Welt des Puppet. Die Community hat ein Gesicht und hier sitzt ein wichtiger Teil davon am gleichen Tisch. Abschließend in einer Flipper-Bar. Ich habe seit meiner Kindheit nicht mehr geflippert 😇

Stadt: Gent lohnt allein schon wegen der Architektur. Die beleuchteten Sakralbauwerke bei Nacht am Kanal sind beeindruckend.

Neuer Laptop: Ich habe mir einen Framework 13 als Dienstgerät gewünscht.
Die Do-It-Yourself-Edition kommt als Paket mit mehreren Kleinpaketen. Gehäuse, Mainboard, Deckel und Display sind eine Einheit. Tastatur-und-Unterhälfte-Oberseite kommen als Einheit, RAM, NVMe, Bezel und eine Kiste mit Erweiterungskarten. Der Zusammenbau war leicht - ausgenommen die Festplattenschraube. Die war zu fest angezogen und wir konnten sie im ersten Anlauf nicht lösen. Wir mussten zum Elektronikladen laufen und das kleine iFix-it-Schraubendreherset kaufen. Beim RAM hat es zwei Steckversuche gebraucht, bis der Laptop ihn akzeptiert hat.
Das Gerät ist leicht und schick. Das Bios ist beeindruckend schlank, so etwas habe ich noch nicht gesehen. Software und Setup sind nochmal einen eigenen Artikel wert, da habe ich über Jahre entwickelte Vorstellungen, wie ich mein Arbeitsgerät mag.

Nach dem geglückten Start freue ich mich auf die nächsten Tage. Wenn es eure Nerdseele befriedigt, was ich mit der Laptopsoftware treibe, wäre das nochmal ein eigener Artikel. Bis zum nächsten Mal!

betadots: Scaling Puppet Infrastructure

2024-02-05T09:55:29+00:00

In large environments with many nodes one must take care of Puppet server scaling.
When and if scaling is needed, depends on the number of nodes and on code complexity and size.
This article describes the different ways of tuning Puppet server infrastructure.

Table of content:

Puppet Server tuning
1. Performance tuning a single node
2. Scaling horizontally
PuppetDB tuning
Analyzing performance win
Distributed Puppet agent runs
Summary

Scaling the hard way:

Scaling beyond JRuby limits
Multiple Puppet Server instances

Puppet Server tuning

Performance tuning single node

We sometimes see the usage of horizontal scaling, even when there is no need to do it.
As scaling horizontally needs further infrastructure components like Load Balancer and Puppet Server Compilers, we usually first ask to do performance tuning on the single node instance.

A properly configured and optimized single Puppet server should be able to handle up to 2500 nodes - using default runinterval of 30 minutes.

There are three different possibilities for tuning, which sometimes rely upon each other:

Java Version

Puppet server package has a dependency on java openjdk (headless).
On most Linux distributions this will install Java 1.8.

Please ensure to upgrade to Java 17 and set the java alternative accordingly:

e.g (on Almalinux 8)

alternatives --set java /usr/lib/jvm/java-17-openjdk-17.0.9.0.9-2.el8.x86_64/bin/java

Number of JRuby instances

With its default configuration a Puppet Server spins up 2 JRuby instances.
Each instance is able to handle a single Puppet Agent request.
When there are more requests, these get queued.

Each JRuby instance needs 512 MB of Java Heap RAM.
Just to be sure: it is not possible to run more than 32 JRuby instances on a single node!

Adopting the number of JRuby instances takes place in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf within the jruby-puppet section by setting the max-active-instances parameter:

jruby-puppet: {
    ...
    max-active-instances: 8   # <------- JRuby instances
    ...
}

Please note that increasing the number of JRuby instances causes the Java process to need more Java HEAP RAM.

Puppet Server Java Heap Size

The Java engine should have enough RAM to be able to spin up and maintain the JRuby instances.
If you forget to increase Java Heap size, you will see Java out-of-memory error messages in the Puppet server logfile.

Increasing Java Heapsize takes place in /etc/default/puppetserver (Debian based) or /etc/sysconfig/puppetserver (RedHat based).

The Java Heap size is provided as a Java argument:

# /etc/[sysconfig|default]/puppetserver
JAVA_ARGS="-Xms18g -Xmx18g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger

In this example we have set the Java Heap size (upper and lower limit) to 18 GB RAM.

# Lower limit
-Xms18g
# Upper limit
-Xmx18g

It is considered best practice to set upper and lower limit to the same value, so Java reserves the RAM upon start up.

Please note that the maximum possible value is limited by the amount of the system RAM.
If you increase the Java heap size beyond system RAM, you will find Kernel out-of-memory errors in the journal.

Besides this one must be aware that Java switches memory handling when using more than 32 GB of heap, which reduce the amount of objects. Also see the blog post from Codecentric or Atlassian.

Reserved Code Cache

When Puppet servers receive a request from a node the Puppet Code is loaded into code cache in memory.
The default value is set to 512 MB RAM.

A larger Puppet code base or complex code might need a larger Code cache setting.
Code cache is configured as Java argument in /etc/[sysconfig|default]/puppetserver:

# /etc/[sysconfig|default]/puppetserver
JAVA_ARGS="-Xms18g -Xmx18g -XX:ReservedCodeCacheSize=1024m -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger

In this example we have set the code cache size to 1024 MB RAM.

-XX:ReservedCodeCacheSize=1024m

Please note that the maximum possible value is at 2048m.

Please ensure that the Puppet server process was restarted after setting all the required configuration options.

A short mathematical sizing rule of thumb:

Java Heap size (M) = ( No of JRuby instances * 512M ) + 512M
Reserved Code Cache Size (M) = No of JRuby instances * 128M

Scaling horizontally

If a single Puppet server is not able to handle the amount of requests - e.g. in large infrastructures exceeding 2000 nodes - one has the option to set up additional compilers.

Please note that each Puppet server infrastructure component should receive the performance tuning settings!

Infrastructure setup

A high performance Puppet infrastructure consists of the following components:

CA Server
Compiler(s)
Load balancer

A Puppet agent sends the request to the Load balancer.
The load balancer passes the request to a free compiler.

From a Puppet agent point of view, the request is sent to the Load balancer and the response is received from the compiler.
This has a special meaning when it comes to SSL certificates and strict SSL validation.

We will discuss this when we come to the compiler setup.

Puppet CA Server

The Puppet CA server is a standalone single instance, which is used to spin up and maintain additional Puppet compilers.
From CA server point of view a compiler is just a node.

All new CSR's must be signed on the CA server.
When you want to scale your Puppet infrastructure the CA Server will be your existing Puppet server.

When we want to sign the compilers certificates including dns_alt_names, we must configure the CA instance, to be able to do this by modifying the /etc/puppetlabs/puppetserver/conf.d/ca.conf file:

We must allow subject alt names setting:

# /etc/puppetlabs/puppetserver/conf.d/ca.conf
certificate-authority: {
    # allow CA to sign certificate requests that have subject alternative names.
    allow-subject-alt-names: true  # <----- enable SAN cert signing

    # allow CA to sign certificate requests that have authorization extensions.
    # allow-authorization-extensions: false

    # enable the separate CRL for Puppet infrastructure nodes
    # enable-infra-crl: false
}

Please ensure that the Puppet server process was restarted after doing all the required changes.

Adding compilers

Compilers should not act as CA Server!
The CA functionality is managed in /etc/puppetlabs/puppetserver/services.d/ca.cfg

Now we need to change the settings so that a compiler does not act as CA server, but pass all CA related requests to the CA server:

# To enable the CA service, leave the following line uncommented
# puppetlabs.services.ca.certificate-authority-service/certificate-authority-service
# To disable the CA service, comment out the above line and uncomment the line below
puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service
puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service

Please not that all compilers (and the CA server) should receive the Puppet code. In most environments we see that the compilers and the CA server have a NFS mount on which the code is deployed on the CA server and used on all compilers.

The NFS share is beyond scope of this document.

The Puppet agent will not connect to a compiler but to the load balancer.

If we keep the default setup, the Puppet agent will refuse to connect to the load-balancer, if DNS ALT Names are missing in the compilers certificate.

The compiler MUST have the Load balancer DNS name configured, prior generating the CSR.

This can be achieved by adding the dns_alt_names configuration setting into 7etc/puppetlabs/puppet/puppet.conf into the agent section.

[agent]
dns_alt_names = loadbalancer.domain.tld,compiler-fqdn.domain.tld

Adding load balancer

Each request to a Puppet Server starts a compilation process with different runtimes. One should not configure the roundrobin distribution algorithm on the load balancer.

Instead we want to distribute the connections to the Puppet compiler with the least work to do - which means the one with the least connections. In HAproxy this setting is called leastconn.

Besides this you do not want to rely on Layer 3 IP connection, but on Layer 7 functionality. Within HAproxy one should add the SSL directive to each backend.

PuppetDB tuning

In most environments PuppetDB is used to store facts, reports, catalogs and exported resources.
The more nodes you have in your infrastructure, the higher the load and the memory requirement on the PuppetDB process.

Tuning PuppetDB

PuppetDB is a HTTP(S) Rest API in front of a PostgreSQL database.
One can say that PuppetDB also is a kind of web service.

Java Heap size

The most important setting is Java heap size.
Per default a PuppetDB is configured to use 512MB Heap size.

Configuration takes place in /etc/sysconfig/puppetdb (RedHat/SLES) or 7etc/default/puppetdb (Debian).

The Java Heap size is provided as a Java argument:

# /etc/[sysconfig|default]/puppetdb
JAVA_ARGS="-Xms1g -Xmx1g ...

In this example we have set the Java Heap size (upper and lower limit) to 1 GB RAM.

# Lower limit
-Xms1g
# Upper limit
-Xmx1g

It is considered best practice to set upper and lower limit to the same value, so Java reserves the RAM upon start up.

We usually receive good results with 1GB or 2GB heap site (depending on the number of nodes)

Database connection pool

Within the PuppetDB configuration file, which is located at /etc/puppetlabs/puppetdb/conf.d/database.conf, one can set the maximum numbers of idle and active PostgreSQL connections within the database section:

maximum-pool-size = 25 # default

Please note that PuppetDB uses two connection pools:

read pool
write pool

The read pool can be set individually by setting maximum-pool-size in the read-database section. The default value is 10.

The maximum-pool-size should be set to the sum of both.

Please check your PostgreSQL configuration (especially the max connection setting) prior increasing the connection pool.

Block facts

Another possibility is to lower the number of facts stored.

This can be achieved by setting facts-blocklist.

facts-blocklist = ["fact1", "fact2", "fact3"]

Command processing threads

Besides this PuppetDB lets you configure the number of threads by setting threads within the command-processing section in /etc/puppetlabs/puppetdb/conf.d/config.ini.
As default PuppetDB will use half the number of cores of the system.

Web server threads

Last but least one can set the max-threads in /etc/puppetlabs/puppetdb/conf.d/jetty.ini.
This specifies the number if parallel possible HTTP and HTTPS connections.

Scaling PuppetDB

In former times, it was best practice to have a single PuppetDB instance on the Puppet Server.
If you have the need for scaling Puppet Servers, one should consider having PuppetDB on each Puppet infrastructure node, which connect to a centralized PostgreSQL database.

Analyzing performance win

Every time one does performance tuning, one also wants to get proof that the new setting has improved performance.
The most simple check is by getting the compile times from puppetserver logfile.

awk '/Compiled catalog/ {print $12" "$14}' /var/log/puppetlabs/puppetserver/puppetserver.log

production 0.07
production 0.05
production 0.05
production 0.04
production 0.04
production 0.03
production 0.03

You can also get highest and lowest compile time

awk '/Compiled catalog/ {print $12" "$14}' /var/log/puppetlabs/puppetserver/puppetserver.log | sort | awk 'NR==1; END{print}'

production 0.03
production 0.07

If you need an average compile time the following command will be helpful

awk '/Compiled catalog/ {print $12" "$14}' /var/log/puppetlabs/puppetserver/puppetserver.log | sort | awk '{total += $2; count++ } END { print total/count }'

0.0442857

Another option is provided by the Puppet Operational Dashboards.

This will setup a Grafana frontend which reads data from an InfluxDB, which gets its content from a Telegraf agent which connects via HTTPS to the Puppet server metrics endpoints.

In Puppet Enterprise the PostgreSQL database also uses SSL certificates, which also allows fetching PostgreSQL metrics.
When running Puppet Open Source, one needs to add an additional telegraf configuration to query the PostgreSQL database using a user and password and push the data into InfluxDB.

Distributed Puppet agent runs

Within the introduction we mentioned that a single server can handle up to 2500 nodes.
This number of nodes requires that Puppet agent runs must be evenly distributed over time.

The default way for a Puppet agent is running as a daemon, which will schedule a Puppet agent run upon start and triggers the next runs every 30 minutes (runinterval config option).

Due to the reason that systems might be starting in parallel one will still see overloaded Puppet server infrastructure.
As a result one will recognize that the Puppet agent run takes very long time. If the Agent did not receive a reply from Puppet server within 5 minutes (http_read_timeout config option), it will stop the actual request and repeat again in 30 minutes.

The config options can be modified in puppet.conf config file in section agent.

Please reconsider if Puppet should run every 30 minutes.
In some infrastructure Puppet is running in noop mode as changes may only be deployed during maintenance window.
In this case one still wants the node to regular check its configuration, but maybe 4 times a day only, so people can identify within the reporting that the systems are still in desired state.
Doubling the runinterval option reduces the load on the Puppet server to its half, so the Puppet server can handle double number of nodes.

But one still will see huge and many load spikes on the Puppet server.
It still can happen that to many nodes try to request for their configuration in parallel.

We usually recommend to customers to run the agent via cron.
One run will be done at reboot and the other runs will take place as recurring cron entries.
To distribute the agent runs over time one can make use of the fqdnrand function inside Puppet. This function generates a unique number for a hostame or FQDN.

One can use the reidmv puppet_run_scheduler module to configure Puppet accordingly.

Please note that within Puppet Enterprise environments the Puppet agent on the Puppet servers must be running as agent daemon service!

Summary

Tuning and scaling a Puppet server is very simple, as it is a single Java process and a web service only.

Tuning is mostly related to the Java process and memory handling and is usually limited by the amount of RAM and numbers of CPUs being available.

Scaling can be achieved by using simple, already established solutions to distribute web traffic amongst several servers.

Distributing the Puppet agent runs ensures evenly used Puppet servers which allows more nodes to get managed.

Scaling beyond JRuby limits

We already mentioned that you can not spin up more than 32 JRuby instances.

We tried it. Be sure. It is not working at all.

But what to do if you have high end (e.g. HPC) hardware?

72 cores
248 GB RAM

Usually you want to slice the big hardware e.g using KVM or Docker and run multiple VMs or Puppet server containers for example by using Voxpupuli CRAFTY.

Normally we see such a hardware only within virtualization or container based environments or within a HPC platform.

In our specific use case the usage of KVM, Docker, Podman or Kubernetes was not possible. Don't ask.

We talked about this in the Puppet community Slack channel and people told us that the only option is to spin up multiple Puppet server instances by "some specific symlinking and copying of config files".

Just to be sure: this is not something you can configure out-of-the-box, but needs lots of changes also to Puppet internal scripts.

Multiple Puppet Server instances

You want it the hard way. Here we go..

Please note that this is just first findings, not automated and of course unsupported!

Requirements:

big hardware
multiple IP addresses on the node
- one for the main Puppet CA server
- one for the HAproxy loadbalancer
- and one for each compiler
DNS configured for each IP
Puppet CA server configured and running

Assumptions:

the main installation is used for the Puppet CA server
compilers will use copy of this installation
main CA server FQDN: puppetserver.domain.tld
compiler FQDN: puppetcompiler1.domain.tld
load balancer FQDN: puppetbalancer.domain.tld

First we need to stop the running Puppet Server, then we can copy several directories and make modifications to configuration files and scripts.
Afterwards we can start the Puppet server processes and then we can install and configure HAproxy.

systemctl stop puppetserver

Files and directories needed

Each Puppet server instance needs several files and directories copied from the main installation:

# Service config
cp -pr /etc/sysconfig/puppetserver /etc/sysconfig/puppetservercompiler1
# Puppet agent config
cp -pr /etc/puppetlabs/puppet /etc/puppetlabs/puppetcompiler1
# Puppet server config
cp -pr /etc/puppetlabs/puppetserver /etc/puppetlabs/puppetservercompiler1
# Puppet server application
cp -pr /opt/puppetlabs/server /opt/puppetlabs/servercompiler1
# Puppet server logging
cp -pr /var/log/puppetlabs/puppetserver /var/log/puppetlabs/puppetservercompiler1
# Puppet server service
cp -pr /usr/lib/systemd/system/puppetserver.service /usr/lib/systemd/system/puppetservercompiler1.service

Config files

/etc/puppetlabs/puppet/puppet.conf

The main Puppet configuration from the main Puppet CA Server should only receive an entry for the server:

puppet config set --section main server puppetserver.domain.tld

/etc/sysconfig/puppetservercompiler1

Change the paths:

...
INSTALL_DIR="/opt/puppetlabs/servercompiler1/apps/puppetserver"
CONFIG="/etc/puppetlabs/puppetservercompiler1/conf.d"
# Bootstrap path
BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetservercompiler1/services.d/,/opt/puppetlabs/servercompiler1/apps/puppetserver/config/services.d/"
...

/etc/puppetlabs/puppetservercompiler1/services.d/ca.cfg

Disable CA

# To enable the CA service, leave the following line uncommented
#puppetlabs.services.ca.certificate-authority-service/certificate-authority-service
# To disable the CA service, comment out the above line and uncomment the line below
puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service
puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service

/etc/puppetlabs/puppetservercompiler1/conf.d/global.conf

Adopt path

global: {
    # Path to logback logging configuration file; for more
    # info, see http://logback.qos.ch/manual/configuration.html
    logging-config: /etc/puppetlabs/puppetservercompiler1/logback.xml
}

/etc/puppetlabs/puppetservercompiler1/conf.d/metrics.conf

Adopt the server id

metrics: {
    # a server id that will be used as part of the namespace for metrics produced
    # by this server
    server-id: puppetservercompiler1
    registries: {
        ...

/etc/puppetlabs/puppetservercompiler1/conf.d/puppetserver.conf

Adopt paths

jruby-puppet: {
    ruby-load-path: [/opt/puppetlabs/puppet/lib/ruby/vendor_ruby]

    # Adopt path
    gem-home: /opt/puppetlabs/servercompiler1/data/puppetserver/jruby-gems

    # Adopt paths
    gem-path: [${jruby-puppet.gem-home}, "/opt/puppetlabs/servercompiler1/data/puppetserver/vendored-jruby-gems", "/opt/puppetlabs/puppet/lib/ruby/vendor_gems"]

    # Adopt path
    server-conf-dir: /etc/puppetlabs/puppetcompiler1

    server-code-dir: /etc/puppetlabs/code

    # Adopt path
    server-var-dir: /opt/puppetlabs/servercompiler1/data/puppetserver

    # Adopt path
    server-run-dir: /var/run/puppetlabs/puppetservercompiler1

    # Adopt path
    server-log-dir: /var/log/puppetlabs/puppetservercompiler1

    ...
}
...

/etc/puppetlabs/puppetservercompiler1/conf.d/webserver.conf

Adopt Path and IP and/or Port

webserver: {
    access-log-config: /etc/puppetlabs/puppetservercompiler1/request-logging.xml
    client-auth: want
    ssl-host: 10.110.10.102 # <---- add compiler1 IP and/or
    ssl-port: 8141          # <---- use another port
}

/etc/puppetlabs/puppetservercompiler1/logback.xml

Adopt paths

<configuration scan="true" scanPeriod="60 seconds">
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%t] [%c{2}] %m%n</pattern>
        </encoder>
    </appender>

    <appender name="F1" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!-- TODO: this path should not be hard-coded -->
        <file>/var/log/puppetlabs/puppetservercompiler1/puppetserver.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>/var/log/puppetlabs/puppetservercompiler1/puppetserver-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <!-- each file should be at most 200MB, keep 90 days worth of history, but at most 1GB total-->
            <maxFileSize>200MB</maxFileSize>
            <maxHistory>90</maxHistory>
            <totalSizeCap>1GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%t] [%c{2}] %m%n</pattern>
        </encoder>
    </appender>

    <appender name="STATUS" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/var/log/puppetlabs/puppetservercompiler1/puppetserver-status.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>/var/log/puppetlabs/puppetservercompiler1/puppetserver-status-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <!-- each file should be at most 200MB, keep 90 days worth of history, but at most 1GB total-->
            <maxFileSize>200MB</maxFileSize>
            <maxHistory>90</maxHistory>
            <totalSizeCap>1GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <!-- note that this will only log the JSON message (%m) and a newline (%n)-->
            <pattern>%m%n</pattern>
        </encoder>
    </appender>

    <!-- without additivity="false", the status log messages will be sent to every other appender as well-->
    <logger name="puppetlabs.trapperkeeper.services.status.status-debug-logging" level="debug" additivity="false">
        <appender-ref ref="STATUS"/>
    </logger>

    <logger name="org.eclipse.jetty" level="INFO"/>
    <logger name="org.apache.http" level="INFO"/>
    <logger name="jruby" level="info"/>

    <root level="info">
        <!--<appender-ref ref="STDOUT"/>-->
        <!-- ${logappender} logs to console when running the foreground command -->
        <appender-ref ref="${logappender}"/>
        <appender-ref ref="F1"/>
    </root>
</configuration>

/usr/lib/systemd/system/puppetservercompiler1.service

#
# Local settings can be configured without being overwritten by package upgrades, for example
# if you want to increase puppetserver open-files-limit to 10000,
# you need to increase systemd's LimitNOFILE setting, so create a file named
# "/etc/systemd/system/puppetservercompiler1.service.d/limits.conf" containing:
#   [Service]
#   LimitNOFILE=10000
# You can confirm it worked by running systemctl daemon-reload
# then running systemctl show puppetserver | grep LimitNOFILE
#
[Unit]
Description=puppetserver compiler1 Service
After=syslog.target network.target nss-lookup.target

[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/puppetservercompiler1
User=puppet
TimeoutStartSec=300
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5
PIDFile=/run/puppetlabs/puppetservercompiler1/puppetserver.pid

# https://tickets.puppetlabs.com/browse/EZ-129
# Prior to systemd v228, TasksMax was unset by default, and unlimited. Starting in 228 a default of '512'
# was implemented. This is low enough to cause problems for certain applications. In systemd 231, the
# default was changed to be 15% of the default kernel limit. This explicitly sets TasksMax to 4915,
# which should match the default in systemd 231 and later.
# See https://github.com/systemd/systemd/issues/3211#issuecomment-233676333
TasksMax=4915

#set default privileges to -rw-r-----
UMask=027


ExecReload=/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver reload
ExecStart=/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver start
ExecStop=/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver stop

KillMode=process

SuccessExitStatus=143

StandardOutput=syslog

[Install]
WantedBy=multi-user.target

Scripts to adopt

Unluckily several scripts must be modified, too.

Mostly due to hardcoded paths.

/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver

#!/bin/bash

#set default privileges to -rw-r-----
umask 027

set -a
if [ -r "/etc/default/puppetservercompiler1" ] ; then
    . /etc/default/puppetservercompiler1
elif [ -r "/etc/sysconfig/puppetservercompiler1" ] ; then
    . /etc/sysconfig/puppetservercompiler1
elif [ `uname` == "OpenBSD" ] ; then
    JAVA_BIN=$(javaPathHelper -c puppetserver)
    JAVA_ARGS="-Xms2g -Xmx2g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"
    TK_ARGS=""
    USER="_puppet"
    INSTALL_DIR="/opt/puppetlabs/servercompiler1/apps/puppetserver"
    CONFIG="/etc/puppetlabs/puppetservercompiler1/conf.d"
else
    echo "You seem to be missing some important configuration files; could not find /etc/default/puppetservercompiler1 or /etc/sysconfig/puppetservercompiler1" >&2
    exit 1
fi
...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/foreground

#!/usr/bin/env bash

restartfile="/opt/puppetlabs/servercompiler1/data/puppetserver/restartcounter"
cli_defaults=${INSTALL_DIR}/cli/cli-defaults.sh
...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/reload

#!/usr/bin/env bash
set +e

restartfile="/opt/puppetlabs/servercompiler1/data/puppetserver/restartcounter"
reload_timeout="${RELOAD_TIMEOUT:-120}"
timeout="$reload_timeout"
realname="puppetservercompiler1"

...

initial="$(head -n 1 "$restartfile")"
pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetservercompiler1/conf.d")"
kill -HUP $pid >/dev/null 2>&1

...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/start

#!/usr/bin/env bash
set +e
env

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetservercompiler1/conf.d")"

restartfile="/opt/puppetlabs/servercompiler1/data/puppetserver/restartcounter"
start_timeout="${START_TIMEOUT:-300}"

real_name="puppetservercompiler1"

...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/stop

#!/usr/bin/env bash
set +e

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetservercompiler1/conf.d")"
realname="puppetservercompiler1"

...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/cli-defaults.sh

INSTALL_DIR="/opt/puppetlabs/servercompiler1/apps/puppetserver"

if [ -n "$JRUBY_JAR" ]; then
  echo "Warning: the JRUBY_JAR setting is no longer needed and will be ignored." 1>&2
fi

CLASSPATH="${CLASSPATH}:/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.jar:/opt/puppetlabs/servercompiler1/data/puppetserver/jars/*"

Adoptions to scripts from main installation

As we now have changed the way on how to determine the correct pid, we must also do this adoption at the main puppet server cli commands.

/opt/puppetlabs/server/apps/puppetserver/cli/apps/reload

...

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d")"

...

/opt/puppetlabs/server/apps/puppetserver/cli/apps/start

...

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d")"

...

/opt/puppetlabs/server/apps/puppetserver/cli/apps/stop

...

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d")"

...

Start the stack

systemctl start puppetserver
systemctl start puppetservercompiler1

Happy hacking and success on performance tuning your Puppet server infrastructure.

Martin Alfke
ma@betadots.de

betadots: HDM Foreman integration

2024-01-25T08:55:17+00:00

Analyzing Hiera Data can be very difficult.
HDM - Hiera Data manager is a web UI, which let's you analyze Hiera Data in a user-friendly way.

We now have built and released the HDM Foreman integration, which consists of two packages:

Foreman HDM Smart Proxy plugin: smart_proxy_hdm
Foreman HDM plugin: foreman_hdm

HDM Foreman Smart Proxy

Foreman needs to read data from HDM. HDM offers an API endpoint to allow remote systems to query for data and structure.
The Foreman HDM Smart Proxy plugin needs to know where it can find the HDM API endpoint and how to authorize.

HDM Foreman Plugin

The Foreman HDM plugin is an extension to the node view and adds a new tab which shows node hiera data.
Node data are fetched via HDM Smart Proxy from HDM API endpoint.

Requirements

First you need a HDM - Hiera Data manager installation. The installation can take place on an individual node or a Puppet Server.
HDM needs access to

PuppetDB to read environments, nodes and facts
Puppet Code to read hiera config file and hiera data

Besides this HDM requires to only make use of facts and trusted variables in hiera.yaml file.
Any variable which gets set during catalog compilation can not be evaluated by HDM and HDM will display these hierarchies as unresolvable.

The HDM Foreman integration needs at least Foreman 3.6 or newer. We don't build packages for older versions.
Older versions must use HDM integration from ruybgems.org.

Installation

You must install the packages from the Foreman Plugins repository and initialize the database:

dnf install -y rubygem-foreman_hdm rubygem-smart_proxy_hdm
foreman-rake db:migrate

Next you need to configure the HDM Smart Proxy:

# /etc/foreman-proxy/settings.d/hdm.yml
# HDM Smart Proxy
:enabled: https
:hdm_url: 'http://<HDM IP>:<HDM Port>'
:hdm_user: '<HDM API User Email>'
:hdm_password: '<HDM API User Password>'

HDM User and Password must be taken from an HDM API user!
You can omit these settings if you have disabled HDM user authentication.

Now the smart proxy must be restarted:

systemctl restart foreman-proxy

Screenshots

HDM Data view

Foreman Data View

Happy hacking on Puppet and Hiera data,

Martin

Example 42: Emerging from a year long apnoea

2024-01-19T00:00:00+00:00

The last post on example42’s blog was from December 2022, more than a year ago. I was introducing tp desktop which is a quick and easy way to manage your desktops with Puppet and I was actively working on the new version of Tiny Puppet which was expecting to deliver the huge promise of being able to install EVERY application on EVERY Operating System (this is already happening for years) in EVERY way (OS packages, upstream repo packages, source code, release tarballs, containers…) with a single Puppet module. Works on this release peaked at last Configuration Management Camp and then have been brutally interrupted by new activities we had to deliver for customers and these are the main reasons why also this blog has been silent for so long. 2023 has been a particular year for example42: almost no public activities, but a lot of work for customers, which resulted in the best year ever, commercially speaking, for example42’s mother company Lab42 srl (who said that Puppet is dead?). Still, I think that lack of time is the excuse of the procrastinator, and even if this year has begun as active and busy as the previous one, I want to commit more time to public (and open source, as always) activities on Puppet. So, what’s next? No promises yet, let facts speak for themselves, when they will be ready. For the moment I can anticipate the conferences where I will be present in the next months, see you there: AIDays in Milan, Italy, January 30, 2024 FOSDEM in Brussels, Belgium, February 3-4, 2024 Config Management Camp in Gent, Belgium, February 5-7, 2024 Incontro Devops Italia 2024 in Bologna, Italy, March 15, 2024 KubeCon Europe2024 in Paris, Frances, March 1922, 2024 Google Next’24 in Las Vegas, USA, April 9-11, 2024 At the Config Management Camp I’ll be presenting a talk about Strategies for Puppet code upgrade and refactoring, join me there if you want to know more about how to handle code upgrades and refactoring in a sane way. Keep puppetizing, because if you have to manage a system whose lifecycle is longer than a few weeks, Puppet is still the best way to do it. Alessandro Franceschi

Example 42: Emerging from a year long apnoea

2024-01-19T00:00:00+00:00

Example 42: Emerging from a year long apnoea

2024-01-19T00:00:00+00:00

Example 42: Emerging from a year long apnoea

2024-01-19T00:00:00+00:00

betadots: Puppet container version schema update

2024-01-03T16:08:30+00:00

Community: we hear you!

Several people have asked to please change the version schema on puppetserver-container and puppetdb-container.

Puppet Inc version schema

Prior the repositories have been handed over from Puppet Inc to Puppet Community, the container images were using the Puppet server and PuppetDB versions, which were used inside the container.

<puppet.major>.<puppet.minor>.<puppet.patch>

Example usage:

docker run --name puppet --hostname puppet puppetlabs-puppetserver:7.2.0

Name	Description
puppet.major	Describes the contained major Puppet version (7 or 8)
puppet.minor	Describes the contained minor Puppet version
puppet.patch	Describes the contained patchlevel Puppet version

Initial voxpupuli version schema

The original version schema did not show if changes to the container image have been done.
At Voxpupuli it was decided to use a new version schema:

v<major>.<minor>.<patch>-<puppet release>

Example usage:

docker run --name puppet --hostname puppet ghcr.io/voxpupuli/container-puppetserver:v1.0.0-7

Name	Description
major	Describes the major version of the base container (Ubunutu 22.04) or incompatible changes
minor	Describes new features or refactoring with backward compatibility
patch	Describes if minor changes or bugfixes have been implemented
puppet release	Describes the contained major Puppet release (7 or 8)

At Voxpupuli Crafty we had a discussion on how to improve the version schema.
Additionally we received feedback from customers, that they were unhappy with the new version schema as it was unclear, which exact Puppet version is inside the containers.

New version schema

The new version schema has the following layout:

<puppet.major>.<puppet.minor>.<puppet.patch>-v<container.major>.<container.minor>.<container.patch>

Example usage:

docker run --name puppet --hostname puppet ghcr.io/voxpupuli/container-puppetserver:7.13.0-v1.2.0

Name	Description
puppet.major	Describes the contained major Puppet version (7 or 8)
puppet.minor	Describes the contained minor Puppet version
puppet.patch	Describes the contained patchlevel Puppet version
container.major	Describes the major version of the base container (Ubunutu 22.04) or incompatible changes
container.minor	Describes new features or refactoring with backward compatibility
container.patch	Describes if minor changes or bugfixes have been implemented

Many thanks to the community for the great feedback and ideas for improvement.

Happy hacking on Puppet in containers.

Martin Alfke