The Foreman Blog

Screencast: Locations & Organizations

2013-01-07T20:06:00.001+02:00

One of the most important features that will be released soon as part of Foreman 1.1 is locations & organizations. We will probably do a deep dive (which will get recorded) pretty soon, but I recorded a quick screencast to show how they work. Enjoy!

The volume in the recording is low, but it gets very degraded when I try to increase the gain so make sure to crank up your output.

Parameterized Class Support in Foreman

2012-11-26T14:34:00.000+02:00

We've got Parameterized Class support in Foreman now, and it'll be in the next release! I've recorded a small screencast demonstrating its use:

You can find the mentioned wiki page at wiki:Parameterized_class_support

Debian Packaging, the Transparent way

2012-11-01T20:17:00.000+02:00

So I've been managing the Debian packages for about 8 months now, and every so often I get asked if there's anything people can do to help. I have to answer "Not really" because the way we're building the Debian packages is somewhat arcane.

At least, it was. This blog is to tell you all about how it's now much more open.

Packaging in the open

The first piece of the puzzle starts with our foreman-rpms repo (github.com/theforeman/foreman-rpms)
in this repo we store all the packaging data for the foreman packages, both deb and rpm. In the case of Debian, you'll see a structure like:

debian/<distro>/<type>/<package>

For example, the foreman-proxy nightly package for squeeze is at:

debian/squeeze/nightly/foreman-proxy

Now, the eagle-eyed and quick-witted among you will already have thought "That's quite a lot of code duplication". You're absolutely correct, but this method allows us to package each distro in it's own way, if need be. At the moment, it's all very similar, but it might not stay that way. It also makes it trivial to add in more distributions later (say, quantal or wheezy).

So, now you know where our package data is, you'll want to know; how does it get used? Well, here we turn to the king of automation - Jenkins.

Jenkins

You can find our Jenkins server at ci.theforeman.org. In there, you will find a set of interesting packaging jobs, specifically:

packaging_build_foreman-proxy_matrix
packaging_build_foreman_matrix
packaging_build_foreman-proxy_individual
packaging_build_foreman_individual

There's two types of jobs here. Let's start with the "individual" jobs. These are what they sound like - they take a set of parameters about what package the are building and build the required package(s). For example, if I feed it the options

arch = 64bit
distro = squeeze
type = nightly

Then it checks out the foreman-rpms repo, and changes to the appropriate directory in the repo, according to those options. It then runs 'build.sh' found in the packaging directory, which uses Pbuilder to build the package for the requested settings. At the end of the job (if it built successfully) the package is uploaded to our Debian repository at deb.theforeman.org

The matrix jobs, as you can guess, call the individual jobs many times over - once for all the possible combinations of the options. Thus we can have the matrix job track the Foreman git repo, and rebuild the devel packages whenever a commit is made.

How to get involved

Now that things are more open, it's easy to get involved. Here's a workflow you could
use:

1. Get a build box

You don't have to go through the trouble we did of configuring pbuilder. Just build a
machine with the OS you'd like to run the packages on

2. Install basic packages

You'll need 'build-essential' and 'devscripts' as a minimum (well, on Squeeze
anyway :P)

3. Check out the repo

Fork the foreman-rpms repo and check it out to your buildbox. If you're not trying
to update an existing distro, copy the closest match to a new repo.

4. Build the package

You don't need to use 'build.sh' for this (that's crafted towards pbuilder and other
small automated task we need to do in the final packages). Just run "debuild -us -uc"
and see if it builds. If it doesn't either
a. Install the missing build-dependencies, or
b. Fix the broken packaging

5. Test

You can either test on your buildbox, or for best results, on another machine with
the same OS installed. Hopefully the package will install and work ;)

6. Contribute

Commit your changes to your repo and send us a pull request on foreman-rpms. We'll
test your changes and merge if it looks good.

Conclusion

Hopefully that demystifies our package building architecure a bit. If nothing else,
this blog serves to remind me how we set it all up when I can't remember in six
months time ;)

Epilogue

If anyone is interested in replicating the whole chain, you can get our pbuilder
puppet configuration from our infrstructure repo at github.com/theforeman/foreman-infra

This repo contains the puppet code we use to build the various parts of our
infrastructure, including the Debian build slaves for Jenkins. The only thing you won't
find there is the GPG key to sign the packages, for obvious reasons ;)

Open source infrastructure for the Foreman project.

2012-10-23T14:35:00.000+02:00

Overview

We started working on a new project to create build and test pipelines for releasing code and running tests, and small, but dense infrastructure arose from working on that project. Instead of keeping the Puppet code locked up, we decided specifically to make it public. There are lots of benefits of doing this, but it made sense specifically for us so others can help manage the infrastructure on an unfettered basis. Lots of other benefits come with public Puppet code, too, like thinking more acutely about storing private information in the repositories and reducing the ability of manual changes to cause issues in the test environment.

The tools

Jenkins lives in the center of the infrastructure and is key to the development process; ensuring our tests pass, building packages, and validating our installer. We had been using Travis CI, a great project that is changing the way open source projects test their code. There were a few issues with Travis; Foreman isn't a "standard" Rails application in that it requires some gems with native extensions that were hard to get installed on Travis. Additionally, the ability to run tests on different operating systems is key, and Travis only supports Ubuntu right now. Lastly, it's not possible to run builds without pushing to the repository, which makes testing out new configurations a burden. We wanted to be able to build more complex pipelines like building development packages after tests pass. That's certainly possible on Travis, but Jenkins provides a central place to build flexible jobs and provides facilities like build slaves, which are great for building Debian packages and RPM's and testing across different platforms and databases.

Here's a complete rundown of the different tools we are using right now:

- Puppet & the Foreman (of course)
- Jenkins
- pbuilder
- reprepo
- mock
- Redmine & friends (for theforeman.org)

The implementation

We currently have less than a dozen machines; several builders, a few testing machines, Jenkins (only runs the interface, no builds), and a server that runs the website. The CI infrastructure runs on a physical box with KVM & libvirt for virtualization, provisioned via the Foreman. Servers are grouped via the "hostgroup" feature in Foreman and given the appropriate classes via the ENC. Despite the low number of instances, being able to dynamically provision new builders in a few minutes is endlessly convenient.

What's left?

Sensitive data

There is still a lot of different pieces the need to be addressed as the project grows. One of them is that we don't have an easy way to store sensitive data in the public repository. One option to handle that is GPG encyrption and then storage in the public repo using some variation of this technique, with Foreman as the provider of the parameters. Most of the files that need to remain private, like the private key for accessing slaves, are stored on the Jenkins master; we don't need to distribute them too widely so manual placement works for now.

Contribution

It's really easy to contribute to the infrastructure codebase right now; fork the repo, make your changes, submit a pull request. What's not so easy is getting those changes out onto the puppet master - this is still a manual task that I run each time a change gets made. Ultimately the deployment to the master will probably be handled via Capistrano and Jenkins on a post-receive hook from the Github repository which will completely alleviate this problem. Using Capistrano is also great because it allows easy rollback should a deployment to the master go sideways.

The takeaway

As far as I'm concerned we're just getting started! There is a lot more work to do on both the infrastructure and automation sides. Now that most of the initial work to get the project off the ground is complete, incremental changes will be in order to keep the dependencies of the builders up to date and ensure things running smoothly. Spending the time to think through the architecture and build a system that will work as the community around Foreman grows has absolutely been a worthwhile use of time.

More posts to come soon about the work that the team has been doing on automated package building and repository promotion - keep an eye out!

Foreman 1.0 Demo talk

2012-07-19T09:33:00.002+03:00

Last week I had the pleasure of presenting Foreman 1.0 at redhat offices in Brno, CZ.

It was done during our lunch and learn sessions (people like it becuase you get free pizza or so) which could explain a few of the interesting background noises during the talk.

as usually, demos don't work out as planned, but I decided to leave it all, so you can enjoy for yourself.

this by no means its a complete demo of 1.0 features, but maybe a more general overview and its features in 1.0.

many thanks to inecas and lzap for taking care for the details and capturing the talk on video.

EC2 provisioning using Foreman

2012-05-31T12:09:00.000+03:00

One of foreman goals, is to provide a simple and familiar process to provision systems, regardless of where they are located.

We've now added the ability to provision systems in EC2, alongside with the existing virtualization providers such as RHEVM, libvirt, VMWare etc.

In this blog, I'll try to describe step by step what is required in order to provision a new instance in EC2.

Requirements

You should be using a recent version of foreman, either directly from git, or using the nightly, see for git instruction, or use the debian, redhat or fedora nightly packages.
Have a working foreman server, this should include operating system definitions and unattended mode enabled In addition storeconfigs data must not be stored in the foreman database.
Amazon valid EC2 access and secret keys.
Security Group which allows foreman to SSH to the instance.

Configuring AWS

Click on the more tab, and select Compute Resources.
Compute Resources are services that can generate a host, e.g. VMWare, libvirt, openstack etc.

Click on New Compute Resource and fill in the information about your new compute resource, normally the name should represent something meaningful to you, such as a combination of the ec2 region and the account used.

if everything is entered correctly, you should be able to get back a list of regions and select the region that you would like to deploy to.

Foreman would then automatically create a new set of SSH keypairs, which would be used in order to configure the instance (you may remove them later on).

Then, the next step is to define which images are allowed to use and assign them to Foreman Operation systems / architectures.

Click on the image tab and select New Image.

Since foreman would SSH to the instance (at least for now, we've decided using ssh first, cloud-init later), it is very important that you define the correct user, that is configured on the ami (normally the ubuntu user, or ec2-user) and of course, the ami id.

Foreman is now ready to create your instance, however, in order to automate fully puppet to load upon instance launch, we would need to create a little post script, this is where the provisioning templates comes into play.

Configuring Provisioning Templates

Add or edit a new provisioning template, More => Provisioning Templates => New

Select Finish and paste the following content in

Don't forget to associate the template, (in the association tab) and set a default per OS (in the OS settings)

and then add a the snippets too

etc-hosts

puppet.conf

An important note about UUID's for certnames, if you want to use this feature, please make sure that you enable use_uuid_for_certificates under more => settings, if not, you can simply use <%= @host.name %> for the certname.

additionally, it is not compatible with storeconfigs at this time.

master_bootstrap

and if you want to provision a whole puppet master in EC2, you can use the following snippet to get it up and running

Now if you ask your self how variables like ntp-server get resolved, they are simply Foreman smart vars

Actual instance launch

Goto to the Hosts tab, click on New Host, among other settings, make sure you select your compute resource, image and hardware profile

Primary tab

Operating System Tab

Add caption

Progress Bar

EC2 console

As always, since this is a new feature, any feedback, comments etc are welcomed!

Foreman demonstration and breakdown at NYC puppet usergroup

2012-05-01T17:12:00.000+03:00

A couple of weeks ago, I had the pleasure of visiting NYC puppet user group. Here is a video of the recorded talk, many thanks to Brian Gupta for making it all happen! slides can be found at http://www.slideshare.net/ohadlevy/foreman-presentation-at-nyc-puppet-users

VNC support built in foreman

2012-03-21T16:00:00.003+02:00

Now that oVirt integration is done, it was important to have a way to see the host display:

oVirt Installation - part 1

2012-01-30T22:51:00.001+02:00

In this post, we'll see how to install oVirt. for those of you who ask oVirt?! well, taken from its website:

THE VIRTUAL DATACENTER MANAGEMENT PLATFORM Provision, manage, and monitor hosts, hypervisors, virtual machines, storage, and network

So really, oVirt is a management eco system for virtual machines, fairly similar to v-sphere, but open source and runs on the very popular KVM hypervisor, oVirt is the open source version, of the redhat's RHEV-M.

My motivation for this series of posts, beside promoting alternative and cool open source software is the integration between oVirt and Foreman, so provisioning of vm's would get a whole new meaning when you have a virtual data center managed by oVirt where you could plug in puppet into the loop.

I've tested the oVirt management (oVirts core/backend) on Fedora 16, mostly because RPM's for RHEL clones has not made by the community just yet :)

It is recommended to use something with a minimum of 2GB of RAM, 4GB better, as oVirt runs under jboss, however, it is not a problem to run oVirt on a virtual machine.

So enough about that, its awesome and open source, how do I install it? :)

You should know be able to access the UI via http://<your host>:8080 and login using the credentials above.

In a follow up post I'll describe how to configure oVirt, and how to starts creates new VM's.

Re-provision a host without PXEboot

2012-01-29T20:58:00.000+02:00

So, yet another feature not many people know about Foreman.

it is really easy to reprovision (reinstall, or even to a whole new os) an existing system without PXE.

Foreman supports the following provisioning templates:

provision - kickstart, preseed, jumpstart, yast..
PXELinux - templates for syslinux / pxelinux
gPXE - templates for {g,i}PXE
finish - for OS's that requies a second stage script (debian, solaris)
PXEGrub - templates for GRUB PXE
script - custom entry for what ever you might want to do with foreman templates

All of these templates are available usually at: foreman/unattended/<name>

the default script template that comes with foreman does the following :

gets the kernel/initrd and reconfigure grub, so upon next reboot, you can provision your host.

for example, put your host in build mode, and on your host simply execute

Recovering Lost DHCP reservations

2012-01-16T16:06:00.001+02:00

Someone asked me today on IRC how to recover his dhcp reservations, as he was using foreman to create those, and his dhcp server died.

Since we are slow on delivering #944, here is a short snippet that would get the job done.
This script validates all DHCP records, if one is missing, it would recreate it.

if you are using 0.5 or newer, simply change the shebang line to be something like:
#!/usr/bin/env /usr/share/foreman/script/rails runner -e production

What about conflicting records?

In the case of a conflict, its possible to do remove the conflicting reservations and simply recreate the correct one.. however, you should be sure you can remove the old ones before..

dhcp_record.conflicts(&:destroy)
dhcp_record.create

Foreman CLI is now a gem

2012-01-11T09:23:00.000+02:00

Brian Gupta did a great job with coming up with a CLI for foreman, and just released it as a gem.

To install, simply run:

gem install foremancli
foremancli -h

Usage: foremancli [options] ...
    -d, --debug                      Output more information
    -u, --user USER                  Foreman user
    -p, --pass PASSWORD              Foreman password
    -s, --server URL                 Foreman Server URL
        --json                       JSON output
        --yaml                       YAML output
        --status                     Foreman status
        --custom COLLECTION          Custom COLLECTION string, see: http://theforeman.org/projects/foreman/wiki/API
        --architectures [filter]     Retreive a list of architectures
        --common_parameters [filter] Retreive a list of common_parameters
        --config_templates [filter]  Retreive a list of config_templates
        --domains [filter]           Retreive a list of domains
        --environments [filter]      Retreive a list of environments
        --fact_values [filter]       Retreive a list of fact_values
        --hosts [filter]             Retreive a list of hosts
        --hostgroups [filter]        Retreive a list of hostgroups
        --media [filter]             Retreive a list of media
        --puppetclasses [filter]     Retreive a list of puppetclasses
        --reports [filter]           Retreive a list of reports
        --roles [filter]             Retreive a list of roles
        --settings [filter]          Retreive a list of settings
        --lookup_keys [filter]       Retreive a list of lookup_keys
        --dashboard [filter]         Retreive a list of dashboard
        --operatingsystems [filter]  Retreive a list of operatingsystems
        --subnets [filter]           Retreive a list of subnets
        --ptables [filter]           Retreive a list of ptables
        --users [filter]             Retreive a list of users
        --auth_source_ldaps          Retreive a list of auth_source_ldaps
        --hypervisors                Retreive a list of hypervisors
        --lookup_values              Retreive a list of lookup_values
        --smart_proxies              Retreive a list of smart_proxies
        --statistics                 Retreive a list of statistics
        --usergroups                 Retreive a list of usergroups
        --audits                     Not implemented
        --bookmarks                  Not implemented
    -h, --help                       Show this message

     ENVIRONMENT VARIABLES:

     FOREMAN_SERVER                  Foreman Server URL
     FOREMAN_USER                    Foreman user
     FOREMAN_PASSWORD                Foreman password

     CLI options take precendence over ENVIRONMENT VARIABLES

     FILTERS:

     Please see:
     http://theforeman.org/projects/foreman/wiki/Search_API

     Examples:
     --hosts "domain = domain.com AND class = squid"
     --hosts "domain = domain.com AND facts.architecture = x86_64 AND \
       class = module::class
     --classes "name = squid"
     --domains "name = domain.com"

happy hacking

fosdem 2012

2012-01-08T16:30:00.001+02:00

I've finally wrapped up all my bookings, and I'll be attending both fosdem days and the following the #monitoringsucks hackfest.

It looks like we have a bunch of Foreman users there as well, so I hope we could get together, either during Friday(unless there is a devops dinner) or the following days (I'll get to Brussels during Friday morning).

Anyone has a recommendation of for where/when we can meet?

Hope to see you guys there!

Creating a new host using foreman API

2012-01-01T11:00:00.001+02:00

Using foreman API is fairly simple, in here I'll show an example using curl.

Using this simple script, you could automate your VM/Bare metal provision process + Puppet configuration in one simple step.

Create a new Host

In this example, I've hidden most of the logic in the host group attribute in foreman.
meaning that it already knows the Provisioning and Puppet attributes, but its not a problem to define extend the script not to relay on a given hostgroup, or simply override certain default attributes ( such as memory size, or host operation system etc).

Best way to figure out the additional attributes (besides RTFM) is simply to look at foreman log during the creation POST command.

a typical response from foreman, would include the ipaddress (that was automatically assigned), mac address (that was auto generated if its a vm), etc etc.

Note that in order to use the hostgroup, you would need to know its ID, that's easily done if you simply look at the URL while you edit an existing hostgroup, for example, if your URL is https://foreman/hostgroups/1-base/edit, then the ID is 1.

Delete a Host

Getting foreman search results into your Puppet manifest

2012-01-01T10:42:00.000+02:00

Lets say you want to know all of the hosts your monitoring host need to monitor, or maybe, the hosts to which your database needs to allow access to, traditionally, the solution to this problem was using Puppet storeconfigs.

In this blog post, I mentioned how you could utilize foreman search language to get customized results.

While storeconfigs is a great solution, and if it works for you, by all means, please do keep using it, but in this post I would like to show you how to use Foreman to query for similar data + foreman data as well.

Lets say, we want to allow VPN access only to client hosts which ran puppet in the last week.

You could easily change the search conditions, for example, to get a list of hosts without any puppet failures, simply change the query to status.failed = 0.

we could easily search for conditions based on facts, puppet classes, owner, reports and combination of them.

the output from the puppet function, may include complex data, such as Arrays and Hashes as well, and it depends on the query object used, for example, host lists would mostly be an Array, however, host facts would be a hash, for example:

which you could utilize either in templates or versions of puppet that supports hashes.

Quick start

Install and setup foreman (Foreman puppet modules might be a quick starting point).
If you are not using the official foreman installer, download and put the following file in your modules lib directory, and ensure you are using pluginsync.
adjust the file to point to your foreman server.
use it in your manifest.

Advance Search tips in Foreman

2012-01-01T10:04:00.000+02:00

Foreman uses the wonderful scoped_search gem written by Willem van Bergen and Amos Benari, and while the search auto completer is pretty good, there are cases where it's worth knowing about some of the extra features.

Foreman Search functionality can be used both in the web interface, and via the API, in a follow up blog I'll describe how to use the API and search via puppet directly as an alternative to storeconfigs - UPDATED: here

Search Per Page

First of all, each page gets its own search field, looking at page in the host tab is not the same as in the reports tab, make sure you look at the options the auto completer suggest.

Bookmarks

Foreman supports the ability to make search bookmarks, allows users to quickly jump to predefined search conditions.

Once a bookmark is saved (make sure you save it via the bookmark icon next to the search bar), it would appear as a drop down under the main tab.

We provide a few bookmarks by default, you can try using them already by clicking on the dropdowns in the main tabs, for example under hosts or reports.

Bookmarks can be label as public, this would mean other users would see them as well.

Free text

If you ignore the auto completer, and just type stuff, Foreman would try searching for that attribute as free text across multiple fields.

for example, if you just typed 12 in the hosts page, that would search for all hosts with 12 in their ip address, mac address or maybe name, in general we try to keep the fields we search for to a sane list for performance reasons, if you know you are searching for an ip address, you probably want to type ip ~ 12.

Case sensitivity

When querying using = and != the we are looking for an exact match, the match is case sensitive. When running ~ (like) and !~ (unlike) the match is case insensitive.

Using quotes

In the search syntax white spaces are used as a delimiter. Here are some examples of the way a query will be interpreted:

description ~ “created successfully” => list all notifications that contains “created successfully”
description ~ created successfully => list all notifications that contains “created” and at least one of it’s text fields contains “successfully”
description !~ created successfully => list all notifications that doesn't contains “created” and at least one of it’s text fields contains “successfully”

In the second and third example successfully is an additional term that is interpreted as a free text search

Wildcard ‘_’, ‘%’ and ‘*’

The search terms are translated to SQL queries. ~’ -> LIKE ‘!~’ -> NOT LIKE

LIKE statement supports two wildcards ‘_’ and ‘%’.

’_’ is a wildcard for a single character replacement. For example the search name ~ fo_ will match both ‘foo’ and ‘for’.

The ‘%’ and ‘*’ wildcard will replace any number of characters including 0. For example the search name ~ corp% will match both ‘corp’ and ‘corporation’. The more common ‘*’ wildcard is not an SQL wildcard but can be used in the search.

When the ‘like’ or ‘unlike’ search is processed, a ‘%’ wildcard is added at the beginning and the end of a value if it isn’t there in the first place.

For example: The search name ~ foo will become “name LIKE ‘%foo%’” The search name ~ foo% will become “name LIKE ‘foo%’”

Date-time search query syntax

We can handle many time and date formats, here are some examples of the date and time formats that scoped_search accepts:

“30 minutes ago” , ”1 hour ago” , ”2 hours ago”, Today, Yesterday

“3 weeks ago” , ”1 month ago” , ”6 days ago” , ”July 10,2011″

The date can have different separators, “10-July-2011″ is going to read the same as “10/July/2010″ or “10 July 2011″

Month name could be the full name or a three letter abbreviation, Jan will read the same as January.

Many other formats are also acceptable, however it is not recommended to use ambiguous formats such as “3/4/2011″

The date time operators are ‘=’, ‘<’ and ‘>’ the operators should be read as ‘at’, ‘before’ and ‘after’. This is how the search term interpeted:

The right hand part of a date-time condition is parsed and translated into a specific date-time, “30 minutes ago” is translated to “now – 30 minutes”.

last_report > "2011-07-01 12:57:18 EDT” should be read as created after this time.

Note that in the same way: last_report > “30 minutes ago”, should read “created after 30 minutes ago” not “created more then 30 minutes ago”.

A search query like installed_at = Yesterday is translated into a period query, it will be translated in run time to the respective dates. For example, if running on Jan 1, it could be translated into “(installed_at >= Jan 1,2012 00:00) and (installed_at < Dec 31,2011 00:00)”.

Searching for NULL values

if you are searching for hosts without hostgroups, you could try:
not has hostgroup