The Foreman Blog

Presenting the new Foreman Blog!

2015-11-05T20:33:00.001+02:00

Some while ago, I realized that the entire Foreman website was written in Markdown and stored in GitHub, where any contributor can help to improve it - but not the blog! An area almost entirely composed of text; an area where guest posts would be hugely valuable; an area where fixes to old popular posts would be fantasic; all written on Blogger, where we can't easily get contribution from the community. Sigh.

Well, no longer! Today we're unveiling the new Foreman Blog!

What's new?

In terms of content - very little. The old posts have been imported from Blogger via their XML export tool, and then converted to Markdown via the awesome Pandoc tool. Images were downloaded from blogger too, and Gists loaded via the jekyll-gist gem which was already part of the website stack. However, the vast majority of the text is unchanged.

The URL has changed though - it's now http://theforeman.org/blog

What's changed?

We've tried pretty hard to be consistent. Obviously the style of the pages now comes under the general styling of the overall Foreman website, so that's a plus. Later this week we'll redirect blog.theforeman.org to theforeman.org so that the permalinks stored in search engines will continue to work.

There is an RSS feed, of course. You can get it at /feed.xml and there's an RSS icon at the top of the blog index. Please do update your RSS readers if you're using the old Blogger feed. Post comments appear to be present and correct, so there should be no changes or continuity loss from Blogger.

Please do update your bookmarks and feeds, as this will be the last post sent to the Blogger platform.

Great! How can I help?

As part of my role as the new Community Lead, I'll definitely be posting more about all sorts of Foreman related things, as well as my own activities to keep the community informed. In addition, the main goal for changing to Markdown was to enable community contribution, so ...

If you see any bugs or problems with any old blog posts, please do let us know in the issue tracker or even better, open a Pull Request to fix the problem (see Contributing for a guide on that). You can find the raw text of the posts here.

If you want to rework an old blog post for a newer version of Foreman, please feel free to do so - many of the older blog posts (such as the one on EC2 provisioning) are very popular, but were written for much older versions of Foreman. The community will no doubt make good use of updated versions. PRs are very much welcome.

I'd also like to encourage guest authorship on the blog. If you have a blog post of your own that you'd like to reshare on the Foreman blog as part of this, please do open a PR (or send me the text of the blog if you're not into Git).

Above all enjoy the new blog, and let us know about what you like/don't like in the comments on the new blog.

Foreman Community Newsletter - September 2015

2015-09-15T13:01:00.000+03:00

Foreman 1.9 and 1.9.1 released

Foreman 1.9.0 was published on August 18th followed by a bugfix 1.9.1 release on September 11th.

New features in this version include:

A few small web UI enahncements such as new search-enabled dropdown menus throughout the application and full screen editor for parameters
Template editing forms have now capability to preview edits on the rendered template.
Foreman's logging has become more granular so it's now easier to enable debugging of specific subsystems.
Two new email notifications for listing recent audit activities and for notifying when a host has built.
Libvirt has gained a new capability for using cloud-init when image provisioning.

The full list of the changes is available in release notes for both Foreman 1.9 and Foreman 1.9.1:
http://theforeman.org/manuals/1.9/index.html#Releasenotesfor1.9
http://theforeman.org/manuals/1.9/index.html#Releasenotesfor1.9.1

Foreman 1.8 bugfix release

Foreman 1.8.3 has been released on August 19th with half a dozen bug fixes. All components have been updated.

Full release notes are on the website here:
http://theforeman.org/manuals/1.8/index.html#Releasenotesfor1.8.3

This will probably be the last 1.8 series release due to the release of 1.9.0, unless there is a severe security issue or a significant number of regressions get fixed.

Katello releases

Katello project released two new versions this month. General availability of Katello 2.3 (Berliner Weisse) has been announced on August 21st. Visit the release notes page to see the full list of changes:
http://www.katello.org/docs/2.3/release_notes/release_notes.html

There was also a bugfix release of Katello 2.2.4 that includes several patches:
http://www.katello.org/docs/2.2/release_notes/release_notes.html

Remote execution plugin released

The Foreman Remote Execution Team has released a first version of the plugin that allows batch execution of scripts on multiple hosts. A functionality demo of the first version is linked in the Media section.

Plugin releases this month

New media

Foreman community demo (held on Thursday, August 27th)
Foreman remote execution plugin deep-dive
Foreman deployments plugin pre-release demo

Foreman Community Newsletter - August 2015

2015-08-09T18:31:00.002+03:00

Foreman celebrated its 6 year anniversary

The first commit was on July 13, 2009. To celebrate, we got some new T-shirts.

Foreman 1.9 RC3 is out, 1.9.0 and 1.8.3 expected soon

Several blocker bugs found in the 2 previous release candidates have delayed the release of 1.9.0, which is expected to be out around August 17th. Read the announcement or the release notes for more details.
Foreman 1.8.3, a bug fix release, will also be released at approximately the same time.
We appreciate your help testing the release candidates and reporting bugs!

Katello 2.3 RC2 is ready for testing

Katello 2.3 is planned to be released in alignment with Foreman 1.9. Check out the announcement, or the release notes for a full list of updates.
Please help us with testing this RC so the final release will be as stable as possible!

Hammer 0.3 released

This release adds a notification for deprecated options and several bug fixes, and is compatible with Foreman 1.9.

New and updated plugins this month

New smart_proxy_dns_route53 1.0.0 released July 10. This smart proxy plugin allows foreman to manage records in Amazon's Route53 service.
foreman_discovery 3.0.0 released July 28, with 4.0.0 planned for release soon.
foreman-tasks 0.7.2 released July 31.
foreman_chef 0.1.6 released July 15.
foreman-xen 0.1.2 released July 31.
foreman_hooks 0.3.8 released July 30.
foreman-docker 1.4.0 released July 23.
foreman_templates 2.0.0 released July 6.

New media and blogs

This month’s community demo was held Thursday, August 6th.
Daniel gave a talk about Foreman at PuppetNYC.
Shlomi blogged about running OpenLDAP in a container for testing foreman LDAP functionality.
Arnold Bechtoldt wrote a blog post about using Foreman to deploy Windows Server on vSphere.

Upcoming events

Julius Bloch will be giving a lecture about Foreman at FrOSCon (in German) on August 23rd.
Devops Days Tel Aviv will be at October 7-8, call for papers is open until September 1st.
The dates for FOSDEM and Config Management Camp have been announced, block your calendars January 30 - February 2 , 2016 and look out for Foreman-related talks and workshops.

You can follow us on Twitter: @ForemanProject. If you tweet about foreman, please use the hashtag #theforeman so we can see it!

We also have a Facebook group which you can join for updates & discussion.

Foreman Community Newsletter - June 2015

2015-07-09T13:52:00.001+03:00

Foreman 1.8.2 bug fix release

A new bug fix release has been published on June 16th. Mostly fixes bugs around network interfaces
for unmanaged hosts but also other areas were touched including:

prevented extra tokens creation with Foreman Discovery
missing VMware SCSI controller field restored
fact uploads from Windows hosts fixed
errors viewing dashboard/reports list in multi-org mode fixed

For more information see the official announcement

Katello 2.2.1 and 2.2.2 bug fix releases

Two new bug fix versions of Katello have been released in June. List of fixes can be found in Katello documentation, For upgrade instructions see 2.2.1 and 2.2.2 announcements.

Foreman 1.9 branched out, RC1 ready for testing

We started preparations for upcoming 1.9 release. A develop branch was cloned to 1.9-stable and we have
already released RC1. The most interesting features follows:

new select boxes with inline searching, see video and Shlomi's blog post
fullscreen editing for parameters
partition tables auditing, scoping per organizations and locations, file uploads and editor improvements, first presentation in community demo
logging improvements including more details in messages, configurable output (e.g. syslog) and more
more email notifications
cloud-init support on libvirt
copying fact values, short video demonstration here

We appreciate help with RC testing. More information about how to install and report bugs in the announcement on mailing list

At the same time, Katello 2.3 has been branched out so the release can be aligned.

New community-template branches

Community templates repository has been migrated to follow the same branch schema as other repositories. Master branch was renamed to develop and we have stable branches for 1.7, 1.8, 1.9. If you use foreman_templates plugin for provisioning templates synchronization, you can use branch option to specify correct version.

Plugin releases

foreman_chef 0.1.4 and smart_proxy_chef 0.1.4 bug fixes released
foreman_salt 3.0.0 and smart_proxy_salt has reached 2.1.3 adding support salt 2015.05 and Ruby 1.8.7 compatibility
new foreman_bootdisk 6.0.0. release is compatible with Foreman 1.9+
foreman-xen 0.1.1 released restoring compatibility with Foreman 1.8

Quick updates

Foreman media page has been updated with new videos (Lifecycle Management with Foreman, Puppetcamp 2015: selling management and developers on puppet)
New version of Kafo (0.6.11) has been released, fixing validations issues
Another case study recorded and published by Daniel, this time with a consulting company Inuits
Shlomi has published introduction to OpenSCAP plugin on his blog
Daniel's blog about deploying CoreOS cluster using Foreman
You can follow us on twitter using @ForemanProject and #theforeman

CoreOS cluster deployments with Foreman

2015-06-11T17:05:00.004+03:00

This is an authorized repost of CoreOS cluster deployments with Foreman

As Major Hayden mentioned more than a year ago, deploying CoreOS is a bit of a different beast than deploying other operating systems. In this case, we are going to do it by PXE booting the image, then applying a cloud-config script which will will set the SSH keys, core user password, CoreOS version, and register in etcd. We are going to pass parameters that will set these options. That way we can define a host group, with certain parameters, such as the authorized keys, etcd discovery url, and virtual disk. This will simplify booting hosts in our deployment so that creating a new CoreOS node in the cluster will be reduced to three clicks, New Host -> Hostgroup: CoreOS cluster > Submit. As in my previous tutorial for unattended Atomic deployments, I will assume you have Foreman installed, and a PXE Smart Proxy in the network (or networks) you want to launch your cluster. If not, please go to theforeman.org and get a default installation. In my opinion, Libvirt is the easiest way to get this PXE "enabled" network. The PXE templates are already in community-templates , make sure to add the snippet too. You can create them manually by going to Hosts > Provisioning Templates > New template. However, it is much easier to install the foreman_templates plugin, then run:

foreman-rake templates:sync

And you'll get all of the templates in the community-templates repository. Create a new operating system with the following options, in this case it will use CoreOS 647.0.0 from the stable channel.

Time to create the host group. Go to Configure > Host groups > New host group, and create a group with the following parameters. The network must be the one you can PXE boot on, and you can add a parameter ssh_authorized_keys, value should be your public ssh key, usually located in .ssh/id_rsa.pub. I did not add it here as I have a global parameter ssh_authorized_keys with that value. Get a discovery code by going to discovery.etcd.io/new , and put the value you got in etcd_discovery_url.

That should be enough for the group. Now create a new host in that cluster, and as soon as it boots, it will connect to etcd. Remember the URL we used for discovery? Go to that URL and you should see all hosts that have registered in the cluster.

Enjoy it, and please point out any mistakes on the comments section, or let me know on Twitter.

Unattended deployments of Fedora and RHEL Atomic with Foreman

2015-06-11T17:01:00.004+03:00

This blog post is an authorized repost of Unattended deployments of Fedora and RHEL Atomic with Foreman

Project Atomic is a new initiative to have a family of well-known, enterprise-tested operating systems ready for massive container deployments. Atomic operating systems focus on:

Minimal size
Immutable
Easy to update and rollback
Container cluster and runtime provided (currently via Docker and Kubernetes)

It comes with a set of tools. ostree, fleet, kubectl to manage your OS updates, network configurations and cluster health. As I manage my virtual machines using Foreman, and I used to spend some time developing the Docker plugin, the project piqued my interest, as it helps me work with containers more efficiently. For this deployment, I will assume you have a Foreman host with a Smart Proxy with at least TFTP. I will be using domains in the examples, but you could use IPs instead. I think it could be possible to skip the TFTP part for PXE too, but I have not got that far yet. You need a subnet in which you can PXE boot hosts, an example of such a subnet on Libvirt can be found on Dominic Cleal's blog. Step one, download Fedora 22 Atomic iso or RHEL 7 (installer) Atomic iso in your Foreman host. For the Fedora case, it'll be possible to fetch the content straight from the repo through ostreeupdate , however for the moment we will need the image to get vmlinux, initrd, and a few other files.

# wget https://dl.fedoraproject.org/pub/alt/stage/current/Cloud_Atomic/x86_64/iso/Fedora-Cloud_Atomic-x86_64-22.iso -O fedora-atomic.iso

Mount these images in a public directory so that they can be reached from the virtual machine. To keep any existent vmlinux available for non-atomic hosts, we will copy and rename it to vmlinuz_atomic from the mounted iso. By default /var/www/html/pub/atomic would work:

# mkdir /var/www/html/pub/atomic
# mount -o loop fedora-atomic.iso /var/www/html/pub/atomic/
# cp /var/www/html/pub/atomic/isolinux/vmlinuz /var/lib/tftpboot/vmlinuz_atomic

We will need now an installation medium in Foreman pointing to this location. Go to Host > Installation media and create a mirror:

Create the operating system. We will go back to this operating system to associate it with the appropriate partition table and templates afterwards. For the moment, just make sure you choose the right major version 7 for RHEL, 22 in the case of Fedora, as these are the only Atomic ones. Go to Hosts > Operating systems and click on New operating system.

Create a new partition table in Foreman to provide an initial /boot and / in the Atomic virtual machine. Go to Hosts > Partition tables, and click on New partition table.

zerombr
clearpart --all --initlabel
part /boot --size=300 --fstype="ext4"
part pv.01 --grow
volgroup atomicos pv.01
logvol / --size=3000 --fstype="xfs" --name=root --vgname=atomicos

Head to /config_templates (or Hosts > Provisioning templates) and create a new PXE template. Choose a name, then go click on the type tab and select PXELinux. Associate the template with the operating system you created previously, and use this as the content:

DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
kernel vmlinuz_atomic
append initrd=<%= @host.medium.path %>isolinux/initrd.img repo=<%= @host.medium.path %> ks=<%= foreman_url('provision')%> ks.device=bootif network ks.sendmac
IPAPPEND 2

Stay on provisioning templates, and create the kickstart. To do so, click on New Template, choose a type 'provision', associate it with the operating system you created previously, and add this code in the editor:

lang <%= @host.params['lang'] || 'en_US.UTF-8' %>
keyboard <%= @host.params['keyboard'] || 'us' %>
timezone --utc <%= @host.params['time-zone'] || 'UTC' %>

# Partition table should create /boot and a volume atomicos
<% if @dynamic -%>
%include /tmp/diskpart.cfg
<% else -%>
<%= @host.diskLayout %>
<% end -%>


bootloader --timeout=3
<% if @host.operatingsystem.name =~ /.*Fedora.*/ -%>
ostreesetup --nogpg --osname=fedora-atomic --remote=fedora-atomic --url=<%= @host.medium.path %>/content/repo/ --ref=fedora-atomic/f<%= @host.os.major %>/<%= @host.architecture %>/docker-host
<% else -%>
ostreesetup --nogpg --osname=rhel-atomic-host --remote=rhel-atomic-host --url=file:///install/ostree --ref=rhel-atomic-host/<%= @host.os.major %>/<%= @host.architecture %>/standard
<% end -%>
services --disabled cloud-init,cloud-config,cloud-final,cloud-init-local
rootpw --iscrypted <%= root_pass %>

reboot

%post
(
# Report success back to Foreman
curl -s -o /dev/null --insecure <%= foreman_url %>
) 2>&1 | tee /mnt/sysimage/root/install.post.log

exit 0

%end

This template will pull the content from the --ref you specify, and the URL would be $FOREMANSERVER/pub/atomic/content/repo/. If you visit this URL, you should be able to find the docker-host file at the end of the hierarchy, specifically heads/fedora-atomic/f22/x86_64/docker-host. If you cannot find docker-host there because you're using this tutorial for Fedora 23 and it has changed, I would recommend you to peruse your /pub/atomic folder and find the correct URL. Currently this is the structure for RHEL 7 and Fedora 22. It's time to associate these templates with the operating system, and deploy the host. Go back to Hosts > Operating systems, and click on your Atomic operating system to associate the templates:

We're all set. Time to deploy the host. I'm doing this in Libvirt as it's where my PXE network is configured. Go to Hosts > New host and choose a name for your Atomic host. The Puppet options are irrelevant, as Puppet is not able to modify the Atomic ostree for the moment. Select the right domain and subnet, to ensure you're booting in a PXE-enabled network. Something I found is that Anaconda tends to get stuck when I have tried to provision Atomic systems with less than 1GB of RAM, so I would recommend to assign that amount of RAM to your Atomic host at least. The operating systems tab should look similar to this. Remember this is network-based provisioning.

If everything went well, your system will PXE boot and start Anaconda right away. If you have VNC access to the machine, it will look similar to this:

After Anaconda finishes the installation, you should be able to SSH into that machine using the root password you provided in Foreman.

We're done! I suggest you use this host as a Kubernetes master, minion, or as a Docker host. For now, I will investigate how to pass the proper parameters through Foreman to provide Atomic/Kubernetes cluster provisioning.

Foreman community survey results

2015-02-23T13:57:00.000+02:00

February has been an intense month for the Foreman team. We gave a few talks at FOSDEM, got in touch with the community at cfgmgmtcamp, and overall had a great time.

After all of these events - we will go over them in the February newsletter - we are now ready to discuss what the results of the community survey have been. First and foremost, THANK YOU to everyone that replied. We are incredibly thrilled to hear how all of you are using the tool.

The reason why we launched the survey was essentially to understand what features should we focus on and what are we missing. I think the results will be instrumental to Foreman's development in the future. We had a total of 118 responses.

Let's discuss some insights:

Half of our user base have been using Foreman for one year or less. It surprised me so many people joined us during the last year, and expected most users to have been using Foreman for longer. Hopefully we can keep up launching new features so that we this ratio remains similar in 2016.

1.7.x was around for about one month and a half at the time the survey was released. We support the two latest minor versions (1.7 and 1.6 in this case) so 86% of the installations are officially supported. If you are using 1.5 or older versions, please take into account your installation security might be compromised and upgrade. Tell us on #theforeman or #theforeman-dev (IRC channels on Freenode) if there is any particular bug or feature in new releases that holds you back.

The vast majority of installations (86%) are under 1000 nodes. We love feedback from our biggest users as they have unusual performance problems that lead to improvements for both small and large installations. The needs of most users are covered by less than 1000 nodes, which Foreman can deal with very well. Larger installations (100k nodes even!) are not unheard of, and JJ (John Jawed) from eBay/PayPal talked about his experiences with Foreman and Puppet at Puppetconf 2014 managing a very large deployment.

I would have expected a larger proportion of Debian/Ubuntu users, we devote some time to its packaging and testing. I think this graph does show an over representation of users of Red Hat based servers, partly because the survey was initially sent to a series of employees at Red Hat.

This question helps us to prioritize features. For instance, RBAC is a important concern for installations with more than 10 users, however they make up 20% of all installations.

More or less expected, although usage of Hammer is a bit low (< 50%). Our guess is that we might not publicize as much as it deserves. If you have been looking for a good Foreman CLI, you can find instructions about how to install it in the Github repository . It also supports Katello through hammer-cli-katello and has a pluggable interface so extensions can be easily written for any plugin.
API v1 is still used by 12% of the respondents, we would like to remind you v1 is deprecated and urge you to update your scripts to use v2 which is considered stable now. If there is any feature you are missing, please tell us in the mailing list or #theforeman-dev on IRC.

This graph was the most surprising of all. I would have expected Openstack or EC2 to be number one, but they are not even in the top 3. Perhaps the lack of support of advanced features, or volumes are responsible in this case. Similarly, VMware support is not very good yet most of our users with compute resources use it. GCE is completely broken due to changes in the Foreman API, and we did not catch it. Docker's case is particularly interesting as it surpasses Openstack in usage, even though it is not useful to deploy clusters yet. We expect usage can only increase.

Most people were interested about a meet up in their area. It would be interesting to know in particular which areas would have a large enough group of users to organize an event. If you would be interested, please send an email to the foreman-users mailing list.

The graphs seem to support that people are happy with their config management tools. However the configuration of a whole infrastructure remains an unsolved problem. Results for the multi host deployment interest make us confident when exploring this area. Petr Chalupa explained at cfgmgmtcamp '15 some plans that Foreman will follow in this area.

By far the most requested feature. The question is unintentionally vague, but what I had in mind when I wrote the question was mainly scheduling of Puppet runs / Salt highstate, improving what we already have, getting a console, and perhaps adding hooks to make Foreman a gateway for the dbus API.

This one graph is a bit skewed, as most users who already use Foreman are already happy with their current configuration management solution. However, we do realize one of the main drivers for new users would be to stop being as Puppet focused as we are now and integrate better with other solutions.

Most users are not managing installations where containers are important yet. However the industry has its eyes set on containers for the future, so it's not a surprise people would like to have it in Foreman already to fiddle with it.

OpenSCAP is a new feature in Foreman, and most users will be unfamiliar with it. Governmental procedures for IT security are often more standardized than at private companies, however we recommend migrating your compliance rules to OpenSCAP to get better security compliance at your data center and manage it through Foreman.

Improved networking has continually been high on the list of requests for future Foreman releases. We're happy to report that 1.8 will feature a completely redesigned UI in the Network tab, allowing use of separate provisioning networks, bonds, aliases and so on. We've also merged the creation of "physical" devices on virtual machines into this UI, removing the duplication of workflow on the VM tab. Please do give us feedback on this new UI and see the list of things we still want to implement at ticket #2509

As expected, improvements to features we already have are very favoured by the community. However we remain open to new, different contributions that allow people from other communities to start using Foreman.

Here is a distribution of the percentage of users that use each Foreman plugin. We only asked for plugins in the official list in the wiki , so if you have created a plugin and you cannot find it there, please add it. As expected very generic plugins such as discovery and default_hostgroup are very widely used, and more specific features like Xen, Salt, are less used. The Salt plugin has already overtaken Chef, and Foreman in the future should be able to easily assimilate more configuration management tools such as Ansible, and remove the Puppet dependencies if you just want to use it as an inventory.

To finish, I would like to list some features you asked for in the survey very frequently. We will take these seriously for future releases, as 1.8 is almost out of the door:

Multi-factor authentication, required for government deployments.
Hyper-V as a compute resource - most likely this can be implemented as a plugin
Dashboard customization (on the making at #8106)
Provisioning in VMware data store clusters.
Better documentation, especially for plugins and the CLI.
Up to date screencasts
Determining what Puppet classes are causing failed Puppet runs in the whole deployment, dashboard is not enough.
Ansible support and less dependency on Puppet
Support to deploy in Docker clusters and not just individual containers
IPv6 support.

Thanks again for responding to the survey, our community drives Foreman success!

Foreman Community Newsletter - January 2015

2015-01-30T14:38:00.002+02:00

Foreman 1.7.2 released

Foreman 1.7.2 has been released at the end of January. This is a bugfix release, the full list of fixes can be found in the release manual.

Katello 2.1 release is approaching

The release is expected any day now. Keep your eyes on the project website at http://katello.org for details.

New and updated plugins this month

foreman_docker – enables integration with Docker, version 1.0 released .
foreman_discovery – enables hardware discovery, version 2.0RC1 released.
foreman_chef – provides integration with chef, updated to version 0.1.1.
foreman_salt – salt support, updated to version 1.1.1.
foreman_abrt – handling of ABRT reports, updated to version 0.0.6.

New media

The Foreman team is collecting data about our user base. You can help by filling out community usage survey.
We had our second joint demo of Foreman and Katello teams in January. You can can watch the recording of the demo here.

Foreman Community Newsletter - December 2014

2014-12-23T17:50:00.003+02:00

Foreman 1.7.0 and 1.7.1 released

A new version of the Foreman went out in early December, followed by a bugfix release.
Headline features of the releases are:

enhancements to smart class parameters, providing better control over default values and merging of values across matchers
a new build review page that should help you check for common errors when switching a host into build mode
support for specifying bonds, VLANs and aliased interfaces on a host-importing interfaces them from facts
two existing features have been moved into plugins (foreman_host_rundeck, smart_proxy_chef) and have their separate lifecycle now

Detailed information about the changes in the releases is available in our Foreman 1.7.0 and Foreman 1.7.1 release notes.

This release also marks the end of Debian 6 (Squeeze) support, and Ruby 1.8.7 in the Foreman application (not the proxy or installer).

Thanks to all our contributors!

Hammer 0.1.4 released

After a long period of silence new versions of Hammer CLI and Hammer CLI Foreman were released. They bring new commands for managing external user groups and LDAP auth sources, YAML and JSON exports and enable for using names when associating multiple resources at once. There are also new translations available.
Read the release notes for further details: hammer_cli, hammer_cli_foreman

Katello 2.1 RC1 released

The Katello team announced a release candidate of Katello 2.1. Users may test out the new features in the next release. Katello 2.1 is a plugin to Foreman 1.7. Upgrades from previous versions of Katello have not been tested but an upgrade procedure to 2.1 should be available soon.

For more information, please be sure to visit the project's website http://www.katello.org

Features in development

Networking

Bunch of networking improvements is approaching merge. This change brings more flexibility into specifying primary and provisioning interfaces. The UI has been simplified and split up into a NICs overview table and modal windows showing details.

Multihost deployment

Work on multihost deployment continue. December was in the name of defining a data model. The current state is thoroughly described in a deepdive on this topic.

Moving to Rails 4

We've started with first steps towards moving to Rails 4 this month. Stay tuned for more information.

New plugins this month

foreman_digitalocean - enables provisioning and managing of DigitalOcean droplets in Foreman
foreman_host_overview - extends the properties table for a host to include VNC & RDP links, as well as the host's certificate name
foreman_host_rundeck - creates a yaml representation of rundeck on hosts

New media

Multihost deployments data model deepdive

Foreman Community Newsletter - November 2014

2014-12-02T01:21:00.000+02:00

Foreman 1.7.0-RC2 released

Second release candidate for Foreman 1.7 was released. It fixes a number of issues found by folks testing RC1 and will likely become the 1.7.0 release within a week.

Much more information about the changes in Foreman 1.7 is available in
our release notes.

Foreman 1.6.3 released

Foreman 1.6.3 has been released to fix a handful of bugs. Check the release notes for full list. This will probably be the last update of the 1.6 series.

Foreman Discovery Image 2.0 released

New major version of the Discovery Image was released. A minimized version of CentOS 7.0 is used to PXEBoot unknown bare-metal hosts and registering them in Foreman via Discovery plugin. The image does support both Foreman 1.6 and upcoming 1.7.

To download 2.0 series follow the link.For upgrade instructions see the announcement on the list.

Features in progress

A lot of effort was put into design and development of new features. The following lines are partly links to reports of current status and heads up of what you can look for in near future and partly invitationto sharing your ideas, feedback and testing.

Networking

Recently, we've been working on Foreman networking refactoring. If you want to know more about upcomming changes in networking setup of the host, how to set primary and provisioning interfaces and what improvements are prepared for NIC objects on bare metals see the detailed summary on the list. Also don't miss the networking unification deepdive that will guide you through the changes.

Multi-host orchestration

is in early stage of development, which means you can help it a lot to become awesome. If you are interested in this topic, there is a detailed design plan and discussion here on the mailing list, covering defining new terms, stories and plans. And let us know what do you think.

Foreman Live CD

was mentioned in the last newsletter. Since then we have upgraded Foreman Live CD image to be based on CentOS 7 and are extending Foremen installer with wizards for easier setup of networking and authentication. Detailed report should land on the list soon.

OpenSCAP plugin

OpenSCAP is a toolkit for security compliance automation build on top of SCAP standard. OpenSCAP ecosystem includes low-level C library, command-line scanner, graphical interface (scap-workbench), hardening guidances (scap-security-guide), Anaconda installer add-on, and integration with Spacewalk project. Work on integration with Foreman has just started. See announcement on the list for more.

Meeting together

There are a few interesting Foreman related conferences out there

cfgmgmtcamp.eu - we even have a dedicated track for all Foreman things there
FOSDEM - config management, don't forget ti visit visit IaaS and Virtualisation rooms
devconf CZ - meet people form Red Hat and Fedora communities in Czech Republic
T3chfest - it's mainly geared towards university students

Hope to see you there!

New media

Sprint 31 Demo
Salt integration deep dive
Networking unification

Both clips are availabe on our media page.

Foreman Community Newsletter - October 2014

2014-10-31T11:43:00.000+02:00

Important security updates

Foreman 1.6.1, 1.6.2 and 1.5.4 have been released to fix an important security issues affecting the core and the smart proxy (foreman-proxy) identified as CVE-2014-3590, CVE-2014-3653 and CVE-2014-3691. For more security-related information visit our Security Page.

Katello 2.0 released

The Katello team announced the official release of Katello 2.0 on October 6. This release had been a long time coming and brings with it many new features including a change to Katello architecture and deployment as a Foreman plugin. Katello was designed to bring distributed content management alongside the configuration and provisioning management that Foreman provides. More information can be found in the Release Notes.

Foreman Live CD

People responsible for the foreman-staypuft plugin which aims for OpenStack installations using Foreman developed nice Live CD/appliance which allows users to run Foreman from memory (what we call “stateless” mode). We have taken this work and published it in foreman-live git repository, integrated build process in our Jenkins and started with nightly builds of the live cd. It is currently in alpha stage and expect rough edges, but In the upcoming months we will work on improving the Live CD.

New Foreman discovery image

We redesigned how we build foreman discovery image and we are going to release new version 2.0 which will be smaller, faster, more debugging options and with the same features as 0.6 release. It will work with Foreman 1.6 and 1.7. To check the image out, see our pre1 build which was announced this month.

FOSDEM 2015 is approaching

The FOSDEM and ConfigManagementCamp 2015 Call for Proposals is open until December 1 and we welcome all proposals related to Foreman. As usual, FOSDEM will take place at ULB Campus Solbosch on Saturday 31 January and Sunday 1 February 2015 followed by ConfigManagementCamp on Moday 2 and Tuesday 3 at Gent, Belgium.

Dropping Ruby 1.8.7 support for core

Foreman supports Ruby 1.8.7 from the day one, but since we expect Ruby 2.2 during Christmas and we plan to upgrade to Rails 4.0, it’s the time to drop support of Ruby 1.8.7 for Foreman core (not Smart Proxy, hammer or the Installer). We started with turning off unit and integration tests against Ruby 1.8.7 and removing various related workarounds across our code base. Also we have started to run our tests against Ruby 2.1.

New plugins this month

foreman-cloudstack adds CloudStack compute resource
foreman-abrt integrates Automated Bug Reporting Tool from Fedora and Red Hat compatible hosts
foreman-custom-banner adds user-defined colored top banner
foreman-graphite sends performance metrics to Graphite.
foreman-hammer-ssh hammer SSH plugin, execute SSH commands across multiple foreman managed hosts.

Featured media

Sprint 29 demo
Sprint 30 demo
ABRT plugin deep dive

All the clips are published on our media pages.

What to expect in November

Next month, Foreman core team will be working on various features like mentioned Live CD, better Docker integration, adding http(s) proxy support to foreman-proxy, UEFI support, improving discovery, foreman-tasks plugin availability, katello and bastion split and bug fixing.

Of course, we will be delivering the most anticipated artifact Foreman 1.7.0 beginning December. Expect bunch of Release Candidates during November. More info in our schedule wiki page.

Foreman Community Newsletter - September 2014

2014-10-01T11:11:00.000+03:00

Foreman 1.6.0 released!

After some testing in late August with the release candidate versions, Foreman 1.6.0 is out! Some of the highlights are:

Smart Proxy plugins support
API v2 is now declared stable, will be the default in 1.7.0
External user groups for LDAP and Kerberos group support
Support for RHEL7 (CentOS and Scientific will be supported when SCL 1.1 becomes available)
Configurable and extensible dashboard
Puppet directory environment support

The full list of release notes can be found in the manual, as usual. There are two CVEs explained in the security advisories and scheduled to be fixed in 1.6.1.

New plugins

foreman_graphite adds graphite support for your Foreman infrastructure instances. Configure where to send your performance metrics through a YAML file.

hammer-cli-foreman-ssh gives Hammer parallel remote execution for your nodes in Foreman. Authentication is done through SSH keys and parameters to filter subsets of hosts are available.

Rundeck support will be moved out of core for 1.7. foreman_host_rundeck will allow users to get a YAML representation of hosts for Rundeck.

foreman_salt provides node classification in Salt, and an interface for Salt keys, highstate runs and more. It needs to run with a smart proxy plugin as well. Setup instructions are in the wiki.

foreman_abrt provides reports forwarding to ABRT, also needs to run with a plugin in smart proxy.

hammer_cli_foreman_bootdisk gives Hammer a set of commands for foreman_bootdisk a plugin to Foreman for ISO/USB booting support.

FOSS outreach program and Foreman

Foreman will be again taking an intern for the December-March cycle of the FOSS outreach program (formerly OPW). You can find more information about the program here. Here are the proposed projects, you are welcome in #theforeman in Freenode to ask questions.

Featured media

Mitchell Anicas wrote a great tutorial on how to setup Foreman 1.5 to manage an installation of Ubuntu 14.04 nodes. It is part of a series on Puppet and Configuration management. Check it out in the DigitalOcean community site.

We also discovered recently an article by Anthony Lapenna on Foreman and Puppet infrastructure setup, the scalable way. Check it out if you want ideas for a solid Foreman and Puppet infrastructure.

Sprint 29 demo was recently published, if you want to follow the latest from the Foreman development team. It covers EFI and QEMU setup in Foreman, bond and fencing network devices support, foreman_host_rundeck, foreman_graphite, hammer-cli-foreman-ssh, changes in the code base to use gem friendly_id and puppet server support.

Foreman Community Newsletter - May 2014

2014-05-29T11:22:00.004+03:00

Foreman 1.5.0 released!

Thanks to our many contributors and testers during the Foreman 1.5 development cycle, as with their help we released Foreman 1.5.0 in early May.

The full set of release notes are in the Foreman 1.5 manual:

Foreman 1.5 release notes

An update release (1.5.1) is planned shortly as a follow-up for some bugs found in the release.

Recent articles

Introduction to Foreman

Felix Massem has written an instructional series of articles for codecentric introducing Foreman and showing how they carry out automated provisioning with Foreman and Puppet.

Systemmanagement mit Puppet und Foreman (German)

Mattias Giese presented at Linux-Tage in March on the subject of Foreman and integrating it with other systems management projects such as Puppet, Spacewalk, Icinga and PuppetDB. Audio and slides available in German.

Also linked below is a published paper from Mattias on the subject of easy system administration with Foreman and Puppet.

OpenNebula integration

Achim Ledermüller and Sebastian Saemann from NETWAYS have released an early version of foreman-one, a new plugin providing OpenNebula compute resource integration for Foreman.

Currently this is pending on inclusion of a patch for OpenNebula in Fog, the cloud library that Foreman uses to work with compute resources. Enthusiastic users can run foreman-one with this patch in a development environment. We'll provide updates as soon as this is more generally consumable.

Read more in Sebastian's article on OpenNebula.org:

OpenNebula and Foreman integration (English)
Foreman, Fog und OpenNebula (German)

Recent videos

While we've mostly been consumed by the 1.5 release, the past few weeks added some new features to show off in the regular sprint demo.

This includes changes to the admin account and installer CLI support, improved Chef support (linking hosts to Chef proxies) and a new drag-and-drop dashboard page for plugins to extend.

Foreman Sprint 23 demo

The "staypuft" plugin project is designed to extend Foreman to install hosts running OpenStack services. They've released a video showing the end to end process (note, no audio). Packages for the adventurous are available in the nightly repositories, starting with foreman-installer-staypuft.

Staypuft installation demo

Foreman Community Newsletter - April 2014

2014-05-01T00:54:00.000+03:00

Foreman 1.5, here we come!

We're excited to be just a week or two out from releasing Foreman 1.5.0, with the usual raft of new features.

The highlights in this release are:

config groups for managing sets of Puppet classes assigned to a host or host group (think "profiles"), watch demo
end-to-end host provisioning integration with FreeIPA, so hosts are joined to the realm automatically, watch demo
fine-grained role and filter system for authorizing access to users and groups
a trio of compute resources (oVirt, VMware, libvirt) get image provisioning support
new installer support for a range of plugins and all compute resources

The full set of release notes are in the Foreman 1.5 manual:

Foreman 1.5 release notes

The latest release at the time of writing is 1.5.0-RC2, which is available in the repositories, or keep an eye on the foreman-announce mailing list for the final release.

Foreman 1.5.0-RC2 announcement and notes
foreman-announce mailing list

foreman-announce+subscribe@googlegroups.com to subscribe

Ubuntu 14.04 packages

With the release this month of Ubuntu 14.04 (Trusty Tahr), new packages for this LTS version have been prepared and are available for Foreman 1.5 in the usual repositories.

If you test them out and have feedback, please e-mail the foreman-users list, drop into IRC or file bug reports.

There is a known issue with the installer in RC2, so please take note of the workaround in the announcement or wait until 1.5.0 final.

Discovery Image 0.5

Recently the Discovery images based on oVirt Node have had some major enhancements, including support for a new CentOS 6 based image (in addition to the Fedora based one). There are improvements particularly around network interface handling, and a big rework of the build automation for the images.

Read more of Lukas' announcement below, or get copies of the Discovery Image 0.5 from our website.

Updates from the plugin community

The first plugins to add new compute resources are coming into being, ready to accompany Foreman 1.5 which provides this feature.

Docker and containers are the hot topics of the year, so Amos Benari's new foreman_docker plugin will be of interest to anybody looking to deploy containers from their usual Foreman interface. It's available as a gem and RPM for Foreman 1.5 or above.

XenServer support has been requested a few times and the initial work to add it to Foreman was done by ooVoo. It's now been bundled up into a foreman_xen project to help incubate and improve the support, as the code is still in its early stages. Testers and contributors interested in this are very much encouraged to join in.

foreman_xen

Futher watching

Sprint 22 demo

The last few weeks included major work on Discovery, CLI id/name enhancements, work on the OpenStack/Foreman installer, FreeIPA realm integration, config groups and compute resource plugins (Docker).

Watch sprint 22 demo

More talks

Other talks that have taken place recently:

CLI deep dive - unified name searches - Tomas delves into the details showing how the Hammer CLI will support both names and IDs better.
Foreman and Chef integration - from ChefConf 2014, explaining the foreman_chef plugin and handler integration available

Foreman Community Newsletter - March 2014

2014-03-28T17:47:00.000+03:00

Welcome to the March 2014 community newsletter. Feedback always appreciated, if there's a topic you'd like to see more (or less!) of in these editions.

Call for testing: Foreman 1.5 authorization system

Our Foreman 1.5 development cycle is coming to a close in a couple of weeks, so the aim until the end of April is to finish off the last major features and harden it for release.

To that end, we're putting out a call for testing, focused on the major improvements to the user authorization system in Foreman 1.5. In particular, we're interested in environments that rely on user filters, user groups and want fine-grained access controls.

More information about what's changed, how to test it over the next two or three weeks and report bugs is available on our call for testing page:

Call for testing: authorization system for Foreman 1.5

Foreman 1.4.2 released

Last week, we released an update to Foreman 1.4 to fix two security issues and a number of bugs. This includes updates to support Puppet 3.5 and Facter 2.0, both of which are being released by Puppet Labs imminently.

Foreman snapshot plugin

Greg Sutcliffe has announced the first (alpha) release of a new plugin called foreman_snapshot, designed to make images or snapshots on compute resources from host groups. The idea is to use host groups as a form of template, creating an image with most of the configuration already applied, and then boot new hosts quickly from this predefined image.

So far it's implemented for libvirt and OpenStack compute resources, with more to come. Note that it is currently API-only and requires Foreman 1.5/nightly.

Further watching: Sprint 21

Sprint 21 included the new authorization system and other large features such as CLI enhancements that will make up the Foreman 1.5 release. The demo of the work from this sprint is now available and includes foreman_snapshot's libvirt support, Hammer CLI enhancements, LDAP group support, a live CD of Foreman and more work on the configuration groups UI. Watch the video here:

Foreman Sprint Demo 21 (see description for full agenda)

OpenStack Foreman Installer

RDO has offered Foreman as an installation method for OpenStack for the last couple of releases, but a team of developers are working on a major update for the installation and setup process of Foreman itself (called "OFI" for "OpenStack Foreman Installer" or "Staypuft").

This is based on a new plugin to Foreman and coupled with the Discovery plugin to quickly provision physical hosts. See these two videos for two aspects of the combined installer, and follow the foreman-dev mailing list for updates:

As a further part of this work, a new live CD is being developed that contains Foreman pre-installed with Discovery ready for provisioning. Follow the foreman-live project here and watch the demo here.

GNOME OPW and events

The Foreman project is proud to be part of this summer's GNOME Outreach Program For Women, helping women get involved in open source projects.

While the Foreman application period is just wrapping up (by 31st March), we've seen a number of new faces get involved in the project over the past couple of weeks and are very happy to have received patches to some of our projects.

Next month, there's a talk on Foreman and Chef integration at #ChefConf 2014 (San Francisco), on Thursday 17th April. Tickets for the conference are still available!

Foreman Community Newsletter - February 2014

2014-02-28T23:54:00.002+02:00

With a number of events and the Foreman 1.4 release, February's been busy.

Videos and presentations

Early in February, we enjoyed an excellent meetup of Foreman users and developers in Belgium for the combination of FOSDEM and the Foreman track at CfgMgmtCamp.

Lots of visitors stopped by the stand at FOSDEM shared with oVirt, where a demo of the newly released Foreman 1.4.0 was on display. At CfgMgmtCamp, a series of interesting talks was held over the two days, with subjects from Foreman deployment through to plugin development.

Recordings of some presentations are now available:

Foreman Project, Ohad Levy
Foreman integration with Chef and others, Marek Hulán
Build your own image: Foreman discovery plugin - Lukas Zapletal

A number of slide decks used in the other presentations at CfgMgmtCamp are available on the website:

Videos > Presentations (see the bottom of the page)

Foreman 1.4.0 and 1.4.1 released

At the start of February (ready for FOSDEM), Foreman 1.4.0 was released and followed up with a bug fix release (1.4.1) a few weeks later. Users are urged to upgrade to 1.4.1 to fix a few important bugs, such as errors when deleting compute resources and compute profile fixes.

Key new features are compute profiles, image user-data (cloud-init) support, web UI enhancements, Kerberos support, a new plugin registration and extension framework, plus lots of new provider support in the smart proxy. Lots more information is available in the Foreman 1.4 release notes.

Further watching

After the bug fix push for Foreman 1.4, another sprint has been completed with some new features already in the works and being committed ready for Foreman 1.5 in a few months' time.

This includes improved host registration, i18n support in Hammer, plugin support in the installer, config groups for Puppet classes, oVirt and libvirt imaging support and Docker compute resources. Have a look at the demos here:

Foreman Sprint Demo 20 (see description for full agenda)

The team regularly holds "deep dive" sessions to explore a new feature or topic. Join the Foreman Google+ community to keep up with these events. Here are the recent recordings:

Dynamically generated CLI: Tomas shows ongoing work to dynamically generate the CLI based on API documentation
Rewriting orchestration to Dynflow : Ivan shows how orchestration actions can be written to use Dynflow

Foreman Community Newsletter - January 2014

2014-01-28T22:15:00.003+02:00

Welcome to a new year, already with lots of exciting developments.

Foreman 1.4.0 about to be released

We're very close to announcing the general availability of Foreman 1.4.0, with contributions from over eighty individuals, and

At the time of writing, 1.4.0-RC2 is available for download and is stable, with just one or two known bugs to be fixed in 1.4.0 final. The final release should be available on Thursday 30th January.

Key new features are compute profiles, image user-data (cloud-init) support, web UI enhancements, Kerberos support, a new plugin registration and extension framework, plus lots of new provider support in the smart proxy.

Lots more information is available in the Foreman 1.4 release notes. Keep an eye on the website or the foreman-announce list for the final availability.

Upcoming events

This weekend and coming week, the largest ever group of Foreman users and developers will congregate in Brussels, Belgium for the FOSDEM and CfgMgmtCamp events.

Just before FOSDEM will be a CentOS Dojo event in Brussels, at which Julien Pivotto will give a talk on Lifecycle Management with Foreman.

Moving on to FOSDEM, there will be some Foreman-related talks:

Config management: Foreman integration with Chef and others, Marek Hulán
Virtualisation and IaaS: Foreman Project, Ohad Levy
Testing and automation: Automation in the Foreman infrastructure, Greg Sutcliffe

On Sunday evening, if you're still in the Brussels area, please join us for a meal. Reply to the Google+ event for numbers.

At CfgMgmtCamp, we'll hold a dedicated Foreman track with a variety of talks. The complete schedule has fifteen talks on a variety of Foreman-related subjects, including:

Deploying Foreman in enterprise environments, Nils Domrose
Realtime remote management with MCollective and Foreman, Sam Kottler
Hammer - a CLI tool for Foreman, Tomas Strachota
Managing Foreman with Foreman, Ewoud Kohl van Wijngaarden

Lastly, at AsiaBSDCon 2014 in early march, Martin Matuška will give a talk on Foreman and FreeBSD provisioning.

Katello installer: alpha testing

The team behind Katello is making good progress in adapting it to be a Foreman plugin, with a new combined installer.

If you're interested in trying it out in the early stages and report bugs, read Eric's e-mail to foreman-dev for more details.

Further watching

The last development sprint was completed at the beginning of December, and the sprint demo shows lots of features that are either available in nightly or upcoming work.

This includes user data support, an update on the permissions system changes, compute profiles, dynamic API bindings and Puppet future parser support.

Foreman Sprint Demo 18

The team regularly holds "deep dive" sessions to explore a new feature or topic. Join the Foreman Google+ community to keep up with these events. Here are the recent recordings:

Organization and location inheritance: Joseph shows a change to allow a parent/child organization/location concept
Dynflow tools and concepts (1/2), Dynflow executors (2/2): Petr's two part deep dive on Dynflow for task orchestration

Foreman Community Newsletter - November 2013

2013-11-25T11:09:00.000+02:00

Welcome to the November 2013 issue of the Foreman newsletter.

CfgMgmtCamp.eu Foreman track announced

Join the Foreman team at CfgMgmtCamp.eu in Ghent, Belgium on 3-4th February 2014 for a series of talks on all aspects of the project.

CfgMgmtCamp.eu will host a selection of the biggest configuration and systems management projects for two days after FOSDEM 2014, and is a short train ride from Brussels.

The call for papers is open until 1st December, and we'd love to hear about how you use Foreman in your organization or projects.

Foreman 1.3.1 update released

The first update to Foreman 1.3 has been released, fixing a number of bugs found in 1.3.0.

More information in the announcement and release notes.

Foreman Discovery oVirt-based image

Lukas Zapletal has announced a new image based on oVirt Node which contains a far better range of drivers and firmware for systems, and is RPM-based. It works as a plugin to oVirt, so it extends the existing image building mechanism, which you can replicate and customize.

Further watching

A recording of the latest additions to Foreman, available already in nightly packages, has been posted. It features nested fact support, host creation via the Hammer CLI and updates on the Katello and MCollective plugins.

Foreman Sprint Demo 16

The team regularly holds "deep dive" sessions to explore a new feature or topic. Join the Foreman Google+ community to keep up with these events. Here are the recent recordings:

OpenStack provisioning with plugins: a walkthrough of provisioning RDO OpenStack hosts with foreman_setup and foreman_openstack_cluster
New oVirt Node based image for Discovery: demoing of the new oVirt Node image and how it's built
Plugin registration and menus: a new mechanism for plugins to register and extend Foreman's menu
System vs Host: a discussion of whether to rename "host" to "system" (background)

Foreman Community Newsletter - October 2013

2013-10-22T11:53:00.001+03:00

Welcome to the October 2013 issue of the Foreman newsletter.

Foreman 1.3.0 and 1.2.3 released

With the help of everybody who tested the 1.3 release candidates last month, Foreman 1.3.0 was released in mid-October with a whole set of new features. Special thanks go to the 53 contributors who created this release, across all of our projects.

The new kafo-based Foreman installer features, along with Amazon EC2 VPC support, a preview of the Hammer command line tools and experimental Google Compute Engine support.

Lots more information, including upgrade notes is available in the Foreman 1.3 release notes in the manual.

Foreman 1.2.3 security release

A security fix release was made for Foreman 1.2.3, which fixes an authenticated SQL injection. All 1.2 users are urged to update as soon as possible. Read the announcement for more info.

Chef support in the making

A team of developers (Romain Vrignaud and Marek Hulán) are working on adding support for the Chef configuration management tool to Foreman. Their first steps are to import facts and reports, before working out how to provide full management capabilities equivalent to Foreman's support for Puppet.

Chart their progress and join in via:

Chef reporting wiki page
chef-handler-foreman GitHub project

Discussions have also started about integration with Salt. Do get in contact via the foreman-dev mailing list or IRC channels if you can help out with either.

Katello project moves into Foreman

The Katello project (for content and subscription management) has announced that the project will become a Foreman plugin in the future, with efforts now going into modifying the existing codebase to run inside Foreman.

We'd like to welcome the Katello team into the Foreman community - this will be an exciting project to watch in the coming months.

Foreman Community Newsletter - September 2013

2013-09-22T06:36:00.000+03:00

This is the first edition of what I hope will continue as a monthly overview of events in and around the Foreman project, news from our plugin community and must-read articles or tips. Comments and suggestions are most welcome.

Foreman 1.3 release candidates out!

The first release candidates (RC) are out for the Foreman 1.3 release, only a few days behind the target quarterly schedule agreed at the roadmap meeting earlier this year.

Headline features include a new CLI (Hammer), new foreman-installer and many compute resource enhancements (EC2 VPC, association, GCE support).

Read the RC announcement or the 1.3 release notes for more information, and file feedback in Redmine.

Plugin community news

There are some exciting new plugins being created to extend Foreman's capabilities. We feature three this month:

Clustered Foreman with foreman_memcache

For users running Foreman in a cluster configuration, foreman_memcache gives your instances a shared session and cache store. It's seen an update to add a configuration file now and is now available in the plugins yum repo.

More information on the foreman_memcache project page.

Build ISOs with foreman_bootdisk

This plugin wraps up Foreman's g/iPXE support into a user-friendly button to download ISO and USB images with fully static networking, so you only need DNS and connectivity to Foreman to boot new hosts. We'd like to expand support to host group images in the future.

More information on the foreman_bootdisk project page.

Tidying PuppetDB with puppetdb_foreman

PuppetDB stores exported data and catalogs from all of the hosts in your infrastructure, but when you delete a host in Foreman, that data lingers until it expires. Daniel Lobato's plugin contacts PuppetDB when a host is deleted and triggers the tidyup immediately, the equivalent of puppet node deactivate.

More information on the puppetdb_foreman project page.

Getting plugins

All of the featured plugins above, and more, are available as RPMs in our repo at yum.theforeman.org/plugins.

The full known list and installation instructions are on the plugins wiki.

Recent talks

+Marek Hulán shows the new kafo project built for the Foreman installer: Kafo deep dive
+Greg Sutcliffe explains the work to remove the core Puppet dependency: Removing Puppet from core

Screencast: Locations & Organizations

2013-01-07T20:06:00.001+02:00

One of the most important features that will be released soon as part of Foreman 1.1 is locations & organizations. We will probably do a deep dive (which will get recorded) pretty soon, but I recorded a quick screencast to show how they work. Enjoy!

The volume in the recording is low, but it gets very degraded when I try to increase the gain so make sure to crank up your output.

Parameterized Class Support in Foreman

2012-11-26T14:34:00.000+02:00

We've got Parameterized Class support in Foreman now, and it'll be in the next release! I've recorded a small screencast demonstrating its use:

You can find the mentioned wiki page at wiki:Parameterized_class_support

Debian Packaging, the Transparent way

2012-11-01T20:17:00.000+02:00

So I've been managing the Debian packages for about 8 months now, and every so often I get asked if there's anything people can do to help. I have to answer "Not really" because the way we're building the Debian packages is somewhat arcane.

At least, it was. This blog is to tell you all about how it's now much more open.

Packaging in the open

The first piece of the puzzle starts with our foreman-rpms repo (github.com/theforeman/foreman-rpms)
in this repo we store all the packaging data for the foreman packages, both deb and rpm. In the case of Debian, you'll see a structure like:

debian/<distro>/<type>/<package>

For example, the foreman-proxy nightly package for squeeze is at:

debian/squeeze/nightly/foreman-proxy

Now, the eagle-eyed and quick-witted among you will already have thought "That's quite a lot of code duplication". You're absolutely correct, but this method allows us to package each distro in it's own way, if need be. At the moment, it's all very similar, but it might not stay that way. It also makes it trivial to add in more distributions later (say, quantal or wheezy).

So, now you know where our package data is, you'll want to know; how does it get used? Well, here we turn to the king of automation - Jenkins.

Jenkins

You can find our Jenkins server at ci.theforeman.org. In there, you will find a set of interesting packaging jobs, specifically:

packaging_build_foreman-proxy_matrix
packaging_build_foreman_matrix
packaging_build_foreman-proxy_individual
packaging_build_foreman_individual

There's two types of jobs here. Let's start with the "individual" jobs. These are what they sound like - they take a set of parameters about what package the are building and build the required package(s). For example, if I feed it the options

arch = 64bit
distro = squeeze
type = nightly

Then it checks out the foreman-rpms repo, and changes to the appropriate directory in the repo, according to those options. It then runs 'build.sh' found in the packaging directory, which uses Pbuilder to build the package for the requested settings. At the end of the job (if it built successfully) the package is uploaded to our Debian repository at deb.theforeman.org

The matrix jobs, as you can guess, call the individual jobs many times over - once for all the possible combinations of the options. Thus we can have the matrix job track the Foreman git repo, and rebuild the devel packages whenever a commit is made.

How to get involved

Now that things are more open, it's easy to get involved. Here's a workflow you could
use:

1. Get a build box

You don't have to go through the trouble we did of configuring pbuilder. Just build a
machine with the OS you'd like to run the packages on

2. Install basic packages

You'll need 'build-essential' and 'devscripts' as a minimum (well, on Squeeze
anyway :P)

3. Check out the repo

Fork the foreman-rpms repo and check it out to your buildbox. If you're not trying
to update an existing distro, copy the closest match to a new repo.

4. Build the package

You don't need to use 'build.sh' for this (that's crafted towards pbuilder and other
small automated task we need to do in the final packages). Just run "debuild -us -uc"
and see if it builds. If it doesn't either
a. Install the missing build-dependencies, or
b. Fix the broken packaging

5. Test

You can either test on your buildbox, or for best results, on another machine with
the same OS installed. Hopefully the package will install and work ;)

6. Contribute

Commit your changes to your repo and send us a pull request on foreman-rpms. We'll
test your changes and merge if it looks good.

Conclusion

Hopefully that demystifies our package building architecure a bit. If nothing else,
this blog serves to remind me how we set it all up when I can't remember in six
months time ;)

Epilogue

If anyone is interested in replicating the whole chain, you can get our pbuilder
puppet configuration from our infrstructure repo at github.com/theforeman/foreman-infra

This repo contains the puppet code we use to build the various parts of our
infrastructure, including the Debian build slaves for Jenkins. The only thing you won't
find there is the GPG key to sign the packages, for obvious reasons ;)

Open source infrastructure for the Foreman project.

2012-10-23T14:35:00.000+02:00

Overview

We started working on a new project to create build and test pipelines for releasing code and running tests, and small, but dense infrastructure arose from working on that project. Instead of keeping the Puppet code locked up, we decided specifically to make it public. There are lots of benefits of doing this, but it made sense specifically for us so others can help manage the infrastructure on an unfettered basis. Lots of other benefits come with public Puppet code, too, like thinking more acutely about storing private information in the repositories and reducing the ability of manual changes to cause issues in the test environment.

The tools

Jenkins lives in the center of the infrastructure and is key to the development process; ensuring our tests pass, building packages, and validating our installer. We had been using Travis CI, a great project that is changing the way open source projects test their code. There were a few issues with Travis; Foreman isn't a "standard" Rails application in that it requires some gems with native extensions that were hard to get installed on Travis. Additionally, the ability to run tests on different operating systems is key, and Travis only supports Ubuntu right now. Lastly, it's not possible to run builds without pushing to the repository, which makes testing out new configurations a burden. We wanted to be able to build more complex pipelines like building development packages after tests pass. That's certainly possible on Travis, but Jenkins provides a central place to build flexible jobs and provides facilities like build slaves, which are great for building Debian packages and RPM's and testing across different platforms and databases.

Here's a complete rundown of the different tools we are using right now:

- Puppet & the Foreman (of course)
- Jenkins
- pbuilder
- reprepo
- mock
- Redmine & friends (for theforeman.org)

The implementation

We currently have less than a dozen machines; several builders, a few testing machines, Jenkins (only runs the interface, no builds), and a server that runs the website. The CI infrastructure runs on a physical box with KVM & libvirt for virtualization, provisioned via the Foreman. Servers are grouped via the "hostgroup" feature in Foreman and given the appropriate classes via the ENC. Despite the low number of instances, being able to dynamically provision new builders in a few minutes is endlessly convenient.

What's left?

Sensitive data

There is still a lot of different pieces the need to be addressed as the project grows. One of them is that we don't have an easy way to store sensitive data in the public repository. One option to handle that is GPG encyrption and then storage in the public repo using some variation of this technique, with Foreman as the provider of the parameters. Most of the files that need to remain private, like the private key for accessing slaves, are stored on the Jenkins master; we don't need to distribute them too widely so manual placement works for now.

Contribution

It's really easy to contribute to the infrastructure codebase right now; fork the repo, make your changes, submit a pull request. What's not so easy is getting those changes out onto the puppet master - this is still a manual task that I run each time a change gets made. Ultimately the deployment to the master will probably be handled via Capistrano and Jenkins on a post-receive hook from the Github repository which will completely alleviate this problem. Using Capistrano is also great because it allows easy rollback should a deployment to the master go sideways.

The takeaway

As far as I'm concerned we're just getting started! There is a lot more work to do on both the infrastructure and automation sides. Now that most of the initial work to get the project off the ground is complete, incremental changes will be in order to keep the dependencies of the builders up to date and ensure things running smoothly. Spending the time to think through the architecture and build a system that will work as the community around Foreman grows has absolutely been a worthwhile use of time.

More posts to come soon about the work that the team has been doing on automated package building and repository promotion - keep an eye out!

Foreman 1.0 Demo talk

2012-07-19T09:33:00.002+03:00

Last week I had the pleasure of presenting Foreman 1.0 at redhat offices in Brno, CZ.

It was done during our lunch and learn sessions (people like it becuase you get free pizza or so) which could explain a few of the interesting background noises during the talk.

as usually, demos don't work out as planned, but I decided to leave it all, so you can enjoy for yourself.

this by no means its a complete demo of 1.0 features, but maybe a more general overview and its features in 1.0.

many thanks to inecas and lzap for taking care for the details and capturing the talk on video.