Blog Archive on t0rrant's tech blog

OpenStack - the beauty of the Beast - part 1

Wed, 04 Nov 2020 12:27:00 +0100

Introduction

Almost two years ago I embarked on a challenge which led me through the depths of OpenStack and the hardship of managing a multi-region OpenStack deployment. OpenStack is a beast, really, it can get enormous even with just "basic" cloud features. For example, if I want to launch a Virtual Machine (VM) in the cloud and have it display a static webpage via HTTP it will, at least, need the following from the cloud provider:

a way to identify a cloud user
a way to store images to boot the VM
hypervisor nodes to launch the VM in
some kind of scheduler to select to which hypervisor the VM should be allocated to
network virtualization on top of the physical network layer, so our user network is separated from other users
some way to access the VM from the exterior
a way for all of the services to communicate with each other
a place to store the state of the cloud at any given moment

That said, there is a lot of service interaction, and they are amazingly independent services that can do their part on their own, and in the end it works (mostly) as expected and it is somewhat straightforward to use.

We can use OpenStack for managing VMs (nova), containers (magnum) or bare metal (ironic) hosts. In this first part of presenting my OpenStack experience we will focusing on VMs and will be covering:

OpenStack overall architecture
Just enough service dependencies to get a working OpenStack cluster
Common difficulties (especially when coming from interacting with older releases)
Basic usage (user perspective)

In a future second (or even third) part we will cover other topics like operating the cloud, dealing with migrations, planning for high availability, and debugging networking issues (insert ominous sounds here).

Architecture

Let us start from a user-centric perspective. As we can see in Figure 1 the user interacts with each service independently. However, in order to do that there needs no be some kind of authentication, and it should also exist some kind of service discovery. We can depict a possible interaction as follows:

Cloud User communicates with the Identity service, authenticating against it and receiving a session token after a successful authentication.
Cloud User communicates with the Identity service using the previously received token and asks for all of the known public service endpoints.
Cloud User communicates with the desired service through its public endpoint, using the same token it received previously.

OpenStack service interaction from the user perspective

So we can see that the Identity service is a keystone for OpenStack's workflow, and is no surprise that it is called Keystone ;-)

Although there are obvious dependencies for using each service, the beauty of OpenStack is that a failure of one service does not mean a complete failure of another service. We may, however, lack some functionality of the service, for example, if your identity plane crashes spectacularly you may not be able to launch new VMs but your VMs' will still be up and running, and the network is also reachable.

This way of interacting is the usual CLI interaction path, you can, however, not see any of this and use the Dashboard service, also known as Horizon, so that you can see in a web application all of the available services and all of the available features from those services, acting as a proxy for the previous interaction.

Services

In this section we will see every service, one by one, in more detail.

Identity
Image
Block Storage
DNS
Network
Compute
Dashboard
Service Communication
Service State

Identity

Keystone is an OpenStack service that provides API client authentication, service discovery, and distributed multi-tenant authorization by implementing OpenStack’s Identity API. [1]

The central piece of every OpenStack cloud, it is responsible for keeping records of users, generating authentication tokens and other access mechanisms like API keys, maintaining a registry of available services for the cloud, and ensure individual service authentication and access control for different users to different projects (aka tenants).

Image

The Image service (glance) project provides a service where users can upload and discover data assets that are meant to be used with other services. This currently includes images and metadata definitions. [2]

This service is responsible for maintaining images that will be used to launch VM instances. They can be public, usually made available by administrators, private or shared, so one tenant can upload an image and share it with other tenants.

Block Storage

Cinder is the OpenStack Block Storage service for providing volumes to Nova virtual machines, Ironic bare metal hosts, containers and more. [3]

This services is responsible to store volumes created from cinder images so we can boot our VMs with persistent root storage, persistent volumes to attach to instances for additional storage, or even create encrypted volumes using another OpenStack service responsible for secrets management we will not cover here (barbican).

DNS

Designate is a multi-tenant DNSaaS service for OpenStack. It provides a REST API with integrated Keystone authentication. It can be configured to auto-generate records based on Nova and Neutron actions. Designate supports a variety of DNS servers including Bind9 and PowerDNS 4. [4]

With DNSaaS incorporated in an OpenStack cloud deployment you can have every instance you launch to have valid DNS records for their interfaces, and records for floating IPs you create, making it simple to manage DNS for your projects.

Network

Neutron is an OpenStack project to provide "network connectivity as a service" between interface devices (e.g., vNICs) managed by other OpenStack services (e.g., nova). It implements the OpenStack Networking API. [5]

This service is what gives network connectivity to and from your VM instances. It enables users to create Routers, making it possible to create multiple networks within your projects, connect your routers to the outside network, assigning public IPs to your VM instances, announcing Border Gateway Protocol (BGP) routes among other network operations, like creating firewall rules outside Operating System (OS) space, vital to your services operations.

Compute

Nova is the OpenStack project that provides a way to provision compute instances (aka virtual servers). Nova supports creating virtual machines, baremetal servers (through the use of ironic), and has limited support for system containers. Nova runs as a set of daemons on top of existing Linux servers to provide that service. [6]

This is the service responsible for allocating VMs to hypervisors, and it depends on keystone, glance and neutron components to operate on a basic level.

Dashboard

Horizon is the canonical implementation of OpenStack’s Dashboard, which provides a web based user interface to *OpenStack services including Nova, Swift, Keystone, etc. [7]

It is basically a way for users to interact with the OpenStack cloud in a simple graphical way, avoiding to perform complex operations through the OpenStack API or the OpenStack CLI. It is also an easy way to look at resources available to individual projects, and resource details all in one place.

Service Communication

All of the above services have to communicate with each other, or even between different agents for the same service. This is achieved through message queueing and at this time it supports the following backends:

RabbitMQ
Qpid
ZeroMQ

OpenStack does not support message-level confidence, such as message signing. Consequently, you must secure and authenticate the message transport itself. For high-availability (HA) configurations, you must perform queue-to-queue authentication and encryption. [8]

Service State

State for the above services has to be kept in an atomic, consistent, isolated and durable way. So we should use a database backend which is compliant to this model. OpenStack considers using either PostgreSQL or MySQL as the database backend, I personally like MySQL (or MariaDB) as it provides high availability options out of the box.

The choice of database server is an important consideration in the security of an OpenStack deployment. Multiple factors should be considered when deciding on a database server. [9]

Openstack Landscape

After this fast run-down of services we can look at another representation of an OpenStack cloud, as we can see in Figure 2.

OpenStack service architecture

Here we can see three layers:

External Layer
Service Layer
Physical Layer

In the External Layer we have our cloud users, which interact with the OpenStack cloud through either the Horizon Dashboard or through its REST API, either directly or through the openstackclient wrapper. These interactions usually start with asking the Keystone service for a token, and using that token to interact with the other cloud services. The service discovery is also made via the Keystone service, as every service must register with the Keystone endpoint.

We could consider the Dashboard to be part of the service layer, although I like to see it more like the wrapper to the REST API that it is, a web front-end, so we will leave it in a limbo between the External and the Service layer

In the Service Layer we have several distinct Planes where specific operations occur.

In the Control Plane we have all of our services' controllers (neutron, designate, nova) or standalone services (keystone, cinder, glance). These are responsible for, respectively, either delegate actions to other service agents (i.e: delegate VM creation to specific compute nodes), or take direct actions themselves (i.e: upload a cloud image to the storage backend).

In the Compute Plane we have the compute nodes, which receive VM allocation requests in the nova agents, and create those VMs using the appropriate hypervisor.

The Network Plane is probably the most complex, and is responsible for plugging and unplugging ports, create networks or subnets, and provide IP addressing. These plug-ins and agents differ depending on the vendor and technologies used in the particular cloud. OpenStack Networking ships with plug-ins and agents for Cisco virtual and physical switches, NEC OpenFlow products, Open vSwitch, Linux bridging, and the VMware NSX product [10] , via neutron agents and plug-ins. It can also be responsible for announcing BGP routes (via a specific neutron agent), and manage DNS zones and records (via a designate agent).

The Storage Plane is where cloud images (glance) and volumes (cinder) are effectively stored. Openstack supports a wide variety or storage backends, although in their documentation they say the most common are Ceph RBD, NFS or Local Disk (LVM).

The Control, Compute and Network Planes interact through two channels. A messaging queue (usually RabbitMQ) is used for services to publish and collect messages in specific service queues, where operations to perform and their results are sent to. A database (usually MySQL or PostgreSQL) is used to store the state of each service, describing the state of the whole OpenStack cloud state. Obviously these channels are very important for a smooth operation, sometimes being an operational bottleneck, and so they should be separate services and ideally clustered.

Finally, the Physical Layer is where the services actual run be it VMs, bare metal machines or containers. In Figure 2 we can see a correspondence between the Service Layer and the Physical Layer through their colours (sorry for not choosing colorblind safe colours):

Control Plane runs in Controller Nodes
Compute Plane runs in Compute Nodes
Network Plane runs in Network Nodes
Storage Plane runs in Ceph Cluster
State Storage runs in MariaDB Cluster
Message Queue runs in RabbitMQ Cluster

And with all of these services configured properly we have a minimal working OpenStack Cloud, neat!

For a broader view of what OpenStack can provide, we can look at Figure 3 for a list of all of the projects that exist until the day of this post.

Map of OpenStack projects [11]

Conclusion

In this first post we took a look at different OpenStack components, namely the bare minimum to have a functional cloud:

manage users, projects and their access tokens (Identity)
manage images to boot VM instances from
manage volumes to store arbitrary data, attaching and detaching from VM instances
manage network components such as routers, networks, subnets, DNS
launch VM instances from an available image or volume

This article and it referred resources should give you everything you need to be able to deploy a minimal and functional OpenStack cluster.

Feel free to leave comments below, any improvement to this and other articles is always welcome.

Thanks for reading!

References

[1]	--, Keystone, the OpenStack Identity Service - OpenStack Docs [link]

[2]	--, Welcome to Glance's documentation! - OpenStack Docs [link]

[3]	--, OpenStack Block Storage (Cinder) documentation - OpenStack Docs [link]

[4]	--, Designate, a DNSaaS component for OpenStack - OpenStack Docs [link]

[5]	--, Welcome to Neutron’s documentation! - OpenStack Docs [link]

[6]	--, OpenStack Compute (nova) - OpenStack Docs [link]

[7]	--, Horizon: The OpenStack Dashboard Project - OpenStack Docs [link]

[8]	--, Message queuing - OpenStack Docs [link]

[9]	--, Databases - OpenStack Docs [link]

[10]	--, Networking service overview - Openstack Docs [link]

[11]	--, Introduction, Openstack Docs [link]

Integration and Compliance with InSpec

Sun, 11 Oct 2020 13:37:00 +0100

Introduction

Chef InSpec is an open-source framework for testing and auditing your applications and infrastructure. Chef InSpec works by comparing the actual state of your system with the desired state that you express in easy-to-read and easy-to-write Chef InSpec code. Chef InSpec detects violations and displays findings in the form of a report, but puts you in control of remediation. [1]

After we got past that well marketed description on what InSpec is, let us be more specific on what you can do with it.

When developing Chef Cookbooks you usually have a default InSpec test file created for you which is also defined in the kitchen.yml manifest, i.e:

t0rrant@testing:~$ chef generate cookbook mycookbook
....
Your cookbook is ready. Type `cd mycookbook` to enter it.
....
Why not start by writing an InSpec test? Tests for the default recipe are stored at:
test/integration/default/default_test.rb
...
t0rrant@testing:~$ cd mycookbook/

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# test/integration/default/default_test.rb
# InSpec test for recipe mycookbook::default
# The InSpec reference, with examples and extensive documentation, can be
# found at https://www.inspec.io/docs/reference/resources/
unless os.windows?
# This is an example test, replace with your own test.
describe user('root'), :skip do
it { should exist }
end
end
# This is an example test, replace it with your own test.
describe port(80), :skip do
it { should_not be_listening }
end

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# kitchen.yml
...
verifier:
name: inspec
...
suites:
- name: default
verifier:
inspec_tests:
- test/integration/default
attributes:
t0rrant@testing:~/mycookbook$

So you can see that it is fairly simple to start smashing keys and add more tests for your cookbook.

Now imagine that besides that specific cookbook you have a running system, in production, with some of the same integration requirements. Let us assume the following requirement:

the machine should be serving MySQL from port 3306

according to those requirements we could define some rules for the system:

For the sake of simplicity, we assume the target system is running a debian-based Linux distribution

the package mariadb-server should be installed
the user mysql must exist
the mysqld process should be listening on port 3306

This can be easily described as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


describe package('mariadb-server') do
it { should be_installed }
end
describe user('mysql') do
it { should exist }
end
describe port(3306) do
it { should be_listening }
its('processes') { should include 'mysqld' }
end

The result of having that evaluated successfully should be something like:

Package mariadb-server
✔ is expected to be installed
User mysql
✔ is expected to exist
Port 3306
✔ is expected to be listening
✔ processes is expected to include "mysqld"

Cool, but wait... what are we actually testing here? Configuration, not functionality. These tests do not tell us if the MySQL server is actually working or that the SSH server is properly accepting Public Keys and not Passwords.

This shows that the system is compliant for our specific rules, but what we probably want is also a way to test for functionality.

As a first step let us organize these tests and create a profile for them. Let us call it database.

Creating InSpec Profiles

Chef InSpec supports the creation of complex test and compliance profiles, which organize controls to support dependency management and code reuse. Each profile is a standalone structure with its own distribution and execution flow. [2]

Here we will only cover the basic aspects of InSpec profiles, you should check the docs [2] for more information.

Profile Structure

A profile can be seen as a directory with the following structure:

t0rrant@testing:~/inspec/profile$ tree
.
├── controls
│   ├── control1.rb
│   └── control2.rb
├── files
│   └── example_input.yml
├── inspec.yml
└── README.md

The README.md file should contain information about the profile, I usually leave some examples of a successful execution on the profile.

The inspec.yml file has general information about the profile, including inputs (we will get to that later), the profile name, supported platforms, the profile's version, among others.

The controls directory contains the InSpec tests which will be applied when executing the profile.

The files directory can contain additional files, accessible by the profile, or just arbitrary files useful outside of the testing scope, and it is optional.

We can also have a libraries directory in which we can have InSpec resources extensions and helper functions, and is also optional.

Database Profile

Let us create the previously mentioned structure with two control files (one for compliance and another for integration testing), the main inspec.yml file describing our profile and an input.yml (which we will use to pass custom values to variables we will be using in the integration testing).

t0rrant@testing:~/database$ tree
.
├── controls
│   ├── compliance.rb
│   └── integration.rb
├── files
│   └── input.yml
├── inspec.yml
└── README.md

First the inspec.yml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


# inspec.yml
name: database
title: Basic Database
maintainer: Manuel Torrinha
copyright: Manuel torrinha
copyright_email: torrinha@implement.pt
license: MIT
summary: Verify that mariadb Server is configured and working properly
version: 1.0.0
supports:
- platform-family: linux
inspec_version: "~> 4.0"
inputs:
- name: db_host
description: Hostname or IP address for accessing the database
type: string
value: '127.0.0.1'
- name: db_port
description: Port on which the database should be listening
type: numeric
value: '3306'
- name: db_user
description: Username used to access the database
type: string
required: true
- name: db_password
description: Password used to access the database
type: string
required: true

Looks good, we defined four input variables so that we can test different settings for the database connection. And also we made the db_user and db_password mandatory in order for any control that depends on either of them fails if they are not defined.

Next let us create the tests for compliance in controls/compliance.rb:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# controls/compliance.rb
control 'mariadb-compliance' do
title 'Check for proper configuration'
describe package('mariadb-server') do
it { should be_installed }
end
describe user('mysql') do
it { should exist }
end
describe file('/etc/my.cnf') do
it { should_not be_more_permissive_than('644') }
end
describe port(3306) do
it { should be_listening }
its('processes') { should include 'mysqld' }
end
end

As you can see we just did the same as in the introduction, plus we also check for the main config file's permissions.

Finally, we will create the tests for integration in controls/integration.rb:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# controls/integration.rb
control 'mariadb-integration' do
title 'Checks for proper DB connection'
sql = mysql_session(input('db_user'), input('db_password'), input('db_host'), input('db_port'))
describe sql.query('show databases;') do
its('exit_status') { should eq(0) }
end
end

Here we are testing if the connection works, by executing a common query to list the existing database, which should not fail if we can authenticate successfully.

Now let us test this database profile:

t0rrant@testing:~/database$ inspec exec . --log-level=error
Profile: Basic Database (database)
Version: 1.0.0
Target: local://
✔ mariadb-compliance: Check for proper configuration
✔ System Package mariadb-server is expected to be installed
✔ User mysql is expected to exist
✔ File /etc/my.cnf is expected not to be more permissive than "644"
✔ Port 3306 is expected to be listening
✔ Port 3306 processes is expected to include "mysqld"
× mariadb-integration: Checks for proper DB connection
× Control Source Code Error ./controls/integration.rb:1
Input 'db_user' is required and does not have a value.
Profile Summary: 1 successful control, 1 control failure, 0 controls skipped
Test Summary: 4 successful, 1 failure, 0 skipped

As we can see the compliance control seems ok, however we can't run the integration tests because we did not define neither db_user or db_password. So, let us create the file files/input.yml and define them there:

1
2
3


# files/input.yml
db_user: 'avaliduser'
db_password: 'a6v8a8l2li0dp01as0s0w6r0d'

t0rrant@testing:~/database$ inspec exec . --input-file=files/input.yml --log-level=error
Profile: Basic Database (database)
Version: 1.0.0
Target: local://
✔ mariadb-compliance: Check for proper configuration
✔ System Package mariadb-server is expected to be installed
✔ User mysql is expected to exist
✔ File /etc/my.cnf is expected not to be more permissive than "644"
✔ Port 3306 is expected to be listening
✔ Port 3306 processes is expected to include "mysqld"
✔ mariadb-integration: Checks for proper DB connection
✔ Command: `mysql -uavaliduser -pa6v8a8l2li0dp01as0s0w6r0d -h 127.0.0.1 --port 3306 -s -e "show databases;"` exit_status is expected to eq 0
Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped
Test Summary: 5 successful, 0 failures, 0 skipped

Looks ok, although we may not want the output showing the whole command, including the user and password used to access the database, so let us modify the integration test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# controls/integration.rb
control 'mariadb-integration' do
title 'Checks for proper DB connection'
sql = mysql_session(input('db_user'), input('db_password'), input('db_host'), input('db_port'))
 describe 'A simple SQL query' do
 subject do
 sql.query('show databases;')
 end
 it 'should execute successfully' do
 expect(subject.exit_status).to eq(0)
 end
 end
end

Here we refer to RSpec syntax and use the subject block to define our test and the expect method for the assertion, while using the it block to validate the subject and customize the output through the title. Let us run the profile again:

t0rrant@testing:~/database$ inspec exec . --input-file=files/example_input.yml --log-level=error
Profile: Basic Database (database)
Version: 1.0.0
Target: local://
✔ mariadb-compliance: Check for proper configuration
✔ System Package mariadb-server is expected to be installed
✔ User mysql is expected to exist
✔ File /etc/my.cnf is expected not to be more permissive than "644"
✔ Port 3306 is expected to be listening
✔ Port 3306 processes is expected to include "mysqld"
✔ mariadb-integration: Checks for proper DB connection
✔ A simple SQL command should execute successfully
Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped
Test Summary: 6 successful, 0 failures, 0 skipped

Nice, now we don't see anything sensitive and can safely show the output of the test.

Conclusion

In this post we saw how to create InSpec tests, and that we are able to test both configuration settings, and functionality using InSpec. We also saw how to customize a test's output so we can hide sensitive information.

The files described in this post are available here

Feel free to leave comments below, any improvement to this and other articles is always welcome.

Thanks for reading!

References

[1]	--, An Overview of Chef InSpec - Chef Web Docs [link]

[2]	(1, 2) --, About Chef InSpec Profiles - Chef Web Docs [link]

Linux Networking Introduction

Thu, 18 Jun 2020 23:00:00 +0100

this article is intended for an audience which does not necessarily have a Computer Science or Information Technology background and are curious about how things work. The goal is that by the end of the article, as someone just arriving to the Linux universe, you will be able to have basic understanding of how the Linux Operating System (OS) works, how you can connect two or more Linux systems through a network, simple best practices to keep your system safe, and other (maybe useful) tips. Take everything with a pinch of salt as I will be simplifying some of the topics.

Requirements

To fully follow this course, you should have at least two computers available. If you do not have a spare old computer gaining dust in the attic you can use VirtualBox (take a look at this guide on how to set it up) or use a similar software with either NAT Networking or Bridged network configuration.

We assume those machines have either Ubuntu 20.04 (Focal Fossa) or Debian 10 (Buster) installed. The requirements for either of them with a Desktop environment are:

2 GHz single core processor
2 GiB RAM (system memory)
10 GB of hard drive space

of course the more the merrier. Remember that these requirements are the recommended minimum for a usable Linux OS.

The OS installation itself is outside of the scope of this article, however if you are to follow the defaults you should at least be able to follow this course =)

Introduction

During the SARS-CoV-2 pandemic I was reached out by people wanting to start using Linux as an alternative everyday-use Operating System, most of them were familiar with Ubuntu or some other Debian-based variation, or by people who now had to interact with Linux systems. Eventually some reached a point where they had to open a command-line to follow a set of instructions they did not really understand. Among those, there were some who were curious enough to ask "what does this do?" or "why does this do what it does?". Those that made those questions eventually caught themselves delving deeper into Linux internals and how everything fits together.

This made me think it would be worthwhile to write a small course-like article introducing new users into this complex universe that is Linux.

I have given this article the title Linux Networking Introduction because in today's perspective what would we really do with a Linux system without networking? But mostly because the people that have a need to interact with a Linux system usually do it by means of a network connection.

That said, in this course we will go through:

A Bit of Linux History
An Overview of Linux Kernel Components
An Overview of the Linux Filesystem
User Authentication and Security
File Attributes and Ownership
Bash, your friendly command-line interpreter
Networking 101
Secure Communications and SSH
Setting up your machine for remote access

This is a fairly extensive article, so I invite you to take a break between sections, take some time to sink all in and try your own version of the examples.

Have a nice ride :-)

A Bit of Linux History

Before talking specifically about Linux we can go a bit further back to 1969 at AT&T Bell Labs, where Dennis Ritchie, Kenneth Thompson, and other colleagues began developing a small operating system for a very specific minicomputer. That operating system was named Unix as a result of cooperation between Bell Labs and the academic community, in 1979 the "seventh edition" (V7) version of Unix was released, the base for every still existing Unix system. [1]

In 1984, Richard Stallman's Free Software Foundation (FSF) began the GNU project, which had the objective of creating a freely used, read, modified and redistributed version of the Unix operating system. The FSF created useful (and still used today) components. However, the FSF had trouble with developing an operating system kernel to replace the Unix one (Hurd) and while they were developing it the Linux Kernel was released, filling the kernel gap in the GNU Project. [2]

In 1987, Andrew S. Tanenbaum developed MINIX, as a teaching tool for his textbook Operating Systems Design and Implementation published in the same year. This was used by many students taking introductory courses on computer operating systems, and one of them was Finnish student Linus Torvalds.

In 1991, Linus Torvalds started a new project, which would later be called Linux. After a couple of months working on it he reached out to the MINIX community for help, this marked what became the start of one of the biggest open-source software projects:

Hello everybody out there using minix -
I'm doing a (free) operating system (just a hobby, won't be big and
professional like gnu) for 386(486) AT clones. This has been brewing
since april, and is starting to get ready. I'd like any feedback on
things people like/dislike in minix, as my OS resembles it somewhat
(same physical layout of the file-system (due to practical reasons)
among other things).
I've currently ported bash(1.08) and gcc(1.40), and things seem to work.
This implies that I'll get something practical within a few months, and
I'd like to know what features most people would want. Any suggestions
are welcome, but I won't promise I'll implement them :-)
Linus (torv...@kruuna.helsinki.fi)
PS. Yes - it's free of any minix code, and it has a multi-threaded fs.
It is NOT protable (uses 386 task switching etc), and it probably never
will support anything other than AT-harddisks, as that's all I have :-(.

You can check the full conversation (plus more recent comments) in Google groups

8 years later, Linus Torvalds stated that Linux at the time had millions of users, thousands of developers, and a growing market. It is used in embedded systems; it is used to control robotic devices; it has flown on the space shuttle. I'd like to say that I knew this would happen, that it's all part of the plan for world domination. But honestly this has all taken me a bit by surprise I was much more aware of the transition from one Linux user to one hundred Linux users than the transition from one hundred to one million users. [3]

Tux - the Linux mascot

23 years later, Linux is now the world’s largest and most pervasive open source software project in the history of computing. The Linux Kernel is the largest component of the Linux operating system and is charged with managing the hardware, running user programs, and maintaining the security and integrity of the whole system [5] as we will see throughout this course.

An Overview of Linux Kernel Components

In Figure 2 we can see how different hardware parts connect to make your computer's architecture. As a bare minimum you should have a Central Processing Unit (CPU), some amount of Random Access Memory (RAM), some amount of storage in the form of a Hard Disk Drive (HDD) or Solid State Drive (SSD), and some peripherals (i.e: your monitor, keyboard and mouse). You can have more peripherals, different kinds of storage, or even more hardware connected to the Input/Output (I/O) Bus (like network cards and graphics cards), but you cannot have less in order for an operating system to function properly.

Computer architecture overview

The operating system is what makes every one of these components work together, and some at all.

In Figure 3 we can see an overview of the different layers of abstraction that make the Linux Kernel architecture.

The bottom layer (the first), we have already mentioned earlier, represents every component physically connected to your computer.

Linux Kernel architecture overview

The middle layer represents the Kernel Space, this is a very special layer as it can interact directly with all the hardware connected to your computer, it has the most privileges and can:

manage which process is running on the CPU, through Process Management
manage memory allocation for individual processes, and even memory shared between processes, through Memory Management
read and write storage devices through File Systems
draw images on your screen, send data through a network card, control a wide LED panel, or any other connected device through Device Drivers
interact with other devices connected to a network through Networking

The top layer, and the one you usually interact with, is User Space. Here is where we have System Applications that can perform special administrative tasks, User Applications like a browser or a music player, and Tools, which may go somewhere in between the previous two, available to all and may support applications, like software libraries.

In sum, in the Hardware layer is every device connected to you computer and every thing that makes your computer a computer. In the Kernel Space layer is every Kernel component that interacts directly with Hardware. In the User Space layer we have applications and tools that allow us to reach the Hardware, always going through the Kernel Space layer.

An Overview of the Linux Filesystem

When migrating from another operating system such as Microsoft Windows to another; one thing that will profoundly affect the end user greatly will be the differences between the filesystems. [7]

A filesystem is the methods and data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk. The word is also used to refer to a partition or disk that is used to store the files or the type of the filesystem. Thus, one might say I have two filesystems meaning one has two partitions on which one stores files, or that one is using the extended filesystem, meaning the type of the filesystem. [7]

The difference between a disk or partition and the filesystem it contains is important. A few programs (including, reasonably enough, programs that create filesystems) operate directly on the raw sectors of a disk or partition; if there is an existing file system there it will be destroyed or seriously corrupted. Most programs operate on a filesystem, and therefore won't work on a partition that doesn't contain one [7]

So we can look at the Linux Filesystem as a set of folders, or directories, that each have some associated function. They can contain configuration files necessary for a proper operation of the Linux operating system, they can contain applications (executable files), or they can contain arbitrary files such as our family photos or personal documents.

The most top-level directory, which contains every other directory, is called the root directory and is represented by a forward slash (/).

The following directories usually exist in / :

Directory	Description
/bin	Essential command binaries
/boot	Static files of the boot loader
/dev	Device files
/etc	Host-specific system configuration
/home	User home directories
/lib	Essential shared libraries and kernel modules
/media	Mount point for removable media
/mnt	Mount point for mounting a filesystem temporarily
/opt	Add-on application software packages
/proc	Process information pseudo-file system
/root	Home directory for the root user
/run	Operating system runtime information
/sbin	Essential system binaries
/srv	Data for services provided by this system
/tmp	Temporary files
/usr	Secondary hierarchy
/var	Variable data

For operating systems using the systemd init system some of these directories still exist for compatibility, however in another form. They "point" to their homonyms under /usr secondary hierarchy directory, these are:

/bin, /sbin/ which all point to /usr/bin
/lib which points to /usr/lib

For more information you can see systemd documentation regarding file system hierarchy.

User Authentication and Security

In order for the kernel to be able to identify who is trying to access, modify or execute anything in the system it has to have some type of authentication system. We will have two important categories, users and groups.

User information is usually stored in your filesystem in the file /etc/passwd. In this file each line represents a user and its information separated by a colon (:).

account:password:UID:GID:GECOS:directory:shell

Each field description can be seen in the table below:

Field	Description
account	The user's name. Should not contain uppercase letters
password	Encrypted user password or asterisks
UID	Numerical identification of the user
GID	Numerical identification of the user's primary group
GECOS	Optional field used for information purposes only. It usually contains the user's full name
directory	The user's home directory
shell	The command line interpreter to be used after login.

an example of a real user could be:

t0rrant:K3xcO1Qnx8LFN:1001:1001:Manuel Torrinha,,,:/home/t0rrant:/bin/bash

Group information is usually stored in your filesystem in the file /etc/group. In this file each line represents a group and, like the /etc/passwd file, its information separated by a colon (:).

group_name:password:GID:user_list

Each field description can be seen in the table below:

Field	Description
group_name	The name of the group
password	Encrypted group password. If empty, no password is needed
GID	The numeric group identification
user_list	a list of usernames that are members of this group, separated by commas (,)

an example of a real group could be:

ourgroup::132:t0rrant

With this information the kernel can establish who as access to what, as we will see how it works for files in the next section, File Attributes and Ownership.

Shadow Passwords

You may have noticed that instead of an encrypted password in your /etc/passwd file you have something like:

t0rrant:x:1001:1001:Manuel Torrinha,,,:/home/t0rrant:/bin/bash

The /etc/passwd file, which contains information about all users, including their encrypted password, is readable by all users, making it possible for any user to get the encrypted password of everyone on the system. Though the passwords are encrypted, password−cracking programs are widely available. To combat this growing security threat, shadow passwords were developed [8]

When shadow passwords are enabled the password field in the /etc/passwd file is replaced by an "x" and the user's actual password (encrypted) is stored in the /etc/shadow file. This file is only readable by the root user which makes it a safer place to keep such sensitive information.

File Attributes and Ownership

On a Linux system, every file is owned by a user and a group user. There is also a third category of users, those that are not the user owner and don't belong to the group owning the file. [11]

For each of these user categories, we can enable or disable read, write, or execute permissions.

The owner and group user for a file is represented by its identifier (id), but more commonly you will see the name corresponding to that id for readability.

The permissions for each user category can be defined in octal (8 digit system instead of 10), which conveniently maps to a three digit binary, meaning that we can easily represent every combination of rwx (read, write, execute) with a single octal digit, as we can see in the table below:

Octal	Binary	File Permission
0	000	---
1	001	--x
2	010	-w-
3	011	-wx
4	100	r--
5	101	r-x
6	110	rw-
7	111	rwx

Directories are also files in their essence, but they are distinguished by a special bit which is either on or off, and represented by d or -, respectively.

So if we have a representation like this

drwxr-xr-x t0rrant t0rrant Personal
-rw-r----- t0rrant ourgoup private_file

it means that the directory Personal can be read and executed by its owner t0rrant, everyone in the group t0rrant and everyone else, but can only be written by its owner t0rrant. In octal we would represent this as 755.

The file private_file can be read by its owner t0rrant and everyone in the group ourgoup but can only be written by its owner t0rrant. In octal we would represent this as 640.

Every directory that should have the read permission has to also have the executable permission, or else it will not be accessible.

Besides using the octal notation we can also use a character notation, which is sometimes easier to add or remove permissions from a file. This notation is used by defining one or more of user (u), group (g) or other (o) categories, followed by the plus (+) sign if we want to add, or minus (-) if we want to remove permissions, followed by the permissions we want to add or remove in rwx form.

So if we, for example, wanted to add just writable permissions to the file private_file for members of the group ourgroup we could do so by writting g+w. Seemingly, if we wanted to add executable permissions to the same file to the owner and group we would write ug+x.

File Types

In Windows systems you are probably used to distinguishing a file's type by its extension. If you have a file named myfile.pdf it represents a Portable Document Format (PDF) file, and Windows "knows" which application should open such a file because it has stored somewhere the relationship between file names ending in .pdf and the Adobe Reader application.

The same is not true for Linux systems. In fact, besides the file extension, the file itself has information of what type it is in the first bytes of the file. If we could take a look at the start of the PDF file as it were a normal text file we would see something like this:

 %PDF-1.4

Let us take a look at another file type, Portable Network Graphics (PNG) for example

 �PNG

That � character appears because the content cannot be converted to another readable character, which means that there is something there, just not in a usual readable format.

So we can see that files themselves have information of which type of file they are.

Bash, your friendly command-line interpreter

Bash is the shell, or command language interpreter, for the GNU operating system. The name is an acronym for the Bourne-Again SHell, a pun on Stephen Bourne, the author of the direct ancestor of the current Unix shell sh, which appeared in the Seventh Edition Bell Labs Research version of Unix. [6]

In Linux graphical environments, the case with our Ubuntu 20.04, we can use Bash by launching the Terminal application which is usually associated with the icon seen in Figure 4 or some variation of it.

Gnome Terminal icon

When interacting with such Terminal applications we are presented a prompt followed by a flashing cursor, which represents where the next character we input will appear. The prompt usually has the following appearance:

<current user name>@<machine name>:<current working directory>$

Most of you that have already interacted with a Linux system have probably copy & pasted some instructions to install an application or configure something in your system.

Now you will be able to better understand how those instructions work underneath.

Simple Commands

Here we will see some commands we can enter in Bash that interact with different Linux components.

User System

Identify the user running a command:

t0rrant@testing:~$ whoami
t0rrant
t0rrant@testing:~$

See to which groups a user belongs:

t0rrant@testing:~$ groups
t0rrant sudo
t0rrant@testing:~$

or a more broad user view:

t0rrant@testing:~$ id
uid=1001(t0rrant) gid=1001(t0rrant) groups=1001(t0rrant),27(sudo)
t0rrant@testing:~$ id root
uid=0(root) gid=0(root) grupos=0(root)
t0rrant@testing:~$

File System

List files and directories located in our current working directory:

t0rrant@testing:~$ ls
Desktop Documents Downloads Images Music Videos
t0rrant@testing:~$

or perhaps you have folders with other names, more folders, less folders...

It is also common to have coloured output, in which each colour represents a particular characteristic of the file. It may be that the file is a normal file (white), a directory (blue), an executable file (green), a link to another existing file (teal), a link to a non-existing file (red), a file that is accessible by any user (blue with a green highlight).

List files and directories for a specific location:

t0rrant@testing:~$ ls Images/
dontpanic.jpg
t0rrant@testing:~$

List files, including hidden files:

t0rrant@testing:~$ ls -a
. .bashrc Downloads .profile
.. Desktop Images Videos
.bash_history Documents Music
t0rrant@testing:~$

List files, and see more detailed properties:

t0rrant@testing:~$ ls -l
total 24
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Desktop
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Documents
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Downloads
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Images
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Music
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Videos
t0rrant@testing:~$

Create an empty file or change its modification time, if it exists:

t0rrant@testing:~$ touch myfile
t0rrant@testing:~$ ls -l
total 24
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Desktop
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Documents
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Downloads
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Images
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Music
drwxr-xr-x 2 t0rrant t0rrant 4096 May 13 23:58 Videos
-rw-rw-r-- 1 t0rrant t0rrant 0 May 14 00:05 temp
t0rrant@testing:~$

The echo command outputs whatever we pass to it:

t0rrant@testing:~$ echo Hello World
Hello World

This can be useful to, for example, add content to a file:

t0rrant@testing:~$ echo Hello World > temp
t0rrant@testing:~$

The cat command outputs a file contents:

t0rrant@testing:~$ cat temp
Hello World
t0rrant@testing:~$

The > character we used earlier redirects whatever would go to the output (standard output, or stdout) to a file, the file name is whatever comes after the >. This special character will overwrite the file contents entirely! So we can also use another special character, the >> which appends the content to the end of the file, i.e:

t0rrant@testing:~$ cat temp
Hello World
t0rrant@testing:~$ echo Hello World! > temp
t0rrant@testing:~$ cat temp
Hello World!
t0rrant@testing:~$ echo Goodbye World! >> temp
t0rrant@testing:~$ cat temp
Hello World
Goodbye World
t0rrant@testing:~$

We can also see a list of previously run commands:

t0rrant@testing:~$ history
1 whoami
2 groups
3 id
4 ls
5 ls -a
6 ls -l
7 touch myfile
8 ls -l
9 echo Hello World
10 echo Hello World > temp
11 cat temp
12 cat temp
13 echo Hello World! > temp
14 cat temp
15 echo Goodbye World! >> temp
16 cat temp
17 history

And run one of those commands again with the ! special command:

t0rrant@testing:~$ !1
whoami
t0rrant
t0rrant@testing:~$

We can also run the previous command by using the !! special command:

t0rrant@testing:~$ !1
whoami
t0rrant
t0rrant@testing:~$ !!
whoami
t0rrant
t0rrant@testing:~$

Man Pages

A very (very!) helpful command is man. This command allows you to read the documentation for a specific item, also known as man pages. It makes use of the less command to read those files. You can use the arrow keys to move around, use the h key to see the help screen or the q key to exit.

t0rrant@testing:~$ man ls
LS(1) User Commands LS(1)
NAME
ls - list directory contents
SYNOPSIS
ls [OPTION]... [FILE]...
DESCRIPTION
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is speci‐
fied.
Mandatory arguments to long options are mandatory for short options
too.
-a, --all
do not ignore entries starting with .
-A, --almost-all
do not list implied . and ..
--author
Manual page ls(1) line 1 (press h for help or q to quit)
t0rrant@testing:~$

There are even man pages for the man command, try it out!

Variables

A variable is a "word" you can assign a value to. Shell variables are defined in uppercase, by convention. Bash has two types of variables:

Global (or environment) variables
Local variables

Global variables are available in all Bash sessions, they can be displayed using the env command. These variables contain information that should be persistent across all shells, such as:

the user running the shell, USER
the name of the machine the shell runs in, HOSTNAME
the path for the current working directory, PWD
the path for the previous working directory, OLDPWD
the language used by the system, LANGUAGE
the path for the shell executable we are running, SHELL

Local variables are words specific to an execution environment. What this means is that if you define a variable in one Bash session, it is not available to another Bash session. They are also displayed with the env command, mixed with the environment variables.

Apart from dividing variables in local and global variables, we can also divide them in categories according to the sort of content the variable contains. In this respect, variables come in 4 types: [9]

String variables
Integer variables
Constant variables
Array variables

In order to simplify this subject we will focus on String and Integer variables only.

To assign a value to a variable, in a shell, we do the following:

t0rrant@testing:~$ MYVAR="myvalue"
t0rrant@testing:~$

We should not put spaces around the equal (=) sign, as it will cause the shell to error. It is also a good practice to always surround our String variables with quotes in order to reduce our chances of making errors.

To access a variable's value we use the dollar ($) sign in the form $VARNAME

t0rrant@testing:~$ echo $MYVAR
myvalue
t0rrant@testing:~$

In the case of local variables defining a variable in the form VARNAME=value make the variable only available to the current shell, but not to applications or processes that are child processes of that shell.

t0rrant@testing:~$ bash -c echo $MYVAR
t0rrant@testing:~$

In order to make variables available to child processes we need to use the export command.

t0rrant@testing:~$ export MYVAR="myvalue"
t0rrant@testing:~$ bash -c echo $MYVAR
myvalue
t0rrant@testing:~$

Besides defining and displaying variables, we can interact with variables in interesting ways.

We can concatenate strings:

t0rrant@testing:~$ HELLO="Olá"
t0rrant@testing:~$ WORLD="Mundo!"
t0rrant@testing:~$ HELLOWORLD="$HELLO $WORLD"
t0rrant@testing:~$ echo $HELLOWORLD
Olá Mundo!
t0rrant@testing:~$

We can do arithmetic operations with Integers by surrounding our operation with $(( ... )):

t0rrant@testing:~$ ONE=1
t0rrant@testing:~$ RESULT=$(($ONE+$ONE))
t0rrant@testing:~$ echo $RESULT
2
t0rrant@testing:~$

We can mix it all up:

t0rrant@testing:~$ ONE=1
t0rrant@testing:~$ RESULT=$(($ONE+$ONE))
t0rrant@testing:~$ QUESTION="One plus One"
t0rrant@testing:~$ echo $QUESTION is $RESULT
One plus One is 2
t0rrant@testing:~$

Shell arithmetic is complex and is not the focus of this section, if you would like to know more see the official documentation on the subject.

Special Variables

The shell treats several parameters specially. These parameters may only be referenced, assignment to them is not allowed. [9]

Variable	Value Expansion
$*	Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word with the value of each parameter separated by the first character of the IFS special variable.
$@	Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word.
$#	Expands to the number of positional parameters in decimal.
$?	Expands to the exit status of the most recently executed foreground pipeline.
$-	A hyphen expands to the current option flags as specified upon invocation, by the set built-in command, or those set by the shell itself (such as the -i).
$$	Expands to the process ID of the shell.
$!	Expands to the process ID of the most recently executed background (asynchronous) command.
$0	Expands to the name of the shell or shell script.
$_	The underscore variable is set at shell startup and contains the absolute file name of the shell or script being executed as passed in the argument list. Subsequently, it expands to the last argument to the previous command, after expansion. It is also set to the full pathname of each command executed and placed in the environment exported to that command. When checking mail, this parameter holds the name of the mail file.

Some of these variables are pretty useless for our previous interactions with the shell, however they are very useful for shell scripting, which we will take a look into in next section.

File Editing

One thing we have not covered yet is how to edit files. In Linux you have a lot of different options, be it a Graphical User Interface (GUI) or Command Line Interface (CLI) applications. If you talk to one user you will ear Vim, of course , it's very powerful, others will say Use Emacs, it's the best and most powerful!, well ... I say Use whatever tool you have the time to learn, and trust me when I say those two take some time to learn. I rather start off with something more simple and straight forward. However, if you would really like to know about those two and other CLI text editors you should check openvim and gnu pages.

In Ubuntu 20.04 you should have the following applications available:

nano (CLI)
gedit (GUI)

The most familiar for you to use is probably gedit as it is a windowed application where you can use your mouse to move around and scroll the file.

In this course I will focus on using the CLI, so we will be using nano in our examples.

nano is very simple, you execute the command with the name (or path) of the file you want to edit. If it exists, it is opened and you can edit it. If it does not, it is created.

Every command in nano is a combination of the CTRL key (indicated by ^ in the instructions) or the ALT key (indicated by M- in the instructions) followed by the shortcut key.

For example, let us create a file called test, write something in it, save it and exit.

t0rrant@testing:~$ nano test

our whole terminal is now another application (nano):

 GNU nano 4.8 test
|
^G Get Help ^O Write Out ^W Where Is ^K Cut Text ^J Justify ^C Cur Pos M-U Undo
^X Exit ^R Read File ^\ Replace ^U Paste Text ^T To Spell ^_ Go To Line M-E Redo

On the first screen line we have the application name and our file name. On the second line is where our file contents start, where we see a blinking cursor is just like the one in the terminal.

On the bottom we have some helpful commands. After we wrote something in our file we would save by pressing the CTRL+O combination:

 GNU nano 4.8 test Modified
Hello World!
File Name to Write: test|
^G Get Help M-D DOS Format M-A Append M-B Backup File
^C Cancel M-M Mac Format M-P Prepend ^T To Files

The cursor is now on the bottom, so we could save the file with another name if we wanted, so we press Enter to confirm, to which the application replies how many lines it wrote:

 GNU nano 4.8 test
Hello World!
[ Wrote 1 line ]
^G Get Help ^O Write Out ^W Where Is ^K Cut Text ^J Justify ^C Cur Pos M-U Undo
^X Exit ^R Read File ^\ Replace ^U Paste Text ^T To Spell ^_ Go To Line M-E Redo

Now we exit by pressing CTRL+X.

If we had not saved the file before exiting, nano would ask us if we wanted to save the file, and we would type Y or N and press Enter, which would be followed by a prompt to choose the filename if we typed Y.

Scripting in Bash

Like we saw in File Types, in Linux the beginning of a file is what tells us the type of the file and even what application the file is supposed to be executed with. For Bash scripts we always start the file with a shebang (#!) followed by the path to the env command and bash as its argument

#!/usr/bin/env bash

Imagine we want to create a file named test.sh which should display our username, relative path to itself, and the name of the machine it is running on, its content could be:

1
2


#!/usr/bin/env bash
echo I am $USER, and I just ran $0 at $HOSTNAME

After saving the file and closing nano we need to make that file executable (remember File Attributes and Ownership):

t0rrant@testing:~$ chmod 755 test.sh

And execute it, by indicating a path to the file:

t0rrant@testing:~$ ./test.sh
I am t0rrant, and I just ran ./test.sh at testing

Networking 101

Nowadays we have an assortment of devices connected to the Internet and most of us have something similar to Figure 5 in our homes, a wireless router provided by our Internet Service Provider (ISP) which connects our wired and wireless devices to the Internet.

Common household network infrastructure

Inside our home wireless router there is actually a combination of several components:

a network switch
a network modem
a network router
a DNS server (nameserver)

we will see the function for each of these components as we move along each subsection.

There are four key concepts I would like to present in this section:

Network Communications
Switching
Routing
Domain Name System (DNS)

Network Communication

Network devices communicate with each other in a way that resembles a very awkward and very formal conversation.

Comparing a network communication to a human conversation between Alice and Bob, it could go something like this:

*Alice initiates conversation*

Alice: Hello.
Bob: I acknowledge your message.
Alice: I acknowledge your acknowledgement.

*Now they can actually have a conversation*

Alice: Today is a nice day to learn something new.
Bob: I acknowledge your message.
Alice: I read this great article about cryptography.
Bob: I acknowledge your message.
Alice: Well I have to go to the grocery store to buy
Bob: I acknowledge your message, continue.
Alice: some cheese and bread.
Bob: I acknowledge your message.

*Alice terminates the conversation*

Alice: Goodbye.
Bob: I acknowledge your Goodbye.
Bob: Goodbye.
Alice: I acknowledge your Goodbye.

*End of conversation*

As you can see for every statement Alice makes, Bob acknowledges that statement, and there are two distinct protocols for initiating and terminating a conversation.

With network devices we need:

a way to identify a device
a way to reach a specific device
a way to represent a message
a way to represent which service in the device we want to deliver the message to

The first is called a Media Access Control (MAC) address, and is a unique identifier assigned to every Network Interface Controller (NIC).

The second is called an Internet Protocol (IP) address, and is a numerical assignment for a specific device. For simplicity we will only consider IPv4 addresses which are made up of four numbers, ranging from 0 to 254 and are separated by a dot (.) (i.e: 192.168.0.254).

The third is called a packet, and is what devices send to each other through the network. Each message exchanged between devices can be very small or very large, so these messages may need to be broken down into smaller packets, this is done by each network device.

The fourth is called a network port, and it usually identifies a process or network service waiting for messages (listening). This is a software port, a physical port is where physical cables connect to.

Imagine the IP address as being an address for a certain store in a mall, but it just identifies the building, the port identifies the number of the actual store, the actual endpoint where you can find your wanted service. You would not be going to an electronics store asking for beer right? =P

So, every network packet, should have a source IP address with its port and a destination IP address with its port in order to know where to exactly send the message to and where we expect to be replied to. This is very similar to traditional mail, just more complicated than sending a letter.

The MAC address association with an IP address also exists, but it is treated only within the local network and that is a whole other subject(s).

Switching

To filter and deliver these packets to its correct destination we have the network switch, which connects several network devices and "knows" where to which port to send each packet that arrives at another port.

In Figure 6, we can see an example of a switch with four connected devices. Each device will have a network interface connected to one of the switch's network ports (or the wireless interface) and the switch will know which IP address is associated with which MAC address. In example, if a packet arrives to the switch that has the TV's IP address as destination, only the TV will receive that packet.

Network Switch forming a network of devices

So, having a switch with connected devices defines a network between those devices.

Routing

In our home router we have a modem that connects to our ISP and gives us access to the Internet, but that modem will not do much more by itself, it is just a way to make a path to the ISP.

To have our packets being forwarded to the Internet we need the router.

The main functions of a router are:

connecting multiple networks
forwarding network packets

In Figure 7 we can see an example of two networks connected through the Internet. Every network that connects to the Internet will usually follow this setup.

Two networks connected by routers through the Internet

Our local router makes it possible to connect from our to a different network connected to another router, or a different network connected to the same router.

Sending a packet from our device with a local IP address to another device in a different network is made possible due to a process called Network Address Translation (NAT). I will not delve into it here, but broadly speaking it is a way for your router to hide your device information, exposing only itself in any forwarded packet, while at the same time maintaining a record of the origin for each forwarded packet in order to deliver any response to their proper recipient.

DNS

The Domain Name System (DNS) works somewhat like an old phone list. It is a dynamic set of servers that have records corresponding, among other record types, names to IPv4 addresses (Alias or A records). For example, the name implement.pt matches the A record corresponding to the IPv4 addresses:

104.24.106.10
104.24.107.10
172.67.136.196

which corresponds to three different entry points to access this name.

This record keeping is recursive, meaning that when you contact a certain DNS server it will search its own records for a match, and if it fails to find a match it will ask that information to another DNS server. This goes on until either a record is found, returning it back to the user, or no more DNS servers are available to ask to, in which case something like a "I do not know that record" is returned.

Taking a look at Figure 8 we can see an example of what happened when the device you are using to access this article tried to reach implement.pt.

DNS request flow example

First your device searched its internal records, and probably did not find anything.

Then it asked the next DNS Server, which is usually your router.

Your home router should also not have any records for implement.pt, and if so it asked its default DNS Servers for it. Usually these are your ISP's DNS servers. If they also do not have any record they will ask another, and another, until eventually some DNS server replied with the answer "implement.pt. A 104.24.107.10" and voilá your router can now pass that response to your device, which it did or else you would not be reading this =).

After your device has this information it can finally connect to the server at the correct address and initiate communication with this server at a specified port, in this case port 443.

Secure Communications and SSH

Encryption is the process of converting messages, information, or data into a form unreadable by anyone except the intended recipient. Encrypted data must be deciphered, or decrypted, before it can be read by the recipient. The root of the word encryption—crypt—comes from the Greek word kryptos, meaning hidden or secret. [15]

The first known form of encryption was used around 1900 BC! by an egyptian scribe using non-standard hieroglyphs, and it kept evolving from inscriptions in stone using signatures to authenticate its origin, to reversed-alphabet substitutions and other shifted substitutions. [15]

A very famous one is the Caesar Cipher, named after the roman emperor Julius Caesar, which used a simple substitution with the normal alphabet. For example, we could shift the alphabet left three places:

Plain: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Cipher: XYZABCDEFGHIJKLMNOPQRSTUVW

and if we wanted to send the message "ATTACK AT DAWN":

Plaintext: ATTACK AT DAWN
Ciphertext: XQQXZH XQ AXTK

we would send the message "XQQXZH XQ AXTK" and only who knew the cipher we were using would be able to know the correct message.

Symmetric vs Asymmetric Cryptography

Today’s crypto systems are divided into two categories: [15]

symmetric
asymmetric

Symmetric cryptography uses the a unique key (the secret key) to encrypt and decrypt a message. We can view this as a simple lock which only opens with a correct key, like we can see in Figure 9

Symmetric cryptography analogy

Asymmetric (or public key) cryptography uses one key (public key) to encrypt the message and a different key (private key) to decrypt the message. [15] One of the advantages is that there can be unending copies of the public key without compromising the private key.

By far the best non-technical explanation I have seen was shared by Panayotis Vryonis in his blog and I will share it here with you.

Picture a box with a lock, but this lock is special (see Figure 10). It fits two separate keys (yes, two). The first one can only turn clockwise (from A to B to C), let us call this the private key, and the second one can only turn counter-clockwise (from C to B to A), let us call this the public key. [10]

Asymmetric cryptography analogy

With this we can do two things to an unlocked box:

lock with the private key
lock with the public key

If we lock with the public key, only the person with the private key will be able to open it, and so anyone with the public key will be able to send secret items safely inside the box, knowing that only the owner of the private key will be able to access them.

If we lock the box with the private key, anyone that has the public key and can open the box can be sure that the box was locked by whoever possesses the private key counterpart, which makes them sure that whatever is inside the box it was placed by the one person holding the private key.

For example, Bob has one of these specials boxes, keeps one copy of the secret key for himself, and makes lots of copies of the public key. He then spreads those copies around, his house, office, random mailboxes, subway, and gives copies to all of his friends, including his close friend Alice.

Alice wants to send a secret message to Bob. She has a box with this special lock made by Bob. She can put a message inside the box and turn the lock to the locked position A. Then she can pass the box to anyone (that will eventually deliver the box to Bob, of course) sure that only Bob will be able to use his secret key to move the lock to the unlocked position B.

Now imagine that Alice receives a box from someone, saying it has a message from Bob. But how can Alice be sure it was Bob that sent it? Easily enough, if the public key that Bob gave her can open the box then the box surely came from Bob.

Secure Shell (SSH)

The SSH protocol (also referred to as Secure Shell) is a method for secure remote login from one computer to another. It provides several alternative options for strong authentication, and it protects the communications security and integrity with strong encryption. [12]

SSH protocol [12]

When using SSH, we are using the same concepts we saw earlier about asymmetric cryptography. As we can see in Figure 11, the SSH Client first contacts the SSH Server through an insecure communication channel (the network), so after the SSH Server sends the Client its public key, they use a really crafty way of having a common secret without anyone in that insecure channel being able to know what it is. [14]

In Figure 12 we can see an analogy of this ingenious process, of sharing secrets through an insecure channel, with colors instead of very large numbers (which is what roughly happens in a real-world interaction), this process is called the Diffie-Hellman Key Exchange.

Sharing a common secret through a non-secure channel [13]

The process begins by having the two actors, Alice and Bob, agree on one color that does not need to be kept secret. In this example, the color is yellow.

Each of them also has a secret color that they keep to themselves, in this case red for Alice, teal for Bob.

The crafty part of the process is that Alice and Bob each mix their own secret color together with their initially shared color (yellow), resulting in an orange-tan color for Alice and a light-blue mix for Bob.

They can now exchange the resulting mixed colors publicly, we assume that separating the mixed colors back to their original individual colors takes too long and is very expensive.

Finally, they each mix the color they received from the previous exchange with their own secret color.

The final result is a color mixture (yellow-brown in this case) that is identical for both.

If someone saw the exchange, they would only know the common color (yellow) and the first mixed colors (orange-tan and light-blue), but it would be extremely difficult for them to know the resulting secret color (yellow-brown).

Bringing the analogy back to a real-life exchange using large numbers instead of colors, this determination is computationally expensive. It is impossible to compute in a practical amount of time. [13]

Finally, after this "simple" key exchange the SSH Client and Server are ready to communicate securely.

Setting up your machine for remote access

Now that we have covered BASH interaction, file manipulation, networking and secure communications, we are ready to setup network communications between our two machines and perform interactions between them, in a secure manner.

For this section we will assume our workstation to be called testing and that it has the IPv4 address 192.168.1.10, and our other machine will be called remote-server and it has the IPv4 address 192.168.1.11.

We also assume that a user with the same name exists on both of those machines, in this case the user is t0rrant.

Hostname	IPv4 Address
testing	192.168.1.10
remote-server	192.168.1.11

To find out your real IPv4 addresses you can use the ip command, as such:

t0rrant@remote-server:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: enp3s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
inet 192.168.1.11/24 brd 192.168.1.255 scope global dynamic noprefixroute enp3s1
valid_lft 67476sec preferred_lft 67476sec

From the output above you should notice three things, as you should see something similar in your machine, the interface names are probably different and can be something like eth0, wlan1 or wlo1.

The interface name, which is after each number in the list:

1: lo
...
2: enp3s0
...
3: enp3s1
...

The interface status:

1: lo: ... state UNKNOWN ...
...
2: enp3s0: ... state DOWN ...
...
3: enp3s1 ... state UP ...
...

Interfaces with their state UP should be the ones that are connected, and thus have an IP address (inet):

1: lo: ... state UNKNOWN ...
...
2: enp3s0: ... state DOWN ...
...
3: enp3s1 ... state UP ...
...
inet 192.168.1.11/24 ...

The /24 portion, broadly speaking, defines the size of the network your machine is connected to. What you need is the first part, so in this case we can see that the IPv4 address for the enp3s1 interface in the remote-server is 192.168.1.11.

Server Application Setup

First let us install the server application:

t0rrant@remote-server:~$ sudo apt install openssh-client openssh-server

And now we can start the service

t0rrant@remote-server:~$ sudo systemctl start ssh

see that it is running

t0rrant@remote-server:~$ sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-05-11 15:24:03 WEST; 5s ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 9523 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 9524 (sshd)
Tasks: 1 (limit: 38302)
Memory: 1.4M
CGroup: /system.slice/ssh.service
└─9524 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
mai 11 15:24:03 testing systemd[1]: Starting OpenBSD Secure Shell server...
mai 11 15:24:03 testing sshd[9524]: Server listening on 0.0.0.0 port 22.
mai 11 15:24:03 testing sshd[9524]: Server listening on :: port 22.
mai 11 15:24:03 testing systemd[1]: Started OpenBSD Secure Shell server.

and from our workstation test the connection:

t0rrant@testing:~$ ssh t0rrant@192.168.1.11
t0rrant@odin's password:
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-29-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Mon May 11 15:25:44 WEST 2020
System load: 0.6
Usage of /: 22.3% of 9.32GB
Memory usage: 35%
Swap usage: 0%
Processes: 102
Users logged in: 0
IPv4 address for enp3s1: 192.168.1.11
0 updates can be installed immediately.
0 of these updates are security updates.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
t0rrant@remote-server:~$

SSH Keypair Setup

Entering our password every time we connect to a remote server is more than tedious, it opens the possibility for someone else to be able to "guess" our password. Of course we can find ways to remedy this, we can generate a random password with 20+ characters, we can setup defense mechanisms to "lock out" anyone that fails to enter the correct password some number of times, some other convoluted solution or we could gather the knowledge we gained from secure communications and SSH and make use of asymmetric cryptography not only to perform secure communications, but also to authenticate ourselves with the server.

Remember that section, and remember that if our "box" was locked with our public key, it can only be unlocked with our private key, so if the server would send a message encrypted with our public key we could only send back the same message if we have the correct private key. Essentially what we are doing is receiving one "box" locked with our public key, unlocking it with our private key, and finally we send a new "box" to the server locked with the server's public key, which we have because the server sent it to us on our first connect (as seen in Figure 11).

In Figure 13 we can see a sequence of this process.

SSH authentication with messages codified with public keys

So, let us get on to generating the keys for our special lock =)

When we installed openssh-client it also came with a couple of useful tools, one of them is ssh-keygen, which conveniently lets us generate a pair of keys:

t0rrant@testing:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/t0rrant/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/t0rrant/.ssh/id_rsa
Your public key has been saved in /home/t0rrant/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:ILvTwLzlzZyMS7YFflmMY+WxCG7BjkN4VCRg5tae3WI t0rrant@testing
The key's randomart image is:
+---[RSA 3072]----+
| +.ooo |
| + + o |
| + = = o |
| . * O = * o |
| X E S = |
| % X = |
| + B X |
| + = |
| o |
+----[SHA256]-----+

One other tool is ssh-copy-id, which adds our public key to a server, for a specific user (the one we connect with):

t0rrant@testing:~$ ssh-copy-id t0rrant@192.168.1.11
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 't0rrant@192.168.1.11'"
and check to make sure that only the key(s) you wanted were added.
t0rrant@testing:~$

Let us try a new connection from our testing workstation to our remote-server:

t0rrant@testing:~$ ssh t0rrant@192.168.1.11
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-29-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Mon May 11 15:34:40 WEST 2020
System load: 0.6
Usage of /: 22.3% of 9.32GB
Memory usage: 35%
Swap usage: 0%
Processes: 102
Users logged in: 0
IPv4 address for enp3s1: 192.168.1.11
0 updates can be installed immediately.
0 of these updates are security updates.
Last login: Mon May 11 15:25:44 2020 from 192.168.1.10
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
t0rrant@remote-server:~$

Notice that this time it did not ask us for our user's password, success :)

Now that we have access through our keypair, we can configure the server to not accept password based logins, only accept keypair based ones.

Open (as root) the /etc/ssh/sshd_config file for editing

t0rrant@remote-server:~$ sudo nano /etc/ssh/sshd_config

search for the options PubkeyAuthentication and PasswordAuthentication and change them to yes and no, respectively, as so:

PubkeyAuthentication yes
PasswordAuthentication no

make sure none of those lines start with a #, if they do remove it.

Save and exit by pressing CTRL+O and then CTRL+X.

Finally, restart the application

t0rrant@remote-server:~$ sudo systemctl restart ssh

do not lose or share your private key, and do not forget the passphrase!

Network Interactions

Now that we have a functional client-server connection, what can we really do with it?

First of all, we are getting pretty tired of typing the remote-server IPv4 address, it would be way better if we could simply do

ssh remote-server

and we can! Recalling the network name resolution in the previous section on DNS, when we search for the IPv4 address of a specific hostname, the Linux operating system does not go straight to the DNS servers asking for it. It actually has an internal name resolver which, among other things, reads information from a special file located at /etc/hosts. There we can define our own records, that can be (almost) whatever we want.

If you check its contents (for the sake of simplicity we will ignore IPv6 settings):

t0rrant@testing:~$ cat /etc/hosts
127.0.0.1 localhost
t0rrant@testing:~$

you can see that the name localhost is an alias for the 127.0.0.1 IPv4 address, which is that special address we discussed earlier in Networking 101 that represents our own machine.

As easily as that, we can add an entry for our remote-server, using nano, which leaves the file as such:

t0rrant@testing:~$ cat /etc/hosts
127.0.0.1 localhost
192.168.1.11 remote-server
t0rrant@testing:~$

Note that the hyphen (-) is the only non-alphanumeric character allowed in a hostname (for more information see the man pages for hostname)

And now we can connect to remote-server without typing its IPv4 address:

t0rrant@testing:~$ ssh t0rrant@remote-server
Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-29-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Mon May 11 15:56:12 WEST 2020
System load: 0.6
Usage of /: 22.3% of 9.32GB
Memory usage: 35%
Swap usage: 0%
Processes: 102
Users logged in: 0
IPv4 address for enp3s1: 192.168.1.11
0 updates can be installed immediately.
0 of these updates are security updates.
Last login: Mon May 11 15:34:40 2020 from 192.168.1.10
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
t0rrant@remote-server:~$

Besides connecting to our remote server and interacting with it from within a shell, we can also do other types of operations through SSH.

We can send a file:

t0rrant@testing:~$ scp test.sh remote-server:~/test.sh
test.sh 100% 278 1.2MB/s 00:00
t0rrant@testing:~$

We can retrieve a file:

t0rrant@testing:~$ scp remote-server:~/test.sh test2.sh
test.sh 100% 278 1.2MB/s 00:00
t0rrant@testing:~$

We can run commands remotely:

t0rrant@testing:~$ ssh remote-server ~/test.sh
I am t0rrant, and I just ran /home/t0rrant/test.sh at remote-server
t0rrant@testing:~$

Conclusion

There is no place like 127.0.0.1

In this course we touched several aspects of the Linux ecosystem. Its origins, what components make the Linux Kernel, learned how to use some of its tools, and culminated in communicating between two machines through a network.

These were a lot of concepts to take in at once, so I hope you took your breaks and took your time exploring some of the concepts on your own.

I invite you to explore further, remember that a using a search engine is a good way of finding information but sometimes the Man Pages are even friendlier.

Hopefully this was useful and you could follow and replicate the examples =)

Feel free to leave comments below, any improvement to this and other articles is always welcome.

Thanks for reading!

References

[1]	--, Secure Programming for Linux and Unix HOWTO - The Linux Documentation Project [link]

[2]	Richard Stallman, Linux and GNU - The GNU Project - FSF Foundation [link]

[3]	Linus Torvalds, The Linux edge - Communications of the ACM, Volume 42, Number 4 (1999), Pages 38-39 [link]

[4]	--, History of Linux - Wikipedia [link]

[5]	--, Linux - Linux Foundation [link]

[6]	--, Bash Reference Manual - GNU Software [link]

[7]	(1, 2, 3) --, Linux Filesystem Hierarchy - The Linux Documentation Project [link]

[8]	--, User Authentication HOWTO - The Linux Documentation Project [link]

[9]	(1, 2) --, Bash Guide for Beginners - The Linux Documentation Project [link]

[10]	Panayotis Vryonis, public-key cryptography for non-geeks [link]

[11]	--, Introduction to Linux, Chapter 3, The Linux Documentation Project [link]

[12]	(1, 2) Tatu Ylonen, SSH - Secure Login Connections over the Internet. Proceedings of the 6th USENIX Security Symposium, pp. 37-42, USENIX, 1996.

[13]	(1, 2) --, Diffie-Hellman key exchange - Wikipedia [link]

[14]	Whitfield Diffie and Martin E. Hellman, New Directions in Cryptography. IEEE Transactions on Information Theory, vol. IT-22, no. 6, November 1976. [link]

[15]	(1, 2, 3, 4) Melis Jacob, History of Encryption. Information Security Reading Room, SANS Institute, 2020.

Hacking diskimage-builder for Fun and Profit

Mon, 24 Feb 2020 02:27:00 +0100

Introduction

For the past year I have been responsible for redesigning and keeping a multi-region OpenStack deployment up to date. I will eventually go in-depth on what that entailed, but for now I will focus on my first challenge on that infrastructure, keeping the cloud images provided to the users up to date and customized according to defined local policies.

To achieve this, and after researching my choices, I chose diskimage-builder. It may seem overkill at first glance, but because of its modular architecture it called to my OCD nature and in the end you can make your own modules that can depend on existing modules, so yes it seemed perfect for the job. Besides, it is made by the Openstack team, so it is more than trustworthy.

Before I continue let me reword myself, this is used for generating stripped down Linux bootable cloud images, you can use them wherever you would like that supports booting the resulting filesystem. These images are cloud-ready and should be able to retrieve any metadata you configure it to fetch from your cloud provider.

At the end of this article you should be able to generate your weekly up to date cloud images in whatever CI solution you master =)

Requirements

I recommend having some background prior to delving in this use case, as it can get very complex when you need to debug more advance generation scenarios (i.e: CI). The cloud images provided by your favorite distribution should be enough to satisfy most cloud computing needs. This process helps in reducing the cloud image footprint to your bare needs, and allows you to customize it beyond the distribution's provided cloud images.

That said, I advise:

experience with chroot (I recommend reading the gentoo documentation)
experience with loop devices (specifically using losetup)
Python experience, especially regarding virtual environments
know your Linux File System (LFS)
know your way around manipulating BASH variables)
have minor experience with cloud-init
and as always, use Git

diskimage-builder

As stated in its documentation, diskimage-builder is a tool for automatically building customized operating-system images for use in clouds and other environments.

It includes support for building images based on many major distributions and can produce cloud-images in all common formats (qcow2, vhd, raw, etc), bare metal file-system images and ram-disk images. These images are composed from the many included elements; diskimage-builder acts as a framework to easily add your own elements for even further customization.

One of the main reasons I chose to use diskimage-builder was for its modular design, but I quickly discovered that it is also used extensively by the TripleO project and within OpenStack Infrastructure, which fit perfectly in our Openstack infrastructure.

Elements

"An element is a particular set of code that alters how the image is built, or runs within the chroot to prepare the image." [2]

"Images are built using a chroot and bind mounted /proc /sys and /dev. The goal of the image building process is to produce blank slate machines that have all the necessary bits to fulfill a specific purpose in the running of an OpenStack cloud. Images produce either a filesystem image with a label of cloudimg-rootfs, or can be customised to produce whole disk images (but will still contain a filesystem labelled cloudimg-rootfs). Once the file system tree is assembled a loopback device with filesystem (or partition table and file system) is created and the tree copied into it. The file system created is an ext4 filesystem just large enough to hold the file system tree and can be resized up to 1PB in size." [2]

Take this in... envision it, and now we will split the image process in its ten phases, which each element can go through if it needs to. [3]

root.d
extra-data.d
pre-install.d
install.d
post-install.d
post-root.d
block-device.d
pre-finalise.d
finalise.d
cleanup.d

root.d

Create or adapt the initial root filesystem content. This is where alternative distribution support is added, or customisations such as building on an existing image.

Only one element can use this at a time unless particular care is taken not to blindly overwrite but instead to adapt the context extracted by other elements.

runs:

outside chroot

inputs:

$ARCH=i386 OR amd64 OR armhf OR arm64

$TARGET_ROOT=/path/to/target/workarea

extra-data.d

Pull in extra data from the host environment that hooks may need during image creation. This should copy any data (such as SSH keys, http proxy settings and the like) somewhere under $TMP_HOOKS_PATH.

runs:

outside chroot

inputs:

$TMP_HOOKS_PATH

outputs:

None

Contents placed under $TMP_HOOKS_PATH will be available at /tmp/in_target.d inside the chroot.

pre-install.d

Run code in the chroot before customisation or packages are installed. A good place to add apt repositories.

runs:

in chroot

install.d

Runs after pre-install.d in the chroot. This is a good place to install packages, chain into configuration management tools or do other image specific operations.

runs:

in chroot

post-install.d

Run code in the chroot. This is a good place to perform tasks you want to handle after the OS/application install but before the first boot of the image. Some examples of use would be:

Run chkconfig to disable unneeded services
Clean the cache left by the package manager to reduce the size of the image.

runs:

in chroot

post-root.d

Run code outside the chroot. This is a good place to perform tasks that cannot run inside the chroot and must run after installing things. The root filesystem content is rooted at $TMP_BUILD_DIR/mnt.

runs:

outside chroot

block-device.d

Customise the block device that the image will be made on (for example to make partitions). Runs after the target tree has been fully populated but before the cleanup.d phase runs.

runs:

outside chroot

inputs:

$IMAGE_BLOCK_DEVICE={path}

$TARGET_ROOT={path}

outputs:

$IMAGE_BLOCK_DEVICE={path}

pre-finalise.d

Final tuning of the root filesystem, outside the chroot. Filesystem content has been copied into the final file system which is rooted at $TMP_BUILD_DIR/mnt. You might do things like re-mount a cache directory that was used during the build in this phase (with subsequent unmount in cleanup.d).

runs:

outside chroot

finalise.d

Perform final tuning of the root filesystem. Runs in a chroot after the root filesystem content has been copied into the mounted filesystem: this is an appropriate place to reset SELinux metadata, install grub bootloaders and so on.

Because this happens inside the final image, it is important to limit operations here to only those necessary to affect the filesystem metadata and image itself. For most operations, post-install.d is preferred.

runs:

in chroot

cleanup.d

Perform cleanup of the root filesystem content. For instance, temporary settings to use the image build environment HTTP proxy are removed here in the dpkg element.

runs:

outside chroot

inputs:

$ARCH=i386 OR amd64 OR armh OR arm64

$TARGET_ROOT=/path/to/target/workarea

cloud-init

Cloud-init is a method developed by Canonical for cross-platform cloud instance initialization. It is supported across major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations.

Cloud instances are initialized from a disk image and instance cloud metadata.

Cloud-init will identify the cloud it is running on during boot, read any provided metadata from the cloud and initialize the system accordingly. This may involve setting up the network and storage devices to configuring SSH access key and many other aspects of a system. [4]

You can see here the availability for different Linux distributions as well as compatibility for different cloud execution environments.

We should note that there are two main metadata types, EC2 and Openstack. They both have different API specifications but you can usually find support for Openstack metadata links across different public clouds, or even both, be sure to know your cloud provider!

We will not go much into cloud-init operation, just know that it exist and what purpose it serves. Of course, you are free to explore the documentation, just be careful with the rabbit-hole Alice =)

Creating our Image

For this article we will be building a CentOS 8 cloud-ready image.

CentOS has released its version 8 cloud images last month, conveniently the diskimage-builder developers already added the much needed changes to their centos-minimal element. Fortunately I noticed that the cloud-init package was missing from the generated image (I don't know if it is also missing from the upstream), so we should also add that package to our image, or else we will not have our precious ssh-key added to access any launched instances.

Setup

First we should get the tutorial files:

t0rrant@testing:~$ wget https://implement.pt/files/hacking-diskimage-builder-for-fun-and-profit/dib-tutorial.tar.gz
t0rrant@testing:~$ tar xzf dib-tutorial.tar.gz
t0rrant@testing:~$ cd dib-tutorial

Now we should create a python virtual environment using virtualenv wrapper and install diskimage-builder and its dependencies.

t0rrant@testing:~/dib-tutorial$ mkvirtualenv dib-elements
(dib-elements) t0rrant@testing:~/dib-tutorial$ pip install diskimage-builder

Project Structure

For modularity and learning purposes we will have two elements, centos-eight and my_base. The my_base element will contain settings that may be common between different images, be it CentOS, Debian, Ubuntu, Gentoo, or any others. Whereas the centos-eight will take care only of things specific to CentOS 8.

Let us start with the my_base element.

We should have a directory structure that looks something like this:

 my_base/
├── element-deps
├── environment.d
│   └── 10-cloud-init-datasources
├── package-installs.yaml
├── pkg-map
├── post-install.d
│   └── 01-cloud-init-override
└── README.rst

the element-deps file describes which other elements should be run alongside ours:

1
2
3
4
5
6
7
8
9


base
cloud-init-datasources
dhcp-all-interfaces
enable-serial-console
growroot
openssh-server
package-installs
pkg-map
vm

Besides the "mandatory" base and vm elements, we have some important elements that we do not need to configure.

growroot - grow the root partition on first boot [5]
dhcp-all-interfaces - autodetect network interfaces during boot and configure them for DHCP [6]
enable-serial-console - start getty on active serial consoles. [7]
openssh-server - ensures that openssh server is installed and enabled during boot [8]

Using the cloud-init-datasources element allows us to configure from where does cloud-init fetch metadata to feed to our instance, and the DIB_CLOUD_INIT_DATASOURCES environment variable must be set on image creation. One way of doing this is using the environment.d stage, and in our example we setup only the Openstack data source. Let us create the 10-cloud-init-datasources file inside environment.d:

1

 export DIB_CLOUD_INIT_DATASOURCES="OpenStack"

Using both package-installs and pkg-map elements gives us a flexible way to manage package installation across different distributions. We use pkg-map to map an arbitrary name to a specific package name in a specific distribution family, or release, and with package-installs we define which of those arbitrary names we want installed in our image. So we create the package-installs.yaml file in the root of our element:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


curl:
dnsutils:
git:
htop:
iptables:
man:
nano:
netcat:
ntp:
ping:
resolvconf:
syslog:
tree:
vim:
wget:

and the corresponding package mapping in the pkg-map file, in json format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


{
"release": {
"centos": {
"8": {
"ntp": "chrony"
}
}
},
"family": {
"redhat": {
"curl": "curl",
"dnsutils": "bind-utils",
"git": "git",
"htop": "htop",
"iptables": "iptables",
"man": "man-db",
"nano": "nano",
"netcat": "nc",
"ntp": "ntp",
"ping": "iputils",
"resolvconf": "",
"syslog": "rsyslog",
"tree": "tree",
"vim": "vim-enhanced",
"wget": "wget"
},
"debian":{
"curl": "curl",
"dnsutils": "dnsutils",
"git": "git",
"htop": "htop",
"iptables": "iptables",
"man": "man",
"nano": "nano",
"netcat": "netcat-openbsd",
"ntp": "ntp",
"ping": "inetutils-ping",
"resolvconf": "resolvconf",
"syslog": "rsyslog",
"tree": "tree",
"vim": "vim",
"wget": "wget"
}
}
}

Which in our case will map everything under ["family"]["redhat"] except for the ntp package, which we specified to be chrony for CentOS 8.

For consistency between different elements using the my_base element, we may want to override the cloud-init configuration that comes with each distro's cloud-init version. To do that we should use the post-install.d stage. Let us create the 01-cloud-init-override file in the post-install.d directory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


## DISCLAMER: this file serves as a consistency point for all cloud-init options, if you need to enable modules do it
# here. If you need to add options, use the 50-cloud-init-config or 99-cloud-init file in individual elements
tee /etc/cloud/cloud.cfg <<EOF
cloud_init_modules:
 - migrator
 - seed_random
 - bootcmd
 - write-files
 - growpart
 - resizefs
 - disk_setup
 - mounts
 - update_etc_hosts
 - resolv_conf
 - ca-certs
 - rsyslog
 - mounts
 - apt-configure
 - rh_subscription
 - yum-add-repo
 - package-update-upgrade-install
 - users-groups
 - ssh

cloud_config_modules:
 # Emit the cloud config ready event
 # this can be used by upstart jobs for 'start on cloud-config'.
 - emit_upstart
 - snap
 - snap_config # DEPRECATED- Drop in version 18.2
 - ssh-import-id
 - locale
 - set-passwords
 - grub-dpkg
 - apt-pipelining
 - ubuntu-advantage
 - ntp
 - timezone
 - puppet
 - chef
 - salt-minion
 - mcollective
 - disable-ec2-metadata
 - runcmd
 - byobu

cloud_final_modules:
 - fan
 - landscape
 - lxd
 - ubuntu-drivers
 - rightscale_userdata
 - scripts-vendor
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - ssh-authkey-fingerprints
 - keys-to-console
 - phone-home
 - final-message
 - power-state-change

users:
 - default

disable_root: true

EOF

Note: the files under post-install.d should have executable mode activated, or they will not be executed.

Now we can move on to our centos-eight element, here is our directory structure:

 centos-eight/
├── element-deps
├── package-installs.yaml
├── pkg-map
├── post-install.d
│   └── 50-cloud-init-config
└── README.rst

Taking a look at the dependencies (element-deps file) for our final element, centos-eight:

1
2
3
4
5


centos-minimal
my_base
epel
package-installs
pkg-map

we can see we are including the my_base element we created earlier, two new elements, and we are repeating the use for the package-installs and pkg-map. We include them again as we will want to specify in this element only a special case where we want the cloud-init package to be installed. We do not do this in the my_base element as other distributions may already have cloud-init installed, and they may or may not use the package manager, so we leave that option open.

Let us configure package-install.yaml for centos-eight:

1

 cloud-init:

and also the pkg-map:

1
2
3
4
5
6
7
8
9


{
"release": {
"centos": {
"8": {
"cloud-init": "cloud-init"
}
}
}
}

The centos-minimal element will create a minimal image based on CentOS. The use of this element will require ‘yum’ and ‘yum-utils’ to be installed if you are using Ubuntu or Debian where you generate the image. Nothing additional is needed on Fedora or CentOS. [9]

The epel element installs the Extra Packages for Enterprise Linux (EPEL) repository GPG key as well as configuration for yum. [10]

In this case we want to make sure some of cloud-init's options are added to our already overridden configuration, so we will use the same stage as in the my_base element, but with another priority so that it runs after, in this case we name the file 50-cloud-init-config within the post-instal.d directory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


yum-config-manager --save --setopt=updates.skip_if_unavailable=true
yum-config-manager --save --setopt=extras.skip_if_unavailable=true
tee --append /etc/cloud/cloud.cfg <<EOF
mount_default_fields: [~, ~, 'auto', 'defaults,nofail,x-systemd.requires=cloud-init.service', '0', '2']
resize_rootfs_tmp: /dev
ssh_deletekeys: 0
ssh_genkeytypes: ~
syslog_fix_perms: ~
ssh_pwauth: 0

system_info:
 default_user:
 name: centos
 lock_passwd: true
 gecos: CentOS
 groups: [wheel, adm, systemd-journal]
 sudo: ["ALL=(ALL) NOPASSWD:ALL"]
 shell: /bin/bash
 distro: rhel
 paths:
 cloud_dir: /var/lib/cloud
 templates_dir: /etc/cloud/templates
 ssh_svcname: sshd

ntp:
 enabled: true
 ntp_client: chrony
 conf:
 service_name: chronyd
 servers:
 - 0.europe.pool.ntp.org
 - 1.europe.pool.ntp.org
 - 2.europe.pool.ntp.org
 - 3.europe.pool.ntp.org

timezone: Europe/Lisbon

EOF

That is it for the centos-eight element and we are now (almost) ready to generate our customized cloud-ready image

Generating the image

Now that we have our elements setup, we can setup our environment variables:

we define where disk-image-create can find our custom elements
we define the main distro we are creating
we define the distro's release (this is relevant for *-minimal elements)

(dib-elements) t0rrant@testing:~/dib-tutorial$ export ELEMENTS_PATH=elements
(dib-elements) t0rrant@testing:~/dib-tutorial$ export DISTRO=centos
(dib-elements) t0rrant@testing:~/dib-tutorial$ export DIB_RELEASE=8

Finally we can create our image:

(dib-elements) t0rrant@testing:~/dib-tutorial$ disk-image-create -x --no-tmpfs -o centos8.qcow2 block-device-mbr centos-eight

We can now upload the generated image to our cloud platform, i.e:

(dib-elements) t0rrant@testing:~/dib-tutorial$ openstack image create my-centos-eight --file centos8.qcow2 --disk-format=qcow2 --container-format=bare

Pick up some java, wheat or malt, you've earned it! Then go play with the image you have just uploaded and see if you can customize it even further.

Conclusion

In this article we went through all of the components required to create a cloud-ready image (almost) from scratch. We saw what are elements defined in diskimage builder, and what phases they go through in the image building process. Talked briefly about cloud-init and metadata types, and finished with an in-depth tutorial of building a customized cloud-ready CentOS 8 image.

Hopefully this was useful and you could replicate every example =)

Feel free to leave comments below, any improvement to this and other articles is always welcome.

References

[1]	--, diskimage-builder - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[2]	(1, 2) -- , Developer Guide (Design) - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[3]	-- , Developer Guide (Developing Elements) - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[4]	-- , cloud-init Documentation, cloud-init 20.1. [link]

[5]	--, growroot element - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[6]	--, dhcp-all-interfaces element - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[7]	--, enable-serial-console element - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[8]	--, openssh-server element - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[9]	--, centos-minimal element - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

[10]	--, epel element - Openstack Documentation, diskimage-builder 2.33.1.dev11. [link]

Virtualenvwrapper Installation and Usage

Fri, 08 Mar 2019 14:50:00 +0100

Introduction

In a previous post we saw how we can create isolated python environments so that we have different python packages, python versions, or both, for specific projects, without messing with each other python environment.

Here we will take a look at a very useful tool, Doug Hellmann's virtualenvwrapper, which allows us to create, delete and overall manage our python virtual environments in a seamless way. Like its name suggests, it is essentially a wrapper for Ian Bicking’s virtualenv tool but we will see that it can make our python life much easier.

Installation

A simple way of obtaining virtualenvwrapper is to install it via pip. Depending on your Linux distribution, the virtualenvwrapper.sh script you will need to source may be located in somewhat unexpected locations, even through pip, because of the default location of the python site-packages, i.e:

Ubuntu 18.04:

root@ubu-ws ~# dpkg -S virtualenvwrapper | grep virtualenvwrapper.sh
virtualenvwrapper: /usr/share/virtualenvwrapper/virtualenvwrapper.sh
root@ubu-ws ~# apt remove -y virtualenvwrapper
root@ubu-ws ~# pip install virtualenvwrapper
root@ubu-ws ~# whereis virtualenvwrapper.sh
virtualenvwrapper: /usr/local/bin/virtualenvwrapper.sh
root@ubu-ws ~#

Debian 9

root@deb-ws ~# dpkg -S virtualenvwrapper | grep virtualenvwrapper.sh
virtualenvwrapper: /usr/share/virtualenvwrapper/virtualenvwrapper.sh
root@deb-ws ~# apt remove -y virtualenvwrapper
root@deb-ws ~# pip install virtualenvwrapper
root@deb-ws ~# whereis virtualenvwrapper.sh
virtualenvwrapper: /usr/local/bin/virtualenvwrapper.sh
root@deb-ws ~#

CentOS 7

[root@cen-ws ~]# repoquery -l python-virtualenvwrapper | grep virtualenvwrapper.sh
/etc/profile.d/virtualenvwrapper.sh
/usr/bin/virtualenvwrapper.sh
[root@cen-ws ~]# pip install virtualenvwrapper
[root@cen-ws ~]# whereis virtualenvwrapper.sh
virtualenvwrapper: /usr/bin/virtualenvwrapper.sh
[root@cen-ws ~]#

Keep in mind that it should be installed into the same global site-packages location where virtualenv is installed [1], and so you should install virtualenvwrapper the same way you installed virtualenv package, or just install virtualenvwrapper if you have not installed virtualenv previously.

~# pip install virtualenvwrapper
~# whereis virtualenvwrapper.sh
virtualenvwrapper: /usr/local/bin/virtualenvwrapper.sh
~#

Enable at Shell Startup

Note that on CentOS systems we get an extra script, /etc/profile.d/virtualenvwrapper.sh. For those of you not familiar with the /etc/profile file and the /etc/profile.d directory (take a look at the end of /etc/profile), these locations correspond to system-wide startup files used by the Bourne shell (aka Bash) and Bourne compatible shells (ksh, ash, ...), and are sourced by your shell when you:

login via TTY
login via SSH
launch a new shell

So, we should configure loading of the virtualenvwrapper in either

/etc/profile
/etc/profile.d/virtualenvwrapper.sh
~/.bashrc (user specific)
~/.profile (user specific)

and its content should be [1]:

1
2
3


export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Devel
source /usr/local/bin/virtualenvwrapper.sh

where WORKON_HOME is where your virtual environments will be stored, and PROJECT_HOME is the location of your development project directories (see [3]). The last line just loads all of the virtualenvwrapper methods.

After following the installation instructions above, you have to load the virtualenvwrapper methods, by launching a new shell, or if you already have a shell open you can source your startup file manually, i.e:

 ~$ . .profile

Usage

After loading the virtualenvwrapper.sh, you will have the following commands available:

Command Reference [2]
Command	Description
mkvirtualenv	Create a new named (specified) environment, in the WORKON_HOME directory
mktmpenv	Create a new named (automatically generated) environment in the WORKON_HOME directory
lsvirtualenv	List all of the environments
showvirtualenv	Show the details for a single virtualenv
rmvirtualenv	Remove an environment, in the WORKON_HOME
cpvirtualenv	Duplicate an existing virtualenv environment. The source can be an environment managed by virtualenvwrapper or an external environment created elsewhere.
allvirtualenv	Run a command in all virtualenvs under WORKON_HOME
workon	List or change working virtual environments
deactivate	Switch from a virtual environment to the system-installed version of Python
cdvirtualenv	Change the current working directory to $VIRTUAL_ENV
cdsitepackages	Change the current working directory to the site-packages for $VIRTUAL_ENV
lssitepackages	Shows the content of the site-packages directory of the currently-active virtualenv
add2virtualenv	Adds the specified directories to the Python path for the currently-active virtualenv
toggleglobalsitepackages	Controls whether the active virtualenv will access the packages in the global Python site-packages directory
mkproject	Create a new virtualenv in the WORKON_HOME and project directory in PROJECT_HOME
setvirtualenvproject	Bind an existing virtualenv to an existing project
cdproject	Change the current working directory to the one specified as the project directory for the active virtualenv
wipeenv	Remove all of the installed third-party packages in the current virtualenv
virtualenvwrapper	Print a list of commands and their descriptions as basic help output

In this post we will not go through every command, but we will go through a process similiar to the previous post to get you started. For more info about each command please see the official documentation

Managing your Virtual Environments

First let us see the command you will probably use the most workon that, when you do not pass any arguments, it shows a list of all the available virtual environments (located at WORKON_HOME)

~$ workon
~$

Well, we have not created any yet, so let us create a new one.

~$ mkvirtualenv AdS
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/t0rrant/.virtualenvs/AdS/bin/python2
Also creating executable in /home/t0rrant/.virtualenvs/AdS/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
(AdS) ~$ workon
AdS
(AdS) ~$

Looks good so let us install the sympy package for this environment

(AdS) ~$ pip install sympy
Collecting sympy
Downloading https://files.pythonhosted.org/packages/dd/f6/ed485ff22efdd7b371d0dbbf6d77ad61c3b3b7e0815a83c89cbb38ce35de/sympy-1.3.tar.gz (5.9MB)
100% |████████████████████████████████| 5.9MB 3.5MB/s
Collecting mpmath>=0.19 (from sympy)
Downloading https://files.pythonhosted.org/packages/ca/63/3384ebb3b51af9610086b23ea976e6d27d6d97bf140a76a365bd77a3eb32/mpmath-1.1.0.tar.gz (512kB)
100% |████████████████████████████████| 522kB 4.9MB/s
Building wheels for collected packages: sympy, mpmath
Building wheel for sympy (setup.py) ... done
Stored in directory: /home/t0rrant/.cache/pip/wheels/6c/59/86/478e3c0f298368c119095cc5985dedac57c0e35a85c737f823
Building wheel for mpmath (setup.py) ... done
Stored in directory: /home/t0rrant/.cache/pip/wheels/63/9d/8e/37c3f6506ed3f152733a699e92d8e0c9f5e5f01dea262f80ad
Successfully built sympy mpmath
Installing collected packages: mpmath, sympy
Successfully installed mpmath-1.1.0 sympy-1.3
(AdS) ~$

We can confirm that the sympy package is only available to this environment:

(AdS) ~$ python -m sympy.abc
(AdS) ~$ deactivate
~$ python -m sympy.abc
/usr/bin/python: No module named sympy
~$ workon AdS
(AdS) ~$

Assuming that we have a project called AdS, located in our PROJECT_HOME directory, we can associate the newly created environment to that project, via the setvirtualenvproject command [3]:

(AdS) ~$ cdvirtualenv
(AdS) ~/.virtualenvs/AdS$ setvirtualenvproject $PWD ~/Devel/AdS
Setting project for AdS to /home/t0rrant/Devel/AdS
(AdS) ~/.virtualenvs/AdS$ cdproject
(AdS) ~/Devel/AdS$ ls
AdS_sympy_test.py

As you can see we can now call cdproject which will take us to the project associated to the currently active virtual environment. Neat =)

Note: At this time (virtualenvwrapper v4.8.4) you can only have one project associated to one virtual environment (project foo should work with environment A), or one project associated with any number of projects (project bar should work with environments A, B, ..., Z), you cannot have one virtual environment associated with multiple projects. You can of course activate your desired virtual environment and use it within any directory, you will just not be able to use cdproject to change to the root directory of your project.

Just to check if the sympy is working in our environment:

1
2
3
4
5


# AdS_sympy_test.py
#!/usr/bin/env python
from sympy import test
test("_basic")

(AdS) ~/Devel/AdS$ python AdS_sympy_test.py
============================= test process starts ==============================
executable: /home/t0rrant/.virtualenvs/AdS/bin/python (2.7.15-candidate-1) [CPython]
architecture: 64-bit
cache: yes
ground types: python
numpy: None
random seed: 21534663
hash randomization: on (PYTHONHASHSEED=417663367)
sympy/core/tests/test_basic.py[16] ................ [OK]
================== tests finished: 16 passed, in 0.06 seconds ==================
(AdS) ~/Devel/AdS$

Now we already did everything with this virtual environment so we exit it:

(AdS) ~/Devel/AdS$ deactivate
~/Devel/AdS$

and we can delete the virtual environment as our project development is now finished:

~/Devel/AdS$ rmvirtualenv AdS
Removing AdS...
~/Devel/AdS$ workon
~/Devel/AdS$

And we are done!

Conclusion

In this post we saw how to install and use virtualenvwrapper to manage our Python virtual environments. We followed a development flow similar to a previous post about Python virtual environment, ending with a similar environment. The virtualenvwrapper tool is very versatile, and it is possible to extend it via plugins, which was not covered here. These extensions can allow you to, i.e: create new projects based on predefined templates, or interact with external tools, for more information take a look at the extensions page.

Like always, feel free to post any comments or questions in the comment box below.

References

[1]	(1, 2) --, Installation - virtualenvwrapper 4.8.4.dev1 Documentation. [link]

[2]	--, Command Reference - virtualenvwrapper 4.8.4.dev1 Documentation. [link]

[3]	(1, 2) --, Project Management - virtualenvwrapper 4.8.4.dev1 Documentation. [link]

Creating Custom Resources for Your Cookbooks

Thu, 07 Mar 2019 14:50:00 +0100

Introduction

Like we saw in the previous post, Chef Cookbooks can also provide custom resources that we can use in a dependant cookbooks' recipes. These custom resources can also use other resources from arbitrary cookbooks, or resources builtin to Chef.

A custom resource file is a Ruby file, ending in *.rb, and its syntax is, for example: [1]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


property :property_name, RubyType, default: 'value'
load_current_value do
# some Ruby for loading the current state of the resource
end
action :action_name do
# a mix of built-in Chef resources and Ruby
end
action :another_action_name do
# a mix of built-in Chef resources and Ruby
end

We will generally only have to worry about properties and actions. The load_current_value block can be used in more complex resources, to gather the current state of our resource, if it already exists, thus overriding any properties that we may want.

Development

Before starting to develop our custom resource we should create a regular cookbook, with the difference that our default recipe may not exist, and the recipes directory may not exist at all. This means that our resources cookbook has to have a proper metadata.rb file with proper versioning.

We should create a directory named resources, where our resource files reside. And very importantly remember that the resources made available by our cookbook are named after the cookbook's name and the resource's name. So, if our cookbook is named myapps and we have a resource file named superapp, to use the superapp resource we would call the myapps_superapp resource in the external cookbook.

Let us take as an example the nano text editor, an external Linux application that can be compiled manually and installed to arbitrary directories.

For this example we will call our resource file from_tar.rb and our cookbook pkg, so our directory structure should look something like:

pkg
├── Berksfile
├── metadata.rb
└── resources
   └── from_tar.rb

Our Berksfile is as simple as it gets:

1
2
3


source 'https://supermarket.chef.io'
metadata

And so is our metadata.rb file:

1
2
3
4
5


name 'pkg'
maintainer 'Manuel Torrinha'
maintainer_email 'torrinha at_ implement _dot pt'
description 'Installs tar and one resource to manage remote tar packages'
version '0.1.0'

For our resource we plan it first:

install tar
create extract directory
fetch package
extract package
compile and install

Then we implement it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


property :pkg_source, String, default: 'https://www.nano-editor.org/dist/v3/nano-3.2.tar.gz'
property :prefix, String, default: '/usr/local'
property :configure_flags, Array, default: []
property :creates, String
property :tmp_dir, String, default: '/tmp'
action :install do
basename = ::File.basename(new_resource.pkg_source)
dirname = basename.chomp('.tar.gz')
src_dir = "#{new_resource.tmp_dir}/#{dirname}"
# Install package dependencies
package 'tar'
package 'gcc'
package node['platform_family'] == 'debian' ? 'libncursesw5-dev': 'ncurses-devel'
# Ensure extract dir exists
directory src_dir do
owner 'root'
group 'root'
mode '0755'
recursive true
action :create
end
# Fetch nano to tmp_dir
remote_file basename do
source new_resource.pkg_source
path "#{src_dir}/#{basename}"
mode '0755'
action :create
end
# Unpack nano into src_dir
execute 'unpack nano' do
cwd src_dir
command "tar xfz #{basename}"
creates "#{src_dir}/#{dirname}"
end
# Configure and compile nano
execute 'compile and install' do
cwd "#{src_dir}/#{dirname}"
flags = [new_resource.prefix ? "--prefix=#{new_resource.prefix}" : nil, *new_resource.configure_flags].compact.join(' ')
command "./configure --quiet #{flags} && make -s && make -s install"
creates new_resource.creates
end
end

Note that when we want to use a property value, instead of using the variable name directly we use the namespace new_resource to access them.

We can use this sequence not just for nano, but for any application that:

comes packaged in a '.tar.gz' format
requires execution of configure, and only that, to generate a Makefile
its Makefile supports the install argument

This specific process is actually already implemented by the tar cookbook available at the supermarket.

Testing

Of course that now that we have written our custom resource we want to be able to test it.

Let us start by creating a test cookbook, inside our pkg resource cookbook, which will have a nano_install recipe that uses our resource.

First create the metadata.rb file:

1
2
3
4
5
6
7


name 'test'
maintainer 'Manuel Torrinha'
maintainer_email 'torrinha at_ implement _dot pt'
description 'Cookbook for testing the pkg cookbook'
version '0.1.0'
depends 'pkg'

Then, the recipes/nano_install.rb file:

1
2
3
4
5
6
7
8


pkg_from_tar 'nano' do
pkg_source 'https://www.nano-editor.org/dist/v3/nano-3.2.tar.gz'
prefix '/usr/local'
configure_flags []
creates '/usr/local/bin/nano'
tmp_dir '/tmp'
action :install
end

Next we update our original Berksfile to contemplate our test cookbook:

1
2
3
4
5


source 'https://supermarket.chef.io'
metadata
cookbook 'test', path: './test/cookbooks/test'

And finally we create a .kitchen.yml file for us to use with kitchen and test our resource in a local VM, in this case using Vagrant.

(Note that you will need vagrant and virtualbox installed on your workstation)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


driver:
name: vagrant
provisioner:
name: chef_zero
platforms:
- name: ubuntu-14.04
- name: ubuntu-16.04
- name: ubuntu-18.04
- name: debian-8
- name: debian-9
- name: centos-6
- name: centos-7
suites:
- name: nano
run_list:
- recipe[test::nano_install]

So now our directory structure will now look like this:

pkg
├── Berksfile
├── .kitchen.yml
├── metadata.rb
├── resources
|   └── from_tar.rb
└── test
└── cookbooks
└── test
├── metadata.rb
└── recipes
└── nano_install.rb

And we can now test it with kitchen:

~/pkg$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action Last Error
nano-ubuntu-1404 Vagrant ChefZero Busser Ssh <Not Created> <None>
nano-ubuntu-1604 Vagrant ChefZero Busser Ssh <Not Created> <None>
nano-ubuntu-1804 Vagrant ChefZero Busser Ssh <Not Created> <None>
nano-debian-8 Vagrant ChefZero Busser Ssh <Not Created> <None>
nano-debian-9 Vagrant ChefZero Busser Ssh <Not Created> <None>
nano-centos-6 Vagrant ChefZero Busser Ssh <Not Created> <None>
nano-centos-7 Vagrant ChefZero Busser Ssh <Not Created> <None>
~/pkg$ kitchen test debian-9
-----> Starting Kitchen (v1.24.0)
-----> Cleaning up any prior instances of <nano-debian-9>
-----> Destroying <nano-debian-9>...
Finished destroying <nano-debian-9> (0m0.00s).
-----> Testing <nano-debian-9>
-----> Creating <nano-debian-9>...
(...)
Thank you for installing Chef!
Transferring files to <nano-debian-9>
Starting Chef Client, version 14.11.21
Creating a new client identity for nano-debian-9 using the validator key.
resolving cookbooks for run list: ["test::nano_install"]
Synchronizing Cookbooks:
- test (0.1.0)
- pkg (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 1 resources
Recipe: test::nano_install
* pkg_from_tar[nano] action install
* apt_package[tar] action install (up to date)
* apt_package[gcc] action install (up to date)
* apt_package[libncursesw5-dev] action install
- install version 6.0+20161126-1+deb9u2 of package libncursesw5-dev
* directory[/tmp/nano-3.2] action create
- create new directory /tmp/nano-3.2
- change mode from '' to '0755'
- change owner from '' to 'root'
- change group from '' to 'root'
* remote_file[nano-3.2.tar.gz] action create
- create new file /tmp/nano-3.2/nano-3.2.tar.gz
- update content in file /tmp/nano-3.2/nano-3.2.tar.gz from none to ca6945
(new content is binary, diff output suppressed)
- change mode from '' to '0755'
* execute[unpack nano] action run
- execute tar xfz nano-3.2.tar.gz
* execute[compile and install] action run
- execute ./configure --quiet --prefix=/usr/local && make -s && make -s install
Running handlers:
Running handlers complete
Chef Client finished, 6/8 resources updated in 39 seconds
Downloading files from <nano-debian-9>
Finished converging <nano-debian-9> (1m3.59s).
-----> Setting up <nano-debian-9>...
Finished setting up <nano-debian-9> (0m0.00s).
-----> Verifying <nano-debian-9>...
Preparing files for transfer
Transferring files to <nano-debian-9>
Finished verifying <nano-debian-9> (0m0.00s).
-----> Destroying <nano-debian-9>...
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
Vagrant instance <nano-debian-9> destroyed.
Finished destroying <nano-debian-9> (0m4.10s).
Finished testing <nano-debian-9> (2m8.26s).
-----> Kitchen is finished. (2m9.01s)

You can test all of the flavors by not passing any arguments after kitchen test.

You can also use kitchen converge instead of test, so that the VM is not destroyed after successful convergence, and you can see the state of the VM after applying the run list.

Conclusion

In this post we saw how to design and implement custom resources for Chef, from scratch. We also saw how to test said resources using kitchen and Vagrant.

A working cookbook representing what was reviewed in this post is available here

Feel free to post any questions, or point out any mistakes, in the comment box below.

References

[1]	--, Custom Resources - Chef.io Documentation, 2018. [link]

Hitchhiker's Guide to Chef

Fri, 15 Feb 2019 13:37:00 +0100

Introduction

If you have read my previous posts on Salt (part1 and part2) you will now be reading about a very different approach on configuration management. If you did not read it, you should =) at least the first part, where I write about DevOps and Configuration Management tools in general, converging into Salt.

One of the key differences, from the conceptual point of view, is that Chef has a third element, the Workstation, from where the administrator performs administrative actions, according to predetermined Access Control Lists (ACLs).

Another key difference is the way the Minions (hereby called Nodes) are put into a certain state, which is determined by the Chef client, not by the Chef server, this is somehow useful as the Node applies itself its Run List even if it has lost contact to the Chef server. Salt is also capable of this via a schedule function, but this post is not that of comparison between tools, so I'll leave that for another time and get on to Chef's architecture and workflow for different actions.

Architecture

Chef is complex in terms of its architecture, where it resembles an umbrella organization. I will try to summarize each function, pass on to best practices in terms of interactions between them, and then go through some of its limitations by design.

Chef architecture overview

In Figure 1 we can see three of the most basic interactions, and we can even consider them by the following (oversimplified) order:

From a Chef Workstation, an admin bootstraps a machine
The machine registers itself with the Chef Server, and is now a Chef Node
The Chef Node synchronizes, with the Chef Server, any cookbook that applies to its run list, compiles its resource collection and ultimately converges the node (applies a computed set of recipes in a deterministic order)

And now, the Node is in a certain predetermined state, and will make an effort to stay that way, by periodically synchronizing its run list with the Chef Server and converging itself.

All of this is pretty simple stuff, so let us complicate it a bit by going through each of the components' functions.

Chef Workstation

The Chef Workstation can be configured by:

Installing ChefDK
Registering with the Chef Server:
- a user has to be created and a private_key generated, on the server side
- file ~/.chef/knife.rb should be present in the workstation and properly configured

Knife

knife is a command-line tool that provides an interface between a local chef-repo and the Chef server. knife helps users to manage [1]:

Nodes
Cookbooks and recipes
Roles, Environments, and Data Bags
Resources within various cloud environments
The installation of the chef-client onto nodes
Searching of indexed data on the Chef server

Chef.io makes available an all-in-one knife quick reference commands, you can either get the source files or get a ready to print PDF.

Cookbooks

In Chef, a cookbook is the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything that is required to support that scenario [2]:

Recipes that specify the resources to use and the order in which they are to be applied
Attribute values
File distributions
Templates
Extensions to Chef, such as custom resources and libraries

The reference language used to create cookbooks and defining recipes is Ruby. There is probably as much love for it as there is hate, in this post I will refrain about giving my personal opinion about it.

In this section the Cookbook should be seen as the development of the scenario itself and all properties that are a part of it.

Recipes

Taking word for word from Chef's documentation a recipe is the most fundamental configuration element within the organization [3].

It is mostly a collection of resources
Defines everything that is required to configure part of a system
Must be stored in a cookbook
May be included in another recipe
May include other recipes
Must be part of a run list in order to be executed
The recipe's code is interpreted sequentially, by the order in which it is declared
Resource execution may be delayed or expedited by notifying or be notified by another resource, thus creating an execution sequence different from the order the resources are declared

Attribute Values

Although attributes are used by the chef-client to understand certain node states, within a cookbook we can look at them has the state we want the node to be when we execute a specific recipe.

In a cookbook, attributes can be defined:

In attribute files, any <filename>.rb placed in the attributes directory just under our cookbook's main directory
In the middle of the recipe as we need them

With 1. we define attributes that are static and should not depend on data bag items values or any value that should only be known at runtime. This is because the node attributes are used in the creation of the node object and when this object is rebuilt the node attributes are updated based on attribute precedence, so it should not depend on values that come from other executions, i.e: fetch a value from a data bag or the value of the current dynamically assigned ip of a network interface created by the recipe.

The chef-client uses six types of attributes to determine the value that is applied to a node during a chef-client run [4]:

Attribute precedence, taken from Chef documentation
Attribute Type	Description
`default`	A `default` attribute is automatically reset at the start of every chef-client run and has the lowest attribute precedence. Use `default` attributes as often as possible in cookbooks.
`force_default`	Use the `force_default` attribute to ensure that an attribute defined in a cookbook (by an attribute file or by a recipe) takes precedence over a `default` attribute set by a role or an environment.
`normal`	A `normal` attribute is a setting that persists in the node object. A `normal` attribute has a higher attribute precedence than a `default` attribute.
`override`	An `override` attribute is automatically reset at the start of every chef-client run and has a higher attribute precedence than `default`, `force_default`, and `normal` attributes. An `override` attribute is most often specified in a recipe, but can be specified in an attribute file, for a role, and/or for an environment. A cookbook should be authored so that it uses `override` attributes only when required.
`force_override`	Use the `force_override` attribute to ensure that an attribute defined in a cookbook (by an attribute file or by a recipe) takes precedence over an `override` attribute set by a role or an environment.
`automatic`	An `automatic` attribute contains data that is identified by Ohai at the beginning of every chef-client run. An `automatic` attribute cannot be modified and always has the highest attribute precedence.

Besides attribute values behind provided from attribute files and recipes, they are also collected by Ohai at the start of each chef-client run, i.e: node['hostname'] which returns the hostname for the node, simple enough.

When defining attributes in a attribute file, we can ommit the node module, as such:

default['myapp']['dir'] = '/srv/myapp'

in recipes, however, we can not:

node.default['myapp']['dir'] = '/srv/myapp'

The precedence of the attributes is well documented

File Distributions

In a cookbook, files are managed using the following resources [5]:

cookbook_file resource manages files on the node, based on source files that are located in the files directory in a cookbook, using the source property (for more information on cookbook_file properties see the documentation):

cookbook_file '/etc/sshd_config' do
source 'sshd_config'
owner 'root'
group 'root'
mode '0640'
action :create
end

file resource manages file content directly on the node (for more information on file properties see the documentation):

file '/etc/resolv.conf' do
content 'nameserver 1.1.1.1
nameserver 1.0.0.1'
mode '0644'
owner 'root'
group 'root'
end

remote_file resource transfers a file from a remote location to the node (for more information on remote_file properties see the documentation):

remote_file '/tmp/wordpress-latest.tar.gz' do
source 'https://wordpress.org/latest.tar.gz'
owner 'root'
group 'root'
mode '0755'
action :create
end

template resource manages files that are added to nodes based on files that are located in the templates directory in a cookbook (for more information on template properties see the documentation):

template '/etc/resolv.conf' do
source 'resolv.conf.erb'
owner 'root'
group 'root'
mode '0644'
variables(nameservers: node['nameservers']) # optional
end

Templates

As you can see from the template resource in the File Distributions section, we can pass attributes as variables to our source file, so we can have files on the node generated dynamically according to properties relating to the node itself, cool =)

So taking that example, we could have a template called resolv.conf.erb with the following content:

1
2
3
4
5


# DNS resolver configuration managed by Chef
# do not edit, changes will be overridden!
<% @nameservers.each  do |ns| %>
nameserver <%= ns %>
<% end  %>

and in the default.rb attributes file we could have:

default['nameservers'] = ['1.1.1.1', '1.0.0.1', '8.8.8.8', '8.8.4.4']

when we execute the template resource defined above, in the node at the end of the run we would have the file /etc/resolv.conf with the following content:

1
2
3
4
5
6


# DNS resolver configuration managed by Chef
# do not edit, changes will be overridden!
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4

Extensions to Chef

In this post I will not go in-depth on this subject, just know that these exist:

Libraries	allow for arbitrary Ruby code to be included in a cookbook. The most common use for libraries is to write helpers that are used throughout recipes and custom resources. A library file is a Ruby file that is located within a cookbook’s `/libraries` directory. Because a library is built using Ruby, anything that can be done with Ruby can be done in a library file, including advanced functionality such as extending built-in Chef classes.(for more information see the documentation)
Custom Resources:	are an extension of Chef that adds your own resources; are implemented and shipped as part of a cookbook;are reusable in the same way as resources that are built into Chef (for more information see the documentation)
Resources:	are statements of configuration policies that: describe the desired state for a configuration item; declare the steps needed to bring that item to the desired state; specify a resource type -- such as package, template, or service; lists additional details (also known as resource properties), as necessary; are grouped into recipes, which describe working configurations (for more information see the documentation)

Metadata.rb

Last but definitely not least, the metadata.rb file is one of the fundamental files of a cookbook, every cookbook requires one and is created automatically when running knife cookbook create command on the Chef Workstation, but require a very small amount of metadata information. The file is located at the top of every cookbook directory structure, and its contents provide information that helps both the Chef Server and Client to correctly deploy cookbooks to each node [6].

metadata.rb options, adapted from Chef documentation
Setting	Required	Description
`name`	Yes	The name of the cookbook.
`version`	No	The current version of a cookbook. Version numbers always follow a simple three-number version sequence, i.e: `version '2.0.0'`.
`chef_version`	No	A range of chef-client versions that are supported by this cookbook. All version constraint operators are applicable to this field. i.e: `chef_version '~> 14'`
`depends`	No	This field requires that a cookbook with a matching name and version exists on the Chef server. When the match exists, the Chef server includes the dependency as part of the set of cookbooks that are sent to the node when the chef-client runs. It is very important that the depends field contain accurate data. If a dependency statement is inaccurate, the chef-client may not be able to complete the configuration of the system. All version constraint operators are applicable to this field.
`description`	No	A short description of a cookbook and its functionality.
`long_description`	No	A longer description that ideally contains full instructions on the proper use of a cookbook, including resources, libraries, dependencies, and so on. There are two ways to use this field: with the contents embedded in the field itself or with the contents pulled from a file at a specified path, such as a README.rdoc located at the top of a cookbook directory.
`gem`	No	Specifies a gem dependency for installation into the chef-client through bundler. The gem installation occurs after all cookbooks are synchronized but before loading any other cookbooks. Use this attribute one time for each gem dependency.
`source_url`	No	The URL for the location in which a cookbook’s source code is maintained. This setting is also used by Chef Supermarket. In Chef Supermarket, this value is used to define the destination for the “View Source” link.
`issues_url`	No	The URL for the location in which a cookbook’s issue tracking is maintained. This setting is also used by Chef Supermarket. In Chef Supermarket, this value is used to define the destination for the “View Issues” link.
`license`	No	The type of license under which a cookbook is distributed: Apache v2.0, GPL v2, GPL v3, MIT, or license 'Proprietary - All Rights Reserved (default). Please be aware of the licenses for files inside of a cookbook and be sure to follow any restrictions they describe.
`maintainer`	No	The name of the person responsible for maintaining a cookbook, either an individual or an organization.
`maintainer_email`	No	The email address for the person responsible for maintaining a cookbook. Only one email can be listed here, so if this needs to be forwarded to multiple people consider using an email address that is already setup for mail forwarding.
`supports`	No	Show that a cookbook has a supported platform. Use a version constraint to define dependencies for platform versions: < (less than), <= (less than or equal to), = (equal to), >= (greater than or equal to), ~> (approximately greater than), or > (greater than). To specify more than one platform, use more than one supports field, once for each platform.

Although the only mandatory option is the cookbook's name, I recommend to have at least the following example for a starting cookbook:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


name '<cookbook name>'
license '<license type>'
maintainer '<first and last name>'
maintainer_email '<your e-mail>'
issues_url '<github url>/issues'
source_url '<github_url>'
description 'small description'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '0.0.1'
chef_version '~> 14.0'

This metadata template is almost supermarket ready, missing just the cookbook's dependencies and Operating System (OS) support, so I think it's a great start.

Linting

Linters are very important in a development workflow, a linter verifies a program or program segments against standard libraries. It checks the code for common portability errors. It tests the programming against some tried and true guidelines. linting your code is a necessary (though not sufficient) step in writing clean, portable, effective programs. But lint is not perfect. It will not magically salvage bad code. It will not find all your bugs. [7] I certainly do not have anything to add to this sentence besides the tools we should use for our cookbooks:

cookstyle
foodcritic (deprecated since 25/09/2020)

Tests

According to Chef's documentation, we can use ChefSpec to simulate the convergence of resources on a node:

Is an extension of RSpec, a behavior-driven development (BDD) framework for Ruby
Is the fastest way to test resources and recipes

ChefSpec is a framework that tests resources and recipes as part of a simulated chef-client run. ChefSpec tests execute very quickly. When used as part of the cookbook authoring workflow, ChefSpec tests are often the first indicator of problems that may exist within a cookbook. [8]

It does not make sense to make tests to ensure that, i.e: a package was really installed or the contents of a template file are up to date as the recipes that apply those things already do them. It is however sane to write tests reflecting a set of operations that should be applied to the node according to the current stable state of the recipes, this assures that if some regression, i.e: a wrong file is changed or a package dependency is no longer available, the recipe(s) will fail some test and you will notice that something is wrong before converging a node.

Compliance

InSpec is a free and open-source framework for testing and auditing your applications and infrastructure. InSpec works by comparing the actual state of your system with the desired state that you express in easy-to-read and easy-to-write InSpec code. InSpec detects violations and displays findings in the form of a report, but puts you in control of remediation. [9]

It can be integrated with Chef, but that is out of the scope of this post. Maybe another day.

Kitchen

Kitchen is a tool that allows us to test infrastructure code on one or more platforms in isolation.

"A driver plugin architecture is used to run code on various cloud providers and virtualization technologies such as Vagrant, Amazon EC2, and Docker.

For Chef workflows, cookbook dependency resolution via Berkshelf or Policyfiles is supported or include a cookbooks/ directory and Kitchen will know what to do.

Kitchen is used by all Chef-managed community cookbooks and is the integration testing tool of choice for cookbooks."

We specify every requirement for kitchen in .kitchen.yml file usually located in the root directory of our coookbook, but really kitchen only cares about having that file present in whatever directory you are calling it.

An example file for a .kitchen.yml configuration file running on Openstack would be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


driver:
name: openstack
openstack_auth_url: <%= ENV['OS_AUTH_URL'] %>
openstack_domain_name: <%= ENV['OS_USER_DOMAIN_NAME'] %>
openstack_region: <%= ENV['OS_REGION_NAME'] %>
openstack_project_name: <%= ENV['OS_PROJECT_NAME'] %>
openstack_project_id: <%= ENV['OS_PROJECT_ID'] %>
openstack_username: <%= ENV['OS_USERNAME'] %>
openstack_api_key: <%= ENV['OS_API_KEY'] %>
flavor_ref: <%= ENV['OS_FLAVOR'] %>
key_name: <%= ENV['OS_KEYPAIR_NAME'] %>
provisioner:
name: chef_zero
data_bags_path: "data_bags"
encrypted_data_bag_secret_key_path: "secrets/encrypted_data_bag_secret"
transport:
ssh_key: .<%= ENV['OS_KEYPAIR_NAME'] %>.pem
verifier:
name: inspec
platforms:
- name: ubuntu-18
transport:
username: ubuntu
driver:
image_ref: Ubuntu-18.04-Latest
server_name_prefix: ubuntu-bionic
- name: debian-9
transport:
username: debian
driver:
image_ref: Debian-9-Latest
server_name_prefix: debian-stretch
suites:
- name: defaults
run_list:
- recipe[mycookbook::predefault]
- recipe[mycookbook::default]
- recipe[mycookbook::postdefault]
verifier:
inspec_tests:
- test/integration/predefault
- test/integration/default
- test/integration/postdefault
attributes:
- name: verytastyrun
run_list:
- recipe[mycookbook::saltandpepper]
verifier:
inspec_tests:
- test/integration/salt
- test/integration/pepper
- test/integration/spicy
attributes:

In this configuration we have a 2 suites:

defaults
verytastyrun

We use the Openstack driver, and so we have to configure our environment:

Property Name	Environment Variable	Description
openstack_auth_url	OS_AUTH_URL	URL for the Openstack stackcontroller
openstack_domain_name	OS_USER_DOMAIN_NAME	name of the Openstack domain we will be authentication with
openstack_region	OS_REGION_NAME	Openstack region where we want to launch our instances
openstack_project_name	OS_PROJECT_NAME	The project where we want to launch the instances (note: this is not enough for auth)
openstack_project_id	OS_PROJECT_ID	The project ID
openstack_username	OS_USERNAME	Openstack username to authenticate with
openstack_api_key	OS_API_KEY	API Key corresponding to the previous username

We also configure the environment variable OS_KEYPAIR_NAME which should correspond to the name of the keypair created in Openstack with our OS_USERNAME account, and the private key should be located in .OS_KEYPAIR_NAME.pem in the root directory of our cookbook, as defined by the transport['ssh_key'] yaml key.

Next we specify which provisioner to use, in our case we are testing a chef cookbook so we chose chef_zero. Here we can specify configuration options to be passed to chef, in this example we define options for data bags.

We then define which platforms should be converged, which should be available in Openstack. We can define a prefix for the instance name with the server_name_prefix property.

Finally, in the suites section we define which convergence run lists should be made available to kitchen. In this example we have two:

defaults
verytastyrun

In the run_list property we should specify from which cookbook we want to get each recipe.

A usual command sequence to converge a suite and destroy the converged instances would be:

localhost:~$ kitchen converge <platform name>
localhost:~$ kitchen destroy <platform name>

If, however, we would like to just create one of the machines, i.e: debian, login into it, do some manual testing and then converge the machine we would:

localhost:~$ kitchen create debian
localhost:~$ kitchen login debian
debian-stretch-asdfgh:~$ logout
localhost:~$ kitchen converge debian
localhost:~$ kitchen destroy debian

Chef Server

We have now arrived at the core component of our infrastructure, the Chef Server. In this section I will present, to some extend, the capabilities of the Chef Server and you to interact with it.

Interaction with Chef's API may be done via GET/POST HTTP methods, via the knife tool via plugins, built-in or custom made, or from within the Chef Server itself via the chef-server-ctl. In this section I will focus on operations with the chef-server-ctl tool, however you can assume that every command is similar in syntax with the knife tool.

You can interact with virtually any component of the Chef Server via its API, so I will not post here all of the operations, instead you can check them out in the documentation.

The first thoughts about any of the following components should be "What can I manage and how can I manage it?", so let us get on with the individual components of the Chef Server, what we can manage of each component and how do we do it.

Users

Like any architecture which has an authentication component, we have users. What can we do with it?

Description	Command
Create a user	`$ chef-server-ctl user-create USER_NAME FIRST_NAME [MIDDLE_NAME] LAST_NAME EMAIL 'PASSWORD' [-f USER_NAME.pem]`
Delete a user	`$ chef-server-ctl user-delete USER_NAME`
Edit a user's details	`$ chef-server-ctl user-edit USER_NAME`
View a list of users	`$ chef-server-ctl user-list [--with-uri]`
Show a user's details	`$ chef-server-ctl user-show USER_NAME [--with-orgs]`

Organizations

An organization is the top-level entity for role-based access control in the Chef server. Each organization contains the default groups (admins, clients, and users, plus billing_admins for the hosted Chef server), at least one user and at least one node (on which the chef-client is installed). The Chef server supports multiple organizations. The Chef server includes a single default organization that is defined during setup. Additional organizations can be created after the initial setup and configuration of the Chef server. [10]

Description	Command
Create an organization	`$ chef-server-ctl org-create ORG_NAME "ORG_FULL_NAME" [--association_user USER_NAME] [-f FILE_NAME]`
Delete an organization	`$ chef-server-ctl org-delete ORG_NAME`
View a list of organizations	`$ chef-server-ctl org-list [--all-orgs] [--with-uri]`
Show an organization's details	`$ chef-server-ctl org-show ORG_NAME`
Add a user to an organization	`$ chef-server-ctl org-user-add ORG_NAME USER_NAME --admin`
Remove a user from an organization	`$ chef-server-ctl org-user-remove ORG_NAME USER_NAME`

Groups

A group is used to define access to object types and objects in the Chef server and also to assign permissions that determine what types of tasks are available to members of that group who are authorized to perform them. Groups are configured per-organization. [10]

Individual users who are members of a group will inherit the permissions assigned to the group. The Chef server includes the following default groups: admins, clients, and users. For users of the hosted Chef server, an additional default group is provided: billing_admins. [10]

There is another group named server-admins, it is a global group that grants its members permission to create, read, update, and delete user accounts, with the exception of superuser accounts. The server-admins group is useful for users who are responsible for day-to-day administration of the Chef server, especially user management via the knife user subcommand. Before members can be added to the server-admins group, they must already have a user account on the Chef server. [10]

Description	Command
Add a user to the `server-admins` group	`$ chef-server-ctl grant-server-admin-permissions USERNAME`
Remove a user from the `server-admins` group	`$ chef-server-ctl remove-server-admin-permissions`
List users that belong to the `server-admins` group	`$ chef-server-ctl list-server-admins`

Group administration is done via chef-web, chef-automate or the hosted chef management interface.

Cookbooks

We have already covered what a Cookbook are. In this section the Cookbook should be seen not as the development of the scenario itself but as one or more versions of a already developed scenario which each version should be frozen. Meaning that if any changes should occur to a certain version of the cookbook, the cookbook is no longer on that version. Although not mandatory, this practice makes it harder to break something that depends on a cookbook's version.

To manage cookbooks present on the Chef Server we can perform the following operations, using the knife and chef commands:

Description	Command
Create a new cookbook	`$ chef generate cookbook COOKBOOK_PATH/COOKBOOK_NAME [<options>]`
Delete one or more cookbooks	`$ knife cookbook delete COOKBOOK_NAME [COOKBOOK_VERSION] [<options>]`
Delete one or more cookbook, using a regular expression	`$ knife cookbook bulk delete REGEX -p`
Download a cookbook from the Chef Server to the current working directory	`$ knife cookbook download COOKBOOK_NAME [COOKBOOK_VERSION] [<options>]`
View a list of cookbooks currently available on the Chef Server	`$ knife cookbook list [--all] [--with-uri]`
Generate the metadata for one or more cookbooks.	`$ knife cookbook metadata [<options>]`
Load the metadata for a cookbook from a file.	`$ knife cookbook metadata from file FILE`
View information about a cookbook, parts of a cookbook, or a file associated with it	`$ knife cookbook show COOKBOOK_NAME [COOKBOOK_VERSION] [PART...] [FILE_NAME] (options)`
Test a cookbook for syntax errors	`$ knife cookbook test COOKBOOK_NAME [<options>]`
Upload one or more cookbooks from a local repository to the Chef server	`$ knife cookbook upload [COOKBOOK_NAME...] [<options>]`

Nodes

To manage nodes that exist on a Chef Server we either use the knife command or the web management interface.

Run-lists

A run-list defines all of the information necessary for Chef to configure a node into the desired state. A run-list is [11]:

An ordered list of roles and/or recipes that are run in the exact order defined in the run-list; if a recipe appears more than once in the run-list, the chef-client will not run it twice
Always specific to the node on which it runs; nodes may have a run-list that is identical to the run-list used by other nodes
Stored as part of the node object on the Chef server
Maintained using knife and then uploaded from the workstation to the Chef server, or maintained using Chef Automate

Policies

Policy maps business and operational requirements, process, and workflow to settings and objects stored on the Chef server [12]:

Roles define server types, such as “web server” or “database server”
Environments define process, such as “dev”, “staging”, or “production”
Certain types of data - passwords, user account data, and other sensitive items - can be placed in data bags, which are located in a secure sub-area on the Chef server that can only be accessed by nodes that authenticate to the Chef server with the correct SSL certificates
The cookbooks (and cookbook versions) in which organization-specific configuration policies are maintained

Environments

An environment is a way to map an organization’s real-life workflow to what can be configured and managed when using Chef server. Every organization begins with a single environment called the _default environment, which cannot be modified (or deleted). Additional environments can be created to reflect each organization’s patterns and workflow. For example, creating production, staging, testing, and development environments. Generally, an environment is also associated with one (or more) cookbook versions. [13]

Roles

A role is a way to define certain patterns and processes that exist across nodes in an organization as belonging to a single job function. Each role consists of zero (or more) attributes and a run-list. Each node can have zero (or more) roles assigned to it. When a role is run against a node, the configuration details of that node are compared against the attributes of the role, and then the contents of that role’s run-list are applied to the node’s configuration details. When a chef-client runs, it merges its own attributes and run-lists with those contained within each assigned role. [14]

Attributes

Previously we saw what are attributes and how to define them in a cookbook. However, attributes can also be defined in:

The state of the Node itself
Roles
Environments

We also saw which types of precedence exist, but taking this new information that Roles, Environments, and Node state play a part on this we can view the attribute precedence as a table, as shown in {F#fig-chef-preced}:

Chef attributes overview

For more information on attributes take a look at [4].

Data Bags

Data bags store global variables as JSON data. Data bags are indexed for searching and can be loaded by a cookbook or accessed during a search. [15]

We will usually use the knife tool to manage data bags, although creating data bags manually can be useful to test cookbooks with Kitchen.

Manage data bags using `knife`
Description	Command
Create an empty data bag (or an item)	`$ knife data bag create DATA_BAG_NAME (DATA_BAG_ITEM)`
Create a data bag item a file	`$ knife data bag from file BAG_NAME ITEM_NAME.json`
Edit a data bag item	`$ knife data bag edit admins charlie`
List existing data bags	`$ knife data bag list`
List items from a data bag	`$ knife data bag show DATA_BAG_NAME`
Show contents of a data bag item	`$ knife data bag show DATA_BAG_NAME ITEM_NAME`
Search query for data bag items	`$ knife search DATA_BAG_NAME QUERY_STRING`
Delete a data bag (or an item)	`$ knife data bag delete DATA_BAG_NAME (DATA_BAG_ITEM)`

In order to use manually created data bags in our recipes we can create a directory named data_bags that follows a simple directory/file structure, i.e:

data_bags/
├── db_passwords
│   ├── mysqlroot.json
│   └── slurm.json
├── secrets
│   └── munge.json  
├── service_passwords
│   └── myservice.json
└── user_passwords
├── alice.json
└── bob.json

This way every subdirectory within the data_bags directory corresponds to a data bag, and every .json file corresponds to an item to each of those data bags.

Each data bag item should also follow a specific structure: [15]

1
2
3
4
5
6


{
/* This is a supported comment style */
// This style is also supported
"id": "ITEM_NAME",
"key": "value"
}

Encrypted Data Bags

A data bag item may be encrypted using shared secret encryption. This allows each data bag item to store confidential information (such as a database password) or to be managed in a source control system (without plain-text data appearing in revision history). Each data bag item may be encrypted individually; if a data bag contains multiple encrypted data bag items, these data bag items are not required to share the same encryption keys. [15]

the knife command can encrypt and decrypt data bag items, using the create, edit, from file or show arguments and the options:

Option	Description
`--secret SECRET`	The encryption key that is used for values contained within a data bag item. If `SECRET` is not specified, the chef-client looks for a secret at the path specified by the `encrypted_data_bag_secret` setting in the client.rb file
`--secret-file FILE`	the path to the file that contains the encryption key

To create the secret key, one can use OpenSSL to generate a random number to be used as a secret key:

$ openssl rand -base64 512 | tr -d '\r\n' > encrypted_data_bag_secret

Chef Node

A node is any machine - physical, virtual, cloud, network device, etc. - that is under management by Chef.

The types of nodes that can be managed by Chef include, but are not limited to, the following: [16]

Node Type	Description
Server	A physical node is any active device attached to a network that can run a chef-client and allows that chef-client communicates with a Chef server.
Cloud	A cloud-based node is hosted in an external cloud-based service, such as Amazon Web Services (AWS), OpenStack, Rackspace, Google Compute Engine, or Microsoft Azure. Plugins are available for knife that provide support for external cloud-based services.
Virtual Machine	A virtual node is a machine that runs only as a software implementation, but otherwise behaves much like a physical machine.
Network Device	A network node is any networking device that is being managed by a chef-client, such as networking devices by Juniper Networks, Arista, Cisco, and F5.
Container	Containers are an approach to virtualization that allows a single operating system to host many working configurations, where each working configuration - a container - is assigned a single responsibility that is isolated from all other responsibilities.

The key components, which gather other components, of nodes are the Chef Client and Ohai.

Chef Client

A chef-client is an agent that runs locally on every node. When a chef-client is run, it will perform all the steps that are required to bring the node into an expected state, including: [16]

Registering and authenticating the node with the Chef server
Building the node object
Synchronizing cookbooks
Compiling the resource collection by loading each of the required cookbooks, including recipes, attributes, and all other dependencies
Taking the appropriate and required actions to configure the node
Looking for exceptions and notifications, handling each as required

Ohai

Ohai is a tool that is used to collect system configuration data and is run at the beginning of every Chef run to determine the system's state.

The types of attributes Ohai collects include but are not limited to:

Operating System
Network
Memory
Disk
CPU
Kernel
Host names
Fully qualified domain names
Virtualization
Cloud provider metadata

Attributes that are collected by Ohai are automatic level attributes, in that these attributes are used by the chef-client to ensure that these attributes remain unchanged after the chef-client is done configuring the node. [16]

Chef Supermarket

Chef Supermarket is the site for community cookbooks. It provides an easily searchable cookbook repository and a friendly web UI. Cookbooks that are part of the Chef Supermarket are accessible by any Chef user.

There are two ways to use Chef Supermarket:

The public Chef Supermarket is hosted by Chef and is located at Chef Supermarket
A private Chef Supermarket may be installed on-premise behind the firewall on the internal network. Cookbook retrieval from a private Chef Supermarket is often faster than from the public Chef Supermarket because of closer proximity and fewer cookbooks to resolve. A private Chef Supermarket can also help formalize internal cookbook release management processes (e.g. “a cookbook is not released until it’s published on the private Chef Supermarket”). [17]

Berkshelf

Berkshelf can include multiple Chef Supermarket instances for dependency resolution. Cookbook dependency resolution is performed from the top down. The first source defined in the Berksfile will be searched for the cookbook before the second source.

The Berksfile first looks for the cookbook on the private Chef Supermarket and, if not discovered there, then looks on the public Chef Supermarket. [17]

1
2


source 'https://your_private_supermarket_url'
source 'https://supermarket.chef.io'

Conclusion

In this post we have seen the architecture of the Chef configuration management tool, went through each component, and saw how each component can interact with on another. This was not intended as a first touch tutorial on Chef, but more as a summary and helpful reference for Chef's components and interactions.

Check Chef.io learn site for well-structured and paced tutorials.

References

[1]	--, About Knife - Chef.io Documentation, 2018. [link]

[2]	--, About Cookbooks - Chef.io Documentation, 2018. [link]

[3]	--, About Recipes - Chef.io Documentation, 2018. [link]

[4]	(1, 2) --, About Attributes - Chef.io Documentation, 2018. [link]

[5]	--, About Files - Chef.io Documentation, 2018. [link]

[6]	--, Cookbook Directories and Metadata - Chef.io Documentation, 2018. [link]

[7]	Ian F. Darwin, Checking C programs with lint, O’Reilly & Associates, 1990, ISBN: 9780937175309,0937175307

[8]	--, ChefSpec Documentation,`[link] <https://docs.chef.io/chefspec.html>`__

[9]	--, InSpec Documentation, [link]

[10]	(1, 2, 3, 4) --, Organizations and Groups - Chef.io Documentation, 2018 [link]

[11]	--, About Run-lists - Chef.io Documentation, 2018 [link]

[12]	--, About Policy - Chef.io Documentation, 2018 [link]

[13]	--, About Environments - Chef.io Documentation, 2018 [link]

[14]	--, About Roles - Chef.io Documentation, 2018 [link]

[15]	(1, 2, 3) --, About Data Bags - Chef.io Documentation, 2018 [link]

[16]	(1, 2, 3) --, About Nodes - Chef.io Documentation, 2018 [link]

[17]	(1, 2) --, Chef Supermarket - Chef.io Documentation, 2018 [link]

An Advanced Guide to Salt

Wed, 30 Jan 2019 13:37:00 +0100

Introduction

In this post we will be talking about more advanced scenarios using Salt, specifically how to write your own execution and state modules.

Like we saw before, we configure all of the salt subsystem in files within /srv/salt/.

The location for our custom execution and state modules is in the same place we place our SLS files, however, within two specific directories: _modules and _states, for execution and state modules, respectively.

Execution Module

Let us take the example of creating a custom execution module that can manage users in a system, in a way that we can:

see which home directories have no users associated with
disable a user from login
enable a user to login
force a user to change its password

Let us call the module myuser, so it does not clash with any existing module name. For info on all of salt's existing modules check this list.

First we create the module file in the proper location:

root@master:~# tree /srv/salt
/srv/salt
└── states
└── base
   │  ...
├── _modules
│   └── myuser.py
...

Now we start writing our module. The first thing we should do, after imports, is tell the salt system the proper module name to use:

1
2
3
4
5
6


'''
set the module name
'''
__virtualname__ = 'myuser'
def __virtual__():
return __virtualname__

Now we can implement our first function, let us call it get_orphan_home_dirs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


'''
returns a directory list of directories, which the users that they belong to no longer exist

CLI Example::

 salt '*' myuser.get_orphan_home_dirs
'''
def get_orphan_home_dirs():
ret = dict()
root='/home'
dirs = os.listdir(root)
for dir_ in [d for d in dirs if os.path.isdir(os.path.join(root, d))]:
found = False
for p in [u for u in pwd.getpwall() if (u.pw_uid >= 1000)]: # here we skip system users
if p.pw_dir == os.path.join(root, dir_):
found = True
break
if not found: # we got a culprit
ret[root+dir_] = 'User {} Not Found!'.format(dir_)
return ret

Before we can test this execution module we need to update the cache on the minion:

root@master:~$ salt 'minion-1' saltutil.sync_modules
minion-1:
----------
- modules.myuser
root@master:~$

Now we can test it on the salt-master:

root@master:~$ salt 'minion-1' myuser.get_orphan_home_dirs
minion-1:
----------
/home/does-not-exist:
User does-not-exist Not Found!
root@master:~$

And everything looks good, so we can move on to the next functions, disable/enable user accounts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


'''
disables a user from being able to login to the minion

CLI Example::

 salt '*' myuser.disable_login username
'''
def disable_login(name):
ret = dict()
minion_id = __salt__['grains.get']('id')
try:
pwd.getpwnam(name)
__salt__['shadow.set_expire'](name, 0)
ret['result'] = True
ret['comment'] = None
ret['changes'] = {'disable_login': {'old': '',
'new': name + '@' + minion_id}}
except KeyError:
ret['result'] = True
ret['changes'] = dict()
ret['comment'] = 'User not Present ' + name + ' @ ' + minion_id
return ret
'''
enables a user to login to the minion

CLI Example::

 salt '*' myuser.enable_login username
'''
def enable_login(name):
ret = dict()
minion_id = __salt__['grains.get']('id')
try:
pwd.getpwnam(name)
__salt__['shadow.set_expire'](name, -1)
ret['result'] = True
ret['comment'] = None
ret['changes'] = {'enable_login': {'old': '',
'new': name + '@' + minion_id}}
except KeyError:
ret['result'] = True
ret['changes'] = dict()
ret['comment'] = 'User not Present ' + name + ' @ ' + minion_id
return ret

All of the Salt execution modules are available to each other and modules can call functions available in other execution modules. [1]

The variable __salt__ is packed into the modules after they are loaded into the Salt minion. [1]

The __salt__ variable is a Python dictionary containing all of the Salt functions. Dictionary keys are strings representing the names of the modules and the values are the functions themselves. [1]

Here we make use of the shadow module, you can see details about it here

Let us test these with the user user temp:

root@master:~# salt 'minion-1' myuser.disable_login temp
minion-1:
----------
changes:
----------
disable_login:
----------
new:
temp@minion-1
old:
comment:
None
result:
True

Check that it is disabled:

temp@local:~$ ssh minion-1
Your account has expired; please contact your system administrator
Connection to minion-1 closed by remote host.
Connection to minion-1 closed.

Now we enable it back.

root@master:~# salt 'minion-1' myuser.enable_login temp
minion-1:
----------
changes:
----------
enable_login:
----------
new:
temp@minion-1
old:
comment:
None
result:
True

And check that it works.

temp@local:~$ ssh minion-1
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-131-generic x86_64)
temp@minion-1:~$

Nice, now we just have one task left, make a function that forces a user to change its password:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


'''
Expire the password validity for a user, forcing the user to change the password at next successful login

CLI Example::

 salt '*' myuser.expire_passwd username
'''
def expire_passwd(name):
ret = dict()
minion_id = __salt__['grains.get']('id')
try:
old = spwd.getspnam(name)
__salt__['shadow.set_date'](name, 0)
ret['result'] = True
ret['changes'] = {'expire_passwd': {'old': old.sp_lstchg,
'new': spwd.getspnam(name).sp_lstchg}}
ret['comment'] = None
except KeyError:
ret['result'] = True
ret['changes'] = dict()
ret['comment'] = 'User not Present ' + name + ' @ ' + minion_id
return ret

Let us test it, then.

root@master:~# salt 'minion-1' myuser.expire_passwd temp
minion-1:
----------
changes:
----------
expire_passwd:
----------
new:
0
old:
-1
comment:
None
result:
True

If we did everything right, this will trigger a password change on the user's next successful login. And the user is forced to change to a different password.

temp@local:~$ ssh minion-1
You are required to change your password immediately (root enforced)
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for temp.
(current) UNIX password:
Enter new UNIX password:
Retype new UNIX password:
Password unchanged
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
temp@minion-1:~$ logout
Connection to minion-1 closed.

Looks good =)

State Module

Like we saw in the previous post, to apply a state we use the state execution module, and its function apply. One key difference between using execution modules and state modules is the possibility of using the keyword test that, when True, will not effectively apply the state but rather it will show what it would do and/or which changes it would apply. We will see how to define this behaviour.

Just like for the execution module we should first create the state file in the proper location.

root@master:~# tree /srv/salt
/srv/salt
└── states
└── base
   │  ...
├── _states
│ └── myuser.py
...

And we can start writing our state module. Like the execution module let us call it myuser. As the get_orphan_home_dirs function is informational only, we will be implementing the other three disable_login, enable_login and expire_passwd.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51


def disable_login(name):
ret = dict()
if __opts__['test']:
ret['name'] = 'login'
ret['changes'] = dict()
ret['result'] = None
ret['comment'] = 'Would disable login for user {0}'.format(name)
return ret
ret = __salt__['myuser.disable_login'](name)
ret['name'] = 'login'
return ret
def enable_login(name):
ret = dict()
if __opts__['test']:
ret['name'] = 'login'
ret['changes'] = dict()
ret['result'] = None
ret['comment'] = 'Would enable login for user {0}'.format(name)
return ret
ret = __salt__['myuser.enable_login'](name)
ret['name'] = 'login'
return ret
def expire_passwd(name):
ret = dict()
if __opts__['test']:
ret['name'] = 'login'
ret['changes'] = dict()
ret['result'] = None
ret['comment'] = 'Would enable login for user {0}'.format(name)
return ret
ret = __salt__['myuser.expire_passwd'](name)
ret['name'] = 'login'
return ret

And now we can use these in SLS files, i.e:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


{%- set shell = '/bin/bash' -%}
{%- set home_root = '/home' -%}
{% for username in ['alice', 'bob'] %}
add user {{ username }} and ensure the login is enabled:
user.present:
- name: {{ username }}
- shell: {{ shell }}
- home: {{ home_root }}/{{ username }}
- createhome: True
- password: <some hash previously created with mkpasswd -m sha-256>
myuser.enable_login:
- name: {{ username }}
- require:
- user: add user {{ username }} and ensure the login is enabled
force user {{ username }} to change its password:
myuser.expire_passwd:
- name: {{ username }}
- require:
- myuser: add user {{ username }} and ensure the login is enabled
{% endfor %}
{% for username in ['charlie', 'lucy'] %}
add user {{ username }} and disable its login:
user.present:
- name: {{ username }}
- shell: {{ shell }}
- home: {{ home_root }}/{{ username }}
- createhome: True
- password: <some hash previously created with mkpasswd -m sha-256>
myuser.disable_login:
- name: {{ username }}
- require:
- user: add user {{ username }} and disable its login
{% endfor %}

Before testing our newly created state module we have to update the minion's cache.

root@master:~$ salt minion-1 saltutil.sync_states
minion-1:
----------
- states.myuser
root@master:~$

Now let us test it.

root@master:~$ salt minion-1 state.apply users test=True
minion-1:
----------
ID: add user alice and ensure the login is enabled
Function: user.present
Name: alice
Result: None
Comment: User alice set to be added
Started: 14:23:49.860660
Duration: 0.967 ms
Changes:
----------
ID: add user bob and ensure the login is enabled
Function: user.present
Name: bob
Result: None
Comment: User bob set to be added
Started: 14:23:49.860660
Duration: 0.967 ms
Changes:
----------
ID: add user alice and ensure the login is enabled
Function: myuser.enable_login
Name: alice
Result: None
Comment: Would enable login for user alice
Started: 14:23:48.895331
Duration: 0.473 ms
Changes:
----------
ID: add user bob and ensure the login is enabled
Function: myuser.enable_login
Name: bob
Result: None
Comment: Would enable login for user bob
Started: 14:23:48.895431
Duration: 0.473 ms
Changes:
----------
ID: force user alice to change its password
Function: myuser.expire_passwd
Name: alice
Result: None
Comment: Would force password change for user alice
Started: 14:23:48.895531
Duration: 0.473 ms
Changes:
----------
ID: add user charlie and ensure the login is enabled
Function: user.present
Name: charlie
Result: None
Comment: User charlie set to be added
Started: 14:23:49.860660
Duration: 0.967 ms
Changes:
----------
ID: add user lucy and ensure the login is enabled
Function: user.present
Name: lucy
Result: None
Comment: User lucy set to be added
Started: 14:23:49.860660
Duration: 0.967 ms
Changes:
----------
ID: force user bob to change its password
Function: myuser.expire_passwd
Name: bob
Result: None
Comment: Would force password change for user bob
Started: 14:23:48.896661
Duration: 0.473 ms
Changes:
----------
ID: add user charlie and disable its login
Function: myuser.disable_login
Name: charlie
Result: None
Comment: Would force password change for user charlie
Started: 14:23:49.896761
Duration: 0.473 ms
Changes:
----------
ID: add user lucy and disable its login
Function: myuser.disable_login
Name: lucy
Result: None
Comment: Would force password change for user lucy
Started: 14:23:49.896771
Duration: 0.473 ms
Changes:
----------
Summary for minion-1
--------------
Succeeded: 10 (unchanged=10)
Failed: 0
--------------
Total states run: 10
Total run time: 9.325 ms

The test looks good so now we can apply it.

root@master:~$ salt minion-1 state.apply users
minion-1:
----------
.....
.....
Summary for minion-1
--------------
Succeeded: 10 (changed=10)
Failed: 0
--------------
Total states run: 10
Total run time: 9.325 ms
root@master:~$

Looks good =)

Now we have 4 new users added to minion-1, alice and bob set with a forced password change, and charlie and lucy which exist but cannot login.

Conclusion

In this post we saw how to write both an execution module and a state module for the Saltstack Salt configuration management tool. We also saw how to test these modules and use them in a SLS file.

The files described in this post are available here:

Feel free to leave comments below, any improvement to this and other articles is always welcome.

Thanks for reading!

References

[1]	(1, 2, 3) https://docs.saltstack.com/en/latest/ref/modules/index.html

A Comprehensive Introduction to Salt

Mon, 01 Oct 2018 09:30:00 +0100

Introduction

Recent advances in IT infrastructures, namely for Cloud-based Services, where scalability, elasticity and availability are key factors, have also seen advances in the way Management and Administration of these critical infrastructures are achieved, be it on log analysis and systems monitoring or on configuration management and automation. This is a rising subject, with exponential development evolution as new tools with different approaches have emerged in the last few years. [2]

A conceptual architecture of system configuration tools

Delaet et al. [3] provide a framework for evaluating different System Configuration tools. They also define a conceptual architecture for these tools, illustrated in Figure 1. In essence, this type of tool is composed by a master element (typically residing in a deployment node) containing a set of specialized modules that translate through some form of manifest files, the specific configuration for each component of the remote managed nodes of the infrastructure. Each remote managed node, through some form of local agent conveys to the master detailed information about the node and executes configuration actions determined by the master. The master compiles in a repository a catalog (inventory) with information on the nodes and on how they should be configured.

In their paper, John Benson et al. [1], address the challenge of building an effective multi-cloud application deployment controller as a customer add-on outside of a cloud utility service using automation tools such as Ansible (ansible.com), SaltStack (saltstack.com) and Chef (www.chef.io) and compare them to each other as well as with the case where there is no automation framework just plain shell code.

The authors compare those three tools in terms of performance in executing three sets of tasks (installing a chemistry software package, installing an analytics software package, installing both software packages), the amount of code needed to execute those tasks and different features they have or do not have, such as a GUI and the need for an agent.

Ebert, C. et al. [4] address and discuss this recent organization culture coined DevOps, namely the toolset used at different phases, from Build, to Deployment, to Operations. The authors conclude that "DevOps is impacting the entire software and IT industry. Building on lean and agile practices, DevOps means end-to-end automation in software development and delivery."[4], with which I agree on. The authors also state that "Hardly anybody will be able to approach it with a cookbook-style approach, but most developers will benefit from better connecting development and operations."[4] with which I do not totally agree with, as I will try to demonstrate, by showing that a reactive infrastructure can be created to act autonomously on certain events, without leaving the cookbook-like approach, but through a set of triggered events that can be used to automate software development and delivery. There are already tools, such as SaltStack that makes possible for certain events to intelligently trigger actions, allowing therefore IT operators to create autonomic systems and data centers instead of just static runbooks. The authors also state that a "mutual understanding from requirements onward to maintenance, service, and product evolution will improve cycle time by 10 to 30 percent and reduce costs up 20 percent."[4] pointing out that the major drivers are fewer requirements changes, focused testing, quality assurance, and a much faster delivery cycle with feature-driven teams. They also point out the large dynamics of this DevOps culture as "each company needs its own approach to achieve DevOps, from architecture to tools to culture."[4]

Saltstack architecture

Saltstack took a different approach for its architecture, illustrated in , when compared to the architecture defined by Delaet et al. [3]. Comparing both, there are some similarities but it can also be seen that there is a bidirectional channel of communication between the managed devices and the Salt Master which corresponds to an Event Bus that connects to different parts of the tool. The Runner is used for executing modules specific to the Salt Master but can also provide information about the managed devices. The Reactor is responsible for receiving events and corresponding them to the appropriate state or even to fire another event as a consequence of the former. As stated in their documentation, "Salt Engines are long-running, external system processes that leverage Salt"(https://docs.saltstack.com), allowing to integrate external tools into the Salt system, for example to send to the salt Event Bus messages of a channel on the Slack (slack.com) communication tool (see https://github.com/saltstack/salt/blob/develop/salt/engines/slack.py).

Architecture

For this work we will be using Saltstack's orchestration tool Salt. Its requirements for a device to be managed by it can be as little as having a REpresentational State Transfer (REST) API through which one can interact with. Its overall architecture can be seen in . In this section we will see the architectural, network and hardware requirements, as well as Salt's software requirements and how to use Salt as a full infrastructure management tool.

Monolithic example

The minimum configuration for Salt is one machine, which takes the role of either Master and Minion, or just a Minion, with no master see . Although this does not achieve much, it still allows to test operations, be it configuration management, reactions to certain events, monitoring of processes or to just do some tests with Salt.

Consider the two scenarios represented in Figure 4 and Figure 5:

an Operator within the Control layer
an Operator outside the Control layer, with Control and Infrastructure layers behind a Network Address Translation (NAT)

In the first scenario, the Operator is either within the Main Controller itself or in the network neighbourhood, meaning that it has network access within the private network of which the Main Controller is part of.

In the second scenario, the Operator and the Main Controller are in distinct networks, meaning that the Operator with have to have external access to the infrastructure, either using the Main Controller has a bastion server, or by assigning the minions with a public Internet Protocol (IP) address. We will see that as public IP addresses are not that abundant, using a bastion becomes a necessity. The Salt administration tool, by Saltstack, has several key components which make it a complete framework for managing and administering an IT infrastructure. In each of the following subsections we will see what they are used for and how to make use of them.

"Running pre-defined or arbitrary commands on remote hosts, also known as remote execution, is the core function of Salt" [5]. Remote execution in Salt is achieved through execution modules and returners. Salt also contains a configuration management framework, which complements the remote execution functions by allowing more complex and interdependent operations. The framework functions are executed on the minion's side, allowing for scalable, simultaneous configuration of a great number of minion targets.

"The Salt Event System is used to fire off events enabling third party applications or external processes to react to behavior within Salt" [6]. These events can be launched from within the Salt infrastructure or from applications residing outside of it. To monitor non-Salt processes one can use the beacon system which "allows the minion to hook into a variety of system processes and continually monitor these processes. When monitored activity occurs in a system process, an event is sent on the Salt event bus that can be used to trigger a reactor" [7].

The pinnacle of the two previous components is using that information to trigger actions in response to those kinds of events. In Salt we have the reactor system "a simple interface to watching Salt's event bus for event tags that match a given pattern and then running one or more commands in response." [8].

Operator within the Control layer

Operator outside the Control layer

Most of the information may be static, but it can belong to each individual minion. This information can be either gathered from the minion itself or attributed to it by the Salt system. The first "is called the grains interface, because it presents salt with grains of information. Grains are collected for the operating system, domain name, IP address, kernel, OS type, memory, and many other system properties." [9]. The latter is called the pillar, which "is an interface for Salt designed to offer global values that can be distributed to minions" [10], in a private way relative to every other minion, if need be.

Directory Structures

There are two essential locations for salt related files (excluding service files):

/etc/salt
/srv/salt

In /etc/salt we have the configuration files for salt master and minion, as well as the keys for known minions (if it is a salt master machine) and the key for the salt master (if it is a salt minion machine).

root@master:~# tree /etc/salt
/etc/salt
├── master
├── minion
├── minion_id
└── pki
   ├── master
   │   ├── master.pem
   │   ├── master.pub
   │   ├── minions
   │   │   ├── minion1
   │   │   ├── minion2
   │   │   └── minion3
   │   ├── minions_autosign
   │   ├── minions_denied
   │   ├── minions_pre
   │   └── minions_rejected
   │
   │
   │
   └── minion
   ├── minion_master.pub
   ├── minion.pem
   └── minion.pub

/etc/salt/master: master config file

/etc/salt/minion: minion config file

/etc/salt/minion_id: minion identifier file

/etc/salt/pki/master/master.pem: master private key

/etc/salt/pki/master/master.pub: master public key

/etc/salt/pki/minion/minion_master.pub: master public key

/etc/salt/pki/minion/minion.pub: minion public key

/etc/salt/pki/minion/minion.pem: minion private key

In /srv/salt we store the state, pillar and reactor files, we will read more about this further ahead.

Configuration Files

We will now see basic, and some optional, settings for both salt master and salt minion.

/etc/salt/master

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


# The address of the interface to bind to, here we bind to all
interface: 0.0.0.0
# The port used by the publisher, default 4505
publish_port: 4505
# The user to run salt as
user: root
# Number of hours to keep old job information
keep_jobs: 168
# Timeout for the salt command and api, default is 5 seconds
timeout: 15
# SLS files location
file_roots:
base:
- /srv/salt/states/base/
# Top file to use
state_top: top.sls
# Pillar location
pillar_roots:
base:
- /srv/salt/pillar/base
# Reactor Configurations:
reactor:
- 'salt/auth':
- /srv/salt/reactor/auth-pending.sls
# if we just want to see what actually changes, set this to False
state_verbose: False
# log file location, we leave this to the system's logger
log_file: file:///dev/log
# The level of messages to send to the log file.
# One of 'info', 'quiet', 'critical', 'error', 'debug', 'warning'.
log_level: info
log_fmt_logfile: '%(asctime)-15s salt-master[%(process)d] %(name)s: %(message)s'
log_datefmt_logfile: '%b %d %H:%M:%S'

From reading this configuration file we can see that we will be listening on all interfaces using the default publish port, running as user root, keep job information for a week, increasing the default timeout for salt commands, and logging to the system's logger with a custom-defined format:

[TIMESTAMP] salt-master[PID] SALT_FUNCTION: MESSAGE

Besides this we also set up the base directories for state files, pillar data and reactor states. We will read about this further in this post, in Configuration Management, Storing and Accessing Data and Event System, respectively.

/etc/salt/minion

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# salt master address
master: salt-master.local
# this overrides the salt auto naming (from hostname -f)
minion_id: minion1
# log file location, we leave this to the system's logger
log_file: file:///dev/log
# The level of messages to send to the log file.
# One of 'info', 'quiet', 'critical', 'error', 'debug', 'warning'.
log_level: info
log_fmt_logfile: '%(asctime)-15s salt-minion[%(process)d] %(name)s: %(message)s'
log_datefmt_logfile: '%b %d %H:%M:%S'

As we can see, the most basic configuration file for a salt minion is its salt master network location, it can be configured via a fqdn, a hostname defined in /etc/hosts/ file or via IP address.

We can also see the minion_id setting which forces a minion identifier, instead of letting the salt-minion choose one based on the machine's hostname.

Subsystem Files

Like we saw before, we configure all of the salt subsystem files within /srv/salt/, here we have a tree view of its subdirectories (not ordered on purpose):

root@master:~# tree /srv/salt
/srv/salt
└── states
│ └── base
│    ├── ntp
│    │   ├── etc
│    │   │   └── ntp.conf.tmpl
│    │   └── init.sls
│ └── top.sls
├── pillar
│   └── base
│   ├── ntp
│   │   └── init.sls
│   └── top.sls
└── reactor
   └── auth-pending.sls

The Top File

An infrastructure can be seen as being an applicational stack by itself, by having groups of different machines set up in small clusters, each cluster performing a sequence of tasks. The Top file is were the attribution of configuration role(s) and Minion(s) is made. This kind of file is used both for State and Pillar systems. Top files can be structurally seen has having three levels:

Environment:	(ϵ) A directory structure containing a set of SaLt State (SLS) files.
Target:	(θ) A predicate used to target Minions.
State files:	(σ) A set of SLS files to apply to a target. Each file describes one or more states to be executed on matched Minions.

These three levels relate in such a way that (θ ∈ ϵ) , and (σ ∈ θ). Reordering these two relations we have that:

Environments contain targets
Targets contain states

Putting these concepts together, we can describe a scenario (top file) in which the state defined in the SLS file ntp/init.sls is enforced to all minions.

top.sls:

1
2
3


base:
'*':
- ntp

Remote Execution

Before delving into the more intricate orchestration-like functionalities of salt, let us use the most (probably) basic operation of salt, which you probably will use a lot.

The function is cmd.run, made available from the cmd execution module (see [11] for a list of all available execution modules). And we can use it like this, from a salt master environment:

root@master:~# salt \* cmd.run 'hostname && pwd && ls -la'

This will run the command, or sequence of commands, in every targeted minion and return its result. You can also specify the shell you want to use with the shell argument.

root@master:~# salt \* cmd.run 'hostname && ls -la' shell=/bin/bash
minion3:
minion3.local
/root
total 40
drwx------ 7 root root 4096 Sep 27 16:35 .
drwxr-xr-x 23 root root 4096 Oct 2 07:00 ..
-rw------- 1 root root 2336 Sep 29 14:11 .bash_history
-rw-r--r-- 1 root root 3201 Sep 3 14:19 .bashrc
drwx------ 3 root root 4096 Sep 19 15:21 .cache
drwx------ 3 root root 4096 Sep 27 16:35 .config
drwxr-xr-x 2 root root 4096 Aug 17 17:00 .nano
-rw-r--r-- 1 root root 148 Aug 17 2015 .profile
drwx------ 2 root root 4096 Sep 19 15:21 .ssh
drwxr-xr-x 3 root root 4096 Sep 6 16:43 .virtualenvs
minion1:
master.local
/root
total 40
drwx------ 7 root root 4096 Sep 27 16:35 .
drwxr-xr-x 23 root root 4096 Oct 2 07:00 ..
-rw------- 1 root root 2336 Sep 29 14:11 .bash_history
-rw-r--r-- 1 root root 3201 Sep 3 14:19 .bashrc
drwx------ 3 root root 4096 Sep 19 15:21 .cache
drwx------ 3 root root 4096 Sep 27 16:35 .config
drwxr-xr-x 2 root root 4096 Aug 17 17:00 .nano
-rw-r--r-- 1 root root 148 Aug 17 2015 .profile
drwx------ 2 root root 4096 Sep 19 15:21 .ssh
drwxr-xr-x 3 root root 4096 Sep 6 16:43 .virtualenvs
minion2:
minion2.local
/root
total 40
drwx------ 7 root root 4096 Sep 27 16:35 .
drwxr-xr-x 23 root root 4096 Oct 2 07:00 ..
-rw------- 1 root root 2336 Sep 29 14:11 .bash_history
-rw-r--r-- 1 root root 3201 Sep 3 14:19 .bashrc
drwx------ 3 root root 4096 Sep 19 15:21 .cache
drwx------ 3 root root 4096 Sep 27 16:35 .config
drwxr-xr-x 2 root root 4096 Aug 17 17:00 .nano
-rw-r--r-- 1 root root 148 Aug 17 2015 .profile
drwx------ 2 root root 4096 Sep 19 15:21 .ssh
drwxr-xr-x 3 root root 4096 Sep 6 16:43 .virtualenvs
root@master:~#

Another well used execution module is pkg, it will try to identify the package system present in the minion's system and use it to perform the requested operations.

root@master:~# salt minion\* pkg.latest_version postfix
minion1:
3.1.0-3ubuntu0.3
minion2:
3.1.0-3ubuntu0.3
minion3:
3.1.0-3ubuntu0.3
root@master:~# salt minion\* pkg.install postfix
minion1:
----------
postfix:
----------
new:
3.1.0-3ubuntu0.3
old:
(... suppressed output ...)
minion3:
----------
postfix:
----------
new:
3.1.0-3ubuntu0.3
old:
(... suppressed output ...)
minion2:
----------
postfix:
----------
new:
3.1.0-3ubuntu0.3
old:
(... suppressed output ...)

Configuration Management

Remote execution is very useful, however we may want more complex scenarios to occur, then we can make use of SLS files, states, and state modules [12].

From the tree view seen in Subsystem Files and the configuration of the file_roots option, all of our available states are to be placed in /srv/salt/states/base.

SLS files are mainly written in YAML Ain't Markup Language (YAML) [13] and like most salt related files, can be templated using Jinja [14], by default.

ntp/init.sls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


ntp:
pkg.installed
service.running:
- enable: True
- watch:
- file: /etc/ntp.conf
/etc/ntp.conf:
file.managed:
- source: salt://ntp/etc/ntp.conf.tmpl
- mode: 644
- user: root
- require:
- pkg: ntp
ntp-shutdown:
service.dead:
- name: ntp
- onchanges:
- file: /etc/ntp.conf

ntp/etc/ntp.conf.tmpl:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# This file managed by Salt, do not edit
server ntp1.example.com
server ntp1.example.com
# Only allow read-only access from localhost
restrict default noquery nopeer
restrict 127.0.0.1
restrict ::1
# Location of drift file
driftfile /var/lib/ntp/ntp.drift

This salt state file, when applied, will:

install the ntp package, using the system's package manager
create/change the file /etc/ntp/ntp.conf, on the minion, according to the contents of the /srv/salt/states/base/ntp/etc/ntp.conf.tmpl, on master
make sure the ntp service is stopped if there are changes to the previously managed file.
make sure the ntp service is running and enabled, restarting it if there were changes to the managed file (it should not restart because of the the ntp-shutdown state.

To apply a state we use the state execution module, and its function apply. One key difference between using execution modules and state modules is the possibility of using the keyword test that, when True, will not effectively apply the state but rather it will show what it would do and, in our case, which changes it would apply.

root@master:~# salt --state-verbose=True minion2 state.apply ntp test=True
minion2:
----------
ID: ntp-pkg
Function: pkg.installed
Name: ntp
Result: True
Comment: All specified packages are already installed
Started: 15:26:10.712651
Duration: 928.968 ms
Changes:
----------
ID: ntp-pkg
Function: file.managed
Name: /etc/ntp.conf
Result: None
Comment: The file /etc/ntp.conf is set to be changed
Started: 15:26:11.664815
Duration: 63.205 ms
Changes:
----------
diff:
---
+++
@@ -1,53 +1,20 @@
-# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
+# This file managed by Salt, do not edit
+#
+#
+# With the default settings below, ntpd will only synchronize your clock.
-driftfile /var/lib/ntp/ntp.drift
-
-
-# Enable this if you want statistics to be logged.
-#statsdir /var/log/ntpstats/
-
-statistics loopstats peerstats clockstats
-filegen loopstats file loopstats type day enable
-filegen peerstats file peerstats type day enable
-filegen clockstats file clockstats type day enable
-
-# Specify one or more NTP servers.
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
-server ntp1.somewhere.in.time
-server ntp2.somewhere.in.time
+server ntp1.example.com
+server ntp2.example.com
-# Use Ubuntu's ntp server as a fallback.
-server ntp.ubuntu.com
-# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
-# details. The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
-# might also be helpful.
-#
-# Note that "restrict" applies to both servers and clients, so a configuration
-# that might be intended to block requests from certain clients could also end
-# up blocking replies from your own upstream servers.
-
-# By default, exchange time with everybody, but don't allow configuration.
-restrict -4 default kod notrap nomodify nopeer noquery
-restrict -6 default kod notrap nomodify nopeer noquery
-
-# Local users may interrogate the ntp server more closely.
+# Only allow read-only access from localhost
+restrict default noquery nopeer
restrict 127.0.0.1
restrict ::1
-# Clients from this (example!) subnet have unlimited access, but only if
-# cryptographically authenticated.
-#restrict 192.168.123.0 mask 255.255.255.0 notrust
-
-
-# If you want to provide time to your local subnet, change the next line.
-# (Again, the address is an example only.)
-#broadcast 192.168.123.255
-
-# If you want to listen to time broadcasts on your local subnet, de-comment the
-# next lines. Please do this only if you trust everybody on the network!
-#disable auth
-#broadcastclient
+# Location of drift file
+driftfile /var/lib/ntp/ntp.drift
----------
ID: ntp-pkg
Function: service.running
Name: ntp
Result: None
Comment: Service ntp is set to start
Started: 15:26:11.729479
Duration: 134.939 ms
Changes:
----------
ID: ntpdate
Function: pkg.installed
Result: True
Comment: All specified packages are already installed
Started: 15:26:11.864881
Duration: 25.952 ms
Changes:
Summary for minion2
------------
Succeeded: 4 (unchanged=2, changed=1)
Failed: 0
------------
Total states run: 4
Total run time: 1.153 s

Now for the actual application of the state

----------
ID: ntp-pkg
Function: pkg.installed
Name: ntp
Result: True
Comment: All specified packages are already installed
Started: 15:30:11.488468
Duration: 902.224 ms
Changes:
----------
ID: ntp-pkg
Function: file.managed
Name: /etc/ntp.conf
Result: True
Comment: File /etc/ntp.conf updated
Started: 15:30:12.394500
Duration: 94.999 ms
Changes:
----------
diff:
---
+++
@@ -1,53 +1,20 @@
-# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
+# This file managed by Salt, do not edit
+#
+#
+# With the default settings below, ntpd will only synchronize your clock.
-driftfile /var/lib/ntp/ntp.drift
-
-
-# Enable this if you want statistics to be logged.
-#statsdir /var/log/ntpstats/
-
-statistics loopstats peerstats clockstats
-filegen loopstats file loopstats type day enable
-filegen peerstats file peerstats type day enable
-filegen clockstats file clockstats type day enable
-
-# Specify one or more NTP servers.
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
-server ntp1.somewhere.in.time
-server ntp2.somewhere.in.time
+server ntp1.example.com
+server ntp2.example.com
-# Use Ubuntu's ntp server as a fallback.
-server ntp.ubuntu.com
-# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
-# details. The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
-# might also be helpful.
-#
-# Note that "restrict" applies to both servers and clients, so a configuration
-# that might be intended to block requests from certain clients could also end
-# up blocking replies from your own upstream servers.
-
-# By default, exchange time with everybody, but don't allow configuration.
-restrict -4 default kod notrap nomodify nopeer noquery
-restrict -6 default kod notrap nomodify nopeer noquery
-
-# Local users may interrogate the ntp server more closely.
+# Only allow read-only access from localhost
+restrict default noquery nopeer
restrict 127.0.0.1
restrict ::1
-# Clients from this (example!) subnet have unlimited access, but only if
-# cryptographically authenticated.
-#restrict 192.168.123.0 mask 255.255.255.0 notrust
-
-
-# If you want to provide time to your local subnet, change the next line.
-# (Again, the address is an example only.)
-#broadcast 192.168.123.255
-
-# If you want to listen to time broadcasts on your local subnet, de-comment the
-# next lines. Please do this only if you trust everybody on the network!
-#disable auth
-#broadcastclient
+# Location of drift file
+driftfile /var/lib/ntp/ntp.drift
----------
ID: ntp-pkg
Function: service.running
Name: ntp
Result: True
Started: 15:30:12.587597
Duration: 49.742 ms
Changes:
----------
ID: ntpdate
Function: pkg.installed
Result: True
Comment: All specified packages are already installed
Started: 15:30:12.637827
Duration: 29.567 ms
Changes:
Summary for minion2
------------
Succeeded: 4 (changed=1)
Failed: 0
------------
Total states run: 4
Total run time: 1.077 s

Now if we apply the ntp state again we will see no changes, and no service stop/start.

SLS files can include other SLS files, use externally generated data and depend on states declared on included SLS files.

Each state can require other state completion, be required in other states, watch a certain state, execute unless a given command returns True, execute only if a certain command returns True and more. [2] [15]

Storing and Accessing Data

When planning and implementing this type of orchestration, it is useful to have information that can be shared between states, and which is known only to the intended recipients (i.e: specific minions).

In salt we have the Pillar, Its configuration is similar to the one used by States, as we saw in Subsystem Files its files are located in /srv/salt/pillar/base.

The way the information is defined and assigned is similar to states, but instead of defining State Module executions we define information in the form of YAML, as shown

/srv/salt/pillar/base/ntp/init.sls:

1
2
3
4
5
6


ntp:
servers:
- ntp1.somewhere.in.time
- ntp2.somewhere.in.time
ntp_conf: /etc/ntp.conf
drift_file: /var/lib/ntp/ntp.drift

and we define the attributions of data in the corresponding pillar top file, where all minions are assigned information regarding Network Time Protocol (NTP) server domain names and their NTP service configuration file.

/srv/salt/pillar/base/top.sls:

1
2
3


base:
'*':
- ntp

This information can be used within states or even template files, using the pillar keyword.

Taking the previous SLS file and its managed template, we can now dynamically fill NTP related info.

ntp/init.sls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


ntp:
pkg.installed
service.running:
- enable: True
- watch:
- file: {{ pillar['ntp']['ntp_conf'] }}
{{ pillar['ntp']['ntp_conf'] }}:
file.managed:
- source: salt://ntp/etc/ntp.conf.tmpl
- mode: 644
- user: root
- require:
- pkg: ntp
ntp-shutdown:
service.dead:
- name: ntp
- onchanges:
- file: {{ pillar['ntp']['ntp_conf'] }}

ntp/etc/ntp.conf.tmpl:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# This file managed by Salt, do not edit
{% for server in pillar['ntp']['servers'] %}
server {{ server }}
{% endfor %}
# Only allow read-only access from localhost
restrict default noquery nopeer
restrict 127.0.0.1
restrict ::1
# Location of drift file
driftfile {{ pillar['ntp']['drift_file'] }}

This example could be further enhanced, let us try and make this able to configure ntp servers on a windows machine. For that we need to use another data source in the salt system, the grains, which is generated by the minion itself.

root@master:~# salt win\* grains.get os
windows1:
Windows
root@master:~#

Interesting, so now we can have selective behaviour in our state:

ntp/etc/ntp.conf.tmpl:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


ntp:
{% if grains['os'] == 'Windows' %}
ntp.managed:
- servers: {{ pillar['ntp']['servers'] }}
{% else %}
pkg.installed
service.running:
- enable: True
- watch:
- file: {{ pillar['ntp']['ntp_conf'] }}
{{ pillar['ntp']['ntp_conf'] }}:
file.managed:
- source: salt://ntp/etc/ntp.conf.tmpl
- mode: 644
- user: root
- require:
- pkg: ntp
ntp-shutdown:
service.dead:
- name: ntp
- onchanges:
- file: {{ pillar['ntp']['ntp_conf'] }}
{% endif %}

And now if we apply the state to the windows machine:

root@master:~# salt win\* state.apply ntp test

Event System

In Introduction we saw in Figure 2 a general view of the Salt System, which has a Reactor element, this element is part of a more broad system, the Event System (represented in Figure 6 which is "used to fire off events enabling third party applications or external processes to react to behavior within Salt" [6].

The Event System has two main components:

Event Socket:	from where events are published
Event Library:	used for listening to events and forward them to the salt system

Saltstack Event System Flow Example

"The event system is a local ZeroMQ PUB interface which fires salt events. This event bus is an open system used for sending information notifying Salt and other systems about operations" [8]. This associates SLS files to event tags on the master, which the SLS files in turn represent reactions.

This means that the reactor system has two steps to work properly. First, the reactor key needs to be configured in the master configuration file, this associates event tags SLS reaction files. Second, these reaction files use a YAML data structure similar to the state system to define which reactions are to be executed.

/etc/salt/master:

1
2
3
4
5
6
7


(... suppressed ...)
reactor:
- 'salt/auth':
- /srv/salt/reactor/auth-pending.sls
(... suppressed ...)

this entry means that for every 'salt/auth' event, the reactor state /srv/salt/reactor/auth-pending.sls should be executed. Let us see what that reactor state actually does. [8]

auth-pending.sls:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


{# minion failed to authenticate -- remove accepted key #}
{% if not data['result'] and data['id'].endswith('mydomain.com') %}
minion_remove:
wheel.key.delete:
- match: {{ data['id'] }}
minion_rejoin:
cmd.cmd.run:
- tgt: master.local
- arg:
- ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no "{{ data['id'] }}" 'sleep 10 && systemctl restart salt-minion'
{% endif %}
{# minion is sending new key -- accept this key #}
{% if 'act' in data and data['act'] == 'pend' and data['id'].endswith('mydomain.com') %}
minion_add:
wheel.key.accept:
- match: {{ data['id'] }}
{% endif %}

"In this SLS file, we say that if the key was rejected we will delete the key on the master and then also tell the master to ssh in to the minion and tell it to restart the minion, since a minion process will die if the key is rejected. We also say that if the key is pending and the id starts with ink we will accept the key. A minion that is waiting on a pending key will retry authentication every ten seconds by default." [8]

Conclusion

In this post we saw from where some of the system configuration tools methods come from, and the similarities between then and now. We briefly explored the content management and orchestration tool Salt, saw how to execute commands remotely via the cmd.run execution module function, how to orchestrate more complex scenarios with the state.apply state module function, how to generate, share and fetch static and dynamic information via the grains and the pillar subsystems, and we also saw how to catch certain events and react to them via the Event System. Hope this was useful, and that now you have some grasp of what is salt and how it can be powerful to plan and manage an IT infrastructure. Feel free to use the comment box below to ask any questions.

References

[1]	J. O. Benson, J. J. Prevost, and P. Rad, Survey of automated software deployment for computational and engineering research, 10th Annual International Systems Conference, SysCon 2016 - Proceedings, 2016.

[2]	(1, 2) Torrinha, Adaptive Management and Administration of IT Infrastructures, MSc thesis, Instituto Superior Técnico, Lisbon, Portugal, 2017.

[3]	(1, 2) Delaet, W. Joosen, and B. Vanbrabant, A survey of system configuration tools, International Conference on Large Installation System Adminstration, pp. 1 - 8, 2010.

[4]	(1, 2, 3, 4, 5) Ebert, G. Gallardo, J. Hernantes, and N. Serrano, DevOps, vol. 33, no. 3, pp. 94 - 100, 2016.

[5]	--, Remote Execution - Saltstack Documentation - v2018.3.2, 2018. [link]

[6]	(1, 2) --, Event System - Saltstack Documentation - v2018.3.2, 2018. [link]

[7]	--, Beacons - Saltstack Documentation - v2018.3.2, 2018. [link]

[8]	(1, 2, 3, 4) --, Reactor System - Saltstack Documentation - v2018.3.2, 2018. [link]

[9]	--, Grains - Saltstack Documentation - v2018.3.2, 2018. [link]

[10]	--, Pillar - Saltstack Documentation - v2018.3.2, 2018. [link]

[11]	--, List of execution modules - Saltstack Documentation - v2018.3.2, 2018 [link]

[12]	--, List of state modules - Saltstack Documentation - v2018.3.2, 2018 [link]

[13]	--, YAML website [link]

[14]	--, Jinja2 website [link]

[15]	--, Requisites and Other Global State Arguments - Saltstack Documentation - v2018.3.2, 2018 [link]

Slurm in Ubuntu Clusters - Part 2

Mon, 24 Sep 2018 09:30:18 +0100

Overview

Overview of the slurm infrastructure

In this article we will not go through the installation and configuration of the shared filesystem described above. We assume that the HOME directory is shared between the controller and the compute nodes, in the future I'll write about how to achieve this using NFS (for simplicity) and BeeGFS, for now let us follow with Slurm usage.

Summary of commands

scontrol - used to view and modify Slurm configuration and state.
sacct - displays accounting data for all jobs and job steps in the Slurm job accounting log or Slurm database.
sinfo - show information about the compute nodes status.
squeue - show information about the scheduler's job queue.
smap - show information about slurm jobs, partitions, and set configurations parameters.
sview - graphical user interface to view and modify the slurm state.
salloc - obtain a Slurm job allocation execute a command, and then release the allocation when the command is finished.
srun - run parallel jobs.
scancel - cancel a submitted job.
sbatch - submit a batch script to Slurm.

Changing the state of a Node

With the scontrol command we can control, among other things, the state of each compute node. If, for example, we need to put an idle node down for maintenance we would do the following:

root@controller:~# scontrol update NodeName=compute-1 State=Down Reason='Maintenance'
root@controller:~# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
base* up 3-00:00:00 1 down* compute-1
base* up 3-00:00:00 3 idle compute-[2-4]
root@controller:~# scontrol update NodeName=compute-1 State=Idle
root@controller:~# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
base* up 3-00:00:00 4 idle compute-[1-4]
root@controller:~#

If, on the other hand, the node we want to put down is executing jobs (its state is allocated or mix), we need to put it in the draining state. Which means that the node will no longer accept new jobs and when all the running jobs complete, the node's state will be drained, and we can then put it in the down state.

root@controller:~# scontrol update NodeName=compute-1 State=Drain Reason='Maintenance'
root@controller:~# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
base* up 3-00:00:00 1 drng compute-1
base* up 3-00:00:00 3 idle compute-[2-4]
root@controller:~# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
base* up 3-00:00:00 1 drain compute-1
base* up 3-00:00:00 3 idle compute-[2-4]
root@controller:~# scontrol update NodeName=compute-1 State=Idle
root@controller:~# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
base* up 3-00:00:00 4 idle compute-[1-4]
root@controller:~#

Executing jobs interactively

To execute jobs interactively, there are usually two steps:

Allocate resources
Run parallel job

As an example, we want to run a BASH shell in a compute cluster. We will ask for just 100MB of RAM and 1 cpu.

username@controller:~$ salloc --cpus-per-task=1 --mem=100MB
salloc: Pending job allocation 27564
salloc: job 27564 queued and waiting for resources
salloc: job 27564 has been allocated resources
salloc: Granted job allocation 27564
username@controller:~$ srun --pty /bin/bash
username@node4:~$ hostname
node4
username@node4:~$ exit
username@node4:~$ exit
salloc: Relinquishing job allocation 27564
username@controller:~$

Batch file

If instead of an interactive job we want a job running on its own for a long amount of time, we can create a batch script and submit it to the slurm scheduler queue.

Ley us take the example code for a simple Hello World! program.

helloworld.c

1
2
3
4


#include <stdio.h>
int main(int argc, char** argv){
printf('Hello World!\n');
}

Let us compile it.

username@controller:~$ gcc -o helloworld helloworld.c

And now we can create a batch file to submit the job.

simple_hello.batch

1
2
3
4
5
6


#!/bin/bash

WORKDIR=/home/username/helloworld
cd $WORKDIR
./helloword

Let us submit the job with the help of the sbatch command and see its output.

username@controller:~/helloworld$ sbatch simple_hello.batch
Submitted job 12345
username@controller:~/helloworld$ ls
simple_hello.batch simple_hello-12345.out simple_hello-12345.err
username@controller:~/helloworld$ cat simple_hello-12345.err
username@controller:~/helloworld$ cat simple_hello-12345.out
Hello World!
username@controller:~/helloworld$

It works!

Now let us try a more complex example, making a C program which will run on multiple nodes, via MPI, and outputs the name of the node it is running on and the rank attributed to it [MPITUTO].

helloworld_mpi.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


#include <stdio.h>
#include <unistd.h>
#include <mpi.h>

int main(int argc, char** argv){
// Init the MPI environment
 MPI_Init(NULL, NULL);
// Get the number of processes
 int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
// Get the rank of the process
 int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
// Get the name of the processor
 char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
// Print a hello world message
 printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name,
world_rank,
world_size);
// sleep for 10 seconds, just enough to see the job executing
 sleep(10);
// Finalize the MPI environment
 MPI_Finalize();
}

Compile the code with MPI capabilities.

username@controller:~/helloworld$ mpicc -o helloworld_mpi helloworld_mpi.c
username@controller:~/helloworld$ ls
helloworld_mpi helloworld_mpi.c
username@controller:~/helloworld$

helloworld_mpi.batch

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


#!/bin/bash
#SBATCH --job-name=hello-world-mpi
#SBATCH --output=/home/username/helloworld/helloworld_mpi-%j.out
#SBATCH --error=/home/username/helloworld/helloworld_mpi-%j.err
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --time=72:00:00
#SBATCH --mem=1G
PATH=/path/to/mpidir/bin:$PATH
WORKDIR=/home/username/helloworld
cd $WORKDIR
mpiexec -np 8 ./helloworld_mpi

Now we can submit the batch script as a job to queue.

username@controller:~/helloworld$ sbatch helloworld_mpi.sbatch
Submitted batch job 27560
username@controller:~/helloworld$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
27560 base hello-w_ username PD 0:00 2 (Resources)
username@controller:~/helloworld$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
27560 base hello-w_ username R 0:05 2 node6,node7

On this sequence we can see that the job was submitted to the queue, and it was waiting for resources to be available and run the job. Then, the job started running and was allocated to node6 and node7.

As the job is finished, let us check the job's output:

username@controller:~/helloworld$ cat helloworld_mpi-27560.err
username@controller:~/helloworld$ cat helloworld_mpi-27560.out
Hello world from processor node7, rank 7 out of 8 processors
Hello world from processor node6, rank 5 out of 8 processors
Hello world from processor node6, rank 4 out of 8 processors
Hello world from processor node6, rank 0 out of 8 processors
Hello world from processor node6, rank 2 out of 8 processors
Hello world from processor node6, rank 3 out of 8 processors
Hello world from processor node6, rank 6 out of 8 processors
Hello world from processor node6, rank 1 out of 8 processors
username@controller:~/helloworld$

As we can see, the job was allocated to 7 of node6's CPUs and 1 of node7's CPUs.

As we can also notice, the order of execution of each process is not deterministic, it has to do with the time the process reaches the processor, and also which other processes may be running and yielding in that same processor.

MPI programming is not really in the scope of this article, however if you are interested you can check these tutorials.

Getting job and node information

In part 1 of Slurm in Ubuntu Clusters we saw how to see the general node state with the sinfo command.

username@controller:~$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
base* up 3-00:00:00 1 idle compute-[1-4]
long up infinite 1 idle compute-[1-4]

In the previous section we also saw how to monitor the job submission queue with the squeue command.

There are three more commands that are also very useful to analyse the state of the cluster, and also the job history for a certain user or a set of the users.

The smap command is similar to the squeue, in a way that it also shows the slurm scheduling queue. However it also shows information about Slurm jobs, partitions, and set configurations parameters.

username@controller:~$ smap -c
JOBID PARTITION USER NAME ST TIME NODES NODELIST
27560 base username hellow R 00:00:05 1 node6,node7

The sacct command displays accounting data for all jobs and job steps in the Slurm job accounting log or Slurm database. With it you can see information about past submitted jobs.

username@controller:~$ sacct
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
27560.batch batch 7 COMPLETED 0:0
27560.0 hydra_pmi+ 2 COMPLETED 0:0

The sview command shows a graphical user interface to view and modify the Slurm state. Of course that you can only modify slurm objects to which you have access, i.e: you can cancel your submitted jobs but not one of another user.

To use this command you require X11 forwarding activated on your SSH session, if you are connecting from a windows machine you can use XMing, a X Window System Server for Windows.

If you are using Bitvise SSH Client, you can activate X11 forwarding on the client side by going to the Terminal tab and enable the X11 Forwarding option.

Conclusion

We have now seen how to set up a slurm cluster and how to manage its jobs and nodes.

There are a lot of details that I have skipped for simplicity, but the information present in this article should get you up and running on the way to master slurm clusters.

References

[MPITUTO]

MPI Tutorials [link]

Slurm in Ubuntu Clusters - Part 1

Mon, 17 Sep 2018 09:30:00 +0100

In the High Performance Computing (HPC) world it is usual to have a job scheduler. Its function is to receive job allocation requests, and job to the run on the system. In the image below we can see an example of an architecture that makes use of these systems.

Overview of job scheduling in HPC

In this post we will be looking into how to set up and configure [Slurm Workload Manager], formerly known as Simple Linux Utility for Resource Management [CCE2003]

Controller

The controller machine is where one should submit its jobs. It runs the slurmctld service, which controls the compute nodes. And in our case it will also run the slurmdbd service, which controls the accounting and qos settings for the queues.

Slurm controller components

O.S pkgs:

mailutils
slurm-llnl
sview
mariadb-client
mariadb-server
libmariadb-client-lgpl-dev
python-dev
python-mysqldb
slurm-llnl-slurmdbd

root@controller# apt-get install -y mailutils slurm-llnl sview mariadb-client mariadb-server libmariadb-client-lgpl-dev python-dev python-mysqldb slurm-llnl-slurmdbd

Database

After installing MariaDB we should confine the connections to the DB to localhost as we will be using this only for the slurmdbd installed in the local machine. In the file /etc/mysql/mariadb.conf.d/50-server.cnf we should have the following setting:

1

 bind-address = localhost

If you will be accepting connections from other machines, i.e if you want to have the database running on a separate machine than the one having the slurmdbd, then you should change this setting accordingly.

If you have the need to receive connections from multiple distinct subnets, you have to set this to 0.0.0.0 and create firewall policies blocking all incoming traffic to port 3306 except for those subnets.

Now, let us create the necessary databases and grant the slurm user rights to manage them, as user root:

root@controller# mysql -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.0.36-MariaDB-0ubuntu0.16.04.1 Ubuntu 16.04
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> grant usage on *.* to 'slurm'@'%' identified by 'safepassword'
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> create database slurm_acct_db
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> create database slurm_job_db
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> grant all privileges on slurm_acct_db.* to 'slurm'@'localhost';
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> grant all privileges on slurm_acct_db.* to 'slurm'@'localhost';
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> exit
Bye
root@controller#

Node Authentication

The subsystem responsible for authentication between the slurmd and slurmctl is munge. After installing the slurm-llnl package you should now have the munge package also installed.

First, let us configure the default options for the munge service:

/etc/default/munge:

1

OPTIONS="--syslog --key-file /etc/munge/munge.key"

Next, we create a key, common to all comunicating nodes, in /etc/munge/munge.key.

You can create a secret key using a variety of methods [MUN2018]:

Wait around for some random data (recommended for the paranoid):

root@controller# dd if=/dev/random bs=1 count=1024 >/etc/munge/munge.key

Grab some pseudorandom data (recommended for the impatient):

root@controller# dd if=/dev/urandom bs=1 count=1024 >/etc/munge/munge.key

Enter the hash of a password (not recommended):

root@controller# echo -n "foo" | sha512sum | cut -d' ' -f1 >/etc/munge/munge.key

Enter a password directly (really not recommended):

root@controller# echo "foo" >/etc/munge/munge.key

Now the service can be started:

root@controller# systemctl start munge

As a side note, if you are not able to manage binary files due to some kind of data conversion, i.e: using a configuration manager that may give you a hard time copying raw binary data, you can convert the file contents to a base64 representation of the data.

root@controller# dd if=/dev/random bs=1 count=1024 | base64 > /etc/munge/munge.key

Accounting Storage

The subsystem responsible for the job completion storage, maintaining information about all of the submitted jobs to the slurm system.

This is optional to install to have a functioning slurm system, but in order to have a more solid understanding of what happened with each past job it is necessary to have some record of those job.

After we have the slurm-llnl-slurmdbd package installed we configure it, by editing the /etc/slurm-llnl/slurmdb.conf file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


AuthType=auth/munge
AuthInfo=/var/run/munge/munge.socket.2
StorageHost=localhost
StoragePort=3306
StorageUser=slurm
StoragePass=safepassword
StorageType=accounting_storage/mysql
StorageLoc=slurm_acct_db
LogFile=/var/log/slurm-llnl/slurmdbd.log
PidFile=/var/run/slurm-llnl/slurmdbd.pid
SlurmUser=slurm

Now we can start the slurmdbd service:

root@controller# systemctl start slurmdbd

Central Controller

The central controller system is provided by the slurmctld service. This is the system that monitors the resources status, manages job queues and allocates resources to compute nodes.

The main configuration file is /etc/slurm-llnl/slurm.conf this file has to be present in the controller and all of the compute nodes and it also has to be consistent between all of them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64


# General
ControlMachine=entry-node
AuthType=auth/munge
CacheGroups=0
CryptoType=crypto/munge
JobCheckpointDir=/var/lib/slurm-llnl/checkpoint
KillOnBadExit=01
MpiDefault=pmi2
MailProg=/usr/bin/mail
PrivateData=usage,users,accounts
ProctrackType=proctrack/cgroup
PrologFlags=Alloc,Contain
PropagateResourceLimits=NONE
RebootProgram=/sbin/reboot
ReturnToService=1
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/cgroup
# Timers
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# Scheduler
FastSchedule=1
SchedulerType=sched/backfill
SchedulerPort=7321
SelectType=select/cons_res
SelectTypeParameters=CR_CPU_Memory
# Preemptions
PreemptType=preempt/partition_prio
PreemptMode=REQUEUE
# Accounting
AccountingStorageType=accounting_storage/slurmdbd
AccountingStoreJobComment=YES
ClusterName=mycluster
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
SlurmSchedLogFile= /var/log/slurm-llnl/slurmschd.log
SlurmSchedLogLevel=3
NodeName=compute-1 Procs=48 Sockets=4 CoresPerSocket=12 ThreadsPerCore=1 RealMemory=128000 Weight=4
NodeName=compute-2 Procs=48 Sockets=4 CoresPerSocket=12 ThreadsPerCore=1 RealMemory=254000 Weight=3
NodeName=compute-3 Procs=96 Sockets=2 CoresPerSocket=24 ThreadsPerCore=2 RealMemory=256000 Weight=3
NodeName=compute-4 Procs=96 Sockets=2 CoresPerSocket=24 ThreadsPerCore=2 RealMemory=256000 Weight=3
PartitionName=base Nodes=compute-1,compute-2,compute-3,compute-4 Default=Yes MaxTime=72:00:00 Priority=1 State=UP
PartitionName=long Nodes=compute-1,compute-2,compute-3,compute-4 Default=No MaxTime=UNLIMITED Priority=1 State=UP AllowGroups=long

In this config file we configured 4 compute nodes and 2 partitions.

In the node section we need to give a profile for each machine, which we can get with a combination of the output of /proc/cpuinfo and /proc/meminfo, and of course, knowledge about the underlying architecture.

The RealMemory parameter refers to the available memory you want to have available for resources, you do not need to match it with 100% of the available physical memory (and you should not, keep a few MBs reserved for the operating system).

Now we can start the slurmctld service:

root@controller# systemctl start slurmctld

Compute Nodes

A compute node is a machine which will receive jobs to execute, sent from the Controller, it runs the slurmd service.

Slurm compute node components

O.S. pkgs:

slurm-llnl
slurm-wlm-basic-plugins

Authentication

As mentioned before in Node Authentication, we need to setup the default options for the munge service, create a key, and start the service. On the compute nodes we can just copy the key file from the controller:

root@controller# for i in `seq 1 4`; do scp /etc/munge/munge.key compute-${i}:/etc/munge/munge.key; done

and then start the service:

root@compute-1# systemctl start munge

Compute Node Daemon

The subsystem responsible for launching and managing slurmstepd is slurmd. Slurmstepd is launched for a batch job and each job step, launches user application tasks, manages accounting, application I/O, profiling, signals, etc. [BMISC17]

As mentioned before in Central Controller the file /etc/slurm-llnl/slurm.conf has to be identical to the one created in the controller. Similar to the munge key, you can copy the file from the controller to the compute nodes.

Control Groups provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour. [KRNCGRP]

In the /etc/slurm-llnl/slurm.conf file you may have already noticed the following parameters:

TaskPlugin=task/cgroup
ProctrackType=proctrack/cgroup

The proctrack/cgroup plugin is an alternative to other proctrack plugins such as proctrack/linux for process tracking and suspend/resume capability. proctrack/cgroup uses the freezer subsystem which is more reliable for tracking and control than proctrack/linux. [SLUCGRP]

The task/cgroup plugin is an alternative to other task plugins such as the task/affinity plugin for task management, and provides the following features [SLUCGRP]

The ability to confine jobs and steps to their allocated cpuset.
The ability to bind tasks to sockets, cores and threads within their step's allocated cpuset on a node.
Supports block and cyclic distribution of allocated cpus to tasks for binding.
The ability to confine jobs and steps to specific memory resources.
The ability to confine jobs to their allocated set of generic resources (gres devices).

The task/cgroup plugin uses the cpuset, memory and devices subsystems.

There are many specific options for the task/cgroup plugin in cgroup.conf. The general options also apply. See the cgroup.conf man page for details.

/etc/slurm-llnl/cgroup.conf

1
2


CgroupAutomount=yes
ConstrainCores=yes

CgroupAutomount: In our case we want the cgroup plugins to first try to mount the required subsystems, instead of failing if these subsystems are not mounted.
ConstrainCores: We also want the subsystem to constrain allowed cores to the subset of allocated resources. This functionality makes use of the cpuset subsystem.

Due to a bug fixed in version 1.11.5 of HWLOC, the task/affinity plugin may be required in addition to task/cgroup for this to function properly. The default value is "no". [SLUCGRP]

Now we can start the slurmd service and start accepting jobs.

root@compute-1# systemctl start slurmd

Checkpoint

Let us check if everything is up and running and the slurm components are all communicating:

root@controller# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
base* up 3-00:00:00 1 idle compute-[1-4]
long up infinite 1 idle compute-[1-4]
root@controller#

Next Steps

Now we can start accepting job submissions, in part 2 of Slurm in Ubuntu Clusters we will see how to submit job scripts, allocate resources and see the status of the cluster.

Additional Notes

In order for the e-mail to work properly, we need to have a Mailing Transport Agent (MTA), i.e: postfix, the configuration of postfix is outside this post's scope.

In order to use MPI capabilities, the configuration depends on which implementation of MPI you will be using in your cluster. If you want to keep the default it should work, however performance wise you could be losing by not choosing the right MPI API.

There are three fundamentally different modes of operation used by different MPI implementations [MPISLU]:

Slurm directly launches the tasks and performs initialization of communications through the PMI2 or PMIx APIs. (Supported by most modern MPI implementations)
Slurm creates a resource allocation for the job and then mpirun launches tasks using Slurm's infrastructure (older versions of OpenMPI).
Slurm creates a resource allocation for the job and then mpirun launches tasks using some mechanism other than Slurm, such as SSH or RSH. These tasks initiated outside of Slurm's monitoring or control.

Links to instructions for using several varieties of MPI/PMI with Slurm are provided in the slurm documentation.

Use of an MPI implementation without the appropriate Slurm plugin may result in application failure. If multiple MPI implementations are used on a system then some users may be required to explicitly specify a suitable Slurm MPI plugin. [MPISLU]

References

[CCE2003]

Simple Linux Utility for Resource Management, M. Jette and M. Grondona, Proceedings of ClusterWorld Conference and Expo, San Jose, California, June 2003 [PDF]

[MUN2018]

Munge Installation Guide [link]

[BMISC17]

Slurm Overview, Brian Christiansen, Marshall Garey, Isaac Hartung (SchedMD) [PDF]

[MPISLU]

(1, 2) Slurm MPI Guide [link]

[SLUCGRP]

(1, 2, 3) Slurm cgroup guide [link]

[KRNCGRP]

Kernel cgroup Documentation, Paul Menage, Paul Jackson and Christoph Lameter [link]

Using Python Virtual Environments with Slurm

Mon, 10 Sep 2018 09:35:00 +0100

Using specific packages within a shared resources environment can sometimes be a hassle, some users may need different package versions, or even packages that conflict with each other. Relinquishing the package installation location to local user directories makes this process easier to manage.

In this post we'll see how to use virtual environments to manage your Python packages, this is useful not only to separate your Python packages from the system ones but also to be able to separate packages for each project, and end up making these environments portable and replicable across platforms.

As a plus we'll see how to create a Slurm batch file to submit a job to a Slurm queue

Overview

virtualenv - command used to create our virtual environments.
$HOME/.virtualenvs/ - the chosen directory to contain our virtual environments.
$HOME/.virtualenvs/AdS/ - the directory where this specific virtual environment will be placed in.
sympy - in this example we will be installing the sympy Python package.
source bin/activate - to enter the virtual environment we have to source its bin/activate file.
deactivate - to exit the virtual environment we have to issue the deactivate command.
AdS_sympy.sbatch - Slurm jobs use a batch file with specific headers.

Requirements

To be able to replicate this guide will need:

Virtual Environment

First let's create our virtual environment

t0rrant@bastion01 $ mkdir $HOME/.virtualenvs
t0rrant@bastion01 $ virtualenv $HOME/.virtualenvs/AdS

Now let's get into the virtual environment

t0rrant@bastion01 $ source $HOME/.virtualenvs/AdS/bin/activate
(AdS) t0rrant@bastion01 $

We can now install the sympy package

(AdS) t0rrant@bastion01 $ pip install sympy

Now let's take a look at our test file:

1
2
3
4
5


# AdS_sympy_test.py
#!/usr/bin/env python
from sympy import test
test("_basic")

We only need to be within the virtual environment to actually use the sympy package (i.e: execute our test file) or to install new packages. The virtual environment will be used by the Slurm job as we will see.

So, let us exit the virtual environment:

(AdS) t0rrant@bastion01 $ deactivate
t0rrant@bastion01 $

Slurm Batch File

A quick look into the batch file, we must activate the virtual environment before running our code (deactivating it here is not mandatory as the process will end):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# AdS_sympy_test/AdS_sympy.sbatch
#!/bin/bash
#SBATCH --job-name=AdS_sympy_test
#SBATCH --output=/home/t0rrant/AdS_sympy_test/AdS_sympy_test-%j.out
#SBATCH --error=/home/t0rrant/AdS_sympy_test/AdS_sympy_test-%j.err
#SBATCH --mail-user=<your e-mail should go here>
#SBATCH --mail-type=ALL
#SBATCH --time=1:00:00
#SBATCH --mem=1G
RUNPATH=/home/t0rrant/AdS_sympy_test/
cd $RUNPATH
source $HOME/.virtualenvs/AdS/bin/activate
python AdS_sympy_test.py

Slurm options:

`--job-name`	This will determine the job name which appears in commands like sacct, smap or sview.
`--output`	Determines the location of the file to which the stdout should be redirected to.
`--error`	Determines the location of the file to which the stderr should be redirected to.
`--mail-user`	The e-mail address to which job events should be sent to.
`--mail-type`	The types of events that should be sent, here we use all of them. For more info see [the docs].
`--time`	The amount of time we want allocated for this job.
`--mem`	The amount of memory we want allocated for this job.

Looks good, let's submit it to the queue:

t0rrant@bastion01 $ sbatch AdS_sympy.sbatch
Submitted batch job 27047

Now check for errors and output:

t0rrant@bastion01 $ cat /home/t0rrant/AdS_sympy_test/AdS_sympy_test-27047.err

No errors! =)

t0rrant@bastion01 $ cat /home/t0rrant/AdS_sympy_test/AdS_sympy_test-27047.out
============================= test process starts ==============================
executable: /home/t0rrant/.virtualenvs/AdS/bin/python (2.7.13-final-0) [CPython]
architecture: 64-bit
cache: yes
ground types: python
numpy: None
random seed: 7576360
hash randomization: on (PYTHONHASHSEED=1519633959)
sympy/core/tests/test_basic.py[16] ................ [OK]
================== tests finished: 16 passed, in 0.09 seconds ==================
t0rrant@bastion01 $

As we can see it used the python executable from our virtual environment, used the sympy package also from that virtual environment and successfully ran the basic tests from the sympy package.

UEFI via software RAID with mdadm in Ubuntu 16.04

Fri, 24 Aug 2018 09:35:00 +0100

Introduction

In this post we'll see how to install an UEFI system within a RAID 1 array using mdadm, in Ubuntu Server 16.04.

For this guide we need two images:

systemrescuecd-x86-5.2.2 [download]
Ubuntu 16.04.5-server-amd64 [download]

Disks: 4x 4TB (4096 block native, 512 emulated)

Disk Layout:

100MB EFI vfat /boot/efi
16GB SWAP swap
40GB ROOT ext4 /
(remainder)

Prepare the disks

First we need to prepare the disks for the RAID, to do that we will use the System Rescue CD as it has all the tools we will need, we start with parted:

(here we use MB units to ensure partition alignment, which we can check for in the end, although the partition actual size may differ, for small sizes)

root# parted /dev/sda
(parted) mklabel gpt
(parted) unit MB
(parted) mkpart ESP 0\% 100MB
(parted) set 1 esp on
(parted) mkpart primary linux-swap 100MB 16.1GB
(parted) mkpart primary 16.1GB 56.1GB
(parted) mkpart primary 56.1GB 100\%
(parted) print
Number Start End Size File System Name Flags
1 1049kB 99.6MB 98.6MB fat32 EFI msftdata
2 99.6MB 16.1GB 16.0GB linux-swap Linux RAID
3 16.1GB 56.1GB 40GB Linux RAID
4 56.1GB 4001GB 3945GB Linux RAID
(parted) align-check optimal 1
1 aligned
(parted) align-check optimal 2
2 aligned
(parted) align-check optimal 3
3 aligned
(parted) align-check optimal 4
4 aligned

Copy partition layouts using sgdisk:

root# sgdisk --replicate=/dev/sd{b..d} /dev/sda

Randomize UUID for each partition:

root# sgdisk -G /dev/sd{b..d}

Note that the recommended size for the EFI Partition ranges from 100 to 250 MB.

Create RAID array:

root# mdadm --create --metadata=1.0 --name=hostname:efi /dev/md/efi --level=mirror --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
root# mdadm --create --metadata=1.0 --name=hostname:swap /dev/md/swap --level=mirror --raid-devices=4 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
root# mdadm --create --metadata=1.2 --name=hostname:root /dev/md/root --level=mirror --raid-devices=4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

The other partition is left alone as it is out of the scope of this guide.

Time to reboot to the Ubuntu Server image.

Install Ubuntu 16.04

After choosing/preseeding the usual installer checkboxes, we arrive at partman stage, here we can usually see our raid arrays already assemble. In order to choose the right md device we can switch to a separate tty (usually by Alt+F<2-6>, or Alt+<left/right>) and either check in /etc/mdadm/mdadm.conf for the mdXXX name corresponding to our array, or run:

root# cat /proc/mdstat

followed by:

root# mdadm --detail /dev/mdXXX

so that we can check the name of each md device.

After we are sure of which is the ESP, the swap and the root partitions, we can choose accordingly back in the installer menu (Alt+F1, or Alt+F7)

Here we choose EFI System Partition for the efi, SWAP space for the swap, and ext4 mounted on / for the root filesystem.

All done!

Troubleshooting

But wait, sometimes you can (likely) come across issues when the installer tries to run grub-install.

This may be because you did not boot the installer in UEFI.

If you did and still get the error, keep reading.

Let's reboot into rescue-mode from the Ubuntu installer by choosing rescue a broken system

Now lets choose to execute a shell into the installed system, with the root md device (see above to be sure how to sure the root md device) as the root filesystem.

root# mount /boot/efi

If you get an error, confirm that your UUID is correct in /etc/fstab

Check for installed files in /boot and /boot/efi.

If you have no files there:

root# update-grub
root# grub-install --efi-directory=/boot/efi --uefi-secure-boot

If your UEFI boots without NVRAM entries, you can disable the NVRAM writing via the “Update NVRAM variables to automatically boot into Debian?” debconf prompt when running [ref]:

root# dpkg-reconfigure -p low grub-efi-amd64

If not, you can use efibootmgr to edit the EFI boot entries.

In order to use efibootmgr the system has to be in EFI mode, run the following command to check for the existence of efivarfs:

root# mount | grep efivars
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,relatime)

In this case it is mounted read-only through the sysfs init script, so it needs to be remounted rw manually using the following command:

root# mount -o remount,rw -t efivarfs efivarfs /sys/firmware/efi/efivars

Let's take a look at the EFI entries

root# efibootmgr -v
...

If there is no entry corresponding to /dev/disk/by-partuuid/** -> /dev/sd{a..d}, we need to add them manually:

root# for i in a b c d; do efibootmgr -c -d /dev/sd\${i} -p 1 -L "ubuntu" -l "\\efi\\ubuntu\\shimx64.efi"; done
root# efibootmgr

For more info about efibootmgr check out the Gentoo wiki

Now we should be ready to go!

But wait, there is more...

... if the UEFI writes anything to one of the drives, it may lead to corrupted results once Linux mounts the RAID (since the member drives won’t have identical block-level copies of the FAT32 any more).

To deal with this "external write" situation, and since mdadm has the "--update=resync" assembly option, we can use the following approach, :

Prefer one RAID member’s copy of /boot/efi and rebuild the RAID at every boot. If there were no external writes, there’s no issue. My approach here is I really do not care which version of the ESP gets copied over, if there is something to be written in the ESP and it gets copied over to all RAID members we should be fine.

This requires updating /etc/mdadm/mdadm.conf.

I've tried adding on the RAID’s ARRAY line to keep it from auto-starting:

1

ARRAY metadata=1.0 UUID=XXX

however it seems that mdadm refuses to assemble the array as it is explicitly ignored. So I just delete the ARRAY line corresponding to the efi md device.

We now add the efi partition to the fstab (or modify the existing one):

1

UUID=XXXX-XXXX /boot/efi vfat noauto,defaults 0 0

And finally we can add a systemd oneshot service that will assemble the RAID with resync and mount it:

/lib/systemd/system/efimount.service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


[Unit]
Description=Resync /boot/efi RAID
DefaultDependencies=no
After=local-fs.target
[Service]
Type=oneshot
ExecStart=/sbin/mdadm -A /dev/md/efi --uuid=YY:Y:Y:Y:YYY --update=resync
ExecStart=/bin/mount /boot/efi
RemainAfterExit=yes
[Install]
WantedBy=sysinit.target

We also run:

root# update-initramfs -u

so the initramfs has an updated copy of /etc/mdadm/mdadm.conf

NOTE: XXXX-XXXX is the MD device's uuid, however YY:Y:Y:Y:YYY is the UUID common for each partition /dev/sd{a..d}1 (our ESP partition) that you can get by running:

root# blkid | egrep "/dev/sd.?1"

Now yes, we can reboot and provision the system!

A big thanks to Kees Cook as his blog was one of the very limited resources about this use case on the web.