This example pertains to the AMANDA backup server software. It requires a definition for a sequence of virtual tapes. These are just empty directories that need to be created and then registered using the AMANDA “amlabel” command. All I wanted to do was to say to Puppet, “Create me n virtual disks, where n is an integer variable”. If it was a simple matter of creating an array [ 1..n ] and then passing this to a define, there wouldn’t be a problem, but there’s no way in Puppet to create a variable-length array. One can only do something like this:

$array = [ 1,2,3,4,5 ]

which is no use when the number of elements is not known.

But one simple way to accomplish a variable length sequence is through recursion. Consider this puppet define:

define amanda_create_slot ($backup_set, $slot_path, $slot_num ) {
  if "${slot_num}" > '0' {
    exec { "slot${slot_num}":
      creates => "${slot_path}/slot${slot_num}",
      user => "amandabackup",
      command => "/bin/mkdir $slot_path/slot${slot_num};
	/usr/sbin/amlabel $backup_set $backup_set-${slot_num} slot $slot_num"
    }
    $decrement = inline_template('<%= @slot_num.to_i - 1 %>')
    $amanda_create_slot { "slots-${decrement}": backup_set => $backup_set,
		                                slot_num => $decrement,
			                        slot_path => $slot_path }
  }	
}

It checks that the slot number passed to it is greater than zero, and if so, creates a slot with that index. It then decrements the argument “slot_num”, and then invokes itself again. It will continue to do this until the variable slot_num is reduced to zero, at which point the if-block will fail and the recursive loop will collapse.

In a manifest, this function is initially invoked like this:

amanda_create_slot { "slots": backup_set => Daily,
			      slot_num => "10",
			      slot_path => "/var/amanda/slots" }

}

The result will be a nice series of virtual tape slots, all numbered from 1 to 10.

The way scheduled downtime works in Icinga (these instructions apply to Nagios as well) is described here. Briefly, when downtime is scheduled for a host and/or services, then no alert notifications are sent out during the defined period. Downtime gets scheduled with a start time, an end time, and a duration, and may be either fixed or flexible (the difference is, flexible downtime will start from the moment when a host or service goes down inside of the scheduled period).

Icinga offers a REST API to control it from the command line, but to be honest, it’s a little tricky to use and I can’t work out how to schedule downtime with it, rather than to just disable notifications. So instead, I find the command-file pipe a simpler solution – it requires access to the monitoring server itself, but that’s not really a problem.

The Icinga command-file pipe is a special file known as a FIFO that acts as a pipe into a process, in this case, the daemon itself. Therefore, whatever gets written to this file gets funnelled straight into the Icinga process. It’s location can be found by checking the “command_file” option in the /etc/icinga/icinga.cfg main config file. For example:

/etc/icinga/icinga.cfg:

command_file=/var/spool/icinga/cmd/icinga.cmd

This file is owned by the icinga user, so security is controlled by permissions in that only the icinga user or a member of the icingacmd group can write to this file. For this reason, you’ll need to update /etc/sudoers to permit your Capistrano user to write to the command file as the icinga user. Something like this, using the “tee” command:

/etc/sudoers

ALL=(icinga) NOPASSWD: /usr/bin/tee -a /var/spool/icinga/cmd/icinga.cmd

Then, all that one needs to do is send a formatted string to the command pipe and Icinga will execute the instruction accordingly. The Icinga and Nagios documentation describes the order of each semicolon-separated field.

Setting up a full Capistrano project is beyond the scope of this post, but assuming you have one, the files below will get things working.

This is a helper method to simply execute an Icinga command, for a particular host and taking one parameter, for the number of minutes of downtime.

capistrano/recipes/helpers/icinga_commands.rb:

def icinga_cli_cmd ( icingaCommand, hostName, minutes )
   run <<-CMD
      CMDFILE="/var/spool/icinga/rw/icinga.cmd";
      MINUTES=`expr #{minutes} * 60`
      NOW=`date +%s`;
      STOP=`expr $NOW + $MINUTES`;
      DURATION=$MINUTES;
      AUTHOR="Capistrano";
      COMMENT="Automation";
      CMDLINE="[$NOW] #{icingaCommand};#{hostName};$NOW;$THEN;1;0;$DURATION;$AUTHOR;$COMMENT";
      echo $CMDLINE | sudo -u icinga /usr/bin/tee -a $CMDFILE
   CMD
end

This is the Capistrano recipe to scheduled downtime for a host and all its services.

capistrano/recipes/icinga.rb:

namespace "icinga" do
  task :config do
    close_sessions
    top.load(:string => "set :user, 'deploy'")
  end

  desc "Schedule Icinga Downtime of host and all services"
  task :downtime_host_svc, :roles => :monitor do
    config
    icinga_cli_cmd "SCHEDULE_HOST_SVC_DOWNTIME", "#{hostName}", "#{period}"
  end
end

And within the host definitions themselves, you'll need something like this:

capistrano/deploy/development.rb:

role :monitor, "monitor.example.com"

Your particular setup may involve some tweaking, and this is just an example that can easily be extended to better control Icinga. To invoke this Capistrano recipe from the command line, scheduling 7 minutes of downtime on the "backup" host, execute this command:

$ cap development icinga:downtime_host_svc -s hostName=backup -s period=7

This will send a string similar to this one to the Icinga daemon, and this should also be reflected in the icinga.log file.

[1396341638] SCHEDULE_HOST_SVC_DOWNTIME;backup;1396341638;;1;0;;Capistrano;Automation

Check the effect by viewing the log file and checking the host display in the Icinga web UI.

I use this recipe in a Jenkins job that restores a database snapshot. It means that while the database is being restarted I can suppress warnings. What's the point of being notified of what you already know?

The problem is, if the parallel commands you’re executing are all hitting the same network resource at exactly the same time, you can get timeouts, failures, or possibly even look up the daemon. I’m looking at you, Puppet (sometimes “puppetd -t –splay” just doesn’t cut it).

What I like to do to prevent everything being too simultaneous is to precede the command with a random sleep pause, like this:

sleep $(( $RANDOM*8/1000 )); puppetd -t --noop

This will randomly sleep for any time between 1 and 8 seconds before continuing with the Puppet agent update, saving the Puppetmaster and its precious thread count.

This could similiarly be used during YUM or APT updates, but really, if you have to, you should probably be automating these kinds of solutions.

cfg = rs.config();
cfg.members[1].priority = 2;
rs.reconfig(cfg);

However, with a bit of mongo javascript it’s possible to write a small function to cycle through the member config array, determine the index of a host and then set an attribute.

In this example, we want to configure the Mongo DB nodedb02.example.com as a hidden node. This can be done like this:

cfg = rs.config();
cfg.members.forEach( 
	function(item, i) {
		if ( item.host == "nodedb02.example.com" ) 
		{ 
			cfg.members[i].priority = 0;
			cfg.members[i].hidden = true;
			rs.reconfig(cfg);
		}  
	} 
)

If, on the other hand, you wanted to change the slave delay on a read-only reporting node, this could be accomplished by testing each node to find which has the “hidden” attribute set to “true”:

		if ( item.hidden == "true" ) 
		{ 
			cfg.members[i].slaveDelay=0
			rs.reconfig(cfg)
		}  

Use this function in Mongo JS scripts, be it with Capistrano, Puppet or some other automation technique, and everything got just a bit more hands-off.

In the vi session, enter command mode by pressing ESC. Then, set the size of the tabs with the following command:

  :set shiftwidth=2

To make this the default, put “set shiftwidth=2” in your ~/.exrc file.

Then, to tab shift the next 13 lines, you’d type a command like this:

13>>

When one connects a Mongo database process without credentials, using the “mongo” client command, the implicit “test” database is used. This is a default, and a kind of empty chroot database. Even though it’s an empty database, it still has read-only access to replica information. This can be very handy when automating deployment and monitoring scripts.

There are two commands in particular that are useful for determining the status of a replica set member. These are demonstrated in the following console example, which is fairly self-explanatory.

$ mongo localhost:27017
connecting to: localhost:27017/test
> db.isMaster().secondary
false
> db.isMaster().ismaster
true

This could also be executed non-interactively like this:

mongo localhost:27017 --quiet --eval 'db.isMaster().secondary'

Shell Script Snippet: wait for replica synchronization to finish

When resynchronizing a replica set slave, it will report its status as RECOVERING until it completes, at which point it becomes SECONDARY. The simple example below demonstrates a block of bash to test for this.

while [ "$RESULT" != "true" ]; do
   RESULT=$(mongo localhost --quiet --eval 'db.isMaster().secondary');
   sleep 5;
done

This is over-simplified to demonstrate the logic. In a real-world script you’d have an exit clause to prevent an endless loop in the case when the replica node never recovers.

Other useful commands can be discovered while browsing the mongo console. Find these using commands of the format:

  db.help()

One can use wget or curl for this (this example uses wget) and the external site ipecho.net, which offers an unformatted page displaying the IP address.

   wget -q -O - http://ipecho.net/plain && echo

This expression could be executed inside a subshell and the result written to a variable. From here, this information could be useful for checking that a VPN client is connected.

I’m not going to add each recruiter to my Contacts as it arrives – that would be time consuming to start and difficult to maintain. But when it comes time to make a great big recruiter contact list, I’ve written the Python script below to scrap the entire mailbox and output each unique email address.

#!/usr/bin/python

import imaplib
import sys
import email
import re

#FOLDER=sys.argv[1]
FOLDER='Employment'
LOGIN='example.address@gmail.com'
PASSWORD='xxxxxxxxxxxx'
IMAP_HOST = 'imap.gmail.com'  # Change this according to your provider

email_list = []
email_unique = []

mail = imaplib.IMAP4_SSL(IMAP_HOST)
mail.login(LOGIN, PASSWORD)
mail.select(FOLDER) 

result, data = mail.search(None, 'ALL')
ids = data[0]
id_list = ids.split()
for i in id_list:
	typ, data = mail.fetch(i,'(RFC822)')
	for response_part in data:
		if isinstance(response_part, tuple):
			msg = email.message_from_string(response_part[1])
			sender = msg['from'].split()[-1]
			address = re.sub(r'[<>]','',sender)
# Ignore any occurences of own email address and add to list
	if not re.search(r'' + re.escape(LOGIN),address) and not address in email_list:
		email_list.append(address)
		print address

I’ve hard-coded my email login, password and the mailbox name, although it’s easy enough to modify the script to enter them as argmuents (I’ve commented out a line demonstrating this).

In a later post, I’m going to discuss how I use this script for job-seeking.

In its most basic form, with few options, strace can print out every system call that a process makes, and if the “-f” or “-ff” switches are used, also print out all the system calls from the child processes that the process forks. Like this:

  # strace -f -p <PID>

This provides far too much information though, so it’s often desirable to run strace with filter expressions to narrow the search down, specified with the “-e” switch. The man page provides an exhaustive list of these filters.

To specify a filter for reading STDIN, STDOUT and STDERR, the following expression will do the trick:

  # strace -ff -e trace=write -e write=1,2 -p <PID>

Make use of this how you will.

You’re probably used to pulling up the previous Bash line with the “Up” arrow. You probably also frequently substitute the last word from the previous line with “Ctrl-Underscore”. But next time you need to execute the previous command, but with a different argument, rather than hitting “Up”, deleting the last argument and retyping the new argument, try using !:- in place of the first word (that is, the command).

So if the previous command was:

   # /opt/splunk/bin/splunk start

And you next wanted to execute it again, but with a “stop” argument, do this:

   # !:- stop

The Bash shell will substitute !:- with the text from the previous command. No grabbing for the mouse and copying and pasting. Remember, according to Larry Wall, creator of Perl, Laziness is the second virtue of the great programmer.