Tyler Fitch

Searching by a Node's cookbook's attributes

Fri, 05 Sep 2014 22:15:27 -0700

We’re working on a method for self discovery of a group of servers being setup to be a MongoDB replica set. And a specific replica set among an environment with multiple MongoDB servers and replica sets at that.

TL;DR A cookbook’s node attributes are uploaded back to the server and available to be searched against. Here’s the secret to searching on the Node’s attributes defined in the cookbook. In the cookbook we had node[‘mongodb’][‘replica_set’] but when we wanted to search for it we did mongodb_replica_set:ReplSet1 - chaining the full name of the attributes with underscores. Doesn’t entirely seem intuitive, but it is the way Search works.

Out of all the servers in a Chef server Organization we’re going to use three keys to identify the machines of a single replica set.

1) A Role - http://www.getchef.com/blog/2013/11/19/chef-roles-arent-evil/ For this I like having two roles, mongodb-primary and mongodb-secondary. The difference will be the run_list. Both will have the recipe[‘mongodb’] which will install, configure and startup MongoDB on the servers. But the mongodb-primary role will have the run_list of recipe[‘mongodb’],recipe[‘mongodb::primary’] to install MongoDB and configure the replicaset.

2) The Environment - https://docs.getchef.com/essentials_environments.html

3) A replica set name, set via a cookbook attribute default['mongodb']['replica_set'] = 'ReplSet1'

Not everyone in the Chef community may love Roles and Environments, but this is a great use case for them. Treating the combination of the three like a compound key in a database table, we’re able to identify like machines to be joined together in to the replica set.

Searching for the Nodes

Using the above three values, what do the search commands look like? First up from knife we’d have

This is nice to verify you’ll get the desired search results without having to run a cookbook and debug the search syntax by doing a node converge.

Now we’ve got the search syntax worked out, we’ll want to actually use this in a cookbook to create a replicaset configuration file.

There we have it. We’ve found all the machines we want to be in the same replicaset and configured the replicaset by running the JS file on the Primary machine of the MongoDB replica set.

Remember, only one machine in the replica sets needs to run this script, the primary. AND it will need to run it last, after all the nodes in the replicaset are running (you can’t add a machine to a replica set if it isn’t running and configured to be a mongodb yet).

Sending name/value pairs from Chef cookbook attributes to be dynamic Resource attributes

Mon, 01 Sep 2014 14:05:00 -0700

One of my customers had a question about using name/value pairs from his cookbook’s attributes as settings for a Resource.

TL;DR - The name/value pairs in a cookbook Resource object are more than just a combination of attribute + value. More precisely, they’re a method_name + value combination. So making the method_name come from a dynamic source requires you use send() method of a Ruby Object to invoke the method, not just output a String equal to what the cookbook would look like if everything was hard coded.

Long form:

Let’s start with the hardcoded cookbook recipe and see what that would look like.

Now to convert all those Resource attributes to be dynamic let’s convert them in to Attributes of the cookbook

A first attempt at setting the attributes looked liked this, but it did not work

At first pass this all looked correct to me too. I was stumped until my brilliant co-worker Steve Danna (@SteveDanna) shared some great Ruby/Chef knowledge with me. The correct way to achieve the goal would be this

The key here is the send() method. http://ruby-doc.org/core-2.1.2/Object.html#method-i-send

When you do #{setting} #{value} then setting is just returned as a string and not *calling* the relevant setter method of the Resource object (iis_pool in this case).

I guess this surprised me, because when the cookbook is hardcoded you see the line thirty_two_bit false and it reads like two strings and not that thirty_two_bit is actually a method name being called.

Alas, now I know and you do too.

#ChefConf 2014 BoF - Creating Deploy Jobs in Jenkins

Mon, 28 Apr 2014 23:29:00 -0700

At #ChefConf 2014 I had the honor of leading a Birds of a Feather session on “Creating Deploy Jobs in Jenkins”.

My interest here is in deploying a web app to my Chef managed servers. The problem I need to solve is for when you don’t know what servers you have because in the Cloud the servers come and go. Servers are cattle - not pets, right?

So what are our options?

Doing it manually - bleh: Hardcoding server addresses in a Jenkins deploy job won’t work. Well, it’ll work until you shoot your cow and spin up a new one. Manually updating the Jenkins deploy job with the current server addresses sounds about as fun as herding cats and won’t help you Continuously Deliver.

Use the Chef Server + knife status to lookup what servers you have. When combined with roles + environments, this gets an accurate reading of what’s out there to deploy to. I’ve found the output of knife status is also much friendlier to command-line fu than the knife search output.

knife status “role:webserver AND chef_environment:stage” will give me the following output.

12 mins ago, ec2-57.186.102.29-1396555868.17, ec2-57-186-102-29.us-west-2.compute.amazonaws.com, 192.31.9.43, centos 6.4.
8 mins ago, ec2-57.82.206.136-1396555883.36, ec2-57-82-206-136.compute-1.amazonaws.com, 57.82.206.136, centos 6.4.

So I want to cherry pick the 3rd item of the comma-separated list, the hostname. I could use awk, but found a simple example using cut and went with it.

I loop over the output of the knife status search like so

for h in $(knife status “role:webserver AND chef_environment:load”| cut -d’,’ -f3); do ssh -oStrictHostKeyChecking=no myuser@$h “uptime”; done

The -oStrictHostKeyChecking=no is there because I’ve maybe never deployed to this cow before. Change the ssh “uptime” command to the necessary scp command and you’re deploying.

But it isn’t bulletproof. If a server crashes and doesn’t unregister itself as an active node with the Chef server you’ll try to deploy to a place that is no longer valid. One way to mitigate this is use the timestamp column in the `knife status` result that shows the last time the server phoned home. If all servers are checking in every 15-30 mins and it’s been an hour since the Chef client last checked with the Chef server then skip trying to deploy to it.

Amazon AWS API options:

During the discussion two options for Amazon Web Services were identified. First up, use the AWS API to query for the list of active servers to deploy to. If you tag them or have other means to identify their role in your infrastructure you’ll know what’s active at that time. I imagine other cloud hosting providers will have similar APIs available. Secondly, Kennon Kwok from Chef tipped me off to a nice library from Netflix called Edda which “polls your AWS resources via AWS APIs and records the results. It allows you to quickly search through your resources and show how they have changed over time.” In other words, Edda is wrapper for the AWS API that adds in historical data too (among other things). Nice.

Both of these options are nice a would be more accurate than polling the Chef server. But in my environment I actually have a tool called Scalr fronting my AWS account. So I can create servers but do not actually have the account’s AWS secret key use the AWS API or Edda! It’s a hard knock life in the corporate world of cloud billing and measures to consolidate billing for their numerous AWS accounts.

Lesser known options:

Also discussed were the ideas of using Apache Zookeeper, Serf from Hashicorp or etcd from CoreOS. Zookeeper was not highly rated for WAN setups with machines not on the same networks. And there was not enough experience with Serf or etcd in the room to give them a proper evaluation.

The end result for me?

I’m still proceeding with my initial plan that uses Chef Server + knife status lookup command, but now know a few more options to use in the future to solve to create my deployment jobs in Jenkins.

Thanks to Kennon, Eric from Boston, Mahesh from Dow Jones, the Behance guys and the others that I didn’t get your names but joined in the conversation for the BoF. I had a great time and hope you all did too.

Installing Sun Java 1.5 with Chef

Fri, 28 Jun 2013 21:25:00 -0700

In trying to use the Opscode Chef community cookbook for Java I found it to only work with Java 6 and Java 7.

In trying to create a script to recreate an existing environment I need a scripted install of the Sun 1.5 JDK.

So I had to bail entirely on the community cookbook and go my own route. Here’s how it played out.

Download the installer you need. In my case the .bin file - without the rpm, just the bin.
Upload the .bin to your Artifactory server so everyone can easily get access to it when building their VMs.
Get the installer on to the VM using Chef remote_file command with mode 0700 to ensure the installer is executable.
Run the bash command to run the installer
The installer requires acceptance of the EULA. So scripting this was a pain. Hat tip to Kule on linuxquestions.com - http://www.linuxquestions.org/questions/linux-newbie-8/java-jdk-silent-install-help-891313/#post4433389
The EOH here is important - EOF did not work nor the command attribute.
Create a environment variables $JAVA_HOME just like the Java cookbook would and many Java apps now expect to exist.

One thing this does not do is put the java and/or javac executables in to your $PATH. I’d do this like the Java cookbook does and do symlinks to /usr/local/bin if you need it.