The Adventures of Netman

Introducing UpdateDay patching

2015-01-13T06:09:00.002-08:00

In the school district where I work, we have a large number of laptop carts. Day to day usage of these carts varies greatly. Most are used often at some locations while at the others they will sit in the carts for days or weeks at a time. We are also dealing with a legacy carryover - generic accounts. The transition away from these generic accounts has been slow, mainly due to support issues (i.e. we have a small support staff). Since the laptops are often used with the same account, users find it easy to just close the lid when they put them away at the end of their session. The result of this is that we have laptops that are often not logged-out or shutdown for months at a time. Also, they often leave browsers and other applications open. Patching these computers has become a nightmare. We’ve been using Casper for a few years to deploy software, but one of the drawbacks to using it as it comes out of box its that it will, if you’re not careful, install software while a user is logged in and using the software. This has caused us to receive numerous angry emails and phone calls telling us how evil we are for disrupting their work and ruining their computer. Casper does have the ability to install on startup/shutdown or on login/logout but as I’ve stated above, neither of these happen consistently in our environment. To combat this, we were looking for a method to deliver updates to our users.

Additionally, we will be taking part in the upcoming PARCC online testing later in the school year. As a part of the preparation for these tests we have had to verify that each computer has the prerequisites for the test. In order to prove this, at a glance, we create reports in the JSS. We have Extension Attributes that display the requirements along with other information. We periodically distill this information into reports that we provide to upper management to calm their fears that we will not be ready in time.

After watching Greg Neagle demonstrate AutoPkg last summer at the Penn State MacAdmins conference, we began using AutoPkg with Alister Banks’ JSS-autopkg-addon last summer. For those that are unfamiliar with it AutoPkg is a software package that can be used to download commonly used software updates and package them into an easily deployable format. So now I have in our JSS the latest versions of the software that we use daily. I just needed to find a way to deploy it in a consistent manner.

I discovered the final piece to the puzzle when Rich Trouton released his First-Boot-Package-Install package. This packages, after a restart, will iterate through a series of folders and install the software packages that it finds.

The Solution

Phase 1: Staging the Update Packages

The first part of the solution we needed to find a method to copy the software packages into a “secret location” on the computer and store them in folders to be used by the first boot packages. In order for this to work properly, the packages need to be stored in a numbered folder system. To accomplish this I developed the script below:

#!/bin/bash

pkgname=$4

echo "Package is "$4

basedir=‘/secretlocation/'

waitingroom='/Library/Application Support/JAMF/Waiting Room/'

source=$waitingroom$pkgname

echo "Cached file is "$source

for tens in {0..9}

for ones in {0..9}

foldernum=$tens$ones

update_dir=$basedir$foldernum

#echo $foldernum,$update_dir

if [ ! -d "${update_dir}" ]; then

mkdir -p ${update_dir}

break 2

done

destination=$update_dir/$pkgname

echo "The destination is "$destination

# Move the cached package into the appropriate folder

mv "$source" "$destination"

# Remove the Casper receipt (XML) file

echo "Cleaning Casper receipt" $source".cache.xml"

rm "$source".cache.xml

exit 0

A Casper policy is used to cache the package to the computer. A numbered folder gets created and the packages get moved into the next folder. Then we remove the Casper "receipt" file so that it doesn't think that something is there to install. In the below picture you can see that I have it set to install via a custom trigger during testing. This trigger will be maintained during deployment as you will see later. Pre-deployment we will use the recurring trigger to stage the packages prior to the UpdateDay.

A numbering scheme is used to ensure that the packages get installed in the proper order.
The script checks for the existence of a numbered folder in the $secretlocation and increments its counters until if finds an available one to use (00-99)
The script then moves the package into the numbered folder
The name of the package is passed to the script as $4. This setting is set in Casper Admin.

By setting the parameter name here it is passed to the web interface when setting up the policy. This serves as a reminder to the creator of the policy that they are supposed to enter something here.

The script then moves the package to the secretlocation. The package name is used here : (1) so that we will only move only the intended package (packages cached by any other policies are not disturbed;

The receipt file that Casper creates is deleted so that if a “install all cached packages” policy is run, it will not look for this package. This is repeated for all of the packages that we will be installing.

The last package that we install is a payload-free package that has script that runs a recon. This ensures that the inventory in the JSS has up to date information that we can use to verify the process has completed successfully.

Phase 2: Installing the Update Packages

After the packages are downloaded, we can then move to installing them. Here we’ll be using another policy and script combo called via a custom event "trigger". At deployment time we set the policy to be active at startup so that we can tell the user to simply restart the computer to get the updates. The process can also be started when the trigger is called via a "jamf policy -event updatekickoff" statement via script, command-line, or remote command. To give you a better idea, here is how the policy looks:

Here’s the update_kick.sh script that we’re using to install of the First-Boot-Package-Install packages:

#!/bin/bash

jamf='/usr/sbin/jamf'

secretloc=‘/secretlocation/'

# Check for FBI packages

if [ ! -f "${secretloc}fbi.pkg" ]; then

$jamf policy -event fbi-install

else

if [ ! -f "${secretloc}fbiswu.pkg" ]; then

$jamf policy -event fbi-install

/usr/sbin/installer -dumplog -verbose -pkg "${secretloc}fbiswu.pkg" -target /

$jamf policy -event patching

exit 0

This script checks the $secretlocation for the First-Boot-Package-Install packages. If they are missing it uses a custom trigger, “fbi-install”, to download them to the $secretlocation and then installs the desired version. (We’re installing both versions, with and without Apple Software Updates, onto the computer so that we have them in the future.)

Then, we re-check for any packages waiting to be cached via the trigger “patching". As I’ve stated above, sometimes the laptops sit in the carts for weeks, so they won’t have an opportunity to download the updates. This way they’ll have the ability to pull down the packages. The other computers will also then be able get any last minute packages. Then, once everything is in place, the policy reboots the computer.

One of the reasons why we wanted to move away from Casper's install at startup process is that it doesn't provide any feedback to the user - the computer just sits there and appears to be frozen. Initially we wanted to provide some kind of feedback to the user during the process, but it seems that Apple has removed some of that ability in 10.9. Indeed, when we ran versions of this script using CocoaDialog, JamfHelper and system Notifications we had errors related to WindowServer not being able to start at every line of code. We're using Casper's built-in notifications to give a warning to the user if someone is logged in.

The First Boot Packages use Per Olofsson’s LoginLog application to prevent the user from logging in by displaying a log window overtop of the login window as seen below. This fulfills our desire to provide feedback to the user. It also prevents them from logging in until the installations have finished. After the packages have been installed, the computer restarts and is ready for use.

Phase 3: Adding Self Service to the Process

At the outset, we knew that we would have to come up with a way of updating the staff computers as well. Given the number of and location of the computers we have, there is no way that we would be able to touch all of them. After allowing the above procedure to run for a couple of weeks successfully, we decided to see if we could extend the procedure to be run from Casper's Self Service application. We could then use this to allow our teachers and administrators to install updates at their convenience. It turned out to be easier than we initially thought, a two-step process in fact. All we had to do was add the additional computers to the update package scopes. We then duplicated the "Update Kickoff" policy, gave it an cool official sounding name (2015 Software Update 01), set it for Self Service and released it to our staff.

Lessons Learned and Future Additions

Initially we had a “/sbin/reboot” command at the end of the script, but we quickly learned that it caused the computer to enter a continuous rebooting loop because the policy would never finish.

The first question that was asked after I sent out the instructions for how to update the staff computers was "what about software updates?" We have a Self Service item for installing system updates, but other than a progress bar with minimal text, again there is no feedback to the user. Also, the user has to continually run it to make sure that everything has been installed. As mentioned above, there is a version of the First-Boot-Package-Install packages that installs Apple Software Updates (First Boot Package Install With Automated Apple Software Update). This package will continually check for updates until there are none remaining. We will be creating a Self Service policy, based upon the one described above, to do this. We decided to keep them separate because users sometimes become impatient when an installation seems to take "forever" to finish. This way, they can choose what they will be updating and when it'll be done.

We quickly discovered that missing from this process was the ability to monitor progress. Because this process is done outside of Casper, we couldn't easily use its tools to troubleshoot. We could wait until the next Recon ran, but that wouldn't give us immediate feedback, or tell us what went wrong. Fortunately the process keeps a log file. We could query the log remotely if needed, but that wouldn't give us something that we could capture and analyze easily. Since there is a log file, we're thinking that we should be able to send them to a syslog server.

The road to Reposado

2014-07-23T13:25:00.000-07:00

One of the projects that I've had in the back of my head for a while was to ditch Apple's Software Update Service (SUS) and use Reposado in its place. We currently have several (20) servers in place. During our ongoing evaluation process we've begun to question the necessity of maintaining these servers in their current capacity. We are using them for 3 purposes: Workgroup Manager (MCX), a Casper (JSS) Distribution Point, and Software Update Services.

MCX is being replaced by profiles. As we move more of our computers to 10.8/10.9, it is no longer necessary to continue this service.
We could use (virtually) any server to host our Distribution Points.
Software Update Services only (officially) hosts updates for the same level of the host's operating system or lower. Since our servers are currently at 10.7.5 this obviously creates a problem. We were able to use a kludge to get it to host 10.8.X updates, but this didn't work when 10.9 was released.

So, as you can see, we can possibly do without maintaining Apple branded servers in our environment. As an experiment, on a lonely Sunday afternoon, I decided to install Jamf's Netboot/SUS appliance in our virtual infrastructure. A couple hours and a few hairs later I was servering updates back through 10.4.X. I quietly put this into production and began servicing our 10.9 clients. This would normally be fine, except that we have relatively slow links to our remote sites

Thus this past week I launched my master plan to replace Apple's SUS with Reposado. Below is the process that I used. (Warning: This is the process that I used. I'm sure that there are many other ways to do what I've done.)

Resources used:

Reposado GitHub site (https://github.com/wdas/reposado)
Margarita GitHub site (https://github.com/jessepeterson/margarita)
Bye Bye Apple SUS (http://www.nbalonso.com/bye-bye-apple-sus/)
Reposado Apple Software Update Server (http://signalboxes.net/mac2linux/reposado-apple-software-update-server/)
Setting up Reposado and Margarita on Linux – Part 1 (http://macadmincorner.com/setting-up-reposado-and-margarita-on-linux-part-1/)

Preparation:

Clone (download) Reposado and Margarita repositories.
Download X-Code command line tools

Process:
Create Reposado working directory:
mkdir /usr/local/reposado

Copy items in reposado-master/code directory to this folder.

Create Reposado repository folders:
mkdir -p /Volumes/Data/reposado/html
mkdir /Volumes/Data/reposado/metadata

Change to the Reposado working directory:
cd /usr/local/reposado

Configure Reposado and answer the questions that come up. (Answers in bold) In this example I’ll be using port 80 to serve updates:
./repoutil --configure

Path to store replicated catalogs and updates [None]: /Volumes/Data/reposado/html
Path to store Reposado metadata [None]: /Volumes/Data/reposado/metadata/
Base URL for your local Software Update Service (Example: http://su.your.org -- leave empty if you are not replicating updates) [None]: http://myupdateserver.com

Clone the Apple Software Update library:
./repo_sync

The initial sync will take a couple of hours. While this is running, you should see it creating files and indexes and downloading the updates into it’s file structure. Since this is going to take a while, we can begin to prep our web server.

Open the Apache2 configuration file (/etc/apache2/httpd.conf) to make sure it’s setup properly:

•Make sure it’s listening on port 80 (around line 50?):
#Listen 12.34.56.78:80
Listen 80

•The rewrite module is enabled: (Uncomment if necessary)
LoadModule rewrite_module libexec/apache2/mod_rewrite.so

•Change the default directory. Comment out the default line and add your new directory so that it looks like below:

Change: Directory "/Library/WebServer/Documents"
to: Directory "/Volumes/Data/reposado"

•Some people have found it necessary to make sure that Overrides are enabled (I haven’t found it necessary, but ymmv)
Change: AllowOverride None
to: AllowOverride All

Close the file and edit the default port 80 configuration (/etc/apache2/sites/0000_any_80_.conf) :

Change the default directory here as well:
DocumentRoot "/Volumes/Data/reposado/html”

(This step is probably not necessary, but this is the combination was required in order for me to get this working.)

Create a .htaccess file in your Reposado repository with the following items (check the Reposado wiki for the latest). This will allow you to point all of your clients to [repositoryURL]/index.sucatalog and not worry about making changes for every operating system:

1:  RewriteEngine On  
2:  Options FollowSymLinks  
3:  RewriteBase /  
4:  RewriteCond %{HTTP_USER_AGENT} Darwin/8  
5:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/index$1.sucatalog [L]  
6:  RewriteCond %{HTTP_USER_AGENT} Darwin/9  
7:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-leopard.merged-1$1.sucatalog [L]  
8:  RewriteCond %{HTTP_USER_AGENT} Darwin/10  
9:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-leopard-snowleopard.merged-1$1.sucatalog [L]  
10:  RewriteCond %{HTTP_USER_AGENT} Darwin/11  
11:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-lion-snowleopard-leopard.merged-1$1.sucatalog [L]  
12:  RewriteCond %{HTTP_USER_AGENT} Darwin/12  
13:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L]  
14:  RewriteCond %{HTTP_USER_AGENT} Darwin/13  
15:  RewriteRule ^index(.*)\.sucatalog$ content/catalogs/others/index-10.9-mountainlion-lion-snowleopard-leopard.merged-1$1.sucatalog [L]

Let’s start the web server and check to make everything is OK:
sudo apachectl restart

Check the log files for errors:
tail /var/log/apache2/error_log
tail /var/log/apache2/access_log

After the repo_sync has finished. point one or more of your clients to the new repo and test, test, test:
sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate CatalogURL “http://myupdateserver.com/index.sucatalog”

Verify that your entry was done properly:
defaults read /Library/Preferences/com.apple.SoftwareUpdate CatalogURL
http://myupdateserver.com/index.sucatalog

Test your client:
sudo softwareupdate -l

You should see the computer checking similar to below:

Software Update Tool
Copyright 2002-2010 Apple

Software Update found the following new or updated software:
* SecUpdSrvr2014-003Lion-1.0
Security Update 2014-003 (1.0), 180667K [recommended] [restart]
* SecUpdSrvr2014-002Lion-1.0
Security Update 2014-002 (1.0), 173813K [recommended] [restart]

If you see any errors, check your CatalogURL and try again.

At the server, check the log files to ensure that no errors have occurred. I like to go back 50 or 100 lines or so to make sure I’m seeing the beginning of the process:

tail -n50 /var/log/apache2/error_log
tail -n50 /var/log/apache2/access_log

Congratulations!! You now have a functioning update server!

Now let’s schedule it to sync nightly:

Create a plist file (reposync.plist) in the Reposado working directory with the following contents:

1:  <?xml version="1.0" encoding="UTF-8"?>  
2:  <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">  
3:  <plist version="1.0">  
4:  <dict>  
5:    <key>Label</key>  
6:    <string>com.yourcompany.reposync</string>  
7:    <key>OnDemand</key>  
8:    <true/>  
9:    <key>RunAtLoad</key>  
10:    <false/>  
11:    <key>Program</key>  
12:    <string>/PATH/TO/SCRIPT</string>  
13:    <key>ProgramArguments</key>  
14:    <array>  
15:      <string>/usr/local/reposado/repo_sync</string>  
16:    </array>  
17:    <key>StartCalendarInterval</key>  
18:    <dict>  
19:      <key>Hour</key>  
20:      <integer>02</integer>  
21:      <key>Minute</key>  
22:      <integer>00</integer>  
23:    </dict>  
24:  </dict>  
25:  </plist>

This plist will run repo_sync every night at 2:00 am. Copy it to /Library/LaunchDaemons/com.mycompany.reposync.plist:
cp reposync.plist /Library/LaunchDaemons/com.mycompany.reposync.plist

and launch it:
sudo launchctl load -w /Library/LaunchDaemons/com.mycompany.reposync.plist

Next steps:

Installing Margarita (https://github.com/jessepeterson/margarita)
Duplicating this process across all remaining site servers. (Might turn out to be an excuse to start using Puppet (http://puppetlabs.com/) to manage my servers)
Implement Sean Kaiser’s Multi-Site solution (http://seankaiser.com/blog/2013/05/23/multi-site-reposado/)
Changing the server configurations to pull from my master repo. (Just an idea, only if bandwidth utilization gets out of hand.)

Cleaning up the network monitoring

2014-07-16T08:06:00.000-07:00

Last week I had the pleasure/honor of being able to attend the Penn State Mac Admins Conference. But, before i get to implementing what I learned there are a free things that I need to clean up. One of the issues that I was having prior to leaving was figuring out why our servers weren't recovering properly from power outages. (Before anybody starts screaming "you should have a UPS!", we have them but you can only maintain power for so long). We tend to have seasonal power outages and brown outs at this time of year as well as people being helpful and unplugging the server racks. While things usually recover nicely, we've been having some timing issues. The servers are coming back up before the network. And since they don't have the network available, DNS and DHCP epithet don't or sort of start up. I need to come up d with a way of monitoring this with my current console. I've been using Opsview core for a couple of years to do general system monitoring. It's based on Nagios and had a semi useful web interface, so I can pretty much bend it to my will.

Post PSUMac ToDo List

2014-07-16T06:07:00.001-07:00

Post Conference ToDo List

Last week I had the honor/pleasure of attending the MacAdministrator's Conference at Penn State University. It was a fun filled and brain filling week of techy goodness. Below is the short version of the todo list that I came up with:

Cleaning up and finishing some of my deployment scripts.
Reposado (https://github.com/wdas/reposado)

Multi-Site alterations (http://seankaiser.com/blog/2013/05/23/multi-site-reposado/)

Autopkg (https://github.com/autopkg/autopkg)

JSS addon (https://github.com/arubdesu/jss-autopkg-addon)
Autopkg change notifications (http://seankaiser.com/blog/2013/12/16/autopkg-change-notifications/)

Git (https://about.gitlab.com/gitlab-ce/)

Workstation client (Sourcetree?)
Server solution (Git lab??)

Changes to computer preference management.

Change "defaults write" and plist packages to profiles (http://talkingmoose.net/2014/07/12/posting-my-psu-macadmins-conference-presentation/)

Deep Thoughts for the New Year

2013-01-05T11:04:00.000-08:00

I started this blog a few years ago as a place to put some of my thought and ideas and to document some aspects of my life. Well, this has been a major fail. I hope to fix this this year. Now, on to today's topic.

A lot of people use the new year as a time for reflection and planning goals for the upcoming year. I guess I'm no exception to this. I usually don't make resolutions, but look at things in my life that I could change - part of my never ending quest to be a "better" person.

Since I've been somewhat of a loner most of my life, I don't have a lot of friends. A close friend once told me that I have more friends than I'm aware of and I need to be more receptive to their friendships. Words I've tried to take to heart. I've met some interesting people in the past year, a few I plan to keep around for a while. The others, well, they serve their purpose. Fighting the urge to retire into my shell is something that I constantly fight. While I can fake being comfortable in social situations, its not what I like to do. Mind you, public speaking and performing is not a problem for me. The time that I spent in church during my teen years is responsible for that.

One of my flaws is that I tend to throw myself into my work. For me, work is an extension of some of my interests. Creating something functional from a pile of parts; Finding a unique solution to a problem that I didn't even know that I had; Well, you get the idea.

Over the past few years I've forced myself to get involved with a couple organizations that I didn't know existed prior. Like minded individuals that I can bounce a few of my ideas off of. Many of them are very well known nationally worldwide, at least in some circles. Some of them are published authors and/or renowned speakers as well. I've learned a ton from them. Although I have no idea how they can do so much with the same amount of hours in a day. A lot of it is time management. A lot of it is sheer knowledge and recall ability.

My engineering background, as well as being raised by a basement mad scientist complete with lab coat, has given me a need to learn as much about as many different things as possible. Somewhere along the line I developed the ability to get myself up to speed quickly on new ideas and procedures. There is a lot of depth to go with the breadth mind you. I've learned through the years that others may find my knowledge intimidating, or they don't believe that I know all that I claim to. Through talking to people, I've learned that that is a side affect of successfully working in the IT department of an educational organization.

I remember a quote that I once read that said something along the lines of the truly great minds are not afraid of mistakes as their part of the learning experience. I also know that there are others that have more knowledge, experience and resources than I have. One of the great things about the internet is that people are more than happy to share their experiences and solutions to the problems they've faced. To this end, I've discovered a ton of blogs that I can use for my research and ideas. Why reinvent the wheel when I can take the opportunity to learn from them.

Those that I work with are completely clueless about a lot of the ideas, processes, and products that I bring forward. But, as evidenced by these blogs, they are indeed in use and practiced by many in the field. I try to not be critical of them, but they can be quite frustrating at times.

A few years ago, a former boss tried to impress upon me that I needed to stop creating my own solutions. He was also convinced that any software that is put into production should come out of a box and be backed by a "real" company. He equated open source software to the shareware of the 80's - unpolished, unreliable and unbacked. While this may be true of some products, it is not true for all. The internet was built, and continues to run on, mostly open source software. Many commercial products are based on open source. In fact, many of the more successful open source products have been spun into real companies, while others have been acquired. I know someone who took a leave from work to start such a company with the ultimate goal of acquisition in mind, although 9/11 brought that to a crashing end for him. (No pun or offense intended.) What I've learned through my new contacts is that these products are in use in many of the companies that we're familiar with.

If you're unfamiliar with it, the concept of open source is very successful. It goes something like this: I have an idea, or I have a need or problem that is not solved by off the shelf products. Or, those products don't work the way I need them to and I can't change that. So, I'll come up with my own solution and share it with the world. The only stipulation is that if you come up with new ways to expand it, or solve problems that are in my code, then you must give those changes back to the "community." While some may scoff at the concept of giving away their work, again, it's highly successful.

As my work responsibilities have grown and changed, I've had to take his thoughts to heart. I have had to begin using products that I can turn over to others (with little technical experience) to operate. But, the same crowd-sourced efforts that are used to further open source products can be applied to many of these products. I've tried to walk the balance between commercial products with "real" support yet give me the ability to craft my own customizations. This usually gives me the excuse to keep my programming skills sharp as well as learning more about the systems that I support. However, I simply don't have the time to do it all myself.

I struggle to comprehend how those that I work with do not want to learn about the internals. They're stuck (barely) in maintenance mode. No thought is given to how or why something happens. Or, why does the solution work. Sometimes I take it personally. They're a continual source of aggravation for me. Both their limited skill sets and their unwillingness to grow and expand their own knowledge are baffling. Yet, they seem to be happy this way. How can they be happy not knowing all they could about their chosen profession? While I have technically passed day to day supervision of the underlings to someone else, they don't have the skills to assist them. Or the supervisory skills to keep them on task. While they do need a task master, I still find myself getting too involved with them on a daily basis. This definitely interferes with my ability to get my own tasks done. Yes, another source of frustration.

Regardless of how I try to mask my frustration when I talk to others, it shows in some manner. Some people may think that I'm a mean, cold, uncaring person. My true friends know that this is not true. Still others tell me that I need to calm down because "it's not that important." What? If you don't think that getting your problem (that you've complained to those above me about) resolved is important enough for me to get upset over - you're part of the problem as well. More stress! I'm not proud of this. If anything I sometimes find it embarrassing. I try to be a gentlemen at all times. Well, most times. I have my moments like anyone. I believe that people have many sides. That's what shapes our personalities. Some people verbalize all of their thoughts. I internalize mine. Others don't seem to have any thoughts at all.

It's something that I struggle with daily. How can I find balance. There is part of most people that wants to be around others. Our species did not succeed by being solitary. I can sometimes tolerate being around others for short periods. Unless it becomes an excuse to badger or harass me. Then I can become quite grizzly and off putting. Again, it's not something that I'm proud of and I know that it's a coping mechanism. Sometimes these people need to be put in their place and I may be more than willing to be the one to do it. Usually, afterwards they're more cooperative.

In closing, please be patient with me. I'm a work in progress. Learning to deal with these stressors is something that I need to get a handle on.

Here's to a new year! And, to my true friends: Thanks for being you.

(Mis)Adventures in Networking

2011-10-25T20:51:00.000-07:00

EDIT: Whoops! I never published this blog post.

Wow, it's been a while since my last post. Anyway, if you were looking for an example of Murphy's law, I've got one for you.

We had 2 simple goals: upgrade our internet service and bring up our new WAN connections. This adventure starts on Tuesday, April 19, 2011.

Background:

In the city where I work, we are really locked into our choice of providers. Because of the city's infrastructure, we have 2 options: the incumbent telecom provider and the local cable company who has no interest in providing service to us.

This install is an upgrade to our existing service and also a relocation to another building. It was originally supposed to take place last November. As we were ramping up for the install, we discovered that my former boss didn't place the order. (This after I received a couple of phone calls asking why we were requesting service at 2 sites during the summer.) After a couple of angry phone calls to the telco rep, the order was placed. We were given an install date in early February. A week before the date arrived I sent my weekly pester the project manager email and was told that they were having "unforseen issues and would be unable to meet the date." (According to the rep, they ran out of capacity) The date they gave me was in June. Called the rep, explained to him how this was unacceptable and that I was going to recommend that we take action to terminate our existing contracts. He was then able to get us a March date. As we got close to the March date, we were told that date was only for our end of the connection and it would take a few weeks more to get the other end installed. Finally in late March, we were told that everything was in place and we looked at the calendar to find a suitable date that wouldn't cause too much disruption. We decided on spring break week since the majority of our employees were off anyway.

Tuesday Afternoon:
I met with an engineers from my ISP and my integrator to go over the connections and site prep prior to the installation. He checked the configuration on the router as it was shipped and made some changes/corrections. He made a call to try to bring up the connection, even though it wouldn't be routing any traffic at that point - just to make sure everything was good on the telco end. After spending some time on hold and talking to several people, he was told that our appointment wasn't until 9:00 the next morning and there was nothing that could be done until then. I found this rather entertaining since he works for the same company. While he was doing this the integrator and I busied ourselves with racking switches, connecting cables, checking connections, etc. A couple of hours later we went over our game plan for installation and left for the day.

Wednesday Morning:
We met at the appointed time and set out to make magic!! Our enthusiasm was soon dashed when we couldn't get a link to the demark. We did the usual switch the polarity of the cables, trying new/different cables, trying different SFP modules, nothing was working. At some point during we noticed that we didn't have a link light on the WAN router either - but the connection was up and passing traffic. The ISP engineer continued to work with the telco side to try to troubleshoot the issue.

During a trip to the demark to bang my head against the wall I looked at the fiber box that went to the MDF - it read "SM Fiber to MDF". I thought SM - as in Single Mode Fiber? This should be MultiMode Fiber. I disbelief I opened the box and looked inside. Sure enough, it was Single Mode. WTF! (Bang Head on wall). Funny thing is the vendor that installed the fiber was hired by the telco on the premise that they wouldn't install the line unless dedicated fiber was in place. Now, the demark happens to also be in one of the IDF's in the building. Couldn't we just use that? We were told no. After we put our heads together for a few minutes, I said "F it, this is the only opening we have for over a month that we can do this during the day." I then connected to the fiber that was in the IDF, ran upstairs, moved the cross-connects and bingo, the WAN came up with a link light.

While this was going on, the telco engineer had moved the router to the demark to try to get a connection there. After a while, he was able to get the line up and brought the router back upstairs. We re-racked the router and fired it up. He was unable to pass any traffic across the connection. He gets on the phone again and calls me over after about 15 minutes and puts his cell on speaker. The person on the other line says "Oh, you want to switch your internet service to that line? I asked him why the heck he thought we were going through all of this? His answer (I wish I was making this up): "Oh, I thought you were just testing the line." As Bill Engvall says "I didn't start out the day wanting to be a jackass, but you just pushed my jackass button." Needless to say, we had internet in a couple of minutes. Elapsed time: 3.5 hours.

I wish this was the end of the story, but it's not.

While we did have internet, traffic was sporadic. Our initial troubleshooting lead us to believe it was a DNS issue. And, indeed, we were having trouble with DNS resolution. But, when we tried to use a public DNS server, it only marginally better. It worked fine from the DMZ and from outside. Made a few changes to the routers and firewall and we were able to get access at-first. After about 10 minutes, it stopped working again. Undid the router and firewall changes and it worked again for about 10 minutes.

Hours of troubleshooting later, we were able find the issue, but not a clear solution. When we starting routing traffic through the new WAN, it introduced routing loops at almost every point in the network. Considering we have 22 locations, the problem was quite massive.

Making corrections after deployment (or whoops!)

2011-10-25T20:47:00.000-07:00

Like clockwork, almost as soon as we deployed the new workstations, we began receiving complaints about things that weren't right. Most of the initial complaint were "I need the administrator password so that I can install software and make changes." This was met with a resounding "NO!" It's been an uphill battle and we admittedly have to do some good work to repair our department's reputation. It was most entertaining to get a phone call with a complaint and to be able to tell them "Look at the computer, I'm logged into it right now. And, that program that you said doesn't work does."

The last week or so has been pretty quiet. After they began to see printers and software magically appear on their computers, they stopped complaining.

There was one item that slipped by us: Some of the laptops would be going home with teachers and administrators and they would need to be able to connect to their home wireless networks. We had this locked down to prevent students from changing to an open network that they can see from the school.

After hours of digging and false starts, I found the preference file that needed to be changed:

/Library/Preferences/SystemConfiguration/preferences.plist

My first reaction was it's a plist, it should be easy to script the change. Wrong! It doesn't act like any of the other plist files. Using a "defaults write" command would put the setting into the wrong place in the file. I tried to open it in several plist editors, and it only found some of the settings, none of which I needed to change.

After I got home I spent a couple of hours trying to use sed and awk to change the file, but I couldn't get the regex correct. I was about to give up, when I decided to give Google another try and found my answer:

/usr/libexec/airportd

I ran it with the -h flag to see the options available and I found what I needed:

RequireAdmin (Boolean)
RequireAdminIBSS (Boolean)
RequireAdminNetworkChange (Boolean)
RequireAdminPowerToggle (Boolean)

I could use "/usr/libexec/airportd en1 prefs RequireAdmin=NO" to change all of the settings at once or I could use the others to get more granular if I chose. I wrapped this in a quick BASH script and tested - success!! Here's the code that I created.

#!/bin/sh
#
# Script to allow Users to change the wireless network without requiring a password
#
#

# Turn off all settings
/usr/libexec/airportd en1 prefs RequireAdmin=NO

# To allow granular settings you could do the following:
#/usr/libexec/airportd en1 prefs RequireAdminIBSS=
#/usr/libexec/airportd en1 prefs RequireAdminNetworkChange=
#/usr/libexec/airportd en1 prefs RequireAdminPowerToggle=

exit 0

Tomorrow I get to see if it works with Lion.

Adventures in Managed Computing

2011-10-25T20:29:00.000-07:00

Over the last month we've been working on deploying over 700 new workstations and laptops. This time around, I was able to get support from the upper administration to make them fully managed. I was also able to convince them that we needed to pay for installation. After a few months of negotiating with Apple and my local var we were able to settling on an SOW for the project.

The VAR would:

Install the management and imaging software on a brand new server.
Install and configure the pile of servers that I had in my office.
Image, deliver and deploy the new computers.
Ensure that the computers were integrated into the management system and our online tracking system.

We chose to use Casper for JamfSoftware to manage the computers. Casper has a pretty good reputation in the community and we decided to give it a whirl. After this is settled, we'll begin a SCCM deployment for the PC side of the house.

I'll post about my experiences with Casper in a later post.

That is all for now. Later.

E-Mail Server Update

2010-05-27T06:23:00.000-07:00

On the afternoon my last post, the server went down again. This time it happened as I was "tail-ing" the log file and saw what was happening:



May 17 13:16:27 server lmtpunix[19636]: DBERROR db4: Logging region out of memory; you may need to increase its size

May 17 13:16:27 server lmtpunix[19636]: DBERROR: opening /var/imap/deliver.db: Cannot allocate memory

May 17 13:16:27 server lmtpunix[19636]: DBERROR: opening /var/imap/deliver.db: cyrusdb error

May 17 13:16:27 server lmtpunix[19636]: FATAL: lmtpd: unable to init duplicate delivery database

May 17 13:16:27 server lmtpunix[19636]: DBERROR db4: read: 0x401410, 28: No space left on device

May 17 13:16:27 server lmtpunix[19636]: DBERROR db4: Ignoring log file: /var/imap/db/log.0000000001: No space left on device

May 17 13:16:27 server lmtpunix[19636]: DBERROR db4: DB_ENV->log_put: 1: No space left on device

May 17 13:16:27 server lmtpunix[19636]: DBERROR: error exiting application: No space left on device

May 17 13:16:27 server master[14002]: service lmtpunix pid 19636 in READY state: terminated abnormally

May 17 13:16:27 server lmtpunix[19637]: DBERROR db4: Logging region out of memory; you may need to increase its size

May 17 13:16:27 server lmtpunix[19637]: DBERROR: opening /var/imap/deliver.db: Cannot allocate memory

May 17 13:16:27 server lmtpunix[19637]: DBERROR: opening /var/imap/deliver.db: cyrusdb error

May 17 13:16:27 server lmtpunix[19637]: FATAL: lmtpd: unable to init duplicate delivery database

May 17 13:16:28 server lmtpunix[19637]: DBERROR db4: read: 0x401410, 28: No space left on device

Notice the 5th line: No space left on device. This can't be right, I have about 16 GB of free space (according to df -h). But, lets think about this for a moment. I've been chasing a database corruption problem. If the OS thinks that it is out of space, of course it won't be able to update the db.

Looking back at my notes from a couple of months ago, before this started, I had about 30 GB free. Where did all of the free space go? I talked to the PHB and he finally agreed to let me take the server down over the weekend to work on it. In preparation for this, I also looked at the folders in the mail store and compared them to the FileMaker Pro database that I use to track all of the account info. (For you privacy nuts, I don't keep passwords, if you forget it, I can change it but I cannot see what it is.) I also looked at tuning options for Postfix, Cyrus and Mailman.

Fast forward to Friday night. I purged the mail queue, the stopped the mail server and manually ran the garbage cleanup script. Our webmail users can't understand that unless they click "Purge" the mail doesn't actually get deleted. So, I wrote a script that deletes the files from their Trash folder and rebuilds the index. This happens quietly behind the scenes and they never know it happened unless they try to retrieve something from the Trash. I actually once had someone tell me that's where they store their important emails. I asked them: Do you keep your checkbook in the garbage can at home? And, they went away muttering to themselves.

I rebooted the server into single user mode and fsck'd the drive. It took forever to get through the various steps, to the point that I stopped watching and forced myself to wait 10 minutes between checks. When it got to "Checking Volume Information" it said:

Invalid volume free block count

(It should be 8456596 instead of 4562047)

Repairing Volume

After it was done (roughly 3 hours from start time) I then had 32 G free, not 16 as reported before I started. Yea!!!

From here, I set about rebuilding the Cyrus database and the user databases. I restarted the mail server (sudo serveradmin start) and watched the mail begin to flow. When I felt comfortable, I restarted Mailman and watched it fly through the postings that were waiting to be sent.

Its now a couple of days later and everything seems to be stable.... Except, now the boss want me to get to 100 GB free!!! (le sigh...) Stay tuned for more updates

Its Racing Season

2010-05-26T20:48:00.000-07:00

Believe it or not, the Netman isn't all work, he knows how to have fun also. One of the things that my wife and I like to do is go to races. She's mostly a Nascar fan, but goes to other types of races to entertain me. We have our most extensive season planned and started it off on Mother's Day.

We went to see the Trans-Am series at New Jersey Motorsports Park. I remember growing up watching Mustangs, Corvettes, Camaros and other cars on TV. This was the first time that I've had a chance to see this series and was excited.

We stumbled onto the Club Diner in Bellmawr on our first trip to the track and its become part of our ritual. Usually we have to figure out how to get there. I must have set it as a waypoint in the GPS the last time we went because it took us right to the door. Strange, but I digress.

The entry list had about 20 entries, but when I checked the practice times, most of them didn't have a time listed. When we got there we learned that only 8 cars showed up, but for some reason, they list 12 cars getting points.

Here are a few of my pictures from the race:

Here's Tomy Drissi and Tony Ave getting ready to go out for practice and qualifying. My wife can't wait for the Marmaduke movie to come out and decided that she was going to root for Tomy.

Drissi won the race (surprise, surprise), RJ Lopez came in second and Ave was third. Here they are on the podium spraying champagne and doing the "hat dance."

My personal highlight of the day was getting to see Drissi's owner Paul Gentilozzi. (Below talking on the phone.) I had a chance to talk to him a couple of times and he's much cooler than the TV reporters make him out to be.

Here's a link to my Picasa album. Its not my best work, but here it is. Next up - Arca and Sprint Cup at Pocono.

(More) Email Server Issues

2010-05-17T09:17:00.000-07:00

This past weekend, we had another email outage. This time it looks like it took the server took about 12 hours to rebuild itself. Time for a backup/restore or defrag. But, the PHB says he's going to purchase an Exchange 2010 setup at the start of the next fiscal and doesn't want to have the downtime. Meanwhile, my faithful Frankenserver takes another hit. This server has been in service since 2005 and has run fine for years until recently.

When I left on Friday afternoon, everything looked fine, but I had a feeling that it wasn't. At 6:00 Saturday morning, I got up and checked the server. Sure enough, the db was corrupted again. So I did the usual rebuild:

#serveradmin stop mail
# mv /var/imap /var/imap.old
# mkdir /var/imap
/usr/bin/cyrus/tools/mkimap
reading configure file...
i will configure directory /var/imap.
i saw partition /var/spool/imap.
done
configuring /var/imap...
creating /var/spool/imap...
done
# chown -R cyrusimap:mail /var/imap
# sudo -u cyrusimap  /usr/bin/cyrus/bin/reconstruct
#serveradmin start mail

the last step is usually done with the -i option, but I usually skip it and perform the user folder rebuilds manually with:

# cd /var/spool/imap
#su cyrusimap
$ /usr/bin/cyrus/bin/reconstruct -r -f user/username
user/username
user/username/Deleted Messages
user/username/Drafts
user/username/Sent Messages

I have a text file on my server that has all of the usernames. I wrote a quick script to parse the file and rebuild each user.

After looking at various log files, it looks like Mailman is causing the issues. We have a list that goes out to all of our users (about 2300). (Send to one address and this spiders out to the lists for each location. The Mailman docs refer to this as an umbrella list.)

Our HR department decided to "go green" by sending out death notices and other items via this list. Even though, they've been told to do a delay-send until the evening hours, they keep "forgetting" and sending them in the middle of the day. This usually ends up tying up the server for a couple of hours during which time messages continue to come into the server, but nothing goes or out until it finishes processing the list.

At this point I decided to hold off on the rebuild and wait until I talked to the boss. I got him to agree to hold off on the rebuild until we can clean up the server and verify that it is actually stable (this time).

I spent an hour or so this morning researching various mailman options such as can we delay the sending until X o'clock? Turns out others have asked the same question and the answer ranges from not easily to no. I ran through the various configuration options and found that my SMTP_MAX_RCPTS was set to 500. This causes the server to attempt to send 500 messages in each batch. I changed this to the recommended 10. We shall see if this works.

I was also able to get him to agree to let me purge some of the older emails on the server and set up some quotas.

Stay tuned.....

PICC Day 2

2010-05-11T12:40:00.000-07:00

Day 2 of the PICC conference was little more scattered and varied than the first one. The first session was "The Evolution of Storage Networking and the Current Trends in the Industry." Jacob Farmer, CTO of Cambridge Computer led this session. Jacob's talk was a exactly what the title says. He talked about storage virtualization and how abstraction layers interract with the Operating system and the File System. Most of the concepts were as I imagined they would be, although its nice to know the "real" names for things. I learned that the Equallogic SANs that I have at work are using "spindle virtualization." In short, this means that you don't have to worry about carving up your SAN into various LUNs, each (possibly) having different RAID configurations. When you setup the Equalllogic, you format it (RAID 5, 6, 10, or whatever you want) and then only worry about the size of the partitions that you create. This allows you to easily add additional storage by adding another unit to the "cluster." Personally, I think this is a cooler method of dealing with SANs.

I only stayed for the first half of this talk because I wanted to go to a couple other sessions. Next I went to "Pushing Boulders Uphill: NOAA Updates - High Performance Computing across the WAN." [In my best "Staples" commercial voice] "Wow, that's a long title" The speakers gave an overview of how NOAA deals with their datasets. Its amazing how much data they move around their network on a daily basis. More importantly, some of the discussion was about how the grew their network and some of the tools they use. It was amazing to see how "simple" it was, albeit from a very high level overview. Even better, they use some of the same (open source) tools that I use to monitor their network. I must be doing something right.

Directly after this, Matt Simmons led a discussion on "Keeping Nagios Sane." More than any other, this session demonstrated what community is about. Matt started the session by stating that he wasn't an expert, but these were the things that worked for him. Most of the participants in the session use Nagios and contributed some of their ideas during the discussion. Could any of us be considered "experts"? Maybe. Depending on your definition.

Sometimes, I get a little annoyed when my coworkers or others call me an "expert." What does that mean? I know what I know. But, that's not everything there is to know. Usually I know more about my field than they do. Do I know everything? Of course not. I think what separates us from the paper admins is our ability to get up to speed quickly in a variety of areas. Often, this is driven by our quest for knowledge. Sometimes, we actually need to do something with what we learn. What separates this profession from most others is that there are always several ways to accomplish a task. Some of them are more elegant than others and some are just crazy ideas that lead to some cool discovery. I should probably finish this post before I go off on a rant. BTW, Matt, check out cacti for trending.

Next I went to a session of Budgeting for System Administrators given by Adam Moskowitz. Alan gave some very cogent advice to us regarding the budget process. Sometimes we tend to forget that those above us may not understand the buzzwords. We need to be show them what our needs are in a language that they can understand and relate to. Also, they may not fully understand the inter-relatedness (is that a word?) of the various parts of our projects. Finally, sometimes we have to be willing to compromise and either scale-back or phase in our projects.

After this, my brain needed a break and I attended an "Unconference" This was probably the closest to what most people would imagine would happen if a group of techies got together in a room to just discuss a topic, except most of us were totally worn out at that point. We ended up talking about DNS, specifically, how we each manage it. We were shown Carnegie-Mellon's NetReg by one of the the people who wrote it. How cool is that. I had looked at it a couple of times before, but didn't realize its power. It's not something that I would use in my current environment, but I know its out there if the need arises.

The last session I attended was "An overview of Google's Technologies...." given by Tom Limoncelli. It was kind of fitting that I started and ended with the same speaker. Tom gave a quick overview of Google and how they grew their infrastructure. More cool stuff to keep in the back of my head.

Overall, it was a good conference and I had a blast. I met some new and interesting people. (And not the weirdos that are usually drawn to me) Kudos to William Bilancio, Tom Limoncelli, and the entire planning and organizing committees. They've already begun to consider the next conference and I'm sure this one will be even better.

LOPSA-NJ’s PICC

2010-05-07T08:34:00.000-07:00

I'm at LOPSA-NJ’s PICC conference in sunny New Brunswick, NJ. I've taken Matt Simmons' challenge to blog about the conference (and restart this long neglected blog).The first session I'm attending is Tom Limoncelli's Time Management for System Administrator's - A New Approach. This talk was a whirlwind tour of his classic book Time Management for System Administrators but slightly rearrange, redone and updated.

This was the first time I was able to hear Tom speak formally. I was especially looking forward to this talk since it is based upon the book that radically changed my outlook on doing my job. It was everything I was expecting and more. Sometimes, a refresher course in a subject that you're familiar with can change your way of thinking. A couple of tips that I picked up were:

Keep a single calendar with your work and personal events. And, be sure to write everything down.
A reminder that I have to find a todo list manager or GTD app for my new Droid Incredible. Since the Android platform is a competitor to the iPhone, I don't see Apple supporting syncing to the phone in the near future. (More on the phone in a later post.)
Go over your notes at the end of the day and move incomplete tasks to the next day (or even further out).
Remind my co-workers about the 30 minute rule.
Tom mentioned that he keeps a lab notebook. I tend to keep everything on one pad and things can sometimes get lost.
An interesting idea was to put contact information for the helpdesk onto the desktop wallpaper. I'll have to play around with this.

The second session was also hosted by Tom Limoncelli. This one was titled: Help, Everyone hates our IT Department. This was sort of an extension of the first session. The key bullet points for me were:

How do users get help? And, do they know who to call and when.
Has the scope of support been clearly defined?
What is the definition of an Emergency?

The major takeaway that I had from this session was: If you can manage expectations and communications, it will help to manage the perception of the IT department. (I will probably revise this statement later, after I've had a chance to think about it.)

The evening ended with a talk by David Blank-Edelman - Through the Lens Geeky: How SysAdmins are Portrayed in Pop Culture. His presentation showed several film clips portraying SysAdmins from the 50's through recent films. The good, the bad, the unrealistic (of both sides) were shown.

During the second session I texted one of my friends the title of the session I was in. Her response was : Everyone doesn't hate the IT department, they just misunderstand you. Both Tom's and David's presentations drove this point home. When I get back, I'll be musing over this. How do I change perception? How can we (better) publicize what we do?

Hello World

2008-02-18T15:12:00.000-08:00

I've been thinking about starting a blog for a few months now, and I've finally made the plunge. In this blog I will put down some of my random thoughts and ideas as well as some examples of what a typical day for me is like.