• Show diff of the last commit:

  git diff HEAD^..HEAD

• Roll back to version xyz of a specific file (where xyz is a SHA1 commit ref):

  git checkout xyz path/to/file

• Undo any unstaged changes to your branch:

  git checkout -f

• Undo any staged and working directory changes:

  git reset --hard

• Update submodules after cloning a repository:

  git submodule update --init

• Rebase on current master to pull in new changes:

  git rebase master

• Rebase on current master, but for files that changed, take our version (for some reason, a plain rebase seems to sometimes show conflicts on files that haven’t changed in ages on master):

  git rebase -s recursive -Xtheirs master

• Delete a local branch:

  git branch -d BranchName

• Delete a remote branch from origin:

  git push origin --delete BranchName

At work, we have a handful – currently a really small number – of RPM packages that we need to build and deploy internally for our CentOS server infrastructure. A number of them are just pulled down from specific third-party repositories and rebuilt to have the vendor set as us, and some are internally patched or developed software. We run websites, and on the product side, we’re a Python/Django shop (in fact, probably one of the largest Django apps out there). We don’t deploy our Django apps via RPM, so building and distributing RPMs is definitely not one of our core competencies. In fact, we really only want to do it when we’re testing/deploying a new distro, or when an upstream package is updated.

Last week I pulled a ticket to deploy node.js to one of our build hosts, and we’ve got a few things in the pipeline that also rely on it. I found the puppetlabs-nodejs module on Github that’s supposed to install it on RHEL/CentOS, but it pulls packages from http://patches.fedorapeople.org/oldnode/stable/, and the newest version of nodejs there is 0.6.18, which is quite old. I can’t find any actively maintained sources of newer nodejs packages for RHEL/CentOS (yeah, I know, that’s one down side to the distributions…). However, I did find that nodejs 0.9.5 is being built for Fedora 18/19 in the Fedora build system, is already in the Fedora 18 Testing and Fedora Rawhide repos, but is failing its EL6 builds in their system. The decision I’ve come to is to use the puppetlabs-nodejs module to install it, but try and rebuild the Fedora 18 RPMs under CentOS 5 and 6.

So that’s the background. Now, my current task: to search for an RPM build system for my current job. My core requirements, in no specific order, are:

• Be relatively easy and quick to use for people who have a specfile or SRPM and want to be able to “ensure => present” the finished RPM on a system. i.e., require as little per-package configuration as possible.
• Be able to handle rebuilding “all” of our RPMs when we roll out a new distro version. Doesn’t necessarily need to be automatic, but should be relatively simple.
• Ideally, not need to be running constantly – i.e. something that will cope well with build hosts being VMs that are shut down when they’re not needed.
• Handle automatically putting successfully built packages into a repository, ideally with some sort of (manual) promotion process from staging to stable.
• Have minimal external (infrastructure) dependencies that we can’t satisfy with existing systems.

So, the first step was to research existing RPM build systems and how others do this. Here’s a list of what I could find online, though most of these are from distributions and software vendors/projects, not end-user companies that are only building for internal use.

• Koji is the build system used by Fedora and RedHat. It’s about as full-featured as any can be, and I’m familiar with it from my time at Rutgers University, as it’s used to maintain their CentOS/RHEL packages. It’s based largely on Mock. However, setting up the build server is no trivial task; there are few installations outside of Fedora/RedHat, and it relies on either Kerberos or an SSL CA infrastructure to authenticate machines and clients. So, it’s designed for too large a scale and too much infrastructure for me.
• PLD Linux has a builder script that seems to automate rpmbuild as well as fetching sources and resolving/building dependencies. I haven’t looked at the script yet, but apparently it’s in PLD’s “rpm-build-tools” package.
• PLD Linux also has a CVS repository for something called pld-builder.new. The README and ARCHITECTURE files make it sound like a relatively simple mainly-Python system that builds SRPMS and binary packages when requested, and most importantly, seems like a simple system that uses little more than shared filesystem access for communication and coordination.
• ALT Linux has Sisyphus, which combines repository management and web interface tools, package building and testing tools, and more.
• The Dries RPM repository uses (or at least used… my reference is quite old) pydar2, “a distributed client/server program which allows you to build multiple spec files on multiple distribution/architecture combinations automatically.” That sounds like it could be what I need, but the last update says that it isn’t finished yet, and that was in 2005.
• Mandriva Linux has pretty extensive information on their build system on their wiki and a build system theory page, but it seems to be largely a hodgepodge of shell scripts and cronjobs, and is likely not a candidate for use by anyone other than its designers.
• Argeo provides the SLC framework which has a “RPM Factory” component, but I can’t seem to find much more than a wiki page, and can’t tell if it’s a build automation system or just handles mocking packages and putting them in a repo on a single host.
• Dag Wieers’ repositories use (or used) a set of python scripts called DAR, “Dynamic Apt Repository builder”. They’re on github but are listed as “old” and haven’t been updated in at least 2 years. The features sound quite interesting, and though it’s based on the Apt repo format, it might provide some good ideas for implementing a similar system.

Update four months later: I’ve yet to find a build system that meets my requirements above. For the moment I’m only managing ~20 packages, so my “build system” is a single shell script that reads in some environment variables and runs through using mock to build them in the correct order (including pushing the finished RPMs back into the local repository that mock reads from) and then pushing the finished packages to our internal repository. Maybe when I have some spare time, I’ll consider a project to either make a slightly better (but simple) RPM build system based on Python, or get our Jenkins install to handle this for me.

Well a few days ago I was having a discussion with some co-workers that dovetailed into this topic, and when I started some research, I found (finally after using httpd for years) that the Apache httpd 2.2 configuration file syntax documentation states that httpd supports environment variable interpolation anywhere in the config files (and httpd 2.4 supports it with Defines as well).

Yup, that’s right. All those different Apache configs I’ve worked with for years that define separate vhosts, document roots, rewrite targets, ServerAliases, etc. for www.example.com and www.qa.example.com and www.dev.example.com really only had to be www.${ENV_URL_PART}example.com, and set ENV_URL_PART in the init script or sysconfig file. (Of course this all assumes that you have your different environments served by different httpd instances, which you do, of course…)

For me, this is a very big deal. It means that finally, instead of maintaining separate sets of configs for different environments which are (theoretically, except for those emergencies) kept identical by hand, or updating templates and then re-generating each environment’s configs, we can finally follow the same commit/merge/promotion-between-environments workflow that we use for other production code and Puppet configuration. It also means that those pesky little rewrites and other minor tweaks will make it all the way back to development environments.

So, here’s a little example of how this would work in reality. Let’s assume that we have 3 main environments, prod, qa and dev (though this should work for N environments) and that domains are prefixed with “qa.” or “dev.” for the respective internal environments. We set environment variables before httpd is started, on a per-host basis, depending on what environment that host is in. On RedHat based systems, we’d add the variables to /etc/sysconfig/httpd for production:

HTTPD_ENV_NAME="prod"
HTTPD_ENV_URL_PART=""

or for QA:

HTTPD_ENV_NAME="qa"
HTTPD_ENV_URL_PART="qa."

Those variables will now be available to httpd within the configurations (and also to any applications or scripts that have access to the web server’s environment variables).

Now let’s look at an example vhost configuration file that uses the environment variables:

<VirtualHost *:80>
ServerName example.com
ServerAlias www.example.com
# Aliases including proper environment name
ServerAlias www.${HTTPD_ENV_NAME}.example.com ${HTTPD_ENV_NAME}.example.com
 
ErrorLog /var/log/httpd/example.com-error_log
CustomLog /var/log/httpd/example.com-access_log combined
 
DocumentRoot /sites/example.com/${HTTPD_ENV_NAME}/
 
# Environment-specific configuration, if we absolutely need it:
Include /etc/httpd/sites/${HTTPD_ENV_NAME}/env.conf
 
<Location "/testrewrite">
RewriteEngine on
RewriteRule /foobar/.* http://www.${HTTPD_ENV_URL_PART}example.com/baz/ [R=302,L]
</Location>
 
</VirtualHost>

Every instance of ${HTTPD_ENV_NAME} will be replaced with the value set in the sysconfig file, and likewise with every instance of ${HTTPD_ENV_URL_PART}. This way, we can have one set of configurations and use our normal source control branch/promotion process to both test and promote changes through the environments along with application code, and ensure that any straight-to-production emergency changes (everyone has customer-ordered rewrites like that, right?) make it back to development and qa.

One caveat is that, if the environment variable is not defined, the ${VAR_NAME} will be left as a literal string in the configuration file. There doesn’t seem to be any way to protect against this in httpd 2.2, other than making sure the variables are set before the server starts (and maybe setting logical default values, like an empty string, in your init script which should be overridden by the sysconfig file).

If you’re running httpd 2.4+, you can turn on mod_info and browse to http://servername/server-info?config to dump the current configuration, which will show the variable substitution.

nodejs.spec, diff from Fedora Rawhide nodejs-0.9.5-9.fc18.src.rpm, buildID=377755 (full specfile)

diff --git a/nodejs.spec b/nodejs.spec
index 050ed86..86c0f4b 100644
--- a/nodejs.spec
+++ b/nodejs.spec
@@ -1,6 +1,6 @@
 Name: nodejs
 Version: 0.9.5
-Release: 9%{?dist}
+Release: 10%{?dist}
 Summary: JavaScript runtime
 License: MIT and ASL 2.0 and ISC and BSD
 Group: Development/Languages
@@ -25,7 +25,7 @@ Source6: nodejs-fixdep
 BuildRequires: v8-devel >= %{v8_ge}
 BuildRequires: http-parser-devel >= 2.0
 BuildRequires: libuv-devel
-BuildRequires: c-ares-devel
+BuildRequires: c-ares-devel >= 1.9.0
 BuildRequires: zlib-devel
 # Node.js requires some features from openssl 1.0.1 for SPDY support
 BuildRequires: openssl-devel >= 1:1.0.1
@@ -165,9 +165,13 @@ cp -p common.gypi %{buildroot}%{_datadir}/node
 
 %files docs
 %{_defaultdocdir}/%{name}-docs-%{version}
-%doc LICENSE
 
 %changelog
+* Thu Jan 31 2013 Jason Antman <Jason.Antman@cmgdigital.com> - 0.9.5-10
+- specify build requirement of c-ares-devel >= 1.9.0
+- specify build requirement of libuv-devel 0.9.4
+- remove duplicate %doc LICENSE that was causing cpio 'Bad magic' error on CentOS6
+
 * Sat Jan 12 2013 T.C. Hollingsworth <tchollingsworth@gmail.com> - 0.9.5-9
 - fix brown paper bag bug in requires generation script

v8.spec, diff from Fedora Rawhide 3.13.7.5-2 (full specfile)

--- v8.spec.orig       2013-01-26 16:03:18.000000000 -0500
+++ v8.spec     2013-01-31 09:04:51.068029459 -0500
@@ -21,9 +21,11 @@
 
 # %%global svnver 20110721svn8716
 
+%{!?python_sitelib: %define python_sitelib %(%{__python} -c "import distutils.sysconfig as d; print d.get_python_lib()")}
+
 Name:          v8
 Version:       %{somajor}.%{sominor}.%{sobuild}.%{sotiny}
-Release:       2%{?dist}
+Release:       5%{?dist}
 Epoch:         1
 Summary:       JavaScript Engine
 Group:         System Environment/Libraries
@@ -32,7 +34,7 @@
 Source0:       http://commondatastorage.googleapis.com/chromium-browser-official/v8-%{version}.tar.bz2
 BuildRoot:     %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
 ExclusiveArch: %{ix86} x86_64 %{arm}
-BuildRequires: scons, readline-devel, libicu-devel
+BuildRequires: scons, readline-devel, libicu-devel, ncurses-devel
 
 %description
 V8 is Google's open source JavaScript engine. V8 is written in C++ and is used 
@@ -51,8 +53,13 @@
 %setup -q -n %{name}-%{version}
 
 # -fno-strict-aliasing is needed with gcc 4.4 to get past some ugly code
-PARSED_OPT_FLAGS=`echo \'$RPM_OPT_FLAGS -fPIC -fno-strict-aliasing -Wno-unused-parameter -Wno-error=strict-overflow -Wno-error=unused-local-typedefs -Wno-unused-but-set-variable\'| sed "s/ /',/g" | sed "s/',/', '/g"`
+%if 0%{?el5}
+PARSED_OPT_FLAGS=`echo \'$RPM_OPT_FLAGS -fPIC -fno-strict-aliasing -Wno-unused-parameter -lncurses\'| sed "s/ /',/g" | sed "s/',/', '/g"`
+sed -i "s|'-O3',|$PARSED_OPT_FLAGS,|g" SConstruct
+%else
+PARSED_OPT_FLAGS=`echo \'$RPM_OPT_FLAGS -fPIC -fno-strict-aliasing -Wno-unused-parameter -Wno-error=strict-overflow -Wno-unused-but-set-variable\'| sed "s/ /',/g" | sed "s/',/', '/g"`
 sed -i "s|'-O3',|$PARSED_OPT_FLAGS,|g" SConstruct
+%endif
 
 # clear spurious executable bits
 find . \( -name \*.cc -o -name \*.h -o -name \*.py \) -a -executable \
@@ -198,6 +205,17 @@
 %{python_sitelib}/j*.py*
 
 %changelog
+* Thu Jan 31 2013 Jason Antman <Jason.Antman@cmgdigital.com> - 1:3.13.7.5-5
+- remove -Werror=unused-local-typedefs on cent6
+
+* Wed Jan 30 2013 Jason Antman <Jason.Antman@cmgdigital.com> - 1:3.13.7.5-4
+- define python_sitelib if it isn't already (CentOS 5)
+
+* Wed Jan 30 2013 Jason Antman <Jason.Antman@cmgdigital.com> - 1:3.13.7.5-3
+- pull 3.13.7.5-2 SRPM from Fedora 19 Koji most recent build
+- add ncurses-devel BuildRequires
+- modify PARSED_OPT_FLAGS to work with g++ 4.1.2 on CentOS 5
+ 
 * Sat Jan 26 2013 T.C. Hollingsworth <tchollingsworth@gmail.com> - 1:3.13.7.5-2
 - rebuild for icu-50
 - ignore new GCC 4.8 warning

Well, I have Fedora 18 and OS X 10.8 dual-booting on the laptop, but not as planned. After a day and a half of research, troubleshooting and re-installs, here’s what I found to actually work, in the hope that nobody else will go through the ordeal I went through. Following that are some notes about the new Fedora 18 installer (Anaconda 18), especially important for anyone who’s used Linux for a while. To those who are new to Linux, don’t be dissuaded by the above. Most of the frustration I experienced is because I’ve been using Linux for a relatively long time (about 10 years), had my own ideas about exactly how I wanted things setup (which are decidedly not supported by Fedora), and had some assumptions about the installation process based on earlier versions.

How to get it working:

Forget about rEFInd. This had been the original advice from Matthew Garrett, @mjg59, kernel coder, contributor to the Anaconda project, and all-around authority on booting Linux on EFI/UEFI hardware. My advice, and the method that worked for me:

1. Shrink your Mac partitions and leave as much free space as you want for Fedora. using the Disk Utility tool in OS X (I also created an 8GB VFAT partition that both OSes can read/write to).
2. Download Fedora 18 64-bit DVD image, I chose the KDE version. Verify the sha256 sum if you want (they don’t have a readily visible link to the checksum file. Copy the download link, paste it into your address bar and remove the filename. You should get a directory index that includes a -CHECKSUM file.
3. Per the Installation Guide’s Making Fedora USB Media page, use liveusb-creator to setup the installation image on the USB flash drive (I needed to start it with the --reset-mbr option). You can also use other tools (dd if you’re not on a Fedora-based distro), or a DVD, but this is the method I chose.
4. Due to a bug in liveusb-creator, you may need to manually edit /EFI/boot/grub.cfg on the created USB stick if grub gives you a file not found error. If that happens, please see my bug report above for the action to take (in short, you need to mount the USB stick, chmod u+w /EFI/boot/grub.cfg then edit that file and replace every occurrence of “isolinux” with “syslinux” and every occurrence of “root=live:LABEL=Fedora-18-x86_64-Live-KDE.iso” with “root=live:LABEL=LIVE”).
5. Boot the USB drive (use the alt key when you turn on the laptop to select the USB drive) and just install Fedora normally, letting it do its thing. Select a boot disk and let it put GRUB2 on the EFI partition.

When you boot, it will boot to GRUB. There will be some options for Mac OS there, but they don’t work (more on that below). If you want to boot Mac, hold down the alt/option key when you power on the laptop, which will bring you to the boot disk selector and you can pick the Mac disk. I know it’s not pretty or ideal, but it’s the best option right now.

Making it Better:

GRUB2 tries to automatically detect other OSes and configure them in the boot loader (this is done through /etc/grub.d/30_os-prober, commonly just referred to as os-prober). It tries to boot Mac directly through the xnu_kernel64 module, which not only isn’t installed on the boot partition by default, but just doesn’t work with at least Mountain Lion (10.8). So getting GRUB to boot Mac means either having the bugs in the xnu module fixed, or figuring out how to setup a chainloader to boot from GRUB to Mac. The latter is probably the method I’ll investigate, but for now, since I rarely use Mac, I’m happy having to use the alt key at boot to get there. To remove the annoying, broken Mac OS options from the grub screen, run the following commands as root (they assume you have your EFI partition mounted at /boot/efi which I believe Fedora should do by default:

cp /boot/efi/EFI/fedora/grub.cfg /boot/efi/EFI/fedora/grub.cfg.bak
echo 'GRUB_DISABLE_OS_PROBER="true"' >> /etc/default/grub
grub2-mkconfig > /boot/efi/EFI/fedora/grub.cfg

Thoughts on the Fedora 18 Anaconda Installer

I found a couple of issues with the new Anaconda 18 installer that were either unweildy or confusing for someone who’s been installing Linux for a long time. Overall, the new installer is very nice. It has a clean, even elegant UI, a relatively nice flow from start to completion, and is certainly beginner-friendly. It has fewer options than any Linux installer I’ve ever used before – not even options for package selection, firewall or SELinux configuration, etc. – but I guess this is in line with the goal of making Fedora a desktop OS for the masses. I would have appreciated an “advanced mode” installer that was more like Fedora 17 (or even much older versions), but I guess I’m an edge case, at least in the Fedora community. However, I did find two things especially difficult, both related to the fact that my laptop has two main drives (a 500GB hard drive and a 120GB SSD):

First, the installer prompted me to select a “boot disk”. I guess I should have read the installation guide, but I assumed that nomenclature translated to either “which disk should the automatic partitiioning put yout /boot partition on” or “which disk should I set the bootable flag on in the partition table”. In fact, it means “which disk should I put GRUB on the EFI partition of”. I installed, rebooted, and was shocked – and somewhat distressed – to boot directly to GRUB2 instead of the rEFInd installation I had setup. The installer didn’t have any of the previously-customary “warning: this will overwrite your MBR/EFI boot partition” notices, so I felt safe letting it continue. It turned out that this was the way I ended up going, and it also turns out that there’s a bug in Anaconda that makes it fail installation if you tell it not to write a bootloader to disk (though it’s patched by one line of Python code). But I was deeply distressed that – contrary to the experience of every, admittedly more complicated, Linux installer I’d used before – the Fedora 18 installer overwrote my EFI bootloader (analogous to overwriting the MBR on a BIOS boot machine) without ever warning me or asking for a confirmation.

Secondly, the partitioning tool is clearly designed for only one destination disk. The overview screen lists configured partitions by label and mount point, but not by physical device, so figuring out which partitions are on which physical disks takes a click on each and every partition to view that information in the detail panel. When you create a new partition, it’s automatically put in a LVM volume group spanning all disks. Changing the target of the automatically created volume group requires a few clicks, as does changing the physical disks backing any new volume groups. To assign a newly created partition to a specific disk, you have to click on an unlabeled “tool” icon under the list of partitions, far away from the information on the partition in question. It’s a nice interface for someone who clicks the “partition automatically” button, or who just knows they want to add “an extra partition”, but for anyone who has a specific layout in mind (like having /, /boot and /var, specifically sized, on the SSD and /home on the rotating disk) it takes about 4-5 more clicks and dialogs to add a partition than the last Fedora installer did. Mainly, it’s lacking any sort of Advanced Mode for partitioning that allows the user to quickly and accurately layout a more complex partitioning scheme.

Below are some screenshots from the Fedora 17 and Fedora 18 Installation Guides, which contrast both the overview of all partitions and the individual partition settings:

Fedora 18 Overview, from 9.13. Creating a Custom Partition Layout:

Fedora 17 Overview, from 9.14. Creating a Custom Layout or Modifying the Default Layout:

Fedora 18 Partition Creation/Editing, from 9.13.3. Create LVM Logical Volume:

Fedora 17 Partition Creation/Editing, from 9.14.2. Adding Partitions:

• Scripts must be put in /etc/rc.d/init.d, not in the /etc/init.d symlink. They should have 0755 permissions.
• Scripts must have a Fedora-style chkconfig header (“chkconfig:”, “description:” lines), and may have an LSB-style header (BEGIN INIT INFO/END INIT INFO). See Initscript template.
• Scripts must make use of a lockfile in /var/lock/subsys/, and the name of the lockfile must be the same as the name of the init script. (There is a technical reason for this relating to how sysv init terminates daemons at shutdown). The lockfile should be touched when the daemon successfully starts, and removed when it successfully stops.
• Init scripts should not depend on any environment variables set outside the script. They should operate gracefully with an empty/uninitialized environment (or only LANG and TERM set and a CWD of /, as enforced by service(8), or with a full environment if they are called directly by a user.
• Required actions – all of the following actions are required, and have specific definitions:
  • start: starts the service
  • stop: stops the service
  • restart: stop and restart the service if the service is already running, otherwise just start the service
  • condrestart (and try-restart): restart the service if the service is already running, if not, do nothing
  • reload: reload the configuration of the service without actually stopping and restarting the service (if the service does not support this, do nothing)
  • force-reload: reload the configuration of the service and restart it so that it takes effect
  • status: print the current status of the service
  • usage: by default, if the initscript is run without any action, it should list a “usage message” that has all actions (intended for use)
• There are specified exit codes for status actions and non-status actions.
• They must “behave sensibly”. I’ve found this to be one of the biggest problems with homegrown init scripts. If servicename start is called while the service is already running, it should simply exit 0. Likewise if the service is already stopped. Init scripts must not kill unrelated processes. I don’t know how many times I’ve seen scripts that kill every java or python process on a machine.

I intend to use this as a quick checklist when developing or evaluating init scripts for RedHat/Fedora based systems. In my experience, the biggest problems with most init scripts revolve around poor handling of PID files and lockfiles, mainly:

• Killing processes other than the one that the script started (i.e. killing all java or python processes), usually because the PID isn’t tracked at start
• Starting a second instance of the subsystem because lockfiles aren’t used, or the status function is broken.
• improper exit codes
• either explicitly relying on environment variables (and therefore breaking when called through service(8)), or conversely, not cleaning/resetting environment variables that are used by dependent code or processes.

• The Greatest Battle of the Personal Computing Revolution Lies Ahead – Slashdot. A bit of a rant, but makes some good points that are close to my heart, and unfortunately far from the thoughts of many non-techies.
• Terms of Service; Didn’t Read – an interesting project
• How Patent Trolls Harm the Economy – Slashdot – another issue close to my heart
• How Google Cools Its Armada of Servers » Data Center Knowledge
• Data centers – Google Data centers – A photo tour of Google data centers, by Google, along with a Google+ post about the architecture photographer who did this work.
• DARPA Funds a $300 Software-Defined Radio For Hackers – Slashdot – way cool.
• Honeynet Map – “realtime” map of cybersecurity incidents, from the Honeynet Project.
• Malware Is ‘Rampant’ On Medical Devices In Hospitals – Slashdot – a bit scary, but unfortunately not that hard to guess. I’ve seen (probably unpatched) Windows 2000 workstations on hospital networks.
• To top off the scary posts: Researcher Reverse-Engineers Pacemaker Transmitter To Deliver Deadly Shocks – Slashdot

Custom Tombstone Maker

CustomRoadSign.com

CustomMotelSign.com

GetAMedal.com

]]>