Mundane Essays

Power BI tip: Parameters table for Power Query queries to connect to different databases

2015-03-15T18:09:00.001-05:00

I've been using Microsoft's Power BI to help a wonderful client of mine build self-service BI for their analysts and away from a system where they generate csv data dumps out of a Mainframe and then load them manually into Excel spreadsheets.

Most analysts love to use Excel, so Power BI is a no-brainer for getting them their data but Power Query M, though incredibly powerful, doesn't have a way to switch which DB to use. As a developer who develops against my local machine, tests in a test environment and only then promotes my code to production and gives it to the users, is unacceptable.

Here's a workaround I came up for specifying which Server and DB to use through parameters. It's based on Using dynamic parameter values in Power Query Queries. That approach doesn't explain how to use it to connect to different Servers or DBs, which it turns out is trivial, but it also expects the order of parameters not to change, something that strikes me as much too fragile.

First, create a new Sheet and in it, create a table in Excel. You'll need to columns, Name of Parameter and Value. For dynamically specifying the server and database to use, add two rows, with the Name of Parameter of DB Server and DB. Under the Value enter the server and database you want to use.

Don't forget to (Home ribbon) "Format As Table" then (Table Tools / Design ribbon) set the table name to what you want to call it, Parameters, in this example.

Now, we'll use these parameters in our Power Query queries to the DB. To edit your query: (Power Query ribbon) Show Pane, Right click on the query. Then, in the Query Editor that pops up, (in the Home ribbon), click "Advanced Editor". This will show you the source code of the in Power Query Formula language (usually called "M" by Power BI users). Here's an example query:


let
    Source = Sql.Database("devserver", "maindb", [Query="SELECT * FROM [Reporting].[Users]"])
in
    Source

We're going to change the query to load the parameters and use them instead of the hardcoded "dbserver" and "maindb" in Sql.Database function call.


let
    Parameters = Excel.CurrentWorkbook(){[Name="Parameters"]}[Content],
    DBServer = Table.SelectRows(Parameters, each [Name of Parameter] = "DB Server"){0}[Value],
    DB = Table.SelectRows(Parameters, each [Name of Parameter] = "DB"){0}[Value],

    Source = Sql.Database(DBServer, DB, [Query="SELECT * FROM [Reporting].[Users]"])
in
    Source

You'll then change every query that uses this DB to similarly get those parameters first and then use them in the Sql.Database call. The code duplication is unfortunate. I tried a couple of other approaches that take advantage of Power Query's (creating a power query for each of the parameters and creating a function that passes the parameter name) but Power Query's security model doesn't (at least of March 2015) allow another Power Query query or function when retrieving data from an external source.

We're done. To try it out and use a different Server or DB, modify the Parameters table as necessary then refresh the individual query (right click on the query and refresh) or all your queries (Data ribbon then Refresh All).

If you've never connected to the new server via power query, you'll get a popup asking for them as usual when first connecting to a new database. That popup only works properly when you refresh a single query, so you'll want to do that if it looks like your Refresh All is hanging.

Not a software delivery maturity model

2014-08-22T16:07:00.002-05:00

Not a what?

Over years of building software, I've found that when there's a performance problem, I usually jump to conclusions, put in a crazy fix and observe that it makes no real difference. I then remember what I've learned a dozen times, step back, instrument the systems involved, collect and analyze performance data. Lo and behold, the solution put in place usually has a significant impact.

Optimization matters only when it matters. When it matters, it matters a lot, but until you know that it matters, don't waste a lot of time doing it.

In my experience, the same tendency to guess at problems and apply fixes randomly applies to improving software teams. Here's a process I use to avoid such unilateral process changes and instead talk openly about issues and create consensus before acting.

But we follow [insert methodology here]. Why would we change anything?

At OOPSLA 2005, Glenn Vanderburg presented a paper, A Simple Model of Agile Software Processes – or – Extreme Programming Annealed (PDF) that I've thought about every few months since.

My takeaway: XP practices are a coherent set of practices that support the success factor whose health is necessary for software. Embracing the individual practices isn't the point. What matters is ensuring that those success factor are healthy. If you're aware and mature enough to swap a practice out while maintaining the health of that area, go for it; by doing so, over time you can create a process that better suits your needs than the generic one.

Let's get to it...

For a software delivery organization, here's the list of those success factor that I start with:

Asthetic design: Does it hurt my eyes to use this app?
Company direction: What are our goals? What do we do? What are our values? Does everyone understand them?
Design and architecture: Is your system robust? Is the emergent architecture you have today what you'd use if you were designing the system today? Can you diagnose issues across systems?
Dev: Are the individual systems easy to change? Do they fail too often?
Inter-team collaboration: Do teams work well together?
Intra-team collaboration: Do members of a team feel like they have common goals and priorities?
People: Are we supporting people? Are we giving people information they need? How do well do we make decisions?
Portfolio management: Are we focusing our teams on the right projects?
Product vision: Are we building a system that makes for badass users?
Reliability: Can we run this? Can we recover quickly enough when we fail?
Scalability: Do we understand the scaling characterstics of this system and are they appropriate for what we're doing?
Security: Do we understand the security risks and are they appropriate for this system?
Usability: Are we building a system people enjoy using?

Now what?

As a team, review the list of success factors and come up with ones that work for you. Then use interviews and expert intuition to assess which of the success factors matter most right now and how the team thinks you're doing. Here's an example:

Note that I sorted the list by ratio of competence relative to perceived importance. That gives you a suggested order of what you should tackle first. For the first 3 factors, come up with a list of practices that people on the team has used, seen used or have heard or read about that may help you improve it. With the people who will be impacted by any changes, review current practices and potential practices you could introduce. Finally, pick one that you can try quickly at a small scale. Now experiment.

For example, here are some practices I'd consider implementing when looking for ways to improve code quality:

CI
Code reviews
Coding standard
Pair programming
Pull requests
QA by developers who didn’t develop the code
Refactoring as an explicit, expected step during development
Static code analysis
TDD

What about me?

If you're assessing a software delivery organization, formally or not, come up with your own not a software delivery maturity model success factors list and practice list. It should be a list that you can relate to your own past experiences as a team.

Most importantly, remember that improvement isn't something you do in isolation but is rather a team activity. With the whole team working together, you'll gather better information, have a more nuanced analysis, a better chance of consensus and ownership of the improvement process.

You just don't get us: we're a special snowflake!

I concede that you are unique. You should think of other people's practices as tools in a global toolkit. You get to pick which ones are relevant to you. But don't create all of your practices from scratch or assume that other people's experiences are irrelevant to you.

Immature poets imitate; mature poets steal; bad poets deface what they take, and good poets make it into something better, or at least something different. The good poet welds his theft into a whole of feeling which is unique, utterly different from that from which it was torn; the bad poet throws it into something which has no cohesion. A good poet will usually borrow from authors remote in time, or alien in language, or diverse in interest. -- T.S. Elliot

Doesn't this model imply I may not need some practices already in place?

Absolutely.

But you may not want to stop them: You don't want to kill a practice that people value or enjoy. If you see a conflict between an existing practice and a potential one, be honest; explain to the team where you see the conflict and work with them to resolve it.

Should I only implement changes that improve the weakest factor?

No! Limiting the number of improvement projects your team is implementing indicates that you have a bottleneck in your improvement process, probably a manager. If so, you're going about it the wrong way.

Improvement processes can be going on at once improving various factors.

What if the improvement process takes a long time to implement?

Then implement a different practice. Or find a way to chop it up into smaller chunks. You need lots of small wins, at least at first.

Or do it anyway. The key is to find a way to measure results along the way so you can adapt or abort it if it isn't working.

Why do you say this not a software delivery maturity model?

It isn't a model but rather your model: It only works in context; customize it before using.

Now what?

If you think this might help, use it! Over the next week, draft a version of the table for your team. Share it with your team members and incorporate their feedback. Pick one of the red areas in the table, and introduce one practice that could help improve that area. Set a timeframe for the experiment and iterate.

Shameless plug

I help software delivery organizations get more out of their teams. Drop me a line if you're thinking about those kinds of issues. I'd love to help you figure out how to use this process and otherwise work with your agile software delivery organization for improved results.

Thanks to Stephen Anderson for discussing the process and Jason Rudolph for feedback on making the post more relevant and actionable.

It's your data, not TripIt's, take 2

2014-07-01T10:30:00.000-05:00

Minimal viable products are sometimes too minimal

A couple of weeks ago, I posted a blog post on how I extracted my trip information from TripIt to calculate the number of days I traveled last year. Being the second iteration of the process (the first one was: open the page in TripIt, copy and paste to a text file, calculate manually), it worked, but I knew I could make it less crappy: introducing OpenFlights.org wasn't great: every time I fly, I would have to go back there and reimport the new records. The process included manual work to identify trip start and end dates and to distinguish between business and personal trips. Also, I didn't like that only trips with flights were included as some of my trips were

Let's automate some things

Tonight, I looked a into TripIt's API and once I figured out their OAuth scheme, I had a REPL session working, getting data from TripIt's servers. After a spike, processing a page of the data, I decided not to change the data. I then wrote a script that grabbed all the pages of my trips data, combined all Trip nodes and saved them all to an XML file for import and analysis.

Some code...

My script for exporting TripIt data is over in a github repo. Enjoy! Aside from the authentication, there are only a few of lines code:

xml_out.Trips do # build a Trips node
  for page in (1..pages) # defeat pagination...
    past_trips = t.list.trip({'past' => 'true', 'include_objects' => 'false', 'traveler' => 'true', "page_size" => "30", "page_num" => page})
    node = Nokogiri::XML(past_trips.to_xml.to_s)
    xml_out << node.xpath("//Trip").to_xml # extract each Trip node and add it to the Trips node built above
  end
end

I can haz xml... now what?

Caveat: the data format from TripIt's API isn't pleasant to work with. Use your ETL skills to make it useful. I used Power Query because I want to get better at using it. With Power Query, you define your data source and apply filters, schema manipulations and value manipulations. I did several manipulations to make the data useful: I expanded some XML nodes, moved some columns, removed others and renamed some. I also manipulated the data by defining data types, doing some string manipulations so that partial URIs like /trip/show/id/xxxx would become full trip it URLs I can follow and changing values that match "Raleigh, NC" to "Durham, NC".

Power Query steps

On loading this data to the data model, I added a calculated field (trip duration) and a couple of clicks later, the report I wanted was ready. To update the report, I rerun the script, launch Excel and press Refresh All.

What are you going to do with your TripIt data?

[Self promotion ahead!]

I'm a developer who uses whatever tools I have to quickly produce results. These days, I regularly write scripts to load, manipulate, correlate and analyze data to make financial or operational recommendations based on real data. consulting site for a sample of things I've done recently then give me a call if you have data of your own to wrestle or understand.

Fully loaded what?

2014-06-17T22:14:00.001-05:00

Over my years as a hiring manager I've found that I really like working with contractors. Some of the reasons are what you'd expect: it's nice to work together on a temporary basis on things we both find explicitly and mutually beneficial, the contractors I work with are usually experts on the things I hire them for and can hit the ground running.

One operational benefit that I didn't value until recently was the clarity of their hourly rate: when you work with a contractor, you specifically agree on an hourly rate with them. At least in the US, that kind of cost clarity is non-existent for employees.

Especially in the context of professional services, where you're selling someone's time, knowing someone's hourly cost can is quite relevant in setting the price of their services. In that vein, I've seen people use a rule of thumb to approximate hourly cost: divide their annual salary by 2000. For example, for someone making $120,000, the rule of thumb would say the equivalent hourly cost is $60.

This rule of thumb is utterly wrong. The first page of results for a Google search on calculating the "fully loaded cost" of an employee has a telling example. UNC Research provides a formula to help those submitting budget proposals calculate fully loaded hourly salaries. And the example they use is telling: for a full time EPA employee with a $150,000 in salary, their formula comes up with a very different $157.36/hr rate. Remember, the naive rule of thumb I see used all the time would result in an hourly rate of only $75/hr.

Where does this difference come from? As a sometimes hiring manager, what rate should I use when I am thinking through the difference in cost from hiring an employee or a contractor? I really wasn't sure. And many fellow hiring managers weren't sure either. So I came up with a model to help me think through the costs.

Before I go any further, I'd like to remind you of the quote: "Essentially, all models are wrong, but some are useful." This model is wrong. Every business will incur different costs for benefits. You'll have different time off policies. It's a model that I've used to estimate employees' fully loaded cost. If it seems generous, that is because I've been lucky to have been working in tech where the benefits are incredible. But I find it useful, and think you will too.

The model is based on a few assumptions:

US based.
2014 FICA rates.
Paid time off is based on BLS 1996 averages and updated based on my personal experience.
401(k), state unemployment insurance costs and health care insurance based on a 2013 CNN Money article and updated based on my personal experience.
No superawesome mega wowza perks such as lunch, massages, car washes. Instead I've only included the "normal" ones I've come to expect including a conference budget and a new laptops every couple of years.
No other indirect costs like rent.
I factored in a utilization rate (85%) to reflect the fact that even for a billable person, not every hour of the day will go towards billing. Things that I have seen taking up time of delivery personnel include travel, context switch time, internal meetings, reporting hours and expenses and sales support. I expect that even in a product company you'll have time lost to these sorts of things, though the utilization rate may be higher.

With the model a $150,000 full time salaried employee will cost approximately $181,000 a year. With an expected 1530 utilized hours, that works out to $118/hr.

If you're curious about your own fully loaded costs, make a copy of the model and tweak away.

[Self promotion ahead!]

I was a developer and architect. I wrote Java, JavaScript, Perl, Ruby, bash and even VB, C# and Python once in a while. Then I did agile coaching, people management, project and/or product management and as an executive helped grow a software services firm.

Now I use the experience and my passion for lean and data driven decisions to help software delivery teams (companies or departments) run better. If you'd like help understanding your own costs, utilization rates or otherwise, drop me a line.

It's your data, not TripIt's... an example of dragging information out of your buried data

2014-06-12T22:43:00.002-05:00

Update: I have another blogpost with an example of how to pull your trip it data (using a Ruby library and OAuth) into an XML file.

My tax accountant recently asked me for a rundown of days I had traveled for work in 2013. I use TripIt pro to track my travel, this should be trivial. Hooray for SaaS!

Alas, I log in, and look around... nah, no reports here. The closest thing is the past trips list. Alas I can't filter by a date range or destination or purpose. (Aside: the "Traveler" filter is implemented client side and takes down the 10 listed to 0-10 entries. A minimal viable version of the feature. And now we can ignore it. Bah humbug, I say!) No matter, I can do this myself in a spreadsheet, I'll just export my trips out to a csv...

Thwarted again! TripIt doesn't provide an export from what I can tell. A google search leads me in the right direction. The first link is to a discussion about moving data to other systems for analysis of one's own flight data. It refers to OpenFlights.org being able to import TripIt flight information. A couple of clicks later and yet another sign up to yet another site and a dozen more click I had my TripIt data imported to Openflight.org. And another click later I had it exported to a csv. Finally, my - flight - data is mine!

As I am sure you've guessed, I am not really done shaving yaks. Along the way the data got stripped down to flight information, not trip information. So I have to recreate that.

Finally I have the data my accountant needs. But I don't want to send him csv, but rather just the info he wants: which days did I not work in Ohio, a pivot table later, and I have what he wants. There's actually a bug in this calculation, but I'll leave grumbling about the issues I have with Google Spreadsheets vis a vis Excel for another post.

Unlike my previous post, I didn't write any Python or Ruby to solve my problem though I was tempted (and still am: when I do this next time I'll probably use the ruby library to generate the csv in hopes that I don't have to fill in trip purpose, fill in return automatically and get non-flight trip data in). Using spreadsheets in lieu of real programming, I've been able to tackle a surprising number of problems over the last couple of years. Little ones like the one presented, but also more interesting problems like analyzing sales trends for a client and ball-parking the completion date for a 6+ month data center migration so we could schedule with vendors, business teams and engineering teams. Or tracking down why Akamai CDN usage had gone up over a couple of quarters and prioritizing which technical fixes to put in place first. Or calculating billable utilization rates monthly by team over a couple of years.

If you want to hear more about one of those cases let me know. For now, I encourage you to put your data in spreadsheets and play around: they're good for a lot more than laying out your data into a grid.

[Self promotion ahead!]

I used to be a developer and architect. I wrote Java, JavaScript, Perl, Ruby, bash and even VB, C# and Python once in a while. Then I did agile coaching, people management, project and/or product management and as an executive helped grow a pretty awesome software services firm.

Now I use the experience and my passion for lean and data driven decisions to help software delivery teams (companies or departments) run better. See my consulting site for more, and drop me a line if you'd like to chat about a problem you want to tackle.

Life hack: GCal IM reminders

2013-06-13T19:03:00.001-05:00

I am in a lot of meetings. I am also on IM much if not all the time. Despite all the hoopla around Google Hangouts, Google Calendar doesn't support hangout/google talk reminders.

The chrome desktop notifications aren't quite what I need either: they don't give me a link straight into the hangout I am supposed to be in. Instead, when I click the link in them, it opens up Google calendar in the default week view, I have to scan to right now, find the meeting, click it and then click on the "Join hangout" link therein.

A couple of weeks ago I got fed up with this and decided to hack something together to make life easier. I added hangout link support to gcalcli (see my fork), wrote a script that parses the TSV output finding the links I care about in events coming up in the next 10 minutes, and then wired it up with sendxmpp and cron to send me an IM with the meeting title, start time and either a hangout link or a deep link to the event in google calendar.

(While I was at it, I also put together a mini life hack: a script that shows me all the events I have coming up in the next 4 hours.)

If you too want to get your calendar reminders via IM:

Install my fork of gcalcli (master branch)
Authorize gcalcli to your account by launching it -- it'll walk you through the oauth2 stuff including launching your browser
Install sendxmpp
Make your local install of sendxmpp work with the SSL requirements from google by editing and forcing RC4-MD5.
Download/tweak to suit your needs: gcalcli_parse_tsv, cal_im_reminder and install a cron job

If you enjoy this life hack or have a better way to quickly jump into that hangout, chime in with a comment!

Generate SQL from your migrations

2009-02-20T13:07:00.005-05:00

On one of the projects we're working on, we needed to occassionaly generate SQL from our migrations. Déjà vu, I thought. A few minutes of Googling later, I remembered why: Jay and I had been on a project a couple of years ago whence we had the same need. Jay's code didn't quite work any more due to some ActiveRecord changes, and a search for an alternative implementation turned up nothing.

I took his code and modified it to our needs. A short time later, I had the code pulled out into a rails plugin, migration_sql_generator. Install it (script/plugin install git://github.com/muness/migration_sql_generator.git) and then run the rake task:

rake db:generate:migration_sql

Running this task generates two sql files per migration in db/migration_sql in the form 20090216224354_add_users.sql and 20090216224354_add_users_down.sql.

I've used the plugin with success using the sqlserver adapter, less luck with the mysql adapter (change_column and rename_column blow up because the mysql adapter checks for the presence of a column first) and no luck with the sqlite adapter. Haven't tried it with the postgres adapter.

Easily switch between Ruby 1.8.6 and 1.9.1

2009-02-20T12:05:00.011-05:00

NOTE: I don't keep Ruby Switcher updated anymore. Use RVM instead.

Chad mentioned that he'd gotten ruby 1.9.1 and 1.8.6 side by side on his workstation (his code for this is in his awesome spicy-config repo).

I took some time this morning to get a similar setup working. Now, on my prompt, I type, use_ruby_186 or use_ruby_191 to go back and forth between the Ruby 1.8.6 shipped with Leopard and a self-compiled install of ruby 1.9.1.

Steps for you to get there:

Compile and install ruby 1.9.1:


mkdir -p ~/tmp
cd ~/tmp
curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-p0.tar.gz
tar xzf ruby-1.9.1-p0.tar.gz
cd ruby-1.9.1-p0
./configure --prefix=$HOME/.ruby_versions/ruby_191
make
make install

Install the ruby switcher:


curl -L http://github.com/relevance/etc/tree/master%2Fbash%2Fruby_switcher.sh?raw=true?raw=true > ~/ruby_switcher.sh
echo "source ~/ruby_switcher.sh" >> ~/.bash_profile

Note that I use ~/.gem for my gems. If you don't do that, you'll have to modify the switcher to specify your GEM_HOME. Type gem env and look for GEM PATHS to figure out what you should set it to.

Maatkit installation: the concise guide

2009-02-03T01:10:00.002-05:00

If you use MySQL, odds are you need Maatkit whether you know it or not. It's the Swiss army knife: I mostly use it to parallelize backup and restore of huge databases on multi-core machines, but it's also handy for fake splitting large files, executing sql on multiple tables and a whole lot more. (You also want mytop if you're wondering what's up with MySQL connections.)

Here's the install script if you too want Maatkit at your fingertips (and you do. trust me.):


#!/bin/sh
# tested on Leopard with MySQL installed using the package installer

sudo perl -MCPAN -e 'install DBI::Bundle'
sudo perl -MCPAN -e 'install DBD::mysql'
# you may need to force the install as follows:
# sudo perl -MCPAN -e 'force install DBD::mysql'

mkdir -p ~/tmp
cd ~/tmp
curl -O http://maatkit.googlecode.com/files/maatkit-2725.tar.gz
tar xzvf maatkit-2725.tar.gz
cd maatkit-2725
perl Makefile.PL
sudo make install

No network after copying an Ubuntu VMWare image

2009-01-22T09:24:00.003-05:00

After you copy an Ubuntu (or any Debian based) image, you'll probably lose your network connectivity. After a little bit of digging, it turns out that udev persists the MAC address of the network device in /etc/udev/rules.d/*net.rules . The fix:

sudo rm /etc/udev/rules.d/*net.rules
  sudo shutdown -r now #to reboot

Git: things I love and manipulating remote branches

2008-11-14T09:44:00.008-05:00

I've been happily using Git for months now. The primary benefit for me has been the ability to work offline without messing with SVK, not having to use the svn commands for renames and moves, the ability to use multiple remote servers and better tools to look at branch level history (think gitk).

Branching is much improved, besides. In Subversion, to branch, you copy the current tree elsewhere and then you work in this branch and finally, merge your changes manually back into trunk (to wit, you have to keep track of what was already merged and not reapply those commits). In git, though, that's all handled for you. Another difference is best illustrated by example: you've started a new feature. Several file modifications later, you look at your diff and realize, "I should have started a new branch for this. doh." With Subversion, there's not much you can do automatically to get these changes into a branch. You'd typically now:

generate a diff
undo your changes
create a branch
switch to the branch
apply the diff
back to work

In git on the other hand, you type git branch feature_22.

Sweet, huh?

Well, not altogether. Pushing this local branch to a remote server is altogether too complicated:


git push origin feature_22:refs/heads/feature_22
git fetch origin
git config branch.feature_22.remote origin
git config branch.feature_22.merge refs/heads/feature_22
git checkout feature_22

Sure, you could have created the branch on the remote server first and saved yourself some hassle:


git push origin master:refs/heads/feature_22
git fetch origin
git branch --track feature_22 origin/feature_22
git checkout feature_22

Not exactly straight forward either. Neither are other standard branch related activities. Here's how you delete that remote branch:


git push origin :refs/heads/feature_22 # yeah, that's how you delete a remote branch.  :(
git branch -d feature_22 # delete it locally too

Fortunately, Git has a vibrant community that work to abstract away some of these complications. One of my favorite Git tools is git_remote_branch. By using it, working with remote git branches becomes a breeze:


grb publish feature_22 # publish a local branch
grb create feature_22 # create a remote branch and local branch to track it
grb delete feature_22 # delete the remote and local tracking branch

Ah, that's better!

Mingle, meet Git

2008-09-21T12:12:00.005-05:00

Several weeks back, Adam Monago, Mingle's product manager was visiting our office in Chapel Hill. One of the topics that came up was Git integration with Mingle. Alas, it sounded like it was a ways out. But he explained that the SCM integration was pluggable and that we could write the code and drop it in place, echoing things I'd heard from other ThoughtWorkers.

A couple of days later, Don and I were curious as to just what it'd take to implement the integration. Reverse engineering the interface we had to implement from the Subversion and Perforce plug-ins was anything but fun, but before the day was over, we had rudimentary integration working: we could see commit messages and the list of files modified per check-in. Since then, Don implemented the rest of it including Mingle-based source code browsing.

We've been using the mingle_git plugin for a little while with no problems. For performance reasons, we're not using the source code browsing from inside Mingle. Instead, we use GitHub's source browsing: see the README for instructions on wiring things that way too.

Installation is documented in the README. We've only used this on Mac OS, FreeBSD and Linux. Windows users, you may have luck by using msysgit.

Caveat emptor: if you browse around the code, you'll quickly conclude that it is a spike: we started with the subversion plugin and evolved it to this. Since the SCM integration API is not documented, we can't even be positive that we implemented it correctly (though the evidence suggests that we have). Also note that Mingle makes the assumption that check-in numbers are sequential, and that's an assumption that is not Git-friendly and it shows in the code.

Enjoy, but keep in mind that this is unsupported, use-at-your-own-risk software. For me, that's better than no Git integration. ;)

Note: Mingle's APIs changed from 2.0 to 2.1 and we haven't had a chance to update this plugin yet. If you installed this plugin, and are ready to upgrade to 2.1, you'll want to remove the plugin and then execute this SQL on your Mingle database: delete from plugin_schema_info where plugin_name = 'mingle_git';

Encrypt your client data in 53 minutes

2008-08-17T11:35:00.007-05:00

Thanks to Aaron's continued efforts to make us ever more security conscious, I've been encrypting client data on my laptop. This came up in conversation during erubycon (thanks EdgeCase for a fun conference with engaging evening events). A couple of people asked me to get them started on encrypting their data too.

Here are instructions. They're written for Mac users, but they'd be nearly identical for Linux and Windows users since TrueCrypt is available there too.

Install TrueCrypt.
Create a container based encrypted file. Pick FAT as the file system. I use both keyfiles and a password.
Erase the FAT partition and then format as HFS/ext2/[fs of choice] using Disk Utility/[partition manager of choice]. Name it client_data.
Mount the newly created partition.
Move your sensitive data to /Volumes/client_data. This is by far the slowest part of the process. While you wait, watch The Enemies of Reason (48 minutes -- everything else should take 5 minutes or less).
Modify TrueCrypt preferences to suit your needs. Some suggestions: leave encrypted volumes mounted when quitting TrueCrypt, set Auto-dismount volume after 30 minutes of inactivity in the partition.

As mentioned in my previous blog entry, I use a script to automatically create aliases to allow easy switching to project directories. All I had to do was switch that entry to point to /Volumes/client_data instead of the old ~/work/client_data. Now, when I want to work on client projects or contracts, I launch TrueCrypt, mount the container and open a new terminal window. All my old aliases work just like they used to.

Mac users: if you don't care about the cross-platform compatibility that TrueCrypt offers, and trust Apple to encrypt your data, you could use Disk Utility to create encrypted partitions instead. Some people insist it's more convenient (You know who you are!).

Lazy bash cd aliases

2008-06-29T13:04:00.011-05:00

Those of you who've found value in my last couple of bash specific posts may also like the latest addition to my ~/.bash_profile.

This one iterates through the one or more directories and creates aliases to the subdirectories so I don't have to. Here's the scenario: I've got a directory ~/work/ where I keep work projects, a ~/writings/ where I keep all the writing projects and so on. I used to have aliases to each subdirectory. e.g. alias project1="cd /Users/muness/work/project1". With shame, I admit that I maintained each of these manually. No more!

Install instructions:

curl -L http://github.com/relevance/etc/tree/master%2Fbash%2Fproject_aliases.sh?raw=true?raw=true > ~/.project_aliases.sh
echo "source ~/. project_aliases.sh" >> ~/.bash_profile

Usage instructions:

Move your work projects to ~/work.
Above the source ~/. project_aliases.sh add PROJECT_PARENT_DIRS[0]="$HOME/work".

You may be interested in my blog post over at PragMactic OS-Xer where I describe my motivation for these recent shell scripts.

bash: don't make me think

2008-06-16T19:56:00.008-05:00

Last week I figured out a way to make my life a little bit easier by abstracting the scm I was using and having the prompt indicate whether I was in a Subversion or Git. Thanks to Mike Hommey whose script I tweaked for my needs.

For a long time, I've wanted my iTerm tab title to be more useful. I started by having it display the last process I executed in that tab (this also shows the full path in iTerm's window title bar):


PS1='\[\e]2;\h::\]${PWD/$HOME/~}\[\a\]\[\e]1;\]$(history 1 | sed -e "s/^[ ]*[0-9]*[ ]*//g")\a\]\$ '

Next up I wanted to show the currently running command. Google showed me how:


trap 'echo -e "\e]1;$BASH_COMMAND\007\c"' DEBUG

Instead of trying to explain this code I'll quote the Bash man page:

BASH_COMMAND
The command currently being executed or about to be executed, unless the shell is executing a command as the result of a trap, in which case it is the command executing at the time of the trap.

Some more tweaks followed:

Distinguish between a previously executed command and a currently executing command by decorating them (I chose surrounding the former with braces and the latter with >/<. e.g. [ls] and >vi<).
Show the context of the executing command in the tab. This is just the repository where I executed the command.
Make this coexist with TextMate.

And a screenshot to summarize:

The code is in github/relevance/etc/tree/master/bash/bash_vcs.sh.

Install instructions:


curl -L http://github.com/relevance/etc/tree/master%2Fbash%2Fbash_vcs.sh?raw=true > ~/.bash_dont_think.sh
echo "source ~/.bash_dont_think.sh" >> ~/.bash_profile

Enjoy! Tested with iTerm on Leopard.

Stop the presses: bash said to embrace subversion and git

2008-06-10T19:41:00.009-05:00

Update: I've since updated and moved this script. See my new blog post.

I had a chance to pair with Rob today. On his console window, I noticed something I wanted: his bash prompt had an indication that he was in a Git repository along with the branch he was on. Later, after he'd left, I found myself wondering how that worked and that I simply had to have it, and I wasn't willing to wait a whole day to figure out how he'd done it. Patience is not a virtue. (In that spirit, click here for the code if you're too impatient to read the rest of this blog entry which I spent countless hours writing. Uphill both ways. In the snow.)

A few minutes of googling turned up a blog entry with relevant bash fu. The code there was for changing the prompt such that it indicated whether you were using svn, git, svk or Mercurial along with some repository metadata. A couple of minutes later (it took a while to copy and paste the code since it was interweaved with comments), I found out it didn't work on Leopard. readlink turned out to be the culprit. Instead of figuring out why it wasn't working, I replaced it with something simpler (specifically I used: base_dir=`cd $base_dir; pwd`). Voila, things worked: I now had a hella cool prompt that showed me if I was in a Subversion or a Git repository - I never used Mercurial, and left SVK for Git a couple of months ago. Yay.

Being who I am I had to do something more with this new ability to distinguish between Subversion and Git. And I knew exactly which itch to scratch: 90% of the time I use the same scm commands. I even had shell aliases for them so I wouldn't have to type them. A few minutes later I'd converted my previously Subversion specific aliases to generic ones that worked with Git too. Screenshot says it all:

If you want the same, download the script as ~/.bash_vcs and add source ~/.bash_vcs at the end of your ~/.bash_profile.

Buckets of mice

2008-04-25T10:23:00.012-05:00

We have 24" and 30" pairing stations, an intern, freshly roasted coffee delivered weekly, a fridge stocked with caffeine, more caffeine, beer, and Dave's Insanity sauce. We have cool t-shirts and all manner of radiators up on the walls, whiteboards and the Beta Brite. Music is served by our CI/Music server.

The bookshelves are stocked with reference materials, the occasional bottle of scotch, and recommended readings. On the list of recommended readings we've got The Carpet Makers, The Company, The Omnivore's Dilemma, The Eyre Affair, Behind Closed Doors, Bloodsucking Fiends and The Deadline, Men In Hats, Volume I, Don't Make me Think, The Insane are Running the Asylum.

Our favorite radiator is the Friday radiator. Fridays are special: we don't do billable work. Instead we open source, blog (self reference makes this blog entry extra cool), watch educational movies, play Wii on the projector. Here's today's:

Oh, let's not forget the buckets of mice.

This blog entry inspired by another excellent business card cartoon from Hugh Macleod.

cmd line history meme

2008-04-16T09:06:00.002-05:00

Rob tagged me. So:

muness$ history 1000 | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
102 git
72 ls
63 cd
40 ss
35 rake
24 m
17 up
14 st
12 sc
11 ri

Tag, you're it.

Distributed Retrospectives

2008-04-11T12:10:00.012-05:00

Agile Retrospectives (Esther Derby, Diana Larsen) and Project Retrospectives (Norman L. Kerth) have been a great resource in facilitating - or helping others facilitate - retrospectives. I'll often leaf through them a day or two before a retrospective to remind myself of the framework (more on that in an forthcoming post), and to introduce new activities or create my own.

Something that neither of those books tackle directly is how to facilitate a distributed retrospective. Given that we've been running a few of those lately, I wanted to share some tips:

Your equipment matters. It took us a while to find a speaker phone that worked well for those in the office and those on the other end. It's hard enough to communicate effectively without being in the same room, without muffled voices. We use a Polycom unit with two extensions for a 20x16 room.
Prepare. There's a lot of preparation that goes into running a retrospective in person. You have to familiarize yourself with the project, look for a theme, send out invitations, pick activities (and backup activities in case one isn't working), and collect the tools/resources you need for those activities.
Google Docs is your friend. With its near real time updates for viewers and collaborators, it's been an indispensable tool to take notes. The facilitator or a scribe can update the document and others can see what's happening, on the fly.
It's not enough to use a Google Doc, you need to turn it into an online workspace that makes others feel like they're part of the process. For example, I prepare a google document with activities scheduled for the retrospective. I take care to:
- Clearly state activity names. When we move to the next one, it's enough to say its name and everyone can easily find the appropriate section.
- Describe the activity. Having a written description allows people who miss the spoken description to follow the intent as well as the process involved. Missing a description over a conference call can happen for all sorts of reasons, from people being more easily distracted, to phone problems.
- State the targeted time for the activity. People are even more restless on a conference call. This gives them a sense of the overall meeting's timeline.
- Copy or summarize information from previous retrospective documents for ease of reference.
Share with others. Sharing the document with others as collaborators (rather than just viewers) gives them the confidence that they can correct mistakes.
Don't forget to send an email shortly before the meeting that serves as a reminder, repeats the phone number/conference line details and includes a link to the Google Doc for that meeting.

I've shared a Google Doc template for a 2 hour iteration retrospective for 4 to 8 people you can refer to or use as a starting point for your own living retrospective document. Credit goes to Agile Retrospectives for the activities.

Testing declarative code

2008-04-11T09:20:00.007-05:00

For a while now, I've had my doubts about the usefulness of one aspect of test driven development (which applies equally well to behavior driven development): that it should be done 100% of the time.

The first time I came across this, was at ThoughtWorks. The question was whether we should TDD declarative delegations. I remember a several hour long conversation about the merits of tests for Forwardable based delegations.

class A
  extend Forwardable
  def initialize(b)
    @b = b
  end
  def_delegator :@b, :some_method  # Should I write a test before writing this declaration?!
end

A specification for this declaration would be:

describe "A" do
  it "delegates some_method to b" do
    a = A.new(stub(:some_method => :x))
    a.some_method.should == :x
    # or you could use the Handoff assertion:
    assert_handoff.from(A.new).to(:@b).for_method(:some_method)
  end
end

I didn't get the fuss -- it was clear to me that testing such delegations was wasteful and a maintenance hassle. But I didn't realize that this was part of a larger argument.

Not an isolated case

Since then, I've learned that to some developers this is a general rule. Here's another example, re testing stock ActiveRecord validation declarations, e.g.:

class Model < ActiveRecord::Base
  validates_presence_of :attr_1
end

The test for this would be:

describe "Model" do
  it "validates_presence_of :attr_1" do
    model = Model.new
    model.valid?
    model.errors.on(:attr_1).should == "must not be blank"
  end
  # or you might write a helper that reflects on validates_presence_of declarations
  test_model_validates_presence_of Model, :attr_1
end

It took me a while (too long) to get the general pattern: these discussions I kept finding myself in all had to do with testing declarations.

How did we get here?

Test driven development as best as I can tell is a - pragmatic - descendant of formal specifications:

"A formal specification is a mathematical description of software or hardware that may be used to develop an implementation. It describes what the system should do, not how the system should do it. Given such a specification, it is possible to use formal verification techniques to demonstrate that a candidate system design is correct with respect to the specification."

Writing formal specifications for the majority of the software isn't practical. Writing tests first provides us a simple, practical way to write executable specifications. True, they don't give us the ability to use "formal verification techniques" to prove anything about our code, but they do give us confidence that our code does what our tests say the code does (as much confidence as we have in our tests).

The rule

The rule regarding TDD is: If you can TDD it and it's destined to be production code, you should TDD it.

The problem I find with this rule is that it causes us to write tests like those above that duplicates the declarative code. At best case it goes like this:

  # We write this code first in the spec:
  test_model_validates_presence_of Model, :attr_1 
  # we run it and watch it fail...
  # and then we write this in the model class:
  validates_presence_of :attr_1 
  # rerun the test and watch it pass...

I find this wasteful: We first implement the test_declaration_is_called (in this case, test_model_validates_presence_of) for every declaration. After all that work (and not to mention code that we have to maintain) we end up with a test that tells us what the declaration itself does. But with less readability and intent. Indeed, declarative code is in fact formal specification.

Guidelines not rules

Writing tests first is a great guideline. I do it almost all the time. But I don't do it all the time. It makes me angry when I change declarative code only to have a test that looks exactly the same fail as well.

OK, tests for declarations don't really make me angry. But they do make me wonder if the developer who wrote the test really likes to answer the question, "are you sure that you're sure that you want to do what you just said you want to do?"

Where there's smoke signals...

2008-03-14T10:59:00.005-05:00

In a previous post, I briefly described how we were using cc_campfire_notifier to get build success/failure notifications in campfire. cc_campfire_notifier has lost momentum over the last couple of months. Since we actively rely on the notifier for several projects, we've now forked it as smoke_signals.

The first release offers the following improvements:

Each notification includes a link back to the CruiseControl.rb build (so you have one-click access to the full details for each build).
Recognizes apr_error errors as SVN errors (so you'll spend less time scratching your head over generic build failure messages).
Speaks once into the campfire room whenever an unexpected error occurs (as opposed to speaking once per line in the backtrace).

Check out the notifier's home page for installation instructions or head on over to GitHub to browse the code.

Release early, release often, business edition

2008-03-06T08:46:00.003-05:00

One of the things we preach as software developers is release early, release often. I think ESR is right in that our primary motivation for this is often related to quality:

In the bazaar view, on the other hand, you assume that bugs are generally shallow phenomena—or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.

As I've done more project consulting and management, I've realized that the benefits extend well beyond the technical ones. But I've had a difficult time articulating the business benefits to incremental, frequent releases.

Last week, Ross posted an article that describes one of the primary business benefits to frequent, early releases, Minimising the Speculative Risk of IT Investments:

With each incremental delivery, and every increase in tangible value, the intangible or speculative value decreases. The reduction in speculative value at risk represents a reduction in the total value that can be depressed through delays in delivery. Thus, early delivery reduces the risk of speculative value not being realised. Simultaneously, it reduces the volatility of returns.

What other benefits do you find to the business from releasing early, releasing often?

CC.rb Campfire Notifier Released

2008-02-07T18:19:00.001-05:00

Note: I've forked this notifier to add features and fix some bugs at in Smoke Signals. At Relevance, we make heavy use of Campfire for our distributed projects. One room per project, and we expect everyone to "be in" the relevant room associated with the project they're working on at the time. We use it to communicate: to ask for a pair (we've tried Leopard's Remote Desktop Sharing, but have no settled mostly on emacs + screen), ask questions, request code reviews (we've taken to using the topic for posting ad hoc requests like that), and make announcements.

This virtual space has many disadvantages over being in the same room as I'd been used to on my previous couple of projects, but it did have a nice perk: SVN commit notifications were right there alongside our own discussions. The notices were more useful than I expected, a quick, clear indicator of our pace on a given day.

Naturally, I wanted our other major automated task to notify us of CCMenu is OK, but it's disconnected from our primary virtual space (Campfire, as discussed above). Being lazy, I didn't want to build it myself, so I first looked around for tools that suited my needs. Much to my chagrin, none did, so I took the one that was closest and added to it what I needed: specs. Then came the stuff that was missing: ssl support and support for one room per project. And last but not least, to help deal with configuration hassles and debugging, I added logging.

If you're using CruiseControl.rb and Campfire, give it a whirl. Instructions are in the README.

Pi

2008-02-05T16:53:00.000-05:00

Managers and rules

2008-01-20T18:46:00.001-05:00

Jerry Weinberg, if you don't know him, is the author of more than a dozen books on software development and memorable laws such as the Law of Twins. In Quality Software Management: Systems Thinking, published in '92, he described what we might today call an agile team. (He lists several levels of software team cultures, ranging from Oblivious to Congruent. Along that spectrum, an agile team would fall somewhere along Steering, Anticipating and Congruent.)

At AYE 2007, I had the pleasure of meeting Jerry in person. Something that I was not aware of was his remarkable skill at helping others. Instead of answering a request for advice directly, he'd delve into the details, applying something like the 5 Whys. Within minutes, he would hone in on a rule that the person had imposed on themselves that was at the root of the problem. After expressing the problematic rule, he'd help them relax the rule into guidelines. Several people who've spent more time with him have echoed my gut feel that they too found this ability to help others rare and invaluable.

His approach to helping individuals is based on Virginia Satir's work in family therapy on survival rule transformation. An excerpt from The New People Making:

After you have written down all the rules your family thinks exist and cleared up any misunderstanding about them, go on to the next phase. Try to discover which of your rules are still up to date and which are not. As fast as the world changes, it is easy to have out-of-date rules. Are you driving a modern car with Model T rules? Many families are doing just this. If you find that you are, can you bring your rules up to date and throw away the old ones? One characteristic of a nurturing family is the ability to keep its rules up to date.
Now ask yourself if your rules are helping or obstructing. What do you want them to accomplish? Good rules facilitate instead of limit.

Last weekend, while attending a Satir workshop I was struck by the applicability of Satir's ideas on rules to teams at work. There too, rules can be unstated, ambiguous or inapplicable sometimes hampering our ability to get things done, other times making everyone miserable.

(For a valuable discussion on rules suitable for software development teams, check out the section on Rules in chapter 21 of Software Teamwork:Taking Ownership for Success. To summarize, the rules should be focused on ensuring that a team has "the capabilities and resources to do the right things, rather than on stepwise procedures to be dogmatically followed".)

In big companies the problems are magnified as fully connected networks of people become prohibitively expensive. For example, it'd take each person over a day out of every week for 25 people to stay in meaningful weekly one-on-one contact. (Indeed, the few who take the time to build an extensive network can become stars capable of getting things done faster than their peers could.) Additionally, in a big organization, it's likely that a rule that works for one group will be applied to ones where it doesn't fit.

The role of a manager is to engender an enabling environment. But how? Virginia Satir's approach to rule transformation provides one way. One which fits in well with the management advice in many of the management books I am familiar with, including Behind Closed Doors, Peopleware and Slack.

Here's how I describe the process:

Identify implicit rules and restate the explicit ones.
Explore how they came about.
Express the rules explicitly.
Clear up ambiguities in the team's understanding of each rule.
Evaluate each rule's applicability.

Having done this, you might find that a rule is:

Helpful and reasonable. So, apply it!
Helpful but impossible or untenable. Then transform it into one or more guidelines that aren't prohibitive.
Limiting or obstructing your team. If so, the rule should be challenged, publicly.

Again, quoting The New People Making:

What do you think about your rules? Are they overt, human [i.e. reasonable] and up to date? Or are they covert, inhuman [i.e. prohibitive, impossible] and out of date? If your rules are mostly of the second variety, I think you realize that you and your family [or team] have some important and necessary work to do. If your rules are of the first category, you are probably all having a ball.