If you missed my other blog post, I recently release a Java library named jgit-flow that implements the git-flow branching and merging model introduced by Vincent Driessen in Java.

My hope is that developers will use this library to integrate git-flow workflows inside of their Java apps, and to get the ball rolling, I thought I’d be the first integrator.

If you read my other blog post, you’ll remember that although git-flow is really useful, trying to make it play nice with the maven-release-plugin (MRP) leaves a lot to be desired.

In particular, some of the pitfalls of the maven-release-plugin in the context of git-flow are:

• MRP writes .backup and release.properties files to your working tree which are easily committed when they shouldn’t be
• MRP does a build in the prepare goal and a build in the perform goal causing tests to run 2 times
• MRP and git-flow both create tags leaving you with two tags if you forget to delete one of them
• If something goes wrong, MRP usually leaves you in a bad state and rollback doesn’t work most of the time

So I figured the best plan of action was to write a specific plugin to enable git-flow style release workflows, and after a bit of twiddling, I’m pleased to bring you the JGit Flow Maven Plugin.

Features

While the plugin is primarily used to perform releases, it also provides full git-flow functionality including:

• starting a release – creates a release branch and updates pom(s) with release versions
• finishing a release – runs a maven build (deploy or install), merges the release branch, updates pom(s) with development versions
• starting a hotfix – creates a hotfix branch and updates pom(s) with hotfix versions
• finishing a hotfix – runs a maven build (deploy or install), merges the hotfix branch, updates pom(s) with previous versions
• starting a feature – creates a feature branch
• finishing a feature – merges the feature branch

Differences with the Maven Release Plugin

Although the maven-jgitflow-plugin is based on the maven-release-plugin, it does things a bit differently to provide a cleaner workflow…

• doesn’t create any .backup or release.properties files (or any other files in your working tree)
• makes all changes on branches. This means to “roll-back” you can simply delete the branch
• automatically copies any profiles (-P) and user-properties (-D) passed on the command line to the forked maven process when building
• doesn’t run site-deploy
• provides the ability to completely turn off maven deployment
• provides the ability to completely turn off remote pushes/tagging
• Only builds your project once in the finish goal. e.g. if you do release-start and release-finish together, your tests only run once
• Never clones your project to a temp folder
• auto tracks origin based on maven scm values if origin tracking is not already setup (I’m looking at you Bamboo!)
• master branch is always kept at the latest release version (not a SNAPSHOT)

I’m hoping this plugin will help ease the pain of releasing maven projects with git-flow and help reduce the number of groans when someone says “can you do a maven release of…”

To get started, check out the plugin’s wiki. And when things don’t work, let me know in the plugin’s issue tracker.

If you’re interested in contributing, please fork the Maven JGitFlow Plugin on Bitbucket. When your super cool new feature is ready, just issue a pull request to the develop branch of the main project.

Enjoy!

resources

Wiki: https://bitbucket.org/atlassian/maven-jgitflow-plugin/wiki/Home
Issue Tracker: https://bitbucket.org/atlassian/maven-jgitflow-plugin/issues
Google Group: https://groups.google.com/forum/#!forum/maven-jgitflow-users
Source Code: https://bitbucket.org/atlassian/maven-jgitflow-plugin

The problem of multiple account silos is common across the technology domain, yet is a surprisingly difficult one to resolve. It starts with a simple requirement: “We want to use the same login for multiple systems.”

But what sounds easy on the surface can quickly evolve into a complex blend of concerns across technology, data migration and separate functional teams.

So I am pleased to be able to outline the process and technology solution we used as part of this project, in the hopes that it will mitigate some of the headaches in delivering similar initiatives.

Welcome to a Behind the Scenes look at the creation of Atlassian ID.

Challenges and Learnings

Before diving into the solution and design process, I will comment on some of the key challenges (both technical and non-) and learnings. Our biggest challenges were:

• Multiple teams. By its nature the project required engagement across a number of functional groups within Atlassian. This was made somewhat easier than in most organisations due to Atlassian’s fast moving pace and can-do-it culture, but it was also a double-edged sword. Once we had agreement with the teams, it was necessary to keep up with them as they powered ahead providing new services and capabilities on their system.
• Availability. Having one user base for all your applications can be effectively like putting all your eggs in one basket. You don’t want that basket to break. Availability was a significant challenge to tackle throughout the solution.
• Customer migration. This was a big one. We had seven systems and a large user base that had been diverging for over 10 years. You name the scenario… we had it.

What did we learn from the process? What stood out was:

• Data migration is hard. I have learnt this a couple of times previously in my experience but I was once again reminded. Data migration is hard and always takes more effort than you think. Plan it out…then double.
• SSO is important. From a technical perspective there is an urge to dismiss this as unimportant. The reality is that SSO is hard. It raises a number of security and distributed ‘state’ concerns that need to be dealt with. The alternate argument is that all browsers remember usernames and passwords, meaning only one additional click for users. Surely a single username and password rather than SSO is ‘good enough?’ More often than not the answer is ‘no.’ By giving away SSO you are sacrificing the ability to really provide a seamless customer experience. In addition, you are losing additional technical capabilities that can be extremely useful. The primary one I would cite is the ability to produce finer grained application or services that can act together to form a larger site (micro-sites/services). The ability to divide your applications into a number of smaller modules has significant benefits for maintenance and development speed.
• Authentication is complex. It is quite amazing how complicated a simple concept such as authentication can become, this domain is littered with many different (and often competing) standards and options for implementation. We dealt with a handful of these methods including your standard web based login pages, OpenID, BASIC-AUTH and some other custom authentication mechanisms including of course the Seraph connector (used by Confluence and JIRA).  There are many others and it is an area to tread with caution.
• Design patterns are good. Implementing without a reference architecture is equivalent to exploring uncharted territory, you never know what is going to happen. Having a good design pattern is like having a map with all the pit-falls and dangerous areas to avoid highlighted. If possible always find a relevant design pattern.

Our Solution

We had seven systems that were immediately in scope for the project, each owned by a different business unit here at Atlassian. So the first step was to understand the immediate and potential future requirements of each of these systems and units. They boiled down to the following key capabilities:

• Single sign-on (SSO). The ability to sign-on once and access multiple applications without the need to re-authenticate.
• User aliasing. The ability to ‘harmonize’ non-uniform local user identifiers by the means of aliasing or mapping, i.e., username brendan on system A is the same user as bhaire on system B. This was an important capability in relation to legacy users and migration as it meant we could onboard systems to the solution without imposing the need to rename user accounts on the target system.
• Central identity management. The ability to capture, store and manage user identity information centrally.
• Provisioning. The ability to push user identity events across multiple applications e.g. New User, Updated Profile.
• Federated authentication - The ability to provide authentication services to a 3rd party system external to Atlassian, e.g., OpenID, SAML based auth styles.

We used Crowd to form the backbone of the solution, and extended it using the Interceptor or Gateway architecture pattern. We also settled on SCIM as a standard for the identity management space. The final solution is one that can be used to provide the access and identity management services both to JIRA and Confluence applications as well as commercial off the shelf (COTS), open source and home grown applications.

Are there alternatives? The answer is: yes! There are (at least) two tools that can help track the history of software dependencies in your project while allowing you to keep using git:

• git subtree
• google repo

In this post I will be looking at git subtree and show why it is an improvement – albeit not perfect – over git submodule.

As a working example I run to my usual use case. How do I easily store and keep up to date the vim plugins used in my dotfiles?

Why use subtree instead of submodule?

There are several reasons why you might find subtree better to use:

• Management of a simple workflow is easy.
• Older version of git are supported (even before v1.5.2).
• The sub-project’s code is available right after the clone of the super project is done.
• subtree does not require users of your repository to learn anything new, they can ignore the fact that you are using subtree to manage dependencies.
• subtree does not add new metadata files like submodules doe (i.e. .gitmodule).
• Contents of the module can be modified without having a separate repository copy of the dependency somewhere else.

In my opinion the drawbacks are acceptable:

• You must learn about a new merge strategy (i.e. subtree).
• Contributing code back upstream for the sub-projects is slightly more complicated.
• The responsibility of not mixing super and sub-project code in commits lies with you.

How to use git subtree?

git subtree is available in stock version of git available since May 2012 – 1.7.11+. The version installed by homebrew on OSX already has subtree properly wired but on some platforms you might need to follow the installation instructions.

Let me show you the canonical example of tracking a vim plug-in using git subtree.

The quick and dirty way without remote tracking

If you just want a couple of one liners to cut and paste just read this paragraph.

First add the subtree at a specified prefix folder:

1
git subtree add --prefix .vim/bundle/tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git master --squash

(The common practice is to not store the entire history of the sub-project in your main repository, but If you want to preserve it just omit the –squash flag.)

The above command produces this output:

1
2
3
4
5
6
7
8
9
10
git fetch https://bitbucket.org/vim-plugins-mirror/vim-surround.git master

warning: no common commits

remote: Counting objects: 338, done.

remote: Compressing objects: 100% (145/145), done.

remote: Total 338 (delta 101), reused 323 (delta 89)

Receiving objects: 100% (338/338), 71.46 KiB, done.

Resolving deltas: 100% (101/101), done.

From https://bitbucket.org/vim-plugins-mirror/vim-surround.git

* branch            master     -} FETCH_HEAD

Added dir '.vim/bundle/tpope-vim-surround'

As you can see this records a merge commit by squashing the whole history of the vim-surround repository into a single one:

1
2
1bda0bd [3 minutes ago] (HEAD, stree) Merge commit 'ca1f4da9f0b93346bba9a430c889a95f75dc0a83' as '.vim/bundle/tpope-vim-surround' [Nicola Paolucci]

ca1f4da [3 minutes ago] Squashed '.vim/bundle/tpope-vim-surround/' content from commit 02199ea [Nicola Paolucci]

If after a while you want to update the code of the plugin from the upstream repository you can just subtree pull:

1
git subtree pull --prefix .vim/bundle/tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git master --squash

This is very quick and painless but the commands are slightly lengthy and hard to remember. We can make the commands shorter by adding the sub-project as a remote.

Adding the sub-project as a remote

Adding the subtree as a remote allows us to refer to it in shorter form:

1
git remote add -f tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git

Now we can add the subtree (as before), but now we can refer to the remote in short form:

1
git subtree add --prefix .vim/bundle/tpope-vim-surround tpope-vim-surround master --squash

The command to update the sub-project at a later date becomes:

1
2
git fetch tpope-vim-surround master

git subtree pull --prefix .vim/bundle/tpope-vim-surround tpope-vim-surround master --squash

Contributing back to upstream

We can freely commit our fixes to the sub-project in our local working directory now.

When it’s time to contribute back to the upstream project we need to fork the project and add it as another remote:

1
git remote add durdn-vim-surround ssh://git@bitbucket.org/durdn/vim-surround.git

Now we can use the subtree push command like the following:

1
2
3
4
5
6
7
8
9
10
git subtree push --prefix=.vim/bundle/tpope-vim-surround/ durdn-vim-surround master



git push using:  durdn-vim-surround master

Counting objects: 5, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (3/3), done.

Writing objects: 100% (3/3), 308 bytes, done.

Total 3 (delta 2), reused 0 (delta 0)

To ssh://git@bitbucket.org/durdn/vim-surround.git

  02199ea..dcacd4b  dcacd4b21fe51c9b5824370b3b224c440b3470cb -} master

After this we’re ready and we can open a pull-request to the maintainer of the package.

Without using the subtree command

git subtree is different from the subtree merge strategy. You can still use the merge strategy even if for some reason git subtree is not available. Here is how you would go about it:

Add the dependency as a simple git remote:

1
git remote add -f tpope-vim-surround https://bitbucket.org/vim-plugins-mirror/vim-surround.git

Before reading the contents of the dependency into the repository it’s important to record a merge so that we can track the entire tree history of the plug-in up to this point:

1
git merge -s ours --no-commit tpope-vim-surround/master

Which outputs:

1
Automatic merge went well; stopped before committing as requested

We then read the content of the latest tree-object in the plugin repository into our working directory ready to be committed:

1
git read-tree --prefix=.vim/bundle/tpope-vim-surround/ -u tpope-vim-surround/master

Now we can commit (and it will be a merge commit that will preserve the history of the tree we read):

1
2
3
git ci -m"[subtree] adding tpope-vim-surround"



[stree 779b094] [subtree] adding tpope-vim-surround

When we want to update the project we can now pull using the subtree merge strategy:

1
git pull -s subtree tpope-vim-surround master

Conclusions

After having used submodule for a while I appreciate git subtree much more, lots of submodule problems are superseded and solved by subtree. As usual, with all things git, there is a learning curve to make the most of the feature.

Follow me @durdn and the awesome @AtlDevtools team for more Git rocking.

In short, I don’t trust ‘em.

So command line it is for me, which is fine because I love my command line. Except once in a while, it’s nice to be able to see a “graphical” view of your history, or to have a bit of help when preparing your commits.

Enter tig. tig is a text-mode, ncurses-based UI for git written by Jonas Fonseca. In short, it is the mutt to your Outlook, the Vim to your Emacs, the w3m to your Firefox. I use it all the time, but I haven’t seen many people using it so I thought I’d publicize it a bit. So, why is it awesome?

It’s insanely fast

Seriously. Three letters to type, and it instantly shows up. No fussing around with your mouse, or alt-tabbing to another window, or waiting for a JVM to start. It’s there at your fingertips whenever you need it. It literally loads the 50,000 commits of the JIRA codebase in a fraction of a second.

History view

The default view when launching tig is the history view. It’s basically a git log, with a bit of ASCII-art to represent the history (you can hide the graph with g). Without any argument, it displays the current branch. You can also pass it the name of a branch, or --all for all branches. You can also pass it a range of commits as git log would expect (e.g. tig master..branch). You can navigate through the commits with arrow keys or j/k. Use the / key to search (match is done on summary or author).

Pressing Enter on a commit opens that commit in a split diff view.

You can then navigate in the diff view using the j/k keys. At the beginning of the diff is the list of files that the commit touched. If you select one of them and press Enter, you will directly jump to the beginning of that file in the diff. Press q to close the diff view (and press q a second time to quit tig).

If you press t on a commit in the history view, you will open the tree view. In this view, you browse the content of your repository in exactly the state it was at that point in time (just use the arrow keys and Enter), without having to do a git checkout.

Status view

The is the killer feature in my opinion, and the one I use the most. If you start it with tig status, or if you press shift-S in the main view, you will go to the status view. This is like git status, but interactive. You can see the files that you’ve modified and staged, the files you’ve modified but haven’t staged yet, and the files that are no yet tracked by git. If you select a file that is not staged or untracked and press u, it will stage it. If you select a staged file and press u, it will unstage it. No need to remember that pesky git reset HEAD <file> command. You can also press ! on a file to revert all uncommitted changes to that file.

If you press Enter on a file, you will see a diff view of the changes you’ve made to that file (press q to close that view), making it easy to review what you want to commit or not. Even better: you can navigate in the diff with the j/k keys (line by line), or the @ key (chunk by chunk). If you then press u it will only stage the current diff chunk (a section in the diff that starts with @@). You can also stage a single line with 1. It’s all the power of git add -i, except easy to use.

When you are ready to commit,  press shift-C. and it will open your favourite editor to enter the commit message (make sure you close any open diff view first).

Blame view

Want to know who messed up your beautiful code or introduced that nasty bug? Just do tig blame <file>. It is similar to git blame or Git -> Annotate in IDEA. Just select a line and press Enter, and it will show you the last commit that touched that line.

Pager mode

Got a fancy git log command that you’ve tweaked to display just the information you want? Just pipe its output into tig to have it nicely colorised, and easy to scroll back and forth. Also, if you’re on a line that contains a commit hash, you can just press enter to view that commit. This should work for most Git commands.

How do I get it?

If you use OS X, you can easily install it via homebrew, and most Linux distributions should have a package for it. If not, you can always get the source from http://jonas.nitro.dk/tig/ and compile it yourself. I think it is even available for Cygwin, so there’s no reason not to try it!

Don’t hesitate to look at the man page, or press h inside tig: There’s a lot more you can do with it.

I speak from hard evidence as this is exactly what happens at Atlassian. The Stash team works differently than the Confluence team which works differently from the JIRA team. They all share a similar Agile process but have different approaches to branching, Continuous Integration and team organization.

Differences not withstanding, common patterns are emerging. A recurring paradigm in the industry is the use of a centralized repository with a master branch, several stable lines of development and follow a feature branch workflow where new features and bug fixes can be developed independently. This provides quick integration and allows teams to leverage the efficiency boost that comes with DVCS.

If you read this and ponder a bit you might ask: if a company is following the centralized repository workflow listed above, why would they still need forks?

There are several reasons why forking is useful and vital even within the enterprise. Before I present them let me first backtrack a bit and provide some context and definitions.

Table of Contents

1. What is a Fork?
2. Forks have won the Open Source workflow choice award
3. Do you need Forks in the Enterprise?
4. Guard core components but encourage innovation and adoption
5. Organize cross department collaboration
6. Reduce noise
7. Streamline interaction with contractors
8. Developer to developer interactions behind the firewall
9. Conclusions

What is a Fork?

In recent DVCS terminology a fork is a remote, server-side copy of a repository, distinct from the original. A clone is not a fork; a clone is a local copy of some remote repository. This differs slightly form the general definition of forks in software development but is the meaning that I will refer to in the course of this piece.

Forks have won the Open Source workflow choice award

If you have been living under a rock for the past several years let me reveal to you an indisputable fact: forks have won the Open Source process choice award. They foster participation by drastically lowering the barrier of entry and the friction of code collaboration.

Anybody can fork an open project and the original author does not incur in penalties or burdens because of it, the operation is transparent. He will potentially receive feedback or improvements in the form of pull requests but that’s it.

Do you need Forks in the Enterprise?

A fully distributed de-centralized approach is effective for loosely connected open source teams but what about enterprise teams all working in the same office with a central repository? Are forks useful in this setting?

Forks are surprisingly useful also behind the firewalls of the enterprise.

In abstract terms forks can be used within an organization to manage trust, track the maturity of codebases and facilitate cross team collaboration.

Concrete examples include:

• Guard core components but encourage innovation and adoption.
• Organize Cross department collaboration.
• Reduce noise.
• Streamline interaction with contractors.
• Developer to Developer interactions behind the firewall.

Let’s see in more details each one.

Guard core components but encourage innovation and adoption

Many enterprises have core components that are reused across the company. Often these pieces have stricter policies regarding who can make changes, stability guarantees and heavier review process.

Forks allow to keep the sanctioned code closely guarded but at the same time encourage adoption and innovation.

A non-sanctioned team or a lone developer with interest in the matter can fork the project and start contributing, without requiring supervision and without disrupting the core team’s work. Collaboration still happens as usual with pull requests and feature branches.

By keeping experiments out of the main repositories one can effectively manage and track the maturity of contributions. Crazy unstable hacks remain sealed in their own forks; Solid pieces can be merged into the core components after they are reviewed. Why forks allow this and regular clones do not? Because forks allow the tracking bit: anybody can clone locally and work on their own experiments but pushing this code to a server-side fork allows it to be traceable.

Atlassian has several core libraries shared between the various groups, like for example the Atlassian User Interface. Forks relieve the burden on the Core teams maintaining them. No spurious feature branches and a messy tree appear in their main repository anymore.

Organize cross department collaboration

An alternative but similar scenario is also common: when there is no “core” or ruling team over a certain software component but several departments maintain their own modified versions of the same piece of infrastructure. Forks allow teams to have control and guarantees on their own variations. Collaboration between departments is still easy and transparent with pull requests and the awesome integration features of git.

Reduce noise

When teams grow very large the noise of feature and bug fix branches can effectively become too much for UIs to properly display. Project history becomes so messy that is hard to understand what is going on and what is being merged where. The tools can become less effective in this situation.

Again forks allow sub-teams to collaborate in the open but keep the central repositories where integrations happen as clean as possible.

Streamline interaction with contractors

Another area where forks are valuable is in the interaction with third parties, contractors and freelances. By providing forks as the only access point for contractors to your repositories one can gain several benefits:

• Keep your main repository clean and restricted.
• Integrate third party work work after review at scheduled times.
• Retain the common git collaboration process.

Developer to developer interactions behind the firewall

Let’s not leave out one final and very important piece of the puzzle.

Developer’s personal forks! A developer might be happy to hack, improve and enhance some piece of core infrastructure but he might not want to share his early work with the rest yet. Would you want him to push some mission critical proprietary code out in the open?

In other cases he might want to maintain a slightly different approach to a problem and keep the contribution locked up until he can prove that a certain design decision will reap benefits for his team. For this scenario personal forks are great.

Personal forks allow also for the kind of distributed, uncoordinated kind of interactions that have made DVCS extremely successful in Open Source communities.

Conclusions

Forks are not the answer to all code collaboration woes but offer an effective solution to several concerns. In this piece I presented some ideas that support the statement that forks inside the enterprise help manage trust, the maturity of code bases and facilitate cross team collaboration.

Let me conclude by shouting that Stash’s new release includes first class support for forks in addition to a whole lot of other features.

As usual ping me @durdn or the awesome team @AtlDevtools for more DVCS trend spotting and git content.

(Credits: Branching model image was originally forked from nvie’s git-flow Keynote source)

It is still awesome to me even though I’ve been working with git for a while already.

I am talking about automated merges from stable/maintenance branches to master.

Real World Scenario

Let me give you an example taken from the way the Stash development team works. The Stash team maintains, at any given point in time, several stable code lines that are out in the market (let’s limit the example to 2.1, 2.2 and 2.3 releases) while at the same time they merge feature branches into master in preparation for an upcoming release branch 2.4.

Now check out the magical bit. Whenever a maintenance fix is applied to any of the stable branches it is automatically merged(!!) in cascade to the master branch. No user interaction is required (except in the rare occasion when a conflict arises). (It is understood that a fix is committed to a stable branch only after the change has been peer reviewed, all tests pass, the builds are green and the performance metrics have not been affected).

As you can see a fix applied to the 2.2 code line will automatically be first applied to the 2.3 branch and then from the 2.3 branch to master. All without user interaction. How cool is that?

How is that even possible you ask? Can I have that too please? Well of course.

Automatic Merges are the rule, not the exception

One of the areas where git excels at is merging code lines. git is so good at this that most of the times the merges just work. Except when they don’t. In the case of stable code lines, merge conflicts are even more rare given that the amount of code involved in those fixes is generally small.

How to setup Automated merges, the manual way with hooks

Let me show you how you can implement a similar workflow. First let’s write something that will accomplish it on a local machine.

The first part of the puzzle is to write a script that you can run after you’ve committed a fix on a stable branch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/bin/sh

# Automatically merge the last commit through the following branches:

# 2.1 -} 2.2 -} 2.3 -} master



CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)

LAST_COMMIT=$(git rev-list -1 HEAD)



echo Automatically merging commit $LAST_COMMIT from $CURRENT_BRANCH rippling to master



case $CURRENT_BRANCH in

2.1)

  git checkout 2.2 ＆＆ git merge $CURRENT_BRANCH

  git checkout 2.3 ＆＆ git merge 2.2

  git checkout master ＆＆ git merge 2.3

  git checkout $CURRENT_BRANCH

  ;;

2.2)

  git checkout 2.3 ＆＆ git merge 2.2

  git checkout master ＆＆ git merge 2.3

  git checkout $CURRENT_BRANCH

  ;;

2.3)

  git checkout master ＆＆ git merge 2.3

  git checkout $CURRENT_BRANCH

  ;;

esac

(Please bear with me as I leave the error conditions and the conflicts scenarios out of this example as I just want to highlight the potential for workflow automation. In fact I will show you a more robust, visual and less error prone way to accomplish the same thing if you read on.)

Save it as auto-merge.sh and make it executable with:

1
chmod 755 auto-merge.sh

So after you check out a stable branch and commit a fix – that passes all tests and has been reviewed – run the script and you will get an output like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ git commit -am"PRJ-1 bugfix on 2.1"

Automatically merging commit e42cefd7597b04a619e7f3f1b8a0bdfd3514c296 from 2.1 rippling to master

Switched to branch '2.2'

Updating 032c6a6..e42cefd

Fast-forward

a.txt | 1 +

1 file changed, 1 insertion(+)

Switched to branch '2.3'

Updating 032c6a6..e42cefd

Fast-forward

a.txt | 1 +

1 file changed, 1 insertion(+)

Switched to branch 'master'

Updating 032c6a6..e42cefd

Fast-forward

a.txt | 1 +

1 file changed, 1 insertion(+)

Switched to branch '2.1'

[2.1 e42cefd] PRJ-1 bugfix on 2.1

1 file changed, 1 insertion(+)

Good starting point. Now we want to automate this process for the whole team. If you review all the available local and remote hooks you will quickly realize that the one we need is the update one, which kicks in anytime a single HEAD(i.e. branch) is pushed on to the server.

Catering for error conditions makes writing this in bash less than optimal so I spent some time cooking the update hook in Python. It grew a bit too much for me to show you the source in its entirety here. Check it out on bitbucket at automatic-merge-hook. How to use it? Edit the branches_flow variable (line 71), upload into the hooks folder of your bare repository on your server and you can have automatic merges too:

1
branches_flow = ['2.1','2.2','2.3', 'master']

Done! Now when you create a new commit on say the 2.1 branch you will get a nice ripple merge to master.

Here is the core Python logic that will automatically merge the branch with the next in the list:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
def automatic_merge(head_before_start, latest, branches_flow):

    log.debug('Auto-merging through: %s' % branches_flow)

    reset = False

    for branch in branches_flow:

        onto = next(branch, branches_flow)

        if onto:

            exitcode = call('git checkout %s ＆＆ git merge refs/heads/%s' % (onto, branch))

            if exitcode:

                log.debug('Merge of %s onto %s failed, must reset to original state' % (branch, onto))

                call('git reset --hard %s' % (latest[onto]))

                reset = True

                break



    if reset:

        for to_reset in branches_flow:

            call('git checkout %s ＆＆ git reset --hard %s ＆＆ git checkout %s' % (

                to_reset, latest[to_reset], head_before_start))

        sys.exit('Automated merges failed, reset to original state')

Visual Way Using A Stash Plugin

What if you’re not a fan of fiddling with hooks on the command line or on your remote server via ssh? There is an easier shrink-wrapped almost one-click way to accomplish the same thing using Stash.

It is the Stash Auto Merge Plugin. The way it works is very simple, you install the plugin, you go to the Auto merge menu in the Settings tab and you fill in your automated branching flow using arrows (->):

And then the magic happens automagically. The plugin also handles the edge cases as follows:

• There are no changes to merge – Skip to the next branches
• There is an open pull request for the source and destination branches – break, the changes will be incorporated into the pull request
• There are conflicts – open a pull request

If that is not awesome, I don’t know what is!

Bonus: Sample Branching Model Of The Stash Development Team

I’ve only skimmed over it earlier because it’s not the topic of this post but If you’re very familiar with git development workflows and promise not to be overwhelmed you might be interested to see the full branching model used by Atlassian’s Stash Development team.

If the picture is unclear fear not. The way the Stash team works will be explained in detail soon on this channel. Follow @AtlDevTools or me @durdn on twitter to get it fresh out of the presses.

(In the snippets above I had to use the unicode symbol ＆ in place for the regular ampersand as WordPress is rather zealous with regards of escaping them, they would appear as “&amp;”. Please replace the ＆ with a regular ampersand. The source code on bitbucket is already fine, of course.).

The Problem

I hate it when I make a trivial change, something like:

Then a few minutes later.. *BOOM*

What the heck? How did changing the capitalization of a letter break the database migration tests?

Hmm.. maybe they’re flakey. Let’s re-run the build. *BOOM*

Arggh!

Oh wait. Of course. I know what heinous crime I’ve committed!

Little did I know, one of my co-workers had broken master at some point in the past. My crime: branching from their broken commit. A simple merge from master at this point (providing it’s green) will remedy the situation, even if it does make the history a bit uglier.

Wouldn’t it be nice if we could avoid this whole situation altogether?

The Solution

What if git warned you when you switched to a dodgy ref? Something like:

Then you could easily switch back to a better ref for your branch point, or slap the build breaker until they fix it.

I’ve built a little client-side git hook that does exactly that. It will work work for any git repository built by Bamboo or hosted by Stash, and is super easy to install.

Just run this command from the root of your local clone:

You’ll be prompted for the url and credentials of either:

• the Bamboo server that builds your repository; or
• the Stash server that hosts your repository (if you’ve set up your CI server to notify Stash of build results).

It works by registering a post-checkout hook that hits a little known REST end-point in Bamboo that provides all build results for a particular SHA.

The project is up on Bitbucket and I’m very open to pull requests.

I use Git as my DVCS of choice, and while it certainly makes managing source changes a bit more, well, manageable, it can also feel like the “wild wild west” at times with random branches and commits flying around.

Enter Git Flow. Git Flow makes managing features, releases, and bug fixes really simple and provides structure to the dev cycle. But there’s one problem: There’s no Java implementation, but let’s not get ahead of ourselves…

What Is Git Flow?

Git Flow is a branching and merging model introduced by Vincent Driessen that provides a little bit of structure to your development workflow.

At its core you have two “long running” branches:

• develop – this is where all changes are made (or eventually end up) while in active development.
• master – no development work is done on master, but instead all changes are merged in from the develop branch.

Along with the long running branches, you may have some other short-lived branches as well:

• feature/<some feature> – used to develop new features to be released at some point in the future.
• release/<some version> – used to prepare for a new planned production release.
• hotfix/<some version> – used to prepare for a new unplanned production release specifically to fix up high priority issues.

For more in-depth information about git flow and how git flow can help your business, check out our git flow guide.

JGit Flow Java Library

The JGit Flow Java Library is a general purpose library that implements git flow on top of JGit and is licensed under the Apache 2 License.

Now some people may not like the command pattern that JGit uses for most of its functionality, but I don’t mind it and to keep things familiar and cohesive, JGit Flow uses the same pattern. That being said, we’ve added some abstractions to make it easy to interact with both JGit and JGit Flow instances.

So let’s get into some code already! Say you have a project that you want to git flow enable. It doesn’t even need to be a git project, just a regular project directory will do (or even an empty directory for that matter).

That’s it. One line and you’ve set up git flow for your directory.

The getORInit method essentially checks to see if the folder is already a git project, and if not, runs git init. After that, it checks to see if the directory is already a git flow project and if not, runs git flow init with the default branch names/prefixes and finally returns a git flow instance.

To maintain compatibility with the command line git-flow, JGit Flow uses the same git config keys/values that the CLI uses. This means that you can switch between using JGit Flow and the git-flow CLI with no side effects.

If you need a more customizable approach, here are some other examples:

Once you have a JGitFlow instance, you can use the familiar JGit style command pattern on the instance to carry out git flow operations.

For instance, to start a release, you can do:

All of this is well and good, but you’re probably going to need to mix JGitFlow commands with other JGit commands on the same repository. JGitFlow makes this easy by providing a reference to the Git instance created by JGit.

For instance, if you want to start a release, update a file with the release version and commit it and then finish the release, you could do:

Why A Java Library?

After using git flow for a while and becoming a convert, I found one big wrench in the process. That wrench is called the Maven release plugin.

With its checkouts, tagging, pom rewrites and extraneous release.properties and *.backup files mostly taking place within your working tree, the Maven release plugin doesn’t play well with git flow.

Now that doesn’t mean integrating Maven releases with git flow can’t be done, but it requires some special configuration, and you never quite get it to do exactly what you want.

The best you can do is to configure your pom.xml like this:

Notice lines 11 through 13:

11. tell Maven not to push rewritten POMs to the remote repo, git flow will do that

12. tell Maven not to check out the project from the remote repo as we want to use our local changes

13. give Maven some junk tag name so it doesn’t conflict with the git flow tagging

With this in place, doing a release is as “simple” as:

So after all is said and done, you’ll have created a new release branch, Maven will have:

• messed with your POMs
• maybe committed a release.properties
• run your build and your tests 2 times
• created Java docs
• created a Maven site (that you didn’t ask for)
• and pushed your artifacts the the Maven repo

then git flow merged your release into develop and master, created a tag, and then pushed master, develop and the tag to the remote repo, which now has 1.1-SNAPSHOT as the version in both master and develop.

While this works, there are some obvious flaws here. So I decided that using git flow in a Maven release process probably warrants a specific Maven gitflow-release plugin.

“Shouldn’t be too hard”, I thought. “I’ll just grab whatever java git flow library is out there and wrap a Maven plugin around it”.

To my surprise there was only one git flow java library that was started but never completed.

Clearly there are lots of developers using Maven that could probably take advantage of a git flow plugin, and surely lots of Java projects to benefit from adding git flow support, right?

And so was born the JGit Flow Java Library.

curious about the maven plugin?

When things go wrong

The JGit Flow examples above seem pretty simple from an end user API perspective. However, JGit Flow has to do some complex things under the covers to ensure the proper state exists before carrying out the requested operation—things like making sure branches exist/don’t exist, ensuring that your local tree is not behind the remote repo, ensuring that remote branches exist, etc.

To enable you, as the integrator, to do something meaningful when good things go bad, JGit Flow provides a fine-grained set of exceptions that all inherit from a base exception. This allows you to catch very specific exceptions or just catch a single exception depending on what your code needs to do.

Taking the start release example from above, it could be re-written to catch specific exceptions

Let the flow begin!

Armed with this new library, hopes are that more and more Java projects will use it to integrate git flow into their apps. But don’t stop there! I believe this is just the tip of the iceberg and as JGit Flow gets adopted by more and more projects, I hope to see it evolve beyond the base git flow workflow.

Anything is possible and everything is up for grabs. If you find a bug or have a feature request, by all means add it to the JGitFlow issue tracker.

If you’d like to contribute, please fork the JGit Flow project on Bitbucket. When your super cool new feature is ready, just issue a pull request to the develop branch of the main project.

For more information and usage instructions, see the JGit Flow Wiki on Bitbucket.

Enjoy!

See my follow up blog post about the maven-jgitflow-plugin.

The topic of this post is to confirm what my colleague Charles O’Farrell said very well in the ever popular “Why Git?” article some time ago:

[...] Git is actually the safest of all the DVCS options. As we saw above,
Git never actually lets you change anything, it just creates new objects.

[...] Git actually keeps track of every change you make, storing them in
the reflog. Because every commit is unique and immutable, all the reflog has
to do is store a reference to them.

This means you’re safe from harm and your code is always preserved, but there are situations where you might need some conjuring to get it back.

Let me give you a few real world examples of how to recover from trouble, going from simple to advanced:

Table Of Contents

1. How To Undo A (Soft) Reset And Recover A Deleted File
2. If You Lose A Commit During An Interactive Rebase
3. How To Undo reset –hard If You Only Staged Your Changes

How To Undo A (Soft) Reset And Recover A Deleted File

If you delete a file with git rm and immediately after you git reset your working directory (which is called a soft reset), you will find yourself with a missing file and a dirty working directory like the following:

1
2
3
4
5
6
7
# On branch master

# Changes not staged for commit:

#   (use "git add/rm file..." to update what will be committed)

#   (use "git checkout -- file..." to discard changes in working directory)

#

#   deleted:    test.txt

#

There are a couple of simple ways to go about restoring the file. One is to use a git reset –hard which will recreate the missing files but it will delete any local modifications you might have in your working directory.

The second is to just re-check out the file yourself with git checkout test.txt.

In a slightly different scenario, if the file was removed in an earlier commit, you can recover it by noting down the exact commit where the file was deleted and use the reference to pick the commit immediately before:

1
git checkout commit_id~ -- test.txt

Where the ~ (tilde) sign means the one before this one.

If You Lose A Commit During An Interactive Rebase

Interactive rebase is one of the most useful tools in the git arsenal; It allows to edit, squash and delete commits interactively.

Personally, I love to clean and streamline commits before sharing them with others. It’s nice when each commit is an understandable and logical chunk of work. In the heat of development, or in a local private branch, I might commit furiously for a bit, then when I am satisfied I spend time cleaning up the history.

But what if while in the interactive rebase session you accidentally remove a commit line without realizing it and save?

This situation is slightly annoying cause you have just removed a commit from your history!

If you created a throwaway branch before starting the rebase – which you should always do – you’re fine and you can restart the process all over. But what if you didn’t?

No worries, git has got you covered. Nothing is ever lost. This is a good opportunity to use the reflog, which is an automatic mechanism recording where the tip of all branches has been for the past 30 days.

Say at the start of the rebase I had 4 commits like the following:

1
2
3
4
5
6
$ git log

cfdf880 [2 seconds ago] (HEAD, master) wrote fifth change

ab446e6 [34 seconds ago] changed one line

6e1a130 [25 minutes ago] third change

6566977 [26 minutes ago] second change

5feeb33 [26 minutes ago] initial commit

I rebase the last 4 commits and accidentally remove one line:

1
$ git rebase -i HEAD~4

Editor opens on:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
pick 6566977 second change

pick 6e1a130 third change

pick ab446e6 changed one line

pick cfdf880 wrote fifth change



# Rebase 5feeb33..cfdf880 onto 5feeb33

#

# Commands:

#  p, pick = use commit

#  r, reword = use commit, but edit the commit message

#  e, edit = use commit, but stop for amending

#  s, squash = use commit, but meld into previous commit

#  f, fixup = like "squash", but discard this commit's log message

#  x, exec = run command (the rest of the line) using shell

#

# These lines can be re-ordered; they are executed from top to bottom.

#

# If you remove a line here THAT COMMIT WILL BE LOST.

#

# However, if you remove everything, the rebase will be aborted.

#

# Note that empty commits are commented out

I removed one commit line (pick ab446e6 changed one line), I saved and got this:

1
Successfully rebased and updated refs/heads/master.

Now, if I check the list of commits I see I lost one:

1
2
3
4
5
$ git log

3dd6845 [6 minutes ago] (HEAD, master) wrote fifth change

6e1a130 [32 minutes ago] third change

6566977 [32 minutes ago] second change

5feeb33 [32 minutes ago] initial commit

But the reflog has our lost commit:

1
2
3
4
5
6
7
$ git reflog

3dd6845 HEAD@{0}: rebase -i (finish): returning to refs/heads/master

3dd6845 HEAD@{1}: rebase -i (pick): wrote fifth change

6e1a130 HEAD@{2}: checkout: moving from master to 6e1a130

cfdf880 HEAD@{3}: commit: wrote fifth change

----}} ab446e6 HEAD@{4}: commit: changed one line

[...]

Here ab446e6 is the sha-1 for the commit we lost. Armed with this id we can just cherry-pick it back in the code:

1
2
3
$ git cherry-pick ab446e6

[master 032c6a6] changed one line

1 file changed, 1 insertion(+), 1 deletion(-)

And as you can see the commit is now back in our history:

1
2
3
4
5
6
$ git log

032c6a6 [12 minutes ago] (HEAD, master) changed one line

3dd6845 [12 minutes ago] wrote fifth change

6e1a130 [37 minutes ago] third change

6566977 [38 minutes ago] second change

5feeb33 [38 minutes ago] initial commit

How To Undo reset –hard If You Only Staged Your Changes

I’ll save you the speech to commit frequently – whether in a private short lived branch or a shared one – to protect you from heaps of problems.

But let’s say you didn’t commit this time and you find yourself in the following scenario: you’ve done some work, haven’t committed it yet but have git added it to the staging area.

Then tragedy strikes: you type git reset –hard and immediately realize that you’ve zeroed your local changes(!!) and brought back (in it’s entirety) the previous commit.

This is a tough situation to be in. How do you recover your work? Is it even possible? The answer is yes! The solution involves some low level searching and can be approached in a couple of ways.

Since git creates new objects in the .git/objects folder as soon as something is added to the staging area we can look there for the most recently created objects:

In OSX (and BSD systems) you can do:

1
find .git/objects/ -type f | xargs stat -f "%Sm %N" | sort | tail -5

On Linux(with GNU tools) this should work:

1
find .git/objects/ -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort | tail -5

With tail -n you can cut to the last n results if your repository is very big:

The result should be something like:

1
2
3
4
5
Apr  2 12:26:33 2013 .git/objects//pack/pack-4cf657357973915f6fb6f90a41f69a88c0b08bb7.pack

Apr  2 12:29:56 2013 .git/objects//3d/d6845a69cd59dac0851e1ae0bd69dde950c46b

Apr  2 12:29:56 2013 .git/objects//68/c983f0cefb4bee2f753516ab6352ee2f2b7d29

Apr  2 12:35:23 2013 .git/objects//03/2c6a653cc00c302020293e020d8d68326b112e

Apr  2 15:34:55 2013 .git/objects//df/485610889e98ff773b4440a032ea2ede1338b9

In my case my sample file was lost exactly around 15:34 so I can inspect and retrieve it with:

1
git show df485610889e98ff773b4440a032ea2ede1338b9

Remember that the folder is part of the sha-1 reference, so remove the / slash to compute the correct id.

Note that git has specific low level command to explore dangling objects and to verify their connectivity: git fsck. For an alternative solution to this problem using fsck, check out this very cool answer on Stack Overflow.

Conclusions

There are many more examples and scenarios to cover on this topic so I might come back later with more interesting cases.

One thing to take away from this should be the feeling that it’s very very hard to screw up and lose your data when using git. That’s all for now!

Follow the awesome @AtlDevtools team or me @durdn for more Git rocking.