Git Blame

Updates to "git merge" in upcoming 1.7.10

2012-02-25T00:52:00.000-08:00

The previous entry about a potential incompatible update to the behavior of the git merge command may have been unnecessarily alarming for casual readers and it deserves a bit of clarification. The update is designed to primarily target the interactive use, and not to negatively affect typical uses of the command in scripts.

Let's look at a few example use cases of the git merge command in end user scripts, and how well these scripts will work with the upcoming version.

(1) Your script may be designed to be run by your end users (e.g. it may be an implementation of the git my-merge command) to implement extra checks that are specific to your project or corporate environment before invoking the actual merge, perhaps like this:

#!/bin/sh
# Make sure you do not have uncommitted changes

test 0 = $( (

git diff --name-only

git diff --cached --name-only

) | wc -l ) &&

# Perhaps other project specific checks here...

# ... and finally

git merge "$@"

There is no need to update the above script in preparation for 1.7.10. The script is directly facing the end user, and with 1.7.10, the user can take advantage of the updated behavior that automatically opens the editor to make it easy for him to explain the merge. Also note that in a user-facing script like this, the underlying git merge command is typically passed the command line arguments given by the end user to the script (notice the use of "$@" in the above example). The end user can pass the --no-edit option to the script if he wants to silently conclude the merge.

(2) You may be responsible for the overall QA after receiving various topics from your contributors and co-workers, and may be using a script like the following to regularly merge many topic branches into an integration branch for testing:

#!/bin/sh

if test -f topics

then

# Prepare topics to be integrated just once

list-topics >topics &&

# And checkout the tip of the master to integrate them all

git checkout -B test master || exit

fi
while read topic

git merge "$topic" || exit

done <topics

# All done

rm -f topics

In such a case, you do not have to explain each and every such merge to rebuild the test integration branch, and you do not want the change in 1.7.10 to affect the above script.

And we didn't want to break such a script, either. Notice that the git merge command in this example is not facing the end user—its standard input is connected to the file that lists the topics, from which the surrounding while loop is reading, and not to the user's terminal. Because of this, the command will not automatically open an editor to ask your user to explain the merge, and there is no need to worry about the compatibility.

(3) Your project may run integration test every time its members update the master branch of the central shared repository, and as a part of the integration test, it may merge the updated master to the test repository that has mock data for testing. Such a script may look like this:

#!/bin/sh

# Called from the post-update hook of the central repository

unset GIT_DIR

cd /srv/test.repo &&

git pull /srv/central.repo master || {

logger "The updated master does not cleanly merge -- no automated test done"

git reset --hard exit 1}

make test || {

logger "The updated master does not pass test"

git reset --hard

exit 1

}

If you run such a script interactively, the git pull command will invoke the underlying git merge command interactively, which in turn will open the editor and ask you to explain the merge. But if it is run from a post-update hook, the standard input is not likely to be connected to any interactive terminal (this is also true if such an automated merge is done as part of a continuous integration testing). Hence, in practice, 1.7.10 will not change the behaviour of the git merge command used in such a script, and there is no need to worry about the compatibility.

(4) If your script uses git merge, is run interactively by the end user, and it keeps the standard input to the git merge command connected to the terminal, but for whatever reason you do not need your users to explain the merge, you do need to worry about the compatibility. You could add the --no-edit option to the git merge command invocations in your script as necessary, but there is a quicker way:

#!/bin/sh

GIT_MERGE_AUTOEDIT=no
export GIT_MERGE_AUTOEDIT

# whatever your script did originally

...

git merge foo

git merge bar

...

You can set GIT_MERGE_AUTOEDIT environment variable to no at the beginning of your script, and all the git merge command invocations will work as if they were given the --no-edit option.

Hopefully it is now clearer that there is not much to fear in the update to the git merge command planned in the upcoming 1.7.10 release.

Happy Gitting!

Anticipating Git 1.7.10

2012-02-23T20:16:00.001-08:00

According to the Git Calendar, we still have a few more weeks until the feature freeze for the next release Git 1.7.10, in which there will be one backward-incompatible improvement that potentially can bite people who used the git merge command in their scripts.

[Update: the above phrasing is a bit too alarming; here is a clarification you may want to read after finishing this entry]

Following the advice given in an article at LWN.net by Jake Edge:

Most free software projects discuss planned changes well in advance of their implementation and give users lots of opportunities to try out early versions. But engaging the project is best done with well-reasoned, specific descriptions of problems, missing features, and so on—not endless streams of "Project XYZ sucks!" messages to mailing lists or comment threads.

let's describe what we have decided, why, and how users can use the upcoming release in their work.

Traditionally, when the git merge command attempted to merge two or more histories, it prepared a canned commit log message based on what is being merged, and recorded the result in a merge commit without any user intervention, if the automatic merge logic successfully computed the resulting tree without conflict. When the automatic logic did not manage to come up with a clean merge result, it gave up, leaving the conflicted state in the index and in the working tree files, and asked the user to resolve them and run the git commit command to record the outcome.

Most merges do cleanly resolve, and this behavior resulted in people making their merges too easily and lightly, even when the reason why the merge was made in the first place should be explained. Nobody explained why the merge was made in a merge commit, because in order to do so after git merge already made the commit, they have to go back and run git commit --amend to do so.

Recently in a discussion on the Git mailing list, Linus admitted (and I agreed) that this was one of the design mistakes we made early in the history of Git. And in 1.7.10 and later, the git merge command that is run in an interactive session (i.e. both its standard input and its standard output connected to a terminal) will open an editor before creating a commit to record the merge result, to give the user a chance to explain the merge, just like the git commit command the user runs after resolving a conflicted merge already does.

There are two recommendations we give to end users to adjust to this behaviour change.

(1) When using git merge interactively, there are two cases:

Merging updated upstream into your work-in-progress topic without having a good reason is generally considered a bad practice. Such a merge in the wrong direction should be done only when it is absolutely necessary, e.g. your work-in-progress needs to take advantage of recent advancement made on the upstream. Otherwise, your topic branch will stop being about any particular topic but just a garbage heap that absorbs commits from many sources, both from you to work on a specific goal, and also from the upstream that contains work by others made for random other unfocused purposes. So if you are merging from upstream into your topic branch, you can use the command without any option to take advantage of the new behavior, let it open an editor for you, and justify the merge. You no longer have to amend the commit after you made the merge to explain it.
When you are merging your topic into your own integration (or testing) branch, on the other hand, such a merge is often self-explanatory. The topic is fully cooked, and ready to be pushed out, and that is the reason why you are merging. In such a case, it will be sufficient to run the command with the --no-edit option and accept the canned commit log message without editing.

(2) If you have a script that runs git merge, and you left its standard input and output kept connected to the terminal, it may start to ask whoever is running the script to edit and explain the merge. This may or may not be a desirable thing depending on the reason why the script is making a merge. Often, scripts are used to merge many branches into temporary testing branch in bulk, and want to run unattended if they merge cleanly. In such a case, you do not want to change the old behavior of the script. You do not have to add --no-edit to all your invocations of the git merge command to update such a script. Instead, you can export MERGE_AUTOEDIT=no at the beginning of your script, and the git merge command will silently make commits when there is no conflict.

Please try out the updated git merge before 1.7.10 release is tagged from the master branch, and adjust your scripts and work habit as needed. Also, please suggest improvements to our development mailing list, git@vger.kernel.org, regarding:

the documentation,
error, advice and general messages output from the program,
logic to detect the interactiveness.

One thing that is not open to discussion is that the merge command will launch the editor when interactive by default. This will not change, because, as Linus says, the default matters, and the behavior we had so far was a bad default.

Finally, spread the word so that other users can start adjusting their use of the command early.

[EDIT]

Linus has this to say in his re-share:

This change hopefully makes people write merge messages to explain their merges, and maybe even decide not to merge at all when it's not necessary.

I've been using that git feature for the last few weeks now, and it has resulted in my merges from submaintainers having various notes in them (well, at least if the submainter gave me any). So I'm trying to lead by example.

But if you don't like explaining your merges, this might be annoying. Of course, if you don't explain your merges, you are annoying, so it all evens out in the end. "Karmic balance", so to say.

Note that this new feature will not be in 1.7.9.3 release.

Git 1.7.9.1

2012-02-14T11:35:00.000-08:00

I just tagged and pushed out the first maintenance release to the 1.7.9 released earlier.

It also contains fixes to older bugs and misfeatures, but this release, from my point of view, is primarily to fix user experience kinks in new features introduced in the 1.7.9 release, namely:

Typo in "git branch --edit-description my-tpoic" was not diagnosed.
"git merge --no-edit $tag" failed to honor the --no-edit option.
"git merge --ff-only $tag" failed because it cannot record the required mergetag without creating a merge, but this is so common operation for branch that is used only to follow the upstream, so it was changed to allow fast-forwarding without recording the mergetag.
When asking for a tag to be pulled, "request-pull" did not show the name of the tag prefixed with "tags/", which would have helped older clients.

Hopefully with this maintenance release, there no longer is a reason (or excuse) for users to stay at a version older than 1.7.9 release.

The Double Helix by James D. Watson

2012-02-12T02:16:00.000-08:00

This is a classic.

The scientific race between James D. Watson (the author) and Francis Crick at Cambridge vs Linus Pauling at Cal Tech to solve the structure of DNA is vividly described. The book was written not as an objective science history, but as a record of what the author thought, felt and experienced in the midst of that race, and begins its preface with this:

Here I relate my version of how the structure of DNA was discovered.

Because the author was in the centre of this adventure, there is no other way for him to tell his story than as a personal recollection. Nobody can be an objective third-person observer and reporter of important events around himself that changed the world. And because the book is written from that perspective, the author's adrenaline rush during the fierce competition feels even more real to the readers.

After examining the draft of a paper sent to Peter Pauling (Linus's son, who was then at Cambridge) from their competitor, Linus Pauling, in which Linus described his solution to the puzzle of DNA structure, the Cambridge group concludes that Linus's solution cannot possibly be correct, and congratulates that they haven't lost their race yet. Then:

On our way to Soho for supper I returned to the problem of Linus, emphasizing that smiling too long over his mistake might be fatal. The position would be far safer if Pauling had been merely wrong instead of looking like a fool. Soon, if not already, he would be at it day and night.

When I read this passage, this somehow reminded me of the excitement and tense sense of competition I felt during the early days of Git development. Of course, I was competing with the other  Linus (Torvalds, who is known for his Linux operating system, originally wrote Git and was actively developing it with many other brilliant software developers in collaboration) back then.

When there was an issue to solve, everybody rushed to present his own bright idea, and it was a race to show a clean, clever and useful solution to improve the system. When other guys went in a wrong direction and wasted their time, you had more time to polish your work and beat them to your better solution.

I do not think that the similarity between the way how the scientific race and the open source race work stops there. Even though the participants all want the glory of being the first to reach the right solution, at the highest level, everybody is working collectively towards the same goal, be it the advancement of their scientific field, or the improved user experience of their software. The subtle balance between competition and collaboration is the same in both endeavours.

The book depicts Maurice Wilkins of King's College, the other scientist who shared the Nobel with Watson and Crick, as somebody who had access to good X-ray crystallography data that eventually helped the discovery by Watson and Crick, but didn't solve the puzzle himself even though he was an expert in the field of DNA research.

Given that the way Watson's book is written from his own perspective, I suspect that the aptly titled book The Third Man of the Double Helix : An Autobiography by Wilkins himself is a must-read for anybody who reads this book to see both sides of the coin. It is already on my "To Read" list.

The book was a very satisfying read and I really enjoyed it. It was given by a happy Git user Ben (thanks!) as a present to me the other day, picked from my Amazon wishlist.

The Vertical Farm: Feeding the World in the 21st Century, by Dickson Despommier

2012-01-31T00:31:00.000-08:00

This book was given to me by a happy Git user (Thanks, Michael!) as a gift, picked from my wish list at Amazon.

The book was fascinating and somewhat disappointing at the same time. Perhaps disappointing is not a good word, and I should say somewhat frustrating.

I found that the first half of the main argument of the book very convincing: conventional soil-based agriculture that relies heavily on agro-chemicals (herbicide, insecticide and fertilizer) and causes terrible damage to the environment due to run-off water, is not simply sustainable. The other half of the premise, however, looked rather sketchy, day-dreaming and hand-wavey: multi-story high-tech greenhouses can be built inside city boundaries, and using the state of the art scientific method (e.g. supplying the plants with purified water spiked with necessary nutrients and growing them aeroponically; lighting with OLED that emits only the wavelengths absorbed by chlorophyll; burning plant waste with plasma-arc gassification) sufficient amount of high-quality food can be produced, freeing vast farmland that we have been wasting back to the nature, while reclaiming as much water and energy as possible.

I am reasonably sure that the author has some solid numbers to convince policy people about the benefit and feasibility of his dream, but I found them solely lacking in the book. Here is only one passage with some numbers I remember:

One strawberry farmer who wishes to remain anonymous decided to replace his now destroyed 30 acre farm by constructing a high-tech greenhouse with a 1 acre footprint. Using hydrostackers, he was able to produce the equivalent of 29 acres’ worth of fruit, with year-round production. He elected to return the rest of his farm to its natural setting by simply leaving it alone. Within two years, the understory had returned and the biodiversity of the land improved dramatically.

That is a nice story, but isn't strawberry rather special? It is sufficiently high-value kind of produce that would pay for a farmer to invest in high-tech. If the book had some back-of-the-envelope attempt at some math, like, "To supply a small city of 20,000 people, we would need X tons of grains, Y tons of cabbage, Z tons of ..., and it would need to treat W gallons of black water a year. A vertical farm consisting of N buildings (M stories high) of L sqft footprint would be sufficient for all of the above. The net intake of water and energy, offsetting with the water and energy reclaimed, would be such and such", even if the numbers were qualified with "this is just a back-of-the envelope with my own optimistic assumption that in 5 years technology will sufficiently advance to solve this and that problems", it would have been able to convert me, a mere interested reader, into a believer, somebody who shares the dream.

In the chapter on funding, after discussing that not everything is money and governments will be the likely sources of sponsorship, Dickson alludes to commercial sponsorship possibilities, and there I found this passage:

Google would be my first choice. This giant has an altruistic streak a mile wide. Google could afford to promote the concept with significant financial aid.

This is what made me the most frustrated about the book, because that was exactly what I was feeling while reading earlier parts of the book. Google is a green company; it for example uses many solar photo-voltaic panels (1.6MW), supplying about 30% peak electricity in the four main buildings. Significant number of employees take company shuttles, instead of driving their own cars to work.

If I were to bump into Larry or Sergey at the campus tomorrow and have a chance to chat with them (not likely to happen, as my workplace is on the other side of the campus that is a mile wide), I would love to suggest them that we build an experimental vertical farm near the Mountain View campus, supplying all 20+ cafes and micro-kitchens. Even though the book managed to convert me enough to be very much attracted to the idea, it didn't give me enough to come up with an intelligently-sounding answer to a very basic question, such as "OK, we will go ahead with Dickson's idea—how much space and how much workers do you want?"

Nevertheless, the author's enthusiasm jumps at you from every page, while deep knowledge and experience in his professional area (the author is a microbiologist, ecologist and Professor of Public Health in Environmental Health Sciences at Columbia University) showing through between lines. It was a very pleasant read, and I would highly recommend it to anybody.

Git 1.7.9

2012-01-27T13:29:00.000-08:00

The latest feature release Git 1.7.9 is now available at the usual places.

The release tarballs are found at:

    http://code.google.com/p/git-core/downloads/list

Also the following public repositories all have a copy of the v1.7.9 tag and the master branch that the tag points at:

url = git://repo.or.cz/alt-git.git
url = https://code.google.com/p/git-core/
url = git://git.sourceforge.jp/gitroot/git-core/git.git
url = git://git-core.git.sourceforge.net/gitroot/git-core/git-core
url = https://github.com/gitster/git

As already reported in various places (including lwn.net and Linus Torvalds ), the most notable feature in this release is unarguably the improved support for the pull-request based workflow, in which participants can exchange their histories more securely, even over a potentially untrustable hosting service. This is truly an end-to-end support, starting from the inception of a topic branch with its own description, use of the branch description and the message in a signed tag in a pull-request message, use of the message in a signed tag in the resulting merge commit and verification of the signature when reviewing the resulting history (see this for a tutorial).

There also are many other updates that hopefully make working with Git a more delightful experience. Highlights are:

"Credential helper" support, that allows integration of HTTP authentication with platform-native key-chain implementations;
New progress eye-candy output for a couple of lengthy operations e.g. fsck;
I18N framework enabled for both C and scripted Porcelains;
Improvement in prompted input from an interactive terminal (most notably, reading the username for HTTP authentication);
You can now run "git checkout -B <current branch> <elsewhere>" to correct the tip of the checked out branch;
As another step to support large files better, "git add" stores large files directly into a new packfile without having to hold everything in-core at once; and
"Signed commit" support, that lets a commit object record a GPG signature on it.

Have fun.

Preparing for the next cycle

2012-01-24T22:45:00.000-08:00

While hunting for last-minute embarrasing bugs before the final release of upcoming Git version 1.7.9, those in the Git development community have been busy preparing their own topics to scratch various itches of their own. As we have been in the pre-release feature freeze for about three weeks by now, there have been quite a few such topics, waiting to become part of the release after 1.7.9, being tested in the 'next' branch. I counted 48 patches spanning 23 topics.

Kind	Topics	Patches
Fixes	9	19
UI, Workflows & Features	7	10
Performance	2	6
Internal Implementation	5	13

I've been re-reading them to check their sanity for the past several days, and they all look more or less ready to go. I expect that the next cycle after version 1.7.9 is released will be a short one, just like this cycle leading to 1.7.9 has been.

One of the potentially contentious topic is a change to the git merge command. In the versions up to the upcoming 1.7.9, when the changes from histories being consolidated merge cleanly without conflict, the command records the result without any extra user interaction, using a commit log message that is auto-generated based on the name and contents of the branch being merged.

Linus recently admitted that this is one of the regrettable UI mistakes we made in the early days of Git. This behaviour led too many people to blindly make merge commits that merge into their topic branches, whose purpose is to develop a feature and fix in isolation, of the upstream branch, whose purpose is to integrate assorted topics and move the overall project forward, without justifying why such a merge in a wrong direction was necessary to advance their topics. Of course, git commit --amend command can be used to update the merge commit log message after git merge is done, and the recent git merge command can be given the --edit option to open an editor, but in practice, not many people bother to perform such an extra step only to strive for long term health of the project history.

Just like the git commit command encourages users to explain and justify their non-merge commits by opening an editor and have them describe the commit, the git merge command in the version of Git after the upcoming version 1.7.9, when used interactively, will open an editor, so that the user can further edit the auto-generated merge commit log message and explain why the merge was made.

Oh, I've also been working on a topic that will get ready far in the future, but that will be another article.

Git 1.7.7.6, 1.7.8.4 and 1.7.9-rc2

2012-01-18T19:07:00.000-08:00

Three releases are out on the same day, which is not very unusual for me.

The two maintenance releases are primarily to correct an obscure bug introduced in 1.7.7.1 when you use wildcard pathspec to run "git diff $commit" command.

On the 'master' front, the second release candidate seems to be quite solid and includes the how-to document I earlier posted a preview for.

Hopefully we can have an uneventful final 1.7.9 release by the end of this month.

Using signed tag in pull requests

2012-01-17T14:58:00.000-08:00

[edit: This is where the backstory behind this new feature begins.]
(This is a preview of an howto article meant to be distributed as part of upcoming v1.7.9 release of Git).
A typical distributed workflow using Git is for a contributor to fork a project, build on it, publish the result to her public repository, and ask the "upstream" person (often the owner of the project where she forked from) to pull from her public repository. Requesting such a "pull" is made easy by the git request-pull command.

Earlier, a typical pull request may have started like this:

 The following changes since commit 406da78032179...:

   Froboz 3.2 (2011-09-30 14:20:57 -0700)

 are available in the git repository at:

   example.com:/git/froboz.git for-xyzzy

followed by a shortlog of the changes and a diffstat.

The request was for a branch name (e.g. for-xyzzy) in the public repository of the contributor, and even though it stated where the contributor forked her work from, the message did not say anything about the commit to expect at the tip of the for-xyzzy branch. If the site that hosts the public repository of the contributor cannot be fully trusted, it was unnecessarily hard to make sure what was pulled by the integrator was genuinely what the contributor had produced for the project. Also there was no easy way for third-party auditors to later verify the resulting history.

Starting from Git release v1.7.9, a contributor can add a signed tag to the commit at the tip of the history and ask the integrator to pull that signed tag. When the integrator runs git pull, the signed tag is automatically verified to assure that the history is not tampered with. In addition, the resulting merge commit records the content of the signed tag, so that other people can verify that the branch merged by the integrator was signed by the contributor, without fetching the signed tag used to validate the pull request separately and keeping it in the refs namespace.

This document describes the workflow between the contributor and the integrator, using Git v1.7.9 or later.

A contributor or a lieutenant

After preparing her work to be pulled, the contributor uses git tag -s to create a signed tag [1]:

 $ git checkout work
 $ ... "git pull" from sublieutenants, "git commit" your own work ...
 $ git tag -s -m "Completed frotz feature" frotz-for-xyzzy work

and pushes the tag out to her publishing repository [2]. There is no need to push the work branch or anything else:

 $ git push example.com:/git/froboz.git/ +frotz-for-xyzzy

Then the contributor prepares a message to request a "pull":

 $ git request-pull v3.2 example.com:/git/froboz.git/ frotz-for-xyzzy >msg.txt

The arguments are:

the version of the integrator’s commit the contributor based her work on;
the URL of the repository, to which the contributor has pushed what she wants to get pulled; and
the name of the tag the contributor wants to get pulled (earlier, she could write only a branch name here).

The resulting msg.txt file begins like so:

 The following changes since commit 406da78032179...:

   Froboz 3.2 (2011-09-30 14:20:57 -0700)

 are available in the git repository at:

   example.com:/git/froboz.git tags/frotz-for-xyzzy

 for you to fetch changes up to 703f05ad5835c...:

   Add tests and documentation for frotz (2011-12-02 10:02:52 -0800)

 -----------------------------------------------
 Completed frotz feature
 -----------------------------------------------

followed by a shortlog of the changes and a diffstat. Comparing this with the earlier illustration of the output from the traditional git request-pull command, the reader should notice that:

The tip commit to expect is shown to the integrator; and
The signed tag message is shown prominently between the dashed lines before the shortlog.

The latter is why the contributor would want to justify why pulling her work is worthwhile when creating the signed tag. The contributor then opens her favorite MUA, reads msg.txt, edits and sends it to her upstream integrator.

Integrator

After receiving such a pull request message, the integrator fetches and integrates the tag named in the request, with:

 $ git pull example.com:/git/froboz.git/ tags/frotz-for-xyzzy

This operation will always open an editor to allow the integrator to fine tune the commit log message when merging a signed tag. Also, pulling a signed tag will always create a merge commit even when the integrator does not have any new commit since the contributor’s work forked (i.e. fast forward), so that the integrator can properly explain what the merge is about and why it was made.

In the editor, the integrator will see something like this:

 Merge tag 'frotz-for-xyzzy' of example.com:/git/froboz.git/

 Completed frotz feature
 # gpg: Signature made Fri 02 Dec 2011 10:03:01 AM PST using RSA key ID 96AFE6CB
 # gpg: Good signature from "Con Tributor <nitfol@example.com>"

provided if the signature in the signed tag verifies correctly. Notice that the message recorded in the signed tag "Completed frotz feature" appears here, and again that is why it is important for the contributor to explain her work well when creating the signed tag.

As usual, the lines commented with # are stripped out. The resulting commit records the signed tag used for this validation in a hidden field so that it can later be used by others to audit the history. There is no need for the integrator to keep a separate copy of the tag in his repository (i.e. git tag -l won’t list frotz-for-xyzzy tag in the above example), and there is no need to publish the tag to his public repository, either.

After the integrator responds to the pull request and her work becomes part of the permanent history, the contributor can remove the tag from the publishing repository, if she chooses, in order to keep the tag namespace of her public repository clean, with:

 $ git push example.com:/git/froboz.git :frotz-for-xyzzy

Auditors

The --show-signature option can be given to git log or git show and shows the verification status of the embedded signed tag in merge commits created when the integrator responded to a pull request of a signed tag.

A typical output from git show --show-signature may look like this:

 $ git show --show-signature
 commit 02306ef6a3498a39118aef9df7975bdb50091585
 merged tag 'frotz-for-xyzzy'
 gpg: Signature made Fri 06 Jan 2012 12:41:49 PM PST using RSA key ID 96AFE6CB
 gpg: Good signature from "Con Tributor <nitfol@example.com>"
 Merge: 406da78 703f05a
 Author: Inte Grator <xyzzy@example.com>
 Date:   Tue Jan 17 13:49:41 2012 -0800

     Merge tag 'frotz-for-xyzzy' of example.com:/git/froboz.git/

     Completed frotz feature

     * tag 'frotz-for-xyzzy' (100 commits)
       Add tests and documentation for frotz
       ...

There is no need to fetch and keep a signed tag like frotz-for-xyzzy that is used for frequent "pull" exchange in the long term just for such auditing purposes, as the signature is recorded as part of the merge commit.

Footnotes

1 This example uses the -m option to create a signed tag with just a single liner message, but this is for illustration purposes only. It is advisable to compose a well-written explanation of what the topic does to justify why it is worthwhile for the integrator to pull it, as this message will eventually become part of the final history after the integrator responds to the pull request.

2 The example uses plus at the beginning of +frotz-for-xyzzy to allow forcing the update of a tag, as the same contributor may want to reuse a signed tag with the same name after the previous pull request has already been responded to.

Git 1.7.9-rc1 is out

2012-01-13T15:31:00.000-08:00

This is supposed to be feature complete for the upcoming release, and we are now in the pre-release feature freeze from now on (see tinyurl.com/gitcal for the release schedule calendar).

Minor and obvious bug fixes and documentation updates are appreciated, but the primary focus for the coming few weeks will be on regression hunting and fixes.

I also suspect that the documentation to the new features may still be sketchy, and we may want to fix that by the final release. The "pulling signed tag" topic, for example, may want a new article in Documentation/howto/ area, perhaps lifting some text from here. I may try to do an initial draft over the long weekend.

Git 1.7.9-rc0 and 1.7.8.3

2012-01-06T13:47:00.000-08:00

I just tagged Git 1.7.9-rc0, which is the zeroth release candidate in the sense that it is supposed to be almost feature complete and marks the beginning of feature freeze.

We have achieved quite a lot during this cycle. All the new things that we initially listed that we want to have are in it:

Better and more auditable communication in pull based workflow by supporting electronically signed pull requests that records more meaningful branch description;
More pleasant end-user experience by providing credential helper API to allow platform native keychain implementations to supply authentication material during "git push" and "git pull";
i18n of messages out of the end-user facing programs; and
Better large-contents support.

Even though we didn't originally plan to, this also includes support for signed commits as a part of the first topic in the above list.

I also tagged 1.7.8.3 from the tip of the maintenance track. The fixes in this maintenance release are all included in 1.7.9-rc0 release.

Adding finishing touches to the "pulling signed tags" topic

2012-01-04T16:34:00.000-08:00

[edit: a how-to article will be included in the 1.7.9 release; here is a preview]

One of the more important topics scheduled for the upcoming Git 1.7.9 release is for a contributor to ask the integrator to pull a signed tag and for the integrator to merge such a signed tag to the history. This has already been used in the field by Linus but there wasn't a good automated way to view and verify the signature after such a merge is made.

Until now.

I started working on a follow-up topic to teach "git log --show-signature" about these merge commits, and have a working code in today's proposed updates branch ('pu'). Hopefully we can have this feature, possibly with some clean-ups, in the same 1.7.9 release, if we get enough help from the contributors in the development community.

Knock wood...

Git 1.7.8.2

2011-12-28T16:12:00.000-08:00

The latest maintenance release Git 1.7.8.2 is available.

Again, please note that this is not a release with new features (upcoming 1.7.9 is expected to be released late January next year to include the pulling of signed tags and other goodies).

Please upgrade to get the accumulated bugfixes since 1.7.8.1

Enjoy!

Hopefully the last push-out of the year

2011-12-27T15:50:00.000-08:00

By now I know a bit better than taking the lack of serious regression reports during the holiday weekend as a sign of perfection of the upcoming release, but I will tag 1.7.9-rc0 soonish anyway. As far as I can see, the tip of 'master' branch is now feature complete for 1.7.9, modulo possible bugs and regressions, of course.

We've done quite a lot during this short cycle leading to 1.7.9 and hopefully we can use the next few weeks squashing any regressions, preparing for a solid release by the end of January 2012 (Knock wood...).

A big Thank-you for all the contributors and testers.

Git 1.7.8.1

2011-12-21T14:40:00.000-08:00

The latest maintenance release Git 1.7.8.1 is available. Note that this
is not a release with new features (upcoming 1.7.9 is expected to be
released late January next year to include the pulling of signed tags and
other goodies).

Please upgrade to get the accumulated bugfixes since 1.7.8 release from:

http://code.google.com/p/git-core/downloads/list

and their SHA-1 checksums are:

198e23e6e50245331590a6159ccdbdbe1792422c git-1.7.8.1.tar.gz
8f674dba39d9ae78928abfe9d924b0855e283e98 git-htmldocs-1.7.8.1.tar.gz
b49ce0b4da4f85671693c9b2c6f6a8b8ee65c809 git-manpages-1.7.8.1.tar.gz

In some codepaths (notably, checkout and merge), the ignore patterns recorded in $GIT_DIR/info/exclude were not honored. They now are.
"git apply --check" did not error out when given an empty input without any patch.
"git archive" mistakenly allowed remote clients to ask for commits that are not at the tip of any ref.
"git checkout" and "git merge" treated in-tree .gitignore and exclude file in $GIT_DIR/info/ directory inconsistently when deciding which untracked files are ignored and expendable.
LF-to-CRLF streaming filter used when checking out a large-ish blob fell into an infinite loop with a rare input.
The function header pattern for files with "diff=cpp" attribute did not consider "type *funcname(type param1,..." as the beginning of a function.
The error message from "git diff" and "git status" when they fail to inspect changes in submodules did not report which submodule they had trouble with.
After fetching from a remote that has very long refname, the reporting output could have corrupted by overrunning a static buffer.
"git pack-objects" avoids creating cyclic dependencies among deltas when seeing a broken packfile that records the same object in both the deflated form and as a delta.

Also contains minor fixes and documentation updates.

"Pulling signed tag" is already in use in the field

2011-12-20T22:49:00.000-08:00

[update: with finishing touches, this will be part of 1.7.9 release]

One of the more important features we wanted to have for the next release of Git (1.7.9) is to support a workflow where a pull request for a signed tag is sent to the integrator and git pull in response to the request automatically verifies the GPG signature embedded in the signed tag. See this for the background.

Earlier, a typical pull request was for a branch name in the publishing repository of a contributor, and worse yet, the default request message created by the git request-pull command did not even mention what commit to expect as the result of a requested pull, which meant that it was unnecessarily hard to make sure what was pulled was genuinely what the contributor had produced for the integrator and for third-party auditors.

This feature has been cooking in the next branch of the Git project for a while, and recently graduated to the master branch to become part of the upcoming release. Linus has been using the in-development verison, and I learned that he made a commit (2240a7b (Merge tag 'tytso-for-linus-20111214' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4, 2011-12-14)) that pulls a signed tag in response to a pull request from Ted several days ago already!

It always is satisfying to see that the result of our hard work is used in the field.

The workflow between the contributor (or a lieutenant) and the integrator looks like this:

(contributor)

After preparing the work to be pulled, use "git tag -s" to create a signed tag.

$ git checkout work
$ ... "git pull" from sublieutenants, "git commit" your own work ...
$ git tag -s -m "Completed frotz feature" frotz-for-linus work

And push the tag out to your publishing repository. You do not have to push your work branch or anything else.

$ R=example.com:/git/froboz.git/
$ git push $R frotz-for-linus

Then prepare a pull request message.

$ git request-pull v3.2 $R frotz-for-linus >msg.txt

The arguments are

the version of the integrator's commit you based your work on;
the URL of the repository, to which you have pushed what you want to get pulled; and
the name of the tag you want to get pulled (earlier, you could only write branch names here).

The resulting msg.txt file should begin like so:

The following changes since commit 703f05ad5835cff92b12c29aecf8d724c8c847e2:

Froboz 3.2 (2011-09-30 14:20:57 -0700)

are available in the git repository at:

example.com:/git/froboz.git/ frotz-for-linus

for you to fetch changes up to 406da7803217998ff6bf5dc69c55b1613556c2f4:

Add tests and documentation for frotz (2011-12-02 10:02:52 -0800)

----------------------------------------------------------------
Completed frotz feature

----------------------------------------------------------------
A U Thor (7):
      frotz: do not use --index in the short usage output
      frotz: Add tests and documentation for frotz
      ...

The signed tag message is shown prominently between the dashed lines before the short-log, so you may want to justify why pulling your work is worthwhile when creating the signed tag.

Then open your favorite mailer, read msg.txt in, edit and send it to your integrator.

(integrator)

After seeing such a pull request message, fetch and integrate the tag named in the request.

$ git pull example.com:/git/froboz.git/ frotz-for-linus

It will always open an editor to allow the integrator to fine tune the commit log message for the merge when merging a signed tag. In the editor, the integrator will see something like this:

Merge tag 'frotz-for-linus' of example.com:/git/froboz.git/

Completed frotz feature
# gpg: Signature made Fri 02 Dec 2011 10:03:01 AM PST using RSA key ID 96AFE6CB
# gpg: Good signature from "A U Thor <author@example.com>"

provided if the signature in the signed tag verifies correctly. As usual, the lines commented with # are stripped out. The resulting commit records the contents of the signed tag in a hidden field so that it can be used by others to audit in the future, so the integrator does not have to keep a separate copy of the tag in his repository (i.e. "git tag -l" won't list frotz-for-linus tag in the above example).

After the integrator responds to the pull request and the work becomes part of the permanent history, the contributor can remove the tag from the publishing repository if he or she chooses to in order to keep the tag namespace clean.

(contributor)

$ git push example.com :refs/tags/frotz-for-linus

Moving forward to 1.7.9

2011-12-18T23:59:00.000-08:00

A handful topics that have been cooking on the 'next' branch have now been graduated to the 'master' branch, to be included in the upcoming Git 1.7.9 release. They are all relatively minor fixes and a small feature or two, except for one, which is to stream large files directly to a packfile upon "git add", instead of storing them individually in loose object files and having to later repack them into a single pack, which would doubly be time consuming.

As I wrote earlier, we expect to make Git 1.7.9 a fairly focused release with only small number of new features around high impact themes:

Better and more auditable communication in pull based workflow by supporting electronically signed pull requests that records more meaningful branch description;
More pleasant end-user experience by providing credential helper API to allow platform native keychain implementations to supply authentication material during "git push" and "git pull";
i18n of messages out of the end-user facing programs;
Better large-contents support.

The "'git add' that streams to pack" in tonight's pushout is about the last theme; there is another topic regarding the same theme already in development, but it is not expected to be complete during the 1.7.9 timeframe.

Of course, there are other miscelaneous features and fixes already in the 'master' branch and also still in flight and cooking in the 'next' branch, but as far as the big changes are concerned, we are about 80% feature complete as of tonight. The credential API is expected to hit the 'master' branch in a few days.

Epson WorkForce 545

2011-12-18T23:36:00.000-08:00

My wife wanted a new printer (as the old HP has been acting up), so gave a visit to a local office electronics shop after running a few product searches.

The printer should support Google Cloud Print (as both my wife and I carry Android phones), and also should be supported by CUPS. I ended up getting Epson WorkForce 545 which is an all-in-one wired/wireless unit, simply because HP does not have a good impression on me anymore, and because I never used Kodak printers.

After a few trial-and-error sessions, it was reasonably easy to figure out how to configure it.

I first tried to connect it via WiFi. One glitch was that there didn't seem to be any way to learn the MAC address of the unit (the WiFi router is configured to talk WPA/WPA2 but also to filter connections based on MAC). I however was happy to see that its panel display offered to print diagnostic after it failed to connect and there was its MAC address printed there. After that, it was easy to configure it to authenticate to the WiFi router.

As the unit will sit immediately next to the router, however, I decided to disable WiFi altogether and give it a wired connection with fixed address.

After connecting to the network, interestingly, it was much easier to configure the unit to work with the Google Cloud Print than with CUPS.

A newer Windows box of my wife (I think it runs something called Windows 7) found the printer without me doing anything in particular; just being on the same network segment seemed to be enough, and then the Windows box installed the printer drivers itself.

Visiting the IP address I gave to the unit with the web browser, there were a handful of controls, and the top one was to make it work with Google Cloud Print. It just redirected the browser to google.com for OAuth and I had the printer associated with my Gmail account. From there, I can share the access to the printer with my wife's Gmail account and with my work account.

As I do not print much (and nothing at home), this was the first time I added a network printer to CUPS. After blindly trying random URLs like http://ip-address-of-unit:631, ipp://ip-address-of-unit/, etc., finally figured out that this particular model (or perhaps recent Epson in this class in general) wants to be connected with socket://ip-address-of-unit as its URL, but I did not see this documented anywhere.

An older Windows box my wife uses to control her computerized weaving loom (I think it runs Windows XP) was a different issue. It didn't see and did not want to connect to the wired printer, even though it could see my Linux box that is running samba. Adding an entry for it in /etc/samba/smb.conf was a simple task after figuring out what needs to be done (which unfortunately took too long for my liking). In the printer's section I needed to add use client driver = yes for it to work.

By the way, at the office electronics shop, I saw the new models of Kindle family (including the Fire), Nook color and the Nook tablet. Somehow Fire looked too thick and unwieldy to me, while Nook tablet looked slim and very nice. I didn't buy neither, though...

The Disappearing Spoon: And Other True Tales of... by Sam Kean

2011-12-09T12:44:00.001-08:00

Every once in a while, I receive gifts from satisfied Git friends, chosen from my Amazon Wish list.

And today was such a day.

I enjoy reading history of science. Thanks Miro!

Git 1.7.8

2011-12-02T22:05:00.001-08:00

Git 1.7.8 is finally out.

The most notable improvement from my point of view is tightening of the checks on the data that go and come over the wire between repositories, but it probably is and should be invisible to the end users. Many other small improvements are all over the place.

The release tarballs are found at:

http://code.google.com/p/git-core/downloads/list

and their SHA-1 checksums are:

7453e737e008f7319a5eca24a9ef3c5fb1f13398 git-1.7.8.tar.gz

2734079e22a0a6e3e78779582be9138ffc7de6f7 git-htmldocs-1.7.8.tar.gz

93315f7f51d7f27d3e421c9b0d64afa27f3d16df git-manpages-1.7.8.tar.gz

Have fun.

The development cycle for the next release is expected to last for 8 weeks til the end of January, and some of the major planned topics are listed here.

Buying a new Git feature

2011-11-30T16:51:00.000-08:00

You are a manager of a technology company, and your engineers love Git in general, but Git is not a perfect fit to your organization. Perhaps some work-flow elements your people are used to are not supported nicely by today's Git. Perhaps some class of assets you want to keep track of are not supported well by today's Git.

You are wealthy enough to pay for a developer or two to identify, design and implement necessary changes to Git, but you are not wealthy enough fork Git to maintain such a change yourself forever while the upstream Open Source community continues to improve Git.

What can you do?

Of course, if the changes you initially develop are good enough, they may be merged to the upstream and then you do not have to worry about maintaining your fork yourself. But how would you ensure that the quality of your changes is good enough for upstreaming? Perhaps withhold the payment to your consultants until the changes hit the upstream?

I do not think this is necessarily limited to Git, but applies equally to any useful and active Open Source project.

Git 1.7.8-rc4 and upcoming cycle

2011-11-28T15:42:00.001-08:00

This cycle is taking a bit longer than I had hoped but this should be the last rc before the final.

We had to roll in a fix to the UI for a new feature added in this cycle to "revert/cherry-pick" to avoid costly migration in the later releases (originally, we introduced "--reset" action to discard the in-progress state of a multi-commit revert/cherry-pick sequence, but it was argued that what the action actually did was "--quit" in the sense that it does not reset the state to some known state. Renaming it to "--quit" further opened the door to introduce "--abort" which does revert the state to where the entire revert/cherry-pick sequence started).

The next cycle will have many interesting topics that are already cooking in various doneness, including:

local branch description (in addition to a good discipline of giving descriptive branch names) that can be used in various places including pull request messages, local merge messages and format-patch cover letters;
electronically signed pull requests by asking to pull a signed tag instead of a branch;
signed commit (possibly-if it is found useful, that is);
credential helper API to integrate with platform native keychain implementations;
progress eye-candy for fsck and repack;
"git add" of large contents will send blobs directly to a packfile;
side-by-side diff in gitweb; and
i18n of Git Porcelain messages.

PGP Key-signing and CA fire-and-forget

2011-11-24T02:35:00.001-08:00

Someone at work used to have a kernel.org account but recently needed to re-establish the presense in the web of trust by getting his PGP key signed, so we met and exchanged our key IDs and fingerprints, to mutually sign our keys. I earlier attended a key-signing party and my key has been signed by many other people, and it was a good place to bootstrap his key.

A PGP key has two parts; the public part that you give to others, and the private part that you keep to yourself. The easiest and most common way to distribute public part of the key is to upload it to public keyservers, where other people can find and retrieve your key by specifying the key ID, your name or e-mail address.

When other people want to send a message to you and preserve the secrecy of the message, they only need to use the public part of your key to encrypt the message for you, and PGP guarantees that the encrypted message can be decrypted and read only by whoever holds the corresponding private part of the key unless a complex math problem that is believed to be practically unsolvable can somehow be solved (in other words, "public crypto-system gets broken"). When you want to prove that a message was written by you, you use the private part of your key to electronically sign the message and make the result public, others can check the authenticity of the electronic signature by using only the public part of your key, and again PGP guarantees that the message couldn't have been signed by a person who does not have the private part of the key.

The public part of your PGP key records your name and e-mail address, among other things. It can and often does record more than one pair of name and e-mail (e.g. work address vs personal address). Anybody can generate a PGP key on his or her own and record any name and e-mail address in its public part. If you see a message signed with a PGP key whose public part records my name and address, unless you somehow know that it indeed is the key I created for me and whose private part I have, such a signature has no value. It may have been created by a random person inpersonating me.
If you encrypt a message you want to show only to me, using a random PGP key that records my name and e-mail address to encrypt it would not guarantee that only I would be able to read it, unless you somehow know that the key belongs to me.

Hence, people need a way to validate the authenticity of public keys. People can add electronic signatures to the public part of a PGP key that belongs to another person, vouching that the signer knows the key belongs to the signee. This signature can be made per name and e-mail pair recorded in the public part of the key.

If you see a signature on an unknown public key, signed by public keys that you know belong to people you trust, you can be as sure as you trust these signers that the unknown publiic key belongs to the person it claims to belong to. This "web of trust" extends recursively and I heard that a recent study indicates that all people in the world are connected by 4.74 hops on average.

The only facts I learned when I met the other person for the purpose of key-signing are:

The person looked like his photo in our employee directory, and possessed a photo ID that matches his name;
The achievements by the person described in our employee directory matched what the person I was supposed to be meeting who worked in the Linux kernel project had done; and
The person claimed that a public key belonged to him, and gave me a way to retrieve the public part of this key.

It is not directly the above that I am vouching for by signing the public part of his key, however. I am vouching for the fact that I somehow know that the public key belongs to the person who is in control of the name and e-mail address pair recorded therein. That is not something I checked by meeting the person and chatting with him. I only checked the "name" part, but not the "e-mail address" part.

CA fire-and-forget is a clever scheme to solve this last bit of the problem. Instead of signing the public part of the key for all the name and e-mail pairs and upload the result myself, I make N separate signatures on his public key, one for each pair of name and e-mail address recorded in it. And then I encrypt these N signatures with his public key and send them to the corresponding e-mail addresses. The recipient of these encrypted signatures then decrypt them and upload the result to the public keyservers to complete the cycle.

If the e-mail address belonged to somebody else who does not have the corresponding private part, the encrypted signature would not reach the intended recipient, and the signature would not be decrypted to be uploaded to the public keyservers. I'll see my signature only if the person sitting behind the e-mail address has the private key that corresponds to the public part I have signed.

It is a clever scheme, even though it is a bit cumbersome to use, even with the use of dedicated tools (caff found in signing-party package on some distributions).

Git 1.7.7.4

2011-11-18T14:30:00.001-08:00

Just another maintenance update, this time to fix minor build issues and fix a trivial corner case bug in the git name-rev --all command.

The upcoming feature release 1.7.8 is getting closer, too.

Git 1.7.8-rc3 and being lenient to others while being strict to self

2011-11-17T13:11:00.001-08:00

Hopefully the last release candidate before the real thing.

A big "Thanks" goes to Andrea Arcangeli for reporting an unpleasant regression, me for quickly fixing, and Michael Haggerty for reviewing the proposed fix.

The regression that will not be in the final release was that we broke

  $ git clone --reference=$local_repository $upstream

when the local repository we are borrowing objects from has signed or annotated tags, and the cause of this regression is that a recent topic screwed up implementation that tightens checks for branch and tag (collectively known as "refs") names. When we clone from $upstream while borrowing objects from a $local_repository, we tell the $upstream that objects that are in the $local_repository need not be sent to us, and we discover what objects $local_repository has by reading the output of

$ git ls-remote $local_repository

and adding the result to the set of "extra refs". We internally keeps track of all the "refs" that exist in our repository, and the code that registers the extra refs share the same codepath as the one that finds the branches and tags by reading from .git/refs/{heads,tags} directories. The problem was that the add_ref() function in this shared codepath had a check to error not (not just warn) when it tries to register any "ref" whose name does not conform to the rule. Because an entry for a signed or an annotated tag in the output from ls-remote denotes the object (typically a commit) the tag points at, and because such an entry is marked by adding ^{} at the end of the name of the tag to make sure it will not collide with names of the real refs (that character sequence is invalid), the new check triggered and made the whole clone command fail.

This episode shows two fundamental failures in the topic:

"extra refs" are not real refs, and they shouldn't even need names. The only reason they exist is to let our repository know the objects reachable from them do not need to be transfered into our repository when talking with the outside world. Perhaps we should even consider dropping the name parameter from add_extra_ref() function (but after making sure the code would not make unwarranted assumptions. One such assumption was that they have names and their name must conform to the usual refname rules, which was fixed, but there may be others).
The other use of add_ref() function is used to register existing refs that we find in our repository. While we might not like the name of some of them (nobody stops a user or a tool from creating a randomly named file under .git/refs/{heads,tags} directories after all), it is wrong to error out any operation when talking about what already exists in the repository; the damage is already done. Warning against them to help the user notice and correct is a different story.

The code should be lenient to what it receives and strict in what it produces.

For example, a colon is a forbidden character in a branch name, primarily because a branch with such a name, e.g. a:b, cannot be pushed out to another repository. But if you do not ever push such a branch out, it is not that unreasonable to expect that the following to work, at least for some definition of working:

  $ H=$(git rev-parse HEAD~20) && echo $H >.git/refs/heads/inval:id

  $ git show inval:id

It may be OK for the second line to error out (we cannot do much about the manual echo doing damage to the repository), but where there is no ambiguity (i.e. if there is a ref that is called inval, the above could be a request to show a subdirectory called id in that commit), warning that inval:id is a wrong name but still letting the user what s/he wanted to do would be a far nicer way to deal with a problem like this. After the above sequence, if the following fails only because the repository has a ref with an invalid name, it is even worse:

  $ git show master

and I would have to say it is close to inexcusable.