I’ve long been on the fence regarding PHP’s type coercion and comparison issues, but a recent bug in Mantis Bug Tracker has made me /facepalm for the first time in my long history of working with PHP:

When I click on “Edit” next to 1.2, mantis shows me the 1.20 properties. When I click 1.1 it shows me 1.10!

The offending snippet of code:

foreach( $g_cache_versions as $t_version ) {
   if(( $t_version['version'] == $p_version )
      && ( $t_version['project_id'] == $c_project_id ) ) {
      return $t_version['id'];
   }
}

At first glance, it seems perfectly normal… and then you read the commit log, emphasis mine:

This is due to an incorrect version name comparison in version_get_id whereby the following check between strings was occurring:

if( “1.1″ == “1.10″ ) { … }

PHP evaluates this expression to true because 1.1 and 1.10 are treated as floats. We however need to preserve the string type during this comparison, thus we need to use the === comparison operator instead.

I thought I’d seen it all…

Now if only I could find a job local to Cincinnati… If you are interested in hiring me, or know someone who’s looking for an open source developer, take a look at my résumé.

Anyways, the information and explanations presented in his article are pretty accurate. Until I can get the time to fully document the SVN and Git plugins, or the framework as a whole, consider Chris’s post to be canonical information.

Cheers

The Good King Lisp raised his glass and toasted with the Knights of Lambda, “To much recursion!”

Happy birthday to me.

What’s twenty-four?

• Six by four
• Eight by three
• Twelve by two
• Six past legal
• Three past drinking
• One before cheap insurance
• Space Shuttle Discovery
• SLR cameras

Nifty.

The slides are hosted directly on Google Docs: http://docs.google.com/Presentation?id=dffhxmhs_1273ct3kc8

I’ll be attending it to give a presentation on MantisBT and the Source Integration framework. Specifically, I’ll be covering the myriad of new features that have made it into the project over the past week and a half. I plan to walk through setting up a project in Mantis, creating a new repository on GitHub, linking the Source Integration framework to that, and showing how the branch mapping and auto-resolving features work. Should be interesting.

After the presentation, I plan to post the slides, along with my presentation notes, up here. If anyone decides to video the talk, I’ll also make sure I can get a copy of that as well, but no promises.

Cheers!

The point of this is to show that it’s quite possible to integrate just about any type of version control tool with the Source Integration system; indeed I planned from the beginning to create a generalized framework that would support many different types and paradigms for version control. This should at least be evident in that I have already created extension plugins for Git and Subversion – it should be quite possible to extend the concepts further to Mercurial, Bazaar, CVS, or any other tool.

For the point of brevity, I’ll make the assumption that the developer at least has a fair understanding of PHP, their version control tool, and how events and plugins work in MantisBT. If you are not yet familiar with the plugin system, there is currently a basic introduction in the MantisBT Developer’s Guide, which I’ll hopefully be adding more information to in the near future.

Note: the following few sections dive into the concepts of how the framework was designed; for those just interested in creating extension plugins, skip to the implementation section.

Design Decisions

I tried to approach the design of this system from the standpoint of both generality and simplicity; there are many source control topics that I have either brushed out of the way, or left up to the implementations of other existing projects. The end goal was to have a flexible system that could support almost any type of tool currently available, while also abstracting (or delegating) away as much as possible to simplify the implementation.

The biggest part of the design that I chose to delegate away was the topic of viewing file contents and commit differences. There are so many existing tools out there to handle those tasks within source control repositories, that I figured it was a waste of time to worry about. This means that my system relies on external viewing applications for any repositories in use; for Git, there’s GitHub and Gitweb support, and WebSVN or SourceForge for Subversion.

Another design abstraction that I decided upon was the handling of tool-specific details, such as how changesets are labelled, or how branches and other such topics are implemented. I chose to store as neutral a data model as possible, and delegate any display (or interpretation) of the data to the appropriate extension plugins. For instance, this allows SVN plugins to prepend the standard ‘r’ in front of revision numbers (such as “r123″), while Git plugins can crop the changeset hash at eight characters for easier user consumption.

The resulting design utilizes the Inversion of Control concept to create a “framework” that requires as little tool-specific implementations as possible. The framework does much of the heavy lifting with integrating the data into MantisBT, and delegates only the tasks to extension plugins that require detailed information about the source control tool being integrated.

Much of this design is a direct extension of the plugin system in MantisBT, which I also created using Inversion of Control, and uses its own set of events to handle the delegation of tasks wherever necessary. This allows anyone to create an extension plugin that hooks into this framework and implement the integration for any source control tool, without needing to know much about how the framework integrates within MantisBT.

Data Modelling

For the actual data modelling, I decided upon three major data structures, and one ancillary structure. Each structure is given a simple object-oriented API to work with.

The root of the data model is the Repository object, which contains information on the source control repository itself, such as its name, access URL, and “type”. The type is basically a mapping to a specific extension plugin that knows how to retrieve, store, and interpret the data appropriately; examples of the type include “github” provided by the SourceGitHub plugin, and “websvn” provided by the SourceWebSVN plugin. Each repository also contains a dictionary of arbitrary information that can be used as storage for any tool-specific data; for the GitHub integration, this extra information includes the GitHub user containing the project repository.

Each repository contains a set of Changeset objects, which naturally contains data for each changeset in the repository history. Each changeset can have a revision string, a branch name, author information, timestamp, commit message, and a parent revision string. Most of this is completely optional, meant only for use with tools that support that type of data. For instance, the changeset revision has no meaning when used with a tool like CVS or SCCS, which only tracks revisions at the file level.

Note that I also decided, for the sake of simplicity, to only track a single parent revision for each changeset, even though some tools may have multiple parents for some changesets (such as merge commits in Git). This does throw away a small bit of data, but was done for the sake of maintaining a more simplistic data structure. Give me a good enough argument, and maybe I’ll change my mind.

For each changeset, there may be a set of File objects; these represent any files that have been added, modified, or removed from the repository in that changeset (or potentially any other action you can imagine). These objects track the filename and/or path, the action, and also track a separate revision string from the changeset object. This specifically enables usage with source control tools such as CVS that track file revisions. Note that this object does *not* contain any other information about the file, such as the content or diff, as per my initial decision to keep things simple; the repository itself keeps this information, and for the most part, it’s unnecessary information for a bugtracker to handle.

Lastly, we come to the one ancillary data model, but arguably the one that is most important to the whole point of this framework. For each changeset, there is a set of Bug relationships, that only exist to tie source control changesets to issues in the bugtracker. It also the simplest data in the system, consisting of just a bug ID.

Roughly following the data models described above, are a set of classes defined in Source.API.php (SourceRepo, SourceChangeset, and SourceFile) which contain a set of public variables for the primary data, and static or public methods for simplifying the process of loading, saving, and manipulating groups of objects. It should be rather straightforward to pick it up based on the information above and the comments in the code (IMHO).

Framework Events

Utilizing the event-based plugin system, the source control integration defines a set of events that delegate tool-specific tasks to appropriate extension plugins, such as viewing changeset information and gathering changeset data from the source control tool itself.

The first event is GET_TYPES, which allows extension plugins to “register” themselves with the integration framework, so that it knows what repository types are supported.

The next set delegates the display of certain information: SHOW_TYPE covers the output of an extension plugin’s repository type, SHOW_CHANGESET allows manipulation of the changeset string, including how the branch and revision information is displayed, and SHOW_FILE allows the plugin to modify how files associated with a changeset are displayed, including action and revision information.

Another set delegates the URL links for data when displayed: URL_REPO for the URL to the main page of the repository browser, URL_CHANGESET for a link to a page in the browser for changeset, URL_FILE for a link to the entire contents of files in a given changeset, and URL_FILE_DIFF for a link to the diff output of files from a given changeset.

To handle repository type-specific data, two events allow extension plugins to hook their own form elements: UPDATE_REPO_FORM for displaying the form elements, and UPDATE_REPO for retrieving the form data and storing it in the repository object.

The last set of events allow extension plugins to gather data from the repository itself and create data objects and return them to the system: PRECOMMIT to analyze the incoming data from a repository’s post-commit hook to determine what repository type the data is for, COMMIT to interpret the data and generate objects for an incoming group of changesets, IMPORT_LATEST to pull the latest data from the repository without requiring a post-commit hook, and IMPORT_FULL to wipe and completely re-import changeset data from the repository.

Implementing an Extension Plugin

So how does this design come together when creating an extension plugin to integrate a new source control tool? Well, it’s actually not that complicated; most of the design I discussed in the above sections helps make this portion of the process much simpler and easier to implement. It even goes so far as to give you an initial head start when creating a new extension plugin, with an abstract class to inherit from with most of the plumbing already taken care of for you.

I’m going to cover a lot of this topic using a fake version control tool (“Open Sauce”) as an example; for almost everything, you’ll need to substitute code specific to the tool you’re integrating with Mantis. You may also be interested in reading through the SourceGithub and SourceWebSVN plugins as “real world” examples. Onwards.

Naturally, your extension plugin will still need to be a valid Mantis plugin, so let’s start there. You’ll need to create a new plugin (see the MantisBT documentation for this), and the plugin will need to extend the MantisSourcePlugin class, which is what provides you with the plumbing I mentioned above, such as hooking the framework events, and defining an abstract API for you to implement. For a rough example of the base plugin declaration:

<?php
class SourceOpenSauce extends MantisSourcePlugin {
  function register() {
    $this->name = 'Open Sauce Integration';
    $this->version = 'One Point Oh';

    $this->requires = array(
      'MantisCore' => '1.2.0',
      'Source' => '0.13',
    );
  }
}

So that’s a decent start, but now we’ll need to start filling in all the pieces of the MantisSourcePlugin interface. The interface methods each map to an event defined by the main Source Integration framework, and allow your plugin to take control whenever tool-specific details are needed.

Note: for most of these methods, you will need to check that the repository “type” is the one your plugin has registered, so that it does not try to handle the wrong repositories.

Let’s register our repository type and tell the framework how to display it; this could benefit from using internationalized language strings, but that’s beyond the topic scope:

  function get_types( $p_event ) {
    return array( 'opensauce' => 'Open Sauce' );
  }

  function show_type( $p_event, $p_type ) {
    if ( $p_type == 'opensauce' ) {
      return 'Open Sauce';
    }
  }

So now we can tell the framework how we want changesets and file entries to be displayed; for the purpose of example, we’ll assume that both changesets and files have revision numbers:

  function show_changeset( $p_event, $p_repo, $p_changeset ) {
    if ( $p_repo->type != 'opensauce' ) {
      return;
    }

    return "$p_changeset->branch #$p_changeset->revision";
  }

  function show_file( $p_event, $p_repo, $p_changeset, $p_file ) {
    if ( $p_repo->type != 'opensauce' ) {
      return;
    }

    return "$p_file->action - $p_file->filename #$p_changeset->revision";
  }

Next up is defining how source control data links to the made-up “Open Sauce Webview” application. Note that we are getting the viewer’s base URL from the repo object, which we’ll cover later:

  function url_repo( $p_event, $p_repo, $p_changeset=null ) {
    if ( $p_repo->type != 'opensauce' ) {
      return;
    }

    $t_url = $p_repo->info['viewer_url'];

    if ( !is_null( $p_changeset ) ) {
      $t_rev = $p_changeset->revision;
    } else {
      $t_rev = false;
    }

    return ( $t_rev ? "$t_url/view/$t_rev" : "$t_url/view" );
  }

  function url_changeset( $p_event, $p_repo, $p_changeset ) {
    ...
  }
  function url_file( $p_event, $p_repo, $p_changeset, $p_file ) {
    ...
  }
  function url_diff( $p_event, $p_repo, $p_changeset, $p_file ) {
    ...
  }

Next up is hooking some form elements onto the repository management page, so that the MantisBT administrator can set the base URL for the “Open Sauce Webviewer”:

  function update_repo_form( $p_event, $p_repo ) {
    if ( $p_repo->type != 'opensauce' ) {
      return;
    }

    $t_url = $p_repo->['viewer_url'];

    echo '<tr ', helper_alternate_class(), '><td class="category">',
      'Open Sauce Viewer URL</td><td>',
      '<input name="viewer_url" value="',
      string_attribute( $t_url ), '"/></td></tr>';
  }

  function update_repo( $p_event, $p_repo ) {
    if ( $p_repo->type != 'opensauce' ) {
      return;
    }

    $p_repo->info['viewer_url'] = gpc_get_string( 'viewer_url', '' );

    return $p_repo;
  }

The last piece of the puzzle is the biggest: gathering commit data from the repository and converting it into the generalized data structures provided by the framework. Here, we'll only cover a skeleton of importing the latest commits; you'll need to extrapolate on your own to cover the other methods of bringing in data:

  function import_latest( $p_event, $p_repo ) {
    if ( $p_repo->type != 'opensauce' ) {
      return;
    }

    # randomly make a query to get the latest commit data; our fake
    # viewer "returns" a JSON-encoded payload with commit info...
    $t_data = json_url( $p_repo->info['viewer_url'] . '/latest' );

    # prepare to aggregate the data
    $t_changesets = array();

    # go through all the changesets
    foreach( $t_data->commits as $t_commit ) {
      $t_changeset = new SourceChangeset(
        $p_repo->id,
        $t_commit->revision,
        $t_commit->branch,
        $t_commit->timestamp,
        $t_commit->author,
        $t_commit->message
      );

      # go through all the changeset's files
      foreach( $t_commit->files as $t_file ) {
        $t_file = new SourceFile(
          0, #no changeset id yet
          $t_file->revision,
          $t_file->filename,
          $t_file->action
        );

        #attach the file to the changeset
        $t_changeset->files[] = $t_file;
      }

      # parse any referenced bugs from the commit message
      $t_changeset->bugs = Source_Parse_Buglinks( $t_changeset->message );
      $t_changeset->save();

      # keep track of all the changesets imported
      $t_changesets[] = $t_changeset;
    }

    # return the set of changesets imported
    return $t_changesets;
  }

Wrap It Up

At this point, we now have a basic extension plugin that can pull data from the source control tool and interact with the framework to display the data appropriately. You should now be able to take this to the next level, and create an integration plugin for any source control tool you can imagine. I'd personally love to see support for Mercurial and Bazaar in the near future.

And here's the kicker: if you are willing to release your integration plugin under an open source license (such as BSD or GPL), I'll gladly include it in my main source-integration.git repository along with the framework, as I do for the existing Git and SVN plugins. You'll benefit from any community bug fixes and feature updates, and the community will benefit from having all the plugins in a single location, easy to find and install. I'll even give you developer access to the repository on MantisForge.org in case you need to update it in the future.

Cheers, and happy integrating!

This is how I like to answer these types of questions.

Thank you Explosm, you’ve just made my day!

As the core developer of the new plugin system, I have been working on a variety of plugins. In particular, I have created a vastly improved method of integrating source control repositories within MantisBT. The plugin package is named, aptly enough, Source Integration, and implements a generic framework that will allow integration with multiple repositories, each potentially using any source control systems available, simply by creating an extension plugin for each new tool. Currently, I have implemented integration packages for both Git and Subversion, my two source control tools of choice.

The Source Integration package tracks repository information based on a series of changesets, each of which may have a list of affected files. The data representation is generic enough to cover version control concepts used by all types of tools, from the ubiquitous CVS and Subversion, to modern distributed tools like Git and Hg. However, the system takes the stance of implementing as few details as possible, so it relies on existing repository-viewing tools for tasks such as viewing commit diffs, file contents, tree browsing, etc. Extension plugins handle translating tool-specific information, like history logs or checkin data, into the generalized data objects used by the framework. Extensions also generate URL’s for viewing files and diffs, but everything else is handled automatically by the core framework.

The true benefit of the Source Integration package lies in the amount of repository integration that it implements within MantisBT. When importing changesets from your repository, Source looks at the commit message of each changeset for references to bug numbers in your tracker, and sets up links in the database for any bugs mentioned. When viewing bugs mentioned in commit messages, a new section is displayed after the bugnotes called “Related Changesets”, giving a list of linked changes, including information about the changeset, such as the branch, author, timestamp, and a list of changed files.

Since a picture is worth a thousand words, and I’m not that great of a writer, few screenshots for your viewing pleasure:

List of integrated repositories.

Search for imported changesets that match a set of filter criterion.

List of changesets for a repository or search result.

List of repository changesets attached to the current issue.

Detailed information about the selected changeset.

Installation, Configuration, and Integration

So how do you get this source control goodness in your own MantisBT installation? Well, first, you need to make sure you meet a few basic requirements:

• a development snapshot of MantisBT version 1.2.x – either download a nightly build, or a snapshot from the MantisBT Gitweb – the old 1.2.0a2 release tarball will not work.
• to integrate Git, you need EITHER
• a public GitHub repository, OR
• a Gitweb installation (repo.or.cz works well)
• to integrate Subversion, you need EITHER
• a public SourceForge repository, OR
• a WebSVN installation to view diffs and/or file contents

Note that the Source Integration package relies on external viewers for Git integration, as it does yet support local repository access.

Assuming that you meet the above requirements, you can begin the installation and integration process. The latest version of the plugins can be found at the MantisForge Gitweb by clicking on the ’snapshot’ link next to each repository.

1. Install or upgrade to the latest Mantis 1.2.x (remember, the 1.2.0a2 release will not work)
2. Install the Meta plugin
3. Install the Source plugin
4. Install the appropriate Source extension plugin or plugins:
• SourceGithub for GitHub
• SourceGitweb for Gitweb and repo.or.cz
• SourceSFSVN for SourceForge
• SourceWebSVN for WebSVN


Installed plugins.

5. You’ll now have a new link in your MantisBT menu, named “Repositories”; click on it now


List of integrated repositories.

6. Optional: Click on the “Configuration” link to change the regular expressions used to match bug references in commit messages to match the style used in your project. By default, it will match text like “issue #xxx” or “bugs #xxx, #yyy”, etc. Note that the first regex finds groups of references, and the second regex matches the actual issue numbers.


Default regular expressions for extracting commit message bug references.

7. Fill in the “Create Repository” form, with an appropriate name, and selecting the appropriate repository type from the list of installed extensions


Form used to create a new repository integration.

8. You’ll be taken to a second form with a new set of options based on the repository type you’ve chosen. Fill in the appropriate information, and click “Update Repository” to save the information


Form for updating specific repository information.

9. You should be ready to begin the data import process. Click “Import Latest Data” to begin. Note that for large repositories, you may need to repeat this step, due to limits or timeouts, until it has finished importing the entire repository.

At this point, you should have your repository imported and integrated with your MantisBT installation, but you still need to do some further effort to make sure MantisBT stays up to date with the latest changes from the repository:

• For GitHub, enable remote checkins for address “65.74.0.0″ to allow GitHub’s servers access, and point your repository’s post-receive URL to “http://yoururl.xyz/mantisbt/plugin.php?page=Source/checkin”


Configuring remote checkin options.

• For Gitweb, SourceForge, or WebSVN integration, find your repository’s ID, and setup a cronjob (or something equivalent) to run “curl http://yoururl.xyz/mantisbt/plugin.php?page=Source/repo_import_latest&id=XX”
• For WebSVN, you may optionally use the included post-commit hook found in the plugin directory.

So now you should be done setting up integration between your repository and MantisBT. Congratulations!