We have released version 2.3 of Goby. New in this release is the ability to restrict output to a genomic slice when concatenating multiple sorted alignments. The release also has features to help generate BAM alignments with read groups from Goby alignments. The Goby somatic output format is now quite stable as far as we can tell from our internal benchmarks (comparing performance and somatic calls with Mutect 1.1.4).  You can download this release here or browse the more detailed change log.

We have released version 2.2.1 of Goby. This release includes several bug fixes for the new somatic variation output. New features include the ability of the FDR mode to detect columns to adjust automatically when the column is marked with the “p-value” group. You can download this release here or browse the more detailed change log.

We have released Goby 2.2. Briefly, this version of Goby improves genotype calling and provides a tool designed to detect somatic mutations. We also provide a new option to suggest-position-slices to split alignment files by amount of data to process.  You can download this new version here or see a detailed change log. Feedback welcome on the user forum or GitHub issue tracker.

We have been a bit silent lately as we were working on new major releases of GobyWeb. We set ambitious goals for GobyWeb2 last fall and have been moving towards these goals steadily.  Here’s a summary of features we are now testing internally.

Configuration and maintenance: We have been cleaning up the source code and completely redesigned the configuration of the system to make it easier to install and maintain. This was becoming urgent because we maintain several instances of GobyWeb internally for different projects and groups and sometimes had trouble remembering where to change an item of configuration. Manuele Simi has completely reorganized the configuration files and documented the new structure. GobyWeb 2.0 is currently under test with this new config and will make it much easier to setup a new instance of GobyWeb.

Artifacts: The ability for plugins to download and install their own software or data artifacts. Such plugins don’t include pre-compiled binaries or big tarballs, they now use a mechanism to grab the source code or data archive from the internet or local repository and install on the node used for computation. Artifact support is expected to be released in GobyWeb 2.1. Because we need to adapt a number of existing plugins to use the new artifact mechanism, we need to test these changes thoroughly before we can release 2.1. For users of GobyWeb outside of Weill, these changes mean that GobyWeb 2.1 will be much easier to install. We are essentially removing the need to prepare the /scratchLocal/gobyweb directory described in the current installation instructions and obtaining and compiling third party software. Importantly, artifacts are now responsible for creating indices of genomes for specific organisms needed in an analysis (adding a new organism is as simple as adding a few lines in the organism configuration file). Artifacts will also help to produce perfectly reproducible analysis runs that can be traced back to specific versions of analysis software. Artifacts also make it easier to extend the list of plugins with new tools and functionality (e.g., we will use the Variant Effect Predictor to annotate the effect of genotype changes on transcripts and proteins).

There is a lot more in the work, but I thought I would give you an idea of some of the key developments you can expect in GobyWeb in the following months.

Here’s the prezi I presented at this year’s Rocky’s bioinformatics conference. The talk is mostly about GobyWeb. For more details about GobyWeb, see the pre-print.

After internal testing of GobyWeb 1.9, we pushed a binary distribution of GobyWeb 1.9.1. This version offers a new feature:

• A mechanism to periodically calculate data usage for sets of users. A configuration parameter (intervalForStatisticsInHours, set in webapp/conf/Config.groovy) allows to configure the interval in between two statistics’ calculations. Usage data can be viewed for all users by administror accounts. Individual users can see how much data their account holds (break-down by samples, alignments and group comparison analyses).

• Lucene index view of the results for EdgeR plugin.
•  Increased JVM memory for Bismark plugin (we found one dataset where one chunk ran out of memory with the previous configuration).
• Changed Bismark plugin to support mapping reads longer than 60bp (bismark requires all bases on one line in the fastq file so we changed the way fastq files are formatted before calling Bismark).

Please note that we now license GobyWeb plugins under the LGPL3 license. This change makes it possible for others to contribute new GobyWeb plugins, and facilitates the inspection, reuse and extension the initial set of plugins distributed with GobyWeb.

You can find GobyWeb plugins at GitHub: CampagneLaboratory/gobyweb-plugins.

We have released Goby 2.1.1. This is a maintenance release with one new feature:

We have releasedGobyWeb 1.9 to our local CTSC users last week. Internal testing is ongoing and so far identified only minor issues. The project now has its own user forum, where anybody can subscribe, get announcements and discuss features or ask questions of other users or the development team. We are preparing a binary distribution of 1.9.1 for others to download and install GobyWeb at their site.

To summarize the most prominent new features, GobyWeb version 1.9 offers:

Since we posted an announcement for Goby 2.0 on the samtools developer mailling list, I received a number of questions about differences between Goby alignments and SAM alignments. Many people start by assuming that Goby stores alignments in a way somewhat similar to SAM, but there are key differences, which I attempt to summarize here:

1. Goby supports 5 minute schema changes (i.e., data schema flexibility)
2. Alignments that span a splice site are stored differently
3. Sequence variations are stored explicitly in Goby

Goby supports 5 minute schema changes

Goby stores data with protocol buffers, according to well-defined data schemas. A key feature of protocol buffers is that it supports “5 minute schema changes”. You can add a new field (e.g., a collection of new data structures) to a schema and recompile the code (typically takes about 5 minutes). The recompiled program will seamlessly read old data files, and old programs will be able to read data generated by the recompiled program. This functionality is quite useful when you are developing new programs, or when a community of developers work on various tools that need a common schema, but often also require tool specific extensions. Goby supports 5 minutes schema changes. SAM is limited to extensions via new tags. You can encode structured data in tags, but you are left to write a serialization mechanism between data structures and strings (this will likely take more than 5 minutes).

Alignments that span a splice site are stored differently

Consider this alignment in SAM format (one alignment block spanning a splice site, from a paired-end file):

61PKHAAXX_HWUSI-EAS627_0007:5:38:17885:13714    97      chr1    1231595 255     66M144N9M       =       1231903 0 \
      AGCAGCTCAGGCTCCCACGAGTCCAGCGTCAGGGACCGCACCTTGGAGCAGTGGACACCCAGGCTCCTGTGGATG GGGGGFGGGGGDFGGGGGEGGEGGCGGFCGGGFG?GGEEEEEEGEGCEEEBBDDBBDBCED=CACDC@C9@A:<A     NM:i:0  XS:A:-  NH:i:1  MD:Z:75

The same alignment is stored by Goby in data structures that contain the following (the field names are shown for clarity, but are not stored). Please note that both Goby entries are indexed independently. Storing two entries make it possible to represent gene fusions easily. Imaging the fusion is between two chromosomes, Goby will store the fusion as usual, with splice links pointing at positions on different chromosomes.

{   // first part of the alignment block, before the splice site:
    query_index: 0
    fragment_index: 0  // first part/fragment of the read.
    target_index: 0
    position: 1231594
    query_position: 0
    matching_reverse_strand: false
    multiplicity: 1
    query_length: 75
    query_aligned_length: 66 // this alignment matches the first 66 bases
    target_aligned_length: 66
    mapping_quality: 255
    pair_flags: 97
    pair_alignment_link { // indicates this block is part of a pair,
                          // found at this location:
      target_index: 0
      position: 1231902
      fragment_index: 2 // the pair is another fragment of the same read.
    }
    spliced_forward_alignment_link { // indicates this block is part of a
                                     // spliced alignment, points to second
                                     // entry
      target_index: 0   // indicates the target/chromosome of the block
                        // after the splice
      position: 1231804 // the position of the alignment block
                        // after the slice
      fragment_index: 1 // which part of the read we link to
                        // after the splice
    }
    bam_attributes: "NH:i:1"
    bam_attributes: "NM:i:0"
    bam_attributes: "XS:A:-"
}
{   // second part of the alignment block, after the splice site:
    query_index: 0    // query index match with first alignment, this is the same read.
    fragment_index: 1 // second part of a read
    target_index: 0
    position: 1231804
    query_position: 66
    matching_reverse_strand: false
    multiplicity: 1
    query_length: 75
    query_aligned_length: 9  // this alignment block matches only the 9 last bases
    target_aligned_length: 9
    mapping_quality: 255
    pair_flags: 97
    pair_alignment_link { // indicates this block is part of a pair,
                          // found at this location
      target_index: 0
      position: 1231902
      fragment_index: 2
    }
    spliced_backward_alignment_link { // indicates this block is part of a spliced alignment,
                                      // points back to first entry:
      target_index: 0
      position: 1231594
      fragment_index: 0
    }
    bam_attributes: "NH:i:1"
    bam_attributes: "NM:i:0"
    bam_attributes: "XS:A:-"
}

Sequence variations are stored explicitly in Goby

SAM/BAM rely on CIGAR strings and  MD tags to represent sequence variations. A complicated CIGAR can look like this: 6S23M1I6M1D16M1I47M while the corresponding MD tag will look like this: MD:Z:3G4A7C4C1A2T2^T2C4T3A0G5C44. Goby replaces these codes with explicit data structures.The SAM alignment:

509.6.68.19057.157284	83	chr1	45881869	29	6S23M1I6M1D16M1I47M	=	45881519	-443	TTACCCGCTTTCCTTGCCCAAATTTTAAGTTTCNGGAAAAGGGGAGGGAAATGGGTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTGTGACAGAGTGTCAC	#######################################################ECGGGGGGGGGGGGGGGGGGGGGGGGHHHHHHHHHHHHHHHHHHH	MD:Z:3G4A7C4C1A2T2^T2C4T3A0G5C44	RG:Z:1	XG:i:3	AM:i:29	NM:i:14	SM:i:29	XM:i:10	XO:i:3	XT:A:M

Will be stored as shown below. Please note that while MD tags are optional in SAM, the reference base is typically currently stored in Goby alignments. The read_index field indicates the cycle in the read where the variation was observed.

{
        query_index: 0
        target_index: 0
        position: 45881868
        query_position: 6
        matching_reverse_strand: true
        multiplicity: 1
        query_length: 100
        query_aligned_length: 94
        target_aligned_length: 93
        mapping_quality: 29
        pair_flags: 83
        fragment_index: 0
        insert_size: -443
        read_origin_index: 0
        softClippedBasesLeft: "TTACCC"
        softClippedQualityLeft: "\002\002\002\002\002\002"
        bam_attributes: "XG:i:3"
        bam_attributes: "AM:i:29"
        bam_attributes: "NM:i:14"
        bam_attributes: "SM:i:29"
        bam_attributes: "XM:i:10"
        bam_attributes: "XO:i:3"
        bam_attributes: "XT:A:M"
        sequence_variations {
          to: "T"     // the read contained a T
          from: "G"   // the reference had a G
          position: 4 // at position +4 from the start of the alignment
          to_quality: "\002" // quality of the base was Phred=2
          read_index: 91     // the variation was observed at cycle 91 (the read matches the reverse strand, only a fraction of 100bp aligned).
        }
        sequence_variations {
          to: "T"
          from: "A"
          position: 9
          to_quality: "\002"
          read_index: 86
        }
        sequence_variations {
          to: "T"
          from: "C"
          position: 17
          to_quality: "\002"
          read_index: 78
        }
        sequence_variations {
          to: "A"
          from: "C"
          position: 22
          to_quality: "\002"
          read_index: 73
        }
        sequence_variations {
          to: "T"
          from: "-"
          position: 23
          to_quality: "\002"
          read_index: 71
        }
        sequence_variations {
          to: "T"
          from: "A"
          position: 24
          to_quality: "\002"
          read_index: 70
        }
        sequence_variations {
          to: "N"
          from: "T"
          position: 27
          to_quality: "\002"
          read_index: 67
        }
        sequence_variations {
          to: "-"
          from: "T"
          position: 30
          read_index: 64
        }
        sequence_variations {
          to: "A"
          from: "C"
          position: 33
          to_quality: "\002"
          read_index: 62
        }
        sequence_variations {
          to: "G"
          from: "T"
          position: 38
          to_quality: "\002"
          read_index: 57
        }
        sequence_variations {
          to: "GA"
          from: "AG"
          position: 42
          to_quality: "\002\002"
          read_index: 53
        }
        sequence_variations {
          to: "G"
          from: "-"
          position: 46
          to_quality: "\002"
          read_index: 48
        }
        sequence_variations {
          to: "T"
          from: "C"
          position: 49
          to_quality: "$"
          read_index: 45
        }
}

We’ve posted a feature matrix that compares Goby 1.x, 2.0, BAM, CRAM 0.7 and FASTQ file formats along about 20 dimensions. We welcome suggestions for additions or corrections to this matrix. What other functionality would you like to include in such a comparison?