There has been some confusion regarding Goby, the file sizes it produces and what kind of data it keeps (emails on samtools-devel and SEQAnswers). I realize the confusion is likely due to problems in the conversion tool we provided with earlier versions of Goby (Goby mode called sam-to-compact). I wanted to indicate here that we recently fixed a number of problems with this conversion tool. A much improved one is included in [URL="http://campagnelab.org/software/goby/download-goby/"]Goby 1.9.8.4[/URL].

To help clear some confusion, we have prepared a small benchmark. We obtained simulated paired-end data from GASV (here: http://code.google.com/p/gasv/downloads/detail?name=Example.bam). This file is 80MB and was converted to Goby format as follows (using 1.9.8.4):

java  -Dlog4j.configproperties=goby_1.9.8.4/config/log4j-sample.properties -jar goby_1.9.8.4/goby.jar -m sam-to-compact -i Example.bam -o Example-sorted –sorted

The resulting files are:

This alignment is sorted can be loaded into IGV 2+. One can verify then the type of information preserved in Goby format (variations, quality scores over mutations, pairs, etc). You can load this alignment by URL, in IGV, select open URL and enter:

http://dl.dropbox.com/u/357497/BAM-Goby-tiny-benchmark/Example-sorted.entries

The simulated data can be seen at chr17:1-4,315 (and beyond):

Viewing Example-sorted in IGV (click for larger snapshot)

The input file was:

It is worth noting that the 80M input file is a simulation with very low random coverage. Compression are often much better on less random data files (both for BAM and Goby).

This releases fixes various problems with the sam-to-compact mode and makes it possible to efficiently import sorted BAM files. The previous version was compiled with debugging statements on, which slowed down conversions drastically. This version converts about 40,000 BAM entries per second. The sam-to-compact options have also been redesigned to make simple conversions easier. For instance,

goby 1g sam-to-compact –sorted  -i input.bam -o goby-output

will convert input.bam to Goby alignment format preserving sort order (necessary if you want to load the file into IGV). Neither the –third-party option or the -p options are any longer required. Please see help for new option defaults. This should help a lot of people who are trying to get started with Goby and already have many BAM alignments. Please let us know if you encounter any other issues with SAM/BAM to Goby conversions.

Interesting piece on "Big Data Biology" by Titus Brown.
http://ivory.idyll.org/blog/mar-12/big-data-biology

We have posted revised installation instructions for versions 1.7+ of GobyWeb (versions with the plugin system). Installation instructions for the initial release of GobyWeb were moved here.

Visualizing both region and site estimates of methylation rates with IGV

We posted a new tutorial that explains how to use Goby to estimate methylation rates over regions (requires Goby 1.9.8.3+). The illustration on the left shows estimates of methylation over both regions and individual sites. Files were generated with GobyWeb ready for visualization in IGV. The tutorial explains how to produce these files on the command line with Goby. [tutorial]

We have release Goby 1.9.8.3. This version makes it possible to estimate methylation rates over annotated regions of the genome and fixes a few minor bugs.  The recently released version of GobyWeb (version 1.7.1) requires Goby 1.9.8.3+. See the project change log for more details.

We have released the binary distribution for GobyWeb 1.7.1 (release dated Feb 16 2012). This is a stable release with a few bug fixes and is the first public release to include the plugin system. The plugin mechanism makes it easier to integrate new aligners and analysis tools with GobyWeb. New plugins can be added at run-time and the user interface adapts as needed. The binary distribution can be obtained here. We are in the process of updating the installation instructions (the plugin system has more hooks for configuration and to adapt to a local system).

We have released Goby 1.9.8.2. This version offers the vcf-subset and vcf-compare replacements tools I mentioned in my earlier VCF post.

The release also packs an option to call indels with Goby. We use the method of Krawitz et al (Bioinformatics 2010) to find equivalent indel regions (EIR). This approach can re-conciliate distinct indel observations into canonical  indel boundaries (an EIR). The genotype and compare-groups formats of the discover-sequence-variants mode will output EIRs at a frequency that sum over all the possible indel variations observed at the site that can be explained by that EIR. Of course, there is quite more to the Goby indel calling approach than the Krawitz method. For instance, the approach is integrated with the fast algorithm for local realignment around indels, so that indels that open when realigning end of reads contribute to the frequency of an EIR.

Programmers will find that Goby represents  observed indels at a site in a very similar way to base genotypes. Reading a base or indel frequency at a position in a sample is done with the same API (see the SampleCountInfo class). This makes it easy to support indels in different output formats.

The vcf-compare replacement (new in this release) can keep random samples of positions that differ between input files according to each category of differences it tallies (e.g., missed one allele RA vs RR, missed two alleles AA vs RR, genotypes differ C/T vs A/T where R=G). This is quite useful in inspecting positions in a genome viewer to try and understand differences between calls made by two approaches.

More details about this release are in the ChangeLog.

Stumbled on PLINK/SEQ while looking for a tool to estimate Ti/Tv from VCF files. At the moment, PLINK/SEQ seems limited to some older version of VCF, so it does not quite work with the files Goby generate (4.1), but I am interested in the mention of a binary file alternative to VCF, which could speed up some of the work we do (see section about project creation).

PLINK/SEQ genetics library

Tutorial: working with 1000 Genomes Pilot 3 VCFs. By way of introducing some of the features and approaches of PLINK/Seq, this page provides a tutorial that uses PSEQ and the R interface to PLINK/Seq …

Here's a nice blog-review about studies that discovered variations causing diseases with next-generation sequencing.

In addition to the content, I like the comment of the author that there is no time to write a review paper given the speed of the field development. Obviously what he means (since he has written the material already), is that publishing in a journal has very high overheads for the author(s) that simply slow down communication of information. In this case, I would argue that the comments have served a similar role as peer-review, prompting the author to add links to the papers, adding to the information or modulating the claims of novelty (see comment about RET).