Bioinformatics Zen

Dealing With Big Data In Bioinformatics

Michael Barton — Wed, 21 Oct 2009 16:00:00 PDT

Bioinformatics usually involves shuffling data into the right format for plotting or statistical tests. I prefer to use a database to store and format data as I think this make projects easier to maintain compared with using just scripts. I find a dynamic language like Ruby and libraries for database manipulation like ActiveRecord makes using a database relatively simple.

Using a database however stops being simple when you have to deal with very large amounts of data. Here I’m outlining my experience of analysing gigabytes of data with millions of data points and how tried to improve my software’s performance when manipulating this data. I’ve ordered each approaches with I think is the most pragmatic first.

The simple things

Obvious but sometimes overlooked.

1. Use a bigger computer

Using a faster computer might seem like a lazy option compared with optimising your code, but if the analysis works on your computer it should work the same, but faster, on a more powerful computer. Using a faster computer is probably one of the few things I tried which which didn’t involved modifying my code and therefore shouldn’t introduce any bugs. I used unit tests to make sure the code still worked as expected though.

2. Add database indices

Since I’m using a database, making sure it runs as fast as possible is another cheap way to improve performance. Properly indexed database columns reduce running times when searching or joining tables as an index means rows are looked up much faster. Database indices are also relatively easy to implement, just specify which columns need to be indexed either using SQL or in my case using ActiveRecord.

3. Use a faster language interpreter

Most of the time the standard version of ruby is sufficient for running my code. In last two years though different but faster versions have been created such as REE, JRuby and Ruby 1.9. Therefore when I was encountering long running times from processing millions of database rows I thought it was worth trying a faster Ruby version. I use Ruby 1.9 and it did improve performance. One caveat though was that I had to make my code compatible with the newer version specifically for the CSV library. These code changes were still relatively cheap to implement given the noticeable performance benefits.

Delete stuff

After the above three points I generally had to start digging around in my code - which is bad because changing working code usually creates broken code. A good way to optimise code, without introducing too many new problems, is just to delete it entirely.

4. Delete unnecessary data and analysis

I find that I often generate variables which I think might be useful at some future time. As you might expect just deleting the code that produces these variables removes the time required to compute them. More often than not I never ended up needing the variable anyway.

5. Remove database table joins

I’m using a database because usually I want to compare two or more sets of data and therefore I need to format them in a way that makes them comparable. Once formatted I join each variable in the database and then print the results as a CSV file.

The problem with joining a large number of database records, even with database indices, is that it can take a very long time. The amount of time required also increases the more the data is normalised. To try and fix this I found that I could drop the smaller of two variables I was joining and instead do the join further into my workflow.

For example I had two variables, the first contained millions of entries each one corresponding to a protein residue. The second data contained around only 100 entries each one corresponding to one of twenty amino acids. Merging these two variables in my database required millions of joins and took a long time. Instead I joined my amino acid data to my protein data after I had calculated the mean of each protein residue. This reduced the number of joins from a million down to around 100. I did the join as I plotted it using the merge function in R.

Code optimisation

When I was encountering performance problems I left optimising code as a last resort. There were three reasons for this, the first is that premature optimisation may be the root of all evil. The second reason reason is that the enemy of good-enough code is perfect code - when I start optimising code I tend keep going more than is necessary. Code doesn’t need to be as fast as possible though, just fast enough to get the results I need. The third point is that optimising code, means changing code, which introduces bugs and so the more the code is optimised the more chance of bugs. Code optimisation was a necessity though because my analysis was still taking days to run. I should also point out that my code optimisation was combined with thorough unit testing and benchmarking - which I think is usually how it should be done.

6a. Batch load database query results

One easy way I reducing running times was by batch loading the database table rows rather than trying to load a big table all at once. Pulling all the database records into memory means that most of the running time is spent loading the data into memory rather than actually dealing with it. Batch loading instead pulls subsets of records into memory at a time and each subset is then processed before the next set or rows is retrieved. This means less less memory is used each time. A example of this in Ruby is the ActiveRecord method find_in_batches.

6b. Association loading

Association loading means that when a row of Table A is retrieved from the database, that all the rows associated with it in Table B are also retrieved. This will usually mean that the database is only queried twice, once to find the records from Table A and once to find the records from Table B. The alternative option is to use a loop to retrieve each required row from Table B but this will mean as many database queries as there are rows - and more queries means more running time.

6c. Database querying in loops

I found that large loops which query the database were often the majority of my software’s running time. I improved this by instead moving the database calls, up or out of the loops as much as possible and caching rows in memory before hand. This meant the looping code was looking things up in memory rather then querying the database each time. A similar approach can also be used to avoid object creation inside loops which also seems to improve performance. Combining this approach with the one below was what most improved the running time in my analysis.

6d. Use raw SQL

Object relational management (ORM) libraries like ActiveRecord allow the database to be manipulated using object orientated programming which generally makes using a database a lot easer easier. Using an ORM does however add a performance penalty because it’s an extra layer on top of the database. When I was doing millions of database updates I found that skipping the ORM and directly using raw SQL contributed to a large saving of processing time. The advantages of this technique are neatly outlined by Ilya Grigorik.

That’s it.

Quite a long post I know. I know code performance is a weighty topic and probably what I’m outlining here isn’t the best way to go about dealing with large data. I’m there are better ways better technologies to manage large amounts of data too, e.g. map/reduce or schemaless databases. I’m not a trained computer scientist or a software engineer, but a biologist and what I’ve outlined is what allowed to me to produce the results I need. I’d be happy to read any further suggestions in the comments though.

Data Analysis Using R Functions As Objects

Michael Barton — Thu, 01 Oct 2009 16:00:00 PDT

The R language is useful because of the available statistical and plotting functions in the base and addon packages. Before using any function though it’s usually necessary to get your input data into the format that the function expects. Performing complicated data manipulations with R’s standard methods for accessing and subsetting data can however quickly lead to complex and confusing R scripts.

Processing a complex dataset in someway, for example to find the mean for a range of subsets, generally means creating variables to keep track of a series of levels to subset the data by, then creating a new variable for each subset. Creating many new variables can lead to complex R scripts that are difficult to understand. I’m going to try an outline here an interesting feature of R that allows functions to be treated as objects and can make for much cleaner and shorter R code. For example when I’m discussing “functions as objects” here I mean that the function for calculating standard deviation can be treated in the same was the array of data it works on.

In the examples here I’m using the crab which is part of the MASS package. This data contains morphological measurements on both sexes of two crab species.

library(MASS)
data(crabs)
head(crabs)
   sp sex index   FL  RW   CL   CW  BD
1   B   M     1  8.1 6.7 16.1 19.0 7.0
2   B   M     2  8.8 7.7 18.1 20.8 7.4
3   B   M     3  9.2 7.8 19.0 22.4 7.7
4   B   M     4  9.6 7.9 20.1 23.1 8.2
5   B   M     5  9.8 8.0 20.3 23.0 8.2

Performing functions on subsets of data

I have a function called do_something which I want to use to analyse two columns in the crab data. First I want to subset for each sex in each species and call my function on each data subset. In the example below I’m getting the data for the orange female crabs then calling my function on two of the columns in the data.

  do_something <- function(x,y){
    # Function code goes here ...
  }

  # Subset my data
  orange_girls <- subset(crabs, sex == 'F' & sp == 'O')

  # Call my function
  do_something(orange_girls$CW,orange_girls$CL)

I dislike this code because I have to first create a new variable as a subset of the data before I call my function on it. Secondly when I call the do_something function I have to specify the the columns in data I want using the $ symbol. Instead compare the same functionality but using the with command.

  with(subset(crabs, sex == 'F' & sp == 'O'), do_something(CW,CL))

Here I’m doing exactly the same thing, but the do_something command is called in the context of the data subset which means I don’t have to specify the columns using $. I also don’t have to create a new subset variable beforehand, I can just pass the subset of data as the first argument to the with command. I’m not restricted to just running one function either.

  # Same thing but using a multi-line code block via curly braces
  with(subset(crabs, sex == 'F' & sp == 'O'),{
    do_something(CW,CL)

    # Use more functions on the same subset
    lm(CW ~ CL)
  })

By using curly braces { } I can call multiple functions as part of the same command. I can do this because the curly braced expression is an argument to the with function.

Looping through subsets of data

So far I’m using just a single subset of the crab data, but I will want to call a function on all combinations of sex and species. Usually this means using for loops to iterate through each combination of sex and species to break the data into smaller chunks.

  # Arrays of values for each type of species and sex
  species <- unique(crabs$sp)
  sexes   <- unique(crabs$sex)

  # Loop through species ...
  for(i in 1:length(species)){

    # ... loop through sex ..
    for(j in 1:length(sexes)){

      # ... and finally call a function on each subset
      something_else(subset(crabs, sp == species[i] & sex == sexes[j]))
    }
  }

This is messy because I’ve had to create two arrays just to keep track of the unique values for each species and sex. Furthermore I’ve got two variables i and j which, if I change my loops at some point or add more loops, make the code more fragile and difficult to understand. Furthermore when you look at this code you can see there are nine lines just to get which subset I’m interested in but only one line actually calls my function. Instead of this I can create my own function to automatically iterate through the dataset and call any function I want.

  each <- function(.column,.data,.lambda){
    
    # Find the column index from it's name
    column_index  <- which(names(.data) == .column)

    # Find the unique values in the column
    column_levels <- unique(.data[,column_index])

    # Loop over these values
    for(i in 1:length(column_levels)){

      # Subset the data and call the passed function on it
      .lambda(.data[.data[,column_index] == column_levels[i],])  
    }
  }

The first two arguments of this function are the name of the column I want to subset the data by, and the data frame I want to iterate over. What’s interesting though is that the last argument .lambda is an R function, because R treats functions as objects this allows them to be passed as arguments to other functions.

  # Another function as the last argument to this function
  each("sp", crabs, something_else)

  # Or create a new anonymous function ...
  each("sp", crabs, function(x){

    # ... and run multiple lines of code here
    something_else(x)
    with(x,lm(CW ~ CL))

  })

I’m still looping through each subset of the data but I’ve now moved all the messy code to a separate function which makes the code more readable. Moving repetitive code like this to a separate function is good programming practice as it keeps your code DRY and also avoids the temptation for copy and paste anti-patterns.

My example function isn’t very sophisticated though and can’t create loops for subsets of more than one column. Nevertheless as you might expect there are methods exactly for doing this type of loop-each-subset approach to data analysis. An example is the lapply function in R base, however I find the syntax of this function hard to remember. Instead I prefer to use Hadly Wickham’s excellent plyr package. Plyr is very simple to use, for example:

  library(plyr)

  # Three arguments
  # 1. The dataframe
  # 2. The name of columns to subset by
  # 3. The function to call on each subset
  d_ply(crabs, .(sp, sex), something_else)

Again this is an example of how one function can accept another function as an argument. Compare how concise this code is with the code a few paragraphs up which uses for loops to do the same thing. The plyr package has more functionality than this though and is worth spending some time looking at. Also worth checking out are the foreach and iterators packages which provide similar functionality.

Too long, didn’t read? Here’s a short video

Keyboard, Command Line, And Text Files

Michael Barton — Tue, 08 Sep 2009 16:00:00 PDT

Here I’m outlining an approach to bioinformatics focusing on using three tools: the keyboard to enter commands, the wide range of functions available at the command line, and storing data in easily searched and manipulated text files. The combination of these three can make bioinformatics analysis faster than using graphical user interfaces (GUIs) because typing into the command line is faster than clicking buttons and selecting menus with a mouse. A series text commands is also easier to reproduce and automate than remembering a sequence of GUI commands.

The Keyboard

The best reason for using the keyboard is that it’s faster to type a command compared with doing the same thing with a mouse and GUI. If you’re using the keyboard to write code the more you can also use the keyboard for other actions such as saving the current document the less your “working flow” is broken by reaching for the mouse. Finally once you know the keyboard shortcuts to do your work learning to touch type will also make you work faster.

The Command Line

The command line gives predictable and reproducible results unlike a graphical interface. A workflow in the command line is easier for someone else to test and execute than a list screenshots of which buttons and menu items to click.

The Linux command line has a wide variety of tools ranging from interacting with the operating system to manipulating entries in a text file. Many bioinformatics applications can also be run at the command line and the benefit of using the command line version is that sequences of commands can be chained together. The output of one command becomes the input of the next command to filter and parse the results. In this way sequences of simple commands can be combined together to perform more complex bioinformatics analysis. As these tasks do not require a user with a mouse either they can be performed simultaneously across multiple machines or scheduled to run overnight.

Text Files

Binary files in specific formats can usually only be searched and edited by the program that created them. Plain text files are however open to being searched or manipulated by any tool or program. Therefore different data text formats such as CSV, JSON, YAML, and XML can still be searched and edited using similar command line approaches. Combining the command line with text files makes it relatively simple to perform simultaneous large searches or manipulations across many text based files.

Managing spelling in text and images

In this video I’m illustrating how the spelling of the sulfur across text and image files can be maintained consistently using the command line program sed.

Manipulating data files on the command line

In this video I’m showing how a spread sheet of amino acid data can be converted to comma separated value (CSV) format. The amino acid data is then searched using grep to find for the entry for cysteine. AWK is used to sort by carbon content. Finally I use the R language to create histograms and x/y plots of the data.

Using A Database

Michael Barton — Thu, 19 Feb 2009 16:00:00 PST

My best recommendation for any computational biologist is to learn to use a relational database with a corresponding object relational mapping system (ORM). This sounds complicated, but doesn’t have to be. In bioinformatics, data are distributed as files. Supplementary data are available from journal websites, and a file is easy to attached to an email. The use of data files in programming, however, should be limited wherever possible. A bioinformatics project should instead be built using a database.

Using a database allows all data to be accessed in the same way, whether in a script, at the command line, or through third-party database software. Databases are fast and optimised for searching and joining datasets. Joins between two sets of data that would be difficult when merging two files are made much easier using database relationships.

A simple database workflow first loads all data into the database. Each file usually becomes a table in the database, where each file row is a table row. Analytical scripts make database calls to pull and join different data sets together. Adding indices to a database further increases the speed at which joins are made and data searched.

In contrast using files as the base of the project results in errors when file paths change. Scripts need rewriting if a file format is altered. If the data file has a missing bracket or comma, the resulting script will throw an exception and break. The worst thing about using flat files though, is that they must be parsed and joined at the start of each script. This is repetitive and leads to code duplication across scripts.

SQL is hard

What I haven’t mentioned is that learning to use a database takes time. Understanding how to structure tables and the language to join them together requires effort. Furthermore, writing SQL join statements in scripts requires attaching strings together to create the SQL query, which is complex, hard to maintain, and produces ugly code.

Using object relational mapping (ORM) makes using a database easy and code simpler to write. The phrase “object relational mapping” is jargon for what allows database tables and rows to be treated as in-code variables. Instead of creating verbose SQL statements or reading to the required line in a file, the required data are called in the familiar programming syntax of the language you are used to. This combines the best of efficient data storage, with the language you are skilled in.

Scripting

Michael Barton — Thu, 19 Feb 2009 16:00:00 PST

Scripts differentiate computational research from software production. A script is a file of code with a specific purpose such as running a BLAST search on the E. coli genome. Contrast this with much larger programs designed to manage a variety of inputs and commands. A bioinformatician uses scripts as research tools in the same way a laboratory biologist uses a pipette. In software development, scripting supplements the designing of a software product. The focus is the finished product and scripts there to make source code management or unit testing easier. Since scripts receive comparably less attention as a part of software design, is there best practice for using scripts?

Managing dependency

Scripts are often required to run in a specific order. One script produces a result which is the input to the next script. This means the second script is dependent on the first. Dependency in software equates to increased complexity and requires more work to maintain a project. For example, if there is an undetected bug in one script mistakes are propagated as the next scripts are run. Or if one script in a series is missed, and the output files of a previous iteration still remain, then datasets are mixed between workflow repetitions resulting in unexpected side effects.

Removing the dependencies between workflow steps is difficult. Build files such as Rake, Ant, and make allow dependencies between scripted steps to be formalised: the required previous steps are automatically run first. This is useful to force the requirement that all previous results are deleted before hand, or that all rows in the database are refreshed, or even that the entire analysis is repeated from scratch. Capistrano is a variant where build files can be used to automate repetitive tasks across multiple remote computers. All of this can be managed using single command line calls.

Light and fluffy

Light and simple scripts are easier to maintain. To simplify, a script reads in a set of input data (such as a protein sequence), analyses the data (formatdb on a sequence database followed by BLAST), and then returns to the data (prints the results to the command line). Using this simplification, a script only needs to know what data is coming in, how to analyse the data, and how to return it.

Scripts can be made lighter by reducing the amount of analytical code. Instead of writing the code to call and parse BLAST, use existing code such as in BioPerl. If the code you need doesn’t exist anywhere else, consider writing it as a separate library which can be shared across all your scripts. A script that reads in a the data, calls an external library, then prints the results will be smaller and simpler to understand. Contrast this with a script that reads in data, formats the data, has several lines of a code to interpret and massage the results, then writes output.

Keeping light and simple, and formalising dependencies makes script-based projects easier to manage, maintain, and repeat.

Why Write Good Software

Michael Barton — Mon, 02 Feb 2009 16:00:00 PST

Bioinformatics is far from commercial software development. A bioinformatician’s goal is developing novel scientific research or tools. A software developer is judged on delivering software that people will pay to use. A biologist, whether they use Perl, a pipette or both, is evaluated on their publication record.

In bioinformatics, or any science using a computer, software development is a lesser priority than generating new data. Statistical tests for significance outweigh software testing for reliability. A series of Python scripts for interpreting Chip-chip data are a bioinformatician’s tools; what is important is the publishable prediction of binding sites.

Compare this with commercial software development, for example development of a hotel online line booking system. The developer talks to the hotel to understand the job. A good developer keeps regular meetings with the hotel, to update the project based on the customer’s requirements. The developer maintains the code using common development practices: unit testing, automated building, and source control.

The situation in bioinformatics is different; a hypothesis is made, implemented, and tested. There are no best practices. Methods of research range from a directory of BLAST results with an Excel spreadsheet, to a full application stack with a database backend, revision control, and unit tests. The choice depends on the bioinformatician’s knowledge and experience.

Is good software important for bioinformatics? Either end of the above scenario is rare, and a middle approach is a set of flat files, Perl scripts to parse out required rows, with R scripts to plot the results. If the tools work does the method matter?

Receiving peer review on a manuscript is comparable to getting feedback on delivering a product to a customer. Instead of new feature requests, changes are required as new analyses or the addition of a new data set. Feedback from reviewers is only received after months of work, when the software is developed and mature. The same principles that apply for commercial software can apply for scientific software. Investing 10% extra time in developing versatile and maintainable code saves time later when large changes are required. Using version control is a safety net for making changes to existing code. Unit testing ensures fewer bugs. Automated building makes execution of linear tasks easier. A database enables easier manipulation of large complex data.

Reuse, Contribute, Create

Michael Barton — Sun, 25 Jan 2009 16:00:00 PST

Use existing code

A quick way to get something done is using code that someone else has already written. Many languages have a bioinformatics specific library, such as BioPerl, BioJava, BioPython, or BioRuby. These libraries have functions for many common tasks, such as reading Fasta files, or parsing BLAST results.

One reason to use an existing library over writing your own code is saving time. These libraries are also mature and tested, which means the chance of a bug is much less. If you’re unable to do something in particular and can’t find an answer in the documentation, asking a question on the mailing list will usually result in a suggestion of where to look.

Contribute to existing code

The more specific your requirement the less likely an existing solution. In this case you’ll need to create the necessary fix yourself. After coding something up, being a generous person you will want to contribute the code to a bioinformatics library. This might mean a little work, but by contributing you can save other people time with the same problem.

Contributing code first requires getting the library source code using whatever version control system (VCS) the code is managed with. This can be difficult if you’re never used a VCS before, but is a good change to learn. Once you’ve got the library you’ll need to add your own code, as well as some documentation, and usually a few tests. After this you’ll need to send your update back via the VCS, or submit a patch to the mailing list.

Create new code

Creating a new library should be a last resort, but sometimes the function you want doesn’t fit with any existing libraries. Why is creating a new library a last resort? Because it takes more work than adding to an existing library. Having said that, creating a new library does have benefits, as packaging your code makes it is easier to maintain and use across your own projects. Taking the time to share your code with other people also makes you a good person.

Smaller and simpler is better when creating a new library. The simpler the library, the easier it is for others to use, and for you to maintain. Use a version control system to keep track of changes to the code, git is a good choice. Document the code, and create some web pages highlighting how the library is used. Develop unit tests so that you can make sure the functionality remains the same whenever changes. Make the code open source so that anyone else can contribute. Finally host the library somewhere so that people can get access. There are usually specific resources for each language, for instance in Ruby there is Rubyforge or Github.

This sounds like a lot of work, which is why simpler and lighter libraries are easier is to maintain. The process of creating a new library is a rewarding itself, but also has benefits. Other people may like your library and decide to contribute or fix any bugs. Therefore if you use the library regularly yourself, the investment in creating and maintaining a library will feed back into your own work.

Writing Good Code

Michael Barton — Sat, 24 Jan 2009 16:00:00 PST

Writing good code makes life easier. If there’s a common theme in bioinformatics, it is this: you will write a script, move on to something else, then return to the script in a few months or years time and try to remember how it works. The clearer the code is originally written, the better to remember how it works. Here is a quote “All programming is maintenance programming, because you are rarely writing original code”. This means that most of your time will be spent fixing and improving code, rather than writing fresh. Writing code is personal, and discussing what makes good code is controversial. But I’m going do it anyway and describes what I think are a few basic principles that can help to make code easier to maintain.

Be too descriptive

I think code should err on the side of being too descriptive, rather than being too concise. I mean that code should be loud and expressive about its purpose. An example is choosing variable names.

    # Concise
    seq = File.read('gene.fasta')
    
    # Descriptive
    fasta_gene_sequence = File.read('gene.fasta')

The second example is longer, but leaves no doubt as to what the variable contains. The same can be applied to method names. The more specific a method name the better to remember the function and what is returned.

    # Concise
    def get_seq(file)
      # ...
    end

    # Descriptive
    def read_fasta_from(file)
      # ...
    end

Next are magic numbers, numbers that appear in code, but have no explanation to their meaning. These can particularly annoying if you can’t remember why you used the number and there is no other reference to it.

    # Three, its the magic number
    dna_sequence.scan(3)

    # Descriptive
    nucleotides_per_codon = 3
    dna_sequence.scan(nucleotides_per_codon)

Comments never hurt either, as long as they are correct. Incorrect comments are generally not considered useful. Comments are especially useful when the meaning of the code is not obvious, but going too much commenting can sometimes make code less easy to read

    # Why the chop?
    protein = dna_sequence.translate.chop

    # Some of wikipedia in here...
    # In the genetic code, a stop codon (or termination 
    # codon) is a nucleotide triplet within messenger RNA
    # that signals a termination of translation. Proteins 
    # are unique sequences of amino acids, and most 
    # codons in messenger RNA correspond to the addition
    # of an amino acid to a growing protein chain, stop
    # codons signal the termination of this process,
    # releasing the amino acid chain.
    # Here I am removing the stop codon after translation
    protein = dna_sequence.translate.chop

    # Remove the stop codon after translating
    protein = dna_sequence.translate.chop

Try to follow the indentation guidelines for the language you’re writing in. Indentation makes code easier to read for you and anyone you share the code with.

DRY

DRY means don’t repeat yourself. Code for a single function should exist in a single place. When code needs fixing or maintaining, it only needs to changed once in the one place that it resides. In the short term it’s tempting to copy and paste to save time, but this will be time consuming in the long term when debugging.

For example a common function such as system specific BLAST settings, used across a variety of scripts can be kept in a single file. The can then be called by any script when required. By moving all the common code to a single file, if the BLAST settings change, this is done in just one place.

Books and frameworks

When I used Java, Joshua Bloch’s Effective Java book helped me learn a great deal about how to programme well. When learning Ruby I found the Ruby Way book had many useful examples of how to write in Ruby. I might guess for any popular programming language there is a respected book that illustrates the best practices in the language. These are not the most useful if you’re just starting to learn the language, but as you get more confident they are great for helping to write better, more maintainable code.

In addition to good books, examples of the best practices in a language can be found in popular open source libraries. Rails is a Ruby framework for creating dynamic website. Knowing Rails will come in handy if I ever need to create an interactive website, but practising with Rails also gives an opinionated view of the best way to organise a Ruby project, from people who are experienced in creating them.

Git

Michael Barton — Sat, 10 Jan 2009 16:00:00 PST

About

Git is a version control system or VCS for short. A VCS helps you manage your code by saving changes as versions in a repository. Each version of any file can be retrieved by rolling back the changes to the required version. At the most basic a VCS allows you the freedom to experiment and actively break the code you’re working because the last working version can be reverted to with a single command. Version control is used in software development, and in bioinformatics is useful for keeping track of the scripts and libraries you use in development. Version control using an external server is also a good way to back-up code.

Features

As a VCS git is clean and minimal, working out of a single .git directory in the root of your project. If you want to remove the project from version control, delete the .git directory and all git files are gone. Git managed repositories are small using compression to store the differences between versions. Git is fast at storing the latest version of your code, on even a large repository, it is almost instantaneous. Git repositories are simple to create, and don’t necessarily require an external server to begin tracking versions. If you do use an external git server, pushing and pulling to the server is also very fast. Another feature of git allows you to create branches within your code repository. Branching means copying the code as a duplicate branch of the main “master” branch. The duplicate branch can be modified, committed to and then compared with the original branch. If you are happy with the changes in the new branch you can merge them back into the original master branch. Another option is to leave the alternate branch to work on later since switching back to the master branch will restore the previous state before branching. In this way using a branches is a simple and lightweight way to develop or experiment with new features.

Collaboration

Git is useful for collaborating on shared source code repositories. The collaborative development of the Linux kernel is the reason git was created by Linus Torvalds. A key feature of git is that it is distributed. You are not bound by working from a single source server. I have my copy of the repository and you have yours. I like the changes you are making so I clone your repository as a branch into my own. I can test out the changes you’ve made before merging them into my master branch. If I only want a subset of the changes you’ve made I can use the git cherry-pick command to merge only the changes I want. The website github.com enables a collaborative aspect of developing software with git. Github acts as a git server but also highlights the social links of branches between developers. Other developers’ git repositories can be viewed and downloaded, but also forked into your own github space. This fork acts as a copy of the original repository with the relationship between the two repositories maintained. Github monitors the commits, merges and branching between repositories which can be viewed, compared, or visualised as a network.

Getting started

Latex

Michael Barton — Tue, 23 Dec 2008 16:00:00 PST

LaTeX (pronounced lay-tek) is a document creation system aimed at scientific and technical documents. LaTeX documents are written in plain text using markup to describe which parts should be sections tables or pictures. The LaTeX system parses the markup and formats the text to produce either dvi, postscript or pdf output. As LaTeX is entirely text based, the content can be tracked using a version control system. Plain text files mean that you can work on a document with your favourite editor, and can also be manipulated at the command line using Unix tools. The syntax of LaTeX markup will take an hour or two of practice to learn, but the advantage of creating documents from a marked up source is that the results are consistent and reproducible, which isn’t always the case for graphical document editors.

Features

The main reason for using LaTeX is that it allows you to work on the content of the document, not the formatting. If you were using a graphical editor you format the text as type, but with LaTeX you only have to add the markup to the document and LaTeX takes care of the rest. This can save a lot of time with large documents. The basic features of LaTeX include automatic generation of tables of contents, tables of figures and automatic numbering of sections tables and figures. BibTeX is the companion to LaTeX which adds simple organisation and addition of citations. Citations are added to documents using a simple “cite” command in the text, without the requirement for third party software. One of the benefits of creating documents using LaTeX is that the produced formatting is the result of best practices in typography and document presentation, which means LaTeX documents look better than the average.

Templates and Plugins

LaTeX is free software and available for most operating systems. There is a large LaTeX community which develops themes and modules that can be added to LaTeX documents. Many journals also provide LaTeX templates in which papers can be submitted. There are templates available for writing a thesis or dissertation, and there is likely a specific templates which follows your own institution guidelines. There are many useful third party plugins for adding extras to a document. For example beautiful formating of tables, grouping figures into subfigures, replacing text inside figures and even a framework for including R-code inside a LaTeX document.

Creating a LaTeX document

In this video I’m illustrating how to create a simple LaTeX document with text and headings. An overview of LaTeX document structure is outlined by Andrew Roberts

Creating figures in LaTeX

This screencast illustrates how to add a figure to a LaTeX document. The figure size is changed and a list of figures is added to the document. Andrew Roberts LaTeX site shows different examples of LaTeX figure settings.

Creating tables in LaTeX

This screencast illustrates adding a simple table to a LaTeX document. The table is then formatted to look more “professional” using the booktabs package. More information on tables in LaTeX can be found at Andrew Robert’s website and on the Wikibooks website

Adding references in LaTeX

This screencast shows how to add dynamic references to the text in a document. This includes automatically adding references to tables and figures. The second half of the video illustrates how to add a bibliography to a LaTeX document, and how to cite articles in the text. More information about citations can be found on Andrew Robert’s website.

Vim

Michael Barton — Mon, 22 Dec 2008 16:00:00 PST

About

Doing bioinformatics I only use two tools: the keyboard and the mouse. Since I use these tools all-day, everyday I want to use them efficiently. My opinion is that using the keyboard as much as possible while the mouse as little as possible, is the best way to work at a computer. In a simple example, knowing the keyboard shortcut to run a given command means that I can execute the command quickly without breaking the flow of typing. The text editor Vim takes this to another level and is entirely keyboard driven. There are no drop down menus and everything is performed using the keyboard; there’s no reason for my hands to leave the keyboard. The large number of commands for Vim means there is a fair amount of practice required before you can use it fluently. I think this practice is a great investment though, as being able to use Vim intuitively makes you work faster and more efficiently. The reason for this is because Vim has a huge functionality to be taken advantage of, using just a few quick remembered keystrokes.

Features

As Vim is entirely text-based, without the pretty interface of modern editors, my original opinion was of an archaic hangover from the early days of Unix. Vim is around twenty years old, but is still a sophisticated text editor with a large range of functions. The large range of vim’s functionality means a steep learning curve, but the extensive help documentation is an eloquent and gentle introduction. The commands “:vimtutor” and “:help” are the places to get started for using Vim. Vim’s greatest feature is how easy it is to move, edit and manipulate text. This can sound trivial, but reordering paragraphs in minutes with a mouse, takes just seconds using the keyboard with Vim. This example is applicable to any type of text file, such as Ruby or LaTeX source code, which is what I spend most of my time editing. Another Vim feature are the registers, which act as super-charged clipboards. Not only can text be stored for pasting, but also sequences of commands as well. Stored commands can then be replayed by calling the register, this eliminates performing repetitive actions. Typing “:help usr_10” is a good place to start for learning about using registers for commands, as well as other ways of making large changes quickly.

Customise

One reason not to use Vim, is that it doesn’t have the code orientated features of integrated development environments (IDE) such as Eclipse and Netbeans. But it does. Vim is easy to customise which has lead to a community of developers creating a large number of plugins. Code snippets, project drawers, and fuzzy file finding are just some examples of plugins aimed at using Vim as an IDE. Whatever language, file type, or framework you use, someone will have written a Vim plugin. This gives Vim all the functionality to rival any IDE.

Bioinformatics Zen

Dealing With Big Data In Bioinformatics

The simple things

1. Use a bigger computer

2. Add database indices

3. Use a faster language interpreter

Delete stuff

4. Delete unnecessary data and analysis

5. Remove database table joins

Code optimisation

6a. Batch load database query results

6b. Association loading

6c. Database querying in loops

6d. Use raw SQL

That’s it.

Data Analysis Using R Functions As Objects

Performing functions on subsets of data

Looping through subsets of data

Too long, didn’t read? Here’s a short video

Keyboard, Command Line, And Text Files

The Keyboard

The Command Line

Text Files

Managing spelling in text and images

Manipulating data files on the command line

Using A Database

SQL is hard

Scripting

Managing dependency

Light and fluffy

Why Write Good Software

Reuse, Contribute, Create

Use existing code

Contribute to existing code

Create new code

Writing Good Code

Be too descriptive

DRY

Books and frameworks

Further reading

Git

About

Features

Collaboration

Getting started

Latex

Features

Templates and Plugins

Creating a LaTeX document

Creating figures in LaTeX

Creating tables in LaTeX

Adding references in LaTeX

Vim

About

Features

Customise