Bioinformatics Zen

how to use a database

Michael Barton — Sat, 28 Feb 2009 16:00:00 PST

This video illustrates a minimal approach to using a database as the foundation of a bioinformatics project. Only two files are used: a database.rb file, and a Rakefile. Object relational mapping is provided by using the DataMapper gem. The BioRuby gem is used parse a fasta file of S. cerevisiae proteins. Examples of using data validations and experimental logging are also given.

The text will be more legible in the high definition version.

Example organisation of a bioinformatics experiment from Michael Barton on Vimeo.

Code

using a database

Michael Barton — Thu, 19 Feb 2009 16:00:00 PST

My best recommendation for any computational biologist is to learn to use a relational database with a corresponding object relational mapping system (ORM). This sounds complicated, but doesn't have to be. In bioinformatics, data are distributed as files. Supplementary data are available from journal websites, and a file is easy to attached to an email. The use of data files in programming, however, should be limited wherever possible. A bioinformatics project should instead be built using a database.

Using a database allows all data to be accessed in the same way, whether in a script, at the command line, or through third-party database software. Databases are fast and optimised for searching and joining datasets. Joins between two sets of data that would be difficult when merging two files are made much easier using database relationships.

A simple database workflow first loads all data into the database. Each file usually becomes a table in the database, where each file row is a table row. Analytical scripts make database calls to pull and join different data sets together. Adding indices to a database further increases the speed at which joins are made and data searched.

In contrast using files as the base of the project results in errors when file paths change. Scripts need rewriting if a file format is altered. If the data file has a missing bracket or comma, the resulting script will throw an exception and break. The worst thing about using flat files though, is that they must be parsed and joined at the start of each script. This is repetitive and leads to code duplication across scripts.

SQL is hard

What I haven't mentioned is that learning to use a database takes time. Understanding how to structure tables and the language to join them together requires effort. Furthermore, writing SQL join statements in scripts requires attaching strings together to create the SQL query, which is complex, hard to maintain, and produces ugly code.

Using object relational mapping (ORM) makes using a database easy and code simpler to write. The phrase "object relational mapping" is jargon for what allows database tables and rows to be treated as in-code variables. Instead of creating verbose SQL statements or reading to the required line in a file, the required data are called in the familiar programming syntax of the language you are used to. This combines the best of efficient data storage, with the language you are skilled in.

scripting

Michael Barton — Thu, 19 Feb 2009 16:00:00 PST

Scripts differentiate computational research from software production. A script is a file of code with a specific purpose such as running a BLAST search on the E. coli genome. Contrast this with much larger programs designed to manage a variety of inputs and commands. A bioinformatician uses scripts as research tools in the same way a laboratory biologist uses a pipette. In software development, scripting supplements the designing of a software product. The focus is the finished product and scripts there to make source code management or unit testing easier. Since scripts receive comparably less attention as a part of software design, is there best practice for using scripts?

Managing dependency

Scripts are often required to run in a specific order. One script produces a result which is the input to the next script. This means the second script is dependent on the first. Dependency in software equates to increased complexity and requires more work to maintain a project. For example, if there is an undetected bug in one script mistakes are propagated as the next scripts are run. Or if one script in a series is missed, and the output files of a previous iteration still remain, then datasets are mixed between workflow repetitions resulting in unexpected side effects.

Removing the dependencies between workflow steps is difficult. Build files such as Rake, Ant, and make allow dependencies between scripted steps to be formalised: the required previous steps are automatically run first. This is useful to force the requirement that all previous results are deleted before hand, or that all rows in the database are refreshed, or even that the entire analysis is repeated from scratch. Capistrano is a variant where build files can be used to automate repetitive tasks across multiple remote computers. All of this can be managed using single command line calls.

Light and fluffy

Light and simple scripts are easier to maintain. To simplify, a script reads in a set of input data (such as a protein sequence), analyses the data (formatdb on a sequence database followed by BLAST), and then returns to the data (prints the results to the command line). Using this simplification, a script only needs to know what data is coming in, how to analyse the data, and how to return it.

Scripts can be made lighter by reducing the amount of analytical code. Instead of writing the code to call and parse BLAST, use existing code such as in BioPerl. If the code you need doesn't exist anywhere else, consider writing it as a separate library which can be shared across all your scripts. A script that reads in a the data, calls an external library, then prints the results will be smaller and simpler to understand. Contrast this with a script that reads in data, formats the data, has several lines of a code to interpret and massage the results, then writes output.

Keeping light and simple, and formalising dependencies makes script-based projects easier to manage, maintain, and repeat.

why write good software

Michael Barton — Mon, 02 Feb 2009 16:00:00 PST

Bioinformatics is far from commercial software development. A bioinformatician's goal is developing novel scientific research or tools. A software developer is judged on delivering software that people will pay to use. A biologist, whether they use Perl, a pipette or both, is evaluated on their publication record.

In bioinformatics, or any science using a computer, software development is a lesser priority than generating new data. Statistical tests for significance outweigh software testing for reliability. A series of Python scripts for interpreting Chip-chip data are a bioinformatician's tools; what is important is the publishable prediction of binding sites.

Compare this with commercial software development, for example development of a hotel online line booking system. The developer talks to the hotel to understand the job. A good developer keeps regular meetings with the hotel, to update the project based on the customer's requirements. The developer maintains the code using common development practices: unit testing, automated building, and source control.

The situation in bioinformatics is different; a hypothesis is made, implemented, and tested. There are no best practices. Methods of research range from a directory of BLAST results with an Excel spreadsheet, to a full application stack with a database backend, revision control, and unit tests. The choice depends on the bioinformatician's knowledge and experience.

Is good software important for bioinformatics? Either end of the above scenario is rare, and a middle approach is a set of flat files, Perl scripts to parse out required rows, with R scripts to plot the results. If the tools work does the method matter?

Receiving peer review on a manuscript is comparable to getting feedback on delivering a product to a customer. Instead of new feature requests, changes are required as new analyses or the addition of a new data set. Feedback from reviewers is only received after months of work, when the software is developed and mature. The same principles that apply for commercial software can apply for scientific software. Investing 10% extra time in developing versatile and maintainable code saves time later when large changes are required. Using version control is a safety net for making changes to existing code. Unit testing ensures fewer bugs. Automated building makes execution of linear tasks easier. A database enables easier manipulation of large complex data.

reuse, contribute, create

Michael Barton — Sun, 25 Jan 2009 16:00:00 PST

Use existing code

A quick way to get something done is using code that someone else has already written. Many languages have a bioinformatics specific library, such as BioPerl, BioJava, BioPython, or BioRuby. These libraries have functions for many common tasks, such as reading Fasta files, or parsing BLAST results.

One reason to use an existing library over writing your own code is saving time. These libraries are also mature and tested, which means the chance of a bug is much less. If you're unable to do something in particular and can't find an answer in the documentation, asking a question on the mailing list will usually result in a suggestion of where to look.

Contribute to existing code

The more specific your requirement the less likely an existing solution. In this case you'll need to create the necessary fix yourself. After coding something up, being a generous person you will want to contribute the code to a bioinformatics library. This might mean a little work, but by contributing you can save other people time with the same problem.

Contributing code first requires getting the library source code using whatever version control system (VCS) the code is managed with. This can be difficult if you're never used a VCS before, but is a good change to learn. Once you've got the library you'll need to add your own code, as well as some documentation, and usually a few tests. After this you'll need to send your update back via the VCS, or submit a patch to the mailing list.

Create new code

Creating a new library should be a last resort, but sometimes the function you want doesn't fit with any existing libraries. Why is creating a new library a last resort? Because it takes more work than adding to an existing library. Having said that, creating a new library does have benefits, as packaging your code makes it is easier to maintain and use across your own projects. Taking the time to share your code with other people also makes you a good person.

Smaller and simpler is better when creating a new library. The simpler the library, the easier it is for others to use, and for you to maintain. Use a version control system to keep track of changes to the code, git is a good choice. Document the code, and create some web pages highlighting how the library is used. Develop unit tests so that you can make sure the functionality remains the same whenever changes. Make the code open source so that anyone else can contribute. Finally host the library somewhere so that people can get access. There are usually specific resources for each language, for instance in Ruby there is Rubyforge or Github.

This sounds like a lot of work, which is why simpler and lighter libraries are easier is to maintain. The process of creating a new library is a rewarding itself, but also has benefits. Other people may like your library and decide to contribute or fix any bugs. Therefore if you use the library regularly yourself, the investment in creating and maintaining a library will feed back into your own work.

writing good code

Michael Barton — Sat, 24 Jan 2009 16:00:00 PST

Writing good code makes life easier. If there's a common theme in bioinformatics, it is this: you will write a script, move on to something else, then return to the script in a few months or years time and try to remember how it works. The clearer the code is originally written, the better to remember how it works. Here is a quote "All programming is maintenance programming, because you are rarely writing original code". This means that most of your time will be spent fixing and improving code, rather than writing fresh. Writing code is personal, and discussing what makes good code is controversial. But I'm going do it anyway and describes what I think are a few basic principles that can help to make code easier to maintain.

Be too descriptive

I think code should err on the side of being too descriptive, rather than being too concise. I mean that code should be loud and expressive about its purpose. An example is choosing variable names.

# Concise
seq = File.read('gene.fasta')

# Descriptive
fasta_gene_sequence = File.read('gene.fasta')

The second example is longer, but leaves no doubt as to what the variable contains. The same can be applied to method names. The more specific a method name the better to remember the function and what is returned.

# Concise
def get_seq(file)
  # ...
end

# Descriptive
def read_fasta_from(file)
  # ...
end

Next are magic numbers, numbers that appear in code, but have no explanation to their meaning. These can particularly annoying if you can't remember why you used the number and there is no other reference to it.

# Three, its the magic number
dna_sequence.scan(3)

# Descriptive
nucleotides_per_codon = 3
dna_sequence.scan(nucleotides_per_codon)

Comments never hurt either, as long as they are correct. Incorrect comments are generally not considered useful. Comments are especially useful when the meaning of the code is not obvious, but going too much commenting can sometimes make code less easy to read

# Why the chop?
protein = dna_sequence.translate.chop

# Some of wikipedia in here...
# In the genetic code, a stop codon (or termination 
# codon) is a nucleotide triplet within messenger RNA
# that signals a termination of translation. Proteins 
# are unique sequences of amino acids, and most 
# codons in messenger RNA correspond to the addition
# of an amino acid to a growing protein chain, stop
# codons signal the termination of this process,
# releasing the amino acid chain.
# Here I am removing the stop codon after translation
protein = dna_sequence.translate.chop

# Remove the stop codon after translating
protein = dna_sequence.translate.chop

Try to follow the indentation guidelines for the language you're writing in. Indentation makes code easier to read for you and anyone you share the code with.

DRY

DRY means don't repeat yourself. Code for a single function should exist in a single place. When code needs fixing or maintaining, it only needs to changed once in the one place that it resides. In the short term it's tempting to copy and paste to save time, but this will be time consuming in the long term when debugging.

For example a common function such as system specific BLAST settings, used across a variety of scripts can be kept in a single file. The can then be called by any script when required. By moving all the common code to a single file, if the BLAST settings change, this is done in just one place.

Books and frameworks

When I used Java, Joshua Bloch's Effective Java book helped me learn a great deal about how to programme well. When learning Ruby I found the Ruby Way book had many useful examples of how to write in Ruby. I might guess for any popular programming language there is a respected book that illustrates the best practices in the language. These are not the most useful if you're just starting to learn the language, but as you get more confident they are great for helping to write better, more maintainable code.

In addition to good books, examples of the best practices in a language can be found in popular open source libraries. Rails is a Ruby framework for creating dynamic website. Knowing Rails will come in handy if I ever need to create an interactive website, but practising with Rails also gives an opinionated view of the best way to organise a Ruby project, from people who are experienced in creating them.

git

Michael Barton — Sat, 10 Jan 2009 16:00:00 PST

About

Git is a version control system or VCS for short. A VCS helps you manage your code by saving changes as versions in a repository. Each version of any file can be retrieved by rolling back the changes to the required version. At the most basic a VCS allows you the freedom to experiment and actively break the code you're working because the last working version can be reverted to with a single command. Version control is used in software development, and in bioinformatics is useful for keeping track of the scripts and libraries you use in development. Version control using an external server is also a good way to back-up code.

Features

As a VCS git is clean and minimal, working out of a single .git directory in the root of your project. If you want to remove the project from version control, delete the .git directory and all git files are gone. Git managed repositories are small using compression to store the differences between versions. Git is fast at storing the latest version of your code, on even a large repository, it is almost instantaneous. Git repositories are simple to create, and don't necessarily require an external server to begin tracking versions. If you do use an external git server, pushing and pulling to the server is also very fast. Another feature of git allows you to create branches within your code repository. Branching means copying the code as a duplicate branch of the main "master" branch. The duplicate branch can be modified, committed to and then compared with the original branch. If you are happy with the changes in the new branch you can merge them back into the original master branch. Another option is to leave the alternate branch to work on later since switching back to the master branch will restore the previous state before branching. In this way using a branches is a simple and lightweight way to develop or experiment with new features.

Collaboration

Git is useful for collaborating on shared source code repositories. The collaborative development of the Linux kernel is the reason git was created by Linus Torvalds. A key feature of git is that it is distributed. You are not bound by working from a single source server. I have my copy of the repository and you have yours. I like the changes you are making so I clone your repository as a branch into my own. I can test out the changes you've made before merging them into my master branch. If I only want a subset of the changes you've made I can use the git cherry-pick command to merge only the changes I want. The website github.com enables a collaborative aspect of developing software with git. Github acts as a git server but also highlights the social links of branches between developers. Other developers' git repositories can be viewed and downloaded, but also forked into your own github space. This fork acts as a copy of the original repository with the relationship between the two repositories maintained. Github monitors the commits, merges and branching between repositories which can be viewed, compared, or visualised as a network.

Getting started

vim

Michael Barton — Tue, 23 Dec 2008 16:00:00 PST

About

Doing bioinformatics I only use two tools: the keyboard and the mouse. Since I use these tools all-day, everyday I want to use them efficiently. My opinion is that using the keyboard as much as possible while the mouse as little as possible, is the best way to work at a computer. In a simple example, knowing the keyboard shortcut to run a given command means that I can execute the command quickly without breaking the flow of typing. The text editor Vim takes this to another level and is entirely keyboard driven. There are no drop down menus and everything is performed using the keyboard; there's no reason for my hands to leave the keyboard. The large number of commands for Vim means there is a fair amount of practice required before you can use it fluently. I think this practice is a great investment though, as being able to use Vim intuitively makes you work faster and more efficiently. The reason for this is because Vim has a huge functionality to be taken advantage of, using just a few quick remembered keystrokes.

Features

As Vim is entirely text-based, without the pretty interface of modern editors, my original opinion was of an archaic hangover from the early days of Unix. Vim is around twenty years old, but is still a sophisticated text editor with a large range of functions. The large range of vim's functionality means a steep learning curve, but the extensive help documentation is an eloquent and gentle introduction. The commands ":vimtutor" and ":help" are the places to get started for using Vim. Vim's greatest feature is how easy it is to move, edit and manipulate text. This can sound trivial, but reordering paragraphs in minutes with a mouse, takes just seconds using the keyboard with Vim. This example is applicable to any type of text file, such as Ruby or LaTeX source code, which is what I spend most of my time editing. Another Vim feature are the registers, which act as super-charged clipboards. Not only can text be stored for pasting, but also sequences of commands as well. Stored commands can then be replayed by calling the register, this eliminates performing repetitive actions. Typing ":help usr_10" is a good place to start for learning about using registers for commands, as well as other ways of making large changes quickly.

Customise

One reason not to use Vim, is that it doesn't have the code orientated features of integrated development environments (IDE) such as Eclipse and Netbeans. But it does. Vim is easy to customise which has lead to a community of developers creating a large number of plugins. Code snippets, project drawers, and fuzzy file finding are just some examples of plugins aimed at using Vim as an IDE. Whatever language, file type, or framework you use, someone will have written a Vim plugin. This gives Vim all the functionality to rival any IDE.

latex

Michael Barton — Mon, 22 Dec 2008 16:00:00 PST

About

LaTeX (pronounced lay-tek) is a document creation system aimed at scientific and technical documents. LaTeX documents are written in plain text using markup to describe which parts should be sections tables or pictures. The LaTeX system parses the markup and formats the text to produce either dvi, postscript or pdf output. As LaTeX is entirely text based, the content can be tracked using a version control system. Plain text files mean that you can work on a document with your favourite editor, and can also be manipulated at the command line using Unix tools. The syntax of LaTeX markup will take an hour or two of practice to learn, but the advantage of creating documents from a marked up source is that the results are consistent and reproducible, which isn't always the case for graphical document editors.

Features

The main reason for using LaTeX is that it allows you to work on the content of the document, not the formatting. If you were using a graphical editor you format the text as type, but with LaTeX you only have to add the markup to the document and LaTeX takes care of the rest. This can save a lot of time with large documents. The basic features of LaTeX include automatic generation of tables of contents, tables of figures and automatic numbering of sections tables and figures. BibTeX is the companion to LaTeX which adds simple organisation and addition of citations. Citations are added to documents using a simple "cite" command in the text, without the requirement for third party software. One of the benefits of creating documents using LaTeX is that the produced formatting is the result of best practices in typography and document presentation, which means LaTeX documents look better than the average.

Templates and Plugins

LaTeX is free software and available for most operating systems. There is a large LaTeX community which develops themes and modules that can be added to LaTeX documents. Many journals also provide LaTeX templates in which papers can be submitted. There are templates available for writing a thesis or dissertation, and there is likely a specific templates which follows your own institution guidelines. There are many useful third party plugins for adding extras to a document. For example beautiful formating of tables, grouping figures into subfigures, replacing text inside figures and even a framework for including R-code inside a LaTeX document.

about

Michael Barton — Mon, 22 Dec 2008 16:00:00 PST

The aim of this website is to try and describe the what the best practices are in the field of bioinformatics. I used to write at this web address as a blog, but with less time in my PhD I had less time to write. So instead I've converted this blog into an online book that anyone can read, share and contribute to.

Sharing

All content on this site is free to distribute under a Creative Commons license, and you are encouraged to take and share the material with anyone who who you think may find it useful. For example material can be used verbatim into the teaching material for a bioinformatics degree or course.

contribute

Michael Barton — Mon, 22 Dec 2008 16:00:00 PST

Suggestions to Bioinformatics Zen are welcome and gratefully received. The single requirement is that any contributions you make become part of Bioinformatics Zen to be distributed freely under a Creative Commons license. Any contributions are automatically acknowledged on the site.

The easiest way to make a contribution is to sign up for a free Github account, and fork the Bioinformatics Zen content repository. You can make any edits using the online Github editor, which should be written in the simple markdown syntax. Once you make your changes click the "Pull Request" button and I'll be notified of your contribution.

If you feel adventurous or prefer not to use GitHub, contributions will be accepted from any publicly accessible git repository. Please keep commits to one change per file for the purposes of cherry picking and draft management.

Bioinformatics Zen

how to use a database

Code

using a database

SQL is hard

scripting

Managing dependency

Light and fluffy

why write good software

reuse, contribute, create

Use existing code

Contribute to existing code

Create new code

writing good code

Be too descriptive

DRY

Books and frameworks

Further reading

git

About

Features

Collaboration

Getting started

vim

About

Features

Customise

latex

About

Features

Templates and Plugins

about

Sharing

contribute