PS: It gets better.

Listing objects in your global environment
A simple ls() doesn’t really tell you enough useful information at a glance. Most often I just want to know what I named certain data.frames or functions. This handy little function, called as lsa() will do that for you:

lsa <- function()
 {
    obj_type <- function(x) { class(get(x)) }
  foo=data.frame(sapply(ls(envir=.GlobalEnv),obj_type))
  foo$object_name=rownames(foo)
  names(foo)[1]=&quot;class&quot;
  names(foo)[2]=&quot;object&quot;
    return(unrowname(foo))
}

Listing all functions in a certain package
This can be called with lsp(). The pattern argument will allow you to quickly find the right function if you vaguely remember the name.

lsp <-function(package, all.names = FALSE, pattern)
{
  package <- deparse(substitute(package))
  ls(
      pos = paste("package", package, sep = ":"),
      all.names = all.names,
      pattern = pattern
  )
}

Be sure to throw them both in a new environment (i.e. not the global one) so they don’t get accidentally removed when you clear your variables.

1. Load frequently used packages
   These days I never run R without having to use ggplot2 or plyr so I just include that here (although I hope that someday both packages will become absorbed into the R core).

   library(ggplot2)
   library(plyr)
   

2. Create aliases for frequently used functions

   # Shorten S3 methods so s(obj) instead of summary(obj)
   s <- base::summary;
   h <- utils::head;
   n <- base::names;
   

3. Set your preferred repository

   Hate the menu that asks you to choose a repository when installing a package? Just hardcode it.

   # Get your current repo name
   current_repo <- getOption("repos")
   # change this to your closest one
   current_repo["CRAN"] <- "http://cran.us.r-project.org"
   options(repos = current_repo)
   

4. Create a new environment so you don’t lose your custom startup functions

   I always start a new script with rm(list=ls()) to clear out everything. The unfortunate consequence of this is that it also takes out all the cool new functions from your .rprofile. Get around that by creating a new environment and putting your functions there.

   custom_env <- new.env()
   # If you don't want to clutter this file, leave functions elsewhere.
   sys.source(".my_custom_functions.r", envir = custom_env)
   attach(custom_env)
   

You can also set a range of other options but these are a good start.

Update: As Jason Priem astutely points out, these tricks can impede reproducibility of your work (especially if you fail to load the appropriate libraries & functions in your final script). While these are valuable time savers during the development phase, you certainly want to be more thorough before sharing your code.

The meeting was jointly organized by folks from Nimbios and the remote data analysis and visualization group (rDAV). The idea behind the workshop was to introduce biologists dealing with big-data problems to a variety of analytical (mostly just R) and visualization tools (R and a few other open-source tools). The presentations were either technical (HPC resources, tools, demos) or application oriented.

Of the technical talks (HPC intro, utilities), the one I found most valuable from the workshop was by Pragnesh Patel from rDAV who did an excellent job outlining all the ins and outs of running R on a cluster. Slides from his talk are available here. A more recent summary from his UseR! 2011 presentation is available here.

On the application side, there were a couple of talks from Nimbios scientists. One by Michael Gilchrist on Evolutionary bioinformatics (pdf of slides) and the other by a Nimbios postdoc, William Godsoe on using hpc to build species distribution models [cite]10.1093/sysbio/syq005[/cite].

In addition to R, we also discussed other open-source tools for visualizing large datasets.

• Scientific visualization using VisIT [pdf] - A tutorial for using Visit. There is also a wiki.
• Paraview - Another visualization tool. Both tools can take advantage of HPC resources.
• A tutorial on R visualization – This wasn’t HPC specific. Mostly just examples on how to use ggplot2.

Although I wrote a detailed post on how to use Amazon’s EC2 cloud for HPC, this workshop convinced me to use resources that NSF already provides. Teragrid is a portal that provides access to numerous cluster resources funded by NSF or one of its partners.  Using their XSEDE portal (which has replaced POPS), academics can request an allocation for computing time. For new and exploratory projects, there are ‘starter grants’ where one can get a rather generous allocation within a fairly short time. Larger allocations involve a review process. If the efforts you currently seek time for are being actively funded, the review process is likely to move through faster since it has already been favorably reviewed. Amazon’s computing cluster is still a useful service but there is no need to spend grant money elsewhere when NSF already provides these resources. As more scientists use and acknowledge Teragrid’s resources in their publications, that will provide the incentive and justification for organizations like rDav to continue seeking funding, especially in todays budgetary climate.

Talks

1. John Wilbanks (Creative Commons) presented a great keynote.
   

   Knowledge is a public good and increases in value as the number of people possessing it increase.

   slides

2. Heather Piwowar spoke on data citation challenges
   

   As more datasets become citable, fewer papers will be cited as proxy which is the current practice.

   Related: Data archiving is a good investment [cite]10.1038/473285a[/cite]

3. Jason Priem on altmetrics
   

   Todays journals are still the best scholarly communication system possible using 17th century technology!

   Related: Altmetrics Manifesto

Journals

Gigascience [@] - a new open-access journal that aims to revolutionize data dissemination, organization, understanding, and use. The journal will also publish large datasets alongside papers (also independently?) with citable DOIs

Open Network Biology -  Another new journal that links papers to datasets

Repositories

Pangea – repository for earth sciences and environmental data

Miscellaneous

Nanopub

DataCite Metadata schema for citing research data [pdf]

Other summaries of the meeting

William Gunn’s storify summary

Carl’s summary – day 1, day 2 

Tweets from the meeting

On that note, I’m pretty excited by the announcement of a new R package called RNCEP in the current issue of Methods in Ecology and Evolution [cite]10.1111/j.2041-210X.2011.00138.x[/cite]. The package provides an interface to atmospheric data from National centers for environmental prediction and NCEP/DOE. By encapsulating all the steps from data acquisition and format conversion to interpolation and aggregation from within R, the package provides a way to document an entire workflow as part of an article supplement. As more data repositories open up APIs, similar packages will go a long way towards promoting more open science.

Pinboard -Bookmarks are an essential part of any research/teaching effort. I frequently bookmark pages for field work supplies (to buy later), grant announcements (that I plan to apply for in a future round), tutorials (frequently R related, useful StackOverflow answers), workshops (especially ones with useful handouts), syllabi (ideas for future lecture topics), teaching material, and even papers to read later. Compared with other alternatives, Pinboard is a no nonsense (yet feature rich bookmarking tool). Among all its features the killer (in my opinion) is archival bookmarking ($25 annual fee). The service saves a cache (including images , videos, and overall formatting) of any page you bookmark. This is great for links that change, expire, or disappear long before you actually need to refer to the content.

Pinboard also allows you to automatically archive items from Google reader (any articles you star are saved automatically), Twitter (your own or anyone else’s with the added option to bookmark links alone), and Instapaper (see below).

Instapaper – One can only read for so long a day on a computer and carrying around dead-tree copies of articles gets old real fast. Although I am yet to hop on the ipad bandwagon, I’ve found instapaper to be a great tool to send dense review articles to my kindle for later reading when I am away from internet distractions (Kindle’s painfully slow browser is a blessing).

iEtherPad - Collaborate on notes in realtime (either publicly or privately with a group of collaborators), with authorship colors and automatic versioning. iEtherpad also lets you create a subdomain for your research group.

Mendeley groups – The killer feature of Mendeley, in my opinion, is the ability for small groups of collaborators to share PDFs and cite directly from it. Changes made by any collaborator are automatically pushed to everyone else!

Dropbox (and its alternatives): Although dropbox appears to be the de facto file sharing platform for most researchers, there are also other great alternatives such as Spider Oak (way more secure than dropbox) and Sugar Sync (more free space to begin with) if you end up running out of (free) space. Dropbox, however does increase space with referrals (more for academic users). This Prohacker post explains all the details.

Useful dropbox hacks:

1. DropDAV – A webDAV interface for Dropbox
2. Dropbox Forms – Receive assignments from students and data sheets from your research assistants/collaborators straight into your dropbox
3. Add a widget for a quick drag n drop (OSX users only)

Did I miss something really useful? Add it to the comments below!

Initially I was turned off by the learning curve associated with the ins and outs of setting up R on EC2 but eventually I set up my own Ubuntu box with R, all of my packages and customizations, and saved that as a 64-bit AMI capable of running high memory quadruple extra large instances. This set up has worked really well for me over the last few months.

With the recent release of RStudio, and Rstudio server, I’ve been toying with the idea of running it on an EBS backed instance. Inspired by JD’s tweet, I got around to setting mine up this weekend. Here is a quick walk through.

Assuming you’ve launched and used EC2 services before, start out by launching a newer version of Ubuntu (I’m running 10.04 Lucid) and install the current release of R (2.13).

Next, install RStudio server by following the instructions here (be sure to follow 64-bit).

Once successfully installed, create a new user like so:

sudo adduser username

At this point, be sure to go change your EC2 security group to allow port 8787 on TCP.

If the instructions so far seem complicated or if you’d rather not start from scratch, you can follow instructions here to launch an existing AMI with Rstudio server compatible versions and take it from there.

Next, launch Rstudio from the server using your instance DNS like so:

http://ec2-75-102-193-170.compute-1.amazonaws.com:8787

(be sure to replace the DNS above with your current DNS from the EC2 Dashboard)

Next, login with the username and password set earlier and if everything worked, you should see something like this:

Next, install all the packages you would like. If you require Java backed packages such as glmulti, go ahead and set up Java from the terminal.

After that, you can easily (using GUI menus) save this customized instance by following instructions here. Voila. From now on, whenever you need to run a high-memory instance of R, just launch new instance, choose My AMIs, and once launched, connect to it via the browser using the current DNS. Brilliant!

installed<-installed.packages()
available <-available.packages()
ia <- merge(installed, available, by="Package")[,c("Package", "Version.x", "Version.y")]
updates<-ia[as.character(ia$Version.x) != as.character(ia$Version.y),]
updates

If you would like to install every available update:

update.packages()

If you would like to keep up on new packages being released, I highly recommend following @CRANberries