Depth-First

Big Data in Chemistry: Mirroring PubChem the Easy Way Part 2

2010-02-09T07:20:50-08:00

One of the useful (and unnerving) things about running a blog is that you're forced to face what you don't know (usually very publicly). I've been looking for the simplest way to maintain an up-to-date local copy of PubChem. I previously posted an article describing one way to mirror PubChem through the use of rsync and curlftpfs.

Although this method works, it turns out that there's an even simpler way to do this with wget. For Compounds:

$ wget --mirror --accept "*.sdf.gz" ftp://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/

For Substances:

$ wget --mirror --accept "*.sdf.gz" ftp://ftp.ncbi.nlm.nih.gov/pubchem/Substance/CURRENT-Full/SDF/

The --mirror option bundles several options together relating to timestamping. This is how wget will be able to download only the updates to the PubChem archives directory, rather than downloading the entire PubChem archive every time.

Without any options, the default behavior of wget is to not preserve timestamps and to force a complete download of all files every time.

The --accept option says we only want to download gzipped SDF files (leaving out, for example, text files).

Whenever you're ready to update your local PubChem archive, simple run the two commands above and you're done. You'll have a copy of the PubChem dataset that matches - to within one day - the dataset being used by NCBI itself.

Pretty simple, huh?

Now if I could just figure out how to use a single wget command to mirror both the Compound and Substance directories...

Big Data in Chemistry: Mirroring PubChem the Easy Way

2010-02-08T06:59:18-08:00

PubChem's massive size presents special challenges when working with this chemical dataset. Synchronization in particular requires special care. Although it's very easy to use a tool such as wget to perform a complete, one-time download PubChem's archive files, this approach scales poorly if our goal is to maintain a copy that's always up-to-date. The PubChem dataset's substantial size makes it impractical to download frequently, and especially problematic when an up-to-date local copy is needed quickly.

This article describes a simple way to create a low-maintenance, low-bandwidth, up-to-date local mirror of PubChem using two Unix tools.

What It Does

The method described here will create two directories on your filesystem that will exactly mirror the contents of the PubChem Compound and Substance archives, respectively. A simple command, which can either be run as a nightly cron job or on demand, will efficiently bring these local files up-to-date with PubChem whenever it's run.

Step #1: Create A Workspace and Mount PubChem FTP Site

We're going to need a workspace. In this workspace, we'll first create a mountpoint for the PubChem FTP site archives, then we'll mount the archives:

$ mkdir workspace
$ cd workspace
$ mkdir -p ftp.ncbi.nlm.nih.gov/pubchem
$ curlftpfs ftp.ncbi.nlm.nih.gov/pubchem/ ftp.ncbi.nlm.nih.gov/pubchem/

My Linux distribution (Ubuntu Karmic) gives me the error message:

fusermount: failed to open /etc/fuse.conf: Permission denied

which doesn't seem to matter. The FTP site is mounted, as I can see by listing the top-level entries:

$ ls ftp.ncbi.nlm.nih.gov/pubchem
Bioassay  Compound     data_spec     README      Substance
CACTVS    Compound_3D  publications  specifications

We can unmount the PubChemFTP site with fusermount:

$ fusermount -u ftp.ncbi.nlm.nih.gov/pubchem/

Step #2: Create Synchronization Directories and Transfer Files

Next, let's create two directories to hold the PubChem files - one for Compounds and one for Substances:

$ mkdir substances
$ mkdir compounds

Now comes the magic. We'll use rsync to copy the contents of the mounted FTP archive into each of our local directories. First, we synchronize the Compounds:

$ rsync -r -t -v --progress --bwlimit=500  --include='*/' --include='*.sdf.gz' --exclude='*' ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/ compounds

This is going to take nearly 24 hours.

The option --bwlimit sets the maximum bandwidth (in Mb/S). The --include and --exclude options say that we're only interested in gzipped sd files.

We synchronize Substance records analogously:

$ rsync -r -t -v --progress --bwlimit=500  --include='*/' --include='*.sdf.gz' --exclude='*' ftp.ncbi.nlm.nih.gov/pubchem/Substance/CURRENT-Full/SDF/ substances

This command will take even longer to run.

Step #3: There Is No Step #3

That's really all there is to it. Every time we run the rsync command, we'll synchronize our local copy of the PubChem archive with the one on the PubChem FTP server. PubChem ensures that these archives are always current, so every time we synchronize, we'll have up-to-date files.

Why RSync?

RSync ensures that our synchronizations will be as efficient as possible by only downloading the archive files that change. From time to time, old records are updated in PubChem, and these changes appear as a new archive file that replaces an old archive file. The new file gets an updated timestamp. If you check out the Compounds FTP directory you'll notice several different timestamps reflecting the various updates of existing records. New records appear as new archive files.

The genius of RSync is that it performs an incremental backup; files that haven't changed since our last update are never downloaded.

We can even take this incremental backup idea one step further. Although I don't yet know if PubChem supports it, it's possible to create GZip archives optimized for rsync. This uses a variant of the GZip compression algorithm that makes it possible to transmit only the section of a gzip file that's actually changed, keeping network traffic to an absolute minimum.

This rsyncable archive capability is built into most gzip binary distributions.

Conclusions

Creating and maintaining your own up-to-date, verbatim copy of PubChem is both simple and inexpensive. The trick is to first mount the FTP archive using curlftpfs and then use rsync to perform an incremental backup of the mounted archive. The method described here works equally well as a cron job or an ad hoc command.

Credits: Mirror an FTP Directory with RSync and Curlftps; Rsyncable gzip

I Dare You: Ask Your Toughest Experimental Chemistry Question on Chempedia Lab

2010-02-06T08:25:20-08:00

Nature News is running a story on Matthew Todd and his initiative to develop a more practical treatment for Schistosomiasis by thinking different:

"My funded project is intended to be the kernel, to which anyone can add," Todd says. He hopes that the project will become a successful example of open-source science, and open-source 'wet lab' chemistry in particular, a concept that has been slow to take off.

Call me an optimist, but the problems with getting something like this to work will have less to do with a scarcity of volunteer-minded chemists and more to do with finding them around the world and connecting them to each other.

Chempedia Lab is a service that might have a role to play. It's a question and answer site dedicated to experimental chemistry. Ask a question and get a peer-reviewed answer. No inflated bureaucracy, no lengthy review process, no unaffordable subscriptions, no conflicts of interest, no nagging questions about re-use, no counterproductive rewards system. Just you, your peers, and the information - the way science is supposed to work.

Maybe you're thinking that something like this can't possibly work. If so, I'll leave you with a simple challenge - do the experiment yourself. Ask the toughest question you can think of and see how long it takes to get either exactly the answer you were looking for, or an answer that puts you on the right track. Then ask yourself how you would have answered the same question without Chempedia Lab.

Although Chempedia Lab may not be the best platform, one thing is clear - open science has no chance in the context of traditional scientific communication. That system is simply too cost-ineffective, both in terms of money and time.

My guess is that for every Matthew Todd there are at least a hundred others who would like to start the same kind of initiative, but who feel they lack the funding, the lab space, the staff, or some other critical resource. Thinking different about everything in the way we do chemistry - from who does the research, to where it gets done, to the medium of collaboration - is the key.

PubCouch: Create Your Own Custom PubChem Subset

2010-02-04T15:14:27-08:00

If you've ever worked with the PubChem dataset, you may have found yourself wanting to create a custom subset that filters out certain records. This article, the fourth in a continuing series, shows a very simple way to create a custom PubChem dataset using PubCouch.

The Problem

I really like PubChem. It's the world's largest collection of freely-downloadable chemical structures and an excellent use of taxpayer dollars.

But PubChem has faced some tough tradeoffs over the years., one of the foremost being how inclusive it should be. In other words, when to say 'no' to a substance depositor. I won't rehash the details here, but suffice it to say that the technologies on which PubChem is based are limited in important ways (for example: organometallics).

As part of ongoing work to expand Chempedia, the free chemical substance registry, I became interested in the possibility of building a subset of the PubChem Compound registry that only contained structures that could be safely encoded by the MDL Molfile specification. Call it "PubChem: The Good Parts."

This database was likely to be huge and pretty non-relational. It looked like a perfect job for PubCouch.

A Solution

The software to solve this problem has been built into PubCouch. There are a couple of ways to run it, but I find one of the simplest is to use JRuby:

$ git clone git@github.com:metamolecular/pubcouch.git
$ cd pubcouch
$ ant jar
$ jruby -S rake compounds:snapshot

To get that last part working, you'll need to install JRuby. This is optional; you could also create an Ant task or use some other script. The point is that we're running a pre-packaged PubCouch task called "Compounds".

There's one more thing - you'll obviously need CouchDB installed, and you'll need an empty database called "compounds". The database name can be changed to fit your preferences.

Finally, the way this works is likely to change in the future. To be sure you'll be able to access the code describe here, please use this commit.

Filtering

After running the snapshot task, you'll see some output indicating Compound IDs being checked and written.

Not every compound is being written. Only those passing a specific set of requirements will end up in CouchDB:

No bond annotations other than 'aromatic'.
No multicomponent (disconnected) Compounds.
No undefined stereochemistry.
No charged species.

These happen to be my requirements - yours will probably differ somewhat. To change the applied filter, simply change the method Compounds.StrictFilter.pass. It's that simple.

Fine-Tuning

This is all pretty rough at this point. There are many opportunities to refine the code for flexibility and performance. For example, I initially experimented with CouchDB's bulk update capability, which compresses multiple writes into a single HTTP request. But this actually resulted in more memory/processor usage. My guess is that this was probably less due to CouchDB than it was to the JSON overhead in the JCouchDB library I'm using to talk to CouchDB. Your results may vary.

Conclusions

PubChem is an excellent free resource for raw chemical structures, if filtered correctly. This article showed how to create your own personal subset of PubChem using PubCouch.

PubCouch: Streams Aren't Just for Pipeline Pilot

2010-01-29T10:05:26-08:00

If you've been following along with the development of PubCouch, the CouchDB interface for PubChem, you've probably noticed that only a fraction of the code relates to CouchDB itself. What's the rest of it doing?

This article, the the third in a series on using CouchDB for PubChem data describes how PubCouch transforms PubChem's collection of gzipped archive files into a stream of structure-data records that can be processed as if it were one big SD File.

The Problem

If you want to work with the PubChem dataset, one of the first problems you'll face is how to import the data into your database management system (dbms). The PubChem FTP server contains a rather large collection of archive "bundles", which are simply gzipped SD Files of records within a certain ID range.

In most cases, importing the PubChem database will consist of sequentially reading every Compound and Substance record, applying the appropriate intermediate processing, and storing the result.

So, we have a mismatch in the way PubChem stores its data (multiple gzipped archives) and the way we want to process it (as one big SD File). And by the way, how about not storing a bunch of temporary files, but rather transfer data directly from the archive to our database?

A Solution

One of the reasons Java was chosen as PubCouch's development language is its built-in support for high-performance IO operations. InputStream, the foundation of this support, turns out to be a very versatile class enabling a variety of filtering and reprocessing operations on raw data streams.

Our FTP Client can return at most one raw byte stream from each archive file. By applying a set of filters on this stream, we can get pretty close to where we need to be:

wrap each InputStream in a GZIPInputStream
wrap the result in an StreamReaderReader
wrap the result in a BufferedReader to read line-by-line

These filters alone won't do the job - remember, we want to treat the entire FTP archive as one big SD File.

SequenceInputStream is just what we need. This nifty little class can make a series of InputStreams (i.e., the individual PubChem files) appear as one big InputStream.

Putting this all together, we end up with a chained series of inputs:

[InputStream -> GZIPInputStream] ->
SequenceInputStream ->
StreamReader ->
BufferedReader

We now have a BufferedReader that will for all intents and purposes look like we've just opened a massive SD File. Handing this Reader to an SD File processor will let us capture all Substance or Compound records using a simple conceptual model.

Conclusions

By using Java's support for stream chaining and transformation, PubCouch makes it possible to work with the PubChem FTP archive as if it were one big SD File. This turns out to be useful regardless of how you decide to ultimately represent and store the resulting records. There are still some rough edges in the implementation and possibilities for extending the concept (i.e., random-access), but the idea can be used on many other datasources, and in many other contexts.