Bracing against the wind

gethostbyname command line

noreply@blogger.com (Erik Aronesty) — Fri, 09 Mar 2012 12:55:42 PST

Pasteable script below. I can't believe this doesn't exist. No command line tool to get a host name using the resolver on linux?


#!/usr/bin/perl
use Socket;

$host = shift @ARGV;
die("usage: gethostbyname hostname\n") unless(defined($host));

$packed_ip = gethostbyname($host);

if (defined $packed_ip) {
    $ip_address = inet_ntoa($packed_ip);
    print "$ip_address\n";
    exit 0
} else {
    warn "$host not found\n";
    exit 1
}

Google Scholar Bookmarklet

noreply@blogger.com (Erik Aronesty) — Fri, 09 Mar 2012 22:43:44 PST

This link is a bookmarklet which takes the current query string in google, OR the current selection (if there's no q= parameter), and searches google scholar. Now that Google has removed scholar from the dropdown (sad), this is necessary for me to work the way I've been working.

As predicted

noreply@blogger.com (Erik Aronesty) — Fri, 09 Mar 2012 22:57:48 PST

As predicted 6 months ago, and again last week, Oxford Nanopore is shaking up the industry. Pacbio, Complete Genomics, Life technologies and others are all seeing their stocks drop 5% on the news that sequencing a genome just got 10x cheaper.

Church-Turing thesis and strings

noreply@blogger.com (Erik Aronesty) — Mon, 30 Jan 2012 13:08:25 PST

My empirically determined corollary to Turing's computability thesis is that any problem, no matter how interesting it seems on the surface, can be reduces to a deja-vu inducing set of string handling, vector sorting and hash lookups.

And as new features are added, the problem of writing out and reading files slowly becomes most of what that program does - until the program resembles an actual Turing machine: ploddingly scribbling and reading things from an infinite tape which, because of flooding in Thailand, is infintely expensive.

Embed Tab Size Info in Source Text

noreply@blogger.com (Erik Aronesty) — Mon, 30 Jan 2012 13:14:03 PST

IDEA: a "universal tab stop signature" line you can add to any text file in which tabs may be found and need to be rendered in an editor.

The sig must be in the first or last 128 bytes of a file (so reading the sig is faster)

The syntax could be something like: open paren or brace, % or # sign, open paren or brace, the word "tab" then a ':', then anything except a closing/matching paren or brace, then a percent and then another closing paren/brace. [(\[][%#][(\[]tab:[^\)]*[%#][(\[]

Examples:

C++
// (%(tab:4)%)

Perl:
# (%(tab:4)%)

HTML (for source editing only):


LISP (no parens... too confusing):
; [%[tab:4]%]

SMX (no % signs or parens):
%null([#[tab:4]#])

Uncovering significant miRNA species with RNA-seq

noreply@blogger.com (Erik) — Fri, 23 Dec 2011 14:08:41 PST

Note to self: An easy way to uncover significant miRNA species with Illumina's RNA seq data is to run counts on the small RNA between two cell types, then toss out anything that appears ribosomal, filter for some minimum threshold and run a negative binomial distribution test on them like edge-R or DEseq. The resulting adjusted p-values can illuminate new miRNA species.

Xenograft & Contamination Filter

noreply@blogger.com (Erik Aronesty) — Mon, 19 Dec 2011 07:18:05 PST

Useful code that uses libbamtools to filter out alignments to another genome. Tested with xenograft and bacterial contamination removal...works great.

Connecticut Martin

noreply@blogger.com (Erik Aronesty) — Fri, 21 Oct 2011 01:46:16 PDT

Martin Hedge paused to wipe his glasses with the sleeve of his shirt. The crowd at the coffee shop had gotten louder.

It had never been enough. Even though idle wealthy housewies had long since taken up the pen and written libraries of wisdom and humour, and the girls hanging out at the tattoo parlors had discovered the zen of automotive engineering. Even though the unemployed had long ago sought out physics texts and wrangled secrets of a new energy density theory, it still hadn't been enough. The question remained... what were we working for?

The crowd at the coffee shop, from here, seemed angry, but as Martin walked closer he could see familiar faces, sternly triumphant.

Non "allelic" variation - thinking out loud

noreply@blogger.com (Erik Aronesty) — Thu, 22 Dec 2011 20:37:13 PST

(Later Note: I wrote this before I knew what it's called. The term most people use is "Somatic Mosaicism". Apparently this is a pretty well researched topic... so I can go back to all the biologists that looked at me like I was crazy and tell them .... hmm.)

Here's a link to a good article on the topic:

http://www.sciencedirect.com/science/article/pii/S0168952511000436

Original rant below...

Much of genetics is concerned with "alleles" and "variations". That is, an organism is assumed to be comprised of "one kind of dna". That DNA has 1) inherited alleles from it's parents, and 2) de-novo alleles (also called somatic mutations). The 1000 genomes project estimates at about 30 or so per person. This is probably a very conservative number considering the nature of the 1000 genomes project. IE: all of these mutations are whole-organism, detectible, validated mutations. E-coli error rate estimates would put the range at 30-300, and this might be a better estimate because of how the study was done. Human blastoma cells have has error estimates accurately measured at 10 times that rate. Individual organs may be less sensitive to immune response correction.

But let's assume 30 is our number. It's nice and small. And it's good to have a lower-bound.

That is only the set of variants that went into the "first cell" (fertilized egg) of an organism.

When that egg divides, half the organism has another, different, set of mutations. So 100% of the organism has 30 de-novo mutations and 50% of the organism will have *another* 30 de-novo mutations (30 new ones in that 50%, plus 30 original).

But wait, there's more. When those 2 cells divide in half again, you now get 60 new mutations, 30 from each cell. These 60 will be detectible at the "25%" level ... IE: 25% of the final organism will have them.

High-throughput sequencing can readily detect variation at the "1%" admixture level. That is, commonly detect variation when as little as 1% of the cells have that variation.

So how much variation can we expect, based on a low de-novo mutation rate, detectible at the 1% level?

100%->30, 50% ->30, 25% ->60, 12.5% ->120, 7.25% ->240, 3.12% ->480, 1.6% -> 960

So we can expect about 1000 de-novo variants in a healthy individual, or 32 times the mutation rate. But what if the somatic mutation rate is higher, say, 3000 variants per replication? This may be the case in some organ development.

Thus, at the 1% level, would that be 96000 non-inherited detectable variants. I would call that my "upper bound". In real pileup data... I see around 30% "non-allelic" variation. So if, say, you've got 15000 SNPS (a reasonable number), we would expect 5000 "background" snps.....putting the mutation rate at "156.25" (5000/32). That's smack in the middle of the e-coli based estimate.

Lots of variant callers filter these out.... but I'm interested in them... i think they may be a lot more important than people think.

Multiplayer TD

noreply@blogger.com (Erik) — Fri, 30 Sep 2011 20:24:47 PDT

Looks fun. Seems like it would be cool to play against other players... but they are rarely there. The developer needs to link-in to Armor or Kongregate or Facebook or some other PVP network.

There's probably a market for that... build a PVP portal so that lesser known games that are still fun can have a base of players.

Getting perl readline to work in Ubuntu

noreply@blogger.com (Erik) — Fri, 30 Sep 2011 11:21:22 PDT

I use perl's debugger and psh (simple perl shell). When the up arrow doesn't work, I always forget how to fix it. Hopefully by posting the solution here, it will be easier for me to find:

apt-get install libterm-readline-gnu-perl

I use perl's debugger and psh (simple perl shell). When the up arrow doesn't work, I always forget how to fix it. Hopefully by posting the solution here, it will be easier for me to find:

sam-stats

noreply@blogger.com (Erik Aronesty) — Wed, 28 Sep 2011 13:06:46 PDT

Put up a useful summary statistics from SAM or BAM files. TODO: rewrite in C so it's fast.

Should we try to contact ET?

noreply@blogger.com (Erik Aronesty) — Tue, 23 Aug 2011 15:01:16 PDT

Would it be safe to meet an intelligent alien lifeform from another planets?

Let's assume an ET (extra terrestrial, alien) is more advanced than we are. Let's assume, again, because it's the only experience we have, that we are a good example of an intelligent life form.

Have other species benefited from their contact with humans? Species that are useful to us, like cows and dogs have proliferated in population, but are controlled/used by us. Species that feed on our leavings, like roaches and pigeons do well.

But, given that we're intelligent, and we'd like to think that aliens would regard us as so, we may want to look only at "higher" mammals, with comparable intelligence. Primates certainly don't do well when humans show up. In fact, iut seems we are particularly brutal when dealing with them. Large mammals were nearly driven extinct from North America after human contact. Some very intelligent species, like right whales, with rich communication systems and highly social habits, were driven to near extinction.

OK, but maybe we're talking about "modern man". Maybe we've escaped that brutal past.

We still round up dolphins to kill them. And, to this day, very little effort has been spent in attempting to understand the language and society of the "alien species" we share our home with. Any time even a modicum of effort has been spent it's been met with "shocking" revelations about how other species have complex grammars, notions of fairness, etc. And still most people in the world refuse to believe that other species can feel pain the way humans do - largely (IMO) as a way of justifying abuse and mistreatment.

So.... I would expect to be treated at least as well as we have treated others. Accordingly, someone should *shut down* the SETI program. ASAP.

Oracle fails to crush open source

noreply@blogger.com (Erik Aronesty) — Fri, 05 Aug 2011 01:24:46 PDT

Oracle has repeatedly attempted to acquire it's way into the open source movement, each time charging and licensing technologies it had no hand in developing. Berkeley DB, Sun Grid Engine, Open Office, MySQL, and Java (JRE/JDK), are now all Oracle licensed technologies.

What is admirable is that they make no pretense of being "friends" of open source. They stopped releasing new versions of SGE, and transitioned to a closed-source system immediately. Similar moves are happening in other products at different speeds... with a "what the market will bear" approach.

It took me 6 months to move all my BDB code to SQLite, switch SGE stuff to condor. I never touched MySQL because of the InnoDB creepiness and I'm glad the decision has been vindicated... Postgres was the obvious choice. Oracle's falesly named "Open Office" has thankfully been forked to become the truly open "Libre Office".

The only thing left is Java. And there's nowhere near an adequate replacement. Open JDK is limited to a handful of operating systems. And Java itself may have some questionable licensing - Oracle may be able to seize the whole thing by the fistful.

I've despised Java from the beginning, mostly for it's poor architecture and the way it encourages bad coding (not as bad as python). Watching Oracle roll in and step all over it is, for me, merely another "I told you so" moment.

View the internet as if you are Google

noreply@blogger.com (Erik Aronesty) — Fri, 29 Jul 2011 08:37:20 PDT

Many companies present a different face to search engines than they do to users. It's sometimes interesting to see the internet as if you are a search engine:

To change the Chrome user agent string, add the –user-agent option switch to the chrome.exe program shortcut:

chrome.exe --user-agent=" Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Correlation Heat Map

noreply@blogger.com (Erik Aronesty) — Wed, 22 Jun 2011 09:08:30 PDT

Illumina Quality Score 'B'

noreply@blogger.com (Erik) — Fri, 10 Jun 2011 12:52:20 PDT

When analyzing about 100 PhiX control lanes, by aligning to PhiX and calling any reference mismatch an "error", the quality score 'B' performs higher than 'C' on average. The 'B' score is approximately abot 3 phred-scores lower than the mean score for the whole run. I think the 'B' score represents the equivalent of "I don't know" on the Phred scale. For programs that rely on quality scores to indicate actual likelihoods of error, the 'B' score should be treated as a special case.

This is in addition to the known issue that reported phred scores have higher ovserved error rates overall, especially at the top end of the scale (Q-40's perform like Q-37's should, etc.)

Lightweight Job Queuing Engine and GitHub

noreply@blogger.com (Erik Aronesty) — Sat, 14 May 2011 20:15:29 PDT

I wrote a job queuing daemon. It sits on a head node and any number of compute nodes. Good with EC2. Replacement for condor, OGE (formerly SGE), torque, etc. So far, it's good about crash recovery, very good about latency. Other job queuing systems seem to be sooo slow with latency, you never want to use them. And they all spend so much time worrying about copying i/o files all over the place, when all I ever use is NFS anyway.

https://github.com/earonesty/grun

Written because those other systems are administratively difficult, and have more features than I need. Some day I may choose to go back to condor, or SGE, but for now, it's nice to have a smaller, faster, easier one.

I'm trying github instead of google code. The developers and gangla got me to try it out. I think I might like it. Not sure why it makes you "add" and "commit" separately.

I based the socket system off of ppcgid, since it worked so well in the past.

Dealing with FASTQ's

noreply@blogger.com (Erik Aronesty) — Sun, 01 May 2011 17:37:15 PDT

I posted some code for dealing with FASTq's. They're all written in C++, and each one replaces a tool that's between 3 and 50 times slower. The scoring metric they use for "anchored sequence alignment" has been improved. Also, they contain lots of "do what I mean" features, allowing the user to focus on the results, and not on the mechanics of processing sequencing data.

The essential tasks they accomplish are sequence quality filtering, adapter/primer removal, demultiplexing and paired-end joining (stitching):

They have to be compiled, but they are each "monolithic" ... and require no libraries, headers, etc. I tested fastq-join on Linux and Windows, so I know it compiles easily. The other 2 tools have only been tested on Linux.

See: https://code.google.com/p/ea-utils/

New aligner

noreply@blogger.com (Erik Aronesty) — Tue, 15 Mar 2011 10:03:31 PDT

I'm working on a new aligner i have that doesn't use BTREE. It's fun. Not hard, and finally working.

F-Test for Comparing Variability

noreply@blogger.com (Erik Aronesty) — Sun, 06 Mar 2011 02:43:15 PST

I left my textbook at home, and had to search for a while to find a decent description of how to use an F-test (even though it's somewhat controversial), so I'm posting it for future reference. Wikipedia stuff on statistics appears to be written by people who forget it's supposed to be human-readable.

GATCGGAAGAGCGGTTCAGCAGGAATG

noreply@blogger.com (Erik Aronesty) — Wed, 23 Feb 2011 11:40:58 PST

I knew it was some kind of primer, adapter or barcode - but I wasn't sure which, whether to trim, or if the barcode came before or after.

BLAS*T searches turned up nothing, but Joel answered, within a second, "it's an adapter sequence". He said "I just Googled it".

Reminder to self... Google searching DNA sequences works (NOTE: Bing didn't).

Approximate Line Count for Very Large Files

noreply@blogger.com (Erik Aronesty) — Thu, 10 Feb 2011 03:21:58 PST

When dealing with very large files, the unix tool "wc" can be extremely slow. The alternative, byte size, is often not what I want to look at, especially when trying to estimate the number of reads in a fastq file.

A good estimate (2 sig figs) is, 90% of the time, what I need.

alc is my "approximate" line count tool. It counts the number of lines in a file, just like wc, except it only "samples" the file in a series of segments. By seeking and reading 200K from a dozen places in the file, rather than reading the whole thing, I get a good representative sample, and an accurate-enough count.

A True Teflon Alternative

noreply@blogger.com (Erik Aronesty) — Fri, 04 Feb 2011 18:29:18 PST

We went looking for a nonstick solution that wasn't coated in Teflon or PFTE, or whatever people try to sell it as. Apparently it causes developmental disorders in laboratory animals (click for EPA advisory). I can only wonder what it does to human kids.

The first thing we found was Swiss Diamond... which is a complete scam. Glad to have avoided that before making an expensive mistake, I scoured reviews and sites and we found the Aeternum and Green Earth line of ceramic coated pans (like porcelain, it seems).

I bought the Aeternum and it works exactly as well as advertised. Nonstick, and not made with creepy chemicals. If it lasts or breaks, I'll report it here.

Uncuddled Else

noreply@blogger.com (Erik Aronesty) — Sun, 19 Dec 2010 12:19:10 PST

I don't care how cute the name is. It wastes precious vertical space while adding nothing to readability. Worse yet, the uncuddled else substracts from the implication that it is a part of the prior "if" statement... which it is.