tobym's posterous

Vim syntax file for dot language

Mon, 29 Aug 2011 22:44:00 -0700

Vim syntax file for graph dot language is here: http://www.ocaml.info/vim/syntax/dot.vim (This is more up-to-date than the one at http://www.opensource.apple.com/source/vim/vim-10/vim/runtime/syntax/dot.vim ). Add this to your ~/.vim/ftdetect/dot.vim for file type detection: augroup dot autocmd BufRead,BufNewFile *.gv set filetype=dot autocmd BufRead,BufNewFile *.dot set filetype=dot augroup END

Permalink | Leave a comment »

Autocomplete hostnames in bash gotcha

Wed, 24 Aug 2011 12:39:13 -0700

A while ago I figured out that bash can autocomplete hostnames that are in the hostfile (/etc/hosts). This works when you hit <Tab> after an @ symbol. For example, if you have something like "127.0.0.1 example.com" in /etc/hosts and in a bash prompt you type "ssh user@ex<Tab>", it will expand to "ssh user@example.com".

However, it occasionally seemed to fail. I just realized this is because bash will cache the entries in the hostfile, so if you open a terminal and add an entry to /etc/hosts, that session (and any that are already open) will not pick up the change! Any subsequently opened terminals will, of course. You can solve this issue with the stale terminals with "export HOSTFILE=/etc/hosts", which seems to reset the cache and give you the latest autocompletion options.

Permalink | Leave a comment »

K Combinator in Scala

Fri, 29 Jul 2011 14:56:00 -0700

The K-combinator "manufactures constant functions: (K x) is the function which, for any argument, returns x". I wanted to add some logging in the middle of a series of transforms of a Scala collection, and this seemed like the right way to do it. For Rubyists, it works like Object#tap.

object Functional {
  /*
   *  Adds the k-combinator to an object. This lets you apply a function to the
   *  calling object; useful for accessing entire collections in the middle of
   *  a series of transformations. This lets you do horrible, side-effecting
   *  things to mutable collections, so try to avoid that.
   *
   *  Example:
   *
   *    List(1, 2, 3).map(_ * 2).k{ x => println(x) }.filter(_ > 4)
   *
   *  Prints "List(2, 4, 6)" and returns List(6)
   */
  class KCombinator[T](x: T) {
    def k(f: (T) => Unit): T = {
      f(x)
      x
    }
  }
  implicit def addKCombinator[T](x: T) = new KCombinator[T](x)
}

This implementation pimps objects to add a "k" method that takes a function. It executes the function, returning the base object. This lets you do convenient things like peer into a chain of transformations. You can use it like this:

scala> import Functional._

import Functional._

scala> List(1, 2, 3, 4, 5).k(println).map(_*2).k(println).filter(_>4).k(println)

List(1, 2, 3, 4, 5)

List(2, 4, 6, 8, 10)

List(6, 8, 10)

res0: List[Int] = List(6, 8, 10)

Throw { x => logger.log("Something interesting about x"+x.interestingFact) } in there instead of just (println) and you can see an example of more advanced usage.

Permalink | Leave a comment »

Test Lists In-Order with Scala Specs

Fri, 08 Jul 2011 14:37:00 -0700

In Scala, you can test that two Lists are the same with sameElements:

List(1, 2, 3) sameElements List(1, 2, 3) // true
List(1, 2, 3) sameElements List(3, 2, 1) // false
List(1, 2, 3) sameElements List(1, 2, 3, 4) // false

So I wrote a test using Specs that was supposed to test that two lists had the same elements in the same order.

import org.specs.Specification
class FooSpec extends Specification {
    "these lists" should {
        "be the same" in {
            List(1, 2, 3) must have sameElementsAs(List(1, 2, 3)) // Pass, as expected.
            List(1, 2, 3) must have sameElementsAs(List(1, 2, 3, 4)) // Fail, as expected.
            List(1, 2, 3) must have sameElementsAs(List(3, 2, 1)) // ---> Pass, unexpected!!! <---
        }
    }
}

The trick is to use beTheSameSeqAs:

List(1, 2, 3) must beTheSameSeqAs(List(3, 2, 1)) // Fails, as expected :)

Permalink | Leave a comment »

Windows 7 Outlook Cannot Open Attachment (Solution)

Fri, 01 Jul 2011 12:33:12 -0700

Windows users, have you seen this error when trying to open a picture from Outlook?

Cannot create file: photo.JPG Right click the folder you want to

create the file in and then check properties on the shortcut menu to check your permissions for this folder.

The problem is that your “temporary internet files” folder is filled to capacity. Right-click on “My Computer”, click “Disk Cleanup”, and make sure the temporary folders are checked. Click Okay and a progress bar will pop up showing the status of the files being deleted. Double-click the picture in Outlook again and it will open!

(This is the significantly easier version of the same solution that I’ve seen on the interwebs, that says to open regedit and find the OutlookSecureTempFolder and to clear it yourself.)

Permalink | Leave a comment »

Selecting and Copying with Mouse in MacVim

Wed, 29 Jun 2011 11:54:08 -0700

In which I finally enable 1996-era functionality.

This has bothered me for a long time and I just figured it out. I started using MacVim a while ago but was unable to actually select and copy text like normal. I have had this mapping for ages:

vmap  "*y

which allows me to copy visual-mode-selected text to the system clipboard using Control-c (I haven’t been able to map the actually Command key). MacVim is supposed to give me that ability expect using the mouse and Command-C like every other Mac application. This never worked for me; instead it either did nothing (actually made some seemingly random text disappear because it was really just like hitting “c” in normal mode). Then I discovered this setting:

set mouse=a

That actually enables mouse support in any terminal that supports it, and MacVim is no exception. Just adding that simple set mouse=a gives me the expected mouse copy-paste. Finally I have caught up to 1996!

Permalink | Leave a comment »

MacFUSE and TrueCrypt for Snow Leopard

Tue, 21 Jun 2011 06:35:00 -0700

TrueCrypt requires the MacFUSE filesystem extension. Unfortunately, the latest officially released version (2.0.3) is way out of date. It will install just fine, but after you try to open an encrypted volume in TrueCrypt, it will fail with an error like Failed to load /Library/Filesystems/fusefs.fs/Support/fusefs.kext - (libkern/kext) link error or Can't load kext com.google.filesystems.fusefs.

The fix for me was to install the 64-bit version, 2.1.7 (beta). You can get TrueCrypt 2.1.7 here. The MD5 hash of that dmg is e01786990111e7065a89cedf0a66ebbb. There appears to be another version out there, 2.1.9, but there are scattered bug reports and 2.1.7 works great for me.

Permalink | Leave a comment »

Neo4j 1.3 community edition released under GPLv3 - same as MySQL

Wed, 13 Apr 2011 09:44:27 -0700

The excellent Neo4j graph database community edition has been released under GPLv3. This is the same license the MySQL has thrived under for years. The previous license license was too limiting for open-source developers to commit to neo4j. I believe there was a limit to 1 million nodes before one was forced to move to the commercial license. This is enough for small toy projects or to get a feel for the database, but not for a serious application or even a semi-serious nights-and-weekends project, and I think that for all its growth, this was a big barrier to adoption. Neo4j 1.3 Abisko Lampa supports up to 32 billion nodes/relationships and up to 64 billion properties -- all that is now available to any open-source developer. That's a big step up. I think neo4j's rate of adoption is going to skyrocket now.

Permalink | Leave a comment »

Google finishes ITA acquisition, turns flight search market into winner-take-all

Fri, 08 Apr 2011 11:07:29 -0700

Google's acquisition of ITA Software was cleared by the US Department of Justice. What does this mean for the many websites that use ITA's QPX flight query service? QPX searches a billion combinations of flight fares, schedules, and availability per query; this heavy lifting makes the nice, fast interfaces of Kayak and Hipmunk possible.

For the near future, nothing will really change since Google is supporting ITA's existing commitments and allowing customers to extend their licenses through 2016. After 2016, I imagine it will become a winner-take-all market because Google will just buy the best interface and shut off access to everyone else. Starting now, it's a race to see who can build the best interface to the QPX data. I believe Google would prefer to acquire based on quality of interface, not number of users, as they already have an enormous number of users.

This changes the rules of the flight deal search game, then. Rather than trying to acquire users to generate revenue, a company might want to focus 100% of its energy on building the flat-out best interface imaginable. Nobody is going to dump a large amount of money to acquire users in a market with a 5-year expiration date. This opens up possibilities for small players to out-innovate the existing websites and win the grand prize, being acquired. Alternatively, this opens up a possibility for someone to build an equivalent to ITA itself (no small task!). The latter option is particularly risky; if Google just continues to license ITA's software past 2016, then the market basically stays the same and there's no (less) space for a competitor.

Bonus question - what does this mean for Bing? Bing is a customer of ITA. Google could just shut them off, which is a big hit to Bing's goal of being a "decision engine".

It will be interesting five years from now to check back on this post and see how right or wrong I was.

Permalink | Leave a comment »

Ten Rules for Entrepreneurs

Wed, 16 Mar 2011 15:51:00 -0700

Reid Hoffman's Ten Rules for Entrepreneurs

Seek To Create Disruptive Change. Is this massive and different? Example: Skype
Aim Big. It's the same amount of blood, sweat and tears to build a small company to flip as a big one to change the world.
Build A Network to Amplify Your Company. Your network is a store of distributed intelligence that can enable you. It's important at the board and investor level and the rank and file.
Plan for Both Good and Bad Luck. Opportunity will cross your path and you'll have to pivot to reach it. Plan Bs are intuitively easier to plan for.
Maintain Flexible Persistence. Know when to be persistent, when to be flexible and when to be a combination of both.
Launch Early Enough To Be Embarrassed By Your Creation. You'll probably be wrong about most of of your thesis, and you might as well get the feedback. Perfection is a myth. Get it into the mix. Don't undervalue the importance of time.
Aim High But Don't Drink Your Own Kool-Aid. Find friends who will tell you when your baby is ugly.
A Great Idea for a Product is Important. But a great idea for product distribution is more important.
Pay Attention to Your Culture. Worry about getting hiring right from the very beginning.
These Rules are Not Laws of Nature. You can break them.

via fastcompany.com

Permalink | Leave a comment »

Mac OS X Install rmagick gem and imagemagick

Tue, 15 Feb 2011 10:27:00 -0800

# How to install rmagick gem on Mac OS X. Tested on 10.6.6, assumes you have
# macports installed. Basically http://rmagick.rubyforge.org/install-osx.html,
# but that didn't work for me as-is. Used ruby-1.9.1p378 installed with RVM.

sudo port install tiff -macosx
sudo port clean ImageMagick # (because I started installing it, then stopped)
sudo port install imagemagick  +gs +wmf
gem install rmagick

Permalink | Leave a comment »

ElasticSearch "term" filter/query gotcha

Sat, 05 Feb 2011 12:34:00 -0800

"term" filters and queries are not analyzed. I misinterpreted the docs to mean that the document's field was not analyzed; actually it means the term passed in the filter is not analyzed. This could trip you up if you have a case-sensitive value.

If you index the document { "foo" : "bAr" }, a "term" filter like { "term" : { "foo" : "bAr" } } will not return that document, which might surprise you since "foo" is "bAr" in the document and the query. The reason is that the term "bAr" is not analyzed, but the field "bAr" was. So you have two choices now: first, analyze the term yourself and pass a filter like { "term" : { "foo" : "bar" } } (note the lower-case); second use a query filter like { "query" : { "query_string" : { "query" : "foo:bAr" } } }.

EDIT: prefix filter is also not analyzed.

EDIT 2: @kimchy updated the docs to clarify that prefix filter and prefix query are not analyzed: http://www.elasticsearch.org/guide/reference/query-dsl/prefix-filter.html

Permalink | Leave a comment »

ElasticSearch examples

Fri, 04 Feb 2011 12:23:22 -0800

I wrote some ElasticSearch examples to show how to query for associated documents:

# Example of association mapping in ElasticSearch. Assumes ES is running on localhost:9200 (the default).

# Clear index.
curl -XDELETE 'localhost:9200/testassoc'

# Store 'B' objects.
curl -XPUT 'localhost:9200/testassoc/B/1' -d '{
  "content" : "Lorem ipsum dolor sit amet, consectetuer adipiscing elit"
}'
curl -XPUT 'localhost:9200/testassoc/B/2' -d '{
  "content" : "Vivamus vitae risus vitae lorem iaculis placerat"
}'
curl -XPUT 'localhost:9200/testassoc/B/3' -d '{
  "content" : "Aliquam sit amet felis"
}'

# Store 'A' objects..
curl -XPUT 'localhost:9200/testassoc/A/1' -d '{
  "content" : "something unique to A 1",
  "B" : [1, 2, 3]
}'
curl -XPUT 'localhost:9200/testassoc/A/2' -d '{
  "content" : "something interesting about to A 2",
  "B" : [1]
}'

# Ensure changes are available.
curl -XPOST 'localhost:9200/testassoc/_refresh'

# Find all As that are associated to a given B (#2). Returns A1.
curl -XGET 'localhost:9200/testassoc/A/_search?pretty=true&q=B:2'

# Find all As that are associated to a given B (#2). Returns A1 and A2.
curl -XGET 'localhost:9200/testassoc/A/_search?pretty=true&q=B:1'



# Find all As that are associated to Bs that match a given query. Embedded
# object method.

# Store 'A' objects with embedded B objects. (To conserve space, update mapping
# so embedded object is not stored). Also just index fields of B that you will
# search on.
# Note the embedded objects are stored under a different key ('Bobj' instead of
# 'B') because the indexing above already created a dynamic type mapping for B
# to be an integer, but now we're indexing B as an object.
curl -XPUT 'localhost:9200/testassoc/A/3' -d '{
  "content" : "this is essentially A 1",
  "Bobj" : [
    { "content" : "Lorem ipsum dolor sit amet, consectetuer adipiscing elit" },
    { "content" : "Vivamus vitae risus vitae lorem iaculis placerat" },
    { "content" : "Aliquam sit amet felis" }
  ]
}'
curl -XPUT 'localhost:9200/testassoc/A/4' -d '{
  "content" : "and this is bascially A 2",
  "Bobj" : [
    { "content" : "Lorem ipsum dolor sit amet, consectetuer adipiscing elit" }
  ]
}'

# Ensure changes are available.
curl -XPOST 'localhost:9200/testassoc/_refresh'

# Find all As that are associated to a matching Bs (#2). Returns A3.
curl -XGET 'localhost:9200/testassoc/A/_search?pretty=true&q=Bobj.content:vitae'

# Find all As that are associated to a matching Bs (#1 and #2). Returns A4.
curl -XGET 'localhost:9200/testassoc/A/_search?pretty=true&q=Bobj.content:lorem'

Permalink | Leave a comment »

FB on the Importance of Tools

Wed, 26 Jan 2011 15:24:00 -0800

The original "second DBA" was even only hired because eventually our first database engineer decided she wanted someone else to cover the night shift.

via algeri-wong.com

Good tools multiply the effectiveness of individuals by staggering amounts.

Permalink | Leave a comment »

mysqlimport can't get stat of file

Mon, 24 Jan 2011 07:49:57 -0800

I tried to import some data from a CSV file into MySQL using mysqlimport. It complained a lot even though I passed the absolute path and the permissions were okay:

$ mysqlimport -p <my_db> --ignore-lines=1 /absolute/path/to/data.csv

mysqlimport: Error: 13, Can't get stat of '/absolute/path/to/data.csv' (Errcode: 13), when using table: data

The solution was to add the -L flag (short form of the --local flag). This means to read the file locally from the client host. I wondered how mysqlimport could even work without this flag; turns out the user must have the FILE privilege. Now I still wonder how mysqlimport can work without that flag because my user had that privilege but it failed nevertheless. Also, MySQL server, client, and the data file were on the same machine.

The version that worked is:

$ mysqlimport -p <my_db> --ignore-lines=1 -L /absolute/path/to/data.csv

Permalink | Leave a comment »

Install Ruby on CentOS with RVM

Sun, 09 Jan 2011 21:01:00 -0800

If you see this error: “no such file to load — zlib”, but you have zlib and zlib-devel installed already, it’s because you must pass the paths explicitly.

rvm install 1.9.2 --with-zlib-include=/usr/include/ --with-zlib-lib=/usr/lib/

To get the required development tools like make, use yum groupinstall 'Development Tools' which is similar to apt-get install build-essential on Debian.

Permalink | Leave a comment »

Hack + Math + Expert = Data Scientist

Thu, 30 Dec 2010 23:35:00 -0800

via drewconway.com

Venn diagrams are fun

Permalink | Leave a comment »

Data mining: Classification vs Clustering

Fri, 24 Dec 2010 13:11:00 -0800

Classification

Question: "How likely is person X to buy the newest BMW M5?" By creating a classification tree (a decision tree), the data can be mined to determine the likelihood of this person to buy a new M5. Possible nodes on the tree would be age, income level, current number of cars, mariatl status, kids, homeowner, or renter. The attributes of this person can be used against the decision tree to determine the likelihood of him purchasing the M5.

Clustering

Question: "What age groups like the silver BMW M5?" The data can be mined to compare the age of the purchaser of past cars and the colors bought in the past. From this data, it could be found whether certain age groups (22-30 year olds, for example) have a higher propensity to order a certain color of BMW M5s (75 percent buy blue). Similarly, it can be shown that a different age group (55-62, for example) tend to order silver BMWs (65 percent buy silver, 20 percent buy gray). The data, when mined, will tend to cluster around certain age groups and certain colors, allowing the user to quickly determine patterns in the data.

via ibm.com

Permalink | Leave a comment »

Binary switch on electric heater

Thu, 23 Dec 2010 19:54:53 -0800

This electric heater has actual binary switches for control. 00 for off, 01 for low, 10 for medium, and 11 for high.

Permalink | Leave a comment »

Facebook credits available for purchase at Target

Thu, 23 Dec 2010 19:50:54 -0800

Permalink | Leave a comment »