Hectorgon - Graphics, Books, Travel and Technology

Travel: Villa D'este, Tivoli, Rome

2012-07-22T15:41:00.001-07:00

This beautiful villa outside Rome has gravity powered fountains. All the fountains except two main ones are totally gravity powered.

Travel: Israel

2011-10-25T21:24:00.000-07:00

A short photo tour of Israel including the cities of:

Jerusalem
Tel-Aviv
Haifa
Ceasaria
Nazareth
Jaffa
Capernaum

Wailing wall and temple mount - Jerusalem

Station of the Cross 5 in Jerusalem - Via Dolorosa

Church of the Holy Sepulcher

Herod's swimming pool in Ceasaria

Mary's house in Nazareth

Startup Ecology: Hipchat

2011-06-16T00:16:00.000-07:00

Hipchat is a cloud hosted IM / chat solution with persistant history. It has cool stuff like Git integration, so if any of your team members pushes a change list to the repository it can be posted into a chat room for code reviews.

It has a nice easy to use API call that you can make use of to post notifications, for example when a cron job finishes, to a chat room.

Its a nice tool for use in collaboration. Sort of what Google Wave could have been for enterprise. Worth checking out if you are looking for a better collaboration tool other that gmail chat.

Google Infrastructure Rocks!

2011-06-08T12:14:00.000-07:00

My answer on Quora to: Is Google's software infrastructure obsolete?

In response to this blog post.

As another Xoogler said to me, working at Google (and then moving to open source) is like coming from the future. (I estimate it to be 5-8 years in the future). You know what the infrastructure should look like several years down the road. The Google infrastructure is far more stable and well documented than the open source variants. They work together as a coherent whole. Its much easier to have a stable, massively scalable system built in the Google framework than with the equivalent open source tools.

As a concrete example, Hadoop's 0.20 tutorials use the old mapred interface and when you use it its says DEPRECATED. Thats not great documentation if the most up to date tutorial uses a deprecated interface... on the other hand the internal mapreduce code labs are up to date and have examples of the latest features.

Having contributed to Apache Mahout, the machine learning they use there is at least also 5 years behind the Google equivalents...

Inner joins using Pig are rather slow. The statistics package is not that great. The Google equivalent, sawzall is much much better.

Bigtable is AWESOME. High availability, you know exactly how to optimize for it. All sorts of nice knobs to tweak if you want, but works right out of the box if you don't. You don't have to worry that your data goes missing or corrupted randomly. It. Just. Works.

As Piaw Na said, a large cluster outside of google is maybe 100 nodes. Google scale is many orders of magnitude larger than that. You can run really large jobs without worrying that your 'Hadoop name node' will get DDOSed.

So having seen both worlds, I can tell you not only is the infrastructure NOT obsolete, its also far ahead of the open source variants in terms of stability, reliability, scalability and documentation. The learning algorithms are also far more sophisticated and capable than the open source packages.

Apache Mahout Contribution - Boosting Algorithm

2011-06-01T14:42:00.000-07:00

Woohoo my third Apache Mahout Contribution, code that implements Adaboost. This is a really cool algorithm that makes a strong learner from a bunch of weak learners. In this case the weak learner are just decision stumps (e.g. is age > 50, weight > 500 lbs) etc.

This variant of adaboost is implement from the paper "Boosting with Structural Sparsity" by my ex-colleagues John Duchi and Yoram Singer. I like this variant because it results in smaller models (due to the L1 regularization) and parallelizable training due to the back pruning and feature induction.

Apache Mahout - Online Passive Aggressive Algorithm Patch

2011-05-18T23:02:00.000-07:00

So I submitted my first patch to Apache Mahout, a simple online multi-class classifier. It will be a test bed to see if I should contribute more to this open source machine learning project or just commit to my own repository and open source it.

There are pros and cons to each.
The pro of having my own repository is more control over the design and rapid iteration.

For example, the mahout classifier framework seems to be more geared towards the probabilistic approach (Bayes, Logistic regression), whereas I've been more of the empirical loss minimization approach, where it doesn't make sense to output the scores as probabilities.

The cons is less eyeballs to check for bugs and having to deal with stuff like packaging and support (which are totally optional as an open source contributor, its just a nice thing to do for users).

Well we'll see, so far the experience has been positive, the code gets reviewed quickly and the bug tracking / patch submission process seems quite painless.

Ek'Balam

2011-05-01T12:47:00.000-07:00

Ek'Balam is probably one of the more interesting Mayan ruins easily accessible from Cancun. Its about a two hour bus ride from the airport into the hot jungle of the Yucatan Peninsula.

Ek'Balam

The main attraction is this palatial structure with well preserved sculptures and hieroglyphics depicting an old king's journey into the underworld. The Yucatan Peninsula has very many limestone based sinkholes called cenotes that the Mayans obtained fresh water from. Thus they believed that the souls of the dead journey through the sinkholes and then get renewed.

Here is a picture of the inside of a

cenote

Behind me you can see a big mouth that looks a lot like the stalagtites and stalagmites in the cenote. It depicts the king's journey into the underworld.

Perceptual diff used to heat houses

2011-04-20T22:27:00.000-07:00

So apparently the tool I wrote for comparing images Perceptual Image Diff can be used to heat houses.

I must say its a very novel and funny way to use the algorithm. Kudos to the author for coming up with such a creative and novel use for what was written to be QA software.

Flying robots play ping pong

2011-03-31T18:39:00.000-07:00

You guys know I worked on robots but THIS robot plays ping pong!

Search Amazon.com for quadrotor rc helicopter

Plastic animals

2011-02-16T19:44:00.000-08:00

Theo Jansen has made a new form of "life" made from electricity tube that is powered by air, can walk around, detect sand dunes and water etc. Pretty amazing.

Happy Chinese New Year 2011

2011-02-02T20:35:00.000-08:00

Dear friends and family-
Happy Chinese New Year! Its the year of the rabbit and to usher in the New Year my mom has prepared a feast for us. Here's a video greeting from her with the feast:

Feast components:
Roast pork (kong bak)
Lo Hei (the multi colored carrot dish in the middle)
Ngoh Hiam (fried rolls)
Chap Chai (veggies and bean curd)
Prawns
Glutenous rice and minced meat
Kueh Pai Tee (fried jicama in wheat cups)

Guest Review: The Spontaneous Fulfillment of Desire by Deepak Chopra

2011-01-03T19:39:00.000-08:00

(This is a guest book review from hectorgon.com)
Reviewed by Stella Danker (stelladanker@gmail.com)

This book has one of the best first sentences I have read – “Miracles happen every day.” This is a book to be read slowly and mulled over, to keep close by as a guide, a book from which to extract wisdoms, and through which to create miracles.

There are gems to be mined here, if approached with an open mind and some patience. Better editing would have polished this gem to dazzle a wider audience than the converted like me.

This is my first Deepak Chopra book and there are more than 50 of them. Chopra is a master of the spiritual self-help genre and has been translated into 35 languages. He is a fellow of the American College of Physicians, among other things, and co-founder of
The Chopra Center for Wellbeing in southern California. He writes in this book, from a physics background as well, about the potential that exists in the energy of the universe and our personal power to shape our destinies and make our dreams come true. It is not an airy-fairy leap of faith he is asking of his reader. It is an intelligently put forth quantum leap. Time magazine has called him “the poet-prophet of alternative medicine”.

Chopra requests that the reader persists through the first part of the book and not be tempted to skip to the meaty offerings in the second part of the book. But that is exactly what the reader will want to do because the introduction through to Chapter Two should have been halved. I also urge the reader to persist but suggest that some skimming through would be a good idea.

The physics fascinates but is turgid. Some of the supporting examples are brilliant but some are banal. There are too many personal anecdotes that could have been easily told more succinctly to make an argument for how one thing leads to another seemingly
randomly but actually with a universal purpose to lead us to our destiny.

Chopra delivers the goods from Chapter 3 onwards and the reader is taken mesmerized on a magical, mystery ride though quantum physics to a place offering peace and possibilities. It is an uplifting book in these stressful economic doldrums in which we live. Even if you get nothing else out of this book, it teaches you how to meditate, and offers many mantras. This is a practice that will bring much peace.

There is also much wisdom on how to become extraordinary by understanding the local and non-local mind. “If you really want to break out of the mundane, you must learn to think and dream the impossible.”

Anything that ever happened did so because somebody set an intention for it. This is the “desire” part of the title. Strengthen this intention with a vision and you are on your way to creating a miracle. It has been said that when Mahatma Gandhi was thrown out of
a train in Durban, South Africa, because he was Indian, Gandhi closed his eyes and saw the British Empire crumbling halfway across the world.

There is no such thing as luck or a coincidence. Chopra writes: “Every coincidence becomes an opportunity for creativity. Every coincidence becomes an opportunity for you to become the person the universe intended you to be.” Coincidences are clues sent to us
by the universe. And his book makes us more aware of these and trust that they lead us to where we are meant to be.

Probably the best argument Chopra puts forward on how seemingly random events are synchronized to achieve a deliberate destiny or purpose is in the Big Bang. This is what he calls the Coincidence of the Universe. Absolutely nothing would exist if not
for a “remarkable set of coincidences.” The number of particles and the number of anti-particles had to be just so to begin the chain reactions that would create our planet and the galaxy. Just a little more of this or that, and the whole universe would have collapsed into itself.

The many references in the book to the ancient Vedic texts, the Sanskrit sutras, and to the Greek gods and goddesses are sheer poetry. You want to read this book. Can you pass up the chance that you might actually conjure up magic?

The Spontaneous Fulfillment of Desire: Harnessing the Infinite Power of Coincidence

Stella Danker is a freelance journalist based in New York.

Nexus S

2010-12-18T22:41:00.000-08:00

Got a nice Nexus S from work. My iPhone 3G had been getting slower and slower since iPhone 4 and was practically useless so I was glad to get a new phone. The Nexus S was easy to set up out of the box. The only problem I had was prying the cover off and getting the SIM card in. It didn't come with any setup sheet, I guess they assume people who would get the Nexus S have some level of technical ability.

Having switched from the iPhone I was pleasantly surprised at how fast the Nexus S booted up. It was a lot faster than the iPhone 3G for sure. I had transferred all my contacts via syncing iTunes with my Google contacts, so moving was easy. I had also switched to T-mobile Even More Plus plan (which you can only get in the T-mobile store) and the voice quality went up while my mobile bill went down. Calling my mom international with Google Voice was also a lot clearer, and cheaper, than the AT&T international calling that I had before. Also downloaded the free version of Angry Birds game as my first Android Market foray.

Overall I've had a good impression of it. The keyboard is a bit harder to type in but so far I am pleased with the speed and responsiveness of this new phone.

Review: Stalking the Black Swan

2010-12-18T22:15:00.000-08:00

Originally posted on Hectorgon Reviews - Stalking the Black Swan

This is a review of Stalking the Black Swan: Research and Decision Making in a World of Extreme Volatility (Columbia Business School Publishing).

A Black Swan is a reference to
a concept developed in an earlier book by Nassim Taleb on how people are bad at judging probabilities. For example, if they only see white swans they naturally assume that all swans are white. Until they see a black one. Black Swans in the financial world are extremely unlikely events that almost no one forsees, such as the mortgage crisis in 2008.

In "Stalking the Black Swan", Mr Posner describes how one can use tools such as decision trees to help make decisions that are less gut feeling and more numerical. He uses several case studies such as the crash of credit card companies such as Providian and Capital One and how he had computed their value in the past using such decision trees.

Readers of this blog who are familiar with computer graphics will recognize the statistics and probability concepts quite easily. In fact, the author devotes an entire chapter to Monte Carlo simulation of these decision trees to ultimately compute the valuation of complex financial instruments such as the infamous Collateral Debt Obligations (CDOs) and how mistakes in assumptions such as historical housing pricing data could mislead people to assume that housing prices behaved as before and thus get burnt when the mortgages all crashed at the same time.

Interesting read for people familiar with elementary probability and statistics and gives an insight into how some analysts perform valuations of stocks.

Stalking the Black Swan is available on Amazon.

Review: More Money than God by Sebastian Mallaby

2010-12-09T23:31:00.000-08:00

Original review from hectorgon.com

This is a review of More Money Than God: Hedge Funds and the Making of a New Elite by Sebastian Mallaby.

I was driving in my car listening to NPR when the author Mr Mallaby came on air to discuss his book. Mr Mallaby is an interesting and exciting speaker. He makes a dry subject like finance come to life with interesting anecdotes and insider stories, making you feel as if you're watching the history of hedge funds unfold right before your eyes.

His book More Money Than God details the history of the hedge fund industry from around the early part of the 20th century to the current recession of 2008. His basic premise is that small, boutique hedge funds that are not too big to fail are actually good for the economy as they provide liquidity to the market and generate the long sought after 'alpha', a measure of investor skill as opposed to 'beta' a measure of how well a stock performs just by the market rising. The early hedge funds generated alpha just by the fact that they sat in front of the SEC and got each company's quarterly report by hand rather than waiting for them to be delivered by mail. This gave them a competitive advantage of a few hours to buy or sell a company's stock before their competitors. As the market got more efficient and other people caught on to this innovation, new hedge funds came along and started trying fancier techniques like talking to clients of a company to see how well its doing, or estimating trends or even looking at stock 'momentum'. With the event of the computer, hedge funds like Long Term Capital Management started doing things like finding correlated stocks that may trade in multiple exchanges where it may be too expensive in one exchange and too cheap in another and exploit these inefficiencies to their advantage. He finally describes the ultimate hedge fund of them all, Renaissance Technologies, that were founded by code breakers and statistical machine translation folks who use their advanced computer skills to analyze the stock market to generate 'alpha', in order to get uncorrelated returns from the market.

Mr Mallaby also shows the horrific consequences of hedge funds that got too big like LTCM that had overleveraged themselves and when it came time to liquidate, telegraphed their intent and became the victims of other traders who traded against their position. He also described how George Sorro's Quantum Fund broke the currency pegs in the UK and during the Asian Financial crisis by borrowing more money than a large nations (like the UK's) foreign reserve.

Although dry at times, it provides an interesting casual look into the world of hedge funds that is well research and an entertaining read. Recommended for people who are interested in finance that may not be in the field and are interested in knowing the backstory of what goes on.

23and me Holiday Special

2010-12-02T12:07:00.000-08:00

I just noticed that 23andme, the personal gene testing service, seems to have a $99 holiday special. I took advantage of this during DNA day and it was fairly entertaining. Mostly it tells you interesting things about your heritage e.g. where your mother's ancestors were from and where your father's ancestors were from. It even searches their database for long lost relatives and gives you the option of contacting them. Less interesting (to me) was all the medical stuff, like if you are more or less likely to get some condition.

Deep Mandelbrot Set Zoom

2010-11-10T23:53:00.001-08:00

Now that Benoit Mandelbrot is no longer with us, here's a super deep Mandelbrot set zoom video in commemoration of his passing. I owe a lot of my early math explorations to fractals and the Mandelbrot set has always been a favorite of mine to try out new programming languages with.

Book Review: Destroyer of Worlds by Larry Niven

2010-11-10T23:18:00.001-08:00

Read my Destroyer of Worlds review here.

Book Review: Fleet of Worlds and Juggler of Worlds

2010-11-10T00:42:00.000-08:00

I've posted a new review on Fleet of Worlds and Juggler of Worlds by Larry Niven and Edward M Lerner.

Hectorgon.com

2010-11-08T22:43:00.000-08:00

I've finally registered my own domain name and have my own wordpress hosting.
Please head over to hectorgon.com and check it out!
It will be the new location of my book reviews.
Check out this latest review of Iain M. Bank's Surface Detail.

My very first animation

2010-10-19T17:07:00.000-07:00

Wow blast from the past. I did this animation before doing some real ones in Hollywood.

The CS 418 server over at Cornell seems to be going down I'm told so I uploaded a copy of my first animation to youtube (done as a class project with Bryan Vandrovec):

Here's the same class project done a year later by my friends Steve Colamaria and Paul Zimmons:

At the time we were Babylon 5 fans so here's the full text of our writeup for the class before its gone forever.

Introduction

ISN will now broadcast the exclusive interview with Hector Yee and Bryan Vandrovec, makers of "Shadows" the movie.
The Plot

ISN: Hello Hector and Bryan, Could one of you gentlemen tell me how you came up with the idea for the animation?
Hector: Sure, we were inspired by the graphics of the show Babylon 5, and decided to make our own MPEG of Shadow ships destroying a colony. The plot is remarkably simple - Shadow ships launch from their base at Z'ha'dum with a virtually defenseless war cruiser, phase from hyperspace in the vicinity of the Hyperion colony (named after the Hyperion ftp site), destroy a space station and an innocent trading craft, wait for the cruiser to arrive, and proceed to vaporize the colony form orbit with a massive directed energy weapon.

The Introducion

ISN:The Introduction scene was interesting, almost cow hide like. How did that come about?
Hector: The background Title Screen was rendered using Adobe Photoshop. I drew in a Trutchett tiling (normally used to visualize the randomness of bit patterns) with the help of my friend, Seth. Thanks to Seth Kromholz for the use of his Power Macintosh. The caption for the text was then used to modify the Z-coordinate of the texture map to give the illusion of embossed text.

Z'ha'dum Staging area 2048.1

ISN: Allright, we've had enough of mad cows. Now, how was the Ship flyby scene done?
Hector: Well, I really liked the CGI in Babylon 5, so I decided to use the same rendering engine for the backgrounds. I used Lightwave 3D on a Pentium to render the star scape for all the space scenes. The Z'ha'dum scene was rendered seperately. The Planet is a texture mapped sphere and the stars are polygons with their intensities modified to look like stars. The image was exported to DX and mapped onto a flat polygon as a background.

ISN: How about the ships... how were they done?

Bryan: Ah yes, Hector and I spent quite some time considering our options. After weighing alternatives, we decided that the DX package which would be used to render and animate the scenes didn't quite have the tools to easily generate realistic looking models for the project. The "shadow ships" were an exception, and were in fact done with DX as Hector will explain. I elected instead to use DesignCAD 3D to fill the modeling requirement. As for the spacecraft designs themselves, well, they were inspired by several different sources. Some forms I borrowed from nature. The space station, seen in orbit around the planet in the next scene, was loosely patterned after a jellyfish. The trading vessel came from a concept by the British Interplanetary Society for an interstellar ramjet. The cruiser vessel which ultimately destroys the colony was... well, to be honest I'm not quite sure where that came from. I suppose it was just a few hours of trial and error. Unfortunately there wasn't time to texture drape the ships, but I think they worked out alright regardless.

Hector: I created the shadow ships by parametrically deforming several cones and bump mapping them using the Mark and Compute modules under DX. The hyperspace phasing /cloaking effect was done by radially peturbing the lattice points while varying the opacities.

Hyperion 2048.3

ISN: The Station attack scene was awesome! How was it done?
Hector: Well, the coordination was the hardest part of it. I thought up a scheme to make it easy to coordinate all the motion. We scaled the sequencer so that it gave out a time index that ran from 0 to 1. This ensured that we could have animation independent of the number of frames and that we could re-render with a higher number of frams in the future.

Bryan: I'd have to agree with Hector, coordinating all the motions was quite a task. A system of linear vector interpolation was used to sweep and target the particle beams fired by the two Shadow ships. A DX rotation module was utilized for the station spin, and a combination of time dependent trajectory and scaling functions were used to enhance the perceived depth. The timing was critical as well. We had to be sure that the shield collapse and destruction of both the station and the trading vessel occurred just as the Shadow beams sliced through and penetrated their mid-body sections. Also, both resulting explosions had to be evolved for just the right period of time to acheive the desired effect. Too few frames would result in essentially a firecracker bang, too many would cause overlap and interference with other events in the scene.

ISN:How about the special effects? How were they done?

Hector: The texture maps for the skin of the shadow ships, the shields, and the lasers were created by rendering the objects in a time varying noise field. The explosion was created by rendering three nested isosurface shells in a smoothed noise field.

Bryan: Each of the isosurface shells was differentially upscaled according to time dependent exponential functions. Their opacities were proportionally downscaled to produce the effect of dissipation. A faint spherical shockwave was later added to complete the effect.

Hyperion 2048.31

Hector: Nothing new is exhibited here but close up zooms of the models. The backdrop was done in Lightwave 3D again. This scene was done, as the rest, by first rendering in 640x480 resolution and then using the reduce model to reduce it to 320x240. This ensured that the shadow ships do not dissappear unintentionally due to aliasing. This scene was a filler to show off the details of the models.
Hyperion 2048.32

Hector: Nothing much to say about this scene.
Hyperion Colony

ISN:The colony destruction scene was simply amazing ... what do you have to say about it?
Hector:The colony dome was created by using a DX show connections and tube module on a parabola.

Bryan: I created the colony itself, again using DesignCad. The landscape was done with a custom fractal function I wrote a few weeks ago. It uses a converging pseudo-random number and recursive midpoint subdivision algorithm to simulate the rugged terrain. The clouds were variants off the isosurface macro used in the explosions.

Ending Credits

ISN: Well, we're almost done with the interview. What can you tell us about the ending credits?
Bryan: I wanted to make a parody of the upcoming movie "Independence Day" with a message Have a Nice Day.

Hector: By the way, thanks to Avalon ftp site for the public domain Capitol building DXF.

ISN: Some people have complained about brief flashes of light during the last scene. Is there some sort of a subliminal message in the movie?

Hector: (Shrugs and feigns ignorance). What subliminal message? There are no subliminal messages. Trust me. Watch more Babylon 5.

ISN: Thanks for your time. Do you have anything additional to add?

Hector: Well, yes. The final MPEG compression was done on my home computer, a Pentium running Linux. The reason is that the scratch space on the supercomputer ran out, so I transferred all 999x2 files to my computer and compressed it there.

ISN: Well, that was the interview with CGI artists Hector Yee and Bryan Vandrovec. Thanks for watching the exlusive interview by Inter-Stellar News.

My sekret Google project - self driving cars

2010-10-09T14:04:00.000-07:00

Now that the Google self driving cars story has broken on New York Times and there's a video (look ma no hands) and the official Google blog about it I can finally tell people what I've been working on the past year - the perception system for self driving cars. Yeah, sorry not Street View.

I worked on mostly the perception AI - the machine learning, camera system and some parts of laser and radar sensors to help the car 'see' other cars, pedestrians and traffic lights.

Nice ABC summary:

Video from New York times taken from the inside:

Here's a wikipedia link on the history of driverless cars

Here's a video Robert Scoble managed to get a year ago:

The Sound of Sorting

2010-08-19T10:27:00.000-07:00

Shamayn buzzed me this great youtube on the sound made by different sorting algorithms:

Image Specific Related Searches

2010-05-11T11:17:00.000-07:00

Yay, good news friends and family, my Google Image Search 20% project is live and visible in Singapore, check it out.

Is the thing to the right of the right arrow under "related searches". It tells you what this image is related to.

Book Review: Mycelium Running

2010-03-01T23:48:00.001-08:00

This is a review of Mycelium Running: How Mushrooms Can Help Save the World.

Friends from Cornell will fondly remember the bag of Portobello mushrooms I grew and cooked for one of my random dinners. Well here is a whole book by Paul Stamets on bio remediation using fungi. Apparently fungi can colonize the roots of trees and extend the nutrient gathering capabilities of trees many orders of magnitude. Also, bags of burlap inoculated with the humble oyster mushroom were used to detoxify a dump of diesel fuel! These and other interesting stories on how to cultivate fungi for food, medicine and cleaning up the environment can be found in this easy to read and well illustrated book.

I found out about this book by watching this TED video: