So, in no particular order…

1. Most web applications are database driven, so know about database scaling and performance. Also: Memcached (distributed memory cache), caching proxies like squid, and caching techniques in general.
2. Machine learning and data mining techniques — at least an understanding of their potential. There is way too much to go into here, but play with the open source package called Weka. Also check out the excellent introductory, hands-on book Collective Intelligence by Toby Segaran.
3. Lucene, an open source search engine.
4. If you’re applying (or working at) an interesting startup, you’ll probably have large quantities of data to process, so know about Hadoop, the open source answer to Google’s MapReduce paradigm. Consider in combination with Amazon’s EC2 (see below).
5. The Google File System (GFS) and Google’s BigTable. These projects represent the current cutting-edge in data storage for scalable web applications. But if you want to use them, you’ll have to join Google or use one of these projects that offer some (but not all) of their features: Amazon’s S3 and SimpleDB (see below), MogileFS, Global File System, and Hadoop’s HDFS file system with HBase acting as BigTable, but HBase may not be ready for prime-time just yet. There are, of course, other solutions of varying complexity as well.
6. Amazon web services: S3 (backups, reliable data store; data archive; file serving), EC2 (virtualized, scalable utility computing; file processing; server environments — and you should know something about machine virtualization in general as well), and to a possibly lesser (and unproven) extent SimpleDB (scalable database replacement for some types of applications — you should have experience with MySQL and the SQL language too).
7. Ruby on Rails — even if you’re not working in Ruby or deploying a Rails app, Rails is a powerful environment for rapid prototyping and experimentation, plus a very marketable skill in the current climate.
8. Almost without saying: the obvious frontend interface technologies of HTML, CSS, JavaScript, and Flash/Flex.

All of the above are fairly language agnostic. You should know a couple of programming languages quite well, preferably one scripting language (probably one of Ruby, Perl, Python, or PHP), and one ‘harder’ language such as Java or C++. Be prepared to write code and answer questions in your chosen languages.

When interviewing at a startup, or any place really, make sure to a) explain your thoughts when solving problems (don’t just think to yourself for 5 minutes), b) talk about what excites you (technologically and otherwise) and your awesome side-projects, c) be willing to talk about the flaws as well as the strengths of technologies, d) know something about the technological area of the startup, and e) actually know the subjects you proclaim to understand on your resume.

Everyone is hiring right now (including my employer)! So read up, do some side projects, and good luck!

What do you think about this list? Please suggest technologies and links that I missed!

Color Blender — Very convenient tool to blend any two colors with a varying number of midpoints

A list of colors by name and color code — for those of us who are color blind, this is very helpful for looking up colors by name.

Color Code Chooser — tool helps you manipulate colors in a number of very helpful ways.

Luis von Ahn opened his talk by saying that people spend many millions of human hours on solitaire each year, and that it would be useful if even a small fraction of that time could be harnessed to get people to play games that are also useful. If this is his goal, he has succeeded — some people spend over 40 hours a week on the ESP Game, which has been very successful.

Luis also presented a general approach for turning computationally hard pattern recognition problems into two player games and suggested that many problems can be solved in this way. He is currently thinking about such things as language translation and common-sense knowledge collection.

While it is cool that the ESP Game, if adopted by a major gaming site like Yahoo! Games, could label most of the web’s images in just a few months, the most exciting thing for me is the wealth of training data that this would generate for researchers to make better computer vision algorithms. This would also be true for things like language translation and common-sense knowledge collection — these would empower new algorithms.

Luis ended by pointing out that The Matrix got it all wrong: we’re useless as batteries, but we make great pattern recognition subroutines. That’s why the computers will need to keep us around, at least for the time being. They keep us entertained, and we compute for them. Oddly enough.

I think one of Spore’s largest advancements is its leverage of other player’s content to bootstrap your world. Instead of requiring the game developers to come up with varied designs for thousands of species of life forms or civilizations, Spore finds content created by other players that will work well with your needs, and brings those to you in the form of tools or buildings to be purchased, other life forms, and alien races to encounter. This is brilliant. The game can only get more advanced and varied as players use it.

Spore’s worlds and creatures are procedurally generated. I previously wrote about procedurally generated environments, but I’ve never seen anything like this before. You can create creatures by combining many different parts, each of which has functionality. You can also reshape and mold parts as if they were clay. Then, the system analyzes the morphological structure of your newly created creature and figures out how it might move — how it should walk, fight, eat, mate, and more. The generated movements are plausible and visually pleasing. No motion capture or hand-animation required.

Spore tackles an incredible scale. I watched the video, and at every stage I thought, “wow, that’s a great game!”, then I found out that what I had seen was just the tutorial/prerequisite for the next, even larger stage of game play.

“There will always be things we wish to say in our programs that in all known languages can only be said poorly.”

“There are two ways to write error-free programs; only the third one works.”

This site has more.

The basic idea is that particles of different materials are falling from the sky. You can direct them by creating platforms, mix them for different effects, burn them, and more. There is also a strange bouncy protoplasmic thing that doesn’t like salt. Remember, water puts out fire, fire burns oil and plant, salt kills slug, and sand sinks through water.

The hard thing about web page annotation is figuring out how to lock a highlight or annotation to a specific place on a web page. Especially with dynamically generated pages, the content can change often, so how do you know where to put the annotation? I was excited to see how diigo handled this issue, and they seem to have taken the textual route, analyzing the page’s source via JavaScript and looking for text that matches what was originally annotated. This works when the annotated text is unique, but fails if you happen to annotate text the recurs on the page. However, this still works reasonably well. (If you’re interested, you can grab their plugin and unzip the .jar file to see how some of this stuff works. The interesting text selection code is in the core.js file.) I don’t know how to get around the annotation problem, but will keep an eye on diigo as they continue to refine their beta. Perhaps they will think of something that I haven’t.

Solution Watch and TechCrunch already wrote about Diigo.

Now, a site called Mystickies [via Goggle Blogoscoped] is trying this with floating text boxes — sticky notes. Their service isn’t collaborative or sharable yet, which is a shame, but it’s a start.

There are some other attempts at online annotation, such as the Annotation Engine. However, the Annotation Engine doesn’t have social controls or groups, so the quantity of annotation — spam, graffiti, whatever — grows out of control. It’s also old and kind of broken. I think a good system would use a karma and interest threshold approach, similar to slashdot, to decide what to show you.

Another fun project is wikalong, which I am a big fan of. It’s not on-page annotation, it is a sidebar wiki for every page on the internet. They simplified the annotation problem into something manageable, and then pulled it off.

Read the full post for a partial breakdown of current game AI and some interesting tidbits.

(more…)

I have been playing with Second Life (SL), a realization of the Stephensonian Metaverse. I’m calling this “research,” as I’m surveying the state of MMOG Artificial Intelligence and Artificial Life, but at this point it might as well be play. Yet, Second Life really isn’t a game, it’s more of an online, massively-interactive world. All content is user created with the built-in scripting language (LSL), Havok physics engine, and 3D modeling and animation systems. Users create buildings, vehicles, weapons, clothing, gambling joints, a plethora of shops, and much more. Like first life, it’s very flexible. Some users have even succeeded in making a living selling things in SL and then exchanging the in-game currency for real money.

In terms of AI, the most advanced thing I’ve seen is a tie-in with an implementation of Alicebot. The LSL scripting language lends itself to simple AI because it is based around a model of Finite State Machines — any scripted object can be in a number of different states, and can respond differently to events based on which state it is currently in.

However, what SL really lends itself to is Artificial Life. A user known as Surina Skallagrimson has developed an AL model of fish that swim, shoal (flock), seek out food, gain food preferences, see and evade predators, reproduce, and die if insufficiently fed. She has also met with limited success in passing on genetic information to allow simulated evolution. The main restrictions to progress arise from the many limitations of the LSL scripting language, which has no provisions for long-term memory.

I interviewed Surina in SL (a cool experience) and then she showed me the fish. They flock very naturally, colored by their food preferences. Surina is now working on modifying her code to create birds, which will prey upon the fish. The eggs of AL frogs, created by a different SLer, can also be eaten by the fish. Some of Surina’s schools have existed for months, even though each individual can live for a maximum of four hours. If you want to see the fish for yourself, they swim in the waters around Hypatia 49, 221, but wander as far away as neighboring Themiskyra. A basic SL account is free.

What excites me most about this is the potential for systems like SL to allow people to play with AL in an integrated environment. In such systems, one can actively experiment in 3D on an evolving system. The integration with the Havok physics engine makes complex emergent behavior likely.

A choice quote from a blog article about Surina’s work:

Due to “a bug in the fish causing them to reproduce out of control. It appears the limit for 3 sims full of fish is around 10,000 (Ten thousand) fish… A virus was quickly introduced to kill off the mutant fish before they either crashed the servers or they evolved a way of teleporting to the main grid.”

This all brings me back to a project that I did a while ago in which I created a two-dimensional physics simulation and allowed mass-spring “creatures” to evolve mobility. It was much like the new sodarace system. (I was inspired by the original sodaplay.) Creatures evolved methods of locomotion qualitatively similar to runs, walks, slides, rolls, and hops. AL is exciting because simple rules give rise to incredibly complex behaviors. But we already know that, of course, because we are the result of such a process. (Sorry Kansas Board of Education.) I’d love to see people build complex, evolving AL ecosystems in MMOGs, and I predict that we will see this occur as available in-game CPU time for this sort of process increases.

• Identify trends. For example, BlogPulse. I also experimented with this once and mined common links and phrases from 40,000 LiveJournal posts. This could be implemented on a larger scale, though. Could a program identify a change in rhetoric across an entire region or follow cultural shifts?
• Mine information on people. I’ve seen this done, but never well. (But this is creepy. I’m not sure the world really needs better stalking tools.)
• You can verify links. For example, (shameless plug) my site AbsurdlyCool FreebieFinder finds freebies online while avoiding referral links by verifying links across multiple sites. If a link has an embedded referral ID, it won’t be identical on many sites, and so will be ignored. This could be applied to other domains.

Please comment on other possibilities. What if you have images as well as text?