Gabriel Weinberg's Blog

What makes you weird?

2009-07-06T12:13:00Z

What does most (>75%) of the population (of your country) do that you do not?

Note that I'm not asking what makes you unique. For example, I'm an INTJ, which is one of those 1% personality types. However, it's not like 99% of people are one personality type and then us INTJs are the other one. There are 16 Myers-Briggs types, so this doesn't count, although it certainly contributes to my uniqueness.

I'm looking for (at least almost) binary things. The first thing I thought of is that don't drink, but it turns out 40% of Americans don't drink. So that doesn't count.

Here's some things that seem to count (with my % estimates):

I have a graduate degree (10% of US pop>25). (A bachelor's degree doesn't cut it any more as 27% of the US population over 25 has one.)
I went to a top-10 school (3% of school goers).
I am a business owner (15% of US pop>25).
I did not get a job after school. (I started a business instead.) I can't find a stat here, but I'm sure it is very low.
I'm a stay-at-home Dad (1% of dads).
I blog (12% of Internet users).

What about you?

Things about Web Images I Just Learned

2009-06-16T14:17:23Z

I thought I knew everything you needed to know about Web images. But, of course, I didn't. Here's what I just learned when launching the new icon bar on the homepage of Duck Duck Go. We wanted the it to function sort of like the Apple dashboard (and on the Web like Schmedley's bottom bar).

img{-ms-interpolation-mode:bicubic}. Short version: if you resize images dynamically, they will look bad on IE unless you put this in your CSS.

Longer version: We ended up using the YUI Animation Library to do the animation. But no matter how we did it using 1 image, it always looked terrible on IE. Even if we used an image exactly as big as the big size, and did the smaller image exactly half of the bigger size (which should be easy to resize), it still looked bad.

So then we tried using two images, which sort-of worked, but had its own issues. Sometimes it would slow down the animation. It used almost double the image size and requests (a big no-no), and the actual resizing still looked bad (as opposed to the endpoints)!

This was unacceptable, so I decided to dig deeper on the Web about this issue. It turns out modern browsers use Bicubic interpolation to resize images and make them look good in the process. For whatever reason, IE7+ has decided to turn it off by default. I'm guessing this is because it takes some processing power, but it renders resized images looking terrible so I personally don't think this is a good trade off. Anyway, if you add that above CSS to your page, IE7+ will use this method and your images will look good. I suppose I never hit this before because usually you shouldn't be resizing images dynamically. But there are cases where you want to do it...

Unfortunately, it still doesn't work for IE6, on which you need to use the good ol' AlphaImageLoader (sizingMethod='scale') if you want to support that browser.
Photoshop/Illustrator's 'Save for Web...' does not fully optimize. Perhaps my versions of Photoshop and Illustrator are too old, but I suspect this is still the case with the newer versions. I pretty much used these blind, assuming they were optimizing correctly. And don't get me wrong, it does a decent job, but its just not the best. Instead, run your images through Yahoo!'s smush.it site.
If you really do not need PNG-24, use PNG-8. PNG-8 is really a better GIF. But it is limited in color palette and transparency with respect to PNG-24. That being said, often you don't need the difference, especially for things like icons. When you can, use PNG-8 because you'll get much smaller file sizes.

That being said, you might think you need PNG-24 when you really don't. I did. I had these icons made that had full transparency. I knew, however, they were going to be on a white background, so I really didn't need all the transparency. Yet when I tried to save it as PNG-8, it just looked bad. The colors were all off. So it made me think that I needed PNG-24, but in reality it was Photoshop's optimization stuff that was being poor. In their defense, I wasn't helping them out by setting the white background ahead of time, which leads me to:
If you want to save a PNG-24 image as PNG-8, put in the background first. Once I made a white background layer, Photoshop then did a great job of saving it as PNG-8. And in fact, I could reduced the file size even more by using even less than 256 colors. Of course, I still had to run it through smush.it.
CSS sprites may reduce your page load (and image size further). CSS sprites are a way to group your images into one big file and then split them into separate files via CSS. There is a useful Web site to help you make them at csssprites.com. I couldn't figure out how to use it with my resizing requirements, but in the general case it should be at least tried, especially for icons where the color palette for your icons are similar. You get a win in image size. But you get a bigger win in reducing HTTP requests.
Custom icons are not that expensive. We got $40 custom icons and $10 recolored icons from iconshock.com. We talked to other icon designer firms as well, and prices were similar. Full disclosure: we created more than 3 icons (7), so we got a bit of a bulk discount. I did have a bad experience with iconeden.com, however. So I'd stay away from them.

For more image optimization tips, check out Yahoo!'s presentation.

Update: additional comments can be found here.

A Harsh CSS Environment for Testing Widgets

2009-06-07T16:53:13Z

Embedded widgets can face harsh CSS environments, but usually not this harsh:

#harsh * {

border: thin dotted #00FF00 !important;

display: block !important;

margin: 20 !important;

outline: 1px dotted red !important;

padding: 20 !important;

background: #00ff00 !important;

cursor: move !important;

clear: both !important;

float: left !important;

height: 0 !important;

max-height: 0 !important;

max-width: 0 !important;

min-height: 100px !important;

min-width: 100px !important;

visibility: hidden !important;

width: 0 !important;

bottom: 100px !important;

clip: rect(100px, 50px, 100px, 50px) !important;

left: 100px !important;

overflow: visible !important;

position: absolute !important;

right: 100px !important;

top: 100px !important;

vertical-align: sub !important;

z-index: 100 !important;

color: red !important;

direction: rtl !important;

font: oblique small-caps 900 20px/50px arial !important;

font-size-adjust: .01 !important;

font-stretch: ultra-expanded !important;

letter-spacing: 20px !important;

list-style: decimal inside !important;

text-align: right !important;

text-decoration: blink !important;

text-indent: 100px !important;

text-shadow: #000 30px !important;

text-transform: uppercase !important;

unicode-bidi: embed;

white-space: pre !important;

word-spacing: 20px !important;

border-collapse: separate !important;

border-spacing: 30px !important;

caption-side: bottom !important;

empty-cells: show !important;

table-layout: fixed !important;

}

If your widget looks OK inside <div id="harsh"></div>, then it will probably look OK anywhere. I made this HTML example (view source) for easy testing.

Why does this matter? Suppose a site has a black background and white text, but your widget has a white background but no text color set--none of your text would show.

To deal with a harsh environment, you need some armor:

#armor, #armor * {

border: none !important;

display: block !important;

margin: 0 !important;

outline: none !important;

padding: 0 !important;

background: #fff !important;

cursor: auto !important;

clear: none !important;

float: none !important;

height: auto !important;

max-height: none !important;

max-width: none !important;

min-height: 0 !important;

min-width: 0 !important;

visibility: visible !important;

width: auto !important;

bottom: auto !important;

clip: auto !important;

left: auto !important;

overflow: auto !important;

position: relative !important;

right: auto !important;

top: auto !important;

vertical-align: top !important;

z-index: 1 !important;

color: #000 !important;

direction: ltr !important;

font: normal normal normal 11px/14px tahoma,sans-serif !important;

font-size-adjust: none !important;

font-stretch: normal !important;

letter-spacing: normal !important;

list-style: none !important;

text-align: left !important;

text-decoration: none !important;

text-indent: 0 !important;

text-shadow: none !important;

text-transform: none !important;

unicode-bidi: normal;

white-space: normal !important;

word-spacing: normal !important;

border-collapse: collapse !important;

border-spacing: 0 !important;

caption-side: left !important;

empty-cells: hide !important;

table-layout: auto !important;

}

If you wrap your widget in <div id="armor"></div>, it should work OK. I made another HTML example (view source) for testing this armor.

I tested #armor cross browser using my test systems and browsershots.org. Of course, there are most likely still bugs, so please tell me about them!

To develop #harsh, I used the w3schools CSS Reference, which you can also use to figure out if you want to change the properties in #armor, or apply more thereafter.

To apply additional styling after #armor, use ids instead of classes, e.g. id=""/# and not class=""/. because a particularly harsh use of #id * will override your classes. Of course, if you aren't that paranoid, you could back off the * in #armor and use classes instead.

You could also just use inline styling, i.e. style="". There may also be a better way to do it that I just haven't thought of yet. If you know of one, do tell...

Introducing the Duck Duck Go Blog

2009-05-26T21:27:58Z

Today I started the official blog for Duck Duck Go, aptly named Duck Duck Blog. The inaugural post is What People Search First at Duck Duck Go. The general blog feed is here.

The idea for the new blog is to give you a unique view on search from inside a search engine. In particular, I hope the posts will be on things that you won't find elsewhere and will have a propensity towards data and technical stuff. So add it your blog reader!

I'm still going to post about Duck Duck Go here (on this blog), but will stick to non-search specific things like Perl code and CSS/JS hacks.

Duck Duck Go on CNN Home Page!

2009-05-16T14:50:31Z

Duck Duck Go is on the top of the CNN home page! Well, our logo is :). Sadly, there is no link to us in the story, but I'm not complaining at all. Seriously.

I couldn't be more happy and proud.None of my projects have ever been on CNN, let alone in the lead featured article.I'm even more proud because the other 5 companies pictured all have ridiculous funding. While they have over $150M in combined funding, we are bootstrapped.

Believe it or not, even without the link, we're still getting a decent amount of traffic though. t's funny--the Go of Duck Duck Go is not in the picture so we're getting a lot of requests from people searching "duck duck" in Google and CNN.

Anyway here's the story link for when it is off the home page. And here's a snapshot that documents this was real!

Really Disappointed with MIT Cutting Sports

2009-04-25T01:23:08Z

Today I am really really disappointed in MIT, my alma mater. Yesterday they officially announced that they are cutting eight varsity sports. No longer will we be able to say we offer "the most varsity sports of any university."

The stupidest part about it is the reason: "the severity of the economic downturn." Cutting these sports will apparently save $485,000. Compare that to MIT's endowment of $10B (yes, that is billion). And MIT's endowment actually rose in fiscal year 2008 by 3.2%, or $88M!

So let me get this straight. MIT has $10B and recently made $88M on it, and is cutting something dear to the MIT community that costs a meager $485K. That is pure insanity. What is the endowment for if not to protect things in uncertain times that people care deeply about? Adding insult to injury, it was only a few years ago that MIT spent over $50M on a new athletic center. So you spend all that money and then cut sports at a small fraction of that cost?

Not only that, look at what they are cutting. Among the sports is MIT's pistol team, which has historically been one of the best performing MIT sports. In 2007, they won the national championship, beating out Army and Navy! And that is not the first time. Yes, you read that right--Army and Navy...

To some, perhaps most, this may not seem like a big deal. MIT students don't play sports, right? Wrong! MIT has long prided itself with it's sports offerings and participation. It is also used as a recruiting tool. I've been interviewing students for admission for several years now, and it pretty much comes up in every interview. That stat about our breadth of sports quickly puts to rest any doubts that MIT students do nothing more than academics. Believe it or not, this is paramount on most prospective students' minds.

I guess the bottom line is I just don't get it. It is a trivial amount of money compared to the MIT endowment. I can understand that bureaucracy may have created this problem. That is, the administration asks every department to cut 15%, and this is what the athletic department came up with. But then the administration should step in and revoke this short-sighted idea.

MIT is almost a 150-year-old institution and they are managing it like a public company who are slaves to quarterly earnings. The endowment has been growing steadily for decades. Yes, this might be a bad year. But there will be good ones again soon. Just the same as you shouldn't over-spend in the good years, you shouldn't under-spend in the bad ones. MIT should take a long-term view and consistently maintain the average MIT experience and culture. Why would anyone want to donate to an institution that is managing their endowment so poorly?

How Twitter Helps Me (Specific Use Cases)

2009-04-20T15:02:24Z

The recent surge in twitter press is amazing. A lot of my less technical friends and family have been asking me about it, usually with skeptical tones. Why would you do that? You use it--really?

At first, I too was skeptical. It took me a while to adopt and then even longer to make regular updates. Even now I've only made 322 of them.

Recently I had a long discussion over email about the value of this type of technology. The short version is, upon reflection, I've really come around to its usefulness. So I thought I'd enumerate the use cases where I've found twitter particularly helpful, both personally and professionally.

It keeps an eye out for me. The very first day I started using twitter, I was at the right place at the wrong time. I had showed up 8 hours early for one of our monthly hackathons. I had twittered where I was and my friend almost immediately informed me of my mistake.
It's an extension of my Facebook status. If you use the Facebook status update feature, then you might as well use twitter too. In fact, when I update my twitter feed, it updates my Facebook status automatically (via the Twitter Facebook app).

My set of twitter followers and Facebook friends has a decent amount of overlap, but by no means all. Because they are different sites and different technologies they tend to accumulate slightly different types of associations. In particular, it's common to get Twitter followers who you've never met. But you also get friends who for whatever reason aren't your Facebook friends (at least not yet). By syncing the updates you are reaching more of your friends with your personal updates. I like that because I suck at keeping in touch on a one-to-one basis.
It keeps me closer to customers. I monitor twitter search for mentions of Duck Duck Go (my startup). It is really interesting to see what people say "in the wild" and because of twitter's ease of use you get more (and different) commenting than you find on blogs.
It allows for more (and different) customer communication. I follow the users I find in #3 using my company's twitter feed. I also send out updates about things going on with the site. It turns out that this method of two-way communication generates different feedback than our other methods to connect with customers, e.g. this blog, our feedback page, Facebook, etc. I've even gotten unsolicited bug reports!
It helps me meet more people (in person). I'm an INTJ and have historically been terrible at meeting new people. Twitter really eases that burden for me. As pointed out by another friend, it gives you great ice-breakers when you meet people for the first time (who you've previously connected with on Twitter). For example, I follow a lot of people in the Philly startup community, but have not yet met all of them in person.
It makes me feel more connected. I work from home, by myself. Twitter helps me get to a semblance of a multi-person office.
It helps me spread news/links. You need a decent amount of followers for this one. I don't really have that many, but with the amount I do have it is enough to get benefit when spreading news and links about my company or otherwise. Often people will re-tweet exciting news (re-send my message to their followers).
It helps me keep abreast of news/links. Similar to #7, but from the other side, I often get news/links from people I follow. These are usually things I would have never seen, e.g. info about other startups, local events, etc.Useful enough for you?

How I Got Skype High Quality Video Working

2009-04-15T14:46:20Z

Ever since Eli was born, we've been skyping with my parents regularly. My parents supposedly bought a high quality webcam, but we weren't seeing them great. And I know my built-in laptop cam wasn't up to par, so a few days ago I set out to make it all work.

Skype says this is what you need to make it work:

Latest version of skype.
A Core Duo or similarly fast processor.
One of these Logitech webcams.
The latest webcam drivers.
A fast broadband connection.

I'm not sure if Skype actually checks the drivers to make sure you have one of these webcams, but my parents actually did have one and so I bought one too. I actually bought this one, which has auto face-tracking that I figure would be good fit when Eli gets a bit older. However, once I set it all up, neither direction was what you would call high quality. Mine was certainly better than before, but that was mainly due to the lighting adjustments in the software that came with the webcam.

Anyway, after a lot of messing around, I did get it to work as much as I think possible. Here's how:

Know that Skype "ramps up" to high quality. This wasn't obvious to me initially and caused a lot of time wasting. In particular, Skype apparently does a lot of internal calculations to see what it can send over time. It starts off lower, and then slowly increases the video quality, and in turn, CPU & bandwidth used. So just know that if you don't see it right away, give it a minute or so and see if it improves.
Turn on "technical info". On windows this is under Tools->Opotions->Advanced->Advanced Settings. Once you do this, initiate a video call and then mouse over the incoming video window. A pop-up should appear with all sorts of debugging info that will help you diagnose what is between you and the high quality video. I'm not sure what it all means, but I think you want to concentrate on a few things.

First, look on the bottom where it says "problems" and then gives you some codes. The goal is to get rid of all of these codes. BW stands for BandWidth, and on mine it was critical, i.e. BW_Critical.

The next thing to look for is the video sent and video recv lines. You want to send and receive at 640x480 at a decent frames per sec (FPS>=15). It turns out my parents were sending at 320x240 and both of our FPS were low.

Finally, look for Relays. You want this to be 0. When it is not, you do not have a direct connection with your video partner. Instead, your connection is routed through other computers. I did not have a direct connection, and this turned out to be causing the BW_Critical.
Force 640x480. I was sending at 640x480, but as I said, my parents were not. Turns out while they have a decently fast computer, their processor isn't enough for Skype to trigger the attempt to send video in high quality. After a lot of searching and messing around, I found the solution. You want to put this in your config.xml file:

<CaptureHeight>480</CaptureHeight>
<CaptureWidth>640</CaptureWidth>
<Fps>30</Fps>

On Windows XP, this file lives in Documents and Settings/Login Name/Application Data/Skype/Profile Name/. In general on Windows, you want to find your Application Data folder (on Vista I think replace Documents and Settings with Users) and then follow the same skype path.

After I did this, my parents were sending at 640x480. It turns out their processor was fast enough, and we monitored the CPU usage via the technical call info I talked about above. (You could also do this with the task manager.)
Open the incoming port on your firewall. Go back to Tools->Options->Advanced, but this time select Connection. There is a port # at the top. I opened this port on my firewall and forwarded it to that machine. I also unchecked the box "Use port 80 and 443..." to make sure it was working. Also, if you check "Enable uPnP" you might get this port forwarding set up automatically. Also be aware of any software firewalls you have, and to make exceptions for skype within them. Once I did all that, Relays were now 0, i.e. we had a direct connection.
Plug the webcam directly into the computer. I was using a USB hub. Apparently the newer high quality webcams can transfer a lot of stuff and a shared USB hub can become the limiting factor. Similarly, if you have a lot of USB devices in use, even directly connected to the computer, try unconnecting some of them. You might be being limited by your comptuer's USB bus.
Check your bandwidth speed. You can do so here. I have a great connection due to Verizon FIOS. My parents, less so. Turns out, after all the above, I started sending awesome video, and it was transferring at about 500Kbits/sec! My parents increased their quality as well, but their broadband connection caps at about 150Kbits/sec, and so that became the limiting factor. This showed up in the technical call info problems as BW_Low. They are now looking to upgrade their connection. But if you notice quality does get better, but it is still sort of blurry, check your bandwidth. You can see how much bandwidth skype is currently using by again looking at that technical call info. Again, mine capped around 500Kbits/sec, so I suspect you should expect something similar.

Well, that's it. It was definitely worth it. My parents can now see Eli probably 10 times better, and soon he will be able to see them the same.

Search Engine Optimization (SEO) Tips

2009-04-08T21:04:21Z

I often get asked about Search Engine Optimization (SEO) because I run a search engine. The following is currently my best advice with regards to SEO. Just to be clear, I'm talking about Google here (and by extension Yahoo, Live, etc.) and not Duck Duck Go (my search engine), which has different optimization parameters.

I assume you already know the basics, e.g. don't have duplicate content, have good looking URLs and page titles, etc. If you don't know the basics, read Google's SEO Starter Guide first.

External link (anchor) text is the most important factor. Google often repeats that they use more than 200 ranking factors. I'm not disputing that :). I think they repeat this, however, so that people won't concentrate on the few factors that really matter more than all the others. In my experience, external link text matters way more than anything else. That is, the actual words in the links back to your sites and pages.
If you get link text right, link back sites' pagerank matter way less. Usual SEO advice is to contentrate on link backs from high pagerank sources. If you want to improve your overall rankings I think this is good general advice because pageank is logarithmic, such that a link from a higher pagerank site will have more of an effect than one from a lower pagerank site than you might otherwise think.

However, higher pagerank links are much harder to get than lower pagerank links. And what is often missed is that if you can get even a few lower pageank sites to link back to you using the key phrases you want to rank for, you can rank highly on those search terms even with little or no high pagerank link backs.
Widgets are great strategies. As a result of the above, widgets are great strategies because they help you get link-acks from a variety of sites, often on their front pages and often on multiple pages within the sites. That is, it doesn't matter if they are all low pagerank sites because you can control the link text. But be careful. Google is all over so-called "widget bait."

If you do a widget, don't forget you need that static link in it. That is, it can't all be JavaScript.
Don't guess search term volume. Use Google's Keyword & Trends tools.
Select terms that convert. Ideally, don't guess here either. Use Adwords or another PPC service to test out which terms convert best, and then try to get rankings for them.
Don't bother if you can't get in the top 10. People usually revise their search terms instead of clicking to page 2. So if you don't think you can get in the top 10, try something else, ideally something more specific that would convert better anyway. A good thing to do is to add one more word to the general term, so if you end up ranking well for the specific term you are helping your rankings for the more general one as well. Then if it turns out the specific one was easy (you quickly become #1), you've already gone part way on the general one.
Don't pay any for any general SEO service. Not only do you not need to (because it isn't that hard) but you don't really know what you're getting and as a result you will risk getting blacklisted. The worst is signing up for paid link services. Note I'm not saying don't pay anyone for SEO, because if you need basic help, a consult from someone in the know might be helpful to, for example, tell you how to re-layout your site and to explain this post to you :).
Beware of nofollow links. Nofollow links are links with a special attribute that tell search engines to ignore them. To check a link, view the source of that page in your Web browser and look for rel=nofollow in it. You don't want to waste time getting links on sites that use this attribute. The canonical examples are popular blogs and Wikipedia. Don't waste your time submitting comments and editing Wikipedia articles with your links because it won't help you.
Don't waste your time with Google Sitemaps. Google encourages you to submit sitemaps of your sites. In theory this seems great because you can specify site priority and refresh time. In practice, I've done this repeatedly and seen no change in rankings.
Don't ignore the long-tail. People often concentrate on getting rankings for specific keywords. But I've been just as successful building lots of pages with unique content that end up ranking high on the most random things. Often such pages will be the only things that come up.

How do you do this? First, make sure you have a static site. Sometimes startups don't. Second, look at all the content you can produce or have produced. Can you combine it in interesting ways that people would find useful? For example, at Duck Duck Go we have category pages.
Make as flat a site hierarchy as possible. Pagerank seems to flow logarithmicly from a homepage to its internal pages. So if you have pages you want ranked highly, either you need links back to them directly or have them linked directly from your homepage. And if you take my long-tail advice and make a lot of pages, make a directory of them as flat as possible. For example, if you have 10,000 pages, make 100 pages with 100 links and link to those 100 right off your homepage. Ideally those 100 links would make sense, e.g. categories or alphabetical or something (and not just random).
Use directories instead of subdomains. For example, domain/blog instead of blog.domain.
Less is more. Ranking is distributed across your site, so less pages, less links on them, and less text on them will concentrate your ranking potential on what is left.
Don't do anything black hat. You will get caught, you will not pass go, etc.

Update: additional comments can be found here.

Fast and the Furious Twitter Hilarity

2009-03-23T12:21:14Z

Last night I sent this tweet: "Finally watching The Fast and the Furious: Tokyo Drift." Literally within a minute or two TokyoDriftMovie starts following me on twitter. His bio says:

I'm looking for other people who enjoy The Fast and The Furious: Tokyo Drift Movie as much as I do, maybe that's asking too much. You at least have to enjoy it.

So does he (Rob Jones) have a bot set up that follows anyone who mentions the movie? His profile links to an amazon store page where you can buy the movie. Is he doing this for many movies? If so, he takes really great pride in the details of his work--just check out his twitter stream, e.g. "I stopped keeping track after seeing Tokyo Drift 100 times..."

Duck Duck Go Architecture

2009-03-20T14:49:18Z

I often get asked what Duck Duck Go "runs on." This post basically answers that question by outlining the major moving parts that serve queries, i.e. its architecture. I'll detail in another post what, in particular, makes it fast, i.e. tunables and other specifics.

Caveat: this architecture was designed for maximum query speed for our initial soft launch. While also somewhat designed for eventual scalability, we don't have that much traffic yet (though we are growing at a nice clip). So don't take this as advice like you might get at High Scalability. It's really just for your amusement. However, my last startup did have some scale (relatively speaking of course) so I know a bit about what I'm doing...

DNS served by DNS Made Easy. I used to serve it myself via djbdns, but DNS Made Easy is faster, makes it easier for me to deal with fail-over, and cheap.
All requests come into nginx. I used to use two instances of Apache, one for dynamic requests and one for static files. But nginx is faster, uses less memory, and is more stable.
If a static file, nginx serves it directly, e.g. the home page. It's really good at that.
Otherwise, nginx checks my memcached store. I hadn't used memcached before this, and find it a big win.
If not in memcached store, nginx proxies to FastCGI processes that are running in the background. I hadn't used FastCGI before this, as I always had used mod_perl with Apache.
The FastCGI processes are managed by daemontools (as is memcached). At first I was worried about stability in these processes, but it hasn't proved to be an issue yet.
Internally, the FastCGI scripts are written in Perl and run by the FCGI::Engine Perl module.
The Perl scripts access a PostgreSQL database (when needed) to retrieve our zero-click information, among other things.
The whole thing runs on FreeBSD.
For fail-over and scalability purposes, I have EC2 images that replicate the above except that they run on Ubuntu (since, at the time, FreeBSD wasn't available).
All of our site icons and zero-click info images are hosted on S3.
We also reference some external YUI JS files.

Any questions?

Also, I'd love any feedback on this architecture. I'm always looking for ways to speed it up!

Update: additional comments can be found here.

Twitter: Please Charge me for Biz Tweets instead of Suspending my Account!

2009-03-10T15:40:56Z

Short version: I violated twitter TOS, so my account was suspended (as it should have been). However, I think what I was doing was worthwhile. So what I suggest is a new, paid tweets section (akin to direct messages in functionality, but you pay to send them). So if the receivers never wants to see them, they don't have to. But if they do, the paid tweets are one-click away...

***

Last night the Duck Duck Go twitter account was suspended. I was sleep deprived from the birth of my son, but yet still awake being a human baby monitor. So I quietly used the computer to pass the time (like I'm doing now!).

Lacking SSH access (hospital wifi blocks port 22 for some reason) I was essentially blocked from doing any real work. So I decided to mess around with twitter's search feature, which everyone seems to be talking about lately.

It really is cool. There were a ton of people talking about very specific things I'm really interested in, new search engines being one of them. So of course, I wanted to communicate with them.

So what did I do? Naturally, I found the relevant tweets, and replied to them. I'm still not sure if you can actually reply to a particular tweet, but I did click the reply link from each tweet's individual page. They were all tweets about checking out new search engines.

My reply said something like "@username here's another new search engine, which is live now: http://duckduckgo.com. What do you think? Yes, it's ours :)". Some people replied very quickly with feedback, so I know my tweets at least were somewhat relevant. I'm guessing others would have replied too, but I can't tell for sure because my account was suspended shortly afterwards.

So I naturally went to the twitter support page on account suspensions. It does look like I violated this term: "updates consist of duplicate or repeating links and/or text." That's exactly what I did. OK, fine. I sent an apology to them so hopefully my account will be reinstated soon.

To the bigger question though: what should I have done? Not sent the tweets of course...but is there some way to have the same effect without getting my account suspended? For example, should I have just changed up the message each time? Probably wouldn't have mattered. Sent to less people? But all those people seemed to express equal interest.

Thinking about it some more, I don't really see any way in the current twitter scheme to send tweets like this without risking suspension. And yet, there is value here (on both sides). I clearly want the right people to check out the search engine. And people who post tweets about checking out new search engines, presumably, on average, want to check out a new one!

So here's what I suggest, which I'm sure has been suggested before but I'm too lazy to find out where. (And one more nudge won't hurt anyone...):

Twitter should charge me to send tweets like these, i.e. unsolicited business related tweets. The charging part both would cut down on spam and make them money. But wait, isn't it still some form of spam? Agreed, and, as such, it both messes up the twitter user experience for those receiving them (their reply screen) and those people following our twitter stream, which is annoying.

To fix this issue, twitter should also make a new type of message (like DMs) for this stuff. Instead of 'd ' make it 'p '. Each time I make a paid tweet, charge me $0.05. And on the receiver end, put them in their own window so they don't clutter up anything else. That way only the receiver sees them, and only when they want to.

Update: additional comments can be found here.

If You're Into It

2009-02-14T18:08:35Z

Gabriel Weinberg

Thoughts on Yahoo! BOSS Monetization Announcement

2009-02-12T15:12:18Z

Yesterday Yahoo! announced upcoming usage fees for its BOSS API. The announcement has caused quite a stir on the official discussion group and elsewhere. As the founder of a new search engine that uses BOSS, this news clearly effects me. Here are my initial thoughts on the matter...

It's likely good for me (Duck Duck Go). General search engines should end up monetizing at $10+ CPM. At a cost of about $1 CPM, I'd most likely make more profit in this scheme than in a revenue share, which would probably be >10%.
It does feel like bait and switch. Yahoo! says it isn't because they said monetization was always coming at some point. And that's true. However, the expectation from anyone I've ever talked to about it (and mine) was that it would be a revenue share.

That's not a coincidence. The BOSS terms previously disallowed any third party ads. And whenever anyone asked about it, Yahoo! said its ads are currently being tested on some sites, and are coming soon. My guess is they did try to test it with some sites and either got caught up in cross-group bureaucracy rolling it out or it didn't monetize as well as they had initially hoped (or both...).
It's likely not good for a lot of services. Developers can get screwed here on both the cost and revenue side. I use BOSS as an add-on of sorts, and so I generally fall into the cheapest queries in their system. But others using BOSS are doing things like grabbing the first 1000 results (or 10 results using 10 parallel queries) and then deciding which are the best x to show. Innovative ideas like these cost a lot more in their scheme and may easily push these services' costs above the break even point.

On the revenue side, general search may monetize relatively well, but plenty of other things don't. Facebook gets something like $0.20 CPM. Not everyone using BOSS is building traditional-looking search engines...
It's short-sighted. The second sentence on the BOSS Web site is still "The goal of BOSS is simple: to foster innovation in the search industry." This new scheme is contrary to that goal. Truly innovative search ideas may take a long time to figure out monetization that works. And in the mean time their costs may be significantly higher than their revenue in the proposed scheme.

Fundamentally, as pointed out on the discussion group, this pricing may be completely disconnected from how a particular site makes money, e.g. (from the linked message):

"The amount of $$ I make from my site is not based on the number of results I request (what you are charging me for), but is instead based upon the number of ads that my users click on."
Yahoo should make money on BOSS. I don't think anyone is really disputing this goal. Making money on BOSS would be great for the developers, guaranteeing its longevity and quality of service.
Yahoo should offer revenue sharing as an alternative. Yahoo should offer developers a choice of how they want to pay for BOSS, either this usage pricing or a revenue share from serving Yahoo! ads.
Yahoo should offer both options to all developers. People are talking about grandfathering in current BOSS developers (to what exactly, I'm not completely sure but I suppose it is rev share). I think that would be great, but I think they should offer the rev share alternative to all developers if the true goal is innovation. This will let truly innovative services (with questionable initial monetization) get started. If the developers see fit, they could then switch to the usage pricing.
Revenue sharing could be great for Yahoo's overall CPM. As I said in #1, revenue might make more money in many cases anyway. But beyond that there is potentially a more important network effect at play. The proposed Yahoo/Google deal (that fell through) exposed Yahoo's lower CPM rates. Part of the problem here is they don't have enough advertiser competition on their network, which is a symptom of not enough ads.

Yahoo claims BOSS is serving 10M queries a day, and growing steadily. That's a lot of potential ads, which should improve Yahoo's overall CPM over time. That effect alone, could end up making up whatever differential Yahoo currently sees between the two pricing schemes.
Marginal costs are certainly way below the proposed prices. Yes, Yahoo should make money. And yes, their differential pricing does make sense in the context that those queries they are charging more for do probably cost them more to serve up. But I'd guess that the marginal cost of serving those API calls are orders of magnitude less than the proposed pricing. Think about all caching they can do...
Microsoft, where are you? I've been asking this question in a lot of contexts for years (here's one). There is a great opportunity for Microsoft (and Ask for that matter) to jump in here at a much lower price point given the marginal costs.

Jump On It

2009-02-01T19:57:31Z

And if you like that...check out these.