Update: Vik Singh had the idea for BOSS, and posted Yahoo! Boss - An Insider’s View. It’s money line is this, and describes the big idea succinctly: “I think users should be confident that if they searched in a search box on any page in the whole wide web that they’ll get results that are just as good as Yahoo/Google and only better.”

First, here’s what happened tonight:

Yahoo! is taking a bold step tonight: opening up its index and search engine to any outside developers who want to incorporate Yahoo! Search’s content and functionality into search engines on their own sites. The company that sees just over 20% of the searches performed each day believes that the new program, called BOSS (Build Your Own Search Service), could create a cadre of small search engines that in aggregate will outstrip their own market share and leave Google with less than 50% of the search market.

Might this impact things? He thinks so:

In both cases, Yahoo! BOSS is intended to level the playing field and blow the Big 3 wide open. We agree that it’s very exciting to imagine thousands of new Yahoo! powered niche search engines proliferating. Could Yahoo! plus the respective strengths and communities of all these new players challenge Google? We think they could.

And that part that was music to my ears (emphasis mine):

It is clear, though, that BOSS falls well within the companies overall technical strategy of openness. When it comes to web standards, openness and support for the ecosystem of innovation - there may be no other major vendor online that is as strong as Yahoo! is today. These are times of openness, where some believe that no single vendor’s technology and genius alone can match the creativity of an empowered open market of developers. Yahoo! is positioning itself as leaders of this movement.

Marshall, thanks for the great writeup. Yahoo!, thanks for making me proud.

Some worry that the acquisition will hurt the effort to make Twitter scale. It can’t and won’t.

I believe Twitter’s engineering team is headed up a mountain (they need to switch architectures at a low level), but that they finally know which mountain. True, it’s a tall mountain not quickly climbed. But they finally know their problems and have people in place. Better days ahead.

Others worry that Twitter’s scaling ills will infect Summize. I don’t think that’s possible because they are distinctly different engineering problems. Summize is “fresh search,” an understood and known problem that Summize apparently designed for from the beginning. Twitter, in contrast, evolved a product into a service that no longer matches their architectural model. It didn’t start out as (and therefore wasn’t built to be) a massive-to-massive (when each massively is unique, personal, exponentially expanding) real-time messaging protocol. I believe architectures exist for that problem space, but unfortunately that’s not how Twitter was initially built.

Put briefly, Twitter’s already on the path to health and Summize is immune from Twitter’s disease, so it should all work out fine.

While they are different systems, they may be complimentary. Jettisoning Twitter’s track and reply functionality to Summize’s infrastructure may offer Twitter engineers the headroom they need to roll updates into Twitter’s codebase with a bit of a cushion.

Subscribe to me on FriendFeed.

One feature as a user of FriendFeed is the inline media. MP3s and video can be played right within the page. Click “play” on an audio file to start listening and expose slim player controls. When you play a video the thumbnail grows to standard video player size. For photos / flickr thumbnails, clicking “more” displays the entire set or upload.

Because I’m testing out a new screen capturing process, here are some pictures of the inline media collapsed and expanded so you can see how it works inline.

Here’s what the audio looks like initially.

And here’s how it expands when you click play.

Same for video. Initially you see a thumbnail of the move.

And when you click play the player grows into view.

Works like a charm.

As the name implies, they publish big photos. Not thumbnails or small one-column photos like most news sites (and sites in general), but true large format photos. Generally 990×660. It’s remarkable the greater impact that larger photos can have.

Today’s feature is on California’s Continuing Fires.

There are a LOT of fires burning. Coming home from Golden Gate Park yesterday after Tasty, we crested Twin Peaks and had an eastern view of the entire bay as we drove on Portola Drive. In near unison we all noted the “fire smog.” The air is thick with smoke, even in SF which is currently fairly removed from the fires.

About a month ago, my buddy Matt’s house in the Santa Cruz mountains came within a kilometer or two of being engulfed. If the winds had been normal his house would have been gone. But luckily the winds were anomalously blowing the opposite directly. They evacuated, but were spared.

About 10 days ago, my friend Jud’s mom was evacuated from her home in the Brisbane hills just a few 2 or 3 miles south of SF. She avoided disaster, too.

Last week I flew out of SFO. All flights were delayed because of lack of visibility due to fire smoke in the air across the whole region.

So, take a look at the fires through the Big Picture lens to get a better sense of what’s really going on, and the amazingly tough and dedicated firefighters. There are more than 20,000 hard-core people out there fighting to get it all under control.

Here’s to them.

And here’s the feed for the Big Picture so you can add it to your reader.

This year, my third presenting at @media in London (2006, 2007), Patrick offered me the morning plenary slot. I used the time to talk about a topic of great interest to me: Professional Frontend Engineering.

Over the last three or four years the role of Frontend Engineering has become more important, more respected, more challenging, and more in-demand than ever before, and so I wanted to put a stake in the ground clarifying what we do, how we do it, and why it’s so important to raise it to a professional level. I had four goals:

• Put a stake in the ground.
• Reiterate our values.
• Advocate the discipline.
• Nurture a healthy Web.

The goals were threaded throughout the four sections of the talk:

• Historical Perspective
• Our Beliefs & Principles
• Knowledge Areas & Best Practices
• Why It All Matters

The talk is embedded below (or download: keynote, pdf, quicktime).

I think this topic is critical to the advancement of the Internet. I’ll be writing more about this in these pages in the coming weeks and months, but for now enjoy the slides. And please share your thoughts and feedback in the comments.

| View | Upload your own

The talk embedded below (or download: keynote, pdf, quicktime) covers several well-known optimization practices then quickly moves to review more recent findings and advancements. It concludes with a survey of tools for optimization and links for more information.

Enjoy, and please leave a comment with any thoughts you have.

| View | Upload your own

Next week Tuesday I’ll be presenting an updated “High Performance Web Sites” talk at the inaugural Kings of Code conference in Amsterdam. From there I’m headed to the second half of London Web Week and will be giving a talk called “Professional Frontend Engineering” in the Friday plenary slot at the outstanding @media conference.

The Kings of Code conference is shaping up to be a great event. I’m excited to hear what fellow speakers John Resig, Peter-Paul Koch (PPK), Folke Lemaitre, Nate Abele, Mark Birbeck, and host Robert Gaal have to share with us.

The @media conference is equally impressive. It’s consistently been one of my very favorite events for the last few years. The speakers are insightful and generous, the attendees are smart and engaged, and Patrick and the rest of the organizers put on a warm, welcoming, and action-packed event with lots of time for networking, hallway conversations, and a wee bit of pub-based debauchery. Spread over two days it promises to saturate us all with inspiration and insight.

Please email me, leave a comment below, or shoot me a note of Twitter (follow me) if you’re going to be in the area and want to catch up. If you let me know in advance that our paths will cross I’ll be sure to bring you a little gift.

Now if somebody could please do something about the #$%#@$# exchange rate…

Tomorrow afternoon I’m presenting an internal Tech Talk to designers and engineers at the Yahoo! Taiwan office, hosted by my friend and colleague Aaron Wu. I love the chance to talk to designers and engineers in the same room, and so I’m very much looking forward to the opportunity.

On Saturday I’m offering the keynote at the Open Source Developers’ Conference here in Taipei. My talk is titled “An Insider’s Tour of the YUI Library.” I’ve been experimenting with video clips in my talks lately, and so even though I’m the only member of the YUI team on this trip, I’ll have the video and voices of many from the team with me on stage. I’ve done something similar once before, and it went well then so I’m hoping it goes well again.

Here is some local press coverage of the conference. It’s a trip to see my face surrounded by words I can’t read. If anybody can translate for me, please send me a note or leave a comment (click the images for higher-res copies).

The third event is an interview for that same publication scheduled by Yahoo!’s local “tech PR” team. I’m not used to giving in-person interviews, let alone via translator, so it should be a fun and unique (and flattering) experience. They sent over a few of the questions in advance to set expectations and I gotta say the questions are thought provoking and interesting. (Though I am a little worried about how to translate some of the more fuzzy terminology.)

The fun continues on Monday and Tuesday with my fourth and fifth even is as many days: I have the distinct privilege of address engineering and CS students from both National Taiwan University and the National Chiao Tung University. Each two hour session is part presentation, part on-stage interview with professors, and part question-and-answer. My message is that Frontend Engineering is a first-rate engineering discipline, that industry is hungry for more skills practitioners in the field, and that it’s quite likely the most interesting and stimulating role to play in web and internet development.

I’m exceptionally humbled to be able to speak at such esteemed institutions. I will do my best to live up to the honor. Taiwan: Thank you!

everything in this article is my paraphrasing of speakers’ presentations. not my own words.

(Video coming soon.)

1. We run web applications. We’re only focused on this narrow goal.
2. We handle the entire lifecycle of an app.
3. Apps are run on Google infrastructure.

“It’s hard, but it’s worth it for us.”

“For the first time you can use the same infra we use…Auth, GOS, BigTable”

The Stack

1. Scalable serving infra
2. python runtime
3. SDK
4. Web based admin console
5. DataStore

Demo: App from scratch in 8 minutes.

More details

1. Scalable Serving Infrastructure: fault tolerant (redundant). Fluid: don’t need to schedule needs up front… more servers come online dynamically.
2. Python Runtime and Libraries. All tools are generic, so new languages can be dropped in later. Python used in same python available otherwise. Goal: you can use any language eventually. We don’t want to limit you.
3. SDK: Environment to develop apps locally. Avail for Linux, Mac, Windows today. (But can probably work anywhere.)
4. Admin Console: web-based admin console. (Looks like google finance meets google analytics.) Tools for request logs. Data explorer. Usage/quote numbers. App-version balancing. Can hook up domain (don’t need to run at *.appspot.com).
5. Scalable Datastore. Schemaless object store. Not a clustered sql thing. Instead based on BigTable. (Whitepapers online.) Horizontally scalable. Reacts to hotspots. BigTable instead of SQL is a big change, and may take some time to get used to. But we think you’ll come to like it. Schemaless means you can add a new datatype or entity whenever - no need to update your schema.

Now we’re looking at a Datastore Model Class.

GQL Query example

SELECT *
FROM Story
WHERE title = 'App Engine Launch'
AND author = :current_user
AND rating >= 10
ORDER BY rating, created DESC

Other Notes

Mail Sending API

no setup needed.

Make HTTP Requests

Authenticate with Google Accounts

Frameworks

The whole Django framework.

Guido van Rossum: Creator of Python and member of Google App Engine team

My passion is making life easier for developers. With python i’ve done that for decades. Now i’ve joined GAE team. Excited by potential. (and that python was first picked)

First time that GOogle has let third-party people run software on their infra. That’s fundamentally a big deal.

8:13 PM “We’re offing 100% of the python lang.”

8:14 PM - we don’t offer threads, but you won’t been it because of our scalable arch.

GAE uses a quota system so nobody monopolizes the infra.

me: if it’s so scalable, why do they need the quotes?

What’s Next?

• large upload/download support
• purchase additional capacity
• other language support
• offline processing.

Here are a few of those sites: FireEagle for location data (a single geocode), TripIt for travel data, Delicious for links data (a single URL+ tags), ThingFo for experience data (in 30 chars), Twitter for vitality data (140 chars).

These APIs make possible an undeniable wave of creative hacks within the small orbit of any of the services even individually. This growth testify to the mass variety of niche needs and personal priorities. It seems the ocean of data is really a petri dish.

When these hacks cross-pollenate — when the ins and outs of the data sets start sharing and talking with each other — things get even more interesting.

Those that dismiss mashups as simply “things on a map,” “widgets on a blog,” or “applications on facebook” don’t see the full power. I don’t claim to either, but important coolness seems inevitable when data becomes small and abundant while APIs become prolific and potent. More small pieces fit together more ways.

(Perhaps this is a small part of why Douglas Crockford says that “Mashups are the most interesting innovation in software development in decades.”)