My Blog’s Application

I have a rarely updated blog at my personal site, DLewis.net.  But hey, it’s so easy to build an application, why not!  Here’s the code for mine.  I highlighted the stuff you need to change in red text.  All you need to do is confirm that Google’s blog search indexes your blog, and make the changes below.

WISE.apps.dlewisnet ={
"categories":[ "Blogs" ],
"id": "dlewisnet ",
"name": "DLewis.net ",
"author": "Dan Lewis, User:DNL on Wikia Search ",
"description": "Returns blog posts from www.dlewis.net ",
"regex": /(?:(.*) (?:dlewisnet|dlewis|dan lewis )|(?:dlewisnet|dlewis|dan lewis ) (.*))/i,
"apptitle": "Posts about $1 on DLewis.net ",
"appurl": "http://www.dlewis.net ",
"gen_obj": {"request": {"url": "http://ajax.googleapis.com/ajax/services/search/blogs?v=1.0&callback=WISE.generic.processApiResult", "search": "q", "q_append": " site:dlewis.net "}, "response": {"styles": {"item_style": "", "last_item_style": "", "image_style": "", "url_style": ""}, "size": "3", "rootelement": "responseData", "containerElement": "results", "titleField": "titleNoFormatting", "url": {"field": "postUrl"}, "contentFields": [{"prefix": "", "field": "content", "suffix": "", "limit": "100", "type": ""}]}}};

You’ll note that almost all the red stuff is references to me or my blog.  The short version: Change that stuff over to you and your blog.

The Specifics

You have to change a total of nine things.  The first line is easy — just pick a unique identifier for your WISEApp.  I chose "WISE.apps.dlewisnet" because I’m nearly 100% sure that no one else has any claim on "dlewisnet".

Next is the id line.  Just echo what you did above.

After that, give your app a name.  I chose DLewis.net … for obvious reasons!  Then sign your app on the author line, and in the description line, give a description.  Something short and to the point is perfect.

The regex line is the key part, and the hardest — but it, too, is really easy!  Pick some key terms which should trigger your application.  Separate them with pipes (that’s the | symbol.  It’s shift-backslash, near your enter/return key).   Yes, you need to have it twice — this way, the application will trigger on any search which starts or ends with one of your triggers.  Just remember to keep your triggers specific to your blog.  If you go too broadly, we’re going to have to reject your application.   Me?  I chose "dlewis", "dlewisnet", and "dan lewis".

One other rule for the "regex" field: If you choose multiple triggers, and one of them contains another (e.g. "dlewisnet" contains "dlewis"), put the one that contains the other first.  So I have to have dlewisnet|dlewis and not the other way around.

The "apptitle" field is next.   This is going to display on the search result itself.  The $1 is the query the searcher types in, so leave that in there!  Change the "DLewis.net" to your blog’s name.  In my example, if a searcher enters for "dan lewis copyright", the apptitle will display "Posts about copyright on DLewis.net".

The "appurl" field is the link to your blog.  I chose to send the link to my blog, itself, because I don’t have a good internal search.  You could change it to include the search term — just replace the term with "$1" (no quotes) in whatever URL you provide.

The final change is in the "gen_obj" area.  Change "dlewis.net" to your blog’s domain.  Don’t get creative here or the application won’t work.  We’re piggy-backing of Google’s Blog Search API to get your data, remember.

You Can Build Them For Your Favorite Blogs, Too!

I’m a big baseball fans, and David Pinto’s Baseball Musings is an every-day visit.  There’s no internal search, though, so I built this … in about 90 seconds:

WISE.apps.baseballmusings ={
"categories":[ "Blogs" ],
"id": "baseballmusings",
"name": "Baseball Musings",
"author": " Dan Lewis, User:DNL on Wikia Search ",
"description": " Returns blog posts from www.baseballmusings.com ",
"regex": /(?:(.*) (?: baseballmusings|baseball musings|david pinto )|(?: baseballmusings|baseball musings|david pinto ) (.*))/i,
"apptitle": "Posts about $1 on Baseball Musings ",
"appurl": " http://www.baseballmusings.com ",
"gen_obj": {"request": {"url": "http://ajax.googleapis.com/ajax/services/search/blogs?v=1.0&callback=WISE.generic.processApiResult", "search": "q", "q_append": " site:baseballmusings.com"}, "response": {"styles": {"item_style": "", "last_item_style": "", "image_style": "", "url_style": ""}, "size": "3", "rootelement": "responseData", "containerElement": "results", "titleField": "titleNoFormatting", "url": {"field": "postUrl"}, "contentFields": [{"prefix": "", "field": "content", "suffix": "", "limit": "100", "type": ""}]}}};

Test Your WISEApp

This step is optional, but a good idea.  Go to our Application Testing Grounds and do three things:

1. In the first box, "Query", type in a query which uses one of your triggers — something like "dan lewis copyright" for my first one.  Test for all of your triggers — in my case, I’d also do "dlewis copyright" and "dlewisnet copyright".
2. Paste that entire block of code into the box labeled "Application Definition Object".  Leave the third box blank. We’re making this easy!
3. Hit that "Test Your Application" button and watch as your preview appears.  If it does not work, make sure you replaced the right things with valid info.

Last Step: Submit Your WISEApp to the WISEGuys!

Just email that block of code to wiseguys@wikia.com , and we’ll review it post-haste!

In fact, both of the ones above were built this morning and are already live:

• "dan lewis copyright "
• "david wright baseball musings "

•    Continued work on application framework, including adding tools to show available applications
•    Put up new KT which has admin tools.  Keywords can now be protected –  admin-only,  registered users only, or everyone.
•    Working on fixing encoding problem
•    Performed emergency fixes required by MediaWiki upgrade
•    Deployed version two of the API at ISC
•    Integrating toolbar translations for 9 languages.
•    Continuing work on refactoring toolbar.
•    Working on getting hadoop streaming active.  Will allow us to write MapReduce jobs in perl/python/php/shell, etc.
•    Working on shard prototype.
•    Planning around index expansion
•    UI tweaks
•    Testing alternative UI formats
•    Finished construction and testing of our new protoshard

Search is a 90-10 problem. Today, we have a 90% solution: I could answer all of my unanswered Saturday questions, not ideally or easily, but I could get it done with today’s search tool. [. . .] However, that remaining 10% of the problem really represents 90% (in fact, more than 90%) of the work.

She’s entirely right — although unintetionally humble, as the yeoman’s effort and execution by Google in search cannot be understated.

In building anything, the first barrier is the fundamental one.   Arrington equates search to man’s efforts to fly, and that’s apt.  The Wright Brothers had to figure out the fundamental question: How do we get off the ground?   After that, everything else was gloss — but gloss is, often, incredibly important.   Getting across the Atlantic, those little vaccuum-sealed meals, movies on board, not losing your luggage, in-air WiFi, etc. are all exceptional improvements in flying,  but flying itself had to come first.  That was 90% of the problem, and we should recognize that. The other 10%?  Well, no one wants to fly on a propeller plane.

Google’s search product is (obviously) much better than the propeller plane was an air travel method, but the foundational structure Google created, along with precedessors such as AltaVista, are similarly the first, big step on a path toward much greater things.  Google’s ability to index the web is the gold standard in building a usable, useful search product.  Their algorithm proved that machines can provide, as Mayer noted, an effective — albeit neither ideal nor truly easy — method of finding important information.

The 10% is a different problem.  It’s about improving the experience and quality of search.  That’s evidenced by two things: first, the volume of startups trying very different things, but all in the search sphere and aimed at those goals; and second, the fact that the big guys — Google with their Wikia-like tools, Yahoo with their new bells and whistles, and Microsoft with it’s ever-changing Live product — are diverging from each other.   The question is no longer “who can index/sort it all” — that’s becoming a commodity.  The question is morphing, and we’re not exactly sure what its actual words are.

That’s part of the reason Wikia Search started with four organizational principles — Transparency, Community, Privacy, and Quality — which you can read more about here.  We do not know, nor claim some hidden insight, into how search will mature over the next year, five years, and so on.  We do, however, know that there’s a right way to go about things, and that way will lead to a superior product.   The right way allows everyone to participate in the process and benefit from its fruits.  Let’s get there together.

I tossed “marathon recall” into Wikia Search and the second result, under a now-deleted link to an irrelevant wWikipedia article, was to About.com’s Heather’s Baby Products Blog about the recall.  Browsing around, I came to Britax’s recall-specific website, at http://marathonrecall.com.  I added the page to the “marathon recall” query using the Wikia Toolbar (URL).  We checked the car seat and, good news, the car seat was not part of the recall.

A few days later, a friend asked me about the recall again, and this time, Wikia Search pointed me to the right answer — the Marathon Recall site I entered in before.  I had to visit the site anyway, so I used our Annotation tool to add the step-by-step instructions into the search result itself.    Here’s what it looked like:

On Sunday night, my wife and I were looking at a new car seat — don’t ask — and the new car seat maker claimed that the Marathon only held children up to 48 pounds.   Neither my wife or I believed that, and more importantly, I remembered reading something, somewhere, which said it could hold more.   And for some reason, I thought to check the “marathon recall” entry.  The answer was right there.  So dutifully, I emailed the link to my wife.

What I realized today is that I, entirely accidentally, used Wikia Search as a group/social bookmarking tool.   I built a search result which, for me, has lasting effect; and in going so, I had efficiently created a bookmark that my wife and our friends could use with the same efficacy.    Here’s a step-by-step guide for those who want to do the same:

Step one (optional): Register for a Wikia account. (Click here to register.)

When you add or modify your search result, we’ll record it for you on your profile.    Check out my profile and scroll to “Recent Activity” to see what I mean.

At Wikia Search, Transparency is a core principle.  Search results can come from a lot of places — an algorithm, a user contribution, and more to come — and we pride ourselves in letting the world know where a result came from, and why.  Therefore, when you add to or modify on Wikia Search, we let the world know.  If you do not have a Wikia account or are logged out, your contribution will be attributed to your IP address.  So you are probably best off logging in!

Step two: Add/Edit the URL.   (Hint: Download the Wikia toolbar to make this step incredibly easy.)

If you’re using our toolbar, just click the “Add URL to Keyword” button and enter in a relevant search term.  Rate the URL five stars.  Then hit that “Edit” button and clean up the result.  Finally, use the “Annotate URL” and/or “Comment” tool to explain what parts of the page are specifically important.

Not using the toolbar?  Don’t worry.   Copy the URL you want to add, then go to Wikia Search and search for whatever relevant term you want to add that URL to.  Use the on-page tool to add the URL to that search result.  Mouse over your result and the other tools — edit, comment, annotate, etc. — will be there for you to use.

Step three:  Tell the other person/people what keywords you used, and to try that query on Wikia Search. (Another hint?  If you signed up in Step 1, you can send your friend the URL of your profile page — with your contributions right there.)

That one is self explanatory, but if you want to show me what you’ve built, feel free to email it over to me at dan at wikia dot com.  Maybe I’ll show it off here!

And with that, you’re done!   You’ve created a hyper-relevant, robust search result — and you’ve also created a great bookmark you and your friends can share as easily as you’d share a URL.

Oops.

And no delete button in sight.

Good thing we have one.

(For what it’s worth, Google fixed that query — the math doesn’t appear.  But the similarly worthless “math” appears for things like “seven days a week“. )

Image via the excellent Google Blogoscoped

User Interface

• The new translate tool is up and running at http://re.search.wikia.com/translate.html.  You can translate the interface into around 100 different languages.
• We’re working steadily on the search application framework.
• In progress, a re-workof the core main.js file and its logic flow — cleaning it up an streamlining it.

Operations

• Over half the team was out in San Francisco, adding more hardware to the setup over at ISC.  Once we get more power there, we will have more than doubled our amount of used hardware.

Toolbar

• Added Wikia Search as homepage
• Added Wikia Search to default engine list and as default search engine
• Opened Wikia Search help on first run
• Made bar resizable
• Working on code refactoring, adding new functionality, and integration of other languages

In a battle between two search engines, Google and Wikia Search are letting users rank search results. While Google’s experiment is random, Wikipedia and Wikia Founder Jimmy Wales says Wikia Search is about building a community using simple Web services tools such as ranking and tagging to let users influence search results. Google has no timetable for its search ranking feature and isn’t saying how it works.

That’s from yesterday’s eWeek article by Clint Boulton, titled “Google Search Ranking Feature Threatens Wikia Search“.   I bolded the last five words, but first, I want to say that I absolutely love eWeek’s headline.  They’re mostly right — Google likes to copy what we do.  We’re flattered, honestly.

That said, flattered, yes.  Threatened?  Not really.  Why?  Because their culture endures.

Ben Gomes, a Google engineer, blogged lightly about the “experiment” on Tuesday.  Here is the entirety of his comments on the experiment:

“At this point, I can’t say what we expect from this feature; we’re just curious to see how it will be used.”

In an 850-plus word post, it’s good to know that Google will spend 22 to tell us that (a) they’re collecting data and (b) they aren’t going to share it with us.   That is, they don’t value your privacy — everything you do, they’re watching; and they don’t value transparency — they’re going to keep all that information, all their code, and all their algorithms under lock and key.

Jimmy, in the eWeek article, shows that Google does not value the community aspect that Wikia Search holds dear, either.

I always draw a distinction between things that I would consider to be crowdsourcing—and I consider that to be a very negative term—that aren’t really thinking about a community of people getting together to discuss thoughtfully and work together to build something, but rather it’s atomistic. [. . .]  [Google's experiment is] very different from what we’re doing, which is trying to build all of the tools for a community to come together and take editorial control of the search engine altogether.

Yesterday, I argued that Search needs to be read/write, and that quality demands that users and others be able to explicitly modify search results.   We are, and should be happy to again hear that Google may drop their paternalism down by a marginal amount.  But it’s a baby step toward adopting one — just one — of Wikia Search’s four organizational principles.  And as always, Google’s minor, experimental (!) improvement is made at the permanent cost of the other three.

That’s a quote from an old notepad I found recently.  It’s not verbatim, but close enough  And it’s about nine years old.

In 1995, I entered college.  The Web was, relatively speaking, new.  The first dot-com era was in its pre-manic upswing; broadband access was available but by no means ubiquitous.  As a freshma, I saw the difficulties my university had in wiring the residential buildings for high speed Internet access, going room to room doing things obsolete by today’s best practices: explaining the difference between a CAT-5 cable and a telephone wire, installing ethernet cards, registering each ethernet card’s MAC address with the DHCP server, and poo-pooing irate students who found out that Windowns 95 was not yet supported by the University — it was a Windows 3.1 world!  Students checked email by telnetting into the school’s UNIX shell and issuing what to the unenlightened were archaic commands such as “pine” or “elm”, and sending an attachment was, to most, impossible.  Those who were well-versed in the ways of 1s and 0s used arguably better tools: WS-FTP or, if you were *really* a hacker-type, Eudora.

By the time I graduated in 1999, the process had gotten better: Windows 95 was a supported OS, and a student could self-register with the DHCP server.  So, too, had the experience: Eudora and MS Outlook were common, and Napster was all the rage.  Excite, HotBot, DirectHit, Lycos, and Altavista were making it easy to find interesting information on the web; Yahoo was making it even easier.  Online shopping was also a new revelation — eBay, Amazon, and the neer-do-well Pets.com.  So when I signed up for a Computer Law class, I expected cutting-edge discussions of what the future would bring.  In four years, the Web had grown up.  So, I figured, should have the law.

Nope:  “[A]lmost all websites can be treated like billboards.”

The rational for such a silly sentence — uttered by my professor — was neither ignorance not malice.   It was, rather, correct — at least insofar as the law was concerned.  The Internet posed many new problems, and attorneys and judges alike clung to brick-and-mortor exemplars.  When a car company’s website acted as a big advertisement which offered some additional information about their line of automobiles, it was, indeed, a high-tech billboard.  And when one speaks of websites from the late 1990s of brick-and-mortor companies, that’s probably the norm.  Defenestrating our “the Internet is new!” problem was both efficient and arguably appropriate — as a stop-gap.

But as we know now, and as many of us knew then, the analogy is inapt. The Internet is not read-only. It is not a billboard; not a magazine; not the side of a blimp.  We can interact with it, create our own content, etc.  It is read/write.

Wikia Search believes that the read-only Web, to put it bluntly, sucks.  Quality — one of the core principles of Wikia Search — demands that the Web be read/write.  The Internet is too powerful a medium, the Web too powerful a platform, to relegate ourselves to being glorified billboards.

Search is an integral part of the Web, and it, too, must therefore be read/write.  Google, Yahoo, MS, and Ask are just glorified card catalogs, akin to those found at your local public library.  They are brick-and-mortar solutions to a very new problems, and are necessarily lacking.   We’re changing that.

Toolbar
* Wikia Evolution, the search toolbar launched.  Download it at http://re.search.wikia.com/toolbar/download.html
* Want to tinker with the code?  Grab it via the SVN at http://svn.swlabs.org/re.search/cool/toolbar/ - there’s no documentation yet, but we’re working on it.  The test .xpi (evolution.test.xpi) can be unzipped as per typical for a .zip file.

UI Stuff
(If you’re not on the search-ui miling list, that’s at http://lists.wikia.com/mailman/listinfo/search-ui)
* More work on widget/application framework
* Working on a search engine comparison tool
* Started on a “light” fork to the results UI.

Atlas
(There’s a distribution list for Atlas at http://lists.wikia.com/mailman/listinfo/atlas-l and a wiki page about the project at http://search.wikia.com/wiki/Atlas.  Check out both)
* Updated the Atlas protocol spec for “knuggets”
* A lot of the prototypes are starting to come together.  Atlas-l has a discussion starting at http://lists.wikia.com/pipermail/atlas-l/2008-July/000092.html and the SVN is at http://svn.swlabs.org/atlas/.

Operations
* Assisted with Nutch re-index
* Started a new crawl
* Fixes to KT importer, added the ability to load/populate the new location table
* Built a new 0.1.3 Hbase cluster, loaded with production data snapshot, populated the new location table, setup new KT code (with new features) pointed to the new cluster with new data (kt.search.isc.org/ktdev/)
* Tweaked lots of system monitoring
* Lots of work with the crawler, trying to find the source of very high fetch failure rates
* Deploy-redploy KT /ktdev/, started review of code
* Determining new hardware requirements
* Bind updates
* More work on Grub

Nutch
* Finished test rollout of new indexing and scoring systems.
* Started working on shard management servers.
* Started work on pornography and bad content identification.
* Started integration of kt input into analysis algorithms.
* Documentation, bug fixes, and unit tests for new scoring and indexing frameworks for Nutch.  Working to get final patches submitted and committed into the Nutch core.
* Finished new crawl, working on deployment and roll-out of new indexing and scoring systems to test
* Finished all patches and code documentation for new scoring and indexing systems.  Everything has been submitted to Nutch for inclusion in the Nutch code distribution.
* Finished modifications to FieldIndexer including field filter extension point, and field-basic, field-boost plugins that integration in the arbitrary boosting with the new indexing framework.

Other Stuff
* Improved contact importer for the social tools
* Working on a Facebook application

The paper is a quick read — even with footnotes, it’s a mere 11 pages — so it’s not fair of me to republish it here.  I’ll certainly have more thoughts on it later on, though.