Know It When I See It

Twitter could be better for advertisers than Google.

noreply@blogger.com (Pete Aven) — Sat, 18 Apr 2009 15:37:00 +0000

I've read a ton of stuff over the last few weeks on Twitter and a lot of the articles seem to be in the category of "How can Twitter make money?" And there's been a lot of interesting ideas thrown about. I'm a huge fan of Twitter and I think they actually have multiple money making opportunities available to them, but here I thought I'd explore one in particular, Network Based Marketing.

Now, if you know me, you know I tend not to be a fan of online companies that use advertising (or being acquired) as their sole means of generating cash. But companies pay, and get paid, billions for advertising online. I really think there's an opportunity for Twitter here, so I thought I'd throw in my two-cents since I haven't read it anywhere else.

Let's take a quick look at advertising. From a high-level, there's a few distinct ways for a company to advertise its products/services. A company can:

1) blindly carpet-bomb a region with flyers and see who that brings in to the store/site
2) target distinct groups within a region, send them flyers, and let those customers targeted get the word out to others for them (demographics)
3) target social networks, send flyers to groups based on connections, let those customers targeted get the word out to others in the network for them

( and 4): collaborative filtering ; fits into this list somewhere. its a hybrid of 2 and 3 and can be seen with amazon's : "people who bought X also bought Y" mechanism. There's an opportunity here to, but probably requires a post of its own.)

Continuing, let's take phone companies as an example because they figured out #3 a long time ago, I'm sure you've heard of the "Friends and Family" plans? And since there can be a lot of people in a network, I'll use a fictional character "Ashton" to assist with the following examples.

If you carpet-bomb a region with adverts, Ashton may pick up the advert, say "Oh, that looks interesting." And then forget over time to even go to the store/site to investigate that product he thought he was interested in when he first read/saw/heard the ad.

You may have better luck if Ashton fits within a certain demographic, as then you can tailor the language of an advert to Ashton, but you're likely to still end up with the same results. Worse yet, Ashton might not be recognized by the attributes used to define the demographic. As a potential customer, Ashton may just fall through the cracks, overlooked and missed by the marketing team altogether.

But if Ashton's friend or family member is sent a flyer and that member then either buys the product, investigates it, or talks about it, Ashton will wind up having a strong connection to the product. Some real feeling will be associated with the product/service based on the strength of the connection between Ashton and those other members of his friends and family who share their experiences of the product with him. As more members of Ashton's group adopt products/services, they provide steady reminders to Ashton of their interactions with the product/service and there's a repeating message, delivered from someone he's close with, to get to the store/site and check-out/buy those products.

So, I'm trying to keep this post as succinct as possible. For real detail, go check out Shawndra Hill's, Foster Provost's, and Chris Volinksy's paper on Network Based Marketing. They provide direct, statistical support for the hypothesis that network linkage can directly affect product/service adoption. I also have friends who were able to recreate the experiment at Yahoo! and were able to increase sales of a specific service there by targeting "friends and family". In a nutshell: consumers linked to a prior customer adopt a product/service at a rate 3–5 times greater than groups targeted using the best practices of any firm's marketing team. So I'm not making any of this stuff up, its actually based on science and research.

Phone companies can determine a person's network just by monitoring who you call. They can look at the people you dial most ( and who dials you ) and can pretty easily figure out who's in your actual social network, not an online site, but the group of people you are active in and participate with daily. They've found that instead of canvassing the nation with flyers selling the latest phones and services, they can target "friends and family" (the top n% of people you dial/are-dialed-by most) and have much greater success (Sales!) with their advertisements, by leveraging people's social networks. When advertisers target the right people, the advertisement information diffuses through the network, and other people end up doing marketing for the company, whether they intend to or not.

Now, Google makes a lot of money through advertising. As an advertiser you put your billboards on the streets most traveled and everyone online uses Google for search. But with Google, you have keywords and intent to help target ads. When a person searches, there is an explicit intent to know more about what they're searching for. Google advertising is similar to traditional advertising in a magazine. If you open a magazine about fishing, you don't mind seeing ads for fishing equipment. If you open a fashion mag, you expect to see ads for clothes, cologne, etc. Online, the magazine's always changing as the pages are composed from various search results. Google targets ads based on a user's intent (keyword search terms) and the words in the page being viewed. (I'm sure they also bring demographic info into the mix somewhere.) So Google gets advertisers closer to their market, and if all else fails ...

Google uses a shotgun approach that's slightly more intelligent as the ads blasted at you are based on keywords in your search terms, search results, or emails. This doesn't always work out though. For this post I searched on some sample terms for examples. I searched on "movie times" and an ad/link came up for "Gay Spirituality" in addition to moviefone.com, etc. , so go figure how the algorithm works here. Google does a great job for the most part, but could do better if they didn't have to work within certain constraints.

Google/Yahoo could serve up much better adverts if they could target a person's social network. The tools they have to unlock those networks though are email and chat. Unfortunately for them, most users (us) tend to think of these as private tools. Whenever any company attempts to touch a person's contacts or inbox, issues of Privacy arise and people get seriously riled up about maintaining their anonymity/privacy. All these companies analyze our networks internally, but letting that fact out to the public scares them as they know the response will be negative. They've also been seriously scarred by the AOL fiasco. So they do the best they can internally with all the data they've stored from our interactions, but their hands are tied: It's their kobayashi maru scenario.

Enter Twitter: It's all public. Tweets are all blogs. Now, the naive keep looking at follow/followcount as some indicator of a social network or a person's relevance within a network, but just how a phone company can determine a person's network by phone-number and who's calling who, we can determine a person's network on Twitter by user-id and the use of the @ sign (who's replying to who, how often, etc.). Where ever you have replies back and forth (use of @) , you can increase the relevance of the connection between individuals. Do this a few times over a few connections, and you have a closer approximation to a person's real network, not the false one found in follow/following counts.

With Twitter, the privacy argument is thrown out the window. People who post on Twitter understand its a public forum. So just off simple queries based on @userid, you now have the ability to start serving ads to "friends and family" on Twitter. But wait, it gets even better...

In addition to user-ids and the network, you also have people's conversations. People form networks around similar interests, and they naturally tend to use similar language (either determined by locale, interest, topic, lingo, etc.) within certain groups, and discuss the same products, services, events, etc.. In addition, Tweeters use the # sign to categorize discussions. So there's a tremendous amount of classification info available to enrich people's social networks with. All that classification info can be used to build advertisements that target specific groups, and voila, you now have friends and family plans for any product via Twitter.

So there's definitely an opportunity here for Twitter and a Google/Microsoft/Yahoo search division to partner. Even off of simple keywords in tweets you'd probably get better targeted ads as you could tailor keyword dictionaries based on the network and the domain its in.

Personally, I think that should be a short-term goal. Twitter could target ads better using a combination of a person's network, the context of keywords within tweets, and behavior. Mobile fits in here too of course.

But right now, with some simple social network analysis and some simple keyword usage of their own, Twitter could deliver their own adverts and they'd be way more effective for advertisers than what they're paying for with the traditional approaches used by search companies today. Leveraging what Twitter has for advertising could be a way to bring in some revenue in the short-term, until other opportunities manifest themselves in Twitter's other plans and innovations. Even if they come out with a different, successful business plan tomorrow, it seems there's a huge opportunity for revenue here that shouldn't be overlooked. And as they'll be able to distribute ads with an improved targeting mechanism that will be more effective for advertisers, Twitter shouldn't have to distribute as many ads to still make some serious coin.

I get very excited about the opportunities available to Twitter. I have an interest in using both NLP and SNA to help people find what they're looking for online and in the enterprise. Twitter's dataset is perfect for this! The plan is to post here soon with some demonstrations that leverage the Twitter API. But until then, you all really need to check out what @petewarden has created with Mailana. He's already started to deliver tweeter's social networks based on who's actually communicating with each other. Very cool.

Facebook/Twitter Networks and the Stream

noreply@blogger.com (Pete Aven) — Mon, 09 Mar 2009 18:49:00 +0000

Facebook will be updating its user's home pages very soon, and much has already been made about how these pages are going to become more Twitter-esque in appearance and functionality. Status updates and postings from friends will now take center stage, updating in real time. The tour of the updated home page looks very cool; it's like Twitter on steroids. I'm excited to see what it will look and feel like when it's released, and how it will be used by those on the Book.

But I don't think Facebook or Twitter will immediately dominate the other company for their user base anytime soon. The updated FB home page presents a major shift in the way social network home pages have been used up to now, and it may take some time for people to shift away from the Twitter, and put all their eggs in the FB basket. I could be completely wrong, so it will be interesting to see what happens. But the different types of networks and the strengths of the ties people have with one another on Twitter vs. Facebook may present a challenge to any FB/Twitter plan for FB/Twitter/World dominance.

Here's why: On Twitter, my profile is kind of a sidenote. Not much is made of it by users, and it's not something we look at or groom. It's more like a concise business card providing a little insight into who we are. We can post and never reply to anyone if we wish. When we follow/unfollow/block a fellow Twitterer, it's just opening and closing different pipes of communication. Who follows me or who I follow may something about who I am, but probably doesn't, so I don't think you can assign to much relevance to the connections, as people are just opening and closing circuits to find posts they're interested in.

Many times the follower/following people have never met each other in real life, or even communicated directly with one another on Twitter. Also, networks of communication come and go . Networks of communication are established ad-hoc on Twitter based on common interest areas, goals, and shared experiences. For example: People may only tweet at a conference, because everyone else is. Or at that conference was the only time people had a common interest/goal to tweet with each other about. People come and go on Twitter, and there's a ton of people tweeting, so we get to maintain some level of anonymity.

In a nutshell: Twitter's a personality subscription service that enables ad-hoc interest networks to form and share information. The networks are very dynamic and subject to rapid change.

On Facebook, I have to request a friendship, I have to say how I know that person, they have to confirm and agree to the connection. I may have to provide an email address as proof of my participation in a group. There are lots of pictures of us; some posted by us, some posted by others, we get tagged. Our profiles include much richer information that we enter ourselves when we set up our home pages. We tailor that info for our audience, which includes our families, our friends, our coworkers; all people who get to see each other once we're "friends". These connections have a lot of relevance as they're people we interact with in everyday life, or have interacted with seriously at some point in life.

There's ZERO anonymity, the connections are much more meaningful, and the connections are much more static. The connections I have on FB I had long before there ever was a FB, and will be with me for the rest of my life. These links include shared history and experience that motivate us to stay in touch, either willfully or by obligation. These connections also have levels of trust and reputation associated with them.

Twitter to me is like going to the pub after work, or to a party on Saturday night. You meet up with a group of people. This group may include people you know, maybe not. You have a few drinks, talk a bunch of smack, tell jokes, do something crazy, then go home... to Facebook.

Facebook's where I present my best face. I may not want my Mom or the CEO from my company to know who I was with at the pub. I may not want them to know what we we're talking about earlier, or the jokes that were told, that information could be completely inappropriate for them.

So though Facebook's updated home page and Twitter's standard functionality may look similar, they actually support different groups of people with different motivations. Knowing this, it'll be very interesting to see what happens in the future with their services and what action each company takes to support and capitalize off of its respective audience.

Twitter, Networks, and Finding What I'm Looking For

noreply@blogger.com (Pete Aven) — Sun, 08 Mar 2009 21:01:00 +0000

I thought I'd try and put some of my thoughts on Twitter out there in-a-nutshell style. Enjoy!

Twitter 101
Twitter is a worldwide party that everyone's been invited to. Throughout the duration of the party, people come and go. While present, people contribute to conversations and the party in general in the form of mini blog posts called "tweets". You show up and can choose to walk around and enter/check-out the conversations at will. You can listen to what people have to say, or you can participate and give back by posting your own tweets.

You listen in by choosing to "follow" someone. People have the option of not letting you listen in, and can insure you/others won't ruin their existing conversations with inappropriate remarks by not following and/or "block"ing you. Likewise, if you follow someone and they turn out to be a jackass, or talk about things you don't really care about, you can choose to not follow them anymore as well. In this way you can control the endless stream of information that's generated in tweets by those in attendance and tailor the flow of communication to meet to your personal requirements/desires/etc.

Score-Double-Bonus-Action: You know when you go to a real party, and you're standing with a group of people having a great conversation, and some jackass keeps interrupting with something that's inappropriate, a downer, or just plain offensive? Well, imagine if you had a mute button for that person. Or how about when the conversation you were previously interested in has deteriorated to a point where you want to just walk away, but don't want to seem rude in doing so? Well, Twitter provides us with both options and both are acceptable behavior in Twitterville. Sweet!

I call them conversations, and sometimes they are. We can reply to each other on twitter and when we reply, its a direct reply to the original tweeter as well as a remark shared with those following the conversation. Others may choose to chime in with their own replies, either directly addressing a person, or just contributing to the conversation in general by tagging their comment with a #identifier. Many times though, a "conversation" is very one-sided; it's really a broadcast. (You always have people at the party who talk to show off and/or hear themselves speak, but don't really care or know how to listen to what others have to say.) After you've "follow"ed several people, you really end up using the follow/nofollow/block options to help harvest the large river of broadcasts you want to subscribe too/participate in.

Technology + People = Twitter
I won't say Twitter is simple technology. And simple does not mean easy. Digging a ditch is simple but not easy. You have a shovel, some dirt, you dig, simple right? Well, you're going to get calluses and blisters, the Sun's going to beat down on you, you'll hit rocks as you dig: Not easy stuff! So while Twitter's UI and communication mechanisms seem simple, we know there are some serious challenges too keeping the river of communication flowing, and major props go to the Twitter team there! The mechanisms provided by Twitter that enable people of shared interest to communicate with and find each other are just enough. The rest of the magic and awesomeness that is Twitter and the reason many find it so useful and valuable is really delivered through the people and their tweets.

People who tweet are usually bringing something to the party, in the form of sharing news, information, insights, links to blog posts, and/or other resources that we're all interested in (globally and locally). Some of that info has no other source than word of mouth, and Twitter gives people a real-time broadcast mechanism for information that might otherwise not be reported until later (or at all).

Back to our real party analogy: You know those people/friends who always show up to the party with something to share: beer/jokes/stories to tell/food, etc.? And then there are those who tend to always be the mooches? Well, I haven't found too many mooches on Twitter. Also, if there's a fire down the street, someone usually enters the party and says "did you know so-and-so's home is on fire?". You learn about it then and there and can take action if required, as opposed to learning about it the next day in the paper, online, etc. or when the fire eventually spreads to your home.

On Twitter, existing relationships can be maintained, and new relationships are formed within these networks of constant communication that start with a simple "follow" relationship and naturally evolve over time. My interests lie primarily in technology and new media. Specifically, I follow people who are doing something with XQuery, NLP, Social Network Analysis, Music, and Vlogs. So sometimes these people participate in the same discussions because they move in the same circles for a certain topic area, and many times they don't. But I get to listen and participate in it all.

Imagine going to a party attended by all the people who you consider to be the coolest and most insightful for all the stuff you're interested in and you have the opportunity to talk to them all and/or hear them speak that night. Your interests are somewhat disparate, so these people are standing around having great discussions about the things you appreciate most, but they are also all maintaining their discussions in distinctly separate groups. You want to talk to them all, but during the course of the night, you will only have a chance to participate in 1 or 2 of the discussions at most. On Twitter, we're given the means to participate with everybody we're interested in and in all groups, and oh yeah, no time limit: the party never ends! Woo-Haa!

Twitter provides me with access to others of common interest and an opportunity to hear and be heard. It's a lot of fun, but also extremely useful.

How Twitter Helps Me Find Stuff
1) Twitter provides me with access to people who already have answers.

With Twitter I've been given access to communication networks of common interest. Yesterday I tweeted that I was looking for something. Before I even had a chance to Google it, a person with whom I'd formed a mutual "follow" relationship 2 days prior (around a shared interest in XQuery) replied with a solution immediately. Boom! Immediate answers!, not just a list of links to explore. (and the question/answer had nothing actually to do w/ XQuery.)

2) Twitter creates parallel Googling.

When I Google, I Google alone. Google's great, but Google to me is like the magic 8 ball of search. I enter a term, shake vigorously, and check out the first few pages of the 8 billion links returned in the window. If I don't find what I'm looking for, I change the question, shake again, repeat, etc. and hopefully at some point I finally find an answer I'm satisfied with. On Twitter, I'll have a number of people in my network Googling alongside me for the same resource, entering keywords based on their way of thinking as opposed to mine, and/or seeing things I may miss. Now I don't specifically ask people to Google. But if they have the same interests I do, chances are they're interested in finding the same info and may be looking for it already.

Usually people within a network search for the same things. So what I'm looking for or have found, someone else in my network is likely to look for in the future and/or has found an answer for already. So why even Google everything on the internet when my network will be looking for similar resources within a much smaller domain? An example:

Imagine we're all in the same history class, and the day's assignment is to write a certain essay on George Washington. When we go to do the homework that night, we'll all be looking for the same resources! So we can start with Google, which will give us results based on the keywords in our query, with no context or respect for the network we're in and what may have been searched/found for by us already, or we can tweet and see if anyone from class has already found some resources for us to start reviewing. If I find something, I can post the link on Twitter to be helpful, as I know others in my network from class (you, in this example) will be looking for it too. Maybe someone in the group did a report last year on a related topic, has some info to share, and tweets that. Or maybe someone's Dad just happens to be some sort of George Washington expert and tweets that! Twitter helps expose and take advantage of all that information.

With Twitter our search is augmented by the number of eyeballs, brains, and experiences attached to our follower/following network and people on Twitter tend to want to be helpful. People may have a direct link to us through a "follow"ing relationship, or they may just find us based on our previous tweets using a tool like summize.

3) Twitter helps me find stuff that isn't on the internet.

Google works great if what I'm looking for is on the internet, but on Twitter, through my network, I'm given help to find stuff in the real world that might not be indexed by a traditional search engine. Questions get answered and information gets shared about events and meet ups that occur in the real world for which there may be nothing to search for until someone blogs about it or reports on it after the fact. How about our George Washington expert above? Maybe there's nothing online about our friend's parent who can help us with our report, but Twitter just helped us find that person through a network of shared interest and common goal.

4) Twitter provides me with answers before I even figure out how to ask the question.

One of the qualities I find most valuable and appreciate the most about Twitter, is the IDKWIIBIKIWISI factor. (I don't know what it is, but I'll know it when I see it.)

How many of us spend time looking for that? I spend a lot, hence the name of the blog. But on Twitter, the information finds me, shared by people with similar interests through a simple tweet.

Twitter's just the utility for communication. Twitter doesn't have any logic to provide users with what they're looking for, but it enables people to fuel, view, and direct information so it gets to the people who have an interest in it. The networks that naturally form through "follow"ing expose me to conversations and resources I didn't even know existed, and/or wouldn't even know how to begin searching for. And its all just links, times, dates, addresses, and other resources people within a given interest group share through tweets as they find them. The end result is I get to learn more about the stuff I'm actually interested in, and I don't even have to go out and look for it, just start following and tweeting, and eventually someone finds me and/or shares.

Rocky Mountain News Final Edition

noreply@blogger.com (Pete Aven) — Sun, 01 Mar 2009 18:43:00 +0000

Part moving tribute, part documentary, this is 20 minutes i found fascinating. The people who report and make the paper happen are so dedicated , hard-working, intelligent, and sincere. The video really demonstrates the love and respect reporters have for their jobs and their readers, as well as the value a newspaper brings to a community. The vid also captures some valid insights comparing the value of reporters and papers to bloggers and the internets. Yet the newspaper absolutely refuses to change anything about its business model and that's shown here as well.

It's like watching people on a fast moving train headed for a collapsed tunnel. They know they're going to crash, but they think they're just passengers, so they keep doing what they do best (reporting), until the train finally crashes. Since papers refuse to change absolutely anything about themselves, it appears currently these people only have the choice of trying to find another doomed train to ride.

I'm rooting for a group to wake up, change the newspaper model, take the best of what this group offers, and succeed where EW Scripps fails so miserably.

See for yourself:

Final Edition from Matthew Roberts on Vimeo.

Word 2007 Tip of the Day: Re-Enable Your AddIn

noreply@blogger.com (Pete Aven) — Tue, 29 Jul 2008 00:17:00 +0000

Sometimes Word will disable AddIns. You'll be looking for a Custom Task Pane you've installed, or maybe some Ribbon customization, and the AddIn will seem to have just disappeared. Why does Word do this? Oh, it could be one of many reasons: You may install an AddIn that has the same namespace as another you've already installed, or maybe you monkeyed with the XML in a .docx and Word choked on your edits when it tried to open the document. For whatever reason, when Word experiences a conflict with AddIns or XML within documents, it will disable AddIns and focus on recovering the document.

You don't have to re-install the AddIn. Just go re-enable it.

Tip #3: Re-Enable your AddIn

1) Goto The Button

2)Click Word Options

3)Select AddIns in the left Pane of the "Word Options" dialog.

4) Select COM Add-ins in the dropdown next to "Manage" at the bottom of the "Word Options" dialog, and click "Go...".

5)Check the checkbox next to the AddIn you want to enable and then click "OK".

Most of the time, this re-enables our AddIn successfully until the next time we hammer Word with an AddIn or document it doesn't like.

If the AddIn fails to enable, Word may provide a reason as to why it can't re-enable the AddIn. This may, or may not, be helpful. Or, the AddIn may just fail silently and fail to enable. Most of the time I'm able to re-enable an AddIn without issue, but there's something else we can do to help us identify issues when enabling AddIns. That's a tip for tomorrow though.

Word 2007 Tip of the Day: Document Properties

noreply@blogger.com (Pete Aven) — Mon, 28 Jul 2008 02:48:00 +0000

You can add document properties to Word docs. If you want to do this regularly, you can just add an icon to the Quick Access Toolbar that will display the Document Properties form for you.

Tip #2: Add Document Properties to the Quick Access Toolbar

1) Next to The Button

is an area called the Quick Access Toolbar, in it you'll see icons for saving, redo, and undo.

Click the down arrow on the right side of the QAT.

2) Select "More Commands" from the menu.

The Word Options dialog appears, and it's opened to the Customize tab.

3) Under "Choose Commands From" above the left pane, select "All Commands".
4) Scroll down the left pane and select "Properties".
5) Click the Add button in the center of the panes.

In the right pane, you'll see the icon for Document Properties, added to the already existing icons in the QAT.

6) Click the OK button on the lower right.
7) Now click the Document Properties icon to display the default properties form for Word documents.

You can enter data into this form that will be saved as metadata in a piece named /docProps/core.xml within the .docx package.

Word 2007 Tip of the Day: Enable the Developer Tab

noreply@blogger.com (Pete Aven) — Sat, 26 Jul 2008 22:40:00 +0000

OK peeps, as notes to myself, and to help others who frequently ask, I'm going to try and just post a Word tip of the day on a regular basis. (Not every day, let's not get carried away now.) If we're dealing with something genuinely cool, maybe we'll even give it a Tip of the Hat, and if it sucks, a Wag of the Finger. To Start:

Tip #1: Enable the Developer Tab

1) Goto The Button

2)Click Word Options

3)Check the box: Show Developer Tab in the Ribbon

4) Click OK, and Boo-Yaa!

Goto the Developer Tab and Enjoy!

The plan is to use this space to capture interaction with Word documents through the Word 2007 UI as well as to provide C# and XQuery examples of how to manipulate these docs on the Server or in an AddIn.

Social Search

noreply@blogger.com (Pete Aven) — Fri, 01 Feb 2008 21:31:00 +0000

It's here, check out Delver and this article on Technology Review.

ODF vs. OpenXML: Win or Lose, We Win

noreply@blogger.com (Pete Aven) — Mon, 07 Jan 2008 01:11:00 +0000

Happy New Year All! Ok, I've been meaning to post this for awhile now, and I recently had a friend ask me about it, so here we go.

I'm very excited about XML, XQuery, and XML servers. As one who's toiled in many a RDBMS trying to do something useful with unstructured/semi-structured data, this tech brings some serious heat to the party. Right now, as many know, there's this war going on for what's going to be the document format of the future. It's like the documents are the spice and Baron Harkonnen can be heard saying "HE WHO CONTROLS THE FORMAT, CONTROLS THE UNIVERSE!". ( That's a Dune movie reference people. )

While the last comment is meant to bring levity to the situation, people really seem to take their document formats very, very seriously. With all that's going on in the world, I'm often amazed at the amount of energy put into championing one format over another, but this is the thing: at the end of the day, Open Office documents and Office 2007 documents are just bags of XML now, so win or lose in this battle, we win.

Once we have the XML, we can make the documents into anything we want them to be. With an XML server, XQuery, and minimal knowledge about the formats, it's ridiculously easy to change a document of one format type into the other or even some completely different format altogether. So today I thought I'd provide a simple example.

We'll start with the Excel 2007 spreadsheet from this previous post and an OpenOffice Calc spreadsheet using the parts I'll provide below. You can either create these using XQuery, or just create the individual files manually and zip them up to create the .xlsx and .ods documents. I've named my documents, creatively enough, as Sheet1.ods and Sheet1.xslx. Once you have the pieces that compose the files, load them into you XML server.

For this example, I'm using MarkLogic Server of course. I've loaded the pieces into a directory structure, where the the filename is the name of the directory; Each XML file that comprises the package for the document can be found in the directory named for the file it originated from. An example file in my server has the name "/Sheet1.ods/content.xml".

Ok, it looks like there's a lot here, but there isn't really. And it's hard to publish stuff like this on the web, so, I suggest either A) downloading vim and cutting and pasting the XML and XQuery into a .xml or .xqy file. ( In vim, you'll get syntax highlighting.) or B) Crank up the size of the text in the browser. ( In firefox that's: View -> Text Size -> Increase++ )

Create A Calc Spreadsheet

A simple Calc spreadsheet has two pieces, the manifest and a content.xml file.
Save the following as META_INF/manifest.xml:

<manifest:manifest xmlns:manifest="urn:oasis:names:tc:opendocument:xmlns:manifest:1.0">
<manifest:file-entry manifest:media-type="application/vnd.oasis.opendocument.spreadsheet" manifest:full-path="/"/>
<manifest:file-entry manifest:media-type="text/xml" manifest:full-path="content.xml"/>
</manifest:manifest>

Save the following as content.xml:

<office:document-content office:version="1.1" xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0" xmlns:table="urn:oasis:names:tc:opendocument:xmlns:table:1.0">
<office:body>
<office:spreadsheet>
<table:table table:name="Sheet1" table:print="false">
<office:forms form:automatic-focus="false" form:apply-design-mode="false" xmlns:form="form"/>
<table:table-column table:default-cell-style-name="Default"/>
<table:table-column table:default-cell-style-name="Default"/>
<table:table-column table:default-cell-style-name="Default"/>
<table:table-row>
<table:table-cell table:number-columns-repeated="3"/>
</table:table-row>
<table:table-row>
<table:table-cell office:value-type="string">
<text:p>Player Piano</text:p>
</table:table-cell>
<table:table-cell office:value-type="string">
<text:p>Kurt Vonnegut</text:p>
</table:table-cell>
<table:table-cell office:value-type="string">
<text:p>ISBN</text:p>
</table:table-cell>
</table:table-row>
<table:table-row>
<table:table-cell office:value-type="string">
<text:p>Ubik</text:p>
</table:table-cell>
<table:table-cell office:value-type="string">
<text:p>Philip K. Dick</text:p>
</table:table-cell>
<table:table-cell office:value-type="string">
<text:p>ISBN</text:p>
</table:table-cell></table:table-row>
<table:table-row>
<table:table-cell office:value-type="string">
<text:p>Stories of Your Life and Others</text:p>
</table:table-cell>
<table:table-cell office:value-type="string">
<text:p>Ted Chiang</text:p>
</table:table-cell><table:table-cell office:value-type="string">
<text:p>ISBN</text:p>
</table:table-cell>
</table:table-row>
<table:table-row>
<table:table-cell office:value-type="string">
<text:p>World War Z</text:p>
</table:table-cell>
<table:table-cell office:value-type="string">
<text:p>Max Brooks</text:p>
</table:table-cell>
<table:table-cell office:value-type="string">
<text:p>ISBN</text:p>
</table:table-cell>
</table:table-row>
</table:table>
<table:table table:name="Sheet2" table:print="false">
<table:table-column table:default-cell-style-name="Default"/>
<table:table-row>
<table:table-cell/>
</table:table-row>
</table:table>
<table:table table:name="Sheet3" table:print="false">
<table:table-column table:default-cell-style-name="Default"/>
<table:table-row>
<table:table-cell/>
</table:table-row>
</table:table>
</office:spreadsheet>
</office:body>
</office:document-content>

Zip 'em up, and give the file a .ods extension.
If you have OpenOffice installed you'll be able to open and you'll see:

I use very simple examples for the documents, so you can get an understanding of some of the XML that's in there and we can equally focus on the XQuery used to manipulate the docs. However, if you were to open up Calc or Excel and create these spreadsheets using the applications, when you save and unzip the packages for examination, you'll notice the XML generated for the files will look different. If you do that make sure you update the XQuery in the example below accordingly.

Following simple examples like these, as well as creating documents using their respective applications and examining the deltas in the XML published is a great way to learn more about the ODF and Open XML formats, what XML is required for consumption by each application, and what XML is generated upon save.

Your Office 2007 document should look like:

If it looks strange for some reason, it's because I have the ribbon minimized.

Transformation with XQuery

Finally, here's some XQuery. You can test tranforming for both formats by just switching the $test variable accordingly. If you have the files loaded into your Server, named as I've described above, you should be able to just place the following in a .xqy file. Name the file test.xqy and place it under /Docs in your MarkLogic server. You can then evaluate the file by opening your browser and navigating to http://localhost:8000/test.xqy . I have OpenOffice and Office 2007 installed, so I just open into the appropriate application directly. You can easily change this to just insert the XML as another doc so you can validate for yourself in the Server.

declare namespace ms = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"
declare namespace table = "urn:oasis:names:tc:opendocument:xmlns:table:1.0"
declare namespace office = "urn:oasis:names:tc:opendocument:xmlns:office:1.0"
declare namespace text=  "urn:oasis:names:tc:opendocument:xmlns:text:1.0"
declare namespace manifest = "urn:oasis:names:tc:opendocument:xmlns:manifest:1.0"
declare namespace form="urn:oasis:names:tc:opendocument:xmlns:form:1.0"

define function generate-simple-ods-odf(
$docmanifest as node(),
$content as node()
) as binary()
{
let $manifest :=
  <parts xmlns="xdmp:zip">
  <part>META-INF/manifest.xml</part>
  <part>content.xml</part>
  </parts>
    let $parts := ($docmanifest, $content)
    return
      xdmp:zip-create($manifest, $parts)
}

define function generate-simple-xl-ooxml(
$content-types as node(),
$workbook as node(),
$rels as node(),
$workbookrels as node(),
$sheet1 as node()
) as binary()
{
let $manifest := <parts xmlns="xdmp:zip">
                 <part>[Content_Types].xml</part>
                 <part>workbook.xml</part>
                 <part>_rels/.rels</part>
                 <part>_rels/workbook.xml.rels</part>
                 <part>sheet1.xml</part>
               </parts>
let $parts := ($content-types, $workbook, $rels, $workbookrels, $sheet1)
  return
     xdmp:zip-create($manifest, $parts)
}

define function getOpenXMLRowsFromODF($odfrows as element(table:table-row)*) as element(ms:row)*
{
for $i in $odfrows
let $row := $i/table:table-cell/text:p
return
<ms:row>
{
for $text in $row/text() return
<ms:c t="inlineStr">
 <ms:is>
   <ms:t>{$text}</ms:t>
 </ms:is>
</ms:c>
}
</ms:row>
}

define function getODFRowsFromOpenXML($ofcrows as element(ms:row)*) as element(table:table-row)*
{
for $i in $ofcrows
let $row := $i//ms:t
return
<table:table-row>
{
for $text in $row/text() return
<table:table-cell office:value-type="string">
 <text:p>{$text}</text:p>
</table:table-cell>
}
</table:table-row>
}

(: ---------------- BEGIN OFFICE OPEN XML EXCEL FILES --------------------:)

let $content-types :=
<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">
<Default Extension="rels"
ContentType="application/vnd.openxmlformats-package.relationships+xml"/>
<Default Extension="xml" ContentType="application/xml"/>
<Override PartName="/workbook.xml"
ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/>
<Override PartName="/sheet1.xml"
ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>
</Types>

let $workbook :=
<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"
     xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">
<sheets>
<sheet name="Sheet1" sheetId="1" r:id="rId1" />
</sheets>
</workbook>

let $rels :=
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship Id="rId1"
Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument"
Target="workbook.xml"/>
</Relationships>

let $workbookrels :=
<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">
<Relationship Id="rId1"
Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet"
Target="sheet1.xml"/>
</Relationships>

let $odfrows:= doc("/Sheet1.ods/content.xml")//table:table[1]/table:table-row

let $sheet1 :=
<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">
<sheetData>
{getOpenXMLRowsFromODF($odfrows)}
</sheetData>
</worksheet>

(: ------------------- END OFFICE OPEN XML EXCEL FILES --------------------:)

(: -------------------------- BEGIN ODF CALC FILES ------------------------:)

let $openxml_rows := doc("/Sheet1.xlsx/sheet1.xml")//ms:row

let $docmanifest:=
<manifest:manifest xmlns:manifest="urn:oasis:names:tc:opendocument:xmlns:manifest:1.0">
<manifest:file-entry manifest:media-type="application/vnd.oasis.opendocument.spreadsheet" manifest:full-path="/"/>
<manifest:file-entry manifest:media-type="text/xml" manifest:full-path="content.xml"/>
</manifest:manifest>

let $content :=
<office:document-content xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0" xmlns:table="urn:oasis:names:tc:opendocument:xmlns:table:1.0" office:version="1.1">
<office:body>
<office:spreadsheet>
<table:table table:name="Sheet1"  table:print="false">
<office:forms form:automatic-focus="false" form:apply-design-mode="false"/>
<table:table-column table:default-cell-style-name="Default"/>
<table:table-column table:default-cell-style-name="Default"/>
<table:table-column table:default-cell-style-name="Default"/>
   {getODFRowsFromOpenXML($openxml_rows)}
</table:table>
<table:table table:name="Sheet2" table:print="false">
<table:table-column table:default-cell-style-name="Default"/>
<table:table-row>
<table:table-cell/>
</table:table-row>
</table:table>
<table:table table:name="Sheet3" table:print="false">
<table:table-column table:default-cell-style-name="Default"/>
<table:table-row >
<table:table-cell/>
</table:table-row>
</table:table>
</office:spreadsheet>
</office:body>
</office:document-content>

(: ---------------------------- END ODF CALC FILES ------------------------:)

(: let $test := "officeopenxml" :)
let $test := "odf"

let $package := if($test eq "officeopenxml") then
       generate-simple-xl-ooxml($content-types, $workbook, $rels, $workbookrels, $sheet1)
    else
       generate-simple-ods-odf($docmanifest, $content)
let $filename := if($test eq "officeopenxml") then
       "ExcelTest.xlsx"
     else
       "CalcTest.ods"

let $disposition := concat("attachment; filename=""",$filename,"""")
let $x := xdmp:add-response-header("Content-Disposition", $disposition)

let $x := if($test eq "officeopenxml") then
    xdmp:set-response-content-type("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
else
    xdmp:set-response-content-type("application/vnd.oasis.opendocument.spreadsheet")
return
$package

In the example, we just created new documents, tranforming the content from one spreadsheet into the format of the other. I'm not going to dig too deep into this; You should be able to read the XQuery and see what's going on for yourself. But notice how we just have a variable set to the rows from our original spreadsheet, and when we go to create the new spreadsheet, we just call a function that transforms the rows from one format to the other. It's that simple.

If it helps, I've done the "Hello-World" document example with both OpenOffice and Office 2007, (a couple of times). Those previous posts should help answer some questions if this is all brand new to you. After evaluating the XQuery, you'll see your spreadsheets as follows:

Neither of those files existed in those formats when we began, and when the next format comes along, we'll just transform our saved XML content into that format too. As long as it's XML, the content can be anything we want it to be. The XQuery looks lengthy because of all the nodes we're constructing, but once we're more familiar with the formats, we can re-use pieces we've already stored. I just want to highlight that the amount of XQuery required for a powerful transformation of our content is really minimal.

08's going to be a great year for XML and XQuery and I'm excited to be working with both. My New Year's resolution is to post here regularly. That doesn't necessarily mean often, but regularly. :)

To learn more about ODF:
Open Document Essentials
The ODF Specification

To learn more about Open XML:
Open XML Explained
The Open XML Specifcations

To learn more about XQuery:
x-query.com (get on the mailing list)
w3c XQuery .10
XQuery tutorial

Office 2007 SP1 and Sun ODF Converter

noreply@blogger.com (Pete Aven) — Fri, 14 Dec 2007 21:37:00 +0000

Just an update to a recent post (Office 2007 vs. OpenOffice vs. Novell OpenOffice). I just installed Office 2007 SP1. Previously, Sun's converter didn't support Office 2007, it turns out this was due to the fact you could save as ODF, but not open an ODF document in Word 2007. Well now, you can open an OpenOffice document in Word 2007. I just tried this myself using my same test document from before. I opened the document in Word, saved as an OpenOffice document, closed Word and opened the document in OpenOffice to check the formatting, then closed OpenOffice and reopened the document into Word. The formatting was retained throughout. Very cool.

Sun's official stance on the download page is still "support for Microsoft Office 2007 is planned for one of the next releases." But it looks like you can start using it now with more than just a "hello world" test document.

When I save as .odt from Word, I sometimes have an issue where Word will crash, it's intermittent and I'm always able to save on the second attempt. I just started using this converter, so if I come across any recurring or other issues, or find out about any more updates, I'll post them.

ODF - Hello World with XQuery

noreply@blogger.com (Pete Aven) — Mon, 10 Dec 2007 23:51:00 +0000

This is the "How to create the simple hello-world document for OpenOffice using XQuery" example. I'm using MarkLogic Server for this of course. If you're interested you can download a free copy using a Community License here. I'm also using OpenOffice 2.3.1, available here.

An OpenOffice document is just a zip file. It's actually a .jar file, but we don't care about that right now. We can unzip it and extract the separate XML parts, just how we would with any zip file. In fact, in Windows, you can just change the extension of your OpenOffice document from .odt to .zip, right-click, select "Extract All", then take a look at the files in the folder.

Similarly, we can create an OpenOffice file by creating the required parts and zipping them up into a package. When we say an ODF or OpenOffice document, we're usually referring to the collection of XML documents that make up the .odt you use with OpenOffice. ODF stands for Open Document Format, and is the name given to the XML that OpenOffice is using to create your documents.

This should all sound very familiar. It's very similar to what I've posted on Office Open XML, and a .docx file in Word; We often say an Office 2007 / Word document, but mean the collection of XML files that make up the .docx.

The minimal .odt document has just 2 parts: content.xml and manifest.xml.
We place the main text and body of our document in content.xml, and place the assorted files that compose the document in the manifest.xml. I'll explore the other files in future posts, they of course have to do with styling your document, meta-information about your document (created by, created date, etc.), images, etc. Since we aren't using any other files in this example, this document will have zero formatting and no meta-information associated with it.

Ok, place the following in a file named openODF.xqy under /Docs of your MarkLogic install. You can then evaluate by opening your browser and navigating to http://localhost:8000/openODF.xqy. Your test document will open directly into OpenOffice Writer. You can then mess with the XQuery and XML to create other types of documents. Good times!

Note: To keep the code readable I had to split a couple of nodes across lines. I was able to cut-and-paste this into a .xqy and evaluate with no problems, but I mention in case you run into any issues.

define function generate-odt(
  $docmanifest as node(),
  $content as node()
) as binary()
{
let $manifest :=
 <parts xmlns="xdmp:zip"> 
     <part>META-INF/manifest.xml</part>
     <part>content.xml</part> 
  </parts>
  let $parts := ($docmanifest, $content)
  return 
    xdmp:zip-create($manifest, $parts)
}

let $docmanifest :=
<manifest:manifest
xmlns:manifest="urn:oasis:names:tc:opendocument:xmlns:manifest:1.0">
 <manifest:file-entry
manifest:media-type="application/vnd.oasis.opendocument.text"
manifest:full-path="/"/>
 <manifest:file-entry manifest:media-type="text/xml" manifest:full-path="content.xml"/>
</manifest:manifest>

let $content :=
<office:document-content xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0"
xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0"
office:version="1.1">
   <office:body>
     <office:text>
        <text:p text:style-name="Standard">
          <text:s text:c="5"/>Hello World! This is my first paragraph.
        </text:p>
        <text:p text:style-name="Standard"/>
        <text:p text:style-name="Standard">
          <text:s text:c="5"/>This is another paragraph.</text:p>
     </office:text>
   </office:body>
</office:document-content>

let $package := generate-odt($docmanifest, $content)
let $filename :=  "hello-world.odt"
let $disposition := concat("attachment; filename=""",$filename,"""")
let $x := xdmp:add-response-header("Content-Disposition", $disposition)
let $x := xdmp:set-response-content-type("application/vnd.oasis.opendocument.text")
 return
    $package

For those interested in the ODF format, there's a free book, OpenDocument Essentials, as well as the ODF specification.

The content.xml in a nutshell: <office-document> is our root element.
It's first children are optional and can be <office:scripts>, <office:font-face-decls>, and <office:styles>.

We don't see those here, we'll examine those more in the future. The only required element is <office:body> and this is where the magic happens. It's first child element tells us what type of document we're actually dealing with; we have the choice of:

<office:text>
<office:drawing>
<office:presentation>
<office:spreadsheet>
<office:chart>
<office:image>

We're dealing with text. From there we see it's child element <text:p> , which signifies a paragraph. Now, the only thing funky above is the use of <text:s>, which signifies whitespace. There's a couple of pages on how to handle whitespace in the Essentials book. When you opened the document, you might not have noticed, but each sentence was indented 5 spaces. You can safely remove the <text:s> node for the example above.

Ok, so it's a little more than just a HelloWorld example, but we're not really interested in a one paragraph, one word document. For more fun, we can just start extracting OpenOffice documents and insert the pieces into our XML Server. It's all just XML at the end of the day, and I actually think it's fun to dissect these formats and then transform them into whatever I want. So with ODF and Office Open XML documents in my server, I can write queries to find what I'm looking for and then just deliver the content in any requested format. Sweet!

Office 2007 vs. OpenOffice vs. Novell OpenOffice

noreply@blogger.com (Pete Aven) — Sun, 09 Dec 2007 22:18:00 +0000

Ok, first things first. I work for the Mighty MarkLogic in San Carlos. I'm currently doing a series of blog posts on OOXML and MarkLogic Server for our developer network. Today's post came about as my sole focus for the last few months has been OOXML, but I've been a big fan of OpenOffice for quite some time. I thought I'd repeat some of the examples I'm doing for OOXML with ODF. I have no allegiance to any Office Productivity application, my allegiance, if to anything, is to XML and XQuery.

Now that you know this: The opinions and posts on this blog are mine and mine alone and don't necessarily reflect the opinions of MarkLogic Corp. If they like it, I'll let 'em claim it, but if not, they were out of town when I wrote this, capiche?

The following includes my experience testing OpenOffice 2.3.1 , Novell's OpenOffice 2.3, Novell's OpenOffice.OpenXML translator 1.0.0-2, and the Sun ODF Converter 1.1 for Microsoft Office to see how easily they interoperate with a .docx file, and a .odt file.

Office 2007 Professional out-of-the-box will not open a .odt document, nor save as .odt.
But we can save as .odt if we install and use Sun's ODF Converter.

OpenOffice 2.3.1 out-of-the-box will not open a .docx file, nor save as a .docx.

Novell's 2.3 OpenOffice out-of-the-box will not open a .docx file, nor save as a .docx.
But we can save as .docx and open from a .docx if we install their translator.

We can get Office 2007 to save as a .odt document if we install the Sun ODF Converter. The download and install is quick and simple. A single file, download, double-click, it installs, open Word, and you can save as .odt. Simple! Sun states on their site for the converter that Office 2007 support is coming and they don't explicitly list Office 2007 as supported, but I believe they are well on their way. I was able to take a sample Word document with multiple pages, formatting, color, and a couple of embedded tables - it wasn't just a "hello-world" document - and save as .odt. When I opened in OpenOffice, or Novell's Open Office, it looked perfect. If there was a difference, I didn't see it. It took a few seconds to save, but I'm impressed. The downside is it's just a one-way converter, I still can't open a .odt in Office 2007.

Novell has an OpenXML translator available, that will allow their version of OpenOffice to open and save .docx files. To install their version of OpenOffice, it's an ISO, so you'll need daemon tools or to burn it to disk. Once installed, you can add the converter by opening OpenOffice and going to tools -> Extensions. Click 'add' in the dialog box and add the addin by browsing to its location. Click ok. Restart OO and you'll now have the option to open and save as .docx. Ok, not too simple for an ordinary user, but it works. I think you'd need to be kind of a geek to even know that Novell has their own version of OpenOffice or the translator or how to install, but its cool that its there.

I have that same sample document I described above that originated as a .docx. I opened in Novell's Open Office, and the formatting was off. It lost the indentations for paragraphs, and it shifted a table halfway down the second page.

I then saved the .docx as a .odt in Word 2007. I opened in Novell's OpenOffice, and it looked perfect. I then saved as .docx from within Novell's OpenOffice. I closed NOO. When I re-opened, the formatting was off again, similar to the original document. When I opened in Word 2007, it had lost the indentation and the bottom 1/2 of the still shifted table was missing.

For simple "Hello World" documents, the formatting was fine when I saved as .docx and re-opened in Office 2007 or Novell's OpenOffice, but for other documents, it appears Novell's OpenOffice OOXML support still needs some work.

I just started reading OASIS OpenDocument Essentials. I hope to post some examples soon.

New Media Producers

noreply@blogger.com (Pete Aven) — Thu, 15 Nov 2007 05:07:00 +0000

If you check the mainstream news, chances are you've heard of the writer's strike. I don't know much about it, but my guess is they're "takin it in the shorts", as an ex-boss of mine used to say. But to tell you the truth, I don't watch much tv. ( Though I have discovered Scrubs and watch it late nights before bed, it's like I Love Lucy 2007 ). I peep ATHF, Daily Show, Colbert on a regular. And I've been known to plan my week around Lost; but I'm all about tv on dvd when I like the series. I'd rather blast through the season in 3 days than F around with commercials. (yeah, I haven't got a dvr yet, but again, I'm not around the tv that much to care) Anyway, the must-see tv these days ain't on tv, it's on the web.

There are so many cool shows out there right now. I'm hearing the term New Media Producers more lately, but I'm not sure if that's an apt title. They don't have anything in common with Old Media, and I don't know if they even consider themselves producers. These people are just doin what they do for the love of doin it! It's an exciting time for publishing this type of creativity. There are more than a few of these people out there, but below are some of the shows I enjoy regularly. I have to admit, I got the idea to share these from watchin Epic-Fu, it's my favorite, and it rocks.

My absolute favorite: Epic-Fu! (formely JetsetShow). If you haven't peeped it, check it out. You'll learn more there each week than you will in the news or on the Daily Show. Steve, Zadi, and Rick bring the epicness each week.

Another must-see Chad Vader. This series is hilarious. Can't wait for Season 2. Start with episode1 and work your way up.

If you're into sci-fi at all, you have to check out Galacticast. Rudy and Casey rock!

Ask-A-Ninja always delivers the goods. You got questions, he's got answers.

Another two fun ones are HappySlip (read the "about" for the name) and LisaNova (this chick is nuts, but frick'n hilarious).

And one I realized I haven't watched in awhile, so I have some catching up to do : Tiki Bar TV

Anyway. I spend a lot of time on the computer reading, coding, etc. When I take a break I peep these shows and they're perfect. 3-6 minute blasts of originality with no commercials. Just a bunch of creativity and heart served up freely courtesy of some talented people and the internets. Then its back to the lab.

Goodreads is Sweet

noreply@blogger.com (Pete Aven) — Wed, 14 Nov 2007 05:21:00 +0000

I really like goodreads.com. I just want to put that out there for everyone who's not familiar with it. It's a social network that's actually useful. If you enjoy reading, it gives you a place to share all you've read with others; so you can see what your friends are reading, and what they recommend or didn't like.

Anyway, it turns out that these days as soon as one person has an idea, the VCs rush to fund similar companies. It's like coffee shops, as soon as one opens on the block, you know another will follow shortly as someone wants their piece of the pie. So Shelfari and LibraryThing are two other companies that seem to be doing the same thing and are getting a lot of press lately for themselves by spamming others and complaining about it. I don't know who came first, and Shelfari and LibraryThing might actually be great sites, but I found goodreads first and I dig it so I'm rollin with it.

I'm endorsing them as I find the site very useful. Also, it's refreshing to find a site that seems to put the user first, it's not all about how they can make the money through ads or whatever. In print and on the internets all I'm seeing lately is the other 2 companies and people need to wake up and not sleep on goodreads. It's cool. If you make it over there I'm PT3.

Oliver Selfridge conclusion

noreply@blogger.com (Pete Aven) — Wed, 14 Nov 2007 05:19:00 +0000

Ok, so I had this plan to rant about how screwed up it is that great computer scientists continue to crank out the badassness and deliver the goods while a bunch of fronters get all the media attention and ridiculous valuations for their <airfingerquotes>companies</airfingerquotes> based on what <airfingerquotes>they might be someday</airfingerquotes> or <airfingerquotes>they could be someday</airfingerquotes>. But, it's obvious, and you'll probably find this theme in future rants of mine just manifesting itself in different ways anyway, so I'm calling it off. In conclusion, Oliver's the man. I'm still uplifted just thinking about his presentation.

TrueKnowledge, NLP, and HCI

noreply@blogger.com (Pete Aven) — Sat, 10 Nov 2007 21:50:00 +0000

It's exciting to see NLP present in so many applications these days. Some of the companies I follow and have a keen interest in seeing develop are: Attensity, Powerset, Hakia, and RadarNetworks. But out of the blue last week ( and I'm still not sure how I missed this as I'm usually on top of companies like this ) comes TrueKnowledge. If you haven't peeped their demo yet, you should do so. The site looks great but it's in "private beta" right now. I'm looking forward to testing their tech out once it's finally released.

TK bills themselves as a site for "Question-Answering" and "Enhanced Search". Based on the demo, you can type in a question in English such as, "Who invented Perceptrons?", and it will provide an Answer. It's not just using keywords, it does some parsing of the sentence you enter and then applies some inference rules. In the demo, they ask "What time is it at Google?", and the correct time is returned. Very cool stuff.

So I like asking a question and getting a straight answer, but I have to critique how the answer is returned. This doesn't apply to just TK either. I see other companies doing this, and it's because Engineers come up with this stuff. When we have the Engineer hat on, we do brilliant things, but forget about the little details like user experience. So...

In the demo, when you ask a question of their site, you get a straight answer.

Example: You ask "is J Lo single?", the site returns "No" and a link it thinks will back up it's answer. That parts awesome.

But then there's all this other verbose stuff with the answer. For the same question, the site also returns: "There are 2 defintions of "single" , then provides the 2 definitions,
then goes on to say "I took the meaning of your question to be.." and inserts something from the definitions previously listed. There can then be several links that represent the "inference path" for how the machine came up with the answer it's returning. In their Google-time example, a ton of links are returned.

Something about this nags at me that the HCI or UI design is off somewhere. It looks like they just jammed a bunch of System.outs in their code and blurt out the logging in the answer. They try to gussy it up and anthropomorphize it to make the logging seem helpful, but I'm not sure it really is. Maybe they should have a verbose mode, that will give you all that extra crap if you want it, and a trusted mode, where you just trust TK is giving you the right answer. Or maybe in trusted mode you get a straight answer and a confidence percentage. If the user questions the answer returned or the confidence rating, they can always go into verbose mode to validate the response, but I don't think logging should be on by default. Anyway, for me, I'll dig all that extra info, but I wonder if the average user will, or how all those inference links will look jammed on a cel phone.

Anyway. Back in 2001, I'd type a query in Google and it was like it could read my mind, I loved it. Their search just blew me away. Nowadays, there are a bunch of search engines out there, and I try to check them out occasionally, but I find myself going back to the Goog regularly cause "The devil I know....". But I'm almost always assured that the first few links will be trying to sell me items from ebay or amazon.

I'm all for a site that I can query that will return answers and not links. More power to 'em!

Nopen Social

noreply@blogger.com (Pete Aven) — Fri, 09 Nov 2007 15:13:00 +0000

So Brandy and I were talking last night about Nopen Social and our general agreement with O'Reilly's latest, when she says: "Of course Open Social isn't open, good engineers generally aren't Socially Open, so it makes sense, they couldn't make it any other way."

Brilliant.

Oliver Selfridge pt. 2

noreply@blogger.com (Pete Aven) — Thu, 08 Nov 2007 02:15:00 +0000

One of the greatest things to be a part of at Oliver's presentation, was listening to him reminisce about A.I. history with members of the audience. This wasn't intentional, it just sort of happened naturally as he presented his slides. He'd share a thought, someone in attendance would ask a question, then in response, he and his peers, also in attendance, would bring up past work that validated/invalidated their answers. In doing so, they couldn't help but go off on little sidetrips down memory lane.

Upon reflecting about their time at Bell Labs and DARPA, you could hear the joy and gratitude in his exchanges with McCarthy. They'd describe the work, and mention others working there. They looked back together fondly on a time working with Minsky. Oliver went on to express his sadness at losing Rosenblatt. He told the story of perceptrons and thought it unfortunate Frank didn't get the chance to counter Minsky's arguments against his findings. He also mentioned a brief time of greatness at DEC, and you could sense the early joy and then feel the anger and sadness as his tone changed; He described watching their great products and research efforts take a dive due to poor leadership in the company and he had some choice words for Ken Olsen. You could also hear Oliver's sad frustration at losing Licklider. He thought Licklider could have lived many more years if he would've just taken better care of himself at the end, and he wished they could of had those years. He had a great amount of respect for that man.

They were just brief mentions, but you felt like you were right there with him. I've always been a fan of computer science, and for me, being in that room gave me the chance to meet so many of my heroes. People who I've read about, and who's formulas and algorithms I've studied, all suddenly there live and in 3-d confirming my studies and enriching what I know with those little details that let you know it's real. It was awesome to get to go back to those places with them, just for those short moments.

Well, on that note. I thought of immediately hitting them all up and putting together a book on the history of A.I. Listening to him, I realized we've lost many greats already, and that losing more is a given. I wanted to capture those voices before they were lost. Upon looking them all up after the lecture, I found Nils Nilsson is actually doing just that.

From his homepage:
The Quest for Artificial Intelligence. I am writing a book on the history of AI. I hope to finish by the end of 2008

I'll be the first in line to buy it.

Oliver Selfridge is a P.IM.P. pt. 1

noreply@blogger.com (Pete Aven) — Wed, 07 Nov 2007 02:46:00 +0000

Pretty damn IMportant Person in the world of A.I. !

I've been meaning to share this for a few days, but it'll probably take a few posts. So I'll just start with this: On Thursday, October 25th, I had the privilege of seeing Oliver Selfridge give a presentation to the Symbolic Systems posse at Stanford and it was incredible. I didn't know what to expect, but I walked in and there was John McCarthy, Nils Nillson, and Ed Feigenbaum sitting in the audience. It was a small affair, probably around 40 people, and half of them were O.G. veterans of A.I. There were probably other greats there too that I just didn't know or catch. Half the room was gray hairs and the other half were undergrads. It was kinda like something out of Dark Crystal, all the Mystics gathered and dropped some serious science while us Skeksis waited to merge, but anyway, onto the good stuff....

His lecture was titled "Let's Improve Machine Learning" and I took notes voraciously, so I'll just share some highlights. Also, the experience has given me 3 threads of thought I find myself meditating on, that's why I'll be breakin the posts up:

Part 1) His presentation itself and what it means
Part 2) Hearing the history of A.I. shared first person by himself and the attendees
Part 3) How come so many people ride FB's jock for the valuation, or attend the web2.0 summit but sleep on the greats who actually deliver? In fact, we still use and teach what these people delivered 40+ years later while I'm pretty confident we'll forget some of these dot-com frontrepeneurs after a short time.

Part 1: The Presentation

"I want software to give a shit."

That direct quote sums it up in a nutshell. He himself quoted Norbert Weiner who proposed "a machine can learn" , and Ed Berkeley who said, "a machine, therefore, can think." From this he went on to question how much does software think today? He also asked, well, if we want software to think, how might we do that?

To answer those questions, he focused on giving software a Purpose, and what that entails. His notion of Purpose: "Having Purpose means having a goal, and trying to achieve it."

To have software give a shit, he Proposed "Purpose Loops". Adaptive, learning loops, with an emphasis that software should remember what it has learned. He pointed out that today, most apps reach a goal, but forget how they got there and don't build off of it.

He also stated that Math, while great, should not be an impediment to getting software to care. His background is Math, and he didn't mean to offend any Mathematicians, he just thinks that sometimes scientists get lost in the equations and at that point stop getting things done in regards to advancing learning applications forward.

From there he stated some of the insights and challenges with Purposes, and he often used people and children as a metaphor for what the software could/might do:

Purpose Insights:

1)You can have many at once.
2)There are different kinds.
3)You remember them, and learning is not isolated.
(you build off experiences; sequences of met goals.)
4)You collaborate: sharing what you've learned;
we learn from our experiences and other's;
we don't reach goals or learn in a big-bang fashion, it builds.
5)If you don't reach the goal the first time, you try something else.
6)Learning is self-improvement; what the learner perceives to be important.

Purpose Challenges:

1)You can have a hierarchy of Purpose:

"Win the war!" - The Commander
"Take the city!" - Major
"Take the hill!" - Corporal
"Take out enemy tanks!" - Sgt.
"Stay Alive!" - Private

2)You can have many Purposes at once.
3) The world changes: values, cost, environments, you change...

From this he pointed out, that many times we work to deliver the very best software, but given this, he quotes Minsky: "The best is the enemy of the good". So again, the message was just get out there and make something happen!

4)Sharing; you can't always share and when you can there's no standard way. Language helps, but in some situations, you need examples, instructions, teaching. (He used Tennis as an example.) He also pointed out that many times Sharers have something in common.

Now, here's where it got kind of Matrixy; he proposed apps that help other apps in the way an adult helps a child. When people asked technical questions during the course of this event, he'd often respond: "How would you (insert noun/verb from question here) with a child?" But he made complete sense. It was Brilliant!

Some Challenges with the Challenges:

He provided some more examples where Learning and Sharing are just tough:

1) Learning to raise an eyebrow; or raise each individually.
2) Sometimes you can't teach how to do something, but how to learn how to do something.
3) Counting; how did man learn to count and why, it must be important.
4) How to get to the grocery? Requires maps/models; we often make the map in our head and remember landmarks.
5) Spoonerisms: I'm riding a well-boiled icicle
6) SkyDiving: Trying and failing may not work.
7) Learning a new skill
8) Solving quadratic equations (there are many ways: factoring, graphing, formulas, etc.)

What To Do:

He started to wrap up with proposals for what we can work on to help give software a purpose.
1) A full descriptive taxonomy of Purposes.
2) A language for expressing Purposes.
3) Flexible adaptive modules for our Purpose Loops and Learning.
4) A language for remember what we've learned
5) Hierarchies of Purposes
6) Kinds of Change

A particularly useful problem: Recognition of English Text, there's no ideal way.
(Note: NLP is something I'm keenly interested in and have been focused on for quite some time, so I was right there with him loving it by the time he got to this point.)

In Conclusion:

He concluded with this parting thought:
You can't program a kid, you can only open a door for a kid and show them what's out there, then they care. Do the same for software.

Final Thoughts:

Ok, some of those points above may be concise, but I hope you got something out of it. It was a great presentation, and some of McCarthy's quotes during the presentation were priceless:

When Oliver spoke on learning, McCarthy stated with authority in response:
"You can only learn what your system will permit."

When Oliver spoke on the advancement of machine learning and A.I., McCarthy pointed out:
"Complete success has not been achieved."
which caused many to laugh.

And finally, when people were questioning Oliver's points on scientists getting lost in Math, and not advancing machine learning as quickly as may be possible,
McCarthy blurted out: "It's not the fault of Mathematics!"

So marinate on that for awhile, I'll followup with some more A.I. goodness later.

Urban Challenge Update Final

noreply@blogger.com (Pete Aven) — Sun, 04 Nov 2007 20:23:00 +0000

The Final Winners have been announced.! Details will probably eke out over time, but it looks like a questionable lane change by Stanford, and a traffic infraction by VT, cost them the first place crown. Congratulations to all the Teams! I can't wait to see how they step it up for the next challenge.

1) Tartan Racing (Carnegie Mellon)
2) Stanford Racing
3) Victor Tango (Virginia Tech)

Urban Challenge Update II

noreply@blogger.com (Pete Aven) — Sat, 03 Nov 2007 22:48:00 +0000

Phenomenal!

Spoiler Alert! (just kidding, but you know how people reveal the details of Heroes or Lost, well I'm about to do it with the DARPA Urban Challenge.)

Ok, I haven't gotten anything done all day because I've been watching the race online. 6 of 11 teams completed the course. Stanford crossed the finish first, but we won't know the actual winner until later. Along with the course completion times, the vehicles were awarded and deducted points based on their performance throughout several key tests. DARPA's crunchin the numbers and will probably announce the winner tomorrow morning.

Oh, how far it's come! I remember watching the "webcast" in 2005. Then it consisted of a map with different colored moving dots representing the bots. I was so excited when I saw the red dot of Stanford pass the CMU dot. The reporting was so sparse, I had to wait for official DARPA updates to confirm that they'd actually completed the course before CMU.

Today's webcast was brilliant. They had cameras everywhere and you could watch all the vehicles in action. Like a tv sports event, they even had the banner at the bottom of the screen. On the right of the banner, instead of espn, it had the DARPA emblem. And on the left, it would have numbers for the mission completed while the banner displayed the team name. They even had announcers calling the action. It was spectacular! I hope they release it on DVD, I'd buy it. It was amazing to see autonomous vehicles come to a Stop sign together, pause, then take action to merge into traffic, or continue on their way without hitting the other vehicles. There were some mishaps of course, but all in all, it was amazing! Congratulations to all the teams! I can't wait to find out who won.

Urban Challenge Update

noreply@blogger.com (Pete Aven) — Fri, 02 Nov 2007 13:39:00 +0000

11 Teams Qualified. Tomorrow at 8:30 PT the real challenge begins. Wired has some great posts that give the whole event a more personal touch. Stories there include: teams working together so each bot can qualify, how a track had to be resized for team Oshkosh, and how team Virginia's entry is number 32 in honor of those they lost April 16th at their campus. Best of luck to all 11!

We Can Rebuild Him, We Have The Technology

noreply@blogger.com (Pete Aven) — Thu, 01 Nov 2007 03:48:00 +0000

DARPA @ gadgetoff.
I found myself concerned for the monkey, but they did it to a human too. The Bionic Man is coming....

I love it when you race 'em big DARPA!

noreply@blogger.com (Pete Aven) — Wed, 31 Oct 2007 02:40:00 +0000

Ok, for some reason, a lot of people are sleeping on the DARPA Grand Challenge this year. 35 teams are trying to qualify right now for 20 spots. The teams who qualify, to be announced this Thursday, Nov. 1st, will get to compete on Nov. 3rd in the Challenge. This year, it's an Urban Challenge. What does that mean? Well first, some history:

The first Grand Challenge was in 2004. The challenge was to have a vehicle autonomously navigate itself through a 142-mile desert course in Nevada. Team Red from Carnegie Mellon, headed by Red Whittaker, made it furthest at 7.4 miles. Since the first prize was $1 million, and no team completed the course, the prize money rolled over to the next Challenge in 2005.

In 2005, the course was 132 miles, Team Red had 2 entries and came in 2nd and 3rd behind Stanford Racing, who completed the course in 6 hours and 53 minutes. What's even more spectacular is Team Red was in the lead, and Stanford's vehicle had to pass them to come in first. Stanford's team was led by Sebastian Thrun.

This year, they're at an abandoned Military Base in Victorville CA. The Challenge includes:

Vehicles will attempt to make safe left turns across moving traffic, and pull out at T-intersections with cars arriving from both directions. In other tests, vehicles will follow narrow winding roads, avoiding parked cars and other obstructions. Vehicles must also maneuver into a designated parking spot, and negotiate 4-way intersections and road-blocks.

So this is out of hand. They have professional drivers out there simulating traffic and the vehicles have to autonomously merge and navigate the obstacles to reach the goal line. First prize is 2 million dollars and you can watch the race on a live webcast Nov. 3rd at www.grandchallenge.org . Right now there are all sorts of info on the teams, links to blogs, pictures, etc. on the main site. Epic!

It was just a dream, I used to read DARPA magazine,
Marvin Minsky , John McCarthy in the limousine!

Hello, World!

noreply@blogger.com (Pete Aven) — Tue, 30 Oct 2007 04:26:00 +0000

It's on like Donkey Kong! , as they say, do they say that anymore? Who Cares! I'm bringin it back!! I'm settin up and gettin ready to wreck shop like Sage Francis. For those that don't know, I'll be wreckin shop like a drunk mechanic! This is my forum for all I find interesting; a way to communicate with friends far and near and to share some of my experiences in life and with XQuery. Think of it as Mashups and Machine Learning with Tales from Tagland thrown in.

Why Kiwisi? Because I often find that when I search, I don't necessarily know what I'm looking for, but I know it when I see it. Sorta like the title of this blog, until I think of something better, I'm rollin with it. Also, according to a recent report, many are now experiencing search engine fatigue. Instead of digging around on your favorite search engine for hours trying to find what you're looking for, I'm hoping you might just find something interesting here.

First things First!! I'm rollin with the defaults on this thing. I know it's blah! I'll work on it and improve as we move forward, for now I've got some catching up to do with some of the stuff I want to share with y'all. To keep it interesting, I'm going to try and post frequently, maybe not daily, but I'm definitely making a commitment to this. Peace!