Sean Kearney's Blog

Harvesting data from a Coveo index

Sean Kearney — Tue, 07 Jul 2009 01:51:00 GMT

I’ve been working peripherally with Coveo for the past couple years now. It hasn’t been until recently that I’ve actually needed to start diving into the “API” to try and harvest the data. This, however, proved to be more difficult than one would expect from a Search product!

You see, Coveo wrote a lot of fancy controls (complete with AJAXy functions) that work intimately with their index. They have an “interface editor” that is supposed to help you customize the way these controls function and look. In a perfect world you will tailor your interface using this editor, drop a single user control on your page and you are up and running. However, there are times when these stock controls just don’t cut it; you will either want to change the look, or function, to drastically to be handled with the interface editor. I found myself in such a position.

Coveo lists a few APIs in its online help section (Search Interface API, Converter API, Connector API, and Security API), but none of them tell you how to get access to the raw data!

Before Coveo I was using Lucene for various projects and it had a decent API that you can use to get access to your search results. You were then responsible for rendering these results. This is what I would have expected in terms of an API from Coveo. The Coveo Search Interface API is nothing more than a collection of the aforementioned ASP.NET controls that you can try to use in your Interface. I wasn’t able to find any samples about how to get my data with some API calls, there was however, some MSDN style class documentation.

After some playing around I was able to perform a search using the following code. You need to specify the Collection ID you want to search on as well as the Query to perform. The results will then be populated into the resultArray object. There are many parameters you can set on the SearchBuilder object, which you can read about in the doc. This code will be used in an upcoming post about how to write an autocomplete text box with jQuery and Coveo.

using Coveo.CES.Web.Search;
using Coveo.CES.Web.Search.Controls;
using Coveo.CES.Web.Search.Providers;

// ....

ICESSearchProvider provider = SearchUtilities.CreateDefaultSearchProvider();
SearchBuilder builder = new SearchBuilder();
builder.AddCollection(1234); // add your collection id here
builder.Provider = provider;
builder.ExpandQuery = true;
builder.RetrieveFirstSentences = false;
builder.ExcerptSize = 0;
builder.FilterDuplicates = true;
builder.AddBasicEx pres sion("Search for Me");

if (builder.ExecuteQuery)
{
    QueryWrapper newQueryWrapper = QueryWrapperFactory.GetNewQueryWrapper(builder);

    int totalResults = newQueryWrapper.TotalCount;
    ICESResult[] resultArray = newQueryWrapper.GetResults(0, 100);
}

Coveo with Sitecore Media

Sean Kearney — Mon, 06 Jul 2009 18:27:00 GMT

Out of the box, Coveo does a pretty good job of crawling your website and providing search capabilities. In this particular installation we aren't using the "Sitecore connector" we are simply letting coveo do its thing walking the site. We encountered an issue where Sitecore media library items, mainly PDF files, were not making it into the index. We figured it must be a filter of some sort blocking access to it and this proved to be the case.

Sitecore media (by default) serves media library items with a generic handler extension of '.ashx' and Coveo is configured to ignore all unknown filetypes. What you need to do is go to the "Document Types" tab of your index and add a type for .ashx files. I simply called it "Media Library" and entered in the extension of '.ashx' and left all options as the default. I rebuilt the index and we were all set.

Sitecore FieldRenderer Control and OverrideFieldValue

Sean Kearney — Mon, 22 Dec 2008 19:28:00 GMT

The FieldRenderer Control (Sitecore.Web.UI.WebControls.FieldRenderer) has a method called OverrideFieldValue that takes a string and is supposed to be shown instead of the real fields value, but it doesn't work.

It seems a fix would be to add in a new pipline that accounts for this preset value.

Here's the code:

public class GetOverriddenFieldValue
    {
        // Methods
        public void Process(RenderFieldArgs args)
        {
            Assert.ArgumentNotNull(args, "args");

            if (!string.IsNullOrEmpty(args.FieldValue))
                args.Result.FirstPart = args.FieldValue;
            else
                args.Result.FirstPart = args.Item[args.FieldName];
        }
    }

You then need to drop this in the Pipline in the web.config file so that we can modify the value on the way out.

Look for the following node:

<processor type="Sitecore.Pipelines.RenderField.AddBeforeAndAfterValues, Sitecore.Kernel" />

Now add the following above that:

<processor type="Your.Namespace.GetOverriddenFieldValue, Your.Assembly" />

The bug has been logged with Sitecore so hopefully they release an official patch.

Editors, Sitecore, and SEO

Sean Kearney — Fri, 12 Dec 2008 18:32:00 GMT

Editors (and some sysadmins) that have been around IIS web sites for a few years seem to have a little trouble grasping what Sitecore can do for them. More specifically, they have trouble grasping what it is they don't need to do anymore!

In the purest sense, file based web sites running on IIS serve up the HTML from... you guessed it. Files. These files are typically (and hopefully) organized in folders and there is usually a "default document" whether it be default.htm, index.asp, home.aspx, etc... With Sitecore the concept of folders and default documents goes right out the window. Every item in the site tree can be a page including the "folders!" This can make for a very search engine friendly site.

With traditional web sites there can be many urls to get to the same content. For instance. http://example.com/Products/Default.aspx and http://example.com/Products are the same to us programmers, but to a search engine they are different! IIS is configured so that if a request comes in to http://example.com/Products/ it will look at the file system try to find a preconfigured default document. In this case, it locates the default.aspx file and serves up the content.

If we were implementing this system in Sitecore we would configure the system such that "Products" isn't a plain 'ole folder; we make"Products" a page, which can have child pages! This will result in urls like:
http://example.com/Products.aspx
http://example.com/Products/Widget.aspx

Taking this one step further... Sitecore 6 + IIS7 support turning off the .aspx extension in its urls! This would generate urls that look like:
http://example.com/Products
http://example.com/Products/Widget

In order to disable .aspx extensions for Sitecore 6 sites running in IIS7:
1. Search for "addAspxExtension" in the web.config and set it to false.
2. Search for "<System.WebServer>/<Modules>" node in web.config and add an attribute runAllManagedModulesForAllRequests="true"

Moved Domain Name

Sean Kearney — Sun, 30 Nov 2008 20:01:00 GMT

Almost two years after I thought about phasing out my online moniker I have done it. I have officially moved the carknee.com domain to seankearney.com. Redirects are in place and there shouldn't be any noticable change for the few people that read this site. The Feedburner feed is unchanged and will remain unchanged for the near future. Maybe at some point I will move to rss.seankearney.com, but not yet.

Photo Management and Backup

Sean Kearney — Mon, 24 Nov 2008 02:54:00 GMT

This is a follow up post to "Photo Management" on how I am starting to organize my digital images. After looking around at some of my options (Picasa, Adobe products, buying an iMac, …) I have decided to take the cheap way out. Here is my current workflow, which is a work in progress.

I pull the images from my camera manually and dump them to a temp directory and rename the files from the standard IMG_9753.jpg to the date they were taken on. I then create new directories on my NAS for the Year\Month\[Event|Description] as in “H:\Media\Pictures\2008\11\Party”. I move the files to their new homes on the NAS and fire up Picasa. Once in Picasa I go and “Tag” the images with the people in the picture and any other pertanent info. My final step is to backup my media to Amazon S3.

Maybe I am showing my age a bit, but I have a an issue venturing to far from the old 8.3 DOS file names so I chose the format of yyyy-mm-dd#hh-mm-ss.jpg. In order to do this little task I found a decent piece of software called called AmoK Exif Sorter. The program worked so well I decided to go back and rename ALL my images going back to 2004, which is when I got my first digital camera that stored date/time in the Exif data.

Overall I use Picasa at a very high level. I simply tag, view, and upload my photos using it and that is it! Picasa provides some import and file rename functions, but they weren’t working they I would have liked. I am starting to see that the more work you put in the bigger the benefit you will get out of it though. I spent an hour tagging all the images taken of my son since August and the search capabilities of Picasa work great!

I still have some pieces of the puzzle that don’t fit so well, but this will be a good starting point. I am currently using a Linksys NSLU2 with a USB drive for my NAS. Ideally this will be upgraded to something else, I just can’t decide what yet (ReadyNas, Windows home server, etc…). I failed to mention that I manually backup my single drive NAS to RAID 1 drives on my development server in my home, but I am going to give the Microsoft SyncToy a try for this task. The Amazon S3 backup is extremely important! Every important file you own, not just images, should be backed up offsite! I came close to learning the hardway.

Hope this helps someone out there.

Sitecore Diagrams Part 2

Sean Kearney — Fri, 07 Nov 2008 14:54:00 GMT

We have been doing some pretty fun work with Sitecore 6 here at Hedgehog Development... I started doing some work with Sitecore Template Diagrams a while back and I got to talking with some colleagues and we decided to take this to another level. While I can't get into the specifics (just yet) about the entire project I can show you the diagrams being generated!

The previous version of the diagrams were simply jpg files being generated on the web server. While they were useful for seeing quick representations of your templates, it wasn't too interactive. This new version is tied to the serialization of items in Sitecore and therefore allows you to work outside of a Sitecore context. It is a Windows user control with the ability to move the items around and the future possibility of editing, updating, etc...

This particular screen shot is showing Sitecore template items of sample web site. As you can see we are showing some basic properties of the item as well as building up the sections and fields of the template. We can also enable showing standard values as well as non-template items (content) however, the graphs aren't as much fun.

This is just the tip of the iceberg of what is happening here. As soon as I get clearance I will share more.

Photo Management

Sean Kearney — Wed, 20 Aug 2008 23:07:00 GMT

I am running into a problem. I have never been much of a picture taking kind of guy... that is, until I got a dog and a son. Now I am snapping pictures like a mad man with my Canon Xsi and amassing thousands of pictures named IMG_9321.jpg. I need help!!!!

I have my photos organized in a X:\[year]\[month]\[event] folder structure, but as time goes on I am forgetting what happened when. When my cat loving friends come over, I want to show them a picture of Wilson eating the neighbors cat, but I can never find it. (joke)

I had tried Picasa in the past and it looked promising. I am tempted to get the latest version and give it another shot, but I am wondering what my other options are. I did a little looking around and then started thinking... why not get a mac? Did I really just think that?

I will surely try the free Picasa before jumping ship and spending $2k plus on a macbook pro, but that macbook may come in handy for, say, trying .net development?

It's a sad day for DMB fans

Sean Kearney — Wed, 20 Aug 2008 01:33:00 GMT

What sad news. LeRoi Moore, The Dave Matthews Band saxophonist has died.

I have been a DMB fan since the mid ninetees when they first broke out and to be honest, they were all I listened to for many, MANY, years. I have slowly began to venture away from them over the last couple years, but to this day DMB rules my iPod. Being a sax player (although I haven't touched an axe in years) LeRoi was my favorite musician in the band. I have seen them in concert a few times and LeRoi's solos were always something to look forward to.

"LeRoi Moore on the saxophone"... no more. Sad.

Upgraded Digital DJ Pool

Sean Kearney — Wed, 13 Aug 2008 23:18:00 GMT

DigitalDJPool.com is the online equivalent of a "record pool" more commonly known now as a "Digital Pool." For those who aren't aware, a record pool is a source for DJ's to get promotional music from record labels for use in clubs/radio/etc... The record labels provide the music to "the pool" and the pool can then provide feedback to the label. Now that MP3's are being used more than vinyl, DigitalDJPool.com provides DJ's with promotional MP3's rather than shipping crates of records!

The new site has streamlined some of the major issues members have reported in the past:

New, simpler, rating system
Longer previews
Grouped tracks
and more

DigitalDJPool.com has been around for a few years now and it was due for a major overhaul! The site was originaly powered on the Community Server 1.0 platform and has been upgraded to most of the major CS release up to now. As of now it is running on Community Server 2007.1. If you check out the site you will notice that it isn't using much of Community Server's features and you are correct. As the site has grown we realized that the end users want to download music, not blog or participate in other community oriented things. We weren't ready to jump ship on CS with this release, but it is the most likely scenario. We have a professional license, but with the never ending increase in license costs it doesn't seem like a good path to continue down.

iPhone Part Deux

Sean Kearney — Mon, 09 Jun 2008 22:02:00 GMT

This isn't an iPhone review blog post. There have been hundreds in the last hour since it was announced.

Wow. It has been about 16 months since the I first wrote about the iPhone coming out; I still don't have one. I've read the reviews, played with it, and liked it, but as I wrote, I wasn't going to spend $400+ on a phone. For $199 (maybe cheaper with a 2 year contract) I'll take two please. For $200 you get a fully capable web browser, pocket GPS, email, camera, yada, yada, yada, and, oh yea. a phone too!

July 11th I will finally get rid of my Razr!

Sitecore Template Diagrams

Sean Kearney — Wed, 09 Apr 2008 01:00:00 GMT

With Sitecore templates, unlike C#, you are allowed to inherit from multiple bases. This allows Sitecore developers to be extremely flexible in their implementations, but as with most applications, flexibility adds complexity.

My current Sitecore project has hundreds of templates, inheriting from upwards of five base templates, with up to 6 levels of inheritance! On a single person "team" managing these templates and their inheritance might be doable, but once you have five developers adding or changing templates, fields, sections, bases, etc... it can get out of control and hard to manage really fast! What we needed was the equivalent of a class diagram to lay out or templates and their hierarchy.

Using some good 'ole graph theory -- and some assemblies found on the net -- I now can look at template diagrams like this:

Each box represents a Sitecore template.
The "Number:" can be ignored
The Text at the top is the name of the template
The 2nd row of Text, which isn't indented, is the field section
The indented row of text is the field name

So, the "Highway Marker" template has the following fields:

Color
Text
Side.Height
Side.Length
Number of Sides

Obviously, I can not share the actual template structure of my current project, and this is an extremely trivial example. If you want to see what can be generated from your complex structure drop me an email.

Sorry, but I can not share the code either.

My life will be forever different!!!

Sean Kearney — Tue, 12 Feb 2008 00:00:00 GMT

So, in my last post title, "My life may be forever different," I shared with you that I am getting a dog. I did get the dog, and his name is Wilson. Has my life been changed, absolutely. Having Wilson has been a fun experience with its doses of pain and adjustment. But, I digress, this post isn't about Wilson and how my life has changed. This post is about how my life will surely change...

My wife and I are expecting to be parents this year.

Woo Hoo

Sitecore REST API Service

Sean Kearney — Fri, 18 Jan 2008 22:44:00 GMT

If you are a Sitecore developer, go search the Sitecore Developer Network (SDN) for "REST" and tell me what you find. Nothing. Anyone who develops with Sitecore knows that documentation is lacking, but shouldn't this be something worth mentioning on your site? I digress.

I was looking at the defaul 404 page for Sitecore (/sitecore/notfound.aspx) and I happen to notice the "rest.aspx" file. Interesting. So, I start digging through the SitecoreClient.dll and yada yada yada here are my findings... You can access the REST API from /sitecore/rest.aspx it expects two Querystring parameters, itemPath and _method.

For instance, /sitecore/rest.aspx?itemPath=/sitecore/content/Home/Articles/link-to-us&_method=GetItemData

The other method supported is GetChildInfo as in /sitecore/rest.aspx?itemPath=/sitecore/content/Home/Articles/link-to-us&_method=GetChildInfo

Check out the Sitecore.Rest namespace in Sitecore.Kernel.dll and the Sitecore.Services namespace in the Sitecore.Client.dll

For god sake man... Backup Everything

Sean Kearney — Sun, 06 Jan 2008 14:51:00 GMT

I had a fun Friday. I woke up at my usual time (6am) and dove into my usual routine. While the wife is in the shower I jumped on the computer and my browsers default home page (www.carknee.com) wasn't coming up. humm. Checked a different site on the same server (www.rswarrior.com) and that wasn't responding. Can I ping it? Nope. *dang* I tried pinging a couple other servers in the facility, but they weren't responding either. Double dang. I went to work and figured a bad switch was being replaced and it would be back up when I got to the office.

At 7:30am, still no server so I called the web hosting facility... no answer and no voice mail!?!?!? At 8am I called the owner on his cell phone (it is nice to work with small companies), but no answer. At 8:30am I called again, but this time I got an answer! He said he can't get into the building because the Arson squad is still investigating. SAY WHAT? Apparently the web hosting facility had a gas explosion.

My co-located web server is now off the grid sitting in a war zone. Initial thoughts went as follows.

***.
Was there, and if so how bad was the, fire?
If it survived the fire, would there be water damage?
If there was no fire or water damage, would it survive the pressure of the explosion?
If it survived all that, would it be ok in a room that could have collapsed in on itself?
Oh ***, I'm f'ed.
When were my last backups? Where can I host a backup server before end of business on Friday?
Oh ***... I'm really f'ed

So, around 10am I drove to the web hosting facility (again, nice to work with small local companies). The owner was there, but he had yet to go in. The scene was covered with the local Gas companies workers and the fire marshal still wasn't letting the owner in. I left around 10:45 since there was no way I was getting to get what ever was left of my box.

Numerous conversations with owner over the course of the afternoon, but he was still not allowed in the building. I showed up back at the facility around 5:15pm and he still wasn't allowed in. An asbestos removal team was finishing up and he said I can get in around 7pm. Rather than showing up at 7pm I figured I would just call this time... he picked up and I could sense the relieved tone in his voice! He said, "oh, you didn't know? Your server came back online at 6:30!" Wow. He boarded up the blown out windows and doors, brought in ancillary generators and got some of his facility back up and running. His main generator was out of commission since it ran on natural gas.

I was down for about 12 hours total, but what would I have done if my box hadn't survived this? I had a database backup from a month ago. I had critical web applications on my development PC that I could have deployed, but most everything else would have been gone. Years of data up in a cloud of smoke.

Why don't I have a disaster recovery plan or backup strategy in place you ask?

I never expected my facility to blow up. I have RAID'ed drives and figured worse case a drive would go bad on me. Two can't go bad at once, right?
I did have a backup server running at my home. However, I never stuck to my manual backup process and I ran out of disk space!

Lessons learned:

*** happens! You know that worse case scenario you think about, but figure it can't happen? It happened.
Backup everything --everyday-- and keep it offsite! RAID is great, but it won't save you if your entire server blows up (or gets stolen -- that's another story). Amazon S3 time perhaps?
Stay friendly with the people at your web hosting facility! I was one of the first servers he powered on. If I wasn't onsite, and calling him regularly, my server might have been sitting under rubble for the weekend.

Someone posted this as a joke when they heard of my misfortune

An unwritten rule of thumb in IT management:
if an organization has no database backup
and no database backup strategy,
and no database backup contingency strategy
then the organization has
no organizational intelligence.

Live and learn I guess.