TechHike

The case of the 30 day memcache timeout on app engine

2011-10-28T00:00:00Z

I ran into an interesting problem with Google app engine. It seems that the memcache time limit is enforced differently on the development server versus the production servers.

The documentation says that the time limit for a memcache key is 30 days. The behavior that I have observed is that if you set a memcache key to timeout at 30 days, the development server (on windows) will happily persist the data as expected.

On the production servers, it seems that the behavior is much different. I have notice with a timeout of 30 days, that memcache will not only ignore the persisted key but it will also cause erratic behavior for instances in your application.

Under these conditions a new instance of my application was start for each request! Obviously this adversely effects performance.

Once I changed the memcache timeout to a lesser value (under 30 days), the problem stopped.

iPhone 4S disappointment

2011-10-05T00:00:00Z

Today Apple announced the iPhone 4S. Most people had sky-high expectations for an iPhone 5 or a Facebook deal. The truth came down from on high: new internals, a spec upgrade and some new software.

Immediately most people were disappointed. Has the cupertino machine finally fallen victim to their on secrecy? As usual Apple did not breathe and early word about today’s announcement. That lead everyone to vaulted expectations.

From SplatF:

Cue the whiners. Apple didn’t announce an iPhone 5 today, just an updated version of the iPhone 4, called the iPhone 4S. Apple stock is down a bit in a flat market. Many are claiming to be disappointed. But here’s the thing: You shouldn’t be.

For users, this is still the best phone that money can buy — even better today, with some new features. And for investors, Apple will sell record numbers of these devices, probably earning record profits.

My prediction is that Apple will sell a lot of iPhone 4S units. Face it, it is a great phone. In a month, the missed expectations will lose focus and the facts will shine through.

What if Apple had released the iPhone 4S named the iPhone 5? No one would be complaining.

Should you be a platform developer?

2011-10-04T00:00:00Z

Over the past few years I have seen many developers struggle with a growing dilemma, do I devote my time to platform development?

When I say “platform development” I mean building application on products such as Microsoft SharePoint and Microsoft CRM. The IT world is rapidly moving toward standardizing on platforms such as these. They offer lower IT costs, standardized infrastructure and overall lower cost of ownership. It makes sense from a business perspective, but what do you do as a developer?

The problem here is that we have all been devoting all of our time to becoming better coders, designing systems and generally attacking every problem from the ground up. Not so in the platform world. You begin with a packaged software product, then extend it to meet a customers needs. These extensions can be simply point and click configurations, embedded code such as HTML, CSS and JavaScript, or compiled code which changes behavior in the product.

In the new world of platforms, we are asked to live in a box. That box is called a platform. They bring limitations, boundaries and a sense that we can’t use all the tools that we have worked so hard to learn and build over the years.

Most of my experience is with Microsoft SharePoint and Microsoft CRM but the trend is industry-wide. IT shops have been burned by custom software and they want some predictability over time. I get it.

Customers generally expect the platform to host a wide range of applications and act as a clearing house for all future development activities. The problem comes in when requirements don’t match up to the platforms capabilities. This can be a scale issue, or a mismatch of the smallest detail. These gaps are extremely painful for developers because the client is expecting the fully baked solution, but the platform doesn’t entirely deliver. It’s up to us to build that last 10% of the solution, which takes 50% of the effort. Blarg!

In other scenarios, the product (particularly SharePoint) is a really “big hammer” that solves lots of problems and you have no choice but to use the entire platform. For instance, say a client wants Content Management. Well, SharePoint certainly does that, but it also comes with collaboration, profiling, personalization, single sign-on, search, etc. Sometimes it can feel like you boiling the ocean for a cup of tea.

I have struggled to reconcile these issues and come to the conclusion that there are really two options for developers.

First Option

Swallow your pride and start learning a platform. There are a ton of opportunities right now for platform work, that work is not going to end any time soon and you might as well take the plunge sooner than later. There will be some limitations but in return you will get some opportunities that were not possible before.

Usually enterprises have a portfolio of systems, and your project is just one of them. In the platform world you will need to integrate with other systems in a more reliable and sensible way. Sometimes this integration work is custom (cha-ching!) other times it is not. But the point is that the platform allows you to think at a higher level without being mired in minutia of coding all the components yourself. The upside is that you can use you brain cycles to think about solving bigger problems and delivering more value.

Second Option

Punt. Find a product company you want to work for, or a start-up that seems interesting. If you are in consulting there is a majority chance that you will be moved in to a platform role. If that’s not where you want to be, get out immediately.

I’ve seen too many people struggle through the transition. Always battling to get the custom application development work on any project and generally hating life. Don’t get swept away in the idealized vision that you can buck the trend. See this transition for what it is to you, either a way to move in a new direction or get out before you cause yourself a lot of heartache.

Personal Backup Strategy

2011-10-03T00:00:00Z

There are a million and one ways to backup files from your hard drive. For me, the most important things are making sure that the files that cannot be replaced are safe.

Rather than opt for a relatively complicated system image based backup system such as Windows Home Server or a automated file copy utility such SugarSync, I instead prefer good old fashioned Robocopy and physical DVD media. This might sound archaic in the age of cloud computing but I have found it to be the best way to ensure that I will not lose the most important files and have the least amount of headaches.

My setup

I have a portable 1TB drive which holds all of my media and working files. This includes family pictures, videos and virtual machines. I also have a non-portable 1TB drive that I use for only backups.

For everyday files like word docs, excel spreadsheets, etc. I use Dropbox. These files are important enough that I want then accessible anywhere, anytime but they are not important enough to warrant a physical copy on DVD. Dropbox is great for this purpose, but it’s a big pricey when you go above the 2GB free limit. It runs $99 for 50GB/year and $199 for 100GB/year.

My strategy

Once a week I use Robocopy to backup all important files from my laptop and the portable drive to the backup drive. When a backup is done, all important files are duplicated to the desktop drive. This covers the potential case where my portable drive is either lost or stolen.

Once a month I copy all the very important files to physical DVD discs. These include family photos and videos. I use DVDSpan to burn a set of folders a set of DVD discs. This utility is rather old but it is very useful and has worked fantastically for me. It takes a folder or set of folders and writes them across a set of DVD discs. Granted this won’t work if you have hundreds of Gigabytes to backup.

In my case I am only looking to backup around 50GB of media files. To make things even easier the media files don’t change over time. once a file is archive, it does not need to be archived again.

Then writing files using DVDSpan, it helps to have the files organized into date folders. For example /Photos/2011/2011-01. This way when it’s time to write DVDs, I can simply burn the recent files that have not been archived and ignore the files that I have already archived.

The new discs get sent to the safe-deposit box for safe keeping. I have researched various methods of storing files off site such as Amazon S3 or KeepVault. So far nothing beats driving down to the safe deposit box at the local bank. I have around 60GB of Gigabytes of photos and videos, but the net new data is less than 10GB per month. That means that I can get away with burning roughly 2 DVD discs for each month.

As cloud backup system mature, I may change my approach but for now I’m staying old school.

How to add your last tweet to your blog

2011-09-30T00:00:00Z

I thought it would be nice to see my last tweet on my blog. It turns out that there is an easy way to do this using jquery and the twitter api. These instructions may not work for you if your blogging platform does not allow embedded JavaScript (wordpress.com does not).

To get the tweets for any user in JSON format you can visit http://api.twitter.com/1/statuses/user_timeline/username.json?callback=?

Where username is the twitter username.

Make sure that you first have jquery included in your page like this:

<script src="http://code.jquery.com/jquery-1.6.2.min.js"></script>

Then add this script to pull the last tweet for a username:

<script type="text/javascript">
$(document).ready(function() {
    var u = 'clintsimon';
    var url = 'http://api.twitter.com/1/statuses/user_timeline/'
            + u + '.json?callback=?';
    $.getJSON(url, function(data) {         
        $("#tweet").html(data[0].text));
    }); 
});
</script>

Replace clintsimon with the twitter username that you are interested in.

Next add this markup to your page:

<div id="tweet"></div>

This will act as a container for your last tweet.

As an added bonus I decided to write a little javascript to convert URLs in the tweet to hyperlinks. This mimics the behavior on twitter.com. Also, I added a function that creates hyperlinks for twitter handles such as (@ClintSimon). This link points to the appropriate profile on twitter.com.

<script type="text/javascript">
function urlize(input) {
    var exp = /(\b(https?|ftp|file):\/\/([-A-Z0-9+&@#\/%?=~_|!:,.;]*[-A-Z0-9+&@#\/%=~_|]))/ig;
    output = input.replace(exp,'<a href="$1">$3</a>');
    return output;
}

function tweep(input) {
    var exp = /(\@(\w+?)) /ig;
    output = input.replace(exp,'<a href="http://www.twitter.com/$2">$1</a> ');
    return output;
}
</script>

The final javascript looks like this:

<script type="text/javascript">
$(document).ready(function() {  
    var url = 'http://api.twitter.com/1/statuses/user_timeline/<username>.json?callback=?'
    $.getJSON(url, function(data) {         
        $("#tweet").html(tweep(urlize(data[0].text)));  
    }); 
});
</script>

Cheers!

Why Google Chrome wins

2011-09-30T00:00:00Z

Google's chrome browser is quickly gaining momentum and market share. Recently it has even been projected the before the end of 2011 chrome will surpass firefox as the second most used browser, behind Internet Explorer.

Here's what I think is great about Google Chrome:

It gets out of your way. As the name indicates, the browser is not in your way at all. The content is front and center and the browser is an afterthought.
Excellent JavaScript performance. JavaScript has become the way all web applications are written. In practice, chrome has the best JavaScript performance that I have seen. It blows away Internet explorer, event IE9 (although IE9 has come a long way).
The release schedule rocks. Major versions are released every 6 months and the upgrade process is completely seamless. Chrome is already on version 14. Granted the major releases are not as major as a new IE release, but it is clear that Google is committed to an aggressive release schedule.
Great development tools. The developer tools in Chrome are second only to Firebug. That's saying a lot, firebug is incredible but Chrome is a close second. In some ways I like Chrome better, especially the DOM explorer.
Sync everything. All browser settings, bookmarks and passwords can be stored in the Google cloud. If you roam across machines or rebuild your machines often, this is a big time saver.
HTML5 and CSS3 support. Support for new technologies is great in Chrome, thanks in part to the fact that it is built on top of the popular open source WebKit framework.

This is interesting in a lot of ways. First, Google is not generally thought of as a desktop software company. This is changing and Chrome is a part of that.

For Google, Chrome is strategic in that it gives Google some control (and consistency) over how their services are presented on the desktop. Google is a web software company and Chrome is an attempt at controlling the "last mile" problem. This is important as it gives Google a foot hold on the desktop and a fighting chance against Microsoft.

Chromebooks are also an interesting concept. A computer that is basically just a browser, in a lightweight package. Chrome is obviously a huge part of these machines and Google is banking on the fact that Chrome can become an everyday part of your web experience.

Google's new App Engine pricing

2011-09-29T00:00:00Z

Google App engine is in the process of changing it's pricing model. At present this new pricing model goes into effect on December 1, 2011. There has been quite an uproar among App engine developers. In some cases the new billing model raises monthly costs by more than 300%. This has cause quite a reaction among the community and the backlash is very real

What is changing?

The primary change is that Google is moving from a model where you are charged by CPU time to a model where you are charged by "instance time". This means that google charges are not necessarily based on pure CPU use but rather on the amount of instances of your application the Google scheduler decides to run. To magnify matters, most web applications are IO bound and the current version of Python supported by App Engine is single threaded.

That means that your application is probably using more instances that you think. Because one instance can only handle a fixed amount of traffic based on IO and threading, the scheduler is forced to create instances more often for Python applications. All of these circumstances add up to a much higher cost for developers.

Google has added sliders that allow you to tweak the scheduler behavior. You can now change the maximum number of idle instances and the minimum pending wait time. This allow us to dictate how long is a acceptable waiting time per request and how many instances show be available for traffic spikes.

I'm personally glad these are available, although others disagree. I have found them to be helpful for my small(ish) application because latency is not a concern, so I can set the min latency up to around 700ms. This means that requests can wait to be served from an existing instance. Basically all output from my application is cached, so my request processing time is under 100ms. Setting the min pending latency to 700ms basically means that my application can remain free. I like free.

It appears as though CPU billing was a bad idea in the first place and Google is now correcting for this by changing to instance based billing. In the end, Google needs to pay for the physical resources an application uses and as it turns out web applications use very little CPU as compared to IO with the datastore or other APIs.

So, is the dream of a true "Platform as a Service" dead? Yes and No. The precise billing aspect of PaaS is dead at Google for the moment, but App Engine (and PaaS for that matter) is not defined by this shortcoming. Ultimately Google has recognized that having an instance running costs money even if the CPU usage for that instance is zero. Said another way, memory costs money. An instance will always consume memory if it is running. And not only will it consume memory but Google will need to reserve all the memory that it may need. This is a reality of physics.

I think we have become a little spoiled in that we have come to expect that computing will always become cheaper over time. I wonder how much it would cost to build your own ultra scalable, redundant, highly replicated system?

Optimization

In the new world of App Engine where instances are charged, the emphasis is now on eliminating IO bound activities. That means eliminate datastore calls, URL fetch call, etc. If you have instances that are waiting around for IO activities to complete, then you are running to many instances.

Here are a few ideas: Don't pull more data than you need for the data store. Use "Select key" where possible. Limit your fetches to the smallest possible row sets. Cache results when ever possible. If you are displaying the same data, then cache it per page, per user or per application. * Use a Task Queue. Can this operation run in the background? Task queues can eliminate instances by way of pushing work for a dedicated backend.

The good news...

App Engine was not decommissioned. I know this sounds like little consolation, but many Google projects have been scrapped over the past 6 months and App Engine is not one of them. This shows that Google is willing to make difficult decisions when it comes to App Engine and that they are committed to having a PaaS offering for the long term.

The bottom line is that Google has a right to at least break even on App Engine. The service is incredible, the free tier still exists and the new SLA terms are top notch. App Engine is still a worth competitor in the PaaS market, even with the new pricing.

If This then That

2011-09-26T00:00:00Z

I just started to use a new service that allows you to automate your social life. It's called "If this then that" (http://ifttt.com).

You can setup rules that says things like, if I post a link to twitter then create link post on Facebook. Or, if someone tags me in a picture on Facebook, copy that photo to my dropbox account. These are just a couple of the thousands of possibilities. Serivices can be combined in a multitude of cominations, these even include phone, sms and email.

How would you like to save a voice message to your dropbox by calling ifttt.com? That's possible.

I've done some experiments and found the service to be relatively fast but not exactly "realtime". Most of the time processing happens within about 10 minutes.

Types of consultancies and how to pick one

2011-07-21T00:00:00Z

Over the years I have come across software consulting companies in various states. It occurs to me that there must be a finite set of categories that defines the culture of a consultancy. Here’s a few that I have recognized.

The oak tree. This company is rock solid and has been that way for many years. After establishing itself initially, stable but modest growth has lead to a consistency over time. To be successful at this company, you must deeply agree with the culture and value system. This can be a great place to have a long tenure and create long term professional relationship. Rebels and change agents need not apply.

The Butterfly. This company is always morphing into something new. Things that are important today may have less valuable in the future. Hot and cold streaks plague the company’s history. Take a job here only if you think that the experience you will gain out weighs the risk of a left turn when you least expect it. Often very few people know when the left turn is coming.

The junkyard dog. This company will do anything to keep busy and stay in business. Web site? Mobile? ASP.NET? LAMP? Yep they do that. Take a job here if you want to be a jack of all trades. These are usually good resume builders and companies where you can figure out what technologies you are most passionate about. The downside can be randomization and lack of direction, usually the technology leadership is not necessarily that deeply technical because it is very difficult to be deep in all technologies.

The Corporation. They are Enterprise. They are monumental. They will put a complex plan into place and deliver solid results on time and on budget through deliberate and consistent execution. You’ll excel here if you enjoy structure and thrive when things are planned out to microscopic detail. Here’s to “synergy”!

First impressions: Spotify

2011-07-19T00:00:00Z

Spotify launched in the U.S. on July 14th. Given the hoopla around the service I decided to give it a try. Spotify is a music service that allows users to stream or download music, includes social integration and has support for offline listening. Many people have raved about the service in Europe and there has been a constant chorus for an american launch.

After 5 days of use, here’s my impressions:

Great value. You get all the music you can handle, offline downloading (including offline for iPhone) for just $10 a month. That is fantastic. I have found that most if not all of the music I care about is available. Spotify has deals with all the major labels, so there is a good chance that it will always have the latest music. There is also a $5 per month option when doesn’t allow offline listening and a free ad supported version.

Version 1 application. The Spotify application for windows is good, fast and stable. It is lacking features though. There is no drill down-down by artist or album. Come on, that is obvious and available in all competing products.

I would say the social integration is fairly simple as well. You can see all your Facebook friends who have a Spotify account and view their shared playlists. This seems like the simplest possible social implementation.

Bottom line is that Spotify needs to catch up to Zune and iTunes in terms of features for desktop.

Fast! I have found that playback is extremely fast and reliable. Streaming is particularly good, you mostly can’t tell that a song is streaming at all. Playback of streaming is almost as good as local playback. Skipping and scrubbing are instantaneous, searches are

Many people, including Sean Parker (who is a Spotify investor) say that Spotify is the anwser to music piracy. I think that Spotify has a lot of potential and will be very popular in the short term but it remains to be seen who will win the cloud music wars.

First marathon complete

2011-07-01T00:00:00Z

My marathon journey started in the summer of 2009. I got pneumonia and spent 3 days in the hospital. After recovering, I knew that I needed to get fit and running seemed like a good idea. I started to run regularly, not really thinking that I would get involved in long distance competitive running.

A few months later, I was running 20+ miles a week. I began to think that I may want to do some races in the area, just for fun. I tried a 5K and a 10K. I quickly realized that competitive racing is very different from training. I was surprised at the energy that I had on race day. Lining up with a few hundred people is a exhilarating experience. It gave me a boost that I was not expecting.

In 2010, I ran 2 half marathons. These were a bit of a challenge but I thoroughly enjoyed the challenge and the feeling of accomplishment. By this time, it became clear to me that I was capable of running a full marathon. Why not give it a try?

I choose to run the Seattle Rock n Roll Marathon. The rock n roll marathon has grown to be the largest organized series of competitive running races in the United States. The Seattle event was close to home and I had run the half-marathon there in 2010.

I used The Non-Runner's Marathon Trainer as a guide. This is a 16 week program which is based on a college course taught with the purpose of turning anyone into a marathoner. I highly recommend the book. It has a solid training schedule and many stories of people from all walks of like that eventually completed a marathon.

I followed the program (mostly). I did skip some of the shorter distance running toward the later part of the program, but I did all the long runs. After the program works its way toward an 18 mile run on week 14, it tapers off to race day.

The training was difficult. From a timing perspective fitting a 3 hour run into my schedule was not easy. Plus there is the recover time, which can be considerable. I could not have pulled this off without support for the wife and family.

As I built up to longer distances, I realized a few things. First and foremost, your body is capable of much more than your mind will allow. One of the most important things is to believe in yourself and use the power of positive thinking. Once negative thoughts creep into your head, you are finished. Giving up is the easiest thing to do. Stay positive and don’t allow your mind to dwell on physical pain or

I found that on long runs, it is most important to set a solid pace and short term goals. Running 18 miles can be daunting, especially when you are on mile 3. I generally try to break the run into 3 mile segments and set a time goal for each. Early to middle distances (1 to 10 miles), I try to run each 3 mile segment in 30 minutes each. This gives me a target pace of 10 minutes per mile. I try to block all thoughts outside of meeting these short term goals.

Another interesting thing for the book, is to have some positive phrases that you repeat to yourself when the going gets tough. My favorite is to say to yourself “Hello ______, come run with me.” As in “Hello knee pain, come run with me.” This works beautifully when you are deflated and thinking of giving up on a run. It implies that you will overcome any and all obstacles on the way to completing your goal.

There could be 1,000 reasons to stop but it’s up to you to stay the course and complete the run by not accepting failure. This has helped me tremendously in life. If I can fight through these challenges, then nothing can stand in my way.

For training, I ran mostly on the hills of Snoqualmie Ridge, in the cascade foothills of Washington. The training was brutal, although running in the hills would pay off in the end.

Then came race day. I felt prepared, I was ready. Still there was some doubt, the longest training run in 18 miles and the race in 26 miles. The book claims a 99.9% success rate for people who follow the training, but there was still a lingering doubt at the back of my logical mind. I pushed that aside and the race started.

I was one of 26,000 people starting the race in Tukwilla, WA. What a scene at the starting line!

the start

For the first 13 miles, I stayed on pace and only stopped to drink. I passed the half way point (13.1 miles) in 2 hours, 6 minutes. My best half-marathon is 1 hour, 54 minutes. Needless to say, I was pleased. :)

Feeling good, I tackled the last half of the course. The going got tough around mile 15, with a steep climb. I started to alternate between walking and running, trying to run at least half a mile, then walking a quarter of a mile.

Luckily at mile 18 the course doubles back for a smooth 2 mile downhill run. I focused on hydration, drinking 8 ounces of water or sport drink every 30 minutes. Around mile 19, my stomach became upset and it was hard to drink anything but water.

After mile 20, I was tested severely. At this point, there was a general sense among all runners that we were in for a serious test from here on out. I saw many people pulling out with cramps or sitting/laying down from pure exhaustion. I soldiered through, using my walk/run strategy.

The cruelest part of the race came at mile 22. Here the course weaves close enough to the finish line that I could see people finishing the race at around 3 hours and 45 minutes. My first thought was, “YES!” it’s almost over. My mind was not functioning clearly, we had 5 miles to go.

Soon, I crested a hill and was able to see the last turnaround, almost 2 miles down the road. My heart sunk. I was not sure if my legs could travel that far after such a long day. I recovered about 10 minutes later.

I realized that I could finish the race in under 5 hours if I ran those 5 miles is under 1 hour. It was a slog. I cramped just above the knee in both legs. I was running as far as I could before cramping and getting only about 2 tenths of a mile at a time. I kept plugging away and finally I turned for home.

Just before the finish line, I saw my wife, my 2 kids and my parents cheering me on. It was the best thing I had seen all day.

me on mile 26

After a quick picture and hugs, I headed for the finish line. What a great feeling. I didn’t expect to get emotional but it hit me when I saw the finish line. I had done it! My first marathon. What an experience.

How to get started with OData

2011-06-06T00:00:00Z

Open data protocol or OData is a new Microsoft technology standard which allows for simple data exchange over a REST style interface. The idea is to create a web based interface for complex data stores and in the process free data from silos. This will in theory create a data utopia complete with bunnies, fairies and a side of mash potatoes.

There are a couple of intriguing facets to this proposition. First, since OData is purely REST based, data is available cross platform and also on the client side. JavaScript is a real possibility as a data consumer, mashups just got a whole lot better. To top it all off, natively you can get data in Atom or JSON formats, making JavaScript a first class consumer.

Secondly, Microsoft has bundled a LINQ provider with the .Net Framework. This makes server side integration a snap. You can literal point to an OData endpoint and start writing a LINQ query against it. Bring on the fairies.

To start out, download LINQPad. LINQPad is a lightweight tool that let’s you query a LINQ data source in real-time with minimal setup necessary. It gives you a great interface to run quick LINQ based queries. LINQPad does not require an OData source in fact you can use LINQ to SQL or Entity Framework endpoints among others.

Once you have installed LINQPad, click on new connection. For the data context user “WCF Data Service (OData)”. For the URI use http://odata.netflix.com/catalog. That’s right the Netflix catalog has an OData interface!

OData sources are broken into top level elements called collections. You can think of collections as two dimensional tables with columns and rows. A column can also represent a relationship between two collections.

You can see that in our Netflix service endpoint there are several collections that define the dimensions tracked within the catalog. Genres, People, Titles, etc. This is where the cool starts. You can now construct LINQ queries against the Netflix catalog by typing some LINQ syntax into the main windows and hitting F5.

If you are unfamiliar with LINQ syntax, here are tons of examples: http://msdn.microsoft.com/en-us/vcsharp/aa336746

To use OData from Visual Studio, open a new project, add a new service reference and paste in the Netflix service endpoint URI. VS will automatically understand that this is an OData service and that it should generate a LINQ context class and all the associated classes that define the data collections and data types. Fairies!

Now you can code against this new LINQ data source as you would any other. However, watch out for performance. Remember that you have taken an outside dependency into your project, anytime you do this there are inherent risks.

Furthermore, LINQ makes the query process very transparent which means that you can flagrantly throw queries at you data source without regard for impact to the service. Bottom line is, think about what you are doing. Do I really need to run this extra query? Can I consolidate this data pull into a single query?

Bose IE2 Headphones

2011-04-25T00:00:00Z

A couple months ago, I found myself longing for a new set of headphones. I have a pair of Sennheiser headphones which are rather large but sound great and have a boom microphone. As I realized, those don’t travel very well and I have recently started to work more onsite and in remote locations.

So the search began for new set of quality headphones. I need something compact and with a built in microphone for VoIP calls (Skype, Communicator, etc.)

I thought of using my apple ear buds but the sound quality is just not very good. Eventually I took a look at the Bose IE2 in-ear headphones. At first I was a little skeptical with the design. Take a look at those large pieces of silicone that you need to shove into your ears. Can’t be comfortable right?

Wrong. After buying them I realized that the fit could not be better. I wear these things sometimes for 8 hours straight without any discomfort. Sometimes I even get up from my desk and forget that I have them on.

I am a big fan. It’s Bose, so the sound quality is great. I don’t think the fit could be any better. The microphone works great on the PC or connect to my mobile phone.

One other note, if you are thinking of using this for VoIP with your PC/MAC, look into the headset adapter buddy. This adapter let’s you connect a conventional mobile phone headset into you PC or MAC with separate jacks for both the output sound and microphone.

5 things you should do when estimating a technology project

2011-03-04T00:00:00Z

Estimating a project can be one of the hardest things you do as a technology consultant. Over the years I have done my share of estimation and one thing has become clear to me. Estimation accurately is almost impossible. With lots of experience and some hard knocks, you can accumulate the courage necessary to throw down an estimate. Here’s a few tips that I think might help.

Ask a lot of questions. The first thing you need to do is ask a ton of questions. The only way to get to the bone of the implementation is to understand all the information currently available about the project. Ask the WWWWH (Who, What, When, Where, Why, How) questions.

When you think you have exhausted all the questions in your head, take a walk and forget about the estimation exercise. When you come back you will have more questions, no doubt.

There may very well be details that haven’t been discovered. As the estimator, it is your job to put down a valid estimate. This isn’t the motivation of the sales person, your business lead or the client for that matter. Having a valid estimate creates a framework and a roadmap that the team will follow throughout the project. Without solid questions you are shooting in the dark.

Be detailed. It’s impossible to estimate any complex project by taking a high level view and throwing out numbers. You need to break the project down into digestible pieces in order to make sensible estimates that can hold water. For a budgetary estimate, I like to have my work broken down to no larger then 8 hour chunks. Based on this rule it is difficult to miss something entirely or overestimate something by more than 8 hours :)

Most times, the deeper that you think about the work, the more you realize that there are intricacies that you may not have considered when thinking at a high level. You need to find the bottom of the rabbit hole.
Assume away risk. One of the most important tools you have as an estimator is the assumption. The assumption is beautiful. It is the way that you politely tell a client no. Assumptions are effectively boundaries that define the border of the project scope. You can say what you will do all day, but that doesn’t tell anyone what you won’t do. Basically there is no good way to make the language around you requirements strong enough to stop a nefarious client from take you for a ride. Assumptions allow you to draw a clear line where the requirement stops.
Understand your client. You need to understand the client’s goals, motivation, risk and reward. This is the context that will make you successful in the estimate process and during the delivery phase. Is the client vague? Are they distracted? Are then detail oriented? Anything that will effect the delivery team should be taken into consideration.

To a varied degree this can give you a “gut feel” for how the project will be run and communicated. If there are red flags, then you need to call those out, make assumptions and eliminate the perceived risk.
Build an estimate capable of satisfaction. The idea here is to produce an estimate that will make a client happy. Things change over the course of a project — new requirements, shifting priorities, unknown hurdles. When these changes happen, it is important that your estimate is big enough that you can still make the client happy at the end of the day.
You are probably thinking, “well, why don’t I just make the estimate huge?”. The trick is to make the estimate big enough to accommodate the unknown but also fit into a logical and economical budget number. If you strike this balance it will allow the project team to manage the project to a happy conclusion because your estimates were large enough to make the client happy. Too small an estimate and there is no room for management.

What my guitar taught me

2011-02-27T00:00:00Z

It turns out that my guitar taught me a valuable lesson. Somehow in this crazy house that I live in, ago a week ago, my guitar spontaneously migrated out of my office. One of the kids moved it to the upstairs bonus room without my knowledge. I usually keep my acoustic guitar right next to my work desk. It sits there 99 percent of the time. Occasionally when the mode strikes, I pick it up and strum a tune. Maybe an old song I like or a totally random conglomeration of melodies that I have long forgotten. It’s a little escape for me when my mind is consumed by heavier things It’s always there for the taking at a moments notice.

Well, not any more. An interesting thing happened over the next week. I didn’t think of playing. Not even once. Nada.

That got me thinking about immediacy and the modern world that we live in. We are constantly engaged with TV, movies, iPad, iPhone, DVR, Netflix, Facebook, twitter, foursquare, text messages,and on it goes. I’m not sure if this is our primitive prey behavior coming to the limelight or if it’s our technology fascination or some combination of both. Have you ever seen a cow sleep with its eyes open? That’s because they are a prey animal and it’s not a good thing to forget about your surroundings when you are on the menu.

Here’s the thing, just a few years ago I was just the opposite. Intently focused for hours on end. I remember on one occasion I totally ignored a colleague who was talking to me in at my desk because I was go engrossed in my work. I’m not proud of that one. So what happened to me? At some point I started to change. My brain went from a calm river slowly but surely chugging along to a sprinkler flailing in all direction at once. It seems each of these states has some value depending on the task at hand.

My guitar taunt me that immediacy is king.

How to get started with OData

2011-02-18T00:00:00Z

Using OData

If you are unfamiliar with LINQ syntax, here are tons of examples: http://msdn.microsoft.com/en-us/vcsharp/aa336746

Getting started with Windows Phone 7 application development

2011-02-16T00:00:00Z

Get ready to create your first Windows Phone 7 (WP7) application! With only minimum software development skills, you can create your very own shiny new WP7 app.

Get the tools

The first step is to go get the tools you’ll need to build your application. Microsoft has the best development tools in world (in my humble opinion). Since WP7 application development is an extension of general .NET platform development and Silverlight technology, you can use the standard .NET tools set to development WP7 apps. This includes visual studio 2010, C# or Visual Basic, and most of the .NET framework. This is a fantastic set of tools with a myriad of features design to make application development as easy as possible.

Here’s the kicker. For WP7, these tools are completely free. No need to shell out the green, Microsoft has you covered.

First go here: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=04704acf-a63a-4f97-952c-8b51b34b00ce

Download the installer and run it. The release to web (RTW) installer will download and install Visual Studio 2010, the WP7 emulator, Silverlight 4 tools for Visual Studio, XNA game studio tools and Microsoft Expression blend for Windows Phone. These are all the tools you need, the install is easier than Sunday morning.

Hello World

Now we can start to create our first project. Microsoft makes it super easy to create and run a WP7 project. Launch Visual Studio and go to: File –> New Project

Now pick the type of project you want to create. For now this is not an important choice, so just pick one. The most interesting choices are the Panorama application and the Pivot application. The Panorama is the nifty application where the view scrolls from left to right and the screen to the right has the edge showing. The Pivot application still scrolls but this more like a tabbed interface where all the tabs show up in the heading and you can swipe through the left and right.

For you first project, choose the Panorama application, it’s the best looking and you do want to impress your friends, right?

Once you have created the project, take a look at the contents. Might familiar if you have done any work with C# projects. It’s a basic Silverlight 3 application with some extra juice added to run on the phone. Don’t worry about the juice for now. MainPage.xaml is the entry point for the application. My default it uses MVVM or the Model View View Model pattern. This pattern is somewhat new but it is a way to separate the presentation from the data and logic, which is a good thing. Nice to see the Microsoft is being a bit progressive.

Drumroll please …

Just hit F5. You now have a running WP7 application in under 2 minutes. Notice all the details: fluid animations, scrolling, sample data. In short, this is a working application that you can simply start to customize. This could not be easier. To me this is the power of Microsoft at work in a real way. They have brought together a strong set of technologies here, dev tools,Silverlight and the phone OS. The OS still need to mature a bit, but I think you can see that this is a baseline for building great mobile applications.

Will it blend?

One more thing (like that? I borrowed it from a famous guy I won’t mention here), Expression Blend is in the mix also. For those unfamiliar, Expression Blend is like Visual Studio for designers. It lets creative types work on the same xaml file that developers work on, it understands visual studio projects and it can churn out all the animations, icons and motion that are available in Silverlight and WPF. If you want to develop WP7 applications, make Blend your friend.

To launch Blend, right click on the project inside Visual Studio and choose “Open with Expression Blend”.

You will be greeted with a design surface with tons of bells and whistles. Blend can be daunting when you open it for the first time. It is one of those tools that you will probably never learn in totality because it is complex. The best way to learn is to tackle it.

Building more complex apps

I am certain that there are application on the marketplace that were written in under 2 hours. If you already have a data source and you want to tie to a WP7 app as an alternate access method, general it is very easy. Microsoft gives you an exceptionally good starting point with the built-in WP7 project templates. I have only touched on the true potential here.

Go make some apps!

Code is Creative

2011-02-14T00:00:00Z

Most people think of programmers as the guys guzzling mountain dew in a dark room where pizza slides under the door at around 6PM and a feature is built by 2AM. Although this is can be the case (we’ve all been there), I have found that most of the people that I look up to are not at all like this. More often than not people who have influenced me over the years are people that show a genuine passion for software but also have massive creative energy and a social team mentality to boot. This is a paradox that defies stereotypes. It turns out that creativity is an essential ingredient if you want to make a good software developer.

Starting out a career in technology is a steep mountain and the only way to get anywhere is to start grinding and never stop. It is an endless cycle of solving problems from the most mundane logic games to epic enterprise systems. At some point along the way we start to see patterns and recognize that it is possible to improve this cycle with a little planning and some creative thinking.

Even today, 15 years in, I am still constantly looking to solve that next problem, my brain always straining to fit the puzzle pieces together in a way that will make the picture work. But what is my brain doing deep in the subconscious?

I’ve had this thought many times and while I have no real explanation, I have a good feeling that some creativity is at work. What else can explain it? I have experiences. I have facts. The only thing my brain has to do is fit those together in a way that solves the puzzle. In many ways this much more a creative exercise that an analytical exercise. Maybe analytical thinking is irrevocably tied to creativity in ways that we cannot yet understand.

The true challenge as a software developer is to find creative solutions to daily problems. Learning APIs, patterns and systems is easy compared to applying those constructs to solve a real world problem. Creative thinking is what separates that average coder and the brilliant thinkers that bring us innovations that change our lives.

Using the SharePoint content database as a cache

2008-09-12T00:00:00Z

Data caching is a core concept in the web dominated world that we live in. The premise is that we should not have to go back to retrieve data from a data store for every single web request when the data is unchanged, thereby sharing data across web requests. This offers huge performance advantages because all latency related to data fetching, query processing and transmission is eliminated for requests that use the same data. In the SharePoint world this can be a very valuable technique when querying MOSS page libraries or any data stored in the content database. For high traffic sites, the performance of the SPQuery or SPSiteDataQuery api is not acceptable when executing queries for each request.

The most common data caching implementation is using the HttpContext.Cache collection. This is an ASP.NET facility that allows as an in-memory object cache associated with the ASP.NET application domain (w3wp.exe). For SharePoint applications this cache works great when caching can be scoped to the HttpContext. For applications that need caching that is not scoped based on the HttpContext it is necessary find a different solution. The most common case non-HttpContext caching would be any instance where you are consuming data outside of the ASP.NET context such as SPTimer jobs, stsadm commands, winform applications or console applications. For these cases it is not possible to use the HttpContext cache.

There are a few potential solutions to this problem including using the HttpRuntime static cache, but the best solution (in my opinion) is to use the SPSite instance as a data cache. The idea is that by using the SPSite.RootWeb.AllProperties hashtable, we can store cached data into the content database for later retrieval. This solves the HttpContext problem because for any application to operate it will need to have a SPSite object to provided context for it's operation no matter where it may be running. Because the SPSite object provides context, the data cache is always available.

Here's the class signatures for my implementation of SPSite data caching. By creating a subclass from this base you can implement application independent caching of any data type. The full implement can be found here.

public abstract class RapidCachedDataAdapterBase<TData> where TData : class
{
    protected Guid SiteID;
    protected virtual bool AutoRefreshCache;
    protected virtual string CacheKey;
    protected virtual string CacheExpireKey;
    protected virtual int CacheTimeoutMinutes;

    public virtual TData CachedData;

    public RapidCachedDataAdapterBase(Guid siteID);
    public RapidCachedDataAdapterBase(SPSite site);

    protected abstract TData GetNativeData();
    protected virtual TData GetData();
    protected virtual TData GetCache();

    protected virtual void SetCache(TData data);
    protected virtual string Serialize(TData data);
    protected virtual TData Deserialize(string data);

    protected string SerializeInternal(object data);
    protected T DeserializeInternal<T>(string data);

    public virtual void InvalidateCache();
}

Notes:

GetNativeData() is the only method you must override when creating a subclass. GetNaiveData() provides data from the native datastore to feed the cache, it is called any time that data needs to be retrieved from the native data store.
Serialization to a string is necessary for all cached data types. Since SPSite.RootWeb.AllProperties accepts only limited data types (date, int and string), it is necessary for us to convert our data into a string. My default the base class uses the standard XmlSerializer class. If you are caching a data type that is not xml serializable (such as Hashtable, Dictionary & DataTable) then you need to implement the Serialize() and Deserialize() method within your subclass. You can choose to serialize your data any way you like as long as it can be represented as a string.
The CacheTimeoutMinutes property determines how long the cached data will live. An automatic refresh will be triggered when the timeout period expires. The timeout period is set to 20 minutes by default but you can override this in your subclass.
You need to handle cache invalidation when native data changes. Cache invalidation is supported by the base class, but it is only called automatically if the cache expires due to a cache timeout. You will need to identify the cases where you would like to invalidate the cache and call InvalidateCache() at the appropriated time. Usually you would invalidate the cache when you detect that data has changed, such as when a user changes a list item. In this case you would need to have a feature receiver to detect the change and call InvalidateCache().
The CachedData property used for data access. The CachedData property will either load the cache or retrieve the cache depending on the situation. You can use the CachedData property as the single access for data.
Using SPSite.RootWeb.AllProperties does carry a cost. Because we need to load and manipulate an SPSite object, there is an implied cost associated with this approach. Anytime SPSite is loaded or saved SharePoint needs to communicate with the content database and that takes cycles. This is in sharp contrast to the HttpContext.Cache approach where all data is stored in memory. To mitigate this cost the base class uses the HttpContext.Items collection to cache data for each http request. This means that the SPSite object will only be loaded used once per request.

Here's a sample subclass:

public class FaqDataAdapter : RapidCachedDataAdapterBase<DataTable>
{
    public FaqDataAdapter()    : base(SPContext.Current.Site.ID)
    {
    }

    public FaqDataAdapter(Guid siteID) : base(siteID)
    {
    }

    public FaqDataAdapter(SPSite site) : base(site)
    {
    }

    protected override DataTable GetNativeData()
    {
        DataTable table = null;

        SPSecurity.RunWithElevatedPrivileges(() =>
        {
            using (SPSite esite = new SPSite(SiteID))
            {
                SPList list = esite.RootWeb.GetList("faq");
                table = list.Items.GetDataTable();
            }
        });

        return table;
    }

    protected override string Serialize(DataTable data)
    {
        if (data == null)
            return null;

        var ds = new DataSet();
        ds.Tables.Add(data);
        return SerializeInternal(ds);
    }

    protected override DataTable Deserialize(string data)
    {
        if (string.IsNullOrEmpty(data))
            return null;

        var ds = DeserializeInternal<DataSet>(data);
        return ds.Tables[0];
    }
}

Customize the web.config with a SharePoint XML customization

2008-08-16T00:00:00Z

It is possible to manipulate the web.config file globally by using a little known feature in SharePoint that allows developers to describe a set of changes by using an XML customization file. In fact, both SharePoint and MOSS use this mechanism to publish web.config changes when a web application is created.

Create the customization file

The customization file is used by SharePoint to apply changes to the web.config file when a web application is created. Any changes that are inside the customization file will be applied to all web applications globally. This method of customization does not work for cases where you would like to apply web.config changes to only a specific web application.

In this case I am adding a custom HttpModule to the web.config file. The web.config node will be appended as a child to the configuration/system.web/httpModules node. The id attribute needs to be unique. Everything inside the add node will be appened as a child to the node specified in the path attribute.

<?xml version="1.0" encoding="utf-8" ?>
<actions>

    <add path="configuration/system.web/httpModules" 
        id="{8DD2A7B1-0223-4be8-B1B5-339B642EFB21}">
        <add name="MyHttpModule" 
        type="MyHttpModule, MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=a1df43608d33992f" />
    </add>

</actions>

Copy the customization file to the web front-end

The xml customization file needs to be placed the 12Config directory and it needs to be named webconfig..xml where is a name specified to your customization.

You can take a look at the 12Config directory to see examples of web.config customizations that are applied by SharePoint and MOSS. You will see that you can not only add nodes to the web.config but also remove or update nodes that exist in the web.config file.