OnPreInit.com

Does a policy prohibiting jQuery in Enterprise applications make sense?

noreply@blogger.com (Nariman Haghighi) — Tue, 02 Nov 2010 22:02:00 +0000

I just don’t get it, I actually couldn’t believe it at first; what possible justification would there be for the all-out prohibition of jQuery within an Enterprise? Believe it or not, this is actually happening, on the doorsteps of 2011, and in larger numbers than I would’ve imagined.

Whatever the case may be, here’s an ironclad top 10 list of why avoiding jQuery is a costly mistake:

It’s far and away the most popular and fastest growing JavaScript library on the planet [2]
It’s arguably the safest, most scrutinized library of its kind with no open vulnerabilities
It’s more pervasive than IIS itself – in use by > 28% of all websites on the internet [2] [3]
Even the US government uses it at Whitehouse.gov [2]
It’s distributed freely across the globe through both MS and Google CDN edge-servers [2]
Microsoft has committed to help with QA testing and code development. It also includes jQuery as part of Visual Studio and ASP.NET MVC [2]; on top of that, Microsoft ‘now offers actual tech support for jQuery as part of their Product Support Services (PSS) as jQuery integration has become part of several of the ASP.NET toolkits’ [5]
Nokia is contributing with testing and has also hired one of the core developers, Brandon Aaron [2]
It supports browsers from virtually a decade ago – everything from IE6 and FF2 onward [1]
jQuery solutions are far more supportable than their raw-JavaScript counterparts
Forcing enterprise developers to hand-roll JavaScript is not a solution, just a costlier, more error-prone and less-secure alternative

Hopefully getting the right information in the hands of decision makers is enough to put this to an end.

In what scenarios have you seen jQuery use prohibited?

[1] – Compatibility
http://docs.jquery.com/Browser_compatibility

[2] – jQuery’s triumphant march to success
http://royal.pingdom.com/2010/03/26/jquery-triumphant-march-to-success/

[3] – June 2010 Web Server Survey
http://news.netcraft.com/archives/2010/06/16/june-2010-web-server-survey.html

[4] – Does a policy prohibiting jQuery in Enterprise applications make sense?
http://stackoverflow.com/questions/4082401/does-a-policy-prohibiting-jquery-in-enterprise-applications-make-sense

[5] – Microsoft and jQuery
http://west-wind.com/weblog/posts/807874.aspx?id=807874

What the Enterprise Can Learn From StackOverflow.com – Top 3 Lessons

noreply@blogger.com (Nariman Haghighi) — Tue, 02 Nov 2010 13:26:00 +0000

Update (Nov 3, 2010): I stand corrected, the president of Fog Creek Software informed me on Twitter that ‘the Stack Overflow Product is actually available for enterprises and a few of the top banks are using it internally”. Hopefully we can get more details around this soon…

Stackoverflow has turned the antiquated concept of basic Q & A into a very successful modern franchise. Enterprises grappling with fostering engagement within their knowledge communities should take note. While this platform is not available to to an internal audience, there are key elements that can still help in shaping your unique strategy.

Without further ado, here are the Top 3 StackOverflow Lessons for creating an effective knowledge-base:

Kill Person-to-Person Collaboration

This is the kryptonite of effective knowledge sharing, especially in the Enterprise. Notice that Stackoverflow profiles don’t publish user emails – this is partially to protect the user from unsolicited contact and a flood of questions from newbies, but also to foster person-to-community collaboration. Enterprise Search solutions attempt to ‘open’ content that was previously sealed behind business silos after the fact – well, you can start by not allowing this content to be hidden in the first place. This is poorly done in most organizations, with most users shying away from public distribution lists to carry personal conversations under the fear of inconveniencing others!
Make Participation Highly Addictive

Community up-take is the make-it-or-break-it factor for these applications, meaning they must be highly intuitive (low-barrier to entry) and highly addictive. Stackoverflow achieves this delicate balance through a simple yet competitive reputation system that draws out best-of-breed developers simply to compete and to learn from one another. A slightly refined version of this would work within the Enterprise as well; far too often, the incentive to participate just isn’t there. Before you can reward top participants, you have to start measuring.
Recognize Top Contributors Externally

This one is somewhat controversial (like most new ideas are) but imagine a scenario where the ranking of the top contributors within your organization were opened up externally to prying eyes. Sure, this might be a frightening scenario if you’re worried about exposing (and losing) your top talent, but it’s also the strongest reward you can provide, too. Most traditional incentive/reward programs are completely hidden from the outside world, which misses the point, IMO. As nice of a gesture as a jacket or a $100 gift-certificate may be, the real value to heard-working professionals is recognition of their dedication and devotion within the broader community.

(Now, Stackoverflow promotes ‘flairs’ for different reasons, I think; for one, it’s a highly effective way to spread the word and drive in traffic through a hub-and-spoke model. But I think the implications for external recognition within the Enterprise community are far more profound.)

What’s your Enterprise doing to embrace the lessons from the Web’s top successes?

[1] – Finding Opportunity for Success in the Failure of Others
http://www.readwriteweb.com/start/2010/10/finding-opportunity-for-succes-in-failure-of-others.php

[2] – What is reputation?
http://stackoverflow.com/faq

Measuring the ROI of Enterprise Search – Making the Case with Numbers

noreply@blogger.com (Nariman Haghighi) — Tue, 02 Nov 2010 02:58:00 +0000

Far too often, the Enterprise Search experience is woefully inadequate and falls noticeably short of the carefully tuned and nurtured experience that internet users have come to expect, courtesy of the heated contest between Google, Bing and Yahoo. One factor that’s not helping to close this innovation lag is the complexity involved in building the clean-cut, formal ROI case that often drives project charters within the Enterprise.

NOTE: recognizing that “Enterprise Search” can include a vast spectrum of content sources and content types, each with unique characteristics and limitations, we will focus the discussion here on web-centric content that’s typically served through standard HTML, in both internal and external scenarios.

So how exactly do you build the business case for overhauling your search implementation? A leading Canadian IT consulting firm, NLC, recently tackled the question head on in their ‘Building the Business Case for Enterprise Search’ whitepaper [1] last week. While maintaining that a formal analysis ‘misses’ the real value of Enterprise Search, the paper does go on to provide a formal approach based on the IDC report, The Hidden Costs of Information Work (May 2009).

The proposed formula boils down to measuring:

Search Cost Today = # of knowledge workers X avg. salary (incl. benefits) X % of time spent searching
Potential Annual Benefit = Search Cost Today X The Search Lift Ratio (%)

Once deployed, estimates around ‘The Search Lift Ratio’ can easily be quantified through a combination of A/B testing, measurable and correlated call-to-actions (phone numbers present a challenge here, in some cases), ‘accurate’ time-on-page measurements [5] and traditional survey/feedback polls [4].

While the guidance is sound, it is largely based on estimates and, like similar reports before it [6], focuses primarily on the productivity gain; it does, however, offer a compelling, low-cost approach to building the case without relying on metrics that may be available in other areas.

The “saves you money” argument, by all accounts, is wildly overshadowed by the broader “makes you money” advantages [2], yes; but, recognizing that corporate culture thrives on the the power of hard numbers, can we do better upfront in the ‘reasonably measurable’ areas by relying on other metrics?

Assuming that the business has realized the value of matured analytics, even around legacy search implementations, can we explore these to make a stronger case?

While some of these are domain-specific, here are other areas that may yield useful ROI nuggets:

Reduced Database Volume => License/Hardware Savings: for high-volume applications that rely on a traditional RDBMS alone for ad-hoc queries or search-based functionality, a well-designed search implementation can significantly reduce the load on those database servers, resulting in both hardware and licensing savings. Through standard profiling tools, the volume of traffic that stands be offloaded to a search infrastructure can be quantified beforehand. Similar hardware/licensing savings (and training/staffing can also be realized in consolidating a fragmented search environment.
Measuring Publication Reach => Minimizing Sunk Costs: in scenarios where content was specifically authored and published for an intended audience, you should be able to measure the ‘publication cost’ and compare this to the ‘reach’ of the content (again, based on call-to-actions or accurate ‘time on page’ measurements). In scenarios where the reach is below a certain threshold, there is a clear and measureable ‘sunk cost’ that can be avoided in the future.
Reducing Call Center Volume => Headcount Reductions: assuming that source of call-center calls (both internal and external) are clearly classified (e.g. “I can’t find any information on…”), it should be possible to attribute a portion of call-center volume (by extension, head counts) to inadequate search implementations.

Can you think of others that affect your organization?

References:

[1] – Building the Business Case for Enterprise Search
http://www.nonlinearcreations.com/whitepaper/index.asp?id=23

[2] – In Search of ROI | Search Done Right
http://searchdoneright.com/2008/08/in-search-of-roi/

[3] – How search can add real ROI to eCommerce websites
http://googleenterprise.blogspot.com/2010/01/how-search-can-add-real-roi-to.html

[4] – KISS Insights - It’s so quick to respond that many of your customers will!
http://www.kissinsights.com/tour

[5] – The Problem With Measuring Time Spent on Page
http://blog.webanalyticsdemystified.com/weblog/2007/07/on-netratings-and-time-spent-on-site.html
http://www.marketingpilgrim.com/2007/07/the-problem-with-measuring-time-spent-on-a-web-site.html

[6] – Return on Information: adding to your ROI with Google Enterprise Search
http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en//enterprise/pdf/internal_search_roi.pdf

How to Remove the Evercookie?

noreply@blogger.com (Nariman Haghighi) — Thu, 23 Sep 2010 20:41:00 +0000

One of my colleagues just sent this out – it’s truly amazing [1].

Check out the Web History link that uses a CSS trick to reveal your history (all on the latest FF).

From the author himself [2]:

evercookie is a javascript API available that produces extremely persistent cookies in a browser. Its goal is to identify a client even after they've removed standard cookies, Flash cookies (Local Shared Objects or LSOs), and others.

evercookie accomplishes this by storing the cookie data in several types of storage mechanisms that are available on the local browser. Additionally, if evercookie has found the user has removed any of the types of cookies in question, it recreates them using each mechanism available.

Specifically, when creating a new cookie, it uses the following storage mechanisms when available:

- Standard HTTP Cookies
- Local Shared Objects (Flash Cookies)
- Storing cookies in RGB values of auto-generated, force-cached
PNGs using HTML5 Canvas tag to read pixels (cookies) back out
- Storing cookies in and reading out Web History
- Storing cookies in HTTP ETags
- Internet Explorer userData storage
- HTML5 Session Storage
- HTML5 Local Storage
- HTML5 Global Storage

Art is definitely in the eyes of the beholder!

This is a case of layering several techniques (some known, others new) to create an approach that requires some serious thought to defeat. Much like the MS vulnerability disclosed this week, I think this is more about bringing information to light to promote discussion and progress. As one comment put it, “the bad guys don’t make open source announcements, they keep the code to themselves.”)

[1] - Researcher Claims 'Evercookie' Can't Be Removed
http://threatpost.com/en_us/blogs/frankencookie-developer-builds-bulletproof-web-tracking-tool-092210

[2] - Evercookie -- never forget.
http://samy.pl/evercookie/

[3] - Evercookie: A cookie that undeletes itself from 8 different storages
http://news.ycombinator.com/item?id=1714446

ASP.NET Vulnerability Workaround Not Enough

noreply@blogger.com (Nariman Haghighi) — Tue, 21 Sep 2010 13:08:00 +0000

Microsoft keeps calling this a ‘workaround’ but it’s mitigation at best – sure, for the majority of applications it may be all that’s necessary, but it falls short of cross-the-board prevention. It’s not clear why they’re not more open about the cases that aren’t covered by the suggested workaround but it doesn’t take more than simple reasoning to reveal them. If you’re really concerned about the security of your application, keep reading.

Let’s go back to basics and look at where this issue stems from – all you need to do is differentiate between these three cases…

When the application is passed an encrypted value, it responds in one of three ways:

When a valid ciphertext is received (one that is properly padded and contains valid data) the application responds normally (200 OK)
When an invalid ciphertext is received (one that, when decrypted, does not end with valid padding) the application throws a cryptographic exception (500 Internal Server Error)
When a valid ciphertext is received (one that is properly padded) but decrypts to an invalid value, the application displays a custom error message (200 OK)

And depending on how your application uses encryption, you are still able to do this even with a constant error page – here’s one blogger’s explanation (misses the point, IMO):

How about Microsoft’s workaround?

Well, while the workaround contains a really valuable information, relevant for every system (as for not disclosing the real error), and it will prevent the automated tool released by the researchers to hack your system, it will, by far, NOT protect you from a potential attack!

How so? The workaround assumes that the potential attacker will look for an HTTP error response status (500), or for an error page containing a specific exception message. However, it is enough for attacker to recognize an abnormal, or just different system behavior on certain requests.

Let’s get back to our ASP.NET system that stores an encrypted sensitive information in a cookie. Each request, the system will probably decrypt this information and use it. In case the ciphertext in a cookie is invalid, an exception will be thrown, and the system may act according to one of the following scenarios:

· Return a 500 error response - very user unfriendly!

· Return a default ASP.NET YSOD exception page - extremely bad in production environment!

· Return a page stating only the exception’s message - also very bad!

· Return a constant page, stating there was an error, without providing details– a good practice, this is actually the Microsoft’s workaround

· “Swallow” the exception, and behave like the cookie does not exist. The response may be a redirect to another pager, or just a a slightly changed HTML (instead of user’s name, a “login” link) – This is the way ASP.NET Forms Authentication works.

Note that every one of the possible responses is different from the normal one. Even the last scenario I’ve described above, as clean as it is, still returns a distinctively different response. Therefore, an attacker can take advantage of it, and write a simple script that infers this abnormal behavior to an Invalid Oracle’s answer. It is that simple!

That’s mostly write, except that it’s not sufficient to detect any error condition – you need to be able to discern between two specific error conditions (ciphertext not correctly padded vs. ciphertext is correctly padded but decrypts to an invalid value).

If visitors to your site can do this, you’re still vulnerable. Here’s a scenario: you don’t rely on Forms Authentication, but instead have a custom authentication scheme that relies on encrypted cookies. If an invalid cookie is submitted, you treat the user as anonymous; but if an exception is thrown during decryption, Application_Error kicks in. This scenario, not entirely uncommon, still allows the attacker to replay the request and differentiate between the 3 possible replies.

How does your application respond in these scenarios and how confident are you of the answer?

ASP.NET Padding Oracle Exploit Tool Demo

noreply@blogger.com (Nariman Haghighi) — Mon, 20 Sep 2010 14:40:00 +0000

Update #2: Another good post echoing the “MITIGATE” theme (no signs of “PREVENTION” yet.) And some clarification from SO on how machineKey leaves other files (*.config, /bin/*) vulnerable:

“In ASP.NET 3.5 Service Pack 1 and ASP.NET 4.0 there is a feature that is used to serve files from the application. This feature is normally protected by the machine key. However, if the machine key is compromised then this feature is compromised. This goes directly to ASP.NET and not IIS so IIS's security settings do not apply. Once this feature is compromised then the attacker can download files from your application - including web.config file, which often contains passwords.”

This vector was first published in 2002 (8 years ago) and it’s not unique to .NET either – aspects of JSF, Rails, Django all seem to be affected (is the theme really as simple as encrypted data on the client?).

Update #1: Rinat Abdullin and Vlad Azarkhin offer some good details [4] [5] on this (the only blogs among my subscriptions to even mention the issue so far). Apparently, the difference in response times (among other subtle differences) still allows one to separate errors from accepted responses [6] – though this would require more of a site-specific attack. And the ability to request *.config files might be a secondary exploit for CMS products like DNN or Sitecore that expose the File System through authoring interfaces – though others are suggesting that it can be obtained through WebResource.axd (which uses some kind of key to authenticate incoming request for arbitrary resources).

For a low-level (detailed) explanation of the issue, see Brian Holyfield’s post [6].

Speechless.

Here’s a demo of POET [1] on DNN [2] – still unclear how this can be used to actually request *.config files though (potentially more serious, IMO):

The suggested fix [3], part of standard deployment best-practices, is to mask underlying HTTP error codes:

<customErrors mode="On" defaultRedirect="~/error.html" />

That’s the guidance from Microsoft, anyway; the group that released this video claims that “the setting of CustomErrors is _irrelevant_”.

Lots of confusion over this one, let’s hope it gets cleared up ASAP – stay tuned.

[1] – POET: the tool at the heart of the controversy:
http://netifera.com/research/

[2] – DNN Demo:
http://threatpost.com/en_us/blogs/demo-aspnet-padding-oracle-attack-091710

[3] – Guidance from Microsoft:
http://weblogs.asp.net/scottgu/archive/2010/09/18/important-asp-net-security-vulnerability.aspx

[4] – Details from Rinat:
http://abdullin.com/journal/2010/9/19/details-about-aspnet-security-vulnerability.html

[5] – “Padding Oracle” ASP.NET Vulnerability Explanation
http://blogs.microsoft.co.il/blogs/linqed/archive/2010/09/19/padding-oracle-asp-net-vulnerability-explanation.aspx

[6] – Automated Padding Oracle Attacks with PadBuster
http://www.gdssecurity.com/l/b/2010/09/14/automated-padding-oracle-attacks-with-padbuster/

Azure CDN vs. Amazon CloudFront/S3

noreply@blogger.com (Nariman Haghighi) — Fri, 06 Nov 2009 04:54:00 +0000

Earlier this summer we took a deep look at the CDN market, taking into account both established players (Limelight, Akamai, Level-3) and the emerging pay-as-you-go contenders (Amazon’s Cloud Front, Rackspace’s Cloud Files, SimpeCDN and the like). Today, ~1 week after Amazon’s Relational Database Service was announced, Microsoft responded with its own introduction of a new CDN that will offer 18 edge locations throughout the world [1] :

“Windows Azure CDN has 18 locations globally (United States, Europe, Asia, Australia and South America) and continues to expand. Windows Azure CDN caches your Windows Azure blobs at strategically placed locations to provide maximum bandwidth for delivering your content to users. You can enable CDN delivery for any storage account via the Windows Azure Developer Portal. The CDN provides edge delivery only to blobs that are in public blob containers, which are available for anonymous access.”

Note that the latest addition to the Azure family is in CTP release only. All we know is that, for the Windows Azure platform itself, PDC ‘09 (to be held later this month) is expected to announce new features and will be followed by an official launch in January and first billing cycle in February; it’s very likely that the CDN will be available along similar timelines as well.

Like CloudFront, Microsoft’s CDN doesn’t solve the HTTPS issue in its first release. In terms of pricing, if Windows Azure Platform Pricing is any indication, you can expect to pay ~0.17/Gb for each targeted zone.

As a refresher, Amazon uses 14 edge locations in major markets throughout worldwide: 8 in the United States (Ashburn, VA; Dallas/Fort Worth, TX; Los Angeles, CA; Miami, FL; Newark, NJ; Palo Alto, CA; Seattle, WA; St. Louis, MO), 4 in Europe (Amsterdam; Dublin; Frankfurt; London) and 2 in Asia (Hong Kong, Tokyo).

The North American edge locations are mapped below:

On the issue of TTLs, it’s still unclear whether the Azure CDN will support shorter TTLs required for niche applications:

“The TTL specifies that the blob should be cached for that amount of time in the CDN until it is refreshed by the Blob service. The CDN attempts to refresh the blob from Windows Azure Blob service only once the TTL has elapsed. The default TTL is 72 hours. At PDC 2009, we will allow you to specify the standard HTTP Cache-Control header for your Windows Azure blobs. If this value is specified for a blob, then the TTL period will be set to the value specified in Cache-Control header.”

And lastly, tools you need to get started with either service are the Azure Storage Explorer, the Amazon S3 Firefox Organizer (0.4.8) or CloudBerry Explorer (for S3 and Azure Blob Storage) – just note that, as Microsoft plays catch up here, the Azure tools don’t yet expose the same richness of features around CDN integration.

References:

[1] – Introducing the Windows Azure Content Delivery Network http://blogs.msdn.com/windowsazure/archive/2009/11/05/introducing-the-windows-azure-content-delivery-network.aspx

[2] – Using the New Windows Azure CDN with a Custom Domain http://blog.smarx.com/posts/using-the-new-windows-azure-cdn-with-a-custom-domain

LinkedIn vs. StackOverflow & The Shifting Landscape for Programming Careers

noreply@blogger.com (Nariman Haghighi) — Tue, 03 Nov 2009 14:01:00 +0000

I’ve been curiously watching LinkedIn try to transform itself into an awkward Q&A-style forum. Subscribing to the Senior .NET Developers group recently, I was a bit surprised to see the type of tactical and specific implementation questions being posted.

This is a terrible pairing of technology, IMO. Sure, LinkedIn is great at building and exposing the professional web, but their Q&A offering is almost an after-thought with a feature set that’s at least 5 years dated.

Even worse, none of this dialogue is publically searchable!

We’ve written about the value of StackOverflow reputation rankings as an emerging compliment to traditional resumes and awards. Yes, the current scoring system leaves much to be desired: users are more incentivized to create ‘popular’ content that drives participation than accurate, factual content that doesn’t.

The score itself is less telling than a closer examination of an individual’s responses. Nonetheless, it is best-of-breed and it’s no surprise that they announced a new career site today specifically intended to link hands-on Q&A activity with public CVs.

What is careers.stackoverflow.com? It's a few things:

a completely free, public CV hosting service for programmers, to share the cool stuff you've coded and created with the world.
a way to explicitly link your Stack Overflow profile with your CV, to provide concrete examples of your communication skills and individual expertise to anyone who is interested.
a better way to connect great programmers with the best programming jobs, for those who opt into the small annual listing fee.

Prior to the announcement, bloggers were already starting to sport “My answers @StackOverflow” feeds alongside their content, courtesy of the user-specific RSS feed exposed by StackOverflow and tools like Yahoo Pipes – the new careers site formalizes this intention in a much cleaner way.

Joel had a great post this week on finding out what your company is all about and the theme of helping your users become awesome that really resonated with me – they’re certainly living this out loud with this launch:

If you love to code, too, I encourage you to create your own Stack Overflow CV. Keep it private, or make it public via the URL of your choice -- it's completely free either way. If you think you might be actively looking for a job in the next 3 years, take advantage of our outrageously low promotional pricing of $29 for a 3 year filing. That way, at any point in those 3 years, you can flip a switch and become visible to hiring managers. Or not. It's totally up to you.

It’s refreshing to watch these guys in action and I can’t wait to see how it evolves!

CompilationMode=Never: What does it actually do?

noreply@blogger.com (Nariman Haghighi) — Mon, 02 Nov 2009 00:20:00 +0000

I wanted to take CompilationMode=Never for a test-drive on an existing application that’s plagued by too many control/page assemblies – the goal was to see if this would help improve scalability (by reducing unrecoverable memory consumed by assemblies) and to observe its impact on a high and fluctuating ‘% time in JIT’. It turns out that it’s not as easy as throwing it into web.config.

Yes, you can apply this en masse to an existing website but doing so on pre-existing sites could generate:

The attribute 'codefile' is not allowed in this page.
The attribute 'autoeventwireup' is not allowed in this page.

Why would CompilationMode interfere with AutoEventWireup? In automatically wiring up events, it looks like the framework looks for suitable signatures at runtime using reflection [1] – crazy, I know; and not having assemblies would presumably make this impossible to do. Scott Allen has a couple of good writes-ups on this and you can have a look for yourself in the auto-generated assemblies.

Can you think of any reasons why the events wouldn’t be wired up at compile time?

I tried to debug the framework to see this in action but threw in the towel after a few hours of fighting the symbol server – hours on this is pretty ridiculous, I know, and in complete defiance of Oscar’s #1 rule: stops are in, emotions are out! It was just a bit shocking to see how brittle the entire setup around source server is and how many continue to struggle with it [2] – anyone out there actually using it successfully for mscorlib and System.*.dlls? Have you you applied any service packs? Would love to hear from you. With VS2008 SP1 (9.0.307279.1), Vista SP2, the latest source code component (Dotnetfx_4016_VistaSP2), and following every instruction to a tee, I can step into most assemblies (e.g. System.Web, 2.0.50727.4016) but can’t step into mscorlib (2.0.50727.4200) which contains the reflection calls in question – for reference, the symbols for mscorlib are downloaded from /download/symbols/mscorlib.pdb/4D0B2695F5144B4D8F24004284FE26191/mscorlib.pd_.

So once you’ve manually resolved #2, code-behind files must also be pushed out to a fully-qualified class that can be reference by the Inherits attribute alone.

After that, it works as expected and you see the runtime behaviour of CompilationMode=Never:

   1: // IWebObjectFactory.CreateInstance

   2: public virtual object CreateInstance() {

3:

   4:     // Create the object that the aspx/ascx 'inherits' from

   5:     TemplateControl templateControl = (TemplateControl) HttpRuntime.FastCreatePublicInstance(_baseType);

6:

   7:     // Set the virtual path and TemplateSourceDirectory in the control

   8:     templateControl.TemplateControlVirtualPath = VirtualPath;

   9:     templateControl.TemplateControlVirtualDirectory = VirtualPath.Parent;

10:

  11:     // Give the TemplateControl a pointer to us, so it can call us back during FrameworkInitialize

  12:     templateControl.SetNoCompileBuildResult(this);

13:

  14:     return templateControl;

  15: }

What’s a little surprising, if I’m reading this right, is it seems to generate the type on demand every time without any caching of the instance for re-use across different requests:

   1: /*

   2:  * Faster implementation of CreatePublicInstance.  It generates bits of IL

   3:  * on the fly to achieve the improve performance.  this should only be used

   4:  * in cases where the number of different types to be created is well bounded.

   5:  * Otherwise, we would create too much IL, which can bloat the process.

   6:  */

   7: internal static Object FastCreatePublicInstance(Type type) {

8:

   9:     // Only use the factory logic if the assembly is in the GAC, to avoid getting

  10:     // assembly conflicts (VSWhidbey 405086)

  11:     if (!type.Assembly.GlobalAssemblyCache) {

  12:         return CreatePublicInstance(type);

  13:     }

14:

  15:     // Create the factory generator on demand

  16:     if (!s_initializedFactory) {

17:

  18:         // Devdiv 90810 - Synchronize to avoid race condition

  19:         lock (s_factoryLock) {

  20:             if (!s_initializedFactory) {

  21:                 s_factoryGenerator = new FactoryGenerator();

22:

  23:                 // Create the factory cache

  24:                 s_factoryCache = Hashtable.Synchronized(new Hashtable());

25:

  26:                 s_initializedFactory = true;

  27:             }

  28:         }

  29:     }

30:

  31:     // First, check if it's cached

  32:     IWebObjectFactory factory = (IWebObjectFactory)s_factoryCache[type];

33:

  34:     if (factory == null) {

35:

  36:         Debug.Trace("FastCreatePublicInstance", "Creating generator for type " + type.FullName);

37:

  38:         // Create the object factory

  39:         factory = s_factoryGenerator.CreateFactory(type);

40:

  41:         // Cache the factory

  42:         s_factoryCache[type] = factory;

  43:     }

44:

  45:     return factory.CreateInstance();

  46: }

Anyway, if SharePoint uses it successfully, I’m sure it can work for your CMS/cutting-edge-ASP.NET-stuff too.

Performance benchmarks though will have to wait another day.

References:

[1] – AutoEventWireup & Reflection
http://stackoverflow.com/questions/275965/asp-net-mvc-autoeventwireup-required/276385#276385
http://odetocode.com/articles/406.aspx

[2] – Troubleshooting Framework Debugging
http://stackoverflow.com/questions/1095202/why-cant-i-step-into-this-line
http://social.msdn.microsoft.com/Forums/en-US/refsourceserver/thread/1b74f60c-e961-425c-a38e-362406dd4cfe
http://social.msdn.microsoft.com/Forums/en-US/vsdebug/thread/a8441b7b-017b-4094-8788-6005aa8e69a3
http://social.msdn.microsoft.com/Forums/en/refsourceserver/thread/b22e044c-be8f-4650-98d6-b426193b7b2c
http://social.msdn.microsoft.com/Forums/en-US/refsourceserver/thread/e20ad5f3-3071-4ff6-9f2b-6f4ec22661b8
http://social.msdn.microsoft.com/Forums/en-US/refsourceserver/thread/ceb17913-a983-47a8-b15e-655a65c5f001/

[3] – Microsoft Reference Source Server
http://referencesource.microsoft.com/netframework.aspx
http://referencesource.microsoft.com/serversetup.aspx

[4] – KB944899 (This hotfix addresses a performance problem when stepping through Source Code downloaded via a Microsoft Reference Source Server)
https://connect.microsoft.com/VisualStudio/Downloads/DownloadDetails.aspx?DownloadID=10443&wa=wsignin1.0

The Silo Effect and Other Productivity Killers – A Practical Guide for IT

noreply@blogger.com (Nariman Haghighi) — Wed, 28 Oct 2009 01:35:00 +0000

If we recognize that the greatest innovation comes from working across silos, then how do we reward and incentive this pattern and are companies actively doing this? Some thoughts this weekend triggered readings into the “Silo Effect” [1] and its impact on the pace of progress. In the SaaS world, managing close collaboration between Development, Q&A and Operations is no easy feat, especially for a growing platform. I have to remind myself regularly that a handful of people really can’t run it all. Intuitively, you look for the simplicity of a well-integrated team with common skills and experiences; in practise, teams are highly specialized with varied focuses.

The first stop was an excellent read courtesy of John Simpson and Eric Winquist (of Jama Software [6], Portland, Oregon) in their paper, entitled "Eliminate the Top 3 Productivity Killers and Build Great Products in Half the Time” [5] – some takeaways are highlighted below:

Productivity Killer #1 – The Great Scavenger Hunt is Costly
Information is growing over 66% every year and is constantly changing. The good news is we now have the opportunity to know more about our customers like never before. The bad news is we’re inundated with information – some of it valuable, much of it noise. Where do you store and organize the relevant product information? Do you have the right intelligence captured to make the right decisions and take the right actions?

It’s estimated that employees at U.S. companies waste over 5 billion unproductive hours annually just looking for information2. At $35 per hour for an average knowledge worker, that’s a $175 billion problem in the U.S. alone. As an executive friend used to say, “That’s no pocket change, that’s adult money.”

Did You Know?

Employees spend 25% of their time just searching for information
Employees spend 20 minutes per day recreating information that already exists
42% of employees accidentally use the wrong information at least once per week
The average ramp up time is 45 days for a new employee, as high as 9 months for highly skilled jobs
The productivity loss of IT employee turnover can last from 3- 12 months

There’s Much Hype but the Social Trend is Real
If you’re like us, you might be tired of reading about “the revolution of social networking” – as if life or business wasn’t social before Twitter, Facebook or other Web applications existed. However you feel about this trend, there’s evidence to show the collaborative movement is real and is gaining momentum in the workplace. IBM estimates that within six years, workers will collaborate 80% of their time. You can’t open a prominent business or technology magazine without reading about open innovation and the impact that collaboration is having on business processes. With nearly $1 trillion being invested in R&D worldwide annually, you can understand why it’s a popular issue.

It’s a great read with emphasis on: a) how to better classify information to facilitate cross-silo collaboration, and b) how to cater to more natural forms of day-to-day dialogue. I must say, the 25% of time spent on search really resonated with me – what a powerful stat! I’m relentlessly “negotiating” with reluctant administrators on this dated restriction. I live in email and maliciously hold on to conversations, as I’m sure others do; the ability to instantly query across last decade’s worth of conversations is priceless. Trying to stitch together threads in separate mailboxes with crucial messages lost in some error-prone archiving scheme is, on the other hand, quite expensive. If never ceases to amaze me how these limitations can still be defended? Unlimited email (10-20-30GB, practically speaking), with foolproof-web-based-search-that-works, should be a minimum bar in any organization – what exactly are we waiting for??

(On the paper itself, yes, many of these data points are used in all sorts of collaboration solutions, which ultimately weakens the comparative evaluation for the sake of establishing a business case. I see this all the time and think: yes, it’s a problem, without question, but why is your solution better than X,Y,Z? The StrangeLoop team was a perfect example: yes, we know latency is an issue, but why should we address it with this appliance? I would imagine that would-be customers have identified and understand the problem and that this emphasis is lost on them. Well, here’s a scenario where justifying the business case got my attention. Without question, these are the 3 productivity killers – but from a tool perspective, whether Contour is the answer or whether a TFS/SharePoint paring is an equally viable approach is another debate all together.)

Tearing Down Business Silos[2] was the next stop:

The foundation of a successful organization is an entire team focused on common goals. Silos erode this foundation. Being aware of the fundamental human behaviors that lead to silos and taking steps to overcome them offers fantastic benefits - including more relevant products and services, higher productivity, better use of resources, and more effective and engaged personnel.

So what can we do? Carol has some fantastic suggestions:

#1 –Reward Collaboration: define collaborative performance objectives, make it part of the preview, promote based on this and spread the story throughout the organization!
#2 – Focus on Innovation: triggered by a cross-pollination of ideas; bring together people with diverse perspectives and expertise when setting the agenda.
#3 – Communicate Transparently: enough said; it’s dead-easy and so often missed.
#4 – Encourage Networks: use social networking tools and create visual models – I haven’t seen this formalized well.
#5 – Mix it Up: rotate people, invite other managers to your meetings, make them a member of the group – also very rare.
#6 – Focus on the Customers: stay close to the end-user! Share marketplace information, customer-feedback, the good and the bad. In practise, customer opinions pass through layer upon layer before reaching the Development team.
#6 – Get Personal: collaborative relationships thrive in an environment of personal trust. Most companies do a decent job of this, as they should.

Do these things well and you will avoid scenarios like the one faced by the CIO of a mid-size company (125 reports) who wrote into a recent issue of InfoWorld [7]:

“From the way everyone behaves you'd think my head of application development was Bill Gates and the head of Operations was Larry Ellison. There is no trust at all, and they appear to be out to undermine each other more than they're out to be successful at their own responsibilities.

Worse, the attitude is contagious. The people reporting to them act just like they do. There's no trust, no ability to collaborate ... nothing except a lot of blaming, which seems to have become our core competency.

I've tried lecturing, coaching, berating and arguing, and nothing seems to work.
They really are two talented people and I don't want to get rid of either of them. On the other hand, the situation isn't sustainable. Any thoughts about what else I might try before firing one or both of them?”

InfoWorld’s Bob Lewis responds:

“Before you do anything else, look at the situation you've put them in. Usually, when organizations turn into silos, it's because goals, and any compensation tied to those goals, reinforces silo-oriented behavior.

My guess is that you'll find a lot of win/lose trade-offs. For example, if the budgeting process starts with a fixed pie and when one of them gets something it comes out of the other's budget, it's win/lose, which reinforces organizational rivalry. Result: Silos.

Or, maybe Applications is counted as successful when projects finish on time no matter what, while Operations is counted as successful based on system stability and performance. Which means Applications has an incentive to skimp on software quality assurance, while Operations has an incentive to keep all new code out of production as long as possible. Result: Silos.

While you're looking at structural sources of moat-building, you should also figure out if any of their responsibilities require collaboration. Very likely, they don't. Usually, organizational design starts by trying to draw clear boundaries between departments, to clarify who is responsible for what. The unintended consequence is that because managers have no reason to collaborate, they neither build the habit of doing so, nor see any reason to start.”

“People do what they do for reasons. If you want them to do something different, take away the reasons they're doing what they're doing, and give them reasons to do something different.”

References:

[1] – The Silo Effect (Seven Key Obstacles to Change):
http://www.edwardmorler.com/seven-key-obstacles-to-change.html

[2] – Tearing Down Business Silos
http://www.sideroad.com/Management/business-silos.html

[3] – Breaking Organizational Silos using Enterprise Architecture
http://it.toolbox.com/blogs/ea-for-cio/breaking-organizational-silos-using-enterprise-architecture-30764

[4] – Jama Software (Contour)
http://www.jamasoftware.com/contour/

[5] – Eliminate the Top 3 Productivity Killers and Build Great Products in Half the Time
http://www.jamasoftware.com/media/documents/Central_Hub_Product_Intelligence_Jama.pdf

[6] – Requirements and Bridging the Silos
http://www.requirementsnetwork.com/system/files/Requirements%20and%20Bridging%20the%20Silos%20(Part%202%20of%203).pdf

[7] – Stuck in Silos
http://www.infoworld.com/t/career-advice/stuck-in-silos-707

High CPU in ASP.NET - Finding Large DataTables with the Public SOS

noreply@blogger.com (Nariman Haghighi) — Thu, 15 Oct 2009 23:35:00 +0000

This is yet another one strictly for the dump analysis crowd – what can I say, it’s been a long week! Last episode, we highlighted the lack of support for many useful windbg commands. Well, chief among these was the DumpDataTables command that provides a nice summary of the largest tables and their column counts, typically a nice starting point for high-memory/high-CPU issues in traditional application. So how do we fend for ourselves and reproduce this with the SOS that we have?

Some very helpful support engineers that I’ve been working with over the last few weeks offered the following as a potential substitute:

   1: $$ Run as $><C:\Extensions\DumpLargeTables.txt

2:

   3: .printf "\n===== Printing Large datatables with 1000+ rows with their column names =====\n\n"

4:

   5: .foreach(m_table {!dumpheap -type System.Data.DataTable -short})

   6: {

   7:     .foreach (m_Count {!dumpfield -field nextRowID m_table})

   8:     {

   9:          $$ Change the value of row count here

  10:         .if( ${m_Count} > 1000)

  11:         {

  12:             .printf "Table Name : "; .printf /D "<?dml?><exec cmd=\"!do ${m_table}\">${m_table}</exec> - "

  13:             !dumpfield -field tableName -string ${m_table}

  14:             .printf " Row Count : ${m_Count}"

  15:             .printf "\t|---Columns\n";

16:

  17:             .foreach(m_col {!dumpheap -type System.Data.DataColumn -short})

  18:             {

  19:                 r @$t2 = poi(${m_col}+0x3c)

  20:                 .if(@$t2 == ${m_table})

  21:                 {

  22:                     .printf /D "\t\t|-- <?dml?><exec cmd=\"!do ${m_col}\">${m_col}</exec> - ";

  23:                     !dumpfield -field _columnName -string m_col

  24:                 }

  25:             }

  26:             .printf "\n";

  27:         }

  28:     }

  29: }

Unfortunately, this too relies on a command (!dumpfield) that is also not available in the public SOS.

Through some collaborative trial-and-error, we then came up with the following work-around for accessing the nextRowID field using its offset, in a manor supported by the public SOS:

   1: .foreach (DT {!dumpheap -mt 65242d0c -short}){.echo ${DT};?poi(${DT}+0xd0)}

…which gives you something to work with if you drop it in excel and sort it (original addresses removed):

   1: Evaluate expression: 14 = 0000000e

   2: Evaluate expression: 3 = 00000003

   3: Evaluate expression: 20 = 00000014

   4: Evaluate expression: 29 = 0000001d

   5: Evaluate expression: 11 = 0000000b

   6: Evaluate expression: 28 = 0000001c

   7: Evaluate expression: 1 = 00000001

   8: Evaluate expression: 1 = 00000001

To be version/Method-Table agnostic, I tried the following, which should be comparable to the one above:

   1: .foreach (DT {!dumpheap –type System.Data.DataTable -short}){?poi(${DT}+0xd0)}

…but I see some strange results with the latter, so find your method tables and work from there.

How to read this (we’re simply using the POI function to read the address + offset for nextRowId - d0):

       1: 0:026> !do 0x02c671cc 

       2: Name: System.Data.DataTable

       3: MethodTable: 65242d0c

       4: EEClass: 6515da18

       5: Size: 296(0x128) bytes

       6: GC Generation: 2

       7: (C:\WINDOWS\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)

       8: Fields:

       9:       MT    Field   Offset                 Type VT     Attr    Value Name

      10: 7a5d2a58  40009a3        4 ...ponentModel.ISite  0 instance 00000000 site

      11: 7a5ecd2c  40009a4        8 ....EventHandlerList  0 instance 00000000 events

      12: 7933061c  40009a2      1e4        System.Object  0   shared   static EventDisposed

      13: 65244194  40007a5        c  System.Data.DataSet  0 instance 00000000 dataSet

      14: 6524534c  40007a6       10 System.Data.DataView  0 instance 02c682d0 defaultView

      15: 79332c4c  40007a7       d0         System.Int32  1 instance        3 nextRowID

      16: 65245798  40007a8       14 ...DataRowCollection  0 instance 02c674d8 rowCollection

      17: 652456b8  40007a9       18 ...aColumnCollection  0 instance 02c67388 columnCollection

      18: 65245728  40007aa       1c ...straintCollection  0 instance 02c674a0 constraintCollection

      19: 79332c4c  40007ab       d4         System.Int32  1 instance        2 elementColumnCount

      20: 65244b08  40007ac       20 ...elationCollection  0 instance 02c6845c parentRelationsCollection

      21: 65244b08  40007ad       24 ...elationCollection  0 instance 02c683f8 childRelationsCollection

References:

[1] –
http://blogs.msdn.com/vijaysk/archive/2008/01/15/windbg-scripting-dump-data-column-names-from-a-table.aspx

[2] –
http://blogs.msdn.com/mvstanton/archive/2005/10/11/479861.aspx

[3] –
http://mcfunley.com/205/using-windbg-to-log-exceptions-part-2

The Value of Microsoft’s MVP Designation: Expert, Evangelist or Critic?

noreply@blogger.com (Nariman Haghighi) — Tue, 13 Oct 2009 01:11:00 +0000

Jon Skeet is something of a legend in the Microsoft Development Community – I first stumbled across his name on a C# thread on StackOverFlow, a popular forum for developers, where Jon holds the highest reputational ranking (with a healthy 20K-point lead over the #2 challenger). Something of a Chunk Norris like figure among developers, Mr. Skeet’s fans have gone as far as creating a AskJeeves-styled tribute in his honour [2]:

“It is hard to explain this website without first describing what is Stack Overflow and who is Jon Skeet. Firstly Stack Overflow is a fast-growing website featuring questions and answers on a wide range of topics in computer programming. It has already built up a large community and has an addictive quality that is largely down to its Reputation system.

Meanwhile Jon Skeet is a software engineer who currently works for Google. He has recently gained minor celebrity within the developer community by becoming a prolific contributor to StackOverflow. At the time of writing he has totalled 2,738 answers in the 5 months he has been registered. This works out at roughly 500 questions per month!

Of course being this prolific is astonishing in itself, however it wouldn’t be of much value if the answers were of a poor quality. That is what stands Jon Skeet apart, his answers have a consistent and almost reference quality. It was this that led me to think that it would be fun to create a website that farmed Stack Overflow of all the questions that Jon Skeet has answered and put them in an AskJeeves.com style website. If you want to know what Jon Skeet thinks about a topic, put it in AskJonSkeet.com and see.”

Microsoft MVP: Evangelists, Experts or Critics?

What’s more interesting though is the peripheral debate on the value of the MVP designation in the first place [1], [3], [4] and whether MVPs are Evangelists, Experts or Critics? Developers around the world have chimed in and the usual hyperbole around ‘trust’, ‘good vs. evil’, and ‘Microsoft vs. Google’ rhetoric is once again front stage.

Having worked with a number of MVPs in the SharePoint space, here’s my $0.02:

Politics have an undeniable role in a designation like this;
There are exceptionally talented MVPs and there are ‘how-to’ MVPs that mostly publish starter guides;
The MVP designation, in and of itself, is not a reflection of technical expertise;
Does it have sales value in consulting? Maybe.
Does it have personal marketability? Maybe – this is the incentive provided for enriching the ecosystem; but a popular community endorsement (blog with 10,000 views/month, for instance) would be more compelling, IMO.

To Jon, I doubt this makes one shred of difference in his day-to-day work; he will continue to be seen as a leader, without question.

If you were looking for a strong technical resource, and purely from a capabilities perspective, would you put more emphasis on a StackOverFlow reputation or an MVP designation? Would the answer change if you were a CTO of a medium/large enterprise? What about blog rankings and subscribers as a measure of community clout? Are more objective measurements possible?

Programming Jokes – As Good as it Gets!

On a lighter note, here’s a preview of what’s featured on AskJonSkeet.com; I haven’t material this good since UW’s very own weekly MathNews:

"Jon Skeet once hacked the FBI using an etch-a-sketch"
"Jon Skeet doesn't call a background worker, background workers call Jon Skeet."
"Jon Skeet can determine the next random number in a sequence."
"Superman wears Skeet pajamas to bed!"
"Jon Skeet does not sleep.. He waits."
"Jon Skeet can do pair programming with himself"
"Jon Skeet can believe it's not butter."
"Google is Jon Skeet behind a proxy."

References:

[1] – Google lobs coder's Microsoft badge into rubbish bin
http://www.theregister.co.uk/2009/10/02/google_microsoft_mvp/

[2] – Ask Jon Skeet!
http://askjonskeet.com/home/about

[3] – Microsoft "Most Valuable Professional" Award Banned at Google?
http://www.labnol.org/internet/microsoft-award-banned-at-google/10179/

[4] – Google Employee Reluctantly Gives up Microsoft MVP Status
http://www.techflash.com/seattle/2009/10/google_employee_reluctanctly_gives_up_microsoft_mvp_status.html

Blogger SEO Limitations: Why URL Length & Indexed Tags Matter

noreply@blogger.com (Nariman Haghighi) — Fri, 09 Oct 2009 03:17:00 +0000

Working with blogger over the last few months, 2 things have caused me a bit of unnecessary angst:

Its artificial limit on URL length (truncates URLs somewhere around 40 characters), and;
Its refusal to crawl /search paths for indexing post tags.

Even for an avid user of anything and everything Google - Google Search, Google Apps, Google Reader, Google AdWords, Google AdSense, Google Wave, GMail, Google Analytics, Google Maps, and who knows what else – it was enough to trigger thoughts of Wordpress.

These are serious limitations – and there are others, all unique to Blogger, as far as mainstream platforms go. Why should the Wordpress guys or the do-it-yourself guys, or worse, the have-to-host-it-on-my-home-server guys have an edge on the foremost SaaS player in town, from an on-page perspective at least?

I mean this is Google’s product after all, isn’t it? Could it be that these ‘optimizations’ are not officially endorsed?

You’ll find lots of people telling you that neither limitation matters that much – especially the URL length; some even paint all of ‘SEO’ with the micro-optimization brush, relegating it to the unoriginal who lack substance (Seth said this once, I believe, at the Business of Software conference; may have some truth to it, but for the little guy SEO’s still the make it or break it.)

But according to Google’s Matt Cutts, they do matter, and it’s simply a sliding scale beyond 4 or 5 keywords in the URL [7]:

“We have seen clients use keyword URLs that have 10 to 15 words strung together with hyphens; or blogs - we have seen them even longer there. A typical WordPress blog will use the title of the post as the post slug, unless you defined something different and you can just go on and on and on. Can you give any guidelines or recommendations in that regard?

Matt Cutts: Certainly. If you can make your title four- or five-words long - and it is pretty natural. If you have got a three, four or five words in your URL, that can be perfectly normal. As it gets a little longer, then it starts to look a little worse. Now, our algorithms typically will just weight those words less and just not give you as much credit.

The thing to be aware of is, ask yourself: "How does this look to a regular user?" - because if, at any time, somebody comes to your page or, maybe, a competitor does a search and finds 15 words all strung together like variants of the same word, then that does look like spam, and they often will send a spam report. Then somebody will go and check that out.

So, I would not make it a big habit of having tons and tons of words stuffed in there, because there are plenty of places on a page, where you can have relevant words and have them be helpful to users - and not have it come across as keyword stuffing.

Stephan Spencer: So, would something like 10 words be a bit much then?

Matt Cutts: It is a little abnormal. I know that when I hit something like that - even a blog post - with 10 words, I raise my eyebrows a little bit and, maybe, read with a little more skepticism. So, if just a regular savvy user has that sort of reaction, then you can imagine how that might look to some competitors and others.”

SEO Design Solutions also carries a good write-up of how post tags make a difference – note the example they’re example and test it out [1]:

These keywords are there because they were toggled by users typing in enough searches that a contextual occurrence now binds them under the same umbrella based on the profile of search queries executed.

So, if I wanted to reinforce a broad match topical range of semantics, I could easily implement keywords that were algorithmically related in real time (when looking for tags or additional keyword to include in a post) by using tags or developing related content for the main topic / silo to build into a virtual theme of topical posts.

The “theme” is the main topic and the silos are the supporting articles that reinforce relevance for all things related (based on the array of keyword modifiers). Toggling theme density implies targeting semantic clusters, which is essence is accomplished through the tags becoming links and adding weight to global on page relevance with ease.

The industry thrives off of this evidence-based consensus, turning a blind eye to official announcements from Google itself even. My own research confirmed both strategies make a significant difference so I tracked down Siobhan Quinn, a product manager for blogger, and shared the feedback. Both the JotSpot (Google Sites) and the Pyra Labs (Blogger) acquisitions feel like they haven’t (yet) been absorbed as first-class citizens.

In the end though, it’s not enough to abandon ship! I trust Google will level the playing feel and provide bloggers with a second-to-none publishing experience, free of charge. As for work-arounds (hacks), you’ll have to excuse me if a few ‘index’ pages slip into your feeds over the next few days; here are a few tips [2] that, in addition to the usual template title/meta-tag optimizations, endorse this technique as well:

I generally start with the content - I'll pick a somewhat specific topic to write about (in this case the new adsense email), and I'll write the post. Then I go back and figure out how to market it. Occasionally, if I don't feel a particular inclination to write about something, I do use the Google Suggest (now built into Google front page) to pick a topic to write about. Either way, I almost always use this functionality to refine my keywords when I am finished with the post. How?

Simply go to the front page of Google (not inside of a search, or gmail, etc.) and start typing - see the list of words that pops up? Those are popular search terms that start with what you've typed (note: exactly what is in the box will not appear in the results). Look for words higher in the list - meaning more often searched, but with a lower number of results - meaning less competition. The less letters you type in (the more you can backspace) and still see your "target keyphrase" high in the list, the higher the "value" of the keyword. This does not mean you will get tons of traffic, it just means you will have an easier time getting to the front page of Google.

… on why you’re out of luck with blogger tags [2]:

Remove label and place text links : The label pages in Blogger are automatically set nofollow by Google . So there is no SEO value in keeping the labels in home page . Alternately , you can categorize your posts and include them in single post that can be fixed in homepage . For eg ; Instead of placing the label Blogger tricks and tips , I have written a main post contains all the similarly tagged posts and placed as a text link in my homepage.

In practise, tag links on blogger sport a rel=’tag’ attribute (no sense in deleting this – some ‘related posts’ widgets leverage it); it’s Robots.txt that explicitly blocks /search paths. Interestingly, the blogger search interface accepts wildcards too (also blocked) – had the restricted path been the /search/ folder specifically, this would be accepted:

   1: /search-anything-here/label/some-label

Indispensible Tools:

Need help with your SEO strategy?

Drop me a line or leave a comment.

References:

[1] – SEO Design Solutions
http://www.seodesignsolutions.com/blog/seo-tips/seo-tips-using-related-search-to-optimize-cms-tags/
http://www.seodesignsolutions.com/blog/seo-basics/using-related-search-to-find-googles-most-search-keywords/

[2] – Blogger SEO Tips and Tricks
http://www.techknowl.com/2009/03/blogger-seo-tips-and-tricks.html
http://www.techknowl.com/2009/01/blogger-seo-on-page.html
http://webupd8.blogspot.com/2009/02/blogger-blogspotcom-seo-optimization.html
http://bloggerblackhatseo.blogspot.com/
http://www.moretechtips.net/2009/03/blogger-seo-tip-use-post-labels-as-meta.html
http://tweakandtune.blogspot.com/2009/03/blog-seo-tips.html

[3] – SEO Best Practices for URL Structure
http://www.searchenginejournal.com/seo-best-practices-for-url-structure/7216/

Migrating SharePoint Content to Google Sites via New API

noreply@blogger.com (Nariman Haghighi) — Sun, 04 Oct 2009 04:17:00 +0000

The latest in the “Battle for Enterprise Collaboration” comes courtesy of Barb Mosher of CMS Wire [1] and Adriaan Bloem of CMS Watch [2] in their reviews of the Sites API release that now enables migration of content out of SharePoint and into Google Sites. What both accounts fail to mention is that SharePoint SaaS has a long way to go to match the capabilities of on-premise SharePoint. Success in SaaS form, arguably more relevant than anything that’s happened in the server market over the last decade, will ultimately be predicated on PaaS (Windows Azure vs. Google App Engine) strengths and not the current feature sets around collaboration.

Vendor Lock-in Concerns

In addressing one of the chief concerns with nascent SaaS platforms like Sites, an engineering team at Google responsible for The Data Liberation Front has extended support to allow data the free flow of data in and out of their coloration platform [2]:

The Data Liberation Front is an engineering team at Google whose singular goal is to make it easier for users to move their data in and out of Google products. We do this because we believe that you should be able to export any data that you create in (or import into) a product. We help and consult other engineering teams within Google on how to "liberate" their products. This is our mission statement:

Users should be able to control the data they store in
any of Google's products. Our team's goal is to
make it easier for them to move data in and out.

(Note that the lack of API was one on the barriers cited with the Sites in our first review this summer.)

Meanwhile, Adriaan Bloem, of CMS Watch commented yesterday [3]:

And lastly, who uses Sites, anyway? After Google acquired JotSpot, it was closed for business for a year, and then Sites emerged: a stripped-down version of JotSpot. And after the initial hubbub, there was a roar of silence. @GoogleAtWork occasionally tweets about a corporate site done entirely in Sites, but those are always the sites of Google Enterprise partners. And they're the select few where I could see using Sites for an external site would make any sense (since they're selling it, of course.) Other than that, I read about enterprises switching to Apps -- and they get Sites in the deal, but while everyone uses the (g)mail, and many will use the docs (which have some nice collaborative features), few seem to venture into Sites.

To slightly offset all of this, yes, of course there are scenarios where Sites could be useful. These mainly revolve around small companies, with maybe up to 10 or 20 staff (so governance is less of an issue), who aren't in the business of running fileservers, mailservers, webservers, and social software -- Apps, with Sites, is a viable alternative. Running the infrastructure required for SharePoint (and going through the pains of implementing it to do useful things) would make little sense in that case. And for a lot of organizations, maybe especially the larger enterprises, Gmail is a very cost-effective replacement for maintaining in-house mail servers.

Google, however, hints at examples like "your sales team's Google Sites pages can update automatically when new leads are added to your CRM system"; but you'd have to develop the code to do so, first. And right now, there are too many reasons why you wouldn't really want to do that. Likewise, a migration from Notes to Gmail, I could understand. But from custom-built Domino applications to Sites... that's a completely different story.

Sites as a SharePoint killer? No, definitely not. This assassin has to go through ninja school first.

Google Sites isn’t ready for the enterprise spotlight yet, agreed; but I’m not sure the “reasons why you wouldn’t really want to do that” are clear to me. Comparing on-premise SharePoint to Sites is no apples-to-apples; Sites to SharePoint Online, on the other hand, is a little more realistic and a better question to ask is how and when the on-premise SharePoint that we’ve come to know and love will shine in SaaS form?

We still haven’t seen much on how future generations of non-dedicated SharePoint Online will support code-level customizations – all we know is that today’s version carries very limited support for functional extensions; the most you can do comes in the form of WebParts like Twitter Search that rely on existing HTTP interfaces and the built-in Data View and Data Form WebParts [4]:

(Whether even basic customizations like this would run in SharePoint Online needs to be validated.)

Fundamentally, customizations on either Sites or SharePoint online would look the same at this stage – and exactly why loosely coupled designs like this would be bad investment is unclear to me.

Both products are fairly premature for enterprise adoption; but there is arguably more momentum and a more active ecosystem around Sites at the moment [5] than there is around the wait-and-see SharePoint Online camp.

For a fair comparison of the two SaaS offerings, a detailed review is provided here.

Update (Oct 10, 2009):

Google has just announced a few upgrades to Sites:
Picasa Web Albums integration, site feeds, and page templates in Google Sites

References:

[1] – Google Goes After SharePoint with New Sites API
http://www.cmswire.com/cms/enterprise-cms/google-goes-after-sharepoint-with-new-sites-api-005620.php

[2] – The Data Liberation Front
http://www.dataliberation.org/

[3] – Google Sites with API: Still No SharePoint Killer
http://www.cmswatch.com/Trends/1702-Google-Sites-versus-SharePoint

[4] – Code-Free SharePoint Twitter Search Web Part
http://woodywindy.spaces.live.com/Blog/cns!773832677F575173!558.entry

[5] – Google Site Samples
http://www.steegle.com/websites
http://siteshelp.kccloudsolutions.com/
http://sites.google.com/site/wgpconsulting/Home/links-to-google-sites-examples
http://www.sacvolvoclub.org/google-sites#GSHL

Custom MS Build Task to tame System.Reflection.AmbiguousMatchException: Ambiguous match found.

noreply@blogger.com (Nariman Haghighi) — Thu, 01 Oct 2009 02:50:00 +0000

Update (Oct 3, 2009): Thanks to the reader that pointed out that the MS Build task was using a hard-coded path. The example has been corrected to use a parameterized path that relies on the MSBuildProjectDirectory [3] reserved property.

An innocent “mistake” [1], if you can even call it that, has been punishing developers worldwide for a good part of the last decade – can you spot it?

   1: <%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="WebApplication1._Default" %>

2:

   3: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

4:

   5: <html xmlns="http://www.w3.org/1999/xhtml" >

   6: <head runat="server">

   7:     <title></title>

   8: </head>

   9: <body>

  10:     <form id="form1" runat="server">

  11:     <div>

  12:         <asp:HiddenField runat="server" ID="RecordId" />

  13:     </div>

  14:     </form>

  15: </body>

  16: </html>

   1: using System;

2:

   3: namespace WebApplication1

   4: {

   5:     public partial class _Default : System.Web.UI.Page

   6:     {

   7:         private int recordId = 0;

8:

   9:         protected void Page_Load(object sender, EventArgs e)

  10:         {

  11:             Response.Write(recordId);

  12:         }

  13:     }

  14: }

What’s worse is that the exception thrown doesn’t lend any clues as to what the source of the ambiguity is – even this level of detail requires some custom logging:

System.Reflection.AmbiguousMatchException: Ambiguous match found.
   at System.RuntimeType.GetField(String name, BindingFlags bindingAttr)
   at System.Web.UI.Util.GetNonPrivateFieldType(Type classType, String fieldName)
   at System.Web.Compilation.BaseTemplateCodeDomTreeGenerator.BuildFieldDeclaration
ControlBuilder builder)

So what’s wrong with the code?

Two fields differ only in case. Yup – that’s it. RecordId is both a HiddenField and an integer. C# allows this – not a word is raised during compilation. Based on what I’ve seen and read (it’s widely covered), it only happens only when you pre-compile either a Website or Web application and navigate to the page in question.

Now imagine having a page with 20+ controls and trying to figure out where this is coming from – how’s the reflector supposed to help here?

How can we catch it at compile time?

It’s surprisingly easy, actually:

Until it gets fixed – and yes, it is a defect – here’s a custom task (MemberCaseTask) that you can add to your projects to catch it at build-time as opposed to already-in-staging-and-didn’t-think-to-navigate-to-each-view-after-precompilation-time.

Reference the custom Task in the project/solution script as follows:

   1: <UsingTask TaskName="MemberCaseTask"

   2:            AssemblyFile="C:\Users\admin\Documents\Visual Studio 2008\Projects\ClassLibrary1\ClassLibrary1\bin\Debug\ClassLibrary1.dll" />

   3:  <Target Name="AfterBuild">

   4:      <MemberCaseTask AssemblyPath="$(MSBuildProjectDirectory)\bin" />

   5:  </Target>

Build this file and update the AssemblyFile reference above (note that we copy the original definition of GetNonPrivateFieldType from System.Web.UI.Util) :

   1: using System;

   2: using System.Collections.Generic;

   3: using System.Linq;

   4: using System.Text;

   5: using System.Reflection;

   6: using System.IO;

   7: using Microsoft.Build.Framework;

   8: using Microsoft.Build.Utilities;

9:

  10: namespace ClassLibrary1

  11: {

  12:     public class MemberCaseTask: Task

  13:     {

  14:         [Required]

  15:         public string AssemblyPath { get; set; }

16:

  17:         public override bool Execute()

  18:         {

  19:             foreach (Assembly assembly in GetAllAssemblies())

  20:             {

  21:                 try

  22:                 {

  23:                     foreach (Type type in assembly.GetTypes())

  24:                         foreach (FieldInfo field in type.GetFields(BindingFlags.NonPublic | BindingFlags.Instance))

  25:                             try

  26:                             {

  27:                                 GetNonPrivateFieldType(type, field.Name);

  28:                             }

  29:                             catch (Exception)

  30:                             {

  31:                                 Log.LogError(string.Format("{0} has a field conflict on field {1}.", type.Name, field.Name));

  32:                                 return false;

  33:                             }

  34:                 }

  35:                 catch (System.Reflection.ReflectionTypeLoadException)

  36:                 { }

  37:             }

38:

  39:             return true;

  40:         }

41:

  42:         private IEnumerable<Assembly> GetAllAssemblies()

  43:         {

  44:             foreach (FileInfo file in new DirectoryInfo(AssemblyPath).GetFiles("*.dll", SearchOption.TopDirectoryOnly))

  45:             {

  46:                 Assembly assembly = AppDomain.CurrentDomain.Load(AssemblyName.GetAssemblyName(file.FullName));

  47:                 yield return assembly;

  48:             }

  49:         }

50:

  51:         static Type GetNonPrivateFieldType(Type classType, string fieldName)

  52:         {

  53:             FieldInfo field = classType.GetField(fieldName, BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance | BindingFlags.IgnoreCase);

  54:             if ((field != null) && !field.IsPrivate)

  55:             {

  56:                 return field.FieldType;

  57:             }

  58:             return null;

  59:         }

  60:     }

  61: }

References:

[1] - System.Reflection.AmbiguousMatchException: Ambiguous match found
http://dotnetdebug.net/2006/03/21/ambiguous-match-found-in-a-web-control-a-possible-bug/
http://weblogs.asp.net/pjohnson/archive/2006/08/11/Ambiguous-match-found.aspx
http://www.velocityreviews.com/forums/t153094-error-ambiguous-match-found.html

[2] - How to Write a Task:
http://msdn.microsoft.com/en-us/library/t9883dzc.aspx

[3] - MSBuild Reserved Properties:
http://msdn.microsoft.com/en-us/library/ms164309.aspx

Engineering Recruitment: A Data-Driven Approach to Candidate Assessment

noreply@blogger.com (Nariman Haghighi) — Thu, 24 Sep 2009 07:04:00 +0000

I’ve yet to meet anyone with recruitment responsibilities that isn’t frustrated with the state of the industry. Finding the right people, filtering through them intelligently and keeping them happy is a top priority anywhere you look – so why is it that we’re working with tools and techniques that haven’t changed in decades?

Just last month, I reviewed over 20 resumes for a developer/architect position and could hardly differentiate between them – from the self-described $40/hr to the $90/hr, they all looked the same on paper. And I’m not sure that interviewing each and every one of them would have left me any more convinced.

We’ve all heard “the best people aren’t looking for work” cliché a dozen times but still don’t have a sense for what the remedy is. Are AdWords the solution [6]? The guys at StackOverFlow.com seem to think so: take control of the eyes – where the best people spend their time – and sell targeted ads. (You can appreciate why they’d offer money-back-guarantees under this model.)

What’s everyone else doing?

Jim Holincheck’s blog on Human Capital Management (HCM) introduced me to a project at one of the most innovative recruiting organizations on the planet [1]:

I found Dr. Sullivan's post on ERE enlightening though…It is innovative because of the breadth of factors assessed (though again that is something that has been possible) and how they were leveraging the core competencies of Google in creating this assessment approach. Also, one should not look at it the context of just this one thing that Google is doing [but] in recruitment and talent management in general. It is part of a broader innovation strategy to ensure that it can bring in quality talent to support the business strategy. Is that not what we all want to do?

(Having interviewed at both Google and Microsoft in 2004, I’ve witnessed Google’s effort first-hand.)

Dr. Sullivan’s post on the ERE Recruiting Community goes on to detail Google’s approach [2]:

More than any other organization, Google is credited with changing the game when it comes to recruiting leading-edge talent. Their approaches have forced reactions among nearly every other leading high-technology firm trying to attract the cream of the crop, and encouraged a healthy debate among functional leaders of efficiency versus effectiveness in recruiting.

…on finding the right people who aren’t looking (note that it’s not the first time, either – remember the billboard incident of 2004 [10] that received a lot of press?):

The Google recruiting team continues to come up with creative approaches. One of my favourites occurred in the spring of 2006 when they retooled their search portal to deliver a targeted recruiting message to students and faculty of targeted schools. When individuals would access the Google search portal, Google servers would identify the IP address of the visitor, look up what organization the IP address belonged to, and alter the portal appearance if the visitor was accessing the portal from one of the university campuses Google actively recruits from. The approach, while not new, was implemented in Google’s typical minimalist style. They added a single text line just below the search box that asked whether the visitor was graduating and whether they were interested in a job at Google. The micro-targeting approach was simple and unobtrusive. Another example, while again not being unique, further signifies the extent to which Google is responsive to the labor force. That approach is taking the work to where the workforce already exists, namely the University of Michigan campus. The initiative took private/public cooperation to an entirely new level, ensuring that students would have access to education inherently suited to real employer demands, and that Google would have unfettered access to some of the brightest minds.

…and on their data-driven approach to candidate assessment:

The latest innovation from Google’s recruiting function is so unique that the New York Times wrote a feature story about it. The article, written by Saul Hansell and published January 3, detailed how the search engine company is implementing a new assessment tool that relies on an algorithm to more accurately identify candidates that resemble existing top performers. While many companies seek to screen out candidates, the new Google candidate assessment approach enables Google to “include” candidates that might otherwise be overlooked. The algorithm evaluates a much wider range of potential success predictors than can normally be discerned from most resumes. This innovation recognizes and resolves a major flaw inherent to typical assessment methodologies that rely too heavily on academic grades, SAT scores, degrees from “top” schools, prior industry experience, and subjective interview results.

It’s certainly eye opening (and refreshing, I might add) to look at true innovation alongside bolted solutions of the past. For instance, in building recruitment platforms, we often look at features that enable candidate (or ‘resume’) comparisons using a product like FAST ESP. The intention is the same: if you found one that worked, can you use it find others like it? The results, though, are far less compelling. Experience teaches us that we need to be careful in making the association between a candidate and their resume – search misses the point: it’s the data that matters.

From the New York Times piece, published in Jan, 2007 [3]:

Unfortunately, most of the academic research suggests that the factors Google has put the most weight on — grades and interviews — are not an especially reliable way of hiring good people.

“Interviews are a terrible predictor of performance.”

It’s a great read – Mr.Brock, brought in from GE to improve Google’s screening process, goes on to add [3]:

We wanted to cast a very wide net,” Mr. Bock said. “It is not unusual to walk the halls here and bump into dogs. Maybe people who own dogs have some personality trait that is useful.”

…

“You have to know or at least have a hypothesis why having a dog makes a good computer programmer,” Professor Mumford said. “If you ask whether someone started a club in high school, it is a clear indicator of leadership.

Questions like “what magazines do you subscribe to?” and “what pets do you have?” might sound far fetched but are relatively mild when compared to hand-writing samples [8] and brain-scans [9].

Google’s antics remind me of similar tests that we’ve experienced first-hand [4], [5] for a related but different goal (re-aligning individuals to roles); in both cases, some struggle with accepting the connections that are made:

The OPTIMAX CHARACTER ASSESSMENT tool is a proven system that enhances the potential for optimal performance. Over 20,000 assessments have been conducted in Europe, and in excess of 7,000 assessments in North America.

The Optimax Character Assessment is the best available tool for identifying the psychocybernetical makeup of an individual for the following reasons:

It is scientifically based - founded on a mathematical and logical theory

It is objective and quantifiable - uses mathematical formulas and algorithms

It is well controlled - the test taker cannot manipulate or misunderstand the questions

It is impartial - the results are not influenced by education, experience and knowledge

It is accurate - the accuracy of the results reaches up to +90%

It is reliable - repeated assessments will produce the same results; the assessment will predict all the changes for more than ten years

It is universal - the assessment is non-discriminatory on any basis such as race, religion, sex, age, or disability

It is thorough - the interpretation of the results is based on the combination of over 500 parameters

In addition to business organizations, Dr. Guy has been applying his model with success to Canadian athletes competing in the Olympics. I was so intrigued by my own results that I met with him in person to review them – not only did I find the assessment exceptionally accurate but those who knew me well were impressed too. That “taker(s) cannot manipulate or misunderstand questions” and that “results are not influenced by education, experience and knowledge” are critical components missing in conventional psychometric tools.

In this context, I’d be interested to get his thoughts on the use of his assessment as a recruitment tool; of course, given the manual interpretation of results and the attendant cost, it’s not exactly something that most can employ for front-line (bulk) screening, but the results are nonetheless telling.

There’s no doubt that we will shift towards increasingly scientific methods for not only recruitment but evaluation too, especially in light of the inherent flaws in today’s performance reviews (Chris has a great write-up on this [7]) – as Seth puts it [11]:

“You don't have to like the coming era of hyper-measurement, but that doesn't mean it's not here.”

Interestingly enough, this post started off with the intention of exploring what it means to be successful in the software industry. An ex-colleague has an excellent series here, here and here. (I’m more or less in the Jeff Atwood minimalist camp: the best code is no code at all; most people acknowledge the importance of simplicity, but there’s no shortage of drastically different views on how to achieve it.)

If the abandonment of that track to pursue the recruitment angle thought me anything, it’s that the right person may not have any of the qualities that you’re looking for.

References:

[1] – Google's Recruiting Practices http://blogerp.typepad.com/hcm_research/2007/01/googles_recruit.html

[2] – Google Continues to Innovate in Recruiting and Candidate Assessment http://www.ere.net/2007/01/08/google-continues-to-innovate-in-recruiting-and-candidate-assessment/

[3] – Google Answer to Filling Jobs Is an Algorithm http://www.nytimes.com/2007/01/03/technology/03google.html?_r=2&oref=slogin

[4] – Optimax Character Assessment http://www.optimax.org/tools-character.asp

[5] – Optimax Competency Matrix http://www.optimax.org/tools-comp.asp

[6] – Using AdWords http://www.free-press-release.com/news-why-aren-t-you-using-google-advertising-for-recruitment-1251387060.html http://www.slideshare.net/mspecht/online-recruitment-the-tools-techniques-and-challenges

[7] – A Compelling argument against performance reviews http://blog.chapmanconsulting.ca/2009/09/21/A+Compelling+Argument+Against+Performance+Reviews.aspx

[8] – Knowing you from your writing http://www.deccanherald.com/content/22797/knowing-you-your-writing.html

[9] – Brain scans 'provide clue to leadership skills' http://edition.cnn.com/2009/BUSINESS/09/10/brain.training.leadership/

[10] – Google recruits eggheads with mystery billboard http://news.cnet.com/Google-recruits-eggheads-with-mystery-billboard/2100-1023_3-5263941.html

[11] – Everyone gets paid on commission http://sethgodin.typepad.com/seths_blog/2009/09/everyone-gets-paid-on-commission.html

Windbg Not Getting the Attention it Needs: Doesn’t Work With 2.x

noreply@blogger.com (Nariman Haghighi) — Thu, 17 Sep 2009 17:43:00 +0000

If you’ve never used Windbg, you can probably stop reading – this isn’t for you.

This is for the few, the proud and the brave, banging their heads trying to figure out why CPUs are suddenly spiking; it’s especially for those still supporting legacy apps with literally thousands of DataTables in memory.

With the help of SOS (aka “Son of Strike”), SOSEX and the inherent traps in large-scale .NET deployments (60% CPU is considered normal), what was once an esoteric tool is gaining more and more traction. And because it’s predicated on the fact that you will always have issues that only surface in production (with inadequate instrumentation to tell the whole story), its significance is unlikely to diminish anytime soon.

Windbg is the be-all and end-all in production troubleshooting and it’s surprising that it’s not receiving the attention that it deserves from the CLR team.

There are literally 2 posts describing the use of the !DumpDataTables command. (DataTables are particularly significant as they’re an easy spot to catch uncached data-access or uncapped queries returning 10,000+ rows.) Alik and Tom, both presumably escalation engineers at Microsoft, detail case studies that demonstrate use of the command [1]. But what’s only mentioned in the comments of Alik’s post (can also be derived from Tom’s post through the use of .do –v which is replaced by –da for 2.x) is that !DumpDataTables isn’t available with SOS (externally) for 2.x assemblies as yet.

(For those counting, yes, we’re months into the .NET 4 preview and only a few months away from the 4 year anniversary of the 2.0 release – suffice it to say, it’s not coming.)

!DumpASPNETCache, !DumpDataTables, !DumpAllExceptions, !FindDebugTrue, !ASPXPages, !DumpConfig, !DumpHttpContext and a host of others [6] aren’t available for 2.0 assemblies:

“Doesn’t work with 2.x” is all you get. And to add insult to injury, the latest CLR10\SOS available keeps flashing the ‘tip’ banner telling you how wonderful these commands are.

You might be thinking who cares, you can always script this [3] [4][6]; but even if the output of DumpAllExceptions is equivalent to Tess’ script below, it misses the point:

.foreach (ex {!dumpheap -type Exception -short}){.echo "********************************";!pe –nested ${ex} }

In my experience, add-on scripts take longer to execute, have offsets that need to be uniquely configured, are more verbose and muddied in output and error out frequently when some written text info is displayed upon command execution. (Try Tess’ script for dumping out current and recent ASPX requests, for instance.)

The point is: if a SOS.dll with support for 2.x assemblies has been in use for years by the support team, why not release it on CodePlex without support or warranty and save users endless hours of frustration in having to sift through workarounds?

And why hasn’t the market filled the gap for GUI tools for Windbg, along the lines of SOS Assist, that simplify this [8] instead of leaving us with more of the same? Such a tool needs to be a core citizen of .NET right alongside Visual Studio and the Reflector.

But what about SOSEX?

Definitely, SOSEX is indispensible: its support for displaying value types and revealing the full stack are basic prerequisites for any problem. But it doesn’t provide the commands above; and modernizing Windbg is going to require more than one developer.

Some other notes that might help…

Start with !address open:

How to read this:

~1.5GB out of 2GB (1578940 KB) (max is 32-bit) is in use;
~500MB (518148) available (MEM_FREE)
Largest contiguous block is 46,368 => memory already fragmented!
RegionUsageIsVAD (.NET allocates here ~1.1GB)

More on the implications of the largest contiguous block being less than 64Mb [9]:

Large Object Heap

You should also pay attention to large objects, which lead to memory fragmentation. When .NET based application (as ASP.NET) needs to allocate new memory, it does so looking for (and reserving) chunks of 64 Mb free and contiguous memory: that is the key of this matter. When the Garbage Collector does its job and frees memory for unused (better say, unreachable) objects, it also tries to compact the chunks of memory to have it contiguous for future use, but this is not always possible: when the GC runs, it must temporarily stop all application threads, move around chunks of memory, update memory pointers and then the application can continue it’s job. But for performance reasons this can’t be done with objects larger than 85 Kb, which are seen as big objects from the CLR point of view and are allocated on a special heap, called the Large Object Heap. Objects in LOH are still collected and the memory freed, but since moving around such large chunks of memory is very expensive (and remember during this operation the application is frozen and can’t respond to client’s requests) and would require too much time and effort to the system, at the end the application would be too badly affected. The GC frees that memory but does not compact it, and this leaves some holes in our memory (like a Swiss cheese, if you like it).

Multiply this process during the life of the application and you could end up having lot of free memory but so fragmented (I’ve seen dumps where we still had 80% of available memory that the biggest contiguous free chunk of memory was just 50 Mb, too small for the 64 Mb needed), than the CLR could do nothing else but throw an OutOfMemoryException. Of course if you have lots of big objects you’ll have more chances to run into this problem.

This article has a quite detailed description of GC internals: “Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework” http://msdn.microsoft.com/msdnmag/issues/1100/gci/ and part2 http://msdn.microsoft.com/msdnmag/issues/1200/GCI2/default.aspx.

Update #1: It looks like at least a few others raised this issue in 2006 and 2007 to no avail.

Update #2: John Robbins, Wintellect, echoes the sentiment [9]: “This is especially important to .NET 2.0 because Microsoft dropped huge amounts of functionality in the SOS that ships with .NET 2.0 so digging through those production mini dumps was a complete carpel tunnel inducing exercise.”

References:

[1] – !DumpTables references from MS:
http://blogs.msdn.com/tom/archive/2007/12/26/high-memory-continued.aspx http://blogs.msdn.com/alikl/archive/2009/03/09/windbg-walkthrough-dump-values-of-dataset-or-datatable.aspx

[2] – Sample case:
http://rasmuskl.dk/post/A-WinDbg-Debugging-Journey-NHibernate-Memory-Leak.aspx

[3] – Reading values from a DataTable: http://labs.episerver.com/en/Blogs/Johano/Dates/2008/3/WinDBGSOS-Getting-at-the-values-in-a-DataTable/

[4] – On deprecation of !do address –v in 2.x and automation in Windbg: http://blogs.msdn.com/vijaysk/archive/2008/01/15/windbg-scripting-dump-data-column-names-from-a-table.aspx

[5] – Evolution of !da command (some history): http://blogs.msdn.com/shawnfa/archive/2004/04/30/124218.aspx

[6] – Manual Exceptions Summary:
http://blogs.msdn.com/tess/archive/2009/04/16/net-exceptions-quick-windbg-sos-tip-on-how-to-dump-all-the-net-exceptions-on-the-heap.aspx

[7] – !help output (latest CLR10\SOS from WinDbg 6.11.0001.404 x86):

Object Inspection Examining code and stacks

DumpObj (do)
DumpAllExceptions (dae) DumpStackObjects (dso)
DumpHeap (dh)
DumpVC
GCRoot
ObjSize
FinalizeQueue DumpDynamicAssemblies (dda) DumpField (df)
TraverseHeap (th)
GCRef

Threads (t)
CLRStack
IP2MD
U
DumpStack
EEStack
GCInfo
COMState
X
SearchStack

Examining CLR data structures Diagnostic Utilities

DumpDomain
EEHeap
Name2EE
SyncBlk
DumpASPNETCache(dac)
DumpMT
DumpClass
DumpMD
Token2EE
EEVersion
DumpSig
DumpModule
ThreadPool(tp)
ConvertTicksToDate(ctd)
ConvertVTDateToDate(cvtdd) RWLock
DumpConfig
DumpHttpRuntime DumpSessionStateConfig DumpBuckets
DumpHistoryTable DumpRequestTable
DumpCollection(dc)
DumpDataTables
GetWorkItems DumpLargeObjectSegments(dl) DumpModule
DumpAssembly DumpMethodSig DumpRuntimeTypes
PrintIPAddress
DumpHttpContext DumpXmlDocument(dxd)

VerifyHeap(vh) DumpLog FindAppDomain
SaveModule
SaveAllModules(sam)
GCHandles
GCHandleLeaks FindDebugTrue FindDebugModules
Bp
ProcInfo
StopOnException(soe)
TD
Analysis
Bl
CheckCurrentException(cce) CurrentExceptionName(cen) ExceptionBp
FindTable LoadCache
SaveCache
ASPXPages DumpGCNotInProgress CLRUsage

[8] – SOS Assist:
http://old.thinktecture.com/SOSAssist/Screenshots.htm

[9] – Large Object Heap:
http://blogs.msdn.com/carloc/archive/2006/09/25/770314.aspx

[10] – John Robbins On SOSEX:
http://www.wintellect.com/cs/blogs/jrobbins/archive/2007/06/19/great-sosex-a-phenomenal-net-debugging-extension-to-see-the-hard-stuff-steve-johnson-is-my-hero.aspx
http://msdn.microsoft.com/en-us/magazine/cc164138.aspx

The Google Outage: On Misplaced Rage & Crowd Mentality

noreply@blogger.com (Nariman Haghighi) — Tue, 08 Sep 2009 13:29:00 +0000

There is no shortage of outcries over the now infamous 100-minute outage.

“Little ‘dashboards’ aren't going to cut it,” or so it seems [1].

People want answers – and they want accountability.

Just don’t hold your breath for either.

Accountability will be supplied by the market; yes, once you abandon on-premise infrastructure, "it may be cost-prohibitive to return”; but that’s why the onus is on you, during the selection process, to make sure that the move protects against technologies that promote ‘vendor lock-in’ – and there’s plenty of support to manage this risk.

Pardon the terrible expression, but at the end of the day, expertise will converge into the hands of a few who will offer services that the fragmented provides of today can’t – this doesn’t protect against outages (nothing does!) but it does give you the best industry has to offer and delivers it for less than you pay today.

And of course the opportunity to parade around the standard cloud adoption scarecrows wasn’t lost either [2]:

I had a similar reaction to the newly announced Virtual Private Cloud offering from Amazon. Yes, the company has added some security features enterprises desire. But scroll down to the bottom of the FAQ, and in answer to the question "Does Amazon VPC have a Service Level Agreement (SLA)?" you'll see a two-word answer: "Not currently."

And when you think about it -- how could they? The liability would be ridiculously large if Amazon or Google started offering guarantees that came near the value of some unforeseeable disaster. Until that elephant in the living room is dealt with, few enterprise IT folks will take cloud services seriously.

I’d like to know who this mysterious entity that “offers guarantees… near the value of some unforeseen disaster” is? Does you in-house IT offer this? Or does your managed IT provider offer this?

Because this is what I see when I look at managed SLAs, and it doesn’t strike me as unusual:

Service Level Agreement (SLA) Please Note: The maximum service credit(s) available under this SLA in any calendar month shall not exceed fifty percent (50%) of the Customer’s monthly fees otherwise payable in such calendar month. 1. 100% Infrastructure Availability Acme will use its reasonable best efforts to maintain critical infrastructure availability 100% of the time in any given month. If the Customer experiences downtime due to the unavailability of critical infrastructure, Acme will refund the customer 10% of the monthly fee otherwise payable for each 60 minutes of downtime (up to a maximum of 50% of Customer's monthly fee for the affected Service). Critical infrastructure includes functioning of all power and HVAC including UPS. It does not, however, include the power supplies on the Customer's servers unless they are dual power supply machines. Infrastructure downtime exists when a particular server is shut down due to power or heat problems and is measured from the time the monitoring system detects the outage until the time the problem is resolved, the server is powered back on and the monitoring system records the system as active on the network (i.e. pingable). Credits shall not be provided to the Customer if infrastructure downtime is the result of: a) Temporary Service Suspension as per Section 4 of the MSA; b) Emergency Suspension of Services as per Section 6 or the MSA; c) circumstances beyond Acme’s reasonable control, including but not limited to Customer-provided software or equipment or Force Majeure Events; or d) a breach by the Customer of this Agreement or Acme’s AUP.

2. 100% Network Availability Acme will use its reasonable best efforts to ensure that its core network will be available 100% of the time in any given month. If the core network becomes unavailable and the Customer experiences downtime, Acme will refund 10% of the Customer's monthly fee otherwise payable for each 60 minutes of downtime (up to a maximum of 50% of Customer's monthly fee for the affected server). Network availability includes functioning of all network infrastructure including routers, core switches. It does not include distribution layer switching unless the Customer’s service includes redundant connections to distribution layer switching. Also, it does not include software running on the Customer's server. Network downtime exists when all core network interfaces are unavailable as determined and measured by a third party global monitoring system. These reports will be made available to Customer upon request. Credits shall not be provided to the Customer if network downtime is the result of: a) Temporary Service Suspension as per Section 4 of the MSA; b) Emergency Suspension of Services as per Section 6 or the MSA; c) circumstances beyond Acme’s reasonable control, including but not limited to Customer-provided software or equipment or Force Majeure Events; or d) a breach by the Customer of this Agreement or Acme’s AUP.

An interesting exercise is to think about is how quickly your competitors could force your hand on adoption that you may not be ready for: Competing with the Singleminded.

[1] – Gmail Outage Explanation Doesn't Wash http://www.pcworld.com/businesscenter/article/171317/gmail_outage_explanation_doesnt_wash.html?tk=rel_news

[2] – Gmail follies and Google's enterprise pitch http://www.infoworld.com/t/cloud-computing/gmail-follies-and-googles-enterprise-pitch-540

Latency: Solving 80% of the Problem

noreply@blogger.com (Nariman Haghighi) — Wed, 02 Sep 2009 13:04:00 +0000

There’s a great post describing the “solve 80% of the problem” mantra that’s worth paying attention to, especially in IT, where idealism yields to much tougher budget, risk-appetite, time, and resource constraints; in IT, you can really drown in the last 20%, even on problems much easier than real-time search.

We’ve had a couple of interesting discussions on a very old and widely-covered topic: latency. As it relates to web response times, studies (by Google, Amazon and others) suggest that delays as small as 250-500ms matter to your audience and impact revenue [6].

Two completely separate threads, with two completely different teams, debated issues that ultimately distilled down to latency; participants included upper levels of management:

How to reduce excessive round-trips required to render the homepage;
How to keep users pegged to a particular Data Center without introducing routing delays;

Before looking at solutions, let’s make sure we have a good sense for what latency is and why it matters so much. What may have once been dismissed as a ‘implementation detail’ is increasingly in spotlight as businesses build layers of redundancy, cater to wider audiences, and rely on distributed solutions. Side-stepping text-book definitions [1], a basic description resonates well is offered by Stuart Cheshire in his now infamous “It’s the Latency, Stupid” post:

If you have a network link with low bandwidth then it's an easy matter of putting several in parallel to make a combined link with higher bandwidth, but if you have a network link with bad latency then no amount of money can turn any number of them into a link with good latency.

To see how important this is to the internet elite, look no further than recent headlines [2]:

Miller: The choice of rural North Carolina suggests that the bottom line for Apple is cost, rather than connectivity. The site in Maiden, NC is not far from a large data center by Google, which usually chases cheap power and tax incentives. Power from Duke Energy is about 4 to 5 cents per kilowatt hour, compared to 7 to 12 cents in California. The company also maximized its incentives by pitting Virginia and North Carolina against one another in trying to wring the best tax incentives out of both states (a popular strategy in data center site location).
Some large companies use distributed data centers to manage their latency and content delivery costs. That may be part of Apple’s thinking, since they’re a major customer for CDNs (I believe they use both Akamai and Limelight Networks). Facebook cited latency to Europe as a key factor in its decision to add data centers in Virginia. Before that, MySpace added a data center in Los Angeles to reduce its reliance on CDNs. But in both cases, those companies sought out Internet hubs where they could connect with dozens of other networks to manage their Internet traffic. You don’t get that in rural North Carolina, so Apple seems more focused on cost and scale than on connectivity – which again would suggest a cloud focus.

A big problem that’s only going to get bigger, this much is accepted [4]:

But, as Todd Hoff notes in "Latency is Everywhere and it Costs You Sales - How to Crush it," latency concerns are still very much with us. In fact, the nine sources of latency that he lists suggest that latency is actually a much thornier problem in a world where applications are broken into pieces and often distributed around the world.

Just a few days before these debates stirred up, I stumbled across a Google tech talk on “Multihoming” (operating out of multiple Data Centers) [5], particularly relevant at the time as we had just encountered some of the challenges – even after ‘cheating’, in a sense, by dodging the toughest aspect of multihoming (distributing data).

In his talk, Ryan highlights the fact that CDNs [3] are indeed masters at “edge” caching (removing the latency); we’ve covered this as well, but to recap: it’s one of the most sensible, cost-effective things you can do (when catering to distributed audiences); regardless of what Jeff A. would have you believe [6], a well-selected CDN improves performance and reduces bandwidth expenses at the same time.

He also goes onto highlight that geolocality, "putting stuff near your users so their requests get to it and back, FAST”, is a primary driver for them too: “in the US, we are somewhat spoiled… and it’s easy to ignore the benefits of geolocality. Lots of stuff is here and we have good backbone connectivity to everywhere else. If you’ve ever spent time in India, China… there is a mixed bag of infrastructure at best. Even in Australia, there is good infrastructure but it’s far away.”

(We’ve seen this first-hand: imagine trying to serve-up Microsoft’s SharePoint to remote mine sites using Satellite links – it’s not pretty.)

Even here in North America, round trip from east coast to west cost is 30 ms, without any queuing delays or router delays. “Purely speed of light,” Ryan adds. “If you go through some of the big peering points, PAC-East, PAC-West, MAE-East, MAE-West, “God help you”, “they’re always overloaded”, “they’re going to add another 30-50ms without breaking a sweat.”

When you add this up, it takes just 10 round-trips to create a delay that users notice.

It’s not that we didn’t appreciate this, it's just that it's not always easy to fix, and in this case, the cost/benefit wasn't particularly attractive; latency could be fought more efficiently and more easily on other fronts.

With the two initial problems, we accepted the complexities and agreed that:

Instead of all-out browser caching (for static requests), we can offer most of the benefit (80% ?) without custom development simply by using hourly or daily expirations that coincide with release schedules (otherwise, you need to start thinking about filenames that include embedded release info to invalidate client copies - had this been baked into the initial design, it'd be a different story);
We can keep more than 80%+ of users pegged to a DC without material changes to the application; instead of working on a treatment for the other 20%, we can spend the time reducing more obvious latency issues in a manner that benefits everyone.

Technical Notes:

On keeping users pegged to a particular DC – because the application requires web-server affinity (relies on in-process memory), it also requires data-center affinity. This is normally provided by the BigIP F5 load balancer (GTM, to be specific) through stickiness based on the IP that queried the DNS record. It turns out that some ISPs proxy out these queries, so a particular user, mid-session (behavior varies across browsers) may suddenly see a new IP for the next DNS query, and potentially land at a new DC. And this effect is only amplifiied by short TTLs.
Though there a couple of alternative solutions, choices really boil down to: re-routing users back to the proper DC behind the scenes (and introducing latency, via a 100KM link that all requests for these users must funnel through) or exposing DCs under unique host-names (messy from an SEO, usability perspective).
Which would you choose?

[1] - Latency: The Silent Killer of Application Performance http://www3.villanova.edu/gartner/research/123400/123455/123455.pdf

Latency refers to the time required for a packet to traverse the network from source to destination. When dealing with the WAN, latency is measured as "round trip time" (RTT). In LAN environments, where applications are usually designed and tested, RTT is less than 10 milliseconds (ms). When dealing with cross-continental WANs, RTT increases from 50 ms to 75 ms; in global networks, it will reach 250 ms or more. The bandwidth provisioned is not a significant factor (especially once connections reach T1/E1 rates) in the overall latency because the predominant contributor to RTT (and hence, application delay) is the time "on the wire," which is gated by the speed of light. Therefore, latency (and application performance) is directly proportional to distance from the data center.

[2] - Interview: Apple’s Gigantic New Data Center Hints at Cloud Computing http://www.cultofmac.com/interview-apples-gigantic-new-data-center-hints-at-cloud-computing/14680#more-14680

[3] - What CDN would you recommend? http://highscalability.com/what-cdn-would-you-recommend-0

[4] - Latency (still) matters http://news.cnet.com/8301-13556_3-10024650-61.html

[5] - Multihoming: How Google Serves Data from Multiple Datacenters http://code.google.com/events/io/sessions/TransactionsAcrossDatacenters.html http://highscalability.com/how-google-serves-data-multiple-datacenters

[6] - YSlow: Yahoo's Problems Are Not Your Problems http://www.codinghorror.com/blog/archives/000932.html

[7] – When the Speed of Light is Too Slow http://t1rex.blogspot.com/2005/03/when-speed-of-light-is-too-slow.html http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it

The MPG Illusion & Seth Godin

noreply@blogger.com (Nariman Haghighi) — Sat, 22 Aug 2009 15:04:00 +0000

I’ve been an avid follower of Seth Godin ever since I watched his “why marketing is too important to be left to the marketing department” talk at the Business of Software conference. (If you haven’t made the time to see this, you really should.)

This morning his blog featured a simple quiz, which I must admit had me stumped too, despite my Bachelor of Mathematics degree:

A simple quiz for smart marketers:
Let's say your goal is to reduce gasoline consumption.
And let's say there are only two kinds of cars in the world. Half of them are Suburbans that get 10 miles to the gallon and half are Priuses that get 50.
If we assume that all the cars drive the same number of miles, which would be a better investment:
Get new tires for all the Suburbans and increase their mileage a bit to 13 miles per gallon.
Replace all the Priuses and rewire them to get 100 miles per gallon (doubling their average!)
Trick question aside, the answer is the first one. (In fact, it's more than twice as good a move).
We're not wired for arithmetic. It confuses us, stresses us out and more often than not, is used to deceive.

Surely, there’s a trick, I thought; I immediately started reading too deeply into the subtleties of the implicit consumption associated with new Suburbans tires vs. replacing Priuses outright – this completely missed the point!

Frustrated, I opened up Notepad and wrote it all down:

Let m be number of miles driven by a car...
Let s be the gas consumption (in gallons) for Suburbans (= m/10)
Let p be the gas consumption (in gallons) for Priuses (= m/50)
Let T be the total consumption (in gallons) (= s + p = m/10 + m/50 = 6m/50 = 0.12m)

So in Scenario #1, we have T = m/13 + m/50 = 50m+13m/650 = 63m/650 = 0.097m

And in Scenario #2, we have T = m/10 + m/100 = 11m/100 = 0.11m

Scenario #1 reduced consumption by 0.12-0.097 = 0.023; Scenario #2 only by 0.01; Scenario #1 is 2.3x more efficient! Sure, it all makes sense when it’s drawn out for you [1]:

This very interesting article in Science, “The MPG Illusion” by Richard P. Larrick and Jack B. Soll at the Fuqua School of Business in Duke University (Vol 320, June 20, 2008, p. 1593), points out the mathematically obvious truth that gas used per mile is inversely proportional to miles per gallon, which means that you have a steeper slope at lower MPG ratings, and diminishing returns at higher MPG ratings.

Now, try to think about how this applies to your daily life and where you spend your time, particularly as an application developer.

Successful SR&ED (Scientific Research & Experimental Development) Claims Explained

noreply@blogger.com (Nariman Haghighi) — Wed, 12 Aug 2009 19:24:00 +0000

The technical, logic-oriented side of me has traditionally despised SR&ED; and I’m sure that the ambiguity, subjectivity, procedural vagueness, lack of structure and transparency are nightmares for other likeminded individuals too. In this post, we breakthrough the confusion and deliver a succinct, objective methodology for Canadian IT professionals, most of whom have at one point or another grappled with SR&ED and reluctantly accepted the guidance of consultants as the sole authority.

I’m really curious: what’s been your experience with SR&ED?

Drop me a line if, after reading this, you still aren’t clear about what’s in and what’s out.

Conclusion #1: Outsourcing Claims

There are no shortage of consultants, claiming expertise and exclusive insights into the muddied waters of SR&ED, willing to help you navigate the terrain and maximize your claim potential, even on a contingency basis. In my experience, based on both accepted and challenged claims, these services offer little value-add. At their core, successful SR&ED claims should be transparent enough to be authored by internal resources, assuming they’ve been properly acclimated to the process and its common pitfalls (what this post is intended to help with).

So here it is, in a nutshell; the 6-step guide to successful SR&ED claims:

Familiarize yourself with the fundamentals of the program; understand the difference between IRAP and SR&ED [1].
Read the Guidance on Eligibility of Software Projects for the SR&ED Tax Credits and Developing and Documenting Claims [2]; yes, it’s an old document, but the cases and methodology are very much relevant even for current FY08, FY09 claims.
Group each part of your claim clearly under the three criteria of eligibility: Technological Uncertainty (TU), Technological Content/Chronology (TC) and the Technological Advancement (TA). Note that under the revised T661 form [3] for 2009, TU = 242, TA = 240, TC = 244; the new ordering they’ve suggested is actually counter-intuitive and somewhat harmful, IMO. You will complete a separate Part 2 – Project Information for each project claimed in the year.

Order is paramount: start by succinctly and confidently identifying the Technological Uncertainty (this obstacle is commonly referred to as a “nugget”) as a first-step; TU is the foundation for everything else and it should have the following characteristics (see Update #1 on when TU can be defined up front):

a) Element of Surprise	Must have been an unplanned, unforeseen hurdle; i.e., it caught your team off-guard mid-stride in the delivery phase.
b) Risk	Must have jeopardized successful execution of the project and achieving the stated deliverables; put differently, it introduced new risk in completing previously scheduled work that would have otherwise transpired if it wasn’t compromised by this hurdle.
c) Non-Routine	Routine uncertainties resolved through code or peer discussion and/or optimization, calibration-type exercises are not included; it must be shown that there is a fundamental problem with the underlying technologies. Refer to 6b) for validation through peer reviews. IC97-1 Sec. 3.2: A technological uncertainty arises when the solution is not readily apparent to appropriately skilled and experienced software developers. Uncertainties that arise from lack of diligence or lack of appropriate expertise, such as the failure to use commonly available information, or lack of programming knowledge are not technological uncertainties and relevant to eligibility.

For each nugget, prove Technological Content/Chronology by demonstrating:

a) Experimental Development

This doesn’t necessary mean that you attempted each approach; it means you researched, analyzed and carefully considered potential work-arounds before ultimately accepting or rejecting them. You can apply scientific methodology without actually prototyping each option; applying defined selection criteria to each alternative, and reasoning about why they pass/fail is sufficient.

By showing the evolution of the design (confirmation of approach or changes in design approach or
changes to the specification) resulting from each analysis, prototype or test, the taxpayer also demonstrates
Technological Advancement.

b) Technological Gaps

This is often the hardest criteria to satisfy. When encountering a hurdle (A), experts are trained to immediately jump to solution (B); but merely observing that A doesn’t work and offering B as a working solution is insufficient. You need to: a) understand why A doesn’t work, b) show a systemic progress from A –> B, with discussion and analysis of in-between alternatives, and c) only arrive at B only with a conclusion that demonstrates its efficiency over the others.

Trial-and-error involves executing a series of probes that were not sequenced in a systematic pre-plan. The objective here is to resolve a functional problem (as in routine debugging) rather than to gain understandings that are expected to be more widely applicable. The lesson learned in each iteration of “trial and error” is simply “that an option didn’t work” and they are not applicable in a much broader sense. For each iteration the probe is chosen that is now judged to be the most efficient in resolving the immediate problem. The process proceeds quickly from iteration to iteration.

As stated in IC97-1, “experimentation or analysis in a situation where there is technological uncertainty”, is a strong indication of project eligibility. The claimant should strongly link the experimental work to the Technological Advancement sought and to the Technological Uncertainties declared in the technical description of his project. In this way, the claimant will show clearly that specific experimental work is aimed at resolving the Technological Uncertainty and thus constitutes an attempt at providing the Technological Advancement.

c) Qualified Personnel

SR&ED is not concerned with ramping people up or learning new technology; it does, however, make allowances for not reasonably having access to the required expertise and exploring ideas under business-specific cost, feature constraints.

An appropriately skilled individual is one whose qualifications and/or experience enable him or her to operate competently within a particular field . Such a person could be reasonably expected to produce a functional solution to a problem if such a solution exists within the constraints imposed upon the solution.

The skills expected for a company’s developers are those commonly found within the context of a company’s competitive business environment, and appropriate to the work being undertaken.

The three criteria for eligibility are to be applied with reference to the circumstances of the organization that executes the project. The judgement as to what knowledge of solutions a company should have or how far a company should search for solutions is to be assessed within the context of what it is reasonable to expect of a company conducting the project in a similar business environment.

If a third party executes a project on behalf of a claimant, the applicable business environment against which eligibility will be evaluated is that of the third party. Nevertheless, a claimant should not be denied eligibility simply because third parties exist for whom the same project would be standard practice, .if the claimant did not have reasonable access to that expertise. As noted, the meaning of “commonly available” has to be sensitive to the business’ environment and hence what can be expected of the tax payer’s developer’s project employees. All information technology workers are expected to have knowledge of the commonly available evolving tools of their trade. Workers are expected to make reasonable efforts to learn of new tools that might be applicable to the task at hand, but can be excused for failure to discover a relevant tool not commonly available in the public domain.

Simply learning to solve the problems that are commonly encountered during the first time use of a commonly available tool which happens to be new to the claimant’s business environment does not on its own involve the resolution of a Technological Uncertainties and hence produce a Technological Advancement. In the absence of any other fundamental technological problems, this is simple on-the-job learning.

The redevelopment of a solution which was generally available will usually fail the tests of both Uncertainty and Advancement. When making a claim based upon such a project, the onus will be on the claimant to show that (a) he was in fact unaware of the existing solution, and that (b) he made reasonable efforts to search for available solutions before embarking upon his project. Hint: Often it is a company’s ability to show how he normally searches for generally available solutions or that he did search for them or that he has established make/buy assessment practices that provides useful evidence that a “generally available” solution was not available to him in his business environment.

The business environment in which the software is developed or is to be used can give rise to technological challenges and hence Technological Uncertainties, but business constraints per se are not of themselves Technological Uncertainties.

For example, one claimant may choose for business reasons (e.g. time to market, a perceived need to differentiate himself from a competitor’s product) to attempt technical solutions involving Technological Uncertainty which another claimant might choose to avoid. A small business may try to minimize the development costs of a software system by taking some shortcuts that increase his risks. A larger firm might not take these same shortcuts because of the increased commercial risks. In this scenario, the claimant may be able to provide evidence that the shortcut was the source of a technological challenge and hence of a Technological Uncertainty and hence he sought an advancement.

For example, acceptable evidence might be advanced by showing that component hardware or software chosen because of cost considerations were less capable than those required for the system’s certain success. The resolution of the resulting Uncertainties, if done experimentally, could lead to an Advancement and thus to project eligibility.

For each nugget, prove Technological Advancement by demonstrating:

a) A New Construct, Architecture or Technique	A key here is that a technique equally encompasses do’s and dont’s. Also note that it’s not necessary for the advancement to have been incorporated into the ultimate solution; that the field of knowledge in this realm was advanced is sufficient.
b) Peer Review & Relevancy	Is the material presentation worthy in that an audience of your peers would find the material compelling?
c) Evolution	How does it compare with earlier solutions and what constraints have been overcome?
d) Form/Function vs. Approach?	How widely applicable are your ideas? Simply claiming to have developed the first or best software suite for a given purpose does not in itself prove that the taxpayer has made a technological advancement. A new and unique software suite can be built using only well known combinations of constructs, tools and methods without technological advancement. This is analogous to designing and building a unique and complex office building without making any advancements in the field of civil engineering. Note that an advancement in technology can rarely be described by listing software functions and features at an “end-user” level. Advances are typically made through innovation in software architectures, designs, algorithms, techniques or constructs.

Update #1: Yes, there are certainly eligible projects that don’t fit this mould – be sure to read the guidance carefully. The intention here is to provide a checklist to validate the most common form of eligibility encountered in day-to-day IT. We are trained to avoid risk and to maximize leverage in helping businesses make the most of established tools, processes – this is an inherently different mindset when compared to traditional software vendors, OEMs and ISVs. For instance, in reviewing the section on “Eligibility of work to improve the process of Software Development,” you will find 2 examples that don’t fit the pattern: in both cases, the claimant found no commercially available tools and identified significant obstacles in developing the concept. In such cases, Technological Uncertainty (TU) can be defined up-front, without the element of surprise or risk. In practise, this is difficult to prove for an IT consultant working under billable hours.

[1] – SR&ED and IRAP Primer
http://www.blog.businessready.ca/2008-sred-and-irap-part-two/

[2] – Guidance on Eligibility of Software Projects for the SR&ED Tax Credits and Developing and Documenting Claims
http://www.cra-arc.gc.ca/txcrdt/sred-rsde/pblctns/swgtxt-eng.pdf

[3] – Revised T661 SR&ED Expenditures Claim Form
http://www.cra-arc.gc.ca/E/pbg/tf/t661/t661-08e.pdf

SQL Azure 10GB Limit: Implications & Workarounds

noreply@blogger.com (Nariman Haghighi) — Tue, 21 Jul 2009 18:28:00 +0000

I’ve taken my first deep look at Azure to answer some specific questions that are covered in detail below. My overall impression: Azure (and cloud computing in general) is a powerful paradigm shift for our industry, with the end-result being more emphasis on pure software engineering and less emphasis on operations. Solution providers, consumers, and businesses not in the infrastructure outsourcing field all stand to benefit under this re-alignment – and we should all embrace it. I’m personally very excited to start delivering applications under this new mindset; I think concerns about the longevity of the initiative (or Microsoft’s dedication to it) are misplaced.

As a permanent fixture, Azure represents the end of the “keeping the lights on” mentality that has plagued IT:

“When IT stops being an enabler and simply acts as a cost center or a "necessary evil", then entertaining new projects that leverage one of Gartner Top 10 strategic technologies is unrealistic and doomed for failure or hardship at a minimum. Why? Because this mentality is reactive and sometimes even defensive. This mentality also does not encourage investing in the future whether that is in training employees, establishing and investing in architecture, or addressing problems with long term solutions. Instead, people are rewarded for quick fix fire fighting heroics and at the end of the day it is the user community that suffers. The users get band-aids on top of already outdated systems and are often forced to work in ways in which the technology and systems dictate, instead of the other way around.

It gets worse for the employees in an IT department in charge of "keeping the lights on". Their resumes become stale as their skills are not updated to reflect the newer technologies that innovative companies seek. The reactive nature of the culture can squash innovation and people who have valid solutions may not bring them forward because it would require more than a quick fix. In the end these types of shops become like an assembly line in nature where people clock in, work their shift, and clock out. This is not what must of us envisioned when we enthusiastically enrolled in IT related curriculums in college back in the day.”

Whether it takes 2, 5, or 10 years to permeate isn’t important; but that Microsoft sees 50% of it’s server business moving online in the next 5 years is a telling approximation.

Before delving into the specific questions, I would highly recommend the presentation by Brandon Watson on the state and drivers behind cloud computing – it paints an interesting picture around Amazon’s seasonality and Google’s Ad dependency as important factors in their desire to offload infrastructure costs. Likewise, Microsoft too developed its strategy out of necessity and not choice, which further underscores its importance. (Microsoft’s Business Productivity Online Suite suite isn’t yet running off of Azure, but as adoption increases so to will the need for on-demand scale.)

Implications and workarounds for the 10GB database limit

The work-arounds, for the time being, boil down to: sharding (a term coined by Google Engineers to refer to a method of horizontal partitioning), integration with on-premise SQL or better leverage of Azure Storage. There’s a great article that captures some interesting discussion around this limit [11] and the ramifications for backups; however, the Azure FAQ seems to suggest that backups-in-the-cloud are included. That most of TSQL constructs are supported, with a few obvious exceptions[4], is certainly good news. But the 10GB limit is nonetheless a disappointment in the v1 announcement – it is by far the largest barrier to adoption, particularly for employee-facing applications that collaborate with several internal systems. As a colleague of mine pointed out: that you can connect to on-premise SQL is one thing, but the amount of effort involved to match the performance of co-located servers, particularly with chatty applications, is another. For self-contained, consumer-facing applications, migration is much more realistic. Remember, storage outside of SQL doesn’t have an upper-limit, capacity-wise; but its per-transaction pricing does offer less cost-certainty [1]. If you don’t have existing database assets to leverage, you may opt to build your data model on Azure storage directly, but you would be sacrificing portability and toolset in the process.

On the ability to integrate with on-premise SQL [18]:

“In particular, Nigel did a really nice demo on the compatibility SDS v1 will have with on-premise SQL and how that will make porting existing applications to a cloud environment really easy. A very strong message particularly in conjunction with the ability to run web and worker processes in Windows Azure Compute. Together, these capabilities really provide a great foundation for delivering next generation services based applications.”

How does fault tolerance work, and are you charged for separate Web Role instances required?

Whether separately-billed instances of the Web Role (required to satisfy the 99.95% requirement in the SLA) are required for high-availability is a bit unclear [9] . It seems, on the surface, that a single role can satisfy the SLA requirements through configuration alone, without additional fees [17]:

“The role definition also includes the number of instances that the fabric should deploy, either a specific number or a lower and upper bound. Finally, constraints may be specified, including how many role instances may run in the same node, whether instances of different roles should be co-located, and how to allocate instances across update domains and fault domains.

Update domains and fault domains are optional but useful features. Update domains are used to partition the service during upgrades. A role may specify the number of update domains its instances should be deployed in.

During rolling OS upgrades and service updates, only one update domain will be upgraded at a time. While an update domain is being upgraded, the fabric won't route live traffic to any roles or load balancers inside it. After an update domain has been fully upgraded and all services in it report that they're health, the fabric returns them to service and starts on the next update domain.

Fault domains are similar to update domains. They're basically disjoint failure zones within a single datacenter. They're determined according to the datacenter topology, based on things like rack and switch configurations, which make certain classes of failures likely to affect well-defined groups of nodes. As with update domains, a role may specify the number of fault domains its instances should be deployed in.”

Pricing and Licensing

Note that we haven’t seen the the subscription model or volume licensing just yet [1]:

“Microsoft will offer Azure services in three different ways, including a consumption (pay-as-you-go) model, a subscription model for resellers, and volume licensing to enterprises. The so-called consumption model appears to be priced comparably with the current leader in this market, Amazon.com, which offers a variety of online services, including hosted Windows and Linux services.

Microsoft will charge 12 cents per hour for computer infrastructure services, 15 cents per gigabyte for storage, and 10 cents per 10,000 storage transactions. Users of Microsoft's cloud-based database, SQL Azure, will incur monthly charges of $9.99 for the Web Edition, which supports up to 1GB databases, and $99.99 for the Business Edition, which allows up to 10GB databases. The web developer-oriented .NET Services will cost 15 cents per 100,000 message operations. Additional bandwidth charges apply across the three services as well: 10 cents per gigabyte for incoming data and 15 cents per gigabyte for outgoing data.”

Session, Cache Data

This is a bit of paradigm shift for the typical .NET developer: you no longer have in-process memory (a good thing for scalability) – so where do values in HttpContext.Cache and HttpContext.Session go? The SDK has providers [2] that allow plug into your application “seamlessly” to store these values in Azure Storage (table and blobs) – but there must be a performance penalty associated with this, not to mention the $0.01/10K transactions and $0.15/Gb cost associated with reading/writing to this store. (I use “seamlessly" because objects crossing the wire have to be serializable, so some minor work may be required there.) Whether Velocity eventually services this role remains to be seen; lots of desire to see this happen, mostly for the richness of the API [21]. In terms of leveraging the local 250Gb available to each web role, it appears that there this doesn’t offer much use to developers as it’s not synchronized across multiple instances.

On-Premise Azure

There are no plans for an on-premise version of Azure (storage or otherwise), as far as I can see [20].

Exception Handling

I don’t have a good handle yet on how this is best applied. Some providers will likely emerge that channel diagnostic messages into Azure Storage where they can later be searched, grouped and presented by various ‘Storage Explorer’ like applications [6].

Azure Medium Trust Implications

No major limitations in the custom Medium Trust policy used by Azure, from what I can see [16]: external TCP connections and local writes both seem to be permitted, allowing a full range of possibilities around external integration scenarios using WebRequest, FtpWebRequest, StmpClient [15] and the like.

Sample Deployment of AzureBright: http://www.radicallyfrugal.com

What better test-drive than to take an MVC application and deploy it to the cloud, under a custom domain.

What is AzureBright [1]?

“AzureBright Blog & Forum are two cloud-hosted applications, created for the new CloudApp() contest. Goals: 1.To have a fast, clean (html), scalable, Web 2.0 look and feel, and SEO Friendly, Blog and Forum (StackOverflow.com-like functionality) for the .net community. 2.To share development knowledge and experiences, of new challenges encountered with Windows Azure Service’s Platform.

Technologies used: Windows Azure Storage (Table, Queue, Blob), ASP.NET MVC framework (C#), AJAX using jQuery/json.

Challenges encountered: Paging over data. “Order by” and “Group by” data, Leverage Concurrency, Update/Insert/Delete Batch Transactions, Page Views, and other counters,Voting, Spam, Multiple Worker messages, Membership (manage users and roles)…”

References:

[1] – Pricing/Reselling

http://www.microsoft.com/azure/pricing.mspx
http://windowsitpro.com/article/articleid/102509/microsoft-answers-windows-7-azure-rtm-questions.html

[2] – Session/Cache Providers

http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/2d1340ed-0ad0-456a-b069-aa6b85672102
http://www.aaronlerch.com/blog/2008/11/01/run-aspnet-mvc-on-windows-azure/

[3] – Troubleshooting

http://stackoverflow.com/questions/444528/what-is-my-azure-table-storage-accountname
http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/cbf60a9a-861f-475d-90c6-ba8ad3282d2f

[4] – TSQL Support in SQL Data Services

http://blogs.msdn.com/ssds/archive/2009/07/07/9823115.aspx

[5] – Queue Performance (good blog...)

http://cid-61aea8168d26ea6b.profile.live.com/
http://bstineman.spaces.live.com/blog/cns!61AEA8168D26EA6B!312.entry

[6] – Storage Explorer

http://azurestorageexplorer.codeplex.com/
http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/d4e8f1be-297b-47ff-8135-c26ceafea9c2
http://davidpallmann.blogspot.com/

[7] – 10Gb Limit

http://www.techcrunch.com/2009/07/14/microsofts-azure-gets-a-business-model-and-an-official-release-date/
http://www.sharepointevolved.com/blog/Lists/Categories/Category.aspx?Name=Micorosft%20Azure
http://blogs.msdn.com/windowsazure/archive/2009/07/14/confirming-commercial-availability-and-announcing-business-model.aspx

[8] – SLA

Windows Azure:

Windows Azure has separate SLA’s for compute and storage. For compute, we guarantee that when you deploy two or more role instances in different fault and upgrade domains your Internet facing roles will have external connectivity at least 99.95% of the time. Additionally, we will monitor all of your individual role instances and detect within two minutes when a role instance’s process is not running and initiate corrective action (we will publish by PDC the full details of our uptime promise for individual role instances). For storage, we guarantee that at least 99.9% of the time we will successfully process correctly formatted requests that we receive to add, update, read and delete data. We also guarantee that your storage accounts will have connectivity to our Internet gateway.

[9] – Fault & Update Domains

http://www.azureusergroup.com/profiles/blogs/fault-domains-and-upgrade

[10] – AzureBright
http://azurebright.codeplex.com/
http://www.newcloudapp.com/vote.aspx

[11] – Brent Ozar (SQL Server DBA)

http://www.brentozar.com/archive/2009/07/sql-azure-pricing-10-for-1gb-100-for-10gb/

[12] – TechCrunch Thread

http://www.techcrunch.com/2009/07/14/microsofts-azure-gets-a-business-model-and-an-official-release-date/

[13] – Excellent Presentation by Brandon Watson

http://www.manyniches.com/cloudcomputing/cloud-platforms-whats-going-on/

[14] – Azure FAQ

http://www.microsoft.com/azure/faq.mspx

[15] – Windows Azure - Sending SMTP Emails!

http://blogs.msdn.com/davidlem/archive/2009/01/08/windows-azure-sending-smtp-emails.aspx
http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/ef7a0cd5-5b2c-4ab2-902b-d39cc4b69913

[16] – Windows Azure SDK Trust Policy Reference

http://msdn.microsoft.com/en-us/library/dd179369.aspx

[17] – Deep Technical Details:

http://snarfed.org/space/windows+azure+details

[18] – SQL Azure Team Blog

http://blogs.msdn.com/ssds/archive/2009/03/20/9493117.aspx

[19] – Estimation Spreadsheet:

http://blogs.codes-sources.com/redo/archive/2009/07/14/azure-services-platform-windows-azure-sql-services-net-services-calculez-vous-meme-votre-prix-d-hebergement-mensuel.aspx

[20] – On-premise Azure ruled out:

http://blogs.zdnet.com/microsoft/?p=2340
http://blogs.zdnet.com/microsoft/?p=2340

[21] – Velocity on Azure:

http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/8b4fc071-fe1e-40d2-987a-d6a95b13a34b
http://silverlightuk.blogspot.com/2008/11/windows-azure-cache-based-session.html
http://social.msdn.microsoft.com/Forums/en-US/velocity/thread/a7bbaaa7-de2a-4b24-b41f-cf30f6fb592e
http://blogs.msdn.com/davidlem/archive/2008/12/16/windows-azure-what-happens-in-the-data-center.aspx

PopUrls, PopFly & The Future of Aggregation with Yahoo Pipes

noreply@blogger.com (Nariman Haghighi) — Fri, 17 Jul 2009 19:39:00 +0000

Hot on the trail of Google, Microsoft announced this afternoon that it too would be discontinuing it’s own Mashup Editor, PopFly [1]. Naturally, no one is happier to rejoice in the demise of these two experiments than Yahoo, whose own Pipes offering continues to thrive [2].

Having test-driven Pipes recently, I’m not too surprised. Pipes’ success, much like twitter, lies in its simplicity and elegance; they’ve focused on delivering core data that serves as the essential building block for integrated applications. It’s a simple way to build a unit of logic that’s truly boundary-less.

Now take aggregation service of today like PopUrls [3], for instance:

I’m hard-pressed to see exactly how this exact same data, supplied by public RSS interfaces, can’t be constructed in a personalized fashion using Pipes in a way that can be consumed by existing readers?

What’s really the value-add here, beyond the interface?

And if there is a value-add, why isn’t it exposed for consumption by others?

Yahoo Pipes carries immense potential for the wider development community and it’s no surprise that it’s the lone standing ‘Mashup Facilitator’ among the leaders.

[1] – Washington Post
http://www.washingtonpost.com/wp-dyn/content/article/2009/07/17/AR2009071700741.html

[2] – Pipes Blog
http://blog.pipes.yahoo.net/2009/07/17/microsoft-popfly-and-google-mashup-editor-rip/

[3] – PopUrls (Thomas Marban)
http://en.wikipedia.org/wiki/PopUrls
http://tomatic.com/

Claim code: 6hjuni7rkc

Hosting Your Site on a Content Distribution Network (CDN)

noreply@blogger.com (Nariman Haghighi) — Tue, 14 Jul 2009 20:41:00 +0000

What if you were able to deploy a completely updatable static HTML site in minutes and have it distributed to a global audience for pennies a GB?

A thought occurred to me the other day: offerings like Google Sites and SharePoint Online, though not branded as CDNs per se, likely enjoy reliable access times globally; and both allow arbitrary files to be addressable at user defined URLs using CNAME records. However, with Sites at least, *.HTML, *.CSS, and *.JS files are explicitly prohibited, leaving users struggling to achieve concepts not permitted in the default templates and relegating the products to internal-use scenarios.

But imagine if you had free-reign over static files that leverage the same infrastructure? Imagine if a CDN could host your entire website and not just its static elements. Sure Azure (pricing just announced [14]), EC2 and AppEngine carry promise for offloading applications to the cloud, but the serving of static sites still seems better suited for well-established CDNs.

Welcome to the world that I’ve been immersed in for the last few days: evaluating CDNs as a cost-effective approach for hosting a static site with a gradual ramp-up to ~2M visitors a day. Solution requirements: high-availability, high-scalability and cost-predictability.

Traditional players (Limelight, Akamai and Level 3 – who collectively control 80% of the market) are slowly being challenged by a new breed of pay-as-you-go ‘budget’ CDNs courtesy of pioneers like Amazon, SimpleCDN and the like. Even Limelight re-sellers like Rackspace are undercutting aggressively while offering users a cheap alternative to the same network. In terms of high-level feature set, they’re all roughly comparable; key differences lie mostly in the quality and reach of the distribution networks via so-called “edge” servers around the world.

But there are massive price differences [3] – as large as 10-20x between the least expensive (SimpleCDN) and the most expensive (Akamai). It’s not clear what the noteworthy differentiator between the leading networks is (or whether there is one) – I think they jury’s still out and, in practise, it likely depends largely on the account and their desire to accommodate you. Attempts to quantify performance between the two is a seriously challenged science [10]. The conclusion of “..if you want to really test a CDN’s performance, and see what it will do for your content and your users, you’ve got to run a trial” seems to be the only take-away. And the challengers aren’t without their share of criticism either [6] – both are relatively new (Amazon’s CloudFront is still in beta and doesn’t offer an SLA; it has had some outages) and both have relatively limited distribution networks (SimpleCDN has 10 points of presence in US/Europe and Amazon has 14 that cover Asia as well [9]).

Another consideration: separation of storage from delivery – normally, the CDN can either host your content and charge you a storage fee ($.15/GB, this is pretty standard across the lot, including Azure) or it can access content that’s already hosted and distribute it. With Amazon, you must use their storage services (S3) to be eligible for the CDN, and they will charge you the transfer rates for propagating content from S3 to the CloudFront CDN. SimpleCDN has a product in beta (Lightening) that supports already-hosted content but their mature offering, like Amazon, also relies on a built-in storage service (also at $0.15/GB).

But hosting your own content defeats the purpose of freeing yourself from infrastructure and high-availability concerns; you would have to ensure that the source content is always available or risk requests from the CDN (upon cache expirations) resulting in 404s (this behaviour was confirmed for at least Limelight). And this brings me to my next point…

Cache-Control Headers – Why they matter

Cache-control headers on source content (whether hosted by you or a storage services) are respected by the CDN distribution network. This is the only mechanism you have to expire content located on edge servers, otherwise it defaults to something like 24 hours. Normally, there is no programmatic API to invalidate cached files on edge-servers [5], which creates a tricky balance: to be able to update something, you must define the TTL when you create the file, before it propagates to edge-servers. This is particularly relevant for URLs that can’t be versioned. You can’t selectively update something: either you pay the price of updating it regularly of give up the ability entirely. In our case, we’re looking to employ the CDN in a unique way to host the entire site. In order to do so, we need to refresh HTML every few minutes and cache-controls on the source files allow us to do this. At the same time, we want to employ standard caching practises to ensure that images, CSS files are held as far as the clients and that only HTML requests beyond say x minutes reach the edge-servers and propagate back to the source server.

Can you run the entire website on a CDN?

I’d like to think so, and we’re still trying to see if there are any hidden catches in this scheme. One concern that we’ve run into: how do we ensure that domain.com, www.domain.com all redirect to the index page www.domain.com/index.html? Remember this isn’t a web server where you can just go into IIS and configure the default page. These are CNAME records that point to the custom CDN sub-domain that’s been provisioned for your particular account. In my discussions with Limelight, they mentioned that their engineer can apply this as a custom one-time configuration, which is nice touch. For SimpleCDN and Amazon, I think your only option is to set up DNS WebForward records so that both domain.com and www.domain.com point to content.domain.com/index.html, with the CNAME record for content.domain.com pointing to your CDN. Under this workaround, trying to access content.domain.com will generate a 403 [11]. The lack of a “index file” option in the pay-as-you-go CDNs could be a bit of a hindrance if you’re URL sensitivity.

And where does the HTML come from?

An internal ASP.NET site generates the desired HTML; a scheduled service consumes this HTML and publishes to the storage service (via FTP or SOAP APIs) on a regular interval, with cache headers to expire the content as needed. This publication scheme itself needs some level of redundancy for a solution demanding high-availability, but instead of failures propagating to your end-users they’ll simply result in stale content.

Pricing

Traditional hosting doesn’t stand a chance in competing with these prices and is more than x2 the cost even at the Limelight tier. Managed hosting providers will likely charge bandwidth on the 95th percentile of the transfer rate as opposed to the cumulative transfer amount. This means they take the highest 5% hourly/daily Mbps rate , drop them, and use the next highest Mbps to calculate your monthly invoice. Say you had a commitment to 3-5Mbps burstable to 100Mbps at an overcharge of $350/1Mbps. This carries tremendous cost-risks for a solution that may spike above the 5% barrier but otherwise remain near the initial commitment level. And if you need gigabit burstable network, then costs effectively double from there. By their own admission, managed hosting providers currently lack support for on-demand scaling and are inherently unequipped to compete in these scenarios; not to mention they would require a minimum 12-month commitment that would discourage event-based sites campaigns that are active for only a portion of the year. (Limelight also has a minimum 12-month commitment, but they offer so-called ‘bucket pricing’ that gives you a fixed $/Gb rate that can be used for shorter durations.)

Rackspace Cloud (Formerly Mosso)

We looked briefly at Rackspace for their strategic partnership with Limelight Networks. While access to the full-range of the Limelight reach (60 points-of-presence globally) at .22/GB is compelling, we had no choice but to dismiss this option for two reasons: lack of CNAME support and lack of access for direct uploads to Limelight. That we would have to upload to Limelight via Rackspace introduced a new point of failure in our solution that relies on a frequent update schedule. Others have also highlighted the lack of SSL support with Rackspace [12].

Limelight Networks

I’d also like to take this opportunity to acknowledge the unusually strong first impression that Limelight customer service had on me; these guys are a class-act all the way, from quick turn-arounds to their willingness to accommodate our specifics and for not being afraid to refer us to resellers if the scenario warranted it.

Takeaway

Performance-wise, Limelight is head-and-shoulders above the field surveyed here [12] [13] (by one report ~4.2x faster than CloudFront in US/Europe, ~1.9x faster in Asia and ~3.1x faster globally). If CNAME and SSL aren’t necessary, than Cloud Files is a compelling access point to this network. I think all CDNs could support read-only (i.e. HTTP GETs) sites just as easily; if you need to support POSTs, you have to figure out a way to pass those directly to the source server. Released 3-4 months ago, it seems that LimelightSITE might be specifically geared towards this scenario. Akamai also offers a WhitePaper that describes their take on the same concept (scaling dynamic applications):

Aside from the J2EE “EdgeComputing” support, I think these options are both geared towards scaling the serving of content (even if it’s personalized) and not scaling the processing of data (like an Azure or EC2).

More Info:

If you’re at all interested in keeping up with these subjects you can also subscribe to www.cdnevangelist.com or www.businessofvideo.com for excellent coverage.

References:

[1] – Tools for Amazon’s CloudFront (CloudBerry S3 Explorer & S3 Fox, a FireFox Plug-in)
http://paulstamatiou.com/how-to-getting-started-with-amazon-cloudfront
http://troytolle.blogspot.com/2009/04/tips-on-using-amazon-cloudfront.html
http://www.labnol.org/internet/setup-content-delivery-network-with-amazon-s3-cloudfront/5446/
http://cloudberrylab.com/
http://www.s3fox.net/

[3] – Simple CDN
http://www.simplecdn.com/savings
http://www.simplecdn.com/solutions

Lightening will syndicate content that you host vs. StormFront charging for $149/GB or $.15/MB.

[4] – Amazon CloudFront vs. CacheFly - Performance Benchmark at Sinopop.net
http://sinopop.net/2008/11/20/quick-review-amazon-cloudfront-vs-cachefly-vs-amazon-s3/
http://74.125.95.132/search?q=cache:IVP0f4NsZKAJ:sinopop.net/2008/11/20/quick-review-amazon-cloudfront-vs-cachefly-vs-amazon-s3/+quick-review-amazon-cloudfront-vs-cachefly-vs-amazon-s3&cd=1&hl=en&ct=clnk&gl=ca

[5] – Cache-Control Headers
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=30969&tstart=60
http://blog.bigcurl.de/2008/11/amazon-s3-save-money-by-setting-cache.html
http://www.redmonk.com/jgovernor/2008/11/28/amazon-cloudfront-simple-caching-and-naming/
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=123129&#123129

[6] – CloudFront vs. SimpleCDN (Community Commentary)
http://news.ycombinator.com/item?id=369366

[7] – CDN Pricing Pressure
http://blog.streamingmedia.com/the_business_of_online_vi/2007/11/pricing-pressur.html

[9] – Q: Where are the edge locations used by Amazon CloudFront?
http://aws.amazon.com/cloudfront/faqs/#Where_are_the_edge_locations_used_by_Amazon_CloudFront

[10] – The Microsoft CDN Case Study
http://cloudpundit.com/2008/10/16/the-microsoft-cdn-study/

[11] – ListBucketResult: reply for a CDN request to null
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=108789&tstart=0

[12] – Cloud Files (Rackspace Cloud)
http://stupidsucks.com/2009/05/13/amazon-cloudfront-s3-vs-mosso-cloud-files/
http://techhui.ning.com/profiles/blogs/cloudbased-content-delivery
http://blog.mosso.com/2009/02/a-quantitative-comparison-of-rackspace-and-amazon-cloud-storage-solutions/

[13] – Cloud Files vs. Cloud Front (Performance Reports from Pingdom)
http://www.pingdom.com/reports/nju8qlu8micn/

[14] - Microsoft announces Azure pricing, details
http://news.cnet.com/8301-13860_3-10285904-56.html

Exception-Driven-Development: What are you doing with your Exceptions?

noreply@blogger.com (Nariman Haghighi) — Fri, 19 Jun 2009 19:26:00 +0000

Fantastic article by Jeff Atwood, of Stack Overflow [1], on Exception Driven Development – some highlighted excerpts:

“If you're waiting around for users to tell you about problems with your website or application, you're only seeing a tiny fraction of all the problems that are actually occurring. The proverbial tip of the iceberg.

…

The first thing any responsibly run software project should build is an exception and error reporting facility

… Our exception logs are a de-facto to do list for our team

… Broad-based trend analysis of error reporting data shows that 80% of customer issues can be solved by fixing 20% of the top-reported bugs. Even addressing 1% of the top bugs would address 50% of the customer issues. The same analysis results are generally true on a company-by-company basis too.

… Although I remain a fan of test driven development, the speculative nature of the time investment is one problem I've always had with it. If you fix a bug that no actual user will ever encounter, what have you actually fixed? While there are many other valid reasons to practice TDD, as a pure bug fixing mechanism it's always seemed far too much like premature optimization for my tastes. I'd much rather spend my time fixing bugs that are problems in practice rather than theory.

You can certainly do both. But given a limited pool of developer time, I'd prefer to allocate it toward fixing problems real users are having with my software based on cold, hard data. That's what I call Exception-Driven Development. Ship your software, get as many users in front of it as possible, and intently study the error logs they generate. Use those exception logs to hone in on and focus on the problem areas of your code. Rearchitect and refactor your code so the top 3 errors can't happen any more. Iterate rapidly, deploy, and repeat the process. This data-driven feedback loop is so powerful you'll have (at least from the users' perspective) a rock stable app in a handful of iterations.”

Side-stepping the implementation details (I personally haven’t see strong justification in favour of using anything beyond Enterprise Library + MSMQ + Custom Database), the value is really in what you collect and how you store it rather than the particulars of your approach (whether it be log4net, EL or ELMAH).

At a minimum, the following information should be available in indexed columns for querying:

IP
Web Server
DB Server
Message
Time
Severity
SessionId (ASP.NET’s – crucial for correlating activity)
File Name (Class)
Method Name
Line Number (IL Offset)
Url

Next, the exception should have attached to it, in some form, a set of Extended Properties (typically stored in a different table under a CLOB):

Full Stack Trace (customized, in some instances)
Original Request Headers (all, especially cookie)
Request Total Bytes
Http Method
Url Referrer
ASP.NET Request Cookies (differ from header cookie, if changed)
Form Data
Session Data (particularly IsNewSession)

For session and form data, you have to exercise some caution in putting domain-specific rules in place to avoid unnecessary data (like ViewState, for example; or DataSets stored in Session). It’s prudent to plan on keeping the indexed data for a minimum of 6 month, if possible, and CLOB values for half of that period.

Being able to look back on trends is crucial; if you have time to implement a more elaborate warehousing strategy, all the better. But if you can’t answer basic questions like:

What are the top trending exceptions in the last hour, day, month, or week?
Is the error isolated to one web server or wide-spread? What about database servers?
Are there trends by IP?
Did the user see a Yellow Page of Death or was it a behind-the-scenes exception?
What was the sequence of exceptions for a particular session (series of requests leading up to it)?
What cookies did a request start with; what cookies were assigned, prior to the exception?
What were the form, header values that generated a particular exception?

… then you really should stop development and reconsider your bearings.

[1] – Jeff Atwood: Exception Driven Development
http://www.codinghorror.com/blog/archives/001239.html

[2] – More on ELMAH
http://www.hanselman.com/blog/ELMAHErrorLoggingModulesAndHandlersForASPNETAndMVCToo.aspx
http://code.google.com/p/elmah/

Object Inspection	Examining code and stacks
DumpObj (do) DumpAllExceptions (dae) DumpStackObjects (dso) DumpHeap (dh) DumpVC GCRoot ObjSize FinalizeQueue DumpDynamicAssemblies (dda) DumpField (df) TraverseHeap (th) GCRef	Threads (t) CLRStack IP2MD U DumpStack EEStack GCInfo COMState X SearchStack
Examining CLR data structures	Diagnostic Utilities
DumpDomain EEHeap Name2EE SyncBlk DumpASPNETCache(dac) DumpMT DumpClass DumpMD Token2EE EEVersion DumpSig DumpModule ThreadPool(tp) ConvertTicksToDate(ctd) ConvertVTDateToDate(cvtdd) RWLock DumpConfig DumpHttpRuntime DumpSessionStateConfig DumpBuckets DumpHistoryTable DumpRequestTable DumpCollection(dc) DumpDataTables GetWorkItems DumpLargeObjectSegments(dl) DumpModule DumpAssembly DumpMethodSig DumpRuntimeTypes PrintIPAddress DumpHttpContext DumpXmlDocument(dxd)	VerifyHeap(vh) DumpLog FindAppDomain SaveModule SaveAllModules(sam) GCHandles GCHandleLeaks FindDebugTrue FindDebugModules Bp ProcInfo StopOnException(soe) TD Analysis Bl CheckCurrentException(cce) CurrentExceptionName(cen) ExceptionBp FindTable LoadCache SaveCache ASPXPages DumpGCNotInProgress CLRUsage