How can you read other people’s code?

• Read your team members code
• Read (& refactor) your old code – you probably forgot what you did already
• Read books on software & algorithms
• Participate in open source projects
• Read blogs on software
• Write a programming blog

Getting latest from TFS:

"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\tf.exe" get

(use /preview to preview)

Getting latest from Subversion:

"C:\Program Files (x86)\VisualSVN Server\bin\svn.exe" update "F:\web\Beta"

Building a Visual Studio solution via MSBuild:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe D:\Mises\MisesWeb.sln /p:configuration="Debug"

Checking whether the build succeded:

if %errorlevel% neq 0 exit /b %errorlevel%
REM BUILD OK, GO TO NEXT STEP

Copying build output:

xcopy F:\web\Beta\MisesWeb\bin\* F:\web\Mises\bin\ /Y
REM /Y don’t prompt for overwrite
REM /S recursive
REM /Z restartable mode
REM /D only copy files newer than those at destination
REM More: http://www.computerhope.com/xcopyhlp.htm

How I fix bugs with unit tests:

1. Write a unit test that reproduces the bug so the test fails
2. Fix the bug so the unit test passes
3. Send a link with both to the bug submitter

Why I like it:

• Saves time debugging – just run the unit test instead of recreating the bug scenario every time
• Creates a simple yes/no criteria to verify whether the bug is fixed
• Regression can be detected quickly in the future
• Better communication: bug is documented for other developers looking at this code
• If you don’t have time to fix the bug immediately, write a failing test.  Then you or someone else can fix the bug later
• An organic way to increase code coverage for existing code

The Mises.org Media Management System is part of the Mises.org open source project.  It was started around 2004, and evolved slowly from there, with the latest release around February.  It is used to manage over 6400 audio/visual files.  Total document size is 275GB.  Monthly bandwidth exceeds 10 terabytes.

The front-end features:

• Dynamic, AJAX-based interface build with JQuery UI
• Advanced search and browsing based navigation
• Detects the best media player for the current browser: HTML5, Flash, or Silverlight
• Extensive RSS support.  Integrates with iTunes and iTunes University with iTunes-specific feed extensions
• Optimized for scalable deployment, CDN’s, SEO, external embedding, and social network sharing

The media asset management includes:

• Support for internal and externally hosted content: MP4, MP3, WMA, streaming video, & YouTube
• Integrated drag and drop uploader: management interface allows uploading multi-gigabyte media files straight from the browser us
• Meta data parsing: reads full meta information such as embedded thumbnails, duration, keywords, description, etc.  Able to process all popular audio/video formats as well as PDF.
• Catalog builder:  organizes files in a human-friendly format: http://library.mises.org/media/
• Define arbitrary hierarchical categories
• Versioning and multi-volume support.

Architecture:

• Backend: ASP.Net 4.0, MVC3 (frontend) & Web Forms (backend), Entity Framework, and SQL Server
• Frontend: JQuery UI, HTML5,

How to get it for your site:

• It’s part of the Mises.org CMS
• See http://wiki.mises.org/wiki/MisesWiki:Development for details

Screenshots:

Filtering UI:

Media list:

Light box popover players:

Media detail:

Category browse UI:

Category detail:

RSS feed:

Admin UI – detail page:

Alternative UI:

File system storage:

iTunes U interface:

How I implemented automatic SSL switching:

• SSL proxy: We use CloudFlare as an SSL proxy. Even though we already had SSL configured on our Windows server, CloudFlare makes it easy to share SSL certificates between Windows and Linux servers, which is otherwise quite tricky. But it’s also great if your web server does not have SSL support.
• Relative (scheme-less) Urls: I removed the scheme from all our resources, so they look like <img src=”//images.mises.org/Theme/images/bf_lvmi.png” alt=”" />  We are transitioning to scheme-less URLs for everything.  It allows users to choose HTTPS even if your site is served over HTTP by default.
• Request.Url.Scheme: I used to append the URL scheme used by the request when I was not sure if the client (rss reader, email, templates etc) would support scheme-less URL’s. But this has the disadvantage of not working with cached pages. So far, it seems that scheme-less URL’s are safe for all popular web clients.

I don’t think there was ever much sense in having the http://www. prefix in URI’s, so getting it down to just // is very nice.

How to find non-SSL resources on a page:

One frustrating debugging step for me used to be finding the http resources within a page. I would have to search for http resources in the source code. If the page references scripts that load other resources, this could get complicated. Fortunately, Google Chrome offers a simple solution – the network inspector can export information to a searchable JSON file:

1. Open a new private browsing session in chrome (to disable your extensions, which may inject non-SSL links within the page)
2. Alt-click and “Inspect element”
3. Go to the Network tab
4. Open the URL to inspect
5. Right click inside the Network tab and “Copy all as HAR”
6. Paste the JSON into notepad, and search for http:// resources

One last note: If you use third-party widgets or services which do not fully support SSL, please take the time to contact the vendor and request that they add full support for SSL.

In other words, we are facing the equivalent of the HD TV transition for personal computing. It will come to smartphones, tablets and then desktops. Speculating about the timeline of the transition is of course risky, but I suspect that there will be many high-DPI devices being sold by the end of 2012 – not limited to Apple. Certainly within a few years, we can expect the rest of the industry to follow. Because websites and applications can display content appropriate to the viewer’s hardware, they can gain a competitive advantage without imposing costs on non-HD visitors.

Currently very few websites and desktop applications take advantage of the high-DPI displays. While Apple has demonstrated how it can be done, the specific browser-side implementation may change. For most websites and applications it’s probably too early to invest resources in the transition. However, it is not too early to start updating your creative process to include high-definition graphics and video.  If you decide to upgrade your website six months or a year from now, it will be easier to use pre-prepared HD graphics than have to re-create them from the original PSD files (which may not be easily available.) Apple’s implementation makes it possible to transition to high-DPI by adding a single JavaScript file to your website – without increasing the page size for current users.

What this means in practical terms:

• You should start preparing double resolution graphics and video for new content, especially important elements like headers & front page content.
  • Example: http://images.website.org/header.png should be complemented by http://images.website.org/header_2x.png
• If you target iPad or iPhone users, you should already be updating your website’s technology to support HD graphics & video.
• When initiating projects going forward, you should give increasing consideration to adding support for high-DPI resolutions.

• Instill a sense of professional pride
  • Developers should take part in the technical design
  • Provide interesting, challenging work
  • Developers should own the technical solution
  • Technical architects should be part of the development team
• Respect the developer time
  • Provide a quiet environment dedicated to development work
  • Buy the best tools for the jobs – powerful computers, large monitors, etc.
• Reward learning and exploration
  • Take (reasonable) risks with new technologies
  • Schedule time for self-education and information sharing
• Create a sense of project ownership
  • Delegation “ownership” of functional parts to individual people
  • Assign developers responsibility for follow-up maintenance (don’t just hand it off to a maintenance team)

• Hold people accountable for their work:
  • Use a work unit tracking system (TFS, etc)
  • Make task status publicly visible
  • Hold daily stand ups (Scrum)
  • Consider a public scrum board
  • Provide clear project requirements
  • Lean development approach
    • No time-wasting tasks (useless documentation)
    • Isolate developers from unrelated tasks (no business interruptions)
• Hold high expectations
  • Monitor quality of work with code reviews
  • Regular training & information sharing sessions
  • Consequences for bad work
• Set well defined deadlines
  • Don’t set arbitrary deadlines, but all work should have a deadline
  • Developers should have input on deadlines for their work units

• Instill a sense of collective ownership of the project
  • Hold architectural education sessions
• High-bandwidth communications
  • Entire team should work in physical proximity
• Make quality visible
  • Use continuous integration and automated testing to provide immediate feedback of quality
  • Developers should fix their own bugs
  • Track quality over time
• Explicit mentor roles

• Mentor roles should be explicitly defined
• Mentoring time should be scheduled into the project

 Why choose pure-client side?

My current development approach is MVC 3 “enriched” with the Telerik Extensions for ASP.Net MVC.   ASP.Net MVC HtmlHelpers are much simpler than the traditional .Net drag-and-drop Web Forms approach in that they simply return strings, without the baggage of ViewState management or a complex page event lifecycle.    This provides certain advantages: lightweight rendering, granular state management and simple AJAX – along with auto-generated HTML and IntelliSense.  But any developer-friendly server-side technology also has costs: server-processing time affects performance and flexibility, and server-side controls prevent you from having a true rich-client design which fully separates the user interface and business logic.

Presenting a new alternative to server-side MVC controls, are several client-side UI frameworks.  Much of the rich client movement comes from the open-source world with frameworks like SproutCore/Amber.js, YUI and Cappuccino.  But now that Microsoft is embracing HTML5, the two heavyweight .Net tool vendors are getting on-board as well.

There are three competing HTML5 UI/Ajax UI frameworks I want to mention.  They are pure-JavaScript libraries built on top of jQuery.

jQuery UI: 

jQuery UI provides a bunch of client-side UI widgets, “interactions” and effects.   It is semi-sponsored by industry bigwigs which include such as Microsoft and Adobe.

Pros: From the same team which brought you jQuery.  There are many plug-ins for missing functionality.  Theming is very nice. (Btw, Juice UI is server-side jQueryUI for ASP.Net WebForms – perhaps an intermediate solution?)

Cons: Key components are not included with the framework, such as Grid, Template, DataSource/DataView, Validation.

Cost: Free for commercial and non-commercial use: MIT and GPL licenses

 Telerik Kendo UI:

Kendo UI is a fork of Telerik ASP.Net MVC framework.  The first version of Kendo UI has just been released, but the client-framework is based on the two-year old Telerik MVC framework.

Pros: Complete feature set, with 16 widgets (vs. 8 for jQuery UI), including everything missing from jQuery UI; Visualization; Mobile versions of widgets;

Cons: Just released late 2011; is not jQuery UI;

Cost: GPLv3 and Commercial licenses; closed-source code requires purchasing the commercial license which starts at $399/1yr

ComponentOne Wijmo:

ComponentOne is Telerik’s arch competitor.  Their client-side offering is an “extension” of jQuery UI, and so relies on both jQuery and jQuery UI.  It is more mature (version 2.0) and they are sponsoring jQuery UI development.

Pros: mature; sponsor of jQuery UI development; works well (shares theming) with jQuery UI;

Cons: Free version comes with a subset of controls; replaces existing jQuery UI widgets

Cost: “Open” version with 18 widgets is free (MIT/ GPL v2).  “Complete” version with Grids, Charts, Forms, etc is $299-499

I’m still getting started with this approach, so I’m not sure which framework is best.  They are all good alternatives to a server-side framework (whether Web Forms or MVC).  They all have a learning curve and are probably less efficient from a developer perspective – especially one used to writing server-side controls.  On the other hand, they enable a different development methodology (isolation of UI and the service stack) which may be more efficient overall and better for performance and scalability.

My tentative suggestions:

•  jQuery UI for open-source, small projects, low budgets, simple or non-form heavy UI, and minimal support or learning-curve availability
• Kendo UI to leverage Telerik MVC expertise and maintain compatibility with MVC
• Wijmo when a rich UI is needed as well as compatibility with JQuery UI or the open-source world

Here is a Google search widget for the OS X Dashboard that doesn’t suck — or at least sucks less than what I was able to find.

Google.wdgt

Is it lame that I am posting a dashboard widget in 2012 when they have been totally abandoned by Apple?  Maybe, but I think it’s lame that Apple has abandoned dashboard, as I still use it every day.  Or is there some announcement that I missed?

1: Visualize your users with analytics, personas, and user stories

The most important consideration in web design is to meet the needs of your users. There are two aspects to this:

First, collect statistical data on the demographics and usage patterns of existing users. Use web analytics, (GA), in-page analytics (GA events, crazyegg) to understand exactly how users use the site, what content is popular, and how that relates to revenue goals.  There is no need to rely on hunches and guesses when the relevant information is easily available.

Second: visualize the kind of customers you have and want to have with personas and user stories. Understand their goals and visualize the best way to meet them.

2: Steal ideas from the best

Web design consists mostly of weaving together design patterns from other projects. Coming up with entirely new design paradigms is very rare, difficult and risky. With that in mind, how can we choose the best design patters for a projects? By systematically borrowing from other successful designs.

One of the nice things about web development is that it is much easier to copy successful designs than in other fields. So why not pick the very best websites on the web? There are many websites which have spent millions on market research and high-end design firms. I systematically research successful designs and borrow the best elements from each. Many people make the mistake of only looking at their competition for inspiration.  But looking just like the competition won’t inspire users and limits you to a very narrow sample.

Here are three ways to find sources for inspiration:

• Look for winners of web design awards
• Frequent web design blogs & magazines
• The easiest way: go straight for the websites with the biggest design budgets.

For #3, I sometimes use Wikipedia as a resource to find proxies for design budgets and just go down the list. For example, I use the list of the wealthiest charitable foundations or the largest market caps.  I go straight down the list and note the designs I like.
For example, #5 – the Ford Foundation has a very effective design.  The useful information we can gather goes far beyond visual elements.  By researching their site, press releases and job ads, I was quickly able to identify their technology platform, content management system, the person in charge of their information technology (technical people at non-profits are often willing to share expertise) and some of the contractors they hired to work on it.  The cross-section of people visiting the Ford Foundation and your site is likely to be very small, so you can borrow more liberally than you could from a competitor.

3: Develop a design language

A “design language” is a set of visual concepts which we can use as building blocks for a web design. A design language helps us share ideas visually and then create consistent look and feel for the site, even if multiple people or multiple iterations are involved. A coherent design pattern is very important for a site. I suggest something like a Google Docs document with screenshots of elements from other sites and mockups. Eventually we can weed it down to unique elements representing key building blocks: header, menu, tabular data, form, product list, etc.

A design language is also important for controlling costs, as it is cheaper to pay a designer to create a reusable set of elements than to pay them to create each page from scratch.

4: Aim for the web technologies of the future

The lifetime of the average web design is about three years, though many last much longer. Three years is a long time in web design – paradigms come and go very quickly, as does the need to respond to competitors. That’s why it’s important to find the right balance between targeting emerging technologies while controlling risk. For example, most people don’t have smartphones with full web browsers, but they are likely to in a year or two. It thus makes more sense to invest in full-featured mobile sites and not the last-generation WAP version. At the same time, we don’t know if the Apple, Android, or Windows Phone platform will be most popular in the future, so it’s best to build mobile web versions than native apps in today’s hottest platform.

CPU time

CPU time is the percentage of clock ticks than a processor spends waiting for instructions. This is opposed to wall-clock time, which is the total time the whole computer takes to perform an operation. The “load” on a system is the ratio of clock ticks which are performing operations versus the click ticks spent waiting for instructions over a given time period. Thus, load only makes sense as an average over a particular time period. During any single clock tick, the CPU is either processing an instruction or a HLT.

Idle time

The CPU processes a set of instructions fed from memory. The memory contains a set of opcodes (commands) which tell the CPU which operation to perform and the memory where it is located. The rate of clock ticks is constant – several gigahertz in a modern CPU – and during each clock tick, the CPU processes either an instruction to do some work, or a HLT – an opcode which tells the CPU to turn keep components idle until the next cycle.

Monitoring CPU time

So if we were to “observe” a CPU at the clock-tick level of detail, we would see that it’s always either working or waiting. But we can’t actually do that, since the more closely we monitor clock cycles, the more clock cycles are needed to do the monitoring. So to get an accurate picture of activity, we have to step back to a level where the monitoring tool does not interfere too much with the process being observed.

Input/output overhead

Any given computer task has several components: CPU instructions, memory IO, disk IO, network IO, and many others. Only extremely simple tasks (like calculating π) can fit in the small (but very fast) memory buffers on the CPU itself. Real-world tasks will almost always require the CPU to wait for other components to finish shuffling data back and forth in a state where it is ready for the CPU to work on it. But the ideal scenario is for the CPU to be kept as busy as possible (constant 100% load) until the task is completed. This is the minimum possible time in which a task can be completed. If we were to monitor CPU activity during such a task, we’d see the load jump to 100% during the task than back to the baseline when it’s done.  But if the task is shorter than the measurement frequency of the CPU monitoring tool, it would be an average over two periods, or it might not be detected at all.

Multitasking

The situation is more complicated when there are multiple tasks to run, each off which requires some fraction of CPU time. The operating system will run the tasks concurrently. Both the processing time and the IO of tasks will overlap, but because the operating system takes turns running each task, the total CPU load will be some combination which may be higher or lower than the total of the individual tasks – depending on the other competing resources the tasks use. For example, two disk-intensive tasks will use less CPU than the sum of both running individually because the disk IO will be the bottleneck. But two CPU-intensive tasks would more than double total load because the CPU will have to run both tasks and have to handle the context switching between concurrent tasks.

Further reading

• http://en.wikipedia.org/wiki/CPU_time
• http://en.wikipedia.org/wiki/Load_(computing)
• http://en.wikipedia.org/wiki/No_op
• http://en.wikipedia.org/wiki/Idle_task
• http://en.wikipedia.org/wiki/Computer_performance

An ORM by definition has to do some work to convert a database schema into a native view. ORM’s can try to minimize this performance penalty by defining the transformation at design time or caching the database to native object transformation at different points.

Entity Framework is built on top of ADO.Net, which is just an API that does not “know” anything about the database. To convert .Net code into SQL queries and back into CLR objects, Entity Framework performs a set of operations at different stages: (1) compile time (2) first run (3) each execution. To improve performance, EF allows you to shift some work from step 3 to 1 or 2. But the details can be tricky and understanding the best optimization strategy requires understanding the EF query execution pipeline.

The EF execution pipeline

The following information is based on this MSDN EF Performance page.

There are six steps I want comment on in EF execution:

1. loading metadata
2. generating views
3. preparing the query
4. executing the query
5. tracking
6.  materializing objects

1: Loading metadata: The metadata is the mapping defined in your EDMX file. This is a very expensive operation (see the breakdown here), but it only happens once. EF applications use more memory and have a warm-up penalty which should be ignored in performance analysis if you are not concerned with startup times.  Models with with very large (200+) number of entities have a number of problems – see this and this.

2: Generating views: The local query views are static objects which are cached per application domain. This is also an expensive operation but it can be pre-generated and embedded in the application if you care about first run times.

3: Preparing the query: Each unique query must be compiled into the EF version of a stored procedure before it is executed. Microsoft says that the commands are cached for later executions, but I’m confused about what exactly is cached because profiling shows that the query is compiled every. In any case, Microsoft suggest caching the compiled query to avoid this penalty.

Caching precompiled queries is somewhat unwieldy, so it is only advisable in performance-critical contexts, but it makes a big difference for frequently executed queries.

(Note: Take care when using .Count() or Any() to avoid unnecessary query recompilation and avoid unnecessary enumeration when a simple boolean check will do.)

By the way, all the steps above can be skipped by using direct SQL queries or stored procedures with EF. The performance of direct entity SQL falls between pure ADO.Net and Entity Framework queries, so it is only useful as the occasional exception to LINQ queries against a data store.

4: Executing the query: Query execution time depends on the underlying data source. Entity Framework is just a library built on top of ADO.Net, so pure ADO.Net queries are the a benchmark for any ORM’s built on top of it. Because, ADO.Net will always be faster than any framework built on top of it, it can be used as a fallback when other options have been exhausted.

5: Tracking: tracking is used to track changes for updates. If you only need to read data, you can get a small performance boost by disabling it.

6: Materializing objects: each object returned from the database must be converted into a class instance to be used. There is no way to avoid this penalty, but it is worthwhile to keep in mind that the less data there is, the faster it will materialized – not to mention transferred over the wire.

For best performance, queries should be as specific as possible. I have noticed that ORM’s encourage the bad habit of always selecting an entire row. This is because developers using an ORM tend to think of the database as an object repository rather than as a relational store, whereas raw SQL queries encourage manually selecting the needed rows. (Unless one has the awful habit of “select *”.) So to minimize overhead, pull just the data you need (the MVVM pattern is helpful in this regard.)

Final thoughts:

As the ADO.NET program manager himself has pointed out, Entity Framework is inherently slower than ADO.Net SQL queries. But performance should always be balanced against productivity. There are some applications which are definitely not suitable for an ORM and many others that are. Some operations can only be done using raw SQL and can take forever using an ORM (“truncate table”, temp tables, etc).  I think the best approach is to use an ORM where appropriate and optimize in the specific scenarios where performance is inadequate. I do think that Entity Framework is the best, simplest, and the safest choice out of all .Net ORM’s.

These are just a few tips on Entity Framework performance. There are other tips and much more information in the pages below and the links therein.

More reading:

• MSDN: Performance Considerations (Entity Framework)
• Entity Framework FAQ: Performance
• The Code Project: Performance and the Entity Framework

Further reading:

• Introducing BDD
• A Set of Unit Testing Rules

From a recent presentation: 

<httpcompression directory="%SystemDrive%\inetpub\temp\IIS Temporary Compressed Files" maxDiskSpaceUsage="1024" noCompressionForHttp10="false" noCompressionForProxies="false">
      <scheme name="gzip" dll="%Windir%\system32\inetsrv\gzip.dll" dynamicCompressionLevel="9" />
      <scheme name="deflate" doStaticCompression="true" dll="%windir%\system32\inetsrv\gzip.dll" dynamicCompressionLevel="9" />
      <dynamictypes>
        <add mimeType="text/*" enabled="true" />
        <add mimeType="message/*" enabled="true" />
        <add mimeType="application/x-javascript" enabled="true" />
        <add mimeType="*/*" enabled="true" />
      </dynamictypes>
      <statictypes>
        <add mimeType="text/*" enabled="true" />
        <add mimeType="message/*" enabled="true" />
        <add mimeType="application/javascript" enabled="true" />
        <add mimeType="application/pdf" enabled="false" />
        <add mimeType="application/x-zip-compressed" enabled="false" />
        <add mimeType="text/css" enabled="true" />        
        <add mimeType="*/*" enabled="true" />
      </statictypes>
    </httpcompression>

For some reason text/css was not compressed by default for me and I had to add it.