flowmarks

PowerPivot and SQL Azure

2011-10-05T02:41:00.000-07:00

When PowerPivot was released, I was sceptical. Yes, it could be a great tool for in-house reporting, but not so with a SaaS application. My experience was that SaaS reporting should be embedded in the application. Maintaining a separate reporting systen such as SSRS, or even Telerik Reporting, in addition to the main application would be too much work. And it turned out that a ASP.NET ROLAP control like DevExpress ASPxPivotGrid would be good enough for both standard reports and custom reports, eliminating the need for another reporting system.

This reasoning applies if there are extra costs involved in supporting a new system. PowerPivot, however, is purportedly a self-service system. If it doesn't impose new costs, why not support it ? Unfortunately, even if there are no report authoring costs, there are still system-level problems that have to be solved before PowerPivot can be used in the public internet. First, you need to configure networking to allow secure access to your data. A direct SQL connection to your database is not always a good idea, and data services can take too much time to develop and set up. Once there is access to data, the next problem is to make sure that your database system scales up to the new demand. Alternatively, you may want to implement resource management so that PowerPivot doesn't hog up too much resources. These are hard problems for a small SaaS provider, but interestingly, they are also problems that Azure promises to solve.

With my simple application running in Azure, I was all set to experiment with PowerPivot. The easiest way to use PowerPivot with Azure is to connect directly to a SQL Azure database. This is done in two steps:

In Azure management portal, add your current IP address to the firewall rules for the SQL Azure server.
In PowerPivot, connect to the database. There's a little catch: Ordinary database connection didn't work, PowerPivot just crashed Excel. You need to use "From Other Sources" - "Microsoft SQL Azure"

And there you have it, flexible PowerPivot access to your data. I'm impressed how well it complements web-based reporting. I've used it to answer questions about the data for which the PivotGrid does not adapt easily, such as percent-of-total calculations. The main restriction with PowerPivot is that it suits best for power users with full access to data. Setting up views for row-level security is too much hassle in most cases.

-mika-

Moving to Azure

2011-09-04T06:41:00.000-07:00

Today, I moved my small application to Azure. Rarely have I been this excited. Azure reminds me of the days when I was fifteen and Netscape Navigator was just released. I spend nights in my friend's place browsing the web with a 14kbps modem and a 486. The internet was nothing it is today, but it was exciting.

In my professional career, I have had to run dozens of times to a server facility, installing new servers, putting out fires on existing ones and so on. I have used multiple different customer VPN connections to different server environments, each with its own peculiarities. The hardware costs are reasonable, but the management burden is getting out of control as the business grows. The worst thing is that the quality of system administration is not what it should be, when developers like me work as accidental server, network and database administrators.

Azure promises to fix a lot of this. It is not perfect, and my skills are not yet fit to this new environment. Azure costs can run high compared to a cheap server. I'm more afraid of mistakes, downtime and data loss, but have less control to fix things. There is no simple SQL Azure backup system. I'm not convinced that I could recover a broken webapp fast. At least there is Remote Desktop now.

But once I got my simple app out of a shared physical server to its own corner of the Azure cloud, I knew there is no way back.

-mika-

Flowmarks Events

2011-03-18T08:53:00.000-07:00

Ten years ago I used Excel heavily. The usage pattern was common. What started as a simple spreadsheet quickly evolveld into an information system. Mostly a good information system too: Excel was extremely fast and flexible. The problem with Excel was that a file-based information system worked well only on a local machine with a single user. Once you needed remote access to your data with write privileges, things started to fall apart.

Today, everything worthy of an information system is in the web. Money, time, sports, health.. you name it. Besides remote access and concurrency, webapps offer benefits of specialization that Excel can never do. Money and time management webapps support automated data entry. Sports trackers encourage sharing in social networks. Nutrition and health apps have expert system features like recommendation engines. In specialized applications, out-of-the box reports are usually relevant too. In Excel, you have to define calculations and charts yourself. And then there is the whole smartphone support thing.

But even with all theses improvements, there was something I was missing from the Excel days. Call it versatility and freedom. Compared to Excel, webapps have significant drawbacks:

setting up the account, credential management for each app
learning a different UI for each new app, tolerating with what you get
not having complete control over your data, data loss in the worst case
limited reporting, you can't define the calculations and reports you want
bad performance, downtime
subscription costs, or a free service with no rights to your own data

These drawbacks are acceptable if the payoff is there, but especially with smaller datasets, the benefits may not be worth the costs. Personally, I wanted to track small, ad hoc time series, with pivot table reporting. I didn't really want to use another isolated webapp for this, so I rolled up my own to handle them all.

It is called Flowmarks Events.

You can download it from CodePlex, or just use it at http://events.flowmarks.com.
There is also little documentation and a read-only demo (with book reading data) that has no login requirement.

The events app is a DotNetNuke module. Unfortunately, for licensing reasons, I can't distribute the reporting module that is available in the website.

-mika-

Diving Into HTML5

2011-02-05T06:03:00.000-08:00

Early this year I was working on a user interface refresh and refactoring project with an ASP.NET Web Forms application. In the process I read a great book by Mark Pilgrim, Dive Into HTML5. It presents a nice overview of the new features in HTML5. The most important thing I learned from the book was that HTML5 is something that deserves thought today, not in some distant future.

Data entry forms make a good example. HTML5 provides multiple new web form input types and attributes. In essence, HTML5 has many of the input types and attributes built-in that you would end up adding anyway, using javascript libraries like jQuery & plugins or server-side proprietary controls such as Telerik or DevExpress (for ASP.NET). These include the placeholder and the autofocus attributes for textboxes, specialized input types for common data types (date pickers, email address fields, number selectors like spinboxes and sliders), and also client-side validation support for required fields and email addresses. Many of these are already supported in the latest browsers and should degrade gracefully in older browsers, including IE6.

Yet, there are two problems. First, in order to provide a consistent user experience, you'll need scripted fallbacks for older browsers. Detecting HTML5 support and implementing fallbacks adds extra work. Personally, I would be surprised if more than a tiny fraction of websites implemented fallbacks comprehensively. And this is something that affects UI development for a long time. I would predict that fallbacks could be dropped at earliest in about 2015 when new machines are shipped with Windows 8 and the great majority of browsers supports enough of the HTML5 features.

Second, what if you are using proprietary server-side controls? The situation is unclear, and it is up to the vendors how and when HTML5 is supported. The problem is that using HTML5 features and control's own similar features together, without HTML5 detection and fallback, may cause clashes. For example, I found that HTML5 placeholder and Telerik RadTextBox EmptyMessage didn't play well together in Chrome 8. Consequently, I wouldn't use HTML5 features together with similar server control features until compatibility is assured.

Frankly, the more I think about it, the less it seems possible that HTML5 could ever replace the current nonstandard "advanced" input controls. Advanced controls are not just interchangeable standard parts, they have large and differentiated feature sets. For example, see jQuery UI Datepicker, DevExpress Date and Time Editors and Telerik RadDatePicker. Realistically, it's going to be an either-or choice between some custom control and a HTML5 standard control. And fallbacks are required only if you choose HTML5.

In the end, it is all about options. How valuable is it to use HTML5 during the lifetime of the application? Having the HTML5 option is not cheap, but giving it up means using the lame HTML4 controls or taking the possibly troublesome road of custom nonstandard dependencies. On the other hand, if your application will never see HTML5 capable browsers, or if the HTML5 features would be worse than the current ones, it makes no sense to use HTML5 at all. With these considerations in mind, I chose to use the Telerik input controls in my UI refactoring project. But if the application would not have been so dependent on ASP.NET standard controls, a solution using HTML5 and client-side fallbacks would have been appropriate too.

-mika-

DevExpress ASPxPivotGrid Performance

2010-06-20T05:44:00.000-07:00

In the previous post I introduced DevExpress PivotGrid, a web-based pivot table component. The tool itself is impressive, but I was somewhat suspicious about its performance in real-world reporting applications, and promised to look at this more. There are three potential performance bottlenecks in web-based pivot tables:

Querying data from the data source and getting the results back
Processing data in the application server
Loading and rendering the page in the browser

Query Performance

Raw query performance should be the same for every BI client. Fundamentally, the only thing a client can do to dramatically improve query performance is to cache the results. This is what in-memory engines like VertiPaq are all about. But caching leads to a tradeoff between timeliness and performance. You won't get real-time results from a cache. If your cache expires, say, once in an hour, the average staleness of your data is 30 minutes, which may or may not be too high given your business requirements.

Nevertheless, caching benefits most those source systems that are not well equipped for analytical workloads, like most databases behind line-of-business web applications. These are typically operational applications designed for many small concurrent reads and writes. They are not designed to handle analytical workloads that emphasize read performance. In analytical databases like relational data warehouses and OLAP databases, read performance is boosted by materialized views and heavy indexing, both of which are harmful to operational write performance. Typical analytical workload in a normalized, OLTP-friendly relational schema requires a lot of joins, run-time aggregations and multi-pass queries, which will be slow. On the other hand, read-optimized analytical databases do not usually contain real-time data to begin with, so there is no trade-off in using caching with them either. The message to take home is that if you intend to use PivotGrid for real-time operative reporting without using a cache, you are most exposed to the query bottleneck.

My experience is that query time from relational sources is easily the most dominating factor. Caching makes all the difference between a painful and enjoyable reporting system. Real-time reporting with slow queries is not fun to use, and users more often than not prefer fast stale data to slow real-time data. The hard problem here is to find an optimal cache invalidation policy. I won't go further, but the trade-offs involved will make you think hard about the usage patterns and requirement priorities.

Caching application data with ASP.NET means storing the PivotGrid's datasource object server-side in an application or session level store. Usually caching is done at the application level with System.Web.Caching.Cache, which has features like key dependencies to invalidate the cache. In the code snippet below, SqlDataSource1 retrieves data from cache as long as the CacheKeyDependency does not change.

Dim cacheKeyDependency = keyString
If Cache(SqlDataSource1.CacheKeyDependency) <> cacheKeyDependency Then
 Cache(SqlDataSource1.CacheKeyDependency) = cacheKeyDependency
End If
PivotGrid.DataSource = SqlDataSource1

Application server performance

In web-based analytics, report processing is centralized. With ASP.NET this means that IIS application pool (w3wp.exe) will do all the processing. Once the query bottleneck is solved, it is likely that the application server becomes the next pain point. It is not possible to improve application server performance much with output caching since pivot tables are generated dynamically and HTML output is going to change all the time.

Server-Side Performance Tests

Eventually, it is the request execution time that matters, and you must empirically measure the query time and the application server's contribution. To separate the two, I ran tests with the cache on and off using a setup similar to Bruce Chapman's tests on swiching off the .aspx page extension on DotNetNuke friendly urls. My test loaded eleven different PivotGrids with similar HTML size. HTML size was restricted by having a maximum of ten rows and less than 20 columns in each loaded PivotGrid view.

The performance measures followed can be grouped to:

latency measures: Time to first byte (TTFB), Time to last byte (TTLB)
throughput measures: Requests per second
resource utilization measures: Processor time, Memory, IO

First, caching produces an obvious performance boost. As a matter of fact, I couldn't even get comparable latency and thougput measurements without cache since IIS didn't respond requests reliably then. It is best to visualize the difference with the processor time counter in performance monitor.

Figure 1. Processor time with cache

Figure 2. Processor time without cache

With cache and 100% hit rate, five runs were performed, and the results are shown below. The average TTFB was 1730 ms, and average TTLB only ten milliseconds higher. The small difference tells that there was practically no networking latency in my localhost test setup. Throughput was at 0,6 requests per second, which no one would call scalable. On the other hand, byte throughput was at 1,4 MBit per second, which means that the average response size was as large as 0,8MB.

These results are not generally replicable since the PivotGrid data in the test depends on timestamp values. Measurements taken on a same day should produce the same results though. But strictly, this is not the case. There is some variability even in subsequent runs that complete flawlessly. Median TTFB and TTLB vary considerably, since the number of requests is small (17-18), but average values are fairly consistent. And there are runs that won't complete perfectly even with cache on.

I won't claim to be a ASP.NET performance monitoring specialist, but I think the WCAT results are unambiguous. Overall, the results tell that my PivotGrids easily generate a heavy load on the server, and scalability problems are expectable. Under optimum conditions with 100% cache hit rate I could serve less than one page per second. In single user scenarios, response times are reasonably low though. Without cache, even a single user is too much for the server.

Client-side performance

PivotGrids are fairly static, in the sense that most of the client-side performance depends on loading data, not on javascript performance. And data sizes depend on what you ask for. That said, with sensible row count restrictions and default visible columns, HTML size should be few hundred KB at most.

PivotGrid reloads when you add new fields, and dropdownlists populate on demand. I have not studied the postback mechanism yet, but PivotGrid didn't work inside a UpdatePanel and DotNetNuke compression caused hangups too. Also, if a PivotGrid field is incorrectly mapped to a SQL column name (case-sensitive!), PivotGrid just hangs when the field is needed. Overall, not being able to use some methods of compression is perhaps the most serious client-side performance drawback, since you may have to turn off compression settings that would help other pages in your site.

Update: Setting EnableCallbackCompression to false on the ASPxPivotGrid and ASPxGridView controls solved the DNN compression and UpdatePanel incompatibilitity problem. You may also have to disable built-in callbacks by setting EnableCallbacks=false.

Conclusion

DevExpress PivotGrid is a high-level tool. You can't optimize it much. Instead, optimize queries, use caching, and be careful with the default parameter values and data sizes.

-mika-

DevExpress ASPxPivotGrid

2010-04-29T07:24:00.000-07:00

Lately I've been developing a reporting module for a web application used by our customers. This reporting module is still in very early stages, but it has the basic elements you would expect in a reporting tool: parameterized standard reports, ad-hoc reports, dashboards and charting. Of course, because it is part of our application and not an external service like SSRS, integration with other modules is very tight. For example, reporting can read other modules' settings and customizations and behave accordingly. This helps a lot in creating customer-specific reports without duplicating report code.

In this project I've been using technologies I've written about before: FusionCharts Free, jQuery, Telerik RadControls and Telerik Reporting. These technologies work fine in their respective domains. FusionCharts is great for dashboard visualizations, it has cool animated charts and drill-down support. jQuery is used everywhere in the UI, but not that much on the reporting itself. Telerik RadControls suite is used mostly in the navigation and report filtering controls. And there are a handful of Telerik Reporting reports used by some of our customers.

But the most interesting thing here is the ad-hoc reporting component: DevExpress ASPxPivotGrid.

ASPxPivotGrid is an ASP.NET component for creating crosstabs dynamically from relational or OLAP datasources. It is essentially what is known as a cube browser in the industry. But undoubtedly, a lot more people would call it a "pivot table" just because Excel has a similar functionality. Pretty soon, analysts will be introduced to the new PowerPivot tool. This means that the word "pivot" is destined to become, if not a household name, then at least a corporate buzzword. In my opinion, pivoting live data definitely has spreadsheet-level implications if done right. "Right" means multiple things here: data quality, general availability, high usabilitity, performance, developer support and enterprise-level security, to name a few.

DevExpress has done a lot of things right with PivotGrid. Web platform is even more generally available than Excel. Usability seems good enough, since our users pick up the necessary skills during demo meetings. Charting and drill-down are relatively easy to integrate. Performance is decent for a web control, but sure you can kill your server and browser with data. There is no built-in security, which is understandable for an ASP.NET control. This means that visibility of data is managed by the developer. Localization and theming are harder than they should, though. I'm used to the simplicity found in localizing and theming Telerik controls, even if they do not follow Microsoft's recommendations strictly.

But the most important thing is something I should stress more. After talks with our customers, it became clear that most business value was created by pivoting and integrated charting. No doubt, we showed them everything, but OLAP-style pivoting and instant integrated charts made a lasting impression.

From the developer's perspective, pivot tables are also the easiest tools in the long run, since you get customizations out of the box, which would require a lot of work in static reports.

There is lot more to say about web-based pivoting. In the next week or so, I'll talk more about the key issue of web-based ad hoc reporting: performance.

-mika-

Full-service vs. Self-service Software

2010-03-21T02:01:00.001-07:00

I just finished W.E. Pete Peterson's Almost Perfect, a highly recommended story of the rise and fall of WordPerfect Corporation.

One thing that struck me was the distinction between the old, full-service IBM model and the emerging off-the shelf PC software model:

As large corporations made the transition from the full-service old world to the self-service new world, they were desperately trying to find a way to extract the same level of service from their vendors, even though they were paying much lower prices.. They acted like spoiled rich people who were trying to save money on their vacation by going to a two-star resort instead of the five-star resort they were used to. In my mind I could see them sitting in their boats in the middle of a lake, yelling at the top of their lungs, "Where is my drink, where is my lunch, where is my magazine, and who is going to row my boat?" I imagined myself there as the manager of the resort politely yelling back to them, asking them to row their boat to the shore so I could explain where they needed to go to get what they wanted. Of course, they were not likely to pick up their oars, but I certainly was not going to swim out to their boat and row it for them. It would be foolish to give in to their demands and offer them five-star service at a two-star price.

http://www.wordplace.com/ap/ap_chap14.shtml

In today's world, the question of price point and service level is something that every SaaS vendor has to deal with. But the SaaS model is also more flexible in this respect. Customers who want five star service can buy it to supplement the self-service software. Some customers even want the end result as a service, and it is the SaaS vendor or a third party who actually uses the software to get this result.

I think this flexibility is what customers want. But the considerations raised by Peterson are present today just as they were before.

There must be price discrimination between full-service and self-service. It is unfair to provide higher level of service, for the same price, to those customers who complain the most.
There must be a clear distinction between services that do require custom software modifications and ones that don't. Also, trying to duct-tape custom modifications into self-service software generally leads to great unhappiness.
Responsibility over customer's systems and data must be clearly defined. In the SaaS world, vendors must take more responsibility over data loss due to software errors. But solving issues due to customer's errors or system configuration (browsers, security software etc.) should never be a free service.

-mika-

RadControls Performance Tips

2010-02-16T08:01:00.000-08:00

Recently I was doing some research and looking for tips on how to improve Telerik RadControls performance. On the support forums I found this thread. The thread is a little old, but I answered to it based on my experiences. Having little else to say right now, I thought it might be best to basically repost this answer here.

I've had a problem with customers complaining about the performance of a page using RadScheduler. To be fair, there are other controls besides a RadScheduler in the same page (RadMenu, RadToolbar and RadTreeView). Anyway, the page is heavy on data, and feels sluggish.

I did some benchmarking with YSlow and Chrome Speed Tracer. YSlow told me that my page was about 1MB in total size.

The problematic part is the generated HTML that has to be loaded every time. This is the text you see in the browser's "view source". Of the one megabyte, HTML took about 400KB. It contained:

ViewState ~ 150KB
Script (mostly JSON data) ~ 130KB
HTML ~ 120KB

Even with slower javascript engines it is likely that loading 400K will be a major performance issue, and it certainly pays off to minimize HTML.

Compression
The first thing to do is to use compression in IIS and in your CMS.

DotNetNuke performance options

Just by enabling DotNetNuke compression, I reduced the page size from 400K to 100K, a huge effect. But if this is not enough, you'll have to fine-tune more.

RadCompression

This is perhaps the easiest thing to do. RadCompression HttpModule will automatically compress your Ajax and JSON web service communication and even your ViewState. Setting up RadCompression is just a matter of adding a HttpModule registration to web.config.

Unfortunately, my experience with RadCompression were short-lived this time. RadCompression (2009.02.0826.35) with IIS6 and DotNetNuke compression broke postbacks in Firefox 3.6! Literally, postbacks returned a garbled mess of compressed text to the browser. Chrome and IE were fine, Firefox too with IIS7. The reason for this was that there was two compressors enabled simultaneously: the DotNetNuke GZip compressor and the Telerik RadCompression. I still have to figure out how to enable RadCompression and disable DNN compression on a per page basis, to speed up Ajax requests.

Measuring the performance impact of RadCompression was not that simple and I didn't get a clear picture. But Chrome Speed Tracer indicated a speedup in refreshing UpdatePanels, that was lost after disabling the HttpModule.

RadScriptManager and RadStyleSheetManager

The purporse of these controls is to reduce HTTP traffic by combining requests, for javascript and css respectively.

RadStyleSheetManager is extremely easy to set up. You just add a RadStyleSheetManager control to your page. The net result was about 10 fewer CSS requests in an empty cache. For a primed cache, there was no effect.

RadScriptManager is trickier. You can have only one ScriptManager per page. Because DotNetNuke has one already, you would have to replace it in DNN core code and rebuild. I can't recommend this to anyone. Hopefully in the future DNN Professional editions this is going to be easier.

Minimize inline javascript

Declare only necessary javacript variables in the ASPX. These are usually the ones where you need ClientIDs, like:

var scheduler = $find('<%= RadScheduler1.ClientID %>');

All other javascript should be loaded from external files and cached.

Try to disable ViewState
Easier said than done. For example, I had a hierachical navigation/drilldown menu that displayed the number of events each year, month and day. I can't disable ViewState for this control, because it is tied to the RadScheduler's view (month, week, day, timeline). Surprisingly, I can disable ViewState in the RadScheduler itself without any immediate loss of functionality. This reduced (pre-compression) page size about 10%, but is a risky bet.

Try to limit data size
By showing less data on one screen you can in many cases improve both performance and user experience. But sometimes you need data for things like RadScheduler appointment tooltips, that are stored in appointment attributes and resources. These take easily a lot of space.

Forget old browsers
Even with aggressive tuning, the overall user experience is very much dependent on the browser's capabilities. With IE6 and other older browsers, any page with heavy RadControls barely works, and I wouldn't spend much effort to tweak them.

-mika-

See also:

Top 15 Optimization Tips and Tricks

Top Performance - General Resources

SuperGenPass

2010-01-10T00:01:00.000-08:00

I've been using SuperGenPass (and its predecessor GenPass) for over three years now. SuperGenPass is a javascript bookmarklet that generates a unique password to a web site out of a master password and a domain name, using a MD5 hash. The user interface lets you input the seed password, and populates the generated password to any password field:

In practice, this means that you can increase the security of your passwords while using the same master password on every site, without having to store secrets anywhere. Using SuperGenPass gets you through most of the items in the Bruce Schneier's password rules. Changing passwords gets a little trickier, though, because you have to use two bookmarklets in the transition period: one for the old master password and one for the new one.

Sure, there are other password generators and password managers, but this simple javascript bookmarklet has provided the best user experience so far. Browser compatibility is the only serious problem with the standard SuperGenPass bookmarklet. It didn't work in the Nokia S60 or iPhone browsers, and I remember having problems with IE7. For these situations, there is a static html mobile version that lets you manually generate SuperGenPass passwords on any device. It is also needed when using someone else's computer. A minor annoyance with SuperGenPass is that some websites have idiotic restrictions on password length, and the default 10 character password may be too long or too short.

SuperGenPass is practical for websites only, when a user-generated, non-shared password is needed. I use KeePass for other password management, but wouldn't recommend it for webapps.

-mika-

DevExpress CodeRush and Refactor! Pro

2009-12-30T23:29:00.000-08:00

In July I got a free DevExpress DXperience suite licence. I was asked to write about my experiences with the software, if I found the time to do so. I think it is a fair deal. There is no way I can gain from this offer except by playing around with the software and seeing if it produces a net benefit. I can say that with the hourly rates common in enterprise application development, the time used to learn anything has a considerable opportunity cost. Putting time away to sharpen your saw should be a necessity, but in consulting environment it easily becomes a luxury. Time to learn is time away from short term customer work that brings the money in. Just try telling your boss that you are missing deadlines because of playing around with a new development tool!

Now, half a year has passed and I think I can say something of the tools I use daily, namely CodeRush and Refactor! Pro.

Installation and First Impressions

CodeRush and Refactor! Pro are the first things you see after installing the DevXpress toolkit. They are Visual Studio add-ins that are built upon a common framework called DXCore. There are also a few community plugins using this same framework. A normal user can mostly ignore this plumbing, but from time to time you'll see UI messages that explain how DXCore, not CodeRush, has done something. So, in principle there are three separate products involved: a free framework (DXCore), a commercial refactoring add-in (Refactor! Pro) and a commercial coding assistant add-in (CodeRush) that is always shipped with the Refactor! Pro. If you are not confused enough, there is also a separate free edition (CodeRush Xpress) and a specialiced Refactor editions for C++ and ASP.NET. To keep things simple, I'll will stick with the CodeRush name for the rest of this blog post.

The obvious tool to compare CodeRush against is the JetBrains ReSharper. Unfortunately, I don't have enough experience with ReSharper to say anything interesting (check this thread instead). I tried ReSharper about a year ago and it slowed down Visual Studio too much for my taste. Visual Basic IDE is demanding to begin with, with all the background compilation going on, and adding new extensions will definitely slow it down more. I don't have benchmarks available but I would presume that the amount of background processing is dependent on the lines of code in your solutions. Large solutions should be slower than small ones. Good news is that performance improvements have surely been the most highlighted feature in both CodeRush 9.2 (1) (2) and ReSharper 4.5.

Ultimately, you must compare the productivity gains of IDE extensions to the degradation of default IDE performance. This includes crashes, temporary hangs and general slowdown. I collected some haphazard data after installing CodeRush 9.1. on a machine with the following performance profile:

VS2008 hangs for a few seconds but recovers. This occured repeatedly.
VS2008 hangs permanently or crashes, restart required: happened 3 times during test
VS2008 lost intellisense: did once
VS2008 slows down: noticable, especially correction suggestions update slowly

(I must stress that I did have a few other Visual Studio add-ins during testing but the problems started only after installing CodeRush.)

In the end, for the same reasons that I couldn't use ReSharper, I had to give up using CodeRush 9.1. I need performance and reliability in Visual Studio to do my job adequately. After uninstalling Coderush, VS2008 performance improved but it still crashed every now and then, even with every other add-in uninstalled. It would be unjust to blame 3rd party tools for this, it had something to do with the VS2008 and the environment itself.

CodeRush 9.2

After CodeRush 9.2 became available I decided to try it again. This time with a fresh install of Windows 7 and VS2008. In a fresh system, CodeRush started to perform fast and reliably enough to be useful. I can't remember a single crash. This level of reliability is an absolute requirement for a productivity tool. Until this point, CodeRush had a negative impact on my productivity. I can only recommend that you have a solid Visual Studio environment before installing any add-ins.

Learning curve

Tools like CodeRush and ReSharper are only distractions if you don't learn to use them. And there is a learning curve involved. If found that this learning process started with general confusion, followed by curiosity and experimentation with the features. Finally, there was a kind of settle-down to a subset of features, with some features even disabled.

If I remember right, I first started using the refactoring suggestions right of the editor scroll bar that are similar to ones in ReSharper. These suggestions are mechanical, but they are useful in finding undisposed resources and redundant code. And you can customize them. For example, I didn't see it worthwhile to use String.Format in every concatenation and disabled the suggestion.

But the most visible features like refactoring suggestions are just the tip of an iceberg. I suggest taking a systematic approach to learning CodeRush. This means trying most of the features at least once and seeing what sticks in practice. This way you'll find the features you don't know to look for. I used the cheat sheet as a learning tool, trying one section per day while working on projects. The point is that learning CodeRush is a voluntary discovery process. There are tools to help you, like the Feature Statistics, but in the end you are free to use what you want.

Stats show that my top 5 features for the last week or so were:

Intelliassist 274
IntelliassistCycleForward 35
Rename 25
ShowReferences 24
If .. Then 19
CamelCaseLeft 11

I think I could use MarkerCollect (3) a lot more, but I don't really understand Selection Inversion (6), that is, inverting selected boolean expressions from true to false and vice versa. I think this feature is mostly invoked because of typos. It is easy to hit a wrong shortcut when using CodeRush, and in many cases it might be best to disable the invoked feature or at least remap its shortcut. Selection inversion is particularly dangerous, since it has a semantic effect invisible to the compiler. I have been burned once by an unintended inversion that ended up in the version control system.

Besides learning predefined shortcuts, you'll have to customize CodeRush to use it effectively. CodeRush allows you to customize practically everything. For example, default shortcut keybindings in international keyboards are not always the most natural ones, and you need to rebind them yourself. This effort pays off, since only those keybindings that stick are valuable.

One general impression is that CodeRush is more pleasant to use with C# than with VB. Clearly, C# programmers are a larger audience and also need some of the features more. But CodeRush has improved its VB support too, and it does add useful features like refactorings that are scarce in VB by default.

Keyboard Shortcuts

As mentioned before, there is a great cheat sheet available from the CodeRush inventor himself, Mark Miller. It is divided between Clipboard, Selection, Navigation, Code generation, Miscellaneous and Template shortcuts. Templates, or code snippets, are the most versatile part of the system. They also require the most memorizing, and it is better look at the other shortcuts first.

I have a gut feeling that most programmers are not that much into keyboard shortcuts. Sure, there are global shortcuts that everyone uses, and leveraging them is a good practice. For example, mapping F2 to rename an identifier in Visual Studio is a natural extension to a common Windows rename shortcut. But there is a flipside to this argument: shortcuts that interfere with the more common ones are harmful. For example, I dislike using Ctrl-W to increase a selection is VS because it interferes with using Ctrl-W to close windows and tabs elsewhere.

Even natural shortcuts like F2 can be dangerous. The danger is that you can accidentally rename a method when you really wanted to use another method. This is probably why F2 is disabled by default. But I encourage to enable it. F2 is a special time-saver in ASPX files, where server control identifiers get values like TextBox1 by default. In plain Visual Studio you have to use the property grid to globally change such a server control ID, because there is no rename feature similar to renaming variables in code. But with CodeRush, rename is performed uniformly in both aspx and code-behind.

Other favorites:

Drop and collect marker. These are short-term bookmarks in code. Drop a marker with Alt-Home, go elsewhere in the code, come back to the marker with Esc.
Clipboard history. More slots in VS clipboard. Unfortunately, its shortcut Ctrl+Shift+Insert is hard to use with Logitech keyboards that have this braindead large Delete key and hide the Insert key behind Fn. I suggest Ctrl-Shift-Delete.
Camel Case navigation with Alt-Left and Alt-Right is cool, it works like the default word-wise navigation Ctrl-Left / Ctrl-Right. Selection with Shift.
Show all references using Shift-F12 is better, but much slower than the default behavior.

Templates

Templates, or code snippets attached to keyboard shortcuts, were the last CodeRush feature I eventually discovered. The idea of code snippets is familiar, but it takes a time to get used to them in practice. Spacebar is the magic key in expanding templates, you type the shortcut and hit spacebar, and CodeRush fills in the template. This works great for 'If..Then' template because hitting spacebar after typing 'If' is automatic. For other templates, you have to do some work to remember the shortcut, like 'fe' for 'for each'. I think that ordinary IntelliSense assistance is mostly good enough, and you have to put in some conscious effort to get real benefits from using templates.

Final remarks

A lot of functionality is not mentioned here, mostly because I haven't learned enough yet. I don't use clipboard, wrapping selections, templates or refactorings that much. But the features that I use are valuable and most of the annoyances can be turned off.

An important thing to know is that the current version of CodeRush will be incompatible with Visual Studio 2010. Good news is that the CodeRush team is working on a DXCore rewrite that should work both in VS2008 and in VS2010.

Telerik Code Converter

2009-10-19T00:04:00.001-07:00

Telerik Code Converter is an indispensable tool for every Visual Basic developer.

Visual Basic is a distant second to C# in the .NET ecosystem. Yet, there are strongholds of VB and you may find your job to be in one of them, like I did. For example, VB is strong in DotNetNuke-based development since the core framework itself is written in VB.

Most VB developers use C# as a second native language and it is very common to end up having a multilingual codebase, with projects in both languages. Converting code snippets from one .NET language to another is needed often, since your codebase is the first resource you have when building new stuff.

Also, most public code samples are written only in C# these days.

-mika-

Embedded Reporting with Chart Controls

2009-09-25T05:16:00.000-07:00

I've previously written about embedded reporting in general and looked at Telerik Reporting in more detail. But one subject wasn't touched on these posts, that is, creating operative reports without a reporting engine. Reporting engines provide a lot of features, sure, but you don't always need these features. If the requirement is to embed charts directly to your web application's views, a heavy reporting engine is just getting in your way. Think for example the Google Finance Flash chart pictured below, which for all practical purposes represents the gold standard of interactive time-series plotting:

Would you like if that plot was rendered with an SSRS-style engine, with 'generating report' animation spinning between reloads. Neither would I. Using common report elements like chart controls directly in your web app makes operative reports more responsive and the overall experience more 'webby'. But you won't get a Google Finance chart experience with the current 3rd party chart controls. The high end seems to be more like the Google Analytics -style plot with data point highlighting, which is nice too.

There are two approaches to web reporting: client-side RIA controls and server-side web controls. The distinction between the two is increasingly difficult to define technically, and I won't go into it now. Let's just say that the developer experience is different enough when using a jQuery plugin compared to a ASP.NET AJAX control that you are inclined to contrast and compare the pros and cons of both.

The charts that I've been using as a an exercise are taken from a Campaing Monitor email tracking report, which itself is a shameless visual copy of Google Analytics reports. Kudos for both! There are two charts in this report: a time-series plot with data point highlighting and a pie chart with a legend.

Now, let's try different tools to imitate this report

jQuery Chart Plugins

Javascript source code is available to all users, which makes open source a natural model for javascript-based tools. But it also means that feature-rich and well-supported tools don't just drop out of the sky, they will evolve gradually. The state of the art today is that free jQuery plugins don't either have the features or the quality to substitute a commercial chart control suite. But they are useful enough for many purposes and are simple and fun to play with. They probably also represent the future of embedded reporting. Sooner or later, one such plugin is going have enough lift to really take off. This is simply because jQuery plugins are platform-independent and consequently have appeal to an enormous plugin user and plugin developer bases. But if you are interested in commecial-quality javascript charts today, you should take a look at Emprise Javascript Charts

Technology

The jQuery plugins I tried generally utilize the HTML5 canvas element. IE does not support canvas yet, but Google's ExplorerCanvas does a decent job of emulating its functionality.

Javascript basically has two alternative methods of data access. The most straightforward method is to print data directly to the response, usually as a JSON-formatted value to a javascript variable, or as an HTML table accessible via DOM. jQuery Visualize plugin has taken the HTML table approach while others rely on JSON. But scraping data out of tables is a handy trick you can use with all javascript charting tools. The other method is to remotely load data with XHR or using cross-domain JSON.

Whatever means the client uses in reading the data, it is ultimately generated in a server somewhere. From the server-side view, printing data directly to a response can be easier than serving it through an url, especially if the data is not meant to be public. For example, I get DotNetNuke portal's page-level security for free if I print data.

Developer experience

Aside from technical capabilities, I'm especially interested in the general developer experience. The key question is: Is it simple and reliable enough for your boss? Can you recommend this tool to someone who does not have much time but who is tightly involved with customers and would like to use this first-hand information do reports without delegating them to others?

For jQuery plugins in general, I would say that if your boss has done client-side web development and knows how to use firebug, then jQuery plugins are safe. It is best to set up the environment beforehand, checking that jQuery version and plugin references are correct, and seeing that data access is working. Server-side work involves querying the data and converting it to proper format. As mentioned before, this is not necessarily AJAX, if you just write to the response page. One thing to remember with time values is that javascript requires you to convert .NET ticks to Unix/Javascript ones. Not necessarily something your boss would like learn the hard way.

The best thing in developing with jQuery plugins is the rapid development cycle. 'Installing' and testing multiple plugins in a single page is fast and fun. It took about half a day to evaluate all the plugins in this review. I have grown up to expect increased levels of frustration when dealing with new ASP.NET controls, and finding less frustration in javascript development was a nice surprise. Charting is a good fit for javascript because codebase for a single chart shouldn't grow too big and hairy.

Oh, and one more thing before going to the plugins. If you are interested in Javascript charting, go check Steve Reynolds's reviews:

flot

Flot produces nice looking lines, filled ares and columns, but no pie charts so far. The samples are great, showcasing such features as zooming with overview and colored weekends in time-series plots. Interacting with data and point highlighting with tooltips are supported too. Values for undefined parameters like axis ranges are well-thought.

In my opinion, flot is the best jQuery plugin for the chart types it supports. Solid web development skills are required to use flot, and you shouldn't be afraid to learn from real-world examples.

jqPlot

jqPlot looks very polished for an open source project and has nice features like data point highlighting. But the good first impression fades somewhat when you try it with other browsers than Firefox.

Here's a jqPlot pie chart in FF 3.5, showing email tracking data. The labels are in Finnish and mean "Opened" and "Not opened".

In Chrome, the same chart looks like this:

As you can see, the colors are inverted. If you mistakenly input data as text without parseInt(), FF is ok but Chrome will render the shares wrong:

IE8 with excanvas works like FF, but again, if you input text instead of numbers, the percentage shares start to look interesting, even more so than in Chrome:

Even with these inconsistencies, jqPlot pie charts would be my choice for a jQuery Google Analytics replica.

jgCharts

jgCharts uses the Google Chart API but does not support all features from it. You can't color a pie chart with jgCharts, for example. Yet, most of the shortcomings originate from the Google Chart API. First, jgCharts do not impress visually compared to flot or jqPlot, which is not surprising since the charts are intended to be loaded as ordinary HTML image elements. This means that the API is the only way to customize the charts as CSS and javascript are out of the question. Animations and interactions like drill-down urls are naturally missing too.

Others

jQuery Sparklines didn't impress visually and I couldn't do even basic customizations to a pie chart. There is a reason for that. A sparkline is a type of information graphic characterized by its small size and data density. This is probably the optimal use for Google Chart API too.

Visualize demo charts look nice, but the feature set is quite poor. Interactive features and animations are not supported, and there are only a handful of parameters. All data is scraped from a HTML table, which means that you'll have to explicitly hide the tables you don't want to show. With other controls, the problem is exactly the opposite: you need to do extra work to show data also in a HTML table. But every jQuery plugin can easily scrape data manually from a table once it is there, which is much more flexible than creating tables in the exact format that a plugin needs. For example, in the previous jqPlot and jgCharts pie charts data was scraped from an ASP.NET datagrid. Visualize couldn't use that table directly. In the picture below there is a HTML table in the upper left corner and a corresponding visualize chart with default settings, as seen in Firefox and Chrome:

With IE8, the outcome was not so good, but at least those black spots in percentage shares were gone:

Flash

Flash is the technology of choice for professional web charts, as witnessed by Google Finance and Analytics. Silverlight certainly has potential too, but because of the relatively small install base I decided to drop Silverlight Toolkit from this review.

Technology

The worst part of Flash is the Flash development itself. ActionScript is not exactly known as the language of choice in software development. Developing Flash also requires Adobe Flash Professional or Flex. But if you don't intend to develop Flash chart controls yourself, none of this really matters. Right now and in the near future, Flash chart controls are the most sophisticated ones around, both in their visual outlook and in the features they support. But you should expect Javascript libraries to close the gap pretty soon, at least from an end user's point of view.

Data access methods in Flash are a superset of the methods available for Javascript. You can use Javascript to feed Flash objects and the Flash charting tools provide good Javascript APIs. In addition, you can use the Flash object's parameters for data access. The urls defined in the Flash object are subject to different rules than the urls inside Javascript. Authorized cross-domain data loading is available and is controlled with policy files. Second, some but not all Flash charts support printing data directly to a swf parameter.

Developer experience

If you can avoid Javascript, Flash charting is extremely simple. Well, it is just HTML objects then. Any developer should be able to use these kind of charts. If you cannot avoid Javascript, the experience is similar to jQuery development.

FusionCharts Free

FusionCharts Free (FCF) is a free older version of the popular FusionCharts, the market leader in Flash charting. First time I took notice of FusionCharts when they started to show up in popular web applications, especially RescueTime uses them heavily. Creating the example email tracking charts was no problem in FCF:

FCF has all the polish of a commercial product:

example charts that look good
compelling feature set, but these features are yet very easy to use
high-quality documentation and support forums.
support for multiple server-side technologies

FCF's .NET interface is extremely simple. You replace a literal with a Flash object:

litFC.Text = FusionCharts.RenderChartHTML("/FusionCharts/FCF_Line.swf", strUrl, strXML, chartID, width, height, False);

XML data and settings are provided via an url, or are embedded direcly to a HTML Flash object. Great thing is that .NET interface does not try to act as a wrapper to XML, you generate the xml yourself and pass it as a strXml parameter to the above FusionCharts.RenderChartHTML function. FCF reads xml settings liberally and does not mind if you use settings from different chart types in one setting xml.

Each FCF chart type has a separate swf file, which is a minor complication because you need to know multiple pathnames in using the API. I wrote a helper enumeration to get correct filenames using IntelliSense.

The only problem with FCF is that it is a legacy product. The actively developed product is Fusion Charts 3, and the differences between the two are substantial. Good thing is that you have an upgrade path, but don't expect any new features in the free version.

FCF is the easiest charting tool to work with, and even nontechnical people should be able to draft chart requirements with it.

Open Flash Chart

Open Flash Chart (OFC) is a popular open-source alternative to FusionCharts. OFC sample charts look good, they are definitely the best looking ones from the charts I've tested here. But this does not help too much if you won't get your charts to look good. And unfortunately OFC is not too easy to work with.

OFC documentation is PHP-centric. For other environments, there are support forums but not much more. To use .NET you can compile the shipped .NET library or use an alternative one.

Unlike FCF, OFC has only flash binary, open-flash-chart.swf. Chart type is defined in the JSON-formatted data file. Compared to jQuery plugins, this JSON it is quite complicated to create manually. Good thing is that there are server-side libraries to create OFC charts that handle the conversion to JSON. But you still need to use javascript to pass the data to the chart. This is not as simple as passing XML data directly to the flash object parameters, as in FCF. You need Firebug to debug things, tolerate Firebug/Firefox crashes etc.

The default .NET API acts as a JSON wrapper class and is not nearly as simple as in FCF. The bad thing is that this abstraction leaks, you can't control every setting in JSON with the .NET API, or at least you have to spend a lot of time finding out how to achieve this.

One particular annoyance was that OFC dates use PHP mktime() ticks, that are otherwise the same as Unix/Javascript ticks but are counted in seconds instead of milliseconds.

My first impression was that OFC is hard. I wouldn't let my boss use it.

ASP.NET

Telerik RadChart
Devexpress XtraCharts

It should be noted that the RIA controls I tested are free or open source but the ASP.NET chart controls are commercial products, usually shipped inside a larger toolkit. The controls are well productized and strive for high user productivity while offering the maximum feature set. This is not easy to achieve, and therefore it is important to support the user when the inevitable troubles arise. Money does not buy you a solution to every business charting case, but it does buy you many well-supported scenarios.

Technology

ASP.NET charts are technically using the same client-side web platform as the jQuery charts. In practice, there are probably quite a few constraints in developing a generic .NET chart control that do not exist in pure Javascript development. I wouldn't expect ASP.NET charts to visually match the best Javascript ones. But at least it would be nice to have some interactive features and animations, which are mostly missing now.

Rich data access is the strongest point of ASP.NET controls. You can use native ADO.NET data structures directly instead of serializing to JSON or XML. Another .NET advantage is that the chart controls are usually built as generic all-in-one charting solution for both ASP.NET and WinForms. The same chart control is also used in the vendor's reporting solution which can be useful.

Developer experience

Visual Studio IDE and the .NET tooling support in general are great, sure, but the focus here is on the usability of chart controls. Commercial server-side chart controls typically have lots of features and the learning curve can be steep. In the best case you have decent Visual Studio design-time support, which is intended to work like WYSIWYG for UI programming. In practice this means property grids to make settings and some visual output on how these settings affect the chart. But if you can't get design-time support to work, you'll only see the IntelliSense code comments and external documentation while coding, and the final outcome is visible only in the browser.

Generally, getting an example chart up and running takes more effort in ASP.NET than with jQuery plugins or Flash charts, because you have to set up a IIS web application and prepare an aspx page, a code behind file and a web.config. Development and testing cycle is slowed down by the time it takes to compile the solution and reload the ASP.NET site. Reloading is particularly slow if you have lots of dlls in the /bin folder. But again, this same reload happens when using a Flash chart .NET API, or when doing server-side .NET data access with any front-end charting tool.

I don't like the ASP.NET heavy lifting needed to draw a few charts. But if your boss is an ASP.NET developer, and you provide the environment and a few samples, things should be ok.

XtraCharts

XtraCharts is a part of the DevXpress toolkit. Deployment requires 16mb of DLLs to be loaded in every application restart, only for charting. That's heavy and takes a few seconds in your dev machine.

Demos are nice, I liked the spline plot, but as you can see, the chart is a .png image, which prevents any cool animations or mouseover interactions inside the chart. Outside the chart image you can have tooltips as in the "Weather in London" chart.

For some reason, I couldn't get the Visual Studio designer support work in a DotNetNuke module project. The designer showed an error:

Probably some setting is wrong as I was developing in a Visual Basic project. Working manually with the code-behind file is fine with me, but the bad thing was that I already faced a problem after a few iterations of the pie chart. Below is the chart with labels but no legend:

Setting the legend visible caused the chart to disappear:

Issues like this arise in every complex control, but debugging them makes you appreciate the fast develop/test cycle of the client-side tools.

Telerik Charting

I have used Telerik RadChart previously with Telerik Reporting, and my perception was kind of lukewarm then. But I should have more perspective now.

First the good news. Designer support for Telerik Charting works pretty well, even though the property grid is hard to use when fine-tuning the chart. WYSIWYG works even in the data level: you can put data items in the aspx, and they show directly in the designer, as pictured below!

Having data in aspx also means that you need not to do anything in the code-behind to get a working report. But frankly, I don't find any use for data-in-aspx other than simple demos.

Compared to XtraCharts, some useful chart types are missing, or they don't look as good. For example, a line plot demo has labels but not visible data points. XtraChart demos are more complete, easier to navigate and better-looking than their Telerik counterparts, but the difference is not big. Both tools produce static image-based charts that support only image map interactivity, zooming support in Telerik line charts being an exception. It should be noted that Telerik also offers Silverlight charts that are on par with Flash with respect to pizzazz.

And then the bad news. I couldn't get the Telerik charts working at all in the browser. All that I got was an error message:

BC30560: 'ChartSeriesType' is ambiguous in the namespace 'Telerik.Charting'

Normally, I would debug for an undefined time to get some results. But this time I have to call it off. No more ASP.NET debugging today.

Final words

Programming is a profession of solving real-world problems with tools that are built to look great in demos. But you shouldn't blame the tool if you can't ever achieve the same results in the same time on your own, no matter how much you spend time debugging with the tool. This is the reality. But sometimes you do get positively surprised by a development tool that just works as promised. FusionCharts Free is such a product. The people who designed and implemented it five-to-ten years ago have done a hell of a job. Thank you!

I chose Fusion Charts Free for our company, and it is probable that we upgrade to the latest commercial version in the future. FCF has the looks, simplicity and support required for a business where charting is a nice-to-have but not a critical feature. A great thing is that FCF with inline dataXML data and no mandatory javascript is a fool-proof charting environment that is safe for your boss too.

Open Flash Charts was the runner-up, entirely because of its looks, but it is too complicated for occasional use. If I was doing serious charting I would compare OFC more carefully with the commercial version of Fusion Charts.

jQuery plugins are promising, but so far they are fit for only lightweight charting tasks, mostly because they lack features. Javascript is still not for everyone, but for javascript-savvy programmers the developer experience is extremely productive, until you hit any edge cases.

It makes sense to use .NET chart controls if you can piggyback some of the work you have done with the other controls in Telerik or DevXpress toolkits. Or if you are doing WinForms charts too. But as standalone web charting tool, RIA charts are better.

-mika-

Microsoft Antivirus, Antitrust and the Public Good

2009-09-11T13:23:00.000-07:00

Microsoft Security Essentials has been officially released.

This is good news for all Windows users and especially for the tech people who have gotten used to wrestling with 3rd party security solutions. If a friend calls me and says there is something wrong with his/her laptop, the first thing I presume is that security tools are causing trouble. The usual fix has been to uninstall some old brand-name security solution and replace it with Windows Firewall, Windows Defender and the free and fast Avira Antivir. From now on, I'd use Microsoft Security Essentials instead.

I haven't used an antivirus in my main laptop since I upgraded to 64bit Vista. But I do use antivirus in machines that I don't attend to, like my home server PC and the servers at work. The reason for not using real-time security software is performance. I need it. I'm willing to put effort in hardware upgrades like an SSD drive with clean-installeded Windows 7 to improve developer tool performance, but I'm not going to waste this investment by installing a new resource hog. Instead, I protect the machine with hygiene, by not installing anything unnecessary and keeping the updates fresh. If something goes wrong and an attacker gets through a vulnerability, I have a minimal setup backed up and recoverable fast. So far, I haven't needed it, because natural clean-ups like getting a new machine or installing a clean OS are frequent enough to keep the machine healthy. But I do run an off-line security scanners like Malicious Software Removal tool and ClamWin once a month or so.

For basic users and risk-taking power users, real-time antivirus is good. But even then you don't want to deal with the security software industry. It is just too hard to evaluate the net value of their offerings. You cannot be rational with security software unless you have a reliable cost estimate on being unprotected, and know the costs and benefits of a particular security tool. But this kind of analysis is hard, as the results depend on the value of your time and data in addition to the technical quality of a security tool.

The reason why Microsoft has not included antivirus in Windows already in the late nineties must have something to do with the antitrust regulators. Now, with the popularity of free antivirus rising, Microsoft is more safe to launch this offering. It will probably never be a part of Windows, which is bad for the overall security climate. On the other hand, this will probably be in the best interest of the consumer by keeping some free or commercial alternatives available. Historically, competition has been about the only thing that drives software quality improvements. As a warning example, it is good to keep in mind the internet dark age, 2001-2004, when IE6 didn't have a serious competitor.

Wrapping things up, unless Security Essentials turns out to be a real turkey, I'd recommend the following policies:

For large enterprises, it makes sense to use centrally managed antivirus. But give developers an option to uninstall it in their machines, please!
For nontechnical home users and small business users, who don't need most of the performance most of the time, real-time antivirus like Security Essentials is good. But you need to do backups, always.
For unattended machines, antivirus is a must but you can have a scheduled virus scanner instead of real-time scanning. Free antivirus for Windows Servers is rare, I have used ClamWin for these. Managing ClamWin is tricky if machines can't connect to the update site, like when outbound http requests are blocked by a firewall.
For performance-intensive workstations, minimize the amount of unnecessary software, including security software. You are ultimately responsible for the security, not some black-box application.

-mika-

Experiences with Telerik Reporting

2009-09-06T14:23:00.000-07:00

This is not going to be a too nice and well-rounded review so I'd better start with some background.

I work in a software company that is specialized in event management software. An 'event' here means the same as a 'happening', something that people are going to attend to. I should write about this more later, but briefly, event management software is like Facebook's event invititation feature for organizations. If you think it a while, there are a lot of organizations that do events for a living. Every educational institution is one. Second, inside every large organization there is a HR, marketing and/or public relations department whose one job is to manage events. And then there is the whole mass event industry.

With such diverse clientele it is expectable that the same software features are used to manage completely different data, and sometimes similar data is managed with different software features. Business processes* and the language used to describe them vary a lot. And because reporting is all about written expression, it is not surprising that one report does not fit all users. As I already explained in the last post, different labels on the same categories are often enough to justify a custom-tailored report.

In practice, we usually get an Excel report someone had previously produced manually, before they started to use our software, and from now on, they want this same report to be generated automatically. At the same time, my boss wants a high-level reporting/charting tool. Something that outputs nice charts from a function that takes all the necessary parameters. These reports should be suitable to all customers, with some small customizations passed as parameters.

From these requirements a basic solution is to have few reasonably generic default reports and a bunch of custom reports for ad-hoc situations. The interface to open a report from the application should be stable. You don't want to modify the report calling interface for each new report, because this would be maintenance nightmare. This is because applications and reports are usually developed separately, and you can't trust that the application has always access to the the latest report definitions.

As I started to work out reports, our first choice of reporting tool was Telerik Reporting, just because we had a licence. Compared to SSRS 2005, Telerik Reporting is almost on par with features, at least starting from version Q1 2009. Other option I had was to use Microsoft ReportViewer in local mode, and later on I have started to test DevExpress XtraReports. Comments on these will follow later but the rest of this post is specific to Telerik Reporting, version Q2 2009.

Telerik Reporting Basics

In order to use any reporting tool effectively, you need to have a basic undertanding of the report lifecycle. Report definitions are designed, definitions and data are processed, and finally the resulting report is rendered. You will be confused If you don't realize when your code runs in the report lifecycle. For example, creating a report object means creating an instance of a report definition. All parameters passed to a constructor affect only this definition instance, not the runtime processing.

Familiarity with the lifecycle is needed most often when wiring user parameters to queries:

default values of user parameters are stored in the definition. If the report is processed immediately after it is opened, these default values will become selected values
selected values of user parameters are only known at the runtime processing

Runtime processing behavior is event driven, with events like NeedDataSource to read user parameters, run a query and bind the results to a report element.

Telerik reports definitions are classes inheriting the Report base class. Reports are opened by creating a new report object and assigning it to the report viewer.

Report report = new MyReport()
this.ReportViewer1.Report = report

Because each report definition is a class, you need to use reflection to avoid messing with the interface between application and reporting. The interface I chose was to pass the report class name in the querystring to a report laucher page. This way reports can be opened from the application by redirecting to an url:

/reportlauncher.aspx?type=typename

Simple report parameters could also be added to the url this way.

Parameters

Much of the generalization effort in reporting involves loading available user parameter values dynamically. Available values, usually business categories, would be shown in dropdownlists in the user parameter area. Their effect is to slice the report by parameter value. Telerik reporting supports both single and multivalue parameters, which is good. To populate parameter lists from a datasource, you use the AvailableValues property of the report definition. I had some trouble in getting my datasource right, I couldn't use a dictionary, but arrays and datatables worked fine. Arrays don't have column names like datatables, but the interface requires you to use "Item" as a ValueMember name. This was helpfully instructed in an error message.

Report parameters don't support localization which is a serious shortcoming with no workaround. This is a problem especially with datetime parameters because users are accustomed to certain date formats. Here in Finland the format dd.MM.yyyy is expected but the only supported format in Telerik Reporting is MM/dd/yyyy.

Complex Charts

If you thought that Excel charts are hard to use, think again. Excel team has probably put multiple man-years of work to optimize the usability of charting. On the other hand, if you try to put every setting of a chart into a Visual Studio property grid, you'll get what Telerik charting looks like. This is not a complaint, it is a fact.

Picture on the right is a chart property grid with only a few items expanded. Navigating the grid and finding interdependencies between settings gets hard pretty fast, and it is a trial and error process.

Designing a chart layoyt so that it looks good and clear with as many datasets as possible is a whole topic in itself, a data visualization problem. Thit being said I found that labels and legends are hard to position sensibly. I had a lot of trouble with ugly overlapping labels. Sometimes it is easier to use ordinary textboxes instead of chart labels. One problem I solved this way was positioning the X-axis header and item labels in a bar chart so that they don't overlap each other.

The Evil Wizard

Don't Use Wizard Code You Don't Understand (The Pragmatic Programmer, Tip 50)

Telerik report wizard instructs using the Microsoft DataSet Generator for your queries. Forget it. MSDataSetGenerator is an evil wizard. Once you have generated the code with it, good luck doing any significant changes!

One thing that bugged me was that the wizard-generated adapter has two methods, Fill and GetData with similar signatures. At first, I didn't find any good use for GetData. You would think that once you fill a dataset you can use a reference to it directly. But it turned out that the reference works only for the first time you fill a dataset, but on on subsequent runs.

The problem was that the default user parameter values didn't return any data and I got the dreaded message There is no or empty series, which means that your dataset is empty. I tried to change user parameters and reload the report with no effect. Only after changing Fill to GetData did the dataset refresh.

On top of this all, the dataset wizard/Query Builder does not even work well if you have standard SQL parameters in your query. I don't think you should use it for anything else than playing around.

N-pages and other layout problems

Why is there N pages of duplicates in the report? The reason is that you have defined a datasource for the entire report, with multiple result rows, and the detail section is processed for each result row separately. Take the report datasource away.

Another thing is that you must be careful with the page width. If your report is too wide, blank pages will be rendered. I remember similar issues with SSRS and was expecting this behavior. Page headers are especially tricky, for some reason having a page header generates easily an empty second page in an otherwise one-page report.

Usually blank areas and extra pages have something to do with positioning, but there is at least one exception. If textboxes are invisible and no design mode text shows in the preview, it is probably a localization issue. Revert language to default and remove extra .resx files if you can't get anything else to work.

The latest layout problem that I had was that the report designer canvas didn't show textbox elements that clearly were in the designer file and were visible in the web preview. These elements were outliers that were far separated from others. I had to remove them from the designer file.

Debugging?

Debugging reports is tricky. You cannot debug report code in Visual Studio preview mode. You need to attach the debugger to a windows application or a web application. Now, try your best to make these applications similar, so that they use the same report constructors and all. This is more difficult than it sounds. I have a generic windows test application copied from Telerik samples, but with most of my web applications I need to pass more parameters to the report. So the test app use different constructors than the production app. More than once I've had the trouble that debugging in the windows app is different from debugging in the webapp.

Samples and Forums to the Rescue

God bless, Telerik has provided working examples that look nice, use most of the features and have source files available that actually work. The last thing is important, since if you can't get the examples work in Visual Studio, there is not much point in taking example from them.

Another essential lifeline is to visit the forums. I have definitely needed the forums in every new report task, and usually they have provided an aswer.

Deploying and upgrading

Reports are developed as class library projects in Visual Studio. They will compile to a dll that can be deployed to the application's bin folder. For example, I have copied my report dll directly to DotNetNuke site's /bin.

Upgrading report projects from one Telerik version to other in Visual Studio has never caused any trouble. Reports compile as before. But upgrading requires also that you redeploy both the reports and the Telerik dlls. You must remember to change the Telerik.ReportViewer version number in the application's configuration file, otherwise your reports won't run. This will cause surprises if you have multiple sites and web.config files to maintain.

In Visual Studio, the only small problem I've had was that Telerik ReportViewer didn't show up in the ASP.NET Toolbox in the first installation.

Final Words, Déjà Vu

So, I finished the reports and clients have been happy so far, which is good. But the biggest problem with Telerik Reporting is that I had to do the exact same hoops described here about four months ago. And promptly forgot them afterwards. Wish I had written this blog entry back then. I probably used the same amount of time debugging the same issues now as I did then. This is a real killer, even if its largely my fault for not taking good notes. The knowlegde required to get your reports both work and look good is just very specific. But once you get the first report right, it is straightforward to add similar reports.

Now, my boss wanted a report suite that is easy for him to extend. And he is a CEO/salesman/developer who does not have time to do research. I don't think he has the patience needed to use Telerik Reporting.

Finally, I'd want to point out that I don't dismiss reporting as a job, quite the contrary. But I'm getting more confident that reporting is too important to leave to operational application developers. First, normalized database schemas are not optimal for reporting. Dimensional data models would do a lot to make reporting fun and productive. Second, data quality is not a priority in most CRUD app projects. Schemas are bad, not all tables are even 1NF, policies with null values vary. And third, there are the problems with embedded reporting I have discussed here and in the previous post.

I would recommend separating reporting from applications at all levels:

In the IT department, source reports from a data warehouse, not from operational databases.
As an application developer, make report addresses configurable in your operational application, so that users can choose a report server without having to input parameters twice. Supply default reports for your application as SSRS templates, at least in addition to the embedded reports.

-mika-

* Business process is a term that used to imply how people in different organizational 'tribes' end up doing things in their own way. An evolution-based rationalization would be to say that the origins of business processes are quite random, something like out-of-the-hat management decisions. During the implementation the most harmful initiatives are weeded out, managers publicly (but hopefully not privately) forget their mistakes, and the surviving arrangements are what is left as a business process.

Thoughts On Embedded Reporting & Few Tips

2009-08-28T00:00:00.000-07:00

Some people would argue that preparing business reports is not an innovative area to be in. But in my experience reporting is the area where an individual developer can make a big difference, with relatively small effort. At best, report development is extremely productive, due to two reasons. First, reports are a low hanging fruit. Existing reports are in bad shape in virtually every organization that has no access to a professional database/DW/BI team. Second, it is surprisingly fast to develop for a good reporting system like the SQL Server Reporting Services (SSRS).

But if you are an application developer, SSRS report server is not your choice for operative reporting. It would add another system to administer, another dependency, something that would ultimately be a liability if your business is not to provide infrastucture services. You want to embed reports to your application and minimize any administrative burden. But there are bad news. From the tooling perspective, embedded reporting sucks big time. If you are used to authoring SSRS-style server side reports, then creating, debugging and deploying embedded reports is pain. Your productivity will be abysmal compared to SSRS. You go through this pain in order to get rid of the server dependency and to pass parameters conveniently from the application to the report. In the end, you should have your reports nicely packaged in a deployable dll and your end users shouldn't have to input parameters more than once. These are about the only reasons you should ever consider embedded reporting. Don't do it if you can avoid it!

I have played around with Telerik Reporting in two projects, both times creating a handful of embedded reports for particular customer needs. I also have used DevExpress XtraReports, but not enough to say anything about it yet. So far, I haven't had a chance to develop reports with the long-time market leader Crystal Reports or another well-known solution ActiveReports. With SSRS I have developed report server reports since 2006 but I've never used the Report Viewer control in embedded, or local, mode.

Welcome to the Reporting Country

Reporting is all about compromizing development speed, flexibility and looks. Rapid development is supported by GUIs, 4GL tools (code generators) and DSLs (domain specific languages). A version of Visual Studio shipped with SQL Server, Business Intelligence Development Studio (BIDS), is a prime example of this approach. Reports created with BIDS GUI are actually stored in a special purpose Report Definition Language (RDL), a textual XML format. BIDS has wizards for common tasks like creating crosstabs and charts, and of course for wiring the reports and parameterized queries together.

Now, relying completely on abstract tools is counterproductive if something goes wrong. Abstrations tend to leak on the edges, and you are almost certain to hit a wall sooner or later even with the most mundane business reports. An edge case is any requirement that is not demonstrated in the product documentation or demos, and is potentially difficult to implement. Sometimes you really need to bang your head in frustration, google like mad and read a ton of forums to move a small step forward.

This being said, high-level tools really shine in reporting if they are well designed and don't cause too many surprises. But you shouldn't be blinded and let your customers expect that any report can be finished "in a few hours".

Bottom up is the only way

There is a temptation to design your embedded reports top-down, from the general layout to the details. This may be fine at the design stage, but do not make the mistake of starting to develop from the main report and adding subreports and charts as details to the main report. If it does not work you'll have too many pieces to debug effectively. Instead divide and conquer your reports by using subreports and creating a new report file for each new report item. You can later move these to the main report, if using subreports isn't good enough.

It is surprisingly painless to move report elements and create duplicates of reports as templates once you get the first report working. This will violate the DRY principle, but I'll explain next why it is acceptable and inavoidable in reporting.

Hard-coding is good

Embedded reports are shipped within an application. Most application programmers would like these reports to be generic, so that every customer could use the same reports. Unfortunately, such generic reports are not very useful. Business applications are configured with settings and customized with code changes to fit the customer's business. Reporting is almost always dependent on these customer-specific settings. Say, a customer has a classification scheme that is stored somewhere in the application, preferably so that other customers can easily have their own scheme. These classifications are then shown in data entry forms as options in dropdownlists, listboxes etc. Now, from a the information system's point of view, all these separate categories are the similar, but from a business point of view the same categories make all the difference.

Custom categories are always used as report parameters.

You may try to avoid hard-coding these attribute values in your reports, usually with extra complexity. A common example is pivoting, i.e. turning your attribute values to column headers for calculating aggregates. Usually you need pivoted data to source crosstabs and charts like stacked bars. However, T-SQL PIVOT statement uses a fixed column list, and you need to use dynamic SQL to dynamically pivot your attribute values as columns, which adds complexity. But many times generalization is a disservice to the users who may have to pick 'self-evident' values as parameters in your generic report. Well-defined reports with minimal number of mandatory user parameters are usually the most popular ones.

Then there is the problem of having the whole report depend on particular category values. Let's say a classification represents the role of a person in a project. The roles could be named like 'manager', 'salesman', 'specialist' and 'administrative', but the exact classification is fundamentally a customer's decision. What if your customer wants to report different facts for each role? Hours for the specialist, money for the salesman and number of projects for the administrator. If the only difference between roles is semantic, something that is encoded only in the role's label itself, you cannot really do a generic report.

It is also very hard to get your reports look good if you don't design with specific set of attribute values in mind. Actually, the most generic reports would probably look a lot like an OLAP browser. If you already have one, why bother wasting your time developing a similar tool that is going to be worse anyway?

In the following post I'll discuss more of my experiences with Telerik Reporting.

-mika-

Telerik RadControls Review: First Impressions

2009-07-06T01:00:00.000-07:00

Their marketing is aggressive and sometimes cheesy, but their products certainly are not. The Sofia, Bulgaria based Telerik is perhaps the best known .NET developer tool shop after Microsoft itself *. In early 2008 they released a new control library suite, RadControls For ASP.NET AJAX. The suite is large not only in the number of controls but also in its depth. Each of the 20-something controls has an amazing feature set, and most of the time it is possible to accomplish your objective in multiple ways. This is also the hardest part to swallow. For a newcomer, RadControls is not simple compared to the default ASP.NET controls. It takes time to learn enough of each control to get your job done, and each new version of RadControls has a tendency to break something that worked in the previous version. It is not fun, but it is something you'll probably have to deal with any 3rd party component library.

* DevExpress being the other contender

Writing a comprehensive, yet up-to-date review of a control toolkit is extremely hard and equally foolish. Software is soft, it changes all the time. There is no way on earth that a blogger equipped with a laptop and a quiet evening could even realize what design issues are behind these kinds of products. But I have used the toolkit regularly for about a year, our company has paid good money for it, and consequently, I have the right to complain for any reason. These reasons may be that I have made mistakes, or they may sign that the product has rough edges.

I won't bother you with the basic stuff you can see from the demos or read from the generally high quality Telerik documentation. (Advanced stuff you'll find in the Telerik forums, not in the documentation.) For the most part I've used the server-side APIs, the ones that are used in a classic ASP.NET style by setting up controls and wiring up postback events. Nothing new there. The reason for toolkit's AJAX moniker is that each RadControl now has a client-side API too. This API overlaps the server-side API somewhat but is not a proper subset. This means that you must use the client-side API for client-side features like drag and drop support. And you must learn to use Javascript to manipulate the controls and to make AJAX requests back to the server. Integration with Javascript makes developer experience a little rough but you really can't blame Telerik for it.

Because there may never be "final words" to this review, I'd better begin with the opinionated part:

Telerik RadControls is there to improve your worst stuff, not your best. If you are serious on web development, you'll write the good stuff yourself. Make no mistake about it, and the Telerik suite will be a good companion

Now, we can move on.

1. Does it look good?

Without 3rd party web controls every ASP.NET business app would be using these crappy default controls:

Instead, we can please our customers with these...

Telerik controls do look good, our salesman loves them, and we do get very positive first reactions from our customers. But first reactions won't impress anyone long and the real test is if the users are happy in the long term. Typically in business software satisfaction means silence, as no one is complaining about your application, which leads us to the next topic...

2. Does it cause unexpected support calls?

I've used four major versions (Q1 2008 - Q1 2009) and there has been a couple of support calls. RadControls for ASP.NET AJAX is a new and evolving toolset. Not surprisingly, there is considerable variation in stability and feature-completeness of the different controls.

Some of the issues surface after upgrades, especially with completely new controls like the Scheduler. A few examples.

In Q1 2009 version the Gray skin was dumped. God knows why. Now, if you had defined a skin="gray" in any of the dozens of controls in the code, you would get a nasty runtime error. No excuses here, at least they could have fell back to the default skin.
Scheduler has lived a lot. Q1 2009 had nice improvements in how you can color-code resources. Color-coding is needed for example when you have multiple persons in a calendar view and want each person to have a different color. But again, after this improvement was made the old scheme of coloring things went broke. This was the second time I lost resource coloring in the Scheduler, first time due to CSS changes. But I think the feature has settled now.

As far as I can tell, upgrades have not caused any serious support calls. The most serious ones have come from bugs, and the only fix then has been to revert back to the regular ASP.NET controls.

Let me give a specific example. Perhaps the most commonly used (and abused) radcontrol would be the RadWindow, aka the DHTML popup. You can use RadWindow to get rid of window.open() popup windows that normally would invoke the browser's popup blocker. But we haven't found RadWindow reliable enough to replace all popup windows. There has been problems in situations where the underlying page scrolls down and has in-page anchors. Confused users have lost the RadWindow's toolbar and have not been able to close it. Another problem with all DHTML popups is that the browser window that hosts the DHTML popup has to be large enough for the popup to fit properly. These issues make you wary to use RadWindow. And there are probably better ASP.NET AJAX popup controls around.

Last, browser compatibility is little risky. Last time I tested the Scheduler, there were substantial differences on the size of the month view grid cells across browsers. Because you can define an absolute pixel size for the grid, some browsers show a scrollbar and others do not. This is a minor issue that you face when you try to use a given resolution efficiently. But perhaps the biggest difference between browsers these days is in Javascript performance. If you have multiple data-intensive RadControls in the same screen, you are going to see a HUGE preformance (and usability) difference between, say, Chrome and IE6.

3. Is it developer friendly?

For the developers, installation is straightforward. There is exactly one file to distribute with your web application, Telerik.Web.UI.dll. This dll is quite large, over 13MB and it contains everything: ASP.NET controls, graphics, css and javascript. For the developer machine, the installer will provide example projects and Visual Studio designer support.

As mentioned earlier, developing for the Javascript API is radically different from the server-side .NET API. You should equip yourself with a javascript debugger (Firebug) and find the best setup for javascript development. For example, I mostly work with DotNetNuke modules that are ascx controls. Visual Studio 2008 javascript debugging does not work in this environment at all. Only reasonable setup I have found is to write javascript code in a script block inside the ascx file, not in a separate .js file, and test it with Firebug.

Telerik has really understood the meaning of developer support. They are offering basically free support for everyone, customers and trial users alike. This is amazing and they have done it right. The cornerstone of Telerik support are the forums where Telerik staff is actively present helping their users. Solving the users' actual technical problems creates a lot of real value. For developer tools, there is no better alternative. Simplified examples and otherwise high-quality documentation are just not good enough these days. Users of developer tools are familiar with searching for bits here and there and reading a lot. And writing good questions.

I work in a SaaS company myself, and I know what it means to do both development and support. For most of the time, it sucks. You try to focus on your development effort, keep deadlines and all, but the phone keeps ringing and !-flagged email keeps coming. I don't remember a single day I could have planned my work. The urgent always overrides the important. It takes service design to overcome this mess and I think Telerik has succeeded here well. I mean, they provide better service for free than we could ever manage to provide for money.

On the other hand, I think it is a matter of survival in the developer tools business to have high quality support. It is a global market where only the best can exist. If a commercial tool has worse support than free or open source alternatives, there is no reason to pay anything.

4. What your money does not buy?

If you look at recommendations on how to buy software, there is one thing that Telerik does not have. There is no money-back guarantee. DevExpress products do have a 60-day money back guarantee, so in this respect Telerik is definitely worse.

Trial version of RadControls is fully functional but will occasionally show trial licence information to the end user.

Also, if you want the source code of RadControls included, you'll have to buy a developer licence with subscription and priority support. Telerik Premium collection bundle does have one-year subscription and support, but with the cheapest RadControls for ASP.NET AJAX licence you won't get source code or major (quartal) version updates.

-mika-

Google Notebook End-of-Development

2009-05-23T03:04:00.000-07:00

This is old news: Google is ceasing the developent of Notebook. Personally, this was a sad moment. Even with its rough edges, Notebook is a very unique tool that fits my information needs well. And it still does. I haven't noticed any decrease in the service since this announcement in January. This post tries to explain why Google Notebook is such an excellent application. It also presents a fast spec of the features that I see essential in any notebook app.

The Big Picture

Google Notebook does exactly what it says. It replaces paper notebooks. To be fair, I still use one paper notepad to write down notes during client phone calls but other than that, I use Notebook. I have zero tolerance for Post-It notes and the like. No paper on my desk. Even the paper notepad is just like a temp file and will eventually end up in shredder.

In addition to replacing paper, Notebook substitutes electronic notepads of which there many variants:

Email sent to your own email address
Local text files and Word documents
Online text files like Google docs

All of these are improvements over paper in three areas: management, search and sharing.

1. Management

This is the reason people started to use computers instead of typewriters in the first place. By separating the act of writing from the physical output, the management of written material gets more flexible. You can make identical copies of your documents easily. You can categorize documents without reshuffling drawers. You can sort documents fast by different criteria and transfer large piles of them in small-size storage media or through communication networks. These same networks also enable you to to access documents remotely.

2. Search

Electronic documents can be queried with the help of computing power. Compared to a computer, humans are very slow at doing keyword searches over large bodies of text.

As the number of documents in a collection increases, search becomes more valuable. If you intend to save all your notes in one system like Notebook, search is likely to be the most important feature. It is somewhat surprising to see that Google Notebook has perhaps the worst search engine ever released by Google. The first thing that reveals a crappy search engine is that retrieval is very sensitive to inflectional forms.

A simple example. Three notes:

"cat"
"cats"
"cat cats"

In Notebook, queries for "cat" and "cats" will return different results, except if the result contains both forms. in most cases, even grep would perform better. However, you can use labels in Notebook to somewhat improve search recall.

1. Search for "cats" does not find "cat"

2. Search for "cat" does not find "cats"

3. Search for "cat" will find all notes labeled with "cat".

3. Sharing

Third improvement in electronic documents over paper is the ability to share them more efficiently. There are four modes of sharing:

one reader/one writer: distributed copies of the same files
multiple readers/one writer: write-locked files in a network share
multiple readers/multiple writers: database applications
(one reader/multiple writers is so rare I can't find a single example)

Distributing copies is the traditional method of sharing electronic documents. This is what you do with email or file transfer. The problem is in distributing and merging changes. In any actively developed document you will need a version control system to handle distribution and merging. Most programmers are well aware of this, since program code files are such documents.

When network storage started to get common, people began sharing documents, usually MS Office files, in a network share. The first one who opened the file got the write lock and the rest of the collaborators were just readers. In Excel there was even an option to switch the write access between users, and even secure the write access with an edit password. This scheme works for small groups, but it contributed to the way of managing corporate data that is commonly known as "Excel hell", a sea of somewhat connected Excel documents. A large part of information systems history of the 21st century can be summarized as reconstructing these shared excel files as database-backed web applications.

Finally, database systems are designed for concurrent writing, so that online services like Google Docs, Notebook and Wikipedia allow multiple writers to modify a document at the same time. Of course, this makes version conflicts possible and require some sort of a version control scheme.

I have never used Google Notebook sharing features, since I don't work with other people familiar with the application. There is a notebook-level sharing with edit priviledges to everyone invited, and also an option to publish the notebook as a static web page. Notebook-level sharing is probably a good choice, since you can keep private notebooks easily separated from a shared one and yet move notes easily between each one.

Notebook != Word Processor

There are a few features that make a notebook app unique. First, note taking is not document preparation or word processing. Printing is not a primary concern here*. Notes are all about planning and observing, they are like a personal ERP system.

* before laptops and online apps I used to print summary todo-lists out of my notes

Task management

The closest thing to compare Notebook is a task management application like RememberTheMilk. The key feature in both is the speed an ease of creating new tasks/notes. While task management focuses on due dates and reminders, a notebook focuses on planning with or without time constraints. It is still a useful feature in a notebook app to include due dates and especially have a simple tool to attach formatted 'Now' timestamps. Timestamps are extremely useful in notes but in Google Notebook you either have to write them yourself or use an external tool to generate one to clipboard.

Project management

Google Notebook supports grouping of notes well. This means that every user can customize the browsing experience. There are two levels of grouping: notebooks and sections, which is enough for most users.

For me, a single note corresponds nicely with a small project, especially if the project has some logical end. This kind of a projects consist of tasks that are simple one-line items. A notebook consists of similar projects. I always had a section "Old" in every Notebook. When a project is finished, I moved it to the old category. Sometimes I use an "Active" section on top of the notebook, with all the active projects.

Things to remember

I have found notebook very handy in storing snippets of knowledge that I may need another time but don't want to search the hard way. Software development and system administration is full of such small contextual things. These should be easy to transfer to Google Docs if Notebook is eventually shut down.

Final Words

Notebook is essentially an unstructured or loosely structured information system, with full-text search and the ability to organize notes to relevant categories. Tagging helps to connect notes in different categories.

In the current Internet ecosystem, it is probably hard to make money out of a Notebook-style application. Even Google couldn't sustain it with all the advertising infrastructure they had in place. Notebooks are private and sketchy and I think they should be paid for like their paper counterparts. One dollar/euro a month would be a reasonable price, and the payment should be smooth.

-mika-

A Postmortem of a Failed Trigger Integration

2009-04-26T13:18:00.000-07:00

Last summer I wrote an admittedly noninspiring post on trigger integration between two database instances. Guess what? The solution failed and I had to start over. Actually, all this happened before Christmas, but until now there has not been enough time to write a postmortem.

Good thing is that the new, rewritten solution works fine. I think that the last modifications were made early March and since then nothing has been heard of the client. Nothing, total silence. Now, for those not in the field, silence is about as close to success as you can get in the world of business software. Unless your users have abandoned your software, that is.

But before this bliss there were issues... so what went wrong? Well, the first thing thing that went wrong was that I was unprepared. I should have at least skimmed a book like Enterprise Integration Patterns before starting the project. Second, I should have better understood why our client needed the integration to make better decisions during initial development, especially during source-to-target mapping. Basically this was an application integration (EAI) project that later turned into an operative data integration (ETL) project. There are key differences between the two, and the reason of turning to data-level integration was that one endpoint, a 90's heritage windows client app, was totally out of my control.

Wrong assumption: Simplicity

The original integration plan was built on a premise of simplicity. There would be two applications with two tables on each side (a person and a company tables), and they would stay synchronized via triggers on both ends. When a user adds a person in one application UI, a transaction would be triggered that would write the same data to the other application's person table. Or abort the whole thing, so that there would no insert in either.

Unfortunately, things were not that simple. In a normalized database schema, adding a new person through a UI will probably read and write multiple tables. Phone numbers and email addresses would go to some contact table, profile data like marketing restrictions to a profile table, and there would be a references from persons to a company table for all persons whose employers were known.

Solving all this complexity took time but eventually it worked. During testing all CRUD operations worked fine on both ends. So the increased complexity caused by dependencies did not break the system. But it made it fragile and too weak to withstand the final blow.

After testing, the integration was rolled out into production. Soon afterwards reports started to come in, and finally the client's representative called and said that they couldn't do any work. I took down the integration temporarily, waited until late evening, and started to debug in the production environment.

What I found was a deadlock in the legacy client-server application. There were no reported problems on the new web application side. But the catch was that this old client-server app had actually two fat client UIs, one of which was not available during testing. And this second one wrote to tables in slightly different order than the first, tested client, and would therefore fire triggers in different order than planned and cause a deadlock.

Now, at this point it became clear that the legacy application should not play any role in the integration. Not even the role of firing a trigger when a "Save" button was pressed. The solution needed to be uncoupled from the client app.

Wrong technology: Triggers

Some of the readers might have noticed the words trigger integration at the beginning. Triggers are not in fashion these days. The problem with triggers is that anything unexpected breaks them, and it is painful to debug them. This pretty much goes to declaratice, set-based SQL in general. Most programmers, myself included, are not very productive writing professional quality software with T-SQL. For example, an integration solution should have error logging facilities and behave reasonably well when networking is down. But I wouldn't expect .NET developers to write as high quality exception handling in T-SQL as they do in .NET languages.

Right technology: Asynchronous broker service

The new solution was written in C# as a Windows service. It collects change data in both systems, resolves any conflicts between them and calculates a change set to be written to each database. This would be scheduled to run regularly like once in an hour. There are two major design considerations in building this kind of a synchronization:

how to collect change data
how to resolve conflicts

Avoiding wrong choices here was actually one factor in taking the synchronous path first. In a T-SQL trigger, change data would be stored in inserted and deleted tables. Also, there would be no conflicts because transactions would take care of concurrent updates.

Without going to details here, a user-table level change data capture can use timestamps columns in the tracked table or maintain a separate change table. The problem with timestamps is that you have to trust them. In two servers, clocks will go out-of sync unless a time server is used. Second, delete operations should be lazy, otherwise they cannot be detected.

In our case, time service was provided by Active Directory domain controller and there was no hard deletes. Conflict resolution could be handled by selecting the row with the most recent timestamp. So far, this simple solution has worked well enough.

-mika-

Real-Time Data Warehouse, An Introduction

2009-04-20T11:29:00.000-07:00

This is the first part of an ongoing series on Real-Time Data Warehousing. The series consists of draft chapters of my master's thesis, with added links to relevant sources. When possible, topics discussed are demonstrated with Microsft SQL Server 2008. Hope you enjoy!

-mika-

INTRODUCTION

Real-time data warehouse (RTDW) is an analytical database that supports soft real-time reporting applications. A basic requirement for these applications is continuous availability of fresh data. Soft real-time applications are usually performing operative rather than strategic or analytical tasks. However, real-time reporting with read-only semantics is not an OLTP application. RTDW has to serve concurrently multiple read transactions and one write transaction. The problem is to optimize for read transactions without failing to load data in time. A traditional DW loading methodology is to take the warehouse off-line for maintenance at most once a day. Online maintenance with frequently executed update transactions is much less well known and is considered risky by many experts. It also introduces query contention bottlenecks, isolation anomalies and view maintenance phenomena not seen in a DW where reader and maintenance transactions are never executed concurrently [LYS07, MTK06].

This report focuses on online DW maintenance with soft deadlines. When applicable, Microsoft SQL Server database system is used to present examples of RTDW techniques. No actual experiments or performance benchmarks are planned. Recently published methods like the MeshJoin algorithm for storing streaming updates to database tables [PSV08] are not easily demonstrated with present-day database systems and ETL tools.

There are three sequential phases in a maintenance cycle, each having an effect on the propagation time of changes from the source systems to the DW:

change data capture (CDC) in the source systems [Oracle, MSSQL],
extract, transform and load process (ETL) and
incremental maintenance of materialized views (in the DW or OLAP DBMS).

The purpose of CDC is to avoid full load of the source system to the DW by providing change data for the ETL process. Changes in data can be detected from the transaction log, with the help of triggers or by setting up a transactional replica that can pull changes from its master as needed. Trigger-based CDC has two drawbacks: it requires changes in the user databases and there is an immediate performance penalty each time a trigger is fired. For these reasons, trigger-based CDC has gone out of favor recently. Log-based CDC is less intrusive to the user tables, as the preferred method is to use system tables to store change data acquired from the transaction log. Log-based CDC can be implemented using log APIs published by DBMS vendors. In fact, Microsoft SQL Server 2008 CDC tools are just a set of public stored procedures and functions written on top of the log API. Third alternative is to set up a transactional replica to collect data, which also avoids the drawbacks associated with triggers. What CDC method is best depends mostly on the available tools. As a database research topic, CDC barely does exist. It can be considered more as a part of the skillset of a working database professional. This does not diminish its importance in RTDW: without CDC, ETL loads from the source systems would take a lot more time.

Designing and developing an ETL system takes the most man hours in virtually every DW project. A brief description of the work involved is given in a promptly named article "38 subsystems of ETL" [Kim04]. Also, running an ETL process is usually the most time consuming phase of the maintenance cycle [MTK06]. Consequently, any treatment on DW systems must focus on ETL. In RTDW the natural conceptual model for ETL process is a queuing network of streaming inputs, processing nodes and endpoints (i.e., tables in the DW) [KVP05]. Current ETL tools in the market do not fully reflect this model. To be fair, ETL tools like Microsoft SQL Server Integration Services (SSIS) are modeled as a pipeline of data and control flows, but they do not support continuous input streams. Instead, they are designed to handle large bulk loads efficiently. With the current ETL tools one can simulate a continuous ETL process by increasing the frequency of ETL batch loads. Queuing network models with streaming inputs serve primarily as a model for estimating the highest possible ETL batch frequency. It is likely that running and ETL process once a day presents different challenges than running the same process once a minute, 1440 times a day. Robustness of the ETL process usually becomes more important. In many real-time reporting applications it is better to have approximate real-time data instead of delayed but complete data. For example, if a delay is caused by a queues, system quality could be improved by shedding, i.e. dropping a fraction of the incoming data stream, and loading it later completely [PSV08].

Materialized view is a cached view representing the result of a query. Materialized views can speed up query processing greatly, but to realize the benefits two subproblems must be solved: view matching and incremental view maintenance [LaZ07]. High-availability DW systems must also solve a third subproblem: how to maintain views without taking the system offline? View matching determines if a query can be computed from a view. View maintenance is the process of updating a view when base tables are updated. Incremental view maintenance means updating the view without rebuilding it from scratch. Maintenance of materialized views is a mature area with over 25 years of publications [ShI84]. View matching and efficient incremental view maintenance algorithms for SPJG views are well understood [LaZ07]. SPJG views are composed of selection, projection and inner joins with an optional group-by operation. Latest development is to extend view support to SPOJG view, that is, allow views to also contain outer joins [LaZ07]. Online view maintenance algorithms do not block reader transactions during maintenance [Gar06, KiM07]. This is achieved using two versions of the view, i.e. caching. During maintenance, readers are delivered a cached version of the view. Materialized views can be stored in a relational DW or in a separate OLAP DBMS like Microsoft SQL Server Analysis Services (SSAS). In RTDW view maintenance is triggered after every ETL load, or alternatively, after each change to the base tables. In SSAS/SSIS terminology, the first alternative means that the cube processing is scheduled as a last step in an SSIS package, and the second corresponds to using proactive caching in SSAS. Proactive caching is a configurable, online view maintenance technology that detects changes in the base tables and maintains the dependent views automatically.

Generally, a soft relative deadline on a maintenance cycle time affects the overall read performance of the DW. It is necessary to trade-off read performance in order to achieve low maintenance cycle times. The exact trade-off is determined by reporting application requirements. It is common that soft real-time reporting is a feature that is wanted more than needed, in a sense that ultimately no one is willing to pay the price of developing and operating a RTDW. It should be emphasized that a RTDW system should be considered only after finding direct reporting from OLTP database(s) insufficient. Generally, this is the case when analytical query loads are heavy or distributed queries from heterogeneous source systems are unfeasible. Security issues may also necessitate a centralized DW when source systems are administered by different organizations.

Even when a RTDW is necessary, only a small physical partition of the DW database should be updated continuously. This partition should contain intra-day updates, and it would be regularly merged with the main partition during a daily online maintenance window. Most of the index rebuilding and view maintenance would occur in this maintenance window and the real-time partition would be lightweight in both indexes and materialized views. If analytical queries are executed on an OLAP cube instead of a relational DW, the real-time partition should benefit from a ROLAP storage mode while the vast majority of the cube should use a MOLAP storage [MTK06]. Segregating the real-time (intra-day) data from the rest of the DW has emerged as a best practice to deal with the issues that real-time data warehousing presents [SaB08, MTK06].

REFERENCES

[Gar06] Garca, C. (2006), Real time self-maintenable data warehouse, in 'ACM Southeast Regional Conference', pp. 518-524.

[KVP05] Karakasidis, A.; Vassiliadis, P. & Pitoura, E. (2005), ETL Queues for Active Data Warehousing, in 'Proceedings of the 2nd International Workshop on Information Quality in Information Systems', pp. 28--39.

[Kim04] Kimball, R. The 38 Subsystems of ETL. Intelligent Enterprise

[KiM07] Kim, N. & Moon, S. (2007), 'Concurrent View Maintenance Scheme for Soft Real-Time Data Warehouse Systems', Journal of Information Science and Engineering 23(3), 723--740.

[LaZ07] Larson, P-Å. & Zhou, J. (2007), Efficient Maintenance of Materialized Outer-Join Views., in 'ICDE', IEEE, , pp. 56-65.

[LYS07] Lin, Z.; Yang, D.; Song, G. & Wang, T. (2007), Dealing with Query Contention Issue in Real-Time Data Warehouses by Dynamic Multi-level Caches, in 'CIT', pp. 122-127.

[MTK06] Mundy, J.; Thornthwaite, W. & Kimball, R. (2006), The Microsoft Data Warehouse Toolkit: With SQL Server 2005 and the Microsoft Business Intelligence Toolset, Wiley.

(Especially Chapter 17: Real-Time Business Intelligence )

[PSV08] Polytzotis, N.; Skiadopoulos, S.; Vassiliadis, P.; Simitsis, A. & Frantzell, N.-E. (2008), 'Meshing Streaming Updates with Persistent Data in an Active Data Warehouse', IEEE Transactions On Knowledge And Data Engineering 20(7), 976--991.

[SaB08] Santos, R. J. & Bernardino, J. (2008), Real-time data warehouse loading methodology, in 'IDEAS', pp. 49-58.

[ShI84] Shmueli, O. & Itai, A. (1984), Maintenance of Views., in Beatrice Yormark (Ed.): SIGMOD'84, Proceedings of Annual Meeting, Boston, Massachusetts, June 18-21, 1984. ACM Press, pp. 240-255

Full HD triggers massive upgrades

2009-03-28T07:40:00.000-07:00

So, you have bought a new Full HD TV, Congratulations! So did I, a 40" Sony KDL-40E4000.

Now, isn't it great to download HD content and stream it through wlan to your media center, like PS3?

No, it isn't if your other equipment sucks.

You'll need:

a modern 2008-level multi-core machine, preferably a new 64bit system with at least 8 gigs of RAM and plenty of disk space
802.11n wlan with some QoS implementation to prioritize your HD video streams. 802.11g with about 15mbps max is not good enough for HD if you have other traffic in your wlan.
A gigabit network connection to your media center is always good, too
And if you intend to download HD content, 10mbps internet connection is a minimum
...and you should have a HDMI output in your PC to connect the TV directly, without a media center

Besides money, you'll have to invest some time to learn about uPnP AV and the basics of MPEG-4 and different container formats, at least in order to learn their compatibility with your media center.

All this because of the new TV. Otherwise, no big hurry to upgrade the home desktop/server with all the hassle involved.

-mika-

P.S. For a PS3 uPnP server I recommend the PS3 Media Server. It crashes now and then but it's still better than Windows Media Player 11 or Tversity.

A review of RescueTime: Part 1. Why time tracking?

2009-03-13T13:20:00.000-07:00

Just about a year ago I stumbled upon RescueTime, a time tracking tool for individuals and teams. This is the first part of a two-part review I intend to write on the individual version of the application. I will also discuss the topics of economics of human time and collecting time data to a real-time data warehouse.

This part of the review will focus on economics of time tracking. Tracking life events divides people and even the most accounting-oriented persons have mixed feelings about tracking their time. Knowing your time, expenses, calories etc is both potentially insightful and yet utterly disturbing. Most of us can probably remember periods of our lives when we had to count money more carefully, and the idea of going back to penny-pinching days is indeed disturbing. Likewise, tracking time can be seen as petty waste of time. Something that busy people do only to get money from. So why bother for nothing?

There are a few reasons that make time tracking sensible.

First, time is always a scarce resource. There is a fundamental difference between counting cents and counting seconds: money has no hard upper limit for an individual, but life definitely has. It follows, that if you compare the utility of an extra unit of free time versus an extra unit of money, the marginal utility of money tends to decrease more than the marginal utility of time. There are two factors that amplify this:

employment increases money and decreases unallocated time
getting older decreases one's available time by definition, but it also increases one's wealth from two sources: inheritance from the deceased and interest payments from accrued wealth.

These facts can be summarized as follows:

The more money you have the less wealthy you are in time, relatively.

Second reason to track your time is that you can actually waste time voluntarily. There are places in society that restrict individual's use of time. Like the army for example. I've been in the draft army and I know that tracking every minute of time does not make any sense there. You are stuck in the woods and that's about it. You are not wasting time because you don't have any choice. Wasting time is a concept that presumes individual liberty.

Third, you have to have plans. Without plans there is no difference between time well spent and waisted. Everyone has plans but there are huge differences in commitment. Time tracking is a clear statement that one is committed to his/her plans. At least in the most measurable level.

So far, I have tried to describe an image of the people who should be interested in time tracking. They are relatively well-off, determined people who have long-term plans. Now that is a good clientele. These people are perhaps busy with their current commitments and distractions surrounding them, but they do want to make progress. Time tracking tool has to offer them something they couldn't do themselves, a value proposition. Doing tracking yourself generally means doing tracking with Microsoft Excel, the world's tool of choice for semistructured data management. So any new tool has to beat Excel, but that is only a start.

So what is RescueTime good for?

Even if time tracking creates value, it may not create net value. The costs of tracking might be too high. The first and foremost cost in time tracking is the input time cost. This is where RescueTime has made a big effort. They have really focused on keeping the input costs low. It is not even possible to input timesheets manually to RescueTime, at least not from the UI. This prevents certain scenarios like project billing but I think that the decision is right. First, you have to get reliable, objective data on your time use and only then you can make judgements on more subjective items like projects. After all, project billing is not about counting time, it is all about putting a price to a work agreement.

In part 2, I'll discuss more about RescueTime, its data collector, reporting UI, and the whole data-warehouse centric application model in general. This will rely heavily on my current academic focus, the trickle-loaded real-time data warehouse. Stay tuned!

-mika-

Addicted to a website? A simple method to get rid of the bad habit

2009-03-06T04:11:00.000-08:00

Like it or not, consumer websites are designed to be distractive or even addictive, regardless of the genre. You can get addicted to social media, news sites, financial information, tech sites, blogs, porn videos, image boards, whatever. And just like in substance abuse, it is possible to get rid of the harmful consequences of addiction by abstinence. But abstinence also takes away any good consequences of the addictive substance, so it is a trade-off.

This is simple behavioural economics. There is no reason why anyone of us couldn't find a suitable way to manage one's behavior. I suggest a two-step iterative method presented below. But I also want to point out that testing and comparing the statistical effectiveness of these kinds of self-help methods is completely another matter. Resolving that would take a serious study.

The 2 steps are:

1. Measure you computer use

The rational thing to do when you realize having a bad habit is to measure its effect. RescueTime is a great tool for this, it gives you good reports on how much time you spend on your computer(s). It reports every site you visit + every application you use. You should use it for at least a couple of weeks to gather enough data.

2. Ban the harmful sites

Once you have found that a site or a group of sites is harmful, the next thing to do is to make it hard to access them. There are multiple tools for this, but the simplest one is to route the domain to localhost. This means editing your hosts file.

You can find the hosts file in C:\Windows\System32\drivers\etc for Windows and in /etc/hosts for OS X and Linux. It is named hosts without an extension.

Open the hosts file in a text editor (you will need administrator priviledges) and add one line for each domain in your blacklist, like:

127.0.0.1 finance.google.com

127.0.0.1 icanhascheezburger.com

Now, every access to these domains by any application will be routed to your local machine and you won't get to the site. Nice. If you really need to access any of the banned sites, just comment out the that line with a #. Frankly, I can't figure out an easier way to blacklist sites, but there is a whole industry to make this process easier or user-specific. The user-specific thing is usually called "parental control".

(3. Adjust and iterate)

After taking these steps, measure your internet usage again for a few weeks. There might be bad substitutes that have taken the place of the banned domains. You might consider banning them too. Sometimes it is best to take domains out of the blacklist. After iterating for a while you should have found a balance.

This kind of self-denial is somewhat humiliating. But I think it is a rational way to deal with internet distractions. If you spend much of your life online, adjusting your "diet" can pay off well.

-mika-

Face Time Does Not Scale

2009-02-20T06:37:00.000-08:00

Corporate IT is all about service. If you hold an IT job, your job is to provide a service to someone. No one gets around it, least the senior management whose job is mostly to serve the clients' senior managers AND their own shareholders.

Everybody wants to serve well when their own face is at stake. Bad service is almost always anonymous in some respect: an email from a customer service address, an unknown voice in the support line, an invoice from the phone company.

There is one rule here: The more customers pay, the more personal service they want, and the more they will usually get. If you buy a Windows Vista licence with your laptop, you probably won't get Steve Ballmer to personally answer to your printer troubles. But if Dell buys and preinstalls millions of Vistas, and they have a huge pain in their ass from it, they will raise the question to Mr. Ballmer and he will answer.

As much as we like good personal service, there is a flipside. Production of highly personal service does not scale. It does not scale because people have only 24 hours in a day. As a result, there is a rigid hierarchy and rationing in personal services. As a private citizen and consumer, each of us gets face time primarily from the lower ranks of any organization. And the less we pay, the less personal this service tends to be. This is simple economics of rationing human attention.

But as anyone who has worked in service business knows, there is a huge loophole here. That is, bad customers take a disproportionate amount of resources from a service company. Usually by being a nagging customer, you get more and better quality time. For free. Bad customers take advantage of people's desire to keep a good reputation. They won't use anonymous customer service channels but take direct contact. And it is damn hard to say no to them.

But that is exactly what needs to be said. Sooner or later. The only way to deal with a hard customer is to intentionally decrease the level of service. Otherwise you'll have to decrease the service level of the other customers. I don't think there is a general recipe on how you should actually deal with a bad customer, but I do know one thing. If your manager ignores your complaints about a bugging customer, then it is your manager's fault. Service managers just have to know the amount of service their organization provides to each customer and what the company is getting out of it.

It is always sad to realize that a customer has abused your goodwill. Many times these problem customers just vanish when the issue is taken seriously. If the very people that you have paid the most attention just go away, it means that they really didn't want the service. Not for the full price. But not all hard customers are intentionally jerks, they just might be clueless. The really hard problem in human relations is to take a stand when things are not black and white, which they seldom are.

-mika-

Unit testing DotNetNuke projects just became easier

2009-01-14T02:17:00.000-08:00

Unit testing DotNetNuke projects is hard. A typical DNN module is an ASP.NET Web Application Project that includes all the normal WebForms dependencies + DotNetNuke specific dependencies. Visual Basic is more common than C#. DotNetNuke project team has put a considerable effort to make the latest release of the framework more testable, but there are still a number of roadblocks. Charles Nurse of the core team has posted a good series on creating testable modules for DNN based on the Model-View-Presenter pattern.

However, not everyone has the luxury of rearchitecting their webapps to a more testable design. If unit testing WebForms is hard, unit testing legacy WebForms is downright painful. Unless, you can somehow create tests without changing existing code. This is where mock objects come in. Traditional mocking relies in interfaces that you hand-code for the objects you want to mock. This process can hardly be justified in itself, it is more like a tax to be paid to get unit tests in place. A more flexible mock system such as TypeMock Isolator does not require such effort. Unfortunately, TypeMock has not been too useful to DNN developers because it has not supported VB. Until now, that is.

With TypeMock 5.2 here, there is a new power tool to help DNN developers create testable and more maintainable code. It costs money, but in any serious work your time will cost many times more. I don't know yet if TypeMock will save enough time to justify its cost, but it has certainly raised high hopes.

For more information, check the (yet) unpaid ad below ;)

-mika-

[Update: I received a free licence in January 21st. After I have played around, a review post will follow. Thank you!!! ]

*Begin*

Programming Visual Basic applications?

Typemock have released a new version of their unit testing tool, Typemock Isolator 5.2.
This version includes a new friendly VB.NET API which makes Isolator the best Isolation tool for unit testing A Visual Basic (VB) .NET application.

Isolator now allows unit testing in VB or C# for many ‘hard to test’ technologies such as SharePoint, ASP.NET MVC, partial support for Silverlight, WPF, LINQ, WF, Entity Framework, WCF unit testing and more.

Note that the first 25 bloggers who blog this text in their blog and tell us about it, will get a Free Full Isolator license (C#, VB, and Sharepoint included - worth $139 !!!). If you post this in a VB.NET dedicated blog, you'll get a license automatically (even if more than 25 submit) during the first week of this announcement.

Go ahead, click the following link for more information on how to get your free license.

*End*

Windows Vista greets Moore's law: 64-bit desktop is here

2008-12-25T12:34:00.000-08:00

Windows Vista was sold on security. This pitch didn't work and that is a good thing. You don't sell security. Security is bought out of fear, it is reactionary, not innovative. However you put it, hype on security is lame. You need more than fear to have a compelling value proposition, something like features and convenience. Vista didn't have these and most of us stayed with Windows XP.

But guess what happened in 2008? Moore's law ran over 32-bit operating systems. We have more memory than 32-bit x86 architecture can handle without issues. Many of us started with 2GB of RAM and now have 4GB. Even my 60-year old parents bought a new ultra-cheap desktop with 3GB preinstalled.

In 2009, 4GB is common and 8GB is probably what you really want. This means that you'll need a 64bit operating system. In Windows, your options will mostly be limited to Vista 64-bit right now and Windows 7 later. There is no way around it and that is why Vista 64-bit does not need to be sold. It will sell itself.

Even if it's a forced update, 64-bit Vista promises security and stability improvements compared to XP. All 64-bit drivers will be signed, hopefully resulting in fewer blue screens.

-mika-