Omar AL Zabir blog on ASP.NET Ajax and .NET 3.5

MSDN Day @ Dhaka

omar — Wed, 17 Jun 2009 08:53:57 GMT

Venue. IDB Auditorium E/8-A Rokeya Sharani, Sher-e-Bangla Nagar, Agargaon, Dhaka 1207 Saturday June 20th 9:00 AM to 6:00 PM Microsoft Community in Bangladesh proudly presents Microsoft Day @ Dhaka. This is a special day dedicated to all Microsoft technology professionals and students in Bangladesh. We will be having the best Microsoft community technologists from Bangladesh - Microsoft Most Valuable Professionals (MVPs) delivering sessions at the event. This technology marathon is a great opportunity to learn from the best and network with each other. Both Microsoft developers and networking professionals would find the event worth attending. The event will also feature a demo bonanza with Windows 7 and an extensive Question and Answer panel with the Bangladesh MVPs to answer your queries. http://msdnbangladesh.net/content/msday.aspx Please register soon! Limited sits. I have a session there as well. AUDITORIUM – Dev Track 9:00 - 9:30: Opening Speech - Feroz Mahmood 9:30 - 10:30: Development in ASP.NET [LINQ, Web Forms, Dynamic Data] - Tanzim Saqib 10:30 - 11:15: My First ASP.NET MVC App - Mehfuz Hossain 11:15 - 11:45 : Unit Testing in MVC and Demo of dotnetshoutouts.com - Kazi Manzur Rashid 11:45- 12:30: Developing in Silverlight - Faisal Hossain Khan 12:30 - 1:30: Lunch 1:30 - 2:00 : Introduction to Sharepoint/ MOSS - Jannatul Ferdous 2:00 - 2:45: Production Challenges of ASP.NET Websites - Omar Al Zabir 2:45 - 3:15: Windows Azure - Ashic 3:15 - 3:45: Tea Break 3:45 - 4:30: Overview of Visual Studio Team System 2010 - Mohammad Ashraful Alam 4:30 - 5:30: Features of Windows 7 - IE8 and Windows Live Features - Omi Azad BREAK OUT – IT Pro Track 9:00 - 9:30: Opening Speech - Feroz Mahmood [IN AUDITORIUM] 9:30 – 10:30: Exchange Server 2010 10:30 - 11:30: Windows Server 2008 - Virtualisation & HyperV 11:30 - 12:30: Talking Windows Server 2008 and R2 [Windows Client & Windows Server 2008 NAP – Better Together] - Anwar Hossain (Technical Specialist, MS Bangladesh) 12:30 - 1:30: Lunch 1:30 - 2:15: Session on MS Project & EPM : M. Manzurur Rahman (CEO, ICT Alliance) 2:15 - 3:00: Office 2007 3:00 - 3:30: Tea Break 3:30 - 4:30: Introduction to SQL Server 2008 4:30 - 5:30: Features of Windows 7 - IE8 and Windows Live Features - Omi Azad [IN AUDITORIUM]

Visit my Blog for more details.

ASP.NET AJAX testing made easy using Visual Studio 2008 Web Test

omar — Mon, 25 May 2009 11:44:32 GMT

Visual Studio 2008 comes with rich Web Testing support, but it’s not rich enough to test highly dynamic AJAX websites where the page content is generated dynamically from database and the same page output changes very frequently based on some external data source e.g. RSS feed. Although you can use the Web Test Record feature to record some browser actions by running a real browser and then play it back. But if the page that you are testing changes everytime you visit the page, then your recorded tests no longer work as expected. The problem with recorded Web Test is that it stores the generated ASP.NET Control ID, Form field names inside the test. If the page is no longer producing the same ASP.NET Control ID or same Form fields, then the recorded test no longer works. A very simple example is in VS Web Test, you can say “click the button with ID ctrl00_UpdatePanel003_SubmitButton002”, but you cannot say “click the 2nd Submit button inside the third UpdatePanel”. Another key limitation is in Web Tests, you cannot address Controls using the Server side Control ID like “SubmitButton”. You have to always use the generated Client ID which is something weird like “ctrl_00_SomeControl001_SubmitButton”. Moreover, if you are making AJAX calls where certain call returns some JSON or updates some UpdatePanel and then based on the server returned response, you want to make further AJAX calls or post the refreshed UpdatePanel, then recorded tests don’t work properly. You *do* have the option to write the tests hand coded and write code to handle such scenario but it’s pretty difficult to write hand coded tests when you are using UpdatePanels because you have to keep track of the page viewstates, form hidden variables etc across async post backs. So, I have built a library that makes it significantly easier to test dynamic AJAX websites and UpdatePanel rich web pages. There are several ExtractionRule and ValidationRule available in the library which makes testing Cookies, Response Headers, JSON output, discovering all UpdatePanel in a page, finding controls in the response body, finding controls inside some UpdatePanel all very easy. First, let me give you an example of what can be tested using this library. My open source project Dropthings produces a Web 2.0 Start Page where the page is composed of widgets. Each widget is composed of two UpdatePanel. There’s a header area in each widget which is one UpdatePanel and the body area is another UpdatePanel. Each widget is rendered from database using the unique ID of the widget row, which is an INT IDENTITY. Every page has unique widgets, with unique ASP.NET Control ID. As a result, there’s no way you can record a test and play it back because none of the ASP.NET Control IDs are ever same for the same page on different visits. This is where my library comes to the rescue. See the web test I did: This test simulates an anonymous user visit. When anonymous user visits Dropthings for the first...

Visit my Blog for more details.

Web 2.0 AJAX Portal using jQuery, ASP.NET 3.5, Silverlight, Linq to SQL, WF and Unity

omar — Wed, 08 Apr 2009 18:16:22 GMT

Dropthings – my open source Web 2.0 Ajax Portal has gone through a technology overhauling. Previously it was built using ASP.NET AJAX, a little bit of Workflow Foundation and Linq to SQL. Now Dropthings boasts full jQuery front-end combined with ASP.NET AJAX UpdatePanel, Silverlight widget, full Workflow Foundation implementation on the business layer, 100% Linq to SQL Compiled Queries on the data access layer, Dependency Injection and Inversion of Control (IoC) using Microsoft Enterprise Library 4.1 and Unity. It also has a ASP.NET AJAX Web Test framework that makes it real easy to write Web Tests that simulates real user actions on AJAX web pages. This article will walk you through the challenges in getting these new technologies to work in an ASP.NET website and how performance, scalability, extensibility and maintainability has significantly improved by the new technologies. Dropthings has been licensed for commercial use by prominent companies including BT Business, Intel, Microsoft IS, Denmark Government portal for Citizens; Startups like Limead and many more. So, this is serious stuff! There’s a very cool open source implementation of Dropthings framework available at National University of Singapore portal. Visit: http://dropthings.omaralzabir.com I have published a new article on this on CodeProject: http://www.codeproject.com/KB/ajax/Web20Portal.aspx Get the source code Latest source code is hosted at Google code: http://code.google.com/p/dropthings There’s a CodePlex site for documentation and issue tracking: http://www.codeplex.com/dropthings You will need Visual Studio 2008 Team Suite with Service Pack 1 and Silverlight 2 SDK in order to run all the projects. If you have only Visual Studio 2008 Professional, then you will have to remove the Dropthings.Test project. New features introduced Dropthings new release has the following features: Template users – you can define a user who’s pages and widgets are used as a template for new users. Whatever you put in that template user’s pages, it will be copied for every new user. Thus this is an easier way to define the default pages and widgets for new users. Similarly you can do the same for a registered user. The template users can be defined in the web.config. Widget-to-Widget communication – Widgets can send message to each other. Widgets can subscribe to an Event Broker and exchange messages using a Pub-Sub pattern. WidgetZone – you can create any number of zones in any shape on the page. You can have widgets laid in horizontal layout, you can have zones on different places on the page and so on. With this zone model, you are no longer limited to the Page-Column model where you could only have N vertical columns. Role based widgets – now widgets are mapped to roles so that you can allow different users to see different widget list using ManageWidgetPersmission.aspx. Role based page setup – you can define page setup for different roles. For...

Visit my Blog for more details.

Memory Leak with delegates and workflow foundation

omar — Sat, 14 Mar 2009 08:35:53 GMT

Recently after Load Testing my open source project Dropthings, I encountered a lot of memory leak. I found lots of Workflow Instances and Linq Entities were left in memory and never collected. After profiling the web application using .NET Memory Profiler, it showed the real picture: It shows you that instances of the several types are being created but not being removed. You see the “New” column has positive value, but the “Remove” column has 0. That means new instances are being created, but not removed. Basically the way you do Memory Profiling is, you take two snapshots. Say you take one snapshot when you first visit your website. Then you do some action on the website that results in allocation of objects. Then you take another snapshot. When you compare both snapshots, you can see how many instances of classes were created between these two snapshots and how many were removed. If they are not equal, then you have leak. Generally in web application many objects are created on every page hit and the end of the request, all those objects are supposed to be released. If they are not released, then we have a problem. But that’s the scenario for desktop applications because in a desktop application, objects can remain in memory until app is closed. But you should know best from the code which objects were supposed to go out of scope and get released. For beginners, leak means objects are being allocated but not being freed because someone is holding reference to the objects. When objects leak, they remain in memory forever, until the process (or app domain) is closed. So, if you have a leaky website, your website is continuously taking up memory until it runs out of memory on the web server and thus crash. So, memory leak is a bad – it prevents you from running your product for long duration and requires frequent restart of app pool. So, the above screenshot shows Workflow and Linq related classes are not being removed, and thus leaking. This means somewhere workflow instances are not being released and thus all workflow related objects are remaining. You can see the number is same 48 for all workflow related objects. This is a good indication that, almost every instance of workflow is leaked because there were total 48 workflows created and ran. Moreover it indicates we have a leak from a top Workflow instance level, not in some specific Activity or somewhere deep in the code. As the workflows use Linq stuff, they held reference to the Linq stuffs and thus the Linq stuffs leaked as well. Sometimes you might be looking for why A is leaking. But you actually end up finding that since B was holding reference to A and B was leaking and thus A was leaking as well. This is sometimes tricky to figure out and you spend a lot of time looking at the wrong direction. Now let me show you the buggy code: ManualWorkflowSchedulerService manualScheduler = workflowRuntime.GetService<ManualWorkflowSchedulerService>(); WorkflowInstance...

Visit my Blog for more details.

Optimize ASP.NET Membership Stored Procedures for greater speed and scalability

omar — Fri, 13 Mar 2009 13:23:00 GMT

Last year at Pageflakes, when we were getting millions of hits per day, we were having query timeout due to lock timeout and Transaction Deadlock errors. These locks were produced from aspnet_Users and aspnet_Membership tables. Since both of these tables are very high read (almost every request causes a read on these tables) and high write (every anonymous visit creates a row on aspnet_Users), there were just way too many locks created on these tables per second. SQL Counters showed thousands of locks per second being created. Moreover, we had queries that would select thousands of rows from these tables frequently and thus produced more locks for longer period, forcing other queries to timeout and thus throw errors on the website. If you have read my last blog post, you know why such locks happen. Basically every table when it grows up to hold millions of records and becomes popular goes through this trouble. It’s just a part of scalability problem that is common to database. But we rarely take prevention about it in our early design. The solution is simple, you should either have WITH (NOLOCK) or SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED before SELECT queries. Either of this will do. They tell SQL Server not to hold any lock on the table while it is reading the table. If some row is locked while the read is happening, it will just ignore that row. When you are reading a table thousand times per second, without these options, you are issuing lock on many places around the table thousand times per second. It not only makes read from table slower, but also so many lock prevents insert, update, delete from happening timely and thus queries timeout. If you have queries like “show the currently online users from last one hour based on LastActivityDate field”, that is going to issue such a wide lock that even other harmless select queries will timeout. And did I tell you that there’s no index on LastActivityDate on aspnet_Users table? Now don’t blame yourself for not putting either of these options on your every stored proc and every dynamically generated SQL from the very first day. ASP.NET developers made the same mistake. You won’t see either of these used in any of the stored procs used by ASP.NET Membership. For example, the following stored proc gets called whenever you access Profile object: ALTER PROCEDURE [dbo].[aspnet_Profile_GetProperties] @ApplicationName nvarchar(256), @UserName nvarchar(256), @CurrentTimeUtc datetime AS BEGIN DECLARE @ApplicationId uniqueidentifier SELECT @ApplicationId = NULL SELECT @ApplicationId = ApplicationId FROM dbo.aspnet_Applications WHERE LOWER(@ApplicationName) = LoweredApplicationName IF (@ApplicationId IS NULL) RETURN DECLARE @UserId uniqueidentifier DECLARE @LastActivityDate datetime SELECT @UserId = NULL SELECT @UserId = UserId, @LastActivityDate = LastActivityDate FROM...

Visit my Blog for more details.

Linq to SQL solve Transaction deadlock and Query timeout problem using uncommitted reads

omar — Sat, 07 Mar 2009 07:30:52 GMT

When your database tables start accumulating thousands of rows and many users start working on the same table concurrently, SELECT queries on the tables start producing lock contentions and transaction deadlocks. This is a common problem in any high volume website. As soon as you start getting several concurrent users hitting your website that results in SELECT queries on some large table like aspnet_users table that are also being updated very frequently, you end up having one of these errors: Transaction (Process ID ##) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Or, Timeout Expired. The Timeout Period Elapsed Prior To Completion Of The Operation Or The Server Is Not Responding. The solution to these problems are – use proper index on the table and use transaction isolation level Read Uncommitted or WITH (NOLOCK) in your SELECT queries. So, if you had a query like this: SELECT * FORM aspnet_users where ApplicationID =’xxx’ AND LoweredUserName = 'someuser' You should end up having any of the above errors under high load. There are two ways to solve this: SET TRANSACTION LEVEL READ UNCOMMITTED; SELECT * FROM aspnet_Users WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser' Or use the WITH (NOLOCK): SELECT * FROM aspnet_Users WITH (NOLOCK) WHERE ApplicationID =’xxx’ AND LoweredUserName = 'someuser' The reason for the errors are that since aspnet_users is a high read and high write table, during read, the table is partially locked and during write, it is also locked. So, when the locks overlap on each other from several queries and especially when there’s a query that’s trying to read a large number of rows and thus locking large number of rows, some of the queries either timeout or produce deadlocks. Linq to Sql does not produce queries with the WITH (NOLOCK) option nor does it use READ UNCOMMITTED. So, if you are using Linq to SQL queries, you are going to end up with any of these problems on production pretty soon when your site becomes highly popular. For example, here’s a very simple query: using (var db = new DropthingsDataContext()) { var user = db.aspnet_Users.First(); var pages = user.Pages.ToList(); } DropthingsDataContext is a DataContext built from Dropthings database. When you attach SQL Profiler, you get this: You see none of the queries have READ UNCOMMITTED or WITH (NOLOCK). The fix is to do this: using (var db = new DropthingsDataContext2()) { db.Connection.Open(); db.ExecuteCommand("SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;"); var user = db.aspnet_Users.First(); var pages = user.Pages.ToList(); } This will result in the following profiler output As you see, both queries execute within the same connection and the isolation level is set before the queries execute. So, both queries enjoy the isolation level. Now there’s a...

Visit my Blog for more details.

Strongly typed workflow input and output arguments

omar — Sat, 27 Dec 2008 15:05:45 GMT

When you run a Workflow using Workflow Foundation, you pass arguments to the workflow in a Dictionary form where the type of Dictionary is Dictionary<string, object>. This means you miss the strong typing features of .NET languages. You have to know what arguments the workflow expects by looking at the Workflow public properties. Moreover, there’s no way to make arguments required. You pass parameter, expect it to run, if it throws exception, you pass more arguments, hope it works now. Similarly, if you are running workflow synchronously using ManualWorkflowSchedulerService, you expect return arguments from the Workflow immediately, but there again, you have to rely on the Dictionary key and value pair. No strong typing there as well. In order to solve this, so that you could pass Workflow arguments as strongly typed classes, you can establish a format that every Workflow has only two arguments named "Request” and “Response” and none other. Whatever needs to be passed to the Workflow and expected out of it, must be passed via Request and must be expected via Response properties. Now the type of these arguments can be workflow specific, it can be any class with one or more parameters. This way, you could write code like this: The advantages of these strongly typed approach are: Compile time validation of input parameters passed to workflow. No risk of passing unexpected object in Dictionary’s object type value. Enforce required values by creating Request objects with non-default constructor. Establish a fixed contract for Workflow input and output via the strongly typed Request and Response classes or interfaces. Validate input arguments for the Workflow directly from the Request class, without going through the overhead of running a workflow. If we follow this approach, we create workflows with only two DependencyProperty, one for Request and one for Response. Showing you an example from my open source project Dropthings, which uses Workflow for the entire Business Layer. Below you see the Workflow that executes when a new user visits Dropthings.com, creates a new user and setups all the pages and widgets for the user. It has only two Dependency property – Request and Response. The Request parameters is of type IUserVisitWorkflowRequest. So, you can pass any class as Request argument that implements the interface. Here I have used fancy inheritance to create Request object hierarchy. You don’t need to do that. Just remember, you can pass any class. You don’t even need to use interface for Request parameter. It can be a class directly. I use all these interfaces in order to facilitate Dependency Inversion. Similarly, the Response object is also a class. The Response returns quite some properties. So, it’s kinda handy to wrap them all in one property. So, there you have it, strongly typed Workflow arguments. You can attach properties of the Request object to any activity directly form the...

Visit my Blog for more details.

99.99% available ASP.NET and SQL Server SaaS Production Architecture

omar — Wed, 10 Dec 2008 08:39:00 GMT

You have a hot ASP.NET+SQL Server product, growing at thousand users per day and you have hit the limit of your own garage hosting capability. Now that you have enough VC money in your pocket, you are planning to go out and host on some real hosting facility, maybe a colocation or managed hosting. So, you are thinking, how to design a physical architecture that will ensure performance, scalability, security and availability of your product? How can you achieve four-nine (99.99%) availability? How do you securely let your development team connect to production servers? How do you choose the right hardware for web and database server? Should you use Storage Area Network (SAN) or just local disks on RAID? How do you securely connect your office computers to production environment? Here I will answer all these queries. Let me first show you a diagram that I made for Pageflakes where we ensured we get four-nine availability. Since Pageflakes is a Level 3 SaaS, it’s absolutely important that we build a high performance, highly available product that can be used from anywhere in the world 24/7 and end-user gets quick access to their content with complete personalization and customization of content and can share it with others and to the world. So, you can take this production architecture as a very good candidate for Level 3 SaaS: Here’s a CodeProject article that explains all the ideas: 99.99% available ASP.NET and SQL Server SaaS Production Architecture Hope you like it. Appreciate your vote.

Visit my Blog for more details.

Linq to SQL: Delete an entity using Primary Key only

omar — Thu, 30 Oct 2008 00:27:00 GMT

Linq to Sql does not come with a function like .Delete(ID) which allows you to delete an entity using it’s primary key. You have to first get the object that you want to delete and then call .DeleteOnSubmit(obj) to queue it for delete. Then you have to call DataContext.SubmitChanges() to play the delete queries on database. So, how to delete object without getting them from database and avoid database roundtrip? You can call this function using DeleteByPK<Employee, int>(10, dataContext); First type is the entity type and second one is the type of the primary key. If your object’s primary key is a Guid field, specify Guid instead of int. How it works: It figures out the table name and the primary key field name from the entity Then it uses the table name and primary key field name to build a DELETE query Figuring out the table name and primary key field name is a bit hard. There’s some reflection involved. The GetTableDef<TSource>() returns the table name and primary key field name for an entity. Every Linq Entity class is decorated with a Table attribute that has the table name: Then the primary key field is decorated with a Column attribute with IsPrimaryKey = true. So, using reflection we can figure out the table name and the primary key property and the field name. Here’s the code that does it: Before you scream “Reflection is SLOW!!!!” the definition is cached. So, reflection is used only once per appDomain per entity. Subsequent call is just a dictionary lookup away, which is as fast as it can get. You can also delete a collection of object without ever getting any one of them. The the following function to delete a whole bunch of objects: The code is available here: http://code.msdn.microsoft.com/DeleteEntitiesLinq

Visit my Blog for more details.

How to convince developers and management to use automated unit test for AJAX websites

omar — Mon, 27 Oct 2008 14:00:24 GMT

Everyone agrees that unit testing is a good thing, we should all write unit tests. We read articles and blogs to keep us up-to-date on what’s going on in the unit test world so that we can sound cool talking to peers at lunch. But when we really sit down and try to write unit tests ourselves – “Naaah, this is waste of time, let’s ask my QA to test it; that’s much more reliable and guaranteed way to test this. What’s the point testing these functions when there are so many other functions that we should unit test first?” Had such moment yourself or with someone else? Read on. I had a conversation with our development lead Mike (using a highly generic name since my last post caused some trouble), who runs “the show” in our engineering team. As usual there was reservation in introducing unit test to regular development schedule. Mike also had valid points about lack of powerful tools for doing unit test on AJAX websites. He also had confusion on ‘what’ and ‘how’ to unit test our code so that we aren’t just testing database failures but real user actions that executes both business and rendering logics. So, the discussion has a lot of useful information, that will help you take the right decision when you want to sell unit test to your ASP.NET and/or AJAX development team and finally to higher management so that you can buy enough time for the effort. Friday, Jan 2007 – hallway Omar: Hey Mike, we need to start doing unit testing at least on our web services. We are wasting way too much time on manual QA. Since we are an AJAX shop, unit testing all our web services should give us pretty well coverage. Mike: Sure, that sounds fun. I will do some feasibility check and see how can we chip this in into our next sprint. Friday, Feb 2007 – washroom Omar: Hey Mike, let’s start doing unit tests. I haven’t seen any tests last month. Can we start from this sprint? Mike: Sure, we can surely start from this sprint. Let me find out which tool is the right one for us. Friday, March 2007 – meeting room Omar: Hey Mike, haven’t seen any unit tests in the solution so far. Let’s seriously start writing unit tests. Did you make any plan how you want to start unit testing the webservices? Mike: Yeah, I did some digging around and found some tools. But most of them are for non-AJAX sites where you can programmatically hit a URL or programmatically do HTTP POST on a URL. You can also record button clicks and form posts from the browser. There’s Visual Studio’s Web Test, which does pretty good job recording regular ASP.NET site, but poor on AJAX sites. Moreover, you need to buy Team Suite edition to get that Web Test feature. Besides, recording tests and playing them back really does not help us because all those tests contain hard coded data. We can’t repeat a particular step many times with random data, at least not using any off-the-shelf tools. We need to test things carefully and systematically using random data set and sometimes use...

Visit my Blog for more details.

Solving common problems with Compiled Queries in Linq to Sql for high demand ASP.NET websites

omar — Mon, 27 Oct 2008 13:31:00 GMT

If you are using Linq to SQL, instead of writing regular Linq Queries, you should be using Compiled Queries. if you are building an ASP.NET web application that’s going to get thousands of hits per hour, the execution overhead of Linq queries is going to consume too much CPU and make your site slow. There’s a runtime cost associated with each and every Linq Query you write. The queries are parsed and converted to a nice SQL Statement on *every* hit. It’s not done at compile time because there’s no way to figure out what you might be sending as the parameters in the queries during runtime. So, if you have common Linq to Sql statements like the following one throughout your growing web application, you are soon going to have scalability nightmares: var query = from widget in dc.Widgets where widget.ID == id && widget.PageID == pageId select widget; var widget = query.SingleOrDefault(); There’s a nice blog post by JD Conley that shows how evil Linq to Sql queries are: You see how many times SqlVisitor.Visit is called to convert a Linq Query to its SQL representation? The runtime cost to convert a Linq query to its SQL Command representation is just way too high. Rico Mariani has a very informative performance comparison of regular Linq queries vs Compiled Linq queries performance: Compiled Query wins on every case. So, now you know about the benefits of compiled queries. If you are building ASP.NET web application that is going to get high traffic and you have a lot of Linq to Sql queries throughout your project, you have to go for compiled queries. Compiled Queries are built for this specific scenario. In this article, I will show you some steps to convert regular Linq to Sql queries to their Compiled representation and how to avoid the dreaded exception “Compiled queries across DataContexts with different LoadOptions not supported.” Here are some step by step instruction on converting a Linq to Sql query to its compiled form: First we need to find out all the external decision factors in a query. It mostly means parameters in the WHERE clause. Say, we are trying to get a user from aspnet_users table using Username and Application ID: Here, we have two external decision factor – one is the Username and another is the Application ID. So, first think this way, if you were to wrap this query in a function that will just return this query as it is, what would you do? You would create a function that takes the DataContext (dc named here), then two parameters named userName and applicationID, right? So, be it. We create one function that returns just this query: Next step is to replace this function with a Func<> representation that returns the query. This is the hard part. If you haven’t dealt with Func<> and Lambda expression before, then I suggest you read this and this and then continue. So, here’s the delegate...

Visit my Blog for more details.

Tips and tricks to rescue overdue projects

omar — Mon, 20 Oct 2008 18:08:34 GMT

One of my friends, who runs his own offshore development shop, was having nightmare situation with one of his customers. He's way overdue on a release, the customer is screaming everyday, he's paying his team from his own pocket, customer is sending an ever increasing list of changes and so on. Here's how we discussed some ideas to get out of such a situation and make sure it does not repeat in future: Kabir: Hey, can you help me? My customer is making us work for free for extra two months to fix bugs from our last delivery. We did what he said. But after he saw the output, he came up with hundred changes, which he somehow presents as bugs or missing features and make them look like they are all our fault and making us work for last two months for free. He is sending new changes every week. We have no idea when we will complete the iteration. Omar: I see. Did you get a signed list of requirements from customer before you started the development? Kabir: Of course, I did. He sent us a word document explaining what he wants and we sent him a task breakup with hour estimates and total duration of three months. Now after three months when we showed him the product, he said, it's no where close to what he had expected. Then he sent a gigantic list of things to change. Omar: All of those are bugs? Kabir: Of course not. Most of them are new features. Omar: Then why don't you say those are new features? You have the original word document to prove. Just ask him to show where in the word document did he said X needs to be done? Kabir: Well..., he's tricky. He somehow makes things look like it is obvious that X needs to be done and he's not going to accept a requirement as done until X is done. For example, he said there must be a complete login form in the homepage. So, we did a typical login form with user name, password and OK, Cancel button. Now he says where's the email verification thing? We said, you did not ask for it. He said, "this is obvious, every login form has a forgot password and email verification; I said *complete* login form, not half-baked login form". So, you see, we can't really argue to keep our image. Then, we did the login form exactly how he said. Now he says, where the client side validations of proper email address, username length, password confirmation? We said, you never asked for it! He says, "come on, every single website nowadays has AJAX enabled client side validation, do I have to tell you every single thing? Aren't you guys smart enough to figure this out? You are already doing this for the third time, can't you do it really well this time?" Omar: OK, stop. I see what's your problem. Some customer will always try to make you work more for less money. They will try to squeeze out every bit of development they can for their bucks. So, you have to be extra careful on how much you commit to them and make sure they cannot chip in more...

Visit my Blog for more details.

An Agile Developer's workflow when SCRUM is used

omar — Sat, 11 Oct 2008 07:21:00 GMT

If you are planning to start SCRUM at your company, you might need to train developers and QA to get into the mindset of an Agile developer. SCRUM is only successful when the developers and QA get into the habit of following the principles of SCRUM by heart. So, sometimes you need to offer training or do trial sprints to give some room to your developers how to get used to the working fashion of SCRUM. Giving them a handy workflow diagram that shows how they should work helps soothe the steep learning curve required for non-super star developers. I made such a workflow while I was teaching SCRUM at my friend's company. The following diagram was printed and hung over the desk of each and every developer to help them grasp the culture of SCRUM quickly: We use Flyspray for issue tracking, so you will see the mention of it frequently. You will see the step to "Update Sprint backlog with remaining hours" is missing. This is done kinda verbally and scrum master (sometimes same person who is the product owner) keeps track of it. Hope you find this useful.

Visit my Blog for more details.

ASP.NET website Continuous Integration+Deployment using CruiseControl.NET, Subversion, MSBuild and Robocopy

omar — Mon, 06 Oct 2008 06:59:00 GMT

You can setup continuous integration and automated deployment for your web application using CruiseControl.NET, Subversion, MSBuild and Robocopy. I will show you how you can automatically build the entire solution, email build report to developers and QA, deploy latest code in IIS all using CruiseControl.NET every N minutes. First get the following: CruiseControl.NET Subversion (install the command line tools and add the Subversion bin path to PATH environment variable) Robocopy (Windows Vista/2008 has it built-in, here's the link for Windows 2003) Install .NET Framework. You need it for MSBuild. You will learn how I have configured Continuous Integration and Deployment for my open source AJAX Portal project www.Dropthings.com. The code is hosted at CodePlex. When some developer makes a commit, CruiseControl downloads the latest code, builds the entire solution, emails build report and then deploys the latest web site to IIS 6.0. After installing CruiseControl.NET, go to Programs -> Cruise Control -> CruiseControl.NET Config. Now keep copying and pasting the following XML blocks and make sure you understand each block and make necessary changes: 1: <cruisecontrol> 2: <project name="Dropthings" queue="DropthingsQueue" queuePriority="1"> 3:  7: <workingDirectory>d:\cc\dropthings\code\trunk\</workingDirectory> 8:  9: <artifactDirectory>d:\cc\dropthings\artifact\</artifactDirectory> 10: <category>Dropthings</category> 11:  12: <webURL>http://localhost/ccnet/</webURL> 13: <modificationDelaySeconds>60</modificationDelaySeconds> 14: <labeller type="defaultlabeller"> 15: <prefix>0.1.</prefix> 16: <incrementOnFailure>true</incrementOnFailure> 17: <labelFormat>000</labelFormat> 18: </labeller> 19: <state type="state" directory="State" /> First change the working directory. It needs to be the path of the folder where you will have the solution downloaded. I generally create folder structure like this: D:\CC - Root for all CC.NET enabled projects \ProjectName - Root project folder \Code - Code folder where code is downloaded from subversion \Artifact - CC.NET generates a lot of stuff. All goes here. Next comes the Subversion integration block: 1: <sourcecontrol type="svn"> 2: ...

Visit my Blog for more details.

Using multiple broadband connections without using any special router or software

omar — Sun, 05 Oct 2008 10:29:18 GMT

I have two broadband connections. One cheap connection, which I mostly use for browsing and downloading. Another very expensive connection that I use for voice chat, remote desktop connection etc. Now, using these two connections at the same time required two computers before. But I figured out a way to use both connections at the same time using the same computer. Here's how: Connect the cheap internet connection that is used mostly for non-critical purpose like downloading, browsing to a wireless router. Connect the expensive connection that is used for network latency sensitive work like Voice Conference, Remote Desktop directly via LAN. When you want to establish a critical connection like starting voice conf app (Skype) or remote desktop client, momentarily disconnect the wireless. This will make your LAN connection the only available internet. So, all the new connections will be established over the LAN. Now you can start Skype and initiate a voice conference or use Remote Desktop client and connect to a computer. The connection will be established over LAN. Now turn on wireless. Wireless will now become the first preference for Windows to go to internet. So, now you can start Outlook, browser etc and they will be using the wireless internet connection. During this time, Skype and Terminal Client is still connected over the LAN connection. As they use persisted connection, they keep using the LAN connection and do not switch to the wireless. This way you get to use two broadband connections simultaneously. Here you see I have data transfer going on through two different connection. The bottom one is the LAN which is maintaining a continuous voice data stream. The upper one is the wireless connection that sometimes consumes bandwidth when I browse. Using Sysinternal's TCPView, I can see some connection is going through LAN and some through Belkin router. The selected ones - the terminal client and the MSN Messenger is using LAN where the Internet Explorer and Outlook is working over Wireless connection.

Visit my Blog for more details.

Best practices for creating websites in IIS 6.0

omar — Sat, 04 Oct 2008 10:05:22 GMT

Every time I create an IIS website, I do some steps, which I consider as best practice for creating any IIS website for better performance, maintainability, and scalability. Here' re the things I do: Create a separate application pool for each web application I always create separate app pool for each web app because I can select different schedule for app pool recycle. Some heavy traffic websites have long recycle schedule where low traffic websites have short recycle schedule to save memory. Moreover, I can choose different number of processes served by the app pool. Applications that are made for web garden mode can benefit from multiple process where applications that use in-process session, in memory cache needs to have single process serving the app pool. Hosting all my application under the DefaultAppPool does not give me the flexibility to control these per site. The more app pool you create, the more ASP.NET threads you make available to your application. Each w3wp.exe has it's own thread pool. So, if some application is congesting particular w3wp.exe process, other applications can run happily on their separate w3wp.exe instance, running under separate app pool. Each app pool hosts its own w3wp.exe instance. So, my rule of thumb: Always create new app pool for new web applications and name the app pool based on the site's domain name or some internal name that makes sense. For example, if you are creating a new website alzabir.com, name the app pool alzabir.com to easily identify it. Another best practice: Disable the DefaultAppPool so that you don't mistakenly keep adding sites to DefaultAppPool. First you create a new application pool. Then you create a new Website or Virtual Directory, go to Properties -> Home Directory tab -> Select the new app pool. Customize Website properties for performance, scalability and maintainability First you map the right host headers to your website. In order to do this, go to WebSite tab and click on "Advanced" button. Add mapping for both domain.com and www.domain.com. Most of the time, people forget to map the domain.com. Thus many visitors skip typing the www prefix and get no page served. Next turn on some log entries: These are very handy for analysis. If you want to measure your bandwidth consumption for specific sites, you need the Bytes Sent. If you want to measure the execution time of different pages and find out the slow running pages, you need Time Taken. If you want to measure unique and returning visitors, you need the Cookie. If you need to know who is sending you most traffic - search engines or some websites, you need the Referer. Once these entries are turned on, you can use variety of Log Analysis tools to do the analysis. For example, open source AWStats. But if you are using Google Analytics or something else, you should have these turned off, especially the Cookie and Referer because they take quite some space on the log. If...

Visit my Blog for more details.

Create ASP.NET MVC Controllers under Namespace and specific URL

omar — Sat, 04 Oct 2008 08:06:21 GMT

When you have a lot of controllers, you need to organize them under Namespaces. Also, it is better to put controllers under subfolders in order to organize them properly and have some meaningful URL for them like /Admin/User where Admin is the subfolder and User is the controller. For example, you might have a lot of controllers that are APIs exposed by your web app, not regular pages. So, you might want to put them under /API/ folder. You also want to make sure no one can access those controllers from the root url. For example, no one must call /User/GetUserList instead they must call /API/User/GetUserList ASP.NET MVC default routing and Controller Factory is very greedy, it ignores the subfolders inside the "Controllers" folder. There's a DefaultControllerFactory class in ASP.NET MVC which traverses all controllers under the "Controller" folder and creates a cache using just the class name as the key. So, it ignores any namespace or any subfolder where you have put the controller. So, I created a derivative of the default controller factory and created a new factory that checks if the requested controller belongs to any specific namespace and whether that controller must be inside a specific subfolder. You can map /Admin folder to respond to all controllers that are under then YourWebApp.Admin namespace. Similarly, you can map /API folder to respond to all controllers that are under the YourWebApp.API namespace. None of these controllers can be accessed outside the specified URL. Here's the factory that does it: This controller checks the namespace of the controller being returned by ASP.NET MVC's default implementation and ensures the requested URL is the right url where controllers from the namespace can be accessed. So, this means if the controller matches MvcWebAPI.API.UserController, it ensures the URL being requested must be /MvcWebAPI/API/User. It will return null if the URL was something else like /MvcWebAPI/User or /MvcWebAPI/SomeotherFolder/User. Here's how you use it form Global.asax: You create a mapping for a Namespace and the subfolder it must belong to. Then you register the new Controller Factory as the default controller factory. Now the second catch is, the default Route for the {controller}/{action}/{id} won't work for you. You need to create a specific router that starts with the subfolder name. For example, API/{controller}/{action} Here' two things to notice: The use of PathStartWith routing constraint, which I will explain soon The last parameter which tells the route to include the API namespace for this route. Otherwise it can't find the controllers in the API namespace So, the PathStartsWith routing constraint ensures this route gets hit only when the requested URL is under the /API/ folder. For any other URL, it returns false and thus the routing handler skips this route. It just does a comparison on the AbsolutePath of the current request URL to...

Visit my Blog for more details.

Create REST API using ASP.NET MVC that speaks both Json and plain Xml

omar — Fri, 03 Oct 2008 16:05:51 GMT

ASP.NET MVC Controllers can directly return objects and collections, without rendering a view, which makes it quite appealing for creating REST like API. The nice extensionless Url provided by MVC makes it handy to build REST services, which means you can create APIs with smart Url like "something.com/API/User/GetUserList" There are some challenges to solve in order to expose REST API: Based on who is calling your API, you need to be able to speak both Json and plain old Xml (POX). If the call comes from an AJAX front-end, you need to return objects serialized as Json. If it's coming from some other client, say a PHP website, you need to return plain Xml. Similarly you need to be able to understand REST, Json and plain Xml calls. Someone can hit you using REST url, someone can post a Json payload or someone can post Xml payload. I have created an ObjectResult class which takes an object and generates Xml or Json output automatically looking at the Content-Type header of HttpRequest. AJAX calls send Content-Type=application/json. So, it generates Json as response in that case, but when Content-Type is something else, it does simple Xml Serialzation. Here's the ObjectResult that you can use from Controllers to return objects and it takes care of proper serialization method. Above shows the Json serialization, which is quite simple. XmlSerialization is a bit complex though: Things to note here: You have to force UTF8 encoding. Otherwise it produces UTF16 output. XML Declaration is skipped because that's not quite necessary. Wastes bandwidth. If you need it, turn it on. I have turned on indenting for better readability. You can turn it off to save bandwidth. Some of you might be boiling inside looking at my obscure coding style. I love this style! I am spoiled by jQuery. I wish there was a cQuery. I actually started writing one, but it never saw day light just like my hundred other open source attempts. Now back to Object Serialization, we got the serialization done. Now you can return objects from Controller easily: You can use the test web project to call these methods and see the result: So far you have seen simple object and list serialization. A best practice is to return a common result object that has some status, message and then the real payload. It's handy when you only need to return some error but no object or list. I use a common Result object that has three properties - ErrorCode (0 by default means success), Message (a string data type) and Data which is the real object. When you want to return only a result with error message, you can do this: This produces a result like this: No payload here. So, the return format is always consistent. Those who are consuming service can write a common Xml or Json parsing code to consume both success and failure response. Those who are building API for their website, I humbly request you to return consistent response for both success and...

Visit my Blog for more details.

HTTP handler to combine multiple files, cache and deliver compressed output for faster page load

omar — Thu, 28 Aug 2008 18:43:45 GMT

It's a good practice to use many small Javascript and CSS files instead of one large Javascript/CSS file for better code maintainability, but bad in terms of website performance. Although you should write your Javascript code in small files and break large CSS files into small chunks but when browser requests those javascript and css files, it makes one Http request per file. Every Http Request results in a network roundtrip form your browser to the server and the delay in reaching the server and coming back to the browser is called latency. So, if you have four javascripts and three css files loaded by a page, you are wasting time in seven network roundtrips. Within USA, latency is average 70ms. So, you waste 7x70 = 490ms, about half a second of delay. Outside USA, average latency is around 200ms. So, that means 1400ms of waiting. Browser cannot show the page properly until Css and Javascripts are fully loaded. So, the more latency you have, the slower page loads. Here's a graph that shows how each request latency adds up and introduces significant delay in page loading: You can reduce the wait time by using a CDN. Read my previous blog post about using CDN. However, a better solution is to deliver multiple files over one request using an HttpHandler that combines several files and delivers as one output. So, instead of putting many <script> or <link> tag, you just put one <script> and one <link> tag, and point them to the HttpHandler. You tell the handler which files to combine and it delivers those files in one response. This saves browser from making many requests and eliminates the latency. Here you can see how much improvement you get if you can combine multiple javascripts and css into one. In a typical web page, you will see many javascripts referenced:<script type="text/javascript" src="/Content/JScript/jquery.js"> </script> <script type="text/javascript" src="/Content/JScript/jDate.js"> </script> <script type="text/javascript" src="/Content/JScript/jQuery.Core.js"> </script> <script type="text/javascript" src="/Content/JScript/jQuery.Delegate.js"> </script> <script type="text/javascript" src="/Content/JScript/jQuery.Validation.js"> </script> Instead of these individual <script> tags, you can use only one <script> tag to serve the whole set of scripts using an Http Handler:<script type="text/javascript" src="HttpCombiner.ashx?s=jQueryScripts&t=text/javascript&v=1" > </script> The Http Handler reads the file names defined in a configuration and combines all those files and delivers as one response. It delivers the response as gzip compressed to save bandwidth. Moreover, it generates proper cache header to cache the response in browser cache, so that, browser does not request it...

Visit my Blog for more details.

Loading static content in ASP.NET pages from different domain for faster parallel download

omar — Fri, 01 Aug 2008 18:15:08 GMT

Generally we put static content (images, css, js) of our website inside the same web project. Thus they get downloaded from the same domain like www.dropthings.com. There are three problems in this approach: They occupy connections on the same domain www.dropthings.com and thus other important calls like Web service call do not get a chance to happen earlier as browser can only make two simultaneous connections per domain. If you are using ASP.NET Forms Authentication, then you have that gigantic Forms Authentication cookie being sent with every single request on www.dropthings.com. This cookie gets sent for all images, CSS and JS files, which has no use for the cookie. Thus it wastes upload bandwidth and makes every request slower. Upload bandwidth is very limited for users compared to download bandwidth. Generally users with 1Mbps download speed has around 128kbps upload speed. So, adding another 100 bytes on the request for the unnecessary cookie results in delay in sending the request and thus increases your site load time and the site feels slow to respond. It creates enormous IIS Logs as it records the cookies for each static content request. Moreover, if you are using Google Analytics to track hits to your site, it issues four big cookies that gets sent for each and every image, css and js files resulting in slower requests and even larger IIS log entries. Let's see the first problem, browser's two connection limit. See what happens when content download using two HTTP requests in parallel: This figure shows only two files are downloaded in parallel. All the hits are going to the same domain e.g. www.dropthings.com. As you see, only two call can execute at the same time. Moreover, due to browser's way of handling script tags, once a script is being downloaded, browser does not download anything else until the script has downloaded and executed. Now, if we can download the images from different domain, which allows browser to open another two simultaneous connections, then the page loads a lot faster: You see, the total page downloads 40% faster. Here only the images are downloaded from a different domain e.g. "s.dropthings.com", thus the calls for the script, CSS and webservices still go to main domain e.g. www.dropthings.com The second problem for loading static content from same domain is the gigantic forms authentication cookie or any other cookie being registered on the main domain e.g. www subdomain. Here's how Pageflake's website's request looks like with the forms authentication cookie and Google Analytics cookies: You see a lot of data being sent on the request header which has no use for any static content. Thus it wastes bandwidth, makes request reach server slower and produces large IIS logs. You can solve this problem by loading static contents from different domain as we have done it at Pageflakes by loading static contents from a different domain e.g. flakepage.com. As the...

Visit my Blog for more details.