Ahmeds'

Things I would like to see in a new code base

Ahmed — Fri, 01 Jul 2011 02:09:00 -1200

I was asked by someone recently what I would want to see in a code base that I may've recently inherited. As devs we certainly come across new code bases all the time, some big others small. Here are a few things that I would really like to see:

General code readability & maintainability:

Use of consistent coding conventions
Proper code formatting e.g. no extra spacing or unwanted line breaks (Note: it's easy to reformat code using editors these days though).
Use of a source control system. Although I don't think anyone really develops without a source control these days (or do they?)

Setup and modularity:

Functionality broken into multiple projects/assemblies
Layered architecture to separate concerns
Easy to map namespaces to an assembly. From personal experience, I feel this can cause a lot of confusion sometimes.
Build script for the solution
Deployment scripts and/or packages

Design:

OO basics are applied (assuming an OO programming language is used) i.e. abstraction, encapsulation, inheritance, polymorphism
Less depth in inheritance hierarchies. While I understand this can be hard to get away with in large applications I am against inheritance abuse.
Design patterns
(Any) SOLID principles in use. I would really like to see Single responsibility, Open/closed and Dependency inversion principles.
No obvious code smell e.g. code duplication, God objects, spaghetti code etc
Caching, paging and other general performance considerations in code and database queries
Well known frameworks in use e.g. NHibernate, jQuery, log4net etc
Error handling (where it makes sense) and logging

Testing

Automated unit/integration tests
Mocking frameworks in use
Automated UI testing (if possible)

I'm sure there are other things that could make into this list.

Type Initializer Exception

ahmed — Thu, 09 Dec 2010 14:11:20 -1200

I've blogged about using static constuctors for Singletonsbefore. Static constructors are invoked only once before any (instance or static) member of the class can be executed. Based on this behavior, it provides an ideal thread-safe initialization section in many scenarios.

There is however one particular behavior of static constructors that is good to know. If an unhandled exception occurs in a static constructor then the class is effectively unusable. Any other client code trying to use any static/instance member of the *failed* class can expect a type initialization exception in return. Unlike instance constructors, the runtime will not try to re-execute the static constructor if it failed once since static constructors are only supposed to execute once.

For developers it's something important to keep in mind if you tend to use static constructors frequently. Either explicitly handle the exception or try not to put any code in there that can result in an error e.g. a call to a DB or web service. Stuff like this should be put in a separate initialization method that should be called once.

Bulk Loading with SqlBulkCopy

Ahmed — Sun, 24 Oct 2010 13:10:00 -1200

Bulk copying is the preferred method for loading high volume data. The SqlBulkCopy API in ADO.NET provides the ability to bulk copy using interfaces that .NET developers are more accustomed to e.g. IDataReader and DataTable. It’s similar to the SQL BCP utility however it can provide much more control over the loads. Bulk copying performance over individual inserts is achieved mainly due to a few factors:

Significant reduction in database round trips
Minimal logging at the database level (under certain conditions)
Reduced constraint checking by default

Contrary to popular belief, bulk copy operations can also participate in transactions. SqlBulkCopy has been working quite well for a high volume data loader I developed loading approximate 5 million rows daily on average. Here is an in depth research on SqlBulkCopy API behavior in different scenarios (thanks to the author for all the research): SqlBulkCopy Performance Analysis

Some notes from the article:

SQL Server scales quite well for concurrent loads up to a certain threshold.
Loading into a heap is must faster than an indexed table for obvious reasons. However if indexes are required then it maybe better (in some cases) to disable them during loading and enable them after.
The batch size can impact performance. The bigger the batch size the better.

Generating Sequences In SQL Server

ahmed — Tue, 29 Jun 2010 18:17:53 -1200

SQL Server provides auto identity to generate sequential IDs in a table which is handy. But there is no feature (that I know of in 2005 and before) to generate sequential IDs across multiple tables, similar to the Sequence Generator in Oracle. Recently I came across a requirement where I had to bulk insert parent and child records in 2 separate tables and I did not want to go the Sequential GUID route.

While from the surface it may look simple, the challenge with generating unique sequences is reliability in a multi-user environment. This is why I always prefer the database system providing such functionality. Here is something that I came up with that is simple and worked quite well for my needs. It has 2 main components – a table & a stored proc.

CREATE TABLE [dbo].[EntityIdSequence](
[SeqNo] [bigint] IDENTITY(1,1) NOT NULL,
[Used] [bit] NULL
) ON [PRIMARY]

CREATE PROCEDURE [dbo].[GenerateEntityIdSequences]
    @count int
AS
    SET NOCOUNT ON;
    declare @sequences as table (seqno bigint)

    while (@count > 0)
    begin
        insert into [entityidsequence] values (1);
        insert into @sequences values (scope_identity());    -- get the last generated id
        set @count = @count - 1;
    end

    select seqno from @sequences;

Notice that I didn’t have to specify any special transaction isolation level. And I am able to generate a bunch of unique sequences at a time which can cut down on database roundtrips. The real key here is making use of the scope_identity() system function. Another important thing to remember is that concurrent requests to GenerateEntityIdSequences for multiple sequences could result in skipped sequences e.g. 1,2,4,6,7. But they should always be unique.

We can also reseed if need be. This can be useful if you wanted to turn off auto ID on an existing table although that would require recreating & repopulating the table. Here is the command for reseeding to start generating sequence from 1000:

DBCC checkident(‘EntityIdSequence’, RESEED, 999)

Privacy in the Digital Age

Ahmed — Wed, 02 Jun 2010 17:59:00 -1200

We have entered an era where broadcasting yourself is the new norm. Social networking sites like Facebook & Twitter enable us to share our information with family and frieds which is great. But this also presents some privacy challenges that are a hot topic of discussion these days.

Some people might argue that privacy is not as big of a concern as it's made out to be. Just look at the recent FB decision to share user information with search engines. FB is one of the most popular sites on the planet with 500 million users and growing. Even a small percentage is a lot of private information.

It's true that people are social animals and want to share their information with closed ones. But many people also blindly trust these websites to not misuse their personal information. And that is an important question: can websites that have access to a lot of personal information be trusted? How much control do individuals have over their own information that can be instantly transmitted to 3rd parties around the world for reasons they are completely unaware of.

It seems like there isn't going to be much privacy for consumers in the digital age unless we demand it from organizations (similar to the FB user backlash). Websites should benefit from their users but not at the expense of their privacy.

What Management Should Know

ahmed — Sat, 15 May 2010 10:45:43 -1200

I recently read Agile Software Development with Scrum by Ken Schwaber and Mike Beedle. This book does a great job in delivering an important message i.e. software development is a research/creative process. It cannot be planned for using the same processes more applicable to tasks with predictable outcomes. A single piece of functionality can be implemented in different ways depending on programmer experience and other factors. This is why “sprinting” feels more natural when applied to software development. After each development cycle, that shouldn’t span more than a month, the entire team needs to reevaluate their progress and proceed based on the empirical data available.

While this means trying to get a working quality product out the door it also implies being flexible. This is a difficult proposition for management since it flies in the face of all the nifty looking project plans and customer expectations. This is why the entire company has to be on board with agile software development. It is not just a process limited to the nerds in the corner section cranking out code.

Powered by BlogEngine.NET

admin — Sat, 03 Apr 2010 20:38:59 -1200

I finally moved to a new blogging engine i.e. BlogEngine.NET. I’d been using Community Server(now known as Telligent Community) since 2006 mainly for blogging. I felt it was time to use something more lightweight and community driven. CS has also made some license changes since I started using it.

Anyway, what was important for me was being able to move my existing blog posts and comments which was simple enough by using this script. I did have to tweak the query a little bit to meet my requirements especially since I’d been using a very old version of CS e.g. the section ID for blog posts was 12 in my case. In addition, I had to write the query to import the existing tags. Here is the query:

-- Tags

insert into be_PostTag (PostID, Tag)
    select p.PostID, cat.Name from be_Posts p
    join CommunityServer.dbo.cs_Posts_InCategories pcat on p.cspostid = pcat.PostID
    join CommunityServer.dbo.cs_Post_Categories cat on pcat.CategoryID = cat.CategoryID
    where cat.SectionID = 12

So far I am liking BlogEngine.NET. It’s simple, configurable, fast and comes with a lot of themes and extensions contributed by the community. Looking forward to blogging with BlogEngine.NET.

Team Synergy

Admin — Mon, 08 Mar 2010 17:39:00 -1200

We all know how important team synergy is. Be it a sports team or a corporate one, great things can be achieved by people working together. When people click the impossible becomes possible. People can discuss and debate by setting aside egos. The distinction between work and fun is blurred. Being honest is not considered being too direct. Knowing a team member is more than knowing their name and what feature they are working on. A theme based list exists only known to the team members. Others envy the team. One person leaving the team can have a drastic impact.

In my career I’ve been blessed to have worked with a lot of good folks in awesome teams. It’s important for me (and I’m sure many others) that I look forward to going to work everyday. That includes challenging work and fun people. Having the right mix of people is like having a very delicate balance. You have to be part of one to feel it.

Using LogParser with .NET

Ahmed — Fri, 19 Feb 2010 15:16:00 -1200

In "The Pragmatic Programmer", the authors talk of the Unix philosophy of “small, sharp tools” and the limitations of GUI apps. I strongly agree with them. No doubt GUI apps have their place in the world, but there is something very simple yet powerful about having a console utility that promises to do only one thing and does it well.

One of my favorite console utilities of late has been LogParser. Talk about a fast sql-like query language with the ability to query multiple data sources and a COM API to boot. Since one of the applications I developed recently required IIS log parsing, I simply couldn't ask for more. But the fact that it can query large files efficiently also makes it quite useful for other stuff. Ever have to search a large CSV file? Why even bother opening it in a text editor when you can query and page through the results. How about querying IIS metabase or the Windows Event Log and storing results in a CSV file? You really have to play with it to realize its full potential.

As .NET developers we can also harness LogParser's power through its COM API. While COM Interop is probably not the most desirable option, its the only way to consume LogParser in .NET. There is obviously a performance penalty for this, however I've not experienced any significant performance issues while parsing tons of log files every day (assuming the queries themselves execute quickly). Using the LogParser API is pretty straight forward. Let’s say we are interested in getting the unique hosts from log files. The command line query for this would be:

logparser –i:IISW3C “select distinct to_lowercase(cs-host) from c:\logfiles\W3SVC1\ex100128.log”

Folowing is a C# application that runs the above query for any specified log file. Make sure to install the LogParser utility which will also register the COM DLL. Reference the LogParser.dll from the installation directory and VS will generate the RCWs (runtime callable wrappers) in an assembly named Interop.MSUtil.dll. Here is the code:

using LogQuery = MSUtil.LogQueryClassClass;
using LogRecordSet = MSUtil.ILogRecordset;
using LogRecord = MSUtil.ILogRecord;
using IISW3CInputFormat = MSUtil.COMIISW3CInputContextClassClass;

namespace LogParserTest
{

class Program
{

static void Main(string[] args)
{

if (args == null || args.Length == 0)
{
Console.WriteLine("Please specify a log file path.");
return;
}

try
{

string query = string.Format(@"SELECT DISTINCT TO_LOWERCASE(cs-host) FROM {0}", args[0]);

// Initialize a LogQuery object and use IISW3C input format
LogQuery logQuery = new LogQuery();
LogRecordSet recordSet = logQuery.Execute(query, new IISW3CInputFormat());

// Traverse the result set
for (; !recordSet.atEnd(); recordSet.moveNext())
{

// Get the current record
LogRecord logRecord = recordSet.getRecord();

if (!logRecord.isNull(0))
Console.WriteLine(logRecord.getValue(0).ToString());
}
}
catch (Exception ex)
{
Console.WriteLine("ERROR: {0}", ex.Message);
}

}
}

}

The above example is very simple for demo purpose. For a real world app I would suggest moving the common functionality of querying log files into a base class and delegating the result set processing to derived classes since each query is unique. The derived classes should also expose typed record sets that would be easier to work with.

SQL Short-Circuiting

Ahmed — Fri, 12 Feb 2010 03:12:55 -1200

Recently I worked on a performance issue of a search query in SQL Server involving quite a few parameters. After making some changes, one thing that I wasn’t too sure of was if the OR operator in SQL short-circuits similar to the && or || operator in C# . After some googling it seems that there is no guarantee how a (SQL Server specific) query will be evaluated as discussed in this blog. And this obviously can have a significant impact on queries against large tables. Some additional thoughts on the issue can be found in the comments of this forum post as well. I ended up using dynamic SQL, which wasn’t the most desirable approach but the simplest way to get around this issue.