assumption of advancement

Are stale data evil?

2012-02-27T03:38:51-08:00

When you’re a software engineer who produces software for enterprises like banks or assurances, then it is normal you have huge databases (several gigabytes). Such systems have an operative application where users do the daily business of the company and there are more informative parts (or strategic parts) of the systems which the management uses. At a first glance, there isn’t a problem with those two views, but as you probably know, those companies have for the second part for the management a data ware house solution.

But what if your customer doesn’t want a data ware house solution? Or if he couldn’t afford one? Then you will probably add reports, search views to your application. In this blog post I describe some of the aspects if you’ve to choose this variation.

Stale data as a requirement

Unfortunately the question "how old can the data on this report/search be?" is rarely asked. When the answer is "The report/search has to show the right data", then you have to ask the customer again. The problem is, the data is maybe already stale after the query, because somebody changed some data.

In my experience there are only a few reports, which need as little as possible stale data. But it is essential that you ask this question.

Isolate only as far as needed

Most searches or reports need essential tables in your relational database, so it is important that those searches or reports don’t have an effect to your daily business. You ask yourself maybe now, how those queries could have any impact?

If you use Microsoft SQL Server, then the default isolation level is "Read committed". If a query isn’t that clever made, it could happen, that the query blocks a whole table (Intended Shared Lock which blocks any inserts or updates). If that happen, your users will remark that by waiting while they try to save their data.

When you create a search or an report you have to ask yourself always, which Isolation level you will use. When you use dirty reads (Isolation level "Read uncommitted"), then you’ll probably never generate any locks, but you have to deal with data which is wrong. This because data could be roll backed and the same query wouldn’t bring the roll backed data again.

Conclusion

Stale data or even wrong data on a search or an informational report hasn’t to be wrong or a mistake. Sometimes it’s just good enough to fulfil the requirements and make the customer happy. And that’s what it’s all about.

Anti-Pattern ‘Validation by Execute ‘n’ Rollback’

2012-02-21T12:53:12-08:00

Recently in some reviews I saw an anti-pattern. First you have to know, in the code, there was a validation of the data before it was stored in the database. So far so good. But when I looked at the validation code, I saw the following:

public void Validate()
{
	DoProcess(true);
}

And the persist logic (with some business logic) looked like this:

public void Process()
{
	DoProcess(false);
}

So, I asked myself, what the Boolean means. Well, here is the method signature:

public void DoProcess(bool bValidate)

Hmm, the code looks quite magic so far and the argument (and the method name) itself is more or less an anti-pattern already. But when I checked the DoProcess method, I saw something more or less like this:

public void DoProcess(bool bValidate)
{
	using (SqlConnection con = new SqlConnection(connectionString))
	{
		con.Open();
		SqlTransaction transaction = con.BeginTransaction();

		// Some business logic here...

		if(bValidate)
		{
			transaction.Rollback();
		}
		else
		{
			transaction.Commit();
		}
	}
}

The idea behind this code is, if it could be executed successfully, that means a successful validation. If there are any problems during the execution, those problems are the result of the validation. The execution itself wouldn’t be a problem after a successful validation (now I need a drink…).

Doing it better

It’s easy to find bad things, but it’s sometimes harder to do things right. But in this case, the better way is easy. First, you have to separate your business logic from the data access logic. This means following the separation of concern principle.

The next point is, that you’re able to validate your model. So put the logic where it should be. If you use a domain model, the logic to validate an entity has to be on that entity. Or if you have a table module approach, the logic to validate an entity has to be on the corresponding table module.

Side effects

There are also other problems with this anti-pattern. There are some side effects with the data.

As you know, transactions should be as short as possible to prevent lock problems with your data. But if you have combined the business logic and data access logic, your transactions are much longer than needed. To make it worse, this anti-pattern forces you to do it at least twice.

There are reasonable use cases for dirty reads: For example fast searches. In this case you accept dirty reads and you don’t want any locks on your data in the database. Normally, the chance to have a dirty read isn’t that high, but with this anti-pattern the chances to get a dirty read are significantly higher.

Who should make decisions about technologies?

2012-02-13T08:00:00-08:00

One of the biggest problems of software engineering companies in Switzerland is currently to get new software developers. To get new employees there are several points as for example salary, environment, career possibilities and technologies. The last point looks easy but in reality it isn’t that easy. Why are essential technology decisions (like languages, frameworks, application servers or big libraries) not only made by developers? Why does the management mostly make those decisions?

In this blog post I try to see the technology aspect for creating software from different perspectives. Those views based on experiences and talks I had during the last ten years.

Customer

As expected the technology used for the software is for the customer irrelevant. But for him counts the working functionality and some non-functional requirements as usability or performance.

Product owner or product manager

The product owner neither is interested in the used technology directly. He’s mainly interested to get the functionality in time and in budget. But he’s also interested to get a good quality of the product, else he’ll receive complaints from the users (of the customer).

IT department

The IT department who has to operate the system is somehow interested in technology. They usually don’t like new technology. One reason is for sure "never change a running system". Another reason is the needed knowhow for every technology. They have to concern the cost of maintaining that knowhow.

Software Architect

The software architect (and I am currently one) is special. When he’s interested in technology, he should take responsibility in the project. If not, you have an architect who lives in the "ivory tower". Mainly the architect should concern more about structure, integration of components or other systems, maintainability and quality.

Human resource management

In most of the companies this part get lost. Today it is hard (specially in Switzerland) to get new developers. So, new and fancy technologies make a job or company more attractive. This means, that the human resource management is interested to use up-to-date technologies.

Developer

Well, most developers care about technology. Specially younger developers more because they had the possibility to try those at the university or in little projects with their colleagues. For those developers technology is essential for the daily job and to be motivated. If they have not the possibility to use new technology they will leave the company after one or two years. And that isn’t cheap for the company.

Management

Today most companies use some self-made frameworks. One reason they spent money to create one is the fear of creative developers who want "to fulfil themself". This means for the management that those developers are out of control and create overcomplicated things with a lot of different technologies. I think this fear is unjustified today, but in the days when everybody could call himself a developer it was not only rightless.
But the management has the difficult task to concern all the roles I described before and make a reasonably decision.

Conclusion

It seams there is a fight between the developers and the management who should make decisions about technologies. It is important that the management has all the information it needs and can concern all the risks of an introduction of a new technology. But the management has to be open to all the facts and arguments and should not presume to make a decision based on old experiences, assumptions or prejudices.
If the decisions are too offensive, then you have at the end quite everything in the IT department (for example all java application server products). On the other hand, if the decisions are too defensive, then you will have a stagnation. This will be very expensive to solve because you have to spend a lot of many to get up-to-date again.

TF.exe or maybe the most useful tool for TFS

2012-01-31T14:51:17-08:00

When you’ve to use TFS, then there are moments when you’re missing some features in the UI tools in Visual Studio. In those situations the console tool tf.exe is very useful. In this short blog post I summarized the commands I used the most.

Update an old version of a branch

It could happen, that the production branch or an older feature branch isn’t up-to-date, because somebody forgot one or several merges from the main branch. To fix that, there could be used the following statement:

tf.exe merge /recursive /force /version:T $/Project/Main/Source $/Project/Production/Source

Delete a branch permanently

After a while the number of merged feature branches increase. Normally, you would delete the merged feature branches, so they aren’t any longer visible in Visual Studio. But in my case, I display also the deleted items and this means, that I see all the deleted feature branches. To remove the old merged feature branches, there could be used the following statement:

tf.exe destroy $/Project/Development/FeatureX

Education in companies

2012-01-31T09:00:00-08:00

I do a lot of trainings and courses in the company where I work currently. And I was also in a lot of courses too. But I think, that those courses or trainings don’t work really. The reason, why is that they are just so called "sheep dips".

Sheep dip training doesn’t work

The first time when I heard the expression "sheep dip" was in the excellent book "Pragmatic Thinking and Learning". It means a training where a lot of people receive a lot of information, mostly just by sitting, listening and watching a lot of PowerPoint slides.

A sheep dip is in real the following (from "Pragmatic Thinking & Learning"):

A sheep dip (for real) is a large tank in which you dunk the unsuspecting
sheep to clean them up and rid them of parasites (see picture). The sheep line up (as sheep do) you grab one and dunk in the tank for an intensive, alien, and largely toxic experience. It wears off, of course, so you have to dip them again.

The problem is the exact same with the education trainings in companies. A lot of employees spend time and the revenue is very, very little.

Doing it better

The question is "How can we do education better?". In my experience there are some better ways:

Pair programming
Little workshops (just the team)
Collaboration platforms
Good documentation

One of the best ways to learn is to get feedback fast. In a "sheep dip" training/course there is nearly no chance to have a such feedback. Therefor pair programming is very good, but hard to do. At least one of the two developers has to be experienced, so the knowledge transfer could happen.

Another way I like very much are little workshops. Normally I ask a developer team, if they are interested in some kind of workshops (reviews, designing, architecture, etc.) based on some reviews of their outcome. The feedback what I received so far is very positive.

I think collaboration platforms, like wikis or OneNote, are still underestimated in corporate environments. But I had a chance to participate in a team where they used OneNote. It was really great, because they added all little tricks and tips how to handle problems with the frameworks and tools in this collaborative platform.

The last way is a good documentation. But what is a good documentation? Well, for me it has to be up-do-date, clear, short, contains examples and easy to read. Another point is, that the documentation has to be searchable. It should be easy to find things fast and just in one place.

Services and customer orientation

2012-01-31T08:30:00-08:00

Usually I don’t blog here about personal topics. But as some friends know, I was some days ago in the united states – to be more precise, at the west coast. I visited the cities San Francisco, Las Vegas and Los Angeles. I was a great trip and I enjoyed the cities, the landscape (Grand Canyon) and a lot more.

To be honest, one of the most things which impressed me on the trip was the service and customer orientation of the staff in the hotels, restaurants or taxis. I think there is a gap between what Europe (specially Switzerland) and the USA regarding services and customer orientation. The old world – or Europe – could learn one or more things from the USA.

In the following chapters I try to describe some experiences and conclusion about my experiences during the trip.

Be professional

When I did the helicopter flight to the Grand Canyon, the whole service chain (chauffeur from the hotel, the pilot, the service staff) was just perfect and everything works. They know what they are doing and how the whole process works.

It’s a great experience for a customer when he feels, when a service provider has everything under control and his a master at what he’s doing. Such an experience builds also trust and the customer will return again.

Be personal

This point is where I think the biggest gap between Europe and the USA is. For example at Starbucks they ask you for your first name so they call you when they finished the coffee or bagels. In Switzerland for example you have to guess if the coffee which was finished is for you or not.

One of the most important things to do as a service provider is to make the customer happy. For that you have to build trust and make him feel safe. To build trust, the customer has to know you, so it isn’t unprofessional to speak not only about things which are work related.

Conclusion

It was a great experience to see another way of customer orientation. But there were also some problems, which also exists in Europe. As a service provider you need very good and competent staff. Unfortunately, in some hotels I met a concierge, which wasn’t able to help or understand my need really. And that is what destroy trust, which is unfortunately a really fragile thing.

DataSet and deleted rows

2012-01-31T08:00:00-08:00

Yes, I know, the DataSet isn’t the leading edge technology, but as the company where I work currently, there are several companies who use DataSet as data access technology.

Recently, I had to migrate the DataSet subclasses of the framework of my current employer to .NET 4.0 and add LINQ support. After this migration I made some unit tests and I found a surprising fact: If you using LINQ with a DataSet, then you have to handle deleted rows, which isn’t as you would expecting the behaviour.

First, here a little setup to run the tests:

TestDataSet ds = new TestDataSet();
TestDataSet.BECRow row = ds.BEC.NewBECRow();
row.BecId = Guid.NewGuid();
row.RunDat = DateTime.Now;
row.MachineName = Environment.MachineName;
row.CrtDat = DateTime.Now;
row.CrtUsr = "pw";
ds.BEC.AddBECRow(row);
ds.AcceptChanges();

The old way

In the ages before LINQ you get the preferred rows in the following way out of a typed or untyped DataSet:

string strFilter = string.Format("{0} = '{1}'", ds.BEC.CrtUsrColumn.ColumnName, "pw");
TestDataSet.BECRow[] rowsBEC = (TestDataSet.BECRow[]) ds.BEC.Select(strFilter);
Console.WriteLine("Row count: {0}", rowsBEC.Length);

As you expecting, the result here is one row. The ugly thing is the magic filter string and the casting for the variable rowsBEC. But if you’re using DataSets, you are used to such code.

The new way?

With LINQ, the code could look much nicer:

var query = from r in ds.BEC
			where r.CrtUsr == "pw"
			select r;
Console.WriteLine("Row count: {0}", query.Count());

That’s better and I thought that I had the job done. But I was wrong, seriously wrong. This code behaves differently because you will get also the deleted rows (when no where clause exists) or even worse get a DeletedRowInaccessible exception. This exception occurs, when you try to access a property on a deleted row, which will be the case here, if there are deleted rows in the DataTable.

Why? Well, if you look at the code of the Select method (for example with the tool Reflector) you will see, that there is an implicit filtering on the current rows (added, modified and unchanged).

The solutions

So, how could you fix this issue? There are at least two ways. First, the nearest one:

var query = from r in (TestDataSet.BECRow[]) ds.BEC.Select()
			where r.CrtUsr == "pw"
			select r;

This solution works, but you have to do the ugly cast. Well, because we have a framework and we subclassed every DataSet class, we could fix that by an override in the generic base class.

If you don’t have a framework, where you subclassed all the DataSet classes, then there is an alternative way:

var query = from r in ds.BEC
			 where r.RowState != DataRowState.Deleted && r.CrtUsr == "pw"
			 select r;

The order here is important, because you have to check first the RowState and after that you can add the real where clause. Also the "&&" is important, because it has to break if the row is deleted.

Conclusion

There are some pitfalls with DataSet and LINQ, because you wouldn’t expect that behaviour with deleted rows. It doesn’t make any sense at all, because I nearly never saw a reason why my business logic would like to deal with deleted rows. Also the behaviour is unnatural because you think of the DataSet as an in-memory-database, and on the database you have not to deal with deleted rows, unless you have to implement a trigger.

Speaking at the .NET Usergroup Bern

2011-12-04T11:27:22-08:00

During a Lunch event organized by the .NET Usergroup Bern at the 7 December 2011 in Bern I will speak about the following question:

Is Software design overrated?

Yes, I know, it’s provocative. And obviously wrong. Really? I’m not so sure. After 10 years as software engineer and some years as software architect I saw too often no design, just a transformation from requirements directly in code. And what is the problem with that?

Interested? So come to my talk, here is some more information. The twitter hashtag will be #dnugbesd.

Speaking at the .NET Usergroup Zentralschweiz

2011-11-18T08:17:35-08:00

I’m holding at the .NET Usergroup Zentralschweiz a short talk about "Know your warm-up", see my last blog post. I’ll explain what I developed exactly and how I train new employees with this warm-up. I will also demonstrate the sample application which a new employee develops from scratch.

Come to see me speaking the 24. November, 18:00 at the following address: bbv Software Service AG, Blumenrain 10, Luzern, 1. Stock.

Thanks @danielmarbach and @ursenzler for having me.

Update:
There is a wrap-up on the site of the .NET Usergroup Zentralschweiz with some pictures. The slides are now available.

Know your warm-up

2011-10-31T08:00:00-07:00

I was this month in Berlin at the NHL match between the LA Kings and the Buffalo Sables. Half an hour before the game you could watch both teams doing a warm up session (see photo). They practice the techniques, moves and collaboration which they will use during the game.

After the match I asked myself, how do we do a warm up? How were we introduced into the development environment of our current employer? After how many hours or days had we have to be productive? Fortunately I realized this gap already and I developed a training kit for a duration of a month for new employees for my current employer.

The warm-up

I recognized that the learning curve is quite steep for new employees and so it is a risk for the company to give them immediately productive tasks. Therefore I developed a training program for a duration of a month – the warm-up. During this training the new employees learn the following:

Infrastructure of the company
Frameworks and tools
Responsible persons for the framework and tools
Guidelines and good practices (not best practices)
Realize software with the frameworks and tools based on uniformed requirements

Before the main part of the training starts there a two days of school where I teach them in traditional manner the architecture, the framework and the workflow component.

The main part is the development of a little insurance application with the same tools, frameworks and libraries which the new employee has to use also in the production environment. For this little application I developed the requirements and also a guideline for the responsible person to teach the new employee. I also implemented the application as a reference solution.

The insurance application itself is designed that the first tasks are easier than the last ones. This way gives the new employee constantly success experiences and he will be motivated to go on and have fun with the whole development environment.

And yes, having fun is important to be motivated. Working doesn’t mean that it isn’t allowed to have fun.

Teaching and documentation aren’t enough

There are several slides for the two school days and there are two papers for the school-application as well. Even for the whole framework exist a lot of paper. But this stuff is more or less useless if the new employee hasn’t anybody who is there for him or her. As in agile practices, feedback is the most important thing. Only the contact person, who is responsible for the education, can give the exact feedback which the new employee need. This because the responsible person knows where the new employee currently stays and what the context is.

Conclusion

We had already several new employees which did the warm-up successfully. The feedback of the new employees was also very positive because they could learn the whole development environment in a protected manner and try out things without any risk.

Speaking at the .NET Usergroup Zentralschweiz