Your answers are wrong. Or they will be soon.

As we learn, grow and the world around us changes, it is inevitable that our views and beliefs change as well. One area where software engineers are bound to change opinions is around languages and frameworks.

With time and opportunity to build projects with technologies that were once trending and exciting, we notice that even the latest technology has pros and cons, and is not a perfect utopia like we thought it would be when we were just starting out.

I have had my share of wrong answers. I wrote about one of them in the post about JavaScript back in 2014. To understand why I wrote it in the first place, I wanted to take the trip down the memory lane, look at the evolution of JavaScript, and how it has affected my opinion over the years.

Back in the early days of the web, most web pages were generated on the back-end by web servers, and resulting HTML was sent to a client to be rendered by a browser. Navigation between pages or content required every page to be loaded by the browser. In the mid-2000s, GMail and Google Maps were launched using a new technology called AJAX, shorthand for Asynchronous JavaScript + XML, which allowed for pages to be dynamically updated from the web server without making a full page load. Being obsessed with computers running fast, rendering content without a full page refresh seemed to me so much quicker and almost more magical than the traditional way. I quickly became a fan of this new approach and jumped on any opportunity to implement such behavior.

Around the year 2006, the jQuery library came along and made it very easy to dynamically update the HTML Document Object Model. Besides providing an easy to use fluent API, jQuery also abstracted differences of JavaScript (ECMAScript) implementation of all major browsers. Finally, it made it easy to initiate AJAX requests, parse the data, and then update a portion of a web page. As a side note, very soon after, Microsoft also came out with its AJAX implementation for the ASP.NET Web Forms: ASP.NET AJAX 1.0. It provided JSON serialization features for .NET web-services and allowed developers to use 3rd party JavaScript libraries like jQuery directly to request data formatted as JSON from the back-end. At the beginning of my career as a .NET developer, jQuery with ASP.NET Web Services became my favorite tool when building dynamic web applications.

In the early 2010s, many libraries and frameworks started popping up that allowed engineers to render entire sites with JavaScript and make them look more like native or desktop applications rather than web sites. They were called single-page applications or SPAs. A few of the first ones that come to mind are Backbone, Angular, Ember, and Knockout.js.

My first experience working on a couple of medium-sized SPAs using the Knockout.js library taught me that I could no longer get away with just a surface understanding of JavaScript, which was enough earlier when I putt together sites with jQuery. In addition to reading JavaScript: The Good Parts by Douglas Crockford, I also purchased and read JavsScript Patterns by Stoyan Stefanov. Those two books helped me to become much more comfortable with the language and make my now non-trivial JavaScript projects more maintainable.

As I was becoming more comfortable with writing non-trivial single-page JavaScript applications, Node.js was starting to gain popularity. It opened up a new frontier, and JavaScript developers could now not only create applications in a browser but also on the server. Node.js brought the simplicity of JavaScript to the backend. It was easy to get started, just import some packages and run one command to execute your script. Because I enjoyed the novelty and innovativeness of SPAs, I was naturally gravitating at playing around with Node.js and wanting to use it in my future projects.

Besides being simple to get started with, Node.js had multiple other selling points going for it. First of all, because typically all I/O in a JavaScript engine is performed via events and callbacks, asynchronous in nature and never blocks the executing thread, Node.js is very efficient at processing many requests and is very well suited for web applications. It could handle a much bigger throughput than synchronous web applications. Another selling point was using a single language to write your front end, SPA application and backend to power it. Lastly, because Node.js library came only with a basic core library that helped to write server applications, it relied on the community to write the missing pieces. Very soon, every little bit of non-trial code was published as a node package manager (NPM) package and could be imported into your project with a one line command. Almost like Apple’s trademark “There’s an App for That”, npm had a package for anything you could think of. A large number of available packages allows JavaScript developers to build apps really fast. But that speed did not come for free. Not having a standard library encouraged package developers to rely on other packages, which in turn also relied on other packages. A number of dependencies started to become a major source of frustration. The following quote is from an article that describes this problem really well:

In order to use these 3 methods node_modules needs 1826 files. And that’s just 4 of mentioned 976 installed packages. - What’s really wrong with node_modules and why this is your fault

With a larger amount of dependencies, there is also a higher likelihood that a dependency may need to be updated and may not be compatible with existing code, therefore a large dependency graph is also more costly to maintain in the long term. Around the year 2015, it took me more than a week to put together a clean boilerplate that needed everything for a modern SPA project (React, Redux, Webpack, History, Router, Hot Reloading, etc,.) and only a year to learn that some of the dependencies were obsolete, and I if I wanted to migrate to the next version of Webpack everything had to be redone. It felt like a wasted effort. Luckily thanks to a very helpful project, Create React App, I don’t have to do this again. It takes care of setting up and configuring all the tools necessary for a React SPA and makes it easier to maintain project dependencies.

Using Node.js on large backend projects may be quicker to build, as PayPal reported back in 2013, but due to the dynamic nature of the language, and not having a type system, I could imagine it will be more difficult to maintain it over the long term. Yes, you can write a lot of unit tests to make sure your refactoring did not break dependencies or use TypeScript to add typing. However, the former comes with a good deal of overhead, and it might be just easier to use a statically typed language in the first place rather than going with the latter.

My Soon To Be Wrong Opinions Today

SPAs are a lot more expensive to build and maintain, and they come with a good deal of additional complexity. Given a lot of overhead, I think SPAs should not be a default choice for building web applications, even today. Yes, there is a place for them but you need to make a conscious decision and accept the overhead and tradeoff. Over time I’ve changed my approach and found it’s much more interesting to solve business problems rather than spending a lot of mental energy just so you can achieve an arbitrary faster and smoother user interface.

If you are not building a native-like UI but still need some portion of the page to be dynamically updated, the same technique we used a decade ago, making AJAX calls to the back-end and updating a page with JavaScript, is still effective, simple, and comes with very little overhead.

There are many valid use cases where JavaScript on the back-end is a good choice. But unless your application is taking advantage of the async I/O and event loop, it may be easier to maintain it in the long term with a language that comes with a proper standard library and is statically typed.

I love this quote from Kellan Elliot-McCrea:

Your answers are wrong. Or they will be soon.

As we learn, grow and the world around us changes, it is inevitable that our views and beliefs change as well. One area where software engineers are bound to change opinions is around languages and frameworks.

With time and opportunity to build projects with technologies that were once trending and exciting, we notice that even the latest technology has pros and cons, and is not a perfect utopia like we thought it would be when we were just starting out.

I have had my share of wrong answers. I wrote about one of them in the post about JavaScript back in 2014. To understand why I wrote it in the first place, I wanted to take the trip down the memory lane, look at the evolution of JavaScript, and how it has affected my opinion over the years.

Back in the early days of the web, most web pages were generated on the back-end by web servers, and resulting HTML was sent to a client to be rendered by a browser. Navigation between pages or content required every page to be loaded by the browser. In the mid-2000s, GMail and Google Maps were launched using a new technology called AJAX, shorthand for Asynchronous JavaScript + XML, which allowed for pages to be dynamically updated from the web server without making a full page load. Being obsessed with computers running fast, rendering content without a full page refresh seemed to me so much quicker and almost more magical than the traditional way. I quickly became a fan of this new approach and jumped on any opportunity to implement such behavior.

Around the year 2006, the jQuery library came along and made it very easy to dynamically update the HTML Document Object Model. Besides providing an easy to use fluent API, jQuery also abstracted differences of JavaScript (ECMAScript) implementation of all major browsers. Finally, it made it easy to initiate AJAX requests, parse the data, and then update a portion of a web page. As a side note, very soon after, Microsoft also came out with its AJAX implementation for the ASP.NET Web Forms: ASP.NET AJAX 1.0. It provided JSON serialization features for .NET web-services and allowed developers to use 3rd party JavaScript libraries like jQuery directly to request data formatted as JSON from the back-end. At the beginning of my career as a .NET developer, jQuery with ASP.NET Web Services became my favorite tool when building dynamic web applications.

In the early 2010s, many libraries and frameworks started popping up that allowed engineers to render entire sites with JavaScript and make them look more like native or desktop applications rather than web sites. They were called single-page applications or SPAs. A few of the first ones that come to mind are Backbone, Angular, Ember, and Knockout.js.

My first experience working on a couple of medium-sized SPAs using the Knockout.js library taught me that I could no longer get away with just a surface understanding of JavaScript, which was enough earlier when I putt together sites with jQuery. In addition to reading JavaScript: The Good Parts by Douglas Crockford, I also purchased and read JavsScript Patterns by Stoyan Stefanov. Those two books helped me to become much more comfortable with the language and make my now non-trivial JavaScript projects more maintainable.

As I was becoming more comfortable with writing non-trivial single-page JavaScript applications, Node.js was starting to gain popularity. It opened up a new frontier, and JavaScript developers could now not only create applications in a browser but also on the server. Node.js brought the simplicity of JavaScript to the backend. It was easy to get started, just import some packages and run one command to execute your script. Because I enjoyed the novelty and innovativeness of SPAs, I was naturally gravitating at playing around with Node.js and wanting to use it in my future projects.

Besides being simple to get started with, Node.js had multiple other selling points going for it. First of all, because typically all I/O in a JavaScript engine is performed via events and callbacks, asynchronous in nature and never blocks the executing thread, Node.js is very efficient at processing many requests and is very well suited for web applications. It could handle a much bigger throughput than synchronous web applications. Another selling point was using a single language to write your front end, SPA application and backend to power it. Lastly, because Node.js library came only with a basic core library that helped to write server applications, it relied on the community to write the missing pieces. Very soon, every little bit of non-trial code was published as a node package manager (NPM) package and could be imported into your project with a one line command. Almost like Apple’s trademark “There’s an App for That”, npm had a package for anything you could think of. A large number of available packages allows JavaScript developers to build apps really fast. But that speed did not come for free. Not having a standard library encouraged package developers to rely on other packages, which in turn also relied on other packages. A number of dependencies started to become a major source of frustration. The following quote is from an article that describes this problem really well:

In order to use these 3 methods node_modules needs 1826 files. And that’s just 4 of mentioned 976 installed packages. - What’s really wrong with node_modules and why this is your fault

With a larger amount of dependencies, there is also a higher likelihood that a dependency may need to be updated and may not be compatible with existing code, therefore a large dependency graph is also more costly to maintain in the long term. Around the year 2015, it took me more than a week to put together a clean boilerplate that needed everything for a modern SPA project (React, Redux, Webpack, History, Router, Hot Reloading, etc,.) and only a year to learn that some of the dependencies were obsolete, and I if I wanted to migrate to the next version of Webpack everything had to be redone. It felt like a wasted effort. Luckily thanks to a very helpful project, Create React App, I don’t have to do this again. It takes care of setting up and configuring all the tools necessary for a React SPA and makes it easier to maintain project dependencies.

Using Node.js on large backend projects may be quicker to build, as PayPal reported back in 2013, but due to the dynamic nature of the language, and not having a type system, I could imagine it will be more difficult to maintain it over the long term. Yes, you can write a lot of unit tests to make sure your refactoring did not break dependencies or use TypeScript to add typing. However, the former comes with a good deal of overhead, and it might be just easier to use a statically typed language in the first place rather than going with the latter.

My Soon To Be Wrong Opinions Today

SPAs are a lot more expensive to build and maintain, and they come with a good deal of additional complexity. Given a lot of overhead, I think SPAs should not be a default choice for building web applications, even today. Yes, there is a place for them but you need to make a conscious decision and accept the overhead and tradeoff. Over time I’ve changed my approach and found it’s much more interesting to solve business problems rather than spending a lot of mental energy just so you can achieve an arbitrary faster and smoother user interface.

If you are not building a native-like UI but still need some portion of the page to be dynamically updated, the same technique we used a decade ago, making AJAX calls to the back-end and updating a page with JavaScript, is still effective, simple, and comes with very little overhead.

There are many valid use cases where JavaScript on the back-end is a good choice. But unless your application is taking advantage of the async I/O and event loop, it may be easier to maintain it in the long term with a language that comes with a proper standard library and is statically typed.

Posted in: blog

Tagged with: thoughts

Because it’s a modern factory, the performance of every stage of the widget assembly is measured and then displayed on electronic dashboards. Based on the metrics, you can tell that the assembly line is at half capacity at the current rate of production. Therefore, to get to the target capacity without any optimizations is not possible, and you are not confident that it can even be accomplished.

The simple approach to increase the capacity is to add additional resources to each assembly stage based solely on the current production capacity and utilization. In other words, if one of the assembly stages is running at 50% capacity and produces 100 widgets, you figure out that you will need at least 50% * 5 = 250% more resources to produce 500 widgets. If one of those assembly stages is a physical resource like a paint station, getting to 250% would mean that you would need to buy and add two additional stations to support the 250% capacity increase.

This approach, however, has some flaws. It makes an assumption that your assembly line is already optimized to support the projected capacity. But in reality, there are many areas that could be improved to produce a higher throughput without additional resources. Increasing every resource across the board is inefficient, requires a lot more maintenance, and is expensive. It’s also wasteful if the expected capacity does not materialize.

Another alternative to increase capacity would be to try to dial up your current production assembly line to see how much it can handle. But this is dangerous because if one of the stages does go out of order, it can create downtime for the entire assembly, and your factory must be producing at all times.

Looking at the dashboards, you see that there are slow areas that can be optimized. But you are still not sure if the optimizations are going to make the entire assembly run quicker. In other words, are you actually discovering the bottlenecks or just optimizing blindly?

Luckily, your company has a new branch opening up soon that has the same assembly line installed, a perfect place for you to run some experiments.

So you set up your factory to mimic the production workload and systematically work to figure out what needs to be done to improve the throughput. By gradually increasing workload, you start noticing which stages are starting to break down and create bottlenecks. To resolve a bottleneck, you get to exercise your engineering skills and come up with a solution to make the steps faster. Sometimes, figuring out how to make a portion of an assembly line run faster is not an easy task, and you need to dive deep into understanding how a particular machine works and obtain valuable knowledge in the process. It’s fun and challenging at the same time.

In the process of pushing your assembly line to the limit, you get to learn about the physical limitations of each stage. Perhaps there are some stages that cannot be more optimized and you may have to redesign or upgrade a section of the assembly line. In order to squeeze every last bit of performance, you had to experiment with many different settings, test many different scenarios, and try out different configurations. As a result, you are now more intimately familiar with the system as a whole; there is no more guesswork needed and you are much better positioned to do the following:

• Create a realistic capacity plan.
• Identify major performance bottlenecks in your system.
• Improve the ability to increase or decrease capacity.
• Improve reliability during increased load.
• Improve the performance of the assembly line.

There could also be some other interesting findings. Perhaps, in the past, you thought that some areas of the assembly would start breaking down earlier than expected and had more resources allocated, but these theories were not realized. As a result, you can save the company money by getting rid of the excess capacity that would never get utilized under the current setup.

Finally, the most exciting outcome of performing these experiments is making your assembly line run faster and much more efficiently. Your customers are much happier that they can get widgets faster, and your business is more cost-effective as a result.

This was my attempt to use an analogy describing a load testing project that I was recently involved in. I understand that sometimes you may not be in a position to re-create a production-like environment, but if you are lucky enough to have an application that runs in the cloud and infrastructure specified as code, standing up a production mirror and investing a bit of time on load testing can have some profound results. As a result of load testing, at my work, we were able to increase the performance and capacity of our system by 10x using the same hardware. This will translate into a more reliable system that can handle bursts of traffic and be less expensive in general.

Special mention goes to the awesome load testing tool, Locust.io. It was very easy to use, and it took only a couple of days to write test scripts to mimic critical path interaction with our services.

Imagine you are an engineer working at a widget factory, and because the business is continuously growing, you estimate that you will soon need to support five times the current production capacity.

Because it’s a modern factory, the performance of every stage of the widget assembly is measured and then displayed on electronic dashboards. Based on the metrics, you can tell that the assembly line is at half capacity at the current rate of production. Therefore, to get to the target capacity without any optimizations is not possible, and you are not confident that it can even be accomplished.

The simple approach to increase the capacity is to add additional resources to each assembly stage based solely on the current production capacity and utilization. In other words, if one of the assembly stages is running at 50% capacity and produces 100 widgets, you figure out that you will need at least 50% * 5 = 250% more resources to produce 500 widgets. If one of those assembly stages is a physical resource like a paint station, getting to 250% would mean that you would need to buy and add two additional stations to support the 250% capacity increase.

This approach, however, has some flaws. It makes an assumption that your assembly line is already optimized to support the projected capacity. But in reality, there are many areas that could be improved to produce a higher throughput without additional resources. Increasing every resource across the board is inefficient, requires a lot more maintenance, and is expensive. It’s also wasteful if the expected capacity does not materialize.

Another alternative to increase capacity would be to try to dial up your current production assembly line to see how much it can handle. But this is dangerous because if one of the stages does go out of order, it can create downtime for the entire assembly, and your factory must be producing at all times.

Looking at the dashboards, you see that there are slow areas that can be optimized. But you are still not sure if the optimizations are going to make the entire assembly run quicker. In other words, are you actually discovering the bottlenecks or just optimizing blindly?

Luckily, your company has a new branch opening up soon that has the same assembly line installed, a perfect place for you to run some experiments.

So you set up your factory to mimic the production workload and systematically work to figure out what needs to be done to improve the throughput. By gradually increasing workload, you start noticing which stages are starting to break down and create bottlenecks. To resolve a bottleneck, you get to exercise your engineering skills and come up with a solution to make the steps faster. Sometimes, figuring out how to make a portion of an assembly line run faster is not an easy task, and you need to dive deep into understanding how a particular machine works and obtain valuable knowledge in the process. It’s fun and challenging at the same time.

In the process of pushing your assembly line to the limit, you get to learn about the physical limitations of each stage. Perhaps there are some stages that cannot be more optimized and you may have to redesign or upgrade a section of the assembly line. In order to squeeze every last bit of performance, you had to experiment with many different settings, test many different scenarios, and try out different configurations. As a result, you are now more intimately familiar with the system as a whole; there is no more guesswork needed and you are much better positioned to do the following:

• Create a realistic capacity plan.
• Identify major performance bottlenecks in your system.
• Improve the ability to increase or decrease capacity.
• Improve reliability during increased load.
• Improve the performance of the assembly line.

There could also be some other interesting findings. Perhaps, in the past, you thought that some areas of the assembly would start breaking down earlier than expected and had more resources allocated, but these theories were not realized. As a result, you can save the company money by getting rid of the excess capacity that would never get utilized under the current setup.

Finally, the most exciting outcome of performing these experiments is making your assembly line run faster and much more efficiently. Your customers are much happier that they can get widgets faster, and your business is more cost-effective as a result.

This was my attempt to use an analogy describing a load testing project that I was recently involved in. I understand that sometimes you may not be in a position to re-create a production-like environment, but if you are lucky enough to have an application that runs in the cloud and infrastructure specified as code, standing up a production mirror and investing a bit of time on load testing can have some profound results. As a result of load testing, at my work, we were able to increase the performance and capacity of our system by 10x using the same hardware. This will translate into a more reliable system that can handle bursts of traffic and be less expensive in general.

Special mention goes to the awesome load testing tool, Locust.io. It was very easy to use, and it took only a couple of days to write test scripts to mimic critical path interaction with our services.

Posted in: blog

Tagged with: load-testing

Performance and performance consistency

EBS documentation describes IO1 as “Highest-performance SSD volume for mission-critical low-latency or high-throughput workloads” and with use cases of “Critical business applications that require sustained IOPS performance, or more than 16,000 IOPS or 250 MiB/s of throughput per volume.” IOPS are defined as a unit of measure representing input/output operations per second with operations measured in Kilobytes (KiB) maxim of 256KiB for SSD volumes.

Mark Olsol, Senior Software Engineer on the EBS team, mentioned in one of the Amazon re:Invent talks that between both volume types “… the performance is very similar, it’s the performance consistency that’s different between the two. In benchmark you won’t notice, but you’ll notice it over time.”

Below 16,000 IOPS or 250MiB/s of data throughput both volume types can be configured to have the same amounts of IOPS and, as Mark said, have very similar performance. With the GP2 volume type, IOPS are provisioned by volume size, 3 IOPS per GB of storage with a minimum of 100 IOPS. Provisioning IOPS for the IO1 volume type is not dependent on the disk size as long as it’s below 50 IOPS per 1 GB rate.

The difference in performance consistency that Olson has mentioned, and the actual consistency numbers are specified in the documentation: “GP2 is designed to deliver the provisioned performance 99% of the time” while IO1 “…designed to deliver the provisioned performance 99.9% of the time”. Therefore, below maximum throughput, if your system can tolerate 99% storage performance consistency and does not need 99.9%, GP2 volume type is a much more cost effective option.

Throughput

Throughput measures how much time it takes for a disk to read or write data. The throughput depends on how many IOPS are configured and how much data is read/written per I/O operation (capped at 256KiB). Maximum throughput for GP2 depends on the instance size and I/O size, with the maximum throughput of 250MiB/s achieved at 1000 IOPS x 256KiB at 334 GiB disk size.

If the database workload is data intensive and requires more than 250MiB/s throughput, GP2 will not be the right volume type. Transactional systems don’t usually read/write large amounts of data at once, but still it is possible to hit the 250MiB/s cap with a workload of more than 16KiB I/O size and 16,000 IOPS (5,334 GB disk size). Of course your workload may be different, and you always need to check the average I/O size for your database.

Cost comparison

Given the same number of provisioned IOPS for both drives and throughput of less than 250MiB/s, the performance consistency with IO1 types does not come cheap. And because the performance is very similar, you can save a lot of money if your application doesn’t need 99.9% performance consistency. Here is a table comparing monthly cost of GP2 baseline IOPS with the exact same size and provisioned IOPS of the IO1 type.

Choosing a correct instance type

Another important factor that could limit the performance of your database is an underlying virtual Amazon Elastic Compute Cloud (EC2) instance type. EBS bandwidth varies between different EC2 instance types, and it’s possible that the EC2 bandwidth is less than the maximum amount of throughput supported by your EBS volume. I’ve personally run into this issue when my application’s database instance type was configured at m4.xlarge with dedicated EBS performance of 750 Mbps which translated to about 93.76 MiB/s, which was less than 250MiB/s expected throughput of the storage. EC2 instance type specifications are listed here.

Summary

Taking time to understand your database workload and differences between the two storage types, GP2 and IO1, can potentially reduce costs and improve the performance of your application.

When choosing a Solid State Drive (SSD) type for your low latency, transactional Amazon Relational Database Service (RDS), Amazon Elastic Block Store (EBS) provides two options: General Storage (GP2) and Provisioned IOPS (IO1). The IO1 type is a much more expensive option, and you want to make sure that your database workload justifies that additional cost. We’ll look at reasons why you may need IO1 vs GP2.

Performance and performance consistency

EBS documentation describes IO1 as “Highest-performance SSD volume for mission-critical low-latency or high-throughput workloads” and with use cases of “Critical business applications that require sustained IOPS performance, or more than 16,000 IOPS or 250 MiB/s of throughput per volume.” IOPS are defined as a unit of measure representing input/output operations per second with operations measured in Kilobytes (KiB) maxim of 256KiB for SSD volumes.

Mark Olsol, Senior Software Engineer on the EBS team, mentioned in one of the Amazon re:Invent talks that between both volume types “… the performance is very similar, it’s the performance consistency that’s different between the two. In benchmark you won’t notice, but you’ll notice it over time.”

Below 16,000 IOPS or 250MiB/s of data throughput both volume types can be configured to have the same amounts of IOPS and, as Mark said, have very similar performance. With the GP2 volume type, IOPS are provisioned by volume size, 3 IOPS per GB of storage with a minimum of 100 IOPS. Provisioning IOPS for the IO1 volume type is not dependent on the disk size as long as it’s below 50 IOPS per 1 GB rate.

The difference in performance consistency that Olson has mentioned, and the actual consistency numbers are specified in the documentation: “GP2 is designed to deliver the provisioned performance 99% of the time” while IO1 “…designed to deliver the provisioned performance 99.9% of the time”. Therefore, below maximum throughput, if your system can tolerate 99% storage performance consistency and does not need 99.9%, GP2 volume type is a much more cost effective option.

Throughput

Throughput measures how much time it takes for a disk to read or write data. The throughput depends on how many IOPS are configured and how much data is read/written per I/O operation (capped at 256KiB). Maximum throughput for GP2 depends on the instance size and I/O size, with the maximum throughput of 250MiB/s achieved at 1000 IOPS x 256KiB at 334 GiB disk size.

If the database workload is data intensive and requires more than 250MiB/s throughput, GP2 will not be the right volume type. Transactional systems don’t usually read/write large amounts of data at once, but still it is possible to hit the 250MiB/s cap with a workload of more than 16KiB I/O size and 16,000 IOPS (5,334 GB disk size). Of course your workload may be different, and you always need to check the average I/O size for your database.

Cost comparison

Given the same number of provisioned IOPS for both drives and throughput of less than 250MiB/s, the performance consistency with IO1 types does not come cheap. And because the performance is very similar, you can save a lot of money if your application doesn’t need 99.9% performance consistency. Here is a table comparing monthly cost of GP2 baseline IOPS with the exact same size and provisioned IOPS of the IO1 type.

Choosing a correct instance type

Another important factor that could limit the performance of your database is an underlying virtual Amazon Elastic Compute Cloud (EC2) instance type. EBS bandwidth varies between different EC2 instance types, and it’s possible that the EC2 bandwidth is less than the maximum amount of throughput supported by your EBS volume. I’ve personally run into this issue when my application’s database instance type was configured at m4.xlarge with dedicated EBS performance of 750 Mbps which translated to about 93.76 MiB/s, which was less than 250MiB/s expected throughput of the storage. EC2 instance type specifications are listed here.

Summary

Taking time to understand your database workload and differences between the two storage types, GP2 and IO1, can potentially reduce costs and improve the performance of your application.

Posted in: blog

Tagged with: aws and rds

One of the challenges for large multi-tenant systems that rely on many external services to complete a single request is ensuring that each dependency is configured correctly in production environments. In addition, if a system is geographically distributed, each instance may depend on different versions of external services.

Even with the best intentions, humans who have to configure and maintain these types of complex systems are known to be unreliable.

… one study of large internet services found that configuration errors by operators were the leading cause of outages, where hardware faults (servers or network) play a role in only 10-25% of outages. Designing Data-Intensive Applications, Martin Kleppmann

Due to many possible test scenarios, manual regression testing after each release is not practical, especially if your team is following a continuous delivery approach.

Monitoring of critical paths provides good feedback about already configured tenants and is an important tool when making production releases. However, some geographical regions may be outside of their peak hours and monitoring alone may not provide fast enough feedback about deployments to production. Also, when new tenants are brought online you still have to validate configuration and dependencies in production.

Traditionally automatic end-to-end tests were reserved only for pre-production environments; however, if your production environments rely on a multitude of external services to complete a single request, extending end-to-end tests to production can help to ensure that all tenant dependencies are configured properly. The goal of these production tests is not to run a full regression suite, but to only test critical paths.

Depending on how your end customers interact with your system, automatic end-to-end tests can run against a public facing web user interface (UI) using something like the Selenium Webdriver, or directly against public API endpoints.

Triggered from a Continuous Integration (CI) system, after deployments to production environments, a test suite will provide an immediate detection and warning of any underlying issues. By having clear results of what is failing, production end-to-end tests will also save time for engineers who might otherwise receive bug reports from different channels and will have to sift through the logs to figure out what went wrong. Finally, because all possible critical paths are tested in production after deploys, tests will provide additional confidence about deploys that may not be present by monitoring alone.

In addition to running end-to-end tests by CI, these tests can be triggered manually or automatically after configuration changes to production. Ability to run critical path tests in production on demand will also reduce manual QA overhead needed to verify that that everything is working as expected after a configuration change.

Unfortunately, running tests in production will create test data in production databases. I personally don’t like to delete data from production databases, even if it’s test data, because your database is a system of record, and it’s important for it to stay intact for troubleshooting. Therefore, you would need to keep track of test users/transactions and filter out test transactions from consumers of your data.

Finally, test results will be fed into a monitoring system and displayed on the team’s dashboard. Alerts are also configured on this data.

Putting it all together

When dealing with a lot of external dependencies and configuration permutations in a system, we need to think outside of the box and engineer solutions that can help us to deal with additional complexity. While it’s important to ship software bug free, there are use cases when it’s much more efficient to verify that software is bug free right after it’s deployed to production.

Posted in: blog

Tagged with: thoughts and ideas

In the last few years, as my team grew in size, one of the problems that kept coming up during retrospective meetings was the poor turn around with code reviews. With a smaller team there was no need for an additional process to find code review volunteers; since every engineer had to pitch in and review code daily. But with a larger team, not having a clear process or rules to follow was starting to affect the team’s performance. The issues were identified as follows:

1. It was difficult to find a code review volunteer.
2. After a volunteer was found, sometimes you would still need to follow up if the review was not getting attention.
3. Some people would volunteer less than others.
4. When comments or replies were posted on code reviews, they were not immediately visible because email notifications from GitHub often had to wait until you checked your email. And to get the review process moving along, sometimes you had to message the reviewer directly.

Initial attempts at creating additional process

After multiple discussions with the team, everyone agreed to introduce a simple rule: at the start of a day, each developer will spend 15 minutes on code reviews. This, in theory, would provide more than enough engineers to complete all outstanding reviews and keep review turnaround under 24 hours. A few months later, the results of the experiment were mixed. There was still a lot of delay with getting code reviewed. Sometimes changes requested during code reviews had to be reviewed again, and if your reviewers were already done for the day, it would have to wait another day. Due to the slow feedback cycle, reviews still could take days to get completed. Finally, some engineers were not contributing every day due to being busy with other work.

After another brainstorming session, the team identified that one of the issues with poor turnaround was that outstanding code reviews were not easily visible. Because we work with many GitHub repositories, it is not practical to go into each one to see what code reviews (pull requests) are outstanding. The proposed solution was to use a “pin” feature in Slack, our instant messenger program, which will add a code review link to the team channel’s dashboard. When engineers finished reviewing, they would add a 👍 emoji to the pinned code review, flagging it as done. When two thumbs up appeared on a pinned request, the requestor would merge a code review and unpin the item. This was not a complicated process to follow, but there was still confusion and after another few months, outstanding reviews started to linger on the team’s channel board.

From volunteering to assignment

In an attempt to uncover the underlying problem, one of the engineers extracted data on the number of reviews per person and noticed that reviews were not evenly distributed. Some people did a lot more than others. Asking for volunteers was not working very well and also created a fairness problem.

At this point, it was obvious that we needed to even out the distribution and prioritize assigning engineers with the least number of reviews. We also thought about automating this process. It was not difficult to write a script that would pull review statistics and assign people with the least amount of reviews. However, to save time we looked online to see if someone had already solved this problem, and we found a Pull Reminders commercial application that did exactly what we needed, plus other useful features.

Initially, when we decided to give Pull Reminders a try, we weren’t confident that it would solve the problem. However, after everyone was onboard, we were surprised to learn that the issue with code reviews did not come up again during retrospective meetings. We changed our process from volunteering to assignment, based on the leader-board from the Pull Reminders app. When you need a review, rather than asking or posting in the channel, you will assign two people with the least number of reviews from the leader-board. Pull reminders will take care of notifying and reminding people about outstanding code reviews. The app also improved our communication because it sent personal slack messages when a comment or reply was posted in your code review. This tremendously improved response and turnaround times.

Summary

It’s been a year since we’ve started using Pull Reminders, and I haven’t noticed any confusion or disconnect about code review responsibilities. The majority of reviews are done within a day or two. And we can finally call the problem that caused a lot of discussions and inefficiencies resolved. Most importantly, the new system removed additional rules that everyone had to remember to follow. Now, the system enforces and notifies engineers when they need to review code, and it’s hard to ignore.

Coming up with new rules for everyone to follow is easy but sometimes ineffective. A much better approach is to create a system that makes new rules hard to ignore.

Posted in: blog

Tagged with: thoughts and ideas

After having integrated in-app payments using Apple platform and processing tens of thousands of purchase receipts through the back-end, I’ve learned a few valuable lessons about issues that can only surface in production. Some of these gotchas are either not documented, hard to find in documentation or just strange behavior that only presents itself in a live environment. This is a guide that I wish I had while I was integrating; it would have saved me a few gray hairs.

Empty in_app transactions array in successful purchase receipts

There is a small percentage of successful purchase receipts, when verified via back-end by calling Apple’s API, that do not contain any in-app transactions. In the response received from the API, in_app array field is empty (in_app:[]). The latest advice I found on this issue was from Apple’s technical support engineer in this forum post. He clarified that when the receipt does not contain any transactions, it’s not a valid receipt and you should never call finishTransaction for these types of receipts. There is also another, older solution to this problem that I think is no longer valid and that is asking users to refresh a purchase receipt. As another highly rated support user in that forum post pointed out, the purchase receipt is refreshed before updatedTransactions callback, therefore the receipt should contain updated information and refreshing it is unnecessary.

Strange transaction id formats in purchase receipts

Normally transaction ids are 15 digits like: 400000123456789. But there are times when our server receives a receipt with the transaction in the following, uuid format: FAB60FFD-906D-48CB-8FED-092C4B2707D6

These strange receipt formats when verified with Apple’s back-end also come back with an empty in_app[] array. There is no definite answer on stackoverflow.com or Apple’s developer forums, but it looks like it could be a hack. The best course of action is to not call finishTransaction on a receipt that doesn’t have transactions and is not valid.

Nil SKPayment.applicationUsername for some transactions in production environment

Calling a method on a nil reference guarantees to crash your app and will make your coworkers give you that look that you don’t know what you are doing.

The applicationUsername property of the SKPayment object is one of the fields that may have a value while you are developing and even pass QA. However, for a small percentage of users it will be null in production. And if you didn’t pay careful attention to documentation you are guaranteed to create a really bad experience for a small percentage of your users. In documentation, the description of this property contains an important section that calls out:

The applicationUsername property is not guaranteed to persist between when you add the payment transaction to the queue and when the queue updates the transaction. Do not attempt to use this property for purposes other than providing fraud detection.

What this also is trying to tell you is that sometimes this property will be nil, and if you call a method on a nil property you will crash your user’s device. If you are not careful, after a user makes a payment, his app will crash and will continue crashing on every restart because your application will try to reprocess the payment transaction on every startup. This behavior is impossible to catch testing.

Being careful with nil fields on callbacks

For purchase callbacks:

The following fields are only available when transactionState is SKPaymentTransactionState.purchase or SKPaymentTransactionState.restored.

• SKPaymentTransaction.transactionDate
• SKPaymentTransaction.transactionIdentifier

For product callbacks:

If Apple rejects your app and in-app purchase products during the approval process, product data for rejected items retrieved via SKProductsRequest will contain some invalid information. Unfortunately, I wasn’t able to pinpoint the problem because it was happening only in production, and we didn’t have debug symbols uploaded at that time. So to be on the safe side, I opted in for checking every SkProduct attribute for null before reading a value or calling method on the attribute. This is more of a brute force solution, but at least your SKProducts callback won’t crash your app after Apple rejects in-app products.

Keep a strong reference to SKProductsRequest

Be sure not to miss the “Note” section of the SKProductRequest description about keeping a strong reference to the SKProductRequest object.

If your app can receive multiple requests to retrieve products, I would add SKProductRequest references to a NSMutableSet data structure and remove them after requestDidFinish callback fires.

Here is an example for initiating get products info request.

- (void)getProducts:(NSSet*)productIDs {
  NSLog(@"Requesting %lu products", (unsigned long)[productIDs count]);

  SKProductsRequest* productsRequest = [[SKProductsRequest alloc] initWithProductIdentifiers:productIDs];
  productsRequest.delegate = self;

  @synchronized(_productsRequests) {
    [_productsRequests addObject: productsRequest];
  }
  [productsRequest start];
}

Example interface:

@interface IapAppleDelegate : NSObject<SKProductsRequestDelegate>
{
  NSMutableSet *_productsRequests;
}

Removing references after request has be completed.

- (void)requestDidFinish:(SKRequest *)request
{
  @synchronized(_productsRequests) {
    [_productsRequests removeObject:request];
  }
}

Summary

In-app purchase integration with Apple may not seem super complex but the devil lies in the details, and you want to be very thorough with documentation. It will pay dividends to read about every property that you will be using data from and following advice from the “Notes” and “Important” sections of documentation. Don’t rely on examples found on the web or in GitHub alone, as they might be incomplete and could get you in trouble.

Other things like empty in_app array or strange transaction id formats don’t get mentioned in the documentation and are left for developers to discover when the app goes live. Hopefully this post has been helpful in providing you information on how to deal with undocumented scenarios.

Posted in: blog

Tagged with: in-app-purchase, ios, storekit, and objective-c

This article describes how to use Google Play Developer API with the Google API Client library with Java to refund Google Play In-App Purchases. Hopefully this information and the example script provided will save someone a bit of time.

Unlike Apple App Store, the Google Play platform offers an ability to refund in-app purchase orders, which can be accomplished using one of the following:

• Manually via the Google Play Console on the web.
• Programmatically via a REST API

If you need to refund more than a handful of orders, the REST API is a life saver. Google also provides a library that makes it very easy to get started. The only downside is there aren’t many examples and the documentation just points to the Javadoc references. Having gone through that exercise, here is my own example on how to refund orders using v3 of the client library.

Prerequisite

You will need a Google Service account that has permissions to manage orders.

1. Go to the API Access page on the Google Play Console.
2. Under Service Accounts, click Create Service Account.
3. Follow the instructions on the page to create your service account.
4. Once you’ve created the service account on the Google Developers Console, click Done. The API Access page automatically refreshes, and your service account will be listed.
5. Click Grant Access to provide the service account the rights to manage orders.

If you get lost you can also try this guide from Google’s docs.

Installation

Add the Google API Client library to your project using your favorite build tool.

Maven

Add the following lines to your pom.xml file:

<project>
  <dependencies>
    <dependency>
      <groupId>com.google.apis</groupId>
      <artifactId>google-api-services-androidpublisher</artifactId>
      <version>v3-rev20200223-1.30.9</version>
    </dependency>
  </dependencies>
</project>

Gradle

repositories {
  mavenCentral()
}
dependencies {
  compile 'com.google.apis:google-api-services-androidpublisher:v3-rev20200223-1.30.9'
}

Example Script

import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.androidpublisher.AndroidPublisher;
import com.google.api.services.androidpublisher.AndroidPublisherScopes;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.Collections;
import java.util.Set;
import java.util.logging.Handler;
import java.util.logging.Level;
import java.util.logging.LogRecord;
import java.util.logging.Logger;

class Scratch {
    public static void main(String[] args) {
        enableLogging();

        Set<String> SCOPES = Collections.singleton(AndroidPublisherScopes.ANDROIDPUBLISHER);
        JacksonFactory jsonFactory = JacksonFactory.getDefaultInstance();
        NetHttpTransport httpTransport = new NetHttpTransport();
        GoogleCredential credential = getGoogleCredentials("private service key goes here")
                                        .createScoped(SCOPES);
        AndroidPublisher androidPublisher = new AndroidPublisher
                .Builder(httpTransport, jsonFactory, credential)
                .setApplicationName("com.yourname.app")
                .build();
        try {
            androidPublisher.orders()
              .refund("com.yourname.app","GPA.3333-7777-6666-44444")
              .execute();
        } catch (IOException e) {
            System.out.println(e.getMessage());
        }
    }

    private static GoogleCredential getGoogleCredentials(String privateKey) {
        try {
            return GoogleCredential.fromStream(
              new ByteArrayInputStream(privateKey.getBytes()));
        } catch (IOException e) {
            throw new IllegalArgumentException("Private key for Google Pay is invalid", e);
        }
    }

    public static void enableLogging() {
        Logger logger = Logger.getLogger(HttpTransport.class.getName());
        logger.setLevel(Level.CONFIG);
        logger.addHandler(new Handler() {
            @Override
            public void close() throws SecurityException {
            }
            @Override
            public void flush() {
            }
            @Override
            public void publish(LogRecord record) {
                // Default ConsoleHandler will print >= INFO to System.err.
                if (record.getLevel().intValue() < Level.INFO.intValue()) {
                    System.out.println(record.getMessage());
                }
            }
        });
    }
}

In the above script:

• Refund API returns 204 HTTP Status Code (No content) if a refund was successful.
• Additional logging is enabled by calling enableLogging() . This provides debug logging of NetHttpTransport request/response.

Hopefully this example script will save someone a bit of time.

Posted in: blog

Tagged with: in-app-purchase, android, and java

The use case I’m describing is when a user’s purchased inventory is tied to an organizational (non-apple) user account and is tracked by an external service. For example, when a user buys a song from your mobile app on iOS or Android device, and then logs in to a web version of your app, the song should be instantly available to play in his/her collection. It also works the other way around, where you can buy something on the web, and it should be available on a mobile device. The purchased inventory and the list of available products is independent of the mobile platform. The bottom line: , it will be the responsibility of your inventory service to make sure that items which were already purchased can only be purchased once and not be displayed to the users.

One of the in-app purchase types that is not compatible with cross-platform payments is the App Store’s non-consumable type. Apple developer documentation defines it as:

Non-consumables are purchased once and do not expire, such as additional filters in a photo app.

This type also comes with a an important requirement:

If your app sells non-consumable, auto-renewable subscription or non-renewing subscription products, you must provide users with the ability to restore them.”

If you want to sell a cross-platform product that can only be purchased once, after reading the Apple definition, you may think that non-consumable type is the answer, but the additional requirement may confuse you.

There is also one place (App Store Review Guidelines) where documentation mentions multi platform services, but it doesn’t say anything about non-consumable types.

3.1.3(b) Multiplatform Services: Apps that operate across multiple platforms may allow users to access content, subscriptions, or features they have acquired in your app on other platforms or your web site, including consumable items in multiplatform games, provided those items are also available as in-app purchases within the app.

One of the features of App Store’s StoreKit is the ability to manage inventory of purchased items, meaning it will be responsible for filtering out items that were already purchased. This is handy if a user has lost or bought a new iOS device because it offers the ability to restore purchased items. However, when you let Apple manage your inventory, the purchased inventory will be tied to a user’s iTunes account and not your cross-platform user account. It’s also problematic if a user purchases an item outside of Apple’s platform; Apple would not be able to restore it.

When the inventory is managed by an external service there is no need for the second purchase management system (apple). The inventory travels with your organizational user account, available after login, and does not need to be restored to the device. Therefore, even though Apple does not mention it, non-consumable types are not compatible with the multiplatform purchase inventory.

This is why you can only use a consumable purchase type when you have an external purchased inventory. As soon as a user makes a purchase you top up his/her organizational account with content and consume the item. This inventory service is also used to filter out items that were already purchased because Apple is not a system of record anymore.

Making all your in-app purchase products consumables, even for items that can only be consumed once, might make you nervous, especially if you haven’t gone through the apple approval process before. But logically there is no other way. So hopefully this post save you some anxiety.

Implementing cross-platform purchases in iOS is not very well documented and can cause some anxiety before submitting your app for Apple’s approval. In this post I’m hoping to provide some basic guidance and explanation about which type of in-app products are best suited for that task.

The use case I’m describing is when a user’s purchased inventory is tied to an organizational (non-apple) user account and is tracked by an external service. For example, when a user buys a song from your mobile app on iOS or Android device, and then logs in to a web version of your app, the song should be instantly available to play in his/her collection. It also works the other way around, where you can buy something on the web, and it should be available on a mobile device. The purchased inventory and the list of available products is independent of the mobile platform. The bottom line: , it will be the responsibility of your inventory service to make sure that items which were already purchased can only be purchased once and not be displayed to the users.

One of the in-app purchase types that is not compatible with cross-platform payments is the App Store’s non-consumable type. Apple developer documentation defines it as:

Non-consumables are purchased once and do not expire, such as additional filters in a photo app.

This type also comes with a an important requirement:

If your app sells non-consumable, auto-renewable subscription or non-renewing subscription products, you must provide users with the ability to restore them.”

If you want to sell a cross-platform product that can only be purchased once, after reading the Apple definition, you may think that non-consumable type is the answer, but the additional requirement may confuse you.

There is also one place (App Store Review Guidelines) where documentation mentions multi platform services, but it doesn’t say anything about non-consumable types.

3.1.3(b) Multiplatform Services: Apps that operate across multiple platforms may allow users to access content, subscriptions, or features they have acquired in your app on other platforms or your web site, including consumable items in multiplatform games, provided those items are also available as in-app purchases within the app.

One of the features of App Store’s StoreKit is the ability to manage inventory of purchased items, meaning it will be responsible for filtering out items that were already purchased. This is handy if a user has lost or bought a new iOS device because it offers the ability to restore purchased items. However, when you let Apple manage your inventory, the purchased inventory will be tied to a user’s iTunes account and not your cross-platform user account. It’s also problematic if a user purchases an item outside of Apple’s platform; Apple would not be able to restore it.

When the inventory is managed by an external service there is no need for the second purchase management system (apple). The inventory travels with your organizational user account, available after login, and does not need to be restored to the device. Therefore, even though Apple does not mention it, non-consumable types are not compatible with the multiplatform purchase inventory.

This is why you can only use a consumable purchase type when you have an external purchased inventory. As soon as a user makes a purchase you top up his/her organizational account with content and consume the item. This inventory service is also used to filter out items that were already purchased because Apple is not a system of record anymore.

Making all your in-app purchase products consumables, even for items that can only be consumed once, might make you nervous, especially if you haven’t gone through the apple approval process before. But logically there is no other way. So hopefully this post save you some anxiety.

Posted in: blog

Tagged with: in-app-purchase, ios, and cross-platform

All these minor improvements had an incredible impact. Within five years after the new performance director took over, the team won 60% of the gold medals for the cycling events at the 2008 Olympic Games in Beijing.

If we apply the same philosophy to the software engineering teams, what type of tiny improvements could have a big impact in the long run? Here are a few that I think can be a good start.

Incremental Refactoring

By creating a good habit of leaving the code base better than it was before, chances are that our projects would be easier to maintain in the long run. Besides making the code more expressive, easier to read and modify, engineers can also add missing unit tests, cleaning up legacy comments, or format code. I think the trick is to have a balance and sprinkle improvements a little bit over time. Nobody wants to review a huge refactoring for correctness when it’s unrelated to the task assigned. As a bonus if you could separate refactoring in a separate pull request, your reviewers will thank you. Finally, if you are looking to learn more about refactoring, I highly recommend the book, Refactoring: Improving the Design of Existing Code.

Small Bug Fixes Without Tickets

Not every bug fix needs to go through a process of writing up a ticket, getting it estimated, prioritized and planned. If the bug fix is small enough and the engineer has the context around it, it is worth fixing it along with the primary task. The cost of the bug fix at that moment will be significantly lower and will not require a context switch for another engineer or yourself a few months later. Finally, customers will appreciate quick fixes if it affects them.

Document Root Cause Analysis for Major Incidents

Every failure is a great opportunity to learn from mistakes. Before we can do that, we must first gather all available information about the incident, ask five whys, and create an action plan on how it can be prevented in the future. It’s also valuable to share this information with the stakeholders and team, because they might have a different perspective and offer valuable insights that might not be obvious to the author. In addition to improving processes, documenting incidents is also valuable for tracking purposes; it helps to determine if the team is improving over the long term.

Reviewing Logs After Deployments

Reviewing error logs after each deployment can help identify issues caused by newly released code and get them resolved right away. It’s much cheaper to fix bugs when the context is fresh in the author’s mind. And it could help to save a lot of time and frustration for customers, support and the engineering team. Some bugs could require a data back-fill or other escalation, and if left unchecked, can create a lot of unnecessary work.

Quick Responses to Questions and Requests

When researchers compiled a huge database of the digital habits of teams at Microsoft, they found that the clearest warning sign of an ineffective manager was being slow to answer emails. Responding in a timely manner shows that you are conscientious — organized, dependable and hardworking. And that matters. In a comprehensive analysis of people in hundreds of occupations, conscientiousness was the single best personality predictor of job performance. - Adam Grant No, You Can’t Ignore Email. It’s Rude.

According to Adam Grant, an organizational psychologist, quick responses are the best predictor of job performance. Because our customers and peers are our top priority, it makes sense to try to get a little bit quicker at responding to emails or messages. I can vouch for this myself. I certainly appreciate when someone on my team reviews my code review request right away; it prevents me from switching off to a different task and allows me to get more stuff done.

README Files

Putting together a README file for projects will save time when someone else, or yourself in the future, has to get started on the project. You can find suggestions on how to make good README files here. Small investments into better documentation will pay off in the future.

Do Not Stop Here

This was not meant to be an exhaustive list, just a starting sample of things that may or may not work for your projects. Most important are not the improvements by themselves, but the willingness to adopt a philosophy of continuous aggregation of improvements that will make a difference in the long run.

In his book, Atomic Habits, author James Clear tells a story about a struggling British cycling team that for over a hundred years won only one gold Olympic medal and did not win a single Tour de France, cycling’s biggest race . After the team hired a new performance director, he brought with him a new philosophy, referred to as “the aggregation of marginal gains,” which focused on tiny improvements in every thing that you do. Together with his team, the new director introduced over a hundred small improvements, including redesigning a bicycle seat, testing different fabrics for air dynamics, and rubbing alcohol on the tires to make them grip better. He even brought in a surgeon to teach riders how to wash their hands better so they would not get sick.

All these minor improvements had an incredible impact. Within five years after the new performance director took over, the team won 60% of the gold medals for the cycling events at the 2008 Olympic Games in Beijing.

If we apply the same philosophy to the software engineering teams, what type of tiny improvements could have a big impact in the long run? Here are a few that I think can be a good start.

Incremental Refactoring

By creating a good habit of leaving the code base better than it was before, chances are that our projects would be easier to maintain in the long run. Besides making the code more expressive, easier to read and modify, engineers can also add missing unit tests, cleaning up legacy comments, or format code. I think the trick is to have a balance and sprinkle improvements a little bit over time. Nobody wants to review a huge refactoring for correctness when it’s unrelated to the task assigned. As a bonus if you could separate refactoring in a separate pull request, your reviewers will thank you. Finally, if you are looking to learn more about refactoring, I highly recommend the book, Refactoring: Improving the Design of Existing Code.

Small Bug Fixes Without Tickets

Not every bug fix needs to go through a process of writing up a ticket, getting it estimated, prioritized and planned. If the bug fix is small enough and the engineer has the context around it, it is worth fixing it along with the primary task. The cost of the bug fix at that moment will be significantly lower and will not require a context switch for another engineer or yourself a few months later. Finally, customers will appreciate quick fixes if it affects them.

Document Root Cause Analysis for Major Incidents

Every failure is a great opportunity to learn from mistakes. Before we can do that, we must first gather all available information about the incident, ask five whys, and create an action plan on how it can be prevented in the future. It’s also valuable to share this information with the stakeholders and team, because they might have a different perspective and offer valuable insights that might not be obvious to the author. In addition to improving processes, documenting incidents is also valuable for tracking purposes; it helps to determine if the team is improving over the long term.

Reviewing Logs After Deployments

Reviewing error logs after each deployment can help identify issues caused by newly released code and get them resolved right away. It’s much cheaper to fix bugs when the context is fresh in the author’s mind. And it could help to save a lot of time and frustration for customers, support and the engineering team. Some bugs could require a data back-fill or other escalation, and if left unchecked, can create a lot of unnecessary work.

Quick Responses to Questions and Requests

When researchers compiled a huge database of the digital habits of teams at Microsoft, they found that the clearest warning sign of an ineffective manager was being slow to answer emails. Responding in a timely manner shows that you are conscientious — organized, dependable and hardworking. And that matters. In a comprehensive analysis of people in hundreds of occupations, conscientiousness was the single best personality predictor of job performance. - Adam Grant No, You Can’t Ignore Email. It’s Rude.

According to Adam Grant, an organizational psychologist, quick responses are the best predictor of job performance. Because our customers and peers are our top priority, it makes sense to try to get a little bit quicker at responding to emails or messages. I can vouch for this myself. I certainly appreciate when someone on my team reviews my code review request right away; it prevents me from switching off to a different task and allows me to get more stuff done.

README Files

Putting together a README file for projects will save time when someone else, or yourself in the future, has to get started on the project. You can find suggestions on how to make good README files here. Small investments into better documentation will pay off in the future.

Do Not Stop Here

This was not meant to be an exhaustive list, just a starting sample of things that may or may not work for your projects. Most important are not the improvements by themselves, but the willingness to adopt a philosophy of continuous aggregation of improvements that will make a difference in the long run.

Posted in: blog

Tagged with: software-engineering

The idea is to answer the following three simple questions:

1. What went well this year?
2. What didn’t go so well this year?
3. What did I learn?

What went well this year?

Reading. Last year stumbled upon a great book on reading itself, How to Read a Book. Prior to reading it, my reading was at the elementary level. With the new knowledge acquired from this book and with practice, hopefully I can progress into analytical reading and beyond. Finally, this book also inspired me to look into books with topics outside of my current interest and difficulty and provided techniques on how to approach hard subjects.

On a different note, I picked an excellent tip from one of the interviews with Derek Sivers on how to retain information. In order to remember ideas from the books that he reads, he takes his notes and publishes them on his website. This gives him the ability to access his notes from anywhere. Finally, he has a habit of reviewing his notes for a random book every morning. After struggling with remembering information from books for a long time, I’m eager to try his approach and have already published a few book notes myself.

This year I read: The Three-Body Problem, Joyful Wisdom: Embracing Change and Finding Freedom, Awareness, Stillness Is the Key, A Guide to the Good Life, Atomic Habits, C++ Crash Course, How to Read a Book.

Overall it wasn’t a stellar year in terms of volume, but rather it’s my approach to reading, improving depth and retention that I am most excited about.

Traveling. I had a great time in Hawaii on the island of Maui with my wife and daughter. It was great to spend a couple of weeks at the beach and swim in the warm waters. One of the highlights was when we took a side trip to the Big Island and went snorkeling in the crystal clear waters with beautiful fish and corals. My wife and I were amazed to see my daughter, almost two years old, fearlessly jumping into the water from the boat and swimming in the deep water with her floating device.

Exercise. Besides having many bouts of back spasms and getting sick more than usual, I was able to come back to my workout routine and be more or less consistent. When we traveled I wasn’t able to exercise and actually experienced craving, which was very encouraging.

Workouts per month in 2019

What didn’t go so well this year?

Health. I was sick more than usual this year. Perhaps my immune system is not strong enough, so when my daughter gets sick there is no way I can avoid the germs. Also my sleep has been a bit broken because I tend to wake up many times. I have been experimenting with different things. The latest sleep improvement approach I’m trying at the moment is to keep the temperature down using a sleep system, which hopefully can help me stay asleep. This past year was also stressful at work, so that could have also contributed.

In addition, I’ve been plagued with multiple instances of back spasms. I found a promising book, Pain Free, that I’m in the process of implementing. Finally, Peter Attia M.D. has a set of videos on proper movement, which I’m also planning to practice this year.

Writing. I didn’t publish a single blog post last year. I urgently need to improve my written communication and strongly believe that the only way to do it is through practice. This year’s main goal is to build a writing habit (Q1), and then scale up my writing slowly to two articles per week.

Products. While I’m very fortunate to work at a great company and enjoy what I do, the downside of it is that I don’t build equity, and at the end of the day, I don’t own anything that I build. These past two years, I was solely focused on my day job, so this year I want to change that. I want to start out small and build a small mobile application for fun and to get into a habit of working on side projects.

What did I learn?

Focusing on systems rather than goals. Goals are great but they alone don’t help if you don’t have a system on how to implement them. I think that was my biggest failure point in the past. I would focus on pure willpower and motivation, work extra hard on a new goal, and then eventually burn out and stop. I think designing a system where I can focus on building a habit first, slowly, one day at a time, and then eventually scaling out effort should help me be consistent over the long term. Finally, the most important point of the system is to make it easier to work even when you are not motivated, or you don’t feel like it.

This also somewhat aligns with what I learned from Stoic’s teachings. You want to set internal goals, or goals that you can achieve, rather than external goals, something that is outside of your control. For example, building a successful business is an external goal because there are a lot of things that are outside of your control, and if you set a goal like that you most likely set yourself up for disappointment or frustration. On the other hand, setting aside thirty minutes a day and working on something to the best of your ability is something that is totally up to you and is achievable.

Writing things down is important. If I want to retain ideas from books, I have to write them down and review them periodically. It is a shame that this year I bought a book and could not tell if I had read it before or not. It had only been four years since I bought it. I think I can get an exponential amount of benefit if I can retain this information, otherwise I’m only getting a small amount of benefit. The same goes for podcasts and audio-books. I enjoy consuming the content, and there are many great insights, but all those wash away very fast if I don’t write down the ideas and action steps.

Financial independence. One way to achieve financial independence is to save 25x of your yearly expenses. The choices we make in regards to our lifestyle get us either further or closer to independence. A closet full of clothes would definitely give you something new and exciting to wear every day, but it may cost you six months of your life. That luxury car may cost you years of sitting in the office at a desk and doing something that you don’t enjoy. Every purchase we make is trade. We trade our time, a non-renewable resource, for things that we don’t need. For myself, I would rather spend this time on something that I can enjoy, like spending more time with my family or working on things that interest me.

Sleep. Quality sleep is super important for many aspects of our wellbeing, including recovery, cognitive performance, and immune function. I’ve been tracking my sleep over the past year with an Oura ring device and it made me aware of the following things that impact my sleep:

• Alcohol. This one is pretty obvious, but I didn’t realize that having a few drinks before bed elevates your heart rate for the entire night, similar to a light walk. Having your heart work overtime through the night when it should be resting is probably not doing a lot of good. It also negatively impacts very important deep and REM sleep.

• Big dinner or dinner with lots of carbohydrates within three hours of bedtime. Having pasta for dinner will keep my heart rate elevated for the first half of the night, affect my deep sleep and cause me to wake up in the middle of the night.

• Caffeine in the afternoon or evening. This affects heart rate variability and REM sleep.

• Late screen time. This affects deep sleep and overall quality of sleep. When I was younger I used to play a lot of video games late into the night; little did I know that this was really bad for my health and athletic performance.

• Sporadic bed time. Going to sleep later than usual will significantly decrease sleep quality by reducing deep sleep.

• Movement. Consistent exercise will decrease the resting heart rate and improve overall quality of sleep. This is pretty obvious but good feedback.

The bottom line. I learned a lot of good things last year which will hopefully help me to grow in the future. This year, I’m planning to implement what I’ve learned and make it a year of building new good habits.

Thanks for reading and happy new year!

This is my first attempt at reviewing my yearly progress. Inspired by James Clear’s yearly reviews, and similar to a sprint retrospective in Scrum methodology, it’s an opportunity to look back and see what things worked that I should continue doing or where I can improve. In addition, because I have a hard time remembering things I did in prior years, having a point of reference will be valuable for my future self.

The idea is to answer the following three simple questions:

1. What went well this year?
2. What didn’t go so well this year?
3. What did I learn?

What went well this year?

Reading. Last year stumbled upon a great book on reading itself, How to Read a Book. Prior to reading it, my reading was at the elementary level. With the new knowledge acquired from this book and with practice, hopefully I can progress into analytical reading and beyond. Finally, this book also inspired me to look into books with topics outside of my current interest and difficulty and provided techniques on how to approach hard subjects.

On a different note, I picked an excellent tip from one of the interviews with Derek Sivers on how to retain information. In order to remember ideas from the books that he reads, he takes his notes and publishes them on his website. This gives him the ability to access his notes from anywhere. Finally, he has a habit of reviewing his notes for a random book every morning. After struggling with remembering information from books for a long time, I’m eager to try his approach and have already published a few book notes myself.

This year I read: The Three-Body Problem, Joyful Wisdom: Embracing Change and Finding Freedom, Awareness, Stillness Is the Key, A Guide to the Good Life, Atomic Habits, C++ Crash Course, How to Read a Book.

Overall it wasn’t a stellar year in terms of volume, but rather it’s my approach to reading, improving depth and retention that I am most excited about.

Traveling. I had a great time in Hawaii on the island of Maui with my wife and daughter. It was great to spend a couple of weeks at the beach and swim in the warm waters. One of the highlights was when we took a side trip to the Big Island and went snorkeling in the crystal clear waters with beautiful fish and corals. My wife and I were amazed to see my daughter, almost two years old, fearlessly jumping into the water from the boat and swimming in the deep water with her floating device.

Exercise. Besides having many bouts of back spasms and getting sick more than usual, I was able to come back to my workout routine and be more or less consistent. When we traveled I wasn’t able to exercise and actually experienced craving, which was very encouraging.

Workouts per month in 2019

What didn’t go so well this year?

Health. I was sick more than usual this year. Perhaps my immune system is not strong enough, so when my daughter gets sick there is no way I can avoid the germs. Also my sleep has been a bit broken because I tend to wake up many times. I have been experimenting with different things. The latest sleep improvement approach I’m trying at the moment is to keep the temperature down using a sleep system, which hopefully can help me stay asleep. This past year was also stressful at work, so that could have also contributed.

In addition, I’ve been plagued with multiple instances of back spasms. I found a promising book, Pain Free, that I’m in the process of implementing. Finally, Peter Attia M.D. has a set of videos on proper movement, which I’m also planning to practice this year.

Writing. I didn’t publish a single blog post last year. I urgently need to improve my written communication and strongly believe that the only way to do it is through practice. This year’s main goal is to build a writing habit (Q1), and then scale up my writing slowly to two articles per week.

Products. While I’m very fortunate to work at a great company and enjoy what I do, the downside of it is that I don’t build equity, and at the end of the day, I don’t own anything that I build. These past two years, I was solely focused on my day job, so this year I want to change that. I want to start out small and build a small mobile application for fun and to get into a habit of working on side projects.

What did I learn?

Focusing on systems rather than goals. Goals are great but they alone don’t help if you don’t have a system on how to implement them. I think that was my biggest failure point in the past. I would focus on pure willpower and motivation, work extra hard on a new goal, and then eventually burn out and stop. I think designing a system where I can focus on building a habit first, slowly, one day at a time, and then eventually scaling out effort should help me be consistent over the long term. Finally, the most important point of the system is to make it easier to work even when you are not motivated, or you don’t feel like it.

This also somewhat aligns with what I learned from Stoic’s teachings. You want to set internal goals, or goals that you can achieve, rather than external goals, something that is outside of your control. For example, building a successful business is an external goal because there are a lot of things that are outside of your control, and if you set a goal like that you most likely set yourself up for disappointment or frustration. On the other hand, setting aside thirty minutes a day and working on something to the best of your ability is something that is totally up to you and is achievable.

Writing things down is important. If I want to retain ideas from books, I have to write them down and review them periodically. It is a shame that this year I bought a book and could not tell if I had read it before or not. It had only been four years since I bought it. I think I can get an exponential amount of benefit if I can retain this information, otherwise I’m only getting a small amount of benefit. The same goes for podcasts and audio-books. I enjoy consuming the content, and there are many great insights, but all those wash away very fast if I don’t write down the ideas and action steps.

Financial independence. One way to achieve financial independence is to save 25x of your yearly expenses. The choices we make in regards to our lifestyle get us either further or closer to independence. A closet full of clothes would definitely give you something new and exciting to wear every day, but it may cost you six months of your life. That luxury car may cost you years of sitting in the office at a desk and doing something that you don’t enjoy. Every purchase we make is trade. We trade our time, a non-renewable resource, for things that we don’t need. For myself, I would rather spend this time on something that I can enjoy, like spending more time with my family or working on things that interest me.

Sleep. Quality sleep is super important for many aspects of our wellbeing, including recovery, cognitive performance, and immune function. I’ve been tracking my sleep over the past year with an Oura ring device and it made me aware of the following things that impact my sleep:

• Alcohol. This one is pretty obvious, but I didn’t realize that having a few drinks before bed elevates your heart rate for the entire night, similar to a light walk. Having your heart work overtime through the night when it should be resting is probably not doing a lot of good. It also negatively impacts very important deep and REM sleep.

• Big dinner or dinner with lots of carbohydrates within three hours of bedtime. Having pasta for dinner will keep my heart rate elevated for the first half of the night, affect my deep sleep and cause me to wake up in the middle of the night.

• Caffeine in the afternoon or evening. This affects heart rate variability and REM sleep.

• Late screen time. This affects deep sleep and overall quality of sleep. When I was younger I used to play a lot of video games late into the night; little did I know that this was really bad for my health and athletic performance.

• Sporadic bed time. Going to sleep later than usual will significantly decrease sleep quality by reducing deep sleep.

• Movement. Consistent exercise will decrease the resting heart rate and improve overall quality of sleep. This is pretty obvious but good feedback.

The bottom line. I learned a lot of good things last year which will hopefully help me to grow in the future. This year, I’m planning to implement what I’ve learned and make it a year of building new good habits.

Thanks for reading and happy new year!

Posted in: blog

Tagged with: yearly-review

I really enjoyed this book, and I would highly recommend it to anyone who is trying to work on something that requires a lot of effort. By establishing tiny, incremental improvements in our lives, the benefits will accumulate and grow, like a financial investment. Eventually the investment will reach a critical point, and you will see a massive change. The author, James Clear has an interesting solution on how to form a habit, you want to make a daily task super easy. For example, if you are trying to be a more consistent writer, your initial goal would be just sitting down to write for two minutes every day until you’ve established a habit. Only after you’ve made it your habit to write a little bit daily, can you progress to increase the difficulty of the task..

Atomic Habits

I really enjoyed this book, and I would highly recommend it to anyone who is trying to work on something that requires a lot of effort. By establishing tiny, incremental improvements in our lives, the benefits will accumulate and grow, like a financial investment. Eventually the investment will reach a critical point, and you will see a massive change. The author, James Clear has an interesting solution on how to form a habit, you want to make a daily task super easy. For example, if you are trying to be a more consistent writer, your initial goal would be just sitting down to write for two minutes every day until you’ve established a habit. Only after you’ve made it your habit to write a little bit daily, can you progress to increase the difficulty of the task..

The entire book is full of many other great techniques and stories. After implementing the author’s suggestions in my own life over the last month, I can already tell that they work better than pure motivation. There were many days when I didn’t feel like performing the task at all, but by just telling myself that it’s super easy and it would take just a few minutes, I was able to complete it. I’m looking forward to continuing to work on building my habits, and if my writing habit works, I will report on the progress.

A Guide to the Good Life: The Ancient Art of Stoic Joy

This book was highly recommended by Tim Ferriss on his podcast, and it quickly became one of my favorites. Stoic philosophy provides ancient wisdom that is still relevant today. It explains that without a coherent philosophy of life, humans are more likely to default to pleasure maximization and distractions that will lead to squandering time, or worse, pursuing something that would not fulfill us in the end. Because Stoic philosophy is based on logic, a lot of it makes sense and does not require you to believe in something abstract or intangible.

Stoics believed that the key to happiness is to prevent ourselves from taking for granted things that we already have. Because a lot of human unhappiness comes from a desire to have more, if we practice appreciation for all the things we worked so hard to get, we can trade desire for enjoyment. In addition to being grateful, another technique would be to imagine losing the things that we already have. This would also make us appreciate our existing situation and not take things, relationships or our health for granted.

Another great piece of wisdom that I really enjoyed was the importance of setting achievable goals. Achievable goals mean, if I play a tennis match, winning the match would be outside of my control, and therefore would not be an achievable goal. Why? Because I cannot control my opponent or the environment; for example how good of a player she is, or if the sun is in my face. Rather than focusing on winning the match, my goal should be to play at the best of my ability, whichis something I could certainly achieve. By setting achievable goals, we prevent ourselves from getting disappointed and as a consequence discouraged and frustrated. In Stoic philosophy achievable goals are also called internal goals, versus external goals something that I don’t have control over.

I hope you enjoy this book as much as I did and that your life will be more joyful and fulfilling.

C++ Crash Course

After more than a decade of working with managed, high level languages, I needed to work on a cross platform project written in C++. I quickly discovered that it was hard to find a good reference on modern C++ without it being too overly complex or having to learn previous versions first. This book filled the gap that I had, and I would highly recommend it to any experienced programmer picking up C++ for the first time.

Flour Water Salt Yeast: The Fundamentals of Artisan Bread and Pizza

I give this book honorable mention because after reading it, I was able to consistently bake great tasting and looking bread. I love my carbs and a lot of times bread bought from the store does not satisfy me. If you’ve never baked bread, it is a great exercise where you can step out of your comfort zone and practice your beginner’s mind. It was also fun learning about the fundamentals of bread baking, with its different flower types, fermentation, and hydration.

Posted in: blog

Tagged with: books

I’ve recently updated a tiny utility script (sqsmover) that I wrote when I was learning Go Lang that moves AWS SQS messages between queues. After three years, it is still a userful tool when I need to move messages between queues. I feel like it could be useful to others who use SQS and deadletter queues because at one point something is going to fail and your messages will end up in the deadletter.

In order to polish it, I’ve included the following features:

Help menu

Region flag

Original version had region hardcoded in the code and needed a code change if you used a different region, oops. I’ve modified to allow region to be overwritten as a command line flag and default to us-west-2.

FIFO queues support

When sending messages to FIFO queues MessageGroupId is a required field, and because it is not part of the message you have to explicitly copy it from the original message to the destination. Finally every message in the FIFO queue has a deduplication id. It’s either explicitly set by the caller or automatically hashed based on the message content. Because I want to support all FIFO queues, deduplication id is also copied when moving messages.

Installation

Updated installation options with homebrew tap install brew install mercury2269/homebrew-tap/sqsmover for macOS, shell script for macOS, Linux, or OpenBSD. And compiled binary for Windows. No longer in order to install the app you have to have Go Lang installed locally and issue a go get ... command. This was pretty easy to do with awesome goreleaser project and companion godownloader. GoReleaser automatically creates binary releases and publishes it to github while GoDownloader will create a bash script for one line installation.

Intuitive error messaging

It sucked when your script fails and you don’t know what went wrong, so I’ve included a lot more user friendly error messages that should help users with figuring out what needs to be done.

Progress indicator

For the first version app I used to just output every message moved to the screen. This is not very useful if you have thousands of messages. A better user experience is to include a progress indicator.

Future work

At the moment the utility is using a batch download and upload which moves about 10 messages at a time. It’s pretty fast if you need to move few thousand of messages but if you have a million stuck in the deadletter queue I could imagine it could take a while. To speed up, I’m thinking to introduce multiple receiver and consumer threads and allow users to specify how many threads to use.

That’s it let me know if you come accross any issues using github, thanks!

Posted in: blog

Tagged with: amazon-sqs

When used to best advantage, exceptions can improve a program’s readability, reliability, and maintainability. When used improperly, they can have the opposite effect. - Joshua Block

When used to best advantage, exceptions can improve a program’s readability, reliability, and maintainability. When used improperly, they can have the opposite effect. - Joshua Block

Most of this wisdom comes from Joshua Bloch’s Effective Java book, see the book for the complete discussion on the exceptions.

Use exceptions only for exceptions conditions

// Horrible abuse of exceptions. Don't ever do this!
try {
  int i = 0;
  while(true)
    range[i++].climb();
} catch (ArrayIndexOutOfBoundsException e) {
	...
}

Better idiom for looping through array is: for (Mountain m : range) m.climb();

• Because exceptions are designed for exceptional circumstances, there is little incentive for JVM implementors to make them fast as explicit tests.
• Placing code inside a try-catch block inhibits certain optimizations that modern JVM implementation might otherwise perform.
• Exception based idiom is much slower than state checking.

A well-designed API must not force its clients to use exceptions for ordinary control flow

Throw exceptions appropriate to the abstraction

Problems when a method throwing an exception that has not apparent connection to the task it performs, usually propagates by lower level abstraction.

• Pollutes API of the higher level with implementation details
• If implementation of the higher layer changes, the exceptions that it throws will change too, potentially breaking clients.

Higher layers should catch lower-level exceptions and, in their place, throw exceptions that can be explained in terms of the higher-level abstraction. This idiom is know as exception translation.

// Exception translation
try {
   httpClient.execute(...)
} catch (IOException e) {
   throws PspApiException(...);
}

// Exception chaining (special form of translation)
try {
   httpClient.execute(...)
} catch (IOException cause) {
   throws PspApiException(cause);
}

class PspApiException extends Exception {
   PspApiException(Throwable cause) {
      super(cause);
   }
}

While exception translation is superior to mindless propagation of exceptions from lower layers, it should not be overused.

• Use validate params
• Deal with exceptions on lower level

Use checked exceptions for recoverable conditions and runtime exceptions for programming errors.

The cardinal rule in deciding whether to use a check or an unchecked exception is this: use checked exceptions for conditions from which the caller can reasonably be expected to recover.

Use runtime exceptions to indicate programming errors. Great majority is precondition violation, a failure of the client to adhere to the contract established by the API specification.

Because checked exceptions generally indicate recoverable conditions, it’s especially important for such exceptions to provide methods that furnish information that could help the caller to recover. For example, redemption of the card has failed because there is no money left, that information could be relayed back to player.

More on where to use checked vs unchecked.

Avoid unnecessary use of checked exceptions

Overuse of checked exceptions can make an API far less pleasant to use. Forces caller to handle the exception.

Burden is justified if the exceptional condition cannot be prevented by the proper use of the API and the programmer using the API can take some useful action once confronted with the exception. Unless both of these conditions hold, an unchecked exception is more appropriate.

Favor the use of standard exceptions

IllegalArgumentException (Non-null parameter value is inappropriate) IllegalStateException (Object state is inappropriate for method invocation) NullPointerException (Parameter value is null where prohibited) IndexOutOfBoundsException, ConcurrentModificationException, UnsupportedOperationException

Document all exceptions thrown by each method

To use a method properly you need to know about exceptions that it throws, therefore, it is critically important that you take the time to carefully document all of the exceptions thrown by each method.

Always declare checked exceptions individually, and document precisely the conditions under which each one is thrown using the Javadoc @throws tag

Unchecked exceptions represent programming errors so it is wise to document them so programmers get familiar with all the errors so they can avoid it.

If an exception is thrown by many methods in a class for the same reason, it is acceptable to document the exception in the class’s documentation comment.

Include failure-capture information in detail messages

To capture the failure, the detail message of an exception should contain values of all parameter and fields that “contributed” to the exception.

public IndexOutOfBoundsException(int lowerBound, int upperBound, int index) {
  // Generate a detailed message that captures the failure
  super("Lower bound: " + lowerBound + ", Upper bound: " + upperBound + ", Index: " + index);

  // Save failure information for programmatic access
  this.lowerBound = lowerBound;
  this.upperBound = upperBound;
  this.index = index;
}

Don’t ignore Exceptions

An empty catch block defeats the purpose of exception.

Exceptions code smells

try {
   // 100 lines of code
  ...
} catch (Exception e) {
  //Something failed.. not sure what or where, is it important or is it a bug, who knows ¯\_(ツ)_/¯
}

Ideally try catch around the line where exception is expected or a few lines to make the code more readable.

Catch all unhandled exceptions in the global error handler and log

Unhandled exceptions indicate programming errors, most likely the API was not used correctly and that’s a bug that could be fixed

Posted in: blog

Tagged with: java and exceptions

I found it straight forward to configure expressjs to send http requests to logs and to setup log parsing. If you use log4js, there is a connectlogger that can capture expressjs/connect http requests. Finally, to make sense of logs I prefer to use ELK Stack with Logstash parsing the incoming logs, Elastic Search indexing, and Kibana for functional dashboards.

I always like to include the response times of http requests. It helps with troubleshooting performance issues down the line. Below is the slightly modified default format with response time appended to the end.

var httpLogFormat =
  ':remote-addr - - [:date] ":method :url ' +
  'HTTP/:http-version" :status :res[content-length] ' +
  '":referrer" ":user-agent" :response-time';
app.use(
  log4js.connectLogger(log4js.getLogger("http"), {
    level: "auto",
    format: httpLogFormat,
  })
);

Example log output: 10.201.44.200 - - [Tue, 29 Sep 2015 04:52:53 GMT] "GET / HTTP/1.1" 302 66 "" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36" 2

Logstash Log4js appender

Using log4js-logstash appender, I specify the following in the log4js config.

{
  "appenders": [
    {
      "type": "log4js-logstash",
      "host": "10.0.0.10",
      "port": 5959,
      "fields": {
        "app": "my-app",
        "env": "dev"
      }
    }
  ],
  "replaceConsole": true
}

Fields are nice to have if you want to tag your logs with application name or environment, so you can tell where the logs are coming from.

Logstash parsing

In order to parse our custom log we need create a logstash pattern, and place it into the /etc/logstash/patterns folder.

Here is the pattern for parsing the log format above: EXPRESSHTTP %{IP:clientip} - - \[%{DATA:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|undefined) \"%{URI:referrer}?\" \"%{DATA:agent}\" %{NUMBER:response_time}

Logstash input and filter

    input {
      tcp {
    	codec => "json"
    	port => 5959
    	type => "my-app"
      }
    }
    filter {
      if [type] == "my-app" {
    	mutate {
    	  rename => { "[@fields][app]" => "app" }
    	  rename => { "[@fields][category]" => "category" }
    	  rename => { "[@fields][env]" => "env" }
    	  rename => { "[@fields][level]" => "priority" }
    	  rename => { "category" => "logger_name" }
    	  rename => { "@message" => "message" }
    	}
      }
      if [type] == "my-app" and [logger_name] == "http" {
    	grok {
    	  patterns_dir => "/etc/logstash/patterns"
    	  match => { "message" => "%{EXPRESSHTTP}" }
    	  add_field => ["received_at", "%{@timestamp}"]
    	  remove_field => ["message"]
    	}
    	date {
    	  match => ["timestamp", "EEE, dd MMM YYYY HH:mm:ss zzz"]
    	  remove_field => "timestamp"
    	}
      }
    }

In the above config, I perform a number of renames on the nested fields so it can have short and clear names. And in the final if clause, I checks if the logger_name is marked with “http” and then perform a grok parsing of the logs. It’s also good to add received time and then parse the actual log time as in the above date filter.

Tagged with: logstash

I’m a big believer in continuous improvement and personal growth. But how do you know if you are actually making progress in your life and becoming better? Are you moving towards your goals or are you full of self-delusion that keeps you at bay?

I can tell you from personal experience, that we can be dishonest with ourselves. It may seem like you are actually moving forward by reading books, talking to your friends about your goals, and making up strange tasks that you need to do before you actually do what you need to do. You might tell yourself, I “If I knew this or that, or had this amazing experience, then I would be ready and able to do it. But there is never going to be a perfect moment! It’s always going to be a moving target and we need to start acting toward, and working on, our goals. Bottom line – we need more doing, something that will actually move us towards the goal.

We need to make mistakes in order to learn. But we cannot fail by being safe and not putting ourselves out there. Hence we grow by doing and learning from being foolish.

At this moment I feel like I need to make mistakes and learn from them. I want to challenge myself and accomplish my goals. That is why I’ve decided to try personal coaching. Here are a few things that a good coach should be able to help you with:

Outside Opinion

Your coach is not your friend, she doesn’t know who you are and she is not going to accept your excuses or be nice to you. She will only look at the results that are being produced rather than excuses, and keep you accountable. In addition, she can provide an unbiased opinion about how honest you are being with yourself.

Keep you accountable

It’s one thing to make a promise to yourself that you are not going to keep, but making a commitment about your goals to another person who you respect, will make you work harder.

Your best interests

Your coach’s motivation is to help you succeed with your goals. Her interests are aligned with your best interests. She is there to support you and move you along the path to your goals. We all want the best for ourselves, but we are also very easy on ourselves at times, which does not produce results.

Work out psychological issues

A lot of times the problems that we experience are only in our heads, and fear of failure stops us before we can continue to grow. An experienced coach should be very good at helping you to work out those issues.

Deliberate practice

Just like in sports, a coach will make you do drills that will help you when it’s show time. In addition, she will help you measure your performance.

Getting clear on your goals

Starting from the end result, a coach should be able to help you work out what the most important thing in your life is. Are you chasing someone else’s dreams, or is this something you truly want for yourself?

Enjoy your life now

Goals are great, but if you are not enjoying what you are doing, what is the point of living? A coach can, at times, also remind you to have fun and live in the present if she notices that you are getting burned out.

I’m looking forward to working with a coach and putting myself in a higher gear on the road of greater fulfillment and personal growth.

Posted in: blog

Tagged with: better-programmer

Software Estimation Story

Building software is much like building a house. Imagine you want to build a house and have a friend who is an architect, so you ask him if it’s possible to build a three bedroom house for around $100,000. He thinks about it and says yes, but it depends on the type of house and what’s going to be in it. In your friend’s mind he imagines a simple house, with no thrills, that he could design and build for that amount. But you might have some other ideas. You want to have a nice swimming pool, a playground, a three car garage, bamboo floors, granite everywhere, a sauna, gold plated mirrors, high ceilings, and etc. You also might want a house on a hill with a view. At this point the house might cost ten times the original amount.

There may also be many times during the project when you forgot to mention or thought of some other important feature, like an entertainment room, that you feel you must have! But your friend has already started working on the house and built the frame. And you might have to redo the entire design and knock down walls to accommodate, it which will make the project off budget, and add a lot of additional time.

It’s not always necessary to know everything up front, but it is important to anticipate large features so that your friend can design a plan that will accommodate changes later.

Steve McConnel, in his great book “Rapid Development”, gives a great summary of the software estimation story that all customers and managers must understand:

The basic software-estimation story is that software development is a process of gradual refinement. You begin with a fuzzy picture of what you want to build and then spend the rest of the project trying to bring that picture into clearer focus. Because the picture of the software you are trying to build is fuzzy, the estimate of the time and effort needed to build it is fuzzy, too. The estimate can come into focus only along with the software itself, which means that software-project estimation is also a process of gradual refinement.

In other words, it’s impossible for a programmer to estimate the project until he has a detailed understanding of what the customer wants. Until a specific feature is understood in detail, the rough estimate could be a factor of sixteen differences for that feature.

Another difficult part is that unless you’ve previously built the same house, you probably won’t have all the details, hence you have a fuzzy picture. Software development is a process of discovery and gradual refinement. Developers work with customers to unearth intricate business rules, features, and capabilities. One way you can bring a picture into the focus is by going through the process of defining a product, thinking through the requirements, and creating a design. Let’s go through what we need to have to better understand what we are trying to build.

Defining a product

Detailed understanding means that a developer can take the product definition and create a model of it in code. Most likely a developer is not a domain expert of the business and he needs a lot of help from the customer to understand it. Imagine that you get a job at an unfamiliar organization, where you had to learn how the business works before you can be productive. You will need a lot of time to get up to speed, and so will a developer in order to build your product. It’s also much cheaper to be make mistakes and have invalid assumptions at this stage, because no code is written and it doesn’t cost anything to correct it.

For the product owner at this stage, a good place to start is to define a clear problem/solution statement, along with user stories, tasks, and use cases. User stories is a simple description of a feature told in the following way:

As a <type of user>, I want <some goal> so that <some reason>.

For example, it may look like this: As an incentive compensation specialist I would be able to set up compensation wrappers so that I can create goals for payees. Does this story make a lot of sense to you? If you are not working in the incentive compensation business, I don’t think it will. Stories like these are an invitation for conversations between developer and business experts. During the conversation the developer will identify terms used, a vocabulary for that business domain, and know ubiquitous language in Domain Driven Design. It will also lead to a better understanding of a product, elements, subsystems and entities.

At this stage any additional information that a business expert can present would be helpful. Examples are screenshots of similar application, high level flowcharts, wireframes, onboarding manuals, etc. A developer can write a documentation summary about the problem/solution domain identifying all the entities, business vocabularies, definitions, subsystems, interactions and actors.

Requirements Specification

After defining the product and learning about the business, we can see a better picture, but it’s still not perfect; at this moment there could be a factor of eight differences between the high and low estimate. The next step is to zoom in even closer and define a list of requirements. The amount of features software can have has a tremendous impact on the design, time, and cost of the application. Even if you practice agile methodologies, you still need to gather a minimal set of requirements so you can plan for the future. Just like in the definition stage, it’s much easier, and cheaper, to make changes during requirements rather than realizing you’ve missed an important feature down the line, and have it take months to redesign the application. (Knock down walls).

Input at this stage from the business side should be a list of important features that this project must have. While a developer’s job is to understand how these features fit together and to create a plan that will accommodate future expansions.

Product Design

We are getting much closer to a better estimate at this point; the difference might be a factor of three between high and low estimates. Over my career I’ve learned that nothing brings software into a picture like creating a high level and detailed design. This is the stage where a developer takes everything learned about the product, and starts modeling the domain. During this stage she will make a lot of mistakes and flush out incorrect assumptions. Again, much cheaper to make mistakes here than during coding. Starting with a high level design of the major subsystems and defining a clear communication between them, then modeling your domain and define major objects and abstractions. (For a more detailed analysis on how to model a domain, see Eric Evan’s book, “Domain Driven Design”).

What if I don’t know what I want?

I think the best way to approach this problem is to hire a User Experience (UX) designer who can take vague ideas and create throw away sketches of the product. Each sketch can then be reviewed, refined, or redrawn. This process will flush out the requirements that will crystalize in an initial prototype, and can be tested on users before you start the expensive software development. A good UX designer in this situation will save you a lot of money because she will bring the requirements and prototype into focus, for the fraction of the cost of building software! By sketching, creating multiple prototypes, and then testing it on users, you know exactly what works and there is no chance of building something that your customers can’t use.

I see a tremendous amount of benefit of hiring a UX designer early in any software project. There is a lot of power in sketching, defining, and user testing multiple prototypes without actually building anything. It’s costly to experiment by building software and I don’t think anyone can get the user experience right the first time.

Getting an estimate

After the picture is more clear, a developer will create a list of low level details with a range estimate for how many man-hours it would take to complete each task. You must get estimates on low levels of details, this way you get a more accurate estimate and you can gauge how the project is progressing. A good developer will also provide additional information about the costs that are associated with each task and other alternatives that could be more cost effective. Like a good contractor who would present and educate you about different counter tops, their benefits, and costs, you want your developer to do the same for you when providing a list of estimates.

It’s also very important not to overlook other important tasks that are not coding, but have a significant impact on schedule and budget, like QA, documentation, deployments, setting up dev/production servers/databases, configuration, integration, holidays, training, and sick days. In addition there should be allowance with working with new languages, working with unfamiliar technologies, and frameworks. There is always a learning curve when doing something new and this should be accounted for.

Make estimation part of your plan Estimation should be part of your plan and you need to spend a significant amount of time on it; don’t rush it.

Word on Agile Software Development

I am a big believer in iterative development, rapid delivery, minimal planning, and many other principles Agile Manifesto stands for. Unfortunately, I’ve noticed some developers use Agile as an excuse to have no planning, design, or requirements analysis at all, which results in the costliest approach of them all: code-like-hell. It’s common, and you want to be wary of ballpark estimates with use of an Agile as an excuse to not do to any planning or design.

One of the first questions non-technical entrepreneurs who have an idea ask me is, “How much would it cost me to build an application and how soon can you build it?” And for someone, who is not familiar with software development, it could be very difficult and even frustrating to get a realistic estimate with a timeframe. You might be surprised to learn that a lot of developers and managers don’t understand how to estimate software either! I’ve personally been a part of large scale development efforts, with ballpark estimates, coming from management, where projects never met the deadlines and were delivered way over budget. Hopefully after reading this post you will have a better understanding of what it takes to get a good estimate.

Software Estimation Story

Building software is much like building a house. Imagine you want to build a house and have a friend who is an architect, so you ask him if it’s possible to build a three bedroom house for around $100,000. He thinks about it and says yes, but it depends on the type of house and what’s going to be in it. In your friend’s mind he imagines a simple house, with no thrills, that he could design and build for that amount. But you might have some other ideas. You want to have a nice swimming pool, a playground, a three car garage, bamboo floors, granite everywhere, a sauna, gold plated mirrors, high ceilings, and etc. You also might want a house on a hill with a view. At this point the house might cost ten times the original amount.

There may also be many times during the project when you forgot to mention or thought of some other important feature, like an entertainment room, that you feel you must have! But your friend has already started working on the house and built the frame. And you might have to redo the entire design and knock down walls to accommodate, it which will make the project off budget, and add a lot of additional time.

It’s not always necessary to know everything up front, but it is important to anticipate large features so that your friend can design a plan that will accommodate changes later.

Steve McConnel, in his great book “Rapid Development”, gives a great summary of the software estimation story that all customers and managers must understand:

The basic software-estimation story is that software development is a process of gradual refinement. You begin with a fuzzy picture of what you want to build and then spend the rest of the project trying to bring that picture into clearer focus. Because the picture of the software you are trying to build is fuzzy, the estimate of the time and effort needed to build it is fuzzy, too. The estimate can come into focus only along with the software itself, which means that software-project estimation is also a process of gradual refinement.

In other words, it’s impossible for a programmer to estimate the project until he has a detailed understanding of what the customer wants. Until a specific feature is understood in detail, the rough estimate could be a factor of sixteen differences for that feature.

Another difficult part is that unless you’ve previously built the same house, you probably won’t have all the details, hence you have a fuzzy picture. Software development is a process of discovery and gradual refinement. Developers work with customers to unearth intricate business rules, features, and capabilities. One way you can bring a picture into the focus is by going through the process of defining a product, thinking through the requirements, and creating a design. Let’s go through what we need to have to better understand what we are trying to build.

Defining a product

Detailed understanding means that a developer can take the product definition and create a model of it in code. Most likely a developer is not a domain expert of the business and he needs a lot of help from the customer to understand it. Imagine that you get a job at an unfamiliar organization, where you had to learn how the business works before you can be productive. You will need a lot of time to get up to speed, and so will a developer in order to build your product. It’s also much cheaper to be make mistakes and have invalid assumptions at this stage, because no code is written and it doesn’t cost anything to correct it.

For the product owner at this stage, a good place to start is to define a clear problem/solution statement, along with user stories, tasks, and use cases. User stories is a simple description of a feature told in the following way:

As a <type of user>, I want <some goal> so that <some reason>.

For example, it may look like this: As an incentive compensation specialist I would be able to set up compensation wrappers so that I can create goals for payees. Does this story make a lot of sense to you? If you are not working in the incentive compensation business, I don’t think it will. Stories like these are an invitation for conversations between developer and business experts. During the conversation the developer will identify terms used, a vocabulary for that business domain, and know ubiquitous language in Domain Driven Design. It will also lead to a better understanding of a product, elements, subsystems and entities.

At this stage any additional information that a business expert can present would be helpful. Examples are screenshots of similar application, high level flowcharts, wireframes, onboarding manuals, etc. A developer can write a documentation summary about the problem/solution domain identifying all the entities, business vocabularies, definitions, subsystems, interactions and actors.

Requirements Specification

After defining the product and learning about the business, we can see a better picture, but it’s still not perfect; at this moment there could be a factor of eight differences between the high and low estimate. The next step is to zoom in even closer and define a list of requirements. The amount of features software can have has a tremendous impact on the design, time, and cost of the application. Even if you practice agile methodologies, you still need to gather a minimal set of requirements so you can plan for the future. Just like in the definition stage, it’s much easier, and cheaper, to make changes during requirements rather than realizing you’ve missed an important feature down the line, and have it take months to redesign the application. (Knock down walls).

Input at this stage from the business side should be a list of important features that this project must have. While a developer’s job is to understand how these features fit together and to create a plan that will accommodate future expansions.

Product Design

We are getting much closer to a better estimate at this point; the difference might be a factor of three between high and low estimates. Over my career I’ve learned that nothing brings software into a picture like creating a high level and detailed design. This is the stage where a developer takes everything learned about the product, and starts modeling the domain. During this stage she will make a lot of mistakes and flush out incorrect assumptions. Again, much cheaper to make mistakes here than during coding. Starting with a high level design of the major subsystems and defining a clear communication between them, then modeling your domain and define major objects and abstractions. (For a more detailed analysis on how to model a domain, see Eric Evan’s book, “Domain Driven Design”).

What if I don’t know what I want?

I think the best way to approach this problem is to hire a User Experience (UX) designer who can take vague ideas and create throw away sketches of the product. Each sketch can then be reviewed, refined, or redrawn. This process will flush out the requirements that will crystalize in an initial prototype, and can be tested on users before you start the expensive software development. A good UX designer in this situation will save you a lot of money because she will bring the requirements and prototype into focus, for the fraction of the cost of building software! By sketching, creating multiple prototypes, and then testing it on users, you know exactly what works and there is no chance of building something that your customers can’t use.

I see a tremendous amount of benefit of hiring a UX designer early in any software project. There is a lot of power in sketching, defining, and user testing multiple prototypes without actually building anything. It’s costly to experiment by building software and I don’t think anyone can get the user experience right the first time.

Getting an estimate

After the picture is more clear, a developer will create a list of low level details with a range estimate for how many man-hours it would take to complete each task. You must get estimates on low levels of details, this way you get a more accurate estimate and you can gauge how the project is progressing. A good developer will also provide additional information about the costs that are associated with each task and other alternatives that could be more cost effective. Like a good contractor who would present and educate you about different counter tops, their benefits, and costs, you want your developer to do the same for you when providing a list of estimates.

It’s also very important not to overlook other important tasks that are not coding, but have a significant impact on schedule and budget, like QA, documentation, deployments, setting up dev/production servers/databases, configuration, integration, holidays, training, and sick days. In addition there should be allowance with working with new languages, working with unfamiliar technologies, and frameworks. There is always a learning curve when doing something new and this should be accounted for.

Make estimation part of your plan Estimation should be part of your plan and you need to spend a significant amount of time on it; don’t rush it.

Word on Agile Software Development

I am a big believer in iterative development, rapid delivery, minimal planning, and many other principles Agile Manifesto stands for. Unfortunately, I’ve noticed some developers use Agile as an excuse to have no planning, design, or requirements analysis at all, which results in the costliest approach of them all: code-like-hell. It’s common, and you want to be wary of ballpark estimates with use of an Agile as an excuse to not do to any planning or design.

Posted in: development practice

Tagged with: estimation

There are mission critical pieces in any business application that might call external services via some sort of an external API over the local network or Internet. The problem with delivering messages over the network is that networks sometime have a tendency to drop messages and cause timeouts. There is a great paper on Kyle Kingsbury’s blog where him and Peter Bailis provide a list of evidence that failures can happen in many levels of the network or operation system. And it’s not only network that can fail, the receiving 3rd party applications might also be down, under load and slow to respond.

Processes, servers, NICs, switches, local and wide area networks can all fail, and the resulting economic consequences are real. … The consequences of these outages range from increased latency and temporary unavailability to inconsistency, corruption, and data loss.

Therefore, if we want to have a reliable communication with external services we need to implement some kind of a retry mechanism that can redeliver messages and recover from faults.

Solving Guaranteed Delivery

Message Queue is one of the solutions that can address this problem and guarantee delivery. Rather than calling an external API within the application process, like a web request, you place a message into the reliable and durable message queue that guarantees that your message will be delivered to the consumer at least once. Since putting a message on the queue is usually a fast operation, it also speeds up your application performance. Most message queues guarantee delivery by providing some sort of a mechanism of acknowledging if message has been received by the consumer. And if consumer doesn’t respond after a period of time the message gets returned into the queue so it can get processed again. This basically guarantees that a message will get delivered or retried for a certain pre-configured number of times.

There are numerous alternative Message Queue solutions with each addressing certain problems in it’s own way. Since the main specification for calling critical services is guaranteed delivery and we are not building something like a high throughput trading application, the Amazon SQS provides the best alternative to the self-hosted MQs. One of the benefits is that you don’t have to administer message queue servers and spend a lot of time figuring out how to setup redundant clusters for reliability and worry about network partitions. Of course you loose on speed of placing a message into the queue. But a 20ms average put call to SQS is also good enough for this problem.

Until you figure out if you actually need something faster and spend all the time to learn and setting up the MQ cluster, I think Amazon SQS provides the best bang for your time. It’s easy to understand, has a great SDK and it’s ready to go. It’s also not very expensive. A million calls costs $1. Yes you do have to poll the queue, but you can also use the long polling and that will reduce the number of calls of one consumer to 1 call every 20 seconds (if no activity), which is about 120k calls per month, or about 12 cents.

Putting a message into the queue is trivial and doesn’t need more explanation. What is not trivial is creating a reliable application with workers to process the messages. We also want the ability to start multiple workers so they can process messages from the queue in parallel rather than one at a time. Since many workers can be started on one node and run in parallel we need our service efficiently use the CPU resources meaning not blocking threads while waiting for I/O.

I wasn’t able to find an open source application that can always listen to the queue and process messages as they come in. So I needed to write my own. A good candidate for this task is a Win32 Service, since it provides a platform for always running service that can also self restart itself on fault and boot up with windows automatically.

Creating Message Processor Win32 Service

The windows service must always be running, meaning that each worker will have a main while loop that will continue indefinitely. Also you need to start multiple workers so you have to use some sort of multi threaded solution. My initial version was to new up multiple Threads that invoke an asynchronous method. Like this:

protected override void OnStart(string[] args)
{
      for (int i = 0; i < _workers; i++)
    {
      new Thread(RunWorker).Start();
    }
}

public async void RunWorker()
{
  while(true)
  {
    // .. get message from amazon sqs sync.. about 20ms
    var message = sqsClient.ReceiveMessage();

    try
    {
        await PerformWebRequestAsync(message);
        await InsertIntoDbAsync(message);
    }
    catch(SomeExeception)
    {
        // ... log
        //continue to retry
        continue;
    }
    sqsClient.DeleteMessage();
  }
}

And it was working fine, however there is a problem with this code. The initial threads that were created would exit once the first execution of the method would hit first await. So there was really no point of creating those threads. In addition, I wasn’t passing a cancellation token to threads so I could not signal it to shut down whenever I wanted to gracefully exit the service. Thanks to Andrew Nosenko who pointed out a better and cleaner way of accomplishing the same goal using tasks.

Rather than starting threads manually you start each task and add it to the List collection. This way the threadpool is efficiently managing your threadpool threads and schedules it according to the system resources.

List<Task> _workers = new List<Task>();
CancellationTokenSource _cts = new CancellationTokenSource();

protected override void OnStart(string[] args)
{
  for (int i = 0; i < _workers; i++)
  {
    _workers.Add(RunWorkerAsync(_cts.Token));
  }
}

And inside of the RunWorkerAsync’s while loop you call token.ThrowIfCancellationRequested(); that will throw OperationCancelException and exit the thread when the cancel is requested.

With windows service when you start a service the main Win32 Service thread gives you some time to start your processes and it must return quickly, meaning not to get blocked. So your OnStop method is where you have to call your Task.WaitAll(_workers) which blocks the current thread until all workers have completed their tasks. So once the OnStop method begins you signal the cancellation token to cancel the tasks, and then you call Task.WaitAll and wait until all tasks run to completion. If all tasks have been completed prior to calling WaitAll it would just continue so there is no risk that it could finish faster. The OnStop method looks like this:

_cts.Cancel();
try
{
    Task.WaitAll(_workers.ToArray());
}
catch (AggregateException ex)
{
    ex.Handle(inner => inner is OperationCanceledException);
}

It uses an AggregateException.Handle method which will throw any unhandled exceptions after it finished running. And since we are only expecting OpearationCanceledException it will just return.

Polishing it up

One problem with windows service application is that it’s hard debug, you cannot just attach the debugger to the running win32 service. To work around this problem we will use a topshelf project. Topshelf allows you to run your windows service just like you would run a console application with the ability to debug and step through the code. It also make it easier to configure, install and uninstall the service.

Here is a quick sample code that will make a message processor console application into a Win32 service.

public class MessageProcessor
{
    List<Task> _workers;
    CancellationTokenSource _cts;
    public MessageProcessor()
    {
        _workers = new List<Task>();
        _cts = new CancellationTokenSource();
    }
    public void Start() {  //.. same as above }
    public void Stop() { //.. same as above }
}

public class Program
{
    public static void Main()
    {
        HostFactory.Run(x =>
        {
            x.Service<MessageProcessor>(s =>
            {
                s.ConstructUsing(name=> new MessageProcessor());
                s.WhenStarted(tc => tc.Start());
                s.WhenStopped(tc => tc.Stop());
            });
            x.RunAsLocalSystem();

            x.SetDescription("Amazon SQS Message Processor");
            x.SetDisplayName("AmazonSQSMessageProcessor");
            x.SetServiceName("AmazonSQSMessageProcessor");
        });
    }
}

Once you build an executable, you can run MessageProcessor.exe install from command line and the service will get installed, additional -help will show you all the commands that you can do.

Summary

Incorporating queues in your application architecture can help with guaranteed delivery of the business critical messages. It can also speed up your application process since it will offload the work to the external process. On the downside, your application becomes dependent on another application running in a separate process and it’s more code to maintain and deploy. To ensure you message processor doesn’t become a single point of failure, you will also need to have at least 2 nodes running this windows service for redundancy. However, if your business requires guaranteed delivery for the mission critical API calls, the overhead of maintaining message queue solution is worth it’s weight.

Posted in: message-queue

Tagged with: windows-service, message-queue, and amazon-sqs

Having lack of knowledge and experience in selling, it was very important for me to hear a real story from a person in software industry, who was actually successful at selling his or her consulting services. Luckily, I came across this great interview on The Eventual Millionare podcast where I learned about Ian Altman . He built and sold software consulting businesses before, and he has coined his own method of selling that actually made a lot of sense to me.

I’ve always had this bias about salespeople being sleazy and dishonest, but what Ian found out is that it doesn’t have to be that way. He explains that if you focus on FIT (Finding Impact Together) you actually become a problem solver who works together with the client rather than against him. You focus on results and impact, and if your service doesn’t bring value to the client you walk away.

Enter “Same Side Selling”

I was eager to learn more about his approach and I picked up Ian’s book right away. Since I never closed a sale before, I found Same Side Selling book very informative. Further, I learned that rather than explaining all the things that you offer, if you focus on actual problems, your client would not need to translate it into his own language. And if what you are describing is an actual problem that his business is having, he will be much more likely interested in what you have to say.

The book is full of wisdom on how to sit down with a customer to identify the problem, and figure out if it’s worth solving. Author brings up analogy of puzzle pieces where you need ask a series of well formulated questions to learn about the pieces that your customer has and see if yours fit. Unless you actually get to the underlying issue you won’t know for sure that your solution will bring the results.

Honesty and focusing on the outcome rather than a sale is the best way to get on the same side with a client, and it’s much more likely to benefit both parties in the long run.

This approach of problem solving fits my personality very well, and I will definitely practice it in the near future.

I will have to read this book over again, especially before a meeting with a client, so I can refresh and tune in on the same side selling method. I will let you know how it goes.

Honest Solution

This book has opened my eyes that selling doesn’t have to be adversarial. That figuring out a problem with a client and getting his business should be mutually beneficial. Honest selling helps building reputation and gets repeat business.

The problem is that most software engineers suck at sales. I personally felt like it was never my job, and never bothered to learn it. Only to discover later that if I ever wanted to grow a successful business I would have to eventually take the bull by the horns and do it. And it’s not only business, being a great salesman can help to build a successful career as well.

Having lack of knowledge and experience in selling, it was very important for me to hear a real story from a person in software industry, who was actually successful at selling his or her consulting services. Luckily, I came across this great interview on The Eventual Millionare podcast where I learned about Ian Altman . He built and sold software consulting businesses before, and he has coined his own method of selling that actually made a lot of sense to me.

I’ve always had this bias about salespeople being sleazy and dishonest, but what Ian found out is that it doesn’t have to be that way. He explains that if you focus on FIT (Finding Impact Together) you actually become a problem solver who works together with the client rather than against him. You focus on results and impact, and if your service doesn’t bring value to the client you walk away.

Enter “Same Side Selling”

I was eager to learn more about his approach and I picked up Ian’s book right away. Since I never closed a sale before, I found Same Side Selling book very informative. Further, I learned that rather than explaining all the things that you offer, if you focus on actual problems, your client would not need to translate it into his own language. And if what you are describing is an actual problem that his business is having, he will be much more likely interested in what you have to say.

The book is full of wisdom on how to sit down with a customer to identify the problem, and figure out if it’s worth solving. Author brings up analogy of puzzle pieces where you need ask a series of well formulated questions to learn about the pieces that your customer has and see if yours fit. Unless you actually get to the underlying issue you won’t know for sure that your solution will bring the results.

Honesty and focusing on the outcome rather than a sale is the best way to get on the same side with a client, and it’s much more likely to benefit both parties in the long run.

This approach of problem solving fits my personality very well, and I will definitely practice it in the near future.

I will have to read this book over again, especially before a meeting with a client, so I can refresh and tune in on the same side selling method. I will let you know how it goes.

Honest Solution

This book has opened my eyes that selling doesn’t have to be adversarial. That figuring out a problem with a client and getting his business should be mutually beneficial. Honest selling helps building reputation and gets repeat business.

Posted in: business

Tagged with: business and sales

Here is a simple database migration model that I recently adopted at work. It worked out really well. With little effort, we are now able to create new or upgrade existing database to any specific commit in the source control. And now since production SQL changes are migrated with an automatic batch process, it has streamlined our deploys as well.

Start with a Baseline Script

Baseline script is necessary for creating database from scratch. It’s used for setting new environments or even daily development work where you want to make sure to start with a clean slate.

If you already have an existing database you will need to start by creating a baseline script. The best time to do that is after a fresh deploy to production. That way you know your development version of database should be the same as production, and you now have a clear starting point or baseline. Here are the steps I took.

• Script entire database schema with objects
• Clean up schema if necessary. (I had to remove replication directives, and creation of production users since those are production only settings.)
• Add any necessary seed data for testing purposes. When you rebuild a database it will most likely be used for development or QA, therefore most likely you will need some starting data.

Add Migration scripts

Migration scripts get applied in order in which they are created and only migrate database up. This model does not involve downgrading, simply because we haven’t found a need for it (production database is never downgraded, and local/test version can always be recreated from scratch).

When a developer is working on a feature or a bug that needs database changes, he creates a new migration file with a next sequential number in the file name. For example: if the last migration file is Script-0048.sql next a new migration script will be Script-0049.sql.

They have to be sequential because that’s how we can make sure that migrations are applied in order they were created, and can guarantee consistency between environments.

Version Control your SQL scripts

Next important piece is to version control your scripts. It plays the following roles:

• Source control becomes a mediator, so multiple developers cannot check-in script with the same name. In addition, if there is a conflict with names developers are forced to get latest and change their script name.
• Each branch has it’s own list of migration scripts, and there is no doubt of what your database should look like to match the code base in any branch or individual commit. It simply must have all migration scripts applied to the baseline.
• It keeps track of changes and we want to make sure there are no changes once a migration script is in source control. (more on that in the Rules section)

Keeping track of migrations

How do we know what migrations scripts were applied to the database? Simple, we create a table that keeps track of executed script. That way it’s easy to compare what’s already executed and what scripts need to be applied to get to a specific point in time. A simple table like this will do.

Finally, your migration application takes care of figuring out which scripts are missing, executing missing scripts, and recording applied scripts in the log table.

Two Easy Rules for Stress Free Migrations

• Once a SQL migration script is committed and pushed to the source control it must never change. We do that to eliminate potential inconsistencies between environments, because once a script is out in the wild you can assume it was already applied somewhere and if script changes that environment will never get updated.
• Automate database migrations completely. There is absolutely no reason why you need to manually run the update scripts, it’s frustrating, error prone, and it’s waste of time. You can quickly write a batch process that will execute each script and add a record into the journal table, or you can use existing open source projects like DbUp for that. We’ve opted in for DbUp since it already does exactly that and has other nice features like wrapping all migration scripts in a transaction.

Rebuild or Migrate Your Database With One Click

We’ve created two Powershell scripts that will either create or upgrade local database with all migration scripts in the current source control tree. Rebuild will execute baseline script + migrations. Upgrade will only apply missing migrations and it’s the same script that’s used in production. There is no more need to use shared database, developer can migrate or re-create his version of the database in few seconds. I’ve also had an idea to include a module that will check on application start if the database is out of date and apply needed scripts, I wouldn’t run it in production but it’s perfect for development.

After setting up automatic migrations it was very easy to setup test environments for functional end to end testing with Selenium. Continuous integration server will pull latest from the code base, run database upgrade script, build and publish site, and execute functional tests.

Conclusion: A lot of impact for a little effort

I’ve been part of many overnight deployments that gone wrong due to some missing stored procedure, and felt the agony of chasing down errors at 2AM in the morning. It really doesn’t take long to apply this model, even less so if you choose to use existing open source libraries like DbUp. While there is nothing radical about this practice, I know a lot of companies are still manually deploying their SQL scripts. It’s a small change with big impact that will streamline your development and make production database migration smooth with guaranteed correctness. It worked out great for my company. How do you manage your database migrations?

Posted in: sql

Tagged with: database-migration

This post was inspired by Gene Kim’s talk on continuous delivery and his eye opening book The Phoenix Project.

What is a continuous delivery? It’s a methodology that allows you to get new features and capabilities to market quickly and reliably. It’s a practice that shortens work in progress and allows for rapid feedback on your new features or improvements. It’s automation of many parts including testing, creation of environments, and one button push deployments. It’s a magical unicorn that allows companies to deploy 10 times a day without much effort or risk of breaking stuff!

The reason why his story strikes a chord with me is because I strongly believe that automation can streamline the software development, make employees happy, and help organizations to become high performers. And automation is a big part of continuous delivery methodology. Throughout my career, I’ve participated in many painful deployments that lasted more than a day and usually were throughout the weekend. And I believe nobody should be a part of that because there is a better way that’s within reach of many companies. As a result, developers can focus on doing stuff that matters, companies can deliver new features much more rapidly, and stake holders can see their ideas spring to life very quickly.

Average Horse

First, let’s dive deeper into the normal software delivery practice of an average company.

• Product managers will come up with arbitrary due dates without doing technical capacity planning, making promises that we cannot keep. And in result, when a due date comes, product is rushed with many shortcuts taken, which results in lower quality and more fragile applications, which means more technical debt and unplanned worked later.
• Security is not even in the picture because new features are not getting to market quickly enough.
• Technical debt continues to get stock piled and is rarely paid off. Like financial interest, it grows over time until most of the time is spent on just paying off the interest in the form of unplanned work.
• Deployments are not automated and it takes long time to manually deploy. Therefore deployments are a lot less frequent, that means a huge number of features are being deployed at once. That means finished work does not make it into production for months, sometimes years (scary), and that means no rapid feedback on performance, adoption, or business value. Comparing that to the manufacturing plant, where at the bottleneck station you have a stock pile of work, and you have to stop everything just to catch up. At that point you cannot give any feedback because other stations already finished their work, and it’s very costly to change already made stuff (unplanned work) and the solution is lower quality product (technical debt).
• Due to lack of automated testing, companies have to deploy even more infrequent since it takes an army of QA engineers to regression test the entire application, and it becomes even longer as more features are deployed.
• Failed features don’t get pruned, but rather just left to rot and accumulate more security, technical, and maintance debt.

Unplanned work is not free, it’s actually really expensive. Another side affect is when you spend all your time firefighting, no time or energy left for planning. When all you do is react, there’s not enough time to do the hard mental work of figuring out whether you can accept new work. Which means more shortcuts, more multitasking. - The Phoenix Project

How Unicorns Work

The main idea behind continuous delivery is to reduce work in progress (WIP) that would allow for quicker feedback of what goes into production. For example, if you work on a 9 month long project it will take you longer than 9 months to see your code in production, and if something has a problem it will be very expensive to go back and change some design decisions. Therefore, there is a good possibility that a fix will just be a hack rather than a proper solution, meaning more technical debt accumulating, more problems later. And not to forget that after 9 months of projects it will take the whole weekend and huge amount of agony to release it.

Gene’s term for WIP in IT terms is a “Lead Time” which measures how long it takes to go from code committed to code successfully running in production. And until code is in production is has absolutely no value because it’s just sitting there. Focusing on fast flow will improve quality, customer satisfaction, return on investment, and employee happiness.

But you think it must be crazy to release so much a day, isn’t it dangerous to make all those changes? It’s actually a lot less risky to deploy small changes incrementally because you get rapid feedback. And even if there is a problem it’s much easier to fix small problem sooner rather than later where you are forced to take shortcuts.

To get there, you have to “improve at bottlenecks”, and any improvements that are not done at the bottlenecks are a waist of time. If your deployment process is a bottleneck, you have to automate it, until it becomes one button push deployment. There is absolutely no good reason why a developer cannot push a button which will create a QA environment that exactly matches production with code deployed, and if it passes automated functional tests it can be put into production with another push of the button. If the regression testing is a bottleneck then you need to pay off that debt by writing automated functional tests or end to end system tests.

“Automated tests transform fear into boredom.” –Eran Messeri, Google

To become high performer, you will also need to add monitoring hooks to your applications, so that any developer can add his or her metrics at will. So when you release often, you can get rapid feedback on the performance, adoption, and value. That way you can make an inform decisions and rollback if necessary. It should be extremely easy for developer to add any kind of monitoring metrics to code and data must be accessible from production.

Gene proposed to spend 20% of the time to work on non-functional improvements, or non feature work, and I think if any organization adopted that they would be on their way of becoming a high performing unicorn. I honestly don’t think it’s much to invest comparing to opportunity cost of features not making out for long periods of time and where only 10% of features are successful. How can you test something when you can only release couple times a year?

Finally, you should be deploying to production with your features turned off that way your releases are not at the same time as deployments and turning features on and off is a simple button click.

It’s not art, it’s production. If you can’t out-experiment and beat your competitors in time to market and agility, you are sunk! Features are always a gamble. If you are lucky, ten percent will get the desired benefits. So the faster you get those features to market and test them, the better you will be. - The Phoenix Project

And if you think 10 deploys a day is crazy take for example Amazon with a mean time between deployments of 11.6 seconds (insane). And it’s not just them, companies like Intuit, Target, Google, Etsy, Flikr, Kansas State University and many others have embraced continuous delivery.

It Does Not Have To Be Radical. Small Steps Are Just Fine.

In a perfect world, a company with problems would stop everything to fix the production line and pay off the technical debt. And some companies like EBay had to do that to escape the vicious cycle. I don’t think it has to be so drastic for an average company. I believe if you accept the culture of continuous improvement, and first focus on all the bottlenecks, you can soon get there. For example, you can make small changes that will bring a lot of improvement. For example, if you deployment is a manual process focus on automatically creating packages and create a script that will automatically deploy. If it requires database changes, add scripts to the package and your deployment script will deploy database changes automatically. There is no reason why a DBA has to compile and execute scripts by hand when it can be automated. If your need many QA engineers to regression test the site, why not spend some of their time to write automated tests, I’m sure they would be happier to find new bugs rather then doing senseless testing of the same stuff.

Final Thoughts

Gene urges us to create a culture of genuine learning and experimentation and that’s how best companies get even better. In additional here a great quote if you think it’s not relevant to you

Most of the money spent are on IT Projects these days and even if companies say it’s not their core competency it’s not true. Everyone must learn or you risk irrelevance in 10 years. - Gene Kim

Good luck and see you in the world of unicorns! :)

Posted in: continuous-delivery

If you want to build next generation platforms or web applications you have to get serious about JavaScript. Rich user interfaces, or Single Page Applications, provide a much better user experience and give products an edge over competition. Many new applications like DropBox, Trello, Windows Azure and many others are a great example of amazing user experience. In addition, JavaScript is already very widely used on the server side as well. It’s incredably fast, asynchronous out of the box, and is a perfect backend for your single page applications. Finally, it has a huge ecosystem with almost as many NPM packages than the largest platform, and catching up really fast. As we’ve seen before where community goes that’s where the next most widely used platform will be.

Why Should I Care About JavaScript?

If you want to build next generation platforms or web applications you have to get serious about JavaScript. Rich user interfaces, or Single Page Applications, provide a much better user experience and give products an edge over competition. Many new applications like DropBox, Trello, Windows Azure and many others are a great example of amazing user experience. In addition, JavaScript is already very widely used on the server side as well. It’s incredably fast, asynchronous out of the box, and is a perfect backend for your single page applications. Finally, it has a huge ecosystem with almost as many NPM packages than the largest platform, and catching up really fast. As we’ve seen before where community goes that’s where the next most widely used platform will be.

And it’s not only startups that choose Node.js, recently a company like PayPal announced that they chose Node.js to be an application platform of choice. And some great benefits they reported so far:

Built almost twice as fast with fewer people
Written in 33% fewer lines of code
Constructed with 40% fewer files

From my personal experience I can also say that development JavaScript is real fun, it’s very fast and I enjoy learning functional nature of it. Paul Graham chose Lisp because it allowed them at ViaWeb to ship code faster than competition. I feel like JavaScript has that advantage now as well. It’s also functionals and it has a large number of open source project that you can pick and choose, so you don’t have to reinvent the wheel. So why wouldn’t you use it if you had to choose a platform?

It’s time to get serious about learning JavaScript

Writing large application will require developers to actually pick up few books and finally learn the language. You can no longer get away with hacking together bunch of spaghetti code with global variables. Good news is that now there are a lot more good resources than there was few years ago. Couple great books that I had a pleasure of reding provide invaluable deep knowledge of the language. Another super resource for learning JavaScript is to read existing open source libraries. You can extract a large amount of knowledge for free if you are willing to roll up your sleeves and get uncomfortable. Finally, like any serious developers we want to write our components with help of unit tests. I’ve also found that testability of Javascript is extremely important, browser debugging is somewhat not efficient and you have to jump through different places due to Javascrit’s asynchronous event loop nature. It’s easier to test components individually then debug entire application.

So here are the books that I’ve really enjoyed and will get you up to speed if you are a seasoned pro in other languages like C# or Java.

Great Books

JavaScript Patters: I found this very helpful with breaking down different patterns of object creations, function types, and overall best practices. It packs a lot great stuff in about 200 pages and will give you a quick intro without going too deep into language itself. After I read this book, I was finally able to understand different object creation patterns and why you would use one over another. If you don’t want to get too deep into language but want to actually write clean code and to be able to read existing librarires I think you can get away with this book. Because this book was written in 2010, most of the examples covered are following EcmaScript 3 standard, so it’s great if you want to support old browsers.

Async JavaScript This is an awesome book to pick up next. It explains very well asynchronous functions, async error handling and event loop. This knowledge is a must for any serious JavaScript programmer. Every page of this book 80 page book is densely packed with information and you want to read it slow and enjoy every bit of it. It also dives into of how to make your callbacks cleaner with promises and deferreds. Finally it takes a look at existing async libraries that can make your life a lot more easier when dealing with multiple asynchronous functions. The biggest “aha” moment when I read this book was an understanding of how event loop queue works and why some libraries execute functions with setTimeout.

Effective JavaScript From the legendary Effective series, this book is lives up to it’s standard. It goes deep into the language in a series of 68 topics. It’s very good at breaking down and explaining topics and contains a wealth of knowledge. It actually explains intricate details about JavaScript semicolon insertion, implicit coercions, and a lot of other goodness like implicit binding of “this”. A lot of subjects are already covered in other books, but this book actually explains them in more detail and why things the way they are. It’s not overly complex but it is very dense with information and joy to read. This is a must read for any serious developer who wants a deeper understanding of the JavaScript language.

Great JavaScript Libraries To Read

Ghost is a new, simple bloging platform that’s build on top of Node.js with Express on backend and Backbone on the client. If you are looking to build a full stack applications with JavaScript this project will get you going. I found it a great way to get up to speed on how configuration, modules, and data access setup. Among other things you can learn about authentication, middleware, and see extensive use of deferreds.

Backbone What I like about backbone is that it has extensive suite of Unit Tests, the library itself is about 1700 lines of code, and it has very good comments. It’s pretty incredible that such small library is the most widely used SPA library out there. I start with reading unit tests to understand the specifications. After I have general knowledge I would dive in to pieces that I find most interesting.

Express Whether you are looking to build a Restful API or a traditional web application, express is great a minimalist framework on top of Node.js for that. It’s pretty simple and genius and also has a pretty small code base.

What others JavaScript libraries or books did you enjoy? Send me a line @mercury2269.

Posted in: javascript

Tagged with: javascript, learning, and book-recommendations

After I uninstalled Visual Studio 2012, the deployment package creation script that builds and publishes a project using MSBuild started throwing a lovely exception:

.csproj(795,3): error MSB4019: The imported project “C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\We bApplications\Microsoft.WebApplication.targets” was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.

From the error, I can tell that MSBuild is using the wrong Visual Studio version. My first thought was to to tell MSBuild to use v12 to build target by adding a VisualStudioVersion environment variable /p:VisualStudioVersion=12.0 but that resulted in a new error that was a little more confusing:

C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\Web\Transform\Microsoft.Web.Publishing.AspNetCompileMerge.targets(132,5): error : Can’t find the valid AspnetMergePath

I guess something has changed, and indeed, after searching I found that that yes MSBuild now ships as a part of Visual Stuido.

So rather than having MSBuild shipped as a component of a .NET framework, it is now a stand alone package that comes with Visual Studio and each version corresponds to the Visual Studio version with it’s own toolsets. So the new MSBuild will be under:

On 32-bit machines they can be found in: C:\Program Files\MSBuild\12.0\bin

On 64-bit machines the 32-bit tools will be under: C:\Program Files (x86)\MSBuild\12.0\bin

So if you are building you project using C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe, without Visual Studio 2012 installed, it will no longer work, and you will need to switch your build tool to use a new msbuild 12.0 in C:\Program Files (x86)\MSBuild\12.0\Bin\MSBuild.exe.

That’s it, hopefully this post will save someone some time.

Posted in: msbuild

Tagged with: visual-studio-2013 and msbuild

Before doing anything else, we must first check the state of an entity. If you are using Code First, an entity would be your POCO class that is mapped to the database table.

To get information about an entity in question, and it’s relation to the current DbContext, you call a DbContext.Entry<TEntity> method. It will return a DbEntityEntry<TEntity> object which can be used to check the state and perform additional actions on the entity.

When you create a new instance of a class outside of EF, it will have a detached state because context is not tracking that object.

var customer = new Customer();
var customerEntry = _context.Entry<T>(customer);
Debug.Write(customerEntry.State) // EntityState.Detached

If your object is in any state other than EntityState.Detached, that means EF knows about it and tracks it, so you don’t need to do anything else, DbContext.SaveChanges() would take care of persisting differences into the database.

When at first I was trying to update database from a detached object I thought that I can simply attach it and set it to be modified.

public override void Update(T entity)
{
	var entry = _context.Entry<T>(entity);

	if (entry.State == EntityState.Detached)
	{
		_context.Set<T>().Attach(entity);
		entry.State = EntityState.Modified;
	}
}

And that would work, but only if the object with the same key is not already present in the context. And if it does exist, you get a nice error:

An object with the same key already exists in the ObjectStateManager. The ObjectStateManager cannot track multiple objects with the same key.

So what do you do when an object is already being tracked by EF and you have a detached object, how do you merge them together?

Fortunately, EF does provide a way to update an existing object from a detached object. There are couple different ways you can do it, and it all depends on the context, and what you know about your object.

Here is the first way you can do it:

Querying Database First And Updating Tracked Object

using(var db = new StoreDbContext())
{
	var existingCart = db.Carts.Find(cartId);
	if(existingCart != null)
	{
		//entity is already in the context
		var attachedEntry = db.Entry(existingCart);
		attachedEntry.CurrentValues.SetValues(newCart);
	}
	else
	{
		//Since we don't have it in db, this is a simple add.
		db.Cart.Add(newCart);
	}
}

After querying a database and finding a record, we know that EF is already tracking the existing object. We then get an entry object of the tracked entity and call attachedEntry.CurrentValues.SetValues to update it with new values. This method works like an auto mapper and updates scalar properties from the passed in object. Also if the property values is different from the original, it sets the property state to be modified.

That works, but requires an extra database call to get an existing record. There is also a way to do that without an extra query.

Not Performing An Extra Database Call And Checking Local Context

There are times when you know for fact an entity is already in the database. An example would be when a database generated id of your object is not set to zero or default. In that situation you can save an extra call by querying the local context first. If an entity does exist in the local context you perform a similar update with attachedEntry.CurrentValues.SetValues, and if it does not exist, you can modify a state of your detached object to modified, which would attach the object and update the database.

For example, when I know that the cart id is not zero, that means it already exists in the database:

if(newCart.Id > 0) // Id already assigned, need to update.
{
	//We query local context first to see if it's there.
	var attachedEntity = db.Carts.Local.Find(newCart.Id);

	//We have it in the context, need to update.
	if (attachedEntity != null) {
		var attachedEntry = _context.Entry(attachedEntity);
		attachedEntry.CurrentValues.SetValues(newCart);
	}
	else
	{
		//If it's not found locally, we can attach it by setting state to modified.
		//This would result in a SQL update statement for all fields
		//when SaveChanges is called.
		var entry = db.Entry(newCart);
		entry.State = EntityState.Modified;
	}
}
else
{
	//This is a simple add since we don't have it in db.
	db.Cart.Add(newCart);
}

There is a small difference when you query the database first. EF knows which values have been modified an only those specific value. On the other hand, when you are blindly attaching modified entity, EF doesn’t know what fields have changes and performs an update on all fields.

It Works, But Not Fully…

When you call CurrentValues.SetValues(newCart) it will update all scalar properties on your newCart object and set them to modified. However, navigation properties would not get the same respect. As of today EF does not support of full object graph merging, and leaves that for you to manage on your own. So if you have newCart.Customer navigational property it would not get updated. It’s the second most requested feature for EF at the moment, so I think they would add it in the future release.

So for now you have to manually call SetValues on all your navigational properties in order for them to be updated. There are also other solutions that people have written that might help with updating a full graph.

Finally, if you object graph is not very large, you can get away with getting DbEntityEntry and calling SetValues on each navigational property. But if you do have a large graph and want something automatic I would try something like GraphDiff first.

Detached objects, or objects that are created outside of Entity Framework(EF), don’t have automatic tracking enabled, and updating database from detached objects is not hard, but requires extra knowledge of EF. With this post I’d like to spell out different ways of doing it.

Before doing anything else, we must first check the state of an entity. If you are using Code First, an entity would be your POCO class that is mapped to the database table.

To get information about an entity in question, and it’s relation to the current DbContext, you call a DbContext.Entry<TEntity> method. It will return a DbEntityEntry<TEntity> object which can be used to check the state and perform additional actions on the entity.

When you create a new instance of a class outside of EF, it will have a detached state because context is not tracking that object.

var customer = new Customer();
var customerEntry = _context.Entry<T>(customer);
Debug.Write(customerEntry.State) // EntityState.Detached

If your object is in any state other than EntityState.Detached, that means EF knows about it and tracks it, so you don’t need to do anything else, DbContext.SaveChanges() would take care of persisting differences into the database.

When at first I was trying to update database from a detached object I thought that I can simply attach it and set it to be modified.

public override void Update(T entity)
{
	var entry = _context.Entry<T>(entity);

	if (entry.State == EntityState.Detached)
	{
		_context.Set<T>().Attach(entity);
		entry.State = EntityState.Modified;
	}
}

And that would work, but only if the object with the same key is not already present in the context. And if it does exist, you get a nice error:

An object with the same key already exists in the ObjectStateManager. The ObjectStateManager cannot track multiple objects with the same key.

So what do you do when an object is already being tracked by EF and you have a detached object, how do you merge them together?

Fortunately, EF does provide a way to update an existing object from a detached object. There are couple different ways you can do it, and it all depends on the context, and what you know about your object.

Here is the first way you can do it:

Querying Database First And Updating Tracked Object

using(var db = new StoreDbContext())
{
	var existingCart = db.Carts.Find(cartId);
	if(existingCart != null)
	{
		//entity is already in the context
		var attachedEntry = db.Entry(existingCart);
		attachedEntry.CurrentValues.SetValues(newCart);
	}
	else
	{
		//Since we don't have it in db, this is a simple add.
		db.Cart.Add(newCart);
	}
}

After querying a database and finding a record, we know that EF is already tracking the existing object. We then get an entry object of the tracked entity and call attachedEntry.CurrentValues.SetValues to update it with new values. This method works like an auto mapper and updates scalar properties from the passed in object. Also if the property values is different from the original, it sets the property state to be modified.

That works, but requires an extra database call to get an existing record. There is also a way to do that without an extra query.

Not Performing An Extra Database Call And Checking Local Context

There are times when you know for fact an entity is already in the database. An example would be when a database generated id of your object is not set to zero or default. In that situation you can save an extra call by querying the local context first. If an entity does exist in the local context you perform a similar update with attachedEntry.CurrentValues.SetValues, and if it does not exist, you can modify a state of your detached object to modified, which would attach the object and update the database.

For example, when I know that the cart id is not zero, that means it already exists in the database:

if(newCart.Id > 0) // Id already assigned, need to update.
{
	//We query local context first to see if it's there.
	var attachedEntity = db.Carts.Local.Find(newCart.Id);

	//We have it in the context, need to update.
	if (attachedEntity != null) {
		var attachedEntry = _context.Entry(attachedEntity);
		attachedEntry.CurrentValues.SetValues(newCart);
	}
	else
	{
		//If it's not found locally, we can attach it by setting state to modified.
		//This would result in a SQL update statement for all fields
		//when SaveChanges is called.
		var entry = db.Entry(newCart);
		entry.State = EntityState.Modified;
	}
}
else
{
	//This is a simple add since we don't have it in db.
	db.Cart.Add(newCart);
}

There is a small difference when you query the database first. EF knows which values have been modified an only those specific value. On the other hand, when you are blindly attaching modified entity, EF doesn’t know what fields have changes and performs an update on all fields.

It Works, But Not Fully…

When you call CurrentValues.SetValues(newCart) it will update all scalar properties on your newCart object and set them to modified. However, navigation properties would not get the same respect. As of today EF does not support of full object graph merging, and leaves that for you to manage on your own. So if you have newCart.Customer navigational property it would not get updated. It’s the second most requested feature for EF at the moment, so I think they would add it in the future release.

So for now you have to manually call SetValues on all your navigational properties in order for them to be updated. There are also other solutions that people have written that might help with updating a full graph.

Finally, if you object graph is not very large, you can get away with getting DbEntityEntry and calling SetValues on each navigational property. But if you do have a large graph and want something automatic I would try something like GraphDiff first.

Posted in: entity-framework

Tagged with: c#, entity-framework, code-first, and detached-objects

As a part of my transition from “wantrepreneuer” to entrepreneur, I’ve been indulging in a great amount of business and motivational content. While on my way to work, I was listening to one of my favorite podcasts, The Entrepreneur On Fire. One of the guests recommended a book that he said completely shifted his mindset about business and put all the pieces together. It got me interested, so I picked up a copy from audible.com. This post is dedicated to the review of that audio book, The E-Myth Revisited: Why Most Small Businesses Don’t Work and What to Do About It. The following is a summary of the concepts that I thought were valuable and my reflections on those ideas.

Three Critical Roles

In order to create a business that’s going to stand the test of time and continue growing, you need to have the following three roles: Entrepreneur, Manager, and Technician. All three must be equal in the amount of strength, energy, and time devoted to it. The entrepreneur does the strategic thinking, planning, innovating, and has a vision for the future. The manager makes sure the established process is followed; she makes sure stuff gets done and brings order to the chaos. The technician works on the product itself.

Because software engineers are great at building things, we don’t necessarily notice or pay enough attention to the lack of other equally important roles in our business. We concentrate almost all of our energy on building things, which is only a one third of a successful business.

Work On Your Business Rather Than In It

When you work in your business, you become self-employed. Working on your business means developing it and creating a system so that you can leverage other people’s time and grow your business. The system has to be well-established and easy enough to be followed by your future employees.

This is where I have a lot to learn. While working on the idea, I really need to put together a list of things that can be completed by other people and not try to do everything myself. Even development work can be outsourced. I understand it won’t be as good as something that I would write, but I think it’s up to me to create a system that will have those quality checks built in. As an entrepreneur, there must be other more important tasks like talking to potential customers and figuring out their pain points rather than creating a perfect data access layer.

Divide Roles and Assign Accountability

When starting a new venture, it’s important to write down all the positions that you envision for your business as it matures. For example, you will have a Chief Executive Officer, a Chief Operating Officer, a Vice President of Sales, Vice President of Marketing, Vice President of Operations, and then you will have your managers and other staff. Once you have all those positions written down, you assign people to be responsible for those roles. You might end up with 12 or more roles. If it’s just you and your partner, then each of you will have to assume many roles. The point is to divide the roles and hold everyone accountable. So, if one of your roles is V.P. of Marketing, then it’s your responsibility to bring customers in the door, and you have all the decision-making ability in that domain. It is also important to have a different mindset when performing a particular job; you must make a mental shift and think to yourself that right now I’m performing job x and from that position I’ll be making my decisions and taking action.

When your business starts to grow, you will slowly hire people to handle some of your responsibilities, so you will no longer be responsible for them.

To ensure that I don’t make the same mistake twice, I will work on making sure that roles are clearly divided when I start a new venture. I’ve had many experiences starting with partners and arguing over many different subjects. Now I see that the problem was that we didn’t have clearly established roles and people assigned. If V.P. of Marketing is your role, then it’s your final decision—not the guy who does your development work—on how you want to send out those email campaigns. Finally, if you are not contributing efficiently in your assigned job, you need to be fired or replaced in that role.

Create a System for Average Skills

If your business depends on a person who is a genius, then you really can’t replace that person, and if he or she leaves, you would be out of business. Building a business in a way that enables an average person to perform a job well is very important because now you can hire anyone and your quality won’t suffer. You add safety nets and create checklists to ensure quality is met.

Focus on Building a Business Rather Than Building a Product

Building a system is more important than building a product. A successful business can build many different products and then change them to adapt to the market. If you lock yourself into a specific product, then you are really at a disadvantage because if the market shifts, you will be out of your job.

Posted in: book-review

Tagged with: entrepreneur

I get excited when playing with frameworks that are different and challenging to a widely-accepted view on the subject at hand. One of those commonly accepted notions is that in order to build a content management system (CMS), you need a web server and a database. One guy who challenged that notion is Tom Preston-Werner who, in his original article “Blogging Like a Hacker”, set out to fix his frustrations with existing systems. He wrote:

I was tired of complicated blogging engines like WordPress and Mephisto. I wanted to write great posts, not style a zillion template pages, moderate comments all day long, and constantly lag behind the latest software release.

He broke down the problem and realized that the core of the platform is your content, your layout, and the templating engine that will allow you to format and display your content. A simple, genius solution that he came up with is a static site generator, a project named Jekyll that will take your posts in the form of markdown files and run them through a generator that converts your layout and templates into a fully-generated static site. You can take that and host it anywhere you like because it’s just a simple site with static files.

Jekyll: A Perfect Fit

Markdown Format

I already love markdown format. I think it fits perfectly with the web. It allows you to craft simple web pages without HTML getting in your way, and it’s unified. There are now numerous editors that support markdown, so you get to choose the one you like rather than get stuck with what your CMS or blogging platform has to offer. In addition, your blog posts are physical text files that you can save in source control or back up in something like Dropbox. And if you ever decide to switch platforms, you can easily take and import those files to the new and shiny thing.

Simple HTML Layouts

You can create your own simple HTML files and drop Liquid template variables to display the content. Super simple and powerful.

Here is an example of a post page with template tags.

<div class="home entry">
  <h1>{{ page.title }}</h1>
  <span class="date">{{ page.date | date_to_long_string }}</span>
  {{ content }} ...
</div>

The page tags are also custom, and you can define your own. It’s as flexible as you can imagine. Here is an example:

---
layout: post
title: "Going Database Free Feels Great"
tags: ["jekyll", "cms"]
---

The date comes from the file name. Jekyll uses filename convention to pull the post date, so your post has to start with a date, for example: 2013-09-21-myblogpost.html.md

You can also generate sitemap and feed files, as well as define your own URL structure, and much more.

Super cheap

Did I mention that it’s super cheap to host a static site because you don’t need that fancy web server and database? All you need is an Amazon S3 account, and it will cost you pennies every month to run your site. You can just as easily host your site on Github for free.

Unlimited traffic

How many times have you seen articles on Hacker News that, when you click on them, give you a nice 500-page error? That’s because their Wordpress on the shared hosting cannot handle all the traffic, and it just bogs down. Well, with static sites hosted in the cloud, you get unlimited traffic potential out of the box. It will serve almost any amount of traffic you can throw at it.

Secure as it can get, no need to maintain

The beauty of the static site is that it’s simple and doesn’t have any programming logic. You don’t need to download any patches or security updates-it’s all just static HTML file goodness. Feels like we are back in 1996, and it’s awesome!

Static Site + CDN = FAST!

Now to my favorite part. Being a performance freak, I have to admit that I decided to try it out because I knew that putting a CDN in front of the static site can make it really fast. I tried Amazon Cloud Front at first, but their speed was not very impressive. I was getting 600-700 ms response times, which is slower than the DiscountASP provider I was using before. I found MaxCDN, and they were just what I was looking for. I now see responses in the 50 ms or less range, and www.webpagetest.org reports 96/100 point results. It’s incredibly fast!! Here is a proof:

In addition, MaxCDN is doing a great job of zipping all of my content for me automatically.

Plugins

Jekyll has a big ecosystem, and there are a lot of plugins that are available. One in particular that I found very useful is jekyll-assets. It’s very easy to set up, and it will take care of your minification, combining of css/js files, coffee script conversion, and cache busting, etc.

What about comments

Aren’t comments something that your platform needs? How do you go about that? Well, that’s simple—you can install a 3rd party solution like Disqus, or you can roll your own with Javascript and Windows Azure mobile services. In this day and age, anything that requires a database can be sprinkled on the page with Javascript, and 3rd parties will host your data very cheaply.

Extras

I’ve created a simple publishing application that you can run from the root of your Jekyll site. It will take contents of the site folder, figure out the differences between local copy and Amazon S3, and publish those differences to Amazon S3. Finally, it will purge cache in MaxCDN.

Summary

By keeping things simple and ignoring prior biases, we gain incredible benefits that solve problems perfectly. I’m excited and looking forward to blogging on my new, shiny, statically-generated site.

A New, Fresh Look at Old and Boring

I get excited when playing with frameworks that are different and challenging to a widely-accepted view on the subject at hand. One of those commonly accepted notions is that in order to build a content management system (CMS), you need a web server and a database. One guy who challenged that notion is Tom Preston-Werner who, in his original article “Blogging Like a Hacker”, set out to fix his frustrations with existing systems. He wrote:

I was tired of complicated blogging engines like WordPress and Mephisto. I wanted to write great posts, not style a zillion template pages, moderate comments all day long, and constantly lag behind the latest software release.

He broke down the problem and realized that the core of the platform is your content, your layout, and the templating engine that will allow you to format and display your content. A simple, genius solution that he came up with is a static site generator, a project named Jekyll that will take your posts in the form of markdown files and run them through a generator that converts your layout and templates into a fully-generated static site. You can take that and host it anywhere you like because it’s just a simple site with static files.

Jekyll: A Perfect Fit

Markdown Format

I already love markdown format. I think it fits perfectly with the web. It allows you to craft simple web pages without HTML getting in your way, and it’s unified. There are now numerous editors that support markdown, so you get to choose the one you like rather than get stuck with what your CMS or blogging platform has to offer. In addition, your blog posts are physical text files that you can save in source control or back up in something like Dropbox. And if you ever decide to switch platforms, you can easily take and import those files to the new and shiny thing.

Simple HTML Layouts

You can create your own simple HTML files and drop Liquid template variables to display the content. Super simple and powerful.

Here is an example of a post page with template tags.

<div class="home entry">
  <h1>{{ page.title }}</h1>
  <span class="date">{{ page.date | date_to_long_string }}</span>
  {{ content }} ...
</div>

The page tags are also custom, and you can define your own. It’s as flexible as you can imagine. Here is an example:

---
layout: post
title: "Going Database Free Feels Great"
tags: ["jekyll", "cms"]
---

The date comes from the file name. Jekyll uses filename convention to pull the post date, so your post has to start with a date, for example: 2013-09-21-myblogpost.html.md

You can also generate sitemap and feed files, as well as define your own URL structure, and much more.

Super cheap

Did I mention that it’s super cheap to host a static site because you don’t need that fancy web server and database? All you need is an Amazon S3 account, and it will cost you pennies every month to run your site. You can just as easily host your site on Github for free.

Unlimited traffic

How many times have you seen articles on Hacker News that, when you click on them, give you a nice 500-page error? That’s because their Wordpress on the shared hosting cannot handle all the traffic, and it just bogs down. Well, with static sites hosted in the cloud, you get unlimited traffic potential out of the box. It will serve almost any amount of traffic you can throw at it.

Secure as it can get, no need to maintain

The beauty of the static site is that it’s simple and doesn’t have any programming logic. You don’t need to download any patches or security updates-it’s all just static HTML file goodness. Feels like we are back in 1996, and it’s awesome!

Static Site + CDN = FAST!

Now to my favorite part. Being a performance freak, I have to admit that I decided to try it out because I knew that putting a CDN in front of the static site can make it really fast. I tried Amazon Cloud Front at first, but their speed was not very impressive. I was getting 600-700 ms response times, which is slower than the DiscountASP provider I was using before. I found MaxCDN, and they were just what I was looking for. I now see responses in the 50 ms or less range, and www.webpagetest.org reports 96/100 point results. It’s incredibly fast!! Here is a proof:

In addition, MaxCDN is doing a great job of zipping all of my content for me automatically.

Plugins

Jekyll has a big ecosystem, and there are a lot of plugins that are available. One in particular that I found very useful is jekyll-assets. It’s very easy to set up, and it will take care of your minification, combining of css/js files, coffee script conversion, and cache busting, etc.

What about comments

Aren’t comments something that your platform needs? How do you go about that? Well, that’s simple—you can install a 3rd party solution like Disqus, or you can roll your own with Javascript and Windows Azure mobile services. In this day and age, anything that requires a database can be sprinkled on the page with Javascript, and 3rd parties will host your data very cheaply.

Extras

I’ve created a simple publishing application that you can run from the root of your Jekyll site. It will take contents of the site folder, figure out the differences between local copy and Amazon S3, and publish those differences to Amazon S3. Finally, it will purge cache in MaxCDN.

Summary

By keeping things simple and ignoring prior biases, we gain incredible benefits that solve problems perfectly. I’m excited and looking forward to blogging on my new, shiny, statically-generated site.

Posted in: jekyll

Tagged with: jekyll, cms, and amazon-s3

Why Do We Need TDD

TDD helps you to design your software components interactively. How does it help you? You write specifications or behaviors of your component and then your write a solution for that behavior and then you refactor to make the code clean. It’s extremely valuable in terms of providing safety net so you can have confidence while refactoring, and keep code clean and maintainable while not breaking desired specifications.

It’s Important to Understand What TDD Doesn’t Do

It doesn’t help you to find bugs, or regression test the whole system. It also doesn’t help you to test interaction between components or if your application is configured properly. There are other means to do that like automated integration tests with projects like Selenium.

Common Mistakes or When Safety Net Becomes Maintenance Hell

I believe almost all developers who start with TDD without guidance of an experienced TDD practitioner are destined to repeat common mistakes and fail. I’ve done this myself, even though I’ve read few TDD books and I thought I had it. Over the last 3 years I’ve been working on and off on a big and complex calculation functionality piece and used TDD to create a regression safety net. I wrote a huge amount of tests but when the time came to change functionality it didn’t go as smooth as planned. Any changes in the systems behavior would break a huge amount of tests and I could see that my tests were making things a lot more difficult instead of helping. So I went on to really understand TDD and truly learn it. I learned that writing a safety net that would actually help is not easy, requires a true understanding of the TDD, use of the design patterns to remove duplicatio,n and keeping unit tests DAMP (Descriptive and Meaninful Phrases).

Trully Understanding TDD

The most important thing: Through each unit test you are specifying one behavior of the component and NOT testing methods of your class. With TDD your unit tests or your specifications become the documentation of how your components are expected to work. BDD tries to address this issue by changing the terminology, so you think in terms of behaviors rather than tests. But to me it’s all the same TDD just with different set of mind. When you truly understand TDD you would do BDD naturally. (I’m not an expert, these are just my understanding from what I learned and research)

Your components must be tested outside-in. Starting with the public API of your service for example. We’ll address some of the setup design patterns to make this possible because you will need to have a descent amount of setup code in order to test the whole component. And since this setup code needs to be in every test and it has to be easy to read and understand.

Keys to Success

• Easy to read tests, your tests must be DAMP, they should be easy to read and understand. Extra special care need to be taken when writing test names because you want your test to fail in the future and you want a person who will read your test to understand exactly what happened without looking at internals. BDD addresses it with specifying Given When Then. With frameworks like xUnit I like to follow Subject_Scenario_Result in my method names.

• Tests must specify one thing, and one thing ONLY. Your unit tests verifies the behavior of only one specification. If you throw in assertions about unrelated behaviors you will soon discover that when a certain thing changes many of your tests will fail and you won’t be able to tell what went wrong, therefore defeating the purpose of unit tests.

• Outside in, you don’t want to test each individual method of your classes, but rather component’s public API, for example if you have a service that returns available shipping options and you a public interface for that method is CalculateShippingOptions you want to specify functionality through that method.

• Tests are really specifications or behaviors of your code. Think of it as documentation for your components.

• Mock all external components which functionality you are not trying to test.

• Treat your unit tests code as your production code, refactoring and keeping it clean is mandatory.

• Don’t over-specify your system. Only leave specs that are absolutely necessary in order for your component to work properly. 100% test coverage is meaningless it creates unnecessary work and maintenance burden and it’s not pragmatic.

Design Patterns to Help

Writing tests outside-in will require a lot of setup for each test and we want that setup code to be reused. BDD frameworks address it by reusing scenario sentences. And for TDD we have design patterns that help immensely.

Builder Pattern

First pattern is the Builder Pattern . When you are creating a scenario for your test, rather then setting up objects manually you should have a class that does that for you. For example you would usually build you objects manually like this:

var shippingOption = new ShippingOption();
shippingOption.ServiceLevel = 888;
shippingOption.Name = "Ground Shipping"
... etc
var shippingOptions = new List<ShippingOption>();
shippingOptions.Add(shippingOption);

That setup code can be huge, you can replace it with ShippingOptionsBuilder

public class ShippingOptionsBuilder
{
	private List<ShippingOption> _options;

	public ShippingOptionsBuilder()
	{
		_options = new List<ShippingOptions>();
	}

	public ShippingOptionsBuilder AddGround()
	{
		var shippingOption = new ShippingOption();
		shippingOption.ServiceLevel = 888;
		shippingOption.Name = "Ground Shipping"
		...etc
		_options.Add(shippingOption);
		return this;
	}
	public List<ShippingOptions> Build()
	{
		return _options;
	}
}

So now to build your setup code you simply do this:

var options = new ShippingOptionsBuilder().AddGround().Build();

And if you later on add other options like next day air you would simply add another method and call it like this:

var options = new ShippingOptionsBuilder().AddGround().AddNextDayAir().Build();

It’s very simple but powerful, if your ShippingOption object changes your setup code would only change in one place rather than in many unit tests. And it also help with readability I can quickly read that line and see exactly what we have in our setup.

Suite Fixture Setup

A component might have multiple dependencies and we need a way to abstract the creation of the component so it can be easily changed later without us needing to modify a lot of tests. This is where suite fixture comes in to play.

Let’s say we have a ShippingCalculator that will be tested but it takes a couple of dependencies. We’ll create a fixture so it can be reused across tests.

public class ShippingCalculatorFixture
{
	public IShippingRepository ShippingRepository { get; set; }
	public IZipRepository ZipRepository { get; set; }

	public ShippingCalculatorFixture()
	{
		ShippingRepository = new Mock<IShippingRepository>().Object;
		ZipRepository = new Mock<ZipRepository>().Object;
	}

	internal ShippingCalculator CreateSut()
	{
		return new ShippingCalculator(ShippingRepository, ZipRepository);;
	}
}

So now in your setup you can create an instance of the fixture that will hold reference to your dependencies and also create a system under test component.

[SetUp]
public Init()
{
	Fixture = new ShippingCalculatorFixture();
	ShippingCalculator = Fixture.CreateSut();
}

[Test]
public ExampleOfTheFixtureSetup()
{
	..your arrange
	//Act
	ShippingCalculator.Calculate();
	... your assert
}

For example we need in our test to mock out repository. It’s easy to access it through the fixture object that holds a reference to it.

[Test]
public ExampleOfTheFixtureSetupWithDependency()
{
	..your arrange
	Mock.Get(Fixture.ShippingRepository).Setup(p => SomeMethod()).Returns(true);

	//Act
	ShippingCalculator.Calculate();
	... your assert
}

You can also create an extension method

internal static class MockExtensions
{
		internal static Mock<T> GetMock<T>(this T obj) where T : class
		{
			return Mock<T>.Get(obj);
		}
}

So it can be simplified like this

[Test]
public ExampleOfTheFixtureSetupWithDependency()
{
	..your arrange
	Fixture.ShippingRepository.GetMock.Setup(p => SomeMethod()).Returns(true);

	//Act
	ShippingCalculator.Calculate();
	... your assert
}

Some really good resources that helped me tremendously with my learning were:

http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises/

http://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-benefits/

http://blog.ploeh.dk/2009/01/28/Zero-FrictionTDD/

Great pluralsight course that goes deeper into patterns: http://pluralsight.com/training/courses/TableOfContents?courseName=advanced-unit-testing

Good luck with your mastering Test Driven Development! Please let me know if I miss something or you might think I’m completely wrong.

I’ve been practicing Test Driven Development (TDD) for a while and recently I learned that my tests were not really helping but actually becoming a huge burden. So for the past couple months I’ve been on a journey to truly master the TDD and I’m at the point now where I think I have a much better understanding of it. Through my struggles and discoveries with doing TDD comes out this blog post.

Why Do We Need TDD

TDD helps you to design your software components interactively. How does it help you? You write specifications or behaviors of your component and then your write a solution for that behavior and then you refactor to make the code clean. It’s extremely valuable in terms of providing safety net so you can have confidence while refactoring, and keep code clean and maintainable while not breaking desired specifications.

It’s Important to Understand What TDD Doesn’t Do

It doesn’t help you to find bugs, or regression test the whole system. It also doesn’t help you to test interaction between components or if your application is configured properly. There are other means to do that like automated integration tests with projects like Selenium.

Common Mistakes or When Safety Net Becomes Maintenance Hell

I believe almost all developers who start with TDD without guidance of an experienced TDD practitioner are destined to repeat common mistakes and fail. I’ve done this myself, even though I’ve read few TDD books and I thought I had it. Over the last 3 years I’ve been working on and off on a big and complex calculation functionality piece and used TDD to create a regression safety net. I wrote a huge amount of tests but when the time came to change functionality it didn’t go as smooth as planned. Any changes in the systems behavior would break a huge amount of tests and I could see that my tests were making things a lot more difficult instead of helping. So I went on to really understand TDD and truly learn it. I learned that writing a safety net that would actually help is not easy, requires a true understanding of the TDD, use of the design patterns to remove duplicatio,n and keeping unit tests DAMP (Descriptive and Meaninful Phrases).

Trully Understanding TDD

The most important thing: Through each unit test you are specifying one behavior of the component and NOT testing methods of your class. With TDD your unit tests or your specifications become the documentation of how your components are expected to work. BDD tries to address this issue by changing the terminology, so you think in terms of behaviors rather than tests. But to me it’s all the same TDD just with different set of mind. When you truly understand TDD you would do BDD naturally. (I’m not an expert, these are just my understanding from what I learned and research)

Your components must be tested outside-in. Starting with the public API of your service for example. We’ll address some of the setup design patterns to make this possible because you will need to have a descent amount of setup code in order to test the whole component. And since this setup code needs to be in every test and it has to be easy to read and understand.

Keys to Success

• Easy to read tests, your tests must be DAMP, they should be easy to read and understand. Extra special care need to be taken when writing test names because you want your test to fail in the future and you want a person who will read your test to understand exactly what happened without looking at internals. BDD addresses it with specifying Given When Then. With frameworks like xUnit I like to follow Subject_Scenario_Result in my method names.

• Tests must specify one thing, and one thing ONLY. Your unit tests verifies the behavior of only one specification. If you throw in assertions about unrelated behaviors you will soon discover that when a certain thing changes many of your tests will fail and you won’t be able to tell what went wrong, therefore defeating the purpose of unit tests.

• Outside in, you don’t want to test each individual method of your classes, but rather component’s public API, for example if you have a service that returns available shipping options and you a public interface for that method is CalculateShippingOptions you want to specify functionality through that method.

• Tests are really specifications or behaviors of your code. Think of it as documentation for your components.

• Mock all external components which functionality you are not trying to test.

• Treat your unit tests code as your production code, refactoring and keeping it clean is mandatory.

• Don’t over-specify your system. Only leave specs that are absolutely necessary in order for your component to work properly. 100% test coverage is meaningless it creates unnecessary work and maintenance burden and it’s not pragmatic.

Design Patterns to Help

Writing tests outside-in will require a lot of setup for each test and we want that setup code to be reused. BDD frameworks address it by reusing scenario sentences. And for TDD we have design patterns that help immensely.

Builder Pattern

First pattern is the Builder Pattern . When you are creating a scenario for your test, rather then setting up objects manually you should have a class that does that for you. For example you would usually build you objects manually like this:

var shippingOption = new ShippingOption();
shippingOption.ServiceLevel = 888;
shippingOption.Name = "Ground Shipping"
... etc
var shippingOptions = new List<ShippingOption>();
shippingOptions.Add(shippingOption);

That setup code can be huge, you can replace it with ShippingOptionsBuilder

public class ShippingOptionsBuilder
{
	private List<ShippingOption> _options;

	public ShippingOptionsBuilder()
	{
		_options = new List<ShippingOptions>();
	}

	public ShippingOptionsBuilder AddGround()
	{
		var shippingOption = new ShippingOption();
		shippingOption.ServiceLevel = 888;
		shippingOption.Name = "Ground Shipping"
		...etc
		_options.Add(shippingOption);
		return this;
	}
	public List<ShippingOptions> Build()
	{
		return _options;
	}
}

So now to build your setup code you simply do this:

var options = new ShippingOptionsBuilder().AddGround().Build();

And if you later on add other options like next day air you would simply add another method and call it like this:

var options = new ShippingOptionsBuilder().AddGround().AddNextDayAir().Build();

It’s very simple but powerful, if your ShippingOption object changes your setup code would only change in one place rather than in many unit tests. And it also help with readability I can quickly read that line and see exactly what we have in our setup.

Suite Fixture Setup

A component might have multiple dependencies and we need a way to abstract the creation of the component so it can be easily changed later without us needing to modify a lot of tests. This is where suite fixture comes in to play.

Let’s say we have a ShippingCalculator that will be tested but it takes a couple of dependencies. We’ll create a fixture so it can be reused across tests.

public class ShippingCalculatorFixture
{
	public IShippingRepository ShippingRepository { get; set; }
	public IZipRepository ZipRepository { get; set; }

	public ShippingCalculatorFixture()
	{
		ShippingRepository = new Mock<IShippingRepository>().Object;
		ZipRepository = new Mock<ZipRepository>().Object;
	}

	internal ShippingCalculator CreateSut()
	{
		return new ShippingCalculator(ShippingRepository, ZipRepository);;
	}
}

So now in your setup you can create an instance of the fixture that will hold reference to your dependencies and also create a system under test component.

[SetUp]
public Init()
{
	Fixture = new ShippingCalculatorFixture();
	ShippingCalculator = Fixture.CreateSut();
}

[Test]
public ExampleOfTheFixtureSetup()
{
	..your arrange
	//Act
	ShippingCalculator.Calculate();
	... your assert
}

For example we need in our test to mock out repository. It’s easy to access it through the fixture object that holds a reference to it.

[Test]
public ExampleOfTheFixtureSetupWithDependency()
{
	..your arrange
	Mock.Get(Fixture.ShippingRepository).Setup(p => SomeMethod()).Returns(true);

	//Act
	ShippingCalculator.Calculate();
	... your assert
}

You can also create an extension method

internal static class MockExtensions
{
		internal static Mock<T> GetMock<T>(this T obj) where T : class
		{
			return Mock<T>.Get(obj);
		}
}

So it can be simplified like this

[Test]
public ExampleOfTheFixtureSetupWithDependency()
{
	..your arrange
	Fixture.ShippingRepository.GetMock.Setup(p => SomeMethod()).Returns(true);

	//Act
	ShippingCalculator.Calculate();
	... your assert
}

Some really good resources that helped me tremendously with my learning were:

http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises/

http://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-benefits/

http://blog.ploeh.dk/2009/01/28/Zero-FrictionTDD/

Great pluralsight course that goes deeper into patterns: http://pluralsight.com/training/courses/TableOfContents?courseName=advanced-unit-testing

Good luck with your mastering Test Driven Development! Please let me know if I miss something or you might think I’m completely wrong.

Posted in: tdd

Grouping related files in visual studio.

I’ve noticed that classes that inherit from an interface are grouped together in the Visual Studio like this:

That is pretty neat, that way you don’t have to hunt around for all implementations of the interface and it’s all in one place.

By default Visual Studio groups related .aspx files like .cs and .designer.cs. If you look at the .csproj file files that are nested under the main file have a DependenUpon element under the the parent compile element.

<Compile Include="SiteMap.aspx.designer.cs">
	<DependentUpon>SiteMap.aspx</DependentUpon>
</Compile>

By default Visual Studio does not have an option to group arbitrary files but it could easily be done by editing a csproj file. So it’s like a trick to work around default functionality. If you don’t want to manually update that file there is a small little plugin NestIn that will do that for you.

Nested Settings Class

MiniProfiler has a nested static class so you can access settings by going to MiniProfiler.Settings.SomeSetting. I like that composition for settings and it’s pretty easy to do. All you have to do is to set your classes partial and then create a static Settings subclass underneath the partial class like this:

    partial class MiniProfiler
    {
        /// <summary>
        /// Various configuration properties.
        /// </summary>
        public static class Settings
        {

You can also group your settings class with your main class and that’s what is done in MiniProfiler.

Default Values

MiniProfiler.Settings uses DefaultValue Attributes on properties. It’s important to note that this attribute does not initialize the property and that needs to be done separately.

[DefaultValue(20)]
public static int MaxUnviewedProfiles { get; set; }

Initialization happens using reflection in the static constructor of the Settings class.

static Settings()
{
var props = from p in typeof(Settings).GetProperties(BindingFlags.Public | BindingFlags.Static)
let t = typeof(DefaultValueAttribute)
where p.IsDefined(t, inherit: false)
let a = p.GetCustomAttributes(t, inherit: false).Single() as DefaultValueAttribute
select new { PropertyInfo = p, DefaultValue = a };

foreach (var pair in props)
{
	pair.PropertyInfo.SetValue(null, Convert.ChangeType(pair.DefaultValue.Value, pair.PropertyInfo.PropertyType), null);
}
...

I think that’s a nice way to initialize the properties. It keeps declaration and initialization together and easier to see. I wouldn’t do something like that if you had a small settings class and would just initialize in the constructor. But on the large classes it makes sense.

Hooking up Routes

There is a single handler file that has a static method to initialize itself as a route. And it gets called when MiniProfiler sets up ProfilerProvider. What I found cool about this file is that it handles its own registration into routes.

public static void RegisterRoutes()
{
	var routes = RouteTable.Routes;
	var handler = new MiniProfilerHandler();
	var prefix = MiniProfiler.Settings.RouteBasePath.Replace("~/", string.Empty).EnsureTrailingSlash();

	using (routes.GetWriteLock())
	{
		var route = new Route(prefix + "{filename}", handler)
		{
		// we have to specify these, so no MVC route helpers will match, e.g. @Html.ActionLink("Home", "Index", "Home")
		Defaults = new RouteValueDictionary(new { controller = "MiniProfilerHandler", action = "ProcessRequest" }),
		Constraints = new RouteValueDictionary(new { controller = "MiniProfilerHandler", action = "ProcessRequest" })
		};

		// put our routes at the beginning, like a boss
		routes.Insert(0, route);
	}
}

Keep in mind , in order for static files like .js and .css to get routed to the managed module you will need to turn on

<system.webServer>
	<modules runAllManagedModulesForAllRequests="true" />
</system.webServer>

or if you don’t want to turn it on globally you can do it based on the specific url like this:

<system.webServer>
	...
	<handlers>
	<add name="MiniProfiler" path="mini-profiler-resources/*" verb="*" type="System.Web.Routing.UrlRoutingModule" resourceType="Unspecified" preCondition="integratedMode" />
	</handlers>
</system.webServer>

You can basically accomplish the same thing if you create an ashx handler and configure it in the handlers section. By setting up routes you can at least cover clients that have runAllManagedModulesForAllRequest set to true, so they would have one less thing to worry about. But it would also cause some frustration for people who don’t.

That’s it for now, I’ll list more as I’m working through it.

I’ve been reviewing MiniProfiler library source code and here are things that I though were unusual and neat. The reason why it caught my attention is because it’s not something that I use every day so I wanted to list it here. Also, I know that developers who I work with at my job don’t use these features also so they could probably find it interesting.

Grouping related files in visual studio.

I’ve noticed that classes that inherit from an interface are grouped together in the Visual Studio like this:

That is pretty neat, that way you don’t have to hunt around for all implementations of the interface and it’s all in one place.

By default Visual Studio groups related .aspx files like .cs and .designer.cs. If you look at the .csproj file files that are nested under the main file have a DependenUpon element under the the parent compile element.

<Compile Include="SiteMap.aspx.designer.cs">
	<DependentUpon>SiteMap.aspx</DependentUpon>
</Compile>

By default Visual Studio does not have an option to group arbitrary files but it could easily be done by editing a csproj file. So it’s like a trick to work around default functionality. If you don’t want to manually update that file there is a small little plugin NestIn that will do that for you.

Nested Settings Class

MiniProfiler has a nested static class so you can access settings by going to MiniProfiler.Settings.SomeSetting. I like that composition for settings and it’s pretty easy to do. All you have to do is to set your classes partial and then create a static Settings subclass underneath the partial class like this:

    partial class MiniProfiler
    {
        /// <summary>
        /// Various configuration properties.
        /// </summary>
        public static class Settings
        {

You can also group your settings class with your main class and that’s what is done in MiniProfiler.

Default Values

MiniProfiler.Settings uses DefaultValue Attributes on properties. It’s important to note that this attribute does not initialize the property and that needs to be done separately.

[DefaultValue(20)]
public static int MaxUnviewedProfiles { get; set; }

Initialization happens using reflection in the static constructor of the Settings class.

static Settings()
{
var props = from p in typeof(Settings).GetProperties(BindingFlags.Public | BindingFlags.Static)
let t = typeof(DefaultValueAttribute)
where p.IsDefined(t, inherit: false)
let a = p.GetCustomAttributes(t, inherit: false).Single() as DefaultValueAttribute
select new { PropertyInfo = p, DefaultValue = a };

foreach (var pair in props)
{
	pair.PropertyInfo.SetValue(null, Convert.ChangeType(pair.DefaultValue.Value, pair.PropertyInfo.PropertyType), null);
}
...

I think that’s a nice way to initialize the properties. It keeps declaration and initialization together and easier to see. I wouldn’t do something like that if you had a small settings class and would just initialize in the constructor. But on the large classes it makes sense.

Hooking up Routes

There is a single handler file that has a static method to initialize itself as a route. And it gets called when MiniProfiler sets up ProfilerProvider. What I found cool about this file is that it handles its own registration into routes.

public static void RegisterRoutes()
{
	var routes = RouteTable.Routes;
	var handler = new MiniProfilerHandler();
	var prefix = MiniProfiler.Settings.RouteBasePath.Replace("~/", string.Empty).EnsureTrailingSlash();

	using (routes.GetWriteLock())
	{
		var route = new Route(prefix + "{filename}", handler)
		{
		// we have to specify these, so no MVC route helpers will match, e.g. @Html.ActionLink("Home", "Index", "Home")
		Defaults = new RouteValueDictionary(new { controller = "MiniProfilerHandler", action = "ProcessRequest" }),
		Constraints = new RouteValueDictionary(new { controller = "MiniProfilerHandler", action = "ProcessRequest" })
		};

		// put our routes at the beginning, like a boss
		routes.Insert(0, route);
	}
}

Keep in mind , in order for static files like .js and .css to get routed to the managed module you will need to turn on

<system.webServer>
	<modules runAllManagedModulesForAllRequests="true" />
</system.webServer>

or if you don’t want to turn it on globally you can do it based on the specific url like this:

<system.webServer>
	...
	<handlers>
	<add name="MiniProfiler" path="mini-profiler-resources/*" verb="*" type="System.Web.Routing.UrlRoutingModule" resourceType="Unspecified" preCondition="integratedMode" />
	</handlers>
</system.webServer>

You can basically accomplish the same thing if you create an ashx handler and configure it in the handlers section. By setting up routes you can at least cover clients that have runAllManagedModulesForAllRequest set to true, so they would have one less thing to worry about. But it would also cause some frustration for people who don’t.

That’s it for now, I’ll list more as I’m working through it.

Posted in: miniprofiler

Powershell has great documentation inside of the shell, including examples. To turn on complete documentation add -full argument.

Get-Help [command name or topic]
Get-Help [command name or topic] -full

All powershell commands start with a verb and followed by a noun.

Get-Command

To get a list of commands:

Get-command

All commands related to process or noun

get-command -noun process

All commands that have a verb

get-command -verb get

Commands

List of processes

get-process

To stop process and get help on the command, to see examples.

get-help stop-process -full	

What if?

Almost all commands have -WhatIf argument, you can try a command before executing.

Pipe

get-process | sort-object -property Id

To see the list of all available properties from the command

get-process | get-member

Get-Member

Think of it when you have a question, what do I have, or what does this object have.

Filtering and blocks

get-process | where-object { $_.processname -eq 'powershell' }

Curly braces represent the filter block. The $_ represents the current item.

get process | where-object { $_.CPU -gt 1 } | foreach-object { $_.processname + " is over threashold " }

Outputting to the file

get process | where-object { $_.CPU -gt 1 } | foreach-object { $_.processname + " is over threashold " } | out-file cpuinfo.txt

Execution policy

Get-ExecutionPolicy

You won’t be able to run any scripts if the scripts are restricted. To change execution policy.

Set-ExecutionPolicy remotesigned

Scripts

Once you have your execution policy set to remotesigned you can execute scripts that were created on the local machine. Save scripts with .ps1 extension. And script can be executed by providing a path to the script in the powershell. Like ./somescript.ps1. First dot stands for the current directory.

Great Topics with Get-Help

Get-Help About_Execution_Policies
Get-Help About_Operators
Get-Help About_Common_Parameters
Get-Help About_Pipelines
Get-Help About_Scripts
Get-Help About_*

#Creating scripts to start applications and more.

Typing is better than clicking. Automate tasks that are common with windows explorer.

Get your current location

get-location
$pwd
gl

To change your location:

set-location /.documents
chdir ..
cd ./documents

Push location will save your current location before moving to the new location. Once you are done use pop-location to get back to where you were before.

push-location /
pushd ~/desktop
pop-location
popd

Common use is to control the location inside of the script

Navigation

Drive -> Folder -> file

Powershell treats many things like drives, for example registry, functions and system drives.

get-psdrive

You can cd into certificate stores

cd cert:
dir

Environment

cd env:
dir

Registry

cd hkcu:
dir

Push and pop work on custom drives.

Item commands

This are very useful.

get-command *item

Invoke-item

Equivalent as double click, will open with default registered file. Alias

ii message.txt

Will open folder if set to the folder.

http://bit.ly/psphere to add to the windows explorer open in powershell

Automating projects

cd ~/documents/project
push-location "~/documents/project/test"


function push-project( $project ) 
{ 
	# 1: navigate powershll to the project folder
	pushd "~/document/project/$project"
	# 2: open windows explorer at the project folder
	invoke-item .;
	# 3: open any solution found in the project folder
	invoke-item *.sln;
}

Profiles

Powershell saves scripts into the users profile

$profile

Create a profile script file

new-item -type file -path $profile
invoke-item $profile

You can add the function to your user profile. And you can add an alias for the function

new-alias -name pp -value push-project

There are 4 profile scripts in powershell

$profile | select *

Get-Help *Location
Get-Help *Item
Get-Help About_Providers
Get-Help About_Functions
Get-Help About_Profiles

Get-Help

Powershell has great documentation inside of the shell, including examples. To turn on complete documentation add -full argument.

Get-Help [command name or topic]
Get-Help [command name or topic] -full

All powershell commands start with a verb and followed by a noun.

Get-Command

To get a list of commands:

Get-command

All commands related to process or noun

get-command -noun process

All commands that have a verb

get-command -verb get

Commands

List of processes

get-process

To stop process and get help on the command, to see examples.

get-help stop-process -full	

What if?

Almost all commands have -WhatIf argument, you can try a command before executing.

Pipe

get-process | sort-object -property Id

To see the list of all available properties from the command

get-process | get-member

Get-Member

Think of it when you have a question, what do I have, or what does this object have.

Filtering and blocks

get-process | where-object { $_.processname -eq 'powershell' }

Curly braces represent the filter block. The $_ represents the current item.

get process | where-object { $_.CPU -gt 1 } | foreach-object { $_.processname + " is over threashold " }

Outputting to the file

get process | where-object { $_.CPU -gt 1 } | foreach-object { $_.processname + " is over threashold " } | out-file cpuinfo.txt

Execution policy

Get-ExecutionPolicy

You won’t be able to run any scripts if the scripts are restricted. To change execution policy.

Set-ExecutionPolicy remotesigned

Scripts

Once you have your execution policy set to remotesigned you can execute scripts that were created on the local machine. Save scripts with .ps1 extension. And script can be executed by providing a path to the script in the powershell. Like ./somescript.ps1. First dot stands for the current directory.

Great Topics with Get-Help

Get-Help About_Execution_Policies
Get-Help About_Operators
Get-Help About_Common_Parameters
Get-Help About_Pipelines
Get-Help About_Scripts
Get-Help About_*

#Creating scripts to start applications and more.

Typing is better than clicking. Automate tasks that are common with windows explorer.

Get your current location

get-location
$pwd
gl

To change your location:

set-location /.documents
chdir ..
cd ./documents

Push location will save your current location before moving to the new location. Once you are done use pop-location to get back to where you were before.

push-location /
pushd ~/desktop
pop-location
popd

Common use is to control the location inside of the script

Navigation

Drive -> Folder -> file

Powershell treats many things like drives, for example registry, functions and system drives.

get-psdrive

You can cd into certificate stores

cd cert:
dir

Environment

cd env:
dir

Registry

cd hkcu:
dir

Push and pop work on custom drives.

Item commands

This are very useful.

get-command *item

Invoke-item

Equivalent as double click, will open with default registered file. Alias

ii message.txt

Will open folder if set to the folder.

http://bit.ly/psphere to add to the windows explorer open in powershell

Automating projects

cd ~/documents/project
push-location "~/documents/project/test"


function push-project( $project ) 
{ 
	# 1: navigate powershll to the project folder
	pushd "~/document/project/$project"
	# 2: open windows explorer at the project folder
	invoke-item .;
	# 3: open any solution found in the project folder
	invoke-item *.sln;
}

Profiles

Powershell saves scripts into the users profile

$profile

Create a profile script file

new-item -type file -path $profile
invoke-item $profile

You can add the function to your user profile. And you can add an alias for the function

new-alias -name pp -value push-project

There are 4 profile scripts in powershell

$profile | select *

Get-Help *Location
Get-Help *Item
Get-Help About_Providers
Get-Help About_Functions
Get-Help About_Profiles

Tagged with: powershell

System-level configuration, for the entire PC c:\Program Files (x86)\Git\etc\gitconfig or ~\.gitconfig on MAC. Accessed by:

git config --system

User-level configuration, stored in c:\users<name>.gitconfig

git config --global

Repository level configuration, stored .git/config

git config

Show all global options

git config --global --list

Setting properties on global

git config --global user.name
git config --global user.email

Configuring core editor

git config --global core.editor "'C:/Program Files (x86)/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"

Configuring beyond compare

git config --global diff.tool bc3
git config --global difftool.bc3.path "c:/program files (x86)/beyond compare 3/bcomp.exe"
git config --global difftool.prompt false
git config --global merge.tool bc3
git config --global mergetool.bc3.path "c:/program files (x86)/beyond compare 3/bcomp.exe"
git config --global mergetool.prompt false
git config --global mergetool.keepbackup false
git config --global mergetool.keeptemporaries false

Configuring auto correct for the git commands

git config --global help.autocorrect 1

Use colors to show a lot of git information

git config --global color.ui auto

Auto carriage return line feed. False is for windows only will submit it as is. True will convert. Input is for mac, it will convert crlf into lf.

git config --global core.autocrlf false

To remove a setting

git config --unset user.name

You can also edit the config files yourself.

Working Locally

Creating a local repository

git init
git status
git add file.txt
git status
git commit

Look at the history and see the commit SHA: git log

Add all modified/updated files

get add -u

Commit with message

git commit -m "Message"

To view the diffs between SHA1 hashes

git diff dd2929..333322

Easier, latest commit is know as HEAD

git diff HEAD~1..HEAD
git diff HEAD~1..

Add all files including untracked files

git add -A

Stage as two different commits, add specific files and then commit.

git add file1.txt
git commit

Deleting Files

Delete the file first. Then stage the deletion with

git add -u

Rename the file first then:

git add -A

To clear out unstaged changed to the README.txt file

git checkout README.txt

To do a global clear out, and remove all unstaged changes git reset –hard

To take out the last commit out of the repository and move back to the staging area, that way you can change that commit around

git reset --soft HEAD~1

make changes

git commit -m "reorganized commit"

To delete last commit and discard all the changes

git reset --hard HEAD~1

Remove unstaged files

git clean -n //what would it do
git clean -f //performs the operation

.gitignore

Create a file that will list all of the files and directories that need to be omitted.

Working Remote

clone repository

git clone https://....

see list of commits

git log

commits per line

git log --oneline

see number of commits

git log --oneline | wc -1

see the graph git log –graph

Authors and commit messages

git shortlog

Statistics about authors

git shortlog -sne

Look at the specific commit

git show HEAD
git show HEAD~10
git show 939393

Look at the remotes, origin is the git default where the repository came from

git remote
git remote -v //verbose shows remote  urls

To see the repository config cat .get/config

Display all the branches in the repository

git branch

Display all the remote branches

git branch -r

Look at the tags, known stable versions

git tags

To add a remote repository, is commonly down to evaluate patches and pull requests into your project

git remote add origin https://github.com/....

Once remote repository is added you can

git fetch  //all
git fetch origin

to see changes on remote repository

get log origin/master

Merge remote branch into the your current branch

git merge origin/master

Fast forward means, replay commits that happen if there were no other changes and move the HEAD

Shortcut for git fetch; git merge origin/master is

git pull

To set your branch to mirror some other remote branch, or remote tracking branch, or upstream tracking.

git branch --set-upstream master origin/master

If you can also specify without setting the remote like this. origin is the remote name and master is the remote branch.

git pull origin master

Git commit and add modified files together

git commit -am "Some message"

Push your changes to the remote git push

To remove the origin on the current branch

git remote rm origin

to tag a current version, or stable point

git tag v1.0.0

to create tag with the annotation

git tag -a v1.0

To create a tag with the signature, signed with the password

git tag -s v1.0

To verify a tag

git tag -v v1.0

To push tags to remote (by default git would not push tags)

git push --tags

Branching Merging and Rebasing with Git

list of the commits on the current branch with a graph

git log --graph --oneline

To visualize all branches, decorate applies the branch and tag information

git lot --graph --oneline --all --decorate

Adding aliases in the git config

git config --global alias.lga "log --graph --oneline --all --decorate"
git config --global alias.lg "log --graph --all --pretty=format:'%C(bold)%h%Creset -%C(auto)%d%Creset %s %C(green dim)(%cr)%Creset %C(ul)<%an>'"
git lga
git lg
cat ~/.gitconfig

To Create a new branch git branch feature1

Checkout new branch

git checkout feature1

Big difference between branches and tags is that tags always stay on the same commit where branches move along.

git checkout master

Create a branch from a specific commit

git branch fix1 9929292
git checkout fix1

To rename a branch is to move it

git branch -m fix1 bug1234

To delete a branch

git branch -d bug1234

To force delete a branch

git branch -D bug1234

To create a branch and checkout (shortcut) git checkout -b feature2

To get back a deleted branch. From a dangling commit, a commit that doesn’t have a home. These are kept for 30 days.

git reflog // log of all references
git branch bug1234 5a78c8b //with the first 8 of the SHA

To see head

git show HEAD

Save half completed work is to stash it. Temporary holding area for the changes that are not committed to the branch.

git stash
git status
git stash list //holding area
git stash apply //to pull the changes back
git stash pop //applies it and removes it from the list

Merging

Merge fiture1 into the current branch

git merge feature1
git branch -d feature1 //deletes unused branch

Fast forward merge, is just moving the branch label into the new location.

To resolve conflicts using the merge tool

git mergetool

Compare the repository to the staging area

git diff --cached

Rebasing

You can play back your commits on a certain branch rather creating a branch if there are no conflicts. Rebase current branch on top of master

git rebase master

You can also use mergetool to solve merge conflicts and then continue

git rebase --continue

Cherry Pick

If you need to get one commit and apply it on top of a branch.

git cherrypick sks939ss

#Working with remote

git fetch origin master //get latests
git push origin master

to push a branch to remote

git push origin v1.0_fixes

list remote branches

git branch -r

To delete remote branch, push to the remote without specifying the local name

git push origin :v1.0_fixes_remote_branch_name

Complete Configuration File

Configuring

System-level configuration, for the entire PC c:\Program Files (x86)\Git\etc\gitconfig or ~\.gitconfig on MAC. Accessed by:

git config --system

User-level configuration, stored in c:\users<name>.gitconfig

git config --global

Repository level configuration, stored .git/config

git config

Show all global options

git config --global --list

Setting properties on global

git config --global user.name
git config --global user.email

Configuring core editor

git config --global core.editor "'C:/Program Files (x86)/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"

Configuring beyond compare

git config --global diff.tool bc3
git config --global difftool.bc3.path "c:/program files (x86)/beyond compare 3/bcomp.exe"
git config --global difftool.prompt false
git config --global merge.tool bc3
git config --global mergetool.bc3.path "c:/program files (x86)/beyond compare 3/bcomp.exe"
git config --global mergetool.prompt false
git config --global mergetool.keepbackup false
git config --global mergetool.keeptemporaries false

Configuring auto correct for the git commands

git config --global help.autocorrect 1

Use colors to show a lot of git information

git config --global color.ui auto

Auto carriage return line feed. False is for windows only will submit it as is. True will convert. Input is for mac, it will convert crlf into lf.

git config --global core.autocrlf false

To remove a setting

git config --unset user.name

You can also edit the config files yourself.

Working Locally

Creating a local repository

git init
git status
git add file.txt
git status
git commit

Look at the history and see the commit SHA: git log

Add all modified/updated files

get add -u

Commit with message

git commit -m "Message"

To view the diffs between SHA1 hashes

git diff dd2929..333322

Easier, latest commit is know as HEAD

git diff HEAD~1..HEAD
git diff HEAD~1..

Add all files including untracked files

git add -A

Stage as two different commits, add specific files and then commit.

git add file1.txt
git commit

Deleting Files

Delete the file first. Then stage the deletion with

git add -u

Rename the file first then:

git add -A

To clear out unstaged changed to the README.txt file

git checkout README.txt

To do a global clear out, and remove all unstaged changes git reset –hard

To take out the last commit out of the repository and move back to the staging area, that way you can change that commit around

git reset --soft HEAD~1

make changes

git commit -m "reorganized commit"

To delete last commit and discard all the changes

git reset --hard HEAD~1

Remove unstaged files

git clean -n //what would it do
git clean -f //performs the operation

.gitignore

Create a file that will list all of the files and directories that need to be omitted.

Working Remote

clone repository

git clone https://....

see list of commits

git log

commits per line

git log --oneline

see number of commits

git log --oneline | wc -1

see the graph git log –graph

Authors and commit messages

git shortlog

Statistics about authors

git shortlog -sne

Look at the specific commit

git show HEAD
git show HEAD~10
git show 939393

Look at the remotes, origin is the git default where the repository came from

git remote
git remote -v //verbose shows remote  urls

To see the repository config cat .get/config

Display all the branches in the repository

git branch

Display all the remote branches

git branch -r

Look at the tags, known stable versions

git tags

To add a remote repository, is commonly down to evaluate patches and pull requests into your project

git remote add origin https://github.com/....

Once remote repository is added you can

git fetch  //all
git fetch origin

to see changes on remote repository

get log origin/master

Merge remote branch into the your current branch

git merge origin/master

Fast forward means, replay commits that happen if there were no other changes and move the HEAD

Shortcut for git fetch; git merge origin/master is

git pull

To set your branch to mirror some other remote branch, or remote tracking branch, or upstream tracking.

git branch --set-upstream master origin/master

If you can also specify without setting the remote like this. origin is the remote name and master is the remote branch.

git pull origin master

Git commit and add modified files together

git commit -am "Some message"

Push your changes to the remote git push

To remove the origin on the current branch

git remote rm origin

to tag a current version, or stable point

git tag v1.0.0

to create tag with the annotation

git tag -a v1.0

To create a tag with the signature, signed with the password

git tag -s v1.0

To verify a tag

git tag -v v1.0

To push tags to remote (by default git would not push tags)

git push --tags

Branching Merging and Rebasing with Git

list of the commits on the current branch with a graph

git log --graph --oneline

To visualize all branches, decorate applies the branch and tag information

git lot --graph --oneline --all --decorate

Adding aliases in the git config

git config --global alias.lga "log --graph --oneline --all --decorate"
git config --global alias.lg "log --graph --all --pretty=format:'%C(bold)%h%Creset -%C(auto)%d%Creset %s %C(green dim)(%cr)%Creset %C(ul)<%an>'"
git lga
git lg
cat ~/.gitconfig

To Create a new branch git branch feature1

Checkout new branch

git checkout feature1

Big difference between branches and tags is that tags always stay on the same commit where branches move along.

git checkout master

Create a branch from a specific commit

git branch fix1 9929292
git checkout fix1

To rename a branch is to move it

git branch -m fix1 bug1234

To delete a branch

git branch -d bug1234

To force delete a branch

git branch -D bug1234

To create a branch and checkout (shortcut) git checkout -b feature2

To get back a deleted branch. From a dangling commit, a commit that doesn’t have a home. These are kept for 30 days.

git reflog // log of all references
git branch bug1234 5a78c8b //with the first 8 of the SHA

To see head

git show HEAD

Save half completed work is to stash it. Temporary holding area for the changes that are not committed to the branch.

git stash
git status
git stash list //holding area
git stash apply //to pull the changes back
git stash pop //applies it and removes it from the list

Merging

Merge fiture1 into the current branch

git merge feature1
git branch -d feature1 //deletes unused branch

Fast forward merge, is just moving the branch label into the new location.

To resolve conflicts using the merge tool

git mergetool

Compare the repository to the staging area

git diff --cached

Rebasing

You can play back your commits on a certain branch rather creating a branch if there are no conflicts. Rebase current branch on top of master

git rebase master

You can also use mergetool to solve merge conflicts and then continue

git rebase --continue

Cherry Pick

If you need to get one commit and apply it on top of a branch.

git cherrypick sks939ss

#Working with remote

git fetch origin master //get latests
git push origin master

to push a branch to remote

git push origin v1.0_fixes

list remote branches

git branch -r

To delete remote branch, push to the remote without specifying the local name

git push origin :v1.0_fixes_remote_branch_name

Complete Configuration File

Tagged with: git

The web is constantly evolving. About 7 years ago we didn’t know what jQuery was and AJAX was mostly associated with the cleaning product. Now it’s 2013 and it is no longer enough to sprinkle simple AJAX calls to get some dynamic data without refreshing the page. Modern applications run completely in the browser and provide a very fast and intuitive user interface. In addition, it creates a new user experience that makes it feel fresh and vibrant, while going back to standard web sites with postbacks feel clumsy and slow. No wonder new startups are embracing this concept. It gives you an edge and can make or break a company when dealing with competition.

We also see a rise of backend technologies that focus on returning JSON and embracing RESTful web services. One of the example is recent addition of the ASP.NET Web API framework from Microsoft. And of course a numerous amount of javascript frameworks for building SPAs. Here is a list with comparison of many of them.

New concept, need same good programming principles

Since we are moving away from simple javascript and stepping into the realm of building complex applications in the browser we must apply time tested design principles for building maintainable, testable and readable software. We want our applications to be easy to change and understand. And the user interface get changed more frequently than anything else. Therefore it’s extremely important to have good design, follow patterns and separate your concerns.

Any fool can write code that a computer can understand. Good programmers write code that humans can understand.

• Martin Fowler

Javascript libraries for building SPAs give you a nice starting point, but what I have discovered is that it leaves a lot of open questions and ability to shoot yourself in the foot when you don’t understand the concepts. There are however frameworks like EmberJs that will force you into doing things the predefined way, which would probably be a great starting point for beginners. But again it won’t give you as much flexibility of building a framework that works for you.

Foundation Ingridients

So where do we start, how do we begin this journey of re-inventing ourselves. Well what do you do when you start learning a new language? I usually pick up a good book on that language and learn the basics.

Javascript language

Since Javascript is the language of SPAs you as developer cannot longer get aways with simple functions or jQuery selectors. Just like any language other language you need to give it full credit and pick up a book and learn how to compose your objects, modules, how inheritance works and so on. Luckily there are excellent books out there. One of them that I thought was straight to the point and a must read is:

Learning the patterns

After learning the language, if you are not familiar with MV* patterns used most SPAs you will need to learn and understand them in order to know where things go. It helps that patterns have already been discussed thoroughly for other UI frameworks that are similar to SPAs like WPF and Silverlight. So there are plenty of resources and solutions that already have been hashed out. Rob Eisenberg has compiled a great list of resource on learning the presentation patterns.

Unit tests

Once you know how to separate your concerns and not to mix DOM access code with application logic with you will be able to greatly improve your code quality and maintainability by writing unit tests. You can start by going through a list of articles on testing here http://superherojs.com/

Paying your dues and evolving

Once you pay your dues and learning Javascript language along with patterns and principle and unit testing you will have a good foundation to make a maintainable application that will be easy to change, understand and will stand the test of time.

Finally on a more general note we as developers must learn to evolve. What’s awesome is that all the good programming principles that you’ve learned over the years can be applied at everything. It comes down to learning new patterns and language. And patterns knowledge would last a lot longer than frameworks or even languages. I’m excited to step on this road of learning and discovering. Are you with me?

SPAs bring new fresh user experience

The web is constantly evolving. About 7 years ago we didn’t know what jQuery was and AJAX was mostly associated with the cleaning product. Now it’s 2013 and it is no longer enough to sprinkle simple AJAX calls to get some dynamic data without refreshing the page. Modern applications run completely in the browser and provide a very fast and intuitive user interface. In addition, it creates a new user experience that makes it feel fresh and vibrant, while going back to standard web sites with postbacks feel clumsy and slow. No wonder new startups are embracing this concept. It gives you an edge and can make or break a company when dealing with competition.

We also see a rise of backend technologies that focus on returning JSON and embracing RESTful web services. One of the example is recent addition of the ASP.NET Web API framework from Microsoft. And of course a numerous amount of javascript frameworks for building SPAs. Here is a list with comparison of many of them.

New concept, need same good programming principles

Since we are moving away from simple javascript and stepping into the realm of building complex applications in the browser we must apply time tested design principles for building maintainable, testable and readable software. We want our applications to be easy to change and understand. And the user interface get changed more frequently than anything else. Therefore it’s extremely important to have good design, follow patterns and separate your concerns.

Any fool can write code that a computer can understand. Good programmers write code that humans can understand.

• Martin Fowler

Javascript libraries for building SPAs give you a nice starting point, but what I have discovered is that it leaves a lot of open questions and ability to shoot yourself in the foot when you don’t understand the concepts. There are however frameworks like EmberJs that will force you into doing things the predefined way, which would probably be a great starting point for beginners. But again it won’t give you as much flexibility of building a framework that works for you.

Foundation Ingridients

So where do we start, how do we begin this journey of re-inventing ourselves. Well what do you do when you start learning a new language? I usually pick up a good book on that language and learn the basics.

Javascript language

Since Javascript is the language of SPAs you as developer cannot longer get aways with simple functions or jQuery selectors. Just like any language other language you need to give it full credit and pick up a book and learn how to compose your objects, modules, how inheritance works and so on. Luckily there are excellent books out there. One of them that I thought was straight to the point and a must read is:

Learning the patterns

After learning the language, if you are not familiar with MV* patterns used most SPAs you will need to learn and understand them in order to know where things go. It helps that patterns have already been discussed thoroughly for other UI frameworks that are similar to SPAs like WPF and Silverlight. So there are plenty of resources and solutions that already have been hashed out. Rob Eisenberg has compiled a great list of resource on learning the presentation patterns.

Unit tests

Once you know how to separate your concerns and not to mix DOM access code with application logic with you will be able to greatly improve your code quality and maintainability by writing unit tests. You can start by going through a list of articles on testing here http://superherojs.com/

Paying your dues and evolving

Once you pay your dues and learning Javascript language along with patterns and principle and unit testing you will have a good foundation to make a maintainable application that will be easy to change, understand and will stand the test of time.

Finally on a more general note we as developers must learn to evolve. What’s awesome is that all the good programming principles that you’ve learned over the years can be applied at everything. It comes down to learning new patterns and language. And patterns knowledge would last a lot longer than frameworks or even languages. I’m excited to step on this road of learning and discovering. Are you with me?

Posted in: spa

It’s not hard and you don’t have to look far and re-use something that already works. So I encourage you to take a look at the formalized document by the cofounder of github that precisely identifies semantic versioning. Chances are next time you work on the open source project it will use the same system and you will be already familiar with it.

Enjoy.

If you are using simple incremental versioning like v39, v40 and etc., you are missing out. In software world every aspect has been thoroughly thought through, has a meaning and adds some kind of value. One of those aspects is project versioning. Many companies I worked at didn’t really give any thought on that subject and “homebrewed” some kind of simple incremental system. Most of the time that system would simply increment the version without giving any other information. A much better approach is to have versioning as a precise structure or rules to the project. Where all developers that work on the project understand and can differentiate between major, minor and patch releases.

It’s not hard and you don’t have to look far and re-use something that already works. So I encourage you to take a look at the formalized document by the cofounder of github that precisely identifies semantic versioning. Chances are next time you work on the open source project it will use the same system and you will be already familiar with it.

Enjoy.

Posted in: programming

Tagged with: programming-conecepts

In one of the applications I’m working on at the moment I wanted to only show MiniProfiler for logged in developers, so that required some changes to the default project structure. Since we want to have MiniProfiler visible on the entire site we need to make sure that membership provider gets initialized for all controllers. One way is to add InitializeSimpleMembershipAttribute to the global filters, however that would require you to hit the page twice since the action attribute fires after the Application_PostAuthorizeRequest (where we will be checking Roles) and the provider would not be initialized on the first request.

I’ve opted out to move the initialization logic from Action attribute to the Application_Start() of the Global.asax file. Since that only fires once per application I thought it would be a better place for it. So it looks like this:

public class MvcApplication : System.Web.HttpApplication
{
    protected void Application_Start()
    {
        ...
       InitMembership();
    }
    private void InitMembership()
    {
        Database.SetInitializer<UsersContext>(null);

        try
        {
            using (var context = new UsersContext())
            {
                if (!context.Database.Exists())
                {
                    // Create the SimpleMembership database without Entity Framework migration schema
                    ((IObjectContextAdapter)context).ObjectContext.CreateDatabase();
                }
            }

            WebSecurity.InitializeDatabaseConnection("DefaultConnection", "UserProfile", "UserId", "UserName", autoCreateTables: true);

            if (!Roles.RoleExists("Developer"))
            {
                Roles.CreateRole("Developer");
            }
            if (!WebSecurity.UserExists("sergey"))
            {
                WebSecurity.CreateUserAndAccount("sergey", "password");
            }
            if (!Roles.GetRolesForUser("sergey").Contains("Developer"))
            {
                Roles.AddUserToRole("sergey", "Developer");
            }
        }
        catch (Exception ex)
        {
            throw new InvalidOperationException("The ASP.NET Simple Membership database could not be initialized. For more information, please see http://go.microsoft.com/fwlink/?LinkId=256588", ex);
        }
    }
}

I left the default database creating with entity framework for now, I will most likely remove it later. The important part is that we create a new role “Developer” and assign it to our default user. Also, we can completely remove InitializeSimpleMembershipAttribute from the solution and remove it from AccountController.

In our MiniProfiler we will check if user has that role and if he or she doesn’t we will discard the results.

protected void Application_BeginRequest()
{
    MiniProfiler.Start();
}

protected void Application_PostAuthorizeRequest(object sender, EventArgs e)
{
    if (!Request.IsAuthenticated || !Roles.GetRolesForUser(User.Identity.Name).Contains("Developer"))
    {
        MiniProfiler.Stop(discardResults: true);
    }
}

protected void Application_EndRequest()
{
    MiniProfiler.Stop(); //stop as early as you can, even earlier with MvcMiniProfiler.MiniProfiler.Stop(discardResults: true);
}

Unfortunately this will run MiniProfiler for all requests, one trick that was recommended on stackoverflow is to create a watermark cookie if use is in role so the profiler would only start if that cookie is present.

Enjoy!

ASP.NET MVC 4 introduced an important change in membership providers. Moving away from core membership to a more flexible and simpler SimpleMembershipProvider. Default “Internet Application” template initializes SimpleMembership by including an action filer InitializeSimpleMembership which gets gets added to the Account controller. So whenever you try to access account controller a filer would check if the SimpleMembershipInitializer was initialized and do the necessary work if needed.

In one of the applications I’m working on at the moment I wanted to only show MiniProfiler for logged in developers, so that required some changes to the default project structure. Since we want to have MiniProfiler visible on the entire site we need to make sure that membership provider gets initialized for all controllers. One way is to add InitializeSimpleMembershipAttribute to the global filters, however that would require you to hit the page twice since the action attribute fires after the Application_PostAuthorizeRequest (where we will be checking Roles) and the provider would not be initialized on the first request.

I’ve opted out to move the initialization logic from Action attribute to the Application_Start() of the Global.asax file. Since that only fires once per application I thought it would be a better place for it. So it looks like this:

public class MvcApplication : System.Web.HttpApplication
{
    protected void Application_Start()
    {
        ...
       InitMembership();
    }
    private void InitMembership()
    {
        Database.SetInitializer<UsersContext>(null);

        try
        {
            using (var context = new UsersContext())
            {
                if (!context.Database.Exists())
                {
                    // Create the SimpleMembership database without Entity Framework migration schema
                    ((IObjectContextAdapter)context).ObjectContext.CreateDatabase();
                }
            }

            WebSecurity.InitializeDatabaseConnection("DefaultConnection", "UserProfile", "UserId", "UserName", autoCreateTables: true);

            if (!Roles.RoleExists("Developer"))
            {
                Roles.CreateRole("Developer");
            }
            if (!WebSecurity.UserExists("sergey"))
            {
                WebSecurity.CreateUserAndAccount("sergey", "password");
            }
            if (!Roles.GetRolesForUser("sergey").Contains("Developer"))
            {
                Roles.AddUserToRole("sergey", "Developer");
            }
        }
        catch (Exception ex)
        {
            throw new InvalidOperationException("The ASP.NET Simple Membership database could not be initialized. For more information, please see http://go.microsoft.com/fwlink/?LinkId=256588", ex);
        }
    }
}

I left the default database creating with entity framework for now, I will most likely remove it later. The important part is that we create a new role “Developer” and assign it to our default user. Also, we can completely remove InitializeSimpleMembershipAttribute from the solution and remove it from AccountController.

In our MiniProfiler we will check if user has that role and if he or she doesn’t we will discard the results.

protected void Application_BeginRequest()
{
    MiniProfiler.Start();
}

protected void Application_PostAuthorizeRequest(object sender, EventArgs e)
{
    if (!Request.IsAuthenticated || !Roles.GetRolesForUser(User.Identity.Name).Contains("Developer"))
    {
        MiniProfiler.Stop(discardResults: true);
    }
}

protected void Application_EndRequest()
{
    MiniProfiler.Stop(); //stop as early as you can, even earlier with MvcMiniProfiler.MiniProfiler.Stop(discardResults: true);
}

Unfortunately this will run MiniProfiler for all requests, one trick that was recommended on stackoverflow is to create a watermark cookie if use is in role so the profiler would only start if that cookie is present.

Enjoy!

Posted in: asp.net

Tagged with: asp.net-mvc-4 and miniprofiler

Here is an example of a static cache that does clean up every 1000 saves.

public class QueryCache
{
    //static field that we need to make sure is thread safe
    private static int cacheCount;
    private const int MAX_ITEMS = 1000;
    static readonly System.Collections.Concurrent.ConcurrentDictionary<string, object> _cache = new System.Collections.Concurrent.ConcurrentDictionary<string, object>();
    public static void SaveToCache(string key, object value)
    {
        if(Interlocked.Increment(ref cacheCount) == MAX_ITEMS)
        {
            CleanUpCache();
        }
        _cache[key] = value;
    }
}

Since multiple threads can be writing to the cache at the same time, we have to make sure we don’t have a collision when incrementing a cacheCount file. Interlocked class has some other useful methods like Interlock.Exchange to set the variable to a specific value atomically.

When multiple threads are writing to the static field you need to make sure that the writing operation is atomic or only one thread can write at a time. One of the ways to ensure you static fields are thread safe is to use Interlocked class part from the System.Threading namespace.

Here is an example of a static cache that does clean up every 1000 saves.

public class QueryCache
{
    //static field that we need to make sure is thread safe
    private static int cacheCount;
    private const int MAX_ITEMS = 1000;
    static readonly System.Collections.Concurrent.ConcurrentDictionary<string, object> _cache = new System.Collections.Concurrent.ConcurrentDictionary<string, object>();
    public static void SaveToCache(string key, object value)
    {
        if(Interlocked.Increment(ref cacheCount) == MAX_ITEMS)
        {
            CleanUpCache();
        }
        _cache[key] = value;
    }
}

Since multiple threads can be writing to the cache at the same time, we have to make sure we don’t have a collision when incrementing a cacheCount file. Interlocked class has some other useful methods like Interlock.Exchange to set the variable to a specific value atomically.

Posted in: asp.net

Tagged with: thread-safety

There is no Nuget package for MiniProfiler.WCF library so you have to download source code from GitHub and compile it under Release configuration. Then take MiniProfiler.dll and MiniProfiler.WCF dlls and place them in your common folder where you keep your external libraries.

Service setup

Next, in your WCF Service project register wcfMiniProfilerBehavior extensions under extensions element (you can find out the version number with right click and going to properties on the MiniProfiler.Wcf.dll file. )

    ...
        <extensions>
          <behaviorExtensions>
            <add name="wcfMiniProfilerBehavior" type="StackExchange.Profiling.Wcf.WcfMiniProfilerBehavior, Miniprofiler.Wcf, Version=2.0.4.0, Culture=neutral" />
          </behaviorExtensions>
        </extensions>
     </system.serviceModel>

And in your WCF Service project you need to add the following endPointBehavior under behaviors element.

    <behaviors>
    ...
          <endpointBehaviors>
            <behavior>
              <wcfMiniProfilerBehavior />
            </behavior>
          </endpointBehaviors>
    </behaviors>

Oh yea and don’t forget to reference a MiniProfiler.Wcf in your Service Project. You will need to register MiniProfiler assembly in other projects where the actual profiling is going to take place, for example your data layer would have a reference to that file and wrap a connection with a dbprofiled connection, for more setup details see the official site.

Client Setup

If you are using proxies for your client setup you would register the following configuration in your client’s web.config.

      <system.serviceModel>
        <extensions>
          <behaviorExtensions>
            <add name="wcfMiniProfilerBehavior" type="StackExchange.Profiling.Wcf.WcfMiniProfilerBehavior, Miniprofiler.Wcf, Version=2.0.4.0, Culture=neutral" />
          </behaviorExtensions>

And behaviors

    <behaviors>
      <endpointBehaviors>
        <behavior>
          <webHttp/>
          <wcfMiniProfilerBehavior />
        </behavior>
      </endpointBehaviors>
    </behaviors>

Just like in the Sample Project.

If you are using ChannelFactory without generating proxies you would need to add a behavior in the following way, for example:

ChannelFactory = new ChannelFactory<T>(Binding, Endpoint);
ChannelFactory.Endpoint.Behaviors.Add(new WcfMiniProfilerBehavior());
Channel = ChannelFactory.CreateChannel();

That’s almost it, just add the standard initialization to Application_BeginRequest and etc. mine looks like this

protected void Application_BeginRequest(Object sender, EventArgs e)
{
    if (Request.IsLocal)
    {
        MiniProfiler.Start();
    }
}

protected void Application_EndRequest(Object sender, EventArgs e)
{
    MiniProfiler.Stop();
}

private void InitProfilerSettings()
{
    // some things should never be seen
    var ignored = MiniProfiler.Settings.IgnoredPaths.ToList();
    ignored.Add("WebResource.axd");
    ignored.Add("/Styles/");
    MiniProfiler.Settings.IgnoredPaths = ignored.ToArray();

    MiniProfiler.Settings.SqlFormatter = new StackExchange.Profiling.SqlFormatters.SqlServerFormatter();
}

Also, need to add handler activation into web.config under system.webserver

    <system.webServer>
    ...
    <handlers>
    <add name="MiniProfiler" path="mini-profiler-resources/*" verb="*"
    type="System.Web.Routing.UrlRoutingModule" resourceType="Unspecified" preCondition="integratedMode" />
    </handlers>

Finally, adding Mini Profile rendering scripts in your master page.

    <%= StackExchange.Profiling.MiniProfiler.RenderIncludes() %>

First of all MiniProfiler is awesome! And lucky for me some smart people made it work with WCF. Since there is no official guide on setting up the project I had to get the examples and work from there. Here is what I had to do in order to get it working.

There is no Nuget package for MiniProfiler.WCF library so you have to download source code from GitHub and compile it under Release configuration. Then take MiniProfiler.dll and MiniProfiler.WCF dlls and place them in your common folder where you keep your external libraries.

Service setup

Next, in your WCF Service project register wcfMiniProfilerBehavior extensions under extensions element (you can find out the version number with right click and going to properties on the MiniProfiler.Wcf.dll file. )

    ...
        <extensions>
          <behaviorExtensions>
            <add name="wcfMiniProfilerBehavior" type="StackExchange.Profiling.Wcf.WcfMiniProfilerBehavior, Miniprofiler.Wcf, Version=2.0.4.0, Culture=neutral" />
          </behaviorExtensions>
        </extensions>
     </system.serviceModel>

And in your WCF Service project you need to add the following endPointBehavior under behaviors element.

    <behaviors>
    ...
          <endpointBehaviors>
            <behavior>
              <wcfMiniProfilerBehavior />
            </behavior>
          </endpointBehaviors>
    </behaviors>

Oh yea and don’t forget to reference a MiniProfiler.Wcf in your Service Project. You will need to register MiniProfiler assembly in other projects where the actual profiling is going to take place, for example your data layer would have a reference to that file and wrap a connection with a dbprofiled connection, for more setup details see the official site.

Client Setup

If you are using proxies for your client setup you would register the following configuration in your client’s web.config.

      <system.serviceModel>
        <extensions>
          <behaviorExtensions>
            <add name="wcfMiniProfilerBehavior" type="StackExchange.Profiling.Wcf.WcfMiniProfilerBehavior, Miniprofiler.Wcf, Version=2.0.4.0, Culture=neutral" />
          </behaviorExtensions>

And behaviors

    <behaviors>
      <endpointBehaviors>
        <behavior>
          <webHttp/>
          <wcfMiniProfilerBehavior />
        </behavior>
      </endpointBehaviors>
    </behaviors>

Just like in the Sample Project.

If you are using ChannelFactory without generating proxies you would need to add a behavior in the following way, for example:

ChannelFactory = new ChannelFactory<T>(Binding, Endpoint);
ChannelFactory.Endpoint.Behaviors.Add(new WcfMiniProfilerBehavior());
Channel = ChannelFactory.CreateChannel();

That’s almost it, just add the standard initialization to Application_BeginRequest and etc. mine looks like this

protected void Application_BeginRequest(Object sender, EventArgs e)
{
    if (Request.IsLocal)
    {
        MiniProfiler.Start();
    }
}

protected void Application_EndRequest(Object sender, EventArgs e)
{
    MiniProfiler.Stop();
}

private void InitProfilerSettings()
{
    // some things should never be seen
    var ignored = MiniProfiler.Settings.IgnoredPaths.ToList();
    ignored.Add("WebResource.axd");
    ignored.Add("/Styles/");
    MiniProfiler.Settings.IgnoredPaths = ignored.ToArray();

    MiniProfiler.Settings.SqlFormatter = new StackExchange.Profiling.SqlFormatters.SqlServerFormatter();
}

Also, need to add handler activation into web.config under system.webserver

    <system.webServer>
    ...
    <handlers>
    <add name="MiniProfiler" path="mini-profiler-resources/*" verb="*"
    type="System.Web.Routing.UrlRoutingModule" resourceType="Unspecified" preCondition="integratedMode" />
    </handlers>

Finally, adding Mini Profile rendering scripts in your master page.

    <%= StackExchange.Profiling.MiniProfiler.RenderIncludes() %>

Tagged with: asp.net-wcf and miniprofiler

The following example will work with SqlCommand but would not work with the DbCommand because DbCommand.Parameters doesn’t expose a Value property which is a short hand notation that returns the last SqlParameter Value.

    using(var cn = SqlTools.GetSqlConnection("Default"))
    {
        var cmd = cn.CreateCommand();
        cmd.Parameters.Add(new SqlParameter("@Password", SqlDbType.NVarChar, 128)).Value = password;

So I needed to modify this code in about 650 places to use object initializer like this:

    cmd.Parameters.Add(new SqlParameter("@UserName", SqlDbType.NVarChar, 256) { Value = username });

With a little bit of research I wrote a regular expression for Visial Studio Replace to fix this issue. Which looks like this:

Find what: new SqlParameter\({[^)]*}\)\)\.{[A-Za-z]+} = {[^\}]+};

Repleace with what: new SqlParameter(\1) { \2 = \3 });

And vua-lah!, it is all fixed.

One important feature to notice is that you use curly braces to mark the values that you are extracting from the string, this is different from other flavors where you use parenthesis.

Update 04-23-2013

Visual Studio 2012 is using a different syntax to capture strings. The above expression would have to be slightly modified, as follows:

Find what: new SqlParameter\(([^)]*)\)\)\.([A-Za-z]+) = ([^\}]+);

Repleace with what: new SqlParameter($1) { $2 = $3 });

Regular expressions are very handy especially when you are refactoring and there is a lot of manual changes. Visual Studio Replace with regex came in really handy when porting our code base at work to use MiniProfiler. Since MiniProfiler returns a DbConnection rather than SqlConnection, there are few small shorthands that no longer worked with DbConnection.

The following example will work with SqlCommand but would not work with the DbCommand because DbCommand.Parameters doesn’t expose a Value property which is a short hand notation that returns the last SqlParameter Value.

    using(var cn = SqlTools.GetSqlConnection("Default"))
    {
        var cmd = cn.CreateCommand();
        cmd.Parameters.Add(new SqlParameter("@Password", SqlDbType.NVarChar, 128)).Value = password;

So I needed to modify this code in about 650 places to use object initializer like this:

    cmd.Parameters.Add(new SqlParameter("@UserName", SqlDbType.NVarChar, 256) { Value = username });

With a little bit of research I wrote a regular expression for Visial Studio Replace to fix this issue. Which looks like this:

Find what: new SqlParameter\({[^)]*}\)\)\.{[A-Za-z]+} = {[^\}]+};

Repleace with what: new SqlParameter(\1) { \2 = \3 });

And vua-lah!, it is all fixed.

One important feature to notice is that you use curly braces to mark the values that you are extracting from the string, this is different from other flavors where you use parenthesis.

Update 04-23-2013

Visual Studio 2012 is using a different syntax to capture strings. The above expression would have to be slightly modified, as follows:

Find what: new SqlParameter\(([^)]*)\)\)\.([A-Za-z]+) = ([^\}]+);

Repleace with what: new SqlParameter($1) { $2 = $3 });

Posted in: asp.net

Tagged with: regex

When I first started my programming career I have to admit I was pretty mediocre. Even though I could type about 45 words per minute (WPM) I was still typing with 3 fingers and looking at the keyboard. In the beginning typing uncommon characters made me feel awkward and frustrated and my WMP went out the window. I think my real WMP while writing code was about 10-13. I think i really decided to take the bull by the horns when I realized that it actually interfered with my coding. I’m not good at multitasking and when I write my code I need to keep in the flow of one task and that thinking about task at hand. Looking at the keyboard while typing will slow you down and slow down the flow of your thoughts and eventually your brain will get frustrated and get distracted.

It’s pretty amazing how much such a basic skill of touch typing can improve the quality of your career. I can tell that it made my job a lot more pleasurable. I actually look forward to writing code since it doesn’t require much effort. Also, I can tell that I’m relying on IDE much less and it just feels great.

For me working doing 15 minutes of practice every day worked magic, but you gotta stick with it for at least for couple weeks. Also don’t cheat yourself and making sure every finger is doing it’s proper job. You will actually see results pretty fast.

So I challenge you to take 15 minutes out of your day and head over to Typing for developers and stick to taking lessons for about 2 weeks.

That’s it and enjoy!

It’s sad to see many professional programmers typing with two or three fingers. Like in any profession you should become a master of your craft, and for us developers touch typing is a basic skill that you must master. Like a chef who has never learned how to properly prepare vegetables it’s not efficient, makes your already difficult profession harder and just not professional.

When I first started my programming career I have to admit I was pretty mediocre. Even though I could type about 45 words per minute (WPM) I was still typing with 3 fingers and looking at the keyboard. In the beginning typing uncommon characters made me feel awkward and frustrated and my WMP went out the window. I think my real WMP while writing code was about 10-13. I think i really decided to take the bull by the horns when I realized that it actually interfered with my coding. I’m not good at multitasking and when I write my code I need to keep in the flow of one task and that thinking about task at hand. Looking at the keyboard while typing will slow you down and slow down the flow of your thoughts and eventually your brain will get frustrated and get distracted.

It’s pretty amazing how much such a basic skill of touch typing can improve the quality of your career. I can tell that it made my job a lot more pleasurable. I actually look forward to writing code since it doesn’t require much effort. Also, I can tell that I’m relying on IDE much less and it just feels great.

For me working doing 15 minutes of practice every day worked magic, but you gotta stick with it for at least for couple weeks. Also don’t cheat yourself and making sure every finger is doing it’s proper job. You will actually see results pretty fast.

So I challenge you to take 15 minutes out of your day and head over to Typing for developers and stick to taking lessons for about 2 weeks.

That’s it and enjoy!

Posted in: better-programmer

Tagged with: better-programmer

• using a typed accessor SqlDataReader.GetInt32(int columnOrdinal)
• geting a value directly with named column like this SqlDataReader[“columnName”] or with column ordinal SqlDataReader[0] and then casting it
• call SqlDataReader.GetOrdinal(“columnName”) to get column position and then combine the result and call the typed accessor method like SqlDataReader.GetInt32(SqlDataReader.GetOrdinal(“columnName”)).

For performance benefits using Typed Accessors like reader.GetInt32 or reader.GetDateTime is always faster since there is no type conversion or unboxing occurs. But these typed accessors only accept column ordinal or column position and most of the times you don’t want to hard code your column position since that could change at any time and break your application.

I wrote a series of tests that test each scenario in the context of:

1. Getting one record 100000 times, that emulates retrieving of one row.
2. Getting 20k records 1000 times, or simulating retrieving large amount of rows

The results are as follows:

Getting one row over 100k times:

There is really no difference or any significant performance advantage when accessing one row. When you run the test multiple times numbers change slightly and there is no clear winner. So we can say if you are only getting one record you can use any one of those accessors and cast as you please.

Getting 20k records over 1000 times

Typed accessor proves to be the fastest. Right behind is casting by using column orderinal accessor. Casting with column name is probably 30% slower than casting typed casting. And finally if we call reader.GetOrdinal once and cache the column position we get as fast performance as typed accessor alone.

Summary

If your query returns only one row or small amount of rows it doesn’t matter which method you use. It would only make sense if you have large number of rows and using reader.GetOrdinal to cache column number and then using typed accessors would yield the best performance.

If you want to see for yourself you can download the source code and run it against the AdventureWorks database.

Coming back to ADO.NET, I sometimes forget of different ways of accessing values from the SqlDataReader, cost of a type unboxing and which method is to use when. For example, here are different ways we can access values:

• using a typed accessor SqlDataReader.GetInt32(int columnOrdinal)
• geting a value directly with named column like this SqlDataReader[“columnName”] or with column ordinal SqlDataReader[0] and then casting it
• call SqlDataReader.GetOrdinal(“columnName”) to get column position and then combine the result and call the typed accessor method like SqlDataReader.GetInt32(SqlDataReader.GetOrdinal(“columnName”)).

For performance benefits using Typed Accessors like reader.GetInt32 or reader.GetDateTime is always faster since there is no type conversion or unboxing occurs. But these typed accessors only accept column ordinal or column position and most of the times you don’t want to hard code your column position since that could change at any time and break your application.

I wrote a series of tests that test each scenario in the context of:

1. Getting one record 100000 times, that emulates retrieving of one row.
2. Getting 20k records 1000 times, or simulating retrieving large amount of rows

The results are as follows:

Getting one row over 100k times:

There is really no difference or any significant performance advantage when accessing one row. When you run the test multiple times numbers change slightly and there is no clear winner. So we can say if you are only getting one record you can use any one of those accessors and cast as you please.

Getting 20k records over 1000 times

Typed accessor proves to be the fastest. Right behind is casting by using column orderinal accessor. Casting with column name is probably 30% slower than casting typed casting. And finally if we call reader.GetOrdinal once and cache the column position we get as fast performance as typed accessor alone.

Summary

If your query returns only one row or small amount of rows it doesn’t matter which method you use. It would only make sense if you have large number of rows and using reader.GetOrdinal to cache column number and then using typed accessors would yield the best performance.

If you want to see for yourself you can download the source code and run it against the AdventureWorks database.

Posted in: ado.net

Tagged with: ado.net and sqldatareader

It was very easy to do in FunnelWeb since it embraces ASP.NET MVC. All I had to do is to create an extra property in the PageModel model and annotate it with regular expression validation that looks for the key answer to the question I’ve provided. (?i) in the regular expression annotation means case insensitive comparison.

   [Required]
   [StringLength(50)]
   [DisplayName("Spam check")]
   [HintSize((HintSize.Medium))]
   [Description("What colour is grass?")]
   [RegularExpression("(?i)^green$", ErrorMessage = "Please confirm that you are human and enter a correct answer. What colour is grass?")]
   public string SpamCheck { get; set; }

We don’t add any extra code in our controllers since it already checks if model is valid and if not it would return back with validation error messages. One more thing we need to do is to modify our view to display the spam check answer. We customize our views in the FunnelWeb by coping them into our themes folder and then making changes there keeping originals intact. So I’ve copied _EditComments.cshtml from Views/Shared folder and placed under my Theme folder in the Views/Shared folder and added the following to display the spam check.

<div class="editor-label">
  @Html.LabelFor(m => m.SpamCheck)
</div>
<div class="editor-field">
  @Html.EditorFor(m => m.SpamCheck, Html.AttributesFor(m => m.SpamCheck))
  @Html.ValidationMessageFor(m => m.SpamCheck) @Html.HintFor(m => m.SpamCheck)
</div>

And that’s it, with minimal modification to our blogging platform we eliminated all those annoying spam bots posting garbage.

Update 11-26-12

Since putting up that spam filter I haven’t gotten a single spam comment in last 24 hours. Nice, no more need to administer your comments daily, one less thing to worry about!

I’ve been getting a ridiculous amount of spam on my blog and even though FunnelWeb checks with akismet, a lot of spam messages are still getting through. Most spam comments are placed by bots that look for input fields and then submit garbage in hopes that it somehow becomes a link. We can make it a little harder for it by including a simple question that humans can easily answer, which should reduce amount of spam.

It was very easy to do in FunnelWeb since it embraces ASP.NET MVC. All I had to do is to create an extra property in the PageModel model and annotate it with regular expression validation that looks for the key answer to the question I’ve provided. (?i) in the regular expression annotation means case insensitive comparison.

   [Required]
   [StringLength(50)]
   [DisplayName("Spam check")]
   [HintSize((HintSize.Medium))]
   [Description("What colour is grass?")]
   [RegularExpression("(?i)^green$", ErrorMessage = "Please confirm that you are human and enter a correct answer. What colour is grass?")]
   public string SpamCheck { get; set; }

We don’t add any extra code in our controllers since it already checks if model is valid and if not it would return back with validation error messages. One more thing we need to do is to modify our view to display the spam check answer. We customize our views in the FunnelWeb by coping them into our themes folder and then making changes there keeping originals intact. So I’ve copied _EditComments.cshtml from Views/Shared folder and placed under my Theme folder in the Views/Shared folder and added the following to display the spam check.

<div class="editor-label">
  @Html.LabelFor(m => m.SpamCheck)
</div>
<div class="editor-field">
  @Html.EditorFor(m => m.SpamCheck, Html.AttributesFor(m => m.SpamCheck))
  @Html.ValidationMessageFor(m => m.SpamCheck) @Html.HintFor(m => m.SpamCheck)
</div>

And that’s it, with minimal modification to our blogging platform we eliminated all those annoying spam bots posting garbage.

Update 11-26-12

Since putting up that spam filter I haven’t gotten a single spam comment in last 24 hours. Nice, no more need to administer your comments daily, one less thing to worry about!

Posted in: funnelweb

Tagged with: funnelweb and spam

This time however I’ve decided to move out my music library from home server to the media server and was stuck since all my music library tracks were mapped to the UNC \hpserver and new location of my music would be something like M:\Music. Unfortunately you can’t do that with iTunes and you are stuck with going one by one and relocation your track.

Luckily few days ago I’ve stumbled across Scott Hanselman’s post on how to remove dead tracks from iTunes. After looking at the code it was pretty easy to add functionality to relocate tracks automatically and I’ve modified the sample project he provided to do just that. Thanks to him for bringing iTunes api to light!

Now I still have my library with play counts and all the data just like it was before but under a new location!

Here is the link to the modified source code if you ever want to move your itunes library to a new location.

iTunes is my least favorite software when it comes to managing music library. However sometimes you have no choice and have to deal with it. Couple times in the past I had to completely recreate my iTunes library from scratch since I could not move it to the newly installed system. Recently when I’ve build a new PC I was a bit smarter and did the research and mapped my library to the exact same folder and everything worked and everything transferred perfect.

This time however I’ve decided to move out my music library from home server to the media server and was stuck since all my music library tracks were mapped to the UNC \hpserver and new location of my music would be something like M:\Music. Unfortunately you can’t do that with iTunes and you are stuck with going one by one and relocation your track.

Luckily few days ago I’ve stumbled across Scott Hanselman’s post on how to remove dead tracks from iTunes. After looking at the code it was pretty easy to add functionality to relocate tracks automatically and I’ve modified the sample project he provided to do just that. Thanks to him for bringing iTunes api to light!

Now I still have my library with play counts and all the data just like it was before but under a new location!

Here is the link to the modified source code if you ever want to move your itunes library to a new location.

Posted in: itunes

Tagged with: itunes

SQLite is one of the little databases that you can use in your project when you don’t want to have a full blown database and want something simple, quick and awesome. It’s very easy to distribute or share with your team since it’s just a single file and can be checked into source control. Could be used as a development database, one off database for your desktop applications or just about anything else. It’s definitely nice to keep in your development tool belt.

Installation

There is none, just add reference to SQLite C# drivers and it will create a database for your on first run. Simplest way is to go to Nuget package manager console and run:

Install-Package System.Data.SQLite

This will install both versions for x86 and x64 so you can build it for any CPU. There are also specific versions as well in case if you don’t want extra files Install-Package System.Data.SQLite.x86 or System.Data.SQLite.x64.

Model

In our sample we’ll save and retrieve customers from the database so we’ll create a folder “Model” and add a Customer class which will represent our data model.

namespace SQLiteDemo.Model
{
    public class Customer
    {
        public int Id { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public DateTime DateOfBirth { get; set; }
    }
}

Simple Data Layer

For this demo I’ll use a console application and for simplicity sake I’ll create a folder “Data” for data access layer. In you production application you will most likely want to put your data access layer into a separate project.

We’ll also use a repository pattern for our Data Access Layer(DAL) since it’s the most commonly used and easily to understand.

We’ll add an interface ICustomerRepository which will have two methods to save and retrive customers and create a base class SqLiteBaseRepository which will handle common logic between our repositories like creating database connections.

public interface ICustomerRepository
{
    Customer GetCustomer(long id);
    void SaveCustomer(Customer customer);
}

public class SqLiteBaseRepository
{
    public static string DbFile
    {
        get { return Environment.CurrentDirectory + "\\SimpleDb.sqlite"; }
    }

    public static SQLiteConnection SimpleDbConnection()
    {
        return new SQLiteConnection("Data Source=" + DbFile);
    }
}

Using Dapper as our simple Object-relational mapping (ORM)

Dapper is easy to use, fast and fits well for our simple data access layer. It has a simple API on top of already familiar SQL statements.

In your package manager console

Install-Package Dapper

In Customer Repository implementation (SqlLiteCustomerRepository) “SaveCustomer” method will first check if database file already exists and execute a create table script which if nothing is found, it will output a file to the environment runtime directory where your executable is located.

public void SaveCustomer(Customer customer)
{
    if (!File.Exists(DbFile))
    {
        CreateDatabase();
    }

    using (var cnn = SimpleDbConnection())
    {
        cnn.Open();
        customer.Id = cnn.Query<long>(
            @"INSERT INTO Customer
            ( FirstName, LastName, DateOfBirth ) VALUES
            ( @FirstName, @LastName, @DateOfBirth );
            select last_insert_rowid()", customer).First();
    }
}

private static void CreateDatabase()
{
    using (var cnn = SimpleDbConnection())
    {
        cnn.Open();
        cnn.Execute(
            @"create table Customer
                (
                    ID                                  integer identity primary key AUTOINCREMENT,
                    FirstName                           varchar(100) not null,
                    LastName                            varchar(100) not null,
                    DateOfBirth                         datetime not null
                )");
    }
}

Above Dapper maps our object to table field names in the query and also returns a long datatype with a newly inserted scoped identity. With SqLite scope identity has a different method last_insert_rowid() and we use that to get inserted customer id and assign it to our customer.

To retrieve customer we first check if database file exists first and then let dapper handle the rest.

public Customer GetCustomer(long id)
{
    if (!File.Exists(DbFile)) return null;

    using (var cnn = SimpleDbConnection())
    {
        cnn.Open();
        Customer result = cnn.Query<Customer>(
            @"SELECT Id, FirstName, LastName, DateOfBirth
            FROM Customer
            WHERE Id = @id", new { id }).FirstOrDefault();
        return result;
    }
}

And there you have it, an easy way to have a single file database with a simple ORM in your repository layer.

Source code

Here is the link to the source code if you would like to run the solution and try it for yourself. Source Code

Summary

SQLite is one of the little databases that you can use in your project when you don’t want to have a full blown database and want something simple, quick and awesome. It’s very easy to distribute or share with your team since it’s just a single file and can be checked into source control. Could be used as a development database, one off database for your desktop applications or just about anything else. It’s definitely nice to keep in your development tool belt.

Installation

There is none, just add reference to SQLite C# drivers and it will create a database for your on first run. Simplest way is to go to Nuget package manager console and run:

Install-Package System.Data.SQLite

This will install both versions for x86 and x64 so you can build it for any CPU. There are also specific versions as well in case if you don’t want extra files Install-Package System.Data.SQLite.x86 or System.Data.SQLite.x64.

Model

In our sample we’ll save and retrieve customers from the database so we’ll create a folder “Model” and add a Customer class which will represent our data model.

namespace SQLiteDemo.Model
{
    public class Customer
    {
        public int Id { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public DateTime DateOfBirth { get; set; }
    }
}

Simple Data Layer

For this demo I’ll use a console application and for simplicity sake I’ll create a folder “Data” for data access layer. In you production application you will most likely want to put your data access layer into a separate project.

We’ll also use a repository pattern for our Data Access Layer(DAL) since it’s the most commonly used and easily to understand.

We’ll add an interface ICustomerRepository which will have two methods to save and retrive customers and create a base class SqLiteBaseRepository which will handle common logic between our repositories like creating database connections.

public interface ICustomerRepository
{
    Customer GetCustomer(long id);
    void SaveCustomer(Customer customer);
}

public class SqLiteBaseRepository
{
    public static string DbFile
    {
        get { return Environment.CurrentDirectory + "\\SimpleDb.sqlite"; }
    }

    public static SQLiteConnection SimpleDbConnection()
    {
        return new SQLiteConnection("Data Source=" + DbFile);
    }
}

Using Dapper as our simple Object-relational mapping (ORM)

Dapper is easy to use, fast and fits well for our simple data access layer. It has a simple API on top of already familiar SQL statements.

In your package manager console

Install-Package Dapper

In Customer Repository implementation (SqlLiteCustomerRepository) “SaveCustomer” method will first check if database file already exists and execute a create table script which if nothing is found, it will output a file to the environment runtime directory where your executable is located.

public void SaveCustomer(Customer customer)
{
    if (!File.Exists(DbFile))
    {
        CreateDatabase();
    }

    using (var cnn = SimpleDbConnection())
    {
        cnn.Open();
        customer.Id = cnn.Query<long>(
            @"INSERT INTO Customer
            ( FirstName, LastName, DateOfBirth ) VALUES
            ( @FirstName, @LastName, @DateOfBirth );
            select last_insert_rowid()", customer).First();
    }
}

private static void CreateDatabase()
{
    using (var cnn = SimpleDbConnection())
    {
        cnn.Open();
        cnn.Execute(
            @"create table Customer
                (
                    ID                                  integer identity primary key AUTOINCREMENT,
                    FirstName                           varchar(100) not null,
                    LastName                            varchar(100) not null,
                    DateOfBirth                         datetime not null
                )");
    }
}