Morteza Sahragard

My input on the audiobook Fundamentals of Software Architecture

Wed, 02 Oct 2024 08:52:00 GMT

Fundamentals of Software Architecture: An Engineering Approach is a book written by Mark Richards and Neal Ford. Both authors have a longtime track and history in software development and architecture. The paper version of the book contains 419 pages and audiobook one, about 13 hours. The audiobook has been crafted to cover all the topics without the need of having visuals and photos.

You can find the table of contents of the book here. Here is the list of chapters:

Chapter 1: Introduction

Chapter 2: Architectural Thinking

Chapter 3: Modularity

Chapter 4: Architecture Characteristics Defined

Chapter 5: Identifying Architecture Characteristics

Chapter 6: Measuring and Governing Architecture Characteristics

Chapter 7: Scope of Architecture Characteristics

Chapter 8: Component-Based Thinking

Chapter 9: Architecture Styles

Chapter 10: Layered Architecture Style

Chapter 11: Pipeline Architecture

Chapter 12: Microkernel Architecture

Chapter 13: Service-Based Architecture

Chapter 14: Event-Driven Architecture Style

Chapter 15: Space-Based Architecture

Chapter 16: Orchestration-Driven Service-Oriented Architecture

Chapter 17: Microservices Architecture

Chapter 18: Choosing the Appropriate Architecture Style

Chapter 19: Architecture Decisions

Chapter 20: Analyzing Architecture Risk

Chapter 21: Diagramming and Presenting Architecture

Chapter 22: Making Teams Effective

Chapter 23: Negotiation and Leadership Skills

Chapter 24: Developing a Career Path

The authors have listed different Architecture Styles from the very old ones to the new ones which have emerged in the past few years. Specific characteristics of each architectural style have been listed and pros, cons, strengths and weaknesses of each have been discussed too. At the end of each chapter, they rate each aspect of architectural style in terms of deplorability, testability, scalability, reliability and so on, and then reason about their rating. That’s of course opinionated but still so valuable to know and compare different ideas. I found listening to this part of the book so valuable because after being in software development industry for over two decades, I am no longer dealing with all those architectures that I have done in the past. It was so valuable to have an abstract and compact of important characteristics of each architecture to freshen the mind.

The book touches on other topics as well, like diagramming tools like C4 and ArchiMate, Architecture Decision Records (ADR), Negotiation Skills, Time Management, Fallacies of distributed computing etc.

Narration is quite good. The contents of the audiobook are mostly quite straight forward but I have to state that, I wonder if it will be quite helpful for new learners and beginners. If you are one on them, please leave me a comment and talk about your experience.

My input on Designing Data Intensive Applications audiobook

Sun, 19 May 2024 10:34:00 GMT

Designing Data-Intensive Applications is a book written by Martin Kleppmann. The paper version consists of 590 pages, and the audiobook is about 21 hours! So, get ready for a marathon if you choose the audio version!

This book is truly unique. The author has extensive knowledge in the fields of databases and distributed system design. Typically, IT professionals learn about the functionalities of various databases and the problems they address. This book, however, goes beyond that—it delves into the complexities tackled by different databases behind the scenes.

Kleppmann is well-versed in many popular databases, including MongoDB, Cassandra, Microsoft SQL Server, PostgreSQL, and others. He explains real-world issues and discusses existing solutions, detailing the pros and cons of each. He also explains which database systems implement particular approaches.

The content of the book is very comprehensive. It covers different algorithms for finding data on disk, their pros and cons, data replication, master-slave nodes, various real-world implementations, data indexing, cluster management, locks, data consistency and many other topics.

If you have been working on serious distributed applications, you will deeply appreciate and connect with the content. It's fascinating to understand how the things you work on daily or have worked on in the past are actually executed behind the scene.

The narration and structure of the book are excellent. However, I must admit that I struggled to understand some parts, as they were so deep and beyond my database knowledge.

Who is the target audience for this book?

In my opinion, this book is a must-read for database administrators and engineers whose main focus is on databases. Additionally, those working on distributed applications will find this book highly enjoyable and informative.

My input on the Audiobook, Clean Architecture by Robert C. Martin

Tue, 25 Apr 2023 18:57:00 GMT

I have to confess that since the past few years, video and audio trainings have become my main source of learning. Pluralsight, A Could Guru, Udemy, Coursera, YouTube, you name it!

I have been following Robert C. Martin (Uncle Bob) in Twitter for long, but never got a chance to read any of his books. While ago, finally and doubtfully I bought the Audiobook version of his book Clean Architecture. The book subject is very close to the code and implementation, and I was wondering how practical it would be in audio version. Without photo, without code!

Here I am writing my input over the audiobook. I gotta state that I have not read the paper copy of the book and therefore, can’t compare them.

I am not going to talk about the chapters and lessons. You can quickly look into the Table of Contents in the internet. Here is my two cents over the audiobook overall.

1. The audiobook is very well narrated.

2. Being in software development industry for about two decades now, makes it difficult for me to say which level of software development experience or knowledge is needed to be able to get the most out of this book. But one thing for sure, this book (as most other architecture related books) is not meant for juniors or newbies. In other words, certain amount of knowledge and experience is required to be able to easily follow and understand the topics just by listening!

3. This is not an audiobook that you listen while in car, or running on treadmills! Focus is needed to follow the conetnts since narrator frequently refers to the code, and describes the structure, constructors, inputs, and outputs etc.

4. Robert C. Martin has his own way of teaching style, and he is a person who knows what he is talking about. That makes the whole journey of listening the audiobook fun.

Do I recommend this book?

Depends! Assuming you have already checked the Table of Contents of the book in the internet, if you are interested to refresh your mind about the mentioned topics, or want to listen to the audiobook out of fun, the I would recommend it.

If you are already confident with most of the topics in the book, or recently read a similar book, or you are looking to learn things that you think can’t be found in the internet or in other book, then I would not recommend it because it’s not going to satisfy you.

I am now listening to his Clean Agile audiobook and will write my feedback over it soon too.

Put a limit on consumption based azure functions scaling

Sun, 13 Dec 2020 12:19:00 GMT

Needless to say, one of the coolest features and biggest advantage of consumption based azure functions is the fact that how they scale out without we having to do anything about it. As of writing this post, consumption based functions can scale out up to 200 instances and premium ones up to 100. Apparently, if you need even more instances, that's also possible but you need to contact azure support team.

So far so good but in real world scenarios we need to control scaling of instances. More often than not, azure functions deal with databases or other services which either can't scale as much or financially is not cost effective to do so. Over-scaling azure functions could put other resources under pressure and ultimately exceed the threshold and bring everything down.

During the past few years, Azure team introduced several workarounds to limit the scaling which was mostly adding some settings to the configuration, but truth be told, they never worked well for me and some others I know.

One good old technic to control the load is to put the requests in a Queue and then start processing them from there. It's possible to limit the number of items which could be picked up from a Queue in parallel and that kind of lets us control the processing load. This is a good proven technic but not applicable everywhere and sometimes is a bit of overhead.

Now I will tell you a true story that occurred to me while ago. An external client suddenly pushed (they were not supposed to do) massive amount of data to our EventHub. That caused azure functions listening to the EventHub scale out dramatically and they all put requests into a Queue. Since the consumption of the data from the Queue was much slower than enqueuing items, the Queue after some time ran out of capacity and rejected receiving new items. That caused us a bit of work to recover the possible lost data.

Fortunately, recently a new feature was added to azure functions to tackle this issue and it works quite well. IF you open an azure function page in the portal, you will find Scale out menu item in the left. There is possible to put a limit oon maximum number of instances if you need.

Of course, it could also be done via Azure Cli

az resource update --resource-type Microsoft.Web/sites -g <resource_group> -n <function_app_name>/config/web --set properties.functionAppScaleLimit=<scale_limit>

More info here.

Last word

While architecting Serverless applications it's wise to put a cap limit on all possible resources because of Murphy's law. Anything that can go wrong will go wrong!

Disaster recovery in case of Database/Container deletion and data corruption in Azure Cosmos DB

Wed, 16 Sep 2020 13:09:00 GMT

No matter what our strategy is to protect our resources and avoid accidental deletion, we should always be prepared for crisis management and know which steps to follow in case a disaster occurs. When it comes to disaster recovery, databases come on top of the list. Here we are going to discuss Azure Cosmos DB.

Before we go any further, here is the reference page in azure documentation.

https://docs.microsoft.com/en-us/azure/cosmos-db/online-backup-and-restore

First of all, what would you do if you accidentally delete a Container?!

According to the documentation "Azure Cosmos DB automatically takes a backup of your data for every 4 hours and at any point of time, the latest two backups are stored". That means at the time of deletion in worst case there are two backups available, one for about 4 hours ago and the other for 8 hour ago.

These settings could be changed in the portal as you see here:

Even if we set the backup settings here but backups are not directly available to us. To restore a backup we need to open a support ticket and azure support to do so. Pay attention that backups are made at account level.

Another fact to be aware of is that "If the container or database is deleted, Azure Cosmos DB retains the existing snapshots of a given container or database for 30 days".

What should we do if we accidentally deleted a Database or Container in Azure Cosmos DB?

1. The first thing to do is to go to the portal and increase Retention time to preferably 7 days! That gives azure support enough time to react and restore the backup before it's overridden with new backups! According to the documentation this action "It’s best to increase your retention within 8 hours of this event". I do not exactly know what "It’s best" means in this context and what will happen if it goes beyond 8 hours but anyway better to do this immediately.

A question may arise here is that what if we notice the wrong deletion of the Database/Container sometime later?! For instance, we delete the container on Friday and notice the mistake on Monday? My first thought would be that we should adjust number of backups and retention time based on our business needs. If the data is critical and chance of corruption is high, better to create backups more frequently and retain them longer.

2. Second actin is to open a support ticket and ask them to restore the database.

What should we do if we accidentally deleted an Azure Cosmos DB Account?

Microsoft recommends we should NOT re-create the account with the same name! According to the documentation "Because it not only prevents the restored data to use the same name, but also makes discovering the right account to restore from difficult". So keep that in mind. We just create a support ticket.

What about data corruption? What if a buggy application corrupts the items inside a Container? What if we accidentally deleted/updated wrong data in a Container?

1. First thing to do is the same as deleting a Container which we explained earlier. We should increase retention time. In this scenario knowing the time of incident is very important. Since the Container is up and running, the new backups get created from the corrupted Container and therefore, we should know the time of incident to restore the right backup.

2. Open a support ticket.

In the end, I would highly recommend to read the reference page in the documentation since it contains some other important points.

How to get notified about planned maintenance for Azure services

Fri, 05 Jun 2020 09:10:00 GMT

One of the biggest advantages of using Cloud services is the fact that we no longer need to maintain the Hardware and infrastructure. Using PaaS offerings takes it even further and we no longer need to care about updating services, applying security patches and so on.

The fact that Cloud providers take care of updating and maintaining services does not necessarily mean we as a service consumers will not be affected at all. In fact, in most cases we will be affected but depending of the type of the business, the impact could be minor or significant.

If we are dealing with a mission critical application, we need to know when maintenance happens by the Cloud provider and get ready for the consequences. Generally, I believe it's always good to know about such incidents.

For instance, imagine we know there is going to be a maintenance operation going on, on our database server at a particular time. If we see some strange or different behavior from the database at that particular time, at least we know that it could be the consequence of the maintenance and no need to start debugging and digging deep into details to discover the issue. We may also avoid the deployments and releases while a maintenance is being done on critical resources.

Fortunately, It's so easy to get notified about planned maintenance for Azure services. We just need to use Azure Service Health.

To do so, select the Service Health resource in the portal and then go to Service issues menu item.

There you can set alerts to be notified about planned maintenance for different Azure resources in advance.

The following is an example alert email I have received.

Real world use-cases to set lifetime to the documents in Cosmos DB

Mon, 13 Apr 2020 16:54:00 GMT

While creating a container in Cosmos DB and also while upserting (insert/update) a document in the database, we can set a lifetime or so called Time to Live (ttl) to the documents. The lifetime is in seconds and the documents get deleted from the database after the given seconds are passed from the last modified time of the documents. But why would be a real use-case to leverage this feature?

Cosmos DB is Microsoft's NoSQL database which is a distributed database in nature. The only thing we need to do while working with this database is to have a right partitioning strategy and provisioning enough throughput. Database sharding and scaling and pretty much everything else is handled by Microsoft seamlessly behind the scene. Having all that said, makes it an ideal database for persisting IoT telemetry data. You can also read my recent post Lesson learned after 2 years working with Cosmos DB to know more about my practical experiences with this database.

In IoT scenarios we typically receive volumes of telemetry data points in mediums like Azure IotHub, EventHub or even normal http endpoints. Cosmos DB has a great integration with other data related technologies in azure and is highly available and globally distributed database. That makes it a perfect location to persist telemetry data at beginning to use it later for further processing. Now let's talk about what is typically done by IoT telemetry data. Three use-cases come into my mind.

1. Realtime processing scenarios

There are some pieces of data that may need to be processed asap and possibly reacted immediately upon. For instance, the data may be related to the temperature of a device or a system. If the temperature exceeds a threshold, we may need to act upon that and take some immediate actions.

Another example could be realtime reporting like tracking a moving car or Escooter or Ebike etc.

2. Reporting scenarios

IoT telemetry data has typically a tight relationship with time. The data either indicates an occurring of an event at a certain time or carries special data at a certain time (like electricity consumption of a smart meter). In such scenarios, typically the data gets processed and harmonized and finally persisted ideally in the time-series date stores like Time Series Insights or TimeScaleDB. Then we can provide super fast reports based on time buckets.

3. Archiving scenarios

The data may just need to be stored somewhere as an archive or source of truth for future possible incidents. The data may not be even needed after a certain time or if a device or machine breaks or goes out of existence. For archiving purposes, data warehouse technologies are probably good feet.

Considering 3 scenarios above, after a certain time, we may really not need to keep storing data in our Cosmos DB. For instance, GPS data points of an EBike from last year may really not be needed to be available in Cosmos DB now. If it is going to be needed, probably some of other 3 scenarios mentioned above are the better places to lookup. Or even if the EBike is already broken and disposed, we no longer need to keep the data in the database.

Setting a proper lifetime makes sure documents get deleted when they no longer are needed. That brings two benefits along. First, it reduces the number of documents in logical partitions in Cosmos DB and consequently improves queries performance (again very important in IoT scenarios). Second, we do not pay for their storage.

They ttl feature is quite handy. The deletion is handled behind the scene and it uses remaining not used throughput we have provisioned for the container. That means first, it does not cost anything extra for us and second the document may not be deleted immediately when ttl is over. We may need to wait until enough not used throughput is available.

Lifetime can be set while creating a container in Cosmos DB by code or from the portal. It could also be updated later. In this case that life time will be applied to all the documents in the container.

Lifetime could also be assigned while upserting a document. In this case, it will override the lifetime set at container level for this particular document.

I'd like to mention one more tiny point before I wrap up this post. There is one more use I make from this feature. If we need to go to the portal and truncate all the data inside a container in Comsmos DB, we can just set the ttl to something like 1 second and then after all the documents are deleted, disable (or reset) the feature again. That's a handy way of getting rid of all the documents in a container in a few seconds.

Here you can read more about this feature:

Review on the book Hands-On Domain-Driven Design with .Net Core

Sun, 08 Mar 2020 13:27:00 GMT

A few words before starting the book review. I am big fan of Domain-Driven Design (DDD) and therefore, you can expect my review to be a bit opinionated. In a moment you will find out why.

What is DDD? Well, If you look this question up in Google, you will find a lot of accurate and well described explanations but in my words, I’d say DDD is a set of guidelines which help you through building a much better software from negotiating with business all the way down to the implementation in the code.

DDD is not new and in fact, the term was coined but Eric Evans in his very famous book Domain-Driven Design: Tackling Complexity in the Heart of Software in 2003 (around the time I just started working as a junior .NET developer).

My first introduction to DDD was in 2008 when Jeffry Palermo described Onion Architecture in his very famous article series. Emerging Microservices architecture, moving away from traditional layered architectures and embracing architectural styles like Onion/Hexagonal/Clean, huge demand in scalability and distributing more and more applications are some of the reasons that DDD received a lot of attentions in the past few years in my opinion.

If you have been working in software development industry for more than a few years and on sizable software, then probably the following scenario should be familiar to you.

We start a new software and there are tables like Product and User with reasonable number of columns in the database. After a couple of years, there will be tons of columns in those tables, many of them acting as references and constraints. Every time a new requirement arrives, we may need to make some columns nullable or mandatory and consequently having hard time to figure how this change would affect every other feature. Soon after, performance drops and it becomes a big concern. What if we need to scale the databse?!

We have been dealing with such issues since databased have been and heart of many of software and of course they have not been designed to particularly fulfil our business needs.

DDD is a way to retire databases as the first-class citizens and instead replace them with our business domain. I believe everyone needs to learn DDD. Not because it’s a good solution for every problem (because it’s not) but because adopting every piece of advices in DDD can incredibly be helpful, no matter which way we are using to develop our software.

As I mentioned before, DDD is a set of guidelines but when it comes to the lower level implementation, things become opinionated and that was the reason I bought this book to to examine and compare different ideas. Let’s get to the review now.

If you know Alexey Zimarev or happen to follow him in Twitter, you know he is the person who knows what he is talking about when it comes to DDD. Here is my 2 cents after studying his book Hands-On Domain-Driven Design with .NET Core.

Chapter 1: Why Domain-Driven Design

In this chapter he talks about some useful preliminary topics like complexity and domain and backs his discussions up with scientific published letters.

Chapter 2: Language and Context

This chapter is about introducing some fundamental topics in DDD like ubiquitous language, anemic models, being explicit and impacts of context switching in daily works.

Chapter 3: EventStorming

EventStorming is also something I have become a big fan of in the recent years. The technique was introduced by Alberto Brandolini in 2013 and it’s about bringing visibility to the behavior of the system, breaking complex requirements into smaller and relevant pieces and ultimately extracting the requirements out of that. It’s very practical method and compatible with agile methodologies.

Alexey not only introduces EventStorming process in a practical way, he also shares his experiences and teaches how to act as a facilitator of EventStorming sessions. I very much liked this chapter and was my favorite one.

Chapter 4: Designing the Model

As the title implies, this chapter is all about talking about the models. It uses the result of EventStorming chapter and brings it to the design level and also introduces some other elements in EventStorming.

Chapter 5: Implementing the Model

Up to this chapter, it was mostly theoretical discussions but in this chapter and the following ones author get’s his hands dirty with code. It includes topics like Entities and Value Objects, domain events, ensuring valid states and writing unit tests.

Chapter 6: Acting with Commands

This chapter is about designing API layer (or as the author calls it, edge layer) and implementing Commands.

Chapter 7: Consistency Boundary

A talks about Transactional Boundaries in databases and how to bring the consistency to the domain models instead. Introducing Aggregates and Aggregate roots and finally how they protect consistency of their invariant entities.

Chapter 8: Aggregate persistence

It’s about implementing persistence layer using Entity Framework Core and Raven DB.

Chapter 9: CQRS: The read-side

Author introduces CQRS (Command Query Responsibility Segregation), why it’s needed, why databases cannot be optimized for read and write operations and finally how to write queries.

Chapter 10: Event Sourcing

Another very important pillars of DDD. Author talks about what Event Sourcing is, which kind of problems it solves, and finally building an infrastructure for that in code.

Chapter 11: Projections and Queries

Complementary chapter from the previous one and talks about projections, queries, selections, subscriptions, replaying events, checkpoint and so on.

Chapter 12: Bounded Context

This last chapter is about Bounded Context. A very important topic in DDD. I personally would like it if the author had not chosen this subject as the last chapter of the book and focused more on this topic.

Conclusion

Overall, I very much enjoyed reading this book. It’s a practical book for .NET developers and also application architects. As it’s an architectural book, I do not think it would be best book for beginners or absolute beginners in the software development industry.

How to Import and Export data from PostgreSQL

Fri, 31 Jan 2020 17:10:00 GMT

I am dealing with a Azure Database for PostgreSQL now and a few days ago needed to export a database table with millions of records and import them in another table. As a person coming from Microsoft SQLServer world, I though it must have been a trivial work but in the end spent a couple of hours to figure out how to do that.

I am using PgAdmin 4 as a client to manage the Postgre databases. Exporting a small table is very easy task with the tool but for larger ones it just generated an empty file with no data inside!

I looked the issue up in internet and found a lot of hacks and tips and tricks to make it work but ended up with nothing. Later I noticed it's a very typical issue in PgAdmin 4 (v 4.17.0). Nevertheless I prepared a script for the same purpose and will share it here.

First jump to the installation folder of PgAdmin4 command tools. In my Mac book it's located under the following path:

/Applications/pgAdmin 4.app/Contents/SharedSupport

How to export

Here is the script to export the data from a table called tableName to a CSV comma separated file.

./psql -h "my-postgresql-server.postgres.database.azure.com" -p "5432" -U "postgre_user@my-postgresql-server" -d "databaseName" -c "\COPY (SELECT * FROM tableName) TO /Users/morteza/file.csv DELIMITER ',' CSV"

Well, I think it's pretty self explanatory but here is command's details:

-h Postgre server
-p Port
-U database username
-d database name
-c command(select all the rows from the table)
TO local csv file address on the disk

After running the command, you will be prompted to enter the database password.

How to import

How to import data from the csv file into newTableName table.

./psql -h "my-postgresql-server.postgres.postgres.database.azure.com" -p "5432" -U "postgre_user@my-postgresql-server" -d "databaseName" -c "\COPY newTableName FROM /Users/morteza/file.csv DELIMITER ',' CSV"

Lesson learned after 2 years working with Cosmos DB

Sun, 22 Sep 2019 11:46:00 GMT

This is now about 2 years I am working extensively with Cosmos DB and here I would like to share my current view points about this database. First, a brief introduction about Cosmos DB.

Cosmos DB is Microsoft's NoSQL database which is offered as PaaS (Platform as a Service). The database used to be called DocumentDB but after a while Microsoft re-branded that to Cosmos DB. Still in some places like documentations or exception details or source codes, you can find foot prints of DocumentDB.

Cosmos DB is relatively an expensive database. The pricing model of the db is based on Throughput (Request Unit per second) you provision for the database or collections. This post is not about teaching Cosmos DB. Fortunately there is a lot of good materials and documentations available in the internet and could be found in a few seconds by googling. This post is more about sharing experiences and you may need to have at least a bit of experience to understand the concepts.

It used to be only possible to provision throughput at collection (it's also called container) level. So even if there was only a few documents inside a collection or even nothing at all, we needed to pay at least about $24 a mount just for that collection.

If the load on the db is more than what is provisioned, Cosmos DB starts rejecting the requests with 429 http status code and returns a parameter which contain an estimation time for the next retry (in some cases you may receive parts of the query result back and need to retry to get the other parts!). So if you try to query/write data to a collection and receive 429 status code, you need to retry until it succeeds. If the throughput you have provisioned for the db is far less than it should be, the retries may never succeed. I you use Microsoft official SDK for Cosmos DB, the process of retry happens several times by the SDK behind the scene. That's why I recommend you to always use official SKDs. At least, as far as I know, this behavior is unique to Cosmos DB and people coming from other ecosystems and with other (even NoSQL) database experiences get quite annoyed (I have witnessed many times). I have seen many developers changed their mind about using Cosmos Db only for this reason!

Collections are logical concepts and there is no limit how big they could be. That pricing model back then, pushed developers to use the same collections to keep different types of documents. Well, there was nothing wrong with that approach but for some reasons I was not a big fan.

After a while Microsoft offered a new feature and it became possible to provision throughput at database level. So the throughput was shared between all collections in the database. That was a huge improvement in pricing I believe!

Cosmos DB is shining in two senses. First, there is a very good integration and out of the box support between Cosmos DB and all the other data related services in azure like Azure Search, Azure Functions, Big Data services and so on. For instance, in just a matter of a few minutes we can connect Cosmos DB and Azure Search together and implement our own super fast and customized search engine.

Second is the fact that Cosmos DB scales out (horizontally) infinitely and seamlessly behind the scene if required and the hard work is all hidden from the developers and users which is very nice. If you enable replicating data globally, the changes in one database get globally replicated in all Microsoft data centers in up to a second (as promised by Microsoft).

Therefore, no matter how big is the load of the data you want to persist in the database, Cosmos DB can fulfil the job on two conditions:

There is enough throughput provisioned for the database/collection.
There is a right Partitioning strategy.

How much throughput is enough?

Like almost any other resources in azure, there is a Metrics section for Cosmos DB. If there is an expectable or granular load on the DB, you can simply see the consumption of throughput in Metrics and decide of the right amount of throughput.

But imagine the scenario that there is no load on the db at all during the day. Only couple of times a day you you receive a big load of data and need to store in the database. Unfortunately as of writing this post, there is no good solution for this requirement. What we need here is to increase the throughput as much as needed and lower down when then not any longer needed. Dynamic throughput provisioning is not yet offered by Cosmos DB. There is a couple workarounds but are not efficient for all usecases. For instance, if you know exactly when the data is received, you can call an api to bump up the provision and later lower it down when it's no longer needed.

What is it about partitioning?

If you are doing something serious with Cosmos DB, Partitioning is the most important topic you need to focus on. Earlier I mentioned Cosmos DB scales out automatically and seamlessly but that would not be the case if you use wrong partitioning strategy.

If there is going to be only one take away for you from this article, it should be the following section! So pay attention to that.

While creating a collection in Cosmos DB, you need to assign a PartitionKey. PartitionKey should be a property of a document that we store in db. Needless to say that documents in Cosmos DB are just plain JSON objects.

After you create a collection (again, it's also refereed as container), it's NOT possible to change the PartitionKey. Imagine you receive IoT telemetry data from 4 different countries. Germany, France, United Kingdom and Italy. you may be temped to use country name as PartitionKey as follow:

{
id:"123456",
country: "Germany",
city: "Hamburg",
deviceType: "smart-lock",
timestamp: 1569147916
}

Microsoft guarantees that the all the documents with the same PartitionKey are stored in the same Logical Partition and ultimately in a single physical machine. That means if later we query the data within the same partition (e.g. telemetry data only in Germany), the query runs over a dataset in a single physical machine and therefore, it does not have to travel over the network and collect data from different data storages! That brings the maximum speed to us.

So far so good but it's not all the story! Storing all the documents with the same PartitionKey in a single machine, brings up a size limit! As of writing this post, maximum size of a logical partition (and consequently a physical partition) could not exceed 20 GB. If it does, the database crashes and the whole thing goes down and there is no remedy for that! Therefore, the first very important factor to take into account for while choosing a PartitionKey, is the size of the data over the time.

The second factor for choosing a PartitionKey is considering the fact that how data is written into the database. Cosmos DB works the best if it needs to write the data evenly in different partitions. Let's get back to our example.

If the data coming form Germany, France, UK and Rome is even, I mean 25% from each country, then data can evenly distributed and persisted in the database. But what if 60% the data is coming from Germany and 40% from the others?! Then the Germany's partition gets hit the most and results in what's called Hot Spot or Hot Partition! The interesting fact is that even if you increase the throughput, it will not help much!

Well, then we may think it's better to choose city name as partition Key! Hamburg, Munich, Berlin, London, Paris, Rome! Does it help? Only you know and your data pattern. Worth to recap that after creating a collection, it's not possible to change PartitionKey. So we need to create a new collection and choose the right PartitionKey and possibly migrate the data from the old collection to the new one! sounds painful?! Yes it is. Makes it worse to know, there is no backup/restore functionality to move the data in an atomic fashion. So you either need to write your own tool for data migration ot use ETL tools like Azure Data Factory in azure to orchestrate data migration from the old collection to the new ones!

Personally I found it quite useful to have a property in the document called "partitionKey" and then the value will be whatever is appropriate. It could result in a bit of data duplication but also bring flexibility of changing it's value in future if needed.

{
id:"123456",
country: "Germany",
  city: "Hamburg",
  deviceType: "smart-lock",
  timestamp: 1569147916,
  partitionKey: "Hamburg"
}

If you are receiving a huge load of data, you can use time based partitionKeys. For instance, combination of day, hour, minute etc. Then the data each minutes is written in a different partition.

{
  id:"123456",
  country: "Germany",
  city: "Hamburg",
  deviceType: "smart-lock",
  timestamp: 1569147916,
  partitionKey: "Hamburg_20190922_1400_03"
}

If the load is still big enough to get you stuck with Hot Partitions, you may even use a random unique value for each document to help Cosmos DB to persist the data as it wishes and wherever it wishes. That helps a lot to persist the data but what about querying/reading data afterwards?! What if the data you need to query, exists in more tha one partitions? I am afraid, then you need to issue a Cross Partition Query and Cosmos DB needs to ask for the data in all partitions and possibly in different physical machines!

So there is paradox here. Optimizing partitions for write scenario resulted in poor performance in read scenarios. Unfortunately that's a big issue now. The solution here is to have two collections. One optimized for writing and the other for reading. Cosmos DB offers a feature called changed-feed. Using this feature, Cosmos DB generates events where a document changes (insert/update/delete) in the db. We can write code to receive the changes and populate the second collection which is supposed to be optimized for read queries.

I wish Microsoft one day provide an atomic way for doing so. I mean, having two collections which are mirrored/synced in sense of data but with different partitionKeys.

Long story short, while choosing a PartitionKey try to consider size of the data, load of the data while writing and finally read queries.

How to send MQTT messages to an Azure IotHub by MQTT.fx client

Sat, 06 Jul 2019 15:30:00 GMT

As clearly stated in the documentation, “IoT Hub is not a full-featured MQTT broker and does not support all the behaviors specified in the MQTT v3.1.1 standard". If you need to send MQTT messages to an IotHub and for whatever reason cannot use IotHub standard SDK, then the whole process could be a bit tricky.

In this post, I am going to be talking about how to use MQTT.fx client to send messages to an IotHub. MQTT.fx is a client written by Java in order to interact with MQTT message brokers. It could be used for debugging and troubleshooting purposes.

First of all, you need to install the proper MQTT.fx client applications for your operating system.

Then run the client and press the settings button.

Here you can see the first form view. I will explain the values I have changed.

Broker Address: This is IotHub host name. My IotHub is called iothub-prototype so the host name is iothub-prototype.azure-devices.net.

Broker Port: The default port number is 8883.

Client ID: Name of the device. My device is called device01.

Now move to the User Credentials tab.

User Name: If your device is using Symmetric key for authentication (default option), you need to enter the Username and Password here.

Again, according to the documentation, username is composed with the following format:

{iothubhostname}/{device_id}/?api-version=2018-06-30

Therefore, in my case it is:

iothub-prototype.azure-devices.net/device01/?api-version=2018-06-30

Side note: Pay attention that there is no http or https at beginning!

Password: we need a Shared Access Signature (SAS) for this device. There are different ways to generate SAS for an IotHub device but the easiest way I guess is to use Visual Studio Code for the same purpose. If you have Azure IoT Hub Toolkit installed (I would definitely recommend if you have not yet), simply right-click on the device and select Generate SAS Token for Device. Then you will need to associate the life time the SAS and it will be generated and copied into clipboard!

This is an example of a SAS or Password for the device.

SharedAccessSignature sr=iothub-prototype.azure-devices.net%2Fdevices%2Fdevice01&sig=8E%2BL%2FVwM5K%2FRA4fZjUF36Pa2xnoPOqwhWe4ukXrrQE%3D&se=1562457833

So, we need to move to the SSL/TLS tab.

Just check Enable SSL/TLS checkbox and leave everything else as they are.

Now save and close the form and click on Connect button.

If you have entered the correct information, then it should successfully connect to the IotHub.

To publish a message, you need to go to Publish tab. Enter the topic name there. The format of the topic is as follow:

devices/{device name}/messages/events/

In our case it is:

devices/device01/messages/events/

Then write your message and press the Publish button

If everything goes smoothly, on Log tab you should see something similar to the this.

And of course the message should also appear in your IotHub.

ORMs role in Microservices age!

Sun, 09 Jun 2019 15:06:00 GMT

Recently I saw some discussions on Twitter people arguing about necessity of existence of ORMs (Object-relational mapping) at this age and the others are even waiting to have Entity Framework as a binging in Azure Functions. That made me as a person who have been working with ORMs from the early days and with different types of data driven applications, to write my two cents about the topic.

First, why would ever people think they do not need an ORM? Several points come to my mind in this regard:

Movement towards NoSQL databases in the past few years.
Offering everything as a service by third-party company/components resulted in having an API in some cases to perform CRUD operations rather than dealing directly with relational databases.
Movement towards Microservices architecture and having tiny data access layers makes people feel less need of adopting a full fledge ORM.

I will continue by answering some questions from my point of view.

1. Do we still need full-fledge ORMs?

We definitely do. Well, generally people tend to think if they do not need something, no one ever dose! Basically, as long as you need to perform CRUD operations over relational databases, you need to have an ORM of a kind.

2. Why not writing our own data access layer using low level data access API?

For small use cases, it could perfectly work but keep this in mind that like any other parts if the software, if you write code, you also write bug! You also need to probably write test for your code and it means spending time and effort.

As the size of the software grows, you will be tempted to write a code generator to generate data access layer boiler plate code for you and finally you will ended up writing your own ORM! That means reinventing the wheel!

Writing a good ORM is a very difficult task. There is a reason, dedicated teams work for years to build tools like Entity Framework. Going beyond basic CRUD operations and trying to implement feature like connection management/pooling, managing transactions, change tracking, caching, keeping track of database migrations and so on, could be extremely challenging. I am talking out of experience!

3. Why not going with Micro ORMS?

Well, if Micro ORMs fulfill all your requirements, then go with them. Do not forget the fact that there is a reason for having Micro in the name. They do not offer all the features of a full-fledge ORM. If your software may need the missing features, then you will again need to reinvent the wheel!

I would assume that using an ORM or handling data access layer in general, is one of the early decisions needed to be made by an Architect. So somehow an Architect should foresee all the requirements to make the best decision. Despite the fact the data access layer related code in most types of architectures is encapsulated inside Repository classes and is supposed to be replicable easily but I would have to say easily does not always necessarily mean quickly or even without a cost!

4. Is that right to use ORMs in Microservices?

Yes! ORMs nowadays are much faster than a few years ago and have much smaller foot prints. Feel free to use them.

5. What about having and Entity framework binding for Azure Functions?

They are most welcome.

6. Any reason to avoid ORMs?

Well, as I mentioned earlier, choosing an ORM is an architectural decision and when it comes to Architecture, there is no absolute right or wrong decision. Under different circumstances, you may need to make different decisions.

If you need to meet a very tide SLA to perform a transaction in which even milliseconds matter, then you may need to use even low-level API and perform CRUD. Writing a fast middleware for mission critical applications could also be another candidate.

7. What is the best ORM?

If you have several options, the best thing to do is to test different aspects of the ORMs. Performance, features, programming model, memory foot print etc. Then compare the results and make the best decision.

Long story short, use ORMs unless you have a good reason to not to.

How to install and deploy Extensions in Azure Functions v2

Sat, 15 Dec 2018 20:28:00 GMT

If you have been doing Serverless Programming in Micrpsft Azure Stack, you probably know that the new generation of Functions (v2) has been available now for a few months. There are some differences in the new versions compared to the old one that you need to know before migrating your existing Functions.

Although, while creating a new Azure Function you can choose between different Runtime Stacks such as .NET, JavaScript, Java (preview as of writing this post) but it's importatant to know that in the end the Runtime in Azure Functions 2.x is based on .NET Core.

If you already used Azure Function Bindings e.g. for CosmosDB or EventHub or ServiceBus, you already know that the only thing you needed to do before deploying your Functions to the target FunctionApp was to have the right configurations in function.js file (and of course proper values in Application Settings). After deploying the function files to Azure, it just worked. But it's no longer the case in version 2.

In Azure Functions v2 Bindings come as so called Extensions which need to be installed before being able to use them in Functions. For instance, you need to install the extensions Microsoft.Azure.WebJobs.Extensions.CosmosDB to be able to use CosmosDB Binding in your Functions.

Even if you go to Azure Portal and create a new FunctionApp and try to create a new Function which uses such Bindings, you will be prompt to install the Extension before being able to use it the Function. Below you can see a propmt dialog right after you try to create a CosmosDB Trigger Function.

But what does actually happen when installing an Extension?

Even if you are developing a Function with node.js, when you install an Extention, a file called extensions.csproj gets generated in the root folder beside host.json which holds the references and versions of the Extentions. Then a .NET Core command gets executed and the required DLLs get generated inside bin folder. Both extensions.csproj and bin folder are only visible in App Service Editor or Kudo in the portal.

Below you see an example of registration for a CosmosDB and an EventHub inextensions.csproj.

If you create a FunctionApp on your local machine, then you need to execute some commands to install extensions as explained here. Then you need to submit the bin folder alongside other files while deploying to azure FunctionApp.

While using function command line like func init, func new etc, command line generates the required files for you.

Alternatively, you can also create your Functions in portal. Then copy/paste the generated function.js and extensions.csproj files to your local machine (root of your FunctionApp) and run the following .NET Core command from the command-line in root folder of the appication (specially usefull while migrating the old Functions):

dotnet build extensions.csproj -o bin --no-incremental

This .NET Core command generates the required DLLs and bin folder for you. And you already have the right configurations in your function.js file.

Update:

Since I wrote this post, Azure Functions Core Tools has become quite mature. It's now easily possible to install/add extensions locally with command-line. Here you can find how to register extensions locally.

Also worth to mention that while creating a new function, the required entry will be added to the extensions.csproj file after choosing the right template.

Happy Programming!

Why and how to use lazy initialization in C# using Lazy API

Mon, 13 Nov 2017 11:29:03 GMT

First of all, why do we need lazy object initialization? The answer is that when initialization of an object for whatever reason is expensive, we would typically wish to instantiate it once and only when it's required. There is an API introduced in .NET Framework called Lazy<T> for this purpose.

Since Lazy<T> API has been introduced in .NET v 4.0, I have rarely seen developers use it in their source code. It could be related to the fact that now a days, object initialization responsibility has been delegated to the IoC containers. But anyway, there are still legacy codes and some other situations we need to do lazy object initialization in place.

Let's go on with an example. There is a UserDataService class which we need to initialize it once and used it whenever required. The first idea is to have the initialization in the constructor.

class MyClass
{
    private UserDataService _userDataService;
 
    public MyClass()
    {
        _userDataService = new UserDataService();
    }
 
    // Rest of the code
}

Well, it works but the problem is the object will be initialized always with initialization of MyClass.

A famous traditional workaround is to have a Property instead of a Field and do the initialization there:

class MyClass
{
    private UserDataService _UserDataService;
 
    private UserDataService UserDataService
    {
        get
        {
            if (_UserDataService == null)
            {
                _UserDataService = new UserDataService();
            }

            return _UserDataService;
        }
    }
    // Rest of the code
}

Well, it works for most of the cases but we have sort of mixed up the responsibility of a Field and a Property.

Let's solve the problem with Lazy<T> API:

class MyClass
{
    private Lazy<UserDataService> _userDataService;
 
    public MyClass()
    {
        _userDataService = new Lazy<UserDataService>(() => new UserDataService());
    }
    // Rest of the code
}

That's it! With using Lazy<T> API, we moved the declaration to the constructor and after the first call to _userDataService.Value, initialization will occur.

One interesting fact about Lazy<T> is that it is Thread-Safe by nature. That means if several threads hit the following line, the first thread does the initialization and the others will use the same instance.

_userDataService = new Lazy<UserDataService>(() => new UserDataService());

Worthwhile to say that there is another class called ThreadLocal<T> which dose the similar job but every thread will have their own initialization and instance.

More information about Lazy<T> like exception handling and so on, could be found here.

Happy Programming!

Ideas behind INotifyPropertyChanged implementation

Thu, 26 Oct 2017 12:19:00 GMT

INotifyPropertyChanged is a standard interface available form the early versions of .NET but still so popular. The general idea behind the implementation of this interface is awareness of the changes in the value of the properties in a class.

Now a days, it's heavily used in the implementation of the MVVM patterns and generally two way binding scenarios. The signature of this interface is as below. In case of any changes, we should manually fire the PropertyChanged event.

//
// Summary:
//     Notifies clients that a property value has changed.
public interface INotifyPropertyChanged
{
    //
    // Summary:
    //     Occurs when a property value changes.
    event PropertyChangedEventHandler PropertyChanged;
}

The classic implementation of this interface could be as simple as below:

class User : INotifyPropertyChanged
{
    private string _Name;
    public string Name
    {
        get
        {
            return _Name;
        }
        set
        {
            if (_Name != value)
            {
                _Name = value;
                OnPropertyChanged("Name");
            }
        }
    }
 
    public event PropertyChangedEventHandler PropertyChanged;
 
    private void OnPropertyChanged(string propertyName)
    {
        var pChanged = PropertyChanged;
        if (pChanged != null)
        {
            pChanged(this, new PropertyChangedEventArgs(propertyName));
        }
    }
}

Well, there is nothing fancy about the this implementation. There is a single property called 'Name'. As the value of the property changes, two things happen. First, the new value will be set to the property. Second, OnPropertyChanged method will be invoked with passing the property name as a literal string.

Obviously the first problem here is passing a literal string to the method. That means if in future we decide to change the name of the property to i.e. FullName, we need to manually change the literal string from OnPropertyChanged("Name") to OnPropertyChanged("FullName") and Visual Studio reformatting tools can't help us in that regard. Of course, this brings overhead in maintenance of our real world classes.

As of C# 6, the new "nameof" operator is available which solves that problem. The new implementation is like below:

public string Name
{
    get
    {
        return _Name;
    }
    set
    {
        if (_Name != value)
        {
            _Name = value;
            OnPropertyChanged(nameof(Name));
        }
    }
}

Now we can take advantage of the compile time type safety and rename our properties safely. Needless to say, the nameof operator is acting at compile time and there is no use of reflection at runtime. That means it does not have any negative performance effect. Nevertheless, the issue still exists for the earlier C# versions.

Using CallerMemberNameAttribute to improve usability:

There is an attribute in .NET Framework under System.Runtime.CompilerServices namespace, called CallerMemberName. We can make a good use of this attribute in conjunction with Optional Parameters feature to simplify our INotifyPropertyChanged implementation.

With using CallerMemberName attribute, the name of the caller of the method (or in our case, "Name") will be passed to the method and therefore there is no need of explicitly doing it.

class User : INotifyPropertyChanged
{
    private string _Name;
    public string Name
    {
        get
        {
            return _Name;
        }
        set
        {
            if (_Name != value)
            {
                _Name = value;
                OnPropertyChanged();
            }
        }
    }
 
    public event PropertyChangedEventHandler PropertyChanged;
 
    private void OnPropertyChanged([CallerMemberName]string propertyName = "")
    {
        var pChanged = PropertyChanged;
        if (pChanged != null)
        {
            pChanged(this, new PropertyChangedEventArgs(propertyName));
        }
    }
}

In this piece of code two things have changed. First, we no longer send the name of the property to the OnPropertyChanged method. Second, we have added CallerMemberName attribute to the signature of the OnPropertyChanged and of course, made it an optional parameter. This optional parameter is set by C# compiler at compile time. That means CallerMemberName attribute is also acting at compile time and therefore there is no negative performance effect at Runtime.

More optimizations:

As you see in every property while setting a value, we actually do 3 things:

Compare the old value with the new one
Set the new value to the property
Calling OnPropertyChanged

In the below implementation we make it just one line. The idea could be used as a base for a design or an architecture.

class User : INotifyPropertyChanged
{
    private string _Name = "";
    public string Name
    {
        get
        {
            return _Name;
        }
        set
        {
            OnPropertyChanged(ref _Name, value);
        }
    }
 
    public event PropertyChangedEventHandler PropertyChanged;
 
    private void OnPropertyChanged(ref string field, string value, [CallerMemberName]string propertyName = "")
    {
        if (!CheckIfEqual(field, value))
        {
            field = value;
            var pChanged = PropertyChanged;
            if (pChanged != null)
            {
                pChanged(this, new PropertyChangedEventArgs(propertyName));
            }
        }
    }
 
    private bool CheckIfEqual<T>(T value1, T value2)
    {
        return EqualityComparer<T>.Default.Equals(value1, value2);
    }
}

Some more:

Not directly related but it is worth to mention there are two other attributes called CallerFilePathAttribute and CallerLineNumberAttribute. The former is used to pass the file path of the caller file (i.e. C:\Users\...\Program.cs) and the latter is used to pass the line number of the invocation in the caller file (an integer). They could come up so helpful in some Logging scenarios.

Happy Programming!

Rethrow Exceptions with ExceptionDispatchInfo

Fri, 04 Aug 2017 08:47:35 GMT

What I am going to be talking about today is neither new nor complicated. But doing code review, I still see many developers and even experienced ones make this mistake while handling exceptions.

Let's start from the scratch. What is wrong with the code below?

public static void DoSomething(int number)
{
    if (number < 0)
    {
        throw new ArgumentException("number must not be negative!");
    }
}
 
public static void Main()
{
    try
    {
        DoSomething(-1);
    }
    catch (Exception ex)
    {
        //Log Exception

        throw ex;
    }

What we do here is to log the exception in case of an error and the rethrow it. But the problem here is that when the line 'throw ex;' is executed, the stack trace of the original exception will be gone and a new one from this point will be replaced!

Content of exception before 'throw ex;':

System.ArgumentException: number must not be negative!\r\n at Program.DoSomething(Int32 number) in C:\\Users\\MyUser\\Documents\\Visual Studio 2017\\Projects\\Console2017\\Console2017\\Program.cs:line 16\r\n at Program.Main() in C:\\Users\\Morteza\\Documents\\Visual Studio 2017\\Projects\\Console2007\\Console2007\\Program.cs:line 24

Content of exception after 'throw ex;':

System.ArgumentException: number must not be negative!\r\n at Program.Main() in C:\\Users\\MyUser\\Documents\\Visual Studio 2017\\Projects\\Console2017\\Console2017\\Program.cs:line 32

Do you see the difference between these two? In the first one we see the original exception which was thrown from DoSomething and in the second one, that's implied that exception was thrown in Main method. This could simply be misleading and waste time during debugging the code.

To rethrow the exception without changing anything, we just need to use only 'throw;' (without ex) as follow:

public static void Main()
{
    try
    {
        DoSomething(-1);
    }
    catch (Exception ex)
    {
        //Log Exception
 
        throw;
    }
}

But this is not end of the story. Sometimes for whatever reason we need to do something with exception outside of the catch block. For example, before C# 6.0, it was not possible to use await keyword in the catch block. Then we would have to do something like the code below (actually I have seen this a lot while reviwing code in different sources):

public async static void Main()
{
    Exception exception = null;
 
    try
    {
        DoSomething(-1);
    }
    catch (Exception ex)
    {
        exception = ex;
    }
 
    if (exception != null)
    {
        await LogExceptionAsync(exception);
 
        throw exception;
    }
}

Well, to be able to use await keyword, we had to play around the exception outside of the catch block. Same problem exists in this scenario but we can't simply use 'throw;' here since we are outside of catch block, and it will lead to a compile error.

This is where ExceptionDispatchInfo class comes to the rescue. With this class we can simply capture the original exception and rethrow it anywhere without losing any data.

public async static void Main()
{
    ExceptionDispatchInfo exceptionInfo = null;
 
    try
    {
        DoSomething(-1);
    }
    catch (Exception ex)
    {
        exceptionInfo = ExceptionDispatchInfo.Capture(ex);
    }
 
    if (exceptionInfo != null)
    {
        await LogExceptionAsync(exceptionInfo.SourceException);
 
        exceptionInfo.Throw();
    }
}

I think the code is crystal clear. We first capture the original exception in the catch block and then throw it at some point later.

Happy Programming!

An introduction to BlockingCollection

Thu, 01 Jun 2017 12:48:28 GMT

BlockingCollection is a class under System.Collections.Concurrent namespace and as the name implies, like any other collections under this namespace, it can also be used in concurrent and multi-tasking scenarios.

According to my experience, .many developers are familiar with ConcurrentBag, CuncurrentDictionary, ConcurrentQueue and ConcurrentStack. But less people know the power and the usage of BlockingCollection.

Before going any further, let's have a look at a classic primary school example of working with a concurrent queue. Imagine the scenario, we have several threads which are adding users' email to a queue and there is one thread which reads the emails from the queue and send them an email.

class EmailService
{
    private ConcurrentQueue<string> _queue = new ConcurrentQueue<string>();
 
    public void AddEmail(string email)
    {
        _queue.Enqueue(email);
    }
 
    public void StartSendingEmail()
    {
        while (true)
        {
            bool isNotEmpty = _queue.TryDequeue(out string email);
 
            if (isNotEmpty)
            {
                SendEmail(email);
            }
            else
            {
                Thread.Sleep(1000);
            }
        }
    }
 
    private void SendEmail(string email)
    {
        //Send email here
    }
}

In the preceding code, we have an AddEmail method which is used by different threads to add an item the queue. In StartSendingEmail method we first try to pick an email from the queue and send an email. If there is no email in the queue, we will wait for 1000 millisecond and then again try to pick a new item from the list if any exists. And this goes on and on.

We are using a simple polling technique here. The problem here is that how to come up with this 1000 millisecond. We have no idea when an email could be added to the queue to pick up. There are some other old threading classes in .NET Framework which could come in use together and solve this problem but here we will take advantage of BlockingCollection.

BlockingCollection is in fact a wrapper around the concurrent collections which have implemented IProducerConsumerCollection<T> interface. The most famous collections are ConcurrentBag, ConcurrentQueue and ConcurrentStack.

The following code is the similar solution for solving the same problem but this time with BlocingCollection.

class EmailService
{
    private BlockingCollection<string> _collection = new BlockingCollection<string>();
 
    public void AddEmail(string email)
    {
        _collection.Add(email);
    }
    public void StartSendingEmail()
    {
        while (true)
        {
            string email = _collection.Take();
 
            SendEmail(email);
        }
    }
 
    private void SendEmail(string email)
    {
        //Send email here
    }
}

Look at the StartSendingEmail method and see how it has been simplified.

BlocingCollection offers a method name Take. This method returns (moves) an item from the collections if any exists and otherwise blocks the thread until a new item is available in future (that means a new email is added to the collection later on). So we no longer need to pause the operation for 1 second and then start polling again, or even care about if the collection is empty or not.

There is also a way to inform a BlocingCollection that no new email is going to be added to the collection. That means BlocingCollection does not have to wait any longer for new items if it's already empty. This could be done by calling CompleteAdding method. After calling this method, if the collection is empty by invoking Take method an InvalidOperationException will be thrown.

Let's add another feature to our code to FinishSendingEmail.

public void FinishSendingEmail()
{
    _collection.CompleteAdding();
}
 
public void StartSendingEmail()
{
    while (true)
    {
        try
        {
            string email = _collection.Take();
 
            SendEmail(email);
        }
        catch (InvalidOperationException)
        {
            // we are done!
            return;
        }
    }
}

By call FinishSendingEmail method, Take method throws an InvalidOperationException exception (if the collection is empty). We just need to handle this exception and exit from the loop.

Now you may ask, what is the order of the items which are picked up from the collection?

At the beginning of this article, I mentioned that BlockingCollection is a wrapper around IProducerConsumerCollection<T> implementations. BlockingCollection by default is using a ConcurrentQueue as an underlying data source. But we can explicitly assign which datasource should be used. That means if we wish to pick up the items (here emails) in LIFO (last in first out) order, it could simply be done by sending an instance of ConcurrentStack to the collection while initializing:


BlockingCollection<string> _collection = new BlockingCollection<string>(new ConcurrentStack<string>());

Here we talked about some fundamental features of BlockingCollection but there is a still a lot offered by this class. for instance, a CancellationToken could be sent to the Take method as a parameter.

Happy Programming!

How to find the implementations of an interface in Visual Studio 2013 & 2015

Wed, 05 Oct 2016 18:52:00 GMT

Now a days, it has become highly unlikely to find a robust software design and architecture which is not utilizing interfaces. During development, it's sometimes really frustrating to find the implementations of a method of an interface.

Specially if you have to do that over and over again everyday.

Fortunately in Visual Studio 2015 a new menu option was added that makes life much easier when we need to find the implementation(s) of methods in an interface.

Let's have an example:

class Program
{
    interface IMyInterface
    {
        void DoSomething();
    }
 
    class MyFirstImplemntation : IMyInterface
    {
        public void DoSomething()
        {
            Console.WriteLine("I am doing something!");
        }
    }
 
    static void Main(string[] args)
    {
        IMyInterface obj = new MyFirstImplemntation();

        obj.DoSomething();
 
        Console.ReadLine();
    }
 
}

If you are using Visual Studio 2015 and wish to see the implementation of DoSomething method, you just need to right click on the method invocation and select 'Go To Implementation' option as follow:

If there is more than one implementations for this interface, you will be shown a list of references in a dialog window and then you can simply choose each reference to see the implementation.

But what to do in Visual Studio 2013?

As far as I know, the best way (out of the box) to see the implementation of a method is to right click on the method invocation and select 'View Call Hierarchy' option (or press Ctrl+K,Ctrl+T).

Then the Call Hierarchy window will be opened for you. If you expand 'Implements xxx' node, you will see all the implementations and by double clicking on any of them, you can navigate to the particular implementation.

'Paste Special': a less well-known feature in Visual Studio

Tue, 09 Aug 2016 19:43:00 GMT

Well, as web developers, we are always dealing with JSON data which are coming from different sources. Either we are serializing our own entities or it's coming from the external sources like third-party services and so on.

If the data is coming from an external source, one common requirement is to always de-serialize it back to the data model to be able to process the data. Creating data model for JSON data is not the most exciting work in the world. Specially when the data model is a bit nested and complex.

Fortunately there is a very nice feature is Visual Studio which makes the life much easier. This feature is called Paste Special.

To take advantage of this feature you need to first Copy the JSON data . Imaging there is JSON data as follow:

{
    "glossary": {
        "title": "example glossary",
		"GlossDiv": {
            "title": "S",
			"GlossList": {
                "GlossEntry": {
                  "ID": "SGML",
		"SortAs": "SGML",
		"GlossTerm": "Standard Generalized Markup Language",
		"Acronym": "SGML",
		"Abbrev": "ISO 8879:1986",
		"GlossDef": {
                   "para": "A meta-markup language, used to create markup languages such as DocBook.",
		"GlossSeeAlso": ["GML", "XML"]
                    },
		"GlossSee": "markup"
                }
            }
        }
    }
}

And we require to create the data model for that. To do so, just Copy the data in memory and go to Visual Studio, create a new class (or an existing one where we intend to have our data model).

From the Edit menu in Visual Studio, select the Paste Special and from the sub menu, Paste JSON As Classes.

Then, the data model will be generated as follow:

public class Rootobject
{
    public Glossary glossary { get; set; }
}
 
public class Glossary
{
    public string title { get; set; }
    public Glossdiv GlossDiv { get; set; }
}
 
public class Glossdiv
{
    public string title { get; set; }
    public Glosslist GlossList { get; set; }
}
 
public class Glosslist
{
    public Glossentry GlossEntry { get; set; }
}
 
public class Glossentry
{
    public string ID { get; set; }
    public string SortAs { get; set; }
    public string GlossTerm { get; set; }
    public string Acronym { get; set; }
    public string Abbrev { get; set; }
    public Glossdef GlossDef { get; set; }
    public string GlossSee { get; set; }
}
 
public class Glossdef
{
    public string para { get; set; }
    public string[] GlossSeeAlso { get; set; }
}

Here we go! As you see the initial data model has been generated for us and we just need to make our customizations.

As you may have noticed, there is one more option under Paste Special menue item, named Paste XML As Classes. This item does the same thing but for XML data. That means you just need to copy your XML data to the memory and from the Paste Special menu item, choose Paste XML As Classes this time, to have your data model generated.

Web Extension Pack extension for Visual Studio 2015

Sun, 03 Jul 2016 10:38:00 GMT

Before Visual Studio 2015, if there was only a single extension for Visual Studio that every web developer had to install that, it was Web Essentials. For every other extensions we usually have to download and install them individually.

But fortunately for Visual Studio 2015 a new extension pack has been introduced which in fact is a collection of very useful other plugins (Web Essentials 2015 included) specially for web development. The name is Web Extension Pack and is another super cool work by Mads Kristensen!

To know the list of plugins and the functionality of each, have a look at the description in the page. Make sure to check out the detail of each extension to make the best use of them.