Thinq LINQ

New Paper.li for Reactive Extensions

Jim Wooley — Fri, 05 Apr 2013 14:23:00 GMT

Paper.li is a site that makes it easy to aggregate blog posts, twitter streams, Google+, Facebook, etc into a magazine format for easy reading. There are virtual papers on almost any topic you could imagine. While I’ve seen messages that some of my tweets and links have been featured, I didn’t recall seeing any on the Reactive Extensions (until Now).

I took the liberty of creating a Paper.li specifically for tracking Reactive Extensions news and announcements. Feel free to follow it by clicking on the widget below. Currently I have it scheduled to only publish once a week, but could make it more frequent if necessary. Also, if you can think of a filter I should add or post I should feature, let me know and I’ll see what I can do.

Beware of Async Sub or Void

Jim Wooley — Tue, 02 Apr 2013 18:18:00 GMT

At my VS Live last week, I gave a survey of Asynchronous programming from .Net 1 through .Net 4.5. As part of the presentation, I showed off this simple example of doing Async starting with a synchronous example. Here’s the beginning example:

    Sub Main()
        DoWorkAsync()
        Debug.WriteLine("All Done")
        Console.ReadLine()
    End Sub

    Sub DoWorkAsync()
        PrintIt()
        Debug.WriteLine("Done Async")
    End Sub

    Public Sub PrintIt()
        Dim text = "Hello World"
        Task.Delay(2000)
        Debug.WriteLine(text)
    End Sub

Before continuing on, try to figure out what you would see in the output window. For those impatient, here’s the output:

Hello World
Done Async
All Done

However, if you try this code, you will notice that the output is all produced prior to the expected 2 second delay (from the Task.Delay). Why is the delay ignored, because the TPL schedules the delay operation on a different thread than the main UI one and then lets the rest of the code operate on the main thread. We can force the delay to pause by changing the PrintIt method as follows:

    Public Sub PrintIt()
        Dim text = "Hello World"
        Task.Delay(2000).Wait()
        Debug.WriteLine(text)
    End Sub

Now, we do delay for the 2 seconds as expected, but we loose any asynchronous behavior that we expected. As a result, we should use the new Async/Await keywords to make our PrintIt method async:

    Public Async Sub PrintIt()
        Dim text = "Hello World"
        Await Task.Delay(2000)
        Debug.WriteLine(text)
    End Sub

Now, if we check our output, we’ll see the following results:

Done Async
All Done
(2 second pause)
Hello World

Notice here that the Done Async and All Done messages appear before Hello World. Why? Because when the line with the Await is encountered, control is passed back to the calling method (DoWorkAsync) and schedules the continuation of the await operation on the thread context of the Task.Delay operation. This is just one of the problems with the “Async Sub” or in C#, “async void” patterns. They are acceptable for fire and forget operations, but can cause issues if you want to rely on structured exception handling, resource disposal, and a number of other useful constructs, you shouldn’t use Async Sub. For more information, see Lucian Wischik’s Async Patterns article or any of a number of excellent articles by Stephen Toub.

So, how do we call the PrintIt operation asynchronously and ensure that it completes prior to continuing the “Done Async” operation? We need to change the signature of the PrintIt method to return a Task rather than returning nothing (void). We then need to move the await up the stack to also Await PrintIt and mark DoWorkAsync as Asynchronous.

    Sub Main()
        DoWorkAsync()
        Debug.WriteLine("All Done")
        Console.ReadLine()
    End Sub

    Async Sub DoWorkAsync()
        Await PrintIt()
        Debug.WriteLine("Done Async")
    End Sub

    Public Async Function PrintIt() As Task
        Dim text = "Hello World"
        Await Task.Delay(2000)
        Debug.WriteLine(text)
    End Function

Now when we run our program, we see the following output:

All Done
Hello World
Done Async

Notice here that the “All Done” message appears before the 2 second delay, but Hello World and Done Async come out in the expected order showing how the DoWorkAsync operation was indeed run asynchronously from the rest of the app. If you want to see the internals of how the compiler interprets the Async and Await keywords, see my earlier post on using ILSpy with Async.

Also, if you are interested in using Async with Asp.Net or WCF, make sure to check out the Async session from Asp.Net Conf 2012 which details some of the potential issues you should consider there.

CSS Disambiguation with WinJS and SPA

Jim Wooley — Thu, 21 Mar 2013 22:00:00 GMT

Recently, I’ve been working to build with some HTML/JS applications for Windows 8. While I could have built them using XAML/VB-C#, I decided to push myself out of my comfort zone to see what I could do with my web experience in these new Windows Store Apps. In addition, I have a number of customers who are looking to minimize code re-writing by leveraging their existing web assets for these native applications, so I figured I might as well dig into them a bit.

I’m used to the standard page per request model and AJAX based models, but the Windows-8 implementation follows more along the lines of the Single Page Application (SPA) model. As a result, some things become easier (like state management across pages) while others become more tricky (make sure to avoid the global namespace as more JavaScript will stay alive in your apps now).

One of the first Issues I came up against was an issue when navigating between pages and finding that styles that are being applied in child pages are sticking when navigating back to the parent page. I’m sure I’m not alone as this Stack Overflow post indicates.This wouldn’t be an issue in a standard web implementation because the page would be re-loaded along with only the styles that page requested. However, in the Windows Store App model, each page that is loaded will bring it’s associated css and js into scope the first time the page is loaded and be retained as we navigate to other pages. Let’s take a look at an example app to see how this works.

Let’s start by creating a new JavaScript app using the Navigation App template. This will create a solution framework with a default.html page that loads a home.html page into a div called “contenthost”

To keep this sample as simple as possible, I’m just going to add one more page to the default template. To do this, create a new folder in the pages folder in the Solution Explorer. I’ll call this folder Detail. In the Detail folder, add a new Page using the Page Control template and call it DetailPage.html.

The Page Control template will generate three files in our folder – DetailPage.html, DetailPage.js and DetailPage.css. This allows us to follow a good coding pattern to keep our concerns separated between layout semantics (html), logic (js), and presentation (css). We’re not going to modify the html of this page, instead we’ll just tweak the css to apply a bit of styling to our h1 tag to set the font to Comic Sans MS. Note, I don’t recommend making this particular change in your apps. This change is just for demonstration purposes! Open the DetailPage.css and add the following to it:

h1{
    font-family:'Comic Sans MS', cursive;
}

With that change, we now need to give the user the ability to navigate from the home page to our new page. In the home.html page, let’s add the highlighted line giving the user an anchor tag to click:

        <section aria-label="Main content" role="main">
            <p>Content goes here.</p>
            <a id="detailButton" href="#">View Detail</a>
        </section>

Finally, let’s add a click handler for the “button” in the home.js file:

  ready: function (element, options) {
      element.querySelector("#detailButton").addEventListener('click',
           function () { WinJS.Navigation.navigate('/pages/Detail/DetailPage.html') });
  }

Now that we're done, we can navigate to our page to view our handy work. Hit F5 to run the app to see your impressive L337 skillz.

Next, navigate to the detail screen by clicking your link.

Notice here, because we used the WinJS.Navigation.navigate method to move to this page rather than just setting the href on the anchor tag, we magically have the default back button to get back to the home page. Try clicking it now.

Compare the first screen shot with this one. Do you notice anything unusual? The page title is now the wrong font. If we look at the Solution Explorer while debugging, the situation starts to make sense. Notice here that the DetailPage.js is still loaded even though we have navigated back to the home page.

As I stated earlier, in WinJS applications, the js and css files are loaded when they are first used and retained in memory rather than being refetched for each page request as is done in web pages. Since the styles are applied in the order that they are fetched, the last version of the font that applied to the h1 is the one that came from DetailPage.css. We can’t just try to set the style explicitly again in the home.css, because it was already previously loaded, so it won’t be re-loaded again.

In this case, if we want to globally modify the style, we should make then change in the default.css located in the css folder of our project. If we want to modify the style only for the lifetime of our individual detail page, we need to preface the style with a page class that is specific to our page. Looking at our DetailPage.html, we can see that the page template wizard generated the necessary class for us already.

<body>
    <div class="DetailPage fragment">
        <header aria-label="Header content" role="banner">
            <button class="win-backbutton" aria-label="Back" disabled type="button"></button>
            <h1 class="titlearea win-type-ellipsis">
                <span class="pagetitle">Welcome to DetailPage</span>
            </h1>
        </header>
        <section aria-label="Main content" role="main">
            <p>Content goes here.</p>
        </section>
    </div>
</body>

Notice that the main div uses the “DetailPage fragment” class. With this we can preface the h1 style in the DetailPage.css file with the .DetailPage class selector to indicate that the style should only be applied to this page:

.DetailPage h1 {     font-family:'Comic Sans MS', cursive;
}

If we run the application again, we should see that when we navigate back to the home page, the font is (thankfully) no longer Comic Sans MS, but has reverted to the oh so much more wonderful Windows 8 default of Segoe UI. If you don’t’ believe me and want to try this out for yourself, feel free to type the code in above, or just download the project and try it yourself. I know, I could have told you about the download link in the beginning, but where’s the fun in that?

Have you run into this issue and found alternative solutions, I’d love to know what you thinq.

Blog evolution

Jim Wooley — Thu, 21 Mar 2013 19:03:00 GMT

Five years ago, I started this blog as a site written in LINQ dedicated to talking about LINQ. In the years that have followed, I’ve expanded the discussions to include a number of LINQ and language design related topics, including Entity Framework, Reactive Extensions, and other items. Over-all, I’ve tried to limit the scope of the topics to LINQ related issues.

Although I still love LINQ and thinq it is a great technology, I’ve been increasingly working with other technologies and as a result, I have shied away from posting about them here for the fear of alienating my LINQ loving fans. Now that LINQ has been around for over half a decade, I suppose it is time to allow the topics on this site evolve as well and start discussing some of the newer things that I’m encountering in the various technologies that I’m working with to spread the love (or love-hate as the case is from time to time) of that technology with the rest of you.

My hope is that you can continue to grow with me and continue to find value in the posts that will come. Of course, I’ll continue to post about LINQ related items as I find the inspiration and time. If you find yourself turned off by the non-LINQ related posts, let me know and I can consider moving the blog off onto another domain. I'm always open to knowing what you thinq.

Unit testing rules of thumb

Jim Wooley — Thu, 07 Mar 2013 16:16:00 GMT

Earlier this week, Uncle Bob stirred up the TDD hornet nest again trying to wade the fine line between RAD and pure TDD with excessive use of IOC’s. In one of the many responses to the twitter debate, Jimmy Bogard responded that you should use testing methodologies, “When it provides value … it depends.” Since I am speaking about Unit Testing in SharePoint at the upcoming VS Live events, I figured I should put in my two cents as well.

In general, I tend to agree with Jimmy’s pragmatic approach. I’m not a test first zealot by any means, and don’t think that trying to achieve 100% code coverage is a panacea. Indeed, even if you do have 100% code coverage, there can be significant business rules that you may not be testing that you should have tests for. Also, your code may well have exception checking, input validation, data binding, etc. code that spending time testing can only decrease your overall ROI.

When designing systems, I tend to use the following rules of thumb to guide me in when to write coded tests for my solutions. This list is likely not inclusive and as always, to every rule there is an exception.

If it’s a core piece of business logic, it should have a test.

When building systems, there will be a number of cases where business rules assert that under certain situations something specific should happen. In those cases, I recommend writing a unit test. The test should be named appropriate to the business defined rule. For example, if the user should be notified when customer’s age a value drops below an acceptable minimum, I would name the test something along the lines of Customer_InvalidWhenAgeBelow18. This way we document the system requirements through the tests and have a quick way of validating our logic directly with the stakeholders to make sure our tests are asserting the correct things. Once we are done, we can give the test report to the customer to show them that all of their assertions are coming up true before handing off the code.

If you hit F5 to debug an issue more than 5 times, it would have been quicker to write an automated test for it.

On most systems that I work with, the act of debugging an issue can be quite time consuming. Just navigating the system to the point where you want to test often involves a number of steps, including logging into the system, creating a new record, setting the test values, trying to submit the values and checking the results. Often, if you took the time to write up a testing harness for these situations, you can bypass many of the manual steps and just test the logic that you need in isolation. Additionally, if you take the appropriate time upfront to set some test objects (fakes or mocks), then they can be reused for similar tests in the future – further reducing your future testing development time and effort.

If there’s a complex calculation that you are automating, include tests for it.

Even the most thorough coder can make simple math errors when coding calculations. Flipping a “+/-“ check or evaluating a complex logical truth table can be a brain twisting task at times. A simple error can ripple through the system causing unexpected errors. In many cases when automating calculations, you know up-front what some of the expected input and output values are. Write specific tests for them up-front and you will be well rewarded down the road. Better yet, write one test and provide a list (in Excel/Xml/etc.) with the valid inputs and outputs and then use a data driven test approach to evaluate the various permutations of the values.

If there is a user detected bug in logic, write a test before fixing the bug.

All too often, I find my developers trying to debug an issue and thinking they’ve found the solution only to realize that they just located a different bug and never actually fixed the true defect. When writing tests, you should aim to write them so that they fail first and only pass once you’ve corrected the underlying defect. I’ve also seen tests that appear to have passed even when the test failed, but the author of the test never knew it because the test was always passing (in most cases, they were swallowing test failure exceptions). By writing your test to fail first and then fixing the implementation, you are assured that your fix solves the problem at hand, and have a built-in regression test for future changes.

Often, when trying to get teams to start using a testing approach, I have to work to convince them that the tests are worth the time and effort. Typically, I find resistance particularly from teams that follow the just “get ‘er done” approach. In the consulting world, we often feel the pressure of producing the requested system at the expense of tests which are typically not included as a payable deliverable. However by building up a testing suite, we typically find the value when a test locates a regression defect prior to shipping the product which would have had a cost if it shipped with the defect. Following this more pragmatic approach and remembering these rules of thumb can provide a sense of safety in refactoring and increase your agility and velocity over time when maintaining systems.

Do you have additional rules of thumb that help to guide you? Let me know what you Thinq.

External USB Monitor for road warriors

Jim Wooley — Mon, 04 Mar 2013 14:12:00 GMT

I’ve been a consultant for a number of years. Typically, I’m working on a customer’s site to provide solutions for them both in what I’m currently working on, and to pay attention to other needs that they may have that I can help to offer solutions for. At the client’s location I’m given a desk to work at, but I’m responsible for my own equipment. The laptop situation is no problem, but I’ve been spoiled by having dual monitors in the past.

Six months ago, I happened upon a great solution to the problem. The AOC 16” USB powered monitor. It is just about the same size as my laptop and offers an additional 16” display at 1366x798 resolution. This is great for email, twitter, and debugging purposes. Being a USB monitor I can’t recommend using it for hi-res video, but then again, you wouldn’t be watching Netflix on the job anyway, right?

JSON Literals for VB

Jim Wooley — Fri, 01 Mar 2013 20:53:00 GMT

One of the stand-out features of VB.Net since VB9 has been the inclusion of XML Literals. Followers of this blog should be well familiar with the concept because I first wrote about XML Literals way back in 2006. With them, we can imbed XML directly into our code as follows:

Dim value = <root>
                <child attrib="foo">bar</child>
            <root>

While XML Literals make the task of working with XML more of a joy than a necessary evil, they do add a certain level of complexity to the language. Any language feature then needs to be maintained moving forward. When I asked Anders about adding them to C#, he pointed to the ongoing maintenance issue along with the supposition that although XML was becoming a de-facto data persistence syntax, would adopting the literals into the language then set a precedence that they would need to support other persistence mechanisms when XML was replaced with some other syntax.

With the rise of REST and decline of SOAP, we have indeed seen the popularity of XML wane in favor of the more compact json syntax. The popularity is increased due to the fact that most JavaScript clients make parsing json into objects trivial. As a result, I have had conversations at conferences joking about the need for json literals in the language as well. At a recent conference an idea came to me which could actually make them (almost) a reality.

At its heart, json is simply a string representation of the JavaScript object structures. These object structures behave much like the .Net dynamic PropertyBag object. As a result, all we really need is a way to embed a long, multi-line string into our VB code and then parse it into a dynamic PropertyBag in order to consume it. Unfortunately, VB doesn’t support multi-line strings. However, it does support multi-line XML. All we need to do is wrap the multi line string inside of an XML element literal:

        Dim jsonLiteral = <js>[
            {author:'Jim Wooley', bookName:'LINQ in Action'},
            {author:'Frank Herbert', bookName:'Dune'},
            {author:'Joe Albahari', bookName:'LINQ Pocket Reference'},
            {author:'Joseph Rattz', bookName:'Pro LINQ'},
            {author:'Charlie Calvert', bookName:'Essential LINQ'}
        ]</js>

With that, we just need to decode the jsonLiteral.Value into a dynamic object. A quick search of Stack Overflow finds a number of handy options for this task. For the sake of this example, I’m just going to use the System.Web.Helpers.Json library that’s part of MVC. We’ll create an extension method to take an XElement and convert it into a dynamic object using the Json.Decode method:

<Extension>
Public Module JsonExtensions
    <Extension>
    Public Function JsonDecode(input As XElement) As Object
        Return Json.Decode(input.Value)
    End Function
End Module

With this in place, we can now operate on the literal just as if it was any other dynamic object type. Here’s the full code for this example.

Option Strict Off

Imports System.Runtime.CompilerServices
Imports System.Web.Helpers

Public Class Test1

    Public Sub TestJsonLiteral()
        Dim jsonLiteral = <js>[
            {author:'Jim Wooley', bookName:'LINQ in Action'},
            {author:'Frank Herbert', bookName:'Dune'},
            {author:'Joe Albahari', bookName:'LINQ Pocket Reference'},
            {author:'Joseph Rattz', bookName:'Pro LINQ'},
            {author:'Charlie Calvert', bookName:'Essential LINQ'}
        ]</js>

        For Each book In jsonLiteral.JsonDecode()
            Console.WriteLine(book.author)
        Next
    End Sub
End Class

<Extension>
Public Module JsonExtensions
    <Extension>
    Public Function JsonDecode(input As XElement) As Object
        Return Json.Decode(input.Value)
    End Function
End Module

I’m sure that there are features that this technique doesn’t cover (including LINQ because Object isn’t directly convertible to IEnumerable). I’m also not sure if this really has any practical benefit. It just is interesting to consider.

INETA Community Champion for 2012 Q4

Jim Wooley — Mon, 04 Feb 2013 20:58:00 GMT

I’ve been active in the technology community since 2000 when .Net was just announced. This started with doing some small talks at user group meetings and helping to update their web site. It didn’t take long before I was running meetings and networking with other groups.

Over the years, I’ve gradually increased my reach by speaking at events increasingly far from my home base, including events as far away as London, Las Vegas, and Redmond. Prior to it’s being disbanded, my speaking involvement lead me to being selected as a member of the INETA Speaker Bureau. I continue to work with INETA as a member of their more localized version, the INETA Community Speakers program.

I’ve also continued reaching out to help mentor other user group leaders through being an INETA Membership Mentor for a number of years. I also participate in UG Leader summits when I’m able.

Recently, I was honored to be recognized by my peers as an INETA Community Champion for Q4 2012. While I never aspired for such recognition, it’s a privilege to have my efforts recognized by the community at large. Having received this award doesn’t mean the work is done. I continue to share my experiences with others to help grow the community as much as I can. Who knows what the future might bring, but I hope I can continue to bring the love of technology to others, just as my mentors have done for me in the past.

Office 2013 RT excludes programming features

Jim Wooley — Tue, 13 Nov 2012 16:29:00 GMT

When I started my programming career, I started with writing Access applications with VBA. While I have moved away from these applications for the most part, I occasionally find the need to come back to office automation tasks in Excel, Word and Access. When reviewing the version of Office that comes with the new Surface tablet running Windows RT and Office 2013 RT, I checked out the developer tool bar to see what it offered. Here’s the new toolbar in Excel 2013 RT:

As a point of reference, here is the toolbar as it appeared in Excel 2010:

Notice here that the Code (including VBA and Macros), Add-ins, and Controls sections have been removed from the RT version of Office. I suspect the issue here is that VBA is really locked into the x86 architecture. Since Windows RT runs on the ARM rather than x86 processors, the amount of effort to migrate this functionality was not worth the effort.

As a result, if you have Office solutions that you want to continue to use on the new platforms, be careful what features you use. If you need to continue building solutions based on office, I suspect that support will be added for the new Office app model based on HTML and JavaScript. While we’ve heard rumblings for some time on the impending demise of VBA, this appears to be a clear sign that continuing to invest time and money on extending the old VBA based solutions appears to be a dead-end solution in the new tablet/mobile world.

Hierarchical Trees from Flat Tables using LINQ

Jim Wooley — Fri, 05 Oct 2012 20:46:00 GMT

I’m often tasked with creating a tree representation of a structure from a flat self-referencing table. For example, in the EF extensions to Northwind, they extended the Employee table so that it has a self-referencing “ReportsTo” column. As you can see from the data below, Andrew Fuller does not report to any other employees, but Nancy, Janet, Margaret, Steven, and Laura all report to Andrew (because their ReportsTo value is the same as Andrew’s EmployeeID). Likewise Michael, Robert, and Anne all report to Steven.

In order to generate a tree representation of these records, we could start with any records that have no direct reports and then lazy load each of their children. Unfortunately for large graphs, the number of database hits will grow exponentially as we add tree levels. Typically with parent-child relationships, we could eager load the children using the DataLoadOptions with LINQ to SQL or .Includes with Entity Framework. However with self-referencing entities, this isn’t allowed. With LINQ to SQL, you will get an InvalidOperationException “Cycles not allowed in LoadOptions LoadWith type graph.”

So, how do we load the tree in one pass and build the object graphs? It’s really not that hard once you realize how reference types (classes) work in .Net. Let’s start by creating a holder for each employee and their associated direct reports:

Public Class HierarchicalEmployee
    Public Sub New(emp As Employee)
           Model = emp
    End Sub
    Public Property Model As Employee
    Public Property DirectReports As New List(Of HierarchicalEmployee)
End Class

Now that we have this type, we can fill it using a simple LINQ request. In order to optimize the next step, we’ll push the values into an in-memory Dictionary indexed by the EmployeeID:

Dim allEmployees = Employees.
    Select(Function(emp) New HierarchicalEmployee(emp)).
    ToDictionary(Function(emp) emp.Model.EmployeeID)

Next, we iterate over the full list. For records that have a ReportsTo value, we’ll add their object pointer to their parent’s DirectReports list:

For Each emp In allEmployees.Values
  If emp.Model.ReportsTo.HasValue Then
    allEmployees(emp.Model.ReportsTo.Value).DirectReports.Add(emp)
  End If
Next

Notice, here we take advantage of the Dictionary’s hash rather than having to iterate over the list each time we want to find the parent record. Finally, instead of returning the full list, we only return the employees that don’t have any children (where the ReportsTo is null).

Dim rootEmployees = allEmployees.Values.
    Where(function(emp) Not emp.Model.ReportsTo.HasValue())

If we want to test this out in LinqPad, just use the Dump method on the resulting rootEmployees. As a result, you’ll see the following in the results pane. Notice Andrew is the only root object. He has 5 direct reports and one of his reports had 3 reports. You could just as easily bind this to a treeview control or output it using your favorite UI tooling.

The nice thing about this solution is that if we check the generated SQL, we will just see a simple (single) SQL request to generate the entire graph. As a summary, here’s the complete code from the LinqPad sample:

Sub Main
    Dim allEmployees = Employees.
        Select(Function(emp) New HierarchicalEmployee(emp)).
        ToDictionary(Function(emp) emp.Model.EmployeeID)

    For Each emp In allEmployees.Values
        If emp.Model.ReportsTo.HasValue Then
            allEmployees(emp.Model.ReportsTo.Value).DirectReports.Add(emp)
        End If
    Next
    
    Dim rootEmployees = allEmployees.Values.
        Where(function(emp) Not emp.Model.ReportsTo.HasValue())
        
    
    rootEmployees.Dump
End Sub

Public Class HierarchicalEmployee
    Public Sub New(emp As Employee)
           Model = emp
    End Sub
    Public Property Model As Employee
    Public Property DirectReports As New List(Of HierarchicalEmployee)
End Class

Aggregate clause issues

Jim Wooley — Tue, 04 Sep 2012 15:33:00 GMT

I was reviewing a Stack Exchange message regarding the Aggregate clause in VB () where they found that the query was issuing multiple requests to the database and occasionally returning the entire database table to memory and using LINQ to Objects over the result. I also found that Frans Bouma blogged about this back in 2008 at . Consider the following LINQ query over Northwind:

Dim query = Aggregate o in Orders
                   into Sum(o.Freight),
                   Average(o.Freight),
                   Max(o.Freight)

This produces the following TSQL Statements in EF. Notice here that the Sum and Avg are performed on the server, but the Max pulls the entire table to memory and does Max on the client. It would seem that this is an issue in the expression tree parser.

SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
SUM([Extent1].[Freight]) AS [A1]
FROM [dbo].[Orders] AS [Extent1]
) AS [GroupBy1]

SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
AVG([Extent1].[Freight]) AS [A1]
FROM [dbo].[Orders] AS [Extent1]
) AS [GroupBy1]

SELECT
[Extent1].[OrderID] AS [OrderID],
[Extent1].[CustomerID] AS [CustomerID],
[Extent1].[EmployeeID] AS [EmployeeID],
[Extent1].[OrderDate] AS [OrderDate],
[Extent1].[RequiredDate] AS [RequiredDate],
[Extent1].[ShippedDate] AS [ShippedDate],
[Extent1].[ShipVia] AS [ShipVia],
[Extent1].[Freight] AS [Freight],
[Extent1].[ShipName] AS [ShipName],
[Extent1].[ShipAddress] AS [ShipAddress],
[Extent1].[ShipCity] AS [ShipCity],
[Extent1].[ShipRegion] AS [ShipRegion],
[Extent1].[ShipPostalCode] AS [ShipPostalCode],
[Extent1].[ShipCountry] AS [ShipCountry]
FROM [dbo].[Orders] AS [Extent1]

For comparison, here’s the queries issued from LINQ to SQL:

SELECT SUM([t0].[Freight]) AS [value]
FROM [Orders] AS [t0]
GO

SELECT AVG([t0].[Freight]) AS [value]
FROM [Orders] AS [t0]
GO

SELECT [t0].[OrderID], [t0].[CustomerID], [t0].[EmployeeID], [t0].[OrderDate], [t0].[RequiredDate], [t0].[ShippedDate], [t0].[ShipVia], [t0].[Freight], [t0].[ShipName], [t0].[ShipAddress], [t0].[ShipCity], [t0].[ShipRegion], [t0].[ShipPostalCode], [t0].[ShipCountry]
FROM [Orders] AS [t0]

Interestingly, if you use From instead of Aggregate, the expression tree parsers seem to be able to handle this better. For example, the original query could be re-written as follows:

Dim query = From o in Orders
                  Group By key = 0
                  into Sum(o.Freight),
                  Average(o.Freight),
                  Max(o.Freight)

This produces the following SQL (using LINQ to SQL):

SELECT SUM([t1].[Freight]) AS [Sum], AVG([t1].[Freight]) AS [Average], MAX([t1].[Freight]) AS [Max]
FROM (
    SELECT @p0 AS [value], [t0].[Freight]
    FROM [Orders] AS [t0]
    ) AS [t1]
GROUP BY [t1].[value]

For the time being at least, I have to agree with Frans that it is best to avoid using the Aggregate keyword in VB when accessing a database. I’ll update this if I hear any updates that are on the horizon.

Unit testing and SMTP

Jim Wooley — Wed, 22 Aug 2012 21:16:00 GMT

Over the past several years, I’ve become a fan of unit testing. I’m not a test-first/TDD zealot by any means, but have found the definite benefits of testing your custom business logic to assert that it does what you say it will. When going to a client who hasn’t done unit testing at all, I often find it hard enough to get them to start testing and don’t want to over burden their tests with mocking frameworks to abstract out some of the external dependencies. As a result, I have no qualms about incorporating both unit testing and integration testing into the mix here.

One of the things that can be tricky and time consuming to test is operations that send automatic emails via the SMTP server. If you send through a real server, the transmission can be a significant bottleneck from a performance standpoint, particularly if the server doesn’t actually exist. In the past, I’ve used local fake SMTP “servers” like the one at http://smtp4dev.codeplex.com/. With this, you simply run the server in the start tray and any time you send an email to your localhost port 25, it will show up there. In your web config, you can specify this quickly by adding the following node for your test project.

  <system.net>
    <mailSettings  >
      <smtp deliveryMethod="Network" >
        <network host="127.0.0.1" port="25" />
      </smtp>
    </mailSettings>
  </system.net>

An even better solution is to change the DeliveryMethod attribute to “SpecifiedPickupDirectory” and then specify a local path that you want new emails to appear in. From your unit test, you can then check that folder for the presence of the new email if you want to confirm that it was “sent” and open it to view the contents. The revised configuration is as follows:

  <system.net>
    <mailSettings  >
      <smtp deliveryMethod="SpecifiedPickupDirectory">
        <specifiedPickupDirectory pickupDirectoryLocation="d:\Temp\smtp"/>
      </smtp>
    </mailSettings>
  </system.net>

One additional item I want to point out is that if you follow the MSDN help for SpecifiedPickupDirectory you will currently get a warning if you use DeliveryMethod=”specifiedPickupDirectory” with a lowercase “s”. Fortunately, VS will ignore the case on that attribute value and work either way.

Async in loops

Jim Wooley — Tue, 21 Aug 2012 13:40:00 GMT

In yesterday’s post, I mentioned that there is a performance penalty of using the new Async/Await feature inside of loops. Today, I want to point out another reason why you might not want to use Async/Await inside of a loop. Consider the following example that takes a web page as a string and finds all of the hyperlinks contained in that page (using a regular expression match) and downloads each of those child pages. Using Tasks from the TPL we can load the child pages like this:

      private void GetChildren(string parent)
        {
            var matches = Regex.Matches(parent, _HtmlRegex, RegexOptions.IgnoreCase);
            foreach (Match match in matches)
            {
                if (_CancelToken.IsCancellationRequested)
                    return;
                string childTarget = match.Groups[1].Value;
                WebClient client = new WebClient();
                Task<string> childTask = client.DownloadStringTaskAsync(new Uri(childTarget));
                childTask.ContinueWith(child =>
                {
                    _FoundUrls.Add(string.Format("{0} ({1})", childTarget, child.Result.Length));
                },
                    TaskContinuationOptions.ExecuteSynchronously);
            }
            TaskButton.IsEnabled = true;
        }

Here we use the DownloadStringTaskAsync passing each found child page url. We use the ContinueWith method to chain operations asynchronously to download the child pages. As a first pass on this using Async/Await, we can simply await the task returner and no longer need the ContinueWith clause. Our code just continues as if it were just a straight code flow:

        private async void GetChildrenAsync(string parent)
        {
            var matches = Regex.Matches(parent, _HtmlRegex, RegexOptions.IgnoreCase);
            foreach (Match match in matches)
            {
                if (_CancelToken.IsCancellationRequested)
                    return;
                string childTarget = match.Groups[1].Value;
                WebClient client = new WebClient();
                string child = await client.DownloadStringTaskAsync(new Uri(childTarget));
                _FoundUrls.Add(string.Format("{0} ({1})", childTarget, child.Length));
            }
            AsyncCtp2Button.IsEnabled = true;
        }

That sure looks cleaner. What’s the issue? The downside of await is that the code execution moves over to a background loop and stops executing until it receives a response. As a result, we loose the performance improvement of issuing multiple requests and processing them as they return. Although the UI thread can continue on, this background process is blocked until it completes. At which time we request the next page that was found.

When writing multiple asynchronous requests, it is often better to use the WaitAny, WaitAll (for blocking operations), or WhenAll for non-blocking operations. Unfortunately, Async/Await only allow for operating on a single task, not a group of tasks. So how can we modify the code to issue multiple requests and operate on the results as they come back? We need to translate each of the requests into tasks and then return a Task of the resulting string arrays. Here’s the resulting code:

        private Task<string[]> GetChildrenAsyncAll(string parent)
        {
            var matches = Regex.Matches(parent, _HtmlRegex, RegexOptions.IgnoreCase);
            var childTasks = matches.OfType<Match>()
                             .Select(match => 
                                 {
                                     string childTarget = match.Groups[1].Value;
                                     var client = new WebClient();
                                     var childTask = client.DownloadStringTaskAsync(new Uri(childTarget));
                                     childTask.ContinueWith(child => 
                                         Dispatcher.BeginInvoke(new Action(() => 
                                             _FoundUrls.Add(string.Format("{0} ({1})", childTarget, child.Result.Length)))));
                                     return childTask;
                                 });

            return Task.WhenAll(childTasks.ToArray());
       }

Ugh. That’s not quite as elegant. I never said async was easy. Async/Await do make the simple cases appear easier. However there are times where a bit more effort is needed to achieve the true results that you want.

As always, if you have recommendations on how to improve this code, I’m always eager to learn ways to improve. If you want to try this sample yourself, feel free to download the async project and kick the tires a bit. The download sample also includes a version using Rx if you want to compare that version as well.

Async Await StateMachine in ILSpy

Jim Wooley — Tue, 21 Aug 2012 04:17:00 GMT

In my Async Programming in .Net talk I like to show what’s really happening under the covers with the new async/await keywords in VS 2012. To show this, I’m currently using ILSpy instead of Reflector because it is free. I noticed an “issue” tonight when trying to locate the MoveNext method of the state machine generated class, but couldn’t find it. To begin, let’s look at the code sample.

Module AsyncWorld
    Sub DoWorkAsync()
        PrintIt()
        Console.ReadLine()
    End Sub

    Public Async Sub PrintIt()
        Dim text = "Hello World"
        Await Task.Factory.StartNew(
            Sub()
                Threading.Thread.Sleep(2000)
                Console.WriteLine(text)
            End Sub)
        Console.WriteLine("Done")
    End Sub

End Module

Ignoring the potential code smell of an Async Sub (async void in C#), let’s take a look at what IL Spy gives us out of the box for the generated classes:

Here we can see that the generated closure is displayed with the contained Lambda, but the underlying state machine that manages the closure is missing. If you want to see the state machine, click on Tools – Options and uncheck the “Decompile async methods (async/await)” option.

Once you deselect that, refresh your project to see the compiler generated state machine:

My, that’s a lot of code for our simple Console Writeline task. Can’t find it? It’s called in the line

taskAwaiter = Task.Factory.StartNew(New Action(Me.$VB$ResumableLocal_$VB$Closure_ClosureVariable_1$1._Lambda$__1)).GetAwaiter().

Remember the closure we saw above? Our code’s still in there. The state machine just lets us call this asynchronously trapping for exceptions as necessary. In most cases, you don’t need to worry about all of this compiler generated code. You should be aware of it though if you are writing code with lots of awaits in a single method, particularly in tight loops. Your performance will suffer in those cases. It is better to work more natively with the TPL Task objects with continuations than using await/async in those cases. For more information, I highly recommend that you check out Steven Toub’s Zen of Async session from Build 2011.

If you want to try this out more, download my async samples including comparative examples using Callbacks, APM, Tasks, Async/Await, and Reactive Extensions (Rx).

SignalR and Reactive Extensions are an Rx for server push notifications

Jim Wooley — Mon, 30 Jul 2012 02:18:00 GMT

Recently, I had the need to build a system where multiple clients needed to be notified as changes happened on the server. While watching Damian Edwards, Brad Wilson and Levi Broderick present on Async in ASP.Net during the AspConf I was introduced to the dynamic simplicity that SignalR brings to the table. I was even more intrigued by the fact that it integrates directly with IObservable and the Reactive Extensions. After using it for a week, I’m truly impressed by what they’ve done with this library. To give you an idea of what I mean, let’s take my ObservableSensor demo which generates random values and see how we can use SignalR to expose these values over a distributed client environment.

Reactive Extensions on the Server

To begin, let’s look at the server. Here we will use the Observable.Generate method to generate some random values with associated random categories and the timestamp when the value was generated:

Option Strict Off

Imports Microsoft.VisualBasic
Imports System.Reactive.Linq
Imports SignalR

Public Class ObservableSensor

    Public Sub New()
        Dim rand = New Random(Now.Millisecond)

        Dim Generator = Observable.Generate(Of Double, SensorData)(
            initialState:=0,
            condition:=Function(val) True,
            iterate:=Function(val) rand.NextDouble,
            resultSelector:=Function(val) New SensorData With 
                                          {
                                              .Time = Now,
                                              .Value = val * 20,
                                              .Category = (CInt(val * 4)).ToString()
                                          },
            timeSelector:=Function(val) TimeSpan.FromSeconds(val))

        Generator.Subscribe(Sub(value)
                                Dim context = GlobalHost.ConnectionManager.GetHubContext(Of ObservableSensorHub)()
                                context.Clients.Broadcast(value)
                            End Sub)
    End Sub
End Class

Public Class SensorData
    Public Property Time As DateTime
    Public Property Category As String
    Public Property Value As Double
End Class

If you recall, we discussed the Observable.Generate implementation last year. The new part here occurs in the Subscribe implementation.

SignalR across the tiers

In this case, we are going to “Broadcast” our newly created methods to anyone listening. In this case, we are publishing our notifications without having a direct connection to the SignalR hub. We can grab a hold of it using the GlobalHost ConnectionManager to get the hub in our AppDomain for a type of ObservableSensorHub. What is this Hub thing you may ask. Well, here is the implementation for the ObservableSensorHub:

Imports SignalR.Hubs

Public Class ObservableSensorHub
    Inherits Hub

End Class

In case you’re wondering, no I’m not missing code here. That’s the complete implementation. We’re just creating a strongly typed instance of the SignalR.Hubs.Hub type for the the ConnectionManager to work with. In this simple application, we’re just going to start generating values when the web application starts. In the Global.asax, add the following implementation:

    Sub Application_Start(ByVal sender As Object, ByVal e As EventArgs)
        Sensor = New ObservableSensor
    End Sub

At this point, we now have a server that can broadcast our random values over HTTP to any clients that wish to subscribe. Before moving to the client, I need to say a bit more about the Broadcast “Method” that we are calling on on the Hub’s Clients. If you look at the type definition of Clients, you will see that there is no Broadcast method. Clients is actually a Dynamic object. At run time, we are declaring that it has a method called Broadcast. The SignalR infrastructure then knows how to translate requests for an invocation method by that name into an HTTP message to be sent to any clients (serializing the results into a JSON object using Json.Net). Remember in Visual Basic, we enable the Dynamic functionality by specifying Option Strict Off at the top of our class definition.

Now how do we consume these messages? Let’s start with a console application. First, make sure that you’ve installed and added the references to the SignalR.Client library. The easiest way is to use NuGet. Rather than a bunch of text, let’s just jump to the code:

Option Strict Off

Imports SignalR.Client.Hubs
Imports System.Reactive.Linq
Imports Newtonsoft.Json.Linq

Module Module1

    Sub Main()
        Dim cn = New HubConnection("http://localhost:5687/")
        Dim sensor = cn.CreateProxy("observableSensorHub")

        sensor.On(Of SensorData)("broadcast", Sub(item) Console.WriteLine(item.Value))

        cn.Start().Wait()
        Console.ReadLine()

    End Sub

End Module

Here, we create a new HubConnection specifying the endpoint for our web application. SignalR does support self hosted servers if you want to use a Windows Service or other back end implementation. As long as the client can see the server over the network, you can wire it up. Second, we create a dynamic proxy specifying the hub type that we created on the server. Note here that the casing of the proxy is important and uses Camel casing even though the implementation on the server used Pascal casing. This is done to make the hubs and methods feel more natural to JavaScript clients.

Next, we specify how to handle the push notifications that the server sends. We do that using the .On method, specifying the type (SensorData) that the method should be deserialized back into. We then specify the name of the method (altering the case as mentioned above) that we are listening for along with the Action that should be invoked as each value is received. In this case, we’ll just output the value that we received to the console window.

RX in the Client

At this point we have Rx pushing messages to the client via SignalR. Let’s take it a step further and add some Rx goodness on the client side as well. In addition to the On method, the Proxy also supports an Observe method which turns the message pump into an IObservable of an array of Objects where each of the method parameters are contained in that array. Since our Broadcast method only has a single parameter of type SensorData, we will grab it by getting the first array element and calling the Json.Net .ToObject implementation to translate it back into our strongly typed object. From there, we work with it just as we would any other Observable sequence. For example, to output only the generated values for Category 1, we could use the following:

        Dim cn = New HubConnection("http://localhost:5687/")
        Dim sensor = cn.CreateProxy("observableSensorHub")

        Dim items = From item In sensor.Observe("broadcast")
                    Let instance = item(0).ToObject(Of SensorData)()
                    Where instance.Category = "1"
                    Select instance

        Using items.Subscribe(Sub(value) Console.WriteLine(value.Value))

            cn.Start().Wait()
            Console.ReadLine()

        End Using

In this case, we Start the connection inside of our subscription’s Using clause along with the Console.ReadLine. Once a key is pressed, the subscription is disposed freeing our resource.

One of the nice things is the flexibility that SignalR offers. Pretty much anything that speaks HTTP can consume these messages from our Rx Server. If we wanted to consume the same messages in a web client, we could use the following Javascript code:

      $(function () {
          // Proxy created on the fly
          var hub = $.connection.observableSensorHub;

          // Declare a function on the chat hub so the server can invoke it
          hub.Broadcast = function (value) {
              $('#values').append('&lt;li&gt;' + value.Time + ': ' + value.Value + '&lt;/li&gt;');
          };

          // Start the connection
          $.connection.hub.start();
      });

In the javascript, we connect to our server by referring to the $.connection.observableSensorHub. Notice the camel cased name here? That’s the SignalR translation in action again. We then specify the method handler for the dynamically invoked “Broadcast” method. Here we just add list item entries to the unordered list.

As I said above, I’m so impressed with SignalR so far. Don’t be surprised to see me post more about it in the future. For now however, if you want feel free to download this sample and try it out yourself. You will need to refresh the nuget packages to get it to run. Additionally, realize that the sample was built with RC builds of Visual Studio 2012, Rx 2.0, and SignalR. I don’t guarantee that the sample will continue to work once these are officially released. If you notice any issues or want to see more details, let me know what you Thinq below.

ThinqLinq Updated to MVC 4

Jim Wooley — Tue, 24 Jul 2012 03:39:00 GMT

As I work more in Visual Studio 2012, I found the need to make a small tweak to this web site, but discovered that the MVC 1 code base that the site was using is no longer a natively supported as part of Visual Studio 2012. As a result, I took the plunge and re-wrote the UI portions of the site using ~~MVC3~~ MVC 4 with the new Razor templating engine. This site is a test-bed of technologies I want to play with anyway. I also upgraded the site from .Net 3.5 to .Net 4. In light of these changes, please let me know if you see anything unexpected on the site.

I’m also in the process of revising the comment system from the home-grown version I was using, to a hosted comment system in the hopes to get more conversation going on the site. Feel free to comment using the new commenting system. If you have problems with the comments, please feel free to let me know on the Contact page. Thank you for your ongoing support.

Edit. Thanks to Amit Patel’s article, I was able to upgrade from MVC 3 to MVC 4 in less than 30 minutes (including deployment). I hope you appreciate the effort.

Windows 8 Live Tiles and LINQ to XML

Jim Wooley — Sun, 22 Jul 2012 22:29:00 GMT

When I started working with Windows 8 Metro development, I was disappointed to see that the native WinRT API’s relied on the older XmlDocument and XMLDom rather than the newer subset of XDocument/XElement and LINQ to XML. I suppose this was necessary because the older version was more natural for C++ and JavaScript options for Metro applications. If you’re like me and want to use LINQ to XML to work with XML in WinRT, all you need to do is pass the XML strings back and forth between your C#/VB code and the native WinRT methods.

Let’s consider the case of custom Tile Notifications. In order to set one, you first get the XML template from the TileUpdateManager. This returns an XmlDocument object with an XML template looking like this:

<tile>
  <visual>
    <binding template="TileWidePeekImage01">
      <text id="1"></text>
     </binding>
  </visual>
</tile>

In order to set the text, we need to locate the “text” element with the attribute id of 1 and set the value. While we could create the Xml from scratch, It might be safer to just set the node’s value and retain the rest of the XML. If you’ve been following my blog for any time, I find the LINQ to XML API’s to be much easier to use than the older XmlDom. Luckily, moving back and forth from XmlDom and XDocument is as simple as calling Parse on the XDocument passing in the .GetXml method of the XmlDocument. To push our changed XDocument back into the XmlDocument, we call LoadXml passing the String representation of the XDocument using ToString. Here’s the code to grab the TileSquareText04 template and set the text value to “Text Line 1”.

Dim tileXml = TileUpdateManager.GetTemplateContent(TileTemplateType.TileSquareText04)

' Transition to XDocument
Dim xTile = XDocument.Parse(tileXml.GetXml())
' Manipulate with LINQ to XML
xTile...<text>.FirstOrDefault(Function(node) node.@id = "1").SetValue("Text Line 1")

' Set XmlDocument
tileXml.LoadXml(xTile.ToString())
Dim notification = New TileNotification(tileXml)
TileUpdateManager.CreateTileUpdaterForApplication().Update(notification)

That’s all it takes. If you don’t like this template, there’s plenty more where this came from. MSDN has a great page showing each of the tile samples along with the template XML you need to use.

Reactive Extensions books

Jim Wooley — Fri, 06 Jul 2012 14:49:00 GMT

I’ve been presenting on Rx for some time know. Often I’m asked when I’m going to write another book. I’ve considered writing one on Rx expanding on my blog posts here, but suspected that the market might be a bit small. At this point there are two books on the market for Rx. The first one is free and serves as a good resource to the Rx methods.

	Lee Campbell’s Introduction to Rx is a good overview of the various Reactive Extensions methods and coding techniques. It doesn’t focus as much on the practical uses of Rx, but would serve as a good companion to anyone trying to determine which method they should use for any given need. Even better, it’s free online at www.introtorx.com or for $0.99 at Amazon.
	Jesse Liberty and Paul Betts released the first book (mostly) dedicated to Rx. It offers more practical examples, including a discussion of how to use Paul’s ReactiveUI MVVM implementation. I was hoping to like this book, but was disappointed by the thoroughness of covering Rx itself. Instead they used nearly a third of their 150 pages discussing C# LINQ features and LINQ to SQL for the Windows Phone. If you are already familiar with the basics or have gone through the free HOL below, you will probably want to skip directly to Lee’s book above. If you aren’t familiar with LINQ and Lambda’s this is a good book to start with as it covers the basics as well. It’s also a fairly short book as I read the entire thing on a single plane trip from Seattle to Atlanta.
	A couple of hands on Labs may be your best be to getting your feet wet and understanding the basics of Rx: Curing the asynchronous blues with the Reactive Extensions for .NET and JavaScript

In addition to these books, I still recommend the learning resources mentioned in my post: Reactive Framework Learning Resources for RxNet and RxJS. I still wonder if there is a market for more books on Rx. What do you Thinq?

LINQ to Database Performance hints

Jim Wooley — Tue, 12 Jun 2012 19:04:00 GMT

Although I’ve been writing primarily about RX here for a while, I still dig into some good old LINQ to SQL / EF quite a bit. Along the way, I’m frequently finding more tidbits to throw in the toolbox and thought I’d share some observations from a recent performance tuning project. (I’ll avoid disclosing the client and will change the database model to Northwind for this discussion protect the guilty.)

In this project, I was brought in at the end of the project as things started going south to try to get it across the finish line. As with many projects, they waited until after the development was complete before they started to consider the performance impact of their code. This violates one of the principal tenants of performance testing:

RULE 1: Test early and often

Performance for larger scale apps needs to be considered from the beginning of the project and measured. If you don’t have an ongoing way of measuring the performance as the application progresses through the development lifecycle, you will find it difficult to pinpoint specific performance bottlenecks over time.

As an example, several years ago I met with core members of the Microsoft CLR team. They shared the fact that they compare performance on each night’s builds and if they find more than a couple microsecond deviations in the performance, the teams were required to identify the source of the performance degradation. When working with systems that have millions of lines of code, if the waited a week, it may become hard to impossible to identify the multiple places where performance has now dropped by 1/100th of a second. Over time these performance penalties build up.

I don’t mean to indicate that your Mom & Pop web site needs to live up to this stringent of a testing standard. Also, I’m not insinuating that you should prematurely over optimize your code, but try to make intelligent decisions and have ways of determining when your decisions negatively impact your performance up front.

Why do I bring this up in the context of LINQ to DB (SQL/EF)? If you’re not testing and monitoring your app, you may find that a nice abstraction layer that you added just killed your performance. In the case of the client’s application, they had nicely abstracted away the ability to load various database structures into a generic repository. They had separate methods to GetCustomerById, GetOrdersForCustomerId, GetOrderDetailsForOrderId, etc. They also had helper methods for validation including ValidateIsCustomerNameUnique. The downside here is that reading through the code, it wasn’t easy to notice where hidden database calls were being sprinkled through the code. This brings up rule number 2 for performance testing with databases:

RULE 2: Profile your application

Profiling your application for external requests, including services is essential to make sure the requests you are making are not causing a performance bottleneck. I highly recommend using some sort of profiling tool, particularly for people who are new to LINQ. LINQ makes it easy to make silly mistakes like n+1 requests to a database when navigating lazy-loaded parent-child relationships.

There are plenty of options for profiling applications. I identified some of them back in my LINQ Tools post. Some of these require adding code to your assemblies, while others simply attach to running processes.

If you have a license for SQL Server, you probably have access to SQL Profiler. To use this tool, create a new profile pointing it to your database and run your application. Once you have executed the code you want to profile, you can view all of the requests made against that database. Pay particular attention to cases where the same SQL is issued multiple times in succession or sequence. The downside here is that you will need to manually trace through your code base to find where each request is made in order to fix the issues.

Another alternative if you have the Ultimate SKU of Visual Studio is to use the Intellitrace feature added in Visual Studio 2010. Using Intellitrace, you can not only identify which SQL requests were issued to the database. You can also select each line in the results and navigate to the line of code that caused the request to be issued. Let’s consider this fairly innocuous code snippet to output the Order date and Product Names associated with each order:

   1:  Using model As New NwindEntities

   2:      For Each o In model.Orders.Take(3)

   3:          Console.WriteLine(o.OrderDate)

   4:          For Each od In o.Order_Details

   5:              Console.WriteLine(od.Product.ProductName)

   6:          Next

   7:      Next

   8:  End Using

Running this and putting a breakpoint on line 8, we can see the following Intellitrace output:

From here, we can see that as we iterate each order item, we are making a separate request to the database for each associated order detail record to get the related product object. Clicking on any of the lines of the output will take you directly to the line of the code that the request was made. Unfortunately, you won’t be able to view the parameters for each request using Intellitrace as you could with SQL Profiler, but that’s less important from a performance tuning perspective than it is to know that you have multiple excessive requests to the database. If you find that the intellitrace output becomes cluttered with non-database related trace items, I often find it helpful to filter the results to only ADO.Net requests:

Fortunately, fixing the code above only requires changing line 2 in the code above to eagerly load the child records:

For Each o In model.Orders.Include("Order_Details").Include("Order_Details.Product").Take(3)

We may want to further optimize this request by not hydrating the entire objects, but rather just fetching the rows that we want as well:

Using model As New NwindEntities
    Dim query = From o In model.Orders
                Select o.OrderDate,
                    ProductNames = From od In o.Order_Details
                                   Select od.Product.ProductName

    For Each o In query.Take(3)
        Console.WriteLine(o.OrderDate)
        For Each p In o.ProductNames
           Console.WriteLine(p)
        Next
    Next
End Using

If you don’t have access to SQL Profiler or Intellitrace, consider one of the other relatively inexpensive profiling tools out there like the MVC MiniProfiler, ORM Profiler, Huagati’s LINQ to SQL Profiler, EF Prof, or at a bare minimum checking the generated code for your queries using LinqPad.

With your application now profiled, hopefully you won’t run into the issues such as found in the following code (which I did see in the customer’s app changing the model to again protect the guilty:

Debug.WriteLine(Customers.FirstOrDefault().CompanyName)
Debug.WriteLine(Customers.FirstOrDefault().ContactName)
Debug.WriteLine(Customers.FirstOrDefault().ContactTitle)
Debug.WriteLine(Customers.FirstOrDefault().Phone)
Debug.WriteLine(Customers.FirstOrDefault().Fax)
Dim CustomerOrders = Customers.FirstOrDefault().Orders
For Each order in CustomerOrders
    ' Load the order into a POCO
Next

Quick quiz boys and girls, without using a profiler how many requests are we making to our database here? Remember that not only will you have a database hit for each call to FirstOrDefault which doesn’t use deferred loading, but you’ll also get a call on GetEnumerator (called internally in the For Each iteration). Thus the count is 7 for the above code, right? Actually, it’s worse because hidden behind the Debug.WriteLine is a trace writer which also writes the value to the database’s log table. As a result we actually have 12 database requests (7 reads and 5 writes) instead of the single query that we should have used. In this case we’re breaking rule #3:

RULE 3: Don’t fetch needlessly

In the above code, we can simply fetch our target customer once and automatically include their associated Orders as follows:

Dim firstCustomer = Customers.Include("Orders").FirstOrDefault()
Debug.WriteLine(firstCustomer.CompanyName)
Debug.WriteLine(firstCustomer.ContactName)
Debug.WriteLine(firstCustomer.ContactTitle)
Debug.WriteLine(firstCustomer.Phone)
Debug.WriteLine(firstCustomer.Fax)
Dim CustomerOrders = firstCustomer.Orders
For Each order in CustomerOrders
    ' Load the order into a POCO
Next

In this case we fetch the firstCustomer once and reuse it rather than calling FirstOrDefault repeatedly. We also use the .Include option to eagerly fetch the child records. In this case, I’m saddened that the original developer didn’t use a bit of brain power to eliminate those extra database hits FOR LOGGING PURPOSES, but assume that it was because they weren’t aware of when their database was being hit. Of course, this brings us back to rule 2 – profiling.

Simply removing excessive database hits will almost always improve your code performance. In one case, I had a situation where a request was taking 10 minutes. After removing the excessive hits, the performance came down to 10 seconds, which was a definite improvement. However, 10 seconds still does not make the application web scale as the database’s CPU is pegged for that amount of time. Sometimes, it is actually best to break the process up a bit to improve performance.

RULE 4: Break up complex queries

To give you an idea of the kind of query that we’re talking about here, consider the following case where we are fetching the employees in USA including their regions and sales information.

Dim empOrders = 
    From e In Employees
    Where e.Country = "USA"
    Select 
        e.FirstName,
        e.LastName,
        Regions = From t In e.Territories
                  Select t.Region.RegionDescription
                  Distinct,
        TopThreeSales = From o In e.Orders
                        From od In o.OrderDetails
                        Select
                            od.Product.ProductName,
                            TotalSale = od.Quantity * od.UnitPrice
                        Order By TotalSale Descending

While this code will compile and run, the performance will start to suffer as larger volumes of data are added to the database. The reason is that both Territories and Orders are child collections from Employees. As a result, SQL will return the Cartesian product between the two sets. In other words, if a single employee is associated with 5 territories and has sold 10 products, the total number of rows returned would be 50. The OR/M is then responsible for splitting out those results again into the correct groupings. If you multiply this against 10,000 employees, you will find that there is a massive amount of excess data that is returned.

In this case, It may be helpful to split your sub queries up into separate database requests. You could do something like the following using the Contains clause to pass the Id’s of the parent record in to the subsequent queries.

Dim empOnly = 
    (From e In Employees 
    Where e.Country = "USA" 
    Select 
        e.EmployeeID, 
        e.FirstName, 
        e.LastName). 
    ToList()

Dim EmpIds = empOnly.Select(Function(e) e.EmployeeID)

Dim TopThreeSales = 
    From o In Orders
    Where EmpIds.Contains(o.EmployeeID)
    From od In o.OrderDetails
    Select 
        o.EmployeeID,
        od.Product.ProductName,
        TotalSale = od.Quantity * od.UnitPrice
    Order By TotalSale Descending

However “Contains” has a couple of hidden surprises that limit our use here. First if you use Contains, you can not make it into a compiled query because the number of parameters vary at run-time. Second, if you have more than 2100 items in your EmpIds collection, you will run into a hard limit in SQL Server which only allows up to 2100 parameters. In this event, it is better to re-apply the original filter and return the new result sets for each of the subsequent queries. In the end we can join the separate queries back together again using LINQ to Objects:

Dim empOnly = 
    (From e In Employees
    Where e.Country = "USA"
    Select 
        e.EmployeeID,
        e.FirstName,
        e.LastName).
    ToList()

Dim Regions = 
    (From e In Employees 
    Where e.Country = "USA"
    From t In e.Territories
    Select 
        e.EmployeeID,
        t.Region.RegionDescription
    Distinct).
    ToList()

Dim TopThreeSales = 
    (From e In Employees 
    Where e.Country = "USA"
    From o In e.Orders
    From od In o.OrderDetails
    Select 
        o.EmployeeID,
        od.Product.ProductName,
        TotalSale = od.Quantity * od.UnitPrice
    Order By TotalSale Descending).
    ToList()
                
Dim combined = 
    From e In empOnly
    Select 
        e.FirstName,
        e.LastName,
        EmpRegions = Regions.Where(Function(reg) e.EmployeeID = reg.EmployeeID),
        Sales =  TopThreeSales.Where(Function(sale) e.EmployeeID = sale.EmployeeID)

In the above referenced project, I was able to take some queries that as a single LINQ statement took 10 seconds to run on the database down to sub-second requests. Your mileage may vary using this technique. If at all unsure, refer back to Rule 2: Profile.

RULE 5: Use ORM by default but Stored Proc where necessary

At times you will find that the generated query is too complex and the database has issues trying to parse the generated SQL. The complexity of your query, particularly the number of joins and depth of the object graph/inheritance model you are traversing can cause issues. In these cases, I have no objections to using Stored Procedures in order to wrangle the otherwise unruly queries. LINQ to DB is great for the 70-80% of crud operations, but there are times, particularly when reporting when you need something else. Thankfully LINQ to SQL and LINQ to EF both support consuming stored procedures when the need arises without the need to write the tedious and potentially error prone custom ADO code yourself.

In addition, LINQ is not a replacement for ETL operations. In one case, we had a situation where saving a single record with 2000 children caused the database to churn for 10 minutes due to 2001 individual hits to the database for the update process (on a single SaveChanges call). We re-wrote that operation using BulkCopy and brought the operation down to 1 second from 10 minutes.

RULE 6: Use Appropriate Sample Data

The last item for today is to make sure when developing to have representative quantities of data in your sample as you will have after a couple of years of production data enters your system. We found a number of cases where complex queries like I described in rule 4 above would perform fine in development where we had thousands of rows of data, but when we applied it against the production data which had millions of rows of data, the performance died. The complex joins which worked fine against smaller data sets no longer worked against the bigger sets. If we had a good approximation of data volumes in development, we would have been able to diagnose and fix this issue before shipping the version to production.

That’s just a couple of my rules of thumb which have helped me diagnose and fix performance issues with LINQ to SQL/EF. If you have any tricks or techniques to add, please let me know what you Thinq.

Using Rx to consume a Task based WCF service

Jim Wooley — Tue, 22 May 2012 02:33:00 GMT

Among the many changes that Dev 11 brings is the new default when adding a service reference to generate Task based proxy methods rather than using the APM flavor (using the BeginXXX – EndXXX model). In this post, we’ll look at creating a simple service and then consuming it using the Reactive Extensions. Let’s start by defining the service interface and implementation:

Imports System.ServiceModel
Imports System.Threading 

<ServiceContract()>
Public Interface ISimpleServicesvc

    <OperationContract()>
    Function DoSomethingCool(input As String) As String

End Interface

Public Class SimpleServicesvc
    Implements ISimpleServicesvc

    Public Function DoSomethingCool(input As String) As String Implements ISimpleServicesvc.DoSomethingCool
        Return (String.Join("", From letter In input.ToCharArray()
               Order By letter
               Distinct))
    End Function

End Class

Essentially here we are just taking a string input and returning the distinct characters sorted. The details of the service in this case are trivial. Our focus here is how to implement the service client. We start by adding a service reference in our client application by right clicking on the project and selecting Add Service Reference. (Alternatively, you can now press Ctrl-Q and request to “Add Service Reference” from there. From the dialog, you can still use the “Discover” button to locate the service as long as it is in your solution.

One thing to note is that the proxy classes are now by default generated using Task based methods rather than the previous IAsyncResult AMP method.

As a result, the definition of the proxy class is as follows:

Public Function DoSomethingCoolAsync(ByVal input As String) 
                 As System.Threading.Tasks.Task(Of String) 
                 Implements SimpleService.ISimpleServicesvc.DoSomethingCoolAsync
    Return MyBase.Channel.DoSomethingCoolAsync(input)
End Function

If we wanted to consume this using the new Async/Await, we could do it as follows:

Private Async Sub SubmitClicked() Handles SubmitButton.Click
   Dim svc = New SimpleService.SimpleServicesvcClient()
   Dim req = Await svc.DoSomethingCoolAsync(InputText.Text)
   OutputText.Text = req
End Sub

Of course, to put the LINQ spin on this, let’s see the Rx version to do the same thing:

Private Async Sub SubmitClicked() Handles SubmitButton.Click
    Dim svc = New SimpleService.SimpleServicesvcClient()
    Dim req = svc.DoSomethingCoolAsync(InputText.Text).ToObservable()
    req.ObserveOnDispatcher().Subscribe(Sub(val) OutputText.Text = val)
End Sub

We start by turning the Task into an Observable producer using the ToObservable extension method. We then subscribe to the observable making sure to return back to the dispatcher thread because the task based service is run on a taskpool thread. Of course in this case, we are subscribing on every button click. With Rx, we could wire the button click and service request up on form navigate and unwire it when navigating from the form as follows:

Private requestDisposable As IDisposable
Protected Overrides Sub OnNavigatedTo(e As Navigation.NavigationEventArgs)
    Dim svc = New SimpleService.SimpleServicesvcClient

    requestDisposable = (From click In Observable.FromEventPattern(Of RoutedEventArgs)(SubmitButton, "Click")
                        From req In svc.DoSomethingCoolAsync(InputText.Text).ToObservable()
                        Select req).
                        ObserveOnDispatcher().
                        Subscribe(Sub(val) OutputText.Text = val)
End Sub
Protected Overrides Sub OnNavigatedFrom(e As Navigation.NavigationEventArgs)
    MyBase.OnNavigatedFrom(e)
    requestDisposable.Dispose()
    requestDisposable = Nothing
End Sub