Li Chen's Blog

Porting a C# Windows application to Linux

Sun, 12 Aug 2018 21:02:51 GMT

I own a Windows application. To expand our customer base, we need to create a Linux edition. In anticipating the demand, we previously decided to place the majority of logics in a few .net standard libraries and this is a big paid-off. However, there are still a few things we need to do so that the same code would work on both Windows and Linux.

Path separator is different between Windows and Linux. Windows uses “\” as separator while Linux uses “/” as separator. The solution is to always use Path.Combine to concatenate paths. Similarly, use Path.GetDirectoryName and Path.GetFileName to split the paths.
Linux file system is case sensitive. The solution is to be very consistent with path names and always use constants when a path is used in multiple places.
In text files, Windows uses \r\n to end lines while Linux uses \r. The solution is to use TextReader.ReadLine and TextWriter.WriteLine. TextReader.ReadLine reads Windows text files correctly on Linux and vice versa. If we have to face line-ending characters explicitly, use Environment.NewLine.
Different locations for program files and program data. Windows by defaults store programs in “c:\Program Files” folder and store program data in “c:\ProgramData”. The exact location can be determined from the %ProgramFile% and %ProgramData% environment variables. Linux, in contrast, has a different convention and one often install programs under /opt and write program data under /var. For complete reference, see: http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/. This is an area we have to branch the code and detect operating system using RuntimeInformation.IsOSPlatform.
Lack of registry in Linux. The solution is to just use configuration files.
Windows has services while Linux has daemon. The solution is to create a Windows Service application on Windows and create a console application on Linux. RedHat has a good article on creating Linux daemon in C#: https://developers.redhat.com/blog/2017/06/07/writing-a-linux-daemon-in-c/. For addition information on Systemd, also see: https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files.
Packaging and distribution. Windows application are usually packaged as msi or Chocolatey package. Linux applications are usually packaged as rpm. This will be the subject of another blog post.

Building .net core on an unsupported Linux platform

Sun, 12 Aug 2018 06:11:00 GMT

Introduction

I need to a product that I own from Windows to Amazon Linux. However, Amazon Linux is not a supported platform for running .net core by Microsoft. Although there is a Amazon Linux 2 image with .net core 2.1 preinstalled and it is possible to install the CentOS version of .net core on Amazon Linux 1, I went on a journey to build and test .net core on Amazon Linux to have confidence that my product will not hit a wall.

.net core require LLVM 3.9 to build. However, we can only get LLVM 3.6.3 from the yum repository. So we have to build LLVM 3.9.LLVM 3.9 requires Cmake 3.11 or later, but we can only get Cmake 2.8.12 from the yum repository. So we have to start from building CMake.

Building CMake

The procedure to build CMake can be found in https://askubuntu.com/questions/355565/how-do-i-install-the-latest-version-of-cmake-from-the-command-line.

Here is what I did:

sudo yum groupinstall "Development Tools"

Sudo yum install swig python27-devel libedit-devel

version=3.11
build=1
mkdir ~/temp
cd ~/temp
wget https://cmake.org/files/v$version/cmake-$version.$build.tar.gz
tar -xzvf cmake-$version.$build.tar.gz
cd cmake-$version.$build/

./bootstrap

make -j4
sudo make install

Building CLang and LVVM

With CMake installed, we can build LLVM. My procedure of building Clang and LLVM is similar to the procedure in https://github.com/dotnet/coreclr/blob/master/Documentation/building/buildinglldb.md.

Please also refer to https://releases.llvm.org/3.9.1/docs/CMake.html for additional information.

cd $HOME
git clone http://llvm.org/git/llvm.git
cd $HOME/llvm
git checkout release_39
cd $HOME/llvm/tools
git clone http://llvm.org/git/clang.git
git clone http://llvm.org/git/lldb.git
cd $HOME/llvm/tools/clang
git checkout release_39
cd $HOME/llvm/tools/lldb
git checkout release_39

Before we start building, we need to patch LLVM source code for Amazon Linux triplet.Otherwise LLVM cannot find the c++ compiler on Amazon Linux.

To patch, find file ./tools/clang/lib/Driver/ToolChains.cpp, find an array that looks like:

"x86_64-linux-gnu", "x86_64-unknown-linux-gnu", "x86_64-pc-linux-gnu",
      "x86_64-redhat-linux6E", "x86_64-redhat-linux", "x86_64-suse-linux",
      "x86_64-manbo-linux-gnu", "x86_64-linux-gnu", "x86_64-slackware-linux",
      "x86_64-linux-android", "x86_64-unknown-linux"

Append "x86_64-amazon-linux" to the last line.

Similar, append "i686-amazon-linux" to "i686-montavista-linux", "i686-linux-android", "i586-linux-gnu"

Now we can build:

mkdir -p $HOME/build/release
cd $HOME/build/release
cmake -DCMAKE_BUILD_TYPE=release $HOME/llvm

make –j4

sudo make install

Building CoreCLR and CoreFx

With Clang/LLVM 3.9 installed, we can now build CoreCLR and CoreFx.

https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md

We need to install the prerequisites first:

sudo yum install lttng-ust-devel libunwind-devel gettext libicu-devel libcurl-devel openssl-devel krb5-devel libuuid-devel libcxx

sudo yum install redhat-lsb-core cppcheck sloccount

mkdir ~/git

git clone https://github.com/dotnet/coreclr.git
git clone https://github.com/dotnet/corefx.git

Go to each directory and check out a version, for eample:

git checkout tags/v2.0.7

Now just follow https://github.com/dotnet/coreclr/blob/master/Documentation/building/linux-instructions.md to the build.

./clean.sh -all
./build.sh -RuntimeOS=linux
./build-tests.sh

Also look at: https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md and

https://github.com/dotnet/corefx/issues/22509

Conclusions

With the steps above, I was able to build and test .net core on Amazon Linux 1 and 2.

Note that .net core requires GLIBC_2.14 to run. To find the version of GLIBC on your version of Amazon Linux, run:

strings /lib64/libc.so.6 | grep GLIBC

If you don’t see 2.14 on the list, .net core will not run. try “sudo yum update” to see if you can update to a later version of GLIBC.

Additionally, since many newer programming languages were build on LLVM, this exercise also allow us to build other languages that require newer version of LLVM than the version in the yum repository.

Configure Open Live Writer to weblogs.asp.net

Sun, 12 Aug 2018 01:32:27 GMT

I have not blogged for a while. When I opened my Open Live Writer, I got error with https://weblogs.asp.net/lichen/xmlrpc.

I searched the web. Most blogs were still referencing the xmlrpc url which no longer exists.

Fortunately, Fixing is easy. Just Add Account and select “Other services”

On the next screen, enter the url of the blog (without xmlrpc).

Open Live Writer and Orchard are smart enough to figure out the rest. This is certainly an improvement over the earlier versions.

If you are curious on how Open Live Writer figured out the post API endpoint, which view the source of your web page and you will see the following lines in the header:

Top k algorithm revisited

Wed, 08 Mar 2017 07:08:23 GMT

3 years ago, I implemented top K operator in LINQ. I was asked recently why I chose Min Heap since there are faster algorithms. To recap, we try to select top k element from a sequence of n elements. A min-heap has the following property:

find-min takes O(1) time.
extract-min takes O(ln k) time where k is the size of the heap.
insert takes O(ln k) time.

For each number in the sequence, I first compare the number to find-min. If the number is smaller, the number is tossed away. If the number is bigger, we do a extract-min followed by an insert. So in the worst scenario, the algorithm runs with time complexity of O(n ln k) and the space complexity of O(k).

If we use max-heap instead, we can heapify n elements in O(n) time. Then we do k extract-max so we have the total time complexity of O(n + k ln n) and a space complexity of O(n).

We could also use Quick Select. It is very similar to Quick Sort that we randomly select a pivot and move it to the right position. Unlike Quick Sort, we can discard the left side of the pivot whenever we have greater then k elements on the right side. This algorithm converge fairly quickly and we have the average time complexity of O(n) and space complexity of O(n). In average case, the space requirement by Quick Select is less than the max heap approach.

So both max-heap and quick select are likely faster than the min-heap approach. Why do I used min-heap then? The reason is that the min-heap approach uses minimum amount of memory and I assume that I will work with large dataset so . Also, if we work with a stream, the min-heap provides a running top k.

Ever wonder on which platform Amazon AWS Lambda in C# is running?

Tue, 14 Feb 2017 07:17:00 GMT

In last December, AWS announced C# support for AWS Lambda using .NET Core 1.0 runtime. Ever wonder on which platform is it running? I am curious too and I did not see it in any official documentation. So I decided to write a small AWS Lambda function to detect the platform:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using Amazon.Lambda.Core;
using Amazon.Lambda.Serialization;
 
// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializerAttribute(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]
 
namespace SysInfoLambda
{
    public class Function
    {
 
        /// <summary>
        /// A simple function that takes a string and does a ToUpper
        /// </summary>
        /// <param name="input"></param>
        /// <param name="context"></param>
        /// <returns></returns>
        public RuntimeInfo FunctionHandler(ILambdaContext context)
        {
            return new RuntimeInfo()
            {
                FrameworkDescription = RuntimeInformation.FrameworkDescription,
                OSArchitecture = RuntimeInformation.OSArchitecture,
                ProcessArchitecture = RuntimeInformation.ProcessArchitecture,
                OSDescription = RuntimeInformation.OSDescription,
                OSPlatform = RuntimeInformation.IsOSPlatform(OSPlatform.Linux) ? OS.Linux :
                    RuntimeInformation.IsOSPlatform(OSPlatform.OSX) ? OS.OSX : OS.Windows
            };
        }
 
    }
 
    public class RuntimeInfo
    {
        public string FrameworkDescription { get; set; }
        public Architecture OSArchitecture { get; set; }
        public string OSDescription { get; set; }
        public Architecture ProcessArchitecture { get; set; }
        public OS OSPlatform { get; set; }
    }
 
    public enum OS
    {
        Linux,
        OSX,
        Windows
    }
 
}

The result? The AWS C# Lambda runs in 64 bit Linux. The extract OS description is: Linux 4.4.35-33.55.amzn1.x86_64 #1 SMP Tue Dec 6 20:30:04 UTC 2016.

NSession is quietly marching towards 1.0 release

Sun, 12 Apr 2015 21:21:29 GMT

My open source project NSession has be around for several years. It allows ASP classic pages to access ASP.NET out-of-process session states the same way that ASP.NET pages access them. Recently, there are a large number of users migrating from Windows 2003 to Windows Server 2008/2012 and encountered an incompatibility with ASP.NET 4.5. I was contacted by users daily publicly or privately. I decided to make additional effort to put finishing touch on it to make it a 1.0 product.

In 0.9.0.1 release, I had:

Fixed the compatibility issues with asp.net 4.5.
Improved the documents.
Reduced hacking to make it less likely to encounter compatibility problem with future versions of ASP.NET.

In 0.9.1.2 release, I had:

Added support for SQL Server session state so that it supports all the built-in out-of-proc session states in ASP.NET and is feature complete for 1.0 release.
Added a diagnostic check list and a diagnostic tool.

Before 1.0 release, I will continue improving installation experience, fixing bugs reported and will do some performance testing.

Beyond 1.0, I will start designing adapter interface to support custom session states. Please visit the project site to propose custom session state providers to support in the work items area.

First look at the Visual Studio Tools for Apache Cordova CTP 3.1

Wed, 25 Mar 2015 23:17:19 GMT

The company that I worked for had an old cross-platform mobile app developed by an outside contractor using PhoneGap 1.0. When I was asked to look at the app a few months ago, I had great difficulty collecting large number of moving pieces: PhoneGap, Android SDK and emulator. When I saw Visual Studio Tools for Apache Cordova (I will call it VSTAC in the remaining of this post), I decide to give it a try since it attempts to install a large number of third-party software for me.

The journey is not exactly easy, but it is certainly far easier than collecting all the pieces myself with the excellent installation document from MSDN. The result is more than pleasant. Here are some of my findings:

1) After the installation, I could not even get a hello world app to work. It turns out that I had an older version of Nods.js. VSTAC skipped node.js installation. After I uninstall the old node.js and reinstall with the one linked from the VSTAC installation page, I was able to get hello world to work.

2) I was surprise to see the Ripple emulator which I was not aware of previously. The Ripple emulator is very fast and VSTAC provides excellent debugging experience.

3) I had to clear my Apache Cordova cache a few times. This and some other useful items are documented in FAQ. Also visit known issues.

4) The application connects to an old soap web services developed with WCF. It does not support CORS. So I had to use Ripple proxy to connect to it but I kept getting 400 error. Fortunately, I was able to hack Ripple proxy to make it work.

5) I then tried to run the app in Google Android emulator. VSTAC supports this scenario as well. I had to uninstall and reinstall some Android SDK components following this and this directions. Then I had to run AVD Manager to create and start a device. Then I had to update my display driver to make sure I have compatible OpenGL ES driver installed. After that, the Google emulator ran beautifully. It was not as fast as Ripple but is acceptable.

So at the end, I want to give a big thank you to the Microsoft VSTAC team. I know this is not easy but the excellent document got me through. It certainly saved me lots of time.

Missing methods in LINQ: MaxWithIndex and MinWithIndex

Sun, 05 Oct 2014 22:50:01 GMT

The LINQ library has Max methods and Min methods. However, sometimes we are interested in the index location in the IEnumerable<T> rather than the actual values. Hence the MaxWithIndex and MinWithIndex methods.

These methods return a Tuple. The first item of the Tuple is the maximum or minimum value just the the Max and Min methods. The second item of the Tuple is the index location.

As usually, you might get my LINQ extension from NuGet:

PM>Install-Package SkyLinq.Linq

Usage examples in the unit test.

ASP Classic Compiler is now available in NuGet

Sun, 28 Sep 2014 19:06:00 GMT

I know this is very, very late, but I hope it is better than never. To make it easy to experiment with ASP Classic Compiler, I made the .net 4.x binaries available in NuGet. So it is now extremely easy to try it:

From the package console of any .NET 4.x web project, run “Install-Package Dlrsoft.Asp”.

To switch from AspClassic to Asp Classic Compiler in the project, add the following section to the system.webServer handlers section:

  <system.webServer>
    <handlers>
      <remove name="ASPClassic"/>
      <add name="ASPClassic" verb="*" path="*.asp" type="Dlrsoft.Asp.AspHandler, Dlrsoft.Asp"/>
    </handlers>
  </system.webServer>

Comment out the section to switch back.

Add a test page StringBuilder.asp:

<%
    imports system
    dim s = new system.text.stringbuilder()
    dim i
    
    s = s + "<table>"
    for i = 1 to 12
        s = s + "<tr>"
        s = s + "<td>" + i + "</td>"
        s = s + "<td>" + MonthName(i) + "</td>"
        s = s + "</tr>"
    next
    s = s + "</table>"
    response.Write(s)
 
%>

This code uses the .net extension so it will only work with Asp Classic Compiler.

Happy experimenting!

SkyLinq binaries are available on NuGet

Sat, 13 Sep 2014 20:30:39 GMT

After much hesitate, I finally published my SkyLinq binaries on NuGet. My main hesitation was that this is my playground so I am changing things at will. The main reason to publish is that I want to use these works myself so I need an easy way to get the latest binaries into my projects. NuGet is the easiest way to distribute and get updates, including my own projects. There are 3 packages:

SkyLinq.Linq is a portal library that contains some LINQ extensions.
SkyLinq.Composition contains my duck-typing implementation. It is similar to Impromptu-Interface but it is much simpler and it uses il.emit instead of LINQ Expressions to generate code. It also contains a LINQ query rewriting example.
LINQPadHost is a simple hosting and executing environment for LINQPad queries. Live demo at http://skylinq.azurewebsites.net/SkyLINQPad.

Sky LINQPad, a minimum viable clone of LINQPad in the cloud

Sat, 13 Sep 2014 20:09:00 GMT

A while ago, I blogged about a simple LINQPad query host. It is fairly easy to put a web face on it. The only change that I had to make is to set the ApplicationBase for the AppDomains that I create as asp.net is quite different to an .exe app. A playground is now running at http://skylinq.azurewebsites.net/SkyLINQPad. One can upload an existing .linq files designed in LINQPad or type some queries directly into the page:

A simple LINQPad query host

Tue, 02 Sep 2014 04:00:45 GMT

I am a big fan of LINQPad. I use LINQPad routinely during my work to test small, incremental ideas. I used it so much so that I bough myself a premium license.

I always wish I can run queries designed in LINQPad in my own program. Before 4.52.1 beta, there was only a command line interface. In LINQPad v4.52.1 beta, there is finally a Util.Run method that allows me to run LINQPad queries in my own process. However, I felt that I did not have sufficient control on how I can dump the results. So I decided to write a simple host myself.

As in the example below, a .linq file starts with an xml meta data section followed by a blank line and then the query or the statements.

<Query Kind="Expression">
  <Reference>&lt;RuntimeDirectory&gt;\System.Web.dll</Reference>
  <Reference>&lt;ProgramFilesX86&gt;\Microsoft ASP.NET\ASP.NET MVC 4\Assemblies\System.Web.Mvc.dll</Reference>
  <Namespace>System.Web</Namespace>
  <Namespace>System.Web.Mvc</Namespace>
</Query>
 
HttpUtility.UrlEncode("\"'a,b;c.d'\"")

The article “http://www.linqpad.net/HowLINQPadWorks.aspx” on the LINQPad website gives me good information on how to compile and execute queries. LINQPad uses CSharpCodeProvider (or VBCodeProvider) to compile queries. Although I was tempted to use Roslyn like ScriptCS, I decided to use CSharpCodeProvider to ensure compatible with LINQPad.

We only need 3 lines of code to the LINQPad host:

using LINQPadHost;
...
string file = @"C:\Users\lichen\Documents\LINQPad Queries\ServerUtility.linq";
Host host = new Host();
host.Run<JsonTextSerializer>(file);

As I mentioned at the beginning. I would like to control the dumping of the results. JsonTextSerializer is one of the three serializers that I supplied. The other two serializers are IndentTextSerializer and XmlTextSerializer. Personally, I found that the JsonTextSerializer and IndentTextSerializer the most useful.

The source code could be found here.

Examples could be found here.

Why every .net developer should learn some PowerShell

Sun, 17 Aug 2014 18:30:24 GMT

It has been 8 years since PowerShell v.1 was shipped in 2006. I have looked into PowerShell closely except for using it in the Nuget Console. Recently, I was forced to have a much closer look at PowerShell because we use a product that exposes its only interface in PowerShell.

Then I realized that PowerShell is such a wonderful product that every .net developer should learn some. Here are some reasons:

PowerShell is a much better language that the DOS batch language. PowerShell is real language with variable, condition, looping and function calls.
According to Douglas Finke in Windows Powershell for Developers by O’Reilly, PowerShell is a stop ship event, meaning no Microsoft server products ship without a PowerShell interface.
PowerShell now has a pretty good Integrated Scripting Environments (ISE). We can create, edit, run and debug PowerShell. Microsoft has release OneScript, a script browser and analyzer that could be run from PowerShell ISE.
We can call .NET and COM objects from PowerShell. That is an advantage over VBScript.
PowerShell has a wonderful pipeline model with which we can filter, sort and convert results. If you love LINQ, you would love PowerShell.
It is possible to call PowerShell script from .net, even ones on a remote machine.

Recently, I have to call some PowerShell scripts on a remote server. There are many piecewise information on the internet, but no many good examples. So I put a few pointers here:

When connecting to remote PowerShell, the uri is : http://SERVERNAME:5985/wsman.
It is possible to run PowerShell in a different credential using the optional credential.
Powershell remoting only runs in PowerShell 2.0 or later. So download the PowerShell 2.0 SDK (http://www.microsoft.com/en-us/download/details.aspx?id=2560). When installed, it actually updates the 1.0 reference assemblies . On my machine, they are in: C:\Program Files (x86)\Reference Assemblies\Microsoft\WindowsPowerShell\v1.0

So the complete code runs like:

using System.Management.Automation; // Windows PowerShell namespace
using System.Management.Automation.Runspaces; // Windows PowerShell namespace
using System.Security; // For the secure password
using Microsoft.PowerShell;

	    Runspace remoteRunspace = null;
            //System.Security.SecureString password = new System.Security.SecureString();
            //foreach (char c in livePass.ToCharArray())
            //{
            //    password.AppendChar(c);
            //}
            //PSCredential psc = new PSCredential(username, password);
            //WSManConnectionInfo rri = new WSManConnectionInfo(new Uri(uri), schema, psc);
            WSManConnectionInfo rri = new WSManConnectionInfo(new Uri(""http://SERVERNAME:5985/wsman"));
            //rri.AuthenticationMechanism = AuthenticationMechanism.Kerberos;
            //rri.ProxyAuthentication = AuthenticationMechanism.Negotiate;
            remoteRunspace = RunspaceFactory.CreateRunspace(rri);
            remoteRunspace.Open();

	    using (PowerShell powershell = PowerShell.Create())
                {
                    powershell.Runspace = remoteRunspace;
                    powershell.AddCommand(scriptText);
                    Collection results = powershell.Invoke();
                    remoteRunspace.Close();
                    foreach (PSObject obj in results)
                    {
                        foreach (PSPropertyInfo psPropertyInfo in obj.Properties)
                        {
                            Console.Write("name: " + psPropertyInfo.Name);
                            Console.Write("\tvalue: " + psPropertyInfo.Value);
                            Console.WriteLine("\tmemberType: " + psPropertyInfo.MemberType);
                        }
                    }
                }

Implement a simple priority queue

Sun, 27 Jul 2014 03:24:02 GMT

.NET framework does not have a priority queue built-in. There are several open source implementations. If you do not want to reference an entire library, it is fairly easy to implement one yourself. Many priority queue implementations use heap. However, if the number of levels of priorities is small, it is actually very easy and efficient to implement priority queues using an array of queues. There is a queue implementation in the .net framework.

My implementation is in the code below. The enum QueuePriorityEnum contains the number of levels of priorities. It is picked up automatically by the PriorityQueue class. The queue support 3 operations: Enqueue, Dequeue and Count. There behavior is modeled after the Queue class in the .net framework.

using System;
using System.Collections.Generic;
using System.Linq;

namespace MyNamespace
{
    // Modify this enum to add number of levels. It will picked up automatically
    enum QueuePriorityEnum
    {
        Low = 0,
        High =1
    }

    class PriorityQueue<T>
    {
        Queue<T>[] _queues;

        public PriorityQueue()
        {
            int levels = Enum.GetValues(typeof(QueuePriorityEnum)).Length;
            _queues = new Queue<T>[levels];
            for (int i = 0; i < levels; i++)
            {
                _queues[i] = new Queue<T>();
            }
        }

        public void Enqueue(QueuePriorityEnum priority, T item)
        {
	    _queues[(int)priority].Enqueue(item);
        }

        public int Count
        {
            get
            {
                return _queues.Sum(q => q.Count);
            }
        }

        public T Dequeue()
        {
	    int levels = Enum.GetValues(typeof(QueuePriorityEnum)).Length;
	    for (int i = levels - 1; i > -1; i--)
	    {
		if (_queues[i].Count > 0)
		{
			return _queues[i].Dequeue();
		}
	    }
            throw new InvalidOperationException("The Queue is empty. ");
        }
    }
}

integrating external systems with TFS

Sat, 19 Jul 2014 04:56:00 GMT

Recently, we need to integrate external systems with TFS. TFS is a feature-rich system and has a large API. The TFS sample on MSDN only scratch the top surface. Fortunately, a couple of good blog posts get me on the write direction:

The key is to use the VersionControlService. We need to reference the following assemblies:

Microsoft.TeamFoundation.Client.dll

Microsoft.TeamFoundation.Common.dll

Microsoft.TeamFoundation.VersionControl.Client.dll

Microsoft.TeamFoundation.VersionControl.Common.dll

The code would be something like:

using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.Framework.Common;
using Microsoft.TeamFoundation.Framework.Client;
using Microsoft.TeamFoundation.VersionControl.Client;

...

TfsTeamProjectCollection pc = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(tfsUri);
VersionControlServer vcs = pc.GetService();

Then to check whether a file exists, we can use:

vcs.ServerItemExists(myPath, ItemType.File)

Or check if a directory exists:

vcs.ServerItemExists(myPath, ItemType.Folder)

To get a list of directories or files, we can use the GetItems method. TFS is far more complicated than a file system. We can get a file, get the history of a file, get a changeset, etc. Therefore, the GetItems method has many overloads. To get a list files, we can use:

var fileSet = vcs.GetItems(myPath, VersionSpec.Latest, RecursionType.OneLevel, DeletedState.NonDeleted, ItemType.File);
foreach (Item f in fileSet.Items)
{
    Console.WriteLine(f.ServerItem);
}

Or get a list of directories:

var dirSet = vcs.GetItems(myPath, VersionSpec.Latest, RecursionType.OneLevel, DeletedState.NonDeleted, ItemType.Folder);
foreach (Item d in dirSet.Items)
{
    Console.WriteLine(d.ServerItem);
}

Converted ASP Classic Compiler project from Mercurial to Git

Sun, 30 Mar 2014 19:09:29 GMT

Like some other open source project developers, I picked the Mercurial as my version control system. Unfortunately, Git is winning in the Visual Studio echo systems. Fortunately, it is possible to contact Codeplex admin for manual conversion from Mercurial to Git. I have done exactly that for my open source ASP Classic Compiler project. Now I can add new examples in response to forum questions and check them in using my Visual Studio 2013. Now I am all happy.

Video review: RESTful Services with ASP.NET Web API from PACKT Publishing

Sat, 22 Mar 2014 20:49:17 GMT

Disclaimer: I was provided this video for free by PACKT Publishing. However, that does not affect my opinion about this video.

Upon request by PACKT Publishing, I agreed to watch and review the video “RESTful Service with ASP.NET Web API” by Fanie Reynders. Prior to the review, I have a EBook called “Designing Evolvable Web APIs with ASP.NET - Harnessing the power of the web” by Glenn Block, Pablo Cibraro, Pedro Felix, Howard Dierking and Darrel Miller from O’Reilly. I also have access to several videos on the Pluralsight.com on the same subject. So I would put my review in perspective with those other materials.

The video from Packt has the length of 2 hours 4 minutes. It gave a nice overview over the ASP.NET Web API. The video is available for watch online or for downloading to watch offline. The video has 8 chapters. It covers the ASP.NET Web API in a clear and accurate way and is surprisingly complete for this short length.

In comparison with other resources, I would recommend you get this video if you have never worked with ASP.NET Web API before and want to get a complete overview in a short time possible.

If you love video, Pluralsight is offering the equally good and slightly longer (3h 15m) “Introduction to the ASP.NET Web API” by Jon Flanders. You would need subscription to access the exercise materials thought. If you do subscribe, Pluralsight also has a couple of intermediate level videos by Shawn Wildermuth.

Lastly, if you want an in depth book that you can use for a extended period of time, it is hard to beat Glenn Block, etc.’s book from O’Reilly.

LINQ query optimization by query rewriting using a custom IQueryable LINQ provider

Fri, 14 Feb 2014 04:23:00 GMT

In my SkyLinq open source project so far, I tried to make LINQ better and created several extension methods (for example, see GroupBy and TopK) that are more memory efficient than the standard methods in some scenarios. However, I do not necessarily want people to bind into these extensions methods directly for the following reasons:

Good ideas evolve. New methods may come and existing methods may change.
There is a parallel set of methods in IEnumerable<T>, IQueryable<T> and to some degree, in IObservable<T>. Using custom LINQ extension breaks the parallelism.

A better approach would be to write the LINQ queries normally using the standard library methods and then optimize the queries using a custom query provider. Query rewriting is not a new idea. SQL server has been doing this for years. I would like to bring this idea into LINQ.

If we examine the IQueryable<T> interface, we will find there is a parallel set of LINQ methods that accepts Expression<Func> instead of <Func>.

There is a little C# compiler trick here. If C# see a method that accepts Expression<Func>, it would create Expressions of Lamda instead of compiled Lambda expression. At run time, these expressions are passed to an IQueryable implementation and then passed to the underlying IQueryProvider implementation. The query provider is responsible for executing the expression tree and return the results. This is how the magic of LINQ to SQL and Entity Frameworks works.

The .net framework already has a class called EnumerableQuery<T>. It is a query provider than turns IQueryable class into LINQ to objects queries. In this work, I am going one step further by creating an optimizing query provider.

A common perception of writing a custom LINQ provider is that it requires a lot of code. The reason is that even for the most trivial provider we need to visit the every possible expressions in the System.Linq.Expressions namespace. (Note that we only need to handle expressions as of .net framework 3.5 but do not need to handle the new expressions added as part of dynamic language runtime in .net 4.x). There are reusable framework such as IQToolkit project that makes it easier to create a custom LINQ provider.

In contrast, creating an optimizing query provider is fairly easy. The ExpressionVistor class already has the framework that I need. I only needed to create a subclass of ExpressionVisitor called SkyLinqRewriter to override a few methods. The query rewriter basically replaces all call to the Queryable class with equivalent calls to the Enumerable class and rewrite queries for optimization when an opportunity presents.

It is fairly easy to consume the optimizing query provider. All we need is to call AsSkyLinqQueryable() to convert IEnumerable<T> to IQueryable<T> and remaining code can stay intact:

An end-to-end example can be found at https://skylinq.codeplex.com/SourceControl/latest#SkyLinq.Example/LinqToW3SVCLogExample.cs.

To conclude this post, I would recommend that we always code to IQueryable<T> instead of IEnumerable<T>. This way, the code has the flexibility to be optimized by an optimizing query provider, or be converted from pull based query to push based query using Reactive Extension without rewriting any code.

Expression tree visualizer for Visual Studio 2013

Mon, 10 Feb 2014 00:27:02 GMT

Expression tree visualizer, as the name indicates, is a Visual Studio visualizer for visualizing expression trees. It is a must if you work with expressions frequently. Expression Tree Visualizer is a Visual Studio 2008 sample. There is a Visual Studio 2010 port available on codeplex. If you want to use it with a later version of Visual Studio, there is not one available. Fortunately, porting it to another version of Visual Studio is fairly simple:

Download the original source code from http://exprtreevisualizer.codeplex.com/.
Replace the existing reference to Microsoft.VisualStudio.DebuggerVisualizers assembly to the version in the Visual Studio you want to work with. For Visual Studio 2013, I found it in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\ReferenceAssemblies\v2.0\Microsoft.VisualStudio.DebuggerVisualizers.dll on my computer.
Compile the ExpressionTreeVisualizer project and copy ExpressionTreeVisualizer.dll to the visualizer directory. To make it usable by one user, just copy it to My Documents\VisualStudioVersion\Visualizers. To make it usable by all users of a machine, copy it to VisualStudioInstallPath\Common7\Packages\Debugger\Visualizers.

If you do not want to walk through the process, I have one readily available at http://weblogs.asp.net/blogs/lichen/ExpressionTreeVisualizer.Vs2013.zip. Just do not sue me if it does not work as expected.

An efficient Top K algorithm with implementation in C# for LINQ

Wed, 15 Jan 2014 05:39:00 GMT

The LINQ library currently does not have a dedicated top K implementation. Programs usually use the OrderBy function followed by the Take function. For N items, an efficient sort algorithm would scale O(n) in space and O(n log n) in time. Since we are counting the top K, I believe that we could devise an algorithm that scales O(K) in space.

Among all the sorting algorithms that I am familiar, the heapsort algorithm came to my mind first. A Heap has a tree-like data structure that can often be implemented using an array. A heap can either be a max-heap or a min-heap. In the case of min-heap, the minimum element of at the root of the tree and all the child elements are greater than their parents.

Binary heap is a special case of heap. A binary min-heap has the following characteristics:

find-min takes O(1) time.
delete-min takes O(log n) time.
insert takes O(log n) time.

So here is how I implement my top K algorithm:

Create a heap with an array of size K.
Insert items into the heap until the heap reaches its capacity. This takes K O(log K) time.
For each the remaining elements, if an element is greater than find-min of the heap, do a delete-min and then insert the element into the heap.
Then we repeated delete-min until the heap is empty and arrange the deleted element in reverse order and we get our top 10 list.

The time it takes to find top K items from an N item list is:

O(N) * t1 + O((K + log N - log K) * log K) * t2

Here t1 is the time to compare an element in step 3 to find-min and t2 is the combined time of delete-min and the subsequent insert. So this algorithm is much more efficient that a call to OrderBy followed by a call to Take.

I have checked-in my implementation to my Sky LINQ project. The LINQ Top and Bottom is at in LinqExt.cs. The internal binary heap implementation is in BinaryHeap.cs. The LINQ example can be found in HeapSort.cs. The Sky Log example has also been updated to use the efficient Top K algorithm.

Note that this algorithm combined OrderBy and Take by GroupBy itself still uses O(N) splace where N is number of groups. There are probabilistic approximation algorithms that can be used to further alleviate memory foot print. That is something I could try in the future.