As you can imagine, I always love to find new functionalities related to HttpClient, and new ways to improve my work with it.

In his blog, Byte Rot, @aliostad shows a way to monitor HttpClient requests using PerformanceCounters, via a DelegatingHandler. Although I refrain from using PerformanceCounters in my code, due to some bad experience with them, I do like his clean and simple approach. The use of DelegatingHandler provides a nice separation of concerns, and I can easily see someone use it with my OneDrive REST API library.

A few days ago, I spent some time refining some old code, and encountered something along the following lines:

function report(reportData) {
    var self = reportData;
    self.StartedText = friendlyPastMoment(reportData.Started);
    if (reportData.Ended == "0001-01-01T00:00:00") {
        self.EndedText = 'Still running...';
    }
    else {
        self.EndedText = friendlyPastMoment(reportData.Ended);
    }

    return self;
}

(the friendlyPastMoment() method is just a small wrapper method around MomentJS, for handling edge cases)

We had a couple more properties handled similarly. Now, this worked just fine when the information displayed to the end-user was static – we just added a few text binding to a couple of span nodes, and the StartedText and EndedText were displayed beautifully.

But we’ve decided that we want to updated the information displayed to the user automatically, which meant that I now had to replace those simple StartedText and EndedText properties into Knockout computed properties. But for that to work, I had to change the implementation – the Started and Ended properties of the report need to be observable properties.

I’ve started with a naïve implementation:

function report(reportData) {
    var self = reportData;
    self.Started = ko.observable(reportData.Started);
    self.Ended = ko.observable(reportData.Ended);

    self.StartedText = ko.computed(function() { return friendlyPastMoment(self.Started()) });
    self.EndedText = ko.computed(function() { 
        if (self.Ended() == "0001-01-01T00:00:00") {
            return 'Still running...';
        }
        else {
            return friendlyPastMoment(self.Ended());
        }});

    return self;
}

But since we had several properties in reportData that needed to implement this behavior, this started to become a code-smell. I was certain that there must be a better way for handling these kind of scenarios.

The Knockout-Mapping plugin was created exactly for such cases – it convert a simple POCO object into its Knockout equivalent, with all properties implemented as observable properties, with arrays and all:

function report(reportData) {
    ko.mapping.fromJS(reportData, {}, this);

    var self = this;

    self.StartedText = ko.computed(function() { return friendlyPastMoment(self.Started()) });
    self.EndedText = ko.computed(function() { 
        if (self.Ended() == "0001-01-01T00:00:00") {
            return 'Still running...';
        }
        else {
            return friendlyPastMoment(self.Ended());
        }});

    return self;
}

As you can see, the first line calls the fromJS method, which takes a regular JavaScript object, an options object (empty {} in the line above), and the destination object. The result will be that this object will have the same properties as reportData, implemented as Knockout observable properties.

Now, it is very easy to add a refresh function to report, which will retrieve the most current information from the server, and update the report:

    self.refresh = function() {
        $.getJSON("api/report?id=" + self.Id(), function (data) {
            ko.mapping.fromJS(data, self);
        });
    };

As you can see, we’re making a getJSON call to the server, and once we get the data back, we just call the fromJS method again, this time passing the data retrieved from the server, and our current report instance.

As a first step, and in preparation for a future move, I’ve purchased my own domain, saguiitay.com. This will be the primary domain going forward (you will be automatically redirected if you reach hear using the old domain – itaysagui.wordpress.com).

In the near future, I’ll decide on which platform to use. I’m tending towards using Jekyll with Github Pages for hosting the pages and images.

LibraryThing provide several APIs, one of them a RESTful API, which can be invoked from C# relatively easily. In this post I’ll describe how to use that API.

First, you need to get a Developer Key. You can do that here: http://www.librarything.com/services/keys.php

Here’s the most important thing you need:

private const string CollectionByUser = "http://www.librarything.com/api_getdata.php?userid={0}&resultsets=books&max=9999&&booksort=title&responseType=json&key=" + DevKey;
private const string WorkUrlByISBN = "http://www.librarything.com/services/rest/1.1/?method=librarything.ck.getwork&isbn={0}&apikey=" + DevKey;
private const string WorkUrlById = "http://www.librarything.com/services/rest/1.1/?method=librarything.ck.getwork&id={0}&apikey=" + DevKey;
private const string WorkUrlByLCCN = "http://www.librarything.com/services/rest/1.1/?method=librarything.ck.getwork&lccn={0}&apikey=" + DevKey;
private const string WorkUrlByOCLC = "http://www.librarything.com/services/rest/1.1/?method=librarything.ck.getwork&oclc={0}&apikey=" + DevKey;
private const string AuthorUrlByCode = "http://www.librarything.com/services/rest/1.1/?method=librarything.ck.getauthor&authorcode={0}&apikey=" + DevKey;
private const string ImageUrl = "http://covers.librarything.com/devkey/" + DevKey + "/{0}/isbn/{1}";

These URLs allow you to retrieve information from LibraryThing, by providing a Developer Key, and the details about which data you want to get:

• CollectionByUser: Retrieve the list of books that a user has added in his collection. Notice that the URL returns the “books” result-set (you can also get the users’ ratings, reviews and tags).
• WorkUrlByISBN: Retrieve information about a specific work (book), provided the book’s ISBN.
• WorkUrlById: Retrieve information about a specific work (book), provided the book’s LibraryThing ID.
• WorkUrlByLCCN: Retrieve information about a specific work (book), provided the book’s LCCN (Library of Congress Control Number).
• WorkUrlByOCLC: Retrieve information about a specific work (book), provided the book’s OCLC (Online Computer Library Center).
• AuthorUrlByCode: Retrieve information about a specific author, provided the author’s LibraryThing author-code.
• ImageUrl: Retrieve the cover image of a book, provided the book’s ISBN, and the requested image size (

Now, if you want to see some code, we’ll start by creating an HttpClient instance:

_handler = new HttpClientHandler();
if (_handler.SupportsAutomaticDecompression)
    _handler.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
_httpClient = new HttpClient(_handler);

To get a book’s details, we’ll make the following call to the LibraryThing API:

var url = string.Format(WorkUrlByISBN, isbn);
string content;
try
{
    var response = await _httpClient.GetAsync(url).ConfigureAwait(false);
    if (!response.IsSuccessStatusCode)
        return null;
 
    content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
}
catch (Exception)
{
    return null;
}

You can then parse the result into a workable object model. The model is a bit complex, so I’ve posted it as Gist: https://gist.github.com/saguiitay/be3868dae2efb12802a2

One method in that Gist requires some explanation – the parsing of the Xml returned from LibraryThing is not straight forward – you need to “decode” it, in order for some Xml entities to be processed correctly. For that, I’ve created an extension method for String (https://gist.github.com/saguiitay/be3868dae2efb12802a2#file-stringextensions-cs), that replaced the XmlEntities into their equivalent character.

All in all, LibraryThing provides an extensive API to their data, which is mostly painless to work with, and provide detailed and valuable information.

You can see the results of using LibraryThing to retrieve information about books and authors in my Windows Phone App “Library”, which you can download here: http://www.windowsphone.com/s?appid=01f350f2-01d1-4210-a83b-9874b71e9496

After downloading NDepend, my initial reaction to their “installation” process was REALLY?! you expect me to manually copy the files, and don’t include an installation/setup?! After ignoring the instructions, and copying everything to my Program Files folder, I’ve decided to give this baby a shot.

I’ve heard of NDepend for a long time now, and even played a bit (5-10 minutes top) with an older version in the past. I was interesting to see what it can do for me in my day-to-day job, so I started to visual tool, and pointed it to the source code of our main product. That was a mistake. This is a 50-ish projects solution, with dependencies from here till tomorrow. Don’t get me wrong, NDepend handled it without any problems (not an easy feat – even ReSharper is having a tough time). My problem was with the sheer amount of information it threw at me – I didn’t know where to start.

So I decided to take a step back, and decided to start small. I reran the tool, this time on my OneDrive REST API library. This time I got something a bit less scary when I’ve opened the dashboard:

As you can see, we got some statistics about the code, including lines of code/comments, numbers of assemblies/types/etc. as well as Complexity indicators. The most important part, at least for now, is the Code Rules.

Apparently, my library violates only a single rule, and only once. Clicking the link there, brings you to the “Queries and Rules Explorer”:

As you can see, NDepend comes with quite a lot of rules – I still need to explore these, but those that probably interest me the most would be “Code Quality” and “Design”. In this case, the critical rule that is violated is under the “Naming Conventions” category. There are some “old fashioned” rules in there, such as requiring private fields to start with “m_” – something I haven’t used or seen in the last few years – luckily you can edit these rules for yourself.

Going back to the critical rule, it appears that I have a type called “HttpException”. The name itself has nothing wrong with it, but System.Web already has such a type (my implementation of HttpException actually inherits from it). In this case, I’ve decided to rename my class to HttpServiceException.

I’ve then decided to move to the Code Quality category, and see what’s going on there. Things didn’t go so smoothly here. A couple of rules bothered me:

• Methods with too many overloads: My HttpException class has 8 constructors, all of which are just passing the arguments to the base class. Sure, I could probably reduce that number, but since this class was mostly auto-generated by ReSharper, and since these 8 constructors came from the .Net Framework itself, I didn’t see the point of changing the code.
• Methods too complex: I’m always happy to simplify my code, especially in an open source project like this one. However, NDepend pointed me to a couple of Async methods, and gave me the build-generated “MoveNext” method as the too-complex methods. Not a lot I can do here. Ever more bothering, is the fact that NDepend doesn’t appear to play-nicely with async-based code.

At t his point, I’ve decided to take a look at the other features of NDepend. I went back to my “monster” project. To be honest, the Code Rules violations scared the hell out of me. I know we don’t write perfect code, but 322 critical violations, 17700 violations total seems a bit harsh.

The Dependency Graph, after some clean-ups, showed me what I already knew about the solution – it’s built by several major areas – some common assemblies, which contains the model of the product, a group of several mostly-external “services”, a middle layer of concrete services, and lastly the actual “implementation” of the system (front-end, NServiceBus endpoints, etc.):

It is nice to see this visually, but it’s nothing I didn’t already know, from designing and building the system. I guess this might come handy when introducing new team members to the source code.

That’s it for now.

All in all, I did find NDepend a useful tool to get some insights into your code. I am still not sure how and if I’ll integrate it into my day-to-day workflow. The main feature that spiked my interest is the Code rules, but I still need to further review it, learn how to better use it, and see if it enhance my productivity in the long run.

After some back-and-forth with the RavenDb support team, I was suggested to compact the database. Sounds easy, if you know what you’re doing, since documentation on that is … let’s say lacking…

Here’s what you do, for my own future reference:

Using cUrl, do a POST request to http://<servername>/admin/compact?database=<DatabaseName>

Here’s a sample call:

cUrl -F 'database=UsersDatabase' http://localhost:8080/admin/compact?database=UsersDatabase

Notice, that if you’re using any form of authentication, you need to provide that too. Here’s how it’s done for NTLM Authentication:

cUrl --ntlm --user username:password -F 'database=UsersDatabase' http://localhost:8080/admin/compact?database=UsersDatabase

NOTICE: Your database will be off-line during the compact process, and won’t be accessible.

Over the last few days, I’ve implemented a .Net Class library to provide .Net access to OneDrive. Although Microsoft provides its own SDK (which is found at http://msdn.microsoft.com/en-US/live/ff621310), that library is not very developer friendly – it’s hard to debug, and harder to extend. I’ve search for alternatives on the internet, but the only usable solution I’ve found was John Hoerr‘s SkyNet project. Although useful as a start, I’ve encountered several issues with it:

• It didn’t support Async patterns
• It’s support for downloading/uploading file was limited (mostly because it relies on RestSharp)
• It’s Object Model wasn’t complete (some objects and methods were missing)
• It wasn’t updated in a while

I’ve decided to implement my own library. I’ve started with the SkyNet repository.

The first thing I did was replace all RestSharp code with simpler HttpClient (from the Microsoft.Net.Http NuGet). This meant that I had to implement a simplified version of the Request object. Nothing too complex – just enough to allow me to organize the code in a sensible manner. That immediately forced me to modify all method to Async, which was great, since I planned to do that anyway.

Then I’ve added a few models, such as Error and UserProfile, along with their relevant methods.

After that, came some extendibility requirements. The code that I’m currently work on (and which will use this Library) makes quite a few requests to OneDrive. I needed a way to throttling all these requests. For the moment, I implemented this by providing callers the ability to wrap the HttpMessageHandler that the HttpClient is using. Thus I’m able to use a TimeSpanSemaphore to throttle all requests.

I’ve also decided to implement auto-token-refresh. I’m using OneDriveRestAPI in a long-running process. That means that the AccessToken returned by OneDrive might expire during my processing. Using a RefreshToken, the library automatically retrieve an updated AccessToken (as long as the user didn’t revoke permissions on OneDrive).

You are of course welcome to send me any feedback, fork and repository, and send pull requests.

I’ll try to keep this up-to-date as much as possible.

Now, it’s time to query all the data.

RavenDb provides Map/Reduce indexes for querying the data. Let’s see some of the basic ones that interest us.

    Repositories/ByName
    from repo in docs.Repos
    select new {
        Name = repo.Name,
        FullName = repo.FullName
    }

    Packages/ByName
    from package in docs.Packages
    select new {
        Name = package.Name
    }

    Packages/ByTag
    from package in docs.Packages
    from tag in package.Tags
    select new {
        Tag = tag
    }

These are all simple indexes, which are there mostly for searching and viewing our data. Much more interesting is to see the relations between repositories and packages. Let’s look at an index that allows us to get a list of repositories that use a certain package. Here’s the index definition:

    Repositories/ByPackage
    from repo in docs.Repos
    from package in repo.Packages
    from tag in package.Tags
    select new {
        PackageName = package.Name,
        PackageVersion = package.Version
    }

Querying this index gives us a nice list of repositories:

We can also create a more complex index, that will tell us which package (regardless of its versions) is the most popularly used:

• MAP:
      from repo in docs.Repos
    from packageRef in repo.Packages
    let package = LoadDocument(packageRef.Id)
    select new {
        PackageName = packageRef.Name,
        Repos = new string[] { repo.FullName },
        Tags = package.Tags,
        Count = 1
    }
• REDUCE:
      from r in results
    group r by r.PackageName into g
    select new {
        PackageName = g.Key,
        Repos = g.SelectMany(x => x.Repos).Distinct(),
        Tags = g.SelectMany(x => x.Tags).Distinct(),
        Count = g.Sum(x => x.Count)
    }
• FIELDS:
  • Tags:
    • Storage: Yes
    • Indexing: Analyze
  • Count:
    • Sort: Int

This index allows us to see which packages are the most widely used:

And even drill further into the data, and for example, see which of the packages that have the “sql” tag is most widely used (and by which repositories):

Here I’ve shown you how to query the data. In my next post, I’ll try to gain some insights from the data, and see if I can see some usage patterns.

• Which NuGet packages are really popular (real usage vs. just download numbers)?
• Which NuGet packages are most popular for specific areas (logging, testing, etc.)?
• Which repositories are using specific NuGet packages?

I’ve shown in my previous post how to retrieve the list of packages used by  specific GitHub repository. And in a different post, I’ve shown how to retrieve the information of a NuGet package. We can use all that to collect all the information we need to answer the above questions.

I’ll start by creating my model:

    public class Repo
    {
        public string Id { get { return "Repositories/" + RepoId; } }
        public string RepoId { get; set; }
        public string Name { get; set; }
        public string FullName { get; set; }
        public DateTime Updated { get; set; }
 
        public int ForksCount { get; set; }
        public int WatchersCount { get; set; }
 
        public IEnumerable<PackageRef> Packages { get; set; }
    }

    public class Package
    {
        public string Id { get { return "Packages/" + Name + "/" + Version; } }
        public string Name { get; set; }
        public string Version { get; set; }

        public string Summary { get; set; }
        public DateTime? Updated { get; set; }
        public IEnumerable<string> Tags { get; set; }
    }

    public class PackageRef
    {
        public string Id { get { return "Packages/" + Name + "/" + Version; } }
        public string Name { get; set; }
        public string Version { get; set; }
    }

As you can see, we have a simple Repository class, with its Name, Full-Name, a bit of metadata, and a list of references to Packages. We also have a Package model, which includes the Name, Version, Tags and some additional fields.

If you’re asking yourself why I’ve modeled things this way, it is because I’ll be using RavenDb as my storage, and this follows RavenDb’s recommendation/conventions for document Ids, relationships and (de)normalization.

After this, it’s just a matter of writing a “crawler”. I’ll show only the pseudo code here, since it’s nothing groundbreaking:

• Search all C# repositories in GitHub
• For each repository:
  • Get list of packages
  • Save repository in the database
• For each collected package:
  • Get package details from NuGet
  • Save package in the database

There are of course a few challenges worth mentioning:

1. The GitHub API returns paged results, so we need to make sure we iterate over the repositories in pages (https://developer.github.com/v3/#pagination)
2. The GitHub API have a rate-limit, so we need to throttle our requests (https://developer.github.com/v3/#rate-limiting)
3. We are working in a big-data world – we’ll have to work with “chunks” of data, and save/load things from our database as we go along. This means that we also need to be careful not to do unnecessary work, by checking the database if we’ve already have all the relevant information instead of re-querying the (GitHub, NuGet) services.

The NuGet Gallery provides an OData feed, which can be used to easily query and search its database. It’s base URL is http://packages.nuget.org/v1/FeedService.svc. I’ll not go over the details and capabilities of an OData feed – you can read about it here. All I require are the details of a specific version of a specific package. So let’s see the code:

    private void UpdatePackageDetails(Package package)
    {
        var url = string.Format("http://packages.nuget.org/v1/FeedService.svc/Packages(Id='{0}',Version='{1}')",
            package.Name, package.Version);
 
        try
        {
            XDocument xdoc = XDocument.Load(url);
 
            var tags = xdoc.Descendants("{http://schemas.microsoft.com/ado/2007/08/dataservices}Tags").FirstOrDefault();
            if (tags != null)
                package.Tags = tags.Value.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
 
            var summary = xdoc.Root.Element("{http://www.w3.org/2005/Atom}summary");
            if (summary != null)
                package.Summary = summary.Value;

            var updated = xdoc.Root.Element("{http://www.w3.org/2005/Atom}updated");
            if (updated != null)
                package.Updated = DateTime.Parse(updated.Value);
        }
        catch (WebException)
        { }
    }

This code assumes that you have the Package class introduced in the previous post, with the Name and Version properties already populated. It’s a very simple piece of code – it creates the URL for the Feed entry of the package/version we need, loads it into an XDocument, and update the package instance with the metadata we require – Tags, Summary, and Updated date.