Jb in a nutshell

A river of T

Jb Evain — Mon, 09 Aug 2010 01:20:00 +0200

Generic collections, iterators and Linq all have in common a single interface: IEnumerable<T>. If you’ve been writing C# code lately, I bet you wrote your fair share of methods taking or returning IEnumerables.

Boo (among other marvels) has a cute syntactic shorthand for it. You can write T* instead of IEnumerable[of T]. Of course people would really like to have that in C# as well. Sure T* would be nice, but it’s already used for pointers. Some other suggestions in the stackoverflow question involve using the # suffix, or {}. On my side, I quite like the idea of re-using ~. It’s only used by the bitwise complement operator in C# (and by finalizers, thanks Scott for the heads up), and if you look at it with a bit of poetry, it kind of look like a river, a stream. And what’s an IEnumerable<T> but a stream of T?

Now discussing wishes for the next version of C# is nice and all, but I felt like hacking a bit on mcs, Mono’s C# compiler tonight, and in less than half an hour of grepping through the code, I had a working version of a patch that would enable this syntactic sugar.

Let’s take a very poorly implemented set of Linq operators:

using System;
using System.Collections.Generic;

static class Enumerable {

	public static IEnumerable<T> Concat<T> (this IEnumerable<T> source, IEnumerable<T> other)
	{
		foreach (var item in source)
			yield return item;

		foreach (var oitem in other)
			yield return oitem;
	}

	public static IEnumerable<TResult> Select<TSource, TResult> (this IEnumerable<TSource> source, Func<TSource, TResult> selector)
	{
		foreach (var item in source)
			yield return selector (item);
	}

	public static IEnumerable<TResult> SelectMany<TSource, TResult> (this IEnumerable<TSource> source, Func<TSource, IEnumerable<TResult>> selector)
	{
		foreach (var item in source)
			foreach (var sub_item in selector (item))
				yield return sub_item;
	}

	public static IEnumerable<T> Where<T> (this IEnumerable<T> source, Func<T, bool> predicate)
	{
		foreach (var item in source)
			if (predicate (item))
				yield return item;
	}
}

With my patch, you could write instead:

using System;

static class Enumerable {

	public static T~ Concat<T> (this T~ source, T~ other)
	{
		foreach (var item in source)
			yield return item;

		foreach (var oitem in other)
			yield return oitem;
	}

	public static TResult~ Select<TSource, TResult> (this TSource~ source, Func<TSource, TResult> selector)
	{
		foreach (var item in source)
			yield return selector (item);
	}

	public static TResult~ SelectMany<TSource, TResult> (this TSource~ source, Func<TSource, TResult~> selector)
	{
		foreach (var item in source)
			foreach (var sub_item in selector (item))
				yield return sub_item;
	}

	public static T~ Where<T> (this T~ source, Func<T, bool> predicate)
	{
		foreach (var item in source)
			if (predicate (item))
				yield return item;
	}
}

I let the reader decides for himself which version is the most readable. It's pretty much the same debate as the one for nullable types. Also note that this is just a toy patch, mainly for the sake of digging a bit in mcs, but it’s an interesting experiment nevertheless.

On IronRuby

Jb Evain — Sat, 07 Aug 2010 14:30:00 +0200

Just as everyone, I learned yesterday via Jimmy the near death experience that IronRuby is undergoing. And spent some time thinking about it.

For those of you who will stumble upon this and don’t follow this blog regularly, let me add a pinch of background. I implemented the .net 3.5 version of System.Linq.Expressions in Mono, I signed the IronRuby contributor agreement on day one, and regularly made sure it runs fine on Mono along with the DLR and IronPython. In the process of doing so, I contributed fixes and improvements to IronRuby. If I’m far for being the largest external contributor, at least I have a clue about how contributing to IronRuby works. And doesn’t work.

Also I spent the last two weeks working on making the IronRuby 1.1 test suite pass on Mono. Which may be the biggest reason I feel like I have something to say, as it involved countless hours of diving into the IronRuby code to extract simple test cases.

But now to the point. The IronRuby team currently consists of one hacker. We don’t know much about the IronPython team. And everyone who wanted to work on .net with their favorite dynamic language is freaking out. To a reason.

The good news is that the code of IronPython, IronRuby and the DLR is open source, and has recently been re-licensed under the Apache2 license. The official message is that IronRuby’s fate is now in the hands of the community.

That doesn’t sound like a bright future. So far, the community has been excluded from the development process of IronRuby. It’s impossible to contribute code to the core compiler of IronRuby, let alone to the DLR which is now part of .net 4.0. The code in github is a mere mirror of an internal TFS repository, and may or may not be up to date. And until IronRuby’s divorce with Microsoft is completely consumed, it will stay like this. So, sure we can contribute to external libraries, but that’s definitely not where the fun lies, and from now on, nor where the real work will be required.

So is the solution a fork?

It’s indeed a possibility. But for knowing first hand, IronRuby’s code source is far from being a simple piece of code to comprehend and to hack on. So what will happen in a couple of months when the buzz will have settled? Who will be able to take over the development of IronRuby? I for one am interested in helping, but I have already a lot on my plate. With Microsoft excluding the community from the core development process from day one, there’s no real knowledge of the insides of IronRuby outside of Microsoft, yet.

If Microsoft employees keep working on IronRuby occasionally, and keep the current setup, the community will still be excluded, and no one outside will contribute, besides Jimmy. IronRuby will simply fall into oblivion.

Yet I think that until Microsoft completely stops funding any work on IronRuby, they will be the only one really able to make something out of it. As Jimmy suggested, letting Microsoft know that there’s an important demand for IronRuby and IronPython, is probably the best course of action. Who knows, there’s maybe still time for Microsoft to correct its course. Special bonus points if we can get Microsoft to not only accept that there’s a demand for IronRuby, but also for accepting external contributions as it would make them move forward faster.

What about this petition (deleted since) to ask the Mono developers to take over IronRuby?

Actually that’s maybe what I’ve found the most tragic, and one of the reasons that triggered this post. It’s a bit along the line of “we don’t quite want to work on it, but we definitely want to have it, so let’s ask for someone else to steer the project”. If I can’t talk for my boss Miguel, I can say that we already have a lot on our plate, and that I don’t see us taking over IronRuby. I of course will continue to make sure it runs fine on Mono. But really, if the community wants IronRuby, it will have to do something about it itself. Either by contributing code or begging Microsoft.

A possibility of course would be for Microsoft to enter a development agreement with the Mono team as they did for Silverlight and Moonlight, and I’ll gladly volunteer to join the effort, but I don’t see that happening.

As a conclusion, and on a personal note, I find the situation extremely disappointing. In my opinion, Microsoft is making a strategic mistake. The Iron* projects were somehow the fun and hip face of the .net development (outside of Mono, of course). They had a true potential to attract developers that would traditionally stay away from Microsoft and .net.

PredicateBuilder revisited

Jb Evain — Tue, 20 Jul 2010 11:15:00 +0200

If you’re querying a LINQ provider, and if you’re dynamically creating the query, you probably stumbled upon the PredicateBuilder class.

It gives you a simple API to get a predicate in the form of an Expression Tree that a LINQ provider expects. You simply And (or Or) them together. For instance:

Expression<Func<User, bool>> is_underage = u => u.Age < 18;
Expression<Func<User, bool>> name_starts_with_a = u => u.Name [0] == 'A'

Expression<Func<User, bool>> a_is_underage = is_underage.And (name_starts_with_a);

Easy and effective. But let’s have a look at their And implementation:

public static Expression<Func<T, bool>> And<T> (this Expression<Func<T, bool>> expr1, Expression<Func<T, bool>> expr2)
{
    var invokedExpr = Expression.Invoke (expr2, expr1.Parameters.Cast<Expression> ());
    return Expression.Lambda<Func<T, bool>>
        (Expression.AndAlso (expr1.Body, invokedExpr), expr1.Parameters);
}

It creates a new expression tree (they are immutable), but the right hand side of the AndAlso node is an invocation of the second expression. If it works great if you compile the expression into an delegate, most LINQ providers won’t be able to optimize this form, as they’ll expect a more simple form, without nested lambdas.

I added to Mono.Linq.Expressions my own version of a PredicateBuilder which doesn’t have this issue. Let’s examine the implementation:

public static class PredicateBuilder {

    public static Expression<Func<T, bool>> True<T> ()
    {
        return Expression.Lambda<Func<T, bool>> (Expression.Constant (true), Expression.Parameter (typeof (T)));
    }

    public static Expression<Func<T, bool>> False<T> ()
    {
        return Expression.Lambda<Func<T, bool>> (Expression.Constant (false), Expression.Parameter (typeof (T)));
    }

    public static Expression<Func<T, bool>> OrElse<T> (this Expression<Func<T, bool>> self, Expression<Func<T, bool>> expression)
    {
        return Combine (self, expression, Expression.OrElse);
    }

    public static Expression<Func<T, bool>> AndAlso<T> (this Expression<Func<T, bool>> self, Expression<Func<T, bool>> expression)
    {
        return Combine (self, expression, Expression.AndAlso);
    }

    static Expression<Func<T, bool>> Combine<T> (Expression<Func<T, bool>> self, Expression<Func<T, bool>> expression, Func<Expression, Expression, Expression> selector)
    {
        CheckSelfAndExpression (self, expression);

        var parameter = CreateParameterFrom (self);

        return Expression.Lambda<Func<T, bool>> (
            selector (
                RewriteLambdaBody (self, parameter),
                RewriteLambdaBody (expression, parameter)),
            parameter);
    }

    static Expression RewriteLambdaBody (LambdaExpression expression, ParameterExpression parameter)
    {
        return new ParameterRewriter (expression.Parameters [0], parameter).Visit (expression.Body);
    }

    class ParameterRewriter : ExpressionVisitor {

        readonly ParameterExpression candidate;
        readonly ParameterExpression replacement;

        public ParameterRewriter (ParameterExpression candidate, ParameterExpression replacement)
        {
            this.candidate = candidate;
            this.replacement = replacement;
        }

        protected override Expression VisitParameter (ParameterExpression expression)
        {
            return ReferenceEquals (expression, candidate) ? replacement : expression;
        }
    }

    static ParameterExpression CreateParameterFrom<T> (Expression<Func<T, bool>> left)
    {
        var template = left.Parameters [0];

        return Expression.Parameter (template.Type, template.Name);
    }

    static void CheckSelfAndExpression<T> (Expression<Func<T, bool>> self, Expression<Func<T, bool>> expression)
    {
        if (self == null)
            throw new ArgumentNullException ("self");
        if (expression == null)
            throw new ArgumentNullException ("expression");
    }
}

Instead of nesting lambdas, it simply rewrites a lambda and inlines the two previous expressions bodies in it. This way, you can compose predicates and make sure they’ll be usable by any LINQ provider without having to deal with it explicitely.

I also named the method AndAlso and OrElse as it’s the terminology used in the Expression Tree API for the logical And (&&) and Or (||).

What’s next for Mono.Linq.Expressions? Probably an IEqualityComparer<Expression> implementation. Any other useful feature you can think of?

Mono.Linq.Expressions

Jb Evain — Wed, 23 Jun 2010 17:30:00 +0200

In the same vein of my little Mono.Reflection library, which complements the System.Reflection namespace with useful features such as an IL disassembler, Mono.Linq.Expressions is a small library for everyone who has to manipulate LINQ Expression Trees.

The first feature I’ve been adding is a simple pretty printer for expression trees. It comes in handy whenever you want a textual representation of an expression tree in a language you know.

For instance, let’s imagine you’re calling a method returning an expression:

Expression<Func<int, int, int>> silly = GetSillyExpression ();

And you want to quickly have a look at what it does. You may as well use a good old Console.WriteLine, but here’s what it will print:

(a, b) => IIF((a > b), {var c; ... }, {var d; ... })

Yeah right. Not very useful isn’t it. So let’s use your internal knowledge of the DLR and print its DebugView:

.Lambda #Lambda1<System.Func`3[System.Int32,System.Int32,System.Int32]>(
    System.Int32 $a,
    System.Int32 $b) {
    .If ($a > $b) {
        .Block(System.Int32 $c) {
            $c = $a #+ $b;
            $c #+= 42;
            $c
        }
    } .Else {
        .Block(System.Int32 $d) {
            $d = $a #- $b;
            $d #-= 42;
            $d
        }
    }
}

Better yet. But it doesn’t look even remotely familiar. Here comes Mono.Linq.Expressions.

Let’s call Console.WriteLine once more, but in a slightly different way:

Console.WriteLine (silly.ToCSharpCode ());

And here’s the output:

int (int a, int b)
{
        if (a > b)
        {
                int c;

                c = checked { a + b };
                checked { c += 42 };
                return c;
        }
        else
        {
                int d;

                d = checked { a - b };
                checked { d -= 42 };
                return d;
        }
}

Now I get it! And it’s indeed plain silly. But that’s not the point.

If you’re implementing a language on top of the DLR, or playing with the new keyword dynamic’s innards, or writing a LINQ provider, you might find it useful.

Mono.Cecil 0.9.2

Jb Evain — Wed, 05 May 2010 17:28:00 +0200

I tagged Mono.Cecil 0.9.2 at the beginning of the week in Cecil’s github. It’s a bug fix release, exactly two weeks after the release of 0.9.1.

It’s fast, stable, and shiny, thanks to all the contributors and testers of the 0.9 line. It’s safe to say that users of the 0.6 line of code are urged to update to this version, and report issues, if any.

parameterof, propertyof, methodof

Jb Evain — Wed, 05 May 2010 17:09:00 +0200

Every C# programmer knows about the keyword typeof, in essence, it returns an instance of System.Type for a given type. It’s used everywhere you have to do reflection on a type known at compilation time. You could achieve pretty much the same with the methods GetType(string) of both Module and Assembly, but the nice thing about typeof is that it’s refactoring proof.

Now, every C# programmer also knows that typeof is the only keyword with such abilities. There’s no parameterof, propertyof, methodof, or eventof to access other metadata members. The web is filled with ways to emulate a type safe reflection though. Lately, expression trees helped a lot. This post presents a proof of concept piece of code for a different approach.

Considering the following types:

public static class Parameter {

    public static ParameterInfo Of<T> (T value)
    {
        return null;
    }
}

public static class Property {

    public static PropertyInfo Of<T> (T value)
    {
        return null;
    }

    public static PropertyInfo Of (MethodInfo method)
    {
        const BindingFlags all = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.Instance;

        foreach (var property in method.DeclaringType.GetProperties (all))
            if (method.Equals (property.GetGetMethod (true)))
                return property;

        return null;
    }
}

public static class Method {

    public static MethodInfo Of<T> (Func<T> value)
    {
        return null;
    }

    public static MethodInfo Of<T1, TRet> (Func<T1, TRet> value)
    {
        return null;
    }

    public static MethodInfo Of (Action value)
    {
        return null;
    }

    public static MethodInfo Of<T> (Action<T> value)
    {
        return null;
    }
}

They are basically just stubs, but let’s use them anyway:

class Program {

    static void Main (string [] args)
    {
        ParameterInfo parameter = Parameter.Of (args);

        Console.WriteLine ("Parameter of args: {0}", parameter);

        PropertyInfo rank = Property.Of (args.Rank);

        Console.WriteLine ("Property of rank: {0}", rank);

        MethodInfo foo = Method.Of<int> (Foo);

        Console.WriteLine ("Foo method: {0}", foo);

        MethodInfo bar = Method.Of<int> (Bar);

        Console.WriteLine ("Bar method: {0}", bar);
    }

    static int Foo ()
    {
        return 42;
    }

    static void Bar (int b)
    {
        return;
    }
}

Of course if you run the code as is, everything will be null. What I did is that I wrote a little Cecil rewriter that will detect the calls to the different Of methods, and rewrite them to something that will give you the expected results.

So for instance, the Parameter.Of call is compiled as:

ldarg.0
call class [mscorlib]System.Reflection.ParameterInfo Parameter::Of<string[]>(!!0)

But is rewritten in two parts. First we emit at the beginning of the method a way to retrieve of the ParameterInfo of the current method:

ldtoken method void Program::Main(string[])
call class [mscorlib]System.Reflection.MethodBase [mscorlib]System.Reflection.MethodBase::GetMethodFromHandle(valuetype [mscorlib]System.RuntimeMethodHandle)
callvirt instance class [mscorlib]System.Reflection.ParameterInfo[] [mscorlib]System.Reflection.MethodBase::GetParameters()
stloc $reflection_parameters

And we simple replace the Of call with a load from the $reflection_parameters array:

ldloc $reflection_parameters
ldc.i4 0
ldelem.ref

For methods it’s a bit different, they are compiled as:

ldnull
ldftn int32 Program::Foo()
newobj instance void class [mscorlib]System.Func`1<int32>::.ctor(object,native int)
call class [mscorlib]System.Reflection.MethodInfo Method::Of<int32>(class [mscorlib]System.Func`1<!!0>)

And we rewrite them to remove the creation of a delegate:

ldtoken method int32 Program::Foo()
call class [mscorlib]System.Reflection.MethodBase [mscorlib]System.Reflection.MethodBase::GetMethodFromHandle(valuetype [mscorlib]System.RuntimeMethodHandle)
castclass  [mscorlib]System.Reflection.MethodInfo

For properties, it’s exactly the same, but we have to inject a call to a method that will retrieve the PropertyInfo the get method belongs to.

If you’re interested you can read the rewriter code. It’s pretty simple, but is nowhere complete, and I doubt anyone would use it anyway, but it’s still a pretty cool hack.

Mono.Cecil 0.9.1

Jb Evain — Mon, 19 Apr 2010 17:38:00 +0200

I pushed to my github repository an update for Mono.Cecil. It contains a week worth of fixes reported by the folks who ported their code to use the new API. The main addition for this release is the support for complex types in custom attributes, which was pretty much the only known thing that was not supported in 0.9, metadata wise.

Thanks to everyone that took the time to update to 0.9 and report issues.

Mono.Cecil 0.9 "cecil/light"

Jb Evain — Mon, 12 Apr 2010 20:28:00 +0200

I started working on Mono.Cecil during the fall of 2004. In its current incarnation, it served me and a lot of people very well. But looking at it now, it aged quite a bit. The code still compiles on .net 1.1, is using old conventions, doesn’t have a real test suite, is quite memory hungry, and is not that optimized. Which doesn’t prevent it to be a useful and wide used library, but looking back; I could have done a lot of things differently.

And doing things differently is basically what I’ve been doing for the past two years in my free time. What originally started as a refactoring of Mono.Cecil for the decompiler, ended up as a rewrite from the ground up. And today I’m excited to make public what is the next version of Cecil, which I’ve been fondly calling “cecil/light”.

Let’s start with a warning; this version contains breaking changes with the previous API. I didn’t promise API stability for the previous code, but this iteration of Mono.Cecil, tagged 0.9, is a huge step towards 1.0 and API stability.

But let’s focus for a while on the bright and new side. Mono.Cecil 0.9 comes with:

A cleaned and genericized API, I took this opportunity to clean some parts I hated in the old API.
A smaller and easier to maintain C#3 code base (Mono.Cecil 0.9 compiled with optimizations by csc is about 250k against almost 400k for 0.6) which only requires a .net 2.0 compatible runtime.
A test suite which is very easy to augment.
Better support for pdb and mdb files and strong name assemblies.
Complete support for PE32+ assemblies.
Bug fixes that weren’t possible without large changes in the old code.
Less memory consumption.
Lazy loading of every metadata element.
Speed and optimizations.
Complete Silverlight support.
A beginning of documentation on a wiki.
A collection of extension methods to add features to Cecil when they’re not necessary to the core assembly.

I ported a few of my projects to this version of Cecil already, and it shows great results. I didn’t spend more than four hours per project to adjust the code in a branch. There’s a migration page on the wiki to help you. If it doesn’t answer your question, reach us on the mono-cecil group.

I took special care in testing this version, and waited to have something stable to make it public, but just like every rewrite, you might face bugs or regressions. I’m confident it won’t take long before it gets really stable. A few people had early access to this code base, and ensured it was at least working as well as the previous version, if not better most of the time.

I moved the development of this version of Mono.Cecil to github while I’ll be maintaining the previous code in the Mono tree, as it’s still used by a lot of tools there, that I haven’t migrated yet.

We’ll also be working on writing more documentation for 1.0, both as a serie of HOWTOs, and as a more traditional class library documentation. But everyone that has already used Cecil should be up to speed promptly.

Thanks to everyone that helped, whether they knew it or not. Many thanks to Rodrigo B. de Oliveira, Carlo Kok, Sébastien Pouliot, and all the contributors to Mono.Cecil. Thanks also to Jeroen Frijters for his fantastic work on his IKVM.Reflection.Emit assembly in which I sometimes took inspiration. Enjoy!

Picture by law_keven some rights reserved

Présentation MonoTouch à Paris

Jb Evain — Tue, 06 Oct 2009 16:56:00 +0200

Certains vont à la Tour Eiffel, d’autres se ruent dans des Starbucks, mais moi, quand je monte conquérir Paris, je vais aux réunions alt.net. C’est souvent amusant, et toujours de bonne compagnie. En plus j’ai l’honneur, que dis-je le privilège d’être en haut de l’affiche de la prochaine réunion. Évidemment, je n’aurai pas de complet bleu, moi j’aime les rayures.

Lundi prochain, 12 octobre, je vais présenter MonoTouch. MonoTouch, c’est un produit de Novell, qui permet d’écrire des applications pour iPhone en C#, comme on le ferait pour le framework .net. Ou presque.

Cette session sera donc l’occasion de voir ce que l’on peut faire avec, de rentrer un peu dans les détails du pourquoi du comment ça marche, et d’en discuter. Parce que c’est vraiment ce qui fait le charme de ces rencontres alt.net. Les discussions post-session, un verre à la main, une cravate dans l’autre.

Notre hôte pour cette session sera Zenika, qui m’a réservé une page rien que pour moi, et surtout qui permet de s’inscrire. Dépêchez vous de vous inscrire, le nombre de place est probablement limité.

Le synopsis de la session:

Aujourd’hui, dans l’informatique, tout le monde ou presque a entendu parler de l’iPhone. Et à plus forte raison depuis que celui-ci supporte le copier-coller. C’est aujourd’hui un acteur important dans le monde de la mobilité, et sa démocratisation rapide en fait une plateforme de choix pour développer des applications, aussi bien pour l’entreprise que pour le particulier.
Si à l’origine développer pour cette platforme signifiait utiliser le langage d’Apple, l’Objective-C, on assiste à la naissance de solutions tierces destinées à fournir d’autres moyens de programmer pour l’iPhone.
Cette session sera l’occasion de voir non seulement comment il est possible de réutiliser son code et ses compétences .net sur l’iPhone grâce à MonoTouch, mais aussi d’en expliquer le fonctionnement.

À lundi prochain donc.

Rebasing System.Reactive to the .net CLR

Jb Evain — Thu, 30 Jul 2009 12:24:00 +0200

There has been a lot of interest towards the Rx framework lately. It’s definitely an interesting piece of software, and the video linked on channel 9 is quite fascinating. The only issue is that the only assembly you can get right now is compiled against Silverlight, making it impossible to use on the traditional .net framework. Impossible, really?

Well, I for one would not recommend it, but if you really can’t wait to try it, you could use this little piece of code which uses Cecil to turn the assembly compiled against Silverlight into an assembly that will work on the .net framework. Of course the assembly will lose its strong name in the process. But at least, it will be usable.

For instance, here’s the sample that Jafar is showing on his blog, running on Mono:

Update: don’t miss the comment Sébastien posted, and how he would do it with Reflexil.