Jb in a nutshell

SyntaxTree

Jb Evain — Wed, 22 Feb 2012 14:00:00 +0100

Or “What I have been up to since last September”.

It is no secret that the middle of 2011 was a bumpy time for the Mono team. But let’s not babble about something that happened months ago. Suffices to say that September the 15th was my last day at Novell.

I started contributing to Mono in 2005. In 2007, at 24, after being part of the two very first editions of the Google Summer of Code, I was hired full-time. I had an amazing time working this long on a project I deeply care about.

I’m proud of what we’ve accomplished during this time. And I’m super happy to see how great my friends at Xamarin are doing. Yesterday I was stuck on a plane for quite a long time, and a guy in front of me tinkered for an entire hour in iCircuit, an amazing application based on MonoTouch. As a member of the original MonoTouch team, it felt awesome!

Yet, after the dust settled, I felt like it was time to try something new. Something I’ve been thinking about for quite a while now. Even though I’ve been doing a lot of thinking and planning, it took quite a leap of faith, and quite a bit of work.

Introducing SyntaxTree

SyntaxTree is a company whose mission is twofold: write great developer tools and help other companies doing so. We obviously have a very strong background on the Microsoft .NET and Mono platforms, but we’ve already been helping clients target some very different platforms. You’re more than welcome to check SyntaxTree’s services page out to have a look at the kind of work we’ve been doing.

And today we announce our first product, UnityVS, bridging two amazing tools, Unity, and Visual Studio.

If you don’t already know about it, Wikipedia defines Unity as “an integrated authoring tool for creating 3D video games or other interactive content such as architectural visualizations or real-time 3D animations”. It uses Mono as a scripting engine and Unity has been part of the Mono community for a long while now.

Our product, UnityVS adds support in Visual Studio for the languages that programmers can use in Unity to write scripts for their game: UnityScript and Boo. UnityVS also adds support for debugging your scripts inside Visual Studio, even when running inside Unity’s editor.

I can not stress enough how big of a Unity fan I am. I think that what those guys are achieving, especially towards beginners, is nothing short of amazing. Would Unity have been around when I was 15, instead of writing PHP and JavaScript code, I’m pretty sure I would have spent nights tinkering in Unity, and my career would have ended up to be quite different.

And they happen to have a fantastic taste when it comes to third party code. Using Mono as a scripting engine? Awesome. Using a JavaScript like language based on the Boo compiler infrastructure? Still awesome. Using Mono.Cecil to the point that they use the word “to cecil [in|out]” as a verb internally? Even more awesome. Using ILSpy’s engine to retarget .net code to ActionScript for their Flash port? Mind blowing.

I’m very excited to work on a product which complements the Unity ecosystem, I hope you’ll like it.

IL fun fact: not is not !

Jb Evain — Mon, 30 Jan 2012 16:00:00 +0100

Picture yourself working on crafting a specific piece of CIL.

You need to write the compiled equivalent of:

bool b = ...;
bool n = !b;

It would be tempting to write:

ldloc b
not
stloc n

Except that it would not always work. not doesn’t negate booleans, it computes the bitwise complement of the value on the stack. It’s also not to be confused with the neg opcode, which negates a value, as in a multiplication by -1.

The usual pattern to negate a boolean is:

ldloc b
ldc.i4.0
ceq
stloc n

But that’s not really fun, is it?

Actual fun fact: there’s a misused “not” in the ECMA 335 in the example of the section 14.5 of the partition II.

Fun fact: C# methods whose bodies span over multiple source files

Jb Evain — Thu, 19 Jan 2012 22:00:00 +0100

While working on Mono.Cecil (your lovely library to analyze and manipulate .net binaries that is used by legions), one thing that struck me as odd for a while, was the fact that a method could have debug symbols for instructions that are defined in multiple files.

Cecil has this type, Instruction. When you’re analyzing a module with debug information (think .pdb or .mdb files), an Instruction may have its SequencePoint property set. A sequence point is nothing but the location of the code in a file that relates to the instruction.

What got me wondering, is that the APIs to retrieve those sequence points make it so that a method can have sequence points in different documents. So be it, this is how I ended up representing it in Cecil, but it’s only recently that I stumbled upon a case where indeed, a C# method had instructions defined in multiple files:

Foo.Bar.cs:

public partial class Foo {
    private List<Bar> _bars = new List<Bar>();
}

Foo.Baz.cs:

public partial class Foo {
    private Baz _baz;

    public Foo (Baz baz)
    {
        _baz = baz;
    }
}

Do you see what’s going on here?

The constructor of Foo is defined in Foo.Baz.cs, but there’s a field initializer in Foo.Bar.cs that is going to be compiled inside Foo’s constructor. When you debug the constructor, you’ll effectively end up jumping between the two files. Crazy right?

Can you think of another case where a C# (or VB.NET for that matters) method would have instructions defined in different files?

Constraining generic constraints

Jb Evain — Fri, 13 Jan 2012 14:30:00 +0100

This is the last part of an update about Mono.Linq.Expressions, a tiny helper library to complement the System.Linq.Expressions namespace for .net 4 and Mono.

And it's not really about Mono.Linq.Expressions itself. If you're interested about it though, you can read the part about fluently creating expression trees, and the other one about combining lambda expressions together.

Proper generic constraints

Have a look at the signature of the factory method Expression.Lambda:

public static Expression<TDelegate> Lambda<TDelegate> (
    Expression body,
    params ParameterExpression [] parameters) {}

And at the signature of Expression<T>:

public sealed class Expression<T> : LambdaExpression {}

Given the post title, and the <h3> a few lines up there, it's obvious where I'm going: those generic parameters have no constraint. A C# compiler will happily compile:

class Potato {}

Expression<Potato> e = Expression.Lambda<Potato> (
    Expression.Constant (new Potato ()));

Of course this won't get you far, but it sure is a heart breaker that because of a little language oddity, you delay to until runtime the problem resolution: this code will throw, Potato sure isn't a Delegate type, and Expression.Lambda, the only way to create an Expression<T> is making sure of that.

So much for type safety, generics!

Another famous issue is the following. Again, for the compiler, it's perfectly legit to write:

int i;
if (!Enum.TryParse ("42", out i))
    Console.WriteLine ("This is so unfair!");

And why is that? Because of the signature of Enum.TryParse:

public static bool TryParse<TEnum> (string value, out TEnum result) where TEnum : struct {}

To sum the issue up, C# doesn't allow generic constraints on delegates and enums. And why am I rumbling about this? Because Mono.Linq.Expressions has this method:

public static Expression<T> Combine<T> (
    this Expression<T> self,
    Func<Expression, Expression> combinator)

And if Expression<T> is not constrained on Delegate, then I sure as hell won't let this stand in my own public API! Which is why the actual signature of the Combine method is:

public static Expression<T> Combine<[DelegateConstraint] T> (
    this Expression<T> self,
    Func<Expression, Expression> combinator) where T : class

And all of this was just a perfect excuse to write a little Mono.Cecil based utility tool, appropriately named patch-constraints, that is used as a post-compile step to fixup the Delegate and Enum constraints of any generic parameter decorated respectively with a DelegateConstraintAttribute or EnumConstraintAttribute, turning them into actual constraints to delegates and enums, which, funnily enough, the C# compiler is more than happy to honor.

Obviously I could have used the most famous Jon Skeet's unconstrained-melody tool, but 2002 called and it wants its ildasm based, IL text parser back.

Mono.Linq.Expressions update 2

Jb Evain — Thu, 12 Jan 2012 12:25:00 +0100

This is the second part of an update about Mono.Linq.Expressions, a tiny helper library to complement the System.Linq.Expressions namespace for .net 4 and Mono.

The first part is about fluent creation of expression trees.

Combining expression trees together

I keep reading questions on StackOverflow about this. How to combine two lambda expression together? If we have:

Expression<Func<User, bool>> isUserOver18 = u => u.Age >= 18;
Expression<Func<User, bool>> isFemaleUser = u => u.Gender == Gender.Female;

If we want to combine this lambda expression with a “and” logical expression, the natural way would be to write:

Expression<Func<User, bool>> isFemaleUserOver18 = u =>
    isUserOver18(u) && isFemaleUser(u);

This works just fine if you compile the expression into a delegate to actually execute this code. But most of the time questions of StackOverflow are about using the resulting lambda expression to create a query for LINQ to a database provider, which will analyze the expression tree and create an according SQL request.

By combining expression trees this way, the LINQ provider may or may not be unable to turn the two invocations into actual SQL.

That's one of the reason I wrote about an updated PredicateBuilder. The obvious solution is to inline the two combined representation of lambda expressions into a new lambda expression tree.

The update of Mono.Linq.Expressions comes with a new type, CombineExtensions, which exposes extension methods that you can use to combine fully created (into lambda expressions) expression trees.

Using those, combining the two expression trees is as simple as:

Expression<Func<User, bool>> isFemaleUserOver18 = isUserOver18.Combine(
    isFemaleUser,
    (left, right) => left.AndAlso(right));

And indeed, if you print the code representation of this expression tree, you'll have both lambda bodies inlined into another one:

user => user.Age >= 18 && user.Gender == Gender.Female

Or if you want to negate the boolean expression:

Expression<Func<User, bool>> isNotFemaleUserOver18 = isFemaleUserOver18.Combine(
    e => e.Not());

The cool thing about those Combine extension methods is that they're completely generic, they don't work only on simple predicates. For instance, you can use those to chain constructions of mathematical expressions.

Mono.Linq.Expressions update

Jb Evain — Wed, 11 Jan 2012 17:00:00 +0100

I just tagged the 1.2 version of Mono.Linq.Expressions, and pushed an updated nuget package.

Mono.Linq.Expressions is a utility library to complement the System.Linq.Expressions namespace, and works with .net 4.0 just as fine as it does with Mono. With a bit over 220 downloads of the nuget package, it's short of roughly 160,400 downloads to be the most downloaded nuget package : a stunning success to put it simply.

This post is the first of a short series to detail what's awesome and new in this version.

Extension methods for a fluent construction of expression trees.

Have you been using the expression tree API to build a representation of code at runtime ? If so you're familiar with the Expression class, and it's load of factory methods.

You're also familiar with this kind of code:

var user = Expression.Parameter(typeof (User), "user");

var isFemaleUserOver18 = Expression.Lambda<Func<User, bool>>(
    Expression.AndAlso(
        Expression.GreaterThanOrEqual(
            Expression.Property(user, "Age"),
            Expression.Constant(18)),
        Expression.Equal(
            Expression.Property(user, "Gender"),
            Expression.Constant(Gender.Female))), user);

If you take some time to parse this code, the intent is to create an expression tree similar to the one the compiler would emit if you were to write:

Expression<Func<User, bool>> isFemaleUserOver18 =
    user => user.Age >= 18 && user.Gender == Gender.Female;

Mono.Linq.Expressions 1.2 contains a code generated series of extension methods to simplify the manual construction of expression trees by fluently chaining the invocations. This allows you to write instead:

var user = typeof (User).Parameter("user");

var isFemaleUserOver18 = Expression.Lambda<Func<User, bool>>(
    user.Property("Age").GreaterThanOrEqual(18.Constant())
    .AndAlso(
        user.Property("Gender").Equal(Gender.Female.Constant())), user);

Not only is the code shorter, but it's also easier on the eyes, and easier to comprehend. Using this, almost the factory methods calls to the Expression class can be written fluently.

TechDays 2011

Jb Evain — Mon, 07 Feb 2011 23:01:00 +0100

Mercredi, j’aurai l’opportunité de présenter Mono et son écosystème pendant les TechDays Microsoft.

Ce sera l’occasion de faire le point sur les dernières nouveautés de Mono, et de montrer comment réutiliser ses compétences .net et partager son code C# pour cibler des plateformes en vogue, comme l’iPhone et l’iPad avec MonoTouch, Android avec MonoDroid et Mac OS X avec MonoMac.

Si vous ne pouvez pas assister à la session, je serai aussi disponible sur le stand alt.net pour discuter de tous ces sujets. Venez nombreux !

Custom Linq Expressions

Jb Evain — Mon, 17 Jan 2011 18:45:00 +0100

Between .net 3.5 and .net 4.0, the Linq Expression Tree API changed quite a bit, going from a simple compiler supporting expressions only, to a full fledged compiler supporting expressions as well as statements, on top of which are implemented all DLR based language. For anyone working closely with this API, the Expression Tree spec is a great source of information, the MSDN isn’t very helpful there.

When the spec comes to describing the object model for nodes, it differentiates several kind of nodes:

The core nodes and the common nodes are those which are implemented inside System.Linq.Expressions, and common nodes are implemented on top of the core nodes, using a process that is called reducing nodes.

A reducible node is simply a node that can be compiled as a compound of core nodes. We’re not going to talk about library specific extensions, which are nodes that you would write if you were to implement, say, a LINQ provider. We’ll focus on reducible nodes.

I took some time last year to implement a few helper nodes that are not available in .net 4.0, but, according to the spec, may be included in a future release. They’re mapped to similar C# constructs, and are pretty useful it you generate complex code using expression trees. Here’s the list of custom expressions I added in Mono.Linq.Expressions, my project to complement the System.Linq.Expression namespace:

Even if those map to a C# statement, they’re suffixed with `Expression` for consistency with the whole System.Linq.Expressions namespace. Using them is as simple as, heh, adding a using directive:

using Mono.Linq.Expressions;

And using the factory methods on the CustomExpression class to create those new nodes:

var d = Expression.Parameter (typeof (DisposableType), "d");

var printAndDispose = Expression.Lambda<Action<DisposableType>> (
    CustomExpression.Using (
        d,
        Expression.Call (typeof (Console).GetMethod (
            "WriteLine", new [] { typeof (object) }), d)),
    d);

The Expression Tree API makes it easy to write custom reducible expressions, all you have to do, is to inherit from Expression, override CanReduce and NodeType, and implement the Reduce method:

public abstract partial class CustomExpression : Expression {

    public override ExpressionType NodeType {
        get { return ExpressionType.Extension; }
    }

    public override bool CanReduce {
        get { return true; }
    }

    /* ... */
}

If we take, for instance, our UsingExpression, mapping to the C# using statement, Reduce is implemented as follows:

public override Expression Reduce ()
{
    var end_finally = Expression.Label ("end_finally");

    return Expression.Block (
        new [] { variable },
        Expression.Assign (variable, disposable),
        Expression.TryFinally (
            body,
            Expression.Block (
                Expression.Condition (
                    Expression.NotEqual (variable, Expression.Constant (null)),
                    Expression.Block (
                        Expression.Call (
                            Expression.Convert (variable, typeof (IDisposable)),
                            typeof (IDisposable).GetMethod ("Dispose")),
                        Expression.Goto (end_finally)),
                    Expression.Goto (end_finally)),
                Expression.Label (end_finally))));
}

Just like a C# using, it will try to execute a block, and whether the block triggered an exception or not, it will call IDisposable.Dispose on the subject of the using statement.

Wasn’t that easy? Even if those nodes are pretty general, writing custom and specific nodes is a powerful and simple way to factorize code when generating code using the Expression Tree API.

In the end, I added in Mono.Linq.Expressions a CustomExpression type, our base class for well, custom expressions. We provide a CustomExpressionVisitor which extends the standard ExpressionVisitor to support our custom expressions, and the CSharpWriter has been updated to support them as well:

[Test]
public void Using ()
{
    var arg = Expression.Parameter (typeof (IDisposable), "arg");

    var lambda = Expression.Lambda<Action<IDisposable>> (
        CustomExpression.Using (
            arg,
            Expression.Call (typeof (Console).GetMethod (
                "WriteLine", new [] { typeof (object) }), arg)),
        arg);

    AssertExpression (@" 
void (IDisposable arg)
{
    using (arg)
    {
        Console.WriteLine(arg);
    }
}
", lambda);
}

Next feature for Mono.Linq.Expressions: using Mono.Reflection to turn a delegate into an expression of this delegate.

IQueryable support for WP7

Jb Evain — Wed, 13 Oct 2010 17:35:00 +0200

Or how to use APIs missing from the WP7 SDK.

Monday saw the official launch of Windows Phone 7, at this occasion, I downloaded the SDK to see how it compares to MonoTouch and MonoDroid. Basically, it’s based on the Compact Framework Base Class Library, on top of which is exposed the Silverlight UI API.

The Compact Framework has historically targeted devices with limited capabilities. The Compact Framework BCL is a trimmed down version of the vanilla .net BCL. One notable missing feature is runtime code generation through System.Reflection.Emit. As such, you can’t use DynamicMethods, nor can you compile assemblies on the device, and nor can you compile Expression Trees. You can expect to find missing tidbits sprinkled throughout the whole BCL. But luckily for us, there’s Mono.

The Mono BCL being licensed under the MIT/X11, you can easily take C# code from our libraries and use it inside your own applications. And that whenever our code is not tied to a runtime feature. This means that you couldn’t bring System.Reflection.Emit to WP7, but that you could probably bring XmlDocument with a bit of work.

Basically, this reminded me of what db4o did to bring support for Expression Trees to the Compact Framework using code from Mono. I figured I’d try to do the same for WP7. If you played with the WP7 SDK, you probably know that you have access to Expression Trees, and the compiler can emit them without any issue. But LambdaExpression and Expression of T don’t have their Compile method you’re used to. And WP7 is missing IQueryable and folks.

So as an example, I extracted our Expression Tree interpreter, which is used, guess what, when you can’t compile Expression Trees at runtime, and our IQueryable support. Here we are, by just referencing this System.Linq.dll, you can write code such as:

using System.Linq;

var data = new [] { 1, 2, 3, 4, 5, 6, 7, 8 };

var query = data.AsQueryable ();
query.Where (i => i % 2 == 0);

int count = query.Count ();

Or:

using System.Linq.Expressions;

var p1 = Expression.Parameter (typeof (string), "x");
var p2 = Expression.Parameter (typeof (string), "y");

Expression<Func<string, Func<string, Func<string>>>> e =
    Expression.Lambda<Func<string, Func<string, Func<string>>>> (
        Expression.Lambda<Func<string, Func<string>>> (
            Expression.Lambda<Func<string>> (
                Expression.Call (
                    typeof (string).GetMethod ("Concat", new [] { typeof (string), typeof (string) }),
                    new [] { p1, p2 }),
                new ParameterExpression [0]),
            new [] { p2 }),
        new [] { p1 });

var f = e.Compile ();
var f2 = f ("Hello ");
var f3 = f2 ("World !");

var result = f3 ();

(Crazy, I know)

You can simply take the assembly and reference it from your projects, or compile yourself the code. Let me know if this has been useful to you.

To conclude, I’d say don’t hesitate to take code from Mono to fill the voids in the WP7 SDK. Who knows if you won’t end up contributing missing parts or filing bugs afterwards.

IsAssignable what?

Jb Evain — Fri, 01 Oct 2010 10:30:00 +0200

I’ve tweeted it before, Jackson tumblered it in return, but after commenting on a StackOverflow answer where the poster got it wrong, commenting:

I always seem to turn that call around.

I felt that I needed to broaden this grand hack. If you’re like me, and apparently like a lot of coders out there, you tend to always get Type.IsAssignableFrom wrong on the first try, then you need this extension method:

public static bool IsAssignableTo (this Type self, Type type)
{
    if (self == null)
        throw new ArgumentNullException ("self");
    if (type == null)
        throw new ArgumentNullException ("type");

    return type.IsAssignableFrom (self);
}

Somehow, even if IsAssignableFrom is more «left to rightish», preserving the side of the arguments a real assignment would have, I understand IsAssignableTo faster. For what it’s worth, I’ve used it extensively while writing our .net 3.5 implementation of System.Linq.Expressions, and it served me well.

Have a great friday.