Jb in a nutshell

What's new to extend Visual Studio 11

Jb Evain — Fri, 23 Mar 2012 15:21:00 +0100

After my somewhat long rant about Visual Studio, I took some time to compile my different notes into one blog post.

The following is the list of information we have about managed APIs that changed or have been added between Visual Studio 2010 and Visual Studio 11 to easily extend Visual Studio with a sane and managed language:

Visual Studio, you're doing it wrong

Jb Evain — Mon, 27 Feb 2012 17:20:00 +0100

If there’s one thing I’ve come to learn about life, it’s that most things are neither completely black nor completely white. There’s an infinity of shade of grays in between.

And if the announce of the Visual Studio (VS from now on) 11 beta taught me something, it is that VS 11 is exactly that. Neither completely black, nor completely white. But very gray. Oh so gray.

I won’t comment about the new look here, as the object of my ire with VS and its apparent roadmap lies at a completely different level. And boy am I annoyed to see where the VS team puts its focus.

Do you remember Steve Yegge’s rant about the need of a programmable platform at Google? It’s a tad long (euphemism of the day), but it’s well worth the read. And with Visual Studio, Microsoft is exactly failing at that.

Visual Studio 11 will be the sixth iteration of Visual Studio with .net support. And for a .net programmer like myself, it’s still a terrible platform to extend. I want to be able to have complete control over my environment by programming it. I’m a firm believer in meta-programming, I want to be able to program my IDE all around, automate tasks, manipulate my projects and my code, and complement the UI myself. Have you ever tried to write a plugin for Visual Studio? To keep on going with euphemisms, let say it’s not a pleasant experience at all.

It’s quite a shame from the company that brought us the .net framework and C#. Most of the APIs to plug into VS are poorly documented COM wrappers from another age. VS 2010 introduced some managed APIs that are addressable from MEF, and they’re actually pretty cool, but they only cover a small surface of what VS has to offer. And even those too are poorly documented, compared to the level of documentation for the .net framework. Roslyn is also going to somewhat improve things. But it’s not planned for VS 11, and it’s also restricted to a subset of what VS offers.

If you want to interact with the project system for system, you can use the disgrace that the MPF (Managed Project Framework) is. It’s a buggy piece of code that you have to dig from codeplex, with terrible documentation. If googling for documentation about it brought you to the MSDN, you can read that the types there are in:

Assembly: MPF.Project.NonShipping (in mpf.project.nonshipping.dll)

How friendly. And because no one really knows about all those obscure APIs, your only source of knowledge is the Microsoft VS extensibility forum. Which is also another source of frustration entirely : you have to wait for MS employees to help you out.

For anyone who wrote eclipse plugins for instance, VS is a huge step back. No wonder the plugin ecosystem of eclipse is so flourishing, it has been designed to be pluggable.

I think it is actually hurting VS itself, and as a consequence, the .net platform as a whole. I really think that the experience of the .net programmer would be quite different if from the beginning VS would have exposed a proper managed API, with the same quality and care that you can find for the .net framework. Or if at least, they would have seriously worked towards that during the years.

Folks from Microsoft said they value feedback for betas. Here’s one: invest in making Visual Studio a great platform for the .net programmer. Expose real managed APIs, let us plug into as much features as possible. .net programmers all around will take care of improving the experience of developing with Visual Studio much faster than what it takes you to add a better search into VS.

Also, don’t value feedback only during betas. Interacting with microsoft connect most of the time result in huge frustration. It’s the only place where I, a customer, have been told that my bug won’t be fixed because someone was scared of getting customer complaints.

SyntaxTree

Jb Evain — Wed, 22 Feb 2012 14:00:00 +0100

Or “What I have been up to since last September”.

It is no secret that the middle of 2011 was a bumpy time for the Mono team. But let’s not babble about something that happened months ago. Suffices to say that September the 15th was my last day at Novell.

I started contributing to Mono in 2005. In 2007, at 24, after being part of the two very first editions of the Google Summer of Code, I was hired full-time. I had an amazing time working this long on a project I deeply care about.

I’m proud of what we’ve accomplished during this time. And I’m super happy to see how great my friends at Xamarin are doing. Yesterday I was stuck on a plane for quite a long time, and a guy in front of me tinkered for an entire hour in iCircuit, an amazing application based on MonoTouch. As a member of the original MonoTouch team, it felt awesome!

Yet, after the dust settled, I felt like it was time to try something new. Something I’ve been thinking about for quite a while now. Even though I’ve been doing a lot of thinking and planning, it took quite a leap of faith, and quite a bit of work.

Introducing SyntaxTree

SyntaxTree is a company whose mission is twofold: write great developer tools and help other companies doing so. We obviously have a very strong background on the Microsoft .NET and Mono platforms, but we’ve already been helping clients target some very different platforms. You’re more than welcome to check SyntaxTree’s services page out to have a look at the kind of work we’ve been doing.

And today we announce our first product, UnityVS, bridging two amazing tools, Unity, and Visual Studio.

If you don’t already know about it, Wikipedia defines Unity as “an integrated authoring tool for creating 3D video games or other interactive content such as architectural visualizations or real-time 3D animations”. It uses Mono as a scripting engine and Unity has been part of the Mono community for a long while now.

Our product, UnityVS adds support in Visual Studio for the languages that programmers can use in Unity to write scripts for their game: UnityScript and Boo. UnityVS also adds support for debugging your scripts inside Visual Studio, even when running inside Unity’s editor.

I can not stress enough how big of a Unity fan I am. I think that what those guys are achieving, especially towards beginners, is nothing short of amazing. Would Unity have been around when I was 15, instead of writing PHP and JavaScript code, I’m pretty sure I would have spent nights tinkering in Unity, and my career would have ended up to be quite different.

And they happen to have a fantastic taste when it comes to third party code. Using Mono as a scripting engine? Awesome. Using a JavaScript like language based on the Boo compiler infrastructure? Still awesome. Using Mono.Cecil to the point that they use the word “to cecil [in|out]” as a verb internally? Even more awesome. Using ILSpy’s engine to retarget .net code to ActionScript for their Flash port? Mind blowing.

I’m very excited to work on a product which complements the Unity ecosystem, I hope you’ll like it.

IL fun fact: not is not !

Jb Evain — Mon, 30 Jan 2012 16:00:00 +0100

Picture yourself working on crafting a specific piece of CIL.

You need to write the compiled equivalent of:

bool b = ...;
bool n = !b;

It would be tempting to write:

ldloc b
not
stloc n

Except that it would not always work. not doesn’t negate booleans, it computes the bitwise complement of the value on the stack. It’s also not to be confused with the neg opcode, which negates a value, as in a multiplication by -1.

The usual pattern to negate a boolean is:

ldloc b
ldc.i4.0
ceq
stloc n

But that’s not really fun, is it?

Actual fun fact: there’s a misused “not” in the ECMA 335 in the example of the section 14.5 of the partition II.

Fun fact: C# methods whose bodies span over multiple source files

Jb Evain — Thu, 19 Jan 2012 22:00:00 +0100

While working on Mono.Cecil (your lovely library to analyze and manipulate .net binaries that is used by legions), one thing that struck me as odd for a while, was the fact that a method could have debug symbols for instructions that are defined in multiple files.

Cecil has this type, Instruction. When you’re analyzing a module with debug information (think .pdb or .mdb files), an Instruction may have its SequencePoint property set. A sequence point is nothing but the location of the code in a file that relates to the instruction.

What got me wondering, is that the APIs to retrieve those sequence points make it so that a method can have sequence points in different documents. So be it, this is how I ended up representing it in Cecil, but it’s only recently that I stumbled upon a case where indeed, a C# method had instructions defined in multiple files:

Foo.Bar.cs:

public partial class Foo {
    private List<Bar> _bars = new List<Bar>();
}

Foo.Baz.cs:

public partial class Foo {
    private Baz _baz;

    public Foo (Baz baz)
    {
        _baz = baz;
    }
}

Do you see what’s going on here?

The constructor of Foo is defined in Foo.Baz.cs, but there’s a field initializer in Foo.Bar.cs that is going to be compiled inside Foo’s constructor. When you debug the constructor, you’ll effectively end up jumping between the two files. Crazy right?

Can you think of another case where a C# (or VB.NET for that matters) method would have instructions defined in different files?

Constraining generic constraints

Jb Evain — Fri, 13 Jan 2012 14:30:00 +0100

This is the last part of an update about Mono.Linq.Expressions, a tiny helper library to complement the System.Linq.Expressions namespace for .net 4 and Mono.

And it's not really about Mono.Linq.Expressions itself. If you're interested about it though, you can read the part about fluently creating expression trees, and the other one about combining lambda expressions together.

Proper generic constraints

Have a look at the signature of the factory method Expression.Lambda:

public static Expression<TDelegate> Lambda<TDelegate> (
    Expression body,
    params ParameterExpression [] parameters) {}

And at the signature of Expression<T>:

public sealed class Expression<T> : LambdaExpression {}

Given the post title, and the <h3> a few lines up there, it's obvious where I'm going: those generic parameters have no constraint. A C# compiler will happily compile:

class Potato {}

Expression<Potato> e = Expression.Lambda<Potato> (
    Expression.Constant (new Potato ()));

Of course this won't get you far, but it sure is a heart breaker that because of a little language oddity, you delay to until runtime the problem resolution: this code will throw, Potato sure isn't a Delegate type, and Expression.Lambda, the only way to create an Expression<T> is making sure of that.

So much for type safety, generics!

Another famous issue is the following. Again, for the compiler, it's perfectly legit to write:

int i;
if (!Enum.TryParse ("42", out i))
    Console.WriteLine ("This is so unfair!");

And why is that? Because of the signature of Enum.TryParse:

public static bool TryParse<TEnum> (string value, out TEnum result) where TEnum : struct {}

To sum the issue up, C# doesn't allow generic constraints on delegates and enums. And why am I rumbling about this? Because Mono.Linq.Expressions has this method:

public static Expression<T> Combine<T> (
    this Expression<T> self,
    Func<Expression, Expression> combinator)

And if Expression<T> is not constrained on Delegate, then I sure as hell won't let this stand in my own public API! Which is why the actual signature of the Combine method is:

public static Expression<T> Combine<[DelegateConstraint] T> (
    this Expression<T> self,
    Func<Expression, Expression> combinator) where T : class

And all of this was just a perfect excuse to write a little Mono.Cecil based utility tool, appropriately named patch-constraints, that is used as a post-compile step to fixup the Delegate and Enum constraints of any generic parameter decorated respectively with a DelegateConstraintAttribute or EnumConstraintAttribute, turning them into actual constraints to delegates and enums, which, funnily enough, the C# compiler is more than happy to honor.

Obviously I could have used the most famous Jon Skeet's unconstrained-melody tool, but 2002 called and it wants its ildasm based, IL text parser back.

Mono.Linq.Expressions update 2

Jb Evain — Thu, 12 Jan 2012 12:25:00 +0100

This is the second part of an update about Mono.Linq.Expressions, a tiny helper library to complement the System.Linq.Expressions namespace for .net 4 and Mono.

The first part is about fluent creation of expression trees.

Combining expression trees together

I keep reading questions on StackOverflow about this. How to combine two lambda expression together? If we have:

Expression<Func<User, bool>> isUserOver18 = u => u.Age >= 18;
Expression<Func<User, bool>> isFemaleUser = u => u.Gender == Gender.Female;

If we want to combine this lambda expression with a “and” logical expression, the natural way would be to write:

Expression<Func<User, bool>> isFemaleUserOver18 = u =>
    isUserOver18(u) && isFemaleUser(u);

This works just fine if you compile the expression into a delegate to actually execute this code. But most of the time questions of StackOverflow are about using the resulting lambda expression to create a query for LINQ to a database provider, which will analyze the expression tree and create an according SQL request.

By combining expression trees this way, the LINQ provider may or may not be unable to turn the two invocations into actual SQL.

That's one of the reason I wrote about an updated PredicateBuilder. The obvious solution is to inline the two combined representation of lambda expressions into a new lambda expression tree.

The update of Mono.Linq.Expressions comes with a new type, CombineExtensions, which exposes extension methods that you can use to combine fully created (into lambda expressions) expression trees.

Using those, combining the two expression trees is as simple as:

Expression<Func<User, bool>> isFemaleUserOver18 = isUserOver18.Combine(
    isFemaleUser,
    (left, right) => left.AndAlso(right));

And indeed, if you print the code representation of this expression tree, you'll have both lambda bodies inlined into another one:

user => user.Age >= 18 && user.Gender == Gender.Female

Or if you want to negate the boolean expression:

Expression<Func<User, bool>> isNotFemaleUserOver18 = isFemaleUserOver18.Combine(
    e => e.Not());

The cool thing about those Combine extension methods is that they're completely generic, they don't work only on simple predicates. For instance, you can use those to chain constructions of mathematical expressions.

Mono.Linq.Expressions update

Jb Evain — Wed, 11 Jan 2012 17:00:00 +0100

I just tagged the 1.2 version of Mono.Linq.Expressions, and pushed an updated nuget package.

Mono.Linq.Expressions is a utility library to complement the System.Linq.Expressions namespace, and works with .net 4.0 just as fine as it does with Mono. With a bit over 220 downloads of the nuget package, it's short of roughly 160,400 downloads to be the most downloaded nuget package : a stunning success to put it simply.

This post is the first of a short series to detail what's awesome and new in this version.

Extension methods for a fluent construction of expression trees.

Have you been using the expression tree API to build a representation of code at runtime ? If so you're familiar with the Expression class, and it's load of factory methods.

You're also familiar with this kind of code:

var user = Expression.Parameter(typeof (User), "user");

var isFemaleUserOver18 = Expression.Lambda<Func<User, bool>>(
    Expression.AndAlso(
        Expression.GreaterThanOrEqual(
            Expression.Property(user, "Age"),
            Expression.Constant(18)),
        Expression.Equal(
            Expression.Property(user, "Gender"),
            Expression.Constant(Gender.Female))), user);

If you take some time to parse this code, the intent is to create an expression tree similar to the one the compiler would emit if you were to write:

Expression<Func<User, bool>> isFemaleUserOver18 =
    user => user.Age >= 18 && user.Gender == Gender.Female;

Mono.Linq.Expressions 1.2 contains a code generated series of extension methods to simplify the manual construction of expression trees by fluently chaining the invocations. This allows you to write instead:

var user = typeof (User).Parameter("user");

var isFemaleUserOver18 = Expression.Lambda<Func<User, bool>>(
    user.Property("Age").GreaterThanOrEqual(18.Constant())
    .AndAlso(
        user.Property("Gender").Equal(Gender.Female.Constant())), user);

Not only is the code shorter, but it's also easier on the eyes, and easier to comprehend. Using this, almost the factory methods calls to the Expression class can be written fluently.

TechDays 2011

Jb Evain — Mon, 07 Feb 2011 23:01:00 +0100

Mercredi, j’aurai l’opportunité de présenter Mono et son écosystème pendant les TechDays Microsoft.

Ce sera l’occasion de faire le point sur les dernières nouveautés de Mono, et de montrer comment réutiliser ses compétences .net et partager son code C# pour cibler des plateformes en vogue, comme l’iPhone et l’iPad avec MonoTouch, Android avec MonoDroid et Mac OS X avec MonoMac.

Si vous ne pouvez pas assister à la session, je serai aussi disponible sur le stand alt.net pour discuter de tous ces sujets. Venez nombreux !

Custom Linq Expressions

Jb Evain — Mon, 17 Jan 2011 18:45:00 +0100

Between .net 3.5 and .net 4.0, the Linq Expression Tree API changed quite a bit, going from a simple compiler supporting expressions only, to a full fledged compiler supporting expressions as well as statements, on top of which are implemented all DLR based language. For anyone working closely with this API, the Expression Tree spec is a great source of information, the MSDN isn’t very helpful there.

When the spec comes to describing the object model for nodes, it differentiates several kind of nodes:

The core nodes and the common nodes are those which are implemented inside System.Linq.Expressions, and common nodes are implemented on top of the core nodes, using a process that is called reducing nodes.

A reducible node is simply a node that can be compiled as a compound of core nodes. We’re not going to talk about library specific extensions, which are nodes that you would write if you were to implement, say, a LINQ provider. We’ll focus on reducible nodes.

I took some time last year to implement a few helper nodes that are not available in .net 4.0, but, according to the spec, may be included in a future release. They’re mapped to similar C# constructs, and are pretty useful it you generate complex code using expression trees. Here’s the list of custom expressions I added in Mono.Linq.Expressions, my project to complement the System.Linq.Expression namespace:

Even if those map to a C# statement, they’re suffixed with `Expression` for consistency with the whole System.Linq.Expressions namespace. Using them is as simple as, heh, adding a using directive:

using Mono.Linq.Expressions;

And using the factory methods on the CustomExpression class to create those new nodes:

var d = Expression.Parameter (typeof (DisposableType), "d");

var printAndDispose = Expression.Lambda<Action<DisposableType>> (
    CustomExpression.Using (
        d,
        Expression.Call (typeof (Console).GetMethod (
            "WriteLine", new [] { typeof (object) }), d)),
    d);

The Expression Tree API makes it easy to write custom reducible expressions, all you have to do, is to inherit from Expression, override CanReduce and NodeType, and implement the Reduce method:

public abstract partial class CustomExpression : Expression {

    public override ExpressionType NodeType {
        get { return ExpressionType.Extension; }
    }

    public override bool CanReduce {
        get { return true; }
    }

    /* ... */
}

If we take, for instance, our UsingExpression, mapping to the C# using statement, Reduce is implemented as follows:

public override Expression Reduce ()
{
    var end_finally = Expression.Label ("end_finally");

    return Expression.Block (
        new [] { variable },
        Expression.Assign (variable, disposable),
        Expression.TryFinally (
            body,
            Expression.Block (
                Expression.Condition (
                    Expression.NotEqual (variable, Expression.Constant (null)),
                    Expression.Block (
                        Expression.Call (
                            Expression.Convert (variable, typeof (IDisposable)),
                            typeof (IDisposable).GetMethod ("Dispose")),
                        Expression.Goto (end_finally)),
                    Expression.Goto (end_finally)),
                Expression.Label (end_finally))));
}

Just like a C# using, it will try to execute a block, and whether the block triggered an exception or not, it will call IDisposable.Dispose on the subject of the using statement.

Wasn’t that easy? Even if those nodes are pretty general, writing custom and specific nodes is a powerful and simple way to factorize code when generating code using the Expression Tree API.

In the end, I added in Mono.Linq.Expressions a CustomExpression type, our base class for well, custom expressions. We provide a CustomExpressionVisitor which extends the standard ExpressionVisitor to support our custom expressions, and the CSharpWriter has been updated to support them as well:

[Test]
public void Using ()
{
    var arg = Expression.Parameter (typeof (IDisposable), "arg");

    var lambda = Expression.Lambda<Action<IDisposable>> (
        CustomExpression.Using (
            arg,
            Expression.Call (typeof (Console).GetMethod (
                "WriteLine", new [] { typeof (object) }), arg)),
        arg);

    AssertExpression (@" 
void (IDisposable arg)
{
    using (arg)
    {
        Console.WriteLine(arg);
    }
}
", lambda);
}

Next feature for Mono.Linq.Expressions: using Mono.Reflection to turn a delegate into an expression of this delegate.