Jb in a nutshell

Présentation MonoTouch à Paris

Jb Evain — Tue, 06 Oct 2009 16:56:00 +0200

Certains vont à la Tour Eiffel, d’autres se ruent dans des Starbucks, mais moi, quand je monte conquérir Paris, je vais aux réunions alt.net. C’est souvent amusant, et toujours de bonne compagnie. En plus j’ai l’honneur, que dis-je le privilège d’être en haut de l’affiche de la prochaine réunion. Évidemment, je n’aurai pas de complet bleu, moi j’aime les rayures.

Lundi prochain, 12 octobre, je vais présenter MonoTouch. MonoTouch, c’est un produit de Novell, qui permet d’écrire des applications pour iPhone en C#, comme on le ferait pour le framework .net. Ou presque.

Cette session sera donc l’occasion de voir ce que l’on peut faire avec, de rentrer un peu dans les détails du pourquoi du comment ça marche, et d’en discuter. Parce que c’est vraiment ce qui fait le charme de ces rencontres alt.net. Les discussions post-session, un verre à la main, une cravate dans l’autre.

Notre hôte pour cette session sera Zenika, qui m’a réservé une page rien que pour moi, et surtout qui permet de s’inscrire. Dépêchez vous de vous inscrire, le nombre de place est probablement limité.

Le synopsis de la session:

Aujourd’hui, dans l’informatique, tout le monde ou presque a entendu parler de l’iPhone. Et à plus forte raison depuis que celui-ci supporte le copier-coller. C’est aujourd’hui un acteur important dans le monde de la mobilité, et sa démocratisation rapide en fait une plateforme de choix pour développer des applications, aussi bien pour l’entreprise que pour le particulier.
Si à l’origine développer pour cette platforme signifiait utiliser le langage d’Apple, l’Objective-C, on assiste à la naissance de solutions tierces destinées à fournir d’autres moyens de programmer pour l’iPhone.
Cette session sera l’occasion de voir non seulement comment il est possible de réutiliser son code et ses compétences .net sur l’iPhone grâce à MonoTouch, mais aussi d’en expliquer le fonctionnement.

À lundi prochain donc.

Rebasing System.Reactive to the .net CLR

Jb Evain — Thu, 30 Jul 2009 12:24:00 +0200

There has been a lot of interest towards the Rx framework lately. It’s definitely an interesting piece of software, and the video linked on channel 9 is quite fascinating. The only issue is that the only assembly you can get right now is compiled against Silverlight, making it impossible to use on the traditional .net framework. Impossible, really?

Well, I for one would not recommend it, but if you really can’t wait to try it, you could use this little piece of code which uses Cecil to turn the assembly compiled against Silverlight into an assembly that will work on the .net framework. Of course the assembly will lose its strong name in the process. But at least, it will be usable.

For instance, here’s the sample that Jafar is showing on his blog, running on Mono:

Update: don’t miss the comment Sébastien posted, and how he would do it with Reflexil.

Getting the field backing a property using Reflection

Jb Evain — Fri, 01 May 2009 10:46:00 +0200

Update: the code has been moved to its own project page.

Let’s consider you’re writing a LINQ provider. And that you need to opimize the following LINQ query:

from Person p in db where p.Age > 18 select p;

Let’s add a constraint. The underlying storage engine stores data according to the field name. That would mean that when generating the query for the underlying storage system, you’ll have to map

p.Age

into something that the underlying storage system will understand. In that case, a field. And all you have is a MemberExpression, giving you a PropertyInfo.

The issue here is that you have no way to get the FieldInfo backing the property. If you think about it, it’s normal. The setter and the getter of a property being traditional methods, they can contain any kind of code. Meaning that you can’t always find a field backing the property.

But in that case, it’s ok, we’re only interested in those forms of properties:

public int Age { get; set; }

private string name;

public string Name {
    get { return name; }
    set { name = value; }
}

Of course here what’s interesting is how to actually get the field. I’ve used the Reflection based CIL reader I wrote about yesterday. I disassemble the body of either a getter or a setter of the property, and if it matches a simple IL pattern, that is, if it looks to be a property backed by a field, I simply return the field.

To do the actual IL matching, I re-implemented something Rodrigo and I wrote when we were working on instrumenting assemblies at db4o. The code itself is pretty neat.

Anyway, that’s another opportunity to write a simple extension method:

public static FieldInfo GetBackingField (this PropertyInfo self)

Again, you’re more than welcome to have a look at the implementation. Don’t forget that it depends on the Reflection based CIL reader.

Reflection based CIL reader

Jb Evain — Thu, 30 Apr 2009 22:25:00 +0200

Update: the code has been moved to its own project page.

As I was writing, earlier this month, when I worked on a static aspect weaver, the first library we used, to programmatically retrieve the CIL bytecode, was a library published by Lutz Roeder (the original author of the most famous Reflector tool), called ILReader.

It suffered from a number of limitation, and you were tied to the whole System.Reflection infrastructure. Which, during the .net 1.0 time, was somewhat limited, and lacked a few features required to get access to every single detail in an assembly, including the CIL bytecode. It evolved since, for instance, starting from .net 2.0, there’s a GetILAsByteArray on a MethodBody used to get the raw CIL code.

Anyway, most of those concerns were addressed by Cecil, but still, for some use-cases, it could be nice to be able to have access to the CIL bytecode at a higher level of abstraction than a plain raw byte array.

On .net, you can use a library also named ILReader, but it has a few checks that are specific to .net, there’s no information about a license of the code, and also, I’m not especially fond of the way instructions are represented.

So last time, for an hack I’ll soon write about, I extracted Mono.Cecil’s Instruction type, and wrote a cute extension method, or rock, as I like to call them. Its signature:

public IList<Instruction> GetInstructions (this MethodBase self)

I would have loved to declare the extension method on the System.Reflection.MethodBody type, to make things more consistent with the methods it already has, but there’s no cross platform way to get a System.Reflection.MethodBase from a System.Reflection.MethodBody.

Anyway, it’s terribly easy to use if you’ve already used Cecil. The only difference is that for branches, the operand is the offset as an integer, not the target instruction. As a sample usage, here’s a (very) incomplete CIL reflection based disassembler:

static void PrintByteCode (MethodInfo method)
{
    foreach (Instruction instruction in method.GetInstructions ())
        PrintInstruction (instruction);
}

static void PrintInstruction (Instruction instruction)
{
    Console.Write ("{0}: {1} ",
        Labelize (instruction.Offset),
        instruction.OpCode.Name);

    switch (instruction.OpCode.OperandType) {
    case OperandType.InlineNone :
        break;
    case OperandType.InlineSwitch :
        var branches = instruction.Operand as int [];
        for (int i = 0; i < branches.Length; i++) {
            if (i > 0)
                Console.Write (", ");
            Console.Write (Labelize (branches [i]));
        }
        break;
    case OperandType.ShortInlineBrTarget :
    case OperandType.InlineBrTarget :
        Console.Write (Labelize ((int) instruction.Operand));
        break;
    case OperandType.InlineString :
        Console.Write ("\"{0}\"", instruction.Operand);
        break;
    default :
        Console.WriteLine (instruction.Operand);
        break;
    }

    Console.WriteLine ();
}

And of course, you’re welcome to have a look at the implementation, under the MIT/X11 license.

pdb2mdb and Mono.Cecil.Pdb

Jb Evain — Mon, 27 Apr 2009 20:07:00 +0200

I always complained about the fact that debug symbols were not portable between different CLR implementations. The .net CLR consumes pdb files, which is an undocumented format. Another file format was added to the ECMA-335 in a late revision. I wrote about this file format a while ago.

To sum up, it was added very late while Mono already started to use its own format (mdb) and the .net CLR doesn’t understand it anyway. So even if it’s not a bad format (it could use some improvements, like a GUID heap similar to the one in a .net assembly), basically no one uses it in the real world.

As mentioned in a recent post, the CCI contains an interesting piece of code, a managed pdb reader, licensed under the Ms-PL. I extracted it, and used it to be able to better share debug symbols between the .net CLR and Mono.

pdb2mdb

Robert Jordan, a long time Mono contributor, first wrote a tool named pdb2mdb, to convert a pdb to a mdb. The issue is that it was based on a combination of COM and the mixed mode assembly ISymWrapper which comes with the .net framework. All in all, it means that this version of pdb2mdb could only run on on the .net framework on Windows.

With the managed pdb reader, it was very easy to write a fully managed pdb2mdb tool. It’s now available in svn, and it will come with every other developer tool, such as ilasm or the linker. It’s very easy to use. Say you’re deploying a .net application on Linux, you have an assembly Foo.dll, and a Foo.pdb file, just use:

pdb2mdb Foo.dll

And the tool will generate a file Foo.dll.mdb, that Mono can use to display line information in stack traces.

Mono.Cecil.Pdb

Mono.Cecil.Pdb is an assembly that you use together with Cecil, to have line information at the IL level. It’s used by tools such as Gendarme, or MoMA, to help diagnose and locate issues.

I’ve integrated the managed reader, and the folks from NDepend were kind enough to beta test it. After a few fixes, the managed reader passed all NDepend tests, and was performing a lot better than its unmanaged counterpart. It’s now the default, and only the pdb writer uses the ISymWrapper approach.

It would be an interesting challenge for someone to try to write a managed writer from the information gathered in the reader. It may not be easy though.

Converting Delegates to Expression Trees

Jb Evain — Wed, 22 Apr 2009 21:03:00 +0200

Back when I was working at db4o, we had fun implementing a mechanism somehow similar to LINQ, to have strongly typed queries expressed using code itself. The implementation uses Mono.Cecil and Cecil.FlowAnalysis to decompile a delegate into an AST, that db4o’s query optimizer can process.

Since .net 3.5, an API, System.Linq.Expressions, can be used to get a representation of a C# lambda expression into an object graph. An expression tree. .net 4.0 will add support for statements to this API, but as far as I know, the language itself hasn’t been updated to produce those new nodes.

Anyway, a few days ago, someone on Stack Overflow, asked how to turn a delegate into a LINQ expression tree. There’s no builtin feature to do that, it’s not a straightforward process. You basically have to decompile the compiled method. I guess it’s a good thing that I’m working on a decompiler, if I need to decompile something.

Tonight I wrote a short spike to verify the feasibility of my idea, and it turns out to be pretty simple. Sample:

static void Main ()
{
    Func<int, int> magic = i => i * 42;

    Expression<Func<int, int>> expression =
        DelegateConverter.ToExpression (magic);

    Console.WriteLine (expression.ToString ());
    // prints: i => i * 42
    Console.WriteLine (expression.Compile ().Invoke (1));
    // prints: 42
}

DelegateConverter is implemented as a simple visitor which walks over a Cecil.Decompiler AST, and generates, if possible, the according Linq Expression Tree. Pretty cool isn’t it?

You can browse the code of the spike. Keep in mind that it’s nowhere to be complete, and that it’s just a proof of concept. Still, I think it’s a pretty cool usage of the Cecil.Decompiler library.

Cecil and the CCI

Jb Evain — Sat, 18 Apr 2009 14:26:00 +0200

Quite a number of friends pinged me about the recent release of the CCI, under the Ms-PL, and were curious about my take on it, and its effect on Cecil and its ecosystem.

First of all, there’s a bit of a story here, and I’ll write it here for those who like me, love software history. Back in the years 2003 and 2004, I was working with Thomas Gil, one of my mentor and programming hero, on one of the first static aspect weaver on .net, AspectDNG, now abandoned. I was actively researching better ways to do CIL injection.

We went from raw IL text manipulation, to Reflection and Reflection.Emit using Lutz’s ILReader library, to RAIL, until I decide to work on Cecil.

In the meantime, I’ve stumbled upon ILMerge, a tool from Mike Barnett, and mailed him to ask what powered the tool, and he put me in contact with Herman Venter, the man behind the CCI effort. I wrote Herman a couple of mails, in a terrible English, and begged him to push for a release of the CCI under a license we could use in AspectDNG. That was in March 2004.

As you can guess, it quite didn’t work out at that time, so I started working on Cecil. A few weeks after, Miguel blogged about the need of such library. He already had the Mono Linker in mind. I mailed him, got SVN access, checked in the beginning of Cecil, got Sébastien interested, etc.

I had the opportunity to be invited by Microsoft to attend an informal AOP workshop the year later, and to met with Herman, which I remember as a very nice person. I am not sure he remembers the terribly shy kid that did a terrible presentation in a terrible English. But all in all, I’m happy that five years later, my request went through.

Now the CCI release in its own CodePlex page is not really a big event, as it was already released and licensed under the Ms-PL, as it’s part of Sandcastle.

Anyway, Cecil is quite mature in its current form, it’s used by a fair number of (known) applications (please help to improve the list), and I’m currently working on two things.

The first one is a refactoring of Cecil, which vastly reduces memory consumption as well as reading/writing time. Hopefully I’ll have a beta in a month or so. We have great plans for this version of Cecil, and it’s consuming a lot of my time, more on this later.

The second one is an extensible decompiler, Cecil.Decompiler, that will greatly benefit from the Cecil refactoring. The time I dedicate to it is a bit phagocyted by the Cecil refactoring right now, but it’s certainly one of my favorite project.

The CCI is a combination of Cecil, the decompiler, and something to write a decompiled AST back, which will be the natural evolution of the Cecil decompiler. Note that the CCI decompilation/compilation process is not extensible. Now that it’s open source, you can hack it yourself, sadly, the CCI code is well, a bit messy to be polite, or not exactly a joy to read. Also you probably won’t be able to contribute back to the CCI.

Anyway, it does its job alright, and so does Cecil. Choice is always good, let’s welcome the CCI in the small family of such tools. I, for one, will surprisingly keep hacking on and with Cecil :)

To conclude on a very positive note, the fantastic thing about this release is that the CCI contains a fully managed PDB reader and writer. That’s great news as so far, we failed to get any details about this file format. This means that we can now implement a fully managed Mono.Cecil.Pdb support, and that’s just great.

UPDATE: it appears that only the PDB reader is fully managed, the PDB writer is just a wrapper over the COM stuff, just like the current implementation of Mono.Cecil.Pdb. Well, at least it’s a start.

Offline

Jb Evain — Sun, 05 Apr 2009 11:52:00 +0200

I’ll be offline until April 20th.

Mono embedder managed tool-chain

Jb Evain — Fri, 09 Jan 2009 12:07:00 +0100

Miguel blogged a list of iPhone applications that were made using Unity3D, and scripted with Mono.

I’m delighted to have cute applications using Mono to show, and one involves raptors!

One of the concern for Unity3D is to reduce the size of the download as much as possible. And Mono certainly weight a little bit in the download. We have a page on our wiki which describes how to reduce the size of the runtime, but am writing today about what Unity3D uses to reduce the size of the different managed parts, be it the core libraries, or the managed part of the game itself.

They are using two different tools to reduce the size of the assemblies. So first of all, once the application is compiled, the first tool they use is the Mono Linker. I already had the occasion to write about the linker, as it is a tool that I started writing during the second Google Summer of Code I spent as a student, and that I’ve worked on when I joined Novell. The linker is today a mature piece of code, that is exercised during every single Mono build, as it is used to produce the Moonlight 2.0 version of our class library.

So they are using the linker, and this makes sure that everything that is not needed by the game or the engine is removed from the assemblies.

When the linking stage is achieved, they pre-compile the assemblies to native code using our AOT (Ahead of Time) compiler. This is necessary, as the iPhone prevents any JIT to run.

After AOT, you end up with a native binary, and the original managed binary, which still contains the intermediate code. If the assembly has been completely AOTed, this intermediate code is no longer necessary.

Here comes the second tool they’re using. This is a tool which is pretty new, and that I wrote for this specific usage. It’s called `mono-cil-strip`, and is now built along the traditional tools that we ship, such as ilasm or the linker. It uses a special mode of Cecil I hacked on, which preserves the original metadata structure of the assembly, but empties every single method body. It’s necessary to keep the native binary in sync with the managed binary, while still removing parts of it.

If you’re compiling your assemblies ahead of time, and looking for some bytes to save, here’s a neat way to do so.

Sadly I don’t have numbers handy, so I’ll encourage you to give it a try, but here we are, every single iPhone application produced using Unity3D went through Cecil (twice!).

And that’s pretty cool :)

The Thing About ExpressionVisitor

Jb Evain — Tue, 16 Dec 2008 14:25:00 +0100

There was a discussion recently about why the class ExpressionVisitor, which every programmer that developed a LINQ provider or code manipulating .net 3.5 expression trees knows about.

I certainly agree that this class could have been made public, as basically everyone ends up embedding it. Even if it could use a little refactoring to visit lists.

Thing is that I have a naming issue with this class ever since I saw it on Matt’s blog. To me it does more than just visiting an expression tree. It allows you to transform it. And when I’m facing an expression tree, I often like to sometimes visit it, and at some other times, transform it.

That’s basically the reason why I have both a:

class ExpressionVisitor {

    public void Visit (Expression expression)
    {
        // ..
    }

    // ...
}

And a:

class ExpressionTransformer {

    public Expression Visit (Expression expression)
    {
        // ..
    }

    // ...
}

Simply because I write a lot of code that would look like:

SomethingExpression VisitSomethingExpression (SomethingExpression expression)
{
    if (Foo (expression))
        return expression;

    if (Bar (expression))
        return expression;

    Baz (expression);
    return expression;
}

And it looks a lot better (and is also a tiny bit faster) this way:


void VisitSomethingExpression (SomethingExpression expression)
{
    if (Foo (expression))
        return;

    if (Bar (expression))
        return;

    Baz (expression);
}

So if they make the current ExpressionVisitor public, having a real non transforming ExpressionVisitor will introduce an interesting naming issue.

I submitted a bug to the DLR CodePlex, as the namespace System.Linq.Expressions will come from the DLR in .net 4. What about voting for it?