jimmy.thinking

IronRuby and IronPython for .NET 4.0 Beta 2

jschementi@gmail.com (Jimmy Schementi) — Thu, 22 Oct 2009 09:20:00 +0000

Quick download links: IronRuby | IronPython | .NET 4.0 Beta 2

I’m pleased to announce the release of the IronRuby and IronPython CTPs for .NET 4.0 Beta 2. This is the third preview of IronRuby and IronPython running under the Dynamic Language Runtime that is built directly into .NET 4.0. As before, this release allows you to use IronRuby objects and types as .NET 4.0 dynamic objects from within C# and Visual Basic code. These binaries are roughly equivalent to the functionality found in IronRuby 0.9 and IronPython 2.6 RC. Take a look at Harry Pierson’s blogpost about this release as well, especially if you’re more into snakes than gemstones.

Here's a small example showing just how powerful the new dynamic feature is for taking advantage of dynamic language functionality in statically typed languages:

Here is the IronPython version as well. It basically loads up the desired scripting engine (in this case, Ruby), loads a script, instantiates the “Mock” class defined in the script, and calls a method on an instance of “Mock” which the C# compiler cannot possibly know of; the Mock class dynamically responds to any method call with a random result. The “dynamic” keyword indicates that the C# compiler should use the DLR to dispatch method calls at runtime, rather than trying to validate the method calls at compile time. This enables a very clean integration between dynamic languages and static languages, and even gives you the ability to define your own dynamic objects. Chris Burrows has a great blog series on how C# “dynamic” works.

Try it out

To try out this release:

Install NET 4.0 Beta 2 or Visual Studio 2010 Beta 2
Download either IronRuby or IronPython
Follow any of the many dynamic walkthroughs online. This would be a good start.

Build from source

To build IronPython or IronRuby for .NET 4.0 Beta 2 from source, get the sources from:

IronPython: CodePlex
IronRuby: GitHub (direct download)

Note: use the "v4" build configurations.

Caveats

Being a “preview” release, these releases haven’t gone through the usual testing that the normal binary releases go through. However, both IronRuby and IronPython will be supported on .NET 4.0 when it ships for real. Also note that the these releases will run only under .NET 4.0 Beta 2.

There is no project-system integration between IronRuby and IronPython in Visual Studio 2010 Beta 2, so don’t expect to find File->New Project->IronPython or something like that; this release is just about the integration between .NET 4.0 and DLR-based languages. If you’d like that to not be the case, please continue to let Microsoft know that on the open http://connect.microsoft.com issues for IronRuby and IronPython Visual Studio 2010 integration.

If you find any issues, please report them on the language’s respective CodePlex sites (IronPython | IronRuby).

IronRuby 0.9 Released!

jschementi@gmail.com (Jimmy Schementi) — Sun, 02 Aug 2009 00:54:00 +0000

I'm pleased to announce a new release of IronRuby: 0.9!

Download IronRuby 0.9

You can also get the source code for this release

It’s been a long ride since IronRuby was announced at MIX ‘07, but now all eyes are focused on getting it to 1.0. The past couple of versions have made vast compatibility and performance improvements, and 0.9 marks the last set of major features that will be added to IronRuby. From now on, all IronRuby languages will be primarily focused on bug fixing and anything else the community deems necessary to call IronRuby a 1.0 release. These improvements be delivered as point-point releases (0.9.1, 0.9.2, etc) until you decide it is ready to call 1.0.

Though IronRuby is breaking away from it’s conference-driven development schedule, this release comes only a week after OSCON 2009 where I talked about using IronRuby with Mono and Moonlight, and scripting open source apps.

What's in the Release?

Library performance was a big focus for this release; basically going though the Ruby Benchmark suite and making sure any obvious slowness was fixed. As I said in my previous post about OSCON, IronRuby is approximately 2x faster than MRI when running the benchmark suite, but in the near future a complete evaluation of IronRuby’s performance will be done and published on the website.

Antonio Cangiano has already published benchmark results between IronRuby 0.9 and Ruby 1.8.6, and things look really good for IronRuby.

On the compatibility front, the Win32OLE Ruby library is now available in IronRuby. This builds on top of IronRuby’s existing COM interop from version 0.5, letting you script any Windows component/application that exposes a COM interface. Though it hasn’t been fully tested yet, this will make things like Watir work on IronRuby.

Lastly, there have been interop improvements with .NET, most notably making Generic types more friendly to all the crazy things Ruby can do to them, and also with other DLR languages, making it really easy to call IronPython from IronRuby.

For more detailed information, please see the CHANGELOG (which includes all commit messages for the release ... not just "syncing to head of tfs")

Want to find out more about the release? Read on:

Calling Python code from Ruby
Win32OLE support
Rubyizing Generics
Some more notable changes
Bugs closed

Calling Python code from Ruby

In an effort to allow Ruby to call Python code easily, IronRuby provides a way to importing files of other DLR languages, giving you access to the DLR scope. Here's a basic usage:

IronRuby#require(path), load(path) allow you to include script files, just like Kernel#require, #load except that they execute the given script against a new Scope and return it, instead of just returning true/false/nil. If the target is an assembly they return the loaded assembly. Keep in mind you can still use Kernel#require, #load to use other language's script files, but if the language requires you obtain an object representing that file (in Python it's called a module) you'll have to get it elsewhere (see below).

There are other methods to help you use other languages from Ruby:

IronRuby#loaded_scripts has also been added to returns a dictionary that holds on loaded scripts and their Scopes, in-case you used Kernel#require to load a script, and need its Scope at a later time.
IronRuby#loaded_assemblies returns an array of assemblies already loaded in the current runtime; though this isn’t needed for DLR language interop it makes sense to include along with IronRuby#loaded_scripts.
IronRuby#globals returns the current ScriptRuntime.Globals scope; this allows you to easily use Python files and work with the modules they define.

Keep in mind these concepts don’t only apply to Python; they can be used with any language that uses first-class scopes.

Win32OLE support

Since IronRuby 0.1 was released we’ve said Win32OLE was not going to be supported, due to the lack of general COM support in the DLR. However, when the DLR implemented a COM binder for the new dynamic feature in C# 4.0, this allowed IronRuby and IronPython to easily implement COM support. IronRuby 0.5 had this support, but in this release we provided win32ole.rb to implement Ruby’s Win32OLE API. Here’s a simple automation example which opens Excel, adds some data, generates a chart, and rotates the chart. taken from the Ruby Pragmatic Programmer’s Guide (aka. the pickaxe book), which now works in IronRuby:

Rubyizing Generics

Note: this feature didn’t make it into 0.9, but it is in the sources and will be part of 0.9.1

IronRuby has great .NET framework integration, so code written with a CLI-based language (C#, VB, F#) can be called from IronRuby, without it feeling like interop. As we work towards a stable 1.0 release and this integration continues to be polished; in this release we most notably made working with generics a bit friendlier to Ruby.

The first improvement has to do with re-opening a generic class. In the past you couldn’t just re-open a generic class and assume anything you added to it would be found on other versions of that generic class (eg. adding methods to List[] wouldn’t have them show up on List[String] and List[Fixnum]). Now the Ruby class corresponding to the CLR generic type definition is a super-class of all instantiations of that type, making this possible:

There's a much more usability improvements to generics and .NET interop in general in this release, so either take a look at the CHANGELOG or keep an eye on the .NET documentation.

Some more notable changes:

Fix to Module#autoload so it doesn’t squash an existing constant
Fix parsing of YAML time strings
Replace DLR closures by a Ruby-specific implementation for performance
Implement StringIO#ungetc – used in ActiveSuppor
Re-implements “tr”, “tr!”, “tr_s”, “tr_s!”. The new implementation is almost 12x faster.
Reimplements “join” to achieve better perf.
Change ComBinder class to internal.
Improves the ExpressionVisitor API
Improves implementation of RubyArray. A loop doing Array#shift is not shifting the entire array content left every iteration.
Add hosting of IronPython to hosting tutorial
Implements interpretation of coalescing expression
Implements Kernel#rand, Kernel#srand.
Improves scope allocation perf.
ironruby tutorial in Silverlight
More fixes to win32ole to get the ADO DBI driver working
String, Array, and Bignum fixes (thanks Daniele Allessandri!)
Explicit x64 support with ir64.exe, and force ir.exe to run in a 32bit process on a 64bit OS.
Implements basic debug views for classes that implement IRubyObject

For more details, see the CHANGELOG.

Bugs closed

Here are all 26 bugs closed since the last release (2009-07-02). You can see more information about each one on CodePlex.

1841 	Rubygems Test Failures in 'hash'
1506 	-X:PrivateBinding does not enable referring to classes that have internal visibility
1533 	Unverifiable code generated by mspec :lang
1678 	ArgumentError when calling System.String..ctor(Char[], int, int)
1211 	Adding a random object to Time which responds to to_f
910 	YAML does not properly deserialize Time values.
1626 	Rubygems failures
964 	overriding unsafe methods
1537 	rand doesn't work collectly
468 	Better error message for running ir.exe on pre-.Net 2.0 SP1
1787 	All-caps .NET method names are mangled strangely
1521 	Access is allowed to internal fields
1502 	alias_method fails for :do
821 	File.expand_path does not support a line number after filename
1509 	Proc.to_s should include line number where the block was declared
1501 	WinForms broken
1400 	$? is not always Process::Status
1345 	load_assembly() should work
1344 	System.Action.new does not work
1306 	Cannot call CLR constructor of builtin type
1184 	public(:foo) does not work correctly for mixed-in methods
1060 	visibility of send :define_method
917 	Passing a Ruby array to a .NET method that expects an IEnumerable derivative fails with GetEnumerator call
783 	Assert in SetMethodBasesNoLock when calling #== on Ruby class inheriting from CLR class which overrides Equals
761 	Wrong behavior when calling redefined methods on object instances
1470 	Can't call the BigIntegerOverload of a method with a DefaultProtocol Attribute on the BigInteger attribute

Enjoy!

Please test out IronRuby 0.9 and let us know if you have any issues. We hope you enjoy this release! Please submit issues to CodePlex and help prioritize features for 1.0.

Technorati Tags: IronRuby,Release,OSCON

IronRuby at OSCON 2009: Mono, Moonlight, and scripting open source apps

jschementi@gmail.com (Jimmy Schementi) — Fri, 24 Jul 2009 01:08:00 +0000

Today I’ll be giving a talk at OSCON 2009 titled “IronRuby 0.9”. For any new readers, IronRuby is an Open Source implementation of the Ruby programming language for the Microsoft .NET Framework; it brings the expressiveness of Ruby to .NET, and the capabilities of .NET to Ruby. I’ll walk you through the talk in this post:

Road to IronRuby 1.0
Running the same application on the desktop and in the browser, on most major platforms (Windows, Mac OS, Linux) and most major browsers (Firefox, Safari, Internet Explorer)
Detour – Gestalt: Running Ruby and Python in the browser, again.
Using code written in any C# and Python from Ruby without it feeling foreign.
How IronRuby has improved in the past 6 months, and how you can help IronRuby become a 1.0 quality implementation.

Road to IronRuby 1.0

It’s been a long ride since IronRuby was announced alongside Silverlight at MIX ‘07, but in a few days the 0.9 version will be available, and all eyes will be focused on getting it to 1.0. The past couple of versions have made vast compatibility and performance improvements, and 0.9 marks the last set of major features that will be added to IronRuby. From now on, all IronRuby languages will be primarily focused on bug fixing and anything else the community deems necessary to call IronRuby a 1.0 release. These improvements be delivered as point-point releases (0.9.1, 0.9.2, etc) until you decide it is ready to call 1.0.

But we’ll dive into the numbers and metrics at the end; let’s talk about why IronRuby is useful in your toolkit.

Cross-platform, cross-browser Ruby implementation

Whenever people hear .NET [they first think about a top-level domain, but when in the context of “Microsoft”] they immediately think “Windows”. But Miguel and the Mono team have shown that isn’t reality. They’ve built a open-source implementation of .NET which runs on Linux, Mac OS, and also Windows. They are also building a open-source Silverlight 2 implementation, called Moonlight, allowing Silverlight applications to run on Linux browsers supporting Mozilla’s plug-in architecture; Silverlight 2 runs on Intel Macs and Windows. Since IronRuby runs on .NET and Silverlight, it also runs on Mono and Moonlight. Let’s explore that a bit:

With Microsoft.NET and Mono, IronRuby can run on Windows and Linux (anywhere Mono runs, including Mac OS).

You’ll notice that real Ruby code runs, it’s running on a Unix environment on Mono, and .NET types can be accessed in a Ruby-esk way, even generic types.

Because IronRuby integrates directly with .NET, you can use any existing .NET code from IronRuby. This means that any frameworks, be them Windows Forms, Windows Presentation Foundation (WPF), or GTK(#), can be used from IronRuby because .NET and Mono [respectively] have CLI bindings for these frameworks. Mono even has a Windows Forms implementation that translates to GTK, so the same application can run unmodified on both implementations. Mono only supports the “Silverlight” subset of WPF, which I’ll elaborate on in a bit.

There’s also a tiny subset of .NET which is part of a browser plug-in called Silverlight; Microsoft’s implementation runs on Windows and Mac, in Firefox, Safari, and Internet Explorer (though any browser which supports ActiveX, Mozilla’s, or Safari’s plug-in architecture should work fine, like Opera or Chrome).

The IronRuby tutorial application runs as a WPF desktop application (on the left), as well as Silverlight application (on the right).

Also, Silverlight 3 supports running applications out of the browser, very simply on Windows and Mac OS by right-clicking on the Silverlight application and selecting “Install”.

Through Moonlight the IronRuby tutorial app also runs on Linux:

Mono also supports running Moonlight applications through GTK on the desktop, just as a WPF application would work. While direct support isn’t in current daily builds of Moonlight, Miguel shows how easy it is to make the same code target the browser with Moonlight and the Linux desktop with Mono, just using C# code:

Note: the latest Moonlight stable release will not work for IronRuby, you must install the latest Moonlight build from their build bot: http://sublimeintervention.com:8123/GetFile.aspx?id=11487919

Detour: Gestalt –- Ruby and Python in the browser

My last blog-post talks about a new prototype from the MIX Online Lab called Gestalt, which lets you write Ruby and Python in place of JavaScript – literally in <script> tags. Here’s a very simple example:

It uses Silverlight to do this, so this will work in Moonlight as well.

Dynamic and Static language interop

IronRuby has great integration with the .NET framework APIs, so C#/VB code can be called from IronRuby without it feeling like “interop”. C# can also call IronRuby through the Dynamic Language Runtime (DLR) Hosting APIs. However, in .NET 4.0 dynamic method dispatch is part of C#, so calling IronRuby from C# is just as easy as a C# method call. Being that IronRuby is built on the DLR, it can interoperate with other DLR languages just as easily, so Python and Ruby play along nicely today, and future DLR languages as well.

A simple way to show that capability is by writing a unit test for a class in .NET; let’s pick “Stack”:

As you can see, creating .NET types, calling methods on .NET classes, and all the other language features Ruby has work against .NET types (even re-opening .NET classes).

To show the reverse, lets look at a very simple REPL implementation:

In less than 100 lines of code, you have a program that runs arbitrary Python and Ruby code. The DLR hosting API lets you run dynamic language code from a static language just from strings.

Expanding on these ideas, let us give the experience a upgrade to the 21st century. John Lam has shown this before at TechEd 2009: an editor that can double as a REPL.

Downlaod it here

This seems like a great idea, and it is even better once you start using it. It works like a normal editor, but selecting a block of code and pressing ctrl-e will run the code, showing the output directly below selected code as a comment, and ctrl-enter runs the current line. The code that gets run gets syntax highlighted. There is no need to have up-down-cursor history, as it’s all right there in the editor. If you need to modify something you’ve already run just, just modify it in the editor and re-run it! This integrates the editor-repl experience, making it a very valuable tool.

Note: this is not a new idea by any means, as TextMate does something similar, but no one seems to explain this as an upgrade to your traditional REPL.

Using this editor, it can be integrated into any .NET project as a lightweight way to experiment with adding scripting support. Using the hosting ideas I presented before, an application written in C# could use IronRuby to allow users to write applications against their object model, or extend it at runtime. Here’s a small example of hosting this editor in Witty, an open-source Twitter client running on .NET:

You can get this version of Witty from GitHub.

While it is great that you can so easily add scripting into a .NET application, it ends up exposing the design deficiencies in the application being scripted. In Witty’s case, the Twitter class (T) is very good, but the Window class (U) isn’t in good shape, as there are a ton of methods on the type, and it’s unclear how they should be used to drive the UI. Jon Galloway wants to merge this into Witty, so hopefully some of these refactorings can be made to enable scripting better.

I extend this offer to all open-source .NET/Mono projects – if you want to improve your architecture and add scripting support, please let me know and I’ll help out.

It's worth noting that .NET 4.0, the next version of the .NET framework, C# supports a "dynamic" keyword which will let you call Ruby or Python (or any DLR language) just by using method calls.

Project status

Compatibility is really good at this point. We run the RubySpec test suite, passing at (85%) and the test suites from popular Ruby frameworks, including Rails, Rake, RSpec, and RubyGems, on every check-in, and can run pretty substantial Rails applications. Increasing compatibility bugs will definitely be the focus of the point releases approaching 1.0; mainly based on what you tell us needs fixing.

This upcoming version will boast support for the Win32OLE library, making existing Ruby applications with interact with COM work as they do in MRI, like Watir.

Performance is also in a very good place. This past version has pushed on the Ruby benchmark suite, and now we are >= 2x better than MRI on almost all benchmarks. Those numbers will be published on http://ironruby.info around the same time IronRuby 0.9 comes out. The above image shows that IronRuby (in blue) is normally significantly faster than MRI (in red), except for a handful of cases. The is not the end-all-be-all metric to measuring performance, but it’s nice to see IronRuby continuing to improve here.

In prior releases, Rails startup has gotten dramatically better, going from 80+ seconds in March to ~20 seconds now. Read more about Rails performance is the IronRuby 0.6 release.

Interesting IronRuby usages

Now that IronRuby is approaching 1.0, we're seeing very interesting usages of Ruby with .NET:

AutoCAD scripting -- Kean Walmsley
magic DSL for Windows UI frameworks -- Thibaut Barrère
Automated conversion of an app written in an in-house language (lang runs on FORTRAN77) and MOTIF UI “to” IronRuby and WPF -- Robert Brotherus
Point of Sale application based on WPF, IronRuby and FoxPro -- Nick Rickets

If you know of more interesting usages, please send them my way!

Thanks!

Thanks for reading this far! Keep an eye out on this blog and the ironruby website for the upcoming 0.9 release.

Technorati Tags: IronRuby,OSCON,Mono,Moonlight,Silverlight

Gestalt: Ruby and Python in the browser, again

jschementi@gmail.com (Jimmy Schementi) — Wed, 22 Jul 2009 03:06:00 +0000

Today Mix Online Lab released a prototype implementation of Ruby and Python in the browser, named Gestalt. Feeling a bit of de ja vu? If not, you can ignore this post and just see what it’s all about. Regardless, I suggest you read on.

A simple Ruby-based web page using Gestalt would look like this:

Gestalt adds “ruby” and “python” to the languages supported by the <script> tag. You can also include other files rather than writing the code in the HTML page:

My first reactions when hearing about this project was “Holy crap, wow, they got Ruby and Python running in the browser, that’s freakin’ awesome!” But I work on the Silverlight integration with IronRuby and IronPython, already letting you write Ruby and Python in the browser instead of JavaScript. Hmm. Not to my surprise, Gestalt uses IronRuby and IronPython in Silverlight to accomplish this. But damn, wouldn’t it be awesome if this is how the world worked?

For any new readers, IronRuby and IronPython are implementations of Ruby and Python for the .NET Framework, utilizing the Dynamic Language Runtime (DLR).

Gestalt is a effort to show what it would be like if there were other standard languages in the browser, other than just JavaScript. The Mix Online Lab focuses on using Microsoft technology in ways that make non-Microsoft developers interested, so this project is right up their alley. They wanted to showcase the DLR languages in Silverlight as a way of showing how our efforts can make the web a better place.

So what are the difference between IronRuby and IronPython’s Silverlight integration and what Gestalt provides?

Again, you can already make IronRuby and IronPython based applications in Silverlight; Gestalt just puts a new spin on how to do that. IronRuby and IronPython use the XAP file for the application packaging, including script files, and provides a development-tool called Chiron to auto-package the app, providing the edit-refresh experience that is familiar for a web browser app. Chiron is also used to write the package to disk so it can be deployed to any web-server. This model works great for getting new and existing Ruby and Python programs running in Silverlight, as the packaging is transparent to a developer. And it doesn’t fight Silverlight’s packaging conventions, so using Silverlight-specific features will be straight-forward from a DLR language.

Gestalt takes a much different approach using <script> tags, making the integration much more familiar to current JavaScript developers. They also handle all the Silverlight <object> tag stuff in gestalt.js, taking away the need to host Silverlight yourself.

Why Gestalt is interesting to me

The thing that I really like about Gestalt is that Chiron is not needed for development. Just edit the file and refresh the browser. No need to run something in the background generating a XAP file. Gestalt.js detects which languages the page’s <script> tags use, and picks a pre-packaged Gestalt-*.xap file, one for Ruby, one for Python, one for both, and one for none. This means your application could look something like this:

index.html (can contain Python and/or Ruby code)
gestalt/
- gestalt.js
- gestalt-rb.xap
- gestalt-py.xap
- gestalt.xap
- gestalt-x.xap

Very clean. The gestalt directory is just dropped onto your web-server (to be shared among all your applications), and gestalt.js is referenced from your HTML file, and your good to go. No running Chiron.exe, or installing Mono on the Mac to run Chiron.exe. I’m jealous.

So what’s the catch?

While Gestalt is a much simpler to get started writing Ruby and Python applications in the browser, it does not support a major feature of Ruby and Python – accessing something like a file-system. For example, including other files at runtime uses the language’s underlying File access to read the file in and run it’s contents. Ruby uses “require” or “load”, and python uses “import” to do this.

IronRuby and IronPython in Silverlight normally support this, as all file reads are redirected to the XAP file. Since Gestalt’s XAP files are opaque to the the developer, they do not support File access. This is only an issue when you want to use existing Ruby or Python code in the browser. For example, existing code usually has more dependencies, especially into portion of the language’s standard library which is written the language itself. Usually you’d just put these files in the folder that Chiron is auto-xapping for you, and that’s it. However, Gestalt has no way of doing this today.

Making it better

A possible solution is to just cache the contents of <script src> tags, and use the languages path resolution mechanism to test if those files have already been downloaded. If it finds a match, serve the file’s contents. If not, it could either fail fast and say that your missing a <script src> reference, or it COULD download the file synchronously (and still cache it) and display a warning that you are missing a <script src> reference.

Unfortunately, this only works for script files; what if you just want to download a txt file? Having <script src=”foo.txt”> is not going to fly. Maybe using <link> tags could work?

Why can’t we just download them synchronously? No! Though the current bits do have a synchronous downloading mode, it’s only experimental and should not be used in production. Synchronous downloads are significantly slower since only one download can occur at a time. Downloads need to be synchronous since “opening a file” is expected to block until it is complete. Also, IronRuby/IronPython do not support continuations (for good reasons), so the code cannot be “paused” while a file is downloaded, and then be “resumed” when done. And even if they did support continuations (as there is a way to get it working for the very specific case of file access), the downloads would still all occur one at a time, making the loading much slower. So downloading “on-demand” for file reads is not a good solution, and that’s precisely why the browser doesn’t do that today … so let’s not fight it.

Is Gestalt going to always be separate from IronRuby and IronPython?

No! :) While Gestalt’s first release was best to be separate, I’m very interested in merging the ideas Gestalt has back into Microsoft.Scripting.Silverlight … in-fact discussions have already begun on the IronPython Mailing List.

In short, Ruby and Python Silverlight applications are a bit too complex, and Gestalt shows us that it doesn’t need to be that way. Work has already begun to merge Gestalt and Microsoft.Scripting.Silverlight, so play with Gestalt and let me know what you think!

Technorati Tags: IronRuby,IronPython,DLR,Silverlight,Browser,Ruby,Python,dynamic languages

IronRuby 0.6 Released!

jschementi@gmail.com (Jimmy Schementi) — Thu, 02 Jul 2009 19:08:00 +0000

The IronRuby team is pleased to announce a new release of IronRuby: IronRuby 0.6!

Download IronRuby 0.6

You can also get the source code for this release

Staying true to our "Conference-driven development schedule," this release is in conjunction with ... wait, there is no conference this time! Just a good ol' monthly release of IronRuby, chock full of awesomeness. And just so happens to be the 4th of July weekend, so hopefully you have some downtime this weekend and can give the new release a spin.

What's in the Release?

Performance has been a major focus to this release, getting startup time quicker and optimizing built-in types. There has been more progress with Ruby compatibility (Cucumber works!) and .NET interop. Silverlight binaries are back in the release, and there are a couple of samples to help you learn IronRuby.

Most of these descriptions are from Tomas's very detailed code review emails, so thanks Tomas! For more detailed information, please see the CHANGELOG (which includes all commit messages for the release ... not just "syncing to head of tfs")

Lazy method compilation
CLR member enumeration
Generic-Method Parameter-Inference
Testing C# with Cucumber
Silverlight building and updated binaries
Samples

Lazy method compilation

Last release adaptive compilation was enabled for IronRuby, which uses a fast-to-start-up interpreter to start up the program, while code which gets run a lot is compiling on a background thread. This gave IronRuby significant performance improvements for large apps like Rails, which has improved startup by 2.5 times.

This release we've added "lazy method transformation" to the bag of startup performance tricks. In IronRuby 0.5, a method was transformed to the DLR AST as soon as it was defined. IronRuby 0.6 postpones the transformation until the first time the method is called. This significantly improves startup time. For example (not NGEN'd):

This has made Rails startup time 30% faster, coming in approximately 20 seconds on my dual-core laptop running Windows 7.

CLR member enumeration

Now methods like Module#instance_methods, etc, include CLR member names. Though it's a lot cooler than that. The array of methods returns contains strings for Ruby methods, and objects of type IronRuby::Clr::Name to represent CLR methods that can be called by either the actual CLR name or a mangled (Ruby-esk) name. ClrName has methods to_s, to_sym, to_str, <=>, inspect, and dump so that it can be used wherever a string can be used. The display string for the name uses single quotes so that you can easily distinguish CLR (dual) names from regular names (plain mutable strings). CLR strings display themselves as single quoted strings, so this fits nicely.

Even cooler, this works well for meta-programming:

A new set of define_method overloads, strongly typed to ClrName, have been added to enable this. They define the real method using the ruby_name and alias it using the clr_name. So both CompareTo and compare_to calls are intercepted.

Generic-Method Parameters-Inference

Thanks to some work by the IronPython team, IronRuby now has parameter inference for generic methods.

This needs some more TLC before it works perfectly, but this lets you use simple LINQ methods from IronRuby.

Testing C# with Cucumber

This release of IronRuby runs the Cucumber testing framework rather well. Try out Cucumber against some C# code.

Given a feature file (this being the addition.feature)

A step_definition file (calculator_steps.rb)

And a DLL:

Cucumber will test the addition feature:

3 scenarios (3 passed)
12 steps (12 passed)
0m0.753s

Silverlight building and updated binaries

This release re-adds Silverlight binaries to IronRuby, contained in the "silverlight" directory of the release. These bits have been integrated from the AgDLR project, and will be maintained in the IronRuby and IronPython source code repositories from now on. The AgDLR GitHub project will redirect to one of those for binaries for both languages in Silverlight. In addition, the SDLSDK CodePlex site will be redirecting to the IronRuby and IronPython CodePlex sites, for downloads, discussion, and issue tracking. AgDLR was a little side project to add new features to DLR Silverlight applications, and play around with Git. I'm very happy to see it merged back in with the languages.

Note: IronPython Silverlight assemblies are included directly in the release. They may be moved out to the separate package that contains IronPython desktop DLLs in the future.

See the integration commit for more information

Another notable change is that you can now build Silverlight binaries out of IronRuby's GitHub repository very easily. Given that you have Silverlight installed at C:\Program Files\Microsoft Silverlight\2.0.40115.0, this will build IronRuby for Silverlight:

msbuild Merlin/Main/Languages/Ruby/Ruby.sln
  /p:Configuration="Silverlight Release
  /p:SilverlightPath="C:\Program Files\Microsoft Silverlight\2.0.40115.0

You can also build IronPython for Silverlight in a similar manner. Aliases for this will be added soon, but if you need a custom build of the DLR languages for Silverlight, this is the way to do it.

Samples

Six samples are included in this release. The first three are desktop samples in the “/samples” directory. The last three are Silverlight samples in the “/silverlight/samples” directory:

Tutorial - An interactive IronRuby tutorial

DiskUse - A small WPF application which visualizes the disk usage for a particular directory

IRPowerShell - a small library and sample applications showing how to interact with PowerShell from IronRuby

Clock – a simple Silverlight sample

Photoviewer – do AJAX programming with IronRuby (ARAX =P)

REPL – Interactive console in the browser.

Some more interesting changes:

Improved DLR Interop: adds support for GetMember/SetMember with method_missing, Binary/Unary operators, and indexers.
Handling of CLR protected and private methods and properties
Reimplemented File.expand_path such that it does not use System.IO.Path.
This allows us to get better compatibility with MRI. The motivating reason was
that RSpec does File.expand_path("filename:linenumber")
Improves the implementation of singleton method dispatch.
These changes improve running time of specs significantly (2x)
Renames Method, UnboundMethod#overloads (plural) to
Method, UnboundMethod#overload (singular).
The old name is still available for now and throws an exception.
Implements adaptively compiled rules
Improves performance of Array#- from quadratic algorithm to linear.
Improves implementation of RubyArray

Bugs closed

Here are all 37 bugs closed since the last release (2009-05-19). You can see more information about each one on CodePlex.

1521  Access is allowed to internal fields
1502  alias_method fails for :do
821   File.expand_path does not support a line number after filename
1509  Proc.to_s should include line number where the block was declared
1501  WinForms broken
1400  $? is not always Process::Status	
1345  load_assembly() should work
1344  System.Action.new does not work
1306  Cannot call CLR constructor of builtin type
1184  public(:foo) does not work correctly for mixed-in methods
1085  Cannot call new on subtypes of builtin classes whose "new" method has optional arguments
1060  visibility of send :define_method
917   Passing a Ruby array to a .NET method that expects an IEnumerable derivative fails with GetEnumerator call
783   Assert in SetMethodBasesNoLock when calling #== on Ruby class inheriting from CLR class which overrides Equals
761   Wrong behavior when calling redefined methods on object instances
1470  Can't call the BigIntegerOverload of a method with a DefaultProtocol Attribute on the BigInteger attribute
1426  The located assembly's manifest definition does not match the assembly reference. (ctp dev10 beta1)
1441  Error with cyrillyc text in Sharepoint	
1352  Test Defects	
814   Allocator underfined for  (TypeError)
572   Error when running Cucumber examples with IronRuby
718   IronRuby ignores RUBYLIB environment variable
727   to_proc not working
1351  redist-libs should have rubygems-1.3.1	
466   ''.split(//, -1) returns [""] instead of []	
940   Can't inherit from abstract classes	
1028  Missing conversion from Duration to Fixnum (ActiveSupport)?
374   irails Foo: undefined method for OpenSLL::Random.random_bytes
459   throw FileNotFoundException => rescue Errno.NoEntryError
499   Pathname#cleanpath messes up the pathname
467   "igem install rails" fails
375   "ir script\server" causes a YAML parser error
461   Generic type conversion from Fixnum to Integer
674   autoload does not use File::SEPARATOR
1021  Time class instance
578   yield fails in eval'd code
605   Array#hash should properly handle recursive arrays

Enjoy!

Please test out IronRuby 0.6 and let us know if you have any issues. We hope you enjoy this release!

-- The IronRuby team

Technorati Tags: IronRuby,Performance,Cucumber,Testing,Silverlight,Release,Ruby

Simple Python testing in Silverlight with unittest

jschementi@gmail.com (Jimmy Schementi) — Mon, 29 Jun 2009 07:55:00 +0000

Since doing a post about Ruby testing of Silverlight applications, I’ve felt a bit guilty about leaving Python fans in the dark. However, like Ruby, it takes very little work to run Python unit test frameworks in Silverlight, since IronPython is a very compatible implementation of Python. Though my results aren’t as exhaustive as the IronRuby examples, here’s one simple example:

http://github.com/jschementi/europycon2009

zip download

The original example is from the Python website’s unittest documentation. Very little bootstrapping is required to get this running in Silverlight; this is u1.py code that is the start-script for the Silverlight app, which runs the tests:

I did have to make a modification to unittest.py; removing a call to sys.exit since Silverlight doesn’t support that, and just returning the exit code.

There’s also a GUI test runner (http://pyunit.sourceforge.net/pyunit.html#GUI), so it might be interesting to port that to Silverlight to get a nicer test run graphic along with the console, but it’s great that the REPL enables this to just work =) It also helped me find some bugs with the REPL, which will make their way to the IronPython source code shortly.

Let me know how it works for you. The next step is to tackle testing Silverlight C#/VB apps with Python.

Technorati Tags: IronPython,Testing

“The” IronRuby Tutorial

jschementi@gmail.com (Jimmy Schementi) — Tue, 23 Jun 2009 08:49:00 +0000

Hidden in yesterday’s commit to IronRuby’s GitHub repository is evidence of some polishing changes to the IronRuby tutorial app, which I’d like to introduce to you peoples:

To run yourself:

Get IronRuby from GitHub and build it (in release mode preferably)
cd Merlin/Main/Languages/Ruby/Samples/Tutorial
ir.exe wpf_tutorial.rb

Isn’t he/she/it a bute!? This app makes Ruby-based tutorials interactive, guiding you through the material with a IronRuby REPL console and other visual aids such as another window, or whatever the tutorial writer sees fit. There is also a tutorial runner for the command-line, and a exporter for static HTML.

The IronRuby tutorial included covers topics such as using the interactive window, built-in libraries, exploring .NET libraries, generics, events, Windows Forms, and Windows Presentation Foundation.

While the tutorial is currently no where near a complete guide to IronRuby, it’s a great place to get started. Also, the first two chapters of the TryRuby tutorial is included as well for testing reasons, but may or may not be ever finished … unless someone wants to make it work, more on how to do that later.

But it can teach all by itself!

While the content of the tutorials can be awesome, and this app can teach the material very efficiently, it can never teach the real experience of building an app in IronRuby. So the IronRuby tutorial app is written in IronRuby as an example of doing all the things a tutorial can’t. Wow, very meta.

Currently, the tutorial app itself covers:

User Interfaces in Windows Presentation Foundation
Ruby as Data: the tutorial domain-specific language
Testing a Ruby application with test-spec
Developing a GUI application interactively
Anything else we can cram into it in the future :)

These area will be covered in a series of future posts, so keep an eye out for them!

Still plenty of work to be done

This app is far from done, so the exploration of IronRuby in WPF has just begun. Here are some known issues, so don’t complain about these =P

The REPL doesn’t have any history via up/down arrow keys.
The “Jump to another section” menu scrolls too much. This list can only be two items deep, so it's needs to change to arbitrary depth to have more nesting.
The content isn’t totally complete, though the IronRuby tutorial is in great shape. If you have suggestions on topics, please let me know.
Jumping between tutorials a bunch makes the memory usage increase to Rails-like proportions (160MB). Not sure why yet. :(
The code isn’t the best, but it works. =P I’ll be reorganizing it in future posts about client development, but it serves its purpose for now.

Of course there are a ton of other things that could be done. See the TODO file for some ideas.

Please try it out and let me know what you like about it, what you don’t like, and feel free to futz around with the code as well … contributions are welcome. Enjoy!

Technorati Tags: IronRuby,WPF,tutorial

IronRuby 0.5 Released

jschementi@gmail.com (Jimmy Schementi) — Thu, 21 May 2009 01:45:00 +0000

I’m pleased to announce a new release of IronRuby: IronRuby 0.5!

Download IronRuby 0.5

You can also check out the source code for this release

Staying true to our "Conference-driven development schedule," this release is in conjunction with RailsConf '09, where IronRuby was demonstrated running real Rails applications and deploying them on IIS. A ton of work has been done in this release to get real Ruby programs running well, like RubyGems, Rake, Sinatra, and Rails.

Special thanks to Daniele Alessandri, Jirapong Nanta, and Ray Vernagus, who contributed code during this release, and everyone else who has submitted bugs, updated the wikis, or built software on-top of IronRuby.

Also, take a look at the IronRuby CTP for .NET 4.0 Beta 1, which has been released as part of Visual Studio 2010 Beta 1.

What’s in the Release?

A big language feature of this past release was adaptive compilation; the ability to start-up quicker with a fast interpreter, and then compile code that gets interpreted more than two times. This brought Rails startup time down from 89 seconds to 34 seconds, giving Rails approximately 2.5x quicker startup. Since this compiles “hot” code, your Rails application will get faster after an initial set of requests. There is a lot more to do for performance, but this was a great accomplishment.

Here are some more interesting changes:

Removes ClrString. CLR strings are referred to as System::String
Removes IronRuby library. "require 'IronRuby'" is no longer needed; IronRuby module is now a built-in module.
Integer/Float operations for all CLR primitive numeric types
System::Char and System::String methods so that they behave like an immutable UTF-8 encoded string
Improves handling of BOM in 1.8 mode
Debugger visualizer for RubyObject so that VS will display the result of to_s in the Watch window.
Adds support for .NET operators (op_* special methods)
Implements bitwise operators on flag enums
Precompilation of simple Ruby method invocations
Improves performance for file IO#read method. Previously we were 10x slower than MRI, now we are approx. 1.5-times faster.
Tons of fixes to File, String, Array, and other built-in types to drive up RubySpec and Ruby application test-suite pass-rates.

For detailed information, please see the CHANGELOG (which is much more verbose now. including all commit messages for the release … not just syncing to head of tfs)

Bugs closed

"ir script\server" causes a YAML parser error
"igem install rails" fails
yield fails in eval'd code
Generic type conversion from Fixnum to Integer
Pathname#cleanpath messes up the pathname
Array#hash should properly handle recursive arrays
ci_files set needed in mspec config
autoload does not use File::SEPARATOR
Time class instance
__FILE__ doesn't return a canonicalized path
… and various old RubyForge bugs during the move to CodePlex

RubySpec status

IronRuby passed the 80% mark on RubySpec during this release:

You can see more IronRuby statistics on http://ironruby.info.

Also during this release IronRuby has been running other Ruby applications test-suites as part of individual developer tests and in continuous integration, such as RubyGems, Rails (including ActiveRecord, ActiveSupport, ActionPack), and Rake.

Next Release

The next release will push harder on application compatibility, especially Cucumber (which has been the most requested application), Rails, and all the other frameworks being tested currently.

Also, the feature which makes IronRuby stand out, .NET integration, has next-to-no documentation, so that will large focus during this next release as well. Keep an eye on http://ironruby.net/Documentation/.NET, but as I said, is very sparse at the moment.

In the meantime, please test out IronRuby 0.5 and let us know all the issues you find. Enjoy!

IronRuby CTP for .NET 4.0 Beta 1

jschementi@gmail.com (Jimmy Schementi) — Thu, 21 May 2009 01:09:00 +0000

We’re pleased to announce the release of the IronRuby CTP for .NET 4.0 Beta 1!

Download IronRuby CTP for .NET 4.0 Beta 1

This is our second preview of IronRuby running under the Dynamic Language Runtime that is built directly into a .NET 4.0 release! As before, this release allows you to use IronRuby objects and types as .NET 4.0 dynamic objects from within C# and Visual Basic code. These binaries are roughly equivalent to the functionality found in IronRuby 0.3. Please also note that the IronRuby CTP for .NET 4.0 Beta 1 will run only under .NET 4.0 Beta 1.

This release also includes the binaries for the IronPython 2.6 CTP for .NET 4.0 Beta 1.

To try out this preview release:

Install some variant of .NET 4.0 Beta 1 or Visual Studio 2010 Beta 1 from here
Download the release
Follow any of the many dynamic walkthroughs online. Here would be a good start.

Enjoy!

IronRuby at RailsConf 2009

jschementi@gmail.com (Jimmy Schementi) — Fri, 15 May 2009 00:06:00 +0000

I spent last week at RailsConf 2009 in Las Vegas representing the IronRuby team. I gave a talk there titled IronRuby on Rails, and spent the 45 minutes talking about the benefits of IronRuby running Rails, or any other Ruby web framework, can provide. This write-up will walk through the four sections of the talk:

Introduction
IronRuby on Rails
- Other libraries and web frameworks
- ActiveRecord SQLServer support
Deploying on IIS
IronRuby ASP.NET MVC
Project status and roadmap

Downloaded all demos

(get slides, or just visit GitHub page)

Introduction

Last year, IronRuby was much like this sprout; much of the plumbing was there (xylem and phloem, if you will =P), but it has only just begun to be useful, like running Rails.

Quickly we learned that growing up is hard to do. As IronRuby was dispatching it’s first Rails request for RailsConf 2008, it began to experience all the stresses of the outside world, like loading activesupport on startup, which loads almost everything in the Ruby standard library.

The good news is that IronRuby is maturing; meaningfully interacting with the world around it, and begin to get good at things. Startup has much improved, and can run real Ruby programs like Rails, RubyGems, Rake, and Sinatra (to name only a few relevant apps).

IronRuby’s real world, the .NET framework, is different environment than the usual Ruby developer is used to, but IronRuby bridges the gap to bring the .NET framework to Ruby developers. An example of this is deploying Ruby web applications just like other .NET framework web applications using IIS.

Due to running real Ruby applications, and playing nice with .NET, people are starting to notice us. Community involvement is starting to pick up, and we’re getting closer to being a “real Ruby implementation”.

IronRuby running Rails

IronRuby running Rails is not new, but doing it well or completely – is. IronRuby can now run real Rails applications, rather than just toy-hello-world examples. This does not mean IronRuby on Rails is ready for production, but it’s a great measure of forward progress.

Get the source code

Special thanks to Chad Fowler and Rich Kilmer for first building this sample!

We’ve been tracking our Rails progress by running though all the different Rails development scenarios, like migrating a database, generating a model, etc. The Rails documentation page on the IronRuby website walks through setting up IronRuby to run Rails, and shows the various scenarios in action, as well as how to run the Rails test suite.

Other Ruby libraries and web frameworks

Rails is not the only thing IronRuby is tracking; we’re running RubyGems, Rake, Sinatra, Rack, and a bunch more ever-increasing Ruby libraries to catch bugs, find performance issues, and use their test-suites to gather more metrics on IronRuby’s completeness. For example, the Sinatra-basesd ironruby.info website runs perfectly well on IronRuby:

Also, for a while now IronRuby has been using and contributing to RubySpec to provide a standard-runnable Ruby specification. IronRuby’s latest numbers are available on ironruby.info.

ActiveRecord support for SQL Server

ActiveRecord SQLServer Adapter

Installation instructions

To run Rails, IronRuby uses a SQLServer adapter for all ActiveRecord database interactions. This adapter will work against either SQLServer or SQLServer Express (which is a free download).

ActiveRecord is useful outside Rails, and especially to a .NET developer for interacting with SQLServer. Using Ruby to manage the evolution of a schema through migrations is extremely simple and takes the pain out of doing database development. This is an example of a migration which adds a “people” table (and knows how to undo itself).

Once your database is created by the above migration, you can fetch and manipulate the data inside of a Ruby REPL:

Also, using ActiveRecord in a desktop application is pretty straight-forward. The following is an WinForms application which databinds an DataGrid to the ActiveRecord “Person” class.

Get the source code

Deploying Rack-based Ruby Application on IIS

Being able to run real Ruby applications is great, but what about deploying them? Most Ruby apps are web-based, so in this case deployment means getting the application on a production web-server and database server. The database server is already covered by SQLServer, so wouldn’t it be great to use IIS as the web-server?

Of course ...

ironruby-rack: deploy Ruby web applications on IIS

Rack is a Ruby web-server interface which almost all Ruby web frameworks build on-top of, like Rails, Sinatra, and Camping. This package enables and Rack-based Ruby application to run on IIS. Today this is only possible for Ruby applications through FCGI, where this implementation integrates directly with ASP.NET (through a HttpHandler), so your application will be “running” whenever IIS is.

If your organization already has an IIS knowledgebase, you probably want to utilize it. Deploying this way requires no retraining, as this solution is purely ASP.NET based; no throwing away of experience. It’s also a great way to sneak Ruby into your organization.

This project is fairly new, so if you want to use it please let me know and I help with any issues.

IronRuby ASP.NET MVC

Another way to get a Ruby web application running on .NET is to use ASP.NET directly. While there is no IronRuby support for ASP.NET WebForms (IronRuby in .aspx pages), IronRuby does have support for building ASP.NET MVC applications.

ironruby-mvc: ASP.NET MVC applications in Ruby

Special thanks to Phil Haack and the ASP.NET MVC team for seeding this implementation, and Ivan Porto Carrero for recently running with this (I’m sure his book will cover this =)).

In a previous post I built a tiny web-service in IronRuby-MVC to persist code typed into a IronPython-scriptable Silverlight calculator (here running outside the browser!).

Project Status

As IronRuby is maturing, more and more people are noticing it and (more importantly) using it. Special thanks to the more consistent contributors:

They either contribute C# code to the IronRuby codebase, documentation on http://ironruby.net and http://wiki.github.com/ironruby/ironruby, or the general ecosystem with active blogs, speaking at user groups, or new projects based on IronRuby. More information about contributing to IronRuby can be found at http://wiki.github.com/ironruby/ironruby/contributing.

To get all contributor projects, check out the ironruby-contrib project

IronRuby will be tracking performance and compatibility as it gets closer to 1.0. With 84% total compatibility on RubySpec, focus is being put on running real apps’ test-suites. Startup performance has also been a focus of improvement as of late, and by building an adaptive interpreter/compiler (like IronPython now has) Rails startup time decreased from about 85 to 30 seconds, without sacrificing startup time! Keep an eye on http://ironruby.info for updates to these types of stats.

Lastly, a little bit after this post is published ironruby-0.5 will be released. Please give it a try and submit any bugs you find to Codeplex.

MacDrive 7 with Windows 7

jschementi@gmail.com (Jimmy Schementi) — Tue, 12 May 2009 21:23:00 +0000

UPDATE: MacDrive 8 has been released, which has support for Windows 7, so the work-around explained in this post is no longer necessary. Also, people have been experiencing issues with it, so my advice is to upgrade.

If you’re running Windows on a Mac, and you’re shuffling files between Mac OS and Windows, MacDrive is the best tool for mounting your HFS (Mac OS’s default file-system format) partitions in Windows. MacDrive costs $49.95, but is well worth it. However, MacDrive does not release versions for pre-release operating systems, and there are a couple speed bumps when trying to use MacDrive on Windows 7. There are various version of this solution posted in a bunch of forums, but no definitive guide. Here’s exactly how I solved it:

1. MacDrive’s installer will only run on Windows XP and Vista. To have it run on Windows 7, you must run as Administrator, run in “Windows Vista” compatibility mode.

2. Also remove the operating system version checks the installer makes with Orca. MacDrive’s installer is an self-extracting exe, so you must extract it first; using 7-zip or any other unzipping tool will do the trick. Once extracted launch Orca and open MacDrive/x86.en-US/MacDrive7.x86.en-US.msi (or the x64 version if that applies to you). Once open, look for the “LaunchCondition” table and delete all rows from it.

3. If you run the main setup.exe from the extracted location it will install fine, but upon reboot you will not see your Mac OS drive. This is because MacDrive simply doesn’t assign the drive a letter. To do so, download Ext2Fsd, unzip, and run Ext2Mgr.exe. Find your HFSJ-formatted partition, right-click and select “Change Drive Letter”. Assign it a letter, exit the application.

4. Open up your “Computer” screen and you should see your Mac OS drive now:

Please post in the comments if this doesn’t work for you, and I’ll update the steps. Enjoy!

Lang.NET 2009

jschementi@gmail.com (Jimmy Schementi) — Tue, 14 Apr 2009 16:03:00 +0000

Today is Lang.NET 2009!

Lang.NET Symposium 2009 is here! A gathering of language geeks, users and implementers alike; an awesome cross-section of developers.

There is a awesome assortment of topics scheduled, from Google Chrome’s V8 JavaScript engine to a bunch of F# talks, and a ton more awesome talks that can be seen on here:

http://langnetsymposium.com/2009/agenda.aspx

Jason Zander is setting up now, and we’re about 10 minutes away from starting, so I’m excited! Videos of the talks will be posted as soon as humanly possible, so keep an eye on the website for video links.

Scripting C# Silverlight apps with IronPython

jschementi@gmail.com (Jimmy Schementi) — Thu, 02 Apr 2009 11:16:00 +0000

Image is unrelated to the post; just an awesome view from my building

« This post is part of the combined
IronRuby and AgDLR release at MIX '09

While many developers love to program in dynamic languages like Python or Ruby, they may not have that freedom in their day-to-day jobs, for a variety of tecnical and business reasons. However, dynamic languages are as useful when embedded in existing software, possibly making the design better than a non-scripting-enabled version. Languages based on the dynamic language runtime are really easy to embed in .NET applications, so hopefully this post inspires you to add scripting support to your applications.

In a previous post, I showed how to build a C# Silverlight application which evaluated a line of dynamic language code and printed the result. This post also touched on the philosophical issues around dynamic and static languages, so definitely give it a read first. Combining this with my most recent post about testing a C# Silverlight Calculator with IronRuby, this post will walk you through adding the ability to define calculator functions in IronPython.

Note: the same practices apply when scripting your application with IronRuby, but it’s been a while since I gave IronPython some love.

Also, Kirill Osenkov, a C# IDE QA'er, wrote up a similar article about Graphing IronPython functions in Silverlight, so check that out as well.

First off, download the Silverlight Calculator solution that we will build upon.

Download Calculator (start)

To cheat or follow along while reading,
download the finished Calculator from the MIX '09 Demos repository.

Sanity check: You'll need ASP.NET MVC 1.0 and Silverlight 3 Tools installed (SL2 might work, might not). Make sure Calculator.Web is the start project, and make sure CalculatorWebTestPage.aspx is the start page (both can be set by right-clicking on the respective files). Also make sure the site opens to CalculatorTestAppTestPage.aspx?test so that the tests run automatically. Compiling and running this solution should show you the following:

This project gives us some UI to type Python code into (Page.xaml will show you exactly what XAML creates this), and adds references to the IronPython and DLR binaries.

Now for the interesting part; building a PythonEngine class to encapsulate all the Python execution. Open Calculator.Tests\Calculator.Tests\tests\python_engine_test.rb (yes, Ruby is sneaking back into this post) and let's write some tests...

Now open CalculatorTestApp\Page.xaml.cs and add this to the CalculatorTestApp namespace to make those tests pass...

Pretty simple. The code sets up a ScriptEngine for IronPython, executes arbitrary Python code in a shared scope, finds all the defined methods in that shared scope, and can call those methods with a single parameter. All in ~23 lines of code. Awesome. Here it is in action...

Now the Calculator is ready to be scripted. The end goal is to:

Execute the code as it's being typed into the Calculator
If it does not produce an error ... find out all the valid top-level methods defined by the code and add buttons to the UI for each method.
Clicking the button will call the appropriate method, sending the calculator's current result as a argument, and the return value will be the new result of the calculator.

So let's write tests for this ...

And the following will make it all pass...

And now giving it a run will produce the following:

Clicking the button's above the Python code will call that function, sending the Calculator's current value as the first argument. Pretty cool! And that's all there is to it. Adding scripting support to applications on the desktop is just as simple. So, don't be afraid of this new-found power, go build awesome stuff with it!

Testing C# Silverlight apps with IronRuby

jschementi@gmail.com (Jimmy Schementi) — Fri, 27 Mar 2009 09:32:00 +0000

http://blog-well.com/2008/03/04/100-resources-for-web-developers/

« This post is part of the combined
IronRuby and AgDLR release at MIX '09

Testing is a great use for dynamic languages, especially if the application being tested is written in a compiled language like C# or VB. Ruby is an especially interesting language for writing tests in due to the popularity of RSpec, a behavior-driven-development testing framework. IronRuby makes it really easy to test .NET applications because of it's .NET integration and ability to run real Ruby programs, like RSpec. However, this becomes much more difficult to do in Silverlight for a variety of reasons. And Silverlight doesn't have good options for testing to begin with, except for the Silverlight Unit Test Framework, but I yearn for something simpler. This post will show you how to test a C# Silverlight application with IronRuby, using a testing framework called Bacon, a lightweight RSpec clone.

Note: A similar exercise could be performed with IronPython and a Python testing framework like unittest or PyUnit, but I have much less experience with writing test code in Python. If anyone else does, please feel free to put together a similar package for IronPython).

Demo: testing a calculator on steroids

Download MIX '09 demos

This example is in the /Calculator directory. Special thanks to Boyan Nikolov for implementing the Calculator control.

This is a example of testing a calculator (I'll explain the right-side with the Python code in the next post); the tests run in their own text-based REPL console, and even let you poke and prod the application after it's finished running the tests. The tests are very succinct and read easily, showing a big advantage to using IronRuby.

The tests do not actually click the buttons, as Silverlight doesn't have extensive UI automation support like WPF does, but the tests call the methods that are hooked to the events, and also test that the controls hook the correct method, making it an equivalent test, and a lot less magic-y.

For another testing example, see the spreadsheet example John Lam put together for his MIX '09 talk.

Test any Silverlight app

Testing any Silverlight application is pretty straight-forward, so let's walk through how to set it up. You can start from either your own Silverlight application, or a new Silverlight application, however you'll need a website/web-application project to be serving the actual Silverlight application; if you're creating a new application simply use the default options.

First, download eggs.xap (I'll explain what it is later) and place it at the same location in your webserver as your application's XAP file is being served; most likely the ClientBin directory. If you put it anywhere else, make sure to account for that in the code below. Note: Eggs.xap will be part of agdlr in 0.5.1, which will come out in a week from this post's date.

Next, download this Web Application project, rename it to whatever you want, and add it to the same solution as your Silverlight application; this will hold our tests.

There is a Sample.Tests/tests directory in that project, with a single sample_test.rb file; that where the actual tests will go. You’ll also see eggs_config.rb; this file defines the list of tests we want to run, since Silverlight can't enumerate the files in a XAP file easily.

The clientaccesspolicy.xml file makes it possible to download the generated XAP file, since our Silverlight application is being served from a different web-server. You can move these tests to the same domain as the application, but for the sake of this demonstration they are different.

To get your application running the tests, add the following C# code to your application at the point which you want tests to run; this might be on Application_Startup, or at some later time ... it's up to you. If you're not sure, just add this code to the method which handles Application_Startup in App.xaml.cs to download eggs.xap and run the tests, only when the query string contains "test".

When it is all said and done, run your application with "?test" at the end of the URL, and you'll see something like this:

The tests run inside a HTML-based IronRuby REPL control, so you can inspect your application after the tests run, maybe to track down a failure or prototype your next test.

Note: if you don't see the REPL in your application, or just see the REPL and not the red menu bar below, the Silverlight control is blocking its view. If you make your Silverlight control "windowless", by adding <param name='windowless' value='true' />" to the Silverlight object tag, or Windowless='True' to the ASP.NET control which creates the object tag, it will allow the REPL to show up. If for some reason you can't make your control windowless, you'll need to set the width and height of the Silverlight control so it doesn't overlap the console.

How it works

Everything above is pretty transparent on what it's doing, except for two parts: (1) what is this "eggs.xap" thing, and (2) how did my tests magically become a XAP file?

Eggs

"eggs.xap" is a small library to run Bacon in Silverlight. It containing bacon.rb, the Bacon framework, eggs.rb, a Bacon test runner for Silverlight, Eggs.dll, a simple .NET API for running the tests, and DLLs required to run IronRuby.

When you added the code to App.xaml.cs, it:

Downloaded eggs.xap
Pulled out Eggs.dll and loaded it
Runs the tests in the supplied XAP via "Eggs.Start"

Eggs.Start loads IronRuby's DLLs and and uses the DLR Hosting APIs to call the Ruby Eggs.run method. This is abstracted by Eggs.dll so the application being tested doesn't need a compile-time dependency on IronRuby.

Chiron.XapHttpHandler

All you did was put tests in a web application project, so how are they getting packaged up into a XAP file? In that project's web.config, there's one very important line:

Yes, Chiron's XAPing functionality is being used, but it's packaged in an ASP.NET HttpHandler, giving any ASP.NET application the ability to auto-XAP a directory when requested, without having to use Chiron's web-server functionality. So when you give Eggs.start the Uri to the tests (http://localhost:35863/Calculator.Tests.xap), it resolves to that project, sees that the Calculator.Tests folder exists but no XAP file exists, so it creates one in memory and responds to the request with it. Here is the implementation if you are curious.

This same strategy can be used to develop Python or Ruby Silverlight applications in Visual Studio.

Now, get to testing!

Now you have no excuses for not testing your Silverlight application! Hit me up on twitter (@jschementi) or the comments if you have any questions.

Silverlight Dynamic Languages SDK 0.5

jschementi@gmail.com (Jimmy Schementi) — Tue, 24 Mar 2009 06:46:00 +0000

In conjunction with MIX ‘09 and the announcement of IronRuby 0.3, I’d like to announce a new release of AgDLR 0.5 – the Silverlight Dynamic Languages SDK.

Download AgDLR 0.5

You can also check out the source code for this release

This is the MIX ‘09 release for Silverlight 2 and Silverlight 3 Beta. It updates the languages to IronRuby 0.3 and IronPython 2.6 pre-alpha, and adds a bunch of new features to Microsoft.Scripting.Silverlight and Chiron. Please read the release notes for details, but I’ll highlight the awesomeness below.

Silverlight 3 Beta

There is a separate package (agdlr-0.5.0-sl3b.zip) which contains binaries specifically for Silverlight 3 Beta. Though the Silverlight 2 binaries will work just fine in SL3, they are slightly different. The Silverlight 3 binaries use the Silverlight Transparent Platform Extension feature to download the DLR assemblies on demand, significantly reducing the size of a IronRuby or IronPython Silverlight application. This feature warrants a post just for itself, so that’ll be coming shortly.

REPL

Adding “console=true” to the initParams value of a Silverlight control will add a dynamic language REPL to any Silverlight app for a explorative developer experience. See samples/python/repl and samples/ruby/repl as well for how to use REPL programmatically ... in three lines of code.

Testing

Having a REPL in the browser is extremely liberating, because all your console applications can run as well. A killer application of this in the browser is testing, so this release has made it really easy to write tests for your Silverlight application, be it written in Ruby, Python, C# or VB, using IronRuby and a small Ruby testing framework, Bacon. This little subproject is called Eggs, and will be elaborated on in a following post.

AgDLR uses Bacon directly to test itself in the browser, making up a test-suite contained in agdlr-0.5.0-test.zip. This suite contains a lot of new tests written in Ruby, as well as ported Python tests from an internal Microsoft DLR Silverlight test suite.

Contributions

Thanks to Dan Eloff for contributing two fixes to Microsoft.Scripting.Silverlight, and Harry Pierson for Chiron.XapHttpHandler, a way to have IIS and the ASP.NET Development WebServer auto-xap dynamic language applications (which is freakin’ awesome, and I’m going to write about it shortly as well).

So, download agdlr-0.5.0 and ...

From http://www.zune.net

IronRuby 0.3 Released

jschementi@gmail.com (Jimmy Schementi) — Sat, 21 Mar 2009 00:53:00 +0000

I’m pleased to announce a new release of IronRuby: IronRuby 0.3!

Download IronRuby 0.3

You can also check out the source code for this release

Staying true to our “Conference-driven development schedule,” this release is in conjunction with MIX ‘09, where Silverlight 3 Beta and ASP.NET MVC 1.0 were announced, to name a few. John Lam spoke at MIX today at 12:30pm in Las Vegas about using dynamic languages in Silverlight for testing, scripting, and application building. And some ASP.NET MVC with IronRuby is thrown in there too.

The demos are also available for download here. Check out http://github.com/jschementi/mix09 for information on what the demos are, and how to use them. Also, throughout the day I will be posting walk-through-style posts about each major usage of dynamic languages in Silverlight.

AgDLR 0.5 Released
Testing in Silverlight
Scripting in Silverlight
Application building in Silverlight 3 (coming soon)
IronRuby and IronPython with ASP.NET (coming soon)

Now that that’s out of the way, what did we do in this release?

RubySpec

IronRuby passes approximately 80% of RubySpec, the best test suite Ruby has today. IronRuby is best at language compatibility, passing 95% of those tests, but worse on the standard libraries with a pass-rate of 77%. Overall the pass-rate as gone up approximately 10% since the last release, and there hasn’t been a major library push since getting Rails running. For IronRuby 1.0, that number is going to be as high as possible, and we’ll ensure that by continuing to work with the RubySpec project to improve the test suite.

IronRuby.info shows up-to-date RubySpec statistics about IronRuby.

.NET inter-operation

Since this release is around MIX, and John’s talk is going to be focused around Silverlight and ASP.NET MVC, it seems like a great time to polish IronRuby’s .NET interop. For example, the release enables calling generic methods, implementing interfaces with IronRuby, and better conversions between .NET and Ruby types. We’ve also started to build a .NET interop test suite, using the same infrastructure RubySpec uses (MSpec), to provide a executable specification of how IronRuby and .NET play together. There will also be a written specification showing what these .NET interop features are good for, but not for this release.

For more specifics of what this release contains, checkout out the CHANGELOG.txt in the binary download.

What’s with the version number?

IronRuby is going to be a bit unconventional-Microsoft (as if that’s something new) and simply release 0.x releases (and maybe 0.x.x if needed) until it gets to 1.0, rather than release something with “Alpha”, “Beta”, or “RTW” attached to the name. That being said, 0.3 is a complete underestimation of where IronRuby is; it’s definitely more than 30% completed. We’ll move through the point releases rather fast, compared to the 7 months it took to get 0.1 (Alpha 1) releases, and another 3-4 months for (Alpha 2). Any maybe even skip some numbers along the way.

What’s next?

While the .NET interop testing/specing work will continue, the next releases will be much more focused on getting real Ruby applications running well; whether they be already existing apps/frameworks like Rails, RubyGems, Rake, etc, or new applications that people are building with IronRuby. We’re starting to run the gems, rake, and rails tests with IronRuby, and this type of discovery with more Ruby libraries will drive what we work on next.

In short, expect to see more binary releases, more often. Until then, Download IronRuby 0.3!

State of the DLR for Silverlight

jschementi@gmail.com (Jimmy Schementi) — Thu, 05 Mar 2009 19:34:00 +0000

If you’re a frequent reader of this blog, you definitely know what’s going on with the DLR in Silverlight. For those who are new, or just don’t know, hopefully this post will clarify some things.

First off: my previous entries about Silverlight since sdlsdk-0.4.0 dropped:

“console=true”: REPLs in Silverlight
Adding scripting to a C# Silverlight app
jimmy hacking (at) Microsoft / Silverlight stuff near bottom
Dynamic Language Testing in Silverlight

It would seem the next release will be focused around REPLs, testing, scripting support, and maybe some server-side integration pieces. And you’d be right. Proof is in the project’s public source code repository.

But what about the dynamic language runtime and the languages themselves?

There has not been a new release of sdlsdk for many months, and that’s simply because the above features are not finished, NOT because the DLR and languages have not releases anything for Silverlight since then! The DLR released dlr-0.9 and IronPython released ironpython-2.0.1, both including Silverlight binaries. Also, Harry Pierson put together a nightly build site for the DLR, which includes IronRuby and IronPython, and had Silverlight-specific builds.

Then why the sdlsdk project?

sdlsdk is the main project and release vehicle for Chiron and Microsoft.Scripting.Silverlight, which makes writing applications in DLR-languages possible. It’s also the place for collaborating and experimenting with crazy usages of Silverlight+DLR. The DLR and languages are parts of other projects, so I can not justify bumping the version number of sdlsdk simply because a new version of IronRuby, IronPython, and the DLR are in there. The easiest way to update sdlsdk’s DLR+languages is to grab a daily build of the DLR and drop the binaries into sdlsdk’s /bin folder. The next sdlsdk release might include a script to automate that, as long as new builds don’t break it. =)

This is possible because the DLR, IronRuby, and IronPython teams use a shared continuous-integration server that runs test suites in Silverlight on each checkin, ensuring that they all continue to work really well.

Common misconceptions

During development the state of these projects could be completely different within a few weeks, so misconceptions form very fast. Here are ones which beg for correction …

Chiron, a special HTTP server, performs language compilation on the server and generates a .xap file.

FALSE. While Chiron is a special HTTP server, which generates a XAP file for any given directory, it does not perform language compilation on the server! Chiron is a development-time tool which only sticks the script files in the XAP, and they are run in the browser. When putting your app into production, you tell Chiron to write the XAP file to disk, and then you can put your app on any web-server you want.

Chiron’s existence shows the limitation Silverlight imposes by requiring a XAP file. DLLs can be downloaded by the AppManifest.xaml outside the XAP, so the XAP just serves as a AppManifest.xaml container. I’m hoping to get this fixed by Silverlight 4. =\ It also shows a limitation in not supporting continuations, which would let the user script pause while asynchronously downloading a script file, and then resume the initial script. I’ve been prototyping ways to get around this, but nothing working yet.

Also, if you’re using another web-server in your app (like you have a ASP.NET app with a Silverlight control), you can just set up a file-system-watcher and tell Chiron to regenerate the XAP to disk everytime you edit a file, but it will be served with the other webserver. This is used by silverline, a Rails plugin I wrote to integration Silverlight and IronRuby.

This is why you can’t include JavaScript files from inside other JavaScript files, and all your dependencies need to be pre-defined in the HTML file, because JavaScript can’t download the script files while the JavaScript engine is running.

Common interface implementations were inconsistent between languages, so I can’t create a solution where advanced users can pick and choose a language.

FALSE, in theory. The DLR hosting API (contained in Microsoft.Scripting.dll) provides a single set of APIs that can be used to host many languages. For example, Microsoft.Scripting.Silverlight uses the DLR hosting APIs to run any DLR language in Silverlight, based on what extension the entry-point script has (.py, .rb, etc ...). There are simple hosting examples here.

It is the language’s responsibility to implement certain interfaces so they are host-able by the DLR. IronRuby and IronPython are existing examples of that, and there is documentation in the hosting spec on how this all works, so new languages can follow suit.

However, there are always going to be language-specific scenarios which have different ways of using the hosting API, or the hosting API doesn’t support the feature and you have to use the language’s hosting API. In those cases, it’s best to re-think your scripting design and make the hosting code as simple as possible.

Also, a couple months ago, IronRuby’s hosting support was no where near where it is today. Again, things change fast. If you have any feedback on DLR hosting, please ask around on the DLR Discussions List.

And the state is ...

I hope that made some things clear; the DLR, IronRuby, and IronPython are committed to supporting Silverlight developers. Unfortunately, the reverse is less strong, so if this is important to you please talk about it on the silverlight.net forums. Things might be a bit nicer when the inner ring of the DLR (Microsoft.Scripting.Core.dll) is part of the System.Core in Silverlight, but we’ll see.

ironruby.info

jschementi@gmail.com (Jimmy Schementi) — Wed, 04 Feb 2009 07:57:00 +0000

DNS seems to have propagated, so now I can tell you about ...

ironruby.info is a new site I’ve been working on for a week or so; a place to track daily information about how IronRuby is doing. It shows simple performance and RubySpec information, compared to Ruby, and random build/source information. There’s a lot more to do, like tracking working set, showing graphs or historical data, and running traces popular Ruby frameworks to show which specs are “most important” to fix … but it provides some useful information already.

Oh, and the code is on github. stats.rb runs nightly collecting the data, and app.rb is the Sinatra website showing the data. Feel free to send me patches/changes, or fork it to track other implementation :)

http://github.com/jschementi/ironruby-stats

Disclaimer: This information is gathered on a Intel Pentium 4 3.5 ghz machine, so performance numbers will be different if you have a faster processor and/or more processors/cores.

Performance

For startup time, it measures the time it takes to start ir.exe empty.rb from the command-line, where empty is a empty file. IronRuby is currently 10x slower for startup on average, regardless of whether you’re using the interpreter or not (though the interpreter is slightly faster … we’re working on fixing that).

The “100,000 Iterations” is just $a *= 2 100,000 times. Nothing fancy yet. Ideally we’d be running the ruby-benchmark-suite here. IronRuby is only 2.5x slower on this, but the interpreter is way slower. So, for now only use the interpreter for straight-line code ... no loops!

RubySpec

For those who don’t know what RubySpec is:

RubySpec is a project to write a complete, executable specification for the Ruby programming language.

IronRuby has been using this specification for a while now to gauge our progress towards being a Ruby-1.8.6 compliant implementation. They are divided into three sections: Language which tests Ruby itself, Core which tests the libraries which Ruby needs to run, like Array, Hash, etc, and Library which tests the Ruby standard library.

While these are a great judge of a Ruby implementation’s progress, they take a bit of time to run yourself (including setup time), and we never gave detailed reports of our own runs. So now, if you’re ever curious to know how IronRuby is doing, you can just check ironruby.info and get the data immediately. Pretty cool. “But wow, that’s a ton of red on the page.” --- yeah, it looks worse than it is, maybe I should use a range of red? Regardless, let’s explain the results.

Pass rate

The pass rates in the ir.exe and ruby.exe columns are “the number of expectations which passed, divided by the total number of expectations which ruby.exe runs”. So the “all specs pass rate” number is the pure pass rate, regardless of how well/poor ruby.exe does. The “diff” column shows how far away IronRuby is from Ruby for each row.

Overall, IronRuby passes 86% of all the specs. Ruby (on Windows) passes 98%, so that leaves us with 12% of the specs left to get to Ruby compatibility on Windows. Not too shabby. The Core tests have the highest failure rate (-18%), followed by the Language tests (-5%), and the Library tests have a strangely-low failure rate (1.87%) since Ruby is running 482 expectations less than IronRuby … I need to investigate that.

Execution time

The Core and Library specs are between 4-5x slower (Ruby’s Core specs report a WAY too long run time right now, usually they’re around 2 minutes). The Language specs are a whopping 50x slower, which is crazy. I have to investigate why.

Fun facts

There’s other fun facts it collects; build time, binary size, repository size. Not that interesting, but still good to see … I’d like to know if our repository size shoots up dramatically.

Anyway, that’s what ironruby.info gives you today. Hopefully this gives you an idea of where we are (getting close on the specs, and putting a lot of work into performance).

Git on Windows 7

jschementi@gmail.com (Jimmy Schementi) — Mon, 12 Jan 2009 03:52:00 +0000

I use Git everyday, whether it be for IronRuby, AgDLR, or random other Rails/Ruby work. Being that I work at Microsoft, I got my hands on earlier build of Windows 7 during the fall. Unfortuantely, git didn’t work fully on Windows 7, failing whenever a merge happened, which made push fail, as well as pull when trying to get other people’s changes). Jim Deville has been tracking this with the Windows team, and even bringing it up to the msysgit maintainer to see if there’s any way around this. It was looking grim, so I left my main development machine on Vista, and was sad.

Windows 7 Beta was released this past Friday, and I managed to download the .iso, get a Product Key, etc, at home during the day on Saturday. Now that it’s released, I’m guessing other people will have this problem ...

I’m not sure of the specific details behind the issue, but msysgit calls into C:\windows\system32\msvcrt.dll during a merge, and the changes to that DLL in Windows 7 breaks msysgit. Last week, John Lam was describing the issue to John Messerly, and realized that the easiest fix would be to copy the Windows Vista msvcrt.dll to the C:\Program Files\Git\bin directory, since Windows DLL resolution will look on the PATH, and “.” is included.

Curt Hagenlocher just reminded me of the fix, and it works!

Pretty awesome. Enjoy Git in Windows 7!

Dynamic Language Testing in Silverlight

jschementi@gmail.com (Jimmy Schementi) — Tue, 06 Jan 2009 08:54:00 +0000

A little less than two months ago I moved the Silverlight and DLR integration (AgDLR) project to http://github.com/jschementi/agdlr, in an effort to get more collaboration. Since then Dan Eloff and Mark Ryall have sent me pull requests, fixing up various error reporting, threading, and HTML bugs. Thanks guys! I promise to pull from you soon!

Anyway, a little while ago, Mark said:

I don't want to get all Captain Agile on you so soon but what do you think I can possibly add in the way of tests to verify that the change doesn't break anything?

Yeah, not having tests kind of sucks, doesn't it? Unfortunately, testing in Silverlight is a windy, confusing, one-lane road, so getting the infrastructure to run tests has taken some time, but I just committed a Silverlight spec runner for bacon (a little Ruby spec framework) and a bunch of specs for Microsoft.Scripting.Silverlight!

Here's a short screencast walking you through running the tests:

Testing AgDLR from Jimmy Schementi on Vimeo.

So, the tests are written in Ruby, but they test the C# code in Microsoft.Scripting.Silverlight ... pretty cool. This is a deviation from my common belief of "test in the language you write the code in", but so what ... I never cared for that way of thinking =)

This test runner can be copied and used to test any Silverlight code; just place your test file in the ruby/test directory and update the test list in ruby/app.rb, and that's all their is to it. I'll probably pull it out into it's own git repository, but for now it's part of AgDLR.

What took so long?

Yeah yeah, two months without tests is crazy, especially since these tests were just written in the last two weeks! This being Microsoft and all, with an entire discipline devoted to testing, you'd think we have a ton of automated tests to verify our Silverlight integration. We'll we do. Kind of. Let me explain.

Currently AgDLR's tests are mostly in-browser tests, with some "sanity" tests running on the command-line. The command-line runner (which I can't release to the public) hosts CoreCLR, but doesn't work exactly like the managed environment in Silverlight, which is why most are in-browser. The in-browser tests are comprised with a lot of "does this feature of Silverlight work in the DLR" tests, as well as more useful the IronPython and IronRuby test suite, and some end-to-end tests of sample applications (DLRConsole, Clock, etc). They run on an internal version of Microsft.Silverlight.Testing, which stop me from just releasing it, again.

Today these tests mainly run in a check-in system (the infamous SNAP, aka "the troll"), which runs tests in parallel over a bunch of machines. In SNAP, the Silverlight tests can take 30+ minutes themselves. On my laptop it takes about 5 hours! Basically useless for anyone wanting to contribute to AgDLR and run tests.

Yeah, that's how it makes me feel too. Why they are like this is an entire different, and possibly inappropriate post, but I'll summarize. The issue is two fold:

(1) The browser is killed after each test

Yeah, you heard me right. And each test has to run in both Firefox and IE. The tests are launched from a custom test harness, which launches the appropriate browser instance, and waits for the browser to say whether the test passed or failed, and gets any extra information like stack traces, error messages, etc. Sometimes this completion detection happens quick, but other times a failure never happens, so the test must timeout (60 seconds), and then it re-runs itself three times to make sure the timeout wasn't some fluke. Definitely no failing-fast happening here. Much of this is fixed in a newer version of Microsoft.Silverlight.Testing, but this is what's running today.

(2) The tests aren't unit tests

The tests exercise Silverlight features from DLR languages, rather than testing the integration between the two. Actually, the DynamicApplication class is never used in the tests!

This makes a good point about testing in general. While end-to-end/sign-off/acceptance tests are wonderful things, they can't be all the tests. If I can't run the tests while developing, then I'm not going to run them period. Having a check-in system run them for you is fine, but I'd like to have a good chance at passing a 1.5 hour job the first time.

Solution: ditch it all

To get tests out to the world, the only sane thing would be to head back to the drawing board, which is what I've done with test/runner. Hopefully by using existing open source projects, a testing paradigm most people understand, and a very small test runner, this testing solution for AgDLR will be kept small and friendly for developers.

So, now that there's a sane way to verify changes to AgDLR don't break, please fork AgDLR and send me pull requests! Next post will probably be about adding continuous integration to AgDLR, so stay tuned.

jimmy hacking (at) Microsoft

jschementi@gmail.com (Jimmy Schementi) — Wed, 03 Dec 2008 10:02:00 +0000

Here's the "script" I wrote up for my talk this evening at WPI, entitled "jimmy hacking (at) Microsoft -- open source programming languages."

Here are the slides

I'm back! For those who are fortunate enough to not know me, I'm Jimmy Schementi, and graduated from WPI in 2007 with a Bachelors and Masters in Computer Science from this amazing group of people who had the privilege to go either bald or crazy because of me. I ran the Game Development Club, and dedicated a year of my life to Neil Heffernan’s Assistment project. Now I work at Microsoft on implementations of Python and Ruby for the .NET platform. I got them to buy me a Mac. That’s me.

I’m going to waste an hour or so of your time talking about implementing two dynamic languages at Microsoft, IronRuby and IronPython. I’ll also touch on what it’s like working at Microsoft, including the challenges of integrating these languages with Silverlight, a platform for the browser.

But first ...

How I started with programming

LOGO. 6^th grade. But the version I used was in English. Of course, I really didn’t know I was "programming", since I was making pretty pictures, but it was still fun. Years later, when I first saw Smalltalk, I could have sword it was LOGO. Anyway, I found a nifty logo environment on the web, and if you want to take a trip down memory lane with me here are some things to try out:

to spiral :size
  if :size > 30 [stop] ; an exit condition
  fd :size rt 15 ; many lines of action
  spiral :size *1.02 ; the tailend recursive call
end

reset
spiral 10

Which will produce a spiral:

... and ...

to tree :arms :length :count
  if :count < 1 [stop]
  repeat :arms [
    lt 360/:arms
    pd fw :length
    tree :arms 2 * :length / :arms :count - 1
    pu bw :length
  ]
end

reset
tree 7 128 4

Will make a snowflake:

LOGO is a pretty advanced language, supporting recursion, functions, parenthesis-less method calls ... ah, feels like Ruby.

Anyway, why am I showing you LOGO? Because of the way you make logo programs: type a line, run a line, see the result. It’s an extremely productive way to program, and what most of this talk is going to be about. Let me show you what I mean ...

Taste of Ruby

To introduce Ruby, I made a screen-cast where I build a tiny unit testing framework entirely in a interactive environment. Here's two small environments in the browser that you can use to follow along: http://jimmy.schementi.com/silverlight/repl or http://tryruby.hobix.com.

Ruby Testing in a REPL from Jimmy Schementi on Vimeo.

This exploring lead to the making of this tiny unit testing framework. A majority of the logic is here, and the rest is for formatting and mocking. They are interesting because of how they redefine the methods defined in tester.rb, feeling very much like aspects, but built into the language. This allows Ruby to be very powerful for meta-programming, as this unit-testing framework is really a little DSL, looking very far from traditional Ruby syntax.

Terminology

I assume no one here as developed any software on the Microsoft stack, especially since most classes use Java, C, C++, or at least require it to be run-able on the CCC Linux machines. However, there are a lot of parallels between the two worlds, so let me make those connections:

Java langauge >>> C#
Java runtime/libraries >>> .NET platform (CLR)
(imagine) Flash + Java applets >>> Silverlight

Dynamic Language Runtime

IronPython 1.0 was released in September 2006, a Python 2.4 compatible implementation on Microsoft .NET, which performed on par with CPython. That project taught us that the CLR is great for dynamic languages, but there are things common to all dynamic languages that we had to roll ourselves. So, the “Dynamic Language Runtime” project was created to initially provide a dynamic language compiler infrastructure for the CLR, but can support static compilation as well.

A shout out to Bill Chiles, since these next couple of slides are based off of one of his talks.

Before VMs, compiler writers built tokenizers, parsers, semantic analyzers, one or more passes of data flow and optimizations, code generated to machine code, and then produced a equivalent binary file. With VMs, we remove the last few phases, so languages tokenize, parse to their syntax trees ...

... then do some semantics analysis, generate MSIL (higher level than machine IS), then write that to disk (an "assembly" in .NET).

With the DLR, the front-end is the same: tokenize to a AST, but then map the language's AST to a Expression Tree, which would include binding and semantic information. And that's it. The DLR will take those Expression Trees and generate MSIL ... no code generation phase at all.

The Dynamic Language Runtime is a derivative work of the common functionality needed to build Python, Ruby, and JavaScript, so this method is successful for real languages.

Let's dive a bit into how the DLR actually generates MSIL:

There are two ways to compile a method call or operation; statically compile to a known method OR to compile to a dynamic expression that describes what you want to do (call, get member, set item, set member, etc).

At runtime the dynamic expression gets bound to a rule (the method "Handle" in this case), which has the specific code needed to perform the operation when given a set of arguments. The rule has a test for when the implementation of the operation is valid (typically a test on the types of the arguments). The DLR caches these rules.

Take "hello " + name for example, which we can think of as x + y ...

A DynamicSite is the runtime object that manages method/operation caches for fast lookup of how to perform an operation. We compile an expression such as “x + y” to a call to a DynamicSite’s invoke method. That method starts out with a body that simply calls UpdateBindingAndInvoke.

The first time through, UpdateBindingAndInvoke asks the language that owns the context for the expression if it can perform the operation given the actual objects x and y. The binder may in turn ask the objects if they know how to perform the operation. If the binder gets a rule or can produce one, it returns it to the site.

The rule has a test for when it should be selected in the cache, and it has a target or implementation of the operation in the site (get member, call object, set item, do addition, etc.).

If we call the DynamicSite with different argument types ...

... we fail the one test for strings, then fall through to update and bind again.

if we had to integers the second time, we might find the new rule, and we recompile the method that holds the code from the rules. Thus whenever the DynamicSite is invoked on two integers or two strings now, it is very fast.

Since the DLR basically takes text and turns it into executable code, we can expose an API for application developers to “host” the DLR and run arbitrary scripts in their application. This “embedding scripting” functionality is very powerful.

Tooling

In static languages, it’s really easy to make a tool that can do “IntelliSense/code completion” for you, since you know all the type information (methods, properties, variables, etc) at development time/compile time, so a editor can use a background compiler to parse your code and see where there are syntax errors, and even do static analysis to find runtime errors like null reference errors, etc.

However, with dynamic languages, the code is compiled at runtime, so when you do foo.something, any tool can’t know what methods “foo” has, unless the entire application is running. As more static languages start picking up dynamic traits, this needs to be fixed. So, let's talk about the REPL:

History of the REPL

Lisp, a 50 year old programming language, has three vital functions: read (read from a command prompt), eval (run as string as lisp code), and print (print to screen). Lisp made popular the environment you saw before, where I type a line of code, and see the result immediately. This pattern maps directly onto those three Lisp functions, so that type of application is called a REPL (Read-Eval-Print Loop).

Today, REPLs are only seen in runtime-compilation languages, aka scripting/dynamic languages. Static languages like Java, C++, C#, etc, do not have the facilities to enable a REPL, outside of shelling out to the compilers. However, C# 5.0 MonoScript, a derivative of C# 3.0, provide APIs for accessing the compiler’s internals, enabling compiling at runtime, which REPLs depend on.

Hit the breaks

Ok. I’m getting bored of this history lesson, so I’m going to abruptly change direction and talk about some cool technology that can be use from dynamic languages, Silverlight.

Silverlight

Silverlight is a 4MB browser-plugin for Windows/Intel Mac OS 10.4+/Linux(through Moonlight) which puts a subset of the .NET framework in the browser. You can use the runtime to manipulate the DOM of the webpage (replace Javascript), or get a vector graphics surface to do fancier graphics/animations. Because it is integrated with the CLR, you can use any .NET language to program in the browser.

To you dynamic languages from Silverlight, you'll need the Silverlight Dynamic Languages SDK: http://codeplex.com/sdlsdk. The public source repository lives here (http://github.com/jschementi/agdlr), if you just want to get the code or even help develop the SDK.

Here I’ll show Ruby running in the browser.

http://jimmy.schementi.com/silverlight/photoviewer

Note: if you don't have Silverlight installed, it'll prompt you to install. This will only work on Windows or Mac OS 10.4+, as the Linux implementation, Moonlight, is not released yet.

This is a simple Flickr client, entirely running on the client (does not require a server). It is mainly Ruby code, with the exception of a JavaScript library called Lightbox which I’m using to do the picture-open animations, since it’s awesome. This shows the interop between Ruby and JavaScript in the browser. It also directly manipulates the Html directly, so a end user wouldn’t know the difference.

I mentioned hosting before … so if I can host the DLR in a Silverlight application, couldn’t I make a little IDE in the browser?

http://silverlight.net/samples/sl2/dlrconsole/index.html

DLRConsole is a Python Silverlight application, which hosts the DLR and lets you write Python or Ruby code, and manipulate a Silverlight surface. Here’s an example of rendering a clock and dragging it around.

rb> require 'lib/clock'
rb> require 'lib/drag'
rb> $c = Clock.show
rb> def drag(obj)
rb|   Drag.new(obj).enable
rb| ende
rb> drag($c.canvas)
rb> $t = canvas.children.first.children.first
rb> drag($t)
rb> class UIElement
rb|   def drag
rb|     Drag.new(self).enable
rb|   end
rb| end
rb> $tt = TextBlock.new
rb> $tt.text = "Monkey patched!"
rb> canvas.children.add $tt
rb> $tt.drag

Back to REPLs

Ok, let’s pull up for a second. I was talking about the REPL before, so can’t we have it here? Sure! The first example I showed (Ruby testing) was entirely written in a Silverlight REPL! In fact, it’s functionality built-into the Silverlight+DLR integration, so you can fire up a REPL against your live running app (much like “python –i foo.py”

I recently talked about this at Seattle CodeCamp v4.0, and wrote about it, so here's the post and demo.

Embedding Ruby in a Existing C# application

I also talked/wrote about this previously, so view the post here.

Rails Integration

There’s an awesome concept that’s lurking here: you have Ruby in the browser! That is, a language that runs either on a server or a user’s local machine can be pushed from a web server to run on the client, much like JavaScript, but it can be the same language on the server. Therefore, it could be the same exact code. I’ve been playing around with this idea for a while, first talked about it at RailsConf 2008, and it's current a Ruby on Rails plug-in called Silverline.

So, here's what I'm talking about ...

http://silverline.schementi.com/client

Note: this demo is fairly contrived, since it will only work for controllers named "ClientController". Also, it looks like it only works in production-mode once, so don't be surprised if the link above doesn't work for you. If you have Rails installed you can run the demos locally for yourself.

This Rails code ...

class ClientController < ApplicationController
  def index
    render :layout => true, :inline => <<-EOS
      <%= link_to_remote 'Show time', :url => {:action => 'time'} %>
      <br /><div id="time_div"></div>
    EOS
  end

  def time
    @time = Time.now
    render :update do |page|
      page.insert_html :bottom, 'time_div', "#{@time.to_s}<br />"
    end
  end
end

... along with a simple template produce this:

Very simple app; hit "Show time" with your forehead and it prints the time. As Firebug shows, each time request goes to the server.

class ClientController < ApplicationController
  client :time

  # ...
end

However, with the silverline plugin, if you add "client :time" to the controller, it will run the "time" action on the client:

Note that clicking on "Show time" this time doesn't send any requests to the server; the code is executing in the browser.

This simple demo shows how a developer can decide to run a Rails action either on the the client or server. Silverline also provides some rendering helpers to render XAML through ERb. Check out the site if you want to know more.

Open Source

Doing Open Source at Microsoft is hard. Like swimming against the current. Sometimes it's even like being the hippies Cartman is spraying. But that's why the people I work with and I do it ... because it's challenging and potentially that much more rewarding. But enough of the sappy talk ...

IronRuby and IronPython are both Microsoft Public License open-source projects, a fully endorsed Open Source Initiative license. The Ms-Pl is an BSD-like license; very short and to the point. In even shorter terms, any Ms-Pl project can be forked, sold, modified, etc, and you owe nothing to Microsoft. IronRuby accepts contributions to the Ruby standard library, but all of IronRuby and IronPython are on track for accepting contributions to the entire codebases shortly.

Ruby and Python themselves are not Microsoft languages. They exist outside Microsoft and have their own thriving communities. Since Microsoft sees value in supplying Ruby and Python on .NET, I get paid to help make that happen. Since those languages are open-source to begin with, and Microsoft is benefiting from having those languages, it's only right for the Iron* implementation to be open source themselves.

Fat lady, sing it

Whew, that's it! This covered a lot, but I hope you have a better understanding of:

why dynamic languages are useful
what the DLR actually does for dynamic languages
why REPLs are the future of productive IDEs
how Silverlight and dynamic languages can blow your freakin' mind
and why in the world Microsoft is relevant to the open source community

Enjoy!

Speaking at WPI on 2008-11-24

jschementi@gmail.com (Jimmy Schementi) — Fri, 21 Nov 2008 20:30:00 +0000

Monday, November 24 I'm speaking at Worcester Polytechnic Institute in Worcester, MA about what I've been doing since graduating in May 2007. There will be a trip down memory lane, Ruby, Python, .NET, Macs, Robots, Music, and spontaneity; it will be epic.

Jimmy hacking (at) Microsoft
Monday, November 21, 2008, 6pm
Fuller Auditorium, Fuller Labs
Worcester Polytechnic Institute
100 Institute Road
Worcester, MA 01609

I'm flying to NY tonight (woot for red-eyes) with Felicia, and then driving/busing (depends how much work is left to do ;)) to Worcester Sunday. Should be a fun couple of days.

If you're in the Worcester-area, stop by! Slides/Demos will be posted shortly after the talk, and I'm looking into recording.

Adding scripting to a C# Silverlight app

jschementi@gmail.com (Jimmy Schementi) — Tue, 18 Nov 2008 06:30:00 +0000

At Microsoft, the people I work with and I are definitely the minority, preaching about the benefits of dynamic languages and using the right tools for the right jobs, as in don't use static languages where you don't need it -- though there are plenty of times where you need it.

And now you ask, "Jimmy, but why would I, someone who gets paid to write C#, use a dynamic language?"

The majority would only have static languages pulled from their cold, dead fingers, and I totally agree with them. Don't change for the sake of change. Though, for certain scenarios, running scripts in a VB/C# application would be useful. For example, a shopping application that has a bunch of business rules, like "when someone has three items in their cart that all have to do with cooking, give them 10% off." These type of rules can change all the time, and traditionally you'd either store the rules in a database and implement a engine to understand the rules, or hand-code them yourself and have to redeploy the system every time you want to change them.

Or, you could save yourself the hassle and store the rules as Python or Ruby code, and then host the DLR in your application to run the code. Want to update the rules? Just update the code, nothing more. And a dynamic language is probably closer to how the domain-expert would represent them as, so they could even write them. Yes, this is just one scenario, but a powerful one for existing static language developers. This was the last part of my Seattle CodeCamp talk -- how to host the DLR in your C# Silverlight application.

Starting with an existing C# Silverlight application that just takes some input and echos it back, I'll extend it to run the code through IronRuby and print the result. First, open the solution, and hit F5 to see that the app just echos what you type in the grey area. This requires you to have installed the Silverlight Tools.

Now add references to the DLR and IronRuby (in the Dependencies folder):

Let's write a little wrapper class run Ruby code. Open App.xaml.cs and add the following class to the SilverlightDLRDemo namespace:

class RubyEngine {
  private ScriptEngine _engine;

  public RubyEngine() {
    var runtime = new ScriptRuntime(
      DynamicApplication.CreateRuntimeSetup()
    );
    _engine = Ruby.GetEngine(runtime);
  }

  public object Execute(string code) {
    return _engine.Execute(code);
  }
}

Note: this exact code won't work on the Desktop; you'd have to just give the ScriptRuntime constructor no parameters. Here I use the DynamicApplication.CreateRuntimeSetup() to use the same ScriptRuntimeSetup object that Microsoft.Scripting.Silverlight uses, which knows how to map File system access to the XAP file, which is useful for dynamic languages to depend on other files. Of course, it's your decision to allow this or not.

You'll also need to add the following "using" statements:

using Microsoft.Scripting.Hosting;
using IronRuby;
using Microsoft.Scripting.Silverlight;

And that's it! That's all we need to run IronRuby code. Now let's hook it up to the page. Open Page.xaml.cs and add an instance variable to the Page class, and initialize it in the Page constructor:

// add to Page class
private RubyEngine _ruby;

// add to Page constructor
_ruby = new RubyEngine();

Lastly, replace the "Result.Text = ..." line with this, which prints the typed code, and then the result computed by the RubyEngine wrapper we just wrote:

Result.Text = "\n\n" + 
  ">> " + Code.Text + "\n" +
  (_ruby.Execute(Code.Text).ToString()) + 
  Result.Text;

Now hit F5 again, and type some Ruby into the TextBox, hit enter, and boom, you're C# Silverlight application is running IronRuby code.

In fact, this is exactly how AgDLR, aka the dynamic language integration with Silverlight, works -- a C# application (Microsoft.Scripting.Silverlight.dll) which hosts the DLR and runs a script file (app.rb or app.py). Take a look for yourself.

I hope that shows you how easy it is to host the DLR from a C# Silverlight application, and think twice when writing those complex rule engines. =)

"console=true": REPLs in Silverlight

jschementi@gmail.com (Jimmy Schementi) — Tue, 18 Nov 2008 01:23:00 +0000

At Seattle CodeCamp v4.0 today, I talked a lot about REPLs, or "Read, Evaluate, Print Loops". They are the true "minimalist" view of programming; type a line, run a line. Most of them are no frills, only a terminal window and a blinking cursor, though they themselves could be really rich development environments. Also, REPLs are a vital tool for runtime-compilation languages, since the methods/variables/classes/etc are only *really* known to exist at runtime.

I started my talk with a Ruby REPL showing off how you can use IronRuby to explore .NET.

Ah, the beauty of two worlds colliding. You can use Ruby-idioms to explore the .NET types, starting with an A, B, or C, in the System namespace, just like they were Ruby classes to begin with. Call .NET methods, even with Ruby method naming conventions; IsInterface as is_interface.

The lurking awesomeness here is this REPL is running inside the browser! This is a HTML-based Silverlight application, which provides an IronRuby REPL. It accomplishes this by using the Dynamic Language Runtime (DLR) hosting APIs, but don't get hung up on the details of how that works, I'll explain later.

This is nothing *new* in my world. DLRConsole has been around for a while now, and lets you write code in the browser to manipulate a Silverlight canvas. Both REPLs are self-contained, as in they only work against code manually loaded into them, like the previous screenshot shows with loading in the clock.rb and drag.rb files. This is fine for trying small things out, but becomes very cumbersome when trying to interact with a live application.

To interact with a live application seamlessly, the bare minimum would be to have a REPL available directly in the application itself. And that's what I've done:

If you build the latest bits, you can add a new initParam option called "console" to enable a REPL in any application:

<object ... >
  ...
  <param name="initParams" value="console=true" />
</object>

This will inject a HTML-based REPL into whatever web page your app is running on. The language of the REPL is set to whatever language your start script is. To make the console pretty, add this stylesheet to your page. Eventually this stylesheet will be included in the templates script/sl produces.

In the screenshot above, I did not type/click anything on the UI; the code I typed into the REPL set the search term and "clicked" the search button. This type of integration with your application could easily produce Waitr-like automation, for testing and whatnot. I'd really like to see Waitr ported to IronRuby. =

But there's plenty more to-do: this console is very simple.

It only supports Ruby currently; though simple Python statements will work (white space is a bitch =P).
It supports single and multi-line statements, but since the text input is implemented on top of a <input> tag, browser history gets in the way =P
History works, but never resets itself.
If you make a typo, and it's not clear that you've intended for the statement to be completed (like typing "class <enter>"), you'll have to force the console to run the command with <ctrl>+<enter>

Please, play around with this new "usefulness" and let me know what you think. I'd prefer you to leave comments on github, so everyone can participate in the discussion. The code for the REPL lives in Console.cs, so if you want to fix something on the list above, or see something else broken, fix it! If you find more bugs, please post them to Codeplex under the 0.x.0 release.

Hope this little feature makes your live a bit easier!

agdlr: Silverlight + DLR + Open Source

jschementi@gmail.com (Jimmy Schementi) — Mon, 17 Nov 2008 21:23:00 +0000

The "Silverlight Dynamic Languages SDK" Codeplex project has existed since March 2008, with signed binary releases and source code drops every month or so. Though it's a good ship vehicle for the Silverlight+DLR integration, it's not really an open source project -- mainly as it lacks a public source repository. That changes today.

AgDLR Source Repository

This repository will contain the sources to Microsoft.Scripting.Silverlight.dll and Chiron.exe, as well as Ruby/Python libraries for writing Silverlight applications. Any "feature" work on those pieces of code will be committed to the public repository first, and eventually make its way into MIcrosoft's internal source control so the DLR/Iron* languages don't break it. This repository takes a binary dependency on the Iron* languages as well as the DLR, since this isn't the place to change that code. The source code for IronRuby, IronPython, and the DLR are available elsewhere, but releases on the Codeplex page for this project will still contain source drops of everything.

The repository is hosted on http://github.com, which is a collaborative development site based around git, a "fast, efficient, distributed version control system ideal for the collaborative development of software". You can browse the sources on the website, look at commits, fork your own version of the repository, or just download a snapshot. If you're new to git, just install it (for windows, see here, and here). If you'd like to learn more about Github, or Git, Github guides is a good place to start.

If you'd like to contribute to AgDLR, fork your own version of the repository, commit your changes to your version, and send me a pull request on github. We'll take it from there.

I've already started to work on a new feature which I talked about at Seattle CodeCamp, but that's the next post.