Whiteknight's Blog

Entity Framework Code Only Migrations

2013-01-26T00:00:00-08:00

I’ve been using Entity Framework 4.4 at work a lot recently, and as part of that I’ve been running into some questions about how to do this or that with some of the new features, particularly code first. Sometimes I’m able to find the answers I need from the Googles, but sometimes I’ve got to sit down with VisualStudio and find the answers through good old-fashioned trial and error. Then, I figure what’s the point of having a tech-related blog in the first place if you can’t share the things you’ve learned there. I’ll be sharing bits of what I’m learning as I go.

Code-Only Migrations

The new Entity Framework releases have a feature called code-first, where you can write pure csharp or VB code objects (“Plain Old Code Objects”, or POCO), and have the Entity engine automatically discern from those classes the shape of your DB tables and generate a change script to create them. Most tutorials on the topic explain the process through the use of the Package Manager Console in Visual Studio. I have slightly different requirements and so I’m going to try to do the same exact process using the csharp APIs directly.

Here’s a short but helpful blog post where I started my search:

http://romiller.com/2012/02/09/running-scripting-migrations-from-code/

Create Your DbContext and POCO Classes

I won’t go into detail about that here. There are plenty of cool resources for this purpose elsewhere. For the purposes of the rest of this post, I’ll assume you’ve got a DbContext subclass called “MyDbContext”. Even though you may not like to have it, your DbContext subclass must provide a parameterless constructor to work with the Package Manager Console tools.

Create a Configuration

A Migration Configuration is a class that derives from System.Data.Entity.Migrations.DbMigrationsConfiguration. You can create one of these automatically through the Package Manager Console with the Enable-Migrations command, or you can just create it in code yourself:

namespace MyProgram.Migrations
{
    using System;
    using System.Data.Entity;
    using System.Data.Entity.Migrations;
    using System.Linq;
    using System.Reflection;
    using MyProgram;

    internal sealed class MyConfiguration : DbMigrationsConfiguration<MyProgram.MyDbContext>
    {
        public Configuration()
        {
            AutomaticMigrationsEnabled = false;

            // These things are not strictly necessary, but are helpful when the assembly where
            // the migrations stuff lives is different from the assembly where the DbContext
            // lives. For instance, you may not want to run migrations from a separate
            // development-time console program, and not have that code included in production
            // assemblies.
            MigrationsAssembly = Assembly.GetExecutingAssembly();
            MigrationsNamespace = "MyProgram.Migrations";

        }

        protected override void Seed(MyProgram.MyDbContext context)
        {
            // TODO: Initialize seed data here
        }
    }
}

Create a Migration

Next step is to create a migration. A migration is any class which derives from System.Data.Entity.Migrations.DbMigration. You can create one of these manually, but it’s much easier to create them through the Package Manager Console with the Add-Migration command.

Add-Migration MyMigration

Or, if you need some more options (if your solution has multiple projects, etc):

Add-Migration -Name MyMigration -ProjectName MyProject -ConfigurationTypeName MyProject.Migrations.MyConfiguration

You may also need to specify -StartupProjectName, if your migrations live in a library assembly.

Als, you can specify a separate connection string from what is provided by the default parameterless constructor of your DbContext by specifying -ConnectionStringName (for a named connection string in your app.config/web.config file) or -ConnectionString and -ConnectionProviderName to use a value which is not in your app.config/web.config file.

What do all these options mean? Let’s consider a solution with two projects:

MyProgram.sln
    - MyProgram      (a .exe which references MyProgram.Core.dll)
    - MyProgram.Core (a .dll Class Library)

The project MyProgram.Core.dll contains our DbContext instance and the MyProgram assembly has the app.config with connection string information.

If we want our migrations to live in MyProgram.Core we can use this command as our base (plus any other options we need to add):

Add-Migration MyMigration -ProjectName MyProgram.Core -StartupProjectName MyProgram ...

If, on the other hand, we want the migrations to live in MyProgram, the .exe instead of the .dll, we can use this version:

Add-Migration MyMigration -ProjectName MyProgram -StartupProjectName MyProject ...

If you do not specify -ProjectName or -StartupProjectName, the Add-Migration command will attempt to use whichever project you have flagged as the “default startup project” in the solution explorer (whichever project runs when you press F5).

What if I want to separate my migrations out into a different assembly entirely, one which isn’t included in my production deployment? Here’s another example solution:

MyProgram.sln
    - MyProgram                 (the production deployed .exe)
    - MyProgram.Core            (where our DbContext and model classes live)
    - MyProgram.DbMigration     (where our migration code will live, references MyProgram.Core.dll)

In this case, we can use a command like this:

Add-Migration MyMigration -ProjectName MyProgram.DbMigration -StartupProjectName MyProgram ...

You’re going to have to play with some of the options for different configurations. If the Add-Migrations command says something’s wrong, try tweaking your values and adding more info to the commandline.

Run the Migrations (Some Recipes)

Now that you’ve got migrations and a configuration, you can run the migrations manually. Here are some snippets from a console program which does exactly this:

private void DoDbUpdate()
{
    DbMigrator migrator = new DbMigrator(new MyConfiguration());
    migrator.Update();
}

Let’s take a minute to step back and ask how this all works. You build your assembly and run it. The DbMigrator class uses reflection to read out all classes from your assembly, and find the ones which are subclasses of DbMigration. Each DB migration has a name, which is a combination of a timestamp and the name you gave it in the Add-Migration command. In the database, there’s a table (or will be, after you run your first migration) called dbo.__MigrationHistory (it may be under the “System Tables” folder). That table holds information about migrations you have already ran. When you call DbMigrator.Update(), it searches for all migrations, removes the ones which already have entries in the table, and orders them according to timestamp. This is the list of pending migrations. You can get that list like this:

private void DoDbUpdate()
{
    DbMigrator migrator = new DbMigrator(new MyConfiguration());
    foreach (string migration in migrator.GetPendingMigrations()
        Console.WriteLine(migration);
    migrator.Update();
}

You can also get the raw SQL script which is going to be used:

private void GetDbUpdateScript()
{
    DbMigrator migrator = new DbMigrator(new MyConfiguration());
    MigratorScriptingDecorator scripter = new MigratorScriptingDecorator(migrator);
    string script = scripter.ScriptUpdate(null, null);
    Console.WriteLine(script);
}

Running the scripting decorator clears out the list of pending migrations from the migrator. If you want to generate the script first (for logging) and then run the migration, you need to create two migrators:

private void GetDbUpdateScriptAndUpdate()
{
    MyConfiguration myConfig = new MyConfiguration();
    DbMigrator migrator = new DbMigrator(myConfig);
    MigratorScriptingDecorator scripter = new MigratorScriptingDecorator(migrator);
    string script = scripter.ScriptUpdate(null, null);
    Console.WriteLine(script);

    migrator = new DbMigrator(myConfig);
    migrator.Update();
}

Another thing we could try is to create a logging object, and use a logging decorator to log progress. This mechanism will also output the raw SQL text, but will do so piecewise intermixed with other information (so you’ll need to filter out what is and what is not part of the SQL script):

public class MyLogger : System.Data.Entity.Migrations.Infrastructure.MigrationsLogger
{
    public override void Info(string message)
    {
        // Short status messages come here
    }

    public override void Verbose(string message)
    {
        // The SQL text and other info comes here
    }

    public override void Warning(string message)
    {
        // Warnings and other bad messages come here
    }
}

Once we have a logger, we can use it in our migration:

private void DoDbUpdateWithLogging()
{
    DbMigrator migrator = new DbMigrator(new MyConfiguration());
    MigratorLoggingDecorator logger = new MigratorLoggingDecorator(migrator, new MyLogger());
    logger.Update();
}

We can update to a specific migration, or we can rollback to a specific migration by name. Remember, the “name” used by the migrator is a combination of the timestamp and the name you gave it at the console.

private void UpdateOrRollbackTo(string name)
{
    DbMigrator migrator = new DbMigrator(new MyConfiguration());
    migrator.Update(name);
}

And what if you want to completely trash the DB, undo all migrations, delete everything, and start over?

private void CompletelyTrashDb()
{
    DbMigrator migrator = new DbMigrator(new MyConfiguration());
    migrator.Update("0");
}

What’s My Use Case?

So what exactly is my use-case here? Why don’t I just stick with the Package Manager Console like many other tutorials do? I have a few criteria:

I need to seed the new DB with a lot of complex data, pulled from another source, which needs to be updated regularly.
I may have more than one DB, for multiple instances of my application. Connection strings for all of these may be kept in another DB or a file or somewhere else. All of these need to be kept in sync, and a script that runs a migration on all targets is better than a command which runs on only one and needs to be manually updated.
I’d like the ability to log the SQL scripts which are used for the migration, for various purposes.
I’d like to be able to do some scripted unit testing where we create and migrate a test DB from scratch, seed it with test data, and use that for testing. I would like these temporary test DBs to be identical to the production ones.

Overall I think the new Entity Framework Code-First features are really cool, and remind me very closely of the equivalent db migrations scripts in Rails, but we have a little bit more control over it here because we can incorporate the DbMigration process into our application logic.

Working On a New Project

2012-12-09T00:00:00-08:00

As they say on my son’s favorite Thomas The Tank Engine, yesterday an idea flew into my funnel. I was doing some work on my bathroom when I got an idea for a new website that I would like to make. I want to make this site for a few reasons: First, I’m going to be using the new ASP.NET MVC framework at my new job eventually, and I wanted to practice with it. Second, I have been getting motivated to do more programming in general, and a new project that I’m hot on seems like just the thing to get me moving again. Third, my idea for this site is relatively straight-forward but should offer some good practice and interesting technical challenges. Fourth and finally, this website I’m thinking about is actually something that I would like to use myself (even if nobody else joins me).

I still don’t have a new laptop yet, though I’ve been shopping for one in earnest. I figure I’ll have it shortly after the holidays. In either case, for the time being I’m stuck with my current crappy laptop, which exclusively runs (the decidedly not-crappy) Linux. If I’m going to make a new ASP.NET MVC website on this box, it’s going to have to use Mono.

And that’s fine. I’ve used Mono before and I like it plenty despite it’s shortcomings.

I scratched out a few ideas and designs in my notebook. I decided I wanted to use some kind of dependency injection/inversion of control/service locator feature. I’ve used Unity in the past and loved it, but I wouldn’t be against using Ninject or something else too. I also want to use some kind of ORM to make persistance a little easier.

Now, I can already hear some people mumbling to themselves about all the many flaws of ORMs. I won’t even bother to list them or link to the (many) pages where they are discussed on the interwebs. Use your imagination. In any case, I’m not detered and ORMs actually make good sense for the project I’m thinking about, if I can find the right one.

I thought about using MongoDB, but after thinking hard about work flows and data relationships in my site, I think a regular, SQL-based relational DB would just be a better fit in this instance. I’d probably like to stick with MySQL or MS SQL Server, initially. (This is not to say anything about the relative merits of one type of DB over another, just that one type seems to be a more natural fit for this particular problem domain and I’d like not to be shoehorning in the wrong software for the wrong reasons. Don’t get my involved in your holy war.)

The problem, I discovered, is finding a good ORM that’s worth using, doesn’t introduce more hassle than it saves, and actually works (with examples) on Mono. So far, my search is proving to be a little bit fruitless.

NHibernate seems like a common and popular choice, but the large amounts of required XML configuration make me sick to my stomach. I would far prefer something that I can do in pure C# code without large amounts of externa config.
Castle ActiveRecord builds an ActiveRecord-like interface on top of NHibernate. In theory you get all the power of NHibernate without the XML headaches. However, this package is listed on the castle website as being “Archived” and “no longer being worked on”. Also, I can’t find any real examples of using it on Mono. I’m not going to start a new project (which presumably could be active for years) by starting on an old and unmaintained foundation.
db4o It looks to me like this little project uses it’s own custom DB file format and doesn’t connect to existing databases. I think I’d really like to stick with an existing DB, and not use something custom.
Linq-To-SQL, probably using the SQLMetal code generator, would seem like a decent option except it doesn’t seem to be well-supported on Mono, and there’s the issue of having to generate a whole bunch of pure-data objects, which will need to be laboriously mapped to and from my actual object type definitions. I’ve seen the kinds of morass that this kind of situation can lead to, and I’m not interested in going this route if it is even possible to traverse.
Simple.Data Is a newer option which uses all sorts of fancy modern C# features to provide an extremely flexible, extremely natural-looking interface for accessing a database. It’s supposed to work on Mono, but I’ve not figured out a good way to get it (and a MySQL connector, and prerequisites) installed in a reasonable way for Mono. The docs suggest NuGet, but I can’t get NuGet working on my box (and the NuGet devs don’t seem to care about Mono too much)

Overall, the experience of trying to get this project working on Mono has been frustrating. I understand what a big engineering task Mono is in general, and how much work goes into getting a diverse ecosystem of software to work together nicely on a VM that’s supposed to be cross-platform with low barriers to entry. I get all that. Although, for the purposes of this project I really wanted to start writing some code sooner than later and not have to fight with so much infrastructural stuff. I suppose I have a few options: I can wait till I get a new laptop and do things on a windows partition or VM instead. Or, I can keep fighting with this setup to try and get things to work. Finally, I guess I could port over my idea to Ruby on Rails, another platfrom that I’m interested in learning more about.

More IO Work?

2012-11-21T00:00:00-08:00

I might not be too bright. Either that or I might not have a great memory, or maybe I’m just a glutton for punishment. Remember the big IO system rewrite I completed only a few weeks ago? Remember how much of a huge hassle that turned into and how burnt-out I got because of it? Apparently I don’t because I’m back at it again.

Parrot hacker brrt came to me with a problem: After the io_cleanup merge he noticed that his mod_parrot project doesn’t build and pass tests anymore. This was sort of expected, he was relying on lots of specialized IO functionality and I broke a lot of specialized IO functionality. Mea culpa. I had a few potential fixes in mind, so I tossed around a few ideas with brrt, put together a few small branches and think I’ve got the solution.

The problem, in a nutshell is this: In mod_parrot brrt was using a custom Winxed object as an IO handle. By hijacking the standard input and output handles he could convert requests on those handles into NCI calls to Apache and all would just work as expected. However with the IO system rewrite, IO API calls no longer redirect to method calls. Instead, they are dispatched to new IO VTABLE function calls which handle the logic for individual types.

First question: How do we recreate brrt’s custom functionality, by allowing custom bytecode-level methods to implement core IO functionality for custom user types?

My Answer: We add a new IO VTABLE, for “User” objects, which can redirect low-level requests to PMC method calls.

Second Question: Okay, so how do we associate thisnew User IO VTABLE with custom objects? Currently the get_pointer_keyed_int VTABLE is used to get access to the handle’s IO_VTABLE* structure, but bytecode-level objects cannot use get_pointer_keyed_int.

My Answer: For most IO-related PMC types, the kind of IO_VTABLE* to use is staticly associated with that type. Socket PMCs always use the Socket IO VTABLE. StringHandle PMCs always use the StringHandle IO VTABLE, etc. So, we can use a simple map to associate PMC types with specific IO VTABLEs. Any PMC type not in this map can default to the User IO VTABLE, making everything “just work”.

Third Question: Hold your horses, what do you mean “most” IO-related PMC types have a static IO VTABLE? Which ones don’t and how do we fix it?

My Answer: The big problem is the FileHandle PMC. Due to some legacy issues the FileHandle PMC has two modes of operation: normal File IO and Pipe IO. I guess these two ideas were conflated together long ago because internally the details are kind of similar: Both files and pipes use file descriptors at the OS level, and many of the library calls to use them are the same, so it makes sense not to duplicate a lot of code. However, there are some nonsensical issues that arise because Pipes and files are not the same: Files don’t have a notion of a “process ID” or an “exit status”. Pipes don’t have a notion of a “file position” and cannot do methods like seek or tell. Parrot uses the "p" mode specifier to tell a FileHandle to be in Pipe mode, which causes the IO system to select a between either the File or the Pipe IO VTABLE for each call. Instead of this terrible system, I suggest we separate out this logic into two PMC types: FileHandle (which, as it’s name suggests, operates on Files) and Pipe. By breaking up this one type into two, we can statically map individual IO VTABLEs to individual PMC types, and the system just works.

Fourth Question: Once we have these maps in place, how do we do IO with user-defined objects?

My Answer: The User IO VTABLE will redirect low-level IO requests into method calls on these PMCs. I’ll break IO_BUFFER* pointers out into a new PMC type of their own (IOBuffer) and users will be able to access and manipulate these things from any level. We’ll attach buffers to arbitrary PMCs using named properties, which means we can attach buffers to any PMC that needs them.

So that’s my chain of thought on how to solve this problem. I’ve put together three branches to start working on this issue, but I don’t want to get too involved in this code until I get some buy-in from other developers. The FileHandle/Pipe change is going to break some existing code, so I want to make sure we’re cool with this idea before we make breaking changes and need to patch things like NQP and Rakudo. Here are the three branches I’ve started for this:

whiteknight/pipe_pmc: This branch creates the new Pipe PMC type, separate from FileHandle. This is the breaking change that we need to make up front.
whiteknight/io_vtable_lookup: This branch adds the new IOBuffer PMC type, implements the new IO VTABLE map, and implements the new properties-based logic for attaching buffers to PMCs.
whiteknight/io_userhandle: This branch implements the new User IO VTABLE, which redirects IO requests to methods on PMC objects.

Like I said, these are all very rough drafts so far. All these three branches build, but they don’t necessarily pass all tests or look very pretty. If people like what I’m doing and agree it’s a good direction to go in, I’ll continue work in earnest and see where it takes us.

New Job

2012-11-20T00:00:00-08:00

One thing that’s been eating up my time (and energy, and attention) lately has been a hunt for a new job. I had intended to look around passively for a while because I wasn’t in any big hurry. However, once the recruiters got word that I was looking, things started to move much more quickly. Suddenly I’m getting dozens of phone calls every day, and dozens of emails. I was doing phone screens and going on interviews. All of this and I was trying to not impact my current job so much.

First thing’s first: I’m not leaving WebLinc because I’m unhappy with it. Also, I don’t think they’re unhappy with me. WebLinc is a great place to work, and I’m thankful for the time I’ve spent there. If you’re in the Philadelphia area and you know ColdFusion, Ruby on Rails, and/or have solid web fundamentals (JS/CSS/HTML) or graphic design experience (or sales, or project management, etc) you should consider applying. It’s a very hip young organization with great talent, a rapidly growing and diverse clientele, and some real opportunity to do cool things. Also, there’s a cool bar/restaurant on premises, and the company has a good (and growing) relationship with open-source and the developer community at large. If you’re young and talented and care about the craft of web development, definitely consider WebLinc in your job search. You won’t be disappointed.

So why am I leaving? WebLinc historically has had two main platforms: ColdFusion and ASP.NET. Between the two, the ColdFusion team has had some of the biggest project successes and the more demonstrated ability to scale up the size of it’s team. When you’re a company that’s growing as fast as WebLinc, the ability to scale up your team quickly, to meet deadlines and to keep to budgets are all very important. The ColdFusion team was doing these things better than .NET (for a variety of reasons, not the least of which were endemic to the platform itself). This lead to more sales for the ColdFusion team, and a larger, steadier stream of work. At some point the decision was made to devote resources going forward into ColdFusion (and a small, but growing, Ruby on Rails team) and not devote new resources to .NET. This has nothing to do with the relative theoretical merits of ASP.NET vs ColdFusion and, in my opinion, has nothing to do with the quality of developers they had working on those platforms. The reasons why one team was doing better than the other team aren’t really worth exploring at this point, but from a business perspective it was clear where effort and resources needed to be devoted going forward.

I started looking around for a variety of reasons. I could have stayed for a while in my current position, riding the waves of boom and bust that are inherent in any job that bills hourly for maintenance. I could have started transitioning over towards the Ruby on Rails team but chose not to go in that direction, yet. Instead of sitting around and hoping things went well, I wanted to take a little bit more control of my situation. I started looking passively at first, but once the recruiters got involved things started moving quickly and the rest is history. I’ve written a few notes up about my job hunt and my dealings with various recruiters. These notes may turn into additional blog posts (now that I have time and energy to try blogging more).

In early December I’ll be starting at Halfpenny Technologies, a small but growing company involved with electronic medical records and related areas. The team is small, the company is growing rapidly, and they have some very real and very interesting technical challenges bubbling to the forefront. At the interview they were throwing around words like “ownership” and “leadership”, and were talking about some very interesting new technologies. Combine that with a few other factors, and the decision was actually an easy one for me to make.

I have a good idea about what kinds of work I’m going to be doing there but I don’t want to talk about it quite yet. In reality, you don’t know anything until you start working and get knee-deep in code. I also don’t know what their policies and attitudes are on blogging and public commentary, but I’ll say what I can when I’m confident enough to say it.

So starts a new chapter in my career, one that I’m hoping lasts quite a while and takes me in some cool new directions. Again, I’ll post more when I have more to say.

September Status

2012-09-14T00:00:00-07:00

First, some personal status:

Personal Status

I haven’t blogged in a little while, and there’s a few reasons for that. I’ll list them quickly:

Work has been…tedious lately and when I come home I find that I want to spend much less time looking at a computer, especially any computer that brings more stress into my life. Also,
My computer at home generates a huge amount of stress. In addition to several physical problems with it, and the fact that I effectively do not have a working mouse (the built-in trackpad is extremely faulty, and the external USB mouse I had been using is now broken and the computer won’t even book if it’s plugged into the port), I’ve been having some software problems with lightdm and xserver crashing and needing to be restarted much more frequently than I think should be needed. We are planning to buy me a new one, but the budget won’t allow that until closer to xmas.
The io_cleanup1 work took much longer than I had anticipated. I wrote a lot more posts about that branch than I ever published, and the ones I did publish were extremely repetitive (“It’s almost finished, any day now!”). Posting less means I got out of the habit of posting, which is a hard habit to be in and does require some effort.

I’m going to do what I can to post something of a general Parrot update here, and hopefully I can get back in the habit of posting a little bit more regularly again.

`io_cleanup1` Status

io_cleanup1 did indeed merge with almost no problems reported at all. I’m very happy about that work, and am looking forward to pushing the IO subsystem to the next level. Before I started io_cleanup1, I had some plans in mind for new features and capabilities I wanted to add to the VM. However, I quickly realized that the house had some structural problems to deal with before I could slap a new coat of paint on the walls. The structure is, I now believe, much better. I’ve still got that paint in the closet and eventually I’m going to throw it on the walls.

The io_cleanup branch did take a lot of time and energy, much more than I initially expected. But, it’s over now and I’m happy with the results so now I can start looking on to the next project on my list.

Threads Status

Threads is very very close to being mergable. I’ve said that before and I’m sure I’ll have occasion to say it again. However there’s one remaining problem pointed out by tadzik, and if my diagnosis is correct it’s a doozie.

The basic threads system, which I outlined in a series of blog posts ages ago goes like this: We cut out the need to have (most) locks, and therefore we cut out many possibilities of deadlock, by making objects writable only from the thread that owns them. Other threads can have nearly unfettered read access, but writes require sending a message to the owner thread to perform the update in a synchronized, orderly manner. By limiting cross-thread writes, we cut out many expensive mechanisms that would need to be used for writing data, like Software Transactional Memory (STM) and locks (and, therefore, associated deadlocks). It’s a system inspired closely by things like Erlang and some functional languages, although I’m not sure there’s any real prior art for the specifics of it. Maybe that’s because other people know it won’t work right. The only thing we can do is see how it works.

The way nine implemented this system is to setup a Proxy type which intercepts and dispatches read/write requests as appropriate. When we pass a PMC from one thread to another, we instead create and pass a Proxy to it. Every read on the proxy redirects immediately to a read on the original target PMC. Every write causes a task to dispatch to the owner thread of the target PMC with update logic.

Here’s some example code, adapted from the example tadzik had, which fails on the threads branch:

function main[main](var args) {
    var x = 1;
    var t = new 'Task'(function() { x++; say(x); });
    ${ schedule t };
    ${ wait t };
    say("Done!");
}

Running this code on the threads branch creates anything from an assertion failure to a segfault. Why?

This example creates a closure and schedules that closure as a task. The task scheduler assigns that task to the next open thread in the pool. Since it’s dispatching the Task on a new thread, all the data is proxied. Instead of passing a reference to Integer PMC x, we’re passing a Proxy PMC, which points to x. This part works as expected.

When we invoke a closure, we update the context to point to the “outer” context, so that lexical variables (”x”, in this case) can be looked up correctly. However, instead of having an outer which is a CallContext PMC, we have a Proxy to a CallContext.

An overarching problem with CallContext is that they get used, a lot. Every single register access, and almost all opcodes access at least one register, goes through the CallContext. Lexical information is looked up through the CallContext. Backtrace information is looked up in the CallContext. A few other things are looked up there as well. In short, CallContexts are accessed quite a lot.

Because they are accessed so much, CallContexts ARE NOT dealt with through the normal VTABLE mechanism. Adding in an indirect function call for every single register access would be a huge performance burden. So, instead of doing that, we poke into the data directly and use the raw data pointers to get (and to cache) the things we need.

And there’s the rub. For performance we need to be able to poke into a CallContext directly, but for threads we need to pass a Proxy instead of a CallContext. And the pointers for Proxy are not the same as the pointers for CallContext. See the problem?

I identified this issue earlier in the week and have been thinking it over for a few days. I’m not sure I’ve found a workable solution yet. At least, I haven’t found a solution that wouldn’t impose some limitations on semantics.

For instance, in the code example above, the implicit expectation is that the x variable lives on the main thread, but is updated on the second thread. And those updates should be reflected back on main after the wait opcode.

The solution I think I have is to create a new dummy CallContext that would pass requests off to the Proxied LexPad. I’m not sure about some of the individual details, but overall I think this solution should solve our biggest problem. I’ll probably play with that this weekend and see if I can finally get this branch ready to merge.

Other Status

rurban has been doing some great cleanup work with native PBC, something that he’s been working on (and fighting to work on) for a long time. I’d really love to see more work done in this area in the future, because there are so many more opportunities for compatibility and interoperability at the bytecode level that we aren’t exploiting yet.

Things have otherwise been a little bit slow lately, but between io_cleanup1, threads and rurban’s pbc work, we’re still making some pretty decent progress on some pretty important areas. If we can get threads fixed and merged soon, I’ll be on to the next project in the list.

io_cleanup1 Lands!

2012-08-27T00:00:00-07:00

FINALLY! The big day has come. I’ve just merged whiteknight/io_cleanup1 to master. Let us rejoice!

When I started the project, months ago, I had intended to work on the branch for maybe a week or two at the most. Get in, clean what I could, get out. Wash, rinse, repeat. That’s exactly why I named the branch “io_cleanup1”, because I intended it to just be the first of what would be a large series of small branches. Unfortunately as I started cleaning I was lead to other things that needed to go. And those things lead elsewhere. Before I new it I had deleted just about all the code in all the files in src/io/* and started rewriting from the ground up.

Sometimes sticking with a plan and breaking up projects into small milestones is a good thing. Othertimes when you know what the final goal is and you’re willing to put in the effort, it’s good to just go there directly. That’s what I ended up doing.

To give you an idea of what my schedule was originally, I had intended to get this first branch wrapped up and merged before GSOC started, so that I could keep my promise of implementing 6model concurrently with that program. With GSOC over last week (I’ll write a post-mortem blog entry about it soon), I’ve clearly failed at that. I’m extremely happy with the results so far and given the choice I would not go back and do things any differently. The IO system was in terrible condition and it desperately needed this overhaul. I wish it hadn’t taken me so long, but with a system that’s so central and important, it was worthwhile taking the extra time to make sure things were correct.

Where to go from here? My TODO list for the near future is very short:

Threads
6model
More IO work

The Threads branch, the magnum opus of Parrot hacker nine is 99.9% of the way there. If we can just push it up over the cliff, we should be able to merge soon and open up a whole new world of functionality and cool features for Parrot. I’m already planning out all the cool additions to Rosella I’m going to make once threads are merged: Parallel test harness. Asynchronous network requests, an IRC client library. The addition of a real, sane threading system opens up so many avenues to us that really haven’t been available before. Sure there are going to be plenty of hiccups and speedbumps to deal with as we really get down and start to use this system for real things, but the merge of the threads branch represents a huge step forward and a great foundation to build upon.

I’m going to be putting forward as much effort as I can to getting this branch wrapped up and merged. Some of the remaining problems only manifest on hard-to-test platforms, which is where things start to get tricky. As I mentioned in an email to parrot-dev a while ago, test reports on rare platforms are great, but if we can’t take action on the reported failures we can get ourselves into something of a bind. The capability to find problems on those platforms and the capability to fix problems on those platforms are two very different capabilities. But, most of the time that’s a small issue and we’re going to just have to find a way to muscle through and get this branch merged one way or the other. If we can merge it without purposefully excluding any platforms, that would be great.

Before anybody thinks that I’m done with IO and that system is now complete, think again. There is still plenty of work to be done on the IO subsystem, and all sorts of cool new features that become possible with the new architecture and unified type semantics. I want to separate out Pipe logic from FileHandle into a new dedicated PMC type. Opening FileHandles in “p” mode for pipes is clumsy at best, and I want a more sane system. And while I’m at it, 2-way and 3-way pipes would make for a great feature addition (we can’t currently do these in any reliable way).

The one thing that has changed most dramatically in the new IO system is buffers. The buffering subsystem has not only been rewritten but completely redesigned. Instead of being type-specific they are now unified and type independent. Buffers are their own struct with their own API. Instead of having a single buffer that is used for both read and write, handles now have separate read and write buffers that can be created and managed independently. I want to create a new PMC type to wrap these buffers and give the necessary management interface so they can be used effectively from the PIR level and above.

Finally, the whiteknight/io_cleanup1 branch tried to stay as backwards compatible as possible, so many breaking changes I wanted to make had to wait until later. In the future expect to see many smaller branches to remove old broken features, old crufty interfaces, and old bad semantics. We’ll make these kinds of disruptive changes in much smaller batches, with more space between them.

Parrot 4.7.0 "Hispaniolan" Released!

2012-08-22T00:00:00-07:00

On behalf of the Parrot team, I’m proud to announce Parrot 4.7.0, also known as “Hispaniolan”. Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 4.7.0 is available on Parrot’s FTP site, or by following the download instructions at http://parrot.org/download. For those who would like to develop on Parrot, or help develop Parrot itself, we recommend using Git to retrieve the source code to get the latest and best Parrot code.

Parrot 4.7.0 News:

- Core
    + Added .all_tags() and .all_tagged_pmcs() methods to PackfileView PMC
    + Several build and coding standards fixes

The SHA256 message digests for the downloadable tarballs are:

4360ac3dffafffaa00bce561c1329df8ad134019f76930cf24e7a875a4422a90 parrot-4.7.0.tar.bz2
c0bffd371dea653b9881ab2cc9ae5a57dc9f531dfcda0a604ea693c9d2165619 parrot-4.7.0.tar.gz

Many thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next scheduled release is 18 September 2012.

The release is indeed out a day late. It’s not that I forgot about it, it’s just that I can’t read a calendar and HOLY CRAP, IT’S WEDNESDAY ALREADY? When did that happen? So, and I can’t stress this enough, Mea Culpa.

io_cleanup1 Done?

2012-07-22T00:00:00-07:00

This morning I made a few last commits on my whiteknight/io_cleanup1 branch, and I’m cautiously optimistic that the branch is now ready to merge. The last remaining issue, which has taken the last few days to resolve, has been fixing readine semantics to match some old behavior.

A few days ago I wrote a post about how complicated readline is. At the time, I thought I had the whole issue under control. But then Moritz pointed out a problem with a particular feature unique to Socket that was missing in the new branch.

In master, you could pass in a custom delimiter sequence as a string to the .readline() method. Rakudo was using this feature like this:

str = s.readline("\r\n")

Of course, as I’ve pointed out in the post about readline and elsewhere, there was no consistency between the three major builtin types: FileHandle, Socket and StringHandle. The closest thing we could do with FileHandle is this:

f.record_separator("\n");
str = f.readline();

Notice two big differences between FileHandle and Socket here: First, FileHandle has a separate record_separator method that must be called separately, and the record separator is stored as state on the FileHandle between .readline() calls. Second, FileHandle’s record separator sequence may only be a single character. Internally, it’s stored as an INTVAL for a single codepoint instead of as a STRING*, even though the .record_separator() method takes a STRING* argument (and extracts the first codepoint from it).

Initially in the io_cleanup1 branch I used the FileHandle semantics to unify the code because I wasn’t aware that Socket didn’t have the same restrictions that FileHandle did, even if the interface was a little bit different. I also didn’t think that the Socket version would be so much more flexible despite the much smaller size of the code to implement it. In short, I really just didn’t look at it closely enough and assumed the two were more similar than they actually were. Why would I ever assume that this subsystem ever had “consistency” as a driving design motivation?

So I rewrote readline. From scratch.

The new system follows the more flexible Socket semantics for all types. Now you can use almost any arbitrary string as the record separator for .readline() on FileHandle, StringHandle and Socket. In the whiteknight/io_cleanup1 branch, as of this morning, you can now do this:

var f = new 'FileHandle';
f.open('foo.txt', 'r');
f.record_separator("TEST");
string s = f.readline();

…And you can also do this, which is functionally equivalent:

var f = new 'FileHandle';
f.open('foo.txt', 'r');
string s = f.readline("TEST");

The same two code snippets should work the same for all built-in handle types. For all types, if you don’t specify a record separator by either method, it defaults to “\n”.

Above I mentioned that almost any arbitrary string should work. I use the word “almost” because there are some restrictions. First and foremost, the delimiter string cannot be larger than half the size of the buffer. Since buffers are sized in bytes, this is a byte-length restriction, not a character-length restriction. In practice we know that delimiters are typically things like “\n”, “\r\n”, ”,”, etc. So if the buffer is a few kilobytes this isn’t a meaningful limitation. Also, the delimiter must be the same encoding as the handle uses, or it must be able to convert to that encoding. So if your handle uses ascii, but you pass in a delimiter which is utf16, you may see some exceptions raised.

I think that the work on this branch, save for a few small tweaks, is done. I’ve done some testing myself and have asked for help to get it tested by a wider audience. Hopefully we can get this branch merged this month, if no other problems are found.

IO Cleanups Home Stretch

2012-07-10T00:00:00-07:00

I had made a round of fixes with regards to encodings in the whiteknight/io_cleanup1 branch a few days ago. Rakudo hacker Moritz was able to take a look at Rakudo’s spectests and verify that more tests were indeed passing because of it. The remaining test failures represent the changing semantics for the read method and what appear to be two genuine regressions or bugs.

Hopefully I will be able to get all these things sorted out this week before I go away on a mini vacation next weekend. Otherwise I can’t imagine this branch gets merged before the 4.6 release this month.

A few days ago I wrote a post about readline and some of the intricacies involved in that, and some of the weird semantics that I was attempting to unify. It turns out that some of these semantics are a major cause in one of the last bugs in the branch. Let’s look at some code in master to see where the hangup is. First, readline on a Socket:

METHOD readline(STRING *delimiter    :optional,
                INTVAL has_delimiter :opt_flag) {
    INTVAL idx;
    STRING *result;
    STRING *buf;
    GET_ATTR_buf(INTERP, SELF, buf);

    if (!has_delimiter)
        delimiter = CONST_STRING(INTERP, "\n");

    if (Parrot_io_socket_is_closed(INTERP, SELF))
        RETURN(STRING * STRINGNULL);

    if (buf == STRINGNULL)
        buf = Parrot_io_reads(INTERP, SELF, CHUNK_SIZE);

    while ((idx = Parrot_str_find_index(INTERP, buf, delimiter, 0)) < 0) {
        STRING * const more = Parrot_io_reads(INTERP, SELF, CHUNK_SIZE);
        if (Parrot_str_length(INTERP, more) == 0) {
            SET_ATTR_buf(INTERP, SELF, STRINGNULL);
            RETURN(STRING *buf);
        }
        buf = Parrot_str_concat(INTERP, buf, more);
    }

    idx += Parrot_str_length(INTERP, delimiter);
    result = Parrot_str_substr(INTERP, buf, 0, idx);
    buf = Parrot_str_substr(INTERP, buf, idx, Parrot_str_length(INTERP, buf) - idx);
    SET_ATTR_buf(INTERP, SELF, buf);
    RETURN(STRING *result);
}

We can ignore the fact that this implementation of readline doesn’t call Parrot_io_readline like every other PMC does. Or that if we did call that function the program would throw an exception because Parrot_io_readline doesn’t support sockets anyway. Whatever. Moving on…

For comparison, let’s look at the version from the Handle PMC (which is inherited by FileHandle):

METHOD readline() {
    STRING * const string_result = Parrot_io_readline(INTERP, SELF);
    RETURN(STRING *string_result);
}

The Socket version takes a delimiter parameter which is a STRING. When doing readline on a Socket, you can pass in any arbitrary string which is used as the token for end of line. With FileHandle, you don’t seem to have that. However, you can definitely use custom delimiters with FileHandle. However, we clearly don’t take a delimiter here and we aren’t passing one in as an argument to Parrot_io_readline like we do in the branch. Let’s see how it’s done instead. Here’s a snippet from Handle PMC:

    ATTR INTVAL    record_separator;  /* Record separator (only single char supported) */

We don’t need to look at any other code. This is the smoking gun. Socket.readline() can take any arbitrary STRING to use as a record separator, but FileHandle.readline() can only use a single codepoint, which it doesn’t take as an argument.

So that’s the problem right there. When I standardized the readline mechanics between types, I picked the FileHandle semantics. This was probably the wrong decision, because not only could Sockets use a more general mechanism but Rakudo relies on that behavior in its spectests. This does raise a question about why nobody ever expected this same behavior from FileHandle, or why the difference was not considered some kind of bug. It really goes to show how immature our IO system has been for all these years, and how we had all just grown accustomed to the arbitrary, inconsistent, nonsensical behaviors. It just works for some basic usages, so nobody ever complains about it. That time is, thankfully, coming quickly to an end.

Fixing this issue is actually going to take some serious work. Several function signatures are going to need updating to take a STRING delimiter instead of an INTVAL codepoint, and a major chunk of buffering logic is going to need to be rewritten to work on substrings instead of on individual codepoints. This, in turn, is going to require a heck of a lot more testing.

Last night I started putting in some of the changes necessary to use a substring terminator instead of a single codepoint. Most of what I’ve already done has been modifying function signatures. The real changes need to occur deep within the buffering logic and will require a little bit more time.

I’m looking forward to getting this branch fixed up and merged back to master so I can get to work on my next project. I think 6model is going to be the next thing I dig into, before I find something else that annoys me enough to put in a huge amount of effort to rewrite it. I’ll post more updates about my future projects and plans as I go.

HTML with Rosella Xml and Net

2012-06-30T00:00:00-07:00

HTML is a derivative of SGML, just like XML is. Sure, they look pretty much the same for the most part, but there are a few key differences that prevent HTML from being parsed exactly like XML. Part of the reason why I like XHTML so much is that it’s more usable with more parsers, including many of simpler and full-featured XML parsers. Simplicity in parsing was one of the original motivations of the XML design, at least in comparison to a full SGML parser or even something like a full HTML parser.

But that’s all besides the point.

I’ve been in something of a backyard gardening kick lately. We bought our house only a few short months ago, and are only half way through the first summer growing season in my modest little garden. My plans for next year are much more expansive. I’ve finally talked my wife into letting me buy some cherry trees to plant. She was also pretty willing to get a few grape vines planted (especially when I sketched out the beautiful wooden arbor they would be growing on). She put her foot down when I started talking about blueberries, apples and pears, however. And another garden bed or two for more vegetables. For some reason she’s convinced that we need some measure of open space in our little plot so the kid has somewhere to run and play. Some people have weird priorities.

This is all sort of besides the point too.

Getting the things I need for all this gardening work I’ve talked myself into is not cheap. Cherry tree seeds actually do grow on trees so that’s not a big deal, but other things like fertilizers, soil amendments, tools, materials for building a grape trellis and raised garden beds, not to mention a longer hose to reach all the new things that are going to require regular watering all cost money. And maybe a sprinler, like one of those fancy ones on an electronic timer. I can avoid some of that cost by getting things used and at discount on sites like Craigslist. So I’ve been going there. Every day.

And it’s tedious. I have to sort through hundreds of listings for things I don’t want, in categories that seem far too course. Sometimes, because things often get incorrectly categorized, I have to look in other related categories too, sorting through things that are even less relevant on average to try and find the occasional gem. This is all on top of the hardware-related problems I have being unable to use the trackpad on my laptop so web navigation on sites without keyboard shortcuts is an extreme pain. I start to think to myself: I can do better, I’m a programmer! For some values of “better” and “programmer”.

Enter Rosella. Now with Parrot, Winxed and Rosella I can use the Net library to fetch the text of the HTML code of the page. After some hacking in the last few days, I can parse that code with my Xml library (set in a new lenient mode) and start to work with it in a meaningful way:

function main[main]() {
    var rosella = load_packfile("rosella/core.pbc");
    Rosella.initialize_rosella("xml", "net", "string");

    var ua = new Rosella.Net.UserAgent.SimpleHttp();
    var response = ua.get("http://philadelphia.craigslist.org/w4m/");
    var doc = Rosella.Xml.read_string(response.content, false);

    doc.get_document_root()
        .get_children_named("body")
        .get_children_named("blockquote")
        .get_children_named("p", "row":[named("class")])
        .map(function(node) {
            return {
                "title": node.first_child("a").get_inner_xml(),
                "link":  node.first_child("a").attributes["href"],
                "price": node.first_child("span", "itempp":[named("class")]).get_inner_xml(),
                "has_pic": !Rosella.String.null_or_empty(
                    node.first_child("span", "itempx":[named("class")]).get_inner_xml()
                )
            };
        })
        .filter(function(obj) {
            return indexof(obj["title"], "compost") >= 0;
        })
        .map(function(obj) {
            return Rosella.String.format_obj("<a href='{link}'>{title} for {price}</a>", obj);
        })
        .foreach(function(string s) { say(s); });
}

That second argument to Rosella.Xml.read_string tells the parser to go into “non-strict” mode, which is basically my attempt to fudge the XML parsing rules to allow for the SGML nonsense in HTML. Without that, the parser will blow up pretty early in the parse because of unbalanced tags. The XML parser by default does not handle tags which are not balanced and which do not have the trailing slash to indicate a standalone tag, and the Craigslist source is filled with those kinds of things.

All I need to do is set this scraper up on a timer, and have it send me results somehow. If I set up a small server with mod_parrot and some kind of tool for generating RSS feeds, I could have this output neatly delivered to me on a regular basis. Considering that mod_parrot is moving along so smoothly and RSS is just another XML format, I think this is a pretty reasonable idea.

So, I started working on that. As of last night, I’ve sketched out two small libraries, one for RSS feeds and one for the competing standard, Atom. These libraries are thin wrappers around the XML library to deal with the specifics of RSS and Atom. Here’s an example of consuming an RSS feed:

var rss = Rosella.Rss.read_url("http://www.parrot.org/rss.xml");
rss
    .channels()
    .first()
    .items()
    .foreach(function(i) {
        say(Rosella.String.format_obj("{title} (by {creator}) : {description}", i));
    });

You can do almost exactly the same thing with an Atom feed too, if you’ve got one of those instead. Right now RSS and Atom are implemented in two separate libraries, but I may combine them together for simplicity and to avoid unnecessary code duplication.

I’m working on an interface to write and publish feeds as well, though that’s not quite ready yet. You can bet that when I’ve got that working, I’ll be setting up a copy of mod_parrot to use it with.

I’ve been sort of kicking around the idea of a specialized HTML parsing library, which would more or less be an SGML parser with some schema information. I’m not sure I want to get into that hassle because HTML is a pretty messy thing and it will take a huge amount of effort to get something that works most of the time. But, if you’re willing to put up with a little bit of oddity, the Xml library works well enough for many cases.

Sockets and Encodings

2012-06-28T00:00:00-07:00

The io_cleanup1 branch is nearing completion, though as always the last few details are what holds everything up. In the past few days all the remaining tests in the parrot repo were passing. The coding standards tests, as usual, the last to be resolved. Then I started building and testing other things on the branch: Winxed builds and tests fine. So does Rosella. Then I looked at NQP and Rakudo. Both built fine, but Rakudo was failing two socket-related spectests.

That’s not entirely unexpected. Even though my intention was to make this branch as painless as possible there were still some unavoidable changes to interfaces and semantics. There are a few places where older semantics are surrounded by large /* HACK! */ comments, but for the most part I’ve tried to make everything sane. That’s why I wasn’t surprised to see Rakudo failing a few tests. I was much more surprised that Rakudo built without any problems the first time I tried it. I figured the test failures represented some kind of semantic mismatch, and getting Rakudo passing again would have been as easy as getting the old semantics returned, with a note about a future update path.

It turns out this wasn’t exactly the case. For one test it was the simple difference in the way we read on streams with multibyte encodings. This was expected and we can fix it to use the old behavior if that’s what Rakudo prefers. For the second failing test, it’s not that there’s a semantic difference per se, but instead there is a glaring and serious bug in master that was corrected in the new branch. Here, I’m going to explain what’s going on.

Look at this code:

Parrot_io_recv_handle(PARROT_INTERP, ARGMOD(PMC *pmc), size_t len)
{
    Parrot_Socket_attributes * const io = PARROT_SOCKET(pmc);

    /* This must stay ASCII to make Rakudo and UTF-8 work for now */
    STRING * res    = Parrot_str_new_noinit(interp, len);
    INTVAL received = Parrot_io_recv(interp, io->os_handle,
                                     res->strstart, len);

    res->bufused = received;
    res->strlen  = received;

    return res;
}

This is a pared-down version of the code behind the recv method on Socket. It creates a new string with the specified length pre-allocated, then passes the buffer to the low-level recv C API (which has been abstracted a little to account for platform differences).

Notice the comment there in the middle which says the string uses the ASCII encoding, for use by Rakudo. This is what I saw, and this is the semantic I followed in the new system: When you read from a socket by default in the new system, the string is encoded as ASCII unless you specify differently.

Just for my own verification, I had to look at the Parrot_str_new_noinit function to verify that the string was, in fact, being set to ASCII:

Parrot_str_new_noinit(PARROT_INTERP, UINTVAL capacity)
{
    STRING * const s = Parrot_gc_new_string_header(interp, 0);
    s->encoding = Parrot_default_encoding_ptr;

    Parrot_gc_allocate_string_storage(interp, s,
        (size_t)string_max_bytes(interp, s, capacity));

    return s;
}

Elsewhere in the system, we have this:

Parrot_default_encoding_ptr = Parrot_ascii_encoding_ptr;

So yes, the string returned by the Socket does indeed use the ASCII encoding in master. And, after double-checking, the version in the io_cleanup1 branch was using ASCII also. However, in the new branch Rakudo’s test fails because of an exception about a lossy conversion of non-ascii data into the the lower bit-width format. A quick check shows that both systems create an ASCII string buffer and both systems call the same recv function to fill it. So where’s the problem? What the hell?

For comparison, here’s the snippet of code from the new branch that reads data into a STRING, possibly using a buffer:

bytes_read = Parrot_io_buffer_read_b(interp, buffer, handle, vtable,
                                   s->strstart + s->bufused, byte_length);
s->bufused += bytes_read;
STRING_scan(interp, s);

We’re reading out a number of bytes, appending them into the string’s pre-allocated storage and updating the number of bytes actually used. That’s all the same as in master. However, the last line, STRING_scan does not appear in master. What is it?

STRING_scan() loops through the data in the string to verify that it correctly matches the string’s encoding. For instance, if the string is encoded as ASCII, STRING_scan will loop through to make sure all character values are lower than 128. If the string is UTF-16, STRING_scan verifies that we have an even number of bytes and that each value is an acceptable codepoint.

master doesn’t do this, which means there is a bug. In master, we don’t scan the string after recv but before we return it to the user, which means we can have non-ASCII data in a string marked with the ASCII encoding. The Rakudo test puts UTF-8 data into the socket on the server side, and then reads out a string and encodes that to UTF-8 to verify that it comes out correctly. However in the new branch we actually check that the string is valid before giving it out to user code, and it isn’t, so we throw an exception.

Combine that with the fact that the Socket PMC has no way to change the encoding it uses in master, which means all Sockets used in Parrot master are potential sources of bugs.

Two nights ago I added methods to Socket to get/set the encoding to use, and everybody’s favorite Moritz created a branch for Rakudo to use it. Last night I did some playing with default encodings. Tonight and into the weekend I’m hoping to wrap up the last few details to get the Rakudo spectest passing like normal again. Hopefully, if all goes well, we can start talking about a merger within the next week or two.

Reading a Line of Text

2012-06-13T00:00:00-07:00

In terms of usage, there aren’t too many IO-related features in Parrot’s user interface more straight-forward than the readline method. It does exactly what you tell it to do: read a line of text from the given file and return that line of text as a Parrot string. Easy.

Tonight I was looking at some of the old code to get an idea about expected semantics for some tests that need fixing. Let’s look at some code:

.sub read_a_line
    .param string type
    $P0 = new [type]
    $S0 = $P0.'readline'()
    .return($S0)
.end

.sub test_readline
    $S0 = 'read_a_line'('FileHandle')
    say $S0
    $S0 = 'read_a_line'('Socket')
    say $S0
    $S0 = 'read_a_line'('StringHandle')
    say $S0
.end

The valid types for this are, as usual, "FileHandle", "Socket" and "StringHandle". Notice that we’re reading a line from the object of the given type before we’ve opened, connected or initialized. Pretend, in order to save myself some typing, that I’ve set up exception handlers and the like above. So, what happens?

For FileHandle we throw an exception. You can’t read from a closed handle.
For StringHandle, we throw an exception for the same reason.
For Socket we return null because…whatever. (in the test suite we test that when converted to a floating-point number, that it’s 0.0. Again, whatever).

So that’s a little bit weird that socket does something different from the other two, but fundamentally it’s a pretty different type so I suppose some differences can be allowed.

Now, let’s try something slightly different:

.sub read_a_line
    .param string type
    $P0 = new [type]
    $P0.'open'("foo.txt", "r")
    $P0.'print'("This is \n test text")
    $P0.'close'()
    $S0 = $P0.'readline'()
    .return($S0)
.end

With this example we can only operate on FileHandle and StringHandle because Socket doesn’t have an .open() method like those two do. What does this do for those two types?

For FileHandle we throw the same exception, you still can’t read from a closed handle.
For StringHandle you can read like normal without any indication that the handle is closed!

So that’s weird to say the least that StringHandle has two different behaviors. Socket has yet another problem, in a slightly different way. The method Socket.readline() returns null when not open, but if you pass a Socket to the Parrot_io_readline method, it always throws an exception because apparently readline on a Socket isn’t supported! And because readline on a Socket uses a completely different code path from FileHandle the two types use completely different buffering mechanisms with subtly different semantics (StringHandle, because it uses the in-memory string buffer, does it in a third way).

To recap: What is conceptually a simple operation, read in some text until we find a delimiter, is done in three completely different ways by three different types, each with different error-handling semantics depending on both history, state, and the interface used. If anybody was wondering why I wanted to rewrite this subsystem, here’s part of the reason.

Actually, I kind of lied. It’s really not a simple operation which is all the more reason we should share common code. It’s a clear case of an algorithm where the hard parts should be encapsulated inside a clean interface so that different types can avoid needing to reimplement it over and over again (with differences, bugs and complications). That’s the way it really should be, but some of the complications in the code are a little hard to live with. Here’s the general algorithm for readline on a FileHandle, as it’s implemented in Parrot master:

The filehandle requires a buffer for this, so create (and fill) a buffer if one isn’t configured.
Create a new, empty STRING header.
Treating the buffer like an encoded STRING, scan the buffer looking for the end of the delimiter or the end of the buffer, whichever comes first.
Allocate/reallocate enough space in the STRING header to hold all the data we’ve found in the buffer.
Append all the characters we’ve found to the STRING.
If we’ve found the delimiter, we’re done. Return it to the user.
Otherwise, check if we are at the end of file for the input. If so, go to 8. If not end of file, go to 9.
Check that the last codepoint is complete and has all its bytes. If so, return the STRING to the user. If not, throw an exception about a malformed string.
Check that the last codepoint is complete and has all its bytes. If so, go to 10. Otherwise, go to 11.
Refill the buffer and go to 3.
Determine how many more bytes we need to read to complete the last codepoint.
Refill the buffer, and check that we have at least that many bytes available to read. If so, go to 13. Otherwise, throw an exception about a malformed string input.
Read in the necessary number of bytes (1, 2 or 3 at most) from the buffer and go to 3.

If you’re reading an ASCII or fixed8 string the logic obviously collapses down to something a little bit more manageable. Also, this same logic, almost line for line, is repeated in the routine to read a given number of characters from the handle, where characters in a non-fixed-width encoding (like utf8) may need multiple reads to get if we don’t get all the bytes for the character into the buffer in a single go. Notice that the versions provided by StringHandle and Socket are both much more simple and not safe for multi-byte encodings like utf8 or utf16.

In my io_cleanup1 branch, the logic has been simplified substantially, and a single codepath is now used for all three of the major types:

Make sure the handle has a read buffer set up and filled.
Create a new, empty STRING header.
Ask the buffer to find the given end-of-line character. The buffer will return a number of bytes to read in order to get a whole number of codepoints, and a flag that says whether we’ve found the delimiter or not.
Append those bytes to the string header.
If the delimiter is found or if we are at EOF, return the string.
Fill the bufffer and go to #3.

By simply coding the buffer logic to refuse to return incomplete codepoints in response to a STRING read request, the whole algorithm becomes hugely simplified. The readline routine in master takes up 185 lines of C code. In my new branch, the same routine takes up only 47 lines. Of course, this isn’t comparing apples to apples, because I did break up some of the repeated logic into helper routines, and the buffers in my system are obviously a little bit smarter about STRINGs and codepoints, but that’s not exactly the point. The real point is that three large, complicated, hard-to-read functions in master are now a single, much smaller, easier-to-read routine that relies on clear abstraction boundaries to do a difficult job in a much more conceptually simple way.

I’ve also updated the STRING read routine (now called Parrot_io_read_s) to use a similar algorithm and actually share some of the new helper methods. That sharing itself also helps to decrease total lines of code has has other benefits as well.

Notice that there is one small change in these two algorithms, which may or may not need to be worked around if it causes problems. Notice that we don’t read out of the buffer an incomplete codepoint. If we have an incomplete one at the end of the file, the first algorithm will read it in and throw an exception about a malformed string. The second algorithm will ignore those final bytes and successfully return all the rest of the valid-looking data from the buffer instead. In the first algorithm, it then becomes impossible to read the partial data out and make a best effort, while in the second algorithm you can easily get to the data, even if the last codepoint is corrupted and cannot be read. I’d really love to hear what people think about this change, and whether it’s worth keeping or needs to change. I suspect it is better this way but only the users can really say for sure.

IO Rewrite Status

2012-06-08T00:00:00-07:00

I was going to call this post “IO Cleanup Status”, but let’s face facts: This is a complete rewrite of the entire subsystem. I haven’t hardly left a single line of code untouched. It is a full rewrite of the system hiding behind a mostly-similar (though not quite the same) API. I didn’t intend to completely rewrite the whole subsystem when I started the branch, hence the benign-sounding branch name. Following along with our cultural norms, I could have called it whiteknight/io_massacre or something similarly upbeat. Whatever. I’ve known people stuck with un-liked names for their entire lives, so this branch can be misnamed for a few weeks.

So what is the status of this branch, exactly?

At the time of this writing the branch is mostly complete. The major architectural work has all been done, with per-type logic separated out into new IO_VTABLE structures, and buffering logic divorced from FileHandle into a new IO_BUFFER structure. Now you can do things that have never been possible before, like buffering socket input and output, or doing readline with custom line-end characters on all handle types, and a whole bunch of other, increasingly-obscure operations. A lot of the new capabilities are things you didn’t even know we didn’t support before. Now, we do.

We aren’t quite there yet, but the stage is set for some other awesome changes in the future too, which I’ll talk about in more depth when we get there.

The current status of the branch is good. Parrot builds without any huge amount of new warnings and with no errors on my platform. Some platform-specific code needs to be updated for Windows, I’m sure. The one big thing standing in the way is keeping track of file positions through operations like seek and tell. These things are made a little bit more difficult when you have read buffers reading ahead, because the position of the next character to read according to the user may be far different than the position of the file descriptor according to the operating system. Then consider the case when you have a file opened for read and write, with buffers in both directions. The old system had a single buffer per FileHandle which needed to be flushed if you tried to read when the buffer was in write mode, or you tried to write when it was in read mode. If you’re switching back and forth between reading and writing often enough, buffering actually decreases performance when it’s supposed to be a performance enhancer.

The FileHandle has an attribute to keep a pointer to the current cursor location, but I’m not always updating it as often as I should and not always reading it when I should. If you have a file opened for read and write, when you write 5 characters at the current file position you need to increment the read buffer by 5 characters also. When you go to read in 5 characters from the current position, you either need to flush the write buffer first or you can try to read those characters right out of the write buffer. There’s nothing complicated about it, just a lot of bookkeeping to get right and lots of little interactions that need to be tested. It’s helpful that we don’t do seek or tell on some things like Sockets, and we don’t really buffer StringHandles.

The branch is moving along well and if I can find the time to actually sit down and work on it for a dedicate period of time I might be able to get it closer to being done. I’m shooting for being mergable sometime after the coming release.

IO Cleanups Status

2012-06-01T00:00:00-07:00

Tonight I’ve hit something of a milestone with my branch to rewrite the IO subsystem. As of tonight the parrot binary, parrot-nqp and winxed all build in my branch and coretest runs (though fails some tests). The entire build does not complete because of some failures related to dynops, but it does get most of the way through. This means that most of the main-path IO APIs and FileHandle operations are working correctly, which is a relatively small portion of everything that has changed.

With Parrot building, I’m now able to more closely keep track of progress and regressions, and do more live testing as I make new changes. Until this point all my changes have just been mental exercises, so I’m happy to have a little bit more feedback and even some validation.

Of course, just saying that it builds doesn’t really mean anything. Several things are still not implemented or completely wired up. Some operations on files such as seek, peek and tell are still not implemented yet. Several methods on the various PMCs (FileHandle, Socket and StringHandle) have not been updated to use the new system. There are a few regressions I need to address with regards to buffering. Specifically, “line buffering” has been removed from the system during the rewrite and hasn’t been added back. Line buffering in Parrot has never really done much, but it’s just hacky and obscure enough that I’m sure somebody is relying on it.

Some things, like files opened for dual read/write modes or append modes haven’t been completely dealt with in code either. I don’t think there’s a lot of work to do for this, but since the buffering architecture has changed so much from what it used to be and since these modes are relatively rare and not as thoroughly tested I want to spend a little bit of extra time making sure there are no regressions.

Also there are several coding standards tests (especially for function-level annotations and documentation) which fail spectacularly in the branch, and it’s going to take time to update all the old documentation and add docs for all the new functions. I also need to update PDD 22 to reflect the new architecture of the IO system.

I’ve been working on this branch pretty aggressively for the last two weeks and I think I’m about 50% of the way done. That’s not too bad considering the magnitude of the change and the amount of time I’ve had to hack. Within a week or two more, if all goes well, I think the branch might be ready for wider testing and eventually merging.

As usual when we’re talking about changes this big, merges are not something to be rushed. Assuming all goes well and other people like what I’ve been doing, expect to see a brand new IO system in Parrot sometime later this summer.

IO Refactors

2012-05-27T00:00:00-07:00

The IO subsystem is a lot like the garbage collector: So long as it just works we can ignore its faults for quite a long time. The garbage collector had performance and other issues for years before everybody’s favorite bacek went through and finally rewrote it. His effort there saves the rest of us meer mortals from having to touch the GC again for another couple years.

The IO system works reasonably well. It’s got a decent set of features more or less, it implements most of the important operations that our users have needed in the past, it’s not spectacularly slow (and disk or network operation performance almost always outweighs any issues in the code that leads to those things), and we haven’t been getting a lot of error reports or feature requests for it. In short, if it ain’t broke, don’t fix it.

A few days ago I was working on a ticket for moritz to add better integration between our various IO vector PMCs (FileHandle, Socket, etc) and the ByteBuffer PMC. ByteBuffer is what it’s name implies: It’s an array-like type for working with individual bytes in a chunk of memory. It’s like a binary encoded STRING, but it’s not immutable and has a handful of additional features that a raw STRING (or the String PMC) doesn’t. ByteBuffer can be populated from and exported to a STRING, and it is useful for certain types of operations that need to operate on a sequence of bytes without having to worry about strings and encodings and all that other nonsense. Mortiz’s request was a reasonable one so I sat down and made it happen. A few nights ago I merged that work in to master with an “experimental” tag on it.

However while I was in the IO subsystem code making this happen something did break. Not in the code, instead something broke inside my poor little head. The snapping sound you hear is the poor camel’s back under the load of that last piece of straw. I’ve had enough of that system and its inside-out organization and collection of half-ideas and botched refactors. I’ve had my fill of the nonsense and finally decided it was time to make things right.

And before anybody says to me, “hey Mr Whiteknight, you shouldn’t be so mean, somebody probably worked really hard to make this code do what it does”, let me just say two things: First, “Mr Whiteknight” is my father’s handle and Second, I was one of the people who helped put IO where it is today. I don’t feel particularly bad insulting myself or my own work, and my contributions, though well-intentioned at the time, are a big part of why the system is in the condition its in now. First, a brief history lesson.

When I joined Parrot, it sported an IO system based on layers. Layers were arranged in a structure something like a vtable, and IO requests would be fed through the layers. Each layer getting the output of the one before it until the bottom layer actually spat the data out (or, read it in depending on which way you were moving). This worked pretty well when you were trying to do File IO on a file with a particular encoding, with buffering, through an asychrony mechanism, etc. Actually I say it worked well but it was sort of overkill: It was just too much infrastructure for the possible benefits and despite the theory of allowing better code reuse there really weren’t too many different layering combinations that could be set up. Plus, layers start to interdepend and violate encapsulation, then optimization starts prompting a few “short cuts” where layers were flattened together. One of the earlier things I did on Parrot, post-GSOC, was to remove some of the last vestiges of the then-unused layering system from Parrot’s IO.

The IO subsystem has something of a problem where it has a few masters and has to be performance conscious. Many of our programs are still the kind that shuffle data about (very much in the influence of Perl) and IO operation performance mattered when your compiler is reading in HLL code and outputting PIR code, then you’re reading PIR code in and trying to compile it again. Too much nonsense and everybody feels it.

In Parrot at the user level you can do IO in two ways: Through the IO PMCs (FileHandle, mostly) and through opcodes (say, print, etc). The problem, put succinctly, is this: We want to encapsulate logic for writing to files inside the FileHandle PMC, but we don’t want to add new IO-specific VTABLES and we don’t want to incur the costs of method calls on every single IO request. In other words, we didn’t want the print opcode to just be a thin wrapper around the print method on FileHandle. Such a thing, especially if implemented naively, would have killed performance by creating nested runloops and a whole host of other problems.

The way the system is set up is that both FileHandle.print() and the print opcode are both thin wrappers around the real routine Parrot_io_putps, which does all the hard work. And, more importantly, that routine is expected to act transparently (like the print opcode does) on any IO PMC type like Socket or StringHandle. The only real way to do this, if you can’t call a method on the FileHandle and Socket PMC is to use a large switch-statement:

switch (handle->vtable->base_type) {
    case enum_class_FileHandle:
        ...
    case enum_class_Socket:
        ...
    case enum_class_StringHandle:
        ...
    default:
        Parrot_pcc_invoke_method_from_c_args(..., handle, "print", ...);
}

I’ve obviously glossed over all the details, but this is the general form of that routine and several other similar routines in the IO API. You’ll notice several things from even a quick glance at this example:

If we want to add a new IO type to Parrot core we need to add a new entry to the switch statement in every IO API routine that needs to care about PMC type (this is a major part of the reason we don’t yet have a sane, separate Pipe type).
If the user passes in an Object, something defined at the PIR level, we do fall back to calling the method, because we can’t do anything else intelligently.
We can’t really subclass FileHandle or Socket from the user level, because it would fail the base_type test, and wouldn’t be able to handle the low-level structure accesses from that point forward anyway.

Point number 2 is particularly interesting because the FileHandle.print() method calls Parrot_io_putps, which may turn around and call the .print() method. This is a big part of the reason why FileHandle cannot be subclassed in user code. It’s clearly an example of poorly separated concerns and poor encapsulation. Either the method should call the IO API or the IO API should call the method but we can’t be doing both. Actually, I’d far prefer the former, if we can do it in a good, general way.

There are a few other issues worth mentioning, which I’ll just dump rapid-fire without much explanation:

We don’t have a separate Pipe type. Instead, FileHandle can be opened in “pipe mode” to write to a separate process or read output from a separate process.
We have limited buffering, but only on FileHandle and we cannot configure buffers for input and output separately, or use separate buffers.
We don’t really have encodings set up in any consistent way, so it’s very possible, though I haven’t worked out all the details, to write strings with different encodings to a file. This is especially true if we’re using buffers and performing writes through different API routines.
FileHandle logic is considered to be the default and is given deference in the code. Pipe logic is unified with file logic at a very low level. Socket and StringHandle are treated as bolted-on spare parts and don’t benefit from hardly any code sharing or unified architecture. They also don’t have all the same useful features as FileHandle has.
Several functions in the IO subsystem are poorly or inconsistently named and implemented, not to mention the often-times confusing documentation and absurd architectural arrangements.

So that’s the system we’ve got. What do I want to do to fix these issues?

The first thing I’ve suggested is to break up IO functionality into an IO_VTABLE of function pointers, similar to how the STR_VTABLE, the sprintf dispatch mechanism, the packfile segment dispatch table and other similar mechanisms in Parrot work. Each IO request would go through the API routines, which dispatch to a vtable routine (possibly with some intermediate buffering logic). Here’s what it looks like in the branch to do a basic write:

IO_VTABLE * const vtable = IO_GET_VTABLE(interp, handle);
vtable->write_s(interp, handle, str->strstart, str->bufused);

And here’s how to do it with write buffering:

IO_VTABLE * const vtable = IO_GET_VTABLE(interp, handle);
IO_BUFFER * const read_buffer = IO_GET_READ_BUFFER(interp, handle);
Parrot_io_buffer_write_s(interp, handle, vtable, buffer, str);

Internall, the buffer does it’s magic and flushes data out to the vtable if necessary.

The second thing I want to do is break out buffering so that instead of being a detail of the FileHandle PMC a buffer is a separate struct which can be attached to any IO type as desired. And, even better, we can attach multiple buffers to an IO stream, at least one each for input and output, configured separately. The buffering API, which will be cleaned up and properly encapsulated, will take a pointer to the IO_VTABLE for the handle and will pass data through transparently as required. A thin wrapper PMC type, IO_BUFFER, would allow references to buffers to be accessed and configured directly, which would be very useful in some cases.

Imagine, if I may go off on a short tangent, a threaded system where one worker task had a reference to a buffer and continuously made sure it was filled in the background while another worker task read bits and pieces from the buffer very quickly. It would be possible, through careful choice of algorithm, to do such a thing lock-free. Feel free to replace “file” with “socket” or “pipe” in the example above too. Imagine also a system where we can transparently use mmap (or it’s windows equivalent) to map a file to memory as part of the buffer, and keep working with it that way.

The third thing I want to do is start teasing apart the logic for Pipes from the file logic. I’ll create a separate io_vtable for pipe operations, and use that inside FileHandle when we’re in pipe mode. Eventually we’ll be able to create a separate type, divide out all the logic completely, and get to work on really interesting stuff like feasible 2-way and 3-way pipes.

The fourth thing I want to do is start setting up interfaces so that IO operations including buffering, low-level IO, file descriptor manipulation and other things become more accessible at the PIR level so users can make better use of these tools, both in subclasses of the in-built handle PMCs and in custom types which neither derive from nor hold instances of those types.

I’ve started sketching out many of these ideas in the whiteknight/io_cleanup1 branch. cotto seems to agree with the general direction and I haven’t heard any complaints so far, so I’ve had my head down and been working hard on making these ideas reality. As of this writing, I’ve modified just about every single line of code in the subsystem, gotten most of the new architecture and logic into place and set up the vtables for the most important built-in types. I have a few details to finish up before I try to build (and inevitably debug) this new beasts. Ultimately I would like this first round of cleanups to produce no user-visible changes, so the old PMC methods and exported API functions are going to continue doing what they’ve always done. Later rounds of cleanups will add new interfaces and eventually deprecate and remove some of the crufty older ones. I’ll post more updates as this work progresses.

Destructors are Hard

2012-05-23T00:00:00-07:00

««<« HEAD:drafts/gc_destructors.md In my last post I mentioned some of the work I was trying to do with GC finalization and destructors. I promised I would publish a longer and more in-depth post about destructors, what the current state is, what I am doing,

and what still needs to be done.

In my last post I mentioned some work involving the GC, finalization and destructors. Today I’m going to expand on some of those ideas, talk about what the current state of destruction and finalization are in Parrot, some of the problems we have with coming up with better solutions, and some of the things I and others are working on to get this all working as our users expect us to. I apologize in advance for such a long post, there’s a lot of information to share, and hopefully a much larger architectural discussion to be started.

3d0a502723cb7124eb717d7e82bac5ecc567ac31:_posts/2012-05-23-destructors_are_hard.md

Destructors are hard. The idea behind a destructor is a simple one: We want to have a piece of code that is guaranteed to execute when the associated object is freed. Memory allocated on the heap is going to get reclaimed en masse by the operating system when the process exits. However, things such as handles, connections, tokens, mutexes, and other remote resources might not necessarily get freed or handled correctly if the process just exits, or if the object is destroyed without some sort of finalization logic performed on it. Here’s a sort of example that’s been bandied about a lot recently:

function main () {
    var f = new 'FileHandle';
    f.open("foo.txt", "w");
    f.print("hello world");
    exit(1);
}

In this example we would expect that the text "hello world" would be written to the foo.txt file. However, because the text to be written may be buffered (both in Parrot and by the OS), there’s a very real chance that the data won’t get written if we do not call the finalizer for the FileHandle PMC.

Obviously, the brain-dead solution to this particular problem is to manually close or flush the file handle:

function main () {
    var f = new 'FileHandle';
    f.open("foo.txt", "w");
    f.print("hello world");
    f.close();
    exit(1);
}

However, the whole point of having things like finalizers (“destructors”) and GC is to make it so that the programmer does not need to worry about little details like these. The program should be smart enough to find dead objects in a timely manner and free their resources. Beyond that, many programming languages (with special emphasis on Perl6) require the availability of reliable and sane destructors.

In the remainder of this post I would like to talk about why destructors are hard to implement correctly, why Parrot does not currently (really) have them, and some of the ideas we’ve been kicking around about how to add them.

First, let’s cover where we currently stand. Parrot does have destructors, of a sort, in the form of the destroy vtable. That routine is called by the GC when the object is being reclaimed, during the sweep pass. A side-effect of this implementation is that if PMC A refers to PMC B and both are being collected, it’s very possible that A’s destructor tries to access some information in B after B has already been reclaimed. Think about a database connection object that maintains a socket on one side, and a hash of connection information on the other. The socket probably cannot perform a simple disconnect, but instead should send some sort of sign-off message first to alert the server that it can proceed with its own cleanup. The socket PMC would need information from the connection information hash to send this final message, but if the hash had already been reclaimed the access would fail with undefined results.

This situation has lead to more than a few calls for ordered destruction. In one of the most common and severe cases, Parrot’s Scheduler PMC was being relied upon by various managed PMCs. When a Task PMC was destroyed, at least in earlier iterations of the system, it would attempt to send a message to the Scheduler that it was no longer available to be scheduled. Ignore for a moment the fact that the Task could not possibly have been reclaimed in the first place if the Scheduler had a live reference to it, and if the Scheduler was still alive itself.

Because of some of these order-of-destruction bugs, GC finalization (a final, all-encompassing GC sweep path guaranteed to execute all remaining destructors prior to program exit) had been turned off. That and performance reasons. Turning off GC finalization leads to the problem above where data written to the FileHandle is not not flushed before program exit. You are probably now starting to understand the bigger picture here.

Having ordered destruction means essentially that we should be able to have an acyclic dependency graph of all objects in the system with destructors. However, maintaining this in the general case is impossible and attempting to approximate it would be very expensive in terms of performance. In any case, this is just a way to work around the problem of our naive sweep algorithm, which destroys and frees dead objects in a single pass, and not a real solution to the larger problems. A far better idea, recently suggested by hacker Moritz, is a 2-pass GC sweep.

In the 2-pass case the GC sweep phase would have two loops: the first to identify all PMCs to be swept (from a linear scan of the entire memory pool), execute destructors on them and add them all to a list, and the second to iterate over that list (after all destructors had been called) and reclaim the memory. Because of the linked-list setup of the GC, this second pass could, conveivably, be almost free because we could simply append this list of swept items to the end of the free list for an O(1) operation , and the first pass would be no less friendly on the processor data cache than our current sweep would be. This, in theory, solves our problem with ordered destruction, and should allow us to re-enable GC finalization globally without having to worry about these kinds of bugs causing segfaults in the final milliseconds of a program.

So that’s the basics of our current system and our problem with GC finalization, and shows us how we would proceed to make sure destructors were always called as a guarantee of the VM. However, this doesn’t begin to address any of the problems with destructors that will plague their implementation and improvement. I’ll talk about that second subject now.

Destructors, as I said earlier, are hard. In the case of GC finalization, after the user program has executed and exited, it’s relatively easy to loop over all objects and call destructors. It is those destructors which happen during normal program execution that cause problems.

In the C++ language, destructors have certain caveats and limitations. For instance, we can’t really throw exceptions from destructors, because that may crash the program. Not just an “oops, here’s an exception for you to handle”, but instead a full-on crash. In Parrot we can probably be smarter about avoiding a crash but not by much. It’s a limitation of the entire destructors paradigm. Let me demonstrate what I’m talking about.

Let’s say I have this program, which opens up a filehandle to write a message and then starts doing something unrelated to the filehandle but expensive with GC:

function foo() {
    var f = new 'FileHandle';
    f.open("foo.txt", "w");
    f.write("hello world!");
    f = null;       // No more references to f!

    for (int j = 0; j < 1000000; j++) {
        var x = new MyObject(j);
        x.DoSomething();
        x.DoSomethingElse();
        x.DoOneLastThing();
    }
}

Somewhere along the line, when the GC runs out of space, it’s going to attempt a sweep and that means that f is going to be identified as unreferenced, finalized and reclaimed. The question is, where? The thing is that we don’t know where GC is going to run for a few reasons:

We don’t know how many free headers GC has left in the free list before it has to sweep to find more.
We don’t know how many PMCs are being allocated per loop iteration, because the various methods on x could be allocating them internally, and all PCC calls currently generate at least one PMC, and this is a lot of pressure.

So at any point in that loop, any point at all, GC could execute and reclaim the FileHandle f. That calls the destructor, flushes the output, and frees the handle back to the operating system. Good, right? What if there is a problem closing that handle, and the destructor for FileHandle tries to throw an exception (or, if this example isn’t stoking your imagination, imagine that f is an object of type MyFileHandleSubclass with an error-prone finalizer).

There are a few options for what to do here. The first option is that we throw the exception like normal. This means that the loop code with the MyObject variables, which is running perfectly fine and has no reason to throw an exception by itself, is interrupted in mid loop. The backtrace, if we provide one at all, probably points to MyObject but with an exception type and exception message indicative of a failed FileHandle closing. Initial review by the poor developer doing the debugging will show that there are no filehandles trying to close inside this loop and then we get a bug report because a snippet of code which is running just fine exits abruptly with an error condition which it did not cause. The solution for this, wrapping every single line of code you ever write in exception handlers to catch the various possible exceptions thrown from GC finalizers, is untenable from a developer perspective.

A second option is that we somehow disallow things like exceptions from being thrown from destructors, because there’s no real way to catch them rationally. This seems reasonable, until we start digging into details. How do we disallow these, by technical or cultural means? And if we’re relying on cultural means (a line in a .html document somewhere that says “don’t do that, and we won’t be responsible if you do!”), what happens if a hapless young programmer does it anyway without having first read all million pages of our hypothetical documentation? Does Parrot just crash? Does it enter into some kind of crazy undefined state? Obviously we would need some kind of technical mechanism to prevent bad things from happening in a destructor, though the list of potentially bad things is quite large indeed (throwing exceptions, allocating new PMCs, installing references to dead about-to-be-swept objects into living global PMCs, etc) and filtering these out by technical means would be both difficult and taxing on performance. When you consider that even basic error reporting operations at an HLL level, depending on syntax and object model used, may cause a string to be boxed into a PMC, or a method to be called requiring allocation of a PMC for the PCC logic, or whatever, we end up with finalizers which are effectively useless.

A third option is that we could just ignore certain requests in finalizers, such as throwing exceptions. If an exception is thrown at any point we just pack up shop, exit the finalizer and pretend it never happened. This works fine for exceptions, but does nothing for the problem of a finalizer attempting to store a reference to the dieing object into a living object. I don’t know why a programmer would ever want to do that, but if it’s possible you can be damned sure it will happen eventually. Also, when I say “pack up shop”, we’re probably talking about a setjmp/longjump call sequence, which isn’t free to do.

The general consensus among developers is that errors caused by programs running on top of Parrot should never segfault. If you’re running bytecode in a managed environment, the worst that you should ever be able to get is an exception. Segmentation faults should be impossible to get from a pure-pbc example program.

However, as soon as you introduce destructors, suddenly these things become possible. And not just from specifically malicious code, even moderately naive code will be able to segfault by storing a reference to a dieing PMC in a place accessible from live PMCs. Unless, that is, we try to do something expensive like filtering or sandboxing, which would absolutely kill performance.

And this point I keep bringing up about dead objects installing references to themselves in living objects is not trivial. Our whole system is built around the premise that objects which are referenced are alive and objects which are no longer referenced can be reclaimed by GC. Throughout most of the system we dereference pointers as if they point to valid memory locations or to live PMCs. If we turn that assumption around and say that dead objects may still be referenced by the system, then we lose almost all of the benefits that our mark and sweep GC has to offer. Specifically we would either have to install tests for “liveness” around every single PMC pointer access, which would bring performance to a standstill. Otherwise, we need to have a policy that says the user at the PIR level is able to create segfaults without restriction, though officially we declare it to be a bad idea. It’s not just a matter of having to test PMCs to make sure they are alive, the memory could be reclaimed and used for some other purpose entirely! Meerly accessing a reclaimed PMC could cause problems (segfaults, etc) or, if the PMC has already been recycled into something like a transparent proxy for a network resource, send network requests to do things that you don’t want to have happen! The security implications are troubling at best.

The only real solution I can come up with to this problem, and it’s not a very good one, is to add a “purgatory” section to the GC, where we put PMCs during GC sweep but we do not actually free them. The next time GC runs, anything which is still in purgatory is clearly not referenced and can be freed immediately. Anything that is no longer in purgatory has been “resurrected” by some shenanigans and has to be treated as still being alive even though its destructor has already been called. In other words, we take a performance hit and enable zombification in order to prevent segfaults. I don’t know what we want to do here, this is probably the kind of decision best left to the architect (or tech-savvy clergy) but I just want to point out that none of our options are great.

I’ve also brought up the problem with allocating new objects during a finalizer. Why is this a problem? Keep in mind that GC tends to execute when we’ve attempted to allocate an object and have none in the free list. If we have no available headers on the free list, are already in the middle of a GC sweep and ask to allocate a new header, what do we do? Maybe we say that we invoke GC when we have only 10 items left (instead of 0) on the free list, guaranteeing that we always have a small number of headers available for finalization, though no matter what we set this limit at it’s possible we could exhaust the supply if we have many objects to finalize with complex finalizers. Every time a finalizer calls a method or boxes a string, or does any of a million other benign-sounding things PMCs get allocated. If we try to allocate a PMC when there are no PMCs on the free list and we’re already in the middle of GC sweep, the system may trigger another recursive GC run.

Another option is that we could maintain multiple pools and only sweep one at a time. If one pool is being swept we could allocate PMCs from the next pool (possibly triggering a GC sweep in that second pool and needing to recurse into a second pool, etc). Maybe we allocate headers directly from malloc while we’re in a finalizer, keep them in a list and free them immediately after the finalizer exits. We have some options here, but this is still a very ««<« HEAD:drafts/gc_destructors.md real problem that requires very careful consideration. Something like a semi-space GC algorithm might help here, because we could allocate from the “dead space” before that space was freed.

Or we could try to immediately free some PMCs during the first sweep pass, and use those headers as the free list from which to allocate during destructors. This raises some problems because it would be very difficult to identify PMCs which could be freed during the first pass without negating any references which are going to be accessed during the destructors. Also, we run into the (rare) occurance where all the PMCs swept during a particular GC run have destructors, and there are no “unused” headers to immediately free and

recycle for destructors.

real problem that requires very careful consideration. Again, I don’t have an answer here, just a long list of terrible options that need to be sorted according to the “lesser of all evils” algorithm.

3d0a502723cb7124eb717d7e82bac5ecc567ac31:_posts/2012-05-23-destructors_are_hard.md

Let’s look at destructors from another angle. Obviously a garbage-collected system is supposed to free the programmer up from having to manually manage memory (at least) and possibly other resources as well. You make a mess and don’t want to clean it yourself, the GC comes along after you and takes care of the things you don’t wnat to do yourself. On one hand the argument can be made that if you really care about a resource being cleaned in a responsible, timely manner, that you call an explicit finalizer yourself and leaving those kinds of tasks to the finalizer is akin to saying “I don’t care about that object and whatever happens, happens.” After all, if you can’t throw an exception from a destructor and if the destructor is called outside normal program flow with no opportunity to report back even the simplest of success/failure conditions, it really doesn’t matter from the standpoint of the programmer whether it succeeded or silently failed. Further, if the resource is sensitive, you don’t clean it explicitly and Parrot later crashes and segfaults because some uninformed user created a zombie PMC reference, your destructor cannot and will not get called no matter what. If all sorts of things at multiple levels can go wrong and prevent your destructor from running, does it really matter if the destructor gets called at all?

Another viewpoint is that destructors don’t need to be black-boxes, and we don’t care if they have problems so long as they’ve given a best effort to ««<« HEAD:drafts/gc_destructors.md free the resources, those efforts have a decent expected chance of success, and they have an opportunity to log problems in case somebody has a few moments to spare reading through log files. After all, if a FileHandle fails to close in an automatically-invoked destructor, it also would have failed to close in a manually-invoked one and what are you going to do about it? If the thing won’t close, it won’t close. You can either log the failure and keep going with your program (like our destructor would have done automatically) or you can raise hell and possibly terminate the program (like what could happen if an exception is thrown from a destructor). In other words, when you’re talking about failures related to basic resources at the OS level, there aren’t many good options when you’re writing programs at the Parrot level.

I suspect that what we are going to end up with is a system where we allocate a temporary managed pool of PMCs to be available, and allocate all PMCs during a destructor from that pool. After GC, we clear the emergency pool at once. This solution adds a certain amount of complexity to the GC and also does nothing to deal with the zombie references problem I’ve mentioned several times. We’d have to make a stipulation that PMCs allocated during a destructor may not themselves have automatic destructors.

Things start to get a little bit complicated no matter what path we choose. This is the kind of issue where we’re going to need lots more input,

especially from our users.

free the resources and they have an opportunity to log problems in case somebody has a few moments to spare reading through log files. After all, if a FileHandle fails to close in an automatically-invoked destructor, it also would have failed to close in a manually-invoked one and what are you going to do about it? If the thing won’t close, it won’t close. You can either log the failure and keep going with your program (like our destructor would have done automatically) or you can raise hell and possibly terminate the program (like what could happen if an exception is thrown from a destructor). In other words, when you’re talking about failures related to basic resources at the OS level, there aren’t many good options when you’re writing programs at the Parrot level. If you’re not so hot at OS administration, there might not be anything you can do no matter what.

In Parrot we really want to enable PMC destruction and GC finalization. As things stand now you can run destroy vtables written in C, usually without issue. However when we expose this functionality to the user we are talking about executing PBC, in a nested runloop (at least one!), with fresh allocations and all the capabilities of PBC at your disposal. As soon as you open that can of worms, the many problems and problematic possibilities become manifest. The security concerns become real. The performance implications become real. I’m not saying that these are problems we can’t solve, I’m only pointing out that they haven’t been solved already because they are hard problems with real trade-offs and some tough (and unpopular) decisions to be made.

3d0a502723cb7124eb717d7e82bac5ecc567ac31:_posts/2012-05-23-destructors_are_hard.md

Pending Branchwork

2012-05-20T00:00:00-07:00

As I promised in my last post, I have several branches up in the air that need to be worked on. Some branches merged last week after the release. Others are pending to merge soon and some are still in development. In this post I’m going to give a short summary of these things, since I haven’t been posting regular updates like normal.

Already Merged

After the release last week I merged three small branches that brought small changes and appeared to test cleanly with NQP and Rakudo. In short, these were uncontroversial.

whiteknight/gh_675 named after the Github Issue of the same name, this branch removed the can vtable. In all cases in core and in external projects where I looked, the can vtable was simply a redirect to the find_method vtable and a check for null. There’s no need for this added indirection, we can call the find_method VTABLE directly from can opcode.
whiteknight/imcc_file_line This branch removed some very old, long-deprecated IMCC directives. The .line and .file directives were not poorly implemented (as far as IMCC goes) but they weren’t used and weren’t introspectable. The setline and setfile directives (yes, they are directives even though they looked like opcodes!) weren’t used anywhere and weren’t implemented well. I’ve removed all four. Now, we can use the .annotate directive to replace all of these and add other metadata besides in a way that is easy to introspect from within running bytecode.
whiteknight/remove_cmd_ops removed a few command-line arguments from the parrot executable which were non-functional. These arguments have been disconnected since the time of the IMCC API cleanups months ago, and nobody had even noticed. Now they’re gone.

Those things out of the way, here’s a list of some of the branches that are currently unmerged but may be merging soon.

`eval_pmc`

This is one of the most disruptive branches I’ve got going, which is why I’m in no hurry to merge it. Before I can merge it I need to patch both NQP and Rakudo. I submitted patches for these but they weren’t ready to apply and I have to go back and re-do them.

This branch removes the deprecated Eval PMC. The IMCCompiler PMC has already been updated to use a PDD31-compliant interface, which returns a PackfileView PMC instead of an Eval. NQP and Rakudo need to be updated to use this new interface instead of the older VTABLE_invoke one. This update will work in the Parrot master branch just fine, so we can make those updates to NQP and Rakudo and test them thoroughly before we merge the eval_pmc branch in.

`remove_sub_flags`

This is a much bigger and much more disruptive branch. However, because of the fact that NQP and Rakudo don’t really use subroutine flags for their control flow, those two projects won’t really be affected as much as everybody else will be.

The remove_sub_flags branch removes the :load and :init flags from the PIR syntax and replaces them with :tag. The only real way to work with :tag is through the PackfileView PMC, so we need to merge the eval_pmc branch into Parrot first before we can make any further progress on this one. This is a back-burner task and will probably not be touched before the end of the summer.

`whiteknight/gc_finalize`

We’ve received some requests from Rakudo folks that we need to start getting serious about GC finalization. This involves two changes: First is setting the GC to perform a finalization sweep at interp exit, which it currently is not doing. The second is to fix some sweep-related behaviors so the destroy VTABLE can be much more sane and useful.

The whiteknight/gc_finalize branch does both of these things. First, it re-enables GC finalization which had been turned off for so long that the code for it no longer works in master. Second, it moves to a two-stage sweep algorithm, so that we execute all destroy vtables first before we start freeing any resources.

There are still going to be problems with destroy vtables however, and I’m searching for solutions to these. Let me illustrate with a short example. We call GC to sweep typically in response to a request for a new PMC when we have none on the free list. If we have an item on the freelist, we return that immediately and very quickly. If not, we invoke GC to try and free up some headers (or allocate new ones from the OS).

Let’s say we’re programming in Rakudo Perl6 and we have an object with a destructor. For the purposes of our example, it’s a DB connection object. That destructor needs to call a method on a Socket object connecting the client program to the server. As everybody should be aware of now, calling a method in Parrot itself is going to allocate a CallContext PMC.

However, we run into a small problem because we’re in GC because we’re out of PMCs to allocate. So if we try to allocate a new PMC at this point I don’t know exactly what will happen but I can only imagine that the results would not be good. At the worst case, we recursively call into GC which goes back to sweeping, which re-executes finalizers, and we get into an infinite loop.

I won’t go into all the details here, I’ve got another (long) post drafted that discusses these and some other issues related to finalization. This whiteknight/gc_finalize branch solves some of the first few problems but there will be more to come after that.

`whiteknight/gh_663`

The singleton designator for C-level PMCs has been deprecated for some time now, and the whiteknight/gh_663 branch intends to remove them.

Here’s how singletons work in Parrot: The get_pointer and set_pointer vtables are used to manage a single reference to an existing singleton PMC if any. To get the PMC, we invoke the get_pointer vtable without an invocant PMC (the only such occurance of a vtable invoked without an existing PMC reference in the whole codebase that I am aware of). If it returns NULL, a new header is created. If the new header is created, the set_pointer vtable is called on the new object with itself as an argument.

This all happens inside Parrot_pmc_new and is mostly transparent, except for the few bits of code throughout the system which violate this (rather flimsy) encapsulation boundary.

The get_pointer and set_pointer vtables operate on void* pointers, so we even lose typesafety. Plus, we don’t expose get_pointer or set_pointer vtables to PIR code, so there’s absolutely no way to create a singleton class at the user-level using this mechanism. You can do what users of all other languages do and create an accessor and restricted constructor and implement singletons that way. In fact, I think that’s better.

The majority of offending code has been ripped out of this branch, though I’m still seeing some segfaults during the build as a result of bad, unchecked pointer accesses in places where encapsulation has been violoated. I’ve got to spend a little bit more time tracking down some of these failures. Then, assuming NQP and Rakudo aren’t relying on this mechanism, the merge should be relatively painless.

`whiteknight/gh_610`

A while ago, moritz suggested that we improve integration of our ByteBuffer PMC type, especially with our FileHandle and Socket types. We should be able to read a sequence of raw bytes from either of those PMCs into a ByteBuffer and we should be able to write raw bytes from a ByteBuffer into either of those destinations too.

The whiteknight/gh_610 aims to make this a reality. Already I’ve done most of the code work to get this in place, though I haven’t added all the necessary tests and documentation. Plus, a few coding standards tests are failing too.

While looking at this code, I am reminded that the IO subsystem is kind of messy. I’ve tried to clean it up in the past, and made a few small improvements over time. However, without a larger guiding vision to follow, I never really had a great idea of what kind of larger architectural changes to make to really bring this subsystem up out of the mud. After working on this branch, I finally had something like a flash of insight, and think I have a good idea about how to clean things up. This leads me to…

`whiteknight/io_cleanup1`

My idea is a relatively simple one: All our IO operations are controlled by the various PMC types (FileHandle, Socket, StringHandle, etc), but all our IO API functions are currently implemented as ugly (and brittle) switch statements to pick between execution pathways for these different types. A far better idea would be to separate out the different logic behind a virtual function dispatch table (vtable).

I’ve written up some proposed changes in the whiteknight/io_cleanup1 branch, and will start work if other people think it’s a decent idea.

The key points are as follows:

Move all FileHandle-specific logic into src/io/filehandle.c. Do the same for Pipe, Socket and StringHandle types.
Implement a new io_vtable type, which will contain a dispatch table for common operations. Each one of the files created in #1 above will implement the routines for one io_vtable and supporting logic.
Buffering will be refactored. Instead of the FileHandle PMC containing several attributes for buffering, we’ll instead use an io_buffer object to hold buffering details. An encapsulated buffering API will take this buffer structure and the relevant vtable and automatically perform buffering if necessary.
I am going to start separating out Pipe logic from FileHandle, though I’m not planning to create a separate type for it quite yet.

Once these things are done, I think the IO system will be much cleaner and much more hackable. This is lower priority right now until some of my ideas are vetted, but I’m glad I finally have a plan in mind after so many years of staring helplessly at this code.

`whiteknight/sprintf_cleanup`

The engine for our sprintf implementation is sort of old and messy. It’s some very functional and very stable code, but it needs to be brought up to date with our modern coding and organizational standards.

In the whiteknight/sprintf_cleanup branch I make several changes, most of which are entirely internal and should not affect users at all:

I move the files from ‘src/misc.c and src/spf_.ctosrc/string/sprintf.c and src/string/spf_.c respectively.
I’ve cleaned up some header-file nonsense and created a new src/string/spf_private.h header file to hold private data.
I’ve changed the code to use a StringBuilder instead of the older (and now-incorrect) repeated string concatenations. With immutable strings, each concat operation creates a new STRING instead of appending to the pre-allocated buffer, which is extremely wasteful. I haven’t benchmarked this change, but I suspect it has higher performance on longer, more complicated formats.
I’ve fixed a sub-optimal error message at request of benabik in ticket #759.

This branch is almost complete and I’ll probably merge it this weekend. Besides the text of the exception message, there are no visible user changes so it shouldn’t be controversial at all.

Parrot 4.4.0 Banana Fanna Fo Ferret

2012-05-17T00:00:00-07:00

Its existence guarantees nothing in itself, and the catalytic or Promethean moment only occurs when one individual is prepared to cease being the passive listener to such a voice and to become instead is spokesman, or representative.

But it’s important to remember the many dreary years when the prospect of victory appeared quite unattainable. On every day of those years, the “as if” pose had to be kept up, until its cumulative effect could be felt.

– Christopher Hitchens, Letters to a Young Contrarian

On behalf of the Parrot team, I’m proud to announce the 4.4.0 release of Parrot “Banana Fanna Fo Ferret”. Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 4.4.0 is available on Parrot’s FTP site, or by following the download instructions. For those who want to hack on Parrot or languages that run on top of Parrot, we recommend our organization page on GitHub, or you can go directly to the official Parrot Git repo on Github.

Parrot 4.4.0 News:

- Core
    + Most internal calls to libc exit(x) have been replaced with
      Parrot_x_* API calls or PARROT_FORCE_EXIT
- Documentation
    + 'pdd31_hll.pod' made stable in 'docs/pdds/'.
    + Updated main 'README' to 'README.pod'
    + Updated various dependencies, e.g., 'lib/Parrot/Distribution.pm'.
    + Updated all 'README' files to 'README.pod' files.
    + Added 'README.pod' files to top-level directories.
- Tests
    + Update various tests to pull from new 'README.pod'
    + Updated 't/tools/install/02-install_files.t' to pull from new
      'README.pod'
- Community
- Platforms
- Tools
    + pbc_merge has been fixed to deduplicate constant strings and
      merge annotations segments

Alvis Yardley (or a delegate) will release Parrot 4.5.0, the next scheduled monthly release, on June 16th 2012. Subsequent release managers are to be announced. A special thanks to our donors, contributors and volunteers for making this release possible.

Enjoy!

I haven’t been doing enough blogging lately! On Tuesday I put out the 4.4.0 release of Parrot, “Banana Fanna Fo Ferret”. I figured it was a fun play on words. I added a little quote from a favorite writer of mine, Christopher Hitchens. Much of his writings can be pretty inflamatory, but I picked two quotes that related to historical struggles for social progress, and which when read in a certain light (and dramatically out of context) make sense for Parrot too.

The release went off without a problem, and I’ve got a few branches waiting in the environs to be merged. I’m sure I’ll talk about some of those projects if I can get back into a normal blogging rhythm again.

XML Is Hard

2012-04-28T00:00:00-07:00

Last week I promoted the Parse and Json libraries in Rosella to stable status. For both those libraries I wrapped up a few outstanding TODO issues, wrote up some website documentation and added a bunch of unit tests. I figured I would do the same thing for the XML library too. After all I had done the hard part: the first 90% of the library was the recursive descent parser which I had most of.

So today I got to work on that library, trying to put together the last few bits so I could make the library stable. Like I said, I had about 90% of it done already. I spent the time today doing another 90%. I figure I only have about 90% left to go before I have a “real”, usable XML library. Somewhere a mathematician is reading this post and inventing new curse words, but nobody can hear him, because he has no friends.

It turns out that XML is hard.

Anybody can put together a little parser for XML-like tag syntax with attributes, text, and nested tags. That part is dirt simple, and I had that done in an hour or two. It’s once you start getting into DTD declarations and schema validation that things get messy. Honestly, I don’t think I can seriously call Rosella’s XML library “complete” without those things. Or, not without most of them. I can probably get away with only the first 90% or so.

So, what can Rosella’s Xml library do today? Here is a sample of XML text that I can parse into a document object tree without problems:

<?xml version="1.0"?>
<!DOCTYPE foo [
    <!ELEMENT foo (bar, baz)>
    <!ELEMENT bar ANY>
    <!ELEMENT baz (fie)>
    <!ELEMENT fie EMPTY>
    <!ATTLIST fie
                lol CDATA #REQUIRED
                wat CDATA #IMPLIED
                sux CDATA #FIXED "hello!">
]>
<foo>
    <bar/>
    <baz>
        <fie lol="laughing out loud" wat="you talkin bout?" sux="hello!"/>
    </baz>
</foo>

Or, if I want, I can jam all that schema nonsense into a separate file, and load it separately:

<!DOCTYPE foo SYSTEM "foo.dtd">

Although I haven’t integrated Rosella Net yet, to allow loading schemas from a URL. In code, I can do a few things:

var dx = new Rosella.Xml.Document();
dx.read_from_file("foo.xml");
dx.validate();
if (!dx.is_valid()) {
    for (string err in dx.errors)
        say(err);
}
dx.write_to_file("newfoo.xml");

var dtd = new Rosella.Xml.DtdDocument();
dtd.read_from_file("foo.dtd");
var errors = dtd.validate_xml(dx);
if (elements(errors) > 0) {
    for (string err in errors)
        say(err);
}
dtd.write_to_file("newfoo.dtd");

That example shows us loading an XML document from a file and validating it with it’s built-in rules from the !DOCTYPE header. The second part shows us loading a separate DTD definition from a standalone file, and using that to validate the XML document too. In both cases, the validator runs through the document object and returns a whole list of error messages, not just a simple yes/no flag. In both cases, we can also re-serialize the XML and DTD documents back to string and then to file.

So what is left to do? Well, for starters there’s a bunch of syntax in the !ELEMENT tag that I don’t quite handle yet, such as quantifiers and alternations:

<!ELEMENT foo (bar*, (baz|bar), fie?)>

Parsing all that in a way that doesn’t suck is not something I’m looking forward to doing.

Then in attribute lists, there’s some syntax I don’t deal with, such as enumerated values again:

<!ATTLIST foo bar (yes|no)>

The validator I’ve implemented is pretty naive so far, and isn’t set up to do quantifiers anyway. That’s all going to take a while to do. We’re doing some basic validation now, but nowhere near as much as we would expect from a full implementation.

And keep in mind, even when I’m done implementing (mostly) proper XML and DTD parsing, I could still go on to parse other schema languages like XSD which some applications might expect and even prefer. Maybe I could do something like XPath too, which would be very nice. I probably won’t try to do XSLT though: I’m still young and I would like to keep some of my sanity in reserve for my twilight years.

My Json library is about 1300 lines of winxed code long, including whitespace. My Xml library is about 2400 lines of code long and still growing. Json is pretty easy (by design!), but XML is very hard. I’m not going to push the Xml library to become stable any time soon, there’s a hell of a lot of work left on it and I’m not going to rush anything.

Various Updates

2012-04-25T00:00:00-07:00

Here are some updates on various projects I’ve been working on or been planning to work on:

ParrotStore

In my post introducing ParrotStore, I mentioned that I only had support for MySQL, Memcached, and a little bit of stuff working for MongoDB. In the past few days I’ve also added SQLite3 support. Now you can do this, after installing the prequisites, building and installing:

var sqlite3_lib = loadlib('sqlite3_group');
var sqlite3 = new 'SQLite3DbContext';
sqlite3.open("test.sqlite3");
sqlite3.query("INSERT INTO tbl1 (name, number) VALUES ('Andrew', 100)");
var result = sqlite3.query("SELECT * FROM tbl 1");
for (var row in result) {
    for (string colname in row)
        print(colname + "=" + string(row[colname]) + " ");
    say("");
}

SQLite3 offers a bunch of features that I don’t tap into yet, but we have a good start and can do some basic work with it already.

Also, I mentioned that we didn’t support queries with multiple result sets in the MySQL bindings. Well, now we do (and we do in SQLite3 too):

var result1 = mysql.query("CALL my_stored_proc");
var result2 = sqlite3.query("SELECT * FROM tbl1 ; SELECT * from tbl2");

If the query returns one result set, a DataTable object is returned. If it has multple result sets, an array of DataTables is returned instead.

Eval PMC

I went digging through my backlog of old branches last night and found my incomplete branch for removing the deprecated Eval PMC. After updating to current master I gave it a spin and most things looked good. I fixed all the core parrot tests and then moved on to the rest of the ecosystem.

Winxed works fine with the PackfileView PMC instead of the Eval PMC. I made a few of those updates in the past, so it mostly worked out of the gate. Rosella compiled and ran like a charm too.

NQP-rx works fine because it mostly relies on the PCT libraries that ship with Parrot, and which I had already fixed.

The new NQP is a little bit more of a hassle. It took me a little bit of effort to figure out the bootstrapping mechanism, but after a few hours of hacking I had NQP building on the new Parrot using PackfileView instead of Eval. However, one of the regex tests hangs indefinitely now and I’m having trouble tracking that down. this project may get bumped down to a lower priority level until I can either figure out what the problem with NQP is, or until I can enlist some help to fix it.

I would like to merge this branch as soon as NQP is fixed and I can prove that I can build it and Rakudo on the branch.

Sub Flags Cleanup

My remove_sub_flags branch, tasked with removing the old :load and :init flags from Parrot and replacing them with the new :tag() syntax is right where I left it a few weeks ago. I’m down to a relatively small list of test failures, the solution to most of which is to update the syntax in the tests themselves. A handful of tests such as those using the parrot-nqp and winxed compilers are failing because I need to update those compilers first to generate the correct code so the tests can run correctly.

After fixing NQP-rx and Winxed, I need to get started testing out the new NQP and Rakudo. I suspect both of those two things will be made to work without too much effort.

It turns out that the Eval PMC deprecation work overlaps with this slightly, so the things I change for that branch should help reduce failures in this branch too. After I get Eval deprecated and removed, I’ll come back to this branch and see where things stand.

This is such a large and disruptive change that I can’t imagine we would want a merge before the 4.4 release, even if I got all the bugs ironed out. We could be a month or more away from a merge, so I’m not listing this work as high priority.

PCC

Bacek has been doing a lot of refactoring in PCC land, trying to fix some slow and infelicitious aspects of it. I’ve gotten a set of new PCC-related opcodes added to core and have a few more that I want to add, including new variants of set_args, get_params and friends to take explicit context arguments instead of using magical behavior to try and find them automatically. A few patches to IMCC and the new behavior might go in without anybody noticing. I’ve talked more about this in past posts, and I’m sure I’ll have more to say when I start making changes.

Rosella

Rosella is mostly where I want it to be right now. I’m planning to change around the development cycle to stick to supported releases of Parrot and Winxed instead of tracking HEAD for both of them. I’m going to promote one or two more libraries to “stable” status and then put out a release sometime after Parrot 4.4 hits the news stands next month. I’ve already promoted the Parse and Json libraries to stable status. I will probably promote Xml and Net too, since I am pretty happy with both of those two libraries and feel that they are almost ready for general use.

After that, I suspect Rosella is going to take a back seat for a while, so I can focus on some other projects.

Google Summer of Code

GSOC is keeping me pretty busy so far. We accepted 4 projects this summer. The fifth project, which was to do some work on the Jaesop Stage 1 compiler, was lost because the student was accepted to a different organization instead. The four remaining projects are:

Security Sandbox by Justin
Mod_Parrot 2.0 by brrt
LAPACK Bindings by jashwanth
PACT Assembly by benabik

I think these projects will be very cool, and I am looking forward to see what kinds of great code they can produce this summer.

Green Threads

nine has been doing some amazing work on his threading branch. Yesterday he informed me that he had a solution to make green threads work on Windows, and had already implemented part of it. That’s awesome, because I was planning to work on porting the green threads to windows next, but if he’s doing it then I don’t have to.

Some of the performance numbers he’s been getting are pretty impressive for certain tasks. Some benchmarks he has are even showing a significant threading performance improvement over a similar benchmark written in perl5.

I’ve been doing some testing on his branch and things are looking mostly good except for one or two remaining GC-related bugs that need to be ironed out. After that, if we can get some concensus, I would love to start talking a merger shortly after 4.4.

6Model

With Green Threads possibly off my TO-DO list, Eval PMC Deprecation mostly wrapped up and remove_sub_flags on the back burner, I can start moving towards my next project: 6model. And I can do it much earlier than I was expecting. I’m going to mine benabik’s rejected 6model project proposal for some ideas, then I’m going to jump in and try to get things working. I suspect things could get moving pretty quickly, if I can keep my level of free time relatiely high.

Whiteknight's Blog

Entity Framework Code Only Migrations

Code-Only Migrations

Create Your DbContext and POCO Classes

Create a Configuration

Create a Migration

Run the Migrations (Some Recipes)

What’s My Use Case?

Working On a New Project

More IO Work?

New Job

September Status

Personal Status

io_cleanup1 Status

Threads Status

Other Status

io_cleanup1 Lands!

Parrot 4.7.0 "Hispaniolan" Released!

io_cleanup1 Done?

IO Cleanups Home Stretch

HTML with Rosella Xml and Net

Sockets and Encodings

Reading a Line of Text

IO Rewrite Status

IO Cleanups Status

IO Refactors

Destructors are Hard

and what still needs to be done.

recycle for destructors.

especially from our users.

Pending Branchwork

Already Merged

eval_pmc

remove_sub_flags

whiteknight/gc_finalize

whiteknight/gh_663

whiteknight/gh_610

whiteknight/io_cleanup1

whiteknight/sprintf_cleanup

Parrot 4.4.0 Banana Fanna Fo Ferret

XML Is Hard

Various Updates

ParrotStore

Eval PMC

Sub Flags Cleanup

PCC

Rosella

Google Summer of Code

Green Threads

6Model

`io_cleanup1` Status

`eval_pmc`

`remove_sub_flags`

`whiteknight/gc_finalize`

`whiteknight/gh_663`

`whiteknight/gh_610`

`whiteknight/io_cleanup1`

`whiteknight/sprintf_cleanup`