Ookii.org

Dark mode

Wed, 09 Aug 2023 00:37:26 Z

To modernize the appearance of my site a little bit, I've now added a dark mode!

It's triggered by your device's color scheme, so if you've enabled a dark theme on your PC, smartphone, tablet, etch-a-sketch, or whatever you're using to browse this site, you should see the dark version of this site. There is currently no way to trigger it manually, so a browser that supports this is required (fortunately, that's pretty much all modern browsers).

If you're not seeing it, you may have to force reload the stylesheet for this site. On PC, you can do this by pressing CTRL-F5 on this page. On mobile, the only way I'm aware of is to completely clear your browser's cache.

Ookii.CommandLine 4.0

Thu, 20 Jul 2023 20:07:08 Z

When I released version 3.0 of Ookii.CommandLine last year, I wasn't expecting to have another major release so soon. I still had some ideas for new features, like supporting automatic aliases based on prefixes of command line arguments, but they'd only warrant minor revisions.

But then, I started reading about .Net source generators, and I thought, that's something I could use. Ookii.CommandLine used reflection to determine the command line arguments at runtime, so a source generator would be able to do the same job at compile time. That would make it faster, make the library trim-friendly, and allow me to emit warnings and errors for unsupported argument settings, define positional arguments based on the order of the class members rather than explicit numbers, and extract default values from property initializers.

Ookii.CommandLine 4.0 now gives you that option. You can apply the GeneratedParserAttribute to any class that defines command line arguments, and the parser will be generated at compile time.

[GeneratedParser]
partial class MyArguments
{
    [CommandLineArgument(IsPositional = true)]
    [Description("A required positional argument.")]
    public required string Required { get; set; }

    [CommandLineArgument(IsPositional = true)]
    [Description("An optional positional argument.")]
    public int Optional { get; set; } = 42;

    [CommandLineArgument]
    [Description("An argument that can only be supplied by name.")]
    public DateTime Named { get; set; }

    [CommandLineArgument]
    [Description("A switch argument, which doesn't require a value.")]
    public bool Switch { get; set; }
}

That still might not have warranted a major version bump, if trimming hadn't been a goal. Ookii.CommandLine relied on the TypeConverter class to convert argument values from strings to their defined type, and I soon discovered that determining the right TypeConverter for any type at compile time wasn't really possible, and calling TypeDescriptor.GetConverter() at runtime was inherently not compatible with trimming an application. I guess this is why System.CommandLine limits their supported types to just a chosen handful, and everything else requires custom converters.

The solution was ArgumentConverter, a new way to convert argument values, that replaces TypeConverter. Not only is it trim-friendly, it also enables converting arguments using ReadOnlySpan<char>, which means that the parser can avoid allocating strings when splitting arguments provided like -Name:value, another performance improvement.

However, it does mean that any command line arguments classes that relied on a TypeConverter will no longer function correctly without modifications, so the major version needed to change again. While I was at it, I made a few other breaking changes, the biggest of which was probably the removal of constructor parameters as a way to define command line arguments. Now, properties and methods are your only choice. I was never a big fan of the way constructor parameters worked, and how clumsy it was to add attributes to them. They were already de-emphasized in the documentation, and were mostly left over from version 1.0 (sadly lost to time), where they were the only way to define positional arguments. The work needed to support them with source generation meant they ended up on the chopping block this time.

For these and other breaking changes, check the migration guide. As with version 3.0, most users probably won't need to make many changes.

Besides source generation and new argument converters, Ookii.CommandLine 4.0 gives you plenty of other goodness, including the aforementioned automatic prefix aliases, support for C# 11 required properties, a new attribute for value descriptions that allows for localization, canceling parsing with success and access the remaining arguments afterwards, support for nested subcommands, source link integration, and more.

See the change log for everything that's new.

Get it on NuGet or GitHub. You can also try out the parser or subcommands on .Net Fiddle.

Jumbo's JobBuilder: part 2

Tue, 07 Feb 2023 18:54:34 Z

This is fifth article about Jumbo.

Previously, we looked at how Jumbo required you to create your own job configurations, and how I tried to alleviate that with helpers for common job structures.

I wanted a way to make this easier, and while I didn't want to use a separate language like Hadoop's Pig did, I was inspired by its simple, imperative nature: you just specify a sequence of operations, and it turns that into a sequence of MapReduce jobs.

The JobBuilder would end up doing the same thing for Jumbo, but instead you used a sequence of method calls in ordinary C# code. These would be 'compiled' into a job configuration.

This had two key goals:

Let the JobBuilder make certain decisions about job structure, such as when to use a local aggregation step, or how many tasks to use in a channel-input stage.
Allow the user to create tasks using simple methods, instead of having to write a whole class.

The first was accomplished by having the JobBuilder define helper functions for certain operations, such as AccumulateRecords, that knew how to create the kind of job structure appropriate for that operation. That may not seem much better than the old AccumulatorTask approach, but it had one key advantage: you could now incorporate this operation in a larger job structure, rather than that structure having to be your entire job.

It also meant the JobBuilder could apply certain heuristics, such as: if there is only one input task, you don't need a second aggregation stage; everything can be done locally. It's this kind of logic that lets WordCount in the quick start guide run in a single task if the input is small.

The second goal had some complications. I didn't really want to give Jumbo Jet's TaskExecutionUtility (the thing responsible for most of the logic of how to run tasks) the ability to invoke arbitrary functions. It would break too much of the logic of how tasks were constructed, and would require a lot of bespoke code for different types of task methods. No, I wanted to stick to ITask<TInput, TOutput> at that level, which meant JobBuilder had to translate between the two.

That meant dynamically generating MSIL to create task classes that invoked the methods the user specified. These classes would be generated at job submission time, saved to a generated assembly, which the TaskExecutionUtility could just load like normal task type without being aware of any of this. It was a fun challenge to figure out how to do this, and it made it much easier to create custom tasks.

The only limitation is that, unfortunately, if the method you used for your task was not public static, I couldn't emit a direct call in the generated MSIL. In that case, I had to serialize the delegate you pass in, and during task execution, deserialize it and call the method through that, which is much slower. Unfortunately, this meant that using lambdas (which never generate a public method) was possible, but not ideal.

The first iteration of the JobBuilder meant you could now write code like this to create a job:

var builder = new JobBuilder();
var input = builder.CreateRecordReader<Utf8StringWritable>(_inputPath, typeof(LineRecordReader));
var collector = new RecordCollector<KeyValuePairWritable<Utf8StringWritable, Int32Writable>>(null, null, _combinerTasks == 0 ? null : (int?)_combinerTasks);
builder.ProcessRecords(input, collector.CreateRecordWriter(), WordCount);

var output = builder.CreateRecordWriter<KeyValuePairWritable<Utf8StringWritable, Int32Writable>>(_outputPath,
    typeof(TextRecordWriter<KeyValuePairWritable<Utf8StringWritable, Int32Writable>>), BlockSize, ReplicationFactor);

builder.AccumulateRecords(collector.CreateRecordReader(), output, WordCountAccumulator);

...

[AllowRecordReuse]
public static void WordCount(RecordReader<Utf8StringWritable> input, RecordWriter<KeyValuePairWritable<Utf8StringWritable, Int32Writable>> output) { ... }

[AllowRecordReuse]
public static void WordCountAccumulator(Utf8StringWritable key, Int32Writable value, Int32Writable newValue) { ... }

That was better: no more task classes, no more explicitly creating child stages, and the job was defined by basically two function calls: ProcessRecords, and AccumulateRecords. The JobBuilder also kept track of all the assemblies used by each record reader, writer, partitioner, and task you used, and made sure they, and all their non-framework dependencies, would be uploaded to the DFS.

There were two things I didn't like about this, though: all the explicit generic type parameters, and the fact that you have to define your input, output, and channels (through the RecordCollector class) explicitly before calling processing functions. It kind of meant you had to think about your stages backwards, because you had to define their output before you invoked their processing functions.

I wasn't quite satisfied with this. It still wasn't as easy as I wanted it to be. I tried to improve things by just adding even more specific helper methods, so you could write WordCount like this:

var input = new DfsInput(_inputPath, typeof(WordRecordReader));
var output = CreateDfsOutput(_outputPath, typeof(TextRecordWriter<Pair<Utf8String, int>>));

StageBuilder[] stages = builder.Count<Utf8String>(input, output);

((Channel)stages[0].Output).PartitionCount = _combinerTasks;
stages[0].StageId = "WordCountStage";
stages[1].StageId = "WordCountSumStage";

Which was kind of cheating (since I hadn't solved the problem, I just hid it), and the way you customized things like the stage IDs and task count wasn't exactly great.

Eventually, I had an idea that completely changed how the job builder worked. Instead of having to define channels and outputs up front, each operation could directly serve as the input of the next operation, and if that created a channel, the resulting object could be used to customize that channel as well as the stage itself.

This new JobBuilder still had helpers for specific types of operations that could use custom logic (such as adding the local aggregation step), as well as helpers for custom channel types (sorting), and for complex operations like joins. And of course, you could still use custom task classes or manually define child stages if none of the helpers sufficed, combining them seamlessly with the helpers.

Finally, you could specify every step in a straight-forward way:

var job = new JobBuilder();
var input = job.Read(InputPath, typeof(LineRecordReader));
var words = job.Map<Utf8String, Pair<Utf8String, int>>(input, MapWords);
var aggregated = job.GroupAggregate<Utf8String, int>(words, AggregateCounts);
job.Write(aggregated, OutputPath, typeof(TextRecordWriter<>));

This was the kind of simplicity I wanted. There were no more specific job helpers, and every step was explicit while still being easy to use. Even emulating MapReduce was now simple:

var job = new JobBuilder();
var input = job.Read(InputPath, typeof(LineRecordReader));
var mapped = job.Map<Utf8String, Pair<Utf8String, int>>(input, MapWords);
var sorted = job.SpillSortCombine<Utf8String, int>(mapped, ReduceWordCount);
var reduced = job.Reduce<Utf8String, int, Pair<Utf8String, int>>(sorted, ReduceWordCount);
job.Write(reduced, OutputPath, typeof(TextRecordWriter<>));

Here, SpillSortCombine configures the channel in between the map and reduce, without the user needing to know that's what it does. And despite it being an operation that has no stages of its own, you can just apply it directly to DFS input and output; the JobBuilder makes sure a valid job structure is created.

The only limitation I could never get away from is the need for explicit generic type arguments. I was able to make things a bit easier; everywhere you specify a type directly, whether record readers, writers, partitioners, or even tasks (if not using methods), you can specify the "open" generic type (like typeof(TextRecordWriter<>)) above, and the JobBuilder will instantiate it with the correct type for the records.

However, for the actual processing methods, where they use delegates, it was always necessary to specify both the input and output types explicitly, because C# isn't able to determine the type arguments from the arguments of a delegate target (only from the return type). LINQ doesn't suffer from this problem because the type of the input items is known, because you pass in an IEnumerable<T> or IQueryable<T>. Whereas the types of input, words, etc. above (all implementing IStageInput) are not generic, so can't be used to deduce the type.

The only improvement I made was shortening a bunch of type names. Utf8StringWritable became Utf8String, KeyValuePairWritable became Pair, and Int32Writable (and a bunch of similar wrappers) went away in favor of being able to directly use types like int.

So, why not make IStageInput generic? Now the delegates work, but you have to specify explicit types in other places, such as Read above, and when calling Process with a custom task class. There is no way to derive the record type for job.Read(InputPath, typeof(LineRecordReader)). I thought of using a constraint like this:

public IStageInput<TRecord> Read<TRecordReader, TRecord>(string inputPath)
    where TRecordReader : RecordReader<TRecord>

But, you still have to explicitly specify both generic arguments in that case.

Maybe I could do better now, if I gave this some more thought. I can't immediately think of anything that wouldn't have equal drawbacks, though. And, since I'm not developing Jumbo anymore, this is how it will stay.

Still, despite that little annoyance, I accomplished most of what I wanted with the final version of the JobBuilder. It feels like a pretty natural way of defining jobs, and even if you want to customize settings for stages and channels (for example, see advanced WordCount), it works pretty well

It's probably my favorite aspect of Jumbo, something that's really different from anything in Hadoop (at least at the time), and I'm pretty proud of it. And, before I released Jumbo, nobody really knew about it. Since it had no real value to my research, I never published about it. It was only mentioned on one page in my dissertation, but that was it.

That's why I wanted to release Jumbo. It's why I wanted to write these articles about it. Even if nobody cares, the work I did on this is at least preserved now. And that's worth something, even if only to me.

Jumbo's JobBuilder: part 1

Thu, 26 Jan 2023 00:13:38 Z

This is fourth article about Jumbo.

From the moment I started working on Jumbo, I knew I wanted two things: I wanted it to be more flexible than plain MapReduce, and I still wanted it to be easy to use.

While I didn't want to be limited to MapReduce, I also didn't just want to allow arbitrary task graphs, because I felt that was too complex for what I wanted to do. By abstracting MapReduce into a linear sequence of "stages," of which you could have an arbitrary number, I feel like I struck a good balance between allowing alternative job structures, while still not getting too complicated.

Add to that things like making channel operations (such as sorting) optional, and suddenly you could use hash-table aggregation, you could do in one job things that in MapReduce would've required several jobs, and by allowing multiple input stages, you could do things that were traditionally hard to emulate in MapReduce, such as joins.

Still, this added flexibility did come with some complexity. In Hadoop, you write a map function, a reduce function, set up a simple configuration specifying what they are, and you're done. With Jumbo, you had to put a little more thought into what kind of structure is right for your job, and creating a JobConfiguration was a bit more complex.

From the earliest versions of Jumbo Jet I wrote, the job configuration was always an XML file, and it was always a serialization of the JobConfiguration class. However, the structure of that format changed quite a bit. Originally, you needed to add each task individually, which meant that if you had an input file with a 1000 blocks, you needed to add a 1000 TaskConfiguration elements to the configuration. Not exactly scalable.

Eventually, I switched this to having a StageConfiguration instead, and letting the JobServer figure things out from there. But, you still needed to essentially build the job structure manually. At first, this still meant manually creating each stage and channel. Later, I added helpers, which meant that creating WordCount's configuration would look something like this:

var config = new JobConfiguration(new[] { Assembly.GetExecutingAssembly() });
var input = dfsClient.NameServer.GetFileSystemEntryInfo("/input");
var firstStage = config.AddInputStage("WordCount", input, typeof(WordCountTask), typeof(LineRecordReader));
var localAggregation = config.AddPointToPointStage("LocalAggregation", firstStage, typeof(WordCountAggregationTask), ChannelType.Pipeline, null, null);
var info = new InputStageInfo(localAggregation)
{
    ChannelType = ChannelType.File,
    PartitionerType = typeof(HashPartition<Pair<Utf8String, int>>);
};

config.AddStage("WordCountAggregation", localAggregation, typeof(WordCountAggregationTask), taskCount, info, "/wcoutput", typeof(TextRecordWriter));
var job = jetClient.JobServer.CreateJob();
jetClient.RunJob(job, config, dfsClient, new[] { Assembly.GetExecutingAssembly().Location });

Maybe it's not terrible, but it's not super friendly (and this was already better than it started out at). I tried to improve things by creating helpers for specific job types; a job like WordCount was an AccumulatorJob; a two-stage job with optional sorting (like MapReduce) was a BasicJob. This worked, but kind of left you hanging if you wanted a different job structure.

Some things got easier quickly; instead of manually using the JetClient class, I introduced the concept of a job runner, a special class use by JetShell. But defining the structure of the job stayed like this, unless you could use a helper for a predefined job structure, for a long time.

One of my first ideas for how to make creating jobs easier was to adapt LINQ (which was pretty new at the time). I even thought that something like that could have potential for publication. Unfortunately, Microsoft itself beat me to the punch by publishing a paper on DryadLINQ, so that was no longer an option.

After that, I no longer saw this as something I could publish a paper about, but just for my own gratification I still wanted a better way to create jobs.

I thought about alternative approaches. I could still do LINQ, but it would be complicated, and without a research purpose I didn't want to invest that kind of time. Hadoop had its own methods; while single jobs were easy, complex processing that required multiple jobs or joins was still hard even there, and the best solution at the time was Pig, which used a custom programming language, Pig Latin, for creating jobs. I didn't much like that either, since you'd often have to combine both Pig Latin and Java, and that didn't seem like a good option to me.

No, I wanted something that kept you in C#, and the solution I came up with was the JobBuilder, which we'll discuss next time.

Ookii.CommandLine 3.0

Thu, 01 Dec 2022 22:28:09 Z

When I released Ookii.CommandLine for C++, I realized I had quite a backlog of things I wanted to update in Ookii.CommandLine, not just for the C++ version, but for the .Net version as well.

The result is the release of Ookii.CommandLine 3.0 for .Net. This is the biggest release of Ookii.CommandLine yet, with many new features, including support for an additional, more POSIX-like argument syntax, argument validation and dependencies, automatic name transformations, an updated subcommand API, usage help color output, more powerful customization, and more.

Seriously, there's a lot. Even version 2.0, which was a substantial rewrite from the original, wasn't anywhere near as big. Unfortunately, that does mean there's some breaking changes, but I expect most users won't need to make too many changes.

One question you might have is, when is all this new stuff coming to the C++ version? Unfortunately, I don't have a good answer. I'll probably add at least some of the new features to the C++ version, but it probably won't be all at once, and I'm not going to give a timeline either. If there's any feature you want in particular, you should file an issue for it.

Get it on NuGet or GitHub. You can also try it out on .Net Fiddle, or try out subcommands.

Jumbo and .Net Remoting

Thu, 24 Nov 2022 01:11:37 Z

This is the third article about Jumbo.

One of the things I knew I would need for Jumbo was a way for clients and servers to communicate. Clients would have to communicate with the servers, but servers also had to communicate with each other, mostly through heartbeats.

In some cases, writing a custom TCP server and protocol made sense, like client communication with the data server where large amounts of data had to be processed. But for most things, it was just a matter of invoking some function on the target server, like when creating a file on the NameServer or submitting a job to the JobServer. I figured it would be overkill to create completely custom protocols for this. In fact, being able to easily do this was one of the reasons I picked .Net.

WCF existed at the time, but I wasn't very familiar with it, and I'm not sure if Mono supported it back then. So, I decided to use good, old-fashioned .Net Remoting. All I had to do was make some interfaces, like INameServerClientProtocol, inherit from MarshalByRefObject, set up some minimal configuration, and I was in business. This seemed to work great, both on .Net and Mono.

All was well, until I started scaling up on Linux clusters. Suddenly, I was faced with extremely long delays in remoting calls. This was particularly noticeable with heartbeats, which sometimes ended up taking tens of seconds to complete. Now, I wasn't running thousands of nodes, where scaling issues might be expected; I was running maybe forty at the most. A remoting call that takes a few milliseconds to complete shouldn't suddenly take 10+ seconds in that environment.

It took a lot of digging, but I eventually found the cause: it was a bug in Mono. Mono's remoting TCP server channel implementation used a thread pool to dispatch requests. Now, the .Net BCL has a built-in thread pool class, but remoting didn't use that. It used its own RemotingThreadPool, which was used nowhere else.

This thread pool had a fatal flaw. Once it reached a specific number of active threads, on the next request it would wait 500ms for an existing thread to become available, before creating a new one.

That's already not great. Worse, it did this synchronously on the thread that accepts new connections! Which meant that these delays would stack! If a connection attempt comes in, and one is already waiting, it can't accept this connection until the waiting one is done. And if that one then also hits the limit...

Basically, if you get 10 new connection attempts when the pool is already at the limit, the last one will end up waiting 5 seconds, instead of 500ms. This was the cause of my scalability problems, with a few dozen nodes all trying to send heartbeats to the single NameServer and JobServer.

I reported this bug, but it never got fixed. In fact, this problem is still in the code today, at least as of this writing.

I created my own patched version of Mono, which worked around the issue (I think I just removed the delay). But, needing a custom-built Mono wasn't great for the usability of the project. I eventually ended up writing my own RPC implementation, using Begin/End asynchronous methods, which performed better. Still, I refused to merge this into the main branch (trunk, since this was SVN), still waiting for a Mono fix that would never come.

Eventually, I did switch to using the custom RPC implementation permanently, because if I ever did want to release Jumbo, requiring users to patch and compile Mono wasn't really an option. And, it was probably for the better, since .Net Core no longer has remoting, so I would've needed a different solution anyway when making the port.

And yes, just like .Net Remoting, my custom RPC mechanism depends on BinaryFormatter, because it works much the same. And BinaryFormatter is deprecated and insecure. Since I have no interest in further developing Jumbo, that will remain that way, so do consider that if you run Jumbo on any of your machines.

Ookii.CommandLine for C++

Sun, 23 Oct 2022 23:33:37 Z

Today I'm releasing a library that I've had for a long time, but never released: Ookii.CommandLine is now available for C++. It offers the same functionality as its .Net counterpart, but with an API that's suitable for C++.

I've had this for a while, actually, but it was interwoven with another project that contains a lot of pieces that aren't really meant for release. I recently updated it to match Ookii.CommandLine 2.4 functionality, and figured, why not go the extra mile and move it into its own repository.

So I did, and that repository is now available on GitHub. The library is header only, so it can be easily included in any project, as long as you use a compiler that supports C++20.

A tale of two Jumbos

Mon, 17 Oct 2022 22:54:33 Z

This is the second article about Jumbo.

As I said previously, Jumbo originally ran on Mono and the .Net Framework. Part of my hesitation in releasing this project was because of how complicated it could be to get it running on that environment.

When .Net Core started getting more mature with the release of .Net Core 3, I was interested in learning more about it. While I don't use .Net in my day-to-day anymore, I still like it, and .Net Core's promise of cross-platform .Net would've saved me so much hassle back in the day. So, I was curious to see whether I could get Jumbo running on it, just for fun.

Porting was, fortunately, not that difficult. To make things easier on myself, I started in a clean workspace, and copied over Jumbo's components one at a time, created SDK-style projects for them, and fixing build errors until they compiled. Some changes were necessary, but it wasn't too bad.

Changes made for .Net Core

One early difference I ran into was how sockets worked. On Linux, if you listen on an IPv6 address, you automatically also listen on the corresponding IPv4 address. If you subsequently also tried to bind to the IPv4 address, it would fail. Jumbo handled this by only listening on IPv6 on Linux, but on both IPv6 and IPv4 on Windows. This worked great for Mono.

But, I guess .Net Core handled this differently, since it no longer listens on IPv4 automatically on Linux, so I had to change how Jumbo behaved. Fortunately, that only meant changing the default for the associated setting in Jumbo's configuration files.

One of the biggest issues I ran into was the lack of the AssemblyBuilder.Save method. This method, needed to save dynamically generated assemblies to disk (which Jumbo's JobBuilder depends on), doesn't exist in .Net Core (and still doesn't in .Net 6). Fortunately, I was able to find a NuGet package that provided the same functionality, Lokad.ILPack, which saved the day. Unfortunately, it had a bug that prevented it from working for my scenario, but I was able to find and contribute a fix.

Another minor issue was the lack of AppDomains. Jumbo's TaskServer normally runs tasks in an external process, but it could run tasks in an AppDomain if requested, which was used to ease debugging and testing. Since AppDomains don't exist anymore, I had to cut this feature. But, since it isn't used when running Jumbo normally, it wasn't a big loss. The biggest difference is that it does make running the unit tests a bit more fragile, since they now depend on the external TaskHost process.

Then, I discovered that .Net Core can't serialize delegates, but I was able to work around that too. With that, I finally got all the tests working. Not only that; I could run them on Linux now, which was never possible before (old Jumbo's tests depended on NUnit and only ran on Windows).

Another interesting part was the CRC library. Jumbo's DFS calculates checksums a lot, so original Jumbo used a native library, for a faster implementation than the managed version. That library was built as part of Jumbo, either with Visual C++ (on Windows), or g++ (on Linux). Integrating that native build into the .Net Core build process turned out to be a hassle.

I did some speed tests to find out if this would still matter, and tried to optimize the managed fallback algorithm a bit more. And then, I found a CRC32 library for .Net Core that was even faster than my original native implementation, so that solved that problem.

Oh, and Ookii.CommandLine 2.3's inclusion of a .Net Standard 2.0 binary? That's because I needed it for this Jumbo port! Otherwise, I'm not sure when I would've gotten around to that.

Finally, I just had to remove some of the Mono-specific parts from the code (there are probably still some leftovers), and it was good to go. Except...

DfsWeb and JetWeb

The least straight-forward part of the port was, without a doubt, the two administration websites, DfsWeb and JetWeb. These were written using ASP.NET Web Forms (with code-behind files and everything), which just wasn't supported in .Net Core. Not wanting to give up now, I persevered, and rewrote both projects to use ASP.NET Core, replacing the Web Forms with Razor pages. Fortunately, neither site was particularly big, so there were only a few pages to convert.

This also had a big advantage: to use the websites, previously you needed IIS Express (on Windows) or Mono XSP. Now, they could just run using Kestrel, the web server included with ASP.NET Core.

Scripts

Jumbo was meant to run on a Linux cluster, and used a bunch of Bash scripts for deployment and running. These scripts used password-less SSH to start everything on all the nodes in the cluster, and were tailored only for the Linux environment. Running it on Windows, only used for testing purposes, was a mostly manual affair.

When I originally started planning to release Jumbo, back in 2013, I realized that maybe some people would want to be able to try out Jumbo, if only in a one-node configuration, on Windows, without having to manually launch multiple executables. I wrote some PowerShell scripts to facilitate that, but of course, those were for Windows only, so now there were two sets of scripts for basically the same thing.

With the port to .Net Core, there was something else interesting I could use: PowerShell Core! Now, I could adapt the PowerShell scripts to work on both Windows and Linux, and get rid of the old Bash scripts, rather than having to maintain two separate, functionally identical sets of scripts. PowerShell Core even supported remote sessions over SSH now, so could be used to launch Jumbo on multiple nodes in a Linux cluster.

Come to think of it, there's really nothing to stop you from running a hybrid cluster with both Windows and Linux nodes. I've never tried doing that. It could be interesting; it would probably be somewhat difficult to get the configuration right, since e.g. the scripts assume Jumbo is in the same location on every node, and the Windows paths need a drive letter. But it's probably possible to hack something together.

Anyway, that's the story of how Jumbo was brought into the modern age, and is no longer a pain to run on either Windows or Linux. I eventually brought it forward to .Net 6 as well (which was very straight-forward at that point), which is what's available now.

Check out both Jumbo's .Net Core port and the original .Net/Mono version on GitHub, and try to spot the differences!

Jumbo and .Net/Mono

Sat, 08 Oct 2022 20:35:16 Z

This is the first in a series of articles about Jumbo.

The recently released Jumbo comes in two versions: a modern version, and the original. That original version was written in C#, primarily targeting Mono, supporting the official Microsoft .Net Framework mainly for development and testing.

Because it was dependent on rather old versions of Mono (I never tested it on newer versions), and support for running on Windows was limited, it became harder to run it over time. Which is partially why I eventually decided to port it to .Net Core.

But, why is Jumbo written in .Net anyway? Hadoop is written in Java, so if I wanted to learn more about Hadoop, wouldn't it make sense for me to use Java too? Perhaps, but I wasn't super familiar with Java and its tooling, and I also felt it would steer me too much to be exactly like Hadoop, which I didn't want either.

So, I felt I had basically two choices given my skills at the time: C++ or .Net, both of which I was very proficient in.

C++ would've given me performance. But, it also would've increased the amount of work. I knew there were a bunch of things I would need to solve that aren't straight-forward with C++: reading configuration, RPC across a network, dynamically loading code to run jobs, text processing, robust networking, and more. All of those would require me to either find libraries, or roll my own. Possible, but time consuming.

Also keep in mind that this was 2008. C++11 was still called C++0x and not finished yet, with basically no compiler support. A lot of the niceties of modern C++ weren't there yet. And the ecosystem for finding, building, and using libraries in C++ wasn't exactly friendly either, especially if you wanted to work cross-platform.

And I did want to be cross-platform: I had to run this thing on university-owned clusters, which ran Linux, but I was doing my development on a Windows machine. And this was long before WSL made such a thing easy to do; my best options at the time were VMs or Cygwin. The idea of using Visual Studio for Linux development would've sounded ludicrous at the time, and VSCode wasn't even a flicker in anyone's imagination yet.

.Net would be slower, but, I reasoned, probably not any worse than Java. And running it on Linux would be possible with Mono, I believed. It would give me most of what I needed for free (like remoting and reflection), and I was kind of interested to see how some of .Net's features over Java (such as real generics) would alter the design.

My next choice would've been to use Rust, but unfortunately that hadn't been invented yet.

Yeah, I guess part of the point of this post is that this whole endeavour would've been much easier with modern tools. WSL, .Net Core, VSCode, maybe even Rust... it would've been interesting, to be sure. I also used Subversion for source control; git was around, but not so prevalent yet. Only much later did I move the repository over to git.

Anyway, given my choices at the time, I picked .Net. I think it was a good choice, as it allowed me to develop something working reasonably quickly. It also brought its challenges, in particular when dealing with Mono.

At first, I just started in Visual Studio and tried to get some basic DFS parts running before worrying too much about Linux. When I did try to run in on Linux with Mono, it worked surprisingly well.

Building it on Mono was another matter, however. I'm not 100% sure, but I seem to recall that originally, the Mono C# compiler was lacking some features that prevented it. Even after it became possible, I had to keep a separate build system; the Visual Studio (MSBuild) project files were used on Windows, and for Mono I settled on NAnt. I had to keep the two in sync manually. Before that, I would just build on Windows, and run those binaries on Linux.

Mono also limited me in what features I could use. .Net Framework 4.0 was in beta at the time, but I couldn't adopt any fancy new features (like LINQ) until those were supported on Mono, which took a while. Until then, I was basically stuck with mostly .Net Framework 2.0, as most of the 3.x features weren't super relevant for what I was doing.

One of the biggest hurdles with Mono turned out to be Garbage Collection, as Mono's GC at the time was much less advanced than the .Net Framework (and presumably Java's) one. Mono's GC was non-generational, and used a stop-the-world approach (all threads are paused during collection). It did eventually get a better GC, called sgen, but it wasn't stable in time for me to benefit. Meanwhile, the old GC was often taking more than 10% of task execution times, which necessitated the development of record reuse.

Another big challenge was some problems with .Net Remoting, which I'll cover separately in a future post.

Still, Mono was what made this whole thing possible. Without it, I wouldn't have been able to use .Net at all, and any other choice for me at the time would've made development more complicated, slower, and probably less fun. Big thanks to Miguel de Icaza and others who contributed to Mono.

Jumbo

Tue, 20 Sep 2022 23:54:43 Z

Today, I'm releasing something that I've wanted to release for a very long time. It's a project that I worked on during my Ph.D., and while I don't think it'll be terribly useful to anyone, a lot of work went into it that I want to preserve, even if just for myself.

That project is Jumbo, and it's now availabe on GitHub in two flavors: Jumbo for .Net 6+, and the original for .Net Framework and Mono. If you want to play around with it or learn more about it, you probably want the former.

Jumbo is an experimental large-scale distributed data processing system, inspired by MapReduce and in particular Hadoop 1.0. Jumbo was created as a way for me to learn about these systems, and should be treated as such. It's not production quality code, and you probably shouldn't entrust important data to it.

Basically, back when I was getting started with my Ph.D. in 2008, I found myself staring at the code of Hadoop (which wasn't even at version 1.0 yet at the time), and finding I wasn't really getting a good feel of how the whole thing fit together, and what really goes into designing a system like that.

So, some people at my lab suggested I should try building something for myself, which I did. I built, from the ground up, a distributed file system and data processing system, which is Jumbo. It was heavily inspired by Hadoop, and definitely borrows from its design (although no actual code was borrowed). In some aspects, I deviate from Hadoop quite a lot (especially since Jumbo isn't constrained to only using MapReduce).

Building Jumbo taught me a lot: about software design, about distributed processing, about decisions that affect scalability, and more. It's my hope that maybe, someone else interested in these topics might want to look at it and find what I did interesting. If nothing else, I just want to preserve this massive project that I did (still the biggest project I've done where I'm the sole contributor), and have its history available.

I did end up using Jumbo for some research efforts, which you can read about in a few papers as well as my dissertation under the University section of my site.

Jumbo is also the origin of one of my most widely used libraries, Ookii.CommandLine, so it's significant in that respect as well.

Like I said, I've wanted to release Jumbo for a long time. If you look through the original project's commit history you can see a bunch of work done in early 2013 (as I was nearing the end of my Ph.D.) like cleaning stuff up and adding documentation, but I never quite reached a level where I was comfortable doing so. The project, which primarily targeted Mono to run on Linux, wasn't that easy to set up and run.

In 2019, I ported the project to .Net Core, just to see if I could. That version was easier to play around with, and I wanted to release it then too, but I never quite got around to finishing it, until now.

So now, you can look at Jumbo and play around with it on .Net 6+, thanks to this new version. I've also expanded the documentation significantly, so it should be easy to get started and to learn more about how it works. The original Jumbo project for Mono and .Net Framework is only provided to preserve the original history of the project (the new repository only contains the history of the port). You probably shouldn't try and run it (though I obviously can't stop you).

If you want to comment on Jumbo or ask any questions, please use the discussions page on GitHub.

Additional posts about Jumbo

Ookii.FormatC 2.3

Wed, 14 Sep 2022 05:40:49 Z

After I recently released an updated version of Ookii.CommandLine, I figured Ookii.FormatC could also use some love.

This version comes with an optional new dark mode stylesheet, nullable reference types enabled for the library, the ability to write directly to a TextWriter, C# 10.0 keyword support, and a few minor other features and fixes.

Thanks to the ability to write to a TextWriter, you can now do stuff like this:

var formatter = new CodeFormatter()
{
    FormattingInfo = new CSharpFormattingInfo()
};

formatter.FormatCode(SampleCode, Console.Out);

Okay, writing HTML to the console is maybe not the most useful example, but you get the idea.

You can try it out with .Net Fiddle, or look at a sample that also shows the new dark mode in action. The online syntax highlighter has also been updated, and now supports PSParser based PowerShell formatting again.

And yeah, the NuGet package is version 2.3.1, rather than 2.3.0. That's because somehow the package for 2.3.0 ended up with an outdated binary in it. Not sure how that's possible, but it happened.

Ookii.CommandLine 2.4

Tue, 06 Sep 2022 03:05:10 Z

I've released an update to Ookii.CommandLine, my library for parsing command line arguments for .Net.

This new version comes with nullable reference type support (for .Net 6+), a new helper to make parsing easier, more customizability, an easier way to make -Help style arguments, and some bug fixes.

See the full list of changes here.

With the new helper method, you can now just do the following to parse the arguments and write errors and usage to the console if parsing failed:

var parsed = CommandLineParser.Parse<MyArguments>(args);

And if you want to customize parsing behavior, you can still do so with this method:

var options = new ParseOptions()
{
    NameValueSeparator = '='
};

var parsed = CommandLineParser.Parse<MyArguments>(args, options);

Of course, existing code to parse arguments that manually creates an instance of CommandLineParser will continue to work.

Check it out on NuGet or GitHub, or try it out online!

Also, the Visual Studio code snippets (which previously required manual installation) are now available on the Visual Studio marketplace.

Some small site updates

Thu, 25 Aug 2022 19:56:36 Z

I know I don't exactly do much with this site anymore, but I did recently make some small changes.

Primarily, I've reorganized some of the outdated content (like the stuff related to Channel 9) so it's less prominent. I've also updated the information for Ookii.Dialogs to link to the forked project on GitHub. My own version isn't kept up to date, so most people looking for Ookii.Dialogs should probably be using that version instead (the people who are doing this are awesome, by the way, for keeping this project alive).

Oh, and the site now supports https, thanks to Azure's free SSL/TLS certificates.

A new home for ookii.org

Fri, 30 Jun 2017 00:17:59 Z

Ookii.org now has a new home! It's the same site as always, except now it's hosted on Microsoft Azure. This shouldn't make any difference for you, and it probably won't mean this blog will get any more active, but all the existing content is still there.

Comments are now closed

Sat, 27 Sep 2014 20:35:00 Z

Due to the very large volume of spam comments this site receives despite efforts on my part to block them (recaptcha is apparently completely useless), I'm unfortunately forced to disable comments on all posts on this site.

If you wish to contact me about anything pertaining to this site, please use Twitter instead.

I apologize for the inconvenience.

.Net Configuration Section Documentation Generator

Sat, 08 Jun 2013 10:42:49 Z

Today I'm releasing another small tool that I created for personal use that I believe others might find of use. The tool is the .Net Configuration Section Documentation Generator (more proof that I should not be put in charge of naming things), which does what it says on the tin: it generates documentation for configuration sections defined using .Net's ConfigurationSection class.

Basically, it can generate an XSD schema from a ConfigurationSection, to which you can then add annotations to document the elements and attributes of the section, and then generate an HTML documentation file from that schema. The tool can be used with Microsoft .Net and Mono.

As an example, check out the schema for the documentation generator's own configuration section as generated by the documentation generator, and with added annotations. And finally, the documentation file generated from the annotated schema. This is only a very trivial configuration section, but it gives you an idea of what the output of the documentation generator looks like.

More information and downloads here.

Creating a route for a .asmx Web Service with ASP.NET routing

Thu, 30 May 2013 15:56:44 Z

While working on the update to my site (which is now completely finished), I came across a minor problem: I had an old ASP.NET Web Service (you know, the .asmx type, predating even WCF) that I needed to preserve at the same URL for compatibility purposes.

However, that URL was inside a folder on the old site. And in the new ASP.NET MVC site, that folder didn't exist anymore; instead, requests for that path were handled by an MVC Controller. This meant that it was not possible to leave the .asmx file at its original location, because an existing physical directory pre-empts the MVC route for the controller, so it would render all the other pages with the same base URL unviewable.

I figured I'd be able to use RouteCollection.MapPageRoute to redirect the old URL to the .asmx file's new location, but it turned out that this function only works for pages (.aspx files), as it requires that the HTTP handler for the route's target derives from System.Web.Page. Searching the web wasn't able to find a solution for this, but some investigation into the inner workings of ASP.NET routing let me devise my own solution, which I thought I'd share here.

The trick, it turns out, is to create a custom IRouteHandler, one that is able to return the correct IHttpHandler for web services. This turned out to be fairly simple:

using System;
using System.Web;
using System.Web.Routing;
using System.Web.Services.Protocols;

public class ServiceRouteHandler : IRouteHandler
{
    private readonly string _virtualPath;
    private readonly WebServiceHandlerFactory _handlerFactory = new WebServiceHandlerFactory();

    public ServiceRouteHandler(string virtualPath)
    {
        if( virtualPath == null )
            throw new ArgumentNullException("virtualPath");
        if( !virtualPath.StartsWith("~/") )
            throw new ArgumentException("Virtual path must start with ~/", "virtualPath");
        _virtualPath = virtualPath;
    }

    public IHttpHandler GetHttpHandler(RequestContext requestContext)
    {
        // Note: can't pass requestContext.HttpContext as the first parameter because that's
        // type HttpContextBase, while GetHandler wants HttpContext.
        return _handlerFactory.GetHandler(HttpContext.Current, requestContext.HttpContext.Request.HttpMethod, _virtualPath, requestContext.HttpContext.Server.MapPath(_virtualPath));
    }
}

This route handler can then be used to add a route for a .asmx web service:

routes.Add("RouteName", new Route("path/to/your/service", new RouteValueDictionary() { { "controller", null }, { "action", null } }, new ServiceRouteHandler("~/actualservice.asmx")));

One important thing to note is the RouteValueDictionary values I'm passing to the route. If you don't add values for "controller" and "action" to the dictionary, the Html.ActionLink helper method will match this route for any controller or action, leading to completely incorrect URLs if this is the first matching route. Since you probably want to have this route before the MVC default route (which was the case for me), that would be a problem. Setting the values in the route value dictionary resolves this issue.

Of course, you can create your own extension method to make this easier:

public static Route MapServiceRoute(this RouteCollection routes, string routeName, string url, string virtualPath)
{
    if( routes == null )
        throw new ArgumentNullException("routes");
    Route route = new Route(url, new RouteValueDictionary() { { "controller", null }, { "action", null } }, new ServiceRouteHandler(virtualPath));
    routes.Add(routeName, route);
    return route;
}

Which means you can now simply use this:

routes.MapServiceRoute("RouteName", "path/to/your/service", "~/actualservice.asmx");

And voilà, I was now able to preserve the URL to my legacy web service without needing to have a physical file matching that URL. If you're faced with a similar situation, I hope this helps you.

Welcome to the new Ookii.org

Tue, 07 May 2013 19:50:44 Z

As you can see, Ookii.org has gotten a revamp. Now you may wonder, why make such a radical update to a site when the content doesn't change that often? The main reason is that I wanted to get my hands dirty with some modern web technology.

While web development has never been particularly important to me, it was something I knew how to do and I kept somewhat up-to-date with. This dates back to making the site for my high school (at the time in Frontpage). However, as I'd been busy with my Ph.D. in Japan, I had very little time to keep up to date with stuff I didn't have any need to use in my research. Now that I've finished and haven't started my new job yet, it was the ideal time to brush up a little.

Back when I started this site, I wrote it in ASP.NET 2.0. I hand-rolled an AJAX library, because jQuery was years from being invented and actually AJAX support in websites wasn't all that common yet. The AJAX parts of the site used web services that returned data in XML, because no one had thought to start using JSON yet either. "Mobile browsers" at the time meant IE for Windows Mobile, which didn't support script at all and was barely on par with IE6 on the desktop otherwise. And speaking of IE6, supporting that was still very much a priority at the time.

But things have changed, and the goal was to re-familiarize myself with the state of the art. So this site is written in ASP.NET MVC 4, using the Entity Framework for data access. On the client, it uses HTML5 and CSS3, with scripts utilizing jQuery and a few related libraries. The site now has a responsive layout thanks to Zurb Foundation (try it: resize your browser window and watch the layout change), which has the side-effect of making it much more friendly for mobile devices. I've also used Knockout.

I have to say I was surprised at how much web programming has changed. ASP.NET MVC is light-years beyond the old web forms. Not only are the frameworks much better and the HTML/CSS much cleaner (at least, as long as you only target modern browsers), the tooling (in the form of Visual Studio 2012 in my case) also has improved tremendously.

Besides just redesigning the layout, I've also made a new front page that de-emphasizes the blog a little (since I don't update it that often), and instead provides a few more useful links as well as news about my Let's Plays from the associated Twitter account.

Note that not everything uses the new layout. Currently, only the blog has been updated. I intend to move other sections of the site to the new layout over time.

Let me know what you think, or if you find something that's broken.

Ookii.CommandLine 2.2

Wed, 06 Feb 2013 12:11:56 Z

Yes, I realise this blog is all but dead. However, I will soon finally complete my Ph.D. at the University of Tokyo (so, yay me!), and as a result I can now finally release a few things that had been put on hold while I was writing my Ph.D. thesis. In addition, I will probably release a few projects related to my research that may hopefully be of interest to some people.

First up is a small update to Ookii.CommandLine. This update adds command line argument aliases, some updates for usage help generation, and code snippets to help with creating argument classes.

In addition, I used NuGet to create a symbols package and published it on SymbolSource, which you should apparently be able to use as a symbol server in Visual Studio without manually downloading the symbols.

Get the release from CodePlex or using NuGet.

Floppy support in Windows Virtual PC 7

Sun, 17 Oct 2010 06:20:03 Z

Windows Virtual PC 7 is the successor to Virtual PC 2007, and the only version of Virtual PC that can run on Windows 7. It's freely available to anyone, and if you have Windows 7 Professional or Ultimate, you can also download Windows XP Mode, a virtual machine of Windows XP that allows you to run applications that may not otherwise work on Windows 7.

It seems however that Microsoft has removed the UI for using floppy disks with Windows Virtual PC 7. It was there in Virtual PC 2007, and now it's not. However, here's the thing: they only removed the UI, not the functionality itself.

The functionality for using floppy disks is still available in Windows Virtual PC 7, but it's only accessible via the COM object, which is not very convenient. Fortunately, Windows 7 also includes PowerShell, a very powerful shell scripting environment, that allows us to make things a bit easier.

Below, I present three PowerShell scripts: one to create virtual floppy disk images, one to attach floppy disks to a VM, and one to release them. The latter two can be used with both virtual floppy disk images, or a real floppy drive on the host computer.

You can download all three scripts here. Or you can just copy/paste them into a file, whichever you prefer. Note that you'll need to configure PowerShell to allow unsigned scripts to run, or sign the scripts.

Anyway, here are the scripts:

Create-VirtualFloppy.ps1:

param(
    [parameter(Position=0, Mandatory=$true)][string]$Path
)

$vpc = New-Object -ComObject "VirtualPC.Application"
if( -not [System.IO.Path]::IsPathRooted($Path) )
{
    # CreateFloppyDiskImage requires an absolute path
    [Environment]::CurrentDirectory = (Get-Location -PSProvider FileSystem).ProviderPath
    $Path = [System.IO.Path]::GetFullPath($Path)
}
$vpc.CreateFloppyDiskImage($Path, 2)

Attach-VirtualFloppy.ps1:

param(
  [parameter(Position=0, Mandatory=$true)][string]$VirtualMachineName,
  [parameter(Position=1, Mandatory=$true)][string]$FloppyPath,
  [parameter(Mandatory=$false)][Switch]$HostDrive
)

$vpc = New-Object -ComObject "VirtualPC.Application"
$vm = $vpc.FindVirtualMachine($VirtualMachineName)

if( $vm -eq $null )
{
    Write-Error "Virtual machine $vm not found."
}
else
{
    if( $HostDrive )
    {
        $vm.FloppyDrives.Item(1).AttachHostDrive($FloppyPath)
    }
    else
    {
        if( -not [System.IO.Path]::IsPathRooted($FloppyPath) )
        {
            # AttachImage requires an absolute path
            [Environment]::CurrentDirectory = (Get-Location -PSProvider FileSystem).ProviderPath
            $FloppyPath = [System.IO.Path]::GetFullPath($FloppyPath)
        }
        $vm.FloppyDrives.Item(1).AttachImage($FloppyPath)
    }
}

Release-VirtualFloppy.ps1:

param(
  [parameter(Position=0, Mandatory=$true)][string]$VirtualMachineName
)

$vpc = New-Object -ComObject "VirtualPC.Application"
$vm = $vpc.FindVirtualMachine($VirtualMachineName)

if( $vm -eq $null )
{
    Write-Error "Virtual machine not found."
}
else
{
    if( $vm.FloppyDrives.Item(1).ImageFile -ne $null )
    {
        $vm.FloppyDrives.Item(1).ReleaseImage()
    }
    elseif( $vm.FloppyDrives.Item(1).HostDriveLetter -ne $null )
    {
        $vm.FloppyDrives.Item(1).ReleaseHostDrive()
    }
    else
    {
        Write-Warning "No virtual floppy is attached to the virtual machine $VirtualMachineName."
    }
}

Using the scripts is pretty simple. Run PowerShell, navigate to the directory where you put the scripts, and invoke them. Remember that you need to enable execution of unsigned scripts (Set-ExecutionPolicy RemoteSigned), or sign the scripts.

The following example creates a virtual floppy drive image called sample.vfd, attaches it to the Windows XP Mode virtual machine, and then releases it again:

.\Create-VirtualFloppy.ps1 sample.vfd
.\Attach-VirtualFloppy.ps1 "Windows XP Mode" sample.vfd
.\Release-VirtualFloppy.ps1 "Windows XP Mode"

Please note that the virtual machine must be running for attach and release to work.

The following example attaches the host floppy drive A and then releases it again:

.\Attach-VirtualFloppy.ps1 "Windows XP Mode" A -HostDrive
.\Release-VirtualFloppy.ps1 "Windows XP Mode"

And that's it! Now you can use floppy disks with Windows Virtual PC 7 once again.