Filename: lib/mysql_utf8.rb

Put that snippet in your Rails project (I used lib/mysql_utf8.rb) and load it in your environment.

That's all. Now your queries should use Unicode:

$ ./script/console
Loading development environment (Rails 2.3.4)
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:ASCII-8BIT>
>> require 'lib/mysql_utf8'
=> []
>> u = User.find(1)
=> #<User id: 1, name: "Test", ...>
>> u.name.encoding
=> #<Encoding:UTF-8>

Note that if you have BLOB types in MySQL you’ll need to force_encoding back to another type. You could do this in your model.

Update: You can also try out my fork of mysql-ruby that is properly encoding aware. It doesn't just convert everything to UTF-8, but rather to your default encoding, so make sure to set your LANG!

So here we go, the issue:

You’re using Rails, and you see this:

invalid byte sequence in US-ASCII

You might also see:

incompatible character encodings: ASCII-8BIT and UTF-8

That happens all the time when trying to get YARD to parse Rails source. It also happens when your ERB templates and your DB data have mismatched encodings.

The Problem

Ruby has multiple default encoding values: internal (the default encoding for new String objects), external (the default encoding for file data), and script (the default encoding of the content of Ruby scripts). When dealing with IO, you need to make sure both your internal and external encodings match up, especially since Ruby defaults to ASCII-7BIT and not UTF-8. The “script” encoding problems were covered in my last blog post on the subject. 

The Fix

All you need to do is tweak your environment variables (your region / language may differ):

$ export LANG=en_US.UTF-8

Now Ruby will use UTF-8 for your external encodings:

$ LANG=en_US.ASCII-7BIT irb19
>> __ENCODING__
=> #<Encoding:US-ASCII>
$ LANG=en_US.UTF-8 irb19
>> __ENCODING__
=> #<Encoding:UTF-8>

Note, this doesn’t cover your default_internal, but usually this will be handled for you. And if you can’t set your ENV, you can set this stuff right in Ruby:

Encoding.default_internal, Encoding.default_external = ['utf-8'] * 2

This sets both your default internal and external encodings.

Now when Rails (well, ERB) tries to read files on disk, it will default to UTF-8 rather than ASCII, and your UTF-8 data from the DB will work just fine.

If you have this problem the other way around (your DB is ASCII), just reverse everything I said.

Why is the implementation developer a bad choice?

I mentioned there are a number of reasons. Let’s go through some of them:

1. The developer knows exactly what the code does — This makes him a bad choice to author documentation, because his perspective is heavily biased. It is easy for this person to miss points of confusion that someone who is less familiar with the codebase may have upon reading the API documentation.
2. Developers are often not skilled technical writers — Technical writing, or writing in general, is a separate skill that not all programmers have. To get quality documentation you should have someone with the right skill set.
3. The developer doesn’t see the big picture — Usually your API’s implementation will bleed into other APIs or depend on domain knowledge from other parts of your architecture (your foo method takes a Bar widget from some external service). The architect is usually the one who knows when this information is new to the reader or if this has already been covered in another document.

In addition to those points, there’s also a small matter of writing style. Documentation can become hard to follow if multiple writing styles are interchanged arbitrarily through documents. Having multiple developers can lead to style inconsistencies, especially if the developers aren’t that skilled at technical writing. Only a small group of people should be managing and maintaining the documentation, to ensure the phrasing is consistent throughout the API.

Why should it be an API architect?

The architects are the ones who have the least bias when it comes to the functionality of the system. A dedicated technical writer may be proficient at writing but may not have an understanding of the entire system and would suffer from some of the drawbacks that developers do (no “big picture”). Also, since an architect’s main responsibility is to define the API, they’re already done half the work of documentation anyway. They might as well go the extra mile and write it down.

Granted, all of my personal projects benefit others, whether it’s my documentation tool, my OpenCL Ruby bindings or my apparently well received compiler design tutorial I’ve recently written. The dirty little secret is: I do it for myself. I wrote the compiler tutorial because I wanted to learn compiler design. I wrote OpenCL bindings because I wanted to learn OpenCL. I suppose I could even say that my willingness to help others to has a slightly selfish motivation; the joy I get out of being the one to help somebody else is certainly a factor in what motivates me to do the things I do.

Lately, however, I’ve been losing steam on a lot of these projects. I’ve been realizing that my lack of recent motivation is due in part to this looking inwardness that I’ve been guilty of for a while now. Rather than take somebody else’s problem and help them for the sake of their problem, I take my problems and apply them to others. This is hardly a sustainable way to live life. The problem lies in focusing too much on either end of the spectrum. I came to my own problems after being bombarded with contract work and clients looking for solutions to their own problems. That life was equally unsustainable, and likely drove me to the need to focus on myself. The conclusion? It’s just as bad to focus exclusively on my problems as it is to focus exclusively on somebody else’s.

The good thing is I’ve also been remembering another dirty little secret: I like to solve other people’s problems too. In fact, I’ve realized I get just as much joy out of solving someone else’s problem as my own. I write this in part as a confession, but also in hopes of being a footnote to somebody else. Solving your own problems is great, and anyone who has that opportunity should apply it often. But a skill like that should be shared with others as often as it is applied to oneself. The important part is balance. If you’re focusing exclusively on your client or boss’s problems, you need to change that and start writing some code for you. Something perhaps completely selfish. Just don’t quit your day job while doing it; variety is what keeps us fresh and motivated to solve problems, all problems.

I’ve always been interested in compilers and languages, but interest only gets you so far. A lot of the concepts of compiler design can easily go way over most programmers’ heads, even the intelligent ones. Needless to say, I’ve tried, without much success, to write a small toy language/compiler before. I’d usually get caught up at the semantic parsing stage. And again, needless to say, this post is mostly inspired by my latest attempt, though this one has been much more successful (so far).

Fortunately over the last few years I’ve been involved in some projects that helped give me perspective and experience on what’s really involved in building a compiler. The other thing I’ve been lucky to have in my corner this time is the help of LLVM, a tool which I’m hardly qualified to talk too much about, but it’s been quite handy in implementing most of the business end (read: complex aspects) of my toy compiler.

So Why Are You Reading This?

Maybe you want to see what I’ve been doing with my time. It’s more likely, however, that you’re probably interested in compilers and languages as I am, and have probably been hitting similar roadblocks. You’ve probably wanted to try this but never found the resources, or did but couldn’t quite follow. The goal of this article is to provide such a resource and explain in a relatively step by step manner how to create the most basic-but-functional compiler from start to “finish”.

I won’t be covering much theory, so if you haven’t brushed up on your BNF grammars, AST data structures and the basic compiler pipeline, I suggest you do so. That said, I plan on keeping this as simple as possible. The goal, of course, is to make this an easy-to-understand introductory resource for people interested but not experienced with compilers.

What You Will End Up With

If you follow this article, you should end up with a language that can define functions, call functions, define variables, assign data to variables and perform basic math operations. It will support two basic types, doubles and integers. Some of the functionality is unimplemented, so you can have the satisfaction of actually implementing some of this stuff yourself and get the hang of writing a compiler with a little help.

Let’s Get Some Questions Out of the Way

1. What languages do I need to know?

The tools we’ll be using are C/C++ based. LLVM is specifically C++ and our toy language will follow suit since there are some niceties of OOP and the STL (C++’s stdlib) that make for fewer lines of code. In addition to C, both Lex and Bison have their own syntax which may seem daunting at first, but I’ll try to explain as much as possible. The grammar we’re dealing with is actually very tiny (~100 LOC), so it should be feasible.

2. Is this really complicated?

Yes and no. There is plenty of stuff going on here and it might seem scary at first, but honestly, this is about as simple as it gets. We will also use a lot of tools to abstract the layers of complexity and make it manageable.

3. How long will it take?

What we will be building took me about 3 days of toying with, but I have a few failed attempts under my belt and those do have impact on my comprehension. Of course, this will be a “follow me” kind of deal, so it should be much shorter for you. To understand completely everything might take a little longer, but you should be able to run through all of this stuff (and hopefully understand a good amount of it) in an afternoon.

So, if you’re ready, let’s get started.

Angel I

Angel II

Angel III

Angel IV

You can find more of my photos on Flickr.

Barracuda is my latest experiment. I wanted to toy around with the new features of Snow Leopard, and as far as I’ve seen, there are no CUDA or OpenCL wrapper libraries out there for Ruby.

OpenCL, if you don’t know, is an “open” version of CUDA, which is NVIDIA’s proprietary architecture for GPGPU programming. Okay, acronyms out of the way, this means that OpenCL is a way to run small but heavy computations in parallel on your GPU’s hundred or so cores. There’s a great demo of why this is cool at MacResearch (skip to the end of the video).

I should point out that while the library is called Barracuda (did you get it?), there’s currently no CUDA support. I want to add support for CUDA down the road after I figure out what’s involved, though.

Show and Tell Time

Anyway, long story short, I wrote a very basic wrapper for OpenCL which currently only supports signed integers and floats, but will hopefully add some more functionality soon. I know you’re all dying for a demo, so, here’s a benchmark of some integer to float related computation in Ruby, versus the same computation using Barracuda:

require 'barracuda'
require 'benchmark'

include Barracuda

prog = Program.new <<-eof
  __kernel sum(__global float *out, __global int *in, int total) {
    int i = get_global_id(0);
    if (i < total) out[i] = ((float)in[i] + 0.5) / 3.8 + 2.0;
  }
eof

arr = (1..3333333).to_a
input = Buffer.new(arr)
output = OutputBuffer.new(:float, arr.size)

Benchmark.bmbm do |x|
  x.report("cpu") { arr.map {|x| (x.to_f + 0.5) / 3.8 + 2.0 } }
  x.report("gpu") { prog.sum(output, input, arr.size) }
end

The way OpenCL works is you compile a C-like program on the GPU, then run it on the GPU with your data input and get some output; it’s a standard Pipes and Filters architecture. The compilation of the program happens through LLVM, meaning it doesn’t require GCC, so this is not like running RubyInline. Also we’re running these programs on like 200 cores, so there’s that part too. Anyway, “where are the benchmark results?” you must be yelling right now, so here:

Rehearsal ---------------------------------------
cpu   5.510000   0.250000   5.760000 (  5.767632)
gpu   0.630000   0.260000   0.890000 (  0.930966)
------------------------------ total: 6.650000sec

          user     system      total        real
cpu   3.860000   0.020000   3.880000 (  3.931644)
gpu   0.310000   0.040000   0.350000 (  0.366625)

Okay, it’s only about a 10x speed increase. That’s not too impressive. Keep in mind though, (x + 0.5) / 3.8 + 2.0 is a freaking simple computation. I dare someone to hook this up to some Hadoop map-reduce and see some real world results, because they’d probably be much more impressive.

Wanna Try It?

The catch: you can install Barracuda if you’re on OSX 10.6 (Snow Leopard) or have a way to get OpenCL on your machine and hack up the Makefiles. I’d like to add support for other environments but I just finished this build.

If you have the right stuff you can grab it right from rubyforge:

sudo gem install barracuda

Or visit the git repo:

http://github.com/lsegal/barracuda

Brainz is a Artificial Neural Network (ANN) library written by Loren Segal. Neural networks are generally used in pattern recognition, signal processing and other data intensive processing problems. ANN’s benefit by not having to explicitly define the procedural steps involved in the problem, but rather by training the neural network to return the correct output for the respective inputs. This means that the same neural network can be applied to many different problem sets without much (sometimes any) modification, and therefore make a good general solution to a large set of problem domains. The drawback, however, is that these neural networks require large sets of data to be trained and this training process can be processor intensive.

AND, OR, BUT…

Want to see some code? This is also from the readme:

# Define a 2-2-1 neural network
net = Brainz::Network.new(2, 2, 1) 

# We must train the system first
1000.times do
  net.train([0, 0], [0])
  net.train([0, 1], [0])
  net.train([1, 0], [0])
  net.train([1, 1], [1])
end

# Now some tests:
p net.run([0, 1]).map(&:round) # => [0]
p net.run([1, 1]).map(&:round) # => [1]

With a small modification to the training data, you can reprogram that network to calculate any other bit operation with no actual modifications to the algorithm. ScAIry.

Before I get into it, I realize the rest of this post will likely sound a little more full of myself than I’m generally comfortable with being, and I apologize.

What I’m Looking For

I’m ideally looking for a senior / lead position on a team where I’d be managing a small group of people. Something where my responsibility is less to do with writing code. I say this not because I think I deserve the title “senior” (I don’t), but I’ve found that over the years I’m much better at management, high-level design and logistics than I am at being the algorithm or SQL guy. This is a position I’d like to grow into and gain experience at, although I’m quite confident that I’m already prepared for such a job. That said, I’m always able to get down in the trenches and code (sometimes I need to), and I’m pretty good at it too (see my skills below).

I’d rather work for something not entirely web-related, though I realize how pervasive the web is, so I can accept/understand that there will be some crossover. I’ve done the whole JavaScript, Rails, web app thing before, though, and I want to try something new. I’d be seriously interested in iPhone, Android, or desktop application development. You can look at it like this: the more web related it is, the more I’d want to be in a management/lead role where coding was not my main responsibility. The less web related it is, the more I’d want to deal with the coding aspect.

Language? Most people who read my blog know me as a Rubyist. I’m, however, by no means married to Ruby. I’m more turned on by interesting projects than evangelistic languages.

What I’m Not Looking For

A “hip new social networking site”. Please god no. Also, as I said above, I’d prefer something that is less related to the web as a general rule. If your web-related project has a large non-web-related component, that’s something I can tolerate. To be blunt, I’m kind of sick of HTML.

My Experience & Skills

It’s hard to enumerate managerial and leadership skills because they mostly sound like complete bullshit on paper, so I’ll stick to something I can quantify my knowledge of, and that’s technology. Here are some buzzwords for you:

I know a multitude of languages including (from best to worst, only including practical experience): Ruby, Java, PHP*, Python, JavaScript, Objective-C, C/C++, and C# (and more). The last language I learned was Lua, which was earlier this year. I generally learn a new language at least once a year.

I’ve used the following major frameworks: Rails, Sinatra, Cocoa, Prototype, MooTools, jQuery, wxWidgets, .NET (WPF, WinForms), Swing.

If you want to see how I code, you can go to http://github.com/lsegal

I am extremely interested in compilers, languages, HCI, and DSP, though I don’t expect anyone to hire me for that. I am great with developer/workflow/testing tools as well as API design and even documentation (I do author a documentation developer tool). Through school I’ve ended up with a lot of experience in many Software Engineering best practices (Quality Assurance, Process Management, Requirements Gathering, Metrics, Software Architecture, System Design).

Professionally, my experience has mostly been with freelance consulting in various web technologies. I’ve also done a pretty awesome internship at Apple in 2007 working for Apple.com during the big website redesign (and original iPhone launch).

* It’s unlikely I’d take a job in PHP since PHP usually equals web (see above).

Location?

I’m not too concerned with where I end up. I really want to stay in Montreal, but from what I’ve seen so far, salaries in the “software developer” field in this area aren’t all that competitive. I’m currently considering NYC, Seattle and the bay area as potential US candidate cities, though there are probably some other great places to live down there (know one? suggest it!). What about Canadian cities? Well, besides Montreal, I’d consider a few places in B.C., but I’ll be honest, I don’t think I could ever work anywhere in Ontario.

Finally, I’m not too interested in telecommuting. I’d prefer working in person with a team, but if it was really worth it, who knows.

Hire Me

If you know of anyone that might be interested in someone with my skill set and has a project that I might find interesting, send them my way. Alternatively, if you’re capable of employing me and you’re interested, I can forward you my resume. My email is lsegal at soen.ca