The Julia Blog

New York Open Stats Meetup

2012-04-18T00:00:00-07:00

I’ll be giving a talk on Julia at the New York Open Statistical Programming Meetup on May 1st. After my presentation, John Myles White and Shane Conway are going to give followup demos of statistical applications using Julia. Then we’re going to hang out and grab drinks nearby. Thanks to Harlan Harris and Drew Conway for setting the whole thing up!

Announcement:

After a brief hiatus, we are very excited to announce our May meetup will feature one of the hottest new languages in statistical computing: Julia. We are delighted to welcome Stefan Karpinski, one of the creators of Julia, to give an introduction to the language and his perspective on statistical computing.

Julia is a general-purpose, high-level, dynamic language in the tradition of Lisp, Perl, Python and Ruby. It is designed to take advantage of modern techniques for executing dynamic languages with statically-compiled performance. As part of this design, the language has an expressive type system, which programmers may leverage for dispatch and error checking — incidentally providing the compiler with useful type information. Using types is entirely optional, however: “typeless Julia” is a valid and useful subset of the language, similar to traditional dynamic languages, which nevertheless runs at statically compiled speeds.\

Julia is especially good at running Matlab and R-style programs. Given its level of performance, we envision a new era of technical computing where libraries can be developed in a high-level language instead of C or Fortran. We have also experimented with cloud API integration, and begun to develop a web-based interactive computing environment. The ultimate goal is to make cloud-based supercomputing as easy and accessible as Google Docs.

We will also hear from a mix of people who have already started developing in Julia and see some examples of what they have developed.

The meetup will follow our typical schedule: pizza will begin at 6:15pm, Stefan will begin promptly at 7pm, and we will head to The Central Bar around 8:30pm.

Update: You can see the slides for the talk here. There was no video of the talk, but hopefully the slides are informative — there are, among other things, a lot of code examples that should just work if pasted into the Julia repl.

Lang.NEXT Announcement

2012-03-24T00:00:00-07:00

Jeff and I will be giving a presentation on Julia at the upcoming Lang.NEXT conference, a gathering of “programming language design experts and enthusiasts” featuring “talks, panels and discussion on leading programming language work from industry and research.” We are honored and excited to have been invited to speak at an event alongside so many programming language luminaries.

Abstract:

Julia is a dynamic language in the tradition of Lisp, Perl, Python and Ruby. It aims to advance expressiveness and convenience for scientific and technical computing beyond that of environments like Matlab and NumPy, while simultaneously closing the performance gap with compiled languages like C, C++, Fortran and Java.

Most high-performance dynamic language implementations have taken an existing interpreted language and worked to accelerate its execution. In creating Julia, we have reconsidered the basic language design, taking into account the capabilities of modern JIT compilers and the specific needs of technical computing. Our design includes:

Multiple dispatch as the core language paradigm.
Exposing a sophisticated type system including parametric dependent types.
Dynamic type inference to generate fast code from programs with no declarations.
Aggressive specialization of generated code for types encountered at run-time.

Julia feels light and natural for data exploration and algorithm prototyping, but has performance that lets you deploy your prototypes.

Update: You can see the slides for our talk here. Video of the presentation is available here.

Shelling Out Sucks

2012-03-11T00:00:00-08:00

Spawning a pipeline of connected programs via an intermediate shell — a.k.a. “shelling out” — is a really convenient and effective way to get things done. It’s so handy that some “glue languages,” like Perl and Ruby, even have special syntax for it (backticks). However, shelling out is also a common source of bugs, security holes, unnecessary overhead, and silent failures. Here are the three reasons why shelling out is problematic:

Metacharacter brittleness. When commands are constructed programmatically, the resulting code is almost always brittle: if a variable used to construct the command contains any shell metacharacters, including spaces, the command will likely break and do something very different than what was intended — potentially something quite dangerous.
Indirection and inefficiency. When shelling out, the main program forks and execs a shell process just so that the shell can in turn fork and exec a series of commands with their inputs and outputs appropriately connected. Not only is starting a shell an unnecessary step, but since the main program is not the parent of the pipeline commands, it cannot be notified when they terminate — it can only wait for the pipeline to finish and hope the shell indicates what happened.
Silent failures by default. Errors in shelled out commands don’t automatically become exceptions in most languages. This default leniency leads to code that fails silently when shelled out commands don’t work. Worse still, because of the indirection problem, there are many cases where the failure of a process in a spawned pipeline cannot be detected by the parent process, even if errors are fastidiously checked for.

In the rest of this post, I’ll go over examples demonstrating each of these problems. At the end, I’ll talk about better alternatives to shelling out, and in a followup post, I’ll demonstrate how Julia makes these better alternatives dead simple to use. Examples below are given in Ruby which shells out to Bash, but the same problems exist no matter what language one shells out from: it’s the technique of using an intermediate shell process to spawn external commands that’s at fault, not the language.

Metacharacter Brittleness

Let’s start with a simple example of shelling out from Ruby. Suppose you want to count the number of lines containing the string “foo” in all the files under a directory given as an argument. One option is to write Ruby code that reads the contents of the given directory, finds all the files, opens them and iterates through them looking for the string “foo”. However, that’s a lot of work and it’s going to be much slower than using a pipeline of standard UNIX commands, which are written in C and heavily optimized. The most natural and convenient thing to do in Ruby is to shell out, using backticks to capture output:

`find #{dir} -type f -print0 | xargs -0 grep foo | wc -l`.to_i

This expression interpolates the dir variable into a command, spawns a Bash shell to execute the resulting command, captures the output into a string, and then converts that string to an integer. The command uses the -print0 and -0 options to correctly handle strange characters in file names piped from find to xargs (these options cause file names to be delimited by NULs instead of whitespace). Even with extra-careful options, this code for shelling out is simple and clear. Here it is in action:

irb(main):001:0> dir="src"
=> "src"
irb(main):002:0> `find #{dir} -type f -print0 | xargs -0 grep foo | wc -l`.to_i
=> 5

Great. However, this only works as expected if the directory name dir doesn’t contain any characters that the shell considers special. For example, the shell decides what constitutes a single argument to a command using whitespace. Thus, if the value of dir is a directory name containing a space, this will fail:

irb(main):003:0> dir="source code"
=> "source code"
irb(main):004:0> `find #{dir} -type f -print0 | xargs -0 grep foo | wc -l`.to_i
find: `source': No such file or directory
find: `code': No such file or directory
=> 0

The simple solution to the problem of spaces is to surround the interpolated directory name in quotes, telling the shell to treat spaces inside as normal characters:

irb(main):005:0> `find '#{dir}' -type f -print0 | xargs -0 grep foo | wc -l`.to_i
=> 5

Excellent. So what’s the problem? While this solution addresses the issue of file names with spaces in them, it is still brittle with respect to other shell metacharacters. What if a file name has a quote character in it? Let’s try it. First, let’s create a very weirdly named directory:

bash-3.2$ mkdir "foo'bar"
bash-3.2$ echo foo > "foo'bar"/test.txt
bash-3.2$ ls -ld foo*bar
drwxr-xr-x 3 stefan staff 102 Feb  3 16:17 foo'bar/

That’s an admittedly strange directory name, but it’s perfectly legal in UNIXes of all flavors. Now back to Ruby:

irb(main):006:0> dir="foo'bar"
=> "foo'bar"
irb(main):007:0> `find '#{dir}' -type f -print0  | xargs -0 grep foo | wc -l`.to_i
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
=> 0

Doh. Although this may seem like an unlikely corner case that one needn’t realistically worry about, there are serious security ramifications. Suppose the name of the directory came from an untrusted source — like a web submission, or an argument to a setuid program from an untrusted user. Suppose an attacker could arrange for any value of dir they wanted:

irb(main):008:0> dir="foo'; echo MALICIOUS ATTACK 1>&2; echo '"
=> "foo'; echo MALICIOUS ATTACK 1>&2; echo '"
irb(main):009:0> `find '#{dir}' -type f -print0  | xargs -0 grep foo | wc -l`.to_i
find: `foo': No such file or directory
MALICIOUS ATTACK
grep:  -type f -print0
: No such file or directory
=> 0

Your box is now owned. Of course, you could sanitize the value of the dir variable, but there’s a fundamental tug-of-war between security (as limited as possible) and flexibility (as unlimited as possible). The ideal behavior is to allow any directory name, no matter how bizarre, as long as it actually exists, but “defang” all shell metacharacters.

The only two way to fully protect against these sorts of metacharacter attacks — whether malicious or accidental — while still using an external shell to construct the pipeline, is to do full shell metacharacter escaping:

irb(main):010:0> require 'shellwords'
=> true
irb(main):011:0> `find #{Shellwords.shellescape(dir)} -type f -print0  | xargs -0 grep foo | wc -l`.to_i
find: `foo\'; echo MALICIOUS ATTACK 1>&2; echo \'': No such file or directory
=> 0

With shell escaping, this safely attempts to search a very oddly named directory instead of executing the malicious attack. Although shell escaping does work (assuming that there aren’t any mistakes in the shell escaping implementation), realistically, no one actually bothers — it’s too much trouble. Instead, code that shells out with programmatically constructed commands is typically riddled with potential bugs in the best case and massive security holes in the worst case.

Indirection and Inefficiency

If we were using the above code to count the number of lines with the string “foo” in a directory, we would want to check to see if everything worked and respond appropriately if something went wrong. In Ruby, you can check if a shelled out command was successful using the bizarrely named $?.success? indicator:

irb(main):012:0> dir="src"                                                              
=> "src"
irb(main):013:0> `find #{Shellwords.shellescape(dir)} -type f -print0  | xargs -0 grep foo | wc -l`.to_i
=> 5
irb(main):014:0> $?.success?                                                                
=> true

Ok, that correctly indicates success. Let’s make sure that it can detect failure:

irb(main):015:0> dir="nonexistent"                                                              
=> "nonexistent"
irb(main):016:0> `find #{Shellwords.shellescape(dir)} -type f -print0  | xargs -0 grep foo | wc -l`.to_i
find: `nonexistent': No such file or directory
=> 0
irb(main):017:0> $?.success?
=> true

Wait. What?! That wasn’t successful. What’s going on?

The heart of the problem is that when you shell out, the commands in the pipeline are not immediate children of the main program, but rather its grandchildren: the program spawns a shell, which makes a bunch of UNIX pipes, forks child processes, connects inputs and outputs to pipes using the dup2 system call, and then execs the appropriate commands. As a result, your main program is not the parent of the commands in the pipeline, but rather, their grandparent. Therefore, it doesn’t know their process IDs, nor can it wait on them or get their exit statuses when they terminate. The shell process, which is their parent, has to do all of that. Your program can only wait for the shell to finish and see if that was successful. If the shell is only executing a single command, this is fine:

irb(main):018:0> `cat /dev/null`
=> ""
irb(main):019:0> $?.success?
=> true
irb(main):020:0> `cat /dev/nada`
cat: /dev/nada: No such file or directory
=> ""
irb(main):021:0> $?.success?
=> false

Unfortunately, by default the shell is quite lenient about what it considers to be a successful pipeline:

irb(main):022:0> `cat /dev/nada | sort`
cat: /dev/nada: No such file or directory
=> ""
irb(main):023:0> $?.success?
=> true

As long as the last command in a pipeline succeeds — in this case sort — the entire pipeline is considered a success. Thus, even when one or more of the earlier programs in a pipeline fails spectacularly, the last command may not, leading the shell to consider the entire pipeline to be successful. This is probably not what you meant by success.

Bash’s notion of pipeline success can fortunately be made stricter with the pipefail option. This option causes the shell to consider a pipeline successful only if all of its commands are successful:

irb(main):024:0> `set -o pipefail; cat /dev/nada | sort`
cat: /dev/nada: No such file or directory
=> ""
irb(main):025:0> $?.success?
=> false

Since shelling out spawns a new shell every time, this option has to be set for every multi-command pipeline in order to be able to determine its true success status. Of course, just like shell-escaping every interpolated variable, setting pipefail at the start of every command is simply something that no one actually does. Moreover, even with the pipefail option, your program has no way of determining which commands in a pipeline were unsuccessful — it just knows that something somewhere went wrong. While that’s better than silently failing and continuing as if there were no problem, its not very helpful for postmortem debugging: many programs are not as well-behaved as cat and don’t actually identify themselves or the specific problem when printing error messages before going belly up.

Given the other problems caused by the indirection of shelling out, it seems like a barely relevant afterthought to mention that execing a shell process just to spawn a bunch of other processes is inefficient. However, it is a real source of unnecessary overhead: the main process could just do the work the shell does itself. Asking the kernel to fork a process and exec a new program is a non-trivial amount of work. The only reason to have the shell do this work for you is that it’s complicated and hard to get right. The shell makes it easy. So programming languages have traditionally relied on the shell to setup pipelines for them, regardless of the additional overhead and problems caused by indirection.

Silent Failures by Default

Let’s return to our example of shelling out to count “foo” lines. Here’s the total expression we need to use in order to shell out without being susceptible to metacharacter breakage and so we can actually tell whether the entire pipeline succeeded:

`set -o pipefail; find #{Shellwords.shellescape(dir)} -type f -print0  | xargs -0 grep foo | wc -l`.to_i

However, an error isn’t raised by default when a shelled out command fails. To avoid silent errors, we need to explicitly check $?.success? after every time we shell out and raise an exception if it indicates failure. Of course, doing this manually is tedious, and as a result, it largely isn’t done. The default behavior — and therefore the easiest and most common behavior — is to assume that shelled out commands worked and completely ignore failures. To make our “foo” counting example well-behaved, we would have to wrap it in a function like so:

def foo_count(dir)
  n = `set -o pipefail;
       find #{Shellwords.shellescape(dir)} -type f -print0  | xargs -0 grep foo | wc -l`.to_i
  raise("pipeline failed") unless $?.success?
  return n
end

This function behaves the way we would like it to:

irb(main):026:0> foo_count("src")
=> 5
irb(main):027:0> foo_count("source code")
=> 5
irb(main):028:0> foo_count("nonexistent")
find: `nonexistent': No such file or directory
RuntimeError: pipeline failed
    from (irb):5:in `foo_count'
    from (irb):13
    from :0
irb(main):029:0> foo_count("foo'; echo MALICIOUS ATTACK; echo '")
find: `foo\'; echo MALICIOUS ATTACK; echo \'': No such file or directory
RuntimeError: pipeline failed
    from (irb):5:in `foo_count'
    from (irb):14
    from :0

However, this 6-line, 200-character function is a far cry from the clarity and brevity we started with:

`find #{dir} -type f -print0 | xargs -0 grep foo | wc -l`.to_i

If most programmers saw the longer, safer version of this in a program, they’d probably wonder why someone was writing such verbose, cryptic code to get something so simple and straightforward done.

Summary and Remedy

To sum it up, shelling out is great, but making code that shells out bug-free, secure, and not prone to silent failures requires three things that typically aren’t done:

Shell-escaping all values used to construct commands
Prefixing each multi-command pipeline with “set -o pipefail;”
Explicitly checking for failure after each shelled out command.

The trouble is that after doing all of these things, shelling out is no longer terribly convenient, and the code becomes annoyingly verbose. In short, shelling out responsibly kind of sucks.

As is so often the case, the root of all of these problems is relying on a middleman rather than doing things yourself. If a program constructs and executes pipelines itself, it remains in control of all the subprocesses, can determine their individual exit conditions, automatically handle errors appropriately, and give accurate, comprehensive diagnostic messages when things go wrong. Moreover, without a shell to interpret commands, there is also no shell to treat metacharacters specially, and therefore no danger of metacharacter brittleness. Python gets this right: using os.popen to shell out is officially deprecated, and the recommended way to call external programs is to use the subprocess module, which spawns external programs without using a shell. Constructing pipelines using subprocess can be a little verbose, but it is safe and avoids all the problems that shelling out is prone to. In my followup post, I will describe how Julia makes constructing and executing pipelines of external commands as safe as Python’s subprocess and as convenient as shelling out.

Stanford Talk Video

2012-03-01T00:00:00-08:00

Jeff gave his previously announced, invited talk at Stanford yesterday and the video is available here. Congrats, Jeff!

Stanford Talk Announcement

2012-02-27T00:00:00-08:00

I will be speaking about Julia at the Stanford EE Computer Systems Colloquium on Wednesday, February 29 at 4:15PM PST. The title of the talk is Julia: A Fast Dynamic Language For Technical Computing.

Abstract:

Julia is a general-purpose, high-level, dynamic language, designed from the start to take advantage of techniques for executing dynamic languages at statically-compiled language speeds. As a result the language has a more powerful type system, and generally provides better type information to the compiler.

Julia is especially good at running MATLAB and R-style programs. Given its level of performance, we envision a new era of technical computing where libraries can be developed in a high-level language instead of C or FORTRAN. We have also experimented with cloud API integration, and begun to develop a web-based, language-neutral platform for visualization and collaboration. The ultimate goal is to make cloud-based supercomputing as easy and accessible as Google Docs.

Speaker Bio:

Jeff Bezanson has been developing the Julia language for two and a half years with a small distributed team of collaborators. Previously, he worked as a software engineer at Interactive Supercomputing, which developed the Star-P parallel extension to MATLAB. At the company, Jeff was a principal developer of “M#”, an implementation of the MATLAB language running on .NET. He is now a second-year graduate student at MIT. Jeff received an A.B. in Computer Science from Harvard University in 2004, and has experience with applications of technical computing in medical imaging.

The talk will be webcast live.

Edit: the video of the talk can be found here.

Why We Created Julia

2012-02-14T00:00:00-08:00

In short, because we are greedy.

We are power Matlab users. Some of us are Lisp hackers. Some are Pythonistas, others Rubyists, still others Perl hackers. There are those of us who used Mathematica before we could grow facial hair. There are those who still can’t grow facial hair. We’ve generated more R plots than any sane person should. C is our desert island programming language.

We love all of these languages; they are wonderful and powerful. For the work we do — scientific computing, machine learning, data mining, large-scale linear algebra, distributed and parallel computing — each one is perfect for some aspects of the work and terrible for others. Each one is a trade-off.

We are greedy: we want more.

We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled.

(Did we mention it should be as fast as C?)

While we’re being demanding, we want something that provides the distributed power of Hadoop — without the kilobytes of boilerplate Java and XML; without being forced to sift through gigabytes of log files on hundreds of machines to find our bugs. We want the power without the layers of impenetrable complexity. We want to write simple scalar loops that compile down to tight machine code using just the registers on a single CPU. We want to write A*B and launch a thousand computations on a thousand machines, calculating a vast matrix product together.

We never want to mention types when we don’t feel like it. But when we need polymorphic functions, we want to use generic programming to write an algorithm just once and apply it to an infinite lattice of types; we want to use multiple dispatch to efficiently pick the best method for all of a function’s arguments, from dozens of method definitions, providing common functionality across drastically different types. Despite all this power, we want the language to be simple and clean.

All this doesn’t seem like too much to ask for, does it?

Even though we recognize that we are inexcusably greedy, we still want to have it all. About two and a half years ago, we set out to create the language of our greed. It’s not complete, but it’s time for a 1.0 release — the language we’ve created is called Julia. It already delivers on 90% of our ungracious demands, and now it needs the ungracious demands of others to shape it further. So, if you are also a greedy, unreasonable, demanding programmer, we want you to give it a try.