Bryan Summersett

The Latency of Dialogue

2013-09-29T00:00:00-07:00

Check out my latest post, "The Latency of Dialogue," on Medium. It's about our collective fascination with Internet streams, and what we're missing out on because of it.

A transcription of High Density

2013-08-02T00:00:00-07:00

The following is a loose transcription of Horace Dediu's High Density podcast, a spinoff of his excellent Critical Path series. The conversation is between Horace and Tim Bajarin, a technology analyst of many years. Tim offers a compelling summary of the history of the PC industry, one which I find increasingly relevant, as Horace points out, for its parallels to the mobile OS landscape, especially the parallel between IBM and Google's strategy for PCs and Android, respectively. Also crucial is the observation of a continual pattern in platform lifecycles: "hardware, software OS, SDK, innovative apps, and ... explosion." It was also news to me that Alan Kay ultimately had a lot of influence on John Sculley's vision for computing, as Alan was a fellow at Apple during Sculley's tenure.

Enjoy!

Transcript

Horace: I'm joined today by Tim Bajarin. Tim is someone that people in the technology industry have been following for a long time. He's the head of Creative Strategies, although you do more than that. Can you give people some more background to your story?

Tim: I joined Creative Strategies as a PC analyst. When we started, there were no PC analysts. There were four people in a role similar to mine, and we picked up the PC as part of other work. I was able to literally watch it from day one: Don Estridge from IBM and his team, as well as Apple. I've been following them (Apple) since 1971, and have interacted with all of its top leaders over the last 35 years.

H: You're a fountain of knowledge in this industry. I've been an analyst for a short time, but an engineer for a long time. I myself cut my teeth doing software development on the PC. Maybe you can share some stories about how these organizations evolve.

In particular, I'm curious about how technology organizations actually emerge. When I crossed over from technology to business, in a career sense, technology businesses were seen as another kind of normal business. I always thought that it requires a different kind of operating model, one that wasn't dealing with commodities (well, eventually they do), but in the beginning no one knows how to create products, and things are extremely volatile.

How do you think about the history of Apple, etc. Did they know what they were doing? How did they come about creating these mechanisms of production? MS started a certain culture, etc. How did it happen?

Tim: PC industry was a happy accident. Eddie Roberts introduced the first PC in 1974, and it was on the cover of Popular Mechanics. Following this, Bill Gates wrote most of the OS and flew out to Albuquerque.

Steve Jobs and Wozniak saw this happening, and they wanted to buy this PC kit, but Woz didn't have the money. So he figured out how to do it for himself. It started with the garage mentality that Silicon Valley is now famous for.

Jobs had the mentality: if we both wanted it, other people might want it too. So they go and order 400 kits right off the bat, which later become the Apple I, and they went bananas thinking it might be big. Jobs sold his Volkswagen to finance the purchases.

I've dealt with hundreds of entrepreneurs. It always starts with a mindset of people wanting something for themselves, and only later it happens to turn into something akin to market research.

Seasoned businessman took the gamble on financing them later (Mike Markkula). They had to hire engineers, Steve had to handle marketing, but even until '82, '83, it was still led by young people, and these people were not seasoned in business. That was one of the reasons they eventually tapped John Sculley, being a serious leader.

IBM's foray into the PC market was spearheaded by Don Estridge. He went to top management, wrote a business plan, and convinced them of a business plan to back the PC. In the end, the whole endeavor would've failed if it weren't for VisiCalc.

The original Apple II was a hobbyist machine. People used it for bulletin boards, etc., but there were really no significant apps. With the addition of the CP/M board, however, there was more interest. The thing that would change the potential of the PC in the minds of the lay public was VisiCalc. Dan Bricklin and Bob Frankston. I remember using it, and my mind was boggled. Now I can do rows and columns and sum them up. The computer could do it for you! It changed everything. This was first most useful for small businesses, but even corporate accounts were becoming fascinated with its role as a spreadsheet.

Estridge was thinking there was something to this PC.

The real catalyst for VisiCalc, in the end, was the possibility of developing 3rd party applications. The developer kit was what that allowed people to develop their own applications, and this was the same thing with DOS. IBM's PC didn't take off until there was an SDK to create apps. That pattern goes across everything that's done today -- it's a continual pattern: hardware, software OS, SDK, innovative apps, and explosion.

From an IBM standpoint, they were lucky. It was a gamble. Difference was that IBM went at it with a lot of capital.

H: The notion of taking a huge gamble, without knowing necessarily where the product will find a job for itself... It wasn't until the iPhone's apps that the platform took off, and the use cases came with the apps. We're still at an early phase with the iPad (not clear whether it has a natural set of functions about it).

Stepping back, it sounds like these organizations were chaotic; lots of activity to execute on basics. Having adult supervision, or the lack of. When Apple did try to structure themselves into a traditional, rigorous way, they lost that edge. Is there a correlation? IBM could execute better on some dimensions, but they were essentially following in the footsteps of Apple.

Does this mean that when you're trying to optimize, develop a market, maybe the best approach is not to be structured?

T: Yes and no. Sculley did create more structure, and yet Apple continued to grow until 1992. Steve left after 1984. Sculley managed it relatively well, but he took it into a niche area instead of its full potential market. It was already focused on engineering / technical markets, but the introduction of PageMaker, combined with the laser printer, spawned desktop publishing, which also spawned a change as great as VisiCalc. This was also the first foothold in Hollywood from this change.

IBM's design decision was very different: Apple had the vertical integration. In IBM's case, they used off-the-shelf parts. They were forced to create an SDK. Even though they had structure, they didn't have protection from a competitive position.

I've spoken with Rod Canion, who founded Compaq. I asked him, "what made you think about creating a PC clone?" Rod had been at TI, and said he learned one thing: whenever IBM did something, it became a standard, so it was a no-brainer.

Sculley introduced the concept of the CD-ROM to computers, and thought that was the future of computing. HyperCard too, it seems. He brought Negroponte, the CEO of Bell Labs, and a bunch of other people together based on the idea of the CD-ROM, which they all believed could create a new age of communication. Even though IBM left them less controlled, Apple had structure, but kept them in a niche market, Apple didn't get the growth because of the path they took, even though they were structured.

H: They weren't able to pivot to a structured mode that was successful, because they needed to maintain control over the pieces, so they couldn't optimize to the licensing model.

T: Once IBM realized they had nothing proprietary, they tried to make one called PS/2. Rod Canion stood up to IBM, at that time a gnashing decision, and from the industry standpoint, their decision to not back this was the primary reason we have a free, open, PC clone architecture today.

H: So they didn't necessarily want an open architecture in the end. They never had ownership of that market. It's interesting to think about how Android works in a similar way -- whether they have control over the destiny of that franchise.

What about the period of time between '92, when Apple was growing, until when Jobs came back? What happened in the meantime? Was there changing organizational behavior inside Apple: discussions about going back to its creative roots, etc. What did the organization look like between '92 and '97?

T: What was significant was after taking Sculley out, Apple was handed to Michael Spindler. It was curious in that they were largely where Apple is today Presumably he means that they they're leading in their industry. Apple did well in the niche market, and two years after having the lead in desktop publishing, the PC got caught up. Within two years of owning the multimedia technologies space, others were already catching up. Apple needed to go beyond a niche company, be better. They were concerned that they were losing so much ground over the PC, they began thinking that they needed to make it look like a PC. There was confusion over licensing and non-licensing; they did a deal with Motorola. In 3 years of Spindler, Apple was in serious decline, and then eventually brought in Gil Amelio. Mike's a great guy, but wasn't the right person for that period.

Then handed it to Gil, even though he was a board member at Fairchild at the same time. Realized that they had to control things better, and his position was that they couldn't be all things to all people. By that time, they had lost control, and they weren't growing their OS at the core. The OS didn't have a growth pattern, and had the foresight to see that NeXT (UNIX) had the potential of leveraging that OS to move forward. Apple had gone down, was over 1 billion in the red, and we now know they were 6-7 weeks from bankruptcy.

This gets interesting, because when Jobs got back in '97, I asked how he would rescue Apple. He said, "In the last 5 years, we've taken the eyes off the ball. Fundamentally, I'm going to do the best I can for graphics people, engineers." He wanted to go back to the graphics arts, publishing, TV, etc.: the original core customers of Apple's business. The other thing he said is that I'm going to concentrate on industrial design, which I found totally confusing at the time.

And yet a year later came the iMac. 3 years later is the iPod. Eventually he brings out the iPhone. Steve brought back a vision, and luckily, through its board, they brought more structure to that vision, and with that level of structure and vision, Apple is where it is today.

H: People think that they primarily wanted Steve back, but it was more pragmatic than that: there was something very useful and timely about the NeXT offering at the time, not just the OS, but it had a great development environment. It had the Objective-C environment, the OO-orientation of the platform... Novel at the time, and powerful and seen as guiding the future. The NeXT acquisition was driven by the need to reboot and reconfigure the OS 9 franchise, and Steve Jobs provided that.

T: Once they started talking about that, Gil knew they were missing a vision. I dealt with all the leaders besides Steve, and Sculley was the only one who had a good vision. Sculley was highly influenced by Alan Kay, and Alan was extremely influential on his thinking on education, mobile computing, etc. Having said that, at the particular time, John was becoming much more visionary in the context of graphics arts (he was a former architect). John knew everything was going to go digital, and was fascinated with his vision. He said someday we were going to have billions of devices some day... He was the first CEO I dealt with who literally said we're moving from an analog to a digital world, and nothing can stop that. He was becoming more visionary in that role, and the part of the business that needed to grow was they were stuck in these niches while the PC market was exploding.

H: Jobs had more vision, but was extremely pragmatic in how he would get out of that hole.

T: He also mellowed.

H: He also learned a lot with Pixar, and I'm fascinated with how much the Pixar experience influenced his management expertise: how you manage creative people, how you deal with the entertainment industry, etc. Do you find this to be true?

T: Pixar had a huge influence on him, but you have to believe his take on media in general had to change.

H: There's the thought that companies are surfing this wave of technology, and you can't optimize and control a product category. This is a crucial dilemma for a technology company: whether you grab an opportunity and optimize it, or you say perhaps optimization and creativity are orthogonal, and either you create a hybrid sustaining / disruptive, or you choose to be constantly sustaining.

T: There's a couple things going on. Steve understood content was important, and controlling it was also important. Years before, when Akio Morita (Sony) bought a movie studio (Universal), I happened to be at an event with key analysts, and we asked him: why would Sony buy a movie studio? He said, "It's just software." Even then, he understood that what we had in the movies would eventually become digitized like software. Jobs at Pixar understood this as well. The message was clear. Content is king, and it was clear to him then. I believe Jobs was strategically thinking this through from '98 to '00, and eventually he realized he had to build hardware, software, and services together. With the iPod, he had this in mind, because he had to find a way to distinguish himself from the market. The Mac was still within the context of a commoditized market, but in the end, you had to have a certain level of vertical integration to control your destiny.

H: It's hard to define success and failure. It's always nuance and subtlety. We're trying now to think of what went on in his mind. As he mellowed and aged, and saw what he built become corrupted, I wonder how much he thought about how to build an organization that will last in the midst of market forces: sometimes you get lucky, sometimes not, and how do you keep the creative spirit going. It comes back to anecdotes about why people work at Apple. Why do people stay in the organization even when failing? Some of it might be due to a fear of the unknown, but I wonder how the cultural aspects of Apple were shaped by Jobs before and afterwards.

T: You can see this in some of the predominant cultural aspects of Silicon Valley today: they all believe what they're doing will change the world. The Yahoos, etc., all believed they were going to transform the way people enjoy something. There's a tendency for it to be a cliché today, but you go to Apple, and they'll tell you they're there because they have a vision.

H: That's a common theme of startups. What was unique about Apple in particular? Is it a track record, leadership, or was there an operating model where this was the place where stuff gets done?

T: You could look at Apple's historical position: they didn't invent the PC, they reinvented it. They didn't invent publishing, but they reinvented it. They didn't invent the phone, they reinvented it. The driving force is that we can reinvent the watch, the car... there's enough understanding of Apple's track record (which is highly unique -- reinvent technology sectors over and over, and the people there still think there's things to invent and disrupt.)

H: This exceptionalism about Apple divides people. Speaking of the future, how do you see Apple's current structure / team / culture as it exists today? Is it built to last? Second question: what do you think they're going to go with this structure in the future?

T: The current VPs were all schooled in the Jobsian vision. We have examples of this too. After Walt Disney died, people thought the company was going to lose its momentum. Look at Disney now! Of all the companies I track, I still think that Apple has the best chance to do this over the others.

H: What's your take on what they might do with their organizational and cultural makeup; their resources and processes, and priorities? Do you think there's a roadmap out there (watches, television)... How do you see that thinking going?

T: One thing that's intriguing about Apple has always been the fact that they were always forward thinking on user interfaces. The mouse was brought to us by them (popularized); touch UIs; voice UIs. Other areas they continue to disrupt are in user interfaces -- gestures, head tracking.

H: That's an observation I've made as well.

T: They disrupt user interfaces by the nature of the product they deliver. We didn't need touch on a desktop, but we sure did on a smartphone. On a TV, we don't need touch, but we need to reinvent the user interface to make it smarter. I absolutely believe they're going to make a watch. Years before, I highlighted explicitly the watch and car industry as industries they could reinvent. No question they're going to get it right. I have a digital watch, which Seiko introduced in 1981. Automobiles: I want it to be much smarter than it is today. iOS 7 with its car integration is going to be even more important going forward. If we think Apple's done innovating and disrupting, we need to stop and check ourselves.

There was an article about major leaders saying Apple is dead -- Michael Dell, etc. How many times has someone said Apple will not die? If the company has a vision, and the employees believe in it, with a track record like they do, there's something there. If I was going to bet on a big company disrupting markets, it would be Apple.

H: That's a real endorsement. I wanted to make sure that folks can follow you...

T: Twitter is @bajarin. I do Techpinions with my son and three others.

H: For someone in it since the beginning, I'm heartened to hear affirmation that the pioneer is still there. It's a testament to the power of their vision. What made Apple sustain itself, despite being disrupted, was moving forward and having the organization and resources to build the next thing, and the vision is the crucial element. It's causal in the sense that if it didn't exist, the vision would fail. Does this apply to other companies as well?

The presence of vision is crucial, and if you're in the technology business, it's primarily important.

You have a design problem

2013-05-28T00:00:00-07:00

The following is an interpretive transcription of Ryan Singer's UX / UI PeepCode sessions in the style of Paul Arden's It's Not How Good You Are, Its How Good You Want to Be.


You have a design problem.


Ask:
How did they get there?

What's going to happen when they're done?


Before, now, after.



Constrain the domain so the problem is simpler.
Think of the real-world analogy.


Draw information flows:
relations based on navigation trees.


Think about state.
Does it matter?

Is this a micro-domain? It can be transient.
Is this is social thing? The scale is bigger.

Email is UI too.

Split the UX into beginning, middle, and end contexts.
Afterwards, design for the screen with these in mind.



What screen matters most?

Find the simplest screen that:
forces you to be compelling
provides value
and contains no unnecessary detail.

This teaches you about the product.

Doing the most complicated thing first may be tempting,
but this may never be used.

Think of UX like a failing test.
There's always conflict.
Identify the thing that I don't know, and build it out. 
Remove uncertainty.


Consider the naive approach first.

Find the natural way of doing things.
If I did this in the physical world, 
what would its most natural-looking interface be?

(Scribbles on paper?)


Try to communicate using their vocabulary.

What's the dumbest idea that works?
If this were paper, how would I do it? 



Christopher Alexander said
conflict is the starting point.

What conflicts exist?
*Weirdness* is conflict.
Increase confidence by resolving conflicts through design.



Sometimes, it needs to be seen on screen to make a decision.

Confidence is a spectrum.

It's important to find group confidence in a design on a team.
Do this by working side by side on something.




Make the intended action clear.
One way to do this is guarantee there's only one thing to do.

Flows help.
Before, now, after.

Every step needs a concept and implementation.
They must be evident to the user.





Mock (in Photoshop) to explore alternatives.


Criteria for it working:
*reptile stuff*.

When I look at it...
how does it affect me?



Sketching UX doesn't mean doing everything. 
*Use as medicine to ease the pain of worrying.*

Prove clear thinking by sketching. Remove worry.
Just enough to know what you're doing.


Design on the wall is exactly this.
Temporary.
The point is: the designer, programmer work *together*
through the design.

Then they go work.



There's never a moment where everything is built.
Design only when I need to design.
See what's working from there.

Step by step.

To prototype your UI:

*Focus on one UI element at a time.*
Use whatever tool is most convenient. 

PS, Web Inspector.
Rails can be fast if you know it already.

Take screenshots.
Mash it up.
Throw it away.

Write it however you can to make it do what you want.
Don't plan your implementation.

You will externalize your CSS later anyway.
Write it inline. 

Write it the long, easy way.
Start with a mess.
Refactor when your conflicts are resolved.


Photoshop can remove doubt.
Use it to see something.
Mock it quick to see how it will look.
Throw it away.

Stay relevant

2013-05-27T00:00:00-07:00

2005 M.I.A Pitchfork interview, describing the circumstances of her first album:

M.I.A.: The thing that I got exposed to when I met Justine was a lot of middle-class kids making music in England who had everything at their disposal and nothing to say. They didn't rep anything! Yet everybody was having soul-searching issues going, "Oh my Goooooood, who am I, what am I doing? I might have to go to a yoga retreat this year." And I'd be like, "Why don't you just look at what's going on and be a part of the planet, instead of wanting to be what's come before?" Which is how all those bands set themselves up-- they aspired to be [older] bands and forget what's going on [today]."

Analyzing mbostock's queue.js

2013-03-31T00:00:00-07:00

Even with the proliferation of open source software on the web, relatively few developers take the time to study the source code of the software they might use everyday. I believe this can largely be traced back to a lack of culture around code reading, which I will hopefully talk about in a later post. To combat this for now, however, the answer is simple: we just need to practice looking at and studying the works of others, similar to the way painters studied (and copied) the great masters.

Beginning today, I'm beginning a series of "deep dives." First is a look into a tiny JavaScript asynchronous helper library called queue, written by Mike Bostock (of d3.js fame). Many libraries have been written prevent "callback spaghetti:" highly nested callbacks that need to be executed in some order. Many solutions have been implemented, from promises to generators to a combination of both. Mike takes a simple approach: the API comprises one method, defer, which takes in a task (a Node.JS-style function, whose last argument must be a completion callback), along with optional arguments. To get the results back, there's await and awaitAll. These differ only in how they return their results array. Mike mentions the entire source can be compressed down to around 500 bytes, so this walkthrough shouldn't take too long.

Example

First, though, I'd recommend looking at the repo on GitHub, or taking a quick peek at queue.js directly before reading this to get a quick sense of the overall code structure. Here's an example use of the API from the docs, reading two files in parallel and awaiting the combined results:

queue()
    .defer(fs.stat, __dirname + "/../Makefile")
    .defer(fs.stat, __dirname + "/../package.json")
    .await(function(error, file1, file2) {
    console.log(file1, file2);
  });

queue.js

Let's begin. We start with an anonymous self-invoking function. Note that this parenthesis isn't necessarily required; the JS parser just needs to treat this as a function expression rather than a function declaration, so a + or ! would work as well.The surrounding parenthesis helps clarify to the reader that this function will be executed immediately.

(function() {

Next, we check for Node.js-style modules; if these exist, add queue as a module. Otherwise add it to the self object. Assigning to self was new to me, as I typically expect modules to assign to the window global for browsers; however, it turns out most browsers resolve self to window in the global context, and to the Web Worker context in a Web Worker. Thus, assigning to self covers both contexts. Note it may also seem that queue is undefined at this point; however, because JavaScript hoists variable names to the top of functions, the function queue from below is actually available.

  if (typeof module === "undefined") self.queue = queue;
  else module.exports = queue;
  queue.version = "1.0.3";

This is a shorthand for Array.prototype.slice, and now we enter into our main function definition.

  var slice = [].slice;

  function queue(parallelism) {

queue is the object we return to the client at the end of this constructor. This initially seems confusing, as it shadows the name of the function immediately above, but this allows code below to read more naturally. Most other variables are already explained:

    var queue = {},
        tasks = [],
        started = 0, // number of tasks that have been started (and perhaps finished)
        active = 0, // number of tasks currently being executed (started but not finished)
        remaining = 0, // number of tasks not yet finished
        popping, // inside a synchronous task callback?
        error = null,

However, await and all are interesting. In the context of the internals of queue, the await variable is a placeholder for the final callback function. all here is a boolean that determines how the results should be handed back: either as separate arguments, or as an array. Read the documentation for more information.

        await = noop,
        all;

Setting default parallelism is important for later on, as this variable is checked when popping tasks off our queue to ensure we're not executing more than parallelism tasks at a time.

    if (!parallelism) parallelism = Infinity;

Next we have defer, our first public function. defer slurps in its arguments and stores them to our current tasks. defer's API specifies its first argument is our task function, and the rest are optional arguments for that function.

    queue.defer = function() {
      if (!error) {
        tasks.push(arguments);

Next, we increment our count of remaining tasks, and then execute our pop() method, where most of the the magic happens. We'll look at this in a second. Interesting detail: defer tries to executes our tasks immediately, rather than waiting for await, for instance. This means that we can call defer after we await and it should call await again for us. So ends our defer method.

        ++remaining;
        pop();
      }
      return queue;
    };

The await method and awaitAll methods are similar; they simply set the variables we were talking about, and optionally, if there weren't any remaining tasks at this point, immediately notify our callback of our status. These methods also return their parent object, queue, which enables a nice builder pattern-like API. See the examples in the documentation for how this is used.

    queue.await = function(f) {
      await = f;
      all = false;
      if (!remaining) notify();
      return queue;
    };

    queue.awaitAll = function(f) {
      await = f;
      all = true;
      if (!remaining) notify();
      return queue;
    };

Pop is the internal method responsible for popping remaining tasks off the queue and executing them, n at a time, where n is our parallelism value. The popping variable here is used to signal to the task's internal callback function whether or not we're still actively popping tasks off our queue. This comes into play in the case when a task completes synchronously / immediately (i.e., not on the next event loop) so that we know not to fire the pop call again. Now we get to the main while loop for executing our tasks. We continue to spawn new tasks (via started) until we run out of them, or we reach the max parallelism of tasks at the given time.

    function pop() {
      while (popping = started < tasks.length && active < parallelism) {

Now, within the loop, our index is the next started task. We find its arguments, t, then extract its arguments from the input using slice(..., 1), which essentially removes the task function call from the arguments list.

        var i = started++,
            t = tasks[i],
            a = slice.call(t, 1);

Here, callback returns the callback for a given task index, and pushes this as the new last argument; it's a Node.JS convention to have the last argument as the callback. We then increase our active count and execute our task function with a null context and its arguments. Note that this as context would be inappropriate here, as the called tasks shouldn't have any knowledge of the queue internals. So ends our while loop.

        a.push(callback(i));
        ++active;
        t[0].apply(null, a);
      }
    }

Next is the callback constructor implementation. Why do we need this function? Putting the callback directly in the while loop wouldn't work, since JavaScript has function scope, so our task index i would contain the last written index value. Now we see that this constructor serves to close over the task index for the callback function. The returned callback takes in an optional error, and then a result value, which is a Node.JS convention.

    function callback(i) {
      return function(e, r) {

Now we're inside the internal callback function. Decrementing the active count means we can potentially start another task. If we have errors, set our error variable, which will implicitly dequeue all other tasks; we also notify our client immediately that an error has occurred.

        --active;
        if (error != null) return;
        if (e != null) {
          error = e; // ignore new tasks and squelch active callbacks
          started = remaining = NaN; // stop queued tasks from starting
          notify();

Otherwise, if our task executed successfully, store the result for this callback in the original task array. Check if there are any remaining tasks; if there are and we're still popping tasks in our while loop, do nothing, because the loop is dequeueing tasks already. Otherwise, start popping again. Immediately returning / synchronous tasks don't normally occur for tasks that involve I/O or ones that call setTimeout, because these will typically trigger their callback on the next event loop tick. Finally, if there's no remaining tasks, notify the client that we're done.

        } else {
          tasks[i] = r;
          if (--remaining) popping || pop();
          else notify();
        }
      };
    }

Finally notify. This function executes the callback function given by the client in await or awaitAll. If there was a problem, report this as the first argument; otherwise call the callback either with a list of results or results as function arguments, depending on whether the API called was awaitAll or await, respectively.

    function notify() {
      if (error != null) await(error);
      else if (all) await(error, tasks);
      else await.apply(null, [error].concat(tasks));
    }

    return queue;
  }

noop is used as a default value for the await callback. This is an example of the null object pattern and simplifies the implementation in notify, so we don't have to special case our logic if the await callback was null.

  function noop() {}
})();

Conclusion

That wasn't so bad, huh? It's nice to see a library with that much utility in that little code. Hopefully there was at least something you learned from reading the source. If you're interested in reading more async libraries for JavaScript, check out the source to kriskowal's q, Tom and Yehuda's rsvp.js, or even Mozilla's experimental task.js as well.

Please feel free to comment on feedback, style, etc., as I'm looking to create these posts on a somewhat regular cadence.

Steve Yegge and Grok

2012-08-11T00:00:00-07:00

Steve Yegge, best known for his long-winded and colorful articles on software engineering (and for the fairly recent "public" posting lamenting the state of internal tools at Google, posted another today as a thought experiment around conservatism and liberalism in programming language design. All thoughts aside, I found the debate to be an enlightening one in that it's a topic the software community has historically underprivileged: treating software as a social construct that should be studied as one. I like to think about the success of programming languages through this lens, and arguably successful programming language designers do as well: see Larry Wall's motivation for Perl, or Matz's for Ruby (beauty, simplicity). For as much as people can disagree with the assessment Steve places on individual languages' ranking in his spectrum, the fact that he ranked them in a public post so clearly demonstrates to me his genius in understanding that people's egos are attached to the programming languages they use. I should also add that he should be in a position of understanding how people use software in the large, as he (currently?) works on tooling at Google, and previously at Amazon.

The most interesting part of the aforementioned article was his mention of the "Grok" project, one which I've heard a little about through a few of his public appearances. As I searched for more information, it became clear to me that this was something he had been working on internally at Google for some time, and if I were to be more bold, I would suggest that this indeed was the very idea that he pitched internally to Bezos back when he worked at Amazon.

So what is Grok? I'll quote directly from the man himself:

The project's sole purpose in life is to bring toolchain feature parity to all languages, all clients, all build systems, and all platforms. My project is accomplishing this lofty and almost insanely ambitious goal through the (A) normative, language-neutral, cross-language definitions of, and (B) subsequent standardization of, several distinct parts of the toolchain: (I) compiler and interpreter Intermediate Representations and metadata, (II) editor-client-to-server protocols, (III) source code indexing, analysis and query languages, and (IV) fine-grained dependency specifications at the level of build systems, source files, and code symbols.

Yegge's ambition is to improve the state of toolchain support for all editors, in a uniform way, to the point where we can simply plug in the appropriate autocomplete, definition finding logic, etc., rather than coding up from scratch each time someone implements an IDE.

This would a great benefit to desktop-based development systems, but more importantly, I think it would be an amazing feature for the plethora of "cloud" editors entering into the space today (see Cloud9 IDE, Space), as they would be able to focus on providing a great UX experience while not having to burden themselves with the language support problem. Language tooling was part of the reason TextMate 2, for instance, took so long to build.

What we know

The project started work in earnest at Google 4 years ago (2008). This aligns with a researcher's resume where he mentions he worked on the project in 2008 in Kirkland (outside Seattle, and home of Costco!). It went through many phases:

...in four years, from "VC funding" to "acceptance" to "cautious enthusiasm" to "OMG all these internal and even external projects now depend critically on us." The project is still based at Google Kirkland. It grew from 3 engineers, 6 20%ers (2010) to 6 engineers (time unknown) to 12 engineers (present). At one point, they open sourced the Java-based Python indexer used on the project (incorporated into Jython). It used to be publicly available as Google Code Search, but was removed in October 2011. (This happens to coincide with his leaked "platform" memo... Coincidence?). A front-end is still publicly available, albeit only for the Chromium source base.

The most information on the project I could find was a video of a talk he did at the Northwest C++ user's conference in 2010. Disclaimer: this talk is nearly two years old at this point, so the project has clearly gone through a number of iterations since this came out. I've gone and written up a loose transcription of the talk, as I haven't seen much else written about it elsewhere. This is all from the slides and his dialogue, but shouldn't be construed as direct quotes in any case. With that being said:

Grok

Note: the original video is available on Vimeo.

Yegge: Originally pitched at Amazon, now working on this full time at Google. Came from learning how to deal with gigantic codebases; 50M lines or more.

Motivation

Every company uses N >= 3 languages. In practice, they would use 15 without admitting it. SQL, devops might use Ruby to instrument their tests, etc. In production, they might only use 3, but all for different reasons.
Engineers refuse to switch editors / IDEs. Once an Emacs user, always an Emacs user. Many Googlers use Emacs or VIM as their primary development environment, with the Java programmers using primarily IntelliJ.
Only compilers "know" languages. All non-compiler languages are ad-hoc. If it's not built into the IDE, it's probably not widely (pervasively) used, and you get into the problem where some people are more productive than others due to the tools they choose to use.

Language matrix problem

Given N langauges and M editors / IDEs, total toolchain effort is N * M... Any toolchain support for this number of systems is non-trivial. For example, Emacs: cc-engine.el is a huge number of lines of code, and is almost entirely for indentation. This gets replicated for every language out there.

How do you solve matrix problems like this? Use a hub and spoke model.

Grok: a Language-to-client Hub

Moving from matrix to a hub-and-spoke model for editors. Eclipse, and every IDE implementor failed software engineering 101: separate the indexer from the GUI platform. There should be a clean separation from the autocomplete from the GUI portion.

Each IDE becomes a "thin" client. Every IDE supports M languages out of the box.

Side benefits: Eclipse will start faster (less tooling support needed). Consistency across cross-language calls.

Crucially, consistent tools make it easier to switch languages. Devs would then be more likely to use the best language for the job. IDE authors can focus on presentation and editing. This leads to more configurability, scriptability, accessibility in languages.

Additional languages

Why do people choose Java? It's not because of Java the language. C++ was heavily marketed by Bell Labs, AT&T (90's). Sun chose to pump millions of dollars into Java. Two things: tools and performance.

Think that they're choosing a language based on type system, etc. There's a whole class of problems that aren't handled by this (runtime problems and the NullPointerException).

If a language isn't successful, it's because the tools aren't good enough. Through interviews with hundreds of people, online comments, and talks: it always comes down to the tools.

The economy of the game market is actually forcing to switch the language: they need to use Lua to the logic. Bigger companies don't have the same economics in other fields -- people will tend to write everything in one language. If you're writing something to be run once, you should be using a language made for doing quick scripting tasks, rather than trying to implement these tools in a systems language.

How do we do it?

Compiler writers don't care about these things: they want things to be correct, fast. Typically make a IR, then almost immediately get rid of all this information to convert to bytecode, etc. Why do we have two compilers, one for codegen vs. the toolchain? For the most part, they don't want to combine them because they have slightly different goals.

Idea: pry open compilers, run their "guts" on distributed clusters, output a language neutral index. Serve the index via service APIs. Write client plugins, etc.

Clang makes this easy to do.

How we analyze languages

First use case: create LXR-like environment for Java. ctags is basically a regexp-based library. Had to fork Eclipse, but all the code was complected with the indexer and the GUI as previously mentioned. Existing tools use a combination of methods for tooling:

C++: Eclipse CDT (soon to be Clang) use a reference graph
Java: Eclipse JDT builds up IR for IDE use cases: diagnostics, errors, warnings, type graphs. Internally, has a reference graph for callers, callee.
Python: bottom-up, top-down type inferencers, and tended to work quite well for our use cases. We opened sourced the Python indexer
JavaScript: Google's JSCompiler (Closure) for type inferences, etc. This in practice made the job much easier.

The issue is often, when people have a language problem, they tend to want to write some stupid regexp parser. The thought is that parsing is hard. In practice, name resolving and type resolution is 80,90% of the problem... but people never realize that until they've implemented the parser already. So people think parsing is the difficult thing. Many people take the approach the wrong way.

The goal of Grok is not to write a parser. We don't want to reinvent the wheel. For in-house languages we instrument the compiler / interpreter.

Goal: get compiler writers to give us this information

The idea is to put peer pressure on the language / compiler vendors community (ultimately the ones to generate or interpret all the metadata of languages and are the ones in the best position to instrument their software properly). People will start to expect "Grok" support in their editors.

Anecdote: MS to Google people often complained that code exploration will really poor, coming from the world of Intellisense and Visual Studio.

After doing some research, we realized that people often asked for, essentially, SQL support for doing queries on languages... In other cases, they might want sort of an XPath, hierarchical query support. In other contexts, they might also want to treat code as a graph problem. Desired features would be:

Code navigation and browsing: Jump to decl,def,show xrefs, outlines, ..
Interactive editing: Indent, reformat, autocomplete symbols...
Queries (e.g., find subclasses of C where X)
Static analysis (e.g., find memory leaks): Example question: find out how to use exception handling in Java. Want to prove statically-checked exceptions are useless 98% of the time. Now we can do this easily, as we can grep for these things across the entirety of the code base.

Language-neutral index

This is really a key-value store, a multimap. Every identifier gets a key (globally unique). C++ names aren't necessarily global in any case. Have to invent a global namespace by depot, etc.

From our observations, every file generates 100x its file size in metadata. Implementation thus far:

Modeled loosely on Eclipse's IR: Nodes ("bindings") for decls/defs/refs; Typed edges ("associations") between nodes
Example association kinds: SUBCLASS_OF, SUBCLASS_OF, CALLED_BY, IMPLEMENTS, ENCLOSING_SCOPE, FIELD_OF, DEPENDS_ON, INCLUDES

This is sorta language neutral. Some languages don't have classes; instead they have functions, variables. Each time we go through the data model we find new mappings for new languages. In practice, every single abstraction in this edge has at least 2 languages using it: symbols are used the same in Lisp as in Ruby, for example.

Cultural challenges

Even if you deliver this beautiful thing, there's still a lot of cultural problems in getting something out there successfully. (audience: Someone should've done this 30 years ago. Yegge: Someone did do this already in Smalltalk, but it was only for that specific dynamic language.)

Adoption: hard to get devs to use new tools. It's hard to retrain myself to click links rather than grep for everything. At Google, it's well-known that fancy new tools have a slow adoption curve. Google ranks you on how adoption is going.
Speed vs accuracy: Grok's philosophy is that some data is often better than none. On the other hand, Eclipse doesn't scale well for Google-size code bases. Autocomplete sometime halts the entire process when a seemingly simple dependency changes. Doing this remotely is still a tradeoff, as some programmers don't deal well with eventual consistency in their tooling -- some people complain that immediate consistency is crucial to prevent them from building things incorrectly. Java programmers are often expecting 100% IDE accuracy, and this is a cultural expectation.
Prioritization: companies never want to staff this kind of effort.
Managing expectations: Engineers hear about this project and they want it yesterday. There's no good elevator pitch for this kind of thing, but if you sit down with them for 15 / 20 minutes, they will want it at that point. Now, you have to manage expectations. If you promise everything, you'll never be able to meet people's immediate expectation of the thing.

Clang @ Google

Clang enables modern C++ tools on Linux (finally!). Fast, smart, efficient, usable (all of which CDT is not). There was a lot of work to make it work properly.
Google's main code base is recently Clang parse-clean. Chromium can be built with Clang.
An internal clang team at Google is devoted to supporting Clang tools: IDE support, indexing, search, refactoring, analysis

(GCC folks have been late to the game with toolchain support. Stallman is afraid of other people wresting control of the project from him -- which is based in reality, as many companies tried to do so.) This ended up causing poor tooling support.

See Grok in Action

Grok indexes Chromium + WebKit source code currently. The indexer was contributed entirely as a 20% project. We serve Grok index data to Google Code Search. - Currently only Chromium is externally visible. In any source file, hover over indentifiers. Can also show cross-refs (bottom of the page)

Long-term we would like to be able to Grok all open-source code in the world. For any given Java library in the world, find all callers of this function!

Biggest problem is that we're really tied to Google's build system. Critically tied to finding information about build paths, Make, ant, etc... This is something we'd have to rearchitect for publishing out in the open.

What we've built so far

Team of 3 in Kirkland, plus six 20% contributors. We support C++/Java/Python/JS and 2 internal languages. Open-sourced our Python indexer (jython.org). Mostly support Browser/Navigation + simple queries. Flagship clients are Emacs and Code Search. We started work on interactive use case static analysis.

Open-source / open platform

Grok isn't easily / directly open-sourceable. Our indexes and protocols are all internal formats. We also rely heavily on Google's internal build system. The long-term goal is be an external platform and with open standards:

Standardize expected set of compiler artifacts
Standardize schemas (MySQL / NoSQL / XML, etc)
Standardize the services for clients
Open-source our analyzers and plugins as example.

At some point, it would be nice to have people convert their standard IR formats to Grok using tools.

WYSIWYG and merging

2012-08-05T00:00:00-07:00

I just read Joel Spolsky's take on distributed version control. He states the most important thing about Git is that it forces people to think in terms of changes, not versions. As in, bring in Mary's change, bring in his change, as opposed to everyone working towards some base version. GitHub takes this idea and turns it into a service.

However positive this leap may be, it tends to cause usability issues for those using it. When dealing with merging, there's the essential complexity of merging -- that is, merging in changes from multiple sources into working code, and trying to suss out how to deal with the semantics of two changes to a piece of software. This is separate from the accidental complexity of the CLI, documentation, the bevy of tools, the mental model of the Git runtime, etc., which people tend to find so difficult, especially for single-person use cases.

I suspect GitHub's desktop apps were created to alleviate this usability issue. In simple cases, much of this merging complexity can be transparent to users, and the system could do much of the merging automatically. When dealing with programming / markup languages, this doesn't in pratice work very well, as the strictness of the compiler and the language's syntax tries to remove all ambiguity. Git, being the "stupid" versioning system it is, doesn't get in the way of this process. Manual merges are the human-powered mechanism for ensuring that whatever is written down is exactly is what is intended by the author.

In other media, though, ambiguity in merges isn't a big deal, especially when dealing with non-procedural languages. When merging graphics layers in Photoshop, there's basically no possibility of conflict in global state, so people are able to copy paste pretty readily. (In practice, I think this might not be entirely true, as there's probably some dependencies on global settings like bit depth, gamut, etc.)

Tracking document changes in Word is another example of VCS that happens to be quite usable for its problem domain. Seeing the main DVCS princple (changes, not versions!) applied to a WYSIWYG editor, visual or textual, would be interesting. This is basically where Google Wave was heading.

For certain domains, I would like to see an abstracted editor, using Git internally, that makes it totally opaque to the user when commits are occuring. This would be basically equivalent to an undo / redo command history, but with the additional ability for you to bring in others' history in with yours.

Again, this all entails a medium receptive to ambiguous changes, so systems language programming would probably be a bad fit for this. I would expect this would be most useful in the context of GUI programming and front-end development, where merges can be as simple as dragging and dropping. The traditional difficulty of forming abstractions with a visual medium (at least compared to textual programming languages) fortunately makes it that much easier to adapt to a new model of merging changes.

In defense of Flash 4

2012-07-17T00:00:00-07:00

When discussing Adobe Flash, it's important to separate out Flash, the IDE, from Flash Player, the vilified web runtime. The former I'll forever have a soft spot for, as it arguably influenced my becoming the programmer I am today.

When people talk about hating Flash, they typically mean the runtime, and all the baggage that goes along with it: its fan-inducing, CPU-devouring ineffectiveness, its perky installer windows that always seems to appear whenever you least expect it, etc. We've come to expect better of our runtimes in this day and age. Steve Jobs' Flash letter spells out clearly why we don't have Flash on iOS today: its proprietary nature, its security record, performance, battery life, and the fact that it wasn't designed for touch input. Except for the final issue, these are all failings of the virtual machine. Flash, as a platform, has failed to meet the expectations of its consumers, and rather than supporting this sub-par experience, Apple chose to kill it off. That's that.

But with the eventual demise of the Flash ecosystem, let us not forget Flash, the editor, especially in its early forms. It was and is one of the most successful multimedia creation tools for the web. It's interesting to think about why this was. I suspect it was partially because it enabled those without any traditional programming experience to create quite sophisticated web sites and animations. Arguably, this was ultimately because it had an intuitively designed user interface.

I confess: when I was in elementary school, I had an obsession with Pokémon. I had basically memorized the official guidebook to the series -- I had read it so many times the cover had come clear off. Of course, I wasn't alone in my fascination with this Nintendo cash cow. All of the best Pokémon fan pages on the Internet had these over-the-top animated fight sequences as their splash intros, which I thought was the coolest thing ever. Along the way, I eventually acquired a copy of Flash 4 when I realized this was the software everyone had used to make these great scenes. Before I knew it, or knew that programming even existed as a profession for that matter, I was animating my meticulously hand-drawn vector Squirtles across timelines with only the most cursory knowledge of how animation software actually worked. Mind you, at that time, you were encouraged to do everything using a GUI interface, and for most part it handled things quite well. Over the course of a year, I got more ambitious: minute-long action sequences with sound; dancing Pikachus set to the music of Saturday Night Fever; parallax shoot-em-ups made without any real working knowledge of variables. Did I know the inheritence hierarchy of buttons or movie objects, or care about the difference between bitmap or vector graphics? No. All I knew was that I had a vision, and this was giving me some pretty convincing results.

Eventually, I started using event-handlers, learning for the first time how to assign commands to mouse events using Actionscript. I remember this being by far the most frustrating part of the process, as I didn't ever take the time to read documentation for the thing (as if that should ever be a problem). To me, Actionscript was a band-aid, a set of indecipherable hieroglyphics grafted onto this amazing playground of visual fun. I remember painfully learning what a compilation step was, and wondering if the thing I had just typed up was actually going to do what I wanted it to. I remember cryptic errors leading to abandoned ideas. I remember solving the second hardest problem in computer science: naming things, because you needed to script objects on screen. I remember it wasn't as fun, more than anything else.

I was realizing the possibilities and the frustrations caused by using an abstract, textual form when interacting with my graphics. With subsequent versions of Flash, it was as if this style had taken over. Best practices now recommended reading the spec for this new, adult Actionscript: a gargantuan, unholy Javascript and XML lovechild, filled with serious object-oriented programming constructs, and all sort of other paradigms completely foreign to me. To make a button before, you had to add actions to its 'click' marker on a timeline. With Actionscript, I had to think of my button as a living, breathing object, with event handlers for each of the possible touch states I wanted to work with. I had to straddle two mental models of the thing before I was even allowed to think of doing the kind of animation work I once had. Frankly, I was lost in this world. While others found the challenge of a blank text field exhilarating, my young self found better things to do. With the maturation of Actionscript, the role of the programmer was emphasized over the graphic artist and animator. Flash, the IDE, had effectively said I was no longer its target consumer.

Flash 4 was the beginning for me. I think back to how formative this was in my later thinking about computers; how it gave me the confidence to know that I was indeed capable of making complex animations, and how creating on a computer could be as, if not more, rewarding than any pen-to-paper drawing experience. All this without needing to type a single line of code. As I read back on the history of Flash, it's interesting to see its evolution from a seemingly trivial drawing application to the highly-advanced development environment it is today. Somewhere along the way Macromedia set its sights on a more serious customer. Did their developers realize what they had created in the first place?

More features does not a better product make. When I see a dearth of simple creational software today, and especially with the iPad in the hands of so many children, I wonder why where there isn't a more concerted effort in making products that even children can be productive with, like Flash 4. Flash's focus on simplifying animation and interactions got an elementary school kid to make some awesome stuff, some of which would still be difficult for your run-of-the-mill college hire to do in other mediums -- and all because there was a tool with the right abstractions. How shortsighted are we to assume that that we should be teaching everyone "real" programming à la Codecademy or Lua via the iPad app Codea, as if typing programming languages into a terminal was the only valid way to create on a computer. Let's spend some time thinking about how to re-imagine our own tools and languages to serve the jobs of a diverse set of people, even if it means underserving the most technical for a change. In the process, maybe we give some other kids a glimpse at their potential future selves.

Papert's Mindstorms

2012-07-01T00:00:00-07:00

Alan Kay cites Seymour Papert's 1980 thesis Mindstorms so frequently that I figured would eventually have to read it. A few weeks ago, I picked up the revised 1993 edition. Today, educators and technologists tend to emphasize marketplaces and apps, the what, as solutions to educational issues. Few argue for radical changes both in medium and pedagogy, the what and the how. Papert believed this was key. In predicting the rise of the personal computer, he developed a cohesive vision of children teaching computers as the basis for a new digital pedagogy, based on his novel theory of constructionism.

Papert's studied with Jean Piaget for over 6 years, and was famously precocious. Piaget once said, "no one understands my ideas as well as Papert". His most important insight was in how people, and especially children, accumulate knowledge experientially. Piaget concluded that children were capable of abstractions at an early age. The difference between adult reasoning was that these were primitive models, learned through experimentation, and were constantly changing with new experiences. Genetic epistemology described this process of accumulated learning. Recently, this has been linked as the probable function of own neurons.

Papert strongly believed in using experiential learning as a foundation for further, more traditional abstract learning. A mathematician by training, he believed that children using the right object-to-do-with, along with appropriate teacher-as-peer guidance, could help develop mental models of mathematical concepts at a very early age. This informed his theory of constructionism. Papert summarizes experimentation-as-learning as syntonic. This is to be contrasted with more traditional disassociated learning, which emphasizes learning abstractions, then its applications.

Math education in the US is still largely taught using the disassociated model. This tends to disengage students, as they can neither empathize nor apply it to their own lives. He believed this caused our national mathphobia, as generations of parents traumatized by New Math instilled this same learned helplessness in their children.

He stressed teachers act as cultural anthropologists, keeping abreast of cultural trends to ensure using relevant mediums in children's lives. He believed that the keys to successful teaching was in making material meaningful and relevant to students. In practice, this could be accomplished by finding relationships to previous experiences in their lives, and building on that knowledge as a model. He gives a personal anecdote to demonstrate this idea: as a child, he loved gears, and the differential axle in particular. He saw how the two axles could move independent of each other while the drivetrain distributed power accordingly. After awhile, he was able to visualize their motions in his mind's eye. When it finally came time to learn symbolic algebra in grade school, he had little difficulty in intuiting the relationship between variables in an equation, as he already had a working model of these gears in his head. For instance, given:

ax + by = 0

a and b could represent constant ratios of our wheel gears, and x and y the rotational velocities of a given wheel's axle.

Papert later became a professor at MIT, and spoke with the likes of Marvin Minsky regularly. As such, he was able to accurately predict the age of the personal computer. He was an optimist, and believed that this could also radically change education:

[...] the computer presence will enable us to so modify the learning environment outside the classroom that much if not all the knowledge schools presently try to teach with such pain and expense and such limited success will be learned, as the child learns to talk, painlessly, successfully, and without organized instruction.

Important to note is that Papert thought of the computer as disruptive rather than complementary to our educational system. Thirty years later, we haven't seen the incorporation of a pervasively digital pedagogy. (This could be partially explained by the scarcity of engineers and scientists in administrative and educational leadership positions.) Rather, the focus has been on adapting existing curriculum to digital forms. For example, the Khan Academy promotes blended learning classroom models, where short-form video lessons are supplemented by teacher intervention.

(1) significant change in patterns of intellectual development will come about through cultural change, and (2) the most likely bearer of potentially relevant cultural change in the near future is the increasingly pervasive computer presence.

Perhaps Papert overestimated the speed of cultural change, thinking it could keep pace with technology. His knowledge of cultural change came from his wife, Sherry Turkle, who spent time researching the French's adoption of psychoanalysis. Cultural choices tend not to change once a path has been taken, no matter how suboptimal it becomes as time goes on. Change tends to occur over generations, rather than decades. In turn, disruptive ideas are discarded in favor of incremental progress. Many examples of this occur in life: Kuhn's The Structure of Scientific Revolutions describes this phenomenon as it pertains science research. Papert cites the "QWERTY phenomenon," where the arrangement of letters on a keyboard is more due to historical artifact more than critical reasoning. Alan Kay describes computer science today as an incremental pop culture in "Programming and Scaling", and offers guidance on how to avoid this default mode of thinking.

Other roadblocks abound. Experiential learning is realized differently for every individual, and this is at odds with an idealized curriculum. Papert argues objective curriculums are a panacea, as best practices are always relative to a given culture and technology. For example, analytical geometry, typically taught in high school, is done so for historical reasons: only at that point in school can fine motor skills be assumed enough to draw graphs using pen and paper. Today, the merits of cursive is questioned in a world of keyboards and touch screens.

Today's emphasis on summative assessments, coupled with an incentive to teach to the test, point towards an even more idealized curriculum than before. This poses significant problems in more fully realizing Papert's ideas in US classrooms. On the other hand, his vision has seen acceptance in other forms. The One Laptop Per Child project's motivational theory was based on constructionism, and is in millions of classrooms worldwide. LEGO Mindstorms, named after the book, along with its corresponding robotics competitions, emphasizes experiential learning by programming LEGO bricks. The MIT Media Lab was originally created as a center of epistemological research, initially focused on exploring Papert's own principles. Of course, Papert also developed LOGO, an "exploratory mathematical sandbox" and object-to-do-with to help children intuit mathematical concepts. In an upcoming essay, I'll dive into LOGO's programming model, and look to use it as a lens into educational software as a whole today.

Old projects

2012-06-25T00:00:00-07:00

I'm doing some spring cleaning on the site. If you're interested in music synthesizers for iOS, see the project pages for iJam or pdlib. Otherwise, stay tuned for future updates.

Holistic views of nearly everything

2010-10-14T00:00:00-07:00

Finding connections between disciplines fascinates me, like the similarities between work habits of hackers and painters. The benefits of extracurricular research in unrelated fields aren't always immediately realized, but this gain in perspective can lead to many long-term benefits. Lateral thinking and creativity depend on the ability to adapt processes and metaphors to something else. Regarding my ideal university experience, I advocate that engineering majors invest substantial time in studying humanities.

Analytical and procedural thinking are requisites for any successful computer scientist, but the truly innovative are able to call on a wide breadth of knowledge when needed. This can have surprising results. John Holland took the idea of gene replication and adapted it into a well-known algorithm. Alan Kay, trained as a biologist, took the idea of cellular processes as the basis for object-oriented programming: encapsulation serving as an analogue to the cell wall, and signaling between the cells representing something akin to message passing. Understanding the incentives of your customer is largely about being able to empathize with them. As Google has demonstrated, this can't always be solved by refining an algorithm or A/B testing. This realization has led to technology companies emphasizing interaction design, which has its roots in psychology. Systems thinking, now commonly applied in business development, was largely inspired by the work of Donella Meadows, an environmental scientist who wrote 1972's the The Limits of Growth, a study of how exponential forces interact with finite natural resources. (Another one of her books, Thinking in Systems, remains a favorite read of mine as a beginner's guide to systems thinking.)

Breadth in thinking, to me, means reading widely, critically, and then deeply in fields one finds interesting. This requires "learning how to learn," which happens to be one of the tenants of an ideal liberal arts education. Unfortunately, this isn't a focus of many engineering schools. More engineers and technologists-cum-entrepreneurs should find more time to explore these fields, not only for perspectives' sake, but also for the sake of our global society.

Scheming to learn recursion

2010-08-01T00:00:00-07:00

Reading Gödel, Escher, Bach broadened my perspective on recursion and the idea of the pervasiveness of composition of self with self. Recursion is everywhere: in the Piaget's genetic epistemology, where new mental abstractions are defined by existing ones; the actual neurological structure of our brain via feedback loops, and its digital analogue in neural networks; visicious and virtuous cycles in systems thinking and economics; fractals; and also in various programming languages, especially those with lamdba calculus roots.

One languages celebrating recursion is Guy Steele and Gerry Sussman's Scheme, a dialect of of Lisp known for its minimal syntax and library. Scheme was created partially as a teaching aid for introductory computer science courses at MIT. Lisp, in turn, has its roots as a theoretical language in the AI Lab in the late 50's. John McCarthy famously created it without intended it to be implemented; it took a student named Steve Russell to create the first interpreter. The LISP 1.5 Programmers Manual, by McCarthy himself, remarkably implements the entirety of a Lisp interpreter in a single page (see Appendix B's definition of eval).

However, my initial interest in Lisp began with Paul Graham's essays. He waxed eloquent in Hackers and Painters's "Beating the Averages", about how business with limited labor should be using programming languages that maximize their productivity, ideally via higher-level abstractions and meta-languages. Around the same time, I picked up a copy of Daniel Friedman's Essentials of Programming Languages, an academic treatise on programming language theory that assumed a working knowledge of Scheme.

Rather than diving headfirst into the text, I figured it would better to start small and work my way up. What I'm reading now is The Little Schemer, also by Daniel Friedman. This book's style is the complete opposite of Essentials; it assumes no experience programming, and works as a two-way dialogue, where the reader is encouraged to programming questions along the way. The tone is absurd in the vein of Why's Poignant Guide, with pages devoted as a tablemat for PB&J sandwiches. It's a quick read, but worthwhile if you're looking for an entertaining weekend guide to recursive thinking.

It's unfortunate the syntax of Lisp has been a roadblock, as its S-Expressions, enabling homoiconicity, are one of language's strengths. M-Expressions, as McCarthy originally intended Lisp to be written in, were meant to provide a more familiar imperative facade to the language. This never caught on, even with valiant attempts by Apple with its Dylan language. Whether syntax is the reason, CS schools, and most famously MIT, have phased out teaching Scheme in their introductory courses. There's an interesting debate about practicality of programming languages taught at the university level. Papert would argue that the first language you learn tends to influence your view of others, and in this case Scheme might be a good theoretical foundation. Python, on the other hand, has a strong resemblance to what we typically write as pseudocode, which may serve a more enlightened discourse for beginners. In any case, Lambda the Ultimate article explores the issue further.

Learning Perl

2010-07-09T00:00:00-07:00

As there's a decent amount of legacy code at work, I'm starting to learn Perl. The language has had a longer history than some of the popular scripting languages today, so determining best practices involves good ol' investigative work. A good starting point, I've been repeatedly told, is the mammoth (e.g. Programming Perl), which represents the learn-everything-at-once, disassociated approach, and favors those who would love nothing more than to curl up next to an O'Reilly book for an entire weekend. Modern Perl, a more recent beginner's book, which takes an opinionated view in promoting more recent changes to the language, starting with Perl 5.12.

Here are a few observations I've noticed:

Ruby borrows Perl's syntax more than I realized (e.g., postfix conditionals)
The breadth of libraries is the most comprehensive of any language I've used. People really seem to be good about writing good documentation for their code, which is helpful. The CPAN model has since seen rapid adoption by other scripting languages with much success; see Ruby's gem and node.js' npm.
cpanminus is a simpler CPAN; it works exactly how I'd like a package management system to work, without the bevy of features you need to configure when using the perl -MCPAN -eshell CLI.
Moose is an impressive object system, and includes many more meta-class features than one would typically come to expect in an OO language. Traits are another welcomed feature. Unfortuatnely, adding Moose as a dependency also tends to limits its usage in legacy contexts.
Pervasive context-awareness is something that took me some time to understand, but now it seems natural. In addition, dealing with references adds another layer of complexity that can be intimidating to new users having to deal with both at the same time. This is unlike most other scripting languages, which err on the side of simplicity.
Topic variables (@_ and $_) seem like a bad idea for readability, but really cut down on the tedium of dealing with variable names in cases where it's obvious. It's great for short programs, probably not so much for modules.
Perl's argument passing reminds me of Javascript, in that you do most of your work explicitly pushing or popping arguments off of your @_ / arguments stack.
Perl's efforts to be backwards-compatible has left it with a lot of vestigial syntax; in a way it suffers from the same problems C++ does in trying to please everyone. The solution for a lot of teams using C++ is to use a well-defined subset of the features, and I believe this parallels making sure to use the latest features of Perl as well.

Tufte and Infographics

2010-02-10T00:00:00-08:00

I've recently read Tufte's Visual Display of Quantitative Information.Althought it predates the predominance of digital infographics, the text applies to any two dimensional medium.

One of the main ideas is that good information design strives to reveal the greatest number of ideas in the shortest time, with the least ink, in the smallest space.

It's interesting to contrast this with the trend of infographics, which seek to firstly entertain through visual experiences. Tufte would call this "chartjunk," as its design does nothing to contribute to a user's real comprehension of the underlying data.

I would argue for somewhere in between, that infographics serves a job to be done. It allows those seeking a ready-made conclusion enough facts to support a stance. In other words, it serves as a high schooler's persuasive essay, emphasizing and summarizing data to aid a predetermined conclusion. In reality, infographics are often presented as supplements to an editorial in another medium, like on cable or in an article, and serve much the same purpose. A Tufte disciple would find the practice of willful filtering of information reprehensible, and would instead seek to create a graphical representation of the entirety of the data present, with a thought that the display would compel the viewer to her own conclusions. On the whole, I tend to side with information oversharing, as "simplifying" based on presumptions of limited time or critical reasoning skills can often lead to wild misconceptions.

txting

2010-01-08T00:00:00-08:00

There's something haiku-like about the 160-character limit. The poets of our day will tweet.

Happy new year

2010-01-01T00:00:00-08:00

Happy new year everyone!

I've been working on a few projects. One is iJam, a mobile music app for the iPhone. We hope to have it out sometime in the spring. It uses a port of Pure Data, a software synthesizer, called pdlib.

Harmony

2008-12-04T00:00:00-08:00

"Art is harmony." - Seurat.