The essay’s main points are that core Unix concepts such as fork(), accept(), select(), and Unix sockets had lamentably been ignored or neglected by proponents of Ruby and Python; that Unicorn is undoubtedly one of the best, most densely packed examples of Unix programming in Ruby I’ve come across; and that Ruby (and Python) developers should learn more about these paradigms, and consider using them where appropriate.

While I learned much from the essay, and enjoyed reading it and following the “* is Unix” meme it sparked, after I read it I had some questions rolling around my head which I couldn’t quiet. So, since comments are currently disabled on Ryan’s site, I wrote him an email. He was kind enough to respond quickly and at length, illuminating a subject I’d been fuzzy on for years. In fact, I thought his responses were edifying to the degree that they should be published. So after securing his permission, I’m reproducing our dialog here.

If you haven’t read it yet, I suggest reading I like Unicorn because it’s Unix before proceeding.

I wrote Ryan:

Hi Ryan, thanks for your recent piece on Unicorn and Unix. Enjoyed it, and learned… well, I learned that there’s even more important stuff I don’t know. (It’s funny sometimes being a software developer/architect and not having a CS background.)

Anyway, you definitely got me interested in learning more about the Unix programming and concurrency models, thank you for that.

One thing that has me a little confused: I got the impression that a lot of the difference between the Unix and Java/Windows approaches boils down to processes versus threads. Is that right? Is it an oversimplification?

(The process vs. threads thing seems related to the discussions going on about how to write concurrent programs; specifically the debates about shared data vs. message passing. Isn’t IPC essentially message passing?)

If that’s right, then what about Apache? Wasn’t the point of 2.x that it moved to a thread-based model over the process-based model of 1.x? And if so, then… what’s up with that? I’d think there were a ton of very smart, very unixy people involved in that effort; they must have had some good reasons for the switch, right? Or, more likely, I’m missing something.

Thanks!
Avi

And he responded, that night:

On Wed, Oct 7, 2009 at 7:44 AM, Avi Flax wrote:

> Hi Ryan, thanks for your recent piece on Unicorn and Unix. Enjoyed it, and learned… well, I learned that there’s even more important stuff I don’t know. (It’s funny sometimes being a software developer/architect and not having a CS background.)

Tell me about it. I don’t have a high school diploma :)

> Anyway, you definitely got me interested in learning more about the Unix programming and concurrency models, thank you for that. One thing that has me a little confused: I got the impression that a lot of the difference between the Unix and Java/Windows approaches boils down to processes versus threads. Is that right? Is it an oversimplification?

I think that’s about right. There’s a lot of other important differences but, now that I think about it, they all seem to stem from the basic model of concurrency. Java/Windows people pretty much have to use threads because their processes are handicapped. Unix has good processes, so Unix people can use either processes or threads.

Now, the Unicorn piece was talking specifically about threads in Ruby. Native threads are a lot more powerful than Ruby threads. I always choose processes over threads (even native threads) when I can get away with it, but native threads are useful, whereas Ruby threads really aren’t.

> (The process vs. threads thing seems related to the discussions going on about how to write concurrent programs; specifically the debates about shared data vs. message passing. Isn’t IPC essentially message passing?)

That’s precisely what it is. “Message passing” usually implies a specific kind of object encoding and message dispatch, though, whereas IPC is a bit more general. You have to build up messages and dispatch on top of simple binary streams between processes, which is what most message passing systems do but IPC usually implies you build that part yourself.

It’s good to think of Unix IPC as message passing, conceptually.

> If that’s right, then what about Apache? Wasn’t the point of 2.x that it moved to a thread-based model over the process-based model of 1.x? And if so, then… what’s up with that? I’d think there were a ton of very smart, very unixy people involved in that effort; they must have had some good reasons for the switch, right? Or, more likely, I’m missing something.

Process-per-connection — the model used by unicorn and the echo server example and apache’s preforking mpm (I think) — falls down at a certain level of high concurrency. There’s too much overhead in processes to solve C10K (http://www.kegel.com/c10k.html) problems. At that point, you basically have to use native threads or async/events. This doesn’t mean process-per-connection isn’t the right solution for a wide range of problems. Further, the basic technique of forking and sharing a socket between processes underlies async/event solutions — it’s just that each process can handle more than one connection in parallel (without threads). High concurrency servers that use fork(2), a shared socket, and async IO include nginx, memcached, lighttpd, and HAProxy. High concurrency servers that use threads include Apache, Varnish, and Squid. All are stable and production proven.

Here’s what I’m personally trying to forward in all of this: people confuse the idea that threads can always be applied with the idea that threads are always *right*. Just because a general solution exists (threads) doesn’t mean a more specific technique that uses processes isn’t the better solution. I really like process-per-connection where I can get away with it (and Unicorn is a perfect example of where you can get away with it) because it’s so amazingly simple; much simpler than threaded code and definitely much simpler than async/events. I like processes + async IO for highly concurrent systems because you just can’t beat it performance-wise and I personally enjoy working with event-based code. But threads? I can’t think of a problem I like solving with them, even though they can be used to solve a wide range of them.

Thanks,

Ryan

As you can see, Ryan was extremely generous with his time in responding to my email. Thanks Ryan! I learned a lot from this exchange, and I really appreciate it. Congratulations on your new position at GitHub, it’s clearly a perfect match.

You’d never know it based on our [now previous] website, but Arc90 has a long history of consulting success. Companies we’re currently working with or have worked with in the past include Insight, Knovel, McGraw-Hill, GradeGuru, AppFirst, The New York Times, and Wolters Kluwer, among others. So is Arc90 a consulting firm, focusing on strategic consulting, design, and Web development?

And then there’s Kindling, our idea collaboration tool and our first product to market. Kindling is currently being used by organizations large and small: The US Department of Veterans Affairs, LeapFrog, Medtronic, AOL, Symantec, and many more. As far as these customers of Arc90 are concerned, we are a product company responsible for supporting Kindling. So is the future of Arc90 product development – is Arc90 a product company?

Yes. Yes. And yes.

Arc90 is all of the things described above – an R&D shop, a strategic consulting firm, a Web design/ development company, and a product company.

Five years ago, the company was born out of rejection – a rejection of the hostage situation that is often corporate IT, a rejection of the business model of the turn-of-the-century Web company, and a rejection of development-lead software efforts. The company attracts individuals passionate about technology and the Web, those that don’t fit the mold of the 9-5 corporate culture, and those that love what they do and don’t view work as work.

The business has grown steadily over five years, adding clients slowly but consistently, mostly through the scattering model. During this period, we viewed our Web presence as sort of a placeholder; it said nothing about the company and wouldn’t educate the uninformed about our services. A visit to Arc90.com over the last five years left the visitor confused, at best. The site created a fun mystique around the company, but as time went on, we’ve become more visible and this became a problem. It was time to introduce the world to Arc90.

So I’m very happy to announce the relaunch of Arc90.com. Throughout the stages of building the new site, through concept, design, and production, we’ve tried to focus on one specific use-case: a visitor trying to answer the elusive question…Who is Arc90?

This was evident when an e-mail about a company picture day turned into an inter-office Photoshop session.  When everyone was informed that they would be photographed for a redesigned Arc90 site, it brought back memories of elementary school, picture days, and, of course, laser backgrounds.

Check back here for a new and improved Arc90.com (including a “Meet the Team” page) within the next several weeks. Until then, enjoy the remastered photos of Rama Poola, Chris Dary, Rich Ziade and Tim Meaney.

Tyler Gaw worked on the photos of Rama, Chris and Rich; Chris edited the photo of Tim.

To date, our consulting business has grown almost entirely by way of word of mouth. This is a logical and expected outcome for a company that’s done no overt marketing, but it is interesting to see that people who have worked with us in the past tend to work with us again and again. Among that group of people, one consistent theme does emerge: someone works with us, leaves their company and brings us along with them to their new gig. Call it the scattering model of growth. This seems pretty logical for the services business, if people see you as effective in terms of product strategy, design and implementation, they’ll likely join another company with similar needs and will seek partners that have proven their worth. Often, selection of a strategic consulting partner is a very personal decision, and thinking back to the good work that people have done for you is a much stronger connection than an ad in a magazine or the like. In this sense, Arc90 is much like a meme, as my coworker Avi pointed out. Through experience working with us, people get acquainted with the idea of Arc – design-driven development, being passionate about the Web, etc – and carry that forward with them.

So that’s the consulting business, where it’s not uncommon for growth to occur based on pleasing some people at a client where some subset of this group will eventually leave and take your firm with them to their new company. But does this scattering model work for growing a product? I’m convinced that this is a huge potential strength for our idea management and innovation application, Kindling, one that will pay dividends for years to come. Employees working at companies using Kindling is a constantly growing list. Some percentage of those employees, we like to think a pretty high one, will really connect with the product and see the value it creates. Some subset of those people will change jobs in the next year or two, and some subset of those people will pull Kindling along with them into their new company. That group may very well turn out to be our best salespeople.

In thinking about the scattering effect on Arc’s business, I was reminded of an article from a recent issue of Wired, where the economic value-creation of turnover was discussed. Paul Boutin wrote:

“Job-hopping, rather than climbing the career ladder within a corporation, facilitates flows of information and know-how between individuals, firms, and industries. When combined with venture capital, it supports unanticipated recombinations of technologies and skill.” In other words, we have Twitter today because a bunch of engineers who were trained at other companies quit their jobs and brought their expertise to Evan Williams’ side project. It’s like biology: In an ecosystem where microbes are promiscuously swapping genes and traits, evolution speeds up.

That’s exactly the right metaphor – individuals leaving their companies and bringing with them the best parts of their previous role, while leaving behind the frictional parts, is much like the selection process of evolution.

Recently a few people have left Arc after a long time of nearly no turnover. While I’m not happy about it – the people that left are very talented and took a bunch of knowledge out the door with them – I’m an optimist, so I’ll view it as a sign that the job market is loosening-up. An improving job market means more scattering, and more scattering is good for us all.

Note: thanks to flickr user Pensiero for the image.

A few weeks ago, I blogged about the dreaded 1000+ unread items staring me down on Google reader. It’s a deflating feeling to constantly be confronted with so much stuff that I’m failing to catch up to. In reality, I don’t want all this stuff. I just want the good stuff. Of course, technologists react by trying to develop a vaccine that is part smart algorithm (see Techmeme) and/or part social tagging/voting/whatever (Digg and Delicious).

In other words, we want an electronic version of an editorial staff. Well, maybe what we really need is help from.hold onto your hats.other human beings. Maybe we need a way to establish a circle (or circles) of trusted people – friends, colleagues, higher-profile people we know that consistently suggest interesting things – and give them a way to help us cherry-pick the stuff worth focusing on.

The above is the premise behind our proposed panel for next year’s SXSW Interactive. It’s called The Revenge of Editorials. If you think it’s worthwhile, you can help make it a reality by voting it up (speaking of information overload, there are over 2,200 panel suggestions for SXSW).

Speaking of humans (and their inherent value), Behavior Design’s Chris Fahey has another interesting panel called The Human Interface (or: Products are People, Too!). It’s another panel I’d love to see become a reality. The brief pitch:

More and more, users are interacting with web sites and software on a conversational, physical, psychological, and emotional level — just like we’ve always interacted with other people. UX designers, then, must stop thinking about interfaces as dumb control panels and begin using technology to envision interfaces (literally!) as human beings.

Go humans!

From a sales perspective, we’re looking for someone that helps both generate new leads through various means and capable of following up and walking prospective clients and customers through the decision-making process. Ideally, this person has experience with helping build partnerships and target customers for Web-based applications and Web consulting services. It’s important that you understand the viral, social and marketing facets of the Web. We also expect you to help us refine and hone our roadmap and marketing message for both ourselves and our products.

Broadly speaking, we’re looking for someone that is passionate, outgoing, energetic, proactive and willing to wear various hats to help us succeed. If you think you’re the right fit for the job and are interested at working with one of the coolest design and technology shops in New York City, don’t hesitate to get in touch with us.

I had a text file that I wanted to explicitly track versions of — I use Time Machine for incremental backup, which is great, but doesn’t always give me precise enough control of the snapshots, and I don’t always have the snapshots available. Once I read Aristotle’s essay, it only took me a few minutes to download and install Mercurial, and then I started tracking my file with three simple commands: hg init; hg add *; hg commit -m "initial". Quick, easy, and effective. From that point on, my folder was also a repository, and I could track specific revisions of my files by simply typing hg commit -m "update" at any point.

What I think is the killer realization here is that it is now super easy to make any folder into a repository, and instantly start tracking versions of its contents. I’ve used Subversion to make local repositories, and track local files in the past — and it really is many more steps, and much more work. It’s also messier, because the repository and the working copy must be separate entities — you have to think about where the repository should live, as opposed to the actual stuff, the working copy. With a DVSC, no such dichotomy exists. The repository and the actual stuff are one and the same. You can move them, zip them, email them — as long as the metadata directory is preserved (for example, .hg or .git), the folder remains a repository.

I’m thankful to Aristotle for helping me get past the initial barrier to entry, and start actually using a DVCS. I’m looking forward to learning more about them, and using them even more going forward.

(Bonus tip: on OS X, some applications save their data using a special type of file called a “package”, which is actually a directory with a special flag. One example is OmniGraffle, which I use frequently. Because these “files” are actually directories, they can be easily made into self-versioning files with their own self-contained repositories. Just navigate to the directory and run the commands above! From that point on, the file will be a repository and can track its own versions. Pretty useful!)

(I can’t explain exactly why I decided to try Mercurial first, before Git or Bazaar. I’ve read many comparisons of them, and something about those comparisons just made me want to try it. I’m not sure exactly how, but I had built up the impression that Mercurial was somehow simpler, cleaner, more elegant, and more approachable than Git or Bazaar. I don’t know if that’s actually true, but the impression was enough that that’s where I started.)

(Aristotle’s mention of “mental overhead” as a decisive factor in using one system over another is something I’ve been thinking about lately as well, although I’ve used the more unwieldy “cognitive overhead.”)

The problem is, in the real world, things change and by the time you’re done, you’re often pretty far off the mark and usually the first version isn’t right anyway – so you end up making something 2 years late and a hundred features off target. With agile development, you test, evolve and stay in tune with your users and let them guide you. You can also test and refactor more easily because each "story" or feature is smaller, tested and easy to isolate and remove/change. (Or should be.)

One of the principles we try to evangelize at Arc90 is the need to iterate, iterate and iterate some more. Implicit in the quote above is a not-so-subtle confession: it’s hard to get it right the first time around. I’ve often heard usability experts humbly concede that good product design “gets it 30-40% right the first time around.” That’s in the case of good design. How do you get to 70%? You keep questioning, refining and tweaking until you hit just the right notes.

To do this, you have to be equipped for it. And by “equipped” it isn’t only about adopting a particular development methodology. It’s about rebooting your culture, your value system and your tolerance for getting things wrong and accepting risk. It isn’t easy for organizations, especially more established ones, to make that shift.

Most conversations around agile development focus on the “development” part of the discussion. In our experience, we’ve learned that the agile process really shines in evolving the product itself. In other words, agile development provides the roaming ground for sound, nimble design. By being agile, you’re conceding that there is no longer a rigid, stepped process, but rather designers and builders working together and refining all along. It can feel messy at times, but the outcome is almost always invariably better.

Even if you don’t have the luxury of releasing to a broad user population, iterative design brings clarity and direction to even the smallest of teams. Requirements and wireframes are helpful, but nothing beats touching, playing and most importantly validating the real thing. At Arc90, we often call it “prototyping” but that term short-changes the process. What’s really happening is an inching towards a full realization of the real product. It’s all about the small, seemingly obvious in hindsight discoveries that elevates a design to greatness. To find such discoveries, it requires an open mind and a stomach for failure (albeit little failures). Thomas Edison said it best:

I have not failed. I’ve just found 10,000 ways that won’t work.

Instead, I arrived at the show’s Facebook page, which includes updates by the hostess herself. Viewers can also submit questions to celebrity guests, vote on which songs they want bands to play when they appear live, and provide general feedback.

Although an unofficial celebrity endorsement and an appearance on Oprah hurtled Twitter into the mainstream earlier this year, Ms. Chung’s show is very different. MTV’s dual partnership with Facebook and Twitter generates buzz on-screen in various forms, including tweets, content sourced from Facebook profiles and fan pages, audience contribution from polls set to remixed YouTube videos, and round-the-clock updates from Chung’s own Twitter account. Alexa Chung is both a milestone in the convergence between TV and the Web and a fresh infusion of innovation for a TV network that should have caught onto the social media craze a lot earlier.

If you wanted to get a message on MTV a decade ago, your best bet was to write it on a poster and join the cheering crowds outside the network’s New York studio during a TRL taping, in hopes of finding your way into the scope of the cameras sweeping overhead. Lucky viewers could call in for the chance to ask a question to celebrity guests on-air.

MTV canceled TRL earlier this year, after the network’s reputation as a hub of pop-culture influence had long since started to fade. New music was being discovered on MySpace pages and niche music blogs. Popular clips from last night’s TV shows were swapped via YouTube and Hulu links in instant messages and posts on Facebook profiles, defeating the long-standing champ.

With Alexa Chung, MTV has made a totally different show. The iconic TRL set has been converted into something that looks like a loft apartment, with the studio audience scattered around like party guests. There is no countdown. Instead, a talk show format inspired in part by late-night programming, with topics ranging from movies and music to the latest YouTube sensations (whom Chung plans to regularly bring onto the set to see if their singing, dancing, or other oddball talents are for real).

And, of course, the screaming crowds in Times Square are all at home. In their place is the silent, yet deafening, sound of tweets.

One of the things that surprises people about Arc90 is that our development team is right here in our offices in New York City, alongside our project leads, designers and strategists. We’ll often get a “That’s interesting” or “Oh? That’s unusual.” It’s a response that usually leads into a conversation about the nature of conceiving, designing and ultimately building great software.

At Arc90, We Feel Your Pain (In Fact, We Need To Feel Your Pain)

The single most important benefit of building and designing near our clients is, well being near our clients. If you want to deliver something great, team proximity is everything. To get it right, we need to not only have a tight feedback loop with you, we want to become you. This means internalizing all the experience and expertise you bring to an engagement. We don’t just want a “discovery phase.” We want to inherit your instincts, fears and innate knowledge of where things need to go and what pitfalls lay ahead. That’s hard to do across two oceans.

It’s also hard to compartmentalize that dynamic into any single phase. The challenges and unexpected puzzles crop up throughout. This isn’t only about unforeseen “problems” but the inevitable “how can we do this better?” challenges that we can’t help but ask. It’s great to have the client nearby to collaborate on elevating something from passable to truly compelling.

The Value Of First, Second and Third Drafts

When any decision you make needs to be shipped 11,000 miles, it better be the right decision. As we form a strategy and design out an attack plan for an effort, we know it’s going to morph and change and we know there will be missteps along the way. What may feel absolutely sublime in a design comp may fall flat in a prototype. What may shine in a prototype may fizzle out when it’s wired to real data.

We want to be able to make decisions, test decisions and change course, however slightly, as we go. It’s nearly impossible to do that if we’re shipping chunks of an effort in packages halfway across the world. You may well have the most talented group of developers in some far off place, but if they don’t appreciate the rationale and the overarching context of why things are the way they are, you’re going to pay a heavy communication tax.

Ready For This? In-sourcing Is Cheaper

It is a myth that outsourcing is a cheaper way of building quality software. Software can always be built elsewhere, but the basic premise that the early blueprints are 100% dead-on are a fantasy. Once an effort begins, almost invariably, a rat race ensues and the cumbersome task of project managing across geographically dispersed teams kicks in. The result is a frustrating, oftentimes painful endeavor. We’ve seen firsthand how an out-of-synch outsourcing effort can spin out of control and destroy morale. It’s a painful thing to witness.

Great Product 101

Great product hides away all the pain and anguish associated with its inception. It feels organically grown, rather than a visible patchwork of exposed “systems.” To build great product – product that feels intuitive, thoughtful and in tune with the needs of the user – you need the freedom to iterate and be innovative. The tighter the loop, the more room for refinement.

For some efforts, outsourcing may be the right approach. If you can afford to have your product management team, project leads and designers in the same physical location, it can work. If your product is so elaborately defined or prototyped to extreme detail, it may also work. Few companies have either the time or resources to go to such great lengths. Instead, we trudge forward with the resources and time we’ve got.

Ultimately, it’s about bringing product leadership along for the ride. It’s about pairing up the right people – the people that care about the outcome and the people that get you there – all the way through from inception to creation. One of our clients summed up their own struggles with outsourcing: “They just don’t get it.” It wasn’t meant as a jab towards particular talent pools out there. It’s meant to illustrate how truly difficult it is to get everyone in sync throughout an effort.

The world truly is a flatter place today. We are more connected than ever. But connectedness alone doesn’t lay the groundwork for building great product. Until some killer tool comes along that provides a new, more effective way to work together, proximity and tight collaboration between all the players remains the most effective way to build great things.