Paul Stadig

Reading & Writing

2025-12-23T04:58:00.000-05:00

Leslie Lamport knows a thing or two. He's a Turing Award winner and researcher at Microsoft. He has written influential papers on distributed systems and formal verification.

Leslie Lamport says you should learn to write well, because people will judge you by your writing, but more importantly it is the key to thinking well. The better you write the better you think the better you write.

But there's something Leslie didn't say, he talked about why to write well, but he didn't say how to write well except that writing well comes with practice, which isn't horrible advice. (The clip is only two minutes long, so cut the guy some slack!)

To write well you must read well.

Moreover, reading, thinking, writing, thinking, reading is the virtuous spiral of learning. Reading gives you ideas to think and write about. Thinking generates new ideas and integrates them with what you already know. Writing tests and refines those ideas. Each iteration you are just a little better, a little smarter for the effort.

I want to read and write more...a lot more. I've been learning to improve my reading and writing habits.

Reading

People read more and more every day. People read Twitter and Facebook and Instagram. People read mommy blogs. People read emails and texts. But this is not a balanced diet. Much of what we read today is short-form, polemic, snark. It is banal. There can be a place for that, but when fed only junk food your mind will get soft and flabby. Your mind needs to get to the gym.

When I say I need to "read more," what I mean is I need to "read more long-form...mostly books." I'm also thinking mostly of non-fiction. Yes, you can acquire information through YouTube and podcasts, but that will bias you toward recent ideas and a select group of "content creators" with specialized skills and tools. You can't easily skim a YouTube video. YouTube and podcasts are no substitute for books.

Through a book you can learn from people who have been dead for 1,000 years. You can learn from kings as easily as from ordinary people who had only pen and paper. You expose yourself to possibilities outside your sphere of experience. There's a wealth of knowledge available in books, and they aren't all digitized.

While you are learning something else happens, reading (whether fiction or non-fiction) improves your language skills like spelling, vocabulary, and grammar. You can absorb different writing styles from poets, novelists, journalists, mathematicians. From this soup of styles your own writing voice will emerge.

These benefits will only come to you if you read well.

Read with pen in hand.

Research indicates that by engaging with your books you will retain more information longer. Read with pen in hand. Some use their pen for marginalia, but a more effective use will be to write the things you learn on a separate piece of paper in your own words.

To do this you need to summarize the most salient ideas. As you read periodically pause and quiz yourself. What is the author saying? How does this relate to other things you know? This will help you internalize the information as a connected graph of knowledge in your mind. The more connections an idea has the more meaning you impart to it, and the more "handles" you have to grab it and pull it out.

Both for summarizing in my own words and for connecting ideas I sometimes wait until the next day to summarize my reading, without referring back to it. This allows my mind to organize, connect, and highlight the ideas that are important to me. Sometimes I may even re-read something few times over the course of a couple of days to let my mind extract more. (Perhaps only for books that are fertile ground for interesting ideas.)

But after you decide what is important, write it down. You will make best use of the ideas if you keep some notebooks or use a note-taking app. Even if you just throw the notes away you will help your mind retain and comprehend ideas by using your own hand to write the ideas in your own words. You can do this by reading with pen in hand.

Read widely.

The more you connect an idea to other ideas the more valuable it becomes. The more you connect an idea to other ideas the more meaning it receives and the more meaning it gives. Reading widely is a great way to fill your mind with a diversity of ideas.

A recommendation system doesn't help you seek out new ideas. You have to do it. One way to do this is to go to the library, walk up and down the isles, and flip through any book that catches your eye. You may pick up only one idea. That's great! (I'll talk more about how to read ruthlessly later.) Browsing is a fine way to break out of your interests.

Sometimes you can get good recommendations from friends, family, acquaintances. Their interests may be similar to yours, but chances are they are not exactly the same. Ask: "what is the most interesting book you have read recently?" You can try asking this of people you know have very different interests from you. Perhaps even start a monthly email thread with the topic "what have you read recently?" If your friends, family, acquaintances have a hard time answering the question, maybe they'll be encouraged to read more. :)

Reading widely can take a certain amount of humility, especially when you read about ideas that directly oppose your cherished beliefs. You don't have to change your mind, but try to empathize with the people behind the ideas, their struggles, and their experiences that lead them to believe this thing. Think about the light each idea sheds on all the other things you know, and the value you will get by adding to the richness and subtlety of your knowledge web.

A rich web of knowledge will help you find creative solutions to the problems you face. I believe the most revolutionary ideas are born from the unexpected combination of seemingly unrelated ideas. Read widely to fill out that knowledge web.

Read deeply.

Breadth of exposure is helpful for your mind to draw connections between ideas, but so is depth. Your mind is a pattern matching machine. If you give it two ideas it will quickly suss out the similarities and differences.

So if you want to understand the fine distinctions in a subject, read topically. For example, if you want to study the Bible, don't look for the "best" translation, get them all. If you compare passages among translations you'll quickly get to the root of translation disagreements, and all without learning a lick of Greek or Hebrew!

When you read a book on a subject, use it as a starting point. Look through its bibliography and sources cited, and go read them. Use the recommendation system at Amazon to find related books. Go to the library shelves that contain a book and check out those near to it. Ask people for recommendations of books similar to ones you've read.

As you load your mind with information, it will go to work. It performs magic while you sleep, or while you daydream. Let yourself get bored. Let your mind wander. You'll think about ideas, how they are the same, and how they are different. Then write it all down. Reading deeply will give your mind plenty of fodder.

Read ruthlessly

If you feel your reading list getting much, much longer, then I have good news: you don't have to read every word. You don't even have to finish a book. An author has put a lot of work into a book, and you should accept that gift graciously, but the author has no right to demand your time. Unfortunately "speed reading" doesn't help here. You should learn to skim, and be OK with not extracting every idea from a book.

When reading broadly, the point is to find new ideas. If there's a book that looks interesting pick it up. Read the summary. Look through the table of contents. But if the summary and the table of contents don't induce you to read more, then put it back. If you get half way through the book and lose interest, put it back. Take one idea, and put it back. If that idea tickles your fancy, then make it a subject for a deep read.

Ironically, reading deeply also does not require reading every word of every book. As you read topically you'll realize that there's overlapping content. Your first book on a subject may require special attention, but after that you can bound through a book in search of its unique angle.

You alone will have to make an accounting for how you spend your time, and you only have so much of it. Define your reading program based on your own goals and interests, then: Drop. Books. Ruthlessly. Even if it is one that you originally thought would be valuable.

By being ruthless, you will focus on the most potent ideas and build generous knowledge web. No one else will have your unique combination of ideas. Now you must write about them.

Writing

Software engineers undervalue writing. Probably this is so in any career. If you write well (emails, chats, texts, pull requests, documentation, ...), you will distinguish yourself. In addition to distinguishing yourself at work, in this internet age you can improve your chances of being discovered. You can write a niche blog and find an audience. You may not find thousands of readers, but you only need one reader to change your life with a job, a speaking engagement, an opportunity.

I could say a lot of things about writing well, but as I've read about it I've seen four themes: economy, action, warmth, reading.

Economy

Writers advise to avoid adverbs and adjectives, to remove qualifiers ("I think," "some," "many," "possibly"), to avoid cliches, to avoid fluff. Most of this advice reduces to using words economically. Say what you want to say briefly and clearly.

If some slip through, that's OK, because the editing process is 110% about removing words. Pascal famously said "I wish I could have written a shorter letter, but I did not have the time," because concision is hard. You will re-write, re-write, then re-write some more, and it should get shorter each time.

Action

I find this one really tough. As an engineer I usually describe machines and processes. I find it easy to write in a passive voice. (The last sentence was "It is easy for me to write in a passive voice.") When you write passively, you see state of being verbs. You end up with a pronoun as a subject, and the true subject later in the sentence. It's all a bit twisty for people to decode. You may find value in tools like writegood-mode for Emacs.

Warmth

William Zinsser's book On Writing Well is a worthwhile read. In it he advises writing with warmth, which for him is writing about the human element. He reminds me of the great preacher Charles Spurgeon who no matter the passage, Old Testament or New, he would find his way to Jesus. No matter the subject, William will always find his way to a story about people.

Writing about people is one way to achieve warmth. Another way is to write honestly with your own voice. Some people adopt an artificial voice when they write. Perhaps they sound too formal. Find your own voice and be true to it.

Reading

There are two ways that reading is important to writing well. First, reading what others have written will improve your language skills and inspire you. You'll develop an ear for good writing. Second, reading your own writing will help you re-write it. It should sound good to the ear but not in a flowery way, and it should sound like you.

Thoughts on the Death of Brook's Law

2021-05-31T11:32:00.004-04:00

Brooks' Law is wrong.

Brooks' Law (the pithy version) is: "adding people to a late project makes it later." There is some truth to that, but it's more nuanced. A bad manager staffs up a late project (in an act of desperation?) to make a deadline. However, Bertrand Meyer (and Steve McConnell and Barry Boehm) believe—based on empirical evidence—that judiciously adding people to a project can shorten its schedule.

You can shorten a schedule by adding more people...however there's a limit. You can only shorten the schedule for a software project by (up to) 25%. Since adding people to a software project means adding cost, what this is really saying is you can spend more to get your thing up to 25% more quickly. However, the reverse is not necessarily true. Sometimes you can take people off a project, give the remaining people more time, and get your thing for less money; sometimes you cannot.

This has been known for about 20 years and yet Brooks' Law has survived as a folk wisdom, and Bertrand Meyer wants to get the word out. That's why he wrote "In Search of the Shortest Possible Schedule."

I grew up on Brooks' Law, so I'm trying to absorb this. Brooks' Law seems right to me, and in "Brooks' Law Repealed?" Steve McConnell describes an experience that seems familiar:

To those of us who have been around the software-project block a few times, the claim feels true. We’ve participated on projects in which new people are brought on at the end. We know the irritation of having to answer questions from new staff when we’re already feeling overwhelmed about our own work. We’ve seen new hires make mistakes that set the whole project back. And we’ve experienced additional schedule slips even after staff has been added to a late project.

McConnell squares this experience with the new sans Brooks' Law world by pointing out Brooks' Law does apply in certain circumstances, but projects are poorly estimated and poorly tracked. The result is not knowing whether you are in the Brooks' Zone or whether there is enough project left for new hires to pay off the productivity lost to training them.

I'm not sure I buy that. I think the idea of estimation is fundamentally flawed. I can't help but feel like this is saying, "We're doing a bad job. Do better!" I'm more and more convinced that breaking work down, estimating the pieces, and rolling it back up is a terrible way to estimate. It fails to account for variability, and padding estimates is not the solution.

And sure, better project tracking seems like a good thing. It is a necessary first step, but having the data isn't enough, you also have to interpret and extrapolate it. Probably the best thing that can be done with tracking data is to let it empirically drive estimations.

I cannot deny the empirical evidence. You can pay more to shorten a project by up to 25%, but there are some intriguing questions that pop up: Why? Why 25%? Why doesn't it always work backwards? Could you repeat the process with a revised schedule and cut another 25%? How would knowing about this bias estimations?

I think what I take away from this is that Brooks' Law has narrower application than I initially expected and adding people to a project can bring it to completion more quickly.

On Writing (Code)

2019-11-22T08:12:00.000-05:00

Introspection into my own process for making software has led me to believe that writing prose and programming are fundamentally the same. This was further highlighted for me when I read Stephen King's On Writing. I saw interesting parallels between the way he likes to work and the way I like to work. Also my code is a horror story.

King likes to write a first draft as quickly as possible. He does this to get the story out. He doesn't worry about character development or even holes in the plot. Those he will fix up in the second draft. The first draft is about getting onto paper the good bones of a story. Then he lets the first draft sit for a couple of weeks.

When he can come back to the first drift and it looks familiar but not quite—like it was written by his doppelganger—then he has enough distance to make the second draft. That's where any story issues are fixed and things are smoothed out and tightened up. His goal is to make the story 10% shorter.

When I write code:

I like to write code breadth-first so I can see all the moving parts and that everything fits together.
I like to let it sit for a couple of minutes, or hours, or days—whatever I can afford.
I like to smooth things out and tighten things up in the second draft—things like function names, refactoring, etc.

Maybe writing code isn't exactly like writing prose. Maybe it's more like knitting or woodworking or cycling or solving crimes. I think part of the reason that programmers tend to see their work in everything, is that programming is really just process for solving problems and giving form to thoughts. Those skills can be applied to many different domains. Maybe in that way it actually is more like writing than other hobbies?

At least I didn't say that programming is like gardening. I actually appreciate gardening as a hobby because of the fact that it is entirely different than making software!

How to Survive Life Undamaged

2019-11-12T18:15:00.000-05:00

How does one survive life undamaged? Seriously... if you figure it out let me know. I particularly mean self-inflicted damage. You can't guarantee anything about how other people act or think.

Here is what I have tried (am trying):

1. Be willing to listen to others.

You don't have to listen to everyone, but be aware that you risk losing valuable perspective and insight, even if it is insight about how not to do things. You should probably be willing to listen to some people that you have shut out. If you are not uncomfortable, then you are not growing.

Everyone has a story. You can either imagine a person's story, categorize and label them, and treat them as if that story is true, or you can actually listen. You can listen to what someone believes while thinking about why they're wrong and how you will show them, or you can try to inhabit their space and see the world the way they see it. You don't have to stay there, but you will take lessons from it.

2. Be skeptical by default.

To listen to others and inhabit their space does not mean uncritically accepting their ideas. You can be empathetic, you can entertain an idea, without being swept away. You can also question if a person is wrong without imagining they are your enemy.

If some idea pushes through your skepticism, accept it. It is OK to move a little closer towards an "opposite" view.

3. Don't try to convince everyone.

This can be particularly painful when it is someone you respect and/or love. It may take time. Or it may never be. Life is sour sometimes. Don't make it more sour than it needs to be.

If you are skeptical by default, then maybe others are as well? If you are listening to others and inhabiting their space, then you must know that it will not be easy to convince everyone. You can let that bother you. You can turn it into an obsession, or you can focus on the good (what little you estimate there to be) and seek friendship.

These are three things you can try. If you refuse to listen to others and make it your life goal to convince everyone how they're wrong. You will have a life filled with stress and bitterness. Or you could listen--critically--and seek friendship over rightness.

Beyond Techniques

2019-11-06T20:24:00.001-05:00

I firmly believe that writing software is a creative act. I believe that a construction project---or any other linerally presenting process---is the wrong analogy for software development. A much better one is writing prose, which is a process of writing and rewriting and rewriting, and sometimes throwing it all out and starting over. In the same way a software engineer names and renames, factors and refactors, etc., then extends the functionality by doing it all again. This all has to fit into the larger code base in a way that makes for a consistent whole. This requires judgement and taste.

To come at the argument from the other side, if you spent your days rewriting the same code over and over, then you would split it into more generic functions and perhaps a library. If you're solving the same problem that someone else has solved, then you would just reuse their software. This does in fact happen, and yet software engineers still get paid to work, and the value they generate is in the new stuff that requires creativity. Sure, there are some inefficiencies that mean people rewrite software that they could have just reused, and you could argue that the new things to be done are increasingly marginal, but on the whole software engineers would be out of work if there were not new things to create. Ergo writing software is a creative act.

The next reasonable question is: how does one create? I believe that fundamentally, and on average, one cannot directly induce creativity. This isn't entirely satisfying. I wish creativity was a predictable, mechanical process. Is it possible to discover and categorize tools for thinking? I'm still exploring that. Here's an example: to get a fresh perspective on your problem think not about how to solve it, but how to avoid failing to solve it. Instead of trying to figure out how to build a stronger bridge, maybe you'll gain insight by considering how to avoid building a weaker bridge. This technique for finding fresh perspective...does it not make creating into a predictable, mechanical process?

I actually like techniques. I believe the learning process is (or is best) technique based. Think about learning to cook. You start by following the recipe. Then you start to improvise. You understand that this ingredient is for flavor and can be adjusted to taste, but that ingredient provides aeration and messing with it too much will produce a dense, inedible result. Eventually you move beyond techniques and start to construct your own recipes from first principles. A poor education either starts with first principles, which a beginner does not have the experience to appreciate, or focuses too much on techniques divorced from context, which frustrates learners because they don't know how to properly apply techniques. I intended my book Clojure Polymorphism to position itself as a way to explore varied applications of the same tools and techniques, which I hope will guide readers from knowledge into wisdom.

The writing curriculum we use in homeschooling our children is technique based. A child reads an essay and creates an outline of the important points (there's a whole set of techniques for that). Then he will rewrite the essay from the outline applying techniques like "ly"-words, openers, and clinchers. This removes the hand wringing about what to write about. (Incidentally, if you read Ben Franklin's autobiography, this is the way he improved his own writing.) What if computer programming was taught this way? Here's a program that consumes a CSV file, changes some things, then writes it back out. First write out a high level algorithm for the program, then rewrite it using 3 of the 15 techniques you've learned (concurrency, multimethods, asynchronous channels, etc.). Repeat. I think this would produce wiser programmers.

However, eventually you need to move beyond techniques. Leaning on techniques is just shifting the problem elsewhere. Coining techniques is an attempt at mechanizing creativity, but instead of making creating mechanical, you are now faced with choosing which technique to apply to generate a creative solution. Now the creative leap must occur earlier in the process. You can try each technique in a brute force search, but I doubt many (if any) would consider that "creative," nor is it particularly efficient. The experience of repeatedly applying techniques should help you develop judgement and taste, so when you're faced with inventing from first principles, you have some gut sense or guiding aesthetic. That guiding aesthetic is not some set of rules for when to apply which techniques. If it were, then it would become a "library," so to speak. Anybody---or any machine--could evaluate the rules and perform the steps.

Techniques are great for broadening your learning, but they are not self applying. Learning all the techniques does not help you invent new techniques, nor will it induce creativity. You can know how to cut impeccable dovetails and square up lumber and drill straight holes, but that does not mean you will create beautiful furniture.

How does one develop this guiding aesthetic? You read a lot of other people's code. You write a lot of code. You read a lot of books. You work hard. Sorry, that's the best I've got. This is a continuing journey...to be continued.

However you do it, the end goal should be to move beyond techniques.

Virtual Machine Oriented Development

2019-10-30T17:44:00.001-04:00

Most computing devices that we have today---desktop, laptop or phone---are capable of computing any program that can be computed. There's a bit of equivocation there. What is meant is that anything that a human can manually calculate via rote, mechanical process can also be done by computer. This is the Church-Turing Thesis.

I've never really stopped to think, but what would a non-universal computing machine look like?

....

Longevity

Several years back I suffered a bout of jealousy. I thought about engineers in other fields who build roads or buildings or even cars. A civil engineer can imagine something they've build standing still amidst the blur of 100 years passing by. An automotive engineer can imagine a car they've designed still driving the roads in 20 years.

I don't think a single line of code that I wrote 3 or more years ago is still in production, and that's ignoring all the code that I wrote that never made it into production.

These are the kinds of things you start to ponder as you reach the ripe, old, programmer retirement age of 33.

But then a funny thing happened. I played Sam & Max Hit the Road ... on my Android phone.

Here was a game released in 1995 and I was playing it on my Android device in 2014. How did that happen? Well, when LucasArts designed the game "Maniac Mansion," they decided to create a scripting language and write the game in that language, and they used that scripting language for many of the games they made. I have several original LucasArts games on CD. Some are PC versions some are Mac versions.

Over the years as I feel the nostalgia hitting me I'll grab the game files from the CD and download ScummVM for whatever platform I'm on at the time. I copied the game files to my phone and downloaded ScummVM from the Play store. That's how it happened.

....

Data is Code

I had been exposed to Lisp, and even written a lot of Lisp before I finally had my enlightenment about macros and metacircular interpreters. I remember vividly reading Structure and Interpretation of Computer Programs and seeing Scheme put to use creating simple yet powerful abstract interpreters. The author's start "simply" with interpreters that add new programming paradigms to Scheme. Then they proceed to simulating a register machine and writing an assembler and compiler for it. This happens in the last chapter, a space of ~100 pages.

It is a Divine joke that Structure and Interpretation of Computer Programs and The Art of Computer Programming had their titles swapped, because---while I don't wish to denigrate TAOCP which is a depth of amazing riches---SICP is about art, and in a metacircular way it is art.

It is too easy as a Lisper to understand the world this way, but data is always code. In Forth, 5 is not a number, it is an instruction to push the value 5 onto the top of the program stack. Your program receives a program as input. It receives files, network packets, key presses, and mouse clicks. It interprets this program and produces output.

A PDF file can cause a buffer overrun in a PDF reader because each byte is literally an instruction to your program-as-interpreter to "write a value at the current location and move to the next location" (or at least it can be if your program-as-interpreter has flawed semantics).

This is not a property of Lisp, it is a property of the stored program computer, Universal Turing Machine, von Neumann architecture. Code and data are made of the same stuff and stored in the same memory.

....

The Non-Divine Joke

In his talk "The Birth & Death of JavaScript," the 2014 version of Gary Bernhardt extrapolates where JavaScript and asm.js will take the world in 2035 (after an apocalyptic global war, of course). The punch line is that instead of JavaScript running on computers, computers run on JavaScript. This happens through a comical stack of emulators emulating emulators that emulate. Actually I think it's compilers transpiling compilers that transpile transpilers, but...same difference.

But like every joke there's a bit of truth to it.

Paul Graham writes about "Programming Bottom-Up" where you build the language "up" to your program to the point that actually expressing your program becomes somewhat trivial. You're building a domain specific language to solve exactly the problem you have. Again, this is all too natural for Lispers, but everyone does it.

The act of programming is to turn a universal computing machine into a limited computing machine. You build out data types and operations to focus the abilities of the computer into a specific domain. Programmers instinctively understand this, which is why we find it so funny that---in a twist of irony---a universal computing machine emulates a universal computing machine emulating a universal computing machine.

....

Virtual Machine Oriented Development

I started thinking about Virtual Machine Oriented Development because I was concerned about the transience of my legacy. I noticed that there were software products that were still around 20 years after they were written. I started seeing a VM underneath them.

But having thought about it more, I don't think that Virtual Machine Oriented Development is just about legacy. I think it might clarify the design process to be explicit about the fact that we're designing a limited computing machine that analyzes sales data. What are the data types? What are the operations? If you have power users, maybe they'd even like a scripting language that can describe which data to import and then how to analyze it?

You might find then that you've abstracted your problem into a computation model that will become valuable for years. Maybe you'll end up rewriting the interpreter for this language several times, and all the while users can keep using their existing scripts.

....

Conclusion

What does a non-universal computing machine look like? It looks like every program you've ever written.

Speed Reading

2019-08-07T13:41:00.000-04:00

My reading habits are lumpy. I find I'm either not reading anything, or I'm reading seven books at once, and when I am, I wish I could speed read. I have in the past read books on speed reading, and they usually boil down to techniques like increasing your eye span, eliminating regression, eliminating subvocalization, etc. The theory seems to be that your brain can work much faster than your eyes, and you just need to eliminate bad habits, and establish some better ones, so you can get the words into your brain faster.

I do feel like there's something to this. In my experience, my mind tends to wander as I'm reading, and sometimes it's easier to skim since that keeps my mind busier trying to assemble random bits and pieces into a comprehensive whole. That seems to be the idea with this new speed reading book that I picked up called "Speed Reading With The Right Brain." The author claims that by engaging your brain in conceptualizing what you're reading as you're reading, you increase comprehension, and it is through increased speed of comprehension that you achieve increased reading speed. I want to believe, but I'm still skeptical.

What I would love is some reference that approaches speed reading from an empirical approach, you know, with science. What do we know that actually works based on research? Well...you may not like the answer.

Speed Reading is Fake

In looking for an empirically backed approach to speed reading, I came across "So Much to Read, So Little Time: How Do We Read, and Can Speed Reading Help?" This article is based on decades of reading research and cognitive psychology. One of the authors—who passed away from cancer a few days after the first draft—proposed the article "because he felt that it was important to share the knowledge we have gained from experimental science with the general public."

While there may be savants who can read impossibly fast without sacrificing comprehension, controlled studies show that for normal people learning to read faster means comprehending less. When you learn to "speed read" you are actually learning to skim. "Taylor notes that [Evelyn] Wood 'repeatedly stated that her people are not skimming, but rather are reading' (Taylor, 1962, p. 65). Based on recordings of their eye movements, however, Taylor concluded that they closely resembled the eye movement patterns produced during skimming (Taylor, 1965; see also Walton, 1957)." Also from the article:

The speed readers did better than skimmers on general comprehension questions about the gist of the passages but not quite as well as people reading at normal speed. … The advantage of trained speed readers over skimmers with respect to general comprehension of the text was ascribed by Just and colleagues to an improvement in what they called extended inferencing. Essentially, the speed readers had increased their ability to construct reasonably accurate inferences about text content on the basis of partial information and their preexisting knowledge. In fact, when the three groups of participants were given more technical texts (taken from Scientific American), for which background knowledge would be very sparse, the speed readers no longer showed an advantage over the skimmers, even on general questions.

According to the article, learning to speed read may improve your ability to skim, but only for familiar subjects. This isn't necessarily bad news for two reasons:

You can improve your reading speed, just not as dramatically as speed reading advocates suggest.
Learning to skim effectively is a useful skill to learn.

Improve Your Reading Speed

The average person reads between 200-400 words per minute. What is the best way to improve your reading speed? Practice. Unsurprising. Perhaps a little disappointing? There are a couple ways that practice increases your reading speed, but they basically break down to improving your language skills:

better vocabulary
exposure to more writing styles

The broader your vocabulary, the more familiar you are with words and styles, the more quickly you can read. Your eyes fixate less on words with which you are familiar, so they move more briskly across the page. Your familiarity with style allows you to anticipate better how a sentence will end when you've only read part of it. Also, "written language uses some vocabulary and syntactic structures that are not commonly found in speech, and practice with reading can give people practice with these."

The more you do reading, the faster you'll get.[1]

Learn to Skim Effectively

Effective skimming is mostly about trying to extract structure and important ideas from a text. Scan for:

headings
paragraph structure
key words

According to the article, "Research has shown that readers who pay more attention to headings write the most accurate text summaries (Hyönä, Lorch, & Kaakinen, 2002)."[2] You can also do things like:

scan the table of contents
read the first paragraph of each section
read the first sentence of each paragraph

Again from the article:

The eye movements revealed that skimmers tended to spend more time reading earlier paragraphs and earlier pages, suggesting that they used the initial parts of the text to obtain the general topic of passages and provide context for the later parts that they skimmed in a more cursory way. Therefore, effective skimming means making sensible decisions about which parts of a text to select for more careful reading when faced with time pressure. In fact, Wilkinson, Reader, and Payne (2012) found that, when forced to skim, readers tended to select texts that were less demanding, presumably because they would be able to derive more information from such texts when skimming. This kind of information foraging is a useful method of handling large amounts of text in a timely manner.

You know what else helps you skim effectively? Practice. Practice gives you a broader base of knowledge and experience to draw on:

That [knowledge/experience] may be the basis for some anecdotes about the speed-reading abilities of famous people, such as that President Kennedy could pick up a copy of the Washington Post or the New York Times and read it from front to back in a few minutes. However, consider the knowledge and information that someone like Kennedy would bring to the task of reading the newspaper. As president, he was briefed about important world events each day and was involved in generating much of the policy and events reported in the newspaper; thus, he probably had first-hand knowledge of much of what was described. In contrast, the average person would come to such a situation with very few facts at his or her disposal and would probably have to read an article rather carefully in order to completely understand it. To read rapidly, you need to know enough about a topic to fit the new information immediately into what you already know and to make inferences.

Of course, the downside of skimming is that you are skipping over portions of text resulting in lower comprehension. However, if you're looking to get a general overview or find one specific fact, then it can be useful. It may also be good for a first pass at a text before reading in depth.

Conclusion

I do feel as though my mind wanders when I read, and I wonder whether there is a way to better engage my mind when reading. Perhaps conceptualizing or visualizing or some other way of focusing more would help comprehension and speed. If and until I and figure that out, I can improve my reading speed by practicing and getting better at skimming, when skimming makes sense.

Footnotes:
[1] I wonder (off-the-cuff, anecdotally, non-scientifically) whether writing more also improves reading, for the same reason: it improves language skills.

[2] Interestingly, this means that as an author you bear some of the burden for helping readers quickly consume your writing. I had started to shy away from listicle style blog posts, thinking I'd try to contribute to a more high minded discourse that rewarded effort in reading and comprehending. This article has more headings and lists...maybe I'll do a little of both. :)

Engineering Synthesis

2018-09-12T13:27:00.001-04:00

What is the nature of software engineering? How is it different from other kinds of engineering? Why is it so hard?

These are questions I have struggled with for many years. In my work, I have seen more than a few different takes on software engineering. Even when things start out right they seem to end at a sad place, and this has bothered me. Is it really impossible to do software "right?" Or do we just have the wrong idea about how to do it? Software engineering is a relatively new discipline, so maybe we still have some things to learn.

I'm going to draw from several sources here, and try to synthesize some ideas about engineering, science, and art. I feel kind of silly writing all these words summarizing other sources when you could just go watch the videos and read the papers yourself. But for my purposes these sources are a framework for discussing and organizing my thoughts.

Real Software Engineering

"Real Software Engineering" by Glenn Vanderberg
http://www.infoq.com/presentations/Software-Engineering

Glenn Vanderberg is a software practitioner, and he is reacting to the claim that software engineering needs to grow up and become a "real" engineering discipline. But what is "real" engineering?

There are actually a couple of different versions of this talk available online, and in one Vanderberg takes some time to talk about "how did we get here?" He digs up some history on the NATO conference in 1968 whose goal was to define software engineering. He then talks about some commonly believed myths about engineering, about how different engineering disciplines use different methods, then brings it back around to software engineering and applies what we've learned.

There were three big ideas from Vanderberg's talk that stood out to me:

The model of scientists discovering knowledge and engineers then applying that knowledge is wrong.
Software engineering is unique because we spend a lot of time crafting design documents and models and a trivial amount of time actually producing the end product, which is the exact opposite of most other branches of engineering.
Agile methods are the best methods we have and for all practical purposes they are software engineering.

When I first watched Vanderberg's talk years ago, the big idea was the second—about the uniqueness of software engineering—but coming back to it later I was surprised to find this first idea echoed in other sources. Vanderberg gives a examples of advances in knowledge that came not from academics or scientists, but instead from practitioners and engineers. One example is Robert Maillart. He was an engineer who revolutionized the use of reinforced concrete in bridge building. He did this before there were mathematical models to explain the uses and limits of reinforced concrete. Scientific advances are just as likely to come from practitioners as from academics.

My second idea from Vanderberg is that among the kinds of engineering, software engineering has some unique characteristics. If one were to build a skyscraper, one would construct designs, models, blueprints, then those would be handed over to a construction team who would construct the building. The blueprints are relatively cheap to produce. The actual construction is error prone and requires a lot of materials and labor. Looking at this process, it would seem very important to focus as much effort on the architecting of blueprints as possible. Once you've laid the foundation, it is expensive to rethink the footprint of the building.

If I were to apply this process to software engineering I might do something like the following: Hire a system architect to create a design document, and then get a bunch of code monkeys to actually construct the system by writing code. In my interpretation, the requirements and design document are the model and blueprints, the system architect is the architect, and the code monkeys are the construction crew. Vanderberg picked up an insight from Jack Reeves in the 90's: this interpretation is wrong.

Customers do not pay for code, they pay for an executable. They want a working system. That is the constructed product, and it is the compiler not the code monkeys that produces it. The code is the design document and mathematical model. The code monkeys are not the construction crew, they are the architects. Source code and its type systems are a mathematical model that can be formally verified. Using a compiler, I can produce a prototype from that model instantaneously and for free. The source code also contains documentation, and to the extent that it has automated tests (also written in the same language) it is self verifying. Modern high level languages and domain specific languages can even be mostly understood by domain experts.

Software engineering is a unique engineering discipline, because source code is a unique artifact. We should be careful not to take engineering methods from a discipline where constructing a prototype is time consuming and expensive and one is necessarily forced to spend more time on up front design to avoid that cost. This will lead nicely into my third big idea, that agile methods are for all practical purposes the best kind of software engineering we know.

When I say agile methods, I mean agile with a little 'a'. I'm thinking (vaguely) of an incremental tinkering approach, versus a straight line mechanical approach. I'm thinking of a technician approach, versus a technique approach. Or as the original Agile Manifesto said, "people over process." I think they got that right. What is interesting is they were not the only ones to get it right. The original NATO conference on software engineering (1968!) had it right before they had it wrong.

There were two NATO conferences that were a year apart. At the first session Alan Perlis summarized the discussion on system design:

A software system can best be designed if the testing is interlaced with the designing instead of being used after the design.
A simulation which matches the requirements contains the control which organizes the design of the system.
Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself. I think that it is the key of the approach that has been suggested, that there is no such question as testing things after the fact with simulation models, but that in effect the testing and the replacement of simulations with modules that are deeper and more detailed goes on with the simulation model controlling, as it were, the place and order in which these things are done.

What he is saying is:

Test early, test often.
Take a breadth first approach mocking out what you need so you can get a sense for the overall system.
Iteratively refine the system and replace the mocks.

That is suspiciously similar to an incremental development method. Between the 1968 NATO conference and the 1969 NATO conference things changed, and there was a clear tension between those who thought programming was best done by an expert technician, and those who thought programming was best done mechanistically by someone taught a body of scientific techniques. At the end of the 1969 conference, Tom Simpson gave a talk called "Masterpiece Engineering" which is oozing with conflicts of technician vs. technique.

There was definitely a lot of political maneuvering at the NATO conferences. There are some other resources you can investigate if you'd like. The point is the seeds of agile were there, but for some reason we ended up with 33 years of waterfall.

Engineering(,) A Path to Science

"Engineering(,) A Path to Science" by Richard P. Gabriel
http://www.infoq.com/presentations/Mixin-based-Inheritance

"Structure of a Programming Language Revolution" by Richard P. Gabriel
http://dreamsongs.com/Files/Incommensurability.pdf

Richard Gabriel's talk comes from an interesting perspective. He was involved in the Lisp community and has an academic background (he earned a PhD), but is not an academic. After working as a practitioner, he went back to school to earn a Masters of Fine Arts. Upon returning to the technical community, he felt a paradigm shift had happened while he was gone. The conferences he used to attend had been renamed and were now focused on academics instead of practitioners. His entire field--Lisp systems engineering--and its journals had been deleted.

Then he was given the first scientific paper on mix-in inheritance. Being familiar with previous work done on Lisp based inheritance systems, he felt that this paper was using the same terms to describe some of the mechanisms from the Common Lisp Object System, but the terms had different meaning. Gabriel felt he was experiencing incommensurability, that a paradigm shift had happened from an engineering focus to a scientific focus, and now "scientific" papers were being written that described, as new, things that engineers had already known, using the same terms but with different meanings.

The talk is definitely worth watching. It is an interesting personal story intertwined with technical discussions of the previous work versus the paper he had been given. It is an exploration of whether incommensurability can actually happen and to what extent. He also challenges the myth that science always precedes engineering.

I'm honestly not sure whether Gabriel intended his talk and paper to have a single point. Maybe he is mostly interested in relating his personal experience, but this is what I took away:

In general, science does not always precede engineering, and in particular the relationship between computer science and software engineering is even more complex, because the engineers literally create the reality that the scientists study.
There are two approaches to software: the systems approach, and the language approach.
Making engineering subservient to science means throwing away the progress that engineers can and do make.

This was actually the first talk that started the wheels turning for me on the relationship between science and engineering. I had been told in college that scientists expand the body of knowledge and engineers apply that body of knowledge. Gabriel uses as his example the steam engine. When the steam engine was invented the popular theory used to explain its operation was the Caloric Theory of heat, which stated that there was an invisible, weightless, odorless gas called "caloric" that permeated the Universe. The amount of caloric in the Universe is constant, and its interaction with air molecules can explain heat and radiation, and from it you can deduce most of the gas laws. The Caloric Theory was a useful theory with predictive power. When Laplace adjusted Newton's pulse equations to account for caloric, he was able to more accurately predict the speed of sound.

Eventually the Caloric Theory was replaced by Thermodynamics, and amazingly steam engines continued to work! The steam engine was developed by mechanics who observed the relationship between pressure, volume, and temperature. Whether its operation was explained by the Caloric Theory or Thermodynamics made no difference to them. Yet, an engineer's invention can and does spark the curiosity of a scientist to develop a theory to explain how it is that an invention works. This is even more true in the case of computer software.

The second moral I drew from Gabriel's talk is that there are (at least) two approaches to software: a systems approach and a language approach. Gabriel acknowledges that at first he thought the incommensurability that he saw was a difference between an engineering paradigm and a scientific paradigm, but eventually he saw it as a more technically focused conflict between a systems paradigm and a language paradigm. Perhaps what Gabriel means is that you can approach either systems or languages from an engineering or a scientific perspective. However, I tend to see systems versus languages as engineering versus science.

The systems paradigm views software as interacting components forming a whole; real stuff doing real things. The language paradigm views software as abstract signs and rules of grammar conveying meaning. Good design, from a systems perspective, comes from a skilled technician following good design principles (I would even call it aesthetics). Good design, from the language perspective, comes from a relatively less skilled technician working within a language that from the outset excludes bad design through grammatical rules and compilers. The system approach tends to view software as a living organism that is incrementally poked and prodded, changed and observed. The language approach tends to view software as a series of mathematical transformations, preserving meaning. If each of the paradigms were a theory of truth, the systems paradigm would be correspondence, and the language paradigm would be coherence.

I see system versus language as engineering versus science. I view engineering as a bottom up, incremental, tinkering approach, at least when it comes to software and the way I like to practice software engineering. I view science as a top down, formal, mathematical approach. I actually like both, and I think both have their place, but when engineering is made subservient to science, we're actually losing something very important. When engineers are shut out of conferences and journals, there are discoveries that will be left unpublished, and new scientific theories left untheorized. (This was what Gabriel saw happening.)

Computer Programming as an Art

"Computer Programming as an Art" by Donald Knuth
http://dl.acm.org/ft_gateway.cfm?id=1283929&type=pdf

For those with even a cursory exposure to Computer Science, Donald Knuth needs no introduction. Knuth is coming from an academic perspective, but even for an academic his perspective is a bit unique. He has created and maintains several large open source software projects. This is his ACM Turing Award lecture given in 1974. He starts by quoting the first issue of the Communications of the ACM (1959). It claims that for programming to become an important part of computer research and development (to be taken seriously) it needs to transition from being an art to a disciplined science.

The big idea I draw here is: Programming can be art (in the "fine art" sense), which means it is (at least sometimes) a creative endeavor.

Knuth first explores the definition of "art" and "science." He looks at their use over time. Their use was (and is) not consistent. At times "science" and "art" are used interchangeably. "Art" was used to describe something made of human intellect, not nature. Eventually "science" came to mean "knowledge" and "art" came to mean "application." Though even that usage is not universal. To Knuth an "art" is something that is not fully understood and requires some aesthetics and intuition. A "science" is something well understood. Something that can be mechanized and automated. It is something that can be taught to a computer. Can computer programming be taught to a computer?

Knuth does not think that programming can ever be fully automated. However, it is still useful to automate as much as possible, since it advances the artistry of programming. He believes, and cites others, that progress is made not by rejecting art in the name of science, nor science in the name of art, but by making use of both. He makes reference to C. P. Snow's "The Two Cultures" as an example of another voicing concern about separating art and science. At this point when he speaks of art he means something more along the lines of "fine art" than "engineering."

Knuth goes on to talk of creativity, beauty, art, and style. He hits on how sometimes resource constraints can force a programmer to come up with an elegant solution, and this has an artistic aspect to it. He also encourages people to, when it comes to programming, make art for art's sake. Programs can be just for fun.

Knuth's talk is focused on the act of programming, and when he deals with engineering versus science he means with respect to the act of programming. To what extent can the act of programming be made automatic? To what extent must it remain a human act of creativity? This is a little further afield of the previous sources, but Knuth's insistence on seeing programming as a creative act is the big idea I drew from his talk, and is really the point of his talk.

Given that programming can sometimes be a creative act, it raises a lot of questions in my mind. Is programming always a creative act? If programming is a creative act, how should a programming project be managed? Is the high failure rate of software projects related to this? Perhaps this ties back into Tom Simpson's "Masterpiece Engineering" satire. Imagine a project manager with a room full of artists creating Gantt charts and task dependency graphs to plan out the creation of a new masterpiece!

On the other hand, nothing appeals to the ego more than seeing oneself as a grand master of art. There should be a measure of moderation here. I think there is benefit to trying to understand programming as an artistic (or at least "creative") endeavor, whatever that means, but we should not go crazy with hubris.

Better Science Through Art

"Better Science Through Art" by Richard P. Gabriel and Kevin J. Sullivan
https://www.dreamsongs.com/Files/BetterScienceThroughArt.pdf

"Better Science Through Art" by Richard P. Gabriel
https://www.tele-task.de/archive/video/flash/12636/

I have already covered some of Gabriel's background, but I will say that having been involved and educated in both a technical field and an artistic field gives him a unique perspective on the relationship between science, engineering, and art.

I unfortunately don't know much about Sullivan's background, other than he is a professor of computer science at the University of Virginia. His collaboration with Gabriel produced one of my favorite papers ever. I don't know that I can tease out what should be attributed to whom. I will be basing my comments on Gabriel's talk, but I don't intend to attribute everything to him, or to diminish Sullivan's contributions.

The big ideas I drew from this is:

Science, engineering, and art all have at their core "disciplined noticing."
Disciplined noticing is a skill that requires practice.
The creation of knowledge—even in the case of science—requires an abductive leap powered by creative spark.

This is a really great talk, and covers a lot of ground. It is entertaining, insightful, and very worth watching. He attacks some common caricatures of science, engineering, and art, and digs into the actual process behind each. In the end, he finds that there are a lot of similarities to the methods in science, engineering, and art. It is a process of exploration, discovery, and verification. He calls it disciplined noticing.

I have found this to be true in my experience. Just like people have a caricature of science, that it is straight line progress, the monotonic aggregation of knowledge, there's a similar caricature of software development. My experience has been that writing software is a creative, exploratory process. Sometimes I go down an alley, but find that I need to back out and take a different turn. I may write a test, run it, change some code, change a test, run it, think for a while, delete a bunch of code and rewrite it all.

In my experience this process—writing, evaluating, and rewriting—has much more in common with writing a novel than constructing a building.

Conclusion

This long meandering post must come to an end. First of all, I would highly recommend looking at each of these cited sources. They will reward you. Perhaps you may even find that I have seen them through my own preconceived notions, and you may draw an altogether different conclusion from them. So be it.

This "conclusion" is not really a conclusion, but a way-point. I started on this journey to understand the nature of software engineering, how it is different from other kinds of engineering, and why it is so hard. I ended up at a place that intuitively I knew I would end. I will not make an absolute statement. I will say that at least sometimes (and in my experience) software development is a creative process more akin to creative writing.

I have also seen that there is a tremendous amount of creativity in both engineering and science. I believe that at the core of engineering, science, and art is a drive to understand and influence the world, which requires observation, testing, and evaluation. I don't claim to know how to do software engineering "right," but I don't think we will ever do it right if we refuse to see that creativity (which is at times unpredictable) is a key part of the effort.

I have learned that both engineering and science are useful for discovering and validating knowledge. Scientists and engineers should collaborate. Neither should be seen a primary at the expense of the other. They can even be seen as external expressions of the same process sometimes using similar tools and techniques.

I have learned that software is unique in engineering. Whereas a blueprint is a written artifact using specialized notation, the building it describes must be brought into existence through a complex error prone process. Code is written using specialized notation, but the gap from code to execution is much smaller. There are pitfalls and challenges, no doubt, but I would like to see how the nature of what we produce can change how we produce it. I'm still holding out hope that the nature of software can change the face of the human organizations that produce it.

Practically, what this all means is that a software engineering process should be iterative. It should embrace unpredictability and allow space for the creative process. In the same way that a painter never thinks his painting is complete, software should be developed in a way that continuously produces value, so the project could be closed down and the product shipped at any point, and the customer is still happy with the result.

So I end back at the beginning with Vanderberg. I don't think that agile is the last word, but I think it is the best we have so far.

Manufacturing Creativity

2018-09-12T06:21:00.000-04:00

Previously, I've attempted to convince you that making software is a creative act, and I explored the implications for pursuing and managing software engineering. (By the way, science and engineering are also creative acts, and a great exploration of that idea is "Better Science Through Art" by Richard P. Gabriel and Kevin J. Sullivan. Love that paper.)

I've been thinking a lot lately about creativity and how it can be encouraged (even manufactured?). I've also been thinking quite a bit about why people do (or do not) take on ambitious projects, and how to survive a years long ambitious project. I've learned some very interesting things that some day I may write about, but I'd like to share what I've learned about creativity.

What I've discovered about being creative is that even from people in very different lines of work (actors, writers, artists, programmers, scientists, investors) there's a surprising amount of agreement about how it works. I've also discovered that it is not an innate talent that some people have and some do not. Everyone has the tools to be creative.

In many ways this goes all the way back to the very first Clojure Conj in October of 2010. Rich Hickey gave a talk titled "Step Away from the Computer"...actually, it had three titles, and it is best known by one of its other titles "Hammock-Driven Development." I was there in person. I came away with the mistaken impression that the talk was about writing software and solving technical problems. I now know that making software is a creative act, and Rich's talk was about how to be creative.

For Rich, the engine of creativity is the "background mind," which is in contrast to the "waking mind." Your waking mind is your normal mode of operation. It is good at analyzing and thinking critically, but can be too tactical and get stuck in local maxima. Your background mind is good at making connections, thinking abstractly, and synthesizing. It can make the leap past local maxima, unfortunately your background mind cannot be tasked directly. However, you can task it indirectly by obsessively thinking and reading about a particular problem, and, though you can activate it other ways, it is easiest to activate it by sleeping, or relaxing and simulating sleeping (i.e. using a hammock).

So, creativity is an indirect process of a relaxed mental mode that you task by obsessively thinking about a problem, and whose products you only filter after the fact with your normal critical-analytical mental mode. Now here's the surprising part, almost everyone who attempts to describe their creative process describes it similarly. In his essay, "The Top Ideas in Your Mind," Paul Graham says:

Everyone who's worked on difficult problems is probably familiar with the phenomenon of working hard to figure something out, failing, and then suddenly seeing the answer a bit later while doing something else. There's a kind of thinking you do without trying to. I'm increasingly convinced this type of thinking is not merely helpful in solving hard problems, but necessary. The tricky part is, you can only control it indirectly.

John Cleese gave a talk on creativity, and he called the background mind "open mode" and the waking mind "closed mode." In your open mode, you are relaxed, less purposeful, curious, and a bit playful. In your closed mode, you are active, determined, and have a critical eye.

George Land is a business man who investigated how to stimulate and direct creativity. He found there are two kinds of thinking: divergent and convergent. Divergent thinking is creating new ideas. Convergent thinking is judging and evaluating ideas. He did a longitudinal study that found that 98% of 5 year olds exhibit divergent thinking, 30% of 10 year olds, 12% of 15 year olds, and only 2% of adults think divergently. As a person gets older, he or she is taught to use both divergent and convergent thinking at the same time. The result is one criticizes and judges ideas before they can fully develop.

Of peculiar interest to me has been what independent game designer Jonathan Blow—who worked on his successful and influential game Braid for 3.5 years—has said about creativity and surviving ambitious projects. (Maybe someday Rich will talk about how he survived his own ambitious projects: how to maintain motivation day-to-day, how to fund it, how to plan and pace it, how to finish it.) The thoughts about ambitious projects are for another time, but what he says about creativity should be familiar by now. Blow says metaphysically you may not buy into the Greek concept of the Muse—nor may he—but functionally it is real. Creativity feels like something external, and you have to get yourself into a relaxed mode to provide opportunity for new ideas, though you cannot guarantee anything.

Tools and Techniques

I hope to find more resources on direct techniques for stimulating creative (e.g. instead of thinking about solving a problem think about how to make in worse and avoid that), but for now I've found a lot of agreement about how to encourage creativity in an indirect way.

Obsess about your problem. If your subconscious mind (or unconscious mind or background mind or whatever you want to call it) is going to solve a problem for you, then it needs information. Rich has a lot of great advice about this. Write down your problem. Write down what you know. Write down what you don't know. Read about your problem. Read about related problems. Pick apart other solutions. Paul Graham in "The Top Idea in Your Mind" says, "It's hard to do a really good job on anything you don't think about in the shower."

Relax. For Rich this is lying in a hammock and focusing, thinking through all the information you've loaded into your mind. For Blow, a relaxed state of mind is really a pretty active body. He likes to find something purely physical that he can enjoy, like going to a club and dancing. Cleese creates an oasis blocking off time and setting aside other concerns. He gives himself enough time that he can work through all the TODOs that pop into his head. He writes them down for later, and gets back to being relaxed and playful.

Pace yourself. Cleese recommends, if you're going to try to set aside time for creativity, to limit it to no more than an hour and a half, because you'll need a break. If you need more time, then do it again the next day.

Be Playful. George Land found that children are more creative. Cleese finds being in a playful mood conducive to creativity, especially when collaborating with others. Play, imagination, daydreaming all come from or lead to a relaxed state of mind, which accesses your creative mechanism.

Write things down. Rich is big on this. There are several benefits: it helps you think thoroughly, it helps you remember things, it is easy to skim for recall.

Gently keep your mind focused. Cleese says to be successful you must keep your mind gently around the problem. You may wander off, but gently come back to it. Rich uses hammock time not just to relax, but to recall information. Touch each fact with your mind to keep it fresh, and to make it interesting to your background mind.

Have a dogged persistence. Cleese sticks with an problem, and doesn't just take the first idea he comes up with. Sometimes a creative breakthrough requires persisting through the discomfort, even slight anxiety, of an unsolved problem. Rich reminds us that since this is an indirect process it may take days, months, or years for a solution to come.

Anti-techniques

How can you destroy creativity? Easy:

Chase success. Paul Graham says the way to destroy your creativity is to make money the top idea in your mind. It tends to consume all your mental energies. Blow also warns about thinking about success or how others will judge what you do. These things can easily lead to fear, and as Cleese says you need to feel confident to be able to generate ideas.

Obsess about disputes. Paul Graham talks about how Isaac Newton got involved in disputes and regretted the wasted energy. This is really just another form of worrying about what other people think.

Make a schedule. Blow warns about making a schedule, but also admits that we must all deal with schedules. Rich says his techniques don't work under pressure. While Cleese sets aside time to be creative, he recognizes that the process is unpredictable and needs time.

Pre-judge ideas. You must be open, Cleese doesn't call it "open mode" for nothing. Brainstorming forbids judging ideas, and as a technique it gets that much right. George Land found the more we use divergent and convergent thinking together—in other words the more we try to pre-judge ideas—the less creative we will be.

Get distracted. Cleese says you need to create a space free from distractions. For Blow, even the threat of a distraction can prevent him from relaxing, so he'll even spend time at a coffee shop for a few hours before heading into the office.

No humor. According to Cleese humor is about two frameworks coming together to make new meaning, and this is also the core of creativity. If you eliminate humor, then you eliminate creativity.

Live actively and urgently. If you want to ensure no relaxation happens, if you want to ensure that you are in closed mode, then live urgently and actively.

Conclusion

If you're here you are probably a computer programmer (most likely a Clojure programmer). That means you're probably a bit like me. You're good at thinking analytically and logically. You're good a judging solutions based on correctness, performance, etc. You're good at operating in "closed mode." These are great skills, and as Cleese says we need both open and closed mode to succeed, open to generate ideas, and closed to execute on them. We just may need to work on the open mode a bit.

You have the ability to be creative. You have a relaxed, curious, playful, imaginative self. There are some techniques that others have used that may help you access your creativity. They may help you, they may not. You may need to experiment a bit for yourself.

You cannot fully control this process. You can only indirectly stimulate creativity, and you cannot guarantee that your mind will solve the problem you want it to solve. One approach would be to work on several problems at once! You may also find some fruitful connections between the problems.

To be creative you must be persistent, and you must practice. I hope this helps you find those imaginative solutions.

Gardening

2018-05-21T09:00:00.001-04:00

If you're like me, you spend 8+ hours a day in front of a screen. About five years ago, I decided that I needed better hobbies than learning new programming languages and writing code for personal projects. I wanted find ways to learn new skills and connect with people. I've done that by playing board games at local meetups and building a robot, and I've done that with gardening.

Gardening has been incredibly frustrating and incredibly rewarding in a roller-coastery kind of way. I'd like to share my journey with you in an attempt to get you interested in gardening. I'll share some resources I've found interesting and useful.

Why gardening?

I chose gardening for many different reasons. I remember my parents having a garden when I was a kid, and I wanted to have a hobby that my kids could be involved in and excited about. I like to eat things like tomatoes that my wife does not often buy, because no one else (including her) likes them. I wanted to do something outdoors. I wanted to become a little more self-sufficient.

Those are some of my reasons, but maybe you have other reasons. Maybe you'd like to reduce your carbon footprint by producing your own food that doesn't get shipped half way across the world. Maybe you like the idea that food from your garden is essentially tax-free income. Maybe you want to increase the diversity in your diet and/or help preserve and conserve heirloom food varieties that are endangered. Maybe you don't want to grow food but flowers providing you with a vibrant, delicate beauty.

How gardening?

There are many ways to garden from containers to raised beds. One of the things I enjoy about gardening is an entire world of new things to learn. It is a gateway hobby into things like cooking, canning, composting, carpentry, and other words that begin with 'c'.

I have focused mostly on fruits and veggies, since I want to be able to eat from my garden, but I've also grown (and grow more and more) flowers. I've grown some edible flowers and some inedible. It is incredibly satisfying to have some color around the house.

I started small with some containers on my deck. I used a couple of EarthBoxes, then built my own DIY EarthBoxes. I like the sub-irrigated planter (SIP) concept so much that I'm planning on putting in a raised bed SIP in my backyard, automatically fed by rain barrels. If you want to learn more about SIPs, check out https://www.youtube.com/albopepper.

Gardening (like most hobbies) can be as expensive as you let it. You can buy all kinds of gardening gadgets and gizmos. One of my goals is to make gardening as economical as possible. To garden you need:

Plants
Sun
Water
Nutrients

The sun part is pretty easy, since my back yard is south facing. I just need to work around the shadows cast by trees and the deck.

You can buy seeds pretty cheaply, but you can also harvest seeds from your plants, so you don't have to continually buy seed packets. This will only work with open-pollinated (OP) plants. Check out this video to learn about OPs, hybrids, and heirlooms: https://www.youtube.com/watch?v=zkMEmkecSHs Often, it is easier to buy seedlings at a nursery or farmer's market.

You can also plant perennials like strawberries and asparagus. These don't need to be replanted every year. You plant them once and you can harvest for years.

You can obviously water your plants with your tap, but rain barrels are a way save money taking advantage of an abundant resource over our heads. You can buy rain barrels, or you can make your own. My water company even gives a $30 rebate each for up to two rain barrels that I install.

Plants need nutrients, and nutrients can be provided by fertilizer. I still use fertilizer occasionally, but I've opted to make my own compost. Unfortunately I don't have many trees whose leaves I can compost. This is usually the easiest way to make compost. However, I am composting what leaves I have along with grass clippings and cardboard boxes from all my Amazon Prime orders. I compost trimmings from my garden, and kitchen waste. I'm even thinking about getting some composting worms! Here is a video about how ridiculously easy it is to compost: https://www.youtube.com/watch?v=n9OhxKlrWwc

Lessons Learned

I've been gardening about five years, and here are some lessons I've learned.

Time and timeliness.

As a software engineer, I work in a field where I'm constantly learning, and there's a new JavaScript framework every week. I enjoy being more aware of the weather and seasonal rhythms. Plants work on a different timescale. If something goes wrong with the crop this year, I may have to wait another whole year to try again. That can be frustrating, but it can also be an opportunity both to think over a longer timescale and to be very focused on what is happening right now because the stakes are high.

Everything wants to kill your plants.

In container gardening on my deck I've dealt mostly with insects, and there are billions of them. When I moved into raised bed gardening with my strawberry patch, I had to deal with deer eating all the leaves off my strawberries. For the past couple of years it has been impossible for me to grow zucchini or squash, because vine borers have eaten them from the inside out. I'm not necessarily a fan of squishing bugs, but there was nothing more satisfying than digging those buggers out and squishing their fat bodies. It was a kind of anger management program.

The lesson is you need to think about pest management from the beginning. Talk to your neighbors about what pests they've dealt with in their gardens. Or at least be prepared that the first year could be rough until you know what you up against. When you do know what you're up against...research! If you live in the US look up your local cooperative extension website. Virginia's has all kinds of great publications for growing things in my region.

Your plants want to live

Even the sun can sometimes be brutal on your plants. I tried seed starting a couple of years ago. The last step is to "harden off " your plants by gently exposing them to the elements. I was a little less than gentle and nearly killed my plants.

After the hardening off incident I felt like a bad plant daddy, but the amazing thing was my plants came back. They want to live. They are partners in this gardening adventure.

It is satisfying to make things grow

It can sometimes be difficult to diagnose what is wrong with a plant: is it overwatered, underwatered, missing some nutrient, etc? Plants are complicated yet fascinating living things. It is worth the effort to understand them and work with them. One of the most fascinating books I've read is Botany for Gardeners by Brian Capon http://a.co/7SSM1Wi. I really enjoyed Brian's writing style, and it is a very approachable introduction to cellular function, propagation, and the fascinating life of plants.

In the end there is a lot to learn, and it is hard work, but it is so satisfying to nurture a living thing.

It is satisfying to work hard

I have a personal rule for myself that as much as possible I will refuse to have someone else mow my lawn. It saves money. I listen to podcasts and audio books. I like to walk around my house and property (only 1/3 acre but still) and see how things are doing. It can be hard work since my yard is mostly a hill, but I like to get the exercise.

Gardening can be hard work, too. One Sunday afternoon, in addition to mowing and edging, I pulled out two bushes (which if you've ever done, then you know), and planted an apple tree and six red raspberry canes. I was sunburnt and sore, and paid for it the next day, but it was satisfying, and I'm looking forward to the fruit of my labor (literally!).

Play the odds

I recommend starting small, because like any hobby you can get excited and spend a lot of money before you realize it. However, you also have to know that gardening is about playing the odds, so don't start too small. When you start seeds, you put three in each hole, and when they sprout you thin them down to just the strongest of the seedlings. If you buy tomato seedlings from a nursery, don't just buy one, buy two or three. You have to expect that some plants won't survive.

It can also be helpful to plant more than one kind of thing. You may not get everything you want, but you should plant a diverse mix of plants and enjoy whatever you get. If you only plant cucumbers, then horde of cucumber beetles can destroy everything, but if you also have tomatoes, then it's not a total wash.

Conclusion

Have I accomplished my goal of learning new skills and getting to know people? Absolutely! Of the five houses that border mine three are gardeners, and when I'm out early in the morning tending my garden my neighbors are often out, too. I've had chances to get to know them.

I've gotten outdoors. I've gotten plenty of exercise. My kids are involved and excited about gardening. They even eat things they normally wouldn't, because we've grown them ourselves.

If you want a hobby to get you away from the screen and doing something physical in the real world, then give gardening a go.

The Ethics of Software Quality

2017-09-07T09:24:00.002-04:00

Security professionals are in a hard place. If there is a security breach, they take the fall. However, if they do their job right, no one notices. Further, they may even meet resistance to doing their job right because they are being overly cautious, taking too much time, costing too much money, etc., etc.

I think a software professional who wants to create quality software faces the same challenge. You may deliver quality software, but then get accused of taking too long (according to some arbitrary idea someone has) or "gold plating." You get compared to co-workers who write code much faster, even though it may have more bugs. Focusing on speed as a primary metric for software development is a race to the bottom.

This is not to say that there aren't times when something needs to be timeboxed, or a programmer needs to resist "gold plating." It is possible to fall into a trap of tweaking and refactoring ad infinitum. However, I don't find that there is a bright line or objective standard for judging this. Maybe that is because I believe software development to be a creative, exploratory process, so I'm apt to think there's more than a little taste and discernment.

To produce quality software you must take an ethical approach. What do I mean by this? While it seems obvious that there are ethical issues in software development---for example poor quality software wastes time and money, causes frustration, and in the extreme case can cause damage to property and loss of life---that's not what I mean.

What I mean by "ethical approach" (and maybe there's a better term for it) is you must have an intrinsic motivation to create quality software. You have to do it because "it's the right thing." You will rarely get support from managers to produce quality software. You will shoulder the blame for quality issues in your code. If your code is beautiful and functional and bug-free, rarely will anyone even notice, let alone commend you.

How can you develop a "software quality conscience"? I don't have all the answers, but I have a couple of suggestions:

Read good code and read about good code. If it is garbage in, then it will be garbage out.
Surround yourself by other people who care about quality. Find a team of like minded people whether it is at work or not.
Keep things in perspective. I find, as I'm further into my career, that I've had bosses bluster at me to get things done by a certain time ("do or die"), and found that it didn't really have a huge impact on the success or failure of my project or company. Don't be insubordinate or lazy, but don't buy into the hype. Be realistic.

You are responsible for fighting the good fight. So step up.

Continuous Planning

2017-02-27T08:58:00.001-05:00

"In preparing for battle I have always found that plans are useless, but planning is indispensable." -- Dwight D. Eisenhower

There is a tension between engineering on the one hand, and on the other hand those who would like to know when the task will be done. A product must be marketed, documented, sold, and supported. "When it's done," is useless when you're trying to sell to a customer against a market full of competitors. However, the software we write gets more complex each day, and the process for bringing it to life is complex. Complexity means unknowns, and unknowns mean uncertainty. A software project is like a hurricane with a cone of uncertainty preceding it. This tension between the desire to know and the reality of uncertainty is a fundamental part of working a software project (and probably other kinds of projects).

Before going too much farther I will state my assumption: a completion date is an output not an input, and the most effective tool for managing a completion date is changing the amount of work you want to do (i.e. "scope").

You cannot take a date and work backwards. This is no different than taking a date and working forwards. Well actually, there is a big difference. In working forward, you can always push the completion date out. In working backwards you cannot start any earlier than now. The completion date inevitably follows from when you start, how quickly you can work, and how much you are trying to do.

You can spend money on tools, training, consultants, but these each have a time cost.

You can add more people, but in order to establish a context on a project a new person must learn a code base, tools, technologies, personalities of the team, and to do so he or she must take time from an otherwise productive member of the team.

You can have the current team work overtime, but too much of that will cause quality issues and burnout.

You can relax expectations about quality, but that is just trading your future time to get something done more quickly and temporarily.

The best thing you can do to manage a completion date, is to cut the amount of "stuff" you are trying to do, or to rearrange the order of when you will do it, so you get the things you want earlier than you otherwise would have.

Given that a date is an output, as engineers and managers we try to navigate this tension between the desire to know and the reality of uncertainty with planning, but there's a problem with plans: they're useless. Imagine planning a single task. When will it be done? Well, if you ask one engineer she will give you an estimate based on her skill and experience. If the task is ever given to another engineer, then that estimate is invalidated. On top of that, an engineer (or human really) is notorious for estimating only the amount of work she must do. She doesn't think about QA testing, deployment, and data migrations, among other things. Nor does she think to factor in overhead like meetings, filling out time cards, learning new skills, bonding with coworkers, etc.

That is just at the most atomic level of estimation. Once you start to think about collaboration things get more complex. Does our engineer need to get a review from a coworker? That coworker is now being taken off of his task to do the review, which can lead to delays. What if our engineer needs assistance from someone more familiar with a particular technology or part of the code base? What if our engineer wants to brainstorm with another engineer? If our engineer gets delayed then any tasks that were dependent on her task also get delayed.

Now imagine making a plan for a product that spans several teams and tens or hundreds (or thousands??) of people. If you don't know everything that everyone is working on and how they are all related, then you can't with certainty plan out anything, and that is only taking into account everything that can be known, there are still unknowns (like if someone may get sick). This is the uselessness of a plan.

Well, a plan is not entirely useless. It is probably very accurate for the tasks that will be started in a few days, but entirely inaccurate for the tasks that will be started in three or six months.

So, there are two problems with a plan: 1) it must be updated to reflect new information, and 2) it fails to take into account the "cone of uncertainty."

Updating a plan seems easy enough, however the larger the plan the more work it will take to keep updated. One could certainly employ an army of project managers who verify that the task breakdown, estimates, and dependencies have not changed; that you've taken into account every meeting, vacation plan, all the testing, deployment, and overhead. Ideally the plan would be updated continuously (so its more of a "dashboard" than a "plan"). More valuable than knowing that three months ago we thought a task should be complete on such and such a date would be knowing when we think it will be completed as of now with all the latest information we have access to, but that would create quite a drag on the entire team.

Even if you could keep the plan up-to-date, it gives the false impression that one can know precisely when a task will be complete. You may be able to predict the completion date of a task that starts tomorrow, but not for a task that will start in three months. Three months provides plenty of time for both knowns and unknowns to change when the task could even start, let alone when it would complete.

Usually, this uncertainty is handled by "padding" the date, but this is not enough. A single point-in-time completion date conveys certainty, and this is certainly wrong. The completion of a task should always be a range, one that is narrow for the near future and wide for the distant future.

Incidentally, I think even agile burndown charts get this wrong. In my opinion, there is (and should be) variability to a team's velocity. Simply taking some velocity value and running it out a few months to predict a single point in time when a task will be complete is at odds with reality.

What does Continuous Planning look like? Well I don't really know, because I just made it up! At a high level I would summarize it as: plan using real data, with task completion ranges, over as long a term as you want, in aggregate, on average, continuously. The task completion ranges are the key. You can plan over as long a term as you want, however, the ranges will get wider. If you can reduce the variability in your process---and prove it with your data---then you can narrow the ranges. Planning is done in aggregate and on average, because it is impossible to know and manage every possible factor, so we must abstract away much of the minutiae. Finally, to plan continuously implies some kind of tool to facilitate.

To the extent that I have thought about how this would work out practically, this is what I would do:

Each team would estimate their tasks by each member recording the number of hours he or she thinks it would take him or her to complete the task, given that there are no other distractions. This is a kind of pure estimation that engineers usually make. It would be helpful to discuss the task as a team, and try to elicit different opinions on the complexity of the task, so the estimates will be as complete as possible.

Why not use story points? I have been a fan of story points precisely because they abstract away hours. Hours can vary depending on a persons skill and experience. Hours can get lengthened by interruptions and discovering additional complexity. Hours give a false impression that they would map directly to calendar time, and you can accurately predict when a task will complete.

However, the first thing a person asks is how long does X points take. Usually you have to pick as a standard comparison some "golden story" for a certain number of points. People will usually consciously or unconsciously come up with some rule of thumb like, "an eight point story should take about a sprint to complete." So in the end you are estimating in hours, but they're a convoluted form of hours.

Hours are a natural unit for estimations. The danger in using hours is actually trusting the estimate for a precise completion date. We've already rejected precise completion dates with Continuous Planning, and the rest of the process is designed around (automatically) finding an accurate scale with which to judge these estimates. I would actually advocate that the estimates and velocities be hidden variables, and (other than your own estimate) you only see the completion range for a task. This would hopefully reduce some confused expectations around what it means for an estimate to be denominated in hours.

The estimates from each team member would be combined together into a single estimate for the task. The method for combination could be taking an average. It could involve throwing out extreme values first, or doing some sophisticated statistical analysis.

Having done these estimates, a task tracking system would keep track of when tasks started and when they completed, or how long they've been started even if not complete. This actual data can be used to calculate a velocity. The velocity could be calculated at several levels. You could calculate the velocity for an individual task, for a particular team member, for the team as a whole. You could even calculate the velocity for a feature epic cutting across several teams.

To calculate a date range for completion, you can take the average plus or minus a standard deviation for a single velocity calculation over time and get an optimistic and pessimistic velocity. You could get an optimistic and pessimistic velocity by taking the minimum and maximum of the most recent velocity calculations at two different level (task and team, for example). I'm not sure which would work best; it warrants some research.

Tasks would be a hierarchical tree. An epic is really just a task with subtasks. The velocities and estimates can flow up the tree for the purposes of calculating estimated completion ranges for epics.

If you wanted to get fancy, you could draw dependencies between tasks, and the system could then attempt some kind of topological sort of the tasks, and using a prioritized backlog and team assignments to each task, construct a plan for what could be done in parallel, and---based on velocities calculated from real data--calculate a completion range for each task, epic, and the project as a whole.

As you can see there are still many questions to be answered. I think this is an idea worth exploring. In my experience, the usual tools fall flat at resolving the tension between the desire to know and the reality of uncertainty.

To effectively attack this tension requires abstracting away much of the minutiae of detailed planning by embracing the variability of the process. The plan is useless. Planning is indispensable. Therefore, plan continuously

In preparing for battle I have always found that plans are useless, but planning is indispensable.

Read more at: https://www.brainyquote.com/quotes/quotes/d/dwightdei164720.html

In preparing for battle I have always found that plans are useless, but planning is indispensable.
Read more at: https://www.brainyquote.com/quotes/quotes/d/dwightdei164720.html

n preparing for battle I have always found that plans are useless, but planning is indispensable.
Read more at: https://www.brainyquote.com/quotes/quotes/d/dwightdei164720.html

"Clojure Polymorphism" Released!

2016-08-30T14:01:00.000-04:00

From my new blog Real World Clojure. What am I doing with this new blog? I have no idea, but you can follow along.

~ ~ ~ ~

I have released a short e-book (30 pages) titled "Clojure Polymorphism." You can get 50% off by using this coupon link http://www.leanpub.com/clojurepolymorphism/c/ONeJZ629Isy7.
What is this book about?

When it comes to Clojure there are many tutorials, websites, and books about how to get started (language syntax, set up a project, configure your IDE, etc.). There are also many tutorials, websites, and books about how language features work (protocols, transducers, core.async). There are precious few tutorials, websites, and books about when and how to use Clojure's features.

This is a comparative architecture class. I assume you are familiar with Clojure and even a bit proficient at it. I will pick a theme and talk about the tools Clojure provides in that theme. I will use some example problems, solve them with different tools, and then pick them apart for what is good and what is bad. There will not be one right answer. There will be principles that apply in certain contexts.
I this installment, I will pick up the theme of "Polymorphism" looking at the tools of polymorphism that Clojure provides. Then I take a couple of problems and solve them several ways. At the end of it all, we look back at the implementations and extract principles. The end goal is for you to develop an understanding of tradeoffs and a taste for good Clojure design.

I have some ideas for other e-books. Perhaps a concurrency tour of Clojure taking a look at futures, STM, reducers, core.async, etc. Or maybe talk about identity by looking at atom, agent, ref, volatile!, etc. Or maybe look at code quality tools. Or how to organize namespaces. Or adding a new data structure with deftype?

What would you like to see? Contact me. :)

Reducible Streams

2016-08-19T13:15:00.001-04:00

Laziness is a great tool, but there are some gotchas. The classic:

(with-open [f (io/reader (io/file some-file))]
  (line-seq f))

line-seq will return a lazy seq of lines read from some-file, but if the lazy seq escapes the dynamic extent of with-open, then you will get an exception:

IOException Stream closed  java.io.BufferedReader.ensureOpen (BufferedReader.java:115)

With laziness, the callee produces data, but the caller can control when data is produced. However, sometimes the data that is produced has associated resources that must be managed. Leaving the caller in control of when data is produced means the caller must know about and manage the related resources. Using a lazy sequence is like co-routines passing control back and forth between the caller and callee, but it only transfers control for each item, there is no way to run a cleanup routine after the caller has decided to stop consuming the sequence.

A Tempting Solution

One might immediately think about putting the resource control into the lazy seq:

(defn my-line-seq* [rdr [line & lines]]
  (if line
    (cons line (lazy-seq (my-line-seq* rdr lines)))
    (do (.close rdr)
        nil)))

(defn my-line-seq [some-file]
  (let [rdr (io/reader (io/file some-file))
        lines (line-seq rdr)]
    (my-line-seq* rdr lines)))

This way the caller can consume the sequence how it wants, but the callee remains in control of the resources. The problem with this approach is the caller is not guaranteed to fully consume the sequence, and unless the caller fully consumes the sequence the file reader will never get closed.

An Actual Solution

There is a way to fix this. You can require the caller to pass in a function to consume the generated data, then the callee can manage the resource and execute the function. It might look something like:

(defn process-the-file [some-file some-fn]
  (with-open [f (io/reader (io/file some-file))]
    (doall (some-fn (line-seq f)))))

(process-the-file my-file-name do-the-things)

Once upon a time clojure.java.jdbc used to have a with-query-results macro that would expose a lazy seq of query results, and you had these resource management issues. Then it was changed to use this second approach where you pass in functions.

There is a hitch to this approach. Now the callee has to know more about how the caller's logic works. For instance, in the above code you are assuming that some-fn returns a sequence that you can pass to doall, but what if some-fn reduces the sequence of lines down to a scalar value? Perhaps process-the-file could take two functions seq-fn and item-fn:

(defn process-the-file [some-file item-fn seq-fn]
  (with-open [f (io/reader (io/file some-file))]
    (seq-fn (map item-fn (line-seq f)))))

(process-the-file my-file-name do-a-thing identity)

That's better? I still see two problems:

The caller is back to having to know/worry about resource management, because it could pass a seq-fn that does not fully realize the lazy seq before it escapes the with-open
The logic hooks that process-the-file provides may never be quite right. What about a hook for when the file is open? How about when it is closed?

I could argue that this whole situation is worse, since the caller still has to worry about resource management, and now the callee has this additional burden of trying to predict all of the logic hooks the caller might want.

An additional design consequence is that you are inverting control from what it was in the lazy seq case. Whereas before the caller had control over when the data is consumed, now the callee does. You have to break your logic up into small chunks that can be passed into process-the-file, which can make the code a bit harder to follow, and you must put your sharded logic close to the callsite for process-the-file (i.e. you cannot take a lazy sequence from process-the-file and pass it to another part of your code for processing). There are advantages and disadvantages to this consequence, so it is not necessarily bad, it is just something you have to consider.

Another Solution

We can also solve this by using a different mechanism in Clojure: reduction. Normally you would think of the reduction process as taking a collection and producing a scalar value:

(defn process-the-file [some-file some-fn]
  (with-open [f (io/reader (io/file some-file))]
    (reduce (fn [a v] (conj a (somefn v)) [] (line-seq f))))

(process-the-file my-file-name do-a-thing)

While this may look very similar to our first attempt, we have some options for improving it. Ideally we'd like to push the resource management into the reduction process and pull the logic out. We can do this by reifying a couple of Clojure interfaces, and by taking advantage of transducers.

If we can wrap a stream in an object that is reducible, then it can manage its own resources. The reduction process puts the collection in control of how it is reduced, so it can clean up resources even in the case of early termination. When we also make use of transducers, we can keep our logic together as a single transformation pipeline, but pass the logic into the reduction process.

I have created a library called pjstadig/reducible-stream, which will create this wrapper object around a stream. There are several functions that will fuse an input stream, a decoding process, and resource management into an reducible object. Let's take a look at them:

decode-lines! will take an input stream and produce a reducible collection of the lines from that stream.
decode-edn! will take an input stream and produce a reducible collection of the objects read from that stream (using clojure.edn/read).
decode-clojure! will take an input stream and produce a reducible collection of the objects read from that stream (using clojure.core/read).
decode-transit! will take an input stream and produce a reducible collection of the objects read from that stream.

Finally, there is a decode! function that encapsulates the general abstraction, and can be used for some other kind of decoding process. Here is an example of the use of decode-lines!:

(into []
      (comp (filter (comp odd? count))
            (take-while (complement #(string/starts-with? % "1"))))
      (decode-lines! (io/input-stream (io/file "/etc/hosts"))))

This code will parse /etc/hosts into lines keeping only lines with an odd number of characters until it finds a line that starts with the number '1'. Whether the process consumes the entire file or not, the input stream will be closed.

Advantages:

This reducible object can be created and passed around to other bits of code until it is ready to be consumed.
When the object is consumed either partially or fully the related resources will be cleaned up.
Logic can be defined separately and in total (as a transducer), and can be applied to other sources like channels, collection, etc..

Disadvantages:

This object can only be consumed once. If you try to consume it again, you will get an exception because the stream is already closed.
If you treat this object like a sequence, it will fully consume the input stream and fully realize the decoded data in memory. In certain uses cases this may be an acceptable tradeoff for having the resources automatically managed for you.

Summary

Clojure affords you several different tools for deciding how to construct your logic and manage resources when you are processing collections. Laziness is one tool and it has advantages and disadvantages. It's main disadvantage is around managing resources.

By making use of transducers and the reduction process in a smart way, we can produce an object that can manage its own resources while also allowing collection processing logic to be defined externally. The library pjstadig/reducible-stream provides a way to construct these reducible wrappers with decoding and resource management fused to a stream.

Acknowledgments

Special hat tip to hiredman. His treatise on reducers is well worth the read. Many moons ago it got me started thinking about these things, and I think with transducers on the scene, the idea of a collection managing its own resources during reduction is even more interesting.

The March of Technology

2016-05-09T09:07:00.000-04:00

"Our inventions are wont to be pretty toys, which distract our attention from serious things. They are but improved means to an unimproved end, an end which it was already but too easy to arrive at; as railroads lead to Boston or New York. We are in great haste to construct a magnetic telegraph from Maine to Texas; but Maine and Texas, it may be, have nothing important to communicate. Either is in such a predicament as the man who was earnest to be introduced to a distinguished deaf woman, but when he was presented, and one end of her ear trumpet was put into his hand, had nothing to say. As if the main object were to talk fast and not to talk sensibly. We are eager to tunnel under the Atlantic and bring the Old World some weeks nearer to the New; but perchance the first news that will leak through into the broad, flapping American ear will be that the Princess Adelaide has the whooping cough. After all, the man whose horse trots a mile in a minute does not carry the most important messages; he is not an evangelist, nor does he come round eating locusts and wild honey. I doubt if Flying Childers ever carried a peck of corn to mill."

Thoreau, Henry David. Walden, and on the Duty of Civil Disobedience. Project Gutenberg. Web. 09 May 2016. https://www.gutenberg.org/

Or in the words of a more modern philosopher and poet:

Making Fake Things

2016-03-05T15:31:00.001-05:00

Software is fake. There are bits inside a computer represented by a magnetic or electrical charge or mechanical potential or some such thing. But software is not an electrical charge. Electrical charges can represent ones and zeroes and a series of ones and zeroes like "10111101" can represent the JVM opcode "anewarray" or the fraction one-half ("½") in the ISO-8859-1 character encoding or the number -67 in twos-complement. Software is not electrical charges, it is a particular interpretation imposed on electrical charges. An interpretation does not weigh anything. It has no color, taste, temperature, volume, mass, or any physical features. It is fake, but fake things can be useful.

Fake things can represent real things (or other fake things). For example, you can represent a couch with a 3D model in a computer. You can represent cities and towns and roads with fake things. You can also represent fake things with other fake things. JVM opcodes, characters, and numbers are all fake things represented by "10111101", which is fake. Fake things are useful because they can represent real and fake things in a way that can be cheaply manipulated and transported instantly across the world. Fake things also have challenges.

Software is a little unique even among fake things because in making software we are often making something that has never existed before. When someone creates a stove there are hundreds of thousands of others stoves in existence to draw upon. There are wood stoves, electric stoves, and gas stoves. But when someone created a text editor, they created something that had never existed before. Here is how Richard Gabriel describes it:

"But, consider the first people to design and build a text editor. Before that, there was never a text editor. Changes to a manuscript were always made by retyping or retypesetting. How would people want to make textual changes? How would people want to navigate? Searching? - no one ever heard of that before. Systematic changes? Huh? By the way, there were no display terminals, so how do you even look at the manuscript?" -- http://dreamsongs.com/LessonsFromNothing.html

Web applications, virtual currencies, automated theorem provers, and many other software applications had never existed before or were so different in nature from their physical counterparts that they were a unique thing. Making fake things is hard enough, but making things that have never existed before is that much harder. That's not the end of it, though.

Fake things have no real world to help co-design them. Stoves have a real world to help co-design them. There are accessories that are used with stoves that help co-design them. Real things like pots and pans. Stoves have to fit through doorways, nestle between kitchen cabinets, and match the colors on the walls. Text editors have accessories like keyboards and mice that were invented to give real people made of meat a way of manipulating a conceptual world by proxy. Perhaps a mouse has to be compatible with a human hand, but a text editor has to be compatible with the mental model of a text editor that exists in a human mind, a model which no one had ever thought of before. Ultimately making software is a process of collaborating with other humans to dream up some mental model, and then making a fake thing out of software that other humans can use to manipulate that model (assuming they properly understand the mental model).

Which reminds me, collaboration is also a fake thing. Collaboration is about using real things, like vibrating air, to push around fake things, like words. It is about using real things, like markers and whiteboards, to manipulate fake things, like ideas. All of these real things can be replaced by fake things, like video conferencing software and text editors. And fake things like words and ideas can be replaced by other fake things, and all of these fake things can be instantly transported, copied, and manipulated by real people in real (and very distant) places. Collaboration is not a real thing, it is a fake thing produced through the interaction of real people thinking creatively.

And making software is a creative act. Writing software is writing instructions to make a computer do something. You must choose the instructions, determine their order, name things. You develop your own style. Writing software is writing words that have effect. Writing software is as close as you can get to God with words speaking reality into existence, the ultimate creative act. But writing software is not just for telling computers what to do. It is also collaboration with other humans. They must read, understand, modify, and extend what you write. They must understand your vision. You must collaborate with them through your source code.

So, here we are. We have discovered that software is a fake thing, that it is often an entirely new thing, that it is a pure product of the mind, that it is born of collaboration, and it is creative expression. Now what? We must systematically question the constraints we place on ourselves, because those constraints are often meant for real things and our things are fake. Here are a few examples:

A top-down management hierarchy is for making real things, not fake things. Top-down, command-and-control hierarchies are about control and efficiency. Control and efficiency are important for real things, because real things have locality, cost, and scarcity. Software has none of these things. Control and efficiency are important when you are manufacturing the same thing over and over. Software is often exploratory. Software is valuable not because we repetitively make lots of little copies of the same thing, but because we dream up some new way of doing things that has never been done before. Control and efficiency are important when you have a predictable process. A creative process is not predictable. You may think for hours about a problem, sleep on it, and then have the answer pop into your head the instant you wake up. We need to think differently, not just about what we make, but how we make it.

Offices are about locality. An office puts materials, means of production, and managers in the same physical location. Yet with software there is no material and the means of production are mental. There is no reason to be concerned about locality. Ostensibly having a bunch of people in the same office enables them to collaborate, but collaboration is a fake thing. Collaboration does not exist in San Francisco or Saint Louis. It does not weigh 1kg. It is not blue. Having an office for collaboration is a rationalization that projects the past onto the future. Is collaboration different using video conferencing and Google Docs than it is using tables and chairs in an office? Yes, because fake things are different than real things. I do not recommend mixing fake things like video conferencing with real things like offices. It may take getting used to, but embracing the fakeness of collaboration has advantages like hiring people where they want to live instead of trying to convince them to live where you live. It also means having permanent, searchable, modifiable artifacts that can be shared instantly across the world, instead of a whiteboard in a room.

Software can process data, but software is also data. This creates leverage. You can flip a bit, and that bit can flip ten others, and those ten another one hundred, etc. Compilers, build tools, continuous integration, and automated tests are all software doing things to software. "The cloud" has created a lot of leverage because it took something that was real (a machine) and made it fake (a "cloud instance"), and once it is fake it can be manipulated by software. The higher you can climb the mountain of abstraction the more powerful you will become. Before selling to Facebook, WhatsApp had ~450 million active users and ~55 employees. Yahoo has ~12,500 employees. I don't know how many active users they have, but let's just pretend it is ~450 million. Don't be Yahoo.

These are just examples, and you can agree or disagree. My point is, we as an industry can achieve market success and realize our visions much more powerfully, but we must understand the nature of the software we are creating (it is fake), and the newness of what we are doing every day, and its collaborative nature, and the tools that we can take advantage of, and we must have the courage to give up on arbitrary constraints that are optimized for making real things. We must pursue leverage, because leverage will enable us to do amazing things.

The following is a quote from an astronaut speaking about a spacecraft

2014-05-30T14:20:00.000-04:00

"There's so much more elbow room in there compared to the Soyuz," he said. "Instead of just bringing two of your buddies, you can bring six ... It's got modern electronics, it's got modern materials in the heat shield. So technologically, it's a giant leap beyond the Soyuz."

SpaceX reveals new-look passenger spacecraft

One could think of a number of snarky comments that could be made (especially given that Elon Musk is also the CEO of Tesla), but I'm kind of amazed that this is how human beings are speaking about spacecraft these days.

Perhaps it should all be taken with a grain of salt since this quote was spoken at a publicity event by someone who works for SpaceX.

Still...how long until I can take a vacation to Mars?

This, But For Computer Programming

2014-05-10T09:20:00.001-04:00

If you meet a philosopher on a train and ask him his profession, he is likely to lie. It is not that philosophers are especially prone to lying, but rather that philosophy is a peculiar profession. To tell your fellow passenger that you are a philosopher opens up an awkward line of questioning. ... If you take the plunge, however, and accept the label of philosopher, you must be prepared for the disappointment when your listener hears that you don't live in a hut on a mountaintop, haven't uncovered the secret of life, and cannot explain why the world exists. If you are foolish enough to go further and attempt to describe your lifelong attempt to reconcile the epistemology of mathematics with its ontology, be prepared to encounter a look in which boredom and horror are blended equally. Best, therefore, to say simply that you are an architect, and leave it at that.

— A World Without Time, Palle Yourgrau, page 164.

Some possible substitutions:

Philosopher	Computer Programmer
he/him	she/her
live in a hut on a mountaintop	live in your parents basement
uncovered the secret of life	know why my computer is running so slowly
explain why the world exists	install the driver for my printer
reconcile the epistemology of mathematics with its ontology	figure out the best cache invalidation strategy to provide a balance of performance and freshness

As a corollary, I wonder if there is a t-shirt for philosophers that is congruent with the "No, I will not fix your computer" t-shirt for computer professionals, something like "No, I will not fix your worldview."

The Solution for Silicon Valley?

2014-01-14T06:49:00.000-05:00

"People—especially in the financial community—seem to assume that every industry works kind of like a web startup, and that all you need is two hot guys and $25,000 and you're a millionaire in six months. Heavy semi doesn't work like that. Heavy semi is like steel mills and railroads. By the time you can get a serious semi company self sustaining you're looking at a couple hundred million dollars of investment." — Ivan Godard of Out-of-the-Box Computing (https://www.youtube.com/watch?v=uotQn-jrAZU#t=599)

It's a longer conversation to have, but I have opinions about Silicon Valley (that I share with others). However, it is nice to see people trying to create a real business and solve real problems instead of creating the next Instagram clone. The thing is that it takes hard work, many years, and the perspective to step back and work towards fundamental—not incremental—change.

Best of luck to the OOTBC guys! True, a new CPU architecture isn't exactly solving problems of justice and social inequality, but at least it isn't a get rich quick scheme. Finding ways to fill in computation at neglected price points in the market is a stepping stone to solving other problems. Hopefully those problems aren't how to gamify haircuts or something. *sigh*

Let's have some get-rich-slow startups that are truly innovating to solve difficult, fundamental problems.

The Buck Stops Here

2014-01-08T06:19:00.001-05:00

"Rufer explains the thinking behind the process: 'I was signing checks one day and I recalled the saying, "The buck stops here." I thought to myself, that isn't true. In front of me was a purchase order, a note that said the stuff had been shipped, we had received it, and that the price on the invoice matched the puchase order. A check had been prepared. Now, do I have the choice not to sign the check? Nope. So the question isn't where the buck stops, but where it starts—and it starts with the person who needs the equipment. I shouldn't have to review the purchase order, and the individual shouldn't have to get a manager's approval.'"

"First Let's Fire All the Managers," Gary Hamel (https://archive.harvardbusiness.org/cla/web/pl/product.seam?c=573&i=15715&cs=7c855bfce2fd1c3860846954978b1181)

NixOS: A Field Report

2013-12-30T11:03:00.000-05:00

Why?

NixOS is a relatively new Linux distribution based on the Nix package manager. A Nix package is hashed based on its content, dependencies, compiler , etc. and stored immutably in the Nix store. The Nix package manager uses this Nix store, a patched dynamic linker, and a buttload of symlinks to isolate installs. You can have multiple versions of applications and libraries living on the same machine without conflict.

The Nix package manager also maintains a history of installations and—since the Nix store is full of immutable installs of applications and libraries—Nix provides a way to rollback installs to a previously working configuration. NixOS extends this concept all the way to the Linux kernel. When you change the system configuration for your machine you get a new GRUB entry for that entire profile of your computer: kernel, applications, libraries, and even your configuration files. If something goes badly you just reboot and select the GRUB entry for your previous configuration. Check out section 1.4 of the NixOS manual for some cool details http://nixos.org/nixos/manual/#sec-upgrading.

But wait there's more! These installations are atomic, so it either completes successfully or not at all. So basically what we get here is something like an ACID MVCC for the software on your computer.

There are definitely practical benefits to all this. There have been times that I've done some riskier upgrades. Moving to a new Ubuntu release is one of those times. There's always a bit of a question as to what may or may not be broken. I usually back up my files and do a fresh install. I've also been bitten a couple of times by a bad video driver upgrade that was annoying to recover from. That said, I don't know that I've really needed an ACID MVCC for software all that often.

Aside from riskier installs you can also setup a Nix configuration for each project you work on. You can chroot into this Nix environment. The Nix configuration can be version controlled, so that when you checkout an old branch you can use the environment that matches that version of your project. You can also take this Nix configuration and create a VM from it, or send it off to an EC2 node. Admittedly you can do all of this stuff just using the Nix package manager on Linux or OSX without needing NixOS. (Here is how http://zef.me/5966/setting-up-development-environments-with-nix.)

So, really...Why?

So all this stuff is great, but maybe not worth switching to NixOS for your day-to-day desktop machine. I have moved for a couple of reasons. Lately I've been getting disillusioned with the direction Ubuntu is headed. I barely made it through the last LTS upgrade. I prefer to use Gnome and XMonad, but Ubuntu comes with Unity these days and it's hard to get a plain Gnome install (though I made it work). The flip side is that Ubuntu is a really smooth install where most everything just works out of the box, but that's not really enough to keep me there. I'm basically forced to consider some kind of a major change whether it is a change in distribution or the tools I use or something.

I also like to try to change things up every once in a while. I've been on Ubuntu for close to 10 years and Gnome+XMonad for maybe 3 years. I need to shake things up a bit, learn some new stuff, and feel like a hopeless newb. Boy did I ever accomplish that!

Attempt 1: No Gnome & Miserable Failure with Wireless

I downloaded and burned the NixOS 13.10 minimal CD. It booted up fine and gives you hints to walk you through the process. The CD boots with a copy of the manual on one of the Linux consoles so you can jump back and forth to reference the manual as you're installing. The process is pretty simple, too. You format a partition, mount it, run a config generation program, and customize the generated config. Once the config is set you run an install command and boot into your new NixOS installation.

However, Gnome doesn't install. Based on a reading of the docs you might be led to think that it does, but after 13.10 was released this commit was made to the Nix packages: https://github.com/NixOS/nixpkgs/commit/a734f32fa1efce21a008e391cfb10695c4d738cd, so it's not even an option in more recent versions of NixOS.

Coincidentally XMonad doesn't install either. Well it seems like it does, and if you use it without any custom config then it's probably fine, but to use a custom config you need a Haskell compiler and the XMonad packages. For some reason when the XMonad packages install they don't install the dependencies that they need (which seems broken given how Nix is supposed to work). I had to iteratively run `ghc-pkg check` and install a bunch of packages manually. Perhaps I was missing something. Also, not to give away the ending, but this was with 13.10, so it may be a different story with the latest unstable NixOS.

Finally, I had wireless issues. The installation CD booted and I was able to get an install, but once I tried to find some combination of CLI and/or GUI software to configure my wireless adapter I wedged something, and decided to just reboot and reinstall, but the installation CD was hanging when starting the WPA supplicant. It seemed maybe there was some state stored in the wireless card that was causing the WPA supplicant to hang, but I couldn't boot NixOS in order to configure the card. After many hours fiddling with this to try to get the wireless working and being very frustrated I very nearly gave up on NixOS. As a last ditch effort I decided to try using the unstable version of NixOS.

Attempt 2: Using Unstable NixOS

Out of the gate Unstable NixOS was more stable than the 13.10 release. I was able to actually boot the thing! I made it through the installation process. I had decided that I was going to use Xfce+i3 instead of Gnome+XMonad. Changing my desktop and window manager at the same time as changing my distribution was going to make this whole process more "interesting."

Configuring NixOS is relatively simple once you know what the configuration options are, but finding them isn't as easy as it could probably be. The NixOS manual has a section of configuration options at the end. That was an important reference. I don't know if it documents every possible option, so sometimes I would take a look at the Nix package source. There are also configuration options that affect some packages that aren't entirely obvious unless you look at the package source. I think in the future it would be nice (in addition to the way the documentation is currently organized) to have documentation that can be browsed per package, so you could see for a particular package what options affect it.

Having a working distribution is since, but there were still some annoyances. Searching for packages is also a bit unsatisfying. The recommended way to search seems to be `nix-env -qaP '*' | grep NEEDLE` which works, but it's not the most performant, and you also can only search by package name; you can't search by description or configuration options or anything else. Even when you've found the package you want more detail on, it is not obvious how to find the package. You can browse the nixpkgs source, but it is organized by categories. The best way seems to be doing a `find / -name NEEDLE` or something. None of this is really a deal breaker (for me), but just lessons learned on how to find information.

Finally, on the subject of finding information the nix-dev mailing list is a good resource. I ended up searching it with google "site:http://lists.science.uu.nl/pipermail/nix-dev/ NEEDLE". I guess it is also possible to search the Gmane archive http://news.gmane.org/gmane.linux.distributions.nixos.

Adaptations

There were definitely tools that I'm used to using that were available and worked great on NixOS, among them were Conkeror, Emacs, OpenJDK, Ruby, Tmux, and Skype.

There were replacements I had to make. Obviously Gnome+XMonad became Xfce+i3, and this had a cascading effect.

I had a raise-or-run script that used wmctrl to activate a program's window, but wmctrl doesn't work with i3, so I had to replace that script with one that uses i3msg.
I spent a bit of time tuning my i3 config to have bindings similar to the ones I was using with XMonad.
Previously I was using gnome-panel for workspaces, wireless control, sound volume control, date/time, etc.; that got replaced by i3bar and i3status.
I was using swarp to shuffle the mouse cursor out of the way in Gnome+XMonad, and that package wasn't available on NixOS, so I replaced that functionality with xdotool.
I tried to get NetworkManager up and running, and there appear to be configuration options to get it set it up on NixOS, but I ran into issues. I ended up just installing the wpa_gui tool to manage my wireless connections.

I also had a cache of scripts that I used for various things. Some of them were for building programs from source, but I think I can drop those and just use Nix to accomplish the same thing. Others I had to replace '#!/bin/bash' with '#!/usr/bin/env bash' since NixOS doesn't have things at fixed, standard locations like other Linux distributions.

There are some other minor issues to work out. For one my sound card seems to be detected just fine, and Skype will use it, but Xfce doesn't seem to be playing any sounds. I just get a system beep for "sound" events. I think this is an issue with the configuration of Xfce and its sound theme or something. Not really a major issue for me, but I'd like to figure it out eventually.

I would also like to get Dropbox installed. It actually worked fine on 13.10, but for some reason it isn't working so well on unstable. I haven't had a lot of time to look into it, and again not a high priority. I may take the opportunity to look into OwnCloud instead.

Conclusion

I went into this expecting it to be rough at times, and expecting to make some changes to the tools I was used to using. That was no surprise. I was a bit disappointed in how the latest official release of NixOS turned out. The flip side is that the Unstable release has worked great so far, and I'm a little more comfortable following the bleeding edge of NixOS given that (theoretically) I should be able to rollback any changes that bork my system. I'm hoping that turns out to be true.

If you're willing to learn something new and possibly snag your sweater on some rough edges, then NixOS could be a fun experience for you. :) If you use some of the same tools as me, then hopefully what I've discovered can help you move to NixOS. If you want something a little more polished, then you may want to wait on NixOS.

References

My configuration.nix
https://gist.github.com/pjstadig/8183688
Dotfile changes moving from Ubuntu+Gnome+XMonad to NixOS+Xfce+i3
https://github.com/pjstadig/new.dotfiles/compare/455374...master

A Proof of the Braininess of "The Simpsons" and "Futurama"

2013-12-22T09:10:00.001-05:00

Occasionally the mathematics does wind more deeply into the story, most notably in the 2010 “Futurama” episode “The Prisoner of Benda.” The plot turns on a device called the Mind-Switcher, which performs just the function its name suggests; over the course of the episode, as the apparatus is used with greater and greater abandon, the minds of the characters shuttle from body to body like singles switching bedrooms in a French farce. By the end, not a single consciousness remains in its proper skull. What’s worse, the characters can’t just retrace their steps to reunite each mind with its original body; the Mind-Switcher, having operated on a pair of minds, isn’t allowed to switch the same two minds again.

Ken Keeler, who wrote the episode, realized that in order to get everything sorted out it might be necessary to introduce new characters, whose bodies could be used as waystations through which the minds could find their way home. An ordinary writer would have been content simply to find a way out of the episode. But Keeler became obsessed with the problem in its full generality, finally composing a proof that, no matter how wild the original fiesta of Mind-Switching, the damage can always be repaired once two new people are added to the system. This question may sound abstruse, but the part of math it belongs to — “combinatorial group theory” — is one of the hottest things going at the moment, with major advances popping up everywhere from Paris to Los Angeles. Keeler’s theorem isn’t one of those big advances, but it’s a real theorem, certainly the deepest piece of mathematics ever featured in a prime-time sitcom.

-- Jordan Ellenberg, "Mathematics and Homer Simpson" (http://www.washingtonpost.com/opinions/2013/12/20/ef1bfaa6-5b9a-11e3-bf7e-f567ee61ae21_print.html)

The Notion of "Real-Time"

2013-12-16T06:09:00.000-05:00

"I think the current evolution of technical language around web developers has made real-time mean: 'consume information as soon as it is available' and not '_react to the information in a timely manner or this car will crash_'."

-- Alvaro Videla, "Tell me more about your real-time systems" (http://videlalvaro.github.io/2013/12/tell-me-more-about-realtime.html)

Remote != Distributed

2013-07-31T08:46:00.001-04:00

In an age when companies seem to be shutting down their remote work programs, I'm here to tell you that a distributed team can work, and when it works well it is a great experience. I have worked on a distributed team for the past three years. Prior to that I worked for four years in an office that had one full-time remote employee and allowed occasional work from home for others.

Working from home and working remotely are not the same thing as working on a distributed team. If there is a center of gravity at an office and a few remote employees orbit that office, then there is a different experience for some people than for others, and there is a different way of working for some people than for others. There is a natural imbalance that an inanimate system would want to equalize, but in a human organization people pull in opposite directions.

A distributed team is a team where no one is physically co-located. There is no center of gravity at an office. There is no imbalance, because everyone has the same experience, and everyone has the same way of working.

When there is an office/remote split, there is a different experience for some than for others. The office people have shared experiences like water cooler chat and birthday celebrations, and the remote people are left out. The office people gather in a conference room for a meeting and dial in the remote people over a crappy speakerphone. The experiences are different which affects morale and cohesion.

On a distributed team everyone has the same experience. You find new ways to create shared experiences that are different from office experiences and unique to distributed teams. You use tools that allow meetings to be high quality experiences for everyone. These shared experiences support morale and cohesion.

An office/remote split means different ways of working for different people. In-office employees may depend on ad hoc hallway conversations, but remote employees are not privy to those. In-office employees have an "out of sight, out of mind" mentality and think that remote employees aren't productive because they aren't communicating with the remote employees. Remote employees feel disconnected and lack direction for the same reasons.

On a distributed team everyone communicates through the same channels, and everyone can see the activity about who is doing what. Since there aren't two different communication channels there isn't an "out of sight, out of mind" problem.

When there are different experiences, different ways of working, different communication channels at the same company, then two different cultures develop, and that is detrimental. A permanent office/remote split is doomed to failure from the beginning, so it's no wonder that companies are killing their remote work programs.

What does it look like to do a distributed team right? I can tell you what has worked for me and my company. I do not expect that our experiences are universal, but I will try to draw some general principles.

Distributed teams that work well:

Use tools to fix work into a tangible medium of expression (to borrow from copyright law). Office collaboration via ad hoc hallway conversations and whiteboards doesn't work for distributed teams. You have to produce Google Docs, tickets in Jira, etc. These are things to which everyone can have access. Frankly, distributed or not, this is a better way of working than informal ideas banging around in people's heads.
Communicate through many-to-many channels. Of course there is still a need for one-to-one communication, but using many-to-many communication channels means that everyone can feel a part of the company. Just like others can walk by and join in on hallway conversations, people can observe and contribute to Skype calls and chat rooms. It is also a good idea to have logs of these chats that people can catch up on (see above).
Adopt processes that encourage collaboration and productivity. Working on a distributed team may not be for everyone; to a certain extent it does take a self-motivated individual. However, pair programming and daily standups create an environment that encourages collaboration and accountability. We also tend to hire people mostly in North and South America so that the timezones line up better for collaboration. We also meet face-to-face 3-4 times a year, which helps to develop the personal relationships that are necessary to working well together.

Realize that the idea of "remote" working does not work when combined with an office culture and a center of gravity in one location. Working from home or remotely may work temporarily, but eventually gravity will pull you back towards the office. If you try to maintain this unnatural office/remote balance you will enforce different experiences and methods of work and create two different cultures in your company. This cultural divide will lead to jealousy, resentment and other morale and cohesion problems.

This does not mean that it is impossible to function unless everyone reports to the same office. It just means that different tools and processes must be developed to facilitate a distributed team. You must use tools and processes that encourage tangible expression of ideas, many-to-many communication, and collaboration and productivity.

Remote working is isolating and doomed to failure, but a distributed team done right is a joy.

Evaluating CLEAR Wireless Internet

2012-11-28T09:00:00.001-05:00

Here's my current situation: I have Cox High Speed Internet at home in Fairfax County, Virginia, and I have a Verizon Wireless smart phone with the HotSpot feature enabled. I've been intrigued by Clearwire for a while, and I'm always on the hunt for improving my Internet access at home and on the road. Though I've been fairly happy with HotSpot access through Verizon (though not so happy with my HTC Rezound, but that's a story for another time), Clear has a 15-day money back guarantee, and no contract, so it seemed like a no brainer to try them out. Also, I checked their coverage and I should get a good signal at my house.

I think in an ideal world I'd like to decouple my smart phone and my mobile Internet. I'd like to have a smart phone with minutes and texting, and connect it to the Internet (as well as other devices) through a mobile HotSpot, like Clear. However, it doesn't look like that plan is going to come to fruition any time soon. For one thing, Verizon won't allow me to have a smart phone on their network without some kind of data plan.

How does Clear stack up?

The experience signing up for and using Clear has been really great. (I'll let you know how it goes returning the device and getting my money back...but I'm getting ahead of myself.)

I signed up on their website, and received the device the next day. I plugged in the device and it connected to the 4G network. I was able to connect my computer to the HotSpot device without any problems at all.

I tested the "4G Internet Basic" plan which has an advertized speed of 1.5Mbps down and 500Kbps up for $34.99 per month. According to speedtest.net I was getting 1.92Mbps down and 460Kbps up with a ping of 72ms.

It looks like you get what you pay for, and the experience is very smooth, but for me the price/speed isn't as good as I can get elsewhere. The HotSpot on my smart phone gives me 9.93Mbps down and 3.21Mbps up with a ping of 58ms for $30 on top of my unlimited data plan. (I'm grandfathered in on the Verizon plan I'm on. I'm not sure what the numbers would look like if I were to sign up today.) And my Cox High Speed Internet gives me 39.48Mbps down and 12.33Mbps up with a ping of 18ms for $40 per month.

Clear has a "4G Internet Plan" which has an advertized speed of 6Mbps down and 1Mbps up for $49.99 per month, but that is getting on the steep side for me, and the analysis doesn't help Clear.

Comparing my options, $1 will purchase 1Mbps from Cox, 0.333Mbps from Verizon, 0.059Mbps from the 1.5Mbps Clear plan, and 0.122Mbps from the 6Mbps Clear plan.

Conclusion

I would really like to use a service like Clear, and I have no complaints about the Clear experience. However, it is not the best value for me. If Clear was charging $6 per month for their 6Mbps plan, I'd be all over it, maybe even if they charged $10 per month for their 6Mbps plan. However, as it is, I'm going to pass.