Bottom line: No effect.

Research:

I spent most of yesterday and today reading up on basic pharmacokinetics. (Last year, I had a mathematician girlfriend who worked in drug discovery. She explained enough to me that I was able to get a running start researching the details.) Tomorrow, I’m going to simulate a 30 mg dose by altering the timing of when I take two 20 mg tablets.

Bottom line: No effect.

Bottom line: No effect. Today was just like any other day before taking Ritalin.

Confounding factor:

I stayed up almost all of the previous night. To my surprise, “Could Grad School Be Fixed?” got a fair amount of traffic: about 1,000 unique visitors. Despite feeling very sleepy, I kept watching tail -f access.log. I watched it non-stop (sometimes in a background window) for 15 hours straight. I picked up a bunch of basic insights into blogs, partly from noticing patterns in the log file, partly from reading up on various things suggested by what I saw: the Feedburner help, various Google searches, various aggregate data in Google Analytics, a closer look at the HEAD section generated by WordPress, etc.

It felt good to stare at one thing for a long time like that. I’ve only done it once or twice in the past year (since starting grad school). Contrary to my usual daytime experience, I never felt overloaded or distracted. Is this what they call “hyperfocus”?

Some research:

I did some googling, and learned that my experience of Ritalin as “calming” rather than “speeding up” is actually pretty typical for people with ADHD. The “speedy” sensation is more often felt by people with “normal” executive function.

This is sort of obvious when you think about it. The common scolding to “keep it simple” is an attempt to directly oppose the natural urge to make things complicated. Hence the tone.

People want to make things as complicated as they can handle. The musician wants music as complex as he can possibly play, the dancer wants a dance that’s complex enough to be a challenge, the mathematician wants complex puzzles that are just at the edge of his skill, the French cook wants to make dishes that require complex techniques, the gymnast wants to learn more-complex routines, etc. The drive for complexity is just part of the drive to excel. By pushing your talents as far as they can go, you naturally create complexity.

People want other people to keep it simple.

We don’t like other people’s complexity, but we love our own

When you ask a stranger for directions, you generally don’t want the most efficient route, you just want a simple route that will get you there. A guy in San Francisco once asked me to get to the Golden Gate bridge (this was at a gas station diagonally across town), and I drew him “the simple map of San Francisco”: just the three main streets, including the one that eventually gets close to the GG bridge, and from there you can follow the signs. He was delighted! Then I realized that if he had a little more detail and context, he would be able to recover if he got lost, and so I started to tell him some more. He stopped being delighted.

This principle occurred to me while I was installing some open-source software. For pretty much everything that you want to do, there is some other piece or something that you need to download. And then you run into problems installing it, or it’s hard to determine which is the right thing you need, etc. There is much scolding in the forums for people to “read the FAQ”, but the FAQ never tells you all the information that you need to answer a question. Almost anything that people were actually having trouble with could be explained simply, in a couple paragraphs or with a table. But where’s the joy in that—I mean, for the people who make the software? You’re supposed to read lots of little pieces in lots of places and mentally put them together, and thereby grasp the full complexity, see every detail in its true context. I, however, am frustrated, because I don’t want the complexity, I just want to use the program.

It’s partly the joy of complexity, and partly ego trip. If you just answered someone’s question plainly, they wouldn’t partake in all that delicious complexity, and what pride is there in understanding something that can be explained in a paragraph or two? The <20% of complexity that buys you >80% of results is easy, but mmmm, just look at how much delicious hard work there is in covering those last <20% of achievable results. That last, asymptotic 1% is really juicy. Just ask anyone who’s taken real analysis.

More examples

Perl is a seduction by increasing complexity. Look, you don’t need to know all of Perl, you can learn “baby-talk” Perl and get results right away! And now, look here, there’s a special syntax for this, and a clever little trick for that, and a CPAN module, and, and …

Bureaucracies illustrate a different sort of complexity. Bureaucracies exist to make sure that certain things happen reliably—which is to say, to make sure that bad things don’t happen. Pretty soon, though, some borderline cases come up. These need to be ruled on by experts; can’t have laymen in the field deciding these important things. As more and more rules accumulate, delivering the experts’ knowledge of the full complexity of these situations, the people in the field gradually lose their ability respond intelligently to just the present situation. And then you get incidents like when airline passengers are stranded on the tarmac for nine hours because the crew lacked the authority to let the passengers back into the airport.

It’s easy to ridicule other people’s complexity. However, that would miss the point. We all love to make our own kind of complexity. A life of simplicity would be dull. The simple life is not worth living. You could write up some simple rules to handle all of life’s basic physical needs, and you could probably get it all done in a few hours each day, with very little mental effort. And then what would you do?

The moral

As people seeking happiness and synergy in the social world, we should do three things:

1. Provide simple interfaces to services to …
2. Enable other people to make their own lives complicated, and do so in a way that allows us to …
3. Make our own lives complicated (in the particular ways we enjoy).

In other words, simplicity at the interfaces, complexity inside the hubs.

Also, telling people to “keep it simple” just ticks them off.

Bottom line: No effect. Today was just like any other day before taking Ritalin.

Confounding factor:

I stayed up almost all of the previous night. To my surprise, “Could Grad School Be Fixed?” got a fair amount of traffic: about 1,000 unique visitors. Despite feeling very sleepy, I kept watching tail -f access.log. I watched it non-stop (sometimes in a background window) for 15 hours straight. I picked up a bunch of basic insights into blogs, partly from noticing patterns in the log file, partly from reading up on various things suggested by what I saw: the Feedburner help, various Google searches, various aggregate data in Google Analytics, a closer look at the HEAD section generated by WordPress, etc.

It felt good to stare at one thing for a long time like that. I’ve only done it once or twice in the past year (since starting grad school). Contrary to my usual daytime experience, I never felt overloaded or distracted. Is this what they call “hyperfocus”?

Some research:

I did some googling, and learned that my experience of Ritalin as “calming” rather than “speeding up” is actually pretty typical for people with ADHD. The “speedy” sensation is more often felt by people with “normal” executive function.

My first day taking Ritalin:

At the coffeehouse, I mostly wrote a post for a discussion list.

The occasional noise of people speaking and baristas making things with loud machines didn’t particularly bother me. Often it does, especially if I’m trying to concentrate.

As an experiment, I tried multitasking a bit. I interrupted what I was doing to write a personal email and check the Google Analytics page for this blog. I didn’t get confused and flustered and hurried, as I often do. I wasn’t particularly annoyed by slow-loading web pages. I didn’t fear that my train of thought would get lost if I had to pause to wait for something to happen. That loss of train of thought has been a very big problem the last few months.

Bottom line: There was a slight calming and a slight improvement in my ability to stay with a task without “losing my place”.

Confounding factors:

• I got very little sleep the previous night.
• In the morning, I had an experience that got me focused: packers came over and packed up all my stuff for the move tomorrow.
  

  Beforehand, I wanted to wash some dishes and throw out as much stuff as possible, but I was unable to will myself to focus on those things. When the packers arrived, the pressure got me into action. I washed some dishes and threw out some obvious garbage while the packers did the living room.

• I had spent the afternoon writing a summary of a new and somewhat different creationist’s argument, and posting it on a discussion list where his idea had been ridiculed, in my opinion, unfairly. I had skimmed the author’s somewhat rambly treatment the previous day, so my brain was already seeded and the ideas were ready to coalesce into a terse but thorough exposition. In other words, my usual creative process of “seeding, incubation, and trance.” The trance came in the afternoon. As a result, around 6:00 p.m., my mind was pretty clear to start with.
• After taking the Ritalin, I tried writing up an explanation of what’s wrong with the argument, and why it is worthwhile to address objections to evolutionary theory (rather than, say, responding with derision). I was still cruising on cognitive momentum from the preceding couple of days. So, the evening was not a good test of whether Ritalin will help me direct my attention volitionally.

• Divide semesters into two-week intervals (or something about that long). During an interval, you only work on one or two things, and you work on them all day, every day.
• Instead of classes that consist of lectures, work on projects. A little lecturing can work wonders, so lecture occasionally. And then allow time for the wonders. Those happen when the student does something real. The instructor’s main job is to provide some structure by helping define do-able projects that really teach you something, and to help you get unstuck when you get stuck.
• A good daily schedule for a semester: working session in the morning, lunch, working session in the afternoon, 90 minutes in the late afternoon for lectures and presentations. These can be given by professors, researchers, guests, and grad students.
• Instead of meaningless homework problems (math puzzles that have no apparent use, essays that are not interesting to any reader, etc.), the projects should do something real. For example, in a course on software engineering, you should engineer real software. In a course on persuasive writing, you should write with the aim of persuading a target audience of something. In some topics, like history, writing papers might be the only feasible option, but in that case, writing papers is the real thing. Ph.D. students should spend most of their time doing real, publishable research alongside their professors, working as apprentices.
• Close the feedback loop. When the student works on something, let the student see and judge the results. For example, let the journalism student compare his or her story with that of other students in the same class, and get comments from the people interviewed. Let the education student see how well his or her own child students pass a test of what they were taught.
• Separate learning from certification, and dispense with grades. In courses that people take because they want to learn, nobody bothers with grades. (Almost all courses taught outside schools are run this way.) Students cram for exams because their schedules are crammed and because grades are used as carrot/stick every single day. Let actual learning be the carrot. A major project, a certification exam, a thesis and defense—these can provide a basis for prerequisites and granting a degree. If a student does badly in a course, let the only penalty be retaking the course.
• Instead of threatening students with low grades, have a time limit. Students can do as they please for three years, and then they’re done, period. They can go for certification any times they like during that time.

Teaching and learning are well-understood—outside of schools. People are naturally good at teaching and learning. Just see how people really learn, and provide an environment that supports learning instead of opposing it.

“Example is not another way to teach. It is the only way to teach.” —Albert Einstein.

Objections

The academic world exists to cultivate knowledge of a sort that doesn’t lend itself well to measurement and feedback loops. For example, philosophy and literature.

The activity that counts as “doing the real thing” can vary greatly from subject to subject. The main function of philosophy, literature, and history is to build wisdom: the kind of very rich, contextual knowledge that enables you to tell good from bad, important from trivial, and, especially, to notice important things that the uneducated might pass over. An interval of any of these topics can consist of reading a book, writing papers, and then talking about what you might approach differently now. A particularly interesting way that I have seen a course run had the students declare their opinions about the answer to a certain, given question at the beginning of the course, read various readings, and keep a journal of how their answer to that question changed each week.

Don’t you need to take a lot of classes to learn the field before you’re ready to do real projects or research?

No. Lecture-and-exam classes don’t work. We’ve been doing them for hundreds of years now, so there is no more doubt about that.

You pick up all that needed stuff by doing real work. When you find a gap in your knowledge, you fix it. The instructor might give you an exercise to help you fix it. Let this happen naturally, in the course of doing real work. When it arises naturally, the student sees why it’s important. School becomes learning, not engaging in meaningless exercises that theoretically “prepare” you for something—something that occurs so long after the preparation that the preparation is wasted.

With some subjects, there is simply a vast amount of material to memorize, which requires many sleep cycles. It can’t be done in two-week blasts. For example, mathematicians must learn hundreds of theorems, botanists must learn hundreds of plants, etc. This favors long semesters. And long semesters require multiple classes, or you won’t cover a broad enough set of topics.

This is indeed a potential problem. The 90-minute period at the end of each day provides opportunity for memorization-oriented activities. The daily work sessions provide ample opportunity to put that memorization into practice right away, or uncover areas where memorization is needed.

Tournament-level Scrabble players memorize even more than grad students. They often work regular jobs, and regular jobs usually focus on just one thing all day. If Scrabble players can do it with this kind of schedule, surely college students can. But, indeed, the details of how will need to come from experience. I don’t have a really solid answer right now.

Not everyone likes to work on only one or two things at a time. Some people like lectures and taking lots of simultaneous classes.

People like that do exist, but they’re a tiny minority. Virtually every college in the world caters to them and wastes time for the rest of us. Let’s try school in a way that works for most people, somewhere.

There is also the distinct possibility that the lecture-and-exam crowd isn’t really learning anything beyond how to succeed at the lecture-and-exam game, but that is an open question.

This has never been tried. How do you know it would work?

I suppose I don’t. Well, it’s been tried in almost every line of work that doesn’t call itself a school, and it works very well there. It’s standard operating procedure in almost every human culture. The basic formula: do the real thing, but let beginners do it alongside people with more experience so they can get help when they need it.

A sensible way to try this in a college would be with some sort of pilot program with a small set of students and participating professors. Do it on a small scale, see how it goes, repeat, move up to larger scales when you’ve built up some confidence. Of course this is the sensible approach to any new idea. You already know that, and you don’t argue about it. It’s also the way I’m proposing that students learn in school. Think about how insane it would be to run real projects the way classes are ordinarily run: lots of lectures and out-of-context homework and exams—OK, now we’re ready to go live.

Wouldn’t people slack off without grades?

Do people slack off when they pay hundreds of dollars for a real course outside a university?

Do people slack off with grades?

A certain amount of slacking off will always happen. There’s no way around that. But which way reduces it the most? Which way of opposing slacking causes the fewest side-effects? Having a time limit, or counting every homework mistake against the final grade? The better part of learning is making mistakes, and most grading penalizes mistakes. That is, it penalizes the normal learning process. Think about that.

If you have to use a carrot or a stick, then the student’s motive is not learning. Carrots and sticks confuse students about the purpose of school. They don’t learn algebra or biology or history, they learn how to game the system: to get away with doing as little as possible to get someone to make some marks on a page that say, “You passed.” Surely school has a higher purpose than that.

What it was like

I could not concentrate. There was too much noise and interruption in my head.

While working on anything, my mind would “interrupt” and jump to something else every few seconds. One night, I sat in my car and just listened to what was in my head. I heard three things. There was the din of a noisy restaurant. There was a cicada-like buzz, slowly rising and falling in loudness. And there was that relentless interruption: something I need to do, some thought popping up demanding immediate attention.

A typical day

Most days of grad school went like this. Wake up. Hurry to get through wake-up stuff, like getting dressed. It’s hard to do, due to the fact that my attention jumps every few seconds. For example, I intend to start the dishwasher, but something grabs my attention and I don’t make it to the kitchen. This happens over and over.

I prepare for the algebra class I’m teaching: read the book, prepare an example or a way to explain something. I seed my mind by looking at the section, and ideas slowly come. Some days, I would make a quiz or an extra-credit assignment with non-plodding problems designed to help the students “get” whatever they were having trouble with. Hurry to get down to campus. Usually arrive just barely on time, give or take a minute.

Algebra class: Noisy students who don’t want to be there, talking so much I can’t hear myself think. I would have thrown some of them out (as some students even requested), but I was so distracted and flustered from all the noise, I couldn’t pay attention to who was talking, and I didn’t feel confident that I could throw them out fairly and firmly.

Afterward, maybe I get some lunch or go to the library to do my homework. Can’t really think about homework during the day. The block of time is not long enough to get momentum (about an hour or two). I try to read, but I basically can’t. My mind goes loopy after a couple sentences of reading. Trying to focus on anything that requires rigor, or holding more than a single idea in mind at once, is painful.

Afternoon: I want to prep by reading in advance of my real analysis class, but I crave quiet and rest. Fifteen minutes after real analysis comes programming languages. A great class, probably one of the best offered anywhere in the world, but the sudden shift while I’m still processing the real analysis lecture feels like grinding gears in a car. The class is informal and discussion-oriented. Listening to people speak is painful. I can barely follow anything.

After classes, I get away to a secluded place in a library, but by this time, the noise in my head has reached an extreme intensity. I work on homework, but my mind is jumping around too much. It takes me half an hour to read a problem and understand it. Most of the real analysis problems really aren’t hard; I have to read each sentence over and over only because I keep forgetting what I just read. Nearby library books distract me. Occasionally I give in to temptation and pull one off the shelf. Wow, the Mariner Venus probes. Hey, that equation could be a good example show my algebra class how this stuff gets used in real life. OK, back to homework. distract me over and over. In the back of my head, ideas for solving problems in my other two classes are churning, not getting anywhere, just repeating themselves, trying to get attention.

Evening: Eat, not so much because I’m hungry, but to induce a food coma. Lie down in the dark and wait for the noise to die down. Sometimes sleep comes quickly, sometimes it takes hours of agonized waiting.

Total focused work time on a typical day: about five minutes. On most days, I devoted that to the algebra class.

Results

In the programming-languages course, I finished three assignments out of about 25. I did one during Spring Break, and one after the semester (the prof was kind enough to give me an Incomplete). The solution to the one I finished during the semester popped into my head one night, after banging on it for a couple of six-hour sessions the previous two days (and dropping all other classwork to do that). The always-on mental churning produced an epiphany on time. I finished one assignment in real analysis, out of about 13. I did a disappointingly minimal project in my other class, on machine learning. That was the project I had the most ideas for, but they all require sitting down and working for a week or two.

I learned a little, but not much. Got some basic intro to machine learning, learned some basics about the lambda calculus, and got the basic conceptual insight into differential forms. That could all have been done in a week or two, by devoting a couple full days to each topic. Got some meaningful Scheme programming experience.

An extraordinarily unproductive eight months. Also, an almost completely joyless eight months.

Why is grad school such agony?

There are always a thousand causes for anything, but one stands out:

• Time-fragmentation

Trying to switch context several times a day was simply a disaster. People say to work on each class in separate blocks of time, but it just didn’t work that way, at least for me. They are all going, all the time. I can’t just stop one and start another. Hence the scattered jumble of voices and thoughts that never stopped.

The only time I worked on anything for a couple solid days was during the break. Working on something for a week was unthinkable during the semester.

Really, it was attention-fragmentation. The time-fragmentation just induced that. Fragmented attention is about as good as no attention. This is why I retained so little.

The problem is made much worse by the fact that each of the four classes (taking three, teaching one) requires intense concentration and creativity. Concentration when it’s quiet is a pleasure. Trying to concentrate when it’s noisy, even just inside my head, is agony.

Material presented out of context was much less of a problem in grad school than in undergrad. A little more about why some of the topics are considered important would have been a great help for retaining it, but I don’t think this would have saved me from agony. The agony came mostly from having too many things bouncing around in my head at once. Trying to absorb stuff (mostly math) without knowing a reason why people care about it just leveraged the time-fragmentation some.

The summer

It’s now been about nine weeks since Spring Semester finished. The restaurant din and the cicada are gone. For about three weeks, I’d hear words and phrases zip by, about once every fifteen seconds or so. That died out, too.

The rapid-fire internal interruptions are still going, though. Doing anything that involves more than one step is agony. For example, I need to reboot my computer. I click the Apple icon in the upper left. There’s a delay of about half a second. I forget what I was doing. I pause, let my mind go blank, and wait for it to come back (like when you forget your thought in a conversation). Five or ten seconds later, it comes back. Oh, yeah, I was going to reboot. I look through the menu. Recent Items? Oh, I just had a thought. Maybe that USB extension cable is bad. I could test that by—wait, what was I doing, again? Oh, rebooting! There it is, Restart. Wait, why did I want to Restart, again?

Things that hold my attention without a break, I can follow, though. I don’t have a TV, but I’m finding that by watching a TV show on a DVD, after five minutes or so, the cascade of interruptions dies way down and I can just enjoy the show. There is pleasure in that. Afterward, I have some relative calm and focus.

Day by day, the interruption cascade and inability to focus have improved. But it’s been slow progress.

• Programmers who master Scheme, plus some appropriate powerful, refined libraries (yet to be written), become much more productive (at least 2x) than programmers who use “those other” languages, even for regular, workaday jobs like what most programmers get paid for.

I don’t know if the proposition is true, but there it is.  If it is true, that would be wonderful.  It would provide great opportunities for haps (high-abstraction programmers, i.e. programmers who design in layers of simple semantics).  We could develop software faster and better than the faps (flat abstractionless programmers, i.e. programmers who code everything at the level of the base language), and the faps would never be able to catch up, since Scheme doesn’t fit their brains.

Reasons to think that the proposition might be true:

1. Scheme lets you build pretty much any computation you like, represented any way you like (as long as you can live without syntactic sugar).  So, it ought to be possible to make really, really powerful libraries for all sorts of common programming jobs–more powerful than you can make in other languages.  Web stuff, database stuff, etc., would all be handled in a way that focuses on the essence of the problem.  For example, continuations are a natural concept in any data-entry system with multiple screens or steps; and Scheme is the only language that lets you represent and manipulate continuations explicitly.  Thanks to macros, Scheme is a sort of chameleon language: it becomes whatever language you want it to be (minus syntactic sugar).  What’s more, any simple sub-language that you create in Scheme always has an “out” to express odd cases: you always have access to the entire Scheme language.
2. By writing Scheme in a mostly functional style, you should have very few bugs.  The whack-a-mole bug cycle will end.  Just thinking in a functional style prevents a great many bugs from being written in the first place.  For the kinds of bugs that slip through the cracks in human cognition, static analyzers, such as type inferencers, root them out as soon as they occur.  When the properties of programs are provable by automated theorem provers, the vast majority of bugs don’t have a chance.
3. Scheme code is usually not very long.  It’s dense, but short.  By becoming accustomed to the idioms of Scheme, you can think with more leverage.  Since LOC productivity is supposedly about the same in all languages, your feature productivity ought to be extraordinary.
4. The recursive style of coding makes a fair chunk of programming work very systematic: code the simplest possible case, then code the next-more-complex case to leverage the simplest case, and so on.  You don’t think ahead: you just grind through the simplest uncoded piece of the problem, and each time you finish a piece, the next piece (often) becomes easy.  The language itself leads you to break problems down in a way that is easy (once you get used to it) as well as mathematically rigorous.  Code written this way tends to work the first time it’s run.

Reasons to think that the proposition might be false:

1. As Fred Brooks observed in his “No Silver Bullet” article, programs are complicated, not because of the programming language, but because of the irregularity and complexity of real-world problem domains.  When you write a typical business app with fill-in screens, a database, and maybe some middleware, there are just a zillion quirky ways that things can go wrong: wrong in terms of the business logic (think of tax rules), wrong in terms of the database (think of corruption and the difficulties of changing the structure of a database that’s already populated), and wrong in terms of the distributed computing architecture (think of all the ways that networks get out of sync and mess up).  Writing cleaner code might shorten ordinary programs a little bit, but there’s a limit to that, and existing programs (in PHP, Java, etc.) are pretty close to the limit already.
2. Syntactic sugar, particularly the kind in OO, enables intent to be represented in a way that is easy for a programmer to understand.  In Scheme, intent is very hard to indicate, because the same syntax is used for everything.  Explicit tags clutter the code.  When you see three lambdas in a row, you know you’re in for some mental slogging.  In practice, the awesome power of macros, closures, and higher-order functions like FOLD-RIGHT makes Scheme programming slow and thought-intensive.
3. What percentage of bugs are precluded by functional programming?  That is, what percentage are due to internal confusion about state?  10%?  20%?  And does static analysis of functional programs really catch bugs in practice?
4. The great power and flexibility of Scheme makes Scheme programs highly idiosyncratic. It’s hard for one programmer to understand another programmer’s code.  It’s hard for one programmer to hook into another programmer’s code.  Perhaps the reason that Scheme lacks the kind of libraries postulated above is not that they haven’t been written yet, but because extreme flexibility is unfriendly to code re-use.  For example, you can implement OO many different ways in Scheme.  That means that two OO libraries written in Scheme likely can’t work together.  To put this another way, extreme flexibility in a programming language makes software development unmanageable.
5. By insisting on writing programs not as step-by-step instructions, but by specifying a mathematical function to be computed, you place a huge burden on the compiler to solve performance problems.  In functional programming, performance problems are not obvious, and they are severe enough to render the compiled code unusable in the real world.  O(1) algorithms in a procedural language easily turn into O(n^k) algorithms in a functional language.  In practice, when you write Scheme code, only half of your mental bandwidth is on describing the computation simply.  The other half is taken up with devising clever tricks to avoid performance traps.  For example, avoiding APPEND by generating your lists in reverse order, or rewriting your code in continuation-passing style to avoid stack growth.  Linear searches happen frequently in Scheme and must be optimized out for production code, using much cleverness.  Efficient Scheme code tends to be awkward Scheme code.  In Python, list-appending and key-value lookup typically happen in O(1) time using the natural idioms of the language (”the one obvious way to do it”).
6. Scheme is great for building abstractions, but abstractions tend to leak.  The performance difficulties just mentioned are one example.  In practice, dealing with weird gotchas is as much a part of Scheme programming as any other language.  The difference is: programs written using layers of abstraction above the base language tend to be much harder to debug than “flat” programs.  In fact, lack of expressivity is the only way yet found in practice to make programming (somewhat) tractable to the human mind.

Two ways to break down complicated computations

In OO, everything is an “object” that has a list of actions that it can do for you. You instantiate objects, tell them to do stuff, and maybe later they get destroyed. Each object carries a state around with it. To break down complicated computations, you put various elements into different objects, and have the objects call each other in order to orchestrate the computation.

In Scheme, every procedure is a “closure” that is embedded in some context—that is, is bound to the values of the variables in its parent scope when it was instantiated. A closure provides a sort of window into that context at that time. Long after the parent scope has stopped executing, the closure still exists, and is still bound to those variables.

The natural way to break down complicated computations in Scheme

At any given moment in a computation, there are things that you already know (call this OUTER) and things that you don’t yet know (call this INNER). You define a function that bites off the next element from INNER (call this element CAR) and leaves the rest “for later”. Sometimes, to handle CAR, you need some information that you don’t have yet. In that case, you make a closure for what you want to do, and pass it to the code that works on INNER.

I hadn’t previously thought of closures as fitting into the now/later or CAR/CDR pattern that seems to be the essence of Scheme. If I was thinking in terms of now/later, maybe I would have hit on the idea of “reducing” (lately I’ve been learning to call it “folding”) right away. “What can we do now, and what can we defer until later?”

Reading Scheme by translating it into “now and later”

So, you can read Scheme code like this: Every lambda says, “I know now what needs to be done later.” Every procedure-application says, “I know that now is the time to do it.” I’ll try putting this to a test right now. Searching for inner lambdas in my code, the first one I found is in this:

;; Passes INPUT-DATUM to the first TRANSFORMER, the output of that into
;; the second TRANSFORMER, and so on.  Returns a stream.
(define fold-transformers
  (lambda (input-datum transformers)
    (delay
      (cond
        [(null? transformers) '()]
        [(null? (cdr transformers))
          (force (transf->stream input-datum (car transformers)))]
        [else
          (force (stream-mapcan
            (lambda (output-datum)
              (fold-transformers output-datum (cdr transformers)))
            (transf->stream input-datum (car transformers))))]))))

Well, how about that. The lambda says, “Later, when we know the result of running the first transformer (output-datum), feed it to the rest of the transformers.”

The delay in cons-stream is really a lambda, and it just means “Do this later, but only if we later discover that we need the result.”

A lambda with no arguments says, “Do this later.” A lambda with arguments says, “Do this when we know the values of these arguments.” The free variables inside a lambda are a record of “now” that we will use “later”.

call/cc fits this “now and later” framework, too. It provides a different sort of time travel than a closure. (call/cc (k) expr) means “Do expr now, and give expr the ability to jump to now’s INNER.” expr sort of has the ability to go back in time and rearrange things to create a different future.

delay and force obviously fit the now/later framework.

A hard example

Here’s a triple-lambda that frightened me yesterday:

;; Match next n characters exactly.  For example, if param is (I U I),
;; that means the next three characters must be I U I.
(define exact
  (lambda (param)
    (lambda (ss)
      (lambda (yield)
        (pmatch ss
          [(,miu ,subst)
            (cond
              [(starts-with? miu param)
                => (lambda (miu-tail)
                     (yield `(,miu-tail ,subst)))])])))))

Wow. The first lambda means “when we know the substring to match”. The second lambda means “when we know the full string and substitution list to match against.” The third lambda means “when we know who wants to know if we found a match.”

The lambda inside the cond is a hack. It means “when we know what starts-with? returned”, but of course we know that right now. But, time travel is still at play here: the => means “don’t forget the detailed result of the test,” and the name of the lambda’s argument says, “the detailed result is the tail of the MIU string (what’s left after matching param)”.

This method of reading lambdas suggests that maybe by careful naming, you actually can indicate intent in Scheme in a manner analogous to Extreme Programming.

And the now/later way of decomposing computations suggests that functional programming might even be tractable for human programmers.

1. Getting stuck

You ran into a problem: something in the math that you can’t figure out, some information you don’t have. Whatever, you’re spinning in circles or drawing a blank. You’re dead in the water, getting nothing done.

Preventive measure

• When planning (a tiny task, a day, an iteration, the whole project), look for well-mapped paths to get specific elements of the project done. That is, to the extent possible, get things done by reusing known methods rather than by discovering.

Realistically, though, you can’t avoid getting stuck unless you take on only the most unchallenging projects. It’s going to happen. In fact, it’s going to happen just about every day. Multiple times. The main difference between people who complete challenging projects and those who don’t, is not that the former never get stuck, it’s that when they get stuck, they do whatever it takes to get unstuck.

Solutions

1. In most practical cases of getting stuck, you can get unstuck just by asking for help. Ask a friend, ask an expert, have someone come over and pair with you.
2. When asking for help won’t work, you have to use some sort of creative problem-solving. Too broad a topic for this article, but also (a) something creative people like yourself are already good at, and (b) this is not the kind of getting stuck that often leads to projects not getting completed.
3. Work on something else and come back to it.

“When you’re stuck, change something.” —Jacquie Lowell, my first improv teacher.