chris-granger.com

"Is Web3... anything?"

2021-12-09T00:00:00+00:00

The term “web3” seems pretty fluid and is being used as a label for a bunch of disparate ideas. Given that it’s a decentralized group describing a decentralized movement, that lack of focus isn’t exactly surprising. Almost by definition it’s going to end up being different things to different people.

Most of the descriptions I’ve seen focus on mechanisms - block chains, smart contracts, tokens, etc - but I would argue those are implementation details and some are much more likely to succeed than others. (E.g. I think using private keys for authentication/authorization is obviously better if you can get over the UX hump - SSH has shown us that for decades.) The real question is what fundamental capability are people pointing to when they talk about web3?

The best explanation I’ve heard so far is that it’s a protocol/architecture for near-trustless commitments¹, which honestly doesn’t sound like much. But if you think about it, commitments underlie virtually all of modern society, which I think is a big part of why even the well-intentioned rhetoric around the space gets so crazy.

Some examples:

How does someone know you are who you say you are? The government issues you a number of some kind, it makes a commitment that you are unique and this number identifies you.
How do you prove you own a house? The government issues you a deed declaring that some parcel of land and whatever is on it is yours. The government reifies this commitment as a copy of the deed stored in public records.
How do you prove you have the money to buy something? You either physically possess fiat currency, or you have a bank account, with a number that is unique to you and contains your balance. The commitment of that balance is upheld only by the bank itself.

In each of these scenarios the commitments are upheld by authority. I am the government and you trust me, so you’ll believe that this number is this person or this house belongs to them. I am the bank and you will believe that you have $100, because… I say so. In theory, a mechanism like the Ethereum blockchain allows you to accomplish the same kinds of commitments, but without needing to trust the other party to uphold their side of the deal.

Removing the requirement for trust allows for things that weren’t previously reasonable. For example, let’s say you want to fix up your local neighborhood’s park. You have the time, but doing so will take a few thousand dollars in supplies. You could go to your neighbors and try to raise the money from them, but that requires them to trust you with the donations. Unfortunately with trust in low supply these days, I suspect you’d have a difficult time convincing people to hand over a couple hundred dollars each.

Trust-less commitments give us a different way forward though. You could specify the amount needed and contractually state what is allowed to be done with it as well as contingencies for if those obligations aren’t met. Once submitted, that “contract” will be carried out exactly as specified². It is observable to everyone, debuggable, and non-falsifiable¹. So as a community we could come together, raise $2k and require that the person who is going to purchase the physical supplies also puts in 2k as collateral. In the event that the community votes that the funds were misused, the person loses their collateral and everyone is made whole again. Now everyone involved needs very little trust in each other.³ If enough money isn’t raised, everyone gets their money back. If you tried to make off with the 2k, it’s a wash because you just lost the 2k you put up yourself.

Now maybe your local community is tight knit enough that you wouldn’t need that kind of guarantee, but this isn’t about removing trust, it’s about increasing what you can do at a given level of it. You might trust your neighbor with a couple hundred dollars - how about $10,000? Even if you believe in and support the cause of the group, at some point, you’re going to want a guarantee of some kind that in the event of misuse you have some kind of recourse.⁴

This framing of using verified commitments to increase what’s safe at a given level of trust becomes particularly interesting when you apply it to the internet. You’d likely have very little trust of arbitrary folks on the web, but this capability lets you take what small amount of trust you do have and do more with it. It lets groups of people with much less tenuous connections feel safe to go out on a limb and “invest” in people working on projects they happen to find interesting or come together for large scale collective action like improving an entire city’s parks.

In principle a system that can make commitments without the need for authority could underly most forms of ownership and commerce. It can also serve as a foundation for systems of economic incentives that would otherwise require you to trust that individuals never change their mind or become bad - which has historically been a very bad bet. At the end of the day, commitments are the fundamental primitive here. Identity, ownership, economies, etc are all just uses of them.

We’ll have to see if it pans out, but at least in theory, the concepts behind web3 provide an alternate foundation for the inner workings of society, one where guarantees don’t have to be backed by mandated authority (aka governments), but instead cryptographic math that can be automated, observed, and verified. Because of how fundamental that capability is, it’s hard to tell where it would really take hold - it may be that none of these examples are ultimately where something like this finds its place. The current bureaucratic machinery has an incredible amount of inertia after all. But I think it’s relatively unlikely it doesn’t have some place in the world. There are too many second order effects to assume none of them will be significantly better on a meaningful axis.

With all of that being said, the current mechanisms - proof-of-work block chains, smart contracts, tokens, etc - could easily turn out to be bad local maxima or outright dead ends. From a technological standpoint, my investigations of them left a lot to be desired and I was surprised to find out how poorly put together some of it is. But I think it’s a mistake to see the current mechanisms and lose sight of the actual capability they’re attempting to provide.

As a relative outsider, trust-less commitments seem to be at the heart of what web3 really is and that feels much harder to dismiss than owning JPEGs or speculating on currencies.

Nearly trustless/non-falsifiable because with the current systems you have to believe that the network has not had a 51% attack and that the system and contracts are without bugs. Given the size of these networks and their history, by far the greatest risk are the contracts themselves, but just like there’s standard contract legalese, I suspect standard, verified contracts would become the norm and you’d just be putting them together. I believe in practice, it is far more trustless than any other extent system. ↩ ↩²
For better or for worse it’s going to do exactly what was said. There’s a strong argument for needing to find some way to handle extenuating circumstances, as well as creating a set of extremely well vetted contracts for common scenarios, which is the norm in contract law already. ↩
There is still a negative trust scenario here, namely what if the neighbors collude and vote that you failed to improve the park despite having done what was specified. The contract could further specify that some neutral third party acts as an arbiter in that case. This is another reason why I believe it’s not actually about removing trust, but instead increasing what can be done with the trust you do have. ↩
This is ostensibly what the legal system provides, but anyone who’s actually tried to go through small claims court knows that the reality of that route is… painful. And it often ends unsatisfactorily. ↩

Two years of Eve

2016-07-21T00:00:00+00:00

Never heard of Eve? Read more here.

In the pursuit of our goal to make programming more accessible, we’ve built over 30 versions of Eve since we started about two years ago. These experiments explored some really interesting directions and by testing them with people, we learned a lot about problem solving and the motivations for computation in general. Many of our experiments boil down to prototypes of systems that could go on to become their own products - everything from a graphical database explorer to a document editor with embedded and always up-to-date natural language queries. Each of these has helped us understand more of the space we’re trying to navigate and hone in on what really matters for end-user programming, but there’s still a lot more to explore.

One thing all these prototypes showed us is that the core semantics we’ve come up with enable a number of systems that would be useful on their own. In other words, there’s a generally useful platform here regardless of how we ultimately present it. We also learned that generalized tools tend to end up with developers as their first customers even if the target was more end-usery. Both of which suggest that we could potentially treat building Eve as two problems.

A separation of concerns

In technical terms, Eve is a general purpose relational programming language, a temporal database, and an end user environment for working with those. Because we strongly coupled the first two with the last, they’ve had to progress at the same rate, but they now represent fundamentally different levels of research. Early on we stumbled across really nice semantics for thinking about systems, and while we’ve evolved those to make them more expressive and more accessible, they’ve remained relatively the same since. We’ve had to solve some hard problems along the way - aggregation, scoping, time-travel … - but a lot of the work on the “platform” is now engineering. The UI on the other hand requires more fundamental research and a lot of testing.

We learned with Light Table that we can’t just slap a UI onto Javascript and expect it to work; the platform has to allow for the representation. Our semantics have supported a wide variety of representations with very little change, so despite the tight coupling with the UI, we think it’s finally okay to tease the database and language apart from the end user environment. Even if we haven’t figured out the latter, the semantics provide a nice interface between the two. Without doing so, we’re losing a big opportunity to learn by hiding one just because there’s still more work to do on the other.

It’s time to let them progress separately.

Eve the platform

Through all our prototypes, the core has been a language integrated with some form of state management. Early on, we found the properties we were looking for in modern research on relational languages and the core of Eve became a temporal logic language coupled with a relational datastore. When I say “platform” here, that’s what I’m talking about: the amalgam of a relational database and a general purpose programming language. With it, you can build anything from websites to compilers. This combination has some compelling properties to it, from automatic scaling to considerably smaller programs, but arguably its most important property is the reduction in complexity that it provides. Eve’s semantics are simple and easy to reason about. They were explicitly designed for humans. And they try to capture the realities of modern computing.

Regardless of what may happen with the interface, the platform will be useful in a traditional programming context whether as a full application server, a database, or something in between. But if we separate out the UI, we still need to give people some way to use it. Since we’re talking about a traditional programming context, it makes sense to just meet on common ground: a textual syntax and a simple JSON protocol. The audience here is programmers and while most everything else is going to be different than what people are used to, we can at least provide a familiar interface. The past few months we’ve been focused solely on getting this ready and are releasing some alpha stuff on the mailing list with a broad release coming this fall.

An important step toward end users

It might seem odd, but the platform itself represents an important step toward our goal of reaching end users. While a cancer researcher couldn’t care less about a temporal relational language, they do care about the walls that software builds around them; and when the abstractions leak, they’re confronted with the horror underneath. The platform is necessary because we can’t just paper over the complexity anymore.

Writing software is a complex affair. Awhile back I started asking folks in medium-sized startups what infrastructure and tools they use to run the tech side of their business. It was a surprisingly long list. If it takes this much to do a relatively basic set of things, how can we expect someone whose sole job isn’t staring at Vim all day to get their head around it? Over the years, many folks have tried hiding that complexity under menus or interlocking blocks of code, but the reality remains the same - there’s a massive amount of context hidden in what appears to be “just a couple lines of code.” Even as professionals, we are failing to write correct programs and the layers upon layers of abstraction are crashing down on us. If we want to make programming more accessible, we have to find a way to make it manageable in the first place. We can’t just paper over the warts; we have to arrive at a version of computation that matches our reality and focuses on the human instead of outdated ideas of how a computer works. We’ve got a long way to go, but the platform’s explicit goal is to do exactly that.

Another reason the platform is necessary is really counterintuitive: we need developers to like it before end users will. Technology diffuses from technical people to non-technical people over time. If Eve doesn’t find some home with the early adopters of technology, it’s unlikely to find its way to anyone else. We found out that the first users of general tools like this tend to be developers, so it makes sense to engage directly with the people who ultimately end up being our ambassadors if the stars align. Every family has a “sysadmin” and the best way to get them on our side is to provide them something of value. For many of us who have spent years learning our tools, an entirely new interface for programming might be a non-starter, but something familiar that gets the job done faster? Sign me up. As Eve continues to evolve into something more generally accessible, we have to establish the trust and reputation necessary to make the next jump. The platform can help begin the process.

Eve the UI

The language only takes us part of the way though. We’re under no delusion that a textual syntax in a programming context is going to be the thing that ultimately gets us in the hands of an accountant or a cancer researcher. Eve will need a more approachable interface and a way of representing the semantics that helps anchor it to people’s actual problems. It’s not entirely clear what that looks like yet, but getting real-world usage with a smaller class of users is a good step forward. We can’t do another month-long experiment, we need to see people engage with the semantics and see how they apply it on their own. To that end, the focus over the next couple of months is on the tools for the platform. They’ll be bringing some novel ideas to the party, so (useful!) eye candy is forthcoming.

Onward

The past two years have been mind bending trying to navigate an insanely complex space. It’s exciting that we’re at a point where we can start to not only talk about some of what we learned, but begin to put it into practice. There are some big assumptions about computing we intend to challenge this fall, but if nothing else I hope we exposed people to some new (and old!) ideas about how all of this could be different. Here’s to the next two years.

Eve Version 0

2015-08-17T00:00:00+00:00

Never heard of Eve? Read more here.

Today marks a huge milestone for us - the first public release of Eve. After a ton of research, dozens of prototypes, and a slew of tests we’ve finally come to something that we think is the beginning of Eve. It’s still in its infancy with lots missing and assuredly more than a few bugs, but it’s reached a point where we can start to demonstrate the vision we have ahead of us. This is version 0.

To check it out, take a look at our intro tutorial or read through some of our design docs.

What’s in version 0?

Version 0 contains a database, compiler, query runtime, data editor, and query editor. Basically, it’s a database with an IDE. You can add data both manually or through importing a CSV and then you can create queries over that data using our visual query editor.

Notably missing from this release is:

A UI Editor
State
Version control / multiple people working together
Security
Speed

This is not meant to be the release you go build your next website on - it’s still got a lot of changing to do. It is, however, at a point where you can do some interesting things with it and it’s time for us to start getting more eyes on it. Many of these missing pieces actually already exist and just need a little more thought or cleaning before they’re ready. We intend for this list to get smaller pretty quickly and we finally feel like we have a good foundation to build on. As such, we’ll be working in the open from here on out, which means you’ll have the latest and greatest as soon as we do.

How we got here

It’s been a long road to get here and our understanding of both the problem and the task at hand has changed quite a bit since we started a little over a year ago. Our original goal was to build a “better programming,” one that enabled more people to build software. To that end we set out to find a simpler foundation, a language with few parts that could still produce everything from your vacation planner to machine learning algorithms. We ultimately found our answer in research out of the BOOM lab at Berkeley and took off trying to prove that with such a simple language you could still build real software. We’ve built compilers, editors, Turing machines, even a clone of Foursquare to prove that our strategy is workable:

It’s been a mind-bending project that has forced us to evaluate some of our most basic assumptions. Along the way to version 0, we tried everything from a Mathematica-like notebook built on functional programming to a purely spreadsheet-like model. We built dataflow languages and madlib based editors and read through papers from the foundations of computing. One thing that ran through all of this, however, was to make sure we never drank our own kool-aid too much. We dug through research and created postmortems and landscape summaries of all the projects that have come before us. We tested our ideas with actual people and against real projects. That meant that we “threw away” most of what we did to get here. It was the best way to keep ourselves honest.

Fortunately though, we have been able to lean on a wealth of established research to guide us as well. Our language semantics come largely from Dedalus. Our query editor is based on entity-relationship modeling and research on the problems with visual programming. Our experiments in making an efficient runtime are rooted in bleeding edge join algorithms. We even have Joe Hellerstein, the prolific director of the BOOM lab, as an advisor. Leveraging research and learning as much as we can from the past has served us well and given how ambitious what we’re doing is, we’ve needed as much help as we can get.

Where we’re headed

Obviously our first steps are to scratch off the things on the list above, but the latest incarnation of Eve also represents a shift in intent. Originally we imagined ourselves as a sort of modern day Visual Basic 6, but as we collected use cases for Eve we discovered very few people want to build applications or websites. They actually want better tools for thinking and communicating. They want to have their spreadsheets and documents always be up to date, or to set up alerts when certain things happen in the business. They want to create dashboards and models to simulate outcomes. Developers were certainly excited about the idea of creating applications, but even many of them expressed wanting to automate processes or bring a bunch of different types of information together - e.g. alert me when one of my friends is in town. Over the years, programming has become intrinsically tied to the notion of creating programs, but realistically what most people are tying to do is get the computer to do some thinking for them and then communicate the results.

In order to accomplish that, we do need a way to describe processes. We need a way to “program.” But switching the goal from building applications to analyzing and communicating information changes everything. Our current programming tools are awful thinking tools. Instead, they were designed to build complex systems. How much effort does it take to write a program to scan through your facebook friends and check to see if someone who usually isn’t in your area currently is? How hard is it to just write a program to read your emails and respond to certain ones automatically? As we dug more and more into the examples we collected from school teachers, programmers, nuclear physicists and lawyers, what it seems like we need is something more akin to the original vision of Lotus Notes - an environment full of information where communicating that information to people or even other systems is a fundamental primitive. Imagine what we could do just with a version of office where every bit of information was sourced live from a database, where instead of Power Point presentations of status you could throw together a dashboard and send it to everyone in the organization. People aren’t really trying to build the next Facebook, they’re trying to use the information from it in a different way. The tools we need for that look quite different. They look more like Office than Visual Studio.

There are more questions than answers down this path, but it isn’t the first time we’ve been confronted with data that caused us to take an odd road (e.g. a language that only has views). And it provides some insight into why previous attempts to get the masses to program never seemed to align with the masses themselves - most people have no desire to create applications, just solve problems.

When Robert and I created the company, we sat down and tried to summarize what the purpose of it was. We came up with the mission statement of “Empowering the next generation of creators.” Today we’re changing that to “Empowering the next generation of thinkers.” Eve’s version 0 is our first step in that direction - I hope you’ll come along with us.

Coding is not the new literacy

2015-01-26T00:00:00+00:00

Despite the good intentions behind the movement to get people to code, both the basic premise and approach are flawed. The movement sits on the idea that “coding is the new literacy,” but that takes a narrow view of what literacy really is.

If you ask google to define literacy it gives a mechanical definition:

the ability to read and write.

This is certainly accurate, but defining literacy as interpreting and making marks on a sheet of paper is grossly inadequate. Reading and writing are the physical actions we use to employ something far more important: external, distributable storage for the mind. Being literate isn’t simply a matter of being able to put words on the page, it’s solidifying our thoughts such that they can be written. Interpreting and applying someone else’s thoughts is the equivalent for reading. We call these composition and comprehension. And they are what literacy really is.

Coding is not the fundamental skill

When we say that coding is the new literacy, we’re arguing that wielding a pencil and paper is the old one. Coding, like writing, is a mechanical act. All we’ve done is upgrade the storage medium. Writing if statements and for loops is straightforward to teach people, but it doesn’t make them any more capable. Just like writing, we have to know how to solidify our thoughts and get them out of our head. In the case of programming though, if we manage to do that in a certain way, a computer can do more than just store them. It can compute with them.

Reading and writing gave us external and distributable storage. Coding gives us external and distributable computation. It allows us to offload the thinking we have to do in order to execute some process. To achieve this, it seems like all we need is to show people how to give the computer instructions, but that’s teaching people how to put words on the page. We need the equivalent of composition, the skill that allows us to think about how things are computed. This time, we’re not recording our thoughts, but instead the models of the world that allow us to have thoughts in the first place.

We build mental models of everything - from how to tie our shoes to the way macro-economic systems work. With these, we make decisions, predictions, and understand our experiences. If we want computers to be able to compute for us, then we have to accurately extract these models from our heads and record them. Writing Python isn’t the fundamental skill we need to teach people. Modeling systems is.

Modeling is the new literacy

In the same way that composition and comprehension are not tied to paper, modeling is not tied to computers. It can be both physical and mental. It takes place on paper and in Excel or with Legos and balsa wood airplanes. It is an incredibly powerful skill which we can make even greater use of by transposing our models to computers. To understand how we do that, we have to look more deeply at what it means to model.

Modeling is creating a representation of a system (or process) that can be explored or used.

This definition encompasses a few skills, but the most important one is specification. In order to represent a system, we have to understand what it is exactly, but our understanding is mired in assumptions. A fun illustration of this is the first time my co-founder, Robert, was trying to sort a list of names and getting frustrated with it. I asked him a simple question: “What does it actually mean to alphabetize something?” He had never once considered what that really meant. It was some vague concept that he had a mental model for, but he had never been forced to specify the exact result (for example, what do we do with non-English characters?).¹

Defining a system or process requires breaking it down into pieces and defining those, which can then be broken down further. It is a process that helps acknowledge and remove ambiguity and it is the most important aspect of teaching people to model. In breaking parts down we can take something overwhelmingly complex and frame it in terms that we understand and actions we know how to do. The parallel to programming here is very apparent: writing a program is breaking a system down until you arrive at ideas that a computer understands and actions it knows how to do. In either case, we have to specify our system and we do that through a process of iterative crafting.

Creation is exploration

We create models by crafting them out of material.² Sometimes our material is wood, metal, or plastic. Other times it’s data or information from our senses. Either way, we start our models with a medium that we mold. This helps us escape the trap of the blank page - by starting with a hunk of clay (or data, or some physical motion, or Legos…) we don’t have to imagine the leap from nothingness to something. Instead, we can get to work by manipulating what’s in front of us. We can hone, and whittle, and bend, and pull apart, and glue. We can shape it as we see fit.

The process of creating a model is an act of discovery - we find out what pieces we need as we shape our material. This means we needn’t fully specify a system to get started, we can simply craft new pieces as we go. We end up exploring the system as we create it and don’t have to get a “complete” model to gain value. We can simply tinker. We can shave a little off or glue a some on to see what happens. And along the way, we bolster our intuition of how systems behave.

Exploration is understanding

Physical modeling teaches us the value of being able to take things apart, whether that’s removing every single screw and laying the whole engine block out on the garage floor or just removing the alternator and having a look. By pulling something apart we can directly explore what makes it up. This is why interfaces in movies like Iron Man are so compelling - they allow us to just dive in.

Imagine what we could learn if we had the ability to break anything down, to reach inside it, and see what that little bit there does. The more ways we find to represent systems such that we retain that ability, the more power we will have to understand complex things.

We gain understanding through exploration, through fiddling with models and “playing them out” to see what happens. We can make our wheels bigger or plug in a different number for the interest rate and simulate the result. Through this process, we experience and imagine things we might not otherwise be able to, and the more experiences we have, the more we understand how the world works.

Digital freedom

While properties of physical modeling are useful to us as guiding principles, the digital world offers us an opportunity to step out of their limitations. We can freely create copies of parts or craft many different versions of them. We can make changes to individual pieces and the system will adapt to them. We can simulate our models in different contexts, while sophisticated tools verify our expectations. By transposing our models to a computer, we can offload the work necessary to change, simulate, and verify. We can even have our models suggest actions and let other systems perform them for us. As such, we free ourselves to explore without every change requiring us to hit the metaphorical machine shop.

There are a number of tools that already help us do this - from Matlab to Quartz Composer - but Excel is unquestionably the king. Through Excel we can model any system that we can represent as numbers on a grid, which it turns out, is a lot of them. We have modeled everything from entire businesses to markets to family vacations. Millions of people are able to use spreadsheets to model aspects of their lives and it could be argued that, outside of the Internet, it’s the single most important tool available to us on a computer. It gains this power by providing a simple and intuitive set of tools for shaping just one material: a grid of numbers. If we want to work with more than that, however, we have to code.

A fundamental disconnect

Coding requires us to break our systems down into actions that the computer understands, which represents a fundamental disconnect in intent. Most programs are not trying to specify how things are distributed across cores or how objects should be laid out in memory. We are not trying to model how a computer does something.³ Instead, we are modeling human interaction, the weather, or spacecraft. From that angle, it’s like trying to paint using a welder’s torch. We are employing a set of tools designed to model how computers work, but we’re representing systems that are nothing like them.⁴

Even in the case where we are talking specifically about how machines should behave, our tools aren’t really designed with the notion of modeling in mind. Our editors and debuggers, for example, make it difficult to pick out pieces at different depths of abstraction. Instead, we have to look at the system laid out in its entirety and try to make sense of where all the screws came from. Most mainstream languages also make exploratory creation difficult. Exploring a system as we’re building it gives us a greater intuition for both what we have and what we need. This is why languages that were designed with exploration in mind (LISP, Smalltalk, etc) seem magical and have cult followings. But even these suffer from forcing us to model every material with a single tool. Despite having different tools for various physical materials, in programming we try to build nearly everything with just one: the general purpose programming language.

On the surface, it seems desirable to have “one tool to rule them all,” but the reality is that we end up trying to hammer metal with a chef’s knife.⁵ Excel, by contrast, constrains us to the single material that it was intentionally designed to work with. Through that constraint we gain a tool with a very intuitive and powerful interface for working with grids. The problem of course is that Excel is terrible for doing anything else, but that doesn’t mean we should try to generalize a chef’s knife into a hammer. Instead, we should use the right tools for the job and look for a glue that allows us to bring different materials together.⁶

Programming as it exists now forces us to model, but it does so in an unnatural way. And while teaching ourselves how to program will help us learn how to break systems down, it does so at the risk of focusing on the wrong things.⁷ We don’t want a generation of people forced to care about Unicode and UI toolkits. We want a generation of writers, biologists, and accountants that can leverage computers.

How we teach children

We are natural-born modelers and we learn by creating representations that we validate against the world. Our models often start out too simplistic or even obviously wrong, but that’s perfectly acceptable (and arguably necessary⁸), as we can continue to update them as we go. Any parent could give you examples of how this plays out in their children, though they may not have fully internalized that this is what’s happening. A great example is that infants initially have a model of existence that relies on seeing objects. This is why not being able to see their parents is so distressing - it means that they no longer exist.⁹ The notion of object permanence is something that children only fully develop after a couple years of honing their model for how physical objects behave.

So if we do this naturally, what do we have to teach children? The magic of instinct is that we don’t have to be aware of how it actually happens. We understand the world through models, but that doesn’t mean we know how we create them in the first place. As such, we have to teach children how modeling happens,¹⁰ which we can break down into four distinct processes:

Specification: How to break down parts until you get to ideas and actions you understand.

Validation: How to test the model against the real world or against the expectations inside our heads.

Debugging: How to break down bugs in a model. A very important lesson is that an invalid model is not failure, it just shows that some part of the system behaves differently than what we modeled.

Exploration: How to then play with the model to better understand possible outcomes and to see how it could be used to predict or automate some system.

Focusing on these four areas captures the basic process by which we create, hone, and use models and it allows children to become active participants in the process by which they learn. Perhaps even more importantly, focusing on modeling pushes education towards the idea that pedagogy is really guiding children to deliberately create and explore the world around them.¹¹

How we should teach adults

Realistically, we should be teaching ourselves the same things, but unfortunately adult education rarely allows for undirected exploration (we’re just trying to get something done). Instead, we could frame modeling in terms of how we might use models in context. For example, if you’re an accountant that goes through the same process every day to create a report, you could break it down to see how you might automate portions of it. What are the fundamental actions necessary to create that report? Are there tools that can do those actions for you?

If we assume that at some point better tools for modeling come into existence¹², then being able to model some system or process may be all you need to automate it. Once that is the case, a targeted exploration of how you go about modeling specific domains has obvious value as it frees us up to do other tasks. An extreme example of this might be modeling your entire business, from the interactions with customers to fulfillment. At that point you could automate large portions of it and focus on how to grow the business, which itself likely involves exploratory models.

“The computer revolution hasn’t happened yet”

Alan Kay did a talk at OOPSLA in 1997 titled “The computer revolution hasn’t happened yet,” in which he argued that we haven’t realized the potential that computers can provide for us. Eighteen years later, I still agree with him - it hasn’t happened yet. And teaching people how to loop over a list won’t make it happen either. To realize the potential of computers, we have to focus on the fundamental skills that allow us to harness external computation. We have to create a new generation of tools that allow us to express our models without switching professions and a new generation of modelers who wield them.

To put it simply, the next great advance in human ability comes from being able to externalize the mental models we spend our entire lives creating.

That is the new literacy. And it’s the revolution we’ve all been waiting for.

This is used as an argument for why programming can never be made easier. At best, you could potentially say that specification is fundamentally difficult and as such modeling cannot be made easier. Programming currently requires much more than just modeling though, and there’s a great deal of room for improvement in the other areas. Moreover, we have primitive tools for modeling at this point, so there’s plenty of opportunity there as well. ↩
This notion of material comes up in a number of places. Bret Victor’s latest talk discusses creating a “dynamic material.” Papert also talks about how the materials we have at our disposal greatly influence how we think about the world in the beginning of Mindstorms. There’s some important idea about how what we have to work with dictates what we are able to think. ↩
At least not anymore. When we were working on the foundations of computing our focus really did have to be at this level, but if we’re talking about democratizing the ability to control a computer, we can’t expect people to become experts in hardware architecture. The focus has to shift from “how do we make these things work” to “how do we do x with them”. ↩
You learn a lot from the metaphors used in different domains. I talk about material, but in programming we have “modules.” Modules are plug and play, which means that they either have to provide for every possibility or you work around them. In many ways, the idea that we can reuse components without having to change them (or that we can account for all the variances) has caused us more harm than good. Models are made of material, which means they can still be adjusted and honed - they are just a starting point. It seems like that would be a much better way of looking at reuse in systems. We start with something that’s most of the way there and instead of trying to account for every possibility, we just allow for people to reshape it as they see fit. ↩
No one tool will cover every model we want to build, but it may be possible to come up with a small set of mechanisms from which all others could be built. We see this in the physical world with the simple machines. Nearly every tool we use to shape wood, for example, is made from a wedge. The problem is that in the software industry we never made the leap from wedge to saw. ↩
Exploring possible solutions for how we would model using different tools is outside of the scope of this essay, but it’s the basic premise of our work on Eve. The gist is that we think the glue is a language for exploring and translating data from different domains. We then couple that with a system capable of taking data and doing computery things with it (sending it over the network, showing UI, etc). The tools sit on top of this foundation and you model directly in your domain. It’s then up to the system to figure out how to execute the results of the models. ↩
Forcing kids to take classes that focus on general purpose programming runs a very serious risk; it might end up like math education. Programming is only meaningful when put into the context of problems we actually have, otherwise we’re showing people the moral equivalent of calculus - “what’s the point of this crap?” This is not to say we shouldn’t have general programming classes, it’s just that forcing it on kids won’t have the desired effect. ↩
There’s an argument that in order for us to really learn something, we have to go through a series of invalid models to internalize what would initially be unintuitive about the world. While much of nature presents itself simply, there’s almost always a set of complex systems under the covers that wouldn’t make sense based purely on intuition. As Carl Sagan said, “The simplest thought, like the concept of the number one, has an elaborate logical underpinning.” ↩
I saw a hilarious (and somewhat mean) example of this walking around San Francisco. There was a family with a 2ish year old son walking down the sidewalk holding hands. The father let go of his son’s hand and ducked behind a pole. As soon as he was no longer in sight, the little boy started crying. The father stepped from behind the pole and the child stopped. He repeated this several times and every single time the son couldn’t see his father, he wailed. What a different world that must be. ↩
It was pointed out to me that there’s actually an example of this in modern education: the first time you’re graded not just for getting the right answer in a math class, but for also showing your work. By forcing people to show their work, you’re forcing them to internalize how their model for some mathematical principle works. It’s not the most direct way to teach modeling, but it is at least landing in the same ballpark. ↩
For more about this you really should just go read Mindstorms. ↩
And this is exactly what we’re working on with Eve. ↩

Beyond Light Table

2014-10-01T00:00:00+00:00

I have three big announcements to make today. The first is the official announcement of our next project. We’ve been quietly talking about it over the past few months, but today we want to tell you a bit more about it and finally reveal its name:

Eve is our way of bringing the power of computation to everyone, not by making everyone a programmer but by finding a better way for us to interact with computers. On the surface, Eve is an environment a little like Excel that allows you to “program” simply by moving columns and rows around in tables. Under the covers it’s a powerful database, a temporal logic language, and a flexible IDE that allows you to build anything from a simple website to complex algorithms. Instead of poring over text files full of abstract symbols, you interact with domain editors that are parameterized by grids of data. To build a UI you don’t open a text editor, you just draw it on the screen and drag data to it. It’s much closer to the ideal we’ve always had of just describing what we want and letting the machine do the rest. Eve makes the computer a real tool again - one that doesn’t require decades of training to use.

Imagine a world where everyone has access to computation without having to become a professional programmer - where a scientist doesn’t have to rely on the one person in the lab who knows python, where a child could come up with an idea for a game and build it in a couple of weekends, where your computer can help you organize and plan your wedding/vacation/business. A world where programmers could focus on solving the hard problems without being weighed down by the plumbing. That is the world we want to live in. That is the world we want to help create with Eve.

We’ve found our way to that future by studying the past and revisiting some of the foundational ideas of computing. In those ideas we discovered a simpler way to think about computation and have used modern research to start making it into reality. That reality will be an open source platform upon which anyone can explore and contribute their own ideas.

While we have a long way to go before Eve is ready for the spotlight, we’ve shown a few people what we’ve done so far and their reactions have been exciting. Which brings us to my second announcement:

We’ve raised $2.3M from Andreessen Horowitz.

We’ve raised a $2.3M seed round to help us turn Eve into a reality. The round comes from an amazing set of investors: Chris Dixon of Andreessen Horowitz, Sam Altman, Tom Pinckney, Sep Kamvar, and Zubair Quraishi. Given how ambitious this project is, it was important that we find people who are not just interested in handing us money, but deeply care about what we’re actually trying to do. We are on a mission to realize the world I described above and we managed to find people who not only believe in that mission but who have been actively working on it themselves. Chris and Tom had plans to create a company like this and have been thinking about the problem ever since Microsoft killed VB6. Sep has been researching how to make programming more accessible at the MIT Media Lab. Zubair has followed Light Table since the beginning and has been an active member of the community. And Sam’s been looking for a “better programming” since his first company. Each of these people brings something unique to the table and we couldn’t be happier to call them all partners.

This gets us to my third announcement:

We’re hiring!

We raised money to expand the team, as it’s going to take more than the three of us to get Eve ready for the world. So we’re looking for a few developers and a designer to join us.

Working on Eve so far has been an incredible challenge that has taken us through everything from language design and query optimisation to high-dimensional geometry and cognitive science. We are truly working on the “full stack” and there’s no shortage of interesting problems for us to solve. On the engineering side, we’d especially love to find some folks who have experience working on databases, query optimizers, constraint solvers, systems/ops, and crazy UIs. For our designer, we’re looking for someone to own the personality and feel of Eve itself, which presents one heck of a challenge: how do you invite a billion people to program without them ever knowing about it? We’re not sure yet, but we know finding out will certainly be fun.

Realistically, it’s somewhat difficult for me to say exactly what we’re looking for, because no one person is really going to know the breadth of area that we cover. But one thing we do know for sure is that you have to be willing to go way outside of the box with us. If this sounds like your cup of tea, shoot an email to jobs@kodowa.com and introduce yourself - we’d love to meet you.

And Light Table?

Light Table will continue to go on strong. We haven’t talked too much about it lately, but it’s used by tens of thousands of people and still growing. We use it every day to help us build Eve and thanks to the awesome people in the community that has sprung up around it, it gets better every week. Our original goal with our kickstarter was to affect change and with your help we managed to do exactly that, both directly through people using LT and through influencing the broader industry as well. Eve represents the next step for us to take the ideas in Light Table and bring them to far more people. We are stoked for the future that Eve and LT will create together and I hope you’ll join us for the ride.

Toward a better programming

2014-03-27T00:00:00+00:00

this post is based on my talk, "Finding a way out", from Strange Loop 2013

When I built the original prototype of Light Table I didn’t have any grand purpose or goal in mind. I simply had some ideas on how programming could be better and I wanted to see how hard they would be to build. Until fairly recently, it never dawned on me that I’ve actually spent the past decade trying out ideas on how programming could be better, from web frameworks, to Visual Studio, to Light Table and its future. And it wasn’t until I had that realization that I also came to the conclusion that I’d been going about this all wrong. As a matter of fact, I made a classic rookie mistake: I set out to answer a question I didn’t understand.

How do we make programming better?

I kept asking myself “How can we make programming better?”, but I never took a step back and concretely figured out what’s wrong with programming. I’ve always been very reactionary to either my own work or the observations I had of others, and it turns out that has continually led me to local maxima. With Light Table for example, I thought shortening the feedback loop would make a tremendous difference, and while it does make a big impact, it’s overshadowed by the fact that it’s the same old broken version of programming. People still struggled. I still struggled. It didn’t deliver us from the incredible frustration that is writing and debugging software. So if it’s not just the feedback loop, what is it then? What’s wrong with programming?

To answer that I needed more data, not just from my time behind one way mirrors or my own experiences, but from the “real world”. So I started asking everyone who would talk to me two questions: “What is programming and what’s wrong with it?” I talked to people who had never programmed a day in their life, people in the middle of their careers, and even some of the best engineers in the industry. And what I found surprised me.

What is programming?

The answers I got to this were truly disheartening. Not once did I hear that programming is “solving problems.” Instead, I found out that it’s just clumping things together with half-dried glue and praying that it holds water long enough to replace it with a new clump of things. I heard over and over again that we’re plumbers and glue factories, that programming is largely about working around its own deficiencies. This is particularly sad to me, because it turned out to be the only real commonality among the answers: we plumb things together to get something that kind of works. And the “kind of works” part is the source of an unbelievable amount of frustration.

Programming should be about solving problems, but somewhere along the way it turned into us solving the problems around our problems (a friend and ridiculously good engineer likes to call this a “self licking ice cream cone”). The glue factory answer certainly isn’t a desirable one, so I’ve been trying to come up with something workable since. The one I like the best so far is that programming is our way of encoding thought such that the computer can help us with it. At the end of the day, what we’re trying to do is build something that enables us to accomplish something new - programming just happens to be the way we talk to the computer to get it done.

And what’s wrong with it?

The answers to this were why I ended up talking to so many people (around 400 in the end). They tended to fall into one of two categories: they were either so general that they didn’t really provide any information, or they were so tactical that they didn’t apply to anyone else (I get these deadlocks when I’m…). In aggregate, though, some patterns emerged and after hundreds of answers I was able to distill almost everyone’s issues into three buckets for what’s wrong with programming. Those buckets were that it is unobservable, indirect, and incidentally complex.

Programming is unobservable

This one seems to be getting the most attention lately thanks to Bret and our work on Light Table. We can’t see how our programs execute. We can’t see how our changes affect our programs. And we can’t see how our programs are connected together. That basically means we can’t observe anything. The state of the art in observability is a stepwise debugger, which forces us to stop the world and look at a single instant in time. But the truth is that few errors occur in an instant; most are found only by observing the passage of time. And for that, the best we have is print statements. This is ridiculous. We should be able to observe the entire execution of our program, forward, backward, even branched into new futures - not just when our breakpoint hits. Even sadder than that though, is that we seem to have embraced unobservability as an industry best practice. Think about a line of code like this:

person.walk();

What does it do? OOP’s notion of encapsulation is by definition unobservable. I have no idea what person.walk() does. It probably does something sane, like set isWalking to true, but it could also be setting ateCarrots to true and it may have determined that I passed out from exhaustion - I have no idea and no way to tell. We are quite literally throwing darts in the dark and praying that we at least hit the board. We simply cannot see what our programs do and that’s a huge problem whether you’re just starting out or have written millions of lines of beautiful code.

Programming is indirect

Writing a program is an error-prone exercise in translation. Even math, from which our programming languages are born, has to be translated into something like this:

#include <algorithm>
#include <iostream>
#include <iterator>
#include <cmath>
#include <vector>
#include <iterator>
#include <numeric>

template <typename Iterator>
double standard_dev( Iterator begin , Iterator end ) {
   double mean = std::accumulate( begin , end , 0 ) / std::distance( begin , end ) ;
   std::vector<double> squares ;
   for( Iterator vdi = begin ; vdi != end ; vdi++ )
      squares.push_back( std::pow( *vdi - mean , 2 ) ) ;
   return std::sqrt( std::accumulate( squares.begin( ) , squares.end( ) , 0 ) / squares.size( ) ) ;
}

int main( ) {
   double demoset[] = { 2 , 4 , 4 , 4 , 5 , 5 , 7 , 9 } ;
   int demosize = sizeof demoset / sizeof *demoset ;
   std::cout << "The standard deviation of\n" ;
   std::copy( demoset , demoset + demosize , std::ostream_iterator<double>( std::cout, " " ) ) ;
   std::cout << "\nis " << standard_dev( demoset , demoset + demosize ) << " !\n" ;
   return 0 ;
}

Why do we have to see that, instead of this?

And that’s with something that at least translates fairly cleanly. What happens when it doesn’t? All we get in code are symbols. We never see or interact with the real things, just symbolic representations of them. While symbols are certainly important and powerful in some cases, they don’t have to be this opaque:

cards[0][12]

Ah yes, when playing cards, I love it when I get the cards[0][12]. We’re writing a card game and cards have real representations, so why can’t we just see this instead?

Translation is hard and using symbols is error-prone, especially coupled with operations on top of other symbols. This indirectness, this inability to represent things usefully and directly manipulate them, is killing us. A vast number of programming errors are simple translation problems. We had the solution in our head, but in trying to turn it into code we just forgot something or translated it very slightly wrong. We have to get away from that translation. When we do UI, we should do it in a visual editor. When we do math, we should have something like Mathematica at our fingertips. We should express domains in the ways they most naturally present themselves in, not with our own homegrown obfuscations.

Programming is incidentally complex

There is an immense amount of incidental complexity in writing software, by which I mean there’s a bunch of work that needs to be done that isn’t directly related to the real problem you’re trying to solve. Just think about how long it takes to even get something simple to run or the fact that people can spend the better part of a week just trying to set up a dev machine from scratch. These are some simple examples of unnecessary complexity at a systems level, but the truth is incidental concerns are pervasive throughout the entire process of writing software. One of the worst for us right now is at the logic level - managing time in our code. Most people tend to jump immediately to concurrency and parallelism when I say that, but it’s actually more fundamental than that. Every time we add an event handler or create a callback, we’re doing time management. We’re saying at some point later we want this code to execute. Considering the increasingly more complex schemes of interaction that our programs have on top of the proliferation of multiple cores (and the very real concurrency problems that brings), we’re quickly learning that callbacks are a terrible solution to this problem. Every time we’ve found ourselves manually managing something, we’ve come up with a solution that does it for us. We were manually handling binary and so we created Fortran. We were managing memory and so we created garbage collectors. We were managing data constraints and so we got type systems. I think the next big step in terms of removing incidental complexity in code will come from automatically managing time. The implications of which would be tremendous for our ability to cleanly express intent.

There are so many examples of incidental complexity in programming it would be disheartening to try and enumerate all of them, but we have to start addressing them at some point. We should be focused on solving our problems - not solving the problems around solving our problems. At the end of the day, we should be able to go from nothing to a solution in the time it takes us to think of one, not the weeks and months it takes us now.

Chasing local maxima

If you look at much of the advances that have made it to the mainstream over the past 50 years, it turns out they largely increased our efficiency without really changing the act of programming. I think the reason why is something I hinted at in the very beginning of this post: it’s all been reactionary and as a result we tend to only apply tactical fixes. As a matter of fact, almost every step we’ve taken fits cleanly into one of these buckets. We’ve made things better but we keep reaching local maxima because we assume that these things can somehow be addressed independently. The best analogy I’ve heard for what this has resulted in is teacups stacked on top of teacups. Each time we fix something, we push ourselves up some, but eventually we’re working under so many layers of abstraction that the tower starts to lean and threatens to fall down. We have to stop thinking about these issues individually, and instead start imagining what it would take to address them all simultaneously.

The other day, I came to the conclusion that the act of writing software is actually antagonistic all on its own. Arcane languages, cryptic errors, mostly missing (or at best, scattered) documentation - it’s like someone is deliberately trying to screw with you, sitting in some Truman Show-like control room pointing and laughing behind the scenes. At some level, it’s masochistic, but we do it because it gives us an incredible opportunity to shape our world. With a definition for what programming is and a concrete understanding of what’s wrong with it, we have a framework for capturing that opportunity and removing the rest. But in order to do so, we can’t just bolt it on to what we have now. Another layer of abstraction won’t be enough. Instead, we have to address these fundamental problems at the very foundation of our solution. No more teacups and no more compromises.

The people’s programming

I mentioned when I talked with folks that I talked to non-programmers too. I did so because they would provide a very different view and the truth is that most of them are programmers by my definition. They just don’t happen to write “code”. If you use Excel, you’re programming - you’re getting the computer to do work for you based on a process you’ve encoded for it. Excel provides a particularly interesting example, given that it has been massively successful at enabling people to solve problems. It also happens to address all three of our fundamental issues and gives us some evidence that doing so can create an incredibly powerful and approachable environment for people to work in. Excel is inherently observable since it doesn’t have any hidden state and all values are there for you to see and manipulate. It’s also direct. You change values in the grid, drag drop things, do calculations on selections, and so on. And it manages to sidestep a lot of incidental complexity; spreadsheets are timeless, without setup, and don’t even have a notion of being run. Excel achieves this, however, by making a tradeoff in power. There are a lot of things it cannot express very well (or at all) because of the constraints placed on the programming model. The interesting question is whether we can solve our issues in a similar way, but ease some of the constraints to retain more power.

The more we’ve explored this stuff, the more we’ve realized that fixing these problems goes a long way toward making programming more generally palatable. So if the answer to that interesting question ends up being yes, you’d have an environment that gives just shy of a billion people the equivalent of modern day super powers. Imagine what it would be like if virtually everyone with a computer could command it to do even 80% of what a programmer can today. What would the impact of that be? I haven’t the slightest idea, but the more I’ve considered it the more I’ve realized it would be a fundamental shift in what we as a collective would be capable of and that’s certainly a fascinating thing to consider. In the long run, I do believe manipulating computers will be a fundamental skill, but unlike most of the “programming is literacy!” movements lately, I think it’ll have very little to do with writing out ‘if’ statements. The best path forward for empowering people is to get computation to the point where it is ready for the masses. Any attempt to do so with what we have now is destined to fail. It turns out masochism is a hard sell.

A note on culture

One thing my buckets didn’t address is the more societal issues around programming, which are getting a lot of attention lately. There are numerous facets to this problem from the image that programming has, to the way our communities interact, to the biases and prejudices that have developed. I think one of the most interesting aspects to a complete rethinking of programming is the potential for a reboot in the culture of technology. What would it mean if programming wasn’t inherently antagonistic? What would it be like if the only prerequisite for getting a computer to do stuff was figuring out a rough solution to your problem? What if programming was inclusive from the start? In many ways I sincerely hope that whatever way things go, it looks nothing like what we consider programming now. Because the impact of deliberately addressing some of the cultural issues that have developed could be truly world changing.

Great, now what?

We find a foundation that addresses these issues! No problem, right? In my talk at Strange Loop I showed a very early prototype of Aurora, the solution we’ve been working on to what I’ve brought up here. While our strategy has changed significantly since that demo, the notions behind it remain the same - we’re out to find something better, something that isn’t just for programmers or non-programmers, but instead removes that distinction entirely. We’ve made some very real progress in that direction lately, and we’ll be sharing more as it solidifies. But one thing I will say is that what’s coming will have to be seen to be believed.

Light Table is open source

2014-01-07T00:00:00+00:00

Today Light Table is taking a huge step forward - every bit of its code is now on Github and along side of that, we’re releasing Light Table 0.6.0, which includes all the infrastructure to write and use plugins. If you haven’t been following the 0.5.* releases, this latest update also brings a tremendous amount of stability, performance, and clean up to the party. All of this together means that Light Table is now the open source developer tool platform that we’ve been working towards. Go download it and if you’re new give our tutorial a shot!

There’s been a ton of work since the initial release 0.5.0 (about 200 items in the full changelog), so here’s just a few of the highlights:

Plugins

The biggest thing to be released in 0.6.0 is the plugin infrastructure. Given the BOT architecture of Light Table though, “plugin” is a bit of a misnomer - they are capable of fundamentally redefining in or adding anything to Light Table.

Realistically the only distinction between the core code and plugins is which things we ship by default. This gives us an enormous opportunity to redefine what development is. To see what some simple plugins look like, check out the declassifier and CSS plugins. We also added a plugin manager that hooks to a central list of plugins, no need to go hunting all over github.

Inline docs and doc search

This was one of the big things from the original Light Table prototype and video. You can now search docs and get documentation for what’s under your cursor, right inline.

Clojure(Script) nrepl, auto-complete, jump to definition, paredit…

Clojure saw a lot of love in this release from standard editory features like auto-complete and paredit, to reworking the back end to allow for remote nrepl sessions (connect to your server and watch things happen in real time!), all the way to interesting new things like custom watches and eval.

Performance, stability, and polish

Because we wanted to go open source there was a big push to clean things up and get Light Table ready for it’s big unveiling. To that end we spent a ton of time trying to make everything smoother, faster, leaner. In many cases we improved performance by orders of magnitude - auto-complete is now wickedly fast, behaviors load faster, the command and navigate panes now scroll buttery smooth.

We made lots of changes and little improvements that also help it feel like LT does what you’d expect. You can now drop files/folders into the workspace tree, for example, or open the current file in a browser with a command. Along with that we put some time into making the default skin for Light Table more professional, less obtuse, and whole lot more versatile.

On the road again

Getting both plugins and going open source into a single release was a big undertaking, but it gets us closer to the community supported platform we’ve been working towards over the past year and a half. From here on out, anyone can play the game and what that’ll result in is hard to tell, but it’ll certainly be interesting. By getting the platform out, we can now focus a bit more on rethinking the state of the art. And we have some very interesting (and crazy!) ideas for what we think we can do to programming as an industry. I hope you’ll join along with us in reimagining what it means to program.

Light Table is growing

2013-12-11T00:00:00+00:00

I’m happy to finally announce that we have a new addition to the Light Table team: Jamie Brandon will be joining us, coming all the way from sunny old England. Jamie brings some much needed British civility (what else could you possibly learn at Cambridge and Oxford?) to our otherwise lawless duo of merry men. He also happens to be incredibly passionate about getting people the tools they need to do amazing things, the future of technological colaboration, and changing the way we create our world. We’ve been searching for the right person to add to the team for quite some time now and we’ve finally found them. Check out Jamie’s blog post to see what he has to say on the matter.

But we’ve also been growing in other ways. We now have well over 10,000 people using Light Table every month, doing more than 3 years worth of continuous work in that time. And as of the 2013 State of Clojure Survey, we’ve become the 3rd most popular editor for Clojure, being beaten only by Emacs and VIM. But in the startup world, everything seems to be about growth over time, and we’ve done pretty well there too.

We’ve grown our usage by over 20% on average every month for the past year and a half. All of that has come from you guys helping us spread the word, and for that we thank you. It’s been a good year for Light Table and I’ll say that with what’s ahead, things are just getting started.

So with our first year and a half behind us and our team now 50% larger than before, we have some awesome things coming in the not too distant future - a big release in the first week of the new year and maybe even a sneak peek of “Aurora” before too long.

Happy Holidays everyone and I hope you’ll join us in shaking things up in 2014!

Light Table 0.5.0

2013-08-22T00:00:00+00:00

After much ado, I’m happy to announce that Light Table 0.5.0 is out today:

Go give it a try!

Faster, better, stronger

This release was about simplifying the experience of Light Table, making it far more robust, and making sure it’s fast enough to handle whatever you throw at it. We’ve made big strides in each of these areas, from removing some of the UI elements that didn’t add anything (the left vertical menu), to reaching the point where we can open large files as fast vim. In many ways this release is foundational compared to 0.4.0, which was about stretching the limits of eval, but this is an incredibly important step toward plugins - we’re finally at a solid base.

Simple

This go around we spent a lot of time making LT lighter, more predictable, and more helpful. One thing I saw as I started watching people use Light Table is that they felt that it was still “heavier” than something like Sublime, which made it a lot scarier to them. Ultimately, we attributed that feeling to doing some unfamiliar things, one of the biggest of which, was the lack of standard menus. We don’t really gain anything by not doing menus and while there were some technical reasons I couldn’t add them, I’ve modified node-webkit to fix that. This means that the vertical menu is now gone from the left side. You can’t get much “lighter” than LT is now:

There are numerous other little design tweaks that in aggregate make the workflow significantly better and far simpler. For example, transient panels like navigate and commands now slide from the right and are presented in a much more readable fashion. This makes opening them a lot less jarring and doesn’t cause a heavy context switch.

Powerful

This wasn’t just a clean-up release though. We added some very powerful new features, the most interesting of which are “watches” and the behaviors system.

Watches give you an instarepl-like experience that is driven by your code executing normally in your program. You can think of it as a next-gen version of println where whenever a watched expression would normally be executed in your code, the value is streamed back to LT and shown in line. Watches work for all 3 of the core languages (Clojure, JS, and Python)! Here’s a little video of them in action:

The other big improvement is a uniform strategy for settings that has a ton of flexibility and power. Light Table has always had behaviors as first class notions and I came to the conclusion that settings are really just a way to add or remove behaviors. Settings are simply a way to twiddle which behaviors are added to tags. This is what behaviors files let you do in 0.5.0 and it enables all sorts of fun things; you can give all editors instant eval, you can set colors based on syntax, you can set basically anything depending on context… Really, it’s a form of simplistic programming where you can fundamentally change the behavior of LT without writing any logic. This strategy also has a number of non-obvious benefits as well, like making plugin conflicts something that an end-user can deal with - just remove the behaviors that are screwing things up. Behaviors can also be set globally and per workspace, which gives you a lot of opportunity for crafting the experience based on exactly what you’re doing. Similarly, keys are now defined in keymap files that allow you to bind any command based on context.

Fast

With the work we did to add background processing and a good deal of effort toward ensuring everything ran fast, LightTable is now comparable in speed to Vim and faster than Emacs or Sublime in most things. The addition of a threading model means that lots of things that were blocking the UI no longer do so thanks to some magical hackery (given the lack of threads in JS). Everything from opening files, to project-wide search, to watching the file system, should feel a whole lot smoother.

Healing 1000 papercuts

We touched virtually every aspect, every line of code in Light Table with this release, from dramatically simplifying the “deploy” and adding a command line interface to unifying settings and revamping the design. The end result is a release that has a 1000 things better in it, all adding up to something that just feels far better than what we started with. This is the first release that I’m really proud of for Light Table - it’s finally a real contender.

To plugins we go!

The next big step for us is the infrastructure necessary to support end-user plugins for Light Table, which we’ll be rolling into our private beta. I know a lot of people have been asking about getting into the beta after missing the kickstarter. We’ll be talking about our specific plans in that regard in the next couple weeks. Either way, we know plugins are where the magic of LT will really start to shine and there’s no telling where it’ll ultimately take us.

Cancer and Startups

2013-05-20T00:00:00+00:00

About a year and a half ago, my mother’s partner was diagnosed with terminal cancer and given less than six months to live. Since then it has been treatments, pain, and truly miraculous victories - but the word “terminal” is pretty clear; at some point you will lose. I flew home as soon as things started to take a turn for the worse and since then I have had to do things I could never have imagined. I’ve watched a 45 year old person shrink to nothingness. I’ve picked my family up off the floor and tried to get them back on their feet (figuratively and at times literally). I’ve talked with lawyers, and social workers, and nurses, and… But the truth is that there’s nothing I can really do. This kind, compassionate, and strong person who means the world to my family is going to die. And yet there is one thing among all the tragedies that never ceases to amaze me. Every time the doctor or the nurses say there’s only a couple days left, she finds a way to prove them wrong.

I’ve never met a stronger person. She has lasted through doses of poison that would’ve easily killed any one of us “healthy” people, and she has done so with a degree of poise that is truly unfathomable. In our little startup world, the words tenacity and perseverance are thrown around a lot, but in that context they seem hollow and largely meaningless. Tenacity is far more than simply making it through tough times, and it’s not just a matter of finding a way “back to good.” Kristie has shown me that tenacity comes from living for a purpose, from believing in something so fully that it keeps you alive through six rounds of injecting drain cleaner into your veins. By that definition, I haven’t seen much tenacity in the Silicon bubble many of us call home.

People ask me about startups all the time. They ask for tips, what I think of their idea, how to get people interested, how to raise money, how to get into YC, how to ___. In truth, I think there’s a relatively simple test to determine whether or not someone could be a founder; when you’re done talking to them, have they infected you? Is your head spinning with their idea? Regardless of how ridiculous what they just said seems, do you believe even just a little that they could actually do it? And are they out to make a company/name/product or are they on a mission? This last one is the most important, because tenacity is what will keep your startup alive and it’s hard to imagine walking through hell for something you don’t truly believe in. And you will walk through hell - it’s something we don’t talk enough about amidst all the glorifying of startups that we do.

These past few months have certainly tested my resolve. I spend days helping, cooking, cleaning, doing errands, trying desperately to keep things together, to then spend late into the night trying to build a company, community, and product. It’s hard to wipe away your mother’s tears and then once she’s asleep, sit down and build an IDE, but that is what the circumstances require. There are definitely times I wonder if this is really worth it. I could certainly be doing something else. My cousin even pointed it out to me, by asking a simple and rather pertinent question the other day: “Why are you doing this company?” It’s an insightful thing to ask and something I actually think about myself all the time. There are days, days like this one, where I wonder if I should have kept going and become a doctor, or a researcher, or at least done something other than pushing 1’s and 0’s around. I have talents and skills that I could directly apply to saving lives - shouldn’t I be doing that? Isn’t it my moral duty to do that? Could I have saved Kristie, or if not her, others like her? It’s a scary line of questioning, the harbinger of “The Fear.”

And this is the stark truth of startups: you are the last and only line of defense against doubt. There’s no one else to give these questions to. They are yours to stew in. They are yours to try to answer, though they’re unanswerable. They are yours to overcome. And the Fear that they represent is what keeps you up at night, what will make you wonder if this is really the “right” thing to do. I’m not sure anyone can truly explain what that kind of doubt is like and if they did it would likely only hint at the reality. But believe me that when people say startups are hard, they are woefully understating the truth. Yet I’m still doing one, and the reason why is rooted in the answer to the question my cousin asked. I’m doing this because I believe that this is the greatest contribution I can make.

I could’ve become a doctor. All signs pointed to me likely being a very good one. In doing so, I would have gone to work and done my best to save lives every day. In that context, how is some programming environment a greater contribution to the world? Truthfully, it wouldn’t be if I just set out to build an IDE. But that’s not what I did - Light Table is just a vehicle for the real goal. While an IDE probably won’t directly save someone’s life, the things people are able to build with it could do exactly that. My goal is to empower others, to give people the tools they need to shape our lives. Instead of becoming a doctor, I have an opportunity to improve an industry that is unquestionably a part of the future of all fields. Software is eating the world and analytical work is at the core of advances in medicine, hard science, hardware… Human innovation throughout history has been driven by new tools that enable us to see and interact with our mediums in a different way. I’m not building an IDE, I’m trying to provide a better foundation for improving the world.

This is my purpose, this is what keeps the Fear at bay, and this is what Kristie’s lesson drives me toward. If you too decide to wade into the murk of your own startup, do something that matters and do it with purpose. Show some real tenacity - it’s the only way to survive.

Light Table 0.4

2013-04-28T00:00:00+00:00

Today, I’m proud to announce the 0.4 release of Light Table - go download it!

A bit of the future.

With this release our goal was to bring more people into the fold, to increase the number of languages we can eval, and to smooth out some of the core experience. After 0.3, we finally had a chance to move the ball forward some - from live modifying websites directly inside of LT to injecting code into NodeJS and showing matplotlib graphs inline. This is a huge release in virtually every way and I think there’s something for everyone, whether you’re just starting out programming or you’ve been writing python since it was released. Let’s take a look.

Next-gen web dev

One of the things I originally showed in the Light Table videos was the ability to embed your work directly into Light Table itself. Well…

You can now stick a browser in a tab, navigate to any page you want and start live modifying it. The neat thing is that you don’t have to own the page or do any setup. It just works. Cmd/Ctrl+R will refresh the page anytime you need to do so. Here’s the obligatory meta screenshot of me writing this post in Light Table:

The most requested language from our survey was Javascript by a fair margin and a lot of people were hoping for deeper JS integration. As soon as we started to see that, we shifted gears and made it happen. Thanks to some serious intergration with the Chrome-DevTools, we can do some pretty impressive stuff - like inspect Javascript objects inline and through the console:

But the most powerful thing to me is the level of eval we now have. Thanks to the devtools integration, we aren’t just making calls to eval(), we’re patching the running VM. This means that all the things that should be “impossible” to change at runtime, are a single Cmd+Enter from being updated. It magically fixes event handlers, anonymous functions, and a host of other things. No, it’s not perfect, but you sure won’t have to refresh much anymore.

Oh and speaking of Javascript.. you can eval into a NodeJS process now too.

Speaking snake: Python support

SSssssss. Light Table now supports Python eval. It works for both Python 2 and Python 3 and utilizes IPython for a better overall experience if it’s available.

Python support comes complete with inline results/exceptions and integration with the console.

With the IPython integration we were also able to do inline matplotlib/pylab graphs, which is pretty neat:

Documentation

There’s finally a nice set of documentation! Up until this point, we’ve been hesitant to spend much time writing documentation on how to use Light Table, because things were still changing pretty drastically. We’re happy with the general workflow we’ve put together now, though, and so it was time to get some content out there. You can access the docs within Light Table itself from the welcome tab or the Docs: Open Light Table's documentation command.

And then literally everything else

As I said before, this is a huge release - just look at the changelog! Basically every aspect of LT got some love, but here are some of the big ones:

You can now have multiple windows
You can have multiple tabsets! (splits)
The workspace tree-view now tracks the file system and lets you do the standard file operations
You can explicitly create a connection through the connect tab
A greatly improved console
Intra-file token-based Auto-complete!
Dramatic performance improvements (rendering is 4-10x faster)

We fixed a lot of the little things that were missing or slightly off that took away from the overall experience of Light Table. I think you’ll find this release just feels better in every way.

Do you teach programming?

Our goal has always been to provide an environment that helps you get started, but also grows up with you to be a powerful working environment. As of this release, Light Table has a lot to offer as a platform for new programmers - from a dramatically shortened feedback loop to a clean, simple workflow. If you work with a teaching organization of some kind, we’d really love to hear how we can help get your students using Light Table. Shoot us an email at learn@kodowa.com.

The plan

With 0.4 we now see a taste of all three of the original languages we set out to include in Light Table. The plan is for there to be one more public release before we then move into the private beta for our Kickstarter folks. The 0.5 release will be focused on picking up a couple of things that didn’t quite make it into 0.4 and deeper language support, while the beta will be all about opening Light Table up to plugins - the most exciting aspect of this whole thing. We’re proud of this launch, but trust me, there’s even more amazing stuff on the way.

Light Table a year ago

2013-04-12T00:00:00+00:00

A year ago today I put up the original blog post for Light Table. It was just a simple video of a prototype I made in less than a week while I was visiting my grandparents without internet - at the time we had no idea what it would become. To be honest, I thought some would think it was neat, but then it would fade away like most “neat” things on the internet.

It’s been a wild ride this past year, from the kickstarter to joining YCombinator to experimental releases and everything in between. One thing, however, has remained constant through all the ups and downs; you guys. The enthusiasm, the hope for something new and better, the willingness to bear with us as we try things that don’t work or that aren’t quite ready yet. Ultimately, Light Table is a project for and of the community and for that I can’t thank you enough. There have certainly been times where I don’t think we would’ve made it without your support.

A couple days ago, we put out a survey to gather some more information about what people think about the latest release and what they’re most excited to see next. Once again, you guys showed us how amazing of a community you are and we got over 1100 responses in under 24 hours. Over the past couple days Robert and I have been going over what you guys said and we’re happy to say that we were moving in the right general direction already and we’re using that feedback to hone our plans. As a matter of fact, two of the biggest things people wanted to see are coming in the next release - Python eval and deeper JS integration - as well as a host of other things that were mentioned. This next release is slated before the end of april and it’s going to be a big one with some surprises that I think you’ll love.

Thank you again for being a part of this project and for believing in two crazy guys who set out to change 40 years of tradition… We can’t wait to show you what’s next.

A new Light Table experience

2013-02-27T00:00:00+00:00

With our release of the 0.2.0 version of Light Table we laid the foundation we needed to continue moving forward. In this release we wanted to focus on making that foundation a joy to use by taking the time to really craft an experience. To that end, even though 0.3.0 is still an alpha, it will feel much better to anyone who used the previous version and many of the things that seemed manual in the previous incarnations will “just work” now.

Let’s take a peek at what’s new or go download it!

A gorgeous new look

The last version of Light Table was somewhere between cool looking and a retro throwback, which didn’t get anywhere close to the cohesive experience we wanted. The new look is all about drawing you into something that “feels right” and just looks beautiful. We spend a lot of our time in these environments, they should be easy to parse and pleasing to look at. I’m really happy with where we landed on this.

You’ll notice some simple quick animations and a number of visual affordances that make the app feel a lot more polished than it was before. And if your preference is for something else, you have complete freedom when writing your own skin.

An inline results experience

One of the biggest usability problems in 0.2.0 was how results were shown up in the top right. They obscured code, they were hard to interact with when the results were large and the they just generally didn’t scale as you did more and more with LT. One of the more interesting things we’ve done with this release is move result and error reporting inline.

We also did this for the instarepl so that you don’t lose all that screen real estate to another copy of the code on the right. We’ve shortened the default view of the captured values which you can then hover over to see the full text.

JS and CSS eval

You can now eval JavaScript in a browser (or directly in LT)! Just open a JS file and press Cmd/Ctrl-Enter while your cursor is on a block or an expression and you’ll see the result inline. Connecting to a browser is just a matter of including a script tag (use the “Connect to a browser” command to get it) in your page. Here’s a little video of the process:

You can also “eval” css into the browser, allowing you write and test your CSS without having to refresh. The same things works with ClojureScript too, though you need to have some things already on the page for it to work. Some tutorial content for that is forthcoming.

Workspace and tree view

Having the notion of a collection of files that you’re working over is nice, especially since projects rarely confine themselves entirely to a single folder. As such, we went with a “workspace” approach in which you add files and folders into your working set that Light Table can then do more with. Right now this is just holding an index of the file names for quick navigation, but in the future this will include ripping the files into semantic pieces. The workspace tab is basically your typical tree view of this working set.

Since we know what files you’re interested in, we can do fuzzy navigation as popularized by Textmate and carried on in SublimeText. It’s the fastest experience we’ve seen for a strict-file recognition approach and so we went with it too.

In the long run, there will be more efficient means of getting around to the exact things you want, but for now this has proved to be a great placeholder as we start to work on more of the language-focused stuff. We use the fuzzy search everywhere there is a filterable/searchable list and we intend to keep that up; it’s far more efficient than simple substring matching.

Command search

A big thing I wanted to tackle was to come up with a command strategy that works nicely with the keyboard, but also allows for some complex and interesting interactions that remain workable with a mouse. The side command search is what we came up with.

You can quite literally do everything from the command search, from changing the skin of Light Table to toggling Vim mode and starting an instarepl. We want a single canonical way for people to be able to answer “How do I…” - you go to the command search and type a bit of what you’re looking for.

Vim mode and custom key binding

As I’ve stated a few times, I’ve been a Vim user for over a decade and I’m happy to say that Light Table now has a halfway decent Vim mode. It’s still rough around the edges, but it takes a nice step forward. The best part is that it’s just an integration of the excellent work that has been done for CodeMirror’s Vim keybindings, so if you want to make them better, take a look at the work there. Along with the vim keybindings is a little keybinding interface that you can use to bind any key combination in any context to a command. Right now this doesn’t include many textual commands, but they’ll be coming in the near future.

It “just works”

We cleaned up a ton of the little things that started driving us nuts, everything from being able to reorder tabs to making the instarepl more reliable. We wanted to get rid of as many of the papercuts as we could hunt down and our hope is that you’ll find that Light Table just does the right thing when it can. A great example of this is that you’ll never be asked to select a path to connect to a project - it just figures it out.

Toward 0.4.0

With 0.3.0 we now have a base editing experience that we’re happy with and we’re going to start heading into the semantic parts of the LT experience, while also working on really fleshing out the plugin system - the latter of which will be the beginning of our early beta work. As such, we’ll be getting our Kickstarter backers more involved soon to help us figure out exactly what our plugin story will look like. There’s lots of very exciting stuff to come!

The IDE as a value

2013-01-24T00:00:00+00:00

Here’s a video of this talk from the Clojure Conj 2012

In my last post I talked about building a game in ClojureScript using a Component-Entity-System engine and I hinted that the ideas that underly that architecture inspired how we designed Light Table. The reason why stems from the architectural goals of the project:

Runtime modifiability
High contextuality
Infinitely customizable

Games have a relatively similar set of requirements and we saw in the previous post that a Component-Entity-System engine worked nicely to satisfy these kinds of goals. But we’re not building a game. We don’t have a constantly running loop and we aren’t checking state every 16ms. What we have is an event driven application and so we can’t just take the CES approach directly. We need something slightly different.

Meet BOT

What I came up with is what we’re calling a Behavior-Object-Tag engine. It has many of the same properties of a CES engine, but is a better fit for an evented architecture where you don’t have as many variations on the state of similar things (variation, instead, comes from how items react to events). Like with CES, I think the best way to introduce the idea is to walk through each piece to see what its role is.

Objects

Unlike in CES, it’s not particularly likely that, for example, different kinds of editors in Light Table are going to have wildly different state. It’s also not likely that disparate kinds of objects are going to share bits of state. As such, we just use ClojureScript maps to represent objects instead of the groupings of many maps that was used in CES. Here’s an example of what an object definition looks like:

(object* :notifier
         :triggers [:notifo.click :notifo.timeout]
         :behaviors [:remove-on-timeout :on-click-rem!]
         :init (fn [this]
                 [:ul#notifos
                   (map-bound (partial notifo this) notifos)]))

First we give the object type a name, :notifier in this case. Everything thereafter becomes the key-value pairs of the map. All objects inside of LT have a set of triggers (basically events that can be fired), a set of behaviors (things that will react to those events), and an init function that is a bit like a constructor, except what it returns is interpreted as its UI. If we wanted to add a :notifier to Light Table we’d use (object/create :notifier), which simply sticks a copy of that map into a big data structure containing all our objects.

It’s important to note that just like in CES these objects don’t do anything. They’re just state and something outside of them breathes a little life into the party.

Behaviors

So what is the thing we want to compose if it’s not the state? It’s the reactions that objects have when events flow across the system. What tends to define variation in evented applications isn’t the state of objects themselves, but how they react to certain messages. Given that, we want to be able to compose these reactions to create new versions of the same object. A good example is editors. For the most part an editor is an editor whether it happens to contain plain text, JavaScript, or Clojure. But in each of those cases, eval does something different.

Behaviors are a way of creating bite-sized, reusable reactions to messages. Here’s an example of one that could be added to an editor to make it read-only:

(behavior* :read-only
           :triggers #{:init}
           :reaction (fn [this]
                       (set-options this {:readOnly "nocursor"})))

Just like objects, behaviors are maps stored in our big data structure. Here we give it the name :read-only and tell it what triggers it listens for and what reaction it has to them. When an event is raised on an object that has this behavior, the reaction function will be called with the object as the first parameter. In this case, whenever the object (presumably an editor) is done calling its init function, we set it to be read-only.

The way events are traditionally done in most modern platforms, you end up with hidden collections of listeners that contain nameless functions. This hides a vital bit of information - how do you know what’s going to happen at runtime and how would you change that? We could look into that collection but we’d just see [Function]. By comparison, behaviors carry more information and are bound at call time. We can freely modify their reactions at runtime just by replacing the map in the LT data structure. And if we ever want to know what an object is going to do, we just take a look at that object’s set of bound behaviors.

There are a few ways these end up “attached” to an object, which simply means that the behavior’s name ends up in the object’s :behaviors collection. One is by specifying them in the object definition. Here’s the definition of the default editor in Light Table, for example:

(object* :editor
         :tags #{:editor :editor.inline-result :editor.keys.normal}
         :triggers [ ... ]
         :behaviors [:no-wrap
                     :active-on-focus
                     :on-tags-added
                     :on-tags-removed
                     :inactive-on-blur
                     :context-on-active
                     :context-on-inactive
                     :refresh-on-show
                     :highlight-current-line
                     :destroy-on-close]
         :cur-line 0
         :init (fn [] ...))))

Only being able to specify them at object definition time would be pretty limiting though, so you can also add and remove behaviors to any object with a simple function. This allows you to contextualize anything inside of Light Table, based on whatever conditions present themselves at runtime. There is, however, one final way to add behavior to an object: tagging.

Just data.

In the same way that CES turns a game into one big data structure, BOT has enabled us to turn an entire IDE into a data structure that can be simply introspected and manipulated. This allows us to create very interesting views on how objects inside of the IDE behave and we can simply push a couple of tags or behaviors around to fundamentally alter the way the entirety of Light Table works. It is an incredibly flexible system that lives up to the goals we set out. Modifying at runtime is just a matter of modifying the data structure, it’s highly contextual through the use of tags, and it is infinitely customizable - you could replace the guts of LT with something else entirely just by removing behaviors or objects. The lesson in all of this is that building a system’s foundation on data not only makes it easier to reason about, but also enables you to use that system in ways you never could have thought of in the beginning. In our case, it turns Light Table into a nearly limitless development platform that can adapt to the needs of the user.

Dynamic mixins?

A few people have pointed out that BOT seems like an application of dynamic mixins and I think that rings true based on my cursory understanding of the work there. This implementation takes a heavy focus on introspection and runtime recomposition by using data as the underlying system as opposed to language features. This gives you the ability to use the entire language’s facilities toward manipulating the inner workings and that seems far more powerful than the alternatives. In general, I think this approach works very well in any highly variable evented system, which if you consider how few projects finish as what they start out as, is most.

Wait, you’ve only mentioned ClojureScript.

Interestingly enough, as of Light Table 0.2.0, the entire thing is now built on top of ClojureScript and the only Clojure left in the system is used for evaling Clojure. With the help of the Node-Webkit project out of Intel, we now use Node.js for all of our platform interaction and we rely on chromium to present our UI. There are many reasons we went down this path and ultimately it has worked out really well for us. ClojureScript, though it has had its moments, has generally been a joy to use and has allowed us to keep our codebase incredibly tight and small despite all of the stuff we’ve managed to build into LT.

Anatomy of a knockout

2012-12-11T00:00:00+00:00

For the Node Knockout last month, we wanted to build a game in Light Table using ClojureScript. What we ultimately came up with was a fun little racing platformer called ChromaShift. All the code and assets were created during the 48 hours of the competition and in the end we did pretty well, coming in 7th overall. Here’s a video of the game in action:

How do you build a game in ClojureScript?

I’ve built a couple games before in JavaScript that relied on the more standard inheritance based design. But that OOP-centric way of thinking about the entities and logic of the game doesn’t fit very well with ClojureScript. Given how young it is, Clojure(Script) hasn’t been used to create a lot of games and even if you broaden the search to functional languages, you’ll find that it’s still mostly an area of research. After some digging though, I did find a game design theory that actually fits very nicely: the Component-Entity-System engine. Much to my surprise, though, I couldn’t really find a good explanation of the concepts on the internet, which among other things might include an actual example. So, let’s fix that and go through the concepts behind how ChromaShift works.

The traditional way of thinking

The traditional way of thinking about games is fairly intuitive. You represent objects in the game as objects in code. So if you have a player, you’d create a player class that contains all the player’s attributes; things like position, health, ammo, etc. Then you create an update() method that calls other methods like shoot() or jump() that read and change those attributes. To keep things DRY, you’ll probably end up creating some base classes because entities in a game often have many different variations with small differences.

The problem

The problem with this, however, is that in order to actually reuse code as much as possible, you end up being forced into deep unnatural object hierarchies with lots of overridden methods. With thousands of entities in a game, you lose all sense of where things are defined, how they’re changed deeper in the hierarchy, and where the best place to add something really is. This approach also means new combinations of functionality have to be written by programmers, forcing game designers to ask for different variations.

A different approach: Component-Entity-System

A CES engine addresses a number of the problems game devs ran into while trying to build complex games that required a lot of variation in the game objects. As we’ll see, it also ended up being a great way to give content designers a lot more power in shaping the functionality of the game.

A CES engine has 3 parts as the name suggests; components, entities, and systems. I think the best way to go about understanding how it really works is to walk through each of these pieces individually and then see how they fit together.

Entities

We’re actually going to start in the middle, because entities are the easiest thing to understand. They are nothing more than a unique ID. That’s it. You use this ID to link together all of the state related to something in the game. That state is held in components.

Components

Components are little single purpose bits of state. Instead of representing a player as a monolithic object with lots of attributes to cover everything from position to number of lives, you break these different aspects apart into single intentioned objects. For example, a component for position would have an x, y, and maybe an angle of rotation, but nothing else:

(component position [x y a]
           :x x
           :y y
           :a (or a 0))

All that does is basically create a function that returns a map with those values in it. So what we’re doing is exploding all these monolithic objects apart, and turning entities in the game into simple groupings of components. A player for example, might look like this:

:player [(position 10 10)
         (walk 20) ;; set speed to 20 units
         (jump 50) ;; jump height to 50
         (renderable render-player) ;; render the player
         ...]

Here’s a real level from ChromaShift, showing you a more complex example. The tremendous advantage you get from this approach is that you gain composition for free - no overriding, no deep hierarchies, just simple groups of components. As an example, let’s say we have two bad guys in our game; one that walks fast and another that is slower but jumps every 3 seconds. That’s just a matter of giving them both a walker component with different speeds and then giving the jumping one a jumper component:

[:fastguy [(position 10 10)
           (walker 20)] ;;really fast

 :jumpguy [(position 10 10)
           (walker 5)
           (jumper 3)]] ;;jump every 3 seconds

Since objects are just groupings of components, at the core of the CES engine sits a datastructure that maps these entity IDs to the group of components that make them up. Fortunately, that’s very easy to represent as a hash-map or dictionary in most languages. In ClojureScript you just use a map of integer ID to a vector of components:

{1 [{:name "position" :x 10 :y 10 :a 0}
    {:name "health" :lives 3 :health 5}
    ...]

What’s particularly interesting here is that we’ve turned our entire game into a datastructure. By being data-centric, we gain introspection, composition, runtime modification, and a host of other things for free. As I mentioned before, it’s also really easy to build an editor that allows non-programmers to simply create new groupings of components and as a result create fundamentally new objects for the game.

So far, however, all we’ve talked about is state. There’s been no logic in entities or components, which brings us to the final piece of the puzzle: systems.

Systems

Systems are single purpose functions that take in a set of entities which have a specific component (or set of components) and update them. Let’s look at the concrete example of rendering to see what I mean. First off, we need a component to say that this entity is meant to be rendered. We’ll call it renderable:

(component renderable [func]
           :fn func)

The component just takes in a function that will be called when we render the object. So what our system needs to do is to iterate through every renderable entity, get its renderable component, and call that render function. Sounds simple enough, and the code ends up being as straightforward as that explanation:

;; define a function 'renderer' that takes all renderables
(defn renderer [renderables]
  ;; for each renderable entity
  (doseq [e renderables]
        ;; get the entity's renderable component
        (let [rend (as e :renderable)]
          ;; call the stored render function with the entity
          ((rend :fn) e))))

So now that we have a renderer function, we just need to call it in every tick of our game loop with all the entities that have the renderable component:

;;... other systems
(renderer (all-e :renderable)) ;; get all entities that are renderable

And that’s it. Now any entity that gets the renderable component will start rendering with every tick of the game. By doing your logic this way, you ensure that it’s single purpose, easy to modify, and freely available to any object in the game that may need it.

Data and simple functions

This approach turns a game into a data-structure with simple functions that work over it. It allows you to easily combine functionality in interesting and clever ways that you may not initially have realized, and keeps you out of the murky hell that is deep, highly overridden hierarchies. As a bonus, the engine is even quite easy to implement in basically any language. In ChromaShift’s case the core structure and caches for the engine are actually written in JavaScript for performance reasons, while all of the game CES is written in ClojureScript.

A twist: CES and Light Table

The most interesting aspect to all of this, however, is that I haven’t just been telling you how we built ChromaShift for the Node Knockout, but I’ve also been laying the conceptual groundwork to understand how Light Table itself is constructed. In my next post we’ll see that the latest version of Light Table is actually heavily inspired by the concepts of CES engines and takes a novel approach toward applying some of these ideas to an event driven application.

LightTable and the Node Knockout

2012-11-09T00:00:00+00:00

Hey folks! As promised, we’ll be live streaming from my laptop while we do the Node Knockout using Light Table this weekend. We’ll be building a multiplayer platformer game in ClojureScript.

Stream will start right at 4:00pm PST here on Ustream.

Meet the new Light Table

2012-11-05T00:00:00+00:00

We have to start with a picture. Or hey, just go download it and see for yourself.

This is the new Light Table - rewritten from the ground up after having learned a ton about what it’s going to take to make a truly extensible and connected environment. There are a lot of interesting ideas under the hood that make up the core of this new implementation, but the important part is that it is a solid foundation for us to continue forward on. The new code is not only smaller, but far more robust, and allows us to iterate incredibly quickly.

But the changes aren’t just all under the hood. What we have here is far more useable and useful than what we had before. As a matter of fact, I realized the other day that the only time I actually open vim now is when I manage to hose my instance of Light Table. The environment is entirely bootstrapped with all of our work happening by modifying Light Table at runtime. Even I’m amazed at the difference that this has made and it’s something I think you just have to experience to really appreciate - writing whole features without refreshing the screen once is a truly gratifying experience, one we intend to bring to everyone.

##Yeah, yeah, what’s new?

Well, let’s see it in action.

A real app

First and foremost, Light Table is now a real app, not some weird concotion of a clojure server and a browser. It has an executable that you double click to run. Simple as that.

General editing

The 0.2.0 release has more of the general editing features you’d expect in a programming environment. You can open any kind of file, it doesn’t have to be valid/compilable, simple things like incremental find and multiple tabs are there.

This also includes a command bar, which is the source of most of the non-text-in-buffer actions you can do in Light Table. It’s how you spawn an instarepl (the instant evaluation environment I showed in the first video), open files, and connect to projects.

Eval from any file

Unlike the stricter table mode that existed in 0.1.0, you now have the freedom to simply eval from any file you’re working on. While this only supports Clojure and ClojureScript in this initial iteration, we’ll be seeing this used in a lot more places soon.

Eval results show up on the right-hand side of the environment in a little list that you can hover over to make larger. All the results are also saved for you there, so you can simply scrollback through them if you want. Evaling doesn’t block the editor in any way, so keep on keeping on while your process toils away in the background.

Connect to multiple projects

Working with multiple projects with different kinds of files is a non-issue in Light Table. If you try to eval something where no client can currently handle it, it will prompt you to use the connect command to spin up a client for that project.

It’ll do that in the background and show you the result once everything is ready to go. So if you’re working on some crazy computer generated music, with intense graphics, while having your computer write programs for you, you won’t have to skip a beat.

##A new, beautiful brand

We really love what the guys at MODE did for the Light Table logo. They’re some of the best in the business. And it shows.

This also translated into tshirts that look awesome:

##The plan

Two months ago we walked into YC demo day having just launched 0.1.0 and since then we’ve been toiling away at the standard tribulations that startups face - from fundraising to fundamental technology shifts. Now that we’ve waded through the bulk of that, it’s about getting back to what we do best: exploring the future of programming environments. This release was a huge step forward in our ability to deliver to more people and to iterate faster. It is the first move toward supporting more languages and that is where our focus is headed for the next couple of months. We’ll be starting with Javascript first and tackling the problem of both client side JS as well as Node.js itself. To do that, we’ll need to do some research in the way you guys use your tools already, which we’ll be talking about more soon - hopefully you’re up for a visit!

##Issues, comments, concerns?

We’re continuing to use our github to track issues and there are a few we know of already with the release of 0.2.0. There’s also an announcements mailing list which we’ll be making much better use of and the discussion list that we’ll breathe some life into.

##Want to see more?

We’ll be doing the node knockout this weekend and to give people a taste of what Light Table can really do, we’ll be live streaming the entire event. We’re going to be building a game in ClojureScript and it’s going to be awesome. Once the feed is live, I’ll put up another post and tweet about it. I hope you’ll tune in!

Now get out of here and go play with it.

All ideas are old ideas

2012-10-05T00:00:00+00:00

My last post generated some discussion on telling people more about the ideas that came before Light Table. In many ways we’ve been rediscovering the past and since that initial blog post months ago, we’ve learned a ton about the ideas that we’re trying to bring to the industry. In the talks I’ve done recently, I’ve mentioned a bit about all the innovations that happened over the past 40 or so years that simply never made it to the mainstream. These are ideas championed by the lisp machine, structured editors, and the amazing efforts in the tooling surrounding languages like Smalltalk. We are always building on the shoulders of giants and all ideas are old in some way. Let’s deconstruct some of the concepts behind Light Table and look at a few of the projects that inspire us from decades ago.

Execution everywhere

The notion of always having access to an execution environment is something that the smalltalkers and lispers have had for nearly 50 years now. The notion of developing in a REPL is as fundamental to Lisp as s-expressions and for good reason: it helps remove the disconnection we have with the software we’re building. These guys have always believed we should never have to go look somewhere to find out what something does; instead we can just ask the program and see what happens. Being able to simply try something, no matter how small, fundamentally changes the way you do work. You no longer write huge swaths of code at once and pray. Instead you write little bits and see if they work. Smalltalk environments like Squeak embraced this wholeheartedly - giving you “tiny” editors to put code in.

Instead of large editing surfaces, it provides you tools to navigate the code structure and explore information about the running program itself. When your tool is actually connected to the thing you’re building you can do far more interesting things than just provide a nice text editor. The smalltalkers I’ve talked to largely argue that editing is one of the least important parts of their workflow. Compared to those of us who swear by the amazing editors that are Vim and EMACS, that’s a pretty big departure. We all know, though, that most of our time isn’t really spent typing characters into a buffer. Having the ability to “talk” to our software can provide us far more valuable tools and far more efficient workflows.

A modifiable environment

One of the great features of the lisp machine operating systems was the ability to re-evaluate whole parts of the OS at any time. It was a truly self-modifiable system. This meant that when something didn’t work exactly the way you wanted, you could just change it right there. All of our tools as developers should follow a similar pattern - we should be able to try something directly in the environment without jumping through a bunch of hoops.

The truth is that there really isn’t one workflow or interface to rule them all. Our projects and our personalities demand fluidity. The lisp machines and environments like Squeak gave us access to their internals and presented themselves as clay for us to shape without limitation. This level of freedom is something we have to carry with us in our efforts moving forward.

Editors with meaning

Logo took the abstract notion of programming a computer and gave it a concrete entity for people to latch onto: a turtle discovering a new world. This is a fantastic way to bring new programmers into the world of algorithmic thinking and many would argue that it is still one of the best teaching tools we have.

But what it does is far less important than how it does it. The power behind logo is in giving a visualization of what a program does and that visualization is what allows us to better understand what is happening. Bret Victor recently released a piece on Learnable Programming in which he says that until we are able to see our programs, we can’t really reason about them. While I wouldn’t go quite that far, it’s certainly true that we can’t reason about them well. Giving ourselves something concrete to interact with by bringing abstractions directly into our environment is, as I’ve said before, something I believe will make a fundamental difference in how we write software. Environments like HyperCard gave us concrete metaphors to work with and allowed us to reason about programs in a very meaningful way.

Old ideas finding a new home

People like Alan Kay set the ground work for an environment like Light Table to exist; they ushered innovation that fundamentally changed our profession. But the work doesn’t stop with what they did - so much more needs to be done and there’s a lot more to discover down these paths. These environments didn’t quite make it to the mainstream for reasons both intrinsic and historical, but the notions behind them help guide our efforts toward re-imagining the way we do work today. As such, Light Table is as much about rediscovering the past as it is about crafting a new future.

Light Table - Embracing Abstraction

2012-09-25T00:00:00+00:00

Yesterday I gave a talk at Strange Loop about Light Table, in which I stressed the idea that we are not the “recipe writers” we used to describe ourselves as - that instead we are “abstractioners”. We aren’t writing down the steps that the computer needs to follow to make something happen, but instead we’re consuming and subsequently creating new abstractions. If you look at the past 10 years we went from writing everything from scratch to using frameworks for virtually every aspect of our programs. That is a big jump - it means that we have to know far more than just the basic constructs of the language and things like IO. We have a whole new set of layers to understand that span multiple levels of abstraction. As such, we need tools that help us consume and understand the things we’re building on top of, while simultaneously showing us how the new abstractions we’re creating fit into that landscape.

During my talk I showed some of the latest stuff in Light Table, namely the parts that embrace this notion that the fundamental unit of our work is abstraction. One of the things we want to enable is being able to manipulate the environment directly and quickly in meaningful ways. While not quite ready to release yet, I wanted to show what this looks like and give you a taste of what you can really do when you have complete freedom in shaping your tools. So, here’s a video of what I showed:

Light Table reaches 0.1.0

2012-08-17T00:00:00+00:00

The Light Table Playground is getting a huge update today that we’re really excited to show you. New to Light Table? It’s a new kind of reactive IDE, check out the original post to learn more.

A new launcher

While the shell scripts got us this far, it’s time to bid them farewell and move to a much more robust and useable jar-based launcher. The experience of getting the Playground is now to just download the jar and double click it. The server will remain up for as long as the launcher is open - to close, just close the launcher.

A new mode

Previously, the playground included only the Instarepl - a live code evaluation editor that showed you how values were flowing through - but in the latest release it gets a new mode: the “Table”. The Table allows you to connect to existing projects (or create new ones) and then get to work modifying your code.

Once connected to a project, you can explore the namespaces in your project with the namespace browser. The browser lets you open whole namespaces at a time or open single functions out of many different namespaces. It also gives you the ability to quickly reorder functions within a file simply by dragging and dropping them.

One of the things we’re really excited about testing out is the notion of a “Code Document.” When you open something on the table, it’s added to a document that acts as though all of those things were in a single file. This means you can build up a context and work in it just like you would if file organization mapped cleanly to functionality. It’s time we started working with our code in more logical units; as problems to be solved.

Arguably the most important thing to ensure in Light Table is the ability to very quickly try things and be connected to the programs you’re writing. The Table has a scratch surface that is always in the namespace you were last in and you can swap back and forth between real code and scratch code with a single shortcut. No need for a repl that isn’t in sync with all your changes - this is the fastest way to work in a real environment while maintaining that powerful instant feedback.

A new double click experience for Clojure

With this release, Leiningen is built in, allowing you to connect to your projects and create new ones without having to do anything more than start the server. Clojure now has a true double click experience for getting started.

Real work and experiments

With all of these changes we’re now able to do real work in the Playground. As a matter of fact, we’re actually working on Light Table in Light Table now (oh the meta-ness), but some of these ideas definitely change the development workflow quite a bit and are what we consider experiments. There’s a lot to learn from how you guys use this new stuff and we’ll be looking into the best way for us to collect your feedback. Keep in mind this is still very early software, so take care to protect yourself against potential issues.

A couple known problems:

Top level comments aren’t preserved (it’s fine if they’re on a line with other code)
If you use lots of macros not all the forms are well disambiguated

If you run into obvious bugs or issues, you can log them on Github.

Moving forward

As YC is now starting to wrap up (we’ll be talking about that soon), our plans are solidifying a bit and we should be making some big announcements in the next month or so. Stay tuned and hold onto your seats - it’s going to be a wild ride.

Light Table Playground Levels Up

2012-07-09T00:00:00+00:00

I’m happy to announce the release of v0.0.7 of the Light Table Playground. This version is a serious upgrade to the Instarepl that brings the ability to manually evaluate things and work in your own projects.

Here’s a breakdown of what’s new.

A little guide

The first thing you’re greeted with when you open 0.0.7 is what we’re calling the shade. The shade shows you new things that you can do and acts as a sort of interactive change log. Hopefully this will help keep people up to date on what comes out in a more memorable and useful way.

Manual Mode

You now have the ability to enter manual mode in the Instarepl, by simply clicking the live button in the top right, or by pressing Cmd/Ctrl-L.

Once in manual mode, code is no longer evaluated as you type, but instead when you trigger evaluation of the whole editor with Cmd/Ctrl-Enter or a single form using Shift-Enter. This gives you a lot more flexibility to execute only what you want when you want to, while still maintaining the nice data visualization that the Instarepl offers. This means you can play around more with side-effecty code and other bits that you may not want executing after every keystroke.

The sidebar is currently just a simple menu, but will ultimately be the source of a lot of interaction with Light Table’s command set.

Currently it only has two verbs: connect and set. Set allows you to change the font size, which will be persisted between sessions. Connect allows you to hook into other running Light Table clients.

lein-light

The most exciting addition to the playground is the ability to use your own projects as the context for the Instarepl. You do this by using the lein-light leiningen plugin. Just run lein light from one of your projects and use the sidebar’s connect verb to hook into it.

That’s all it takes to get the playground working in your own projects. Coupled with manual mode, you can now get to town building websites, working with data, or making music.

Misc

Lastly there have been many small miscellaneous changes that seek to help performance by being a bit smarter about when and how things get executed. The addition of basic settings is also there and helps persist font size across opens. We’ve also switched to https to be more secure for the downloads.

So there you have it, go run ./light table to upgrade, or go download the playground for the first time and have fun.

Light Table - 24 hours later

2012-06-26T00:00:00+00:00

Yesterday we released the playground to the world. It has been a ridiculous past 24 hours, so ridiculous in fact that I felt I had to share some of the numbers. Within the first 5 hours over 500,000 lines of code had been eval’d by folks using the Light Table Playground. By this morning, exactly 24 hours after the launch, it looked like this:

More than 1.1 million lines of code have been executed! Crazier, however, is the number of active minutes spent using it to do exactly that. In a 24 hour period over 755 active person hours were spent having fun with Clojure. That’s more than 31 days - you guys fit an entire month in a day.

The community has consistently surprised us at every step of this project and once again your enthusiasm, support, and feedback have been nothing short of awe inspiring. We wanted you to see how amazing it really is and thank you for joining the ride with us.

It's playtime - Light Table Playground released

2012-06-24T00:00:00+00:00

I’m happy to announce the official release of the Light Table Playground! You can find instructions for getting it here: http://app.kodowa.com/playground

What is the playground?

The playground is a chance for you to follow along as we test out new things for Light Table. In return, it helps us collect metrics on how people are using our experiments as well as gives us an opportunity to see how certain concepts fare in the real world. All in all, it keeps us connected to you guys and you guys connected to us.

Currently, the playground includes what we call the “instarepl” for Clojure. This is the real-time evaluation and code-flow stuff we showed in the first video. It serves as a fun scratch pad for either learning Clojure or when you’re working and trying out random things. A number of folks have been using it to do 4Clojure or Project Euler problems, which can be a great starting point if you’re new to the language and itching to try something.

Wait, what’s Kodowa?

Kodowa is the official entity behind Light Table. It’s still early days for us, so we’re not quite ready to introduce it fully, but don’t be surprised if you see the name associated with Light Table more as time goes on.

The plan moving forward

The playground is a very stripped down version of Light Table, but it helps us prove out the platform necessary to support the vision we’ve laid out. With the update mechanism built into it, we’ll be releasing new things often with the goal of turning it into a full-fledged Clojure environment over the next few months. At the end of which, we’ll have a solid foundation to continue forward on. Our hope is that you’ll follow along with us, so that we can learn more about you and maybe introduce some new minds to Clojure.

There are very exciting things to come and now that we’ve hit this milestone, we can do a lot more than just tell you about what we’re thinking - we can begin to show you.

Ramblings

2012-06-05T00:00:00+00:00

I love doing presentations and I’m happy to say that we have a few for Light Table lined up throughout the year:

6/7/2012 - Bay Area Clojure Meetup
7/19/2012 - OSCON
9/23/2012 - Strange Loop
11/7/2012 - Oredev

The Clojure meetup will be a really informal demo of Light Table with most of the time spent on Q&A, while the others are more normal talks. I’ll let you know if any more crop up!

45 days later

2012-06-01T00:00:00+00:00

Just a few moments ago our Kickstarter campaign closed with a truly amazing 7,317 backers at $316,720. It’s been such a whirlwind story over the past 45 days from a simple blog post to what is now a funded startup, a successful kickstarter campaign, and a seat at the YC table.

None of that would have been possible without the faith and dedication of all of you. For your support, I give you my sincerest thanks. Now it’s up to us to deliver the future we’ve envisioned. I hope you’re as excited as we are. It’s going to be a wild ride.

The future is specific

2012-05-21T00:00:00+00:00

You can now try Light Table out via the Light Table Playground!

I’m going to make a bold claim: The future of tools isn’t in a better Eclipse or Visual Studio, it’s in easily created domain specific experiences.

If we look at the trends in programminging languages over the past decade, we’ve seen a resurgence of metaprogramming and DSLs. We as an industry are starting to rediscover the power of being able to write language we use to build our software. It doesn’t seem like much of a stretch to apply these notions to our tools. As a matter of fact, wouldn’t it stand to reason that if domain specific languages increase our ability to write and understand our code that domain specific tools would as well?

While generalized editors and IDEs have proven very useful and have helped us get to where we are today, they are necessarily ok at everything and not amazing at any one thing. When I first introduced Light Table, I showed what a general programming environment based on a set of principles might look like. That, however, is not the real power of what we’re building - the real potential is in making it trivial to build domain specific tools. Let me show you what I mean.

Here’s a breakdown of some of the things I showed in the video:

Example one: a benchmarking mode

This is a simple example of a mode created in a few hours to accompany a little benchmarking library I wrote. On the left there are boxes for different versions of the same code. Evaluating the top one sets the baseline run for the benchmark, while evaluating any boxes underneath will count as runs against that baseline.

To the right is the code of the benchmark as well as a graph of the results and a table displaying the actual execution times. Here you can see that one of the lines is tinted red - that’s because the result wasn’t the same as the baseline, which indicates that I probably introduced a bug in my changes.

Example two: a SQL mode (using Korma)

This is a mode for crafting queries using Korma. Korma is a SQL DSL that allows you to compose queries over time. With this mode you can see how the query object is constructed and see the resulting sql as you type.

If you evaluate the Korma buffer, it will then show you the results off to the side in a nice little list. This allows you to quickly mess around with some Korma code and see what it does internally as well as test that the results are what you expect.

Example three: a Flask mode (Python!)

I hinted that my next demo would include an example of Python and here it is. In this case, we have a mode for the Flask microframework that helps you build websites faster by bringing code together and organizing it cleanly into the layers it belongs in. The result is a mode with an embedded browser, boxes of code related to your current route, and the ability to filter out layers.

In traditional web-MVC, the code necessary to serve a single route is spread across many files in many different folders. In a normal editor this means you need to do a lot of context switching to get a sense for everything going on. Instead, this mode replaces the file picker with a route picker, as routes seem like the best logical unit for a website. When you click on one of these, you’re given boxes with all of the code related to serving that route - even going so far as to figure out what templates you referenced and getting the subtemplates out of those. This means you have a single context in which you are able to understand the entirety of the code necessary to serve that page.

Since MVC usually has a specific structure, you can also begin to add some knowledge of the architecture of the website to the tool. In this case, we know where all these functions and bits of code come from and as such can automatically place them in the appropriate architectural layers (routing, modeling, templating). You can then turn those layers on and off to hide the bits of code that aren’t currently pertinent to what you’re doing.

A parallel

It occured to me the other day that what we’re talking about is something like a macro system for tools. To the lispers out there, that statement probably hits home - macros are incredibly powerful. Imagine being able to create these sorts of experiences on a whim instead of needing hundreds of hours to even get something simple working. If you couple that with the generalized editing capabilities I showed last time, you have what we believe to be the future of tools: an environment that you are able to mold to the exact shape of your problem.

An orange glow

2012-05-17T00:00:00+00:00

I’m happy to announce that as of yesterday Light Table has joined a host of amazing startups for the Summer 2012 batch of YCombinator. Under the guidance of YC we believe we have the best possible chance to not just make Light Table a reality, but to help craft a new future for the way we create software.

This does, however, mean that circumstances have changed. We rallied together as a community to ensure that this project had a future. While being a part of YC doesn’t affect our plans or change our use of Kickstarter as a means of accelerating the release of Light Table, many pledged with the explicit purpose of making sure that it had a chance. With YC that chance is now safe and we wanted to let our supporters know there will be no hard feelings if this changes your desire to pledge. We want to thank you all the same for being a part of the push that has gotten us this far. In the end, the important part is that we were able to come together around this ideal and now we can work to make it a reality.

Thank you to everyone who has helped get us here and I look forward to showing you some of the work we’ve been doing in the next few days.

We made it

2012-05-12T00:00:00+00:00

After an amazing showing by the development community, Light Table has hit its goal of $200,000 with 20 days left to go! I want to extend my sincerest thanks to all of you who have helped spread the word, and committed to a future of a better tools. It has been an amazing past couple of weeks that has given me a chance to talk to some of the best and brightest in the business. I think as a group of builders we’re ready to finally see our tools work for us and I’m honored that you’ve put your faith in me to do that.

I’ve been quiet over the past couple weeks as I’ve been putting together a demo to show the real power of what’s to come and I can’t wait to show it to you all. I’ll give you a little hint: I’ve been making a lot of sssSSSSssssSSssss noises lately ;)

On concepts and realities

2012-04-22T00:00:00+00:00

It’s been really interesting seeing how people have responded to the idea of Light Table over the past week and understandably there have been a lot of questions as the ideas have sunk in (for me too). Many of these are questions about the details of exactly how this or that feature will work and that brings me to something I want to address: what I showed is not a final product, but instead an example of what could be built based on the principles I’ve established. This distinction has a few important points.

The principles of Light Table are what matter

Features are necessarily specific to certain contexts, but as long as they are created from a strong and unifying conceptual foundation, they will accomplish the same goals even if by different means. Here’s what Light Tables principles are:

You should never have to look for documentation
Files are not the best representation of code, just a convenient serialization.
Editors can be anywhere and show you anything - not just text.
Trying is encouraged - changes produce instantaneous results
We can shine some light on related bits of code

The problem is that most of our environments today are violating some of these. In my video I showed a prototype I built in under a week. It’s just a sampling of what something designed to uphold these ideas might look like. Was it user tested? No. Has it been confirmed to work in all case? No. Is it exactly what Light Table will look like? Probably not. We have to prove these things work, we have to continue to learn about how we develop, how code comes into being, and how we utilize what’s already written. We’ll learn a lot along the way and I know that will remold not just our understanding of the problem, but also of the product. That means I don’t fully know what Light Table will look like in the end. But so long as we stick to our principles, I believe that we will have something that is just as magical as what I showed and more importantly a highly efficient environment for creating things.

One size does not fit all.

I mean this in virtually every sense. The things I showed in my concept video weren’t meant to fit all scenarios - in that specific case they happen to fit Clojure very well. One advantage of focusing on principles over features though, is that what Light Table does and how it functions doesn’t have to be the same for every language. To me, this is necessary not just for technical reasons, but for cultural ones as well. Our communities are wildly different from language to language, so much so that idioms even contradict each other. One size will never fit all here. The way Clojure developers work is very different than the way C developers do, but that doesn’t mean that Light Table’s principles don’t apply in both cases. We simply have to tap into the community to shape the right experience for each language we tackle. This means lots of user testing, lots of iteration, and tons of exploration. In doing that, we ensure we’re not shoehorning features into contexts that don’t make sense and instead providing something that feels natural to the tasks at hand.

I haven’t thought about everything - and I won’t be able to.

The number of possibilities and edge cases working with something so fundamental is effectively infinite. There will always be scenarios that seem to fall outside of the scope of certain ideas. Ultimately, however, that doesn’t mean that solutions don’t or can’t exist. Light Table is meant to be a platform and it will be open source for that very reason. The team I put together certainly won’t be able to take on everything and it’s up to us as a community to decide what’s worth putting effort into. Novel solutions to problems spring up all the time and while it may seem that live evaluating a compiled language like C is far off, I believe that given the motivation someone will find a way. This is the power of a community full of builders. I may not produce the perfect product, regardless of how hard I try, but that’s not exactly the point either. The point is to lay down the foundation for such a thing to evolve over time, to simply get the ball rolling and prove that we can do better. In the end, the only way to move us forward is for the community to rally behind something much bigger than a single piece of software - we have to rally behind a concept and an ideal. My goal is to take a stab at what that might look like.

If you want to help me achieve it, spread the word and consider pledging on the Kickstarter project.

Light Table's numbers

2012-04-15T00:00:00+00:00

EDIT: Light Table is now on kickstarter!

EDIT: oh, and I set up a mailing list to keep informed.

All I can say is “wow”! The response to Light Table has been far more positive and far greater than I could’ve imagined. To put it into perspective, here are some aggregated numbers at the time of this writing:

There were 105,872 unique visits to the post, just shy of 200,000 pageviews and 80,000 video plays.
It has been viewed in 163 countries (there are only 204 total)
The post on HN stayed on the front page for more than 48 hours (It’s actually still there now) and has amassed more than 1400 points. I believe that makes it one of the largest “launches” in HN history.
The post has more than 1500 points and 469 comments on reddit.com/r/programming.
I’ve gotten hundreds of emails from all sorts of different people, thousands of tweets, and a mountain of comments to go through.

So the big question on most people’s minds is “Now what?”. To be honest, when I put Light Table up, I was mostly just looking to share some revelations I had recently. With that kind of response, however, it’s pretty obvious that the industry needs something here.

I was surprised at the number of people who cried out for a Kickstarter for the project. After some thought and some serious discussions with my best friend (and co-founder), we’ve decided to switch gears from a different project and do exactly that. I’m happy to announce that we submitted our Kickstarter earlier today and are simply waiting for it to be reviewed. In the meantime, I thought I’d post a couple of the FAQ-y type things from our description there:

What languages will it support?

The first two languages it will support are Javascript and Clojure, but the application will be written in such a way that adding new languages can happen through plugins.

Can I script/extend it?

It will be scriptable in Javascript or anything that can compile down to it :) Ultimately the goal of the platform is to be a highly extensible work surface - even the initial core languages will be written as plugins. This allows us to build development interfaces we haven’t even imagined yet.

What about key bindings?

I’m a VIM guy myself, but since we’ll be using the awesome CodeMirror editor, this is something that is easily adapted. If you’re looking for a way to contribute, help improve CodeMirror and its emacs/vim plugins!

Will it be open source?

I’m a firm believer in open source software and open source technologies. I can guarantee you that Light Table will be built on top of the technologies that are freely available to us today. As such, I believe it only fair that the core of Light Table be open sourced once it is launched, while some of the plugins may remained closed source. At some level, this is an experiment in how open source and business can mix - it will be educational for us all.

How can I help in the meantime?

Like I said above, help improve CodeMirror. The better that editor is, the better all editors on the internet can be! Past that, help us spread the word. The more money we get the more people I can involve in the project, the more languages we can support, and the more powerful we can make the entire platform. There’s tremendous potential that we haven’t even scratched the surface of yet!

Light Table - a new IDE concept

2012-04-12T00:00:00+00:00

You can now try Light Table out via the Light Table Playground!

Light Table’s kickstarter has wrapped up!

Despite the dramatic shift toward simplification in software interfaces, the world of development tools continues to shrink our workspace with feature after feature in every release. Even with all of these things at our disposal, we’re stuck in a world of files and forced organization - why are we still looking all over the place for the things we need when we’re coding? Why is everything just static text?

Bret Victor hinted at the idea that we can do much better than we are now - we can provide instant feedback, we can show you how your changes affect a system. And I discovered he was right.

We can do better, and to that end, let me introduce you to

Light Table is based on a very simple idea: we need a real work surface to code on, not just an editor and a project explorer. We need to be able to move things around, keep clutter down, and bring information to the foreground in the places we need it most. Here’s what the default mode looks like:

Light table is based on a few guiding principles:

You should never have to look for documentation
Files are not the best representation of code, just a convenient serialization.
Editors can be anywhere and show you anything - not just text.
Trying is encouraged - changes produce instantaneous results
We can shine some light on related bits of code

Let’s take a look at how these things manifest themselves in Light Table.

Docs everywhere

When you’re looking at new code it’s extremely valuable to be able to quickly see documentation left behind by the author. Normally to do so you’d have to navigate to the definition of the function, but lightable ghosts this information in to the side. Want to know what partial does? Just put your cursor on top of it. This makes sure you never have to worry about forgetting things like argument order ever again.

We work with new code all the time and it’s easy to forget what that function that adds a link on to a page is - we should be able to search all our documentation in place to quickly see what it is. Don’t remember what was in the noir.core namespace? It’s one ctrl-f away.

This is especially handy for finding functions you may not even know exist and seeing their docs right there. No need to look at some other generated documentation.

Instant feedback

In Inventing on Principle, Bret showed us that we could live-edit games and write binary search in an editor that is constantly evaluating and showing you what’s going on. The lispers among us are used to using the REPL to have an environment where we can try things out. But we can do better - we can do it in place and instantaneously. For example here I type the code (+ 3 4) and we immediately see that it evaluates to 7 - no ctrl-enter or anything else.

Light Table takes this idea as far as it can and doesn’t just show you variables to the side, but actually shows you how the code is filled in. This lets you see how values flow through arbitrarily complex groups of functions.

This level of real-time evaluation and visualization basically creates a real-time debugger, allowing you to quickly try various inputs and watch it flow through your code. There’s no faster way to catch bugs than to watch your program work.

We built drafting tables for a reason

Towards the end of my time on the Visual Studio team, I came to the conclusion that windows aren’t a good abstraction for what we do. Other engineers have large tables where they can scatter drawings, tools, and other information around. A drafting table is a much better abstraction for us. We shouldn’t need to limit ourselves to a world where the smallest moveable unit is a file - our code has much more complex interactions that we can better see when we can organize things conceptually.

We saw an example of this with Code Bubbles, but it doesn’t take it far enough - why can’t we embed a running game on our work surface? Then we can interrogate it, ask it questions and have our environment answer them for us.

Code with a little illumination

There’s no reason our tools can’t help us understand how things are organized in our programs. In light mode, Light Table let’s you see what functions are used inside of the one you’re currently working on, not just by highlighting ones in your code, but by also showing you their code to the side.

We shouldn’t have to constantly navigate back and forth to see how the various bits of our code work with one another.

Finally, all of this culminates in the ability to see not just how things I type into a scratch editor evaluate, but how values flow through our entire codebase. Here I find a bug where I wasn’t passing x correctly. I type (greetings ["chris"]) and immediately see all the values filled in not just for the current function but all the functions that it uses as well.

So how does it work?

As you see in the video I have a prototype of this working. It’s built as a web application using Noir, my various ClojureScript libraries, and CodeMirror. I use a slightly modified version of the Clojure compiler to retain some metadata about forms that is currently lost (like column positions and other position data). Past that, it’s just a matter of running over the analysis tree and a healthy sprinkling of magic ;)

A note on languages

It’s no secret that I really like Clojure and as a lisp, it was the easiest language for me to start the prototype with, but there’s no reason this couldn’t be done for any language with a dynamic runtime. The rest is mostly simple analysis of an AST and some clever inference. So could Light Table have used JS instead? Certainly - and hopefully it will get there sooner rather than later.

The opportunity

I’ve always been fascinated with development tools - creating the things used to create. There’s an incredible opportunity to change the way we build, and as a result introduce tremendous value into the world. While I worked on Visual Studio, I began to see the pieces of what we could do here. This is just the beginning; the seeds of what could be. It’s time for us to get out of the box and start reimagining the very things that enable us to create the world we know.

Discuss this on HackerNews

Connecting to your creation

2012-02-26T00:00:00+00:00

edit: there is now a downloadable jar: https://github.com/ibdknox/live-cljs/downloads

After seeing Bret Victor’s talk, Inventing on Principle, the other night, I was curious how hard it would be to build some of things he demoed - so I put together a live ClojureScript game editor today. Here’s a video of it in action:

It basically replicates what was in his demo (minus the Braid graphics). The one thing I didn’t implement was rolling forward and back with a slider, since I thought the projection was far more useful. One interesting difference about this implementation is that since it uses ClojureScript, it has to compile the code that you edit on the fly. Doing this was surprisingly simple by using a hacked-up version of some the repl code. Most of the issues I had with it were related to weirdness with .class files for the compiler getting created and screwing things up.

Some things this demo does:

All changes you make to the code are reflected immediately in the running game
Once focused, clicking the canvas adds a block. Clicking a block removes it.
Pressing “s” will pause the game and show a projection of what you did
Altering the code while paused will alter the projection.
Altering the game while paused will alter the projection (try placing a block in the path)

One thing that struck me as I built this was exactly how much fun it is to mess with something in real-time. Seeing how the path of the guy changes when you put a block in his trajectory is incredibly interesting. Essentially I learned that Victor was right - there’s unquestionable value in connecting yourself with your creation. If you haven’t watched the talk go do it. It’s well worth the hour.

You can get the code here and play around with it yourself.

I didn’t put it up on heroku because the latency kills the real-time aspect of it. If you have lein installed all you need to do is clone the repo, do lein run and open your browser to http://localhost:8074.

Discuss this on HackerNews

Overtone and ClojureScript

2012-02-20T00:00:00+00:00

Lots of folks have been interested in ClojureScript lately, but have had a hard time figuring out what a CLJS app actually looks like. So today I recorded myself building an Overtone controller (that I use on an iPad) using noir, fetch, jayq, and crate. In the end, it looks like this:

Since I don’t narrate in the video, I figured I’d give a breakdown of some of the main ideas below. If you want all the gory details though, you can watch the screencast or look at the code. Now to the fun part.

Getting started

The first step is to generate a new noir project using lein-noir (if you’re new to noir, check out noir’s website)

lein noir new overtoneinterface

Now to set up our project we just need to include our dependencies, which with the wonderful lein-cljsbuild means you do what you always do - add them to your project.clj. ClojureScript dependencies don’t really work any differently than Clojure ones do:

(defproject overtoneinterface "0.1.0-SNAPSHOT"
  :description "FIXME: write this!"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [overtone "0.6.0"]
                 [jayq "0.1.0-SNAPSHOT"]
                 [crate "0.1.0-SNAPSHOT"]
                 [fetch "0.1.0-SNAPSHOT"]
                 [noir "1.3.0-alpha10"]]
  :cljsbuild {:source-path "src"
              :compiler
                {:output-dir "resources/public/cljs/"
                 :output-to "resources/public/cljs/bootstrap.js"
                 :optimizations :simple
                 :pretty-print true}}
  :main overtoneinterface.server)

You see here that we also added a cljsbuild key that defines some properties for how we want out ClojureScript to be generated. They simply tell lein-cljsbuild where to find our source and where to place the output. To then get that into our app, we just need to include the generated javascript file and jquery in our views.common/layout function.

(defpartial layout [& content]
            (html5
              [:head
               [:title "overtoneinterface"]
               (include-css "/css/reset.css")
               (include-css "/css/default.css")
               (include-js "https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js")]
              [:body
               [:div#wrapper
                content]
               (include-js "/cljs/bootstrap.js")]))

Then just modify welcome.clj to get rid of the getting-started content, change /welcome to /, and add a div#piano so we have a container to put our buttons in.

(ns overtoneinterface.views.welcome
  (:require [overtoneinterface.views.common :as common])
  (:use [noir.core :only [defpage]]
        [hiccup.core :only [html]]))

(defpage "/" []
         (common/layout
           [:div#piano]))

I always end up creating a src/myapp/client/ directory where I keep my CLJS. So if you put the following in a main.cljs and fire up lein-cljsbuild, you’ll see a nice little alert box:

(ns overtoneinterface.client.main)

(js/alert "hey!")

Now we’re off to the races.

Using crate and jayq

crate is a ClojureScript implementation of the HTML generation library Hiccup, which represents html as Clojure vectors and maps. We use a special macro called (defpartial ..) to create a function that will create dom objects for us.

(ns overtoneinterface.client.main
  (:require [crate.core :as crate])
  (:use-macros [crate.macros :only [defpartial]]))

(defpartial button [{:keys [label action param]}]
  [:a.button {:href "#" :data-action action :data-param param} label])

One thing to note here is that there’s a special directive for requiring macros in CLJS. Also, any namespace used by that macro must be required as well, or otherwise that code won’t end up in the generated file. Now to do something with it, we’ll use jayq.

jayq is a simple ClojureScript jQuery wrapper that I wrote, which makes it easy to do all your standard dom manipulations like you’re used to.

(ns overtoneinterface.client.main
  (:require [crate.core :as crate])
  (:use [jayq.core :only [$ append delegate data]])
  (:use-macros [crate.macros :only [defpartial]]))

(def $piano ($ :#piano))

(defpartial button [{:keys [label action param]}]
  [:a.button {:href "#" :data-action action :data-param param} label])

(append $piano (button {:label "play note"
                        :action "play-note"
                        :param ""))))

It does, however, add some interesting bits. One of which, is that dom elements created with crate can be referenced by the function that was used to create them. This is actually immensely useful, because it basically gives you named controls for free. For example, we end up adding a click handler for all our buttons like so:

(def $body ($ :body))

(delegate $body button :click
          (fn [e]
            (.preventDefault e)
            (js/alert "clicked!")))

Time to make that handler a bit more interesting. We’re here to make music afterall.

Interacting with the server - fetch

fetch is the next piece of the puzzle which helps us by removing the barrier between the server and the client. In this case, we’re going to use remotes, which are functions defined on the server that are then called by the client. Normally, these would look something like this:

(letrem [result (some-remote-func 2 3 4)]
 (.log js/console result))

But since we want to call these dynamically based on whatever action our button is created with, we’ll need to drop down one level and use (fetch.remotes/remote-callback remote-func params). To do this however, we also need to be able to get a reference to the dom element that was clicked. In jQuery, you usually use this, but in ClojureScript “this” is a symbol just like anything else. For us to get at the js “this”, we’ll simply use the macro (this-as some-symbol-meaning-this ... )

(ns overtoneinterface.client.main
  (:require [crate.core :as crate]
            [fetch.remotes :as remotes]) ;; add fetch.remotes
  (:use [jayq.core :only [$ append delegate data]])
  (:use-macros [crate.macros :only [defpartial]]))

...

(delegate $body button :click
          (fn [e]
            (.preventDefault e)
            (this-as me
              (let [$me ($ me)
                    action (data $me :action)
                    param (data $me :param)
                    params (if (= param "")
                             []
                             [param])]
                (remotes/remote-callback action params)))))

What that does is extract the action and param attributes from our button and then tells fetch to call the remote function whose name is the value of action with the params we give it. On the Noir side, you then just needs to do two things - add the wrap-remotes middleware in server.clj (make sure you restart the server!):

(ns overtoneinterface.server
  (:require [noir.server :as server]
            [noir.fetch.remotes :as remotes]))

(server/load-views "src/overtoneinterface/views/")
(server/add-middleware remotes/wrap-remotes)

(defn -main [& m]
  (let [mode (keyword (or (first m) :dev))
        port (Integer. (get (System/getenv) "PORT" "8080"))]
    (server/start port {:mode mode
                        :ns 'overtoneinterface})))

And define a remote in views/welcome.clj

(ns overtoneinterface.views.welcome
  (:require [overtoneinterface.views.common :as common]
            [overtoneinterface.models.dubstep :as dubstep])
  (:use [noir.core :only [defpage]]
        [overtone.live]
        [overtone.inst.sampled-piano]
        [noir.fetch.remotes :only [defremote]]
        [hiccup.core :only [html]]))

(defpage "/" []
         (common/layout
           [:div#controls]
           [:div#wobble]
           [:div#notes]
           [:div#piano]))

(defremote play-note [n]
  (sampled-piano n))

;;play-note is also just a regular function, meaning you could use
;;it in your clj code like normal..
;;(play-note 60)

If this is the first time your server has loaded overtone.live, it may take a few seconds for it to refresh as it has to startup supercollider and a few other things. Also, if this is your first time ever using the sampled piano, it has to download a pretty large set of samples (this can take an hour). Assuming you have both of those though, clicking the button will cause a tone to be played. In the video, this happens at 11:20.

Adding a bit more.

At this point the fundamentals of the app are there, the rest is just icing on the cake. I clean up the code so that it’s easy to add a bunch of buttons to a container and create more piano keys for us to click:

(def piano-notes (for [note (range 40 60)]
                   {:label (str note) :action "play-note" :param note}))

(defn populate [container buttons]
  (doseq [b buttons]
    (append container (button b))))

(populate $piano piano-notes)

Then I grab the code from the dubstep example in the Overtone repository and drop it in, create a couple more remote functions and we then have the ability to fully control our little dubstep machine. The final welcome.clj looks like this:

(ns overtoneinterface.views.welcome
  (:require [overtoneinterface.views.common :as common]
            [overtoneinterface.models.dubstep :as dubstep])
  (:use [noir.core :only [defpage]]
        [overtone.live]
        [overtone.inst.sampled-piano]
        [noir.fetch.remotes :only [defremote]]
        [hiccup.core :only [html]]))

(defpage "/" []
         (common/layout
           [:div#controls]
           [:div#wobble]
           [:div#notes]
           [:div#piano]))

(defremote play-note [n]
  (sampled-piano n))

(defremote start-dub []
  (dubstep/start-dub))

(defremote stop-dub []
  (dubstep/stop-dub))

(defremote dub-note [n]
  (dubstep/alter-dub :note n))

(defremote dub-wobble [n]
  (dubstep/alter-dub :wobble-factor n))

And here’s the final main.cljs:

(ns overtoneinterface.client.main
  (:require [crate.core :as crate]
            [fetch.remotes :as remotes])
  (:use [jayq.core :only [$ append delegate data]])
  (:use-macros [crate.macros :only [defpartial]]))

(def $body ($ :body))
(def $piano ($ :#piano))
(def $controls ($ :#controls))
(def $notes ($ :#notes))
(def $wobble ($ :#wobble))

(defpartial button [{:keys [label action param]}]
  [:a.button {:href "#" :data-action action :data-param param} label])

(def piano-notes (for [note (range 40 60)]
                   {:label (str note) :action "play-note" :param note}))

(def dub-notes (for [note (range 40 80)]
                   {:label (str note) :action "dub-note" :param note}))

(def dub-wobble (for [w (range 0 8)]
                   {:label (str "w" w) :action "dub-wobble" :param w}))

(def control-buttons [{:label "start" :action "start-dub" :param ""}
                      {:label "stop" :action "stop-dub" :param ""}])

(defn populate [container buttons]
  (doseq [b buttons]
    (append container (button b))))

(populate $piano piano-notes)
(populate $controls control-buttons)
(populate $notes dub-notes)
(populate $wobble dub-wobble)

(delegate $body button :click
          (fn [e]
            (.preventDefault e)
            (this-as me
              (let [$me ($ me)
                    action (data $me :action)
                    param (data $me :param)
                    param (if (= param "")
                            []
                            [param])]
                (remotes/remote-callback action param)))))

And there you have it - a complete overtone controller in about 20 minutes.

Discuss this on HackerNews.

A new leaf

2012-02-19T00:00:00+00:00

I’m currently remodeling my site a bit, so mind the dust while I clean up today. :)

chris-granger.com

"Is Web3... anything?"

Two years of Eve

A separation of concerns

Eve the platform

An important step toward end users

Eve the UI

Onward

Eve Version 0

What’s in version 0?

How we got here

Where we’re headed

Coding is not the new literacy

Coding is not the fundamental skill

Modeling is the new literacy

Creation is exploration

Exploration is understanding

Digital freedom

A fundamental disconnect

How we teach children

How we should teach adults

“The computer revolution hasn’t happened yet”

Beyond Light Table

We’ve raised $2.3M from Andreessen Horowitz.

We’re hiring!

And Light Table?

Toward a better programming

How do we make programming better?

What is programming?

And what’s wrong with it?

Programming is unobservable

Programming is indirect

Programming is incidentally complex

Chasing local maxima

The people’s programming

A note on culture

Great, now what?

Light Table is open source

Plugins

Inline docs and doc search

Clojure(Script) nrepl, auto-complete, jump to definition, paredit…

Performance, stability, and polish

On the road again

Links

Light Table is growing

Light Table 0.5.0

Faster, better, stronger

Simple

Powerful

Fast

Healing 1000 papercuts

To plugins we go!

Links

Cancer and Startups

Light Table 0.4

A bit of the future.

Next-gen web dev

Speaking snake: Python support

Documentation

And then literally everything else

Do you teach programming?

The plan

Links

Light Table a year ago

A new Light Table experience

A gorgeous new look

An inline results experience

JS and CSS eval

Workspace and tree view

Quick, fuzzy navigation

Command search

Vim mode and custom key binding

It “just works”

Toward 0.4.0

Links

The IDE as a value

Meet BOT

Objects

Behaviors

Tags