Of Sterner Stuff

My 2011 Reading Highlights

2011-12-28T10:21:00.000-08:00

Here are some books that I really enjoyed from 2011 (i.e., I read them in 2011 - many of them were published long ago):

Everything is Obvious
The Master Switch: The Rise and Fall of Information Empires
A Soldier of the Great War
Microcosm: E. coli and the New Science of Life
The Rational Optimist: How Prosperity Evolves
All Quiet on the Western Front
Unbroken: A World War II Story of Survival, Resilience, and Redemption

Right now I'm reading these:
The Penguin and the Leviathan: How Cooperation Triumphs over Self-Interest
To Draw Closer to God: A Collection of Discourses

I'm also on Goodreads.

The benefits of 7 billion people

2011-11-19T15:21:00.001-08:00

How many of our technologies and inventions are a result of a large population?

You don't spend $10M on a drug that might cure 50 people...

Connections Matter

2011-11-18T13:15:00.001-08:00

The Algorithms of Wall Street
On May 6, 2010, the Dow Jones Industrial Average fell nearly 1000 points - the largest single-day drop in history - only to regain nearly everything that was lost within minutes.

So what happened? Was there some sort of dramatic news that was reported, and then recanted? A war that almost happened? An election reversed? These are the sorts of dramatic events that seem necessary for such an extreme trading pattern.

In fact, there was no such root cause. While the details are still sketchy, one theory for the cause of the "Flash Crash" is that a single large sell order triggered high-frequency trading algorithms (i.e., computer-directed trades) to start selling. The algorithms were optimized to deal with small changes in price, and as prices started plummeting, the computers continued to trade with each other, driving prices down even further. Where a professional trader could tell that prices were too low, there were no people in the loop - it was just computers buying and selling to each other.

While this is a dramatic example of how eliminating people from a process can prove disastrous, it highlights just how important computer-to-computer interactions are. In fact, I believe that computer-to-computer communication will turn out to be one of a (very) long string of "Connection Revolutions", which I would like to briefly outline. As usual, I will blur the line between biological and technological advances, because I see them as being very similar.

I want to quickly outline each of these revolutions, and possibly follow up with a blog post about each one.

Connection Revolutions

Environment Sensing
Very early in the history of life, organisms gained the ability to sense their environment, and respond to it. These first senses were incredibly rudimentary, and chemical in nature. An organism that recognizes the chemicals that occur near its food source, for example, will be more successful in finding food, and will pass that trait on to its offspring.

Eventually, organisms not only sensed the environment, but intentionally altered the environment by emitting chemicals, thus communicating with other organisms. These communication channels can be surprisingly complex, allowing even one-celled bacteria to determine important information about their environment, and to coordinate their actions.

Multicellularity
This coordination allowed some organisms to become so closely linked that they could specialize, subdivide their tasks, and become multicellular. At first, their larger size allowed them to become bigger, and thus move up the food chain. In time, specialization and intimate intercellular communication made specialized organs possible, allowing bodies to grow much larger, and much, much more complex.

Brains
One of those organs was the brain, composed of cells whose only job is communication. Brains take in stimuli collected from all over the body, process it, and send instructions for how to react. By centralizing this information processing, relations between stimuli could be discovered and responded to.

Language
Some animals learned to signal directly, chemically, visually (think peacocks feathers or flowers), and most importantly for our timeline, aurally. Some animals learned to produce and sense sounds. In the case of humans, our supersized brains allowed us to progress beyond simple grunts and communicate whatever we wanted.

This allowed people to communicate about things other than the immediate present. Concepts and ideas could be discussed and examined. Instructions about how to hunt, plant crops, etc. could be relayed unambiguously. It is possible that language is what enabled our ancestors great migration out of Africa.

Writing
Writing allowed language to be conveyed not just to those immediately present, but across time and space. Vastly more knowledge could be preserved and distributed, making complex societies and technologies possible.

Printing Press
The printing press seems like an incremental improvement to hand-written writing, but its effects were much more important than the technological improvement would predict. By making books financially accessible to everyone, I believe the printing press was a necessary condition for universal literacy, democracy, the Renaissance, and the Industrial Revolution. By making it possible to (nearly) perfectly duplicate even seemingly unimportant books, knowledge about disparate subjects could be gathered, assimilated, and serve as inputs for further knowledge and technology creation.

Mass Media
The rise of mass media (i.e., TV, radio) was another important change. For the first time, media could be produced and distributed simultaneously to entire nations. This had important impact in creating a shared culture, and a shared way of seeing the world. Its masters held great power, and some, like Joseph Goebbels, understood just how revolutionary and powerful it could be.

Internet
The Internet is the next great communication revolution. I definitely want to write another post about this, but I want to outline a few of the important trends that I see from the Internet.

First, the ability to communicate one-to-one with anyone else, immediately. Every person is connected to every other person, and theoretically could communicate with any of them, without intermediaries.

Second, we are storing even unimportant things, digitally, where they can be preserved and searched forever. By aggregating seemingly unimportant things (e.g., which web pages link to which other web pages), amazing things can be built and discovered (e.g., Google).

Third, the Internet allows our things to talk to each other. Never before have our tools been able to tell us, or each other, what they need. We are more and more able to craft an environment that understands us, and can interact meaningfully with us. This has some dangers, like the "Flash Crash", but I believe that the benefits will become more and more apparent.

These changes are incredibly important, and I believe that we are like the readers of the first printed books, seeing a cool new technology, but blind to the societal changes (e.g., the Renaissance, democracy, and the Industrial Revolution) waiting in the wings.

Latent Celebrity

2011-10-18T15:58:00.000-07:00

In the past, society only had the resources to keep track of the currently pertinent. Books, magazines, and newspapers have limited space, and limited resources, so they cover only (what they consider to be) important things and important people.

Now, because so much of our lives are digital, we can store everything, and decide later what is important. Obscure action by obscure people can be stored, and later gain relevance when either the action or the person turns out to have been important, in retrospect. It's as though the data is just waiting for its importance to be revealed - a sort of latent celebrity.

For example, in 1991 a Master's student in Finland was working on a hobby operating system, and asked if anyone wanted to pitch in. The project eventually became Linux. Because the conversation was held on a digital mailing list, we can now study the founding of Linux using the original sources. That is really amazing.

Perhaps just as importantly, there is even a record of similar projects that failed. When publishing is a bottleneck, there is a bias toward focusing on the positive. By having full data on the winners and the losers, we can do much better at finding the source of the success (which might be just random).

There is a dark side to this "latent celebrity" - this storing of everything. As a silly example, digital search and transmission makes it easy for sites like failbook.com to publicize obscure idiocy for the world to see.

I believe that we are just beginning to explore the implications of a digital culture.

Immortal Artifacts

2011-10-18T14:24:00.000-07:00

I found this video very humorous, and probably a very apt description of how our understanding of the past compares to reality.

However, the same lessons don't extrapolate into the future. We are leaving digital artifacts, which are (potentially) immortal. It's very possible that a historian in 3000 A.D. could listen to all of the Beatles' recordings, with exactly the same quality that we can listen to them now. He could watch their movies, and read newspaper articles about them.

Part of what makes another culture feel old or distant is the fact that aging is a lossy process - their paintings fade, their buildings collapse, and their books are lost. How will things change when our artifacts (e.g., the web) can be perfectly preserved indefinitely?

The Second Digital Revolution

2011-10-14T22:49:00.000-07:00

Image by dullhunk

DNA provides a great distinction between data and medium. While our DNA (the medium) dies with us, our genes (the data) can be passed down to our posterity. In fact, some of our genes may be billions of years old. This immortality of genes stems from the fact that they are digital. Genes happen to be encoded in DNA, but they are just base-4 information.

Up until very recently, human beings only created analog artifacts. A painting lasts only as long as the canvas (or the cave) it was painted on. They can be reproduced, but only imperfectly and at great expense.

Books are one step closer to being digital. The English alphabet is, in a sense, a base-26 code. The problem with books is that it is difficult to replicate them.

With computers, we have created a system like DNA - digital information plus a way to replicate that information quickly and faithfully. The first time that sort of system came into being, the output was all of the life on earth. It will be interesting to see what the results are of this second digital revolution.

We are almost certainly wrong

2011-02-04T16:53:00.000-08:00

As individuals, and as societies, we rightfully dismiss a bunch of crazy theories. However, we can't dismiss them with 100% confidence. We are pretty sure that the moon landing wasn't faked, that UFOs aren't stashed at Roswell, that we are not part of a computer simulation (à la Matrix), etc. Let's say there are 100 of these crackpot theories, and that we are 95% certain that each one is incorrect. That seems pretty good, right? We are 95% sure that all of those theories are incorrect, right?

This is where things get strange. Even though we are 95% sure that each of the crackpot theories is incorrect, we shouldn't be 95% confident that all of the theories are wrong. If there is a 5% chance that each one is correct, then the chance that all 100 theories are incorrect is actually only 0.6%! In other words, at least one of the theories is almost certainly correct, and probably quite a few of them are correct.

So does this mean that you should start believing crackpot theories? No - and that's the craziest part of all. The key is that we don't know which of our beliefs are wrong. It is rational to disbelieve all of the crackpot theories and believe that some of your beliefs are wrong.

Rent-A-Shovel

2010-12-06T22:30:00.000-08:00

The Internet is changing the way the world works in a lot of ways. From work to play to hobbies, the Web is touching almost everything that we do. One of the primary tools that the Internet uses to disrupt things is the reduction of transaction costs.

Transaction costs are basically the costs of buying something that are outside of the actual price. Wikipedia gives the example of buying a banana; transaction costs include figuring out what kinds of bananas you like, traveling to the store, waiting in line, etc. (Note that not all of these costs actually cost money - losing time and energy are also real costs)

We have already seen a hint of how the Internet reduces these costs, and how dramatically that can change industries and behavior. Before the Internet, the idea of having hundreds of thousands of people work together to create an encyclopedia would be beyond ludicrous. Just finding that many people willing to help would be incredibly expensive, and then organizing and reconciling their contributions would be very nearly impossible.

Reading about the creation of the original Oxford English Dictionary shows just how difficult a project like this used to be. For years, editors received over 1,000 slips of paper per day from word-lovers around the world, detailing word usage. They stored the slips of paper in 1,029 pigeon-holes in "the Scriptorium", a large shed built specifically to store and organize the submissions. Seventy years (yes, 70!) after the project began, the full dictionary was published. Multiple editors had given the majority of their lives to the project, and most died before it was completed. And all that for "only" 252,200 entries. In less than 10 years, Wikipedia editors have created and collaborated on over 9,000,000 articles (almost 3.5 million for just the English language Wikipedia).

The transaction costs to recruit volunteers and organize information still exist, but the Internet (along with tools like wikis) has reduced them at least 1000-fold, and probably much more. This reduction in costs has made the previously impossible happen, to our benefit.

There are plenty of other transaction costs that the Internet is starting to chip away at. Email and Twitter and Facebook make it much easier for us to communicate with each other. Amazon and Ebay have made it much easier for us to research products and to buy and sell things. Knowing that I can resell almost anything that I buy changes my attitude toward buying things. I see a lot of things as short-term purchases that I can own for as long as I want, and then sell to someone else. That is a new option.

These trends are only going to continue. There are stirrings of other industries where the Internet and its associated technologies are starting to change things. Peer-to-peer lending like Lending Club allows savers to lend money to borrowers, using the Internet as the aggregator and middleman instead of a bank, thereby offering better returns to investors, and better rates to borrowers.

Other companies like Whipcar and airbnb let you rent out your car or a room in your house, respectively. The Internet may allow us to move from an ownership society to a "rentership" society, where the Internet makes it convenient enough and cheap enough to rent things as we need them, from cars to power tools, and maybe even cameras and snow shovels.

The Widening of Frontiers

2010-11-19T22:51:00.000-08:00

We live in an incredible, wonderful time. I call our time the Age of Wikipedia. Our understanding of the world around us has been exploding, and access to that knowledge exploding even faster. A child with access to the Internet has riches of knowledge that da Vinci and Darwin could have only dreamed of.

We are all beneficiaries of mankind's increased knowledge. Our standard of living is directly impacted by new inventions, new manufacturing processes, and new discoveries. But there has been one casualty of this rapid explosion of information.

The Age of Wikipedia has killed the Renaissance Man. Da Vinci made significant contributions to our understanding of anatomy, astronomy, engineering, optics, and hydrodynamics, and made smaller contributions to many other fields.

Today, nearly all of the incredible breakthroughs that da Vinci made are taught in first-year classes in their respective disciplines. Like a balloon that is continually expanding, the frontiers of our knowledge keep growing. The balloon of human knowledge in da Vinci's time was small enough that he could help to expand it in multiple places.

In order to really study something, to make breakthroughs, you have to be at the edges. These days, we have a much larger balloon; it takes a lot more work to get to the edges. It really takes a PhD before you can hope to contribute to a field.

For an aspiring Renaissance Man like me, this can be depressing. When a subject catches my fancy, I often begin by looking it up on Wikipedia. I am quickly immersed by information, and the realization that the waters are much deeper than I realized.

For example, I have recently been enamored with the idea of simple actors (e.g., ants, businesses) working together to build something more complex than any of the individual actors understand (e.g., nests, economies). In my Internet research, I discovered that this is called emergence, and that there are branches of computer science and philosophy and biology and linguistics and sociology that focus on emergence. There is even an Institute for the Study of Coherence and Emergence.

This idea that I had never considered before has an entire industry built around it. There are people who study it for a living. There are websites and magazines about it. This should be enabling and invigorating, but I generally find it depressing. It feels as though I would have to devote a lifetime in order to learn anything "new", anything that humanity doesn't already know.

It's hard to decide what ideas are worth a lifetime.

What is my Fridge Thinking?

2010-11-13T22:05:00.000-08:00

Consciousness is notoriously difficult to define, because we are defining something that we only experience first-hand.

There is no way to measure whether someone else's experience of consciousness is truly the same as ours.

We assume that other people are conscious, because they act like we do, and because we have language which helps us to describe consciousness to each other.

While there have been some fascinating attempts to design tests for consciousness, they can give only circumstantial evidence of consciousness. A sufficiently advanced automaton could pass any test we could design (think The Matrix).

I think that it is probably safe to assume that there are varying levels of consciousness, from a complete unawareness of your own existence or the world around you up to our own awareness not only of ourselves, but awareness that we are aware of ourselves.

While it is easy to put rocks, dirt, etc. on one end of the spectrum and ourselves at the other end, it is very difficult to discern where other things should be placed.

Most people would agree that monkeys are conscious, in most senses of the word, but how about rats? Snakes? Fish? Ants? Trees? Algae? Bacteria? Viruses? We have an intuition about just "how conscious" each organism is, but that guess is really just based on how different from humans they are. We haven't experienced life as an ant, or life as a bacteria - there is a chance that it is much richer with awareness than we expect.

We are also inherently hesitant to grant the possibility of consciousness outside of the realm of life. However, it seems more and more likely that our creations have some sort of consciousness. The Internet has eyes (cameras), ears (microphones), and a brain of billions of interconnected computers. Is it possible that the Internet is, in some sense or other, aware of the world?

And if the Internet, what about individual computers? What about dishwashers and refrigerators? When we designed refrigerators to keep food cold, did we also give them some sort of desire or drive to get rid of heat? In the video above of the "Big Dog" robot, it is very hard to shake the feeling that the robot "wants" to stay upright.

It is possible that all complex systems with interconnected, communicating nodes (brains, networks, etc.) have some level of consciousness. If that is the case, then isn't it also theoretically possible that our economies are conscious? Our cities? Our corporations?

Environment as Information

2010-10-16T21:36:00.000-07:00

Each cell in your body contains your entire genome - all 46 chromosomes, and yet they do many different things. We have skin cells and nerve cells and glial cells and muscles cells, etc. If the cells all have the same DNA, how do they know what to do? How does a skin cell know that it's supposed to be a skin cell, and at a larger level, how does a finger know how to shape itself into a finger?

Many people believe that DNA is a blueprint for our bodies. That if we could read DNA correctly, we could output something like an architect's drawing, that our cells faithfully follow in order to produce the completed body. This description of DNA is typical of what I learned in school:

Every cell in your body has the same "blueprint" or the same DNA. Like the blueprints of a house tell the builders how to construct a house, the DNA "blueprint" tells the cell how to build the organism. Yet, how can a heart be so different from a brain if all the cells contain the same instructions? Although much work remains in genetics, it has become apparent that a cell has the ability to turn off most genes and only work with the genes necessary to do a job. We also know that a lot of DNA apparently is nonsense and codes for nothing.

While the blueprint analogy is helpful, it misrepresents how things really work. Unlike buildings, there is no architect or general contractor in an embryo. There is no cell or group of cells that understands what the body should look like, or what it looks like currently. Cells base their behavior entirely on their local environment, not on any sort of understanding of their larger role in the body. DNA is not a blueprint - it is more like an instruction booklet, and the environment tells the cell which "page" of the booklet to turn to.

For example, the very first cell (or zygote) is in a very unique environment. There is a "page" in the booklet of DNA that gets turned to when a cell isn't connected to any other cells, and is surrounded by the chemicals found in a womb. That page says that it should divide. The next cell has its DNA turned to the page for what to do when it's connected to one zygote-like cell, etc.

The key is that each cell uses its local, unique environment to decide how to act. Its actions then help to create the local environment for the cells around it, telling them how to act. In this way, information cascades through the medium of the environment, and each cell can do what it's supposed to, without only a local understanding of its environment.

If we go back to our finger example, DNA might have instructions that are something like "keep producing skin and bone cells while the ratio of chemical x is above y", where chemical x is produced by your knuckles, for example. When the ratio of the chemical is below the threshold, your fingers stop growing. The specifics of this example are most likely incorrect, but the principle is right - individual cells know nothing except their local environment, and genetic instructions must be based on that.

Many machines are based on this same principle. For example, a thermostat could theoretically be built which measures the volume of a room, and using mathematics figures out the amount of hot air necessary to heat a room of that size, and then pumps in the air.

That would, of course, be silly. Instead, a thermostat measures the temperature of the room, and continues to pump hot air until the room is warm enough. It directly modifies the environment, and then measures the environment to determine whether or not to continue modifications. Information about room size is not necessary, information about how much air has already been pumped is not necessary, even information about the temperature in the room isn't necessary. The only information a thermostat needs is whether or not the room is warmer or cooler than a given temperature. By directly querying the environment, thermostats move from a incredibly complex calculation to a binary decision without any loss of function.

These two examples hint at the incredible ways that environments can be used to interact with systems. They can:

Simplify systems - in both examples, systems are simplified drastically by directly affecting the environment in order to communicate with the system. Instead of cells storing their location in the body, or which part of the DNA they should code for, their instructions simply tell them how to react in various environments. That is a much simpler, and less error-prone way of doing things.
Store information - In a sense, the temperature of a room is storing how much hot air has been pumped into it from a heater. This is information that would be important to a self-contained thermostat system, but it isn't needed because the environment contains the necessary information.
Build complexity - Because all of the components in the system are using their local environment as both input and output, they can build things that are more complex than they are. Cells, economies, and societies all rely on local actors making decisions based on local information, where outputs from one actor become inputs for another. This type of interconnectedness allows for complexity and efficiency that is far beyond what any of the actors could have designed and built from the top-down.
Increase adaptability - Each component in the system is interdependent. Because one component's actions are based on the output of another component, if the output changes, the effects are cascading. The breadth of the cascade is also a function of the interdependence. Some changes will only affect the local environment, while others can change the entire system.
Allow stateless interactions - Thermostats don't need to know anything about how long the heater has been on, or how long it has been since it last turned off. They are stateless because their decision about whether or not to turn on the heat is not based on the past. Each decision is completely independent. Similarly, a cell doesn't know whether it has divided 1 time or 1,000 times. A neuron doesn't know whether it has fired 5 times in the last minute, or 50 times.
Allow certain variables to be abstracted - A thermostat that tried to calculate how much hot air was needed would need to keep track of whether a door to the room was opened (thus increasing the amount of hot air needed), or whether there were warm bodies in the room (thus reducing the amount of hot air needed). By measuring only the temperature, a simple thermostat acts as though it took all of those variables into consideration, without actually knowing them.

Out of Control

2010-10-14T18:57:00.000-07:00

Out of Control by Kevin Kelly is one of the most incredible books that I have ever read.

It is a rambling tome about emergence and technology and complexity and evolution, and it is wonderful. Kelly is the excited narrator who leads the reader into a fantastic new world where creativity and order and intelligence are a result of networked simple parts interacting and learning.

It really is too ambitious to describe in a blog post. Usually books written about current and future technology feel dated and anachronistic within a few years. Out of Control was written in 1994, but the trends that Kelly identified and explored are so deep and long-reaching that the book won't outlive its relevance for a long time.

This book will affect my thinking in many areas and in many ways. I highly recommend it. Just read it.

P.S. Kevin Kelly's new book, What Technology Wants, came out today. He described it as an extension of Out of Control. I can't wait.

You Are Really, Really Rich

2010-10-12T20:30:00.000-07:00

A politician who suggested a healthcare plan that would apply only to the richest 10% of people would be laughed out of office on election day. It would be absurd to look after only those who already have everything. A welfare plan which took money from the rich and gave all of it to the upper-middle class, without giving anything to the poor, would be ridiculous. And yet, that is exactly the system that we have.

Every year, we give the Earned Income Credit to millions of poor Americans. In 2004, $36 billion were transferred from the wealthy to the poor through this system. The maximum credit is earned by those who make $16,420 or less. Therefore, the tax system considers these people poor. Politicians consider these people poor enough that it was worth proposing a tax on richer Americans which would then be used to give money to this group; a proposal which was accepted.

When we take a step back and look at the American "poor" from a global perspective, our priorities seem skewed, to say the least. According to the Global Rich List, someone making $16,420/year is in the top 12% of income earners in the world. Someone at the US poverty line ($10,830) is still in the top 13%. We are taking money from the top 5%, and because we feel so sorry for them, we are giving it to people who are still in the top 15%, and giving absolutely nothing to everyone else.

Sometimes when I am listening to stories on NPR about people who are losing their homes to foreclosure, or who go bankrupt due to medical bills, I start to get a little choked up. I feel sorry for those people. I want to do something to help them.

Then I remember that they aren't the people who really need my help. The difficult experiences that we have in America stink. It's terrible to lose your house. It's terrible to be forced to sell your heirlooms to make ends meet.

It's worse to have to choose which child gets to sleep under the mosquito netting, because you can't afford a $10 bed net for each of them. It's worse to wonder every day whether the water you are drinking will kill you. It's worse to not have enough money to send your children to school.

We have eradicated poverty in America. That is an incredible miracle that we have never really celebrated. People no longer starve to death here. They no longer die of mumps or measles or diarrhea or malaria. We should celebrate that incredible accomplishment. But while we are celebrating, we need to remember that we have brothers and sisters out there who still are starving, and who are still dying from diarrhea and malaria. It's time to move our focus from America to the rest of the world.

The great news is that because you are so (relatively) rich, it is easy to do a lot of good. By cancelling cable, or skipping your morning coffee, you can make a real and meaningful difference to many people who need help.

There are plenty of great ways to help out. I like Kiva and Room to Read, but there are many other ways to do something. But you should do something.

What Happened to Spam?

2010-09-16T19:41:00.000-07:00

Spam used to be a terrible problem. I remember going through my inbox, and deleting 80%+ of my messages. That was just part of checking your email. There was hand-wringing about the death of email, and public outcry, resulting in the ineffectual CAN-SPAM Act.

Today, it is rare if more than one spam email per week makes it through to my inbox. In addition, I can't remember the last time that an email that I wanted was inadvertently sent to my spam folder (a common problem with some early filtering systems).

So, what happened? Did spammers thoughtfully consider their behavior, and decide to change their ways? Did governments crack down on them, scaring them into better behavior? Nope. In reality, the amount of spam sent has continued to increase. It is our filters that have gotten better.

For a long time, the battle between the spammers and the spam filters was fairly equal. Programmers would come up with new ways to thwart spammers, and the spammers would figure out a trick to get around those tools. For example, filter software started to look for terms like "herbal Viagra", and would make any email with that term as spam. Spammers would use an image for the word "Viagra", or would spell it using a similar Unicode character.

While many of these techniques are still valuable, the real breakthrough came when spam filters started using Bayesian classification to identify spam. Where other techniques rely on clever programmers figuring out new tricks, Bayesian filters require no human intervention.

Basically, a Bayesian classifier is fed in a whole bunch of email messages, together with information about whether or not each message is spam or not. It creates its own rules about how to classify messages, and then uses these to determine whether or not incoming emails are spam.

This seems like a crazy way to do things - it seems like a set of rules would be much more effective, but Bayesian filters work better for a few reasons:

1. The rules that they create can be incredibly subtle and would never be noticed by humans. For example, maybe there is a rule that emails from a certain country that have capitalized words in the header are usually spam. A human would never be able to discern that pattern.

2. Their rules are almost impossible to reverse-engineer. Because they are so subtle and complex, spammers cannot figure out why their messages are blocked.

3. It can be user-specific. A personal classifier can be layered on top of a general filter, so that messages that contain your spouse's name, for example, are almost never marked as spam.

4. It can learn. Bayesian filters become better as users give feedback - moving messages to or from the spam folder. In addition, they automatically adjust to any new techniques that spammers use. Spam that gets through is quickly marked as spam, and the filters will learn how to identify it.

The fact that Bayesian filters are our best solution to spam is incredible, and a little unnerving. We have taught our computers to be smarter than us. The best programmer cannot write a program that filters email as effectively as a Bayesian filter. It is one thing to compare a computer's processing speed - multiplying huge numbers or solving the 1,000,000th digit of pi. But it is another thing to realize that computers can now create better solutions to some problems than we can.

An Opposition to Net Neutrality

2010-08-14T16:54:00.000-07:00

Net Neutrality has been in the news lately. Basically, net neutrality supporters want the FCC to make it illegal for internet providers to privilege certain types of Internet traffic over other types. Basically, Google can't pay Comcast to make sure that YouTube videos come through in higher quality than other videos on the web.

On paper, this sounds great. The web, as net neutrality opponents are wont to say, was built on equality, and we shouldn't introduce inequalities. I actually agree that a neutral web is, all things being equal, better.

However, there are some problems with enforcing this equality. I'm going to gloss over the issue that I find most salient, because I also think it is the most obvious. Namely, the question of whether or not it is the government's place to regulate how ISPs operate. They own the networks that the Internet runs on - they paid to lay the fiber, and I think that they have an inherent right to operate it how they choose.

That issue aside, I think that the potentially bigger issue is less obvious. Government regulation so often fails because it focuses on improving the present. Currently, there are not many companies that own much of the Internet infrastructure, and it makes good sense to ensure that they don't do things which are bad for the Internet ecosystem - an ecosystem which has become more and more important to all aspects of our society. Regulation which stops them from ruining the Internet is regulation which seems to help everyone.

The only people that it doesn't help are our future selves. By enforcing net neutrality, we stop Internet providers from doing innovative things, and thereby make laying more Internet cables less lucrative. If Google was willing to pay billions of dollars to prioritize their traffic, you can be sure that ISPs would be laying cable all over the country to make sure that they were able to meet that contract. And that cable would help everyone - not just Google. Sure, maybe Google would get the lion's share of benefit, but would be "trickle-down bandwidth" for everyone.

By letting people decide with their wallets what their real priorities are, you allow companies to do innovative things, and improve the experience for everyone. Technology is magically improving our lives all the time. If we regulate to ensure the status quo, then we will get the status quo. We will miss out on the future that could have been, and the future is (almost) always better.

The Power of Ignorance

2010-04-04T19:45:00.000-07:00

200 years ago, the majority of Americans were rural farmers, living a very different life than the average American today. They grew their own food, made their own clothes, and cooked their own meals.

This self-reliance extended ever further. Every single thing that our rural ancestors owned was either built by them, or by someone nearby. A community of a few dozen people could create every single item owned by every single member of the community. From horseshoes to plows to dresses to seeds, all of the requisite knowledge, resources, and skills were provided by the members of the community.

We live in a very different world. In looking around my modest apartment, I found it very difficult to find items that I or anyone that I know could produce. Even common items, like plastic bags, shoes, and cloth I have no idea how to make. I was washing my hands, and realized that I have no idea how the soap dispenser works.

The urbanization of the world has allowed us to increase our knowledge specialization, and decrease our knowledge density, which I will define as the proportion of the population which need to know how to make or do a certain thing. Living closer together meant that it became more efficient for a baker to make your bread than for you to make it yourself. Because bakers could spend so much of their time making bread, they soon learned more about bread-making than anyone knew before. The Internet continues to decrease our knowledge density, as knowledge can be stored and shared across the world, and across time.

This specialization of knowledge has led to a situation that is unique in human history - products which no individual understands completely. Computer software is a prime example. While there are many people who understand the basic concepts behind how a computer works, there are so many pieces layered on top of and integrated with one another, that understanding them all is impossible. Some people understand how to write a program, others understand how to compile it, others understand how the compiled program interacts with the operating system, others understand how the operating system interacts with the hardware, but I think that it would be fair to say that no one in the world has the knowledge to recreate the entire system. No one knows how a computer works.

At first, this concept is scary. While our collective knowledge has increased many times over, our individual knowledge of basic self-sufficiency has dropped dramatically. We do not know how to create even the things that we use every day. Stranded in the wilderness, we not only could not recreate the tools of today, but we couldn't even create the tools of our ancestors. We would be in much worse shape than the farmer in our introduction.

However, I think that is a price that is worth paying. By specializing our knowledge, we have been able to learn and know and do things that our ancestors couldn't even dream of. We no longer need to know how to do everything for ourselves, and that excess of time, energy, and mental capacity has allowed us to expand the borders of our societal knowledge and ability orders of magnitude faster than at any point in human history. At the same time, we have become much better at providing for our basic needs.

All of these miracles have occurred because we have embraced ignorance, and been willing not to understand.

The Aligning of Opinions

2010-03-13T10:56:00.001-08:00

It is fascinating to me how different electoral systems lead to very different alignments of the populace. America's two-party system, and people's attachment to one party or the other, are almost certainly a result of our First Past the Post electoral system.

What I want to focus on today, however, is not our system of voting, but the actual alignment of the political parties. It seems to me that our two-party system creates an environment where it is almost a certainty that each party will take opposite stands on any controversial issue. This creates some fascinating results. On many issues it seems like the parties betray their fundamental principles simply in order to oppose the other party.

For example, I am really intrigued by the anti-immigration stance of the Republican Party. It seems that a party whose ideals include reduction in government powers, equality of opportunity, and Christianity would be vigorously pro-immigration, and happy to help extend the blessings of America to as many people as possible.

Am I missing something? Why do you think that anti-immigration has become associated with the Republican party?

The Value of an "American Life"

2009-09-19T14:14:00.000-07:00

We have a natural, and just, tendency to value the lives of our countrymen over the lives of others. Citizenship contains an implicit contract to protect and defend one another.

However, we have another implicit contract with the other members of the human race. All religions, all models of morality, contain injunctions to care for one another, regardless of ethnicity. When Jesus Christ taught about what it meant to be moral, he used the example of the Good Samaritan - teaching that compassion and love should cross country borders. We clearly have a greater responsibility to other Americans, but we have some moral responsibility toward everyone.

The recent health care debate in American has left me wondering whether we have skewed this balance too far in favor of our fellow Americans. We certainly have some problems with our health care system. It is inefficient, and unfair. We should fix it. Political blogs and shows are replete with comparisons to the European and Canadian health care systems, and their comparatively better standard of care. But why are we comparing ourselves to these other OECD countries, instead of being compared to all countries in the world?

It could be argued that we should be compared only to this small subset of countries because we are similar to them, and we can learn from their successes. I believe that the other reason is because we don't want to deal with the harsh truth about the rest of the world. We are arguing about whether or not it's worth expanding government coverage to all US citizens, at a cost of $129,000 per year of extended life (that's the amount that Medicare has implicitly decided on). In the rest of the world, people are dying of diarrhea and tuberculosis. The cost per year of extended life for these diseases is $20-$200. We have decided that an American life is 1,000 times more valuable to us than an African life - that we would let 1,000 Africans die instead of 1 American.

This is wrong. I believe that future generations will be aghast at depth of our nationalism. We have the opportunity and the means to save millions of lives in the developing world, and we will be judged for not acting.

Image by goulao on Flickr.

The Coevolution of Man

2009-06-26T20:06:00.000-07:00

Kevin Kelly often writes about how people and tools have evolved together, and how we can no longer truly be distinguished from our technology, and we certainly couldn't survive without our technology.

I definitely agree with that point, and I would argue that some of our technological advances have been so dramatic that they have, in a sense, almost changed us into a different species. The changes wrought by these new technologies are so dramatic that I would argue that people living before and people living after the advent of these technologies have fundamentally different kinds of lives. Technologies like this are obviously very rare, but here are a few:

Language: Language almost certainly progressed very slowly, but language is now almost the sole means by which we think, communicate, etc. When we think about Helen Keller, we feel so desperately sorry for her, not because she was blind or deaf or mute - plenty of people have those afflictions without arousing such deep pity - but because those afflictions shut her out from the world of language. It is when she finally understands language that her life becomes heroic and understandable - before that she is only pitiable. Thinking of what it meant to be a human before language seems impossible.

Writing: Before the advent of writing, all knowledge was local knowledge, both in space and in time. You could never know more than the sum of what your neighbors knew. Ideas and inventions died with their creators. Writing now touches nearly every aspect of our lives. Like language, it seems impossible for any civilization to exist without writing.

Industrial Revolution/Urbanization/Globalization: Before the Industrial Revolution, most people were farmers, living in self-contained communities. Unless you were royalty, you probably made your own shirts, grew your own food, and neither you nor anything that you owned had ever travelled more than a dozen miles. Because of the resources needed to provide everything for yourself, most people only had a few changes of clothes, a very simple diet, and small, sparse homes.

Now, we think nothing of buying shirts from Thailand, TVs from China, bananas from Mexico, and shoes from Singapore - in fact, we could probably buy all of those things at the same store, within a few minutes of our homes. We no longer make our own clothes or grow our own food. Chances are that you are not a farmer - that you don't even know anyone who is a farmer. The lives that we lead now would be wholly foreign to a pre-Industrial farmer.

The Internet: Although it isn't there yet, I believe that the Internet will change the world as much as any of the other technological advances. Just like language and writing enabled not only more conversation, but a different type of conversation, the Internet is not just like a new version of TV or newspapers or books, but it is something new and different. It is changing the way that we do everything - from buying and selling things to finding new friends to learning to working - the Internet democratizes everything. Instead of learning about the Iran elections by watching the news, we can talk with regular people in Iran. Instead of just listening to music by a band that we like, we can write suggestions to them on their MySpace page. Instead of finding a job in a big company, we can be employed by the Internet - as a blogger or an editor or an eBay buyer and seller.

Like many people, the Internet is an integral part of my life, and the things that I do would be dramatically different if it didn't exist. It is truly amazing to me that 10 years ago just over 50% of Americans had ever used the Internet. My school experience seems anachronistic, in that I actually had to research things in a library, using card catalogs to find books. In 7th or 8th grade I spent hours and hours trying to find basic information for a report on Indonesia. I looked through book after book to find and compile a small percentage of the information that is now on the Indonesia Wikipedia page. We are gathering the world's knowledge together, and we are figuring out how to use it. As computers get smarter, and as that data continue to pour in, the Internet will subsume more and more of our lives.

Each of our past advances has enriched the human experience. I am sure that there will be some bumps on the road, but I think that we can be confident in predicting that the Internet will again change for the better what it means to be human.

Collective Decision Making

2009-05-19T19:21:00.000-07:00

In 1984, the U.S. Government started studying 10 sites in the United States to find a location that would act as a repository for nuclear waste from U.S. nuclear reactors. By 1987, Congress had narrowed the list to one location: Yucca Mountain, Nevada.

By almost any measure, Yucca Mountain appears to be the ideal candidate. 80 miles north of Las Vegas, it is a barren desert, without any cities are towns - practically without any living creatures at all. The site was used to test a number of nuclear weapons, and it is owned by the government. We have spent $9 billion studying Yucca Mountain, and haven't found any compelling reasons why it shouldn't be used for nuclear waste disposal.

So, some 22 years after Yucca Mountain was selected, why is our nuclear waste still being stored in much more dangerous temporary locations? Why did Energy Secretary Steven Chu say "the Yucca Mountain site no longer was viewed as an option for storing reactor waste"?

I would argue that Yucca Mountain hasn't happened because Nevada voters didn't want it to happen, and because Nevada voters have power incommensurate with their numbers. In 1980, '84, and '88, Republican presidential candidates won in Nevada by 29 percentage points or more. As the Yucca Mountain issue became more prominent, Nevada voting patterns changed dramatically. Ever since 1988, Nevadans have consistently voted for the winner of the election, who has won by less than 4 percentage points (http://www.fivethirtyeight.com/2008/04/electoral-history-charts.html).

While it could certainly be argued that demographics accounted for much of the change in Nevada politics (a growing Californian population, for example), I would argue that the change could potentially be attributed to collective, subconscious decision making.

Just like our brains "decide" on our breathing, heart rate, etc. without our conscious input, I think that it is possible that our societal "brains" make decisions about our society without any individual directing, or even being cognizant of, the decisions being made. I would argue that there are many decisions that are made at a societal level, with inputs and responses that are not fully understood by the individuals involved.

Why Health Care Costs Will Not Go Down

2009-03-07T12:26:00.000-08:00

We see newspaper headlines almost daily bemoaning the increasing cost of health care. On Thursday, President Obama spoke at the "White House Forum on Health Care Reform", repeating the scary statistics that we hear so often:

You know, we're here to discuss one of the greatest threats, not just to the well-being of our families and the prosperity of our businesses, but to the very foundation of our economy. And that's the exploding cost of health care in America today.

In the last eight years, premiums have grown four times faster than wages. An additional 9 million Americans have joined the ranks of the uninsured. The cost of health care now causes a bankruptcy in America every 30 seconds. By the end of the year, it could cause 1.5 million Americans to lose their homes.(http://www.washingtonpost.com/wp-dyn/content/article/2009/03/05/AR2009030501850.html)

The fact is, he is right. Health care is taking up a larger and larger portion of the average person's budget. Health care costs are increasing. The typical reasons given for high health care costs fall into one of three main camps:

1. People are greedy - this argument says that some group of people - typically insurance companies, doctors, or malpractice lawyers - are too greedy. Costs are increasing because they are taking a bigger and bigger slice of the pie.
2. Government is not involved enough - this argument is that socialized medicine allows the government to set prices and quality of care.
3. Government is too involved - this argument says that if we just let the free market work, then health care costs would come down. The huge influence of Medicare and Medicaid is actually raising prices.

I think that some, or maybe even all, of these camps could be right. I am sure that costs could be lower if we made some changes to the structure of how health care is delivered. However, I believe that there is a much more fundamental reason that health care costs are increasing.

The reason that health care costs are increasing is because health care in the U.S. is so good. Health is the classic inelastic good. People will spend whatever money they have if spending that money will save their life. As the better and better health care saves more and more lives, people live longer, and require more health care.

Since this is the main point of the whole post, I think that it's worth illustrating with an example. In hypothetical country Moribund everyone has a heart attack every 20 years. In Moribund there are two cities: in city A, there are 500 citizens, and a mediocre doctor, whose rate of successful heart operations is 50%. In city B, there are also 500 citizens, but they have a great doctor, whose rate of successful operations is 90%. In both cities an operation costs the society $100 (either the individual or the government - it doesn't really matter).

After 20 years in city A, there will have been 500 heart operations, costing $50,000, but only 250 people will have survived.
In city B, the cost will also have been $50,000, but 450 people will have survived.

After 40 years, city A will have only 125 citizens, and will have spent $75,000 on health care.
In city B, they will have 405 people, and will have spent $95,000 on health care.

Obviously, this little example is not true to the real world, but you get the point. The better health care is, the longer we will live. The longer we live, the more often we will require health care. And because health care is so inelastic, we will pay a greater and greater portion of our income for it.

Image by salimfadhley

The Curse of Poverty

2008-10-15T19:03:00.000-07:00

I have become more and more convinced that I, that we, should be doing more. Despite the recent problems in the economy, despite the talk of a global recession, we have so much.

Like many people, I lost a huge chunk of my net worth over the last few months, having been heavily invested in the stock market. While it has been depressing to watch the news and watch my IRA balance, I have been otherwise unaffected. I still have an apartment, I still have plenty to eat, I still have a TV and a Wii and a computer and access to the Internet. In fact, if I lost all of my savings, all of my possessions, and only had my last paycheck, I would still have more money than 80% of the people in the world make in a year. In fact, even those at the poverty line ($10,400 for one person) in America are in the top 13% of earners in the world!

How To Help

I am a huge fan of Kiva as a way for me to give some of my (relatively) untold wealth to those who have not had the great fortune to be born in a Western country. Please feel free to join the Kiva Mormons team that I founded.

Another great way to help is through fasting. Members of my church forgo two meals for one day each month, and donate the money to help the needy.

There are many, many other ways to help. Find something you are passionate about, and get involved. Just do something.

Our humanity is defined by the way we treat The Other - those without a voice, without opportunity, and without the ability to reciprocate. Christ spent his time with the publicans, the Samaritans, and fisherman. Should we not do more for the Forgotten in our society?

Let Their People Come Review Part 1 - The Morality of Immigration

2008-09-10T23:24:00.001-07:00

Photo from Flickr - celikins

I just finished Lant Pritchett's fantastic Let Their People Come, and I feel compelled to put down a few of the thoughts that I had. I hope this will kick-start my writing on this blog, but that remains to be seen.

I wrote about immigration once before, and reading Mr. Pritchett's book reinforced some of my ideas, and gave me some new ideas.

One of the most powerful arguments, in my mind, is his argument that we treat nationality as a morally legitimate basis for discrimination. He compares labor mobility restrictions to apartheid - "The analogy between apartheid and restrictions on labor mobility is almost exact. People are not allowed to live and work where they please. Rather, some are only allowed to live in places where earning opportunities are scarce . . . The restrictions about who can work where are based on conditions of birth, not on any notion of individual effort or merit"(79).

This is one facet of what I call the "There but for the grace of God" argument, and I think that it is spot on. Any attitude or policy that treats people differently because of the "conditions of birth" is inherently wrong, and the location of birth is no different than race, gender, or ethnicity. He argues that we have as strong a moral obligation to the "Outer Mongolian" as we do to our own countrymen.

While there are serious policy difficulties in determining how to achieve a society without "nation prejudice", Pritchett reminds us that this should be our goal, and we should couch our policy decisions in that framework.

Great video on Ants

2008-06-18T17:07:00.000-07:00

I just saw this great video about ants from a TED talk:

Her focus is on exactly what fascinates me about ants. She talks about how they coordinate a complex organization using very simple tools, and even more importantly - simple brains. No ant understands the organization, or what needs to occur in order to succeed, and yet the coordinate and "make decisions" about resource allocation, and other things.

I think it can be compared to the cells in a body - each individual cell has very, very limited inputs and outputs, and yet they can work together to create movement, sensory perception, and even consciousness, and even more importantly, can link those complex systems together. Truly amazing.

Use Firefox 3!

2008-06-17T11:55:00.000-07:00

Firefox 3 was just released today. It is a fantastic piece of software, with some significant improvements over the already-great Firefox 2.

It is amazingly quick, has some really intuitive features like tagging, dragging text, searching page histories, etc.

Firefox can be download at http://www.firefox.com. You won't regret it.