WorldviewNaturalism.com has finally launched! It’s a small, simple site — an ideal place to send your friends when you want to introduce them to naturalism.

This is probably the last post on CommonSenseAtheism.com, which is now merely an archive of posts.

Common Sense Atheism has closed its doors. Comments are turned off and there will be no new posts.

I will keep the debates page updated, so feel free to notify me of new debates.

The site will remain online as an archive. See the Contents page for a quick view of the site’s main attractions.

You can keep up with my work on a variety of websites at lukeprog.com, which has an RSS feed that will alert you to my new works when they are published. If nothing else, you’ll want to subscribe to that feed so you are notified when WorldviewNaturalism.com goes live. You can also follow my Twitter page.

Common Sense Atheism is closing its doors.

It’s been a great ride, and my interests have now turned elsewhere.

I’ll keep comments open for about a week, and then comments on the site will be closed, but this site will remain online as an archive. I also plan to keep the debates page updated.

Updated February 13, 2012.

Allow me to indulge in some anticipation…

My most eagerly anticipated books

• Frankish & Ramsey (eds.), Cambridge Handbook of Artificial Intelligence (2012)
• Frankish & Ramsey (eds.), Cambridge Handbook of Cognitive Science (2012)
• Holyoak & Morrison, Oxford Handbook of Thinking and Reasoning (2012)
• Thagard, The Cognitive Science of Science: Explanation, Discovery, and Conceptual Change (2012)
• Bostrom, Superintelligence: Groundwork to a Strategic Analysis of Machine Intelligence Revolution (2012)
• Eden et al. (eds.), The Singularity Hypothesis: A Scientific and Philosophical Assessment (2012)
• Yudkowsky, How to Actually Change Your Mind and The Art of Rationality (2012?)
• Drexler, Radical Abundance (2012)
• Greene, The Moral Brain and How to Use It (2012)
• Panskepp & Biven, The Archaeology of Mind (2012)

My most eagerly anticipated films

• Anderson, The Master (2012)
• Blomkamp, Elysium (2012)
• Kaufman, Frank or Francis (2012)
• Nolan, The Dark Knight Rises (2012)
• Cuarón, Gravity (2012)
• Spielberg, Robopocalypse (2013)

What are you most eagerly anticipating?

New at Less Wrong: Can the Chain Still Hold You?

New at Facing the Singularity: Value is Complex and Fragile.

Also new links at my Twitter page.

New at lukeprog.com: Review Articles.

New at Less Wrong: What Curiosity Looks Like.

From Twitter:

• Avoidable errors in Freakonomics.
• Behaviorism at 100.
• Indian girl sacrificed to the gods for a better harvest.
• Double contractions in English.

At Facing the Singularity: Don’t Flinch Away.

New at lukeprog.com: Nonfiction Book Choices for a General Audience.

Alonzo Fyfe: A Basic Review of Desirism, Desirism and Neurobiology.

Luke Barnes summarizes much of the fine-tuning literature, and in particular critiques Victor Stenger’s new book The Fallacy of Fine-Tuning.

From Twitter:

• Photography in the style of traditional Chinese paintings.
• Amazing tightrope acrobatics.
• Berkeley’s Jack Gallant explains (on video) his recent vision reconstruction neuroscience breakthrough.
• A day in the life of a modern woman.
• Life in slow motion.
• Must-see science videos of 2011.
• Fred Astaire was more awesome than I realized.
• Best fails of 2011.
• Cat plays Fruit Ninja on the iPad pretty well.

Since I’m in the mood…

1. Ys by Joanna Newsom, 2006
2. 6 by Supersilent, 2003
3. La Pasión según San Marcos by Osvaldo Golijov, 2001
4. Disappeared by Spring Heel Jack, 2000
5. Whatever You Love You Are by The Dirty Three, 2000
6. Evangelista by Carla Bozulich, 2006
7. The Little Match Girl Passion by David Lang, 2007

(I’m sure I’m missing quite a lot. I’ve consumed far more music than film, but musical opinion is less trustworthy and less well-aggregated than film opinion, especially in jazz and the avantgarde.)

Because I’m in the mood…

1. Memento, Christopher Nolan, 2000
2. Mulholland Dr., David Lynch, 2001
3. Eternal Sunshine of the Spotless Mind, Michel Gondry, 2004
4. Enter the Void, Gaspar Noé, 2009
5. Synecdoche, New York, Charlie Kaufman, 2008
6. 21 Grams, Alejandro González Iñárritu, 2003
7. Dogville, Lars von Trier, 2003
8. Primer, Shane Carruth, 2004
9. Requiem for a Dream, Darren Aronofsky, 2000
10. The Tree of Life, Terrence Malick, 2011
11. Adaptation., Spike Jonze, 2002
12. There Will Be Blood, Paul Thomas Anderson, 2007

New at Facing the Singularity: Superstition in Retreat and Plenty of Room Above Us.

New at lukeprog.com: 12 of my favorite essays.

From Twitter:

• The dolphin jetpack that lets you swim like one.
• Minnesota high school’s “incest prank” is… yeah, I’ll let you fill in the blank.
• Fantastic montage from 250 of this year’s films.
• Jumping off cliffs looks like fun.
• Humorous: A polite exchange between author and editor.
• What Patrick Lin told the CIA about robot ethics.
• Watch it and try not to laugh.
• Wikipedia: List of animals with fraudulent diplomas.
• 7 causes of disagreement.
• What? Video interview with world champion masturbator and his girlfriend.
• Free cabin porn!
• Mr. Plinkett reviews Revenge of the Sith.
• 25 funniest autocorrects of the year. You’re welcome.
• NYT predicts AI in 2048.
• You can do real scientific work — “citizen science” — online.

I’ve finally come to admit that I probably won’t continue to record the Morality in the Real World podcast, which intended to explain the moral theory of desirism. People ask me if I still “believe in” desirism, so let me explain my current thinking. First, a few reminders:

1. Desirism never posited anything more than the standard, reductionistic, scientific picture of the world.
2. Given that most uses of moral terms refer to things that don’t exist (categorical imperatives, divine commands, etc.), my MitRW co-host Alonzo Fyfe several years ago proposed a set of “reforming definitions” for moral terms intended to (1) capture something similar to what most people had meant when using moral terms, but (2) capture a set of processes that actually exist. This is standard practice in moral philosophy: see Rawls, Brandt, Railton, etc.
3. Most moral theories treat acts as the primary objects of moral evaluation, but Alonzo’s reforming definitions made motives (“desires”) the primary object of moral evaluation, ala Adams (1976).
4. Alonzo’s reforming definitions construed (non-moral) “value” as a relation between desires and states of affairs, such that a state of affairs has value just in case it is desired.
5. The existence of a desire is a state of affairs, and according to desirism desires are the primary objects of moral evaluation. A desire is”morally” good, on the desirist view, if it tends to fulfill other desires. This phrase “tends to fulfill” needs quite a bit of fleshing out, which is what we started to do in our podcast. An important point is that this claim does not require that desire fulfillment have any “intrinsic” value: see A Harmony of Desires.
6. Whether you want to call this a theory of “moral realism” or “anti-realism” depends on your attitude toward the meaning of those terms: see Pluralistic Moral Reductionism and Joyce (2011).

Why did I stop talking about morality in the language of desirism?

I once wrote a two-part post called ‘The Greatest Objection to Desirism’. I said that the most common objections to desirism were no good, and that the greatest objection to desirism I knew of was one that nobody (except Alonzo) ever mentioned: the possibility that desires do not exist.

“Desire” in desirism was always a metaphor for “whatever a completed neuroscience tells us about the thing that is sort of like the thing we currently call ‘desire’,” and my studies in the neuroscience of human motivation and agent theory in AI have encouraged my view that something close enough to “desire” exists to support a notion of “value,” while in another sense human motivation in particular works quite differently than the folk theory of desire claims. Overall, I’ve shifted away from finding it useful to talk about human “desires” when I’m not talking casually.

But the larger reason I’ve stopped talking about morality in the language of desirism is that I’m tempted to not use moral terms at all. Moral language is thoroughly confused and corrupted and strongly motivated, and I’m more tempted than ever to abandon the entire language and start with a new one.

Another, more personal reason I’ve stopped talking about desirism is that while desirism remains (in my opinion) one of the best sets of definitions for moral terms for talking about ordinary human interactions — saying things like “Bob stole the truck and that was wrong” — I’ve shifted my focus to what I think is a much larger problem looming over us: intelligence explosion. And I don’t find desirism to be the most useful language when talking about the value problem in the context of intelligence explosion.

So:

Do I think desirism’s factual claims are true? Yes, more or less, though neuroscience continues to refine our notion of what the naive term “desire” might refer to.

Do I think desirism’s proposed reforming definitions are useful? Yes, more useful than every other set of proposed reforming definitions I’ve come across for use in discussing ordinary human moral judgments.

Will I continue to use desirist language on a regular basis? Probably not, because (1) moral language itself is not that appealing to me anymore in serious discussion, and (2) desirism is not the most useful language for discussing the value problems I’m focused on, concerning intelligence explosion.

(Listen to other episodes of Conversations from the Pale Blue Dot here.)

Today I interview philosopher Toby Ord. We discuss the problem of moral uncertainty: What do you do if you aren’t sure which moral theory is correct?

Download CPBD episode 090 with Toby Ord. Total time is 41:33.

Toby Ord links:

• Toby Ord
• Giving What We Can

Links for things we discussed:

• Consequentialism, deontological theory, virtue ethics, utilitarianism
• Moral uncertainty
• Existential risk
• Nick Beckstead interview

Recorded January 2011. The current member count for Giving What We Can is now much larger: 177.

Note: in addition to the regular blog feed, there is also a podcast-only feed. You can also subscribe on iTunes.

Transcript:

LUKE: Dr. Toby Ord is a Postdoctoral Fellow at the University of Oxford in the Philosophy Department and the founder of Giving What We Can, a charitable organization focused on giving to the most cost-effective charities in the world. Toby, welcome to the show.

TOBY: Thanks for having me on.

LUKE: Toby, much of your work concerns consequentialism. I’d like to start by asking you to explain for us what consequentialism is and what some of the major arguments against it have been.

TOBY: Okay. Well, there are three main types of ethical theory that philosophers discuss. One of these is consequentialism, which is a theory that’s fundamentally interested in making the world as good a place as possible and thinks that that’s what ethics is fundamentally about. Another type of theory is one called deontology, which says that ethics is fundamentally about obeying moral rules. Don’t lie, don’t steal, don’t kill, things like that.

Another type of ethical theory is called virtue ethics and says that ethics is fundamentally about being a person of good character — being noble, brave, and generous, having those virtues. These are three broad traditions.

A little bit more about consequentialism itself. The idea is that ethics is about making the world as good a place as possible. That’s a bit vague. You can flesh that out in different ways. There are heaps of different consequentialist theories.

The most famous of these is called utilitarianism, which is the theory that what we should do is act so as to make the world as happy a place as possible, to maximize the sum of happiness among all entities that can be happy. What you could do is you could have other types of consequentialist theory, although utilitarianism is the most famous.

For example, you could say, “Well, we don’t want to maximize the sum. We want to maximize the average.” Or you could say, “We want to maximize a weighted sum in some manner.” Or you could say that, “For each individual, what matters isn’t merely happiness, but it’s something else.” Maybe it’s a preference satisfaction, or maybe it’s some list of features that make a life good.

You could have heaps of modifications like that. You could even say that, “When an act of injustice is performed, that makes the world worse,” so you take that into account. You could have heaps of different theories like this, although utilitarianism is certainly the most important one by consequentialists.

That’s what consequentialism is, and it’s contrasted with some of these other aspects. Some arguments against consequentialism are really just arguments for the other types of theory. For example, people might just think, “Well, consequentialism isn’t very plausible, because ethics is fundamentally about hard and fast rules that you cannot ever break.” Or that, “Ethics is fundamentally about being a person of good character.” They’re positive arguments for the other approaches.

Whereas, some arguments against it are explicit attacks. A good example of that is that they say, “On consequentialist theories you’re allowed to do anything, no matter how abhorrent seeming if it leads to good enough consequences.” They’re attacking it because it lets you do bad things in order to get good outcomes. That’s one form of attack.

The other big form of attack is that it doesn’t allow you to be partial to your friends or your family or to yourself. Instead, it’s generally conceived of as a class of impartial theories where… there’s a large demand on you. If it turns out that you can help other people out more cheaply than you can help yourself, then maybe we have to give a lot of our money away in order to do this, or a lot of our time or something else. These are a couple of ways in which people attack it.

LUKE: Now, Toby, you’ve done some theoretical work which attempts to address some of the complaints against consequentialism, for example, in your bachelor’s thesis, “Consequentialism and Decision Procedures.” Could you explain what the theoretical moves you’ve made are?

TOBY: Yeah, sure. In my [bachelor's] thesis and also in my doctoral thesis, which is an extension of that work, I’ve looked at basically a different conception of consequentialism, or a different way of understanding consequentialism, which broadens its scope compared to how it’s normally construed.

On the normal accounts that you hear about consequentialism… For example, if you go into a philosophy course about it, they’ll tell you that it’s about acts. Then they’ll say that the fundamental principal of consequentialism is something like this: “An act is right if, and only if, it leads to an outcome which is better than all other outcomes.” That’s what they’ll say, and they’ll focus really on acts.

Whereas, I think that you shouldn’t just focus on acts. You should also assess everything else. Rules and character traits or motives can be assessed according to the same principal. The right set of rules is the set of rules that leads to the best outcome, and so on. If you do this, you can get some kind of a partial unification of consequentialism with its main rivals, deontology and virtue ethics. Because you can talk about rules in a similar way to how deontology does, and you can talk about virtues and character traits in similar ways to virtue ethics.

To see how this works, it’s easiest to start with an old objection that some people make to consequentialism, which is a bad objection. But in order to overcome it, you have to think of it, so it’s quite useful. The objection is that they say, “Oh, maybe consequentialism is self-defeating, because if we all behaved like consequentialists, then we’d lead to a worse outcome than if we didn’t. Or even, perhaps, just if one person behaved like consequentialists, that person would lead to a worse outcome than if they behaved in some other way.” Consequentialism would then be self-defeating.

There’s some kind of a thing that has to be answered there, but any consequentialist worth their salt would say, “Well, if it turns out that, empirically, if we believe this theory or if we act on this theory, that would lead to worse outcomes, so much the worse for believing in this theory or acting on this theory. We should endorse whichever theory it is that leads to the best outcomes. We’re not slavish adherents to this idea. We’re only interested in it insomuch as it leads to good outcomes.” That would be this kind of approach.

In my thesis I looked at this and tried to work out how can you spell this out, and what kind of a theory can really deal with this. There are quite a few aspects that are quite technical and maybe less interesting, but the basic idea really is to assess everything in terms of its consequences.

For example, suppose we’re considering what’s the best moral code or professional code for doctors. Is it the code where they say they treat the patient in front of them as being all-important and try to maximize the benefit for the person in front of them? Or is it a utilitarian type of naively calculating approach, where they try to add up the benefits and costs of every action? I think that it’s quite plausible to actually lead to better outcomes if we use the former rather than the latter, because more people will actually go to the doctors.

The idea, though, is that you’re looking, in that case, at a code of conduct, and you’re assessing it in consequentialist terms in a similar way you could assess morals that are being made in consequentialist terms, and you could even assess character traits. Is it good for them if I were being a very generous person? Well, now what would the outcomes be about? It lets us see why rules, character traits, and other things are really important for me to focus on if we are thinking about ethics, without having to forgo or focus on consequences at the same time.

One of the things that I point out in my thesis is that there’s a lot of evidence that the early utilitarians were like Mill and Sedgewick and possibly Bentham actually thought like this all along. There’s just been a bit of confusion, and in fact we have forgotten that they’ve made a whole lot of comments like this. In the 20th century, people who do care about consequentialism got very fixated on acts, but I think if you look before that, you’ll find that there is less of it.

So that’s some kind of approach. It could be seen by people like me as an attempt to unify consequentialism with the best ideas in deontology and virtue ethics.Or it could be seen by my opponents as an attempt to swallow up these other theories within consequentialism, depending on whether they look favorably on the project.

LUKE: Well, Toby I have a series of posts on my blog called “Living without a Moral Code,” and I describe my predicament as being that I’m really motivated to make the world a better place. I really want to do what’s right, maybe it’s my religious upbringing, or something that makes morality so important to me, but the problem is that I don’t know what it is that would make the world a better place, because I’m not very confident about any particular theory of meta-ethics or a normative ethics. Of course, philosophers haven’t come to any consensus on those issues either.

How could someone like myself deal with that kind of fundamental moral uncertainty where it’s not just a matter of not being sure which charity will do the best in helping people or something like that, but not being sure about fundamental moral principles at all, but wanting to be moral? How can we deal with that kind of moral uncertainty?

TOBY: Well, that’s a very big question, and it’s one of the things that I’m quite interested in. Interestingly, there’s been so little focus on that type of question in modern academic ethics, at least analytic ethics, and despite it being a universal predicament.

I mean I’m not sure of what the best moral principles are either, and no one is, or if they are sure, then they shouldn’t be sure. They don’t have enough justification to be sure. We’re all in situations where we should be given some small credence — at least a small credence — to a whole lot of different ethical theories when we think about those things.

Because we’re all in, and it’s particularly interesting to try to reckon how to deal with it. One approach might be to say, “If you look your theories, look at find the theory that has the highest credence in the sort of theory that you support the most, and to just do whatever it says.”

But in some reflection you can see that can’t be right, because if you do that it might be that your credence, let’s say, is evenly split between 100 different theories almost evenly. One of them has slightly more, and that one says to do a certain thing, but all the other theories say that that would be kind of gravely irresponsible to do that thing and that you shouldn’t do it. It seems like you shouldn’t in that case just promote one of them to be believed as if it’s true.

You might think, given that kind of case, “Well, what we need to do instead is not to think of it at the theory choice level, but to think of it as thinking about the acts, one at a time.” In that particular case, if you thought to do the act that has the highest probability of being permissible morally, then you would do the other act in that case.

It would get you out of trouble on that one, but it would run into its own problems. It might be a case where you are unsure between two different theories and you think that one of them is just slightly more probable than the other, but it turns out that, therefore, the act of the first one, let’s say the slightly more probable one, suggests has the highest chance of being permissible if they conflict.

In that case, it might be that the other one says that there’s much more at stake. It says that no, this is a huge difference between doing something that’s absolutely crazy wrong and something that’s really virtuous. Another theory that you’re more confident in says that there’s not much at stake. In that case it seems like you should hedge your bets and worry about the one that says that there’s more at stake.

Then you might think, “Well, OK, how are we supposed to deal with this?” Maybe it’s something more like the way that we deal with empirical uncertainty. What we do in those cases is we try to maximize the expected benefit. We’ve got this expected idea form decision theory about expected benefits, where you multiply the probabilities by how good things would be in the cases in order to work out what you should do.

This is, I think, a more plausible approach, but it turns out that it runs into a whole lot of trouble as well, because it’s very difficult to do comparisons of value between the different theories. Suppose one of the theories is something like utilitarianism and says we should maximize happiness. Let’s suppose that we could do something such as killing 10 people to save the lives of 11 people. Let us suppose there are no other side effects of this action. Utilitarianism says that’s good. Your other theory, let’s say Kantianism, says that’s really wrong to do that.

How are you supposed to make a decision there? How confident would you have to be in utilitarianism before you should do that? It seems like we need some way of comparing how important utilitarianism thinks the benefit of one life is to how important Kantianism thinks…

LUKE: Yeah, it’s hard to see how you could put those two theories on the same scale of value.

TOBY: Yeah. That’s exactly right. That’s a big challenge and there are a couple of different approaches to dealing with that, which are quite theoretical, but one of them is to try to work out a version where you never have to do any explicit comparisons of the values between the theories. You just look at the structure of each theory and you use some clever tricks to try to find examples where they think there’s the same structure between several cases and try to use those tricks in order to rank the theories.

I think that can’t really be done and you have to actually crack open the theories and look inside them and what they’re saying in order to get the right decision. It’s really interesting theoretical stuff there. It’s unfortunately at a very early stage of development, and there are only a few philosophers looking at this. It’s really the last 15 years or something where it’s started to become a topic in its own right.

I don’t have all that much advice as to how to deal with this practically now. Although you can see that, in some cases, these things start to come up. For example, just as the philosopher Peter Singer has an argument to do with us having obligation to donate a large portion of our money to help fight global poverty.

His argument is that it’s analogous to a situation where we saw a child drowning that was easily preventable except we would ruin our suit, say, by wading into the muddy water. Then that’s a case where most people would think that the person would act wrongly if they just walked away in that case. Yet, we tend not to think that when someone could donate an amount of money equivalent to buying a nice suit and save someone’s life, but they’d buy the suit instead. We tend not to judge them as harshly.

Here’s his argument, that there’s an inconsistency in our judgments there, and the considered judgment really that they’re both equally wrong, rather than that it’s now permissible to leave children drowning in ponds. That’s another approach you could take. Here’s his argument and it’s debated a bit in the literature as to whether he’s right about that.

You might then think, “Well, hang on a second. If I’ve got some level of credence that it’s really wrong to let this happen, then maybe I should be donating more money than I’m donating in order to hedge my bets there. Even if I think that there’s a 90 percent chance that he’s wrong and that it’s purely optional to donate money to charity, maybe this serious possibility that a lot of philosophers would say that I’m doing something really wrong. Maybe I should be taking that into account when I’m choosing my action.”

You can see how it could start to influence some things that people think. We’re often very tempted to just say, “Oh, well, as it happens, I think Peter Singer is wrong about that, so I’ll just go ahead and go my own way.” But I think that once you start to really look at how you have to deal with moral uncertainty, you can see that it’s not enough to just say, “Oh, he’s probably wrong.” You would need to be really quite sure.

Similarly, if you were firing a gun through the bushes, you wouldn’t want to be just fairly convinced that there was no one in the bush. You would actually want to, and it would be your responsibility to, do quite a lot of checking to get the probability down very small if there was someone there. I think that there’s a similar situation here.

Hopefully, that’s somehow practical help about how you’re going to weigh these things in.

LUKE: Toby, I think all those ways of dealing with moral uncertainty are way too complicated and I’m just going to become a divine command theorist.

TOBY: [laughs]

LUKE: Well, so you don’t have any answers yet?

TOBY: Not really. It’s a really difficult question, but at least this idea, I think, of moral hedging is quite important. Actually, it’s quite nice. It’s related to an idea of moral trade, which you could have as well.

If you have two, not just one person who is uncertain about something, but imagine two people who have different moral preferences, where they might actually be able to get greater benefits if they did some kind of a swap. Suppose one of them cares more strongly about issue A and the other one cares more strongly about issue B. They might be able to swap votes on the matter in order to get some mutually beneficial outcome.

There are various examples of this, but here’s an example that I know has happened. A friend of mine was in New Zealand and was interested in voting for one party. The election is a two-horse race. Her friend was going to vote for the other party, so they both decided to stay home instead of going to vote.

You could have slightly more extreme examples of that where one person was going to donate, let’s say, $100 to a gun control charity and another person was going to donate $100 to a gun use charity. I’m not sure what they’re called, like the NRA. They could both decide instead to donate the $200 to Oxfam or some other group where they both agreed [on the good] it was doing.

It would seem that that’s the type of win/win trade morally, which is quite interesting to think about those cases. I think there’s quite a lot of interaction between them and moral uncertainty.

LUKE: Well, even if you don’t have the answers yet, I’m glad that you are at least working on those issues of moral uncertainty.

TOBY: Yeah. Some of the cases can be really huge. I’ve got some philosophical examples.

I guess the poverty cases are fairly practical. There are similar issues about vegetarianism. If you hold some credence that it’s wrong to eat meat and you don’t think there’s much chance that it’s right to eat meat, then that could influence how you should decide, even if you think there’s less than a 50 percent chance that it’s wrong.

Similarly, there are some big issues when it comes to evaluating global catastrophes. If it’s something like climate change, perhaps you think that this could cause human extinction or maybe some other event could cause human extinction.

Human extinction is at least as bad as 6.8 billion deaths, but a lot of people think it’s even worse, that all of the future people who could live should count morally. Maybe they think that there’s going to be not just billions of them, but hundreds of billions or trillions. They might think that that component makes this thousands of times more important than we might naively think. Alternative moral theories say that those future people don’t count at all.

You can get some kind of radical uncertainty then about how important it is to avoid this kind of catastrophe. Is it billions of times more important than saving one person’s life or is it trillions of times more important than saving one person’s life? This kind of radical disagreement.

You might think that doesn’t matter because once it’s billions of times more important, it’s important enough, but it does end up mattering. If the chances of these things actually are very small or the level of risk that you can mitigate, you can only change the probability by one in a million or something like that.

LUKE: Well, while we’re working on those issues of moral uncertainty, we might still want to work on actually answering these fundamental moral questions, so that we’re not so uncertain about which normative theory would be right, or that kind of thing. How does one go about deciding between the different normative ethical theories that are on offer, different versions of consequentialism, contractarianism, deontology, virtue ethics?

TOBY: Yeah, that’s a good question. It certainly makes sense, even if you have a theory of moral uncertainty. How to act when you don’t know what’s going on with somebody’s philosophical question? You can see that there’s still a large value of information. Just as we might have a theory which says, “If you don’t know what the card in someone’s hand is when you’re playing poker, the right thing for you to do at the moment is to, let’s say, fold.”

But it could mean that there’s a huge value of information if you can find out something about what card they were holding. That kind of principle still makes sense here, that even better than just acting under uncertainty is removing your uncertainty or lowering your uncertainty, and then you make a more sensible act.

A lot of people in philosophy departments have been trying to argue their cases between these different types of moral theory for quite a while. Although, if you look at the history of philosophy, while it goes back more than 2,000 years, there were very few people doing it in the past and a lot more now. If you look at all the words that have been written about philosophy, I don’t know when the median word was written, but it’s probably something like 1970 rather than the year 1000.

Are these words all of equal quality? Well, the answer is probably no. Also, have there been diminished marginal returns? Is it the case that we’ve worked out a lot of the basic stuff and now, if someone wants to make progress, they’re doing it on a more constrained area? I think that’s probably true. So you can’t just do it by numbers, and I was just pointing that out.

I think that there has been a bit of progress made on some of these questions. Certainly the arguments pile up, even if people haven’t come to firm conclusions about what type of theories are better. The body of sensible argument has been increasing. There’s also, as always in philosophy, a body of stupid arguments on some of these topics, but setting that aside, the amount of sensible argument has also been increasing.

In the case of consequentialist theories, I think that there is a lot of evidence in favor of them at the expense of other theories. Although in some ways, like with my work, it’s not even at the expense of them. It’s possible that in a certain sense the deontological theories and virtue ethics theories are true. While in another sense, I think a deeper sense, the consequentialist theories can be true at the same time.

This can be true. If it is true, that we ought to follow a certain set of rules, the set of rules that we ought to follow is one that’s chosen based on its consequences. If we ought to have certain character, the character that we should have is chosen based on those consequences. We ought to act in certain ways, and the acts be chosen on their consequence.

It’s possible to have an underlying theory that fits those things, but as to how in practice to do it, that’s difficult to say. The obvious answer would be, I think, enroll in a philosophy course and so on. As to how much more certain you’ll be when you come out of the course, I’m not sure. It’s just quite tricky to do.

LUKE: Well, one of the more common ways of arguing between these different normative theories is to argue about whether the conclusions of a particular theory agree with our moral intuitions about the matter.

For example, if happiness utilitarianism tells us that we should kill an innocent person to harvest their organs to save the lives of five other people, but then our moral intuitions tell us that this would be highly immoral to kill this person, this innocent person, then that somehow counts as evidence against happiness utilitarianism.

Why trust our moral intuitions in the first place? Aren’t they just evolved moral prejudices? Do philosophers think that we have a morality module in the brain that can detect moral facts and that’s why our intuitions provide evidence for or against particular moral conclusions? What’s the thinking there?

TOBY: That’s a good question and definitely I don’t have a solid answer to it. Although I should say, this is in some ways a virtue of philosophers is that they often know when they don’t know something. Whereas if you asked them on the street, they might just assumed that they have answers to all of those questions, even though those answers are probably unjustified.

I think our intuitions do count as evidence is favor of certain moral positions. However, I think it’s fallible evidence in the same way as, suppose we want to know the relationship between the height at which you drop a ball and the length of time it takes to hit the ground. We know that there’s a parabolic aspect here, but it might be that our stopwatch that we use to measure the times, we’re not perfectly accurate at using that. It turns out that there’s noise in the data.

If we naively just try to connect all of the dots on the chart we’re making, we would get this very strange-looking line that was roughly parabolic, but had heaps of squiggles on it. I think that that would be a mistake. What we should do in that case it to use Occam’s razor and think that simplicity of your theory is a virtue. In doing so, that kind of approach helps to eliminate the noise when you’ve got fallible measurements.

Similarly, one way that we could try to do moral theory is to say that the truth about ethics is just a combination of the truth about my intuitions about every single scenario. You just add them all together. In that case we’ll come up with a moral theory that, at least for me, there are no strikes against it, because it’s my intuition in every single case.

But it’s a very implausible theory. We know that over time, the moral intuitions of a society change. We would think that if people in the past had used a theory like this, they would get very stupid answers. We also know that it differs between different people. This seems to indicate that there’s at least — assuming there’s something worth tracking at all, which I think that there is, then we know that we have got some kind of a noisy measurement off of it.

We can’t just use data fit as the only criterion. I would try to use something like the fitting of the data and also some notion of theoretical simplicity of the theory just like in science.

Not everyone agrees with that. In fact, that is actually an interesting topic. Your question broaches on several different things to do with normative ethics, epistemology of ethics. How do we come to know moral facts? Some questions about moral psychology and about evolutionary psychology, and also some questions about meta-ethics.

LUKE: Well, Toby, you’re not just a moral theorist, but a moral activist, I guess we might say, in that you’re the founder of the organization Giving What We Can. I spoke about, or I interviewed someone about Giving What We Can in an earlier interview, Nick Beckstead, who launched a chapter of your organization at Rutgers University, but I’d like to ask you how you came to found the organization and what successes you’ve had so far?

TOBY: Sure. Well, yeah, it was something like seven years ago or six years ago, I was writing an essay as part of my graduate studies here at Oxford. The essay was on the topic of, ought we always to forgo a luxury if that would allow us to save someone’s life?

At first you might think, “Yeah, well, obviously we should. That’s a pretty simple essay topic.” But what they were getting at was these cases of global poverty and whether we are perhaps always in a situation like this, where we can forgo a luxury to enable us to save someone’s life, in which case maybe would have to forgo all luxuries in our lives. It turns from something that seems quite trivial to something that seems very challenging for us.

I was looking at this and reading arguments by Peter Singer and others about this. I’d always had some sympathy for this view because of a utilitarian leaning. It really made me think hard about it, and thinking about it practically. Could I live like this? How should I live? I decided to try to work out what I could achieve in my life if I really wanted to. That’s quite a large project, and I certainly haven’t finished working it out, but I’ve done some sketching and one thing to think about is, there’s different areas of your life you can achieve things in.

For example, there’s through your work. There are also the different things that you can achieve through your friends and family, things that you can do through volunteering, and, of course, there are things that you can do by donating money. I felt, about the donating money part, is central to what I was thinking about at the time, and also, things like the type of part that is easy to quantify.

So, how much money could I donate over the course of my life? I worked out that, as a UK academic, I should be able to earn about 1.5 million pounds over my life. If I kept a similar living standard to what I have at the time as a grad student, that I would be able to give away about a million of the 1.5 million, which is quite a lot. It’s quite a nice thing to think about when you’re young, as to, what’s the most I could do if I really wanted to?

I should point out that if you’re an academic in the U.S., that you make more than that. The same in Australia. It seems, actually, like being in the UK is bad country for an academic, as far as donating money goes.

I then thought about this and thought, what could I do about it? And so, I became really interested in cost effectiveness. A colleague of mine, Gaverick Matheny, he sent me a link to a fantastic report done by the Disease Control Priorities Project, where they did some meta-analysis of a whole lot of papers that are being published on the cost effectiveness of different ways of treating various health problems.

And so, it wasn’t just a case of looking at the health problem like AIDS as one big block, but instead, they would say, “Well, there are heaps of different ways to treat or prevent AIDS. So, let’s look at approaches of, let’s say, treating one of the illnesses that you get if you’ve got AIDS and your immune system is lowered and it allows extra illnesses in.”

There’s an illness called Kaposi’s sarcoma, and they found that you can produce about one, what they call “quality-adjusted life year.” That’s the equivalent of a year of life at full health, an extra year of life at full health.

The idea is that it could actually be 10 extra years of life at 10 percent of quality of health, or two extra years of life at half quality of health. Or maybe it’s just taking 10 existing years of life and improving them by 10 percent. The idea is that it’s something which is worth the same amount as an extra year of life at full health. They call that a QALY, a quality-adjusted life year.

It’s a utilitarian type of concept. They wanted to look at, in this book, to measure these things by this and say, if you’re… funding different programs and different interventions, how many of these quality-adjusted life years can you produce for giving that amount of money?

And so, if you have $30,000 and you want to treat Kaposi’s sarcoma, you can produce about one year of quality-adjusted life. Whereas, if you, instead, do different approaches, so, for example, if you, instead, use antiretroviral drugs to fight HIV itself, you can, instead produce about 15 years of life, after the same amount of money.

If you go for education for high-risk groups, you can get it off to about 100 years of life. And then, there’s some other approaches. That’s not the best that we know of in terms of HIV/AIDS. If you look at other areas of health, you can get all the way up to about 10,000 years of life at full health for the same amount of money.

Just looking at HIV/AIDS interventions, they span about three orders of magnitude in total from the least cost effective to the most cost effective. The most cost effective does 1,000 times as much good as the least. If you’re willing to go into other areas, you can do up to 10,000 times as much good.

So this really made me very interested, for two reasons. One reason was that it meant that I could do a huge amount with my money. If I’d taken it seriously, by giving away, it’s about 1.5 million dollars (U.S.), over my life, which is a million pounds, that I would be able to, lead to something like 400,000 years of life at full health, equivalent, over my life, which is amazing.

Caring about where you give it to is also incredibly important. For the average American, it’s, I think, quite possible to give 10 times as much as they would be giving normally, and to be giving it to organizations, which are at least 10 times as effective. If someone did both those things, then it would have 100 times the impact.

I decided to think really seriously about this idea of giving more and giving more effectively, and trying to do, really, the best you can with the donations. From this, I decided to set up a society of people who are taking it seriously and who are willing to donate a significant portion of their income over the rest of their life to the places they thought could most cost-effectively fight causes or effects of poverty in the developing world.

I chose that focus area to narrow it in. I set it up about a year ago. Now we have 80 members, and together the members of the organization have pledged about 15 million U.S. dollars over the courses of their careers. That’s enough money, using these figures I had before, to produce about seven million quality-adjusted life years.

To put that into perspective, seven million years is, if that was all lived in a row, that’s enough time to take us back to the split between the homo and pan. Between the humans and chimpanzees, their common ancestor was about seven million years ago. It’s quite a mind-boggling amount of benefit that could be created by a relatively small group of people.

LUKE: You know, Toby, what I love about Giving What We Can is that it places the emphasis on doing as much objective good as possible. The problem with a lot of charity is that we’re really just giving to the charity and we’re purchasing warm fuzzy feelings about what we’re doing. And so, we’ll give to the charity that gives us warm fuzzies, rather than doing the research and figuring out what actually does the most good, in terms of, you know, quality-adjusted life years, or something like that.

And so, you know, this is just such a great thing to put the emphasis on: “You know what, if you’re going to give to charity, maybe you should give to making the world a better place, rather than purchasing warm fuzzies. If you want warm fuzzies, get them somewhere else. Let’s come together and make the world a better place.”

TOBY: Yeah, well, exactly. As one of my friends puts it, there’s a lot of people who want to make the world a better place, but there aren’t that many people who want to make the world as good a place as possible. You know, there are people who want to make a difference, but not to make as much positive difference as possible.

LUKE: Well, Toby, it’s been a pleasure speaking with you. Thanks for coming on the show.

TOBY: OK, yeah. Thanks very much for having me.

SMBC on the Gettier problem and quantum field theory.

Interview with Peter Boghossian about how professors should challenge students’ groundless (often faith-based) beliefs with facts.

New posts at Facing the Singularity: Not Built to Think About AI and Playing Taboo with “Intelligence”.

From Twitter:

• Programming for human beings: “Literate programming.”
• Mirror image cells could kill millions of people.
• “Whatever doesn’t kill you makes you stronger”? Not true. Here’s Christopher Hitchens.
• A $500 home molecular biology laboratory?
• Nintendo’s Miyamoto is leaving Mario and Zelda to other designers so he can work on smaller, more personal games.
• Quick answers for global warming skeptics.
• Correlation vs. causation.
• Say more, justify less.
• The short, simple case against having ads on Wikipedia.
• The inaugural lecture for the Programme on the Impacts of Future Technology, by Eric Drexler.
• Should we make pandemic bird flu viruses?
• Best of the web 2011 HD.
• Portugal’s alternate gaming history: What happens when a country misses the NES?
• Literally the most amazing college football play I have ever seen in my life.

Congratulations. You figured out the universe runs on physics, not magic.

Given your background and normal human psychology, that may be an impressive feat. It took me more than 20 years.

Please enjoy resisting, debunking and openly mocking religion.

But don’t stop there.

Now you’re living in the real world, and there are real problems here. In the real world, there is no rule that says the good guys win in the end. We live in a world beyond the reach of God, things can go very wrong, and we need your help.

“Critical thinking” is just the start. The rabbit hole of better thinking goes deeper. There are mathematical laws of thought, a mainstream cognitive science of how we depart from them, and experiments showing us how to do better.

Aiding the slow death of religion and superstition is also just a start. We face more pressing concerns, like the fact that every few decades we invent a new technology that can destroy our entire species.

After atheism comes adulthood, with all its challenges and its opportunities. Want to be a hero? Good. We need heroes. Here’s what you do.

An underappreciated post from Paul Christiano:

Hopefully [LessWrong.com] trains its readers to be aware of personal failures of rationality and to correct them over time.

Whenever a small group (all the way down to 2) of rationalists communicate, they should make an equal effort to be aware of failures of their collective rationality (I definitely have failed at this so far in life). In particular: time is valuable, and you can hope to actually learn things from other people. Here are some remarks about communication intended to help you optimize (as opposed to communication intended to help you be happy).

1. Communicate information precisely. I have strong beliefs that contradict the beliefs of other rationalists. I need a better way to express precisely what the disagreement is and my confidence in my beliefs, particularly with people I don’t know well. Right now I feel like I have no hope of correctly updating on the beliefs of others or sharing information in a productive way. To this end, I think making precise testable claims about the future and backing them up with probability estimates (as a regular feature of discussion) would be a good idea, even when your probability estimates are horribly miscalibrated. This would help me much more quickly learn about my own miscalibration, evaluate the miscalibration of others, and at least have a definite scale to make precise communication possible.

2. Communicate preferences precisely. Groupmembers often have strong preferences which they fail to coordinate well. I think groups I have been in would have done better if they consistently used any precise semantics for expressing preferences; for example, assigning monetary values or using some other fixed reference commodity. A bigger problem is that statements of preference carry way to much additional meaning in normal communication. I’m not sure how to divorce statements of preference from this other meaning, but its worth thinking about.

3. Explicitly think about the point of communication. There are important things to talk about, and the marginal benefits of spending 10 minutes on one topic versus another can be quite different. There are a lot of free moments in a day, and I think this calculus is worth spending some of them on. I would like to start directing my conversations towards high value topics much more reliably, especially in conversations with rationalists. This may involve doing things like making explicit notes to structure my conversation.

4. Talking about procedure is OK. Improving your rationality is worth spending time on, and I think improving the quality of your communication and a community’s ability to communicate is worth spending time. Entertain some discussion of process without feeling irresponsible for not moving forward on the meat of whatever issue you are discussing. Perhaps make rational communication the meat of discussion more often.

5. Acquire a lot of data. The effectiveness of strategies for communication or group rationality is a really easy thing to gather data on, at least in principle. Share data (doing this effectively requires improvements in communication).

I’m glad Robert McCauley wrote Why Religion is Natural and Science is Not, so I don’t have to.

New at Facing the Singularity: The Crazy Robot’s Rebellion.

Richard Carrier’s Skepticon IV talk: Bayes’ Theorem, Key to the Universe.

Steve Maitzen‘s new lecture on God and Morality (lecture ends at 39:05).

From Twitter:

• Good, short article on trends in philosophy.
• The disaster movie spectacular series.
• New robot ethics dissertation, with pictures.

New posts at Facing the Singularity: Why Spock is Not Rational and The Laws of Thought.

New at Less Wrong: Hack Away at the Edges and Why study the cognitive science of concepts?

From Twitter:

• The machine that would predict the future.
• Build your own chocolate bar, $4.50 and up.
• Congress is considering a bill for censoring the Internet. Despite public outcry, the bill could pass at any time.
• A blog of great Spotify playlists
• Laser toy. Do want.
• Comparison of artificial brain projects.
• A new research institute in Japan meant for generalist researchers.
• Reuters’ best photos of the year.

Three-hour BCC documentary on the origins of the Abrahamic religions:

I made a dedicated discussion page for Facing the Singularity.

From Twitter:

• Fukushima: The first 24 hours.
• Minute physics (awesome videos).
• This guy knows how to teach math and science. “Here, it’s cool to be smart.“
• Personal trainer starts eating like an American, gains 70 pounds. Will return to sane diet and lose it all.
• Everything is a remix.
• The 10 firms that rule the world.
• Amazing hyper-realist paintings.
• How convenient. “2011 in science” on Wikipedia.
• MuckRock makes it easy to file Freedom of Information requests.
• The science of willpower.
• After 17 years of professional link-finding, Kottke finds the craziest thing he has ever seen on the internet.
• Three awesome films by three awesome directors arrive on Dec. 21: Spielberg’s Tintin, Fincher’s Girl with the Dragon Tattoo, Bird’s Mission Impossible: Ghost Protocol.

I’ve moved Facing the Singularity to its own website: FacingTheSingularity.com.

I’m using the Melville theme for WordPress, but I need to modify it. Anybody wanna help?

Here are the changes I’d like to make to that theme:

1. This header image above the blog title.
2. Larger, darker body text, but same side-margins width.
3. No date or comments or category mention at the bottom of posts.
4. No newer/older posts link at the bottom.
5. No italics for title of blog or links in the footer.
6. ‘Next chapter’ link at the bottom of each post, which will lead to the most recent post.
7. Home page shows the oldest post (‘Preface’).
8. ‘Contents’ page (archives page, renamed) auto-generates a numbered list of all posts, in descending order of age (i.e., oldest post first).
9. ‘RSS’ link in the footer (which links to the RSS feed, not to a page).
10. This elaborate dinkus below the blog title.

Update: Facing the Singularity has a new home at FacingTheSingularity.com.

Update: Facing the Singularity has a new home at FacingTheSingularity.com.

Before I talk about machine superintelligence, I need to talk about rationality, because my understanding of rationality shapes the way I see everything, and it is the main reason I take the problems of machine superintelligence seriously.

If I could say only one thing to the ‘atheist’ and ‘skeptics’ communities, it would be this:

Skepticism and critical thinking teach us important lessons: Extraordinary claims require extraordinary evidence. Correlation does not imply causation. Don’t take authority too seriously. Claims should be specific and falsifiable. Remember to apply Occam’s razor. Beware logical fallacies. Be open-minded, but not gullible. Etc.

But this is only the beginning. In writings on skepticism and critical thinking, these guidelines are only loosely specified, and they are not mathematically grounded in a well-justified normative theory. Instead, they are a grab-bag of vague but generally useful rules of thumb. They provide a great entry point to rational thought, but they are only the beginning. For 40 years there has been a mainstream cognitive science of rationality, with detailed models of how our thinking goes wrong and well-justified mathematical theories of what it means for a thinking process to be “wrong.” This is what we might call the science and mathematics of Technical Rationality. It takes more effort to learn and practice than entry-level skepticism does, but it is powerful. It can improve your life and help you to think more clearly about the world’s toughest problems.

So what is this mainstream cognitive science of rationality? You will find it described in every university textbook on thinking and decision making. For example:

• Baron, Thinking and Deciding
• Hastie & Dawes, Rational Choice in an Uncertain World
• Bazerman & Moore, Judgment in Managerial Decision Making
• Plous, The Psychology of Judgment and Decision Making
• Gilboa, Making Better Decisions

You will also find pieces of it in the recent popular-level books on human irrationality, for example:

• Ariely, Predictably Irrational
• Kahneman, Thinking, Fast and Slow
• Thaler & Sunstein, Nudge
• Tavris & Aronson, Mistakes Were Made: But Not By Me

And you will, of course, find it in the academic journals. Here are links to the Google Scholar results for just a few of the field’s common terms:

• “heuristics and biases“
• “debiasing“
• “affect heuristic“
• “myside bias“
• “base rate fallacy“
• “framing effects“
• “availability bias“
• “conjunction fallacy“

Technical Rationality

There are two parts to technical rationality, normative and descriptive.

The normative part describes the laws of thought and action. Basically: logic, probability theory, and decision theory. Logic and probability theory describe how you should reason if you want to maximize your chances of acquiring true beliefs. Decision theory describes how you should act if you want to maximize your chances of acquiring what you want. Of course, these are not physical laws but normative laws. You can break these laws if you choose, and people often do. But if you break the laws of logic or probability theory you decrease your chances of arriving at true beliefs, and if you break the laws of decision theory then you decrease your chances of achieving your goals.

The descriptive part describes not how we should reason and act, but how we usually do reason and act. The descriptive program includes research on how humans think and decide. It also includes a catalogue of common ways in which we violate the laws of thought and action from logic, probability theory, and decision theory. A cognitive bias is a particular way of violating logic, probability theory, or decision theory. That’s how “bias” is defined (see, e.g., Thinking and Deciding or Rationality and the Reflective Mind, each of which has a table of common biases and which part of logic, probability theory, or decision theory is violated by each of them).

Cognitive scientists also distinguish two domains of rationality: epistemic and instrumental.

Epistemic rationality concerns forming true beliefs, or: having in your head an accurate map of the territory out there in the world. (Epistemic rationality is governed by the laws of logic and probability theory.)

Instrumental rationality concerns achieving your goals, or: maximizing your chances of getting what you want. Or, more formally: maximizing your “expected utility.” Also known as “winning.” (Instrumental rationality is governed by the laws of decision theory.)

In a sense, instrumental rationality takes priority, because the point of forming true beliefs is to help you achieve your goals, and sometimes spending too much time on epistemic rationality is not instrumentally rational. For example, I know some people who would be more likely to achieve their goals if they spent less time studying rationality and more time, say, developing their social skills.

Still, it can be useful to talk about epistemic and instrumental rationality separately. Just know that when I talk about epistemic rationality, I’m talking about following the laws of logic and probability theory, and that when I talk about instrumental rationality, I’m talking about following the laws of decision theory.

So: I’ve explained Technical Rationality and given lots of social proof for its legitimacy. But what is the justification for the normative laws and descriptive claims of Technical Rationality, and how does one become skilled in Technical Rationality? And why should a mastery of Technical Rationality lead you to take the problems of machine superintelligence seriously? I’ll begin to answer these questions in the next post.

Update: Facing the Singularity has a new home at FacingTheSingularity.com.

I’ve learned more in the past year than in the previous 5 years of my life combined. (Largely, because I got really good at independent learning.)

Also, I joined Singularity Institute and discovered an enormous treasure trove of knowledge and analysis that hadn’t been written down or well-organized anywhere.

I’ve been trying to share what I’ve learned in a long series of brief, carefully written, and well-referenced articles, but such articles take a long time to write. It’s much easier to write long, rambling, unreferenced articles.

This new series, ‘Facing the Singularity,’ is my new attempt to rush through explaining as much material as possible. I won’t edit, I won’t hunt down references, and I won’t try to be brief. I’ll just write, quickly.

The subject matter won’t appeal to everyone. The unedited writing certainly won’t. Nevertheless I suspect what I cover will be interesting and informative to many. Besides, this is my personal blog; I’ll write what I want, here.

Personal background

I’ll begin with my personal background. It will help to know who I am and where I’m coming from. That information is some evidence about how you should respond to the other things I say.

I’ve told my personal story before, so here’s the quick version: I was raised a devoted evangelical Christian in Minnesota. I attended a Christian school, enthusiastically read pop-theology in my spare time, played in the worship band, went on short-term missionary trips around the world, experienced what I believed to be the Holy Spirit many times — I was a true believer.

Around age 22 I wanted nothing more than to be like Jesus to a lost and hurting world. Thus, I had to study the Historical Jesus, to figure out what “being like Jesus” meant. That’s when I learned all kinds of disturbing things about the Bible and the origins of Christianity. I wanted to protect my faith, so I read more and more “sophisticated” works of Christian apologetics. I also read a few books by atheists, so I could be “fair” in my investigation. The short story is that the atheists won, and I lost my faith in God completely.

This was horrifying at first, because my relationships, my morality, my sense of purpose, my life plans, and my picture of the world were all grounded in Christianity. Gradually, I built up a new worldview based on the mainstream scientific understanding of the world, and approach called “naturalism.” I launched Common Sense Atheism in November 2008 (at age 23) to explain atheism and naturalism to others, because I was unhappy with the lack of philosophical and rational seriousness displayed by the atheist blogosphere at that time.

My blog became one of the most popular atheism blogs rather quickly. I enjoyed translating the results of professional philosophy of religion for the mass public, and I enjoyed speaking with experts in the field for my podcast Conversations from the Pale Blue Dot.

Morality was always an important topic for me, and eventually I was writing more about morality than I was about philosophy of religion. When I launched my blog I was an error theorist, which is one way of being a moral anti-realist. Shortly after launching Common Sense Atheism, I encountered the work of Alonzo Fyfe, who presented the first theory of naturalistic moral realism that didn’t smell like bullshit to me. I came to accept and then promote his theory of “desire utilitarianism” (aka “desirism”) on my blog.

My moral views have continued to evolve, and my latest statement of moral theory is summarized here. I’ve come to think that moral language is so confused and contentious that we may want to abandon moral terms altogether, and I try to avoid classifying myself as either “moral realist” or “moral anti-realist” because, as always, whether I’m one or the other depends on what is meant by those terms, and people use the terms in a variety of ways. Whether anti-realism or naturalistic moral realism is “true” depends to some degree on one’s attitude toward the use of moral language, as Richard Joyce argues in his paper Metaethical Pluralism, and as I argue in Pluralistic Moral Reductionism. (I still think desirism is one particularly useful way to talk about morality, much better than many others, and it fits within the framework of pluralistic moral reductionism.)

I was also always interested in rationality, at least since my deconversion, during which I discovered that I could easily be strongly confident of things that were total nonsense. How could the human brain be so incredibly misled? Obviously, I wasn’t Aristotle’s “rational animal.” Instead, I was Gazzaniga’s rationalizing animal. Critical thinking was a major focus on Common Sense Atheism, and I spent as much time criticizing poor thinking in atheists as I did criticizing poor thinking in theists.

Intelligence explosion

My interest in rationality inevitably lead me (in mid 2010, I think) to the largest and best treasure trove of articles on the mainstream cognitive science of rationality: Less Wrong. It was here that I first encountered the idea of intelligence explosion and the need for Friendly AI, though I had encountered the mainstream machine ethics literature back in June 2009.

I tell the story of my first encounter with the famous paragraph from I.J. Good on intelligence explosion here. In short:

Good’s paragraph ran me over like a train. Not because it was absurd, but because it was clearly true. Intelligence explosion was a direct consequence of things I already believed, I just hadn’t noticed! Humans do not automatically propagate their beliefs, so I hadn’t noticed that my worldview already implied intelligence explosion.

I spent a week looking for counterarguments, to check whether I was missing something, and then accepted intelligence explosion to be likely (so long as scientific progress continued). And though I hadn’t read Eliezer on the complexity of value, I had read David Hume and Joshua Greene. So I already understood that an arbitrary artificial intelligence would almost certainly not share our values.

My response to this discovery was immediate and transforming:

I put my other projects on hold and spent the next month reading almost everything Eliezer had written. I also found articles by Bostrom and Omohundro. I began writing articles for Less Wrong and learning from the community. I applied to Singularity Institute’s Visiting Fellows program and was accepted. I quit my job in L.A., moved to Berkeley, worked my ass off, got hired, and started collecting research related to rationality and intelligence explosion.

As my friend Will Newsome once said, “Luke seems to have two copies of the ‘Take Ideas Seriously’ gene.”

Fanaticism?

Of course, what some people laud as “taking serious ideas seriously,” others see as an innate tendency toward fanaticism. Here’s a comment I could imagine someone making:

I’m not surprised. Luke grew up believing that he was on a cosmic mission to save humanity before the world ended with the arrival of a super-powerful being (the return of Christ). He lost his faith and with it, his sense of epic purpose. His fear of nihilism made him susceptible to seduction by something that felt like moral realism (desirism), and his need for an epic purpose made him susceptible to seduction by Singularitarianism.

One response I could make to this would be to say that this is just “psychologizing,” and doesn’t address the state of the evidence for the claims I now defend concerning intelligence explosion. That’s true, but again: Plausible facts about my psychology do provide some Bayesian evidence about how you should respond to the words I’m writing in this series.

Another response I could make would be to explain why I don’t think this is quite what happened, though elements of it are certainly true. (For example, I don’t recall feeling that the return of Christ was immanent or that I was on a cosmic mission to save every last soul, though as an evangelical Christian I was theologically committed to those positions. But it’s certainly the case that I am drawn to “epic” things, like the rock band Muse and the movie Avatar.) But I don’t want to make this post even moreso about my personal psychology.

A third response would be to appeal to social proof. There seems to be a class of Common Sense Atheism readers that has read my writing so closely that they have developed a strong respect for my serious commitment to rationality and changing my mind when I’m wrong, and so when I started writing about Singularity issues they thought, “Well, I used to think the Singularity stuff was pretty cooky, but if Luke is taking it seriously then maybe there’s more to it than I’m realizing,” and they followed me to Less Wrong (where I was now posting regularly). I’ll also mention that a significant causal factor in my being made Executive Director of Singularity Institute after so little time with the organization was that Singularity Institute staff could see that I was seriously devoted to rationality and debiasing, seriously devoted to saying “oops” and changing my mind and responding to argument, and seriously devoted to acting on decision theory rather than habit and emotion as often as I could.

In surveying my possible responses to the “fanaticism” criticism above, I’ve already put up something of a defense. But that’s about as far as I’ll take it. I want people to take what I say with a solid serving of salt. I am, after all, only human. Hopefully my readers will take into account not only my humanity but also the force of the arguments and evidence I will later supply concerning the arrival of machine superintelligence.

The debates page has been updated again, pushing the total to over 600.

I’ve written a summary of my in-progress metaethics sequence.

From Twitter:

• Thrilling video art (watch in 720p).
• The Onion: “Man to Sail Around World to Decrease Awareness of Important Issues.”
• My new favorite web design.
• I get two Malick films in the next few years? Praise the gods.
• Oxford’s Nick Bostrom: philosophers can be useful again if they are polymaths who engage in technological forecasting.
• Synthetic biology explained (video).
• List of all Singularity Summit talk videos.
• The surprising cognitive abilities of crows.
• Two spiral galaxies, merging.
• Skinemax is “Koyaanisqatsi for a generation raised on late night television and B-movie VHS tapes.”

New Less Wrong post: Existential Risk.

Draft of a book chapter I wrote with Louie Helm: The Singularity and Machine Ethics.

I’ve begun posting snippets from a forthcoming Friendly AI FAQ for feedback. I’m doing the same thing with an overview analysis of intelligence explosion.

From Twitter:

• Single-molecule car is even smaller than a Smart car.
• How to teach logic
• Playing an addictive puzzle game helps scientists unravel the mysteries of protein folding.
• Flicking your eyes back and forth can improve your memory, but only if you’re strongly right-handed.
• Simple gratitude (watch in 1080p).
• Incredible combination of yoga and breakdancing.

My atheism vs. theism debates page, the most comprehensive such page on the internet, remains one of the most popular pages on Common Sense Atheism. People still send me new debates to add all the time, and I keep them in a list. But my new role as Executive Director at Singularity Institute makes it difficult for me to find time to keep the page updated. Would someone like to volunteer to maintain it for me?

It would work like this: Once a month I send you a list of new debates to add (or corrections to make), complete with audio and video links and so on. You then add these to the HTML table in a Google doc we share, update the debate count, update the ‘most recent update’ section of the page, and let me know when you’re done. Then I upload the updated HTML code to the website.

The volunteer will get a name mention at the top of the page, along with a link to their website (if desired).

Past evidence of reliability on this kind of thing would be appreciated.

Please let me know if you’re interested to do this for me!

If you like both counterculture and futurism, you’ll probably like Extreme Futurist Festival 2011 in L.A. This two-day event (December 16-17, 2011) features futurist speakers and live music, and tickets are only $50 (or $30 if under 21). The event is co-organized by my friend Michael Anissimov of Accelerating Future, and will be held at Courtyard Los Angeles in Marina del Rey, CA: