Games with Words

When is the logically impossible possible?

2010-09-03T00:12:00.001-04:00

Child's Play has posted the latest in a series of provoking posts on language learning. There's much to recommend the post, and it's one of the better defenses of statistical approaches to language learning around on the Net. It would benefit from some corrections, though, and into the gap I humbly step...

The post sets up a classic dichotomy:

Does language “emerge” full-blown in children, guided by a hierarchy of inbuilt grammatical rules for sentence formation and comprehension? Or is language better described as a learned system of conventions — one that is grounded in statistical regularities that give the appearance of a rule-like architecture, but which belie a far more nuanced and intricate structure?

It's probably obvious from the wording which one they favor. It's also less obviously a false dichotomy. There probably was a very strong version of Nativism that at one point looked like their description of Option #1, but very little Nativist theory I've read from the last few decades looks anything like that. Syntactic Bootstrapping and Syntactic Bootstrapping are both much more nuanced (and interesting) theories.

Some Cheek!

Here's where the post gets cheeky:

For over half a century now, many scientists have believed that the second of these possibilities is a non starter. Why? No one’s quite sure — but it might be because Chomsky told them it was impossible.

Wow? You mean nobody really thought it through? That seems to be what Child's Play thinks, but it's a misrepresentation of history. There are a lot of very good reasons to favor Nativist positions (that is, ones with a great deal of built-in structure). As Child's Play discuss -- to their credit -- any language admits an infinite number of grammatical sentences, so any finite grammar will fail (they treat this as a straw-man argument, but I think historically that was once a serious theory). There are a number of other deep learning problems that face Empiricist theories (Pinker has an excellent paper on the subject from around 1980). There are deep regularities across languages -- such as linking rules -- that are crazy coincidences or reflect innate structure.

The big one, from my standpoint, is that any reasonable theory of language is going to have to have, in the adult state, a great deal of structure. That is, one wants to know why "John threw the ball AT Sally" means something different from "John threw the ball TO Sally." Or why "John gave Mary the book" and "John gave the book to Mary" mean subtly different things (if you don't see that, try substituting "the border" with "Mary"). A great deal of meaning is tied up in structure, and representing structure as statistical co-occurrences doesn't obviously do the job.

Unlike Child's Play, I'm not going to discount any possibility of the opposing theories to get the job done (though I'm pretty sure they can't). I'm simply pointing out that Nativism didn't emerge from a sustained period of collective mental alienation.

Logically Inconsistent

Here we get to the real impetus for this response, which is this extremely odd section towards the end:

We only get to this absurdist conclusion because Miller & Chomsky’s argument mistakes philosophical logic for science (which is, of course, exactly what intelligent design does). So what’s the difference between philosophical logic and science? Here’s the answer, in Einstein’s words, “No amount of experimentation can ever prove me right; a single experiment can prove me wrong.”

In context, this means something like "Just because our theories have been shown to be logically impossible doesn't mean they are impossible." I've seen similar arguments before, and all I can say each time is:

Huh?

That is, they clearly understand logic quite differently from me. If something is logically impossible, it is impossible. 2 + 2 = 100 is logically impossible, and no amount of experimenting is going to prove otherwise. The only way a logical proof can be wrong is if (a) your assumptions were wrong, or (b) your reasoning was faulty. For instance, the above math problem is actually correct if the answer is written in base 2.

In general, one usually runs across this type of argument when there is a logical argument against a researcher's pet theory, and said researcher can't find a flaw with the argument. They simply say, "I'm taking a logic holiday." I'd understand saying, "I'm not sure what the flaw in this argument is, though I'm pretty sure there is one." It wouldn't be convincing (or worth publishing), but I can see that. Simply saying, "I've decided not to believe in logic because I don't like what it's telling me" is quite another thing.

Is psychology a science, redux

2010-08-31T09:45:00.000-04:00

Is psychology a science? I see this question asked a lot on message boards, and it's time to discuss it again here. I think the typical response by a researcher like myself is an annoyed "of course, you ignoramus." But a more subtle response is deserved, as the answer depends entirely on what you mean by "psychology" and what you mean by "science."

Two Psychologies

First, if by "psychology" you mean seeing clients (like in Good Will Hunting or Silence of the Lambs), then, no, it's probably not a science. But that's a bit like asking whether engineers or doctors are scientists. Scientists create knowledge. Client-visiting psychologists, doctors and engineers use knowledge. Of course, you could legitimately ask whether client-visiting psychologists base their interventions on good science. Many don't, but that's also true about some doctors and, I'd be willing to bet, engineers.

Helpfully, "engineering" and "physics" are given different names, while the research and application ends of psychology confusingly share the same name. (Yes, I'm aware that engineering is not hte application of physics writ broadly -- what's the application of string theory? -- and one can be a chemical engineer, etc. I actually think that makes the analogy to the two psychologies even more apt). It doesn't help that the only psychologists who show up in movies are the Good Will Hunting kind (though if paleoglaciologists get to save the world, I don't see why experimental psychologists don't!), but it does exist.

A friend of mine (a physicist) once claimed psychologists don't do experiments (he said this un-ironically over IM while I was killing time in a psychology research lab). My response now would be to invite him to participate in one of these experiments. Based on this Facebook group, I know I'm not the only one who has heard this.

Methods

There are also those, however, who are aware that psychologists do experiments, but deny that it's a true science. Some of this has to do with the belief that psychologists still use introspection (there are probably some somewhere, but I suspect there are also physicists who use voodoo dolls somewhere as well, along with mathematicians who play the lottery).

The more serious objection has to do with the statistics used in psychology. In the physical sciences, typically a reaction takes place or does not, or a neutrino is detected is not. There is some uncertainty given the precision of the tools being used, but on the whole the results are fairly straight-forward and the precision is pretty good (unless you study turbulence or something similar).

In psychology, however, the phenomena we study are noisy and the tools lack much precision. When studying a neutrino, you don't have to worry about whether it's hungry or sleepy or distracted. You don't have to worry about whether the neutrino you are studying is smarter than average, or maybe too tall for your testing booth, or maybe it's only participating in your experiment to get extra credit in class and isn't the least bit motivated. It does what it does according to fairly simple rules. Humans, on the other hand, are terrible test subjects. Psychology experiments require averaging over many, many observations in order to detect patterns within all that noise.

Science is about predictions. In theory, we'd like to predict what an individual person will do in a particular instance. In practice, we're largely in the business of predicting what the average person will do in an average instance. Obviously we'd like to make more specific predictions (and there are those who can and do), but they're still testable (and tested) predictions. The alternative is to declare much of human and animal behavior outside the realm of science.

Significant differences

There are some who are on board so far but get off the bus when it comes to how statistics are done in psychology. Usually an experiment consists of determining statistically whether a particular result was likely to have occurred by chance alone. Richard Feynman famously thought this was nuts (the thought experiment is that it's unlikely to see a license plate printed CPT 349, but you wouldn't want to conclude much from it).

That's missing the point. The notion of significant difference is really a measure of replicability. We're usually comparing a measurement across two populations. We may find population A is better than population B on some test. That could be because population A is underlyingly better at such tests. Alternatively, population A was lucky that day. A significant difference is essentially a prediction that if we test population A and population B again, we'll get the same results (better performance for population A). Ultimately, though, the statistical test is just a prediction (one that typically works pretty well) that the results will replicate. Ideally, all experiments would be replicated multiple times, but that's expensive and time-consuming, and -- to the extent that the statistical analysis was done correctly (a big if) -- largely unnecessary

So what do you think? Are the social sciences sciences? Comments are welcome.

Why is learning a language so darn hard (golden oldie)

2010-08-23T09:00:00.001-04:00

I work in an toddler language lab, where we study small children who are breezing through the process of language acquisition. They don't go to class, use note cards or anything, yet they pick up English seemingly in their sleep (see my previous post on this).

Just a few years ago, I taught high school and college students (read some of my stories about it here) and the scene was completely different. They struggled to learn English. Anyone who has tried to learn a foreign language knows what I mean.

Although this is well-known, it's a bit of mystery why. It's not the case that my Chinese students didn't have the right mouth shape for English (I've heard people -- not scientists -- seriously propose this explanation before). It's also not just that you can learn only one language. There are plenty of bilinguals out there. Jesse Snedeker (my PhD adviser as of Monday) and her students recently completed a study of cross-linguistic late-adoptees -- that is, children who were adopted between the ages of 2 and 7 into a family that spoke a different language from that of the child's original home or orphanage. In this case, all the children were from China. They followed the same pattern of linguistic development -- both in terms of vocabulary and grammar -- as native English speakers and in fact learned English faster than is typical (they steady caught up with same-age English-speaking peers).

So why do we lose that ability? One model, posited by Michael Ullman at Georgetown University (full disclosure: I was once Dr. Ullman's research assistant), has to do with the underlying neural architecture of language. Dr. Ullman argues that basic language processes are divided into vocabulary and grammar (no big shock there) and that vocabulary and grammar are handled by different parts of the brain. Simplifying somewhat, vocabulary is tied to temporal lobe structures involved in declarative memory (memory for facts), while grammar is tied to procedural memory (memory for how to do things like ride a bicycle) structures including the prefrontal cortex, the basal ganglia and other areas.

As you get older, as we all know, it becomes harder to learn new skills (you can't teach an old dog new tricks). That is, procedural memory slowly loses the ability to learn new things. Declarative memory stays with us well into old age, declining much more slowly (unless you get Alzheimer's or other types of dementia). Based on Dr. Ullman's model, then, you retain the ability to learn new words but have more difficulty learning new grammar. And grammar does appear to be the typical stumbling block in learning new languages.

Of course, I haven't really answered my question. I just shifted it from mind to brain. The question is now: why do the procedural memory structures lose their plasticity? There are people studying the biological mechanisms of this loss, but that still doesn't answer the question we'd really like to ask, which is "why are our brains constructed this way?" After all, wouldn't it be ideal to be able to learn languages indefinitely?

I once put this question to Helen Neville, a professor at the University of Oregon and expert in the neuroscience of language. I'm working off of a 4-year-old memory (and memory isn't always reliable), but her answer was something like this:

Plasticity means that you can easily learn new things. The price is that you forget easily as well. For facts and words, this is a worthwhile trade-off. You need to be able to learn new facts for as long as you live. For skills, it's maybe not a worthwhile trade-off. Most of the things you need to be able to do you learn to do when you are relatively young. You don't want to forget how to ride a bicycle, how to walk, or how to put a verb into the past tense.

That's the best answer I've heard. But I'd still like to be able to learn languages without having to study them.

originally posted 9/12/07

Tenure, a dull roar

2010-08-17T01:54:00.000-04:00

Slate ran an unfortunate, bizarre piece on tenure last week. FemaleScienceProfessor has a good take-down. Among problems, it repeats the claim that the average tenured professor costs the average university around $11,000,000 across his/her career -- a number that is either misleading, miscalculated, or (most likely) an outright lie. But, as FemaleScienceProfessor points out, tenure itself costs next to nothing, so anyone who says eliminating tenure will save money really means cutting professor salaries will save money but doesn't want to be on the record saying so.

If this seems like deja vu, it is. I just wrote a post about a similarly confused feature in the New York Times. That post is still worth reading (imho).

Which raises the question of why tenure is under attack. I have two guesses: 1) it's a way of ignoring the progressive defunding of public universities, or 2) part of the broader war on science. There are possibly a few people who genuinely think tenure is a bad idea, but not because eliminating it will save money (it won't), because it'll soften the publish-or-perish ethos (yes, the claim has been made), or because it'll refocus universities on teaching (absurd, irrelevant, and beside the point). Which leaves concerns about an inflexible workforce and the occasional dead-weight professor, but that's not on my list of top ten problems in education, and I don't think it should be on anyone else's -- there are bigger fish to fry.

Making data public

2010-08-16T09:00:00.000-04:00

Lately, there have been a lot of voices (e.g., this one) calling for scientists to make raw data immediately available to the general public. In the interest of answer than call, here's some of my raw data:

female	no	English	English	no	no	yes	United States	1148478773	312	0	helped	1	daxed	59	0	1	1
female	no	English	English	no	no	yes	United States	1148478773	312	1	heard	2	blied	33	0	0	2
female	no	English	English	no	no	yes	United States	1148478773	312	2	decelerated	2	lenked	45.4	1	0	2
female	no	English	English	no	no	yes	United States	1148478773	312	3	startled	1	gamped	31.1	1	0	3
female	no	English	English	no	no	yes	United States	1148478773	312	4	prompted	2	henterred	59	0	1	4
female	no	English	English	no	no	yes	United States	1148478773	312	5	engrossed	2	nazored	31.1	0	1	5
female	no	English	English	no	no	yes	United States	1148478773	312	6	obliged	1	ablined	59	1	0	6
female	no	English	English	no	no	yes	United States	1148478773	312	7	tantalized	2	bosined	31.1	1	1	7
female	no	English	English	no	no	yes	United States	1148478773	312	8	bled for	1	breened	31.3	1	1	8
female	no	English	English	no	no	yes	United States	1148478773	312	9	loathed	2	gaubled	31.2	0	0	9
female	no	English	English	no	no	yes	United States	1148478773	312	10	mourned for	1	ginked	31.3	1	1	10
female	no	English	English	no	no	yes	United States	1148478773	312	11	wounded	2	jarined	31.1	0	0	10

Do you feel enlightened? Probably not. Raw data isn't all that useful if you don't know how it was collected, what the different numbers refer to, etc. Even if I told you this is data from this experiment, that probably wouldn't help much. Even showing you the header rows for these data will help only so much:

sex

subject_already

nat_language

prime_language

autism

dyslexia

psychiatric

country

randomID

startTime

trial

word

choice

conclusion

wordClass

whichLocation

because

totalCorrect

Some things are straightforward. Some are not. It's important to know that I record data with a separate row for every trial, so each participant has multiple trials. Also, I record all data, even data from participants who did not complete the experiment. If you're unaware of that, your data analyses would come out very wrong. Also I have some codes I use to mark that the participant is an experimenter checking to make sure everything is running correctly. You'd need to know those. It's key to know how responses are coded (it's not simply "right" or "wrong" -- and in fact the column called totalCorrect does not record whether the participant got anything correct).

The truth is, even though I designed this study myself and wrote the program that outputs the data, every time I go back to data from a study I haven't worked with in a while, it takes me a few hours to orient myself -- and I'm actually relatively good about documenting my data.

So if a law were passed -- as some have advocated for -- requiring that data be made public, one of two things will happen: either people will post uninterpretable data like my mini-spreadsheet above, or they'll spend huge amounts of time preparing their data for others' consumption. The former will help no one. And the latter is expensive, and someone has to pay for that. And this all has to be balanced against the fact that there are very few data sets anyone would want to reanalyze.

There are important datasets that should be made available. And in fact there are already mechanisms for doing this (in my field, CHILDES is a good example). This kind of sharing should be encouraged, but mandated sharing is likely to cause more problems than it solves.

Anonymity

2010-08-10T09:00:00.000-04:00

It seems that most science bloggers use pseudonyms. To an extent, I do this, though it's trivial for anyone who is checking to figure out who I am (I know, since I get emails sent to my work account from people who read the blog). This was a conscious choice, and I have my reasons.

1. I suppose one would choose anonymity just in case one's blogging pissed off people who are in a position to hurt you. That would be mostly people in your own field. Honestly, I doubt it would take anyone in my field long to figure out what university I was at. Like anyone, I write most about the topics my friends and colleagues are discussing, and that's a function of who my friends and colleagues are.

(In fact, a few years ago, someone I knew was able to guess what class I was taking, based on my blog topics.)

2. I write a lot about the field, graduate school, and the job market. But within academia, every field is different. For that matter, even if you just wanted to discuss graduate student admission policy within psychology, the fact is that there is a huge amount of variation from department to department. So I can really only write about my experiences. For you to be able to use that information, you have a have a sense of what kind of school I'm at (a large, private research university) and in what field (psychology).

I read a number of bloggers who write about research as an institution, about the job market, etc., but who refuse to say what field they're in. This makes it extremely difficult to know what to make of what they say.

For instance, take my recent disagreement with Prodigal Academic. Prodigal and some other bloggers were discussing the fact that few people considering graduate school in science know how low the odds of getting a tenure-track job are. I suggested that actually they aren't misinformed about academia per se, but about the difference between a top-tier school and even a mid-tier school. I point out that at a top-tier psychology program, just about everybody who graduates goes on to get a tenure-track job. Prodigal says that in her field, at least, that's not true (and she suspects it's not true in my field, either).

The difference is that you can actually go to the websites of top psychology programs and check that I'm right. We can't do the same for Prodigal, because we have no idea what field she's in. We just have to take her word for it.

3. I suspect many people choose pseudonyms because they don't want to censor what they say. They don't want to piss anybody off. I think that to maintain my anonymity, I would have to censor a great deal of what I say. For one thing, I couldn't blog about the one thing I know best: my own work.

There is the risk of pissing people off. And trust me, I worry about it. But being careful about not pissing people off is probably a good thing, whether you're anonymous or know. Angry people rarely change their minds, and presumably we anger people precisely when we disagree with them.

--------

So why don't I actually blog under my name? I want people who Google me by name to find my academic website and my professional work first, not the blog.

Joining Twitter. Sigh.

2010-08-09T10:56:00.001-04:00

The last few weeks I've been making some changes at this blog. One is to write fewer but higher-quality posts. Hopefully you noticed the latter and not just the former. At the same time, I have been finding more and more articles and posts that demand sharing, but about which I have little or nothing to say, except that you should read it. This has led me to add a twitter feed above the posts. You can read there or follow directly.

We'll see how it goes. Feedback is welcome. After all, I do this for the audience.

UPDATED

Another change: This blog is *relatively* new to FieldOfScience, but posts go back to 2007. Some of those older posts are worth revisiting, and I'll be reposting (occasionally with updates) a few of the better ones from time to time under the label "golden oldies". Again, if people having feelings about this, let me know.

1/3 of Americans can't speak?

2010-08-04T09:22:00.000-04:00

A number of people have been blogging about a recent, still unpublished study suggesting that "a significant proportion of native English speakers are unable to understand some basic sentences." Language Log has a detailed explanation of the methods, but in essence participants were asked to match sentences to pictures. A good fraction made large numbers of mistakes, particularly those who had been high-school drop-outs.

What's going on here? To an extent, this shouldn't be that surprising. We all know there are people who regularly mangle language. But, as Mark Liberman at Language Log points out, at least some of these data are no doubt ascribable to the "paper airplane effect":

At one point we thought we had discovered that a certain fraction of the population is surprisingly deaf to certain fairly easy speech-perception distinctions; the effect, noted in a population of high-school-student subjects, was replicable; but observing one group of subjects more closely, we observed that a similar fraction spent the experiment surreptitiously launching paper airplanes and spitballs at one another.

It's worth remembering that, while many participants in an experiment take it seriously and are happy to help out the researcher, some are just there for the money they get paid. Since we're required to pay people whether they pay attention to the experiment or not, they really don't have any incentive to try hard. Does it surprise anyone that high-school drop-outs are particularly likely to be bad at/uninterested in taking tests?

It's probably relevant that the researchers involved in this study are linguists. There are some linguists who run fabulous experiments, but as a general rule, linguists don't have much training in doing experiments or much familiarity with what data looks like. So it's not surprising that the researchers in question -- and the people to whom they presented the data -- weren't aware of the paper airplane effect.

(I should say that psychology is by no means immune to this problem. Whenever a new method is adopted, it takes a while before there's a critical mass of people who really understand it, and in the meantime a lot of papers with spurious conclusions get written. I'm thinking of fMRI here.)

Honestly, Research Blogging, Get over yourself

2010-08-02T08:00:00.000-04:00

A few years ago, science blog posts started decorating themselves with a simple green logo. This logo was meant to credential the blog post as being one about peer-reviewed research, and is supplied by Research Blogging. As ResearchBlogging.org explains:

ResearchBlogging.org is a system for identifying the best, most thoughtful blog posts about peer-reviewed research. Since many blogs combine serious posts with more personal or frivolous posts, our site offers away to find only the most carefully-crafted about cutting-edge research, often written by experts in their respective fields.

That's a good goal and one I support. If you read further down, you see that this primarily amounts to the following: if the post is about a peer-reviewed paper, it's admitted to the network. If it's not, it isn't. I guess the assumption is that the latter is not carefully-crafted or about cutting-edge research. And that's where I get off the bus.

Peer Review is Not Magic

One result of the culture wars is that scientists have needed a way of distinguishing real data from fantasy. If you look around the Internet, no doubt half or even more than half of what is written suggests there's no global warming, that vaccines cause autism, etc. Luckily, fanatics rarely publish in peer-reviewed journals, so once we restrict the debate to what is in peer-reviewed journals, pretty much all the evidence suggests global warming, no autism-vaccine link, etc. So pointing to peer-review is a useful rhetorical strategy.

That, at least, is what I assume has motivated all the stink about peer-review in recent years, and ResearchBlogging.org's methods. But it's out of place in the realm of science blogs. It's useful to think about what peer review is.

A reviewer for a paper reads the paper. The reviewer does not (usually) attempt to replicate the experiment. The reviewer does not have access to the data and can't check that the analyses were done correctly. At best, the reviewer evaluates the conclusions the authors draw, and maybe even criticizes the experimental protocol or the statistical analyses used (assuming the reviewers understand statistics, which in my field is certainly not always the case). But the reviewer doesn't can't check that the data weren't made up, that the experimental protocol was actually followed, that there were no errors in data analysis, etc.

In other words, the reviewer can do only and exactly what a good science blogger does. So good science blogging is, at its essence, a kind of peer review.

Drawbacks

Now, you might worry about the fact that the blogger could be anyone. There's something to that. Of course, ResearchBlogging.org has the same problem. Just because someone is blogging about peer-reviewed paper doesn't mean they understand it (or that they aren't lying about it, which happens surprisingly often with the fluoride fanatics).

So while peer review might be a useful way of vetting the paper, it won't help us vet the blog. We still have to do that ourselves (and science bloggers seem to do a good job of vetting).

A weakness

Ultimately, I think it's risky to put all our cards on peer review. It's a good system, but its possible to circumvent. We know that some set of scientists read the paper and thought it was worth publishing (with the caveats mentioned above). Of course, those scientists could be anybody, too -- it's up to the editor. So there's nothing really stopping autism-vaccine fanatics from establishing their own peer-reviewed journal, with reviewers who are all themselves autism-vaccine fanatics.

To an extent, that already happens. As long as there's a critical mass of scientists who think a particular way, they can establish their own journal, submit largely to that journal and review each other's submissions. Thus, papers that couldn't have gotten published at a more mainstream journal can get a home. I think anyone who has done a literature search recently knows there are a lot of bad papers out there (in my field, anyway, though I imagine the same is true in others).

Peer review is a helpful vetting process, and it does make papers better. But it doesn't determine fact. That is something we still have to find for ourselves.

****
Observant readers will have noticed that I use ResearchBlogging.org myself for it's citation system. What can I say? It's useful.

Help! I need data!

2010-07-30T14:15:00.000-04:00

Data collection keeps plugging along at GamesWithWords.org. Unfortunately, as usual, it's not the experiments for which I most need data that get the most traffic. Puntastic had around 200 participants in the last month. I'd like to get more than that, and I'd like to get more than that in all my experiments. But if I had to choose one to get 200 participants, it would be The Video Test, which only got 17.

The Video Test is the final experiment in a series that goes back to 2006. We submitted a paper in 2007, which was rejected. We did some follow-up experiments and resubmitted. More than once. Personally, I think we've simply had bad luck with reviewers, since the data are pretty compelling. Anyway, we're running one last monster experiment, replicating all our previous conditions several which-ways. It needs about 400 participants, though for really beautiful data I'd like about 800. We've got 140.

As I said, recruitment has been slow for this experiment.

So... if you have never done this experiment before (it involves watching a video and taking a memory test), please do. I'd love to get this project off my plate.

I liked "Salt," but...

2010-07-28T20:21:00.001-04:00

What's with movies in which fMRI can be done remotely. In an early scene, the CIA do a remote brain scan of someone sitting in a room. And it's fully analyzed, too, with ROIs shown. I want that technology -- it would make my work so much easier!

UPDATE I'm not the only one with this complaint. Though Popular Mechanics goes a bit easy on the movie by saying fMRI is "not quite at the level Salt portrays." That's a bit like saying space travel is not quite at the level Star Trek portrays. There may someday be a remote brain scanner, but it won't be based on anything remotely like existing fMRI technology, which requires incredibly powerful, supercooled and loud magnets. Even if you solved the noise problems, there's nothing to be done about the fact that the knife embedded in the Russian spy's shoe (yes -- it is that kind of movie) would have gone flying to the center of the magnetic field, along with many of the other metal objects in the room.

What are the best cognitive science blogs?

2010-07-27T22:12:00.000-04:00

If you look to your right, you'll see I've been doing some long-needed maintenance to my blog roll. As before, I'm limiting it to blogs that I actually read (though not all the blogs I read), and I have it organized by subject matter. As I did this, I noticed that the selection of cognitive science and language blogs is rather paltry. Most of the science blogs I read -- including many not included in the blog rolls -- are written by physical scientists.

Sure there are more of them than us, but even so it seems there should be more good cognitive science and language blogs. So I'm going to crowd-source this and ask you, dear readers, who should I be reading that I'm not?

Language Games

2010-07-22T12:55:00.000-04:00

Translation Party

Idea: type in sentence in English. The site then queries Google Translator, translating into Japanese and then back again until it reaches "equilibrium," where the sentence you get out is the sentence you put in. Some sentences just never converge. Ten points to whoever finds the most interesting non-convergence.

Sounds of Silence

2010-07-21T09:00:00.001-04:00

My lament that, with regards to discussion of education reform, a trace of small liberal arts colleges has disappeared into the ether appears to have, itself, disappeared into the ether. Seriously, readers, I expected some response to that one. There are parts of my post even I disagree with.

No tenure, no way!

2010-07-20T11:42:00.000-04:00

The New York Times is carrying an interesting but misguided discussion of tenure today. As usual, the first commentator warns that without tenure, academic freedom will die:

As at-will employees, adjunct faculty members can face dismissal or nonrenewal when students, parents, community members, administrators, or politicians are offended at what they say. If you can be fired tomorrow, you do not really have academic freedom. Self-censorship often results.

Mark Taylor of Columbia replies, essentially, "oh yah?"

To those who say the abolition of tenure will make faculty reluctant to be demanding with students or express controversial views, I respond that in almost 40 years of teaching, I have not known a single person who has been more willing to speak out after tenure than before.

Instead, tenure induces stasis, a point to which Richard Vedder, an economist at Ohio University, agrees:

The fact is that tenured faculty members often use their power to stifle innovation and change.

Money

You might, reading through these discussions, almost think that universities have been slowly doing weakening the tenure system because they want to increase diversity, promote a flexible workforce, and reduce the power of crabby old professors. Maybe some administrators do feel that way. But lurking behind all of this discussion is money. Here's Taylor:

If you take the current average salary of an associate professor and assume this tenured faculty member remains an associate professor for five years and then becomes a full professor for 30 years, the total cost of salary and benefits alone is $12,198,578 at a private institution and $9,992,888 at a public institution.

I'm not sure where he's getting these numbers. The numbers at Harvard for the same period is $6,320,500 for salary alone. Assuming benefits cost as much as the salary alone gets us up to our $12,000,000, but that's for Harvard, not the average university. Perhaps Taylor is assuming the professor starts today and includes inflation in future salaries, but 35 years of inflation is a lot. I'm using present-day numbers and assuming real salaries remain constant.

In any case, money seems to be the real factor, mentioned by more or less all the contributors. Here's Vedder:

My academic department recently granted tenure to a young assistant professor. In so doing, it created a financial liability of over two million dollars, because it committed the institution to providing the individual lifetime employment. With nearly double digit unemployment and universities furloughing and laying off personnel, is tenure a luxury we can still afford?

Adrianna Kezar of USC notes that non-tenured faculty are often not given offices or supplies, which presumably also saves the university money.

Professors make choices, too.

So universities save a lot of money by eliminating tenure. And certainly universities need to find savings where they can. What none of the contributors to the discussion acknowledge, beyond an oblique aside by Vedder, is that tenure has a financial value to professors as well as universities. Removing tenure in a sense is a pay cut, and both present and potential academics will respond to that pay cut.

Becoming a professor is not a wise financial decision. The starting salary of a lawyer leaving a top law school is greater than what most PhDs from the same schools will make at the height of their careers should they stay in academia. And lawyers' salaries, as I'm often reminded, can be similarly dwarfed by people with no graduate education that go straight into finance.

Most of us who nonetheless go into academia do so because we love it. The point is that we have options. Making the university system less attractive will mean fewer people will want to go into it. It's really that simple.

Garbage in, Garbage out

2010-07-20T10:46:00.000-04:00

While watching television, have you ever had a fatal heart attack?

If you answered "yes" to this question, you would have been marked as a "bad participant" in Experimental Turk's recent study. The charitable assumption would be that you weren't paying attention. Importantly for those interested in using Amazon Mechanical Turk for research, participants recruited through AMT were no more likely to answer "yes" than participants tested in a traditional lab-based setting (neither group was likely to say "yes").

It's a nice post, though I think that Experimental Turk's analysis is over-optimistic, for reasons that I'll explain below. More interesting, though, is that Experimental Turk apparently does not always include such catch trials in their experiments. In fact, they find the idea so novel that they actually cited a 2009 paper from the Journal of Experimental Social Psychology that "introduces" the technique -- which means the editors and reviewers at this journal were similarly impressed with the idea.

That's surprising.

Always include catch trials

Including catch trials is often taught as basic experimental method, and for good reason. As Experimental Turk points out, you never know if your participants are paying attention. Inevitably, some aren't -- participants are usually paid or given course credit for participation, so they aren't always very motivated. Identifying and excluding the apathetic participants can clean up your results. But that's not the most important reason to include catch trials.

Even the best participant may not understand the instructions. I have certainly run experiments in which the majority of the participants interpreted the instructions differently from how I intended. A good catch trial is designed such that the correct answer can only be arrived at if you understand the instructions. It is also a good way of making sure you're analyzing your data correctly -- you'd be surprised how often a stray negative sign worms its way into analysis scripts.

Sometimes participants also forget instructions. In a recent study, I wasn't finding a difference between the control and experimental groups. I discovered in debriefing that most of the participants in the experimental group had forgotten the key instruction that made the experimental group the experimental group. No wonder there wasn't a difference! And good thing I asked.

The catch trial -- the question with the obvious answer -- is just one tool in a whole kit of tricks used to validate one's results. There are other options, too. In reading studies, researchers often ask comprehension questions -- not because the answers themselves matter (the real interest is in what the participants do while reading), but simply to prove that the participants in fact did read and understand the material.

Similar is the embedded experiment -- a mini experiment embedded into your larger experiment, the only purpose of which is to replicate a well-established result. For instance, in a recent experiment I included a vocabulary test (which you can also find in this experiment I'm running with Laura Germine at TestMyBrain.org). I also asked the participants for their SAT scores (these were undergraduates), not because I cared about their scores per se, but I was able to show that their Verbal SAT scores correlated nicely with performance on the vocabulary test (Math SAT scores less so), helping to validate the our vocab test.

Beyond Surveys

Although I described catch trials mostly in terms of survey-format studies, the same techniques can be embedded into nearly any experiment. I've used them for reading-time, eye-tracking and ERP experiments as well. The practice isn't even specific to psychology/cognitive science. During my brief sojourn in a wet lab in high school, my job was to help genotype genetic knock-out mice to make sure that the genes in question really were missing from the relevant mice and not from the control mice. It probably wouldn't have occurred to the PIs in that lab to just assume the knock-out manipulation worked. Fail that, and none of the rest of the experiment is interpretable.

A version of the catch trial is even seen in debugging software, where the programmer inserts code that isn't relevant to the function of the program per se, but the output of which helps determine whether the code is doing what it's supposed to.

It is true that some experiments resist checks of this sort. I have certainly run experiments where by design I couldn't easily confirm that the participants understood the experiment, were paying attention, etc. But that is better avoided if possible -- which is why when I don't see such checks in an experimental write-up, I assume either (a) the checks were performed but deemed too unimportant/obvious to mention, or (b)

An Odd Omission

If catch trials are a basic aspect of good experimental design, how is it that Experimental Turk and the Journal of Experimental Social Psychology didn't know about it? I'm not sure. Part of it may be due to how experimental design is taught. It's not something you look up in an almanac, and though there are classes on technique (at least in psychology departments), they aren't necessarily that helpful since there are hundreds of types of experiments out there, each of which has its own quirks, and a class can only cover a few.

At least in my experience, experimental design is learned through a combination of the apprenticeship method (working with professors -- or, more often, more experienced graduate students) and figuring it out for yourself. The authors at Experimental Turk, it turns out, come from fields relatively new to experimental design (business, management, and political science), so it's possible they had less access to such institutional knowledge.

As far as the Journal of Experimental Social Psychology... I'm not a social psychologist, and I hesitate to generalize about the field. A lot of social psychology uses questionnaires as instruments. They go to a great deal of difficulty to validate the questionnaires -- show that they are predictive of results on other tests or questionnaires, show that the questionnaires have good test-retest reliability, etc. Many of the techniques they use are ones I would like to learn better. But I actually haven't ever run across one (again, in my limited experience) that actually includes catch trials. Which in itself is interesting.

A clever idea

I should add that while Experimental Turk cites said journal article for suggesting using questions with obvious answers, that's not actually what the paper suggests. Rather, it suggests using instructions telling participants to ignore certain questions. For instance:

Sports Participation

Most modern theories of decision making recognize the fact that decisions do not take place in a vacuum. Individual preferences and knowledge, along with situational variables can greatly impact the decision process. In order to facilitate our research on decision making we are interested in knowing certain factors about you, the decision maker. Specifically, we are interested in whether you actually take the time to read the directions; if not, then some of our manipulations that rely on changes in the instructions will be ineffective. So, in order to demonstrate that you have read the instructions, please ignore the sports item below, as well as the continue button. Instead, simply click on the title at the top of this screen (i.e., "sports participation") to proceed to the next screen. Thank you very much.

That's a clever idea. One of my elementary school teachers actually wrote a whole test with instructions like that to teach the class a lesson about reading instructions carefully (and it worked -- I still do!). So it's a good idea I've never seen used in an experimental setting before, but that doesn't mean it hasn't been used. In any case, the discussion in the paper doesn't mention catch trials or other methods of validating data, so it's hard to know whether they did a thorough literature search.

More training

A bad movie can still make entertaining watching. A bad experiment is irredeemable. If the participants didn't understand the instructions, nothing can be gleaned from the data. And there are so many ways to run bad experiments -- I know, because I've employed many of them myself. There are a lot of datasets out there in psychology that have proven, shall we say, resistant to replication. Some of this has to be due to the fact that experimental design is not as good as it could and should be.

Addendum

As I mentioned higher up, I think Experimental Turk is overly optimistic about the quality of data from AMT. I've run a couple dozen experiments on AMT now, and the percentage of participants that fail the catch trials varies a great deal, from as few as 0% to as many as 20-30%. I haven't made a systematic study of it, but there seem to be a number of contributing factors, some of which are general to all experimental venues (length of the experiment, how interesting it is, how complicated in the instructions are) and some of which are specific to AMT (the more related HITs, the more attractive a target the experiment is to spammers).

All the more reason to always include catch trials.

-----------
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power Journal of Experimental Social Psychology, 45, 867-872

Sell off Harvard Medical School!

2010-07-18T08:14:00.001-04:00

Writing in the Chronicle of Higher Education, Andrew Hacker and Claudia Dreifus contend

Colleges are taking on too many roles and doing none of them well. They are staffed by casts of thousands and dedicated to everything from esoteric research to vocational training—and have lost track of their basic mission to challenge the minds of young people... Spin off medical schools, research centers, and institutes... For people who want to do research, plenty of other places exist—the Brookings Institution, the Rand Corporation, the Howard Hughes Medical Institute—all of which do excellent work without university ties.

Never mind that Howard Hughes is intimately tied to the present university system, let's say we're in favor: sell off Harvard Medical School, Harvard Law School, Harvard Kennedy School, etc., until all that's left is the College. That'd make it what? -- Wellesley + men? (This question is meant to be snarky, but not anti-Wellesley, for which I have the utmost respect, as will be clear in the rest of the post).

It's the money, stupid.

The blogosphere has been rising to the defense of the research university, with posters and commenters focusing on the (alleged) claim that universities use research dollars to fund the loss-leading undergraduate programs. Here's Mike the Mad Biologist:

[O]n a federal grant, usually somewhere between 30-40% of the total grant award doesn't go to the researcher for research costs (salaries, supplies, etc.), but to the institution. Now, some of that money is spent on actual administrative costs, but the rest goes to the university*. So if the university spins off $50 million, or $100, or, in the case of the University of Iowa, $169,175,021 of NIH funding alone (never mind other government sources), that's tens of millions of dollars that have to be recovered. Since I've called for more of a research institute model, I'm not opposed to spinning off research institutes. But I have no idea how universities that receive a lot of research dollars will make up the revenue shortfall.

There's an easy way of answering the question: write to any of the numerous, high-calliber exclusively-undergraduate institutions that makes the American education system so interesting: Wellesley, Swarthmore, Amherst, Grinnell, Oberlin, etc. For the last 150-200 years, such schools have focused on teaching, and teaching caliber is weighted heavily in tenure decisions. I had phenomenal professors. To name a few, Arlene Forman could have taught a turnip to speak Russian, and Jim Walsh delivered spellbinding lectures despite unpromising subject material (e.g., linear algebra). People who had never even attended Ron DiCenzo's classes nonetheless raved about the vicarious experience.

Research University vs. Liberal Arts College

I loved the small liberal arts college experience and wouldn't have traded it for anything. But I have friends who feel the same way about the large research university: the inspirational presence of movers and shakers in the research world, they feel, is irreplaceable. I'm skeptical, but the great thing about the American education system is that it provides both options, something that many (all?) other countries lack. The only distressing thing is that so many students -- along with the Chronicle of Higher Education and essentially every blogger I read and all their commenters -- seem completely unaware that an alternative to the research university exists.

America has research-only institutes. It has undergraduate-only schools. And it has that fabulous hybrid institution: the research university. Arguing that we need to start founding undergraduate-only schools is like saying America really needs subways. Maybe we need more subways (I think we do!), but claiming they don't exist is just ignant, and it's an insult to the ones that exist and the people who made them possible.

Confusing verbs

2010-07-08T16:17:00.000-04:00

The first post on universal grammar generated several good questions. Here's an extended response to one of them:

You said that a 1990's theory was dead wrong because sometimes emotion verbs CAN be prefixed with -un. Then you give examples of adjectives, not verbs, that have been prefixed: unfeared, unliked, unloved. I know these words are also sometimes used as verbs, but in the prefixed versions they are clearly adjectives.

The theory I'm discussing wanted to distinguish between emotion verbs which have experiencers as subjects (fear, like) and those who have experiencers as objects (frighten, confuse). The claim was that the latter set of verbs are "weird" in an important way, one effect of which is that they can't have past participles.

This brings up the obvious problem that "frighten" and "confuse" appear to have past participles: "frightened" and "confused". The author then argued that these are not actually past participles -- they're adjectives. The crucial test is that you can add "un" to an adjective but not a participle (or so it's claimed). Thus, it was relevant that you can say "unfrightened" and "unconfused", suggesting that these are adjectives, but you can't say "unfeared" or "unliked", suggesting that these are participles, not adjectives.

The problem mentioned in the previous post was that there are also subject-experiencer verbs that have participles which can take the "un" prefix, such as "unloved". There are also object-experiencer verbs that have participles which can't be "un" prefixed, like "un-angered" (at least, it sounds bad to me; try also "ungrudged", "unapplauded", or "unmourned"). So the "un" prefixation test doesn't reliably distinguish between the classes of verbs. This becomes apparent once you look through a large number of both types of verbs (here are complete lists of subject-experiencer and object-experiencer verbs in English).

There is a bigger problem, which is that the theory assumes a lack of homophones. That is, there could be two words pronounced like "frightened" -- one is a past participle and one is an adjective. The one that can be unprefixed is the adjective. So the fact that "unfrightened" exists as a word doesn't rule out the possibility that "frighten" has a past participle.

To be completely fair to the theory, the claim that object-experiencer verbs are "weird" (more specifically, that they require syntactic movement) could be still be right (though I don't think it is). The point here was that the specific test proposed ("un" prefixation) turned out to provide different results. It actually took some time for people to realize this, and you still see the theory cited. The point is that getting the right analysis of a language is very difficult, and typically many mistakes are made along the way.

Universal meaning

2010-07-07T11:00:00.000-04:00

My earlier discussion of Evans and Levinson's critique of universal grammar was vague on details. Today I wanted to look at one specific argument.

Funny words

Evans and Levinson briefly touch on universal semantics (variously called "the language of thought" or "mentalese"). The basic idea is that language is a way of encoding our underlying thoughts. The basic structure of those thoughts is the same from person to person, regardless of what language they speak. Quoting Pinker, "knowing a language, then, is knowing how to translate mentalese into strings of words and vice versa. People without a language would still have mentalese, and babies and many nonhuman animals presumably have simpler dialects."

Evans and Levinson argue that this must be wrong, since other languages have words for things that English has no word for, and similarly English has words that don't appear in other languages. This is evidence against a simplistic theory on which all languages have the same underlying vocabulary and differ only on pronunciation, but that's not the true language of thought hypothesis. Many of the authors cited by Evans and Levinson -- particularly Pinker and Gleitman -- have been very clear about the fact that languages pick and choose in terms what they happen to encode into individual words.

The Big Problems of Semantics

This oversight was doubly disappointing because the authors didn't discuss the big issues in language meaning. One classic problem, which I've discussed before on this blog, is the gavagai problem. Suppose you are visiting another country where you don't speak a word. Your host takes you on a hike, and as you are walking, a rabbit bounds across the field in front of your. Your host shouts "gavagai!" What should you think gavagai means?

There are literally an infinite number of possibilities, most of which you probably won't consider. Gavagai could mean "white thing moving," or "potential dinner," or "rabbit" on Tuesdays but "North Star" any other day of the week. Most likely, you would guess it means "rabbit" or "running rabbit" or maybe "Look!" This is a problem to solve, though -- given the infinite number of possible meanings, how do people narrow down on the right one?

Just saying, "I'll ask my host to define the word" won't work, since you don't know any words yet. This is the problem children have, since before explicit definition of words can help them learn anything, they must already have learned a good number of words.

One solution to this problem is to assume that humans are built to expect words of certain sorts and not others. We don't have learn that gavagai doesn't change it's meaning based on the day of the week because we assume that it doesn't.

More problems

That's one problem in semantics that is potentially solved by universal grammar, but not the only. Another famous one is the linking problem. Suppose you hear the sentence "the horse pilked the bear". You don't know what pilk means, but you probably think the sentence describes the horse doing something to the bear. If instead you find out it describes a situation in which the bear knocked the horse flat on its back, you'd probably be surprised.

That's for a good reason. In English, transitive verbs describe the subject doing something to the object. That's not just true of English, it's true of almost every language. However, there are some languages where this might not be true. Part of the confusion is that defining "subject" and "object" is not always straightforward from language to language. Also, languages allow things like passivization -- for instance, you can say John broke the window or The window was broken by John. When you run into a possible exception to the subject-is-the-doer rule, you want to make sure you aren't just looking at a passive verb.

Once again, this is an example where we have very good evidence of a generalization across all languages, but there are a few possible exceptions. Whether those exceptions are true exceptions or just misunderstood phenomena is an important open question.

-------
Evans, N. and Levinson, S. (2009). The myth of language universals: Language diversity and its importance for cognitive science Behavioral and Brain Sciences, 32 (05) DOI: 10.1017/S0140525X0999094X

photo credit

Do Language Universals Exist?

2010-07-06T09:30:00.023-04:00

Is there an underlying structure common to all languages? There are at least two arguments in favor of that position. One is an in principle argument, and one is based on observed data.

Since Chomsky, many researchers have noted that language would be impossible to learn if one approached it without preconceptions. It's like solving for 4 variables with only 3 equations -- for those of you who have forgotten your math, that can't be done. Quine pointed out the problem for semantics, but the problem extends to syntax.

The data-driven argument is based on the observation that diverse languages share many properties. All languages, it is claimed, have nouns and verbs. All languages have consonants and vowels. All languages put agents (the do-ers; Jane in Jane broke the window) in subject position and patients (the do-ees; the window in Jane broke the window) in object position. And so on. (Here's an extensive list.)

Though many researchers subscribe to this universal grammar hypothesis, it has always been controversial. Last year, Evans and Levinson published an extensive refutation of the hypothesis in Behavioral and Brain Sciences. They don't tackle the in principle argument (it's actually tough to argue against, since it turns out to be logically necessary), but they do take issue with the data-based argument.

Rare languages

Evans and Levinson point out that at best 10% of the world's 7,000 or so languages have been studied in any great detail, and that the bulk of all work on language has focused on English. They claim that researchers only believe in linguistic universals because they've only looked at a relatively small number of often closely-related languages, and they bring up counter-examples to proposed universals found in obscure languages.

This argument cuts both ways. The correct characterization of a language is very, very hard. Much of the work I have been doing lately has been an attempt to correctly characterize the semantics of about 300 related verbs in English. Hundreds of papers have been written about these verbs over the last half-century. Many of them have turned out to be wrong -- not because the researchers were bad, but because the problem is hard.

That's 300 verbs in the most-studied language on the planet, and we still have work to do. Evans and Levinson are basing their arguments on broad-scale phenomena in extremely rare, poorly-studied languages.

A friend of a friend told me...

The rare languages that Evans and Levinson make use of are not -- as they readily acknowledge -- well-understood. In arguing against recursion as a linguistic universal, they bring up Piraha, a language spoken in a handful of villages deep in the Amazon. Without discussing recursion in detail, the basic claim is that there are sentences that are ungrammatical in Piraha, and these sentences are ungrammatical because they require recursion.

To my knowledge, there is one Spanish-Piraha bilingual speaker, in addition to two English-speaking missionaries who, as adults, learned Piraha. The claim that Piraha doesn't have recursion is based on the work of one of those missionaries. So the data that sentences with recursion are ungrammatical in Piraha is based on a limited number of observations. It's not that I don't trust that particular researcher -- it's that I don't trust any single study (including my own), because it's easy to make mistakes.

Looking back at English, I study emotion verbs in which the subject of the verb experiences an emotion (e.g., fear, like, love). A crucial pillar of one well-known theory from the 1990s was that such verbs can't be prefixed with "un". That is, English doesn't have the words unfeared or unliked. While I agree that these words sound odd, a quick Google search shows that unfeared and unliked are actually pretty common. Even more problematic for the theory, unloved is a perfectly good English word. In fact, many of these verbs do allow "un" prefixation. The author, despite being an experienced researcher and a native speaker of English, was just wrong.

Even assuming that you are correct in claiming that a certain word or sentence doesn't appear in a given language, you could be wrong about why. Some years ago, Michael Tomasello (and others) noticed that certain constructions are more rare in child speech than one might naively expect. He assumed this was because the children didn't know those constructions were grammatical. For instance, in inflected languages such as Spanish or Italian, young children rarely use any verbs in all possible forms. A number of people (e.g., Charles Yang) have pointed out that this assumes that the children would want to say all those words. Take a look at this chart of all the forms of the Spanish verbs hablar, comer and vivir. The child might be excused for never using the form habriamos hablado ("we would have spoken") -- that doesn't mean she doesn't know what it is.

In short, even in well-studied languages spoken by many linguists, there can be a lot of confusion. This should give us pause when looking at evidence from a rare language, spoken by few and studied by fewer.

Miracles are unlikely, and rare

Some centuries ago, David Hume got annoyed at people claiming God must exist, otherwise how can you explain the miracles recorded in the Bible? Hume pointed out that by definition, a miracle is something that is essentially impossible. As a general rule, seas don't part, water doesn't turn into wine, and nobody turns into pillars of salt. Then consider that any evidence you have that a miracle did in fact happen could be wrong. If a friend tells you they saw someone turn into a pillar of salt, they could be lying. If you saw it yourself, you could be hallucinating. Hume concludes that however strong your evidence that a miracle happened is, that could never be as strong as the extreme unlikelihood of a miracle actually happening -- and, in any case, the chance that the Bible is wrong is way higher than the chance that Moses in fact did part the Sea of Reeds.

(For those of you who are worried, this isn't necessarily an argument against the existence of God, just an argument against gullibility.)

Back to the question of universals. Let's say you have a candidate linguistic universal, such as recursion, that has shown up in a large number of unrelated and well-studied languages. These facts have been verified by many, many researchers, and you yourself speak several of the languages in question. So the evidence that this is in fact a linguistic universal is very strong.

Then you come across a paper that claims said linguistic universal doesn't apply in some language X. Either the paper is right, and you have to toss out the linguistic universal, or it's wrong, and you don't. Evans and Levinson err on the side of tossing out the linguistic universal. Given the strength of evidence in favor of some of these universals, and the fact that the counter-examples involve relatively poorly-understood languages, I think one might rather err on the other side. As they say, extraordinary claims require extraordinary evidence.

The solution

Obviously, the solution is not to say something about "extraordinary claims" and wander on. Evans and Levinson's paper includes a plea to researchers to look beyond the usual suspects and start doing more research on distant languages. I couldn't agree more, particularly as many of the world's language are dying and the opportunity to study them is quickly disappearing.

-------
Evans, N. and Levinson, S. (2009). The myth of language universals: Language diversity and its importance for cognitive science Behavioral and Brain Sciences, 32 (05) DOI: 10.1017/S0140525X0999094X

Friends of science in government

2010-07-01T09:00:00.000-04:00

The Democratic congress continues its support of American basic research. The House subcommittee recommended a 7.2% increase for NSF in the coming year, despite general belt-tightening hysteria. It's not as much as is needed, but it's still a nice change from 2001-2008. Hopefully it'll survive the rest of the legislative process.

Caveat emptor: Is academia a pyramid scheme?

2010-06-23T13:13:00.000-04:00

That's the question on the blogs this week (see here and here). The question arises because each professor will have some number of students during their career (10-20 is common among the faculty I know), whereas the number of professorships increases very slowly. So the number of PhDs being produced far exceeds the number of academic positions.

As pointed out elsewhere, this neglects the fact that many PhD students have no intention of going into academia. Even so, it's clear the system is set up to produce more graduates who want academic jobs than there are jobs available. Prodigal Academic wonders if that's any different from any profession -- generally, there are more people who want the best jobs than there are best jobs to go around. Unlike PA, who doesn't think there's a problem, Citation Needed thinks most people entering graduate school aren't aware of how unlikely it is that they will get a tenure-track job, partly because it isn't in the schools' interest to mention this.

It depends

I largely agree with these fine posts, but I think they overgeneralize. Not all PhD programs are the same. Different fields vary wildly in terms of number of students produced, the likelihood of getting an industry job, etc., and also in terms of the caliber of the program. For instance, nearly every graduate of the psych program at Harvard goes on to get a tenure-track job. A sizable percentage get tenure-track jobs at the top institutions (Harvard, Yale, UChicago, etc.).

On the other hand, at even highly-respected but lower-ranked schools, getting a tenure-track job seems to be the exception. Here I have less personal experience, but a friend from Harvard who is a post-doc at a well-known state school was surprised to discover basically none of the students in that program expected to get an academic job. I've heard similar stories from a few other places.

A common problem

This isn't unique to academia. Many people believe lawyers earn a lot of money. Much fuss is made in the New York Times about how starting salary at a major law first is around $170,000/year (or was, prior to the Great Recession). While basically anyone who graduates from the top three law schools who wants such a job can get one (some go into lower-paying public-interest or public-service work), at most law schools, few if any graduates land such jobs and most lawyers never earn anywhere near that money. As a first approximation, nobody who graduates from law school lands a big firm job, just as, as a first approximation, nobody with a PhD gets a tenure track job at a top research institution.

From my vantage point, the problem is that media (newspapers, movies, etc.) fixate on the prosperous tip of the iceberg. Newspapers do this because their target audience (rather, the target audience of many of the advertisers in newspapers) are people who themselves graduated from Harvard or Yale and for whom getting a tenure-track job or being partner at a major law firm is a reasonably common achievement. Movies and television shows do this for the same reason everyone is beautiful and rich on the screen -- nobody ever said Hollywood was realistic.

This is fine as it goes, but can get people into trouble when they don't realize (a) that the media is presenting the outliers, not the norm, and/or (b) just where their own school/program fits into the grand scheme of things. As Citation Needed points out, it's not necessarily in the interest of less successful schools to warn incoming students that their chances of a job are poor. And, particularly in the realm of undergraduate education, there are certainly there are schools who cynically accept students knowing that their degree is so worthless that the students will almost certainly default on their loans.

What to do

Obviously the real onus is on the student (caveat emptor) to make sure they know what their chances of getting the job they want are prior to matriculating -- and this is true for every degree, not just PhDs. For most schools -- undergraduate and particularly graduate -- you can get data on how graduates fare in the marketplace. This can help determine not only which school to go to but whether it's worth going to school at all (it may not be). But to the extent it is in society's interest that people aren't wasting time and money (often as not, taxpayer money), it is worth considering how, as a society, we can make sure that not only is the information available, but people know that it's available and where to get it.

Do professors teach?

2010-06-23T10:20:00.000-04:00

Luis Von Ahn has an excellent discussion on his blog about the teaching/research balance at major research universities. The comments are worth a read as well, especially this one and Von Ahn's response.

Overnight data on lying and bragging

2010-06-22T14:34:00.002-04:00

Many thanks to all those who responded to my call for data last week. By midnight, I had enough data to be confident of the results, and the results were beautiful. I would have posted about them here on Friday, but in the lead-up to this presentation, I did so much typing I burned out my wrists and have been taking a much-needed computer break.

The study looked at the interpretation of the word some. Under some conditions, people interpret some as meaning some but not all, but other times, it means simply not none. For instance compared John did some of his homework with If you eat some of your green beans, you can have dessert. Changing some to some-but-not-all doesn't change the meaning of the first sentence, but (for most people) changes the interpretation of the second.

This phenomenon, called "scalar implicature" is one of the hottest topics in pragmatics -- a subdivision of linguistic study. The reasons for this are complex -- partly it's because Ira Noveck and his colleagues turned out a series of fascinating studies capturing a lot of people's attention. Partly it's because scalar implicature is a relatively easily-studied test case for several prominent theories. Partly it's other reasons.

Shades of meaning

On most theories, there are a few reasons some might be interpreted as some-but-not-all or not. The usual intuition is that part of why we assume John did some of his homework means some-but-not-all is because if it were true that John did all of his homework, the speaker would have just said so ... unless, of course, the speaker doesn't know if John did all his homework or if the speaker does know but have a good reason to obfuscate.

At least, that's what many theorists assume, but proving it has been hard. Last year, Bonnefon, Feeney & Villejoubert published a nice study showing that people are less likely to interpret some as some-but-not-all in so-called "face-threatening" contexts -- that is, when the speaker is being polite. For instance, suppose you are a poet and you send 10 poems to a friend to read. Then you ask the friend what she thinks, and she says, "Some of the poems need work." In this case, many people suspect that the friend actually means all of the poems need work, but is being polite.

The study

In this quick study, I wanted to replicate and build on Bonnefon et al's work. The experiment was simple. People read short statements and then answered a question about each one. The first two statement/question pairs were catch trials -- trials with simple questions and obvious answers. The small number of participants who got those wrong were excluded (presumably, they misunderstood the instructions or simply weren't paying attention).

The critical trial was the final one. Here's an example:

Sally: 'John daxed some of the blickets.'

'Daxing' is a neutral activity, neither good nor bad.

Based on what Sally said, how likely is it that John daxed ALL the blickets?

As you can see, the sentence contained unknown words ('daxing', 'blickets'), and participants were presented with a partial definition of one of them (that daxing is a neutral activity). The reason to do this was that it allowed us to manipulate the context carefully.

Each participant was in one of six conditions. Either Sally said "John daxed some...," as in the example above, or she said "I daxed some..." Also, "daxing" was described as either a neutral activity, as in the example above, or a negative activity (something to be ashamed of), or a positive activity (something to be proud of).

Results

As shown in the graph, whether daxing was described as positive, negative or neutral affected whether participants thought all the blickets were daxed (e,g, that some meant at least some rather than some-but-not-all) when Sally was talking about her own actions ("I daxed some of the blickets").

This makes sense, if 'daxing' is something to be proud of, then if Sally daxed all of the blickets, she'd say so. Since she didn't, people assume she daxed only some of them (far right blue bar in graph). Whereas if daxing is something to be ashamed of, then even if she daxed all of them, she might prefer to say "I daxed some of the blickets" as a way of obfuscating -- it's technically true, but misleading.

Interestingly, this effect didn't show up if Sally was talking about John daxing blickets. Presumably this is because people think the motivation to brag or lie is less strong when talking about a third person.

Also interestingly, people weren't overall more likely to interpret some as meaning some-but-not-all when the sentence was in the first-person ("I daxed..."), which I had predicted to be the case. As described above, many theories assume that some should only be interpreted as some-but-not-all if we are sure the speaker knows whether or not all holds. We should be more sure when the speaker is talking about her own actions than someone else's. But I didn't find any such affect. This could be because the theory is wrong, because the effect of using first-person vs. third-person is very weak, or because participants were at floor already (most people in all 6 conditions thought it was very unlikely that all the blickets were daxed, which can make it hard detect an effect -- though it didn't prevent us from finding the effect of the meaning of 'daxing').

Afterword

I presented these data at a workshop on scalar implicature that I organized last Thursday. It was just one experiment of several dozen included in that talk, but it was the one that seemed to have generated the most interest. Thanks once again to all those who participated.

--------------
Bonnefon, J., Feeney, A., & Villejoubert, G. (2009). When some is actually all: Scalar inferences in face-threatening contexts Cognition, 112 (2), 249-258 DOI: 10.1016/j.cognition.2009.05.005

Need data by morning

2010-06-16T17:01:00.000-04:00

In preparing a talk for a workshop I've organized tomorrow, I realized there was one simple experiment that would tie several pieces together neatly. Unfortunately, I hadn't run it. But I figured, hey, I can get data quick on Amazon Mechanical Turk.

Turk let me down. I don't know why, but today there aren't a lot of fish biting. So I turn to my usual, pre-Turk subject pool: you. The experiment takes 1-2 minutes -- it's really just 3 questions. As an added inducement to get people to run the experiment now, I'll be posting the results later this week or early next week.