Games with Words

Since one can't be snarky in a response to a review...

2012-05-07T11:47:00.000-04:00

I'll do it here. I am currently revising a paper for resubmission. On the whole, the reviews are fairly reasonable, with the exception of one cranky comment from a reviewer who complains that our literature review is woefully incomplete. This incompleteness seems to be our failure to cite one particular study. The reviewer writes

It is possible that this work is flawed, but it really should be discussed.

It does seem to be a relevant study and we would have cited it, had we known about it. Why didn't we know about it? Because it has never been published. It hasn't even been presented at any of the normal psycholinguistics conferences (though it has appeared at some linguistics conferences). Short of emailing every researcher who might be conducting a study that might be relevant, I'm not sure what this reviewer was expecting of us.

I'd also love to know what the folks who are obsessed with only citing studies published in peer-reviewed journals would say. (It's possible that some of these conferences it has been presented at have pretty thorough review procedures; I wouldn't know.)

The Psychologist on Replication

2012-04-19T11:47:00.000-04:00

The Psychologist solicited opinions on the importance of replication from a number of researchers, including yours truly. See a preview here.

Eadweard J. Muybridge & Google Doodle

2012-04-09T00:57:00.000-04:00

Today's Google Doodle is a fantastic tribute to Muybridge. I haven't found a permalink, but people looking after today can find it archived in a fashion on youtube.

Point-light walkers

2012-03-23T12:37:00.000-04:00

By far the best point-light walker demonstration I've seen is at biomotiolab.ca. I'm classifying this as an illusion (see post label) because, of course, point-light walkers aren't really walking people -- they are just a few white dots moving around the screen. Comparing the male and female versions is particularly fun if you've ever wondered what exactly it is that makes for a stereotypical male or female stride.

It also appears that there is an experiment you can participate in if you want to help with this kind of research.

Fair Use & FedEx

2012-03-21T18:36:00.001-04:00

And now for something completely different:One private citizen's trials and travails trying to convince FedEx to print posters.

I have wanted a map of Hong Kong on my wall for some time. The Survey & Mapping office of the Hong Kong government helpfully provides some free maps for public use on their website. You will notice how the website helpfully includes a "free maps"logo, along with a copyright notice forbidding only commercial use of the map. Presumably they thought this was a good way of providing some publicity for the Special Administrative District.

They did not take into account FedEx Office. I put this map on a USB stick and went to the FedEx Office at Government Center to have it printed. The manager there refused to print it as I didn't have proof of copyright ownership. I showed him the website (particularly where it says "free maps"). He said the fact that the map is free for public use was irrelevant; he needed a signed document from the copyright owner (the government of Hong Kong) stating that I, personally, had the right to print off the map.

His explanation for his refusal was simple: "I can't get between me and the copyright holder." I pointed out that he was getting between me -- who wants to print the map -- and the copyright owner -- who also wants me to print the map. He repeated that even so, he "can't get between me and the copyright holder." This was just repetition, so I pointed out again that the map is clearly labeled for public use. He said that was just "he said/she said" business; what he needed was a signed document.

I'm curious what he would do with a signed document in Chinese, and whether he would require a notarized translation. I realized as I was leaving that at the beginning when the manager was trying to establish whether I had the right to print the map, he had asked me if I was a member of the organization that made the map -- that is the Hong Kong government. I'm curious what would have happened if I had said yes.

The "copyright waiver"

This is not the 1st time that I've had a run-in with the copyright police at FedEx. Last year the Palo Alto FedEx refused to print a poster that I was supposed to present at a conference at Stanford. I study story comprehension in small children, and a common practice is to use stories about familiar characters. In this case, I had stories about Dora the Explorer and a few other cartoon characters. Because my poster showed an example of one of the pictures that we had drawn to go with the stories, FedEx initially refused to print the poster, saying that it violated copyright.

After a long discussion about fair use and noncommercial uses, one of the employees remembered that they have a “copyright release” form that they can use in these circumstances. Unfortunately, they couldn't find any blank copies. One enterprising employee simply wrote the words “copyright release” on a piece of paper and asked me to sign that piece of paper.

I wasn't sure about the wisdom of signing and essentially blank piece of paper (you can see a photo of it on the right), so they came up with another plan, which was to whiteout all the writing on a previously filled out form, which they then copied (not waiting for the whiteout to dry and getting white out all over their copier in the process) and which I signed. Then they printed my poster and I went on to have an otherwise successful conference.

Copyright and FedEx

Clearly somebody has instilled the Fear of the Lord into the employees at FedEx with regards to copyright infringement. FedEx is understandably concerned about their liability, since unlike me, they have actual assets. I also realize that FedEx may not have the resources to have somebody on staff who has been adequately trained to deal with copyright issues ... but in that case, it suggests that maybe they do not have the resources to run a print shop. After all, it is not like they are not making determinations now. They are just doing it randomly and incorrectly.

Are you a Red Sox or Yankees fan?

2012-03-21T18:00:00.000-04:00

If so, a colleague has a short survey for you. It seems she is trying to get together as much data as possible for a talk next week. Apparently there is also an opportunity to win a $50 gift card, though my motivation for participating was in order to help out with some interesting research.

Zeno

2012-03-15T12:59:00.000-04:00

Many people are familiar with Zeno's paradox, though probably not in the form presented by XKCD:

(If you aren't familiar with it or need a refresher, just follow the link above.)

Perhaps this is widely known, but I only recently discovered what the point of Zeno's paradox was: he was trying to prove that motion is impossible. Nothing ever moves and nothing ever changes.

This probably sounds absurd, but it was the basis of a philosophical school of which Zeno was part. Zeno created a number of paradoxes, all of which were meant to demonstrate that if the idea that nothing ever moves or changes is absurd, well then it is no more absurd than the idea that things do move and do change. If motion was possible, you would end up, for instance, with Zeno's never-ending race.

This is just another demonstration that many famous philosophical ideas are often remembered now for reasons very different from the reason for which they were first put forth.

(Insight gleaned from Anthony Gottlieb's excellent The Dream of Reason).

Color illusion -- too cool to believe

2012-03-12T09:00:00.000-04:00

By far the most striking visual illusion I've ever seen. A little bit of color after-effect turns a black-and-white photograph into a vivid color photograph. You may have to do it a few times to convince yourself it is real.

Results: Replication in Psychology

2012-03-07T13:15:00.000-05:00

My paper with Adena Schachner on replication in psychology is now published. The paper contains 3 main sections: a reasonably thorough literature review on replication rates in psychology, a proposal as to how to improve replication rates (primarily through tracking replication rates), and the results of a survey of psychologists on replication practices (many thanks to all who participated). The results of the survey was that while not nearly enough replications are attempted, there are actually more being attempted than we had guessed (or than many of our colleagues that we discussed this project with had guessed).
This paper is part of a larger collection of papers on reimagining the publication and review process, and is more of those papers are printed, I plan to discuss at least some of them.

Pilot data

2012-03-04T10:33:00.004-05:00

I am back from a long semi-silence.I have been trying to finish up a number of projects, which gives me less time to write. Speaking of…

One of the focuses of my work is figuring out how children learn the meaning of verbs. This is made more complicated by the fact that we don't actually have completely solid and uncontroversial definitions of verbs. If we don't know what verbs mean, how can we tell when a child has successfully learned them?

I am working on a large scale project to get better definitions of verbs. We are developing many different tasks, each of which gets at one specific aspect of meaning that is thought to be important for at least some verbs. The traditional method would be to have skilled linguists go through verbs one at a time and consult their own intuitions, and in fact a lot of very good work has been done this way (e.g., Jackendoff's Semantic Structures, among many others). However, there are certain advantages to having this work done by a larger number of people who are naïve to linguistic theory, not the least of which is that there are a very large number of verbs, and one person can't get through them all in any reasonable speed. The one disadvantage of working with naïve participants is that they do not understand linguistic

terminology, so you have to find some other way to explain the task.

I have been developing some such tasks, and I could really use some pilot data to see how well they are working. If you have a little time to spare, I would really appreciate the help. There are 3 in particular I am currently working on:

Like Water off a Duck's Back

Person or Thing of the Year

Simon Says Freeze

There is a comments box at the end where you can leave any feedback and mention anything you noticed or which you found confusing. I do need data on all three, so please don't everyone just do the first one.

Fair warning: These tasks take a bit longer than the ones on my website. My guess is that they will take 20-30 minutes each, but that is a wild guess. If somebody does one and wants to leave a comment about how long it took, that would be helpful for me and also for others who might want to do it.

Many thanks.

Someday we will hopefully have good dictation software. For now, there is Dragon Dictate

2012-02-17T10:01:00.000-05:00

Mary Grover at Salon has distilled the essence of using Dragon Dictate into a brief post. I couldn't possibly do better -- or even as well -- so I refer you to it.

I was assured by several people that if I continued to use DragonDictate and use the vocabulary training feature, eventually the software would learn to do a better job. I made sure to diligently train Dragon on everything that I wrote. Unfortunately, it appears that training function itself is broken. I was suspicious that even when I used very unusual words, it always insisted that it already knew all those words. So I tested it by training on a set of made-up words. When Dragon happily announced that it already knew all of those words, too, I wrote an e-mail to technical support.
I wasn't very optimistic about hearing from technical support, since they had not replied to my previous e-mails when I have had other questions. This time, they replied promptly to tell me the technical support had expired, but that I could pay for extended technical support. Presumably, if I were to pay, they would go back to not answering e-mails.
I spend a lot of time at the computer, and I bought Dragon so that I wouldn't have to spend all of that time typing. I pace when I think, but pacing and typing don't mix. I thought Dragon would give me more flexibility. As of yet, this remains a distant dream.

More on DragonDictate

2011-10-30T13:33:00.000-04:00

DragonDictate continues to do a decent job of writing my email, so long as I don't talk about work. For writing papers, etc., it continues to be of limited use.

I was just dictating notes on how children learn to count. In two back-to-back sentences, I mentioned "subset-knowers". The first time, this was transcribed as "subset-members", and the second time, it was "sunset-whores".

DragonDictate

2011-10-05T20:32:00.000-04:00

I have been doing a great deal of writing lately, though obviously not here. I thought that perhaps at some point in graduate school, I should try getting some of the projects I have done published, and I thought that time was now. Since this requires writing them up, I have been writing. I have gotten a lot of writing done, but I noticed that this came with an increased number of hours spent sitting at my computer. Knowing enough friends who have suffered from repetitive stress injuries, I decided I should take a proactive approach to ergonomics.

One outcome of this process is was that I purchased voice-recognition software, namely Dragon Dictate. This actually complements my preference to pace while I think. My writing style involves a lot of thinking, punctuated by occasional bursts of typing. So being able to write as I pace seemed like a good idea.

I cannot say that this experiment has been an overwhelming success. Based on what I have learned from the documentation, Dragon Dictate seems to place a great deal of faith in transitional probabilities. That is, the hypotheses it makes about what you are saying are based not only on the sounds that you make but based on what words typically come after one another.

Of course, what words typically follow one another depends a great deal on what you are talking about. I suspect that Dragon Dictate was not trained on a corpus involving a great deal of psycholinguistics papers, but in fact it is psycholinguistics papers that I am writing. Dragon Dictate makes a number of very systematic and very annoying errors. For instance, it is absolutely convinced that, no matter how carefully I say the word “verb”, I could not possibly have meant to say that word, and probably meant "four herbs" or some such. In the general case, this is probably the right conclusion. The word “verb” is so rarely spoken, that it is probably a good bet that it even if you think you heard the word “verb”, what was actually spoken was probably something else. However, since almost all my papers are about verbs, I use that word so often that probably the right hypothesis is that no matter what you think you heard, the word I actually uttered was “verb”.

Needless to say, it doesn't do very well with technical terms from semantic and syntactic theory, either.

The upshot is that I spend so much time correcting DragonDictate's mistakes, that it is not clear that I wouldn't be better off just typing the document begin with (you can correct using voice commands, but it is so cumbersome that I usually type instead). Dragon Dictate has a function where you can feed it various documents. The documentation appears to imply that it can learn the relevant word frequencies and transitional probabilities from these documents. I have been feeding at papers I have written, in the hopes that this will help out. So far there has been limited improvement, but I am not sure just how large a corpus of needs. I will keep you updated.

(Written using DragonDictate plus hand correction.)

Ethics

2011-09-19T11:14:00.000-04:00

I'm doing my periodic re-certification on research ethics. One of the questions on one of the quizzes is as follows:

TRUE/FALSE: A good alternative to the current peer review process would be web logs (BLOGS) where postings where [sic] papers would be posted and reviewed by those who have an interest in the work.

Apparently, the correct answer is "false". Presumably because we have much better technology for this kind of thing, rather than using a simple blog?

Nature, Nurture, and Bayes

2011-07-30T15:59:00.000-04:00

I generally have very little good to say about the grant application process, but it does force me to catch up on my reading. I just finished several papers by Amy Perfors, who I think does some of the more interesting computational models of language out there.*

A strange sociological fact about language research is that people generally come in two camps: a) those who don't (really) believe language is properly characterized by hierarchical phrase structure and also don't believe in much innate structure but do believe in powerful innate learning mechanisms, and b) those who believe language is properly characterized by *innate* hierarchical phrase structure and who don't put much emphasis on learning mechanisms. But there's no logically necessary connection between being a Nativist and believing in hierarchical phrase structure or being an Empiricist and believing in relatively simple syntactic forms. In the last few years, Perfors has been staking out some of that (largely) unclaimed territory where hierarchical phrase structure and Empiricism meet.

In "The learnability of abstract syntactic principles," she and her colleagues consider the claim by (some) Nativists that children must have an innate expectation that language be something like a hierarchical context-free grammar because there isn't enough data in the input to rule out alternative grammars. (Empiricists often buck the whole question by saying language is no such thing.) Perfors et al. show that, in fact, with some relatively simple assumptions and a powerful (Bayesian) learning device, the learner would conclude that the most likely representation of English is a hierarchical context-free grammar, based on relatively little input (reproducing what happened in linguistics, where linguists came to the same conclusion). You do have to assume that children have the innate capacity to represent such grammars, but you don't need to assume that they prefer such grammars.

"Joint acquisition of word order and word reference" presents some interesting data bearing on a number of questions, but following the theme above, she notes that her model does not require very much data to conclude that the typical word-order in English is subject-verb-object. She and her colleagues note: "The fact that word order can be acquired quickly from so [little data] despite the lack of bias [for a particular word order] may suggest no need to hypothesize that children are born with strong innate constraints on word ordering to explain their rapid acquisition."

I'm sympathetic to all these points, and I think they bring an important perspective to the question of language learning (one that is not, I should say, unique to Perfors, but certainly a minority perspective). What I can't help wondering is this: she (and others) show that you could learn the structure of language based on the input without (certain) innate assumptions that the input will be of a particular sort. Fine. But why is the input of that particular sort across (most? all?) languages? One thing the Nativist positions Perfors argues against have going for them is that they give a (more or less) principled explanation. Empiricists (typically) do not. (I am aware that some try to give explanations in terms of optimal information structure. What I have seen of this work has not struck me as overwhelmingly convincing, but I admit I haven't read enough of it and that I am willing to be convinced, though my prior on this line of argumentation is fairly low).

*My quasi-journalistic training always makes me want to disclose when I know personally the people I am writing about. But psycholinguistics is a small world. It would be safe for the reader to assume that I know *all* of the people I write about to one degree or another.

*********
Perfors A, Tenenbaum JB, & Regier T (2010). The learnability of abstract syntactic principles. Cognition PMID: 21186021

Maurits, L., Perfors, A., & Navarro, D. (2009). Joint acquisition of word order and word reference Proceedings o the 31st Annual Conference of the Cognitive Science Society, 1728-1733

Statistics for Idiots

2011-06-22T09:31:00.000-04:00

Republicans in the House are proposing to cut funding for food safety programs, despite a rise in food-born illness. Congressman Jack Kingston explains, that the nation's food supply is "99.99 percent safe". Politifact says, "That sounds great, but is it true?"

Actually, it doesn't sound that good to me. Suppose Kingston means that you only have a 0.01% chance of getting ill any particular time you eat (which seems to be the case). And let's say people eat 3 times a day. That gives you a 10.4% chance of getting sick any given year. I'd rather not get sick at all, particularly when many of the illnesses are easily preventable.

NSF fellows can teach again

2011-05-20T18:11:00.000-04:00

I reported last month that NSF was no longer allowing its graduate fellows to teach. According to an email I received earlier today, they are reconsidering the issue:

Each Fellow is expected to devote full time to advanced scientific study or work during tenure. However, because it is generally accepted that teaching or similar activity constitutes a valuable part of the education and training of many graduate students, a Fellow may undertake a reasonable amount of such activities, without NSF approval. It is expected that furtherance of the Fellow's educational objectives and the gain of substantive teaching or other experience, not service to the institution as such, will govern these activities. Compensation for such activities is permitted based on the affiliated institution’s policies and the general employment policies outlined in this document.

New editor at Cognition (eventually)

2011-05-15T09:30:00.000-04:00

There are no doubt many psychologists who don't count Cognition as their favorite journal. I just don't happen to know very many of them. Whenever the topic of favorite journal comes up, Cognition it is. One would think that would argue in favor of continuity; whatever they're doing is working.

That's not apparently how the for-profit publishers of Cognition (Elsevier) feel, as they've decided to find a new editor, apparently without consulting anyone in the field about it. I hope they know what they are doing.

Above average!

2011-05-09T18:40:00.000-04:00

It's often repeated that the median study is cited less 0 times. I haven't been able to find a citation for that, but if it is true, all my papers are now above median. My birth order paper has now been cited. Actually, it was cited last year, but I didn't notice for a while. Granted, it was cited in a paper appearing in Journal of Language, Technology & Entrepreneurship in Africa, which is apparently not a high-impact journal, but a citation is a citation.

For rather boring reasons not related to the data or the review process itself, the birth order paper appeared in a journal that is not widely read by researchers, which probably has reduced its visibility. Certainly, plenty has been published on the topic in the last few years. This is a lesson for the future: it really does matter which journal you publish in, despite the wide-spread use of search engines.

For more on my birth order research, click here.

Survey on Replication

2011-04-23T12:52:00.000-04:00

Are you a researcher working in psychology or related domains (neuroscience, linguistics, etc.)? A colleague and I are conducting a survey on replication in these fields, for inclusion in an upcoming special issue of Frontiers in Computational Neuroscience. You can fill out the survey here.

Photo credit here.

The pace of review

2011-04-23T12:45:00.000-04:00

One of my manuscripts will shortly enter its 7th month of being under review. Apparently one of the three reviewers keeps promising to send in a review but never does. Now the 4+ months a different manuscript languished under review seems speedy.

Ray Kurzweil is convinced that the pace of science is increasing exponentially and will continue to do so. I think he's neglected one rate-limiting step in the process.

What the Best College Teachers Do: A Review of a Vexing Book

2011-04-19T23:31:00.000-04:00

What the Best College Teachers Do is not a bad book. It is engaging and reasonably well-written. The topic is both evergreen and timely, and certainly of interest to college teachers at the very least (as well as to people who rate college quality and to people who use those ratings to decide where to go to school). My issue with this book is that it is incapable of answering the question it sets out for itself.

A problem of comparison

The book is based primarily on extensive research by the author, Ken Bain, and his colleagues. The appendix spells out in detail how they identified good college teachers (a combination of student evaluations, examples of student work, department examinations, etc.) and how they collected information about those gifted individuals (interviews, taped class sessions, course materials, etc.). They analyzed these data to determine what these best college teachers did.

Even assuming that (a) their methods successfully identified superior teachers, and (b) they collected the right information about those teachers' practices, this is only half of a study. Without even looking at their data, I can easily rattle off some things all these teachers had in common:

1. They were all human beings.
2. They were all taller than 17 inches.
3. They all spoke English, at least to some degree (the study was conducted in the USA).
4. Most were either male or female.

Commonalities are not limited to attributes of the teachers, but also to what they do in the classroom:

5. Most showed up to at least half of the class periods for a given course.

6. None of them habitually sat, silent and unmoving, at the front of the classroom for the duration of class.
7. They did not assign arbitrary grades to their students (e.g., by rolling dice).
8. Very few spoke entirely in blank verse.

While these statements are almost certainly true of good college teachers, they do not distinguish the good teachers from the bad ones. Since Bain and colleagues did not include a comparison group of bad teachers, we cannot know if their findings distinguish the good teachers from the bad ones.

Science -- like teaching -- requires training

A good test of teaching ability should pick out all the good teachers. It should also pick out only the good teachers. (A somewhat different cut of the issues is to consider test reliability and test validity). What the Best College Teachers Do focuses entirely on the first issue. As my reductio ad absurdum above shows, having only half of a good test is not a test that is 50% right; it's a useless test.

It's unfortunate that Bain and his colleagues failed in this basic and fundamental aspect of scientific inquiry. Although Bain is now the director of the Center for Teaching Excellence at New York University, he was trained as a historian. This comes out in the discussion of the study methods: "Like any good historians who might employ oral history research techniques, we subsequently sought corroborating evidence, usually in the form of something on paper..." (p. 187).

I would hope that any good historian doing comparative work would know to include a comparison group, but designing a scientific study of human behavior is hard. Even psychologists screw it up. And that's the focus of our training, whereas historians are mostly learning things other than experimental design (I assume).

Circular Definitions

Of course, failing to include a control group is not the only way to ruin a study.You can also make it circular.

Chapter 3 focuses on how excellent teachers prepare for their courses:

At the core of most professors' ideas about teaching is a focus on what the teacher does rather than on what the students are supposed to learn. In that standard conception, teaching is something that instructors do to students, usually by delivering truths about the discipline. It is what some writers call a 'transmission model.' ...

In contrast, the best educators thought of teaching as anything they might do to help and encourage students to learn. Teaching is engaging students, engineering an environment in which they learn.

Here is what the appendix says about how the teachers were chosen for inclusion in the study:

All candidates entered the study on probation until we had sufficient evidence that their approaches fostered remarkable learning. Ultimately, the judgment to include someone in the study was based on careful consideration of his or her learning objectives, success in helping students achieve those objectives, and ability to stimulate students to have highly positive attitudes toward their studies.

It seems that perhaps teachers were included as being "excellent teachers" if they focused on student learning and on motivating students. The researchers then "found" that excellent teachers focus on student learning and on motivating students.

Vagueness and Ambiguity

Or maybe not. I'm still not entirely sure what it means to -- in the first quote -- focus on "what the teacher does" than on "what the students are supposed to learn." For instance, Bain poses the following thought problem on page 52:

"How will I help students who have difficulty understanding the questions and using evidence and reason to answer them."

Is that focusing on what the teacher does or focusing on what the students are supposed to learn? How can we tell? By what metric?

My confusion here may merely mark me was one of those people expecting "a simple list of do's and don'ts" who are "greatly disappointed." Bain adds (p. 15), "The ideas here require careful and sophisticated thinking, deep professional learning, and often fundamental conceptual shifts." That's fine. But if there is no metric I can use to find out whether I'm following these best practices or not, what good does this book do me?

(Also, without knowing what exactly Bain means by these vague statements, there is no way to ensure that his study wasn't circular, as described in the previous section. I gave only one example, but the general problem is clear: Bain defined great teachers by one set of criteria and then analyzed their behavior in order to extract a second set of criteria. If both sets of criteria are loosely and vaguely defined, there's no way even in principle to know whether he isn't just measuring the same thing both times.)

Credible Reviews

So if we don't trust Bain's study, is there anything else in the book worth reading? Maybe. What the Best College Teachers Do is not myopically focused on Bain's own research. He reviews the literature, citing the conclusions from other studies of teaching quality, broadening the scope of the framework outlined in the book. However, this raises its own problem.

In writing a review, the reviewer is supposed to survey the literature, find all the relevant research, determine what the best research is, and then synthesize everything into a coherent whole (or at least, into something as coherent as the current state of the field allows). The reviewer generally does not describe the studies in sufficient detail to allow the reader to evaluate them directly; only a brief overview is provided, with a focus on the conclusions.

If you trust the reviewer, this is fine. That's why reviews from the most respected researchers in the field are typically highly valued, so much so that publishers and editors often solicit reviews from these researchers. Obviously, a review of the latest research on underwater basket weaving by a fifth-grader would not be so highly prized, because (a) we don't believe the fifth-grader did a particularly thorough review, and (b) we don't trust the fifth-grader's ability to sort the wheat from the chaff -- that is, identify which studies are flawed and which are to be believed.

Bain is clearly very smart. He has clearly read a lot. But I do not trust his ability to read scientific literature critically. The only evidence I have of his abilities is in the design of his own study, which is deeply flawed, as described above. If he can't design a study, why should I trust his analysis of other people's studies?

Building a better mousetrap

Criticizing a study is easy, but it's not much of a critique if you can't identify what a better study would look like. Clearly from my discussion above, I would want (a) clear criteria for defining good teaching, (b) clearly-defined measures of teacher behavior, and (c) a group of good teachers and a group of bad teachers for comparison, and probably a group of average teachers as well (otherwise, any differences between good and bad teachers could be driven by bad habits of the bad teachers rather than good habits of the good teachers).

After a set of behaviors that are typical of good teachers -- and which are less frequent or absent in average or bad teachers -- has been identified, one would then identify a new group of good, average, and bad teachers and replicate the results. (The risk is otherwise is one of over-fitting the data: the differences you found between good teachers and the rest were just the result of random chance. This actually happens quite a lot more than many people realize.)

At the end of this process, we should have a set of behaviors that really are particular to the best teachers, assuming that the criteria we used to define good teachers are valid (not an assumption to be taken lightly).

Becoming a good teacher

Whether or not this information would be of any use to those aspiring to be good teachers is unclear. To find out that, we'd actually need to do a controlled study, assigning one set of teachers to emulate this behavior and another set to emulate behavior typical of average teachers. Ideally, we'd find that the first group ended up teaching better. I'm unsure whether that's particularly likely to happen, for a number of reasons.

First, consider Bain's summary of the habits of the best teachers (summarizing, with some direct quotations, from pps. 15-20):

1. Outstanding teachers know their subjects extremely well.
2. Exceptional teachers treat their lectures, discussion sections, problem-based sessions, and other elements of teaching as serious intellectual endeavors as intellectually demanding and important as their research and scholarship.
3. They avoid objectives that are arbitrarily tied to the course and favor those that embody the kind of thinking and acting expected for life.
4. The best teachers try to create an environment in which people learn by confronting intriguing, beautiful, or important problems, authentic tasks that will challenge them to grapple with ideas, rethink their assumptions, and examine their mental models of reality.
5. Highly effective teachers tend to reflect a strong trust in students.
6. They have some systematic program to assess their own efforts and to make appropriate changes.

Much of this list looks like a combination of intelligence and discipline. That is clearly true for #1, and probably true for #2 and #3. To the extent that #4 is hard to do, it probably takes intelligence. And #6 is just a good idea, more likely to occur to smart people and only pulled off by disciplined people. I'm not sure what #5 really means.

If the key to being a good teacher is to be smart and disciplined, this news will be of little help to teachers who are neither (though it may be helpful to people who are trying to select good teachers). In other words, even if we determine what makes a good teacher, than doesn't mean we can make good teachers.

The best teachers

Of course, even if the strategies that good teachers use are ones you can use yourself, that doesn't mean you can use them correctly.

There is an old parable about two young women. One was exceptionally beautiful. She used to sit at her window and gaze out over the field, looking forlorn and sighing with melancholy. Villagers passing by would stop and stare, struck by her heavenly beauty. One such villager was another young woman, who was the opposite of beautiful. Nonetheless, on seeing this example, she went home, sat at her own window, gazed out over the field and sighed. Someone walked by, saw her, and promptly vomited.

Objectification of female beauty and strange fetishization of melancholy aside, the point of this parable is that just because something works for someone else doesn't mean it'll work for you. When I think about the very best teachers I've known, one thing that stands out is how idiosyncratic their methods and abilities have been. One is a high-energy lecturer who runs and jumps during his lectures (yes, math lectures), who is somehow able to turn linear algebra into a discussion class. Another, in contrast, faded into the background. He rarely lectured, preferring to have students work (in groups or individually) on carefully-crafted questions. A third is a gifted lecturer and the master of the anecdote. While others use funny anecdotes merely to keep a lecture lively, when he uses an anecdote, it is because it illustrates the point at hand better than anything else. Over at the law school, there are a number of revered professors famous for their tendency to humiliate students. This humiliation serves a purpose: to show the students how much they have to learn. The students, rather than being alienated, strive to win their professors' approval.

These methods work for each, but I can't imagine them swapping styles round-robin. Their teaching styles are outgrowths of their personalities. Many are high-risk strategies, which if they fail, fail disastrously (don't humiliate your students unless you have the right kind of charisma first).

Are there strategies that will work for everyone? Is there a way of determining which strategies will work for you, with your unique set of strengths and weaknesses? I'd love to find out. But it won't be by reading What the Best College Teachers Do.

The missing linking hypothesis

2011-04-15T11:51:00.000-04:00

Science just published a paper on language evolution to much fanfare. The paper, by Quentin Atkinson, presents analysis suggesting that language was "invented" just one time in Africa. That language first appeared in Africa would be of little surprise, since that's where we evolved. That there was only one point at which it evolved is somewhat more controversial, and also trivially false if one includes sign languages, at least some of which have appeared de novo in modern times (and one could make a case for including spoken creoles in the list of de novo languages).

What still boggles my mind is the analysis that supports these conclusions. In many ways, it seems brilliant -- but I can't escape the feeling that there is something amiss with the argument. The problem, as we'll see, is a series of missing linking hypotheses.

The Data

The primary finding is that the further you go from Africa (very roughly following plausible migration paths), the fewer phonemes the local language has. Hawai'ian -- the language spoken farthest from our African point of origin -- has only 13 phonemes. Some languages in Africa have more than 100.

To support the claim that this demonstrates that language evolved in Africa, one must add some additional data and hypotheses. One datum is that languages spoken by more people have more phonemes. Atkinson argues that whenever a new population migrated away from the parent population, it would necessarily be a smaller group ... and thus their language would have fewer phonemes than the parent group. Keep this up and over time, you end up with just a few phonemes left.

Population genetics

This argument seems to derive a lot of its plausibility from well-known phenomena in population genetics. Whenever a new population branches off (migrates away), it will almost by definition have less genetic diversity than the mother population. And in fact Africa has greater genetic diversity than other continents.

Atkinson tries to apply the same reasoning to phonemes:

Where individuals copy phoneme distinctions made by the most proficient speakers (with some loss), small population size will reduce phoneme diversity. De Boer models the evolution of vowel inventories using a different approach, in which individuals copy any members of their group with some error, and finds the same population size effect.

I see the logic, but then phonemes aren't genes. When ten people leave home to start a new village, they can only take ten sets of genes with them, and even some of that diversity may be lost because of vagaries of reproduction. Those alleles, once gone, are not easily reconstructed.

As far as I can tell, to apply the same logic to phonemes we have to assume a fair percentage of children fail to learn all the phonemic contrasts in their native language. For some reason, this does not prevent them from communicating successfully. In a large population, the fact that many people lack this or that phonemic contrast doesn't matter, as on average, most people know any given phonemic contrast, and thus it is transmitted across the generations. When a small group leaves home, however, it's quite possible that by accident there will be a phonemic contrast that few (or none) of them use. The next generation is then unlikely to use that contrast.

This may be true, but I don't find its plausibility so overwhelming that I'm willing to accept it on face value. I'd actually like to see data showing that many or most speakers of a given language do not use all the phonemic contrasts (beyond the fact that of course some dialects are missing certain phonemes, as in the fact that Californians do not distinguish between cot and caught; dialectical variation probably cannot support Atkinson's argument, but I leave the proof to the reader ... or to the comment section).

Phonemes and Population Size

Atkinson reports being inspired by the relatively recent finding that languages spoken by more people have more phonemes. Interestingly, the authors of that paper note that "we do not have well-developed theoretical arguments to offer about why this should be." It seems to me that Atkinson's analyses depend crucially on the answer to this puzzle, though as I mentioned at the outset, I haven't been able to quite work out all the details yet.

Atkinson's analysis crucially depends on (among things) the following supposition: the current population size of any language community is roughly predicted by the number of branching points (migrations) since the original language (which arose somewhere on the order of 50,000 and 100,000 years ago). I'm still on the fence as to whether or not this is a preposterous claim or very reasonable.

It is certainly very easy to construct scenarios on which this supposition would be false. Civilizations expand and contract rapidly (consider that English was confined to only one part of Great Britain half a millennium ago, or that Celtic languages were spoken across Europe only 2,000 years ago). Relative population size today seems to be driven more by poverty, access to birth control and education, etc., than anything else. Atkinson only needs there to be a mid-sized correlation, but 50,000 years is a very, very long time.

Atkinson also needs it to be the case that the further from Africa a language is spoken, the more branching points there have been. The problem we have is that there is a lot of migration within already-settled areas (Indo-European expansion, Mandarin expansion, Bantu expansion, etc.). So we need it to be the case that most of the branching of language groups happened going into new, unsettled areas, and relatively little of it is a result of invading already-populated areas. That may be true, but consider that all of Africa, Europe, Asia and the Americas were settled by 10,000 years ago, which leaves a lot of time for language communities to move around.

Conclusion

Atkinson put together a very interesting dataset that needs to be explained. His explanation may well be the right one. However, his explanation requires making a number of conjectures for which he offers little support. They may all be true, but this is a dangerous way to make theories. It's a little like playing Six Degrees to Kevin Bacon where you are allowed to conjecture the existence of movies and co-stars. It should be obvious that with those rules, you can connect Kevin Bacon to anyone, including yourself.

Arcadia

2011-04-14T21:22:00.000-04:00

The super-lame New Yorker review of the recent Broadway revival of Stoppard's "Arcadia" moved me to do a rare thing: write a letter to the editor. They didn't publish it, despite the fact -- and I think I'm being objective here -- my letter was considerably better than the review. Reviews are no longer free on the New Yorker website (you can see a synopsis here), but I think my letter covers the main points. Here it is below:

Hilton Als ("Brainstorm", Mar 28) writes about the recent revival of "Arcadia" that Stoppard's "aim is not to show us people but to talk about ideas." Elsewhere, Als calls the show unmoving and writes that Stoppard does better with tragicomedies.

"Arcadia" is not a show about ideas. It is about the relationship people have with ideas, particularly their discovery. Anyone who has spent any amount of time around academics would instantly recognize the characters as people, lovingly and realistically depicted. (Als singles out Billy Crudup's "amped-up characterization of the British historian Bernard Nightengale" as particularly mysterious. As Ben Brantley wrote in the New York Times review, "If you've spent any time on a college campus of late, you've met this [man].")

As an academic, the production was for me a mirror on my own life and the people around me. Not everyone will have that experience. The beauty of theater (and literature) is that it gives us peek into the inner lives of folk very different from ourselves. It is a shame Als was unable to take advantage of this opportunity.

Where the play focuses most closely on ideas is the theme of an idea (Thomasina's) stillborn before its time. If one feels no pathos for an idea that came too soon, translate "idea" into "art" and "scientist" into "artist" and consider the tragedies of artists unappreciated in their time and quickly forgotten. Even a theater critic can find the tragedy in that.

Graduate School Rankings

2011-04-06T09:00:00.000-04:00

There have been a number of interesting posts in the last few days about getting tenure (1, 2, 3). One thing that popped out at me was the use of the National Research Council graduate school rankings in this post. I am surprised that these continue to be cited, due to the deep flaws in the numbers. Notice I said "numbers", not "methodology". I actually kind of link their methodology. Unfortunately, the raw numbers that they use to determine rankings are so error-ridden as to make the rankings useless.

For those who didn't see my original posts on the subject, cataloging the errors, see here and here.